Justin, > For bulk inserts we typically insert directly into the DB too for performance > reasons. As Jeff says, the two main things to be aware of are setting the > label and setting the friendly URL (assuming that content has a friendly > URL). If you have any other calculated/derived properties then you could run > the beforeSave and setDate as a post import process, but it depends on how > complex your custom types are as to what business logic you need to run.
You're making it more complicated than it needs to be. There are more efficient (and cleaner) ways to do it. I have scripts that import thousands (and sometimes millions) of records. You can do it all with createData() as long as you properly batch it. See my answer to Scott below for more details. --- Scott, > So you do bursts of imports, on a scheduled task or something? Whether it's a scheduled task or not is irrelevant (unless you plan to use onComplete in CF10's schedule - which is just awesome). Whatever process you're using now to import the data, when your cfm file is triggered you import, say, the first 100 items - in batches of 100 for this example (*see note below). Whatever the last item was, set that record's unique identifier to a temp variable (or whatever value you want to use to identify the last record). Then call the same import file at the bottom using cflocation and give it the variable for the unique ID. Using that ID, tell your script to start from the next record. Each time the file finishes (and before it calls itself again) the process is flushed from memory (which is the whole point of batching). *note: The batch size depends on how much RAM is available to the JVM. Some clients I have can do batches 50k and higher batches, but it really doesn't matter if you use low numbers. Even it you set it to a low number, it will still import very quickly using this process. The only difference is that you might have to run a query each time at the top of the file (I suggest running a manual query and limiting it to return only 100 records - or whatever your batch size is). There are a bunch of other things I'd do of course, like set a tracking variable that allows me to cancel a long-running import with a click from within the webtop, setting <cfsetting/> at the top of the file with a decent length requestTimeout, using a condition around the cflocation to know if the import process is complete and to stop (don't want a non-ending loop), etc. But you get the idea. I have one client that actually does import millions of records at a time using this process and it is quite fast and efficient (well, it takes many hours, but still faster than other solutions I've done in the past). And there is no need to run all your methods manually and then update each record a second time using setData(). *Note:* For that particular client I find it faster to import the data with no solr indexing. Once complete I index the data in batches using the same batch process. Let me know how it goes for you. -- Jeff Coughlin On Jan 13, 2014, at 6:19 PM, Justin Carter <[email protected]> wrote: > For bulk inserts we typically insert directly into the DB too for performance > reasons. As Jeff says, the two main things to be aware of are setting the > label and setting the friendly URL (assuming that content has a friendly > URL). If you have any other calculated/derived properties then you could run > the beforeSave and setDate as a post import process, but it depends on how > complex your custom types are as to what business logic you need to run. > > Glad you got it sorted though :) > > cheers, > Justin > > -- > Justin Carter > http://www.madfellas.com/blog > http://twitter.com/justincarter > > > On Tue, Jan 14, 2014 at 10:02 AM, Scott Mebberson <[email protected]> > wrote: > Hey guys, > > Yeah. We don't always import using createData or setData. When doing it on > 1000s of rows (daily) it's WAYYYYY to slow. > > On one instance, we wrote a bunch of import scripts based largely in bulk > inserts within MYSQL (CSV straight to MYSQL) and then filled in the gaps that > the framework usually does via createData and setData. > > Lots of testing to get that right though! > > My current instance is slightly different though. But it's all working sweet > now, thanks. > > cheers, > Scott. > > On 14/01/2014, at 9:28 AM, Jeff Coughlin <[email protected]> wrote: > >>> Yeah, the label property will only get set by the framework if setData() is >>> called >> >> or when using createData() - which is what you'd normally use when manually >> importing data into FarCry. >> >> ...Unless you are setting the label in beforeSave() in which case manually >> calling createData() does not call beforeSave(). This is a known bug that's >> just been around for years (can't find the bug number). We just always make >> sure to call beforeSave() manually before using createData(). >> >> And always make sure to call application.fc.factory.farFU.setSystemFU() to >> create the friendly url as well after using createData(), as createData() >> doesn't call that either if also using displaySystemFU.cfm (bug number >> FC-2268). So just to be safe, I always call the aforementioned method for >> FU's after calling createData(). >> >> Good luck. >> >> -- >> Jeff Coughlin >> >> On Jan 13, 2014, at 5:36 PM, Justin Carter <[email protected]> wrote: >> >>> Yeah, the label property will only get set by the framework if setData() is >>> called, so if it was a direct SQL insert then it will bypass the framework >>> :) >>> >>> cheers, >>> Justin >>> >>> -- >>> Justin Carter >>> http://www.madfellas.com/blog >>> http://twitter.com/justincarter >>> >>> >>> On Tue, Jan 14, 2014 at 9:31 AM, Scott Mebberson <[email protected]> >>> wrote: >>> Hey Jeff, >>> >>> Of course. I'll give that a shot, and verify it that it doesn't happen for >>> new instances. I'm working with data imported by someone else, so chances >>> are they didn't script label field up when inserting the data. >>> >>> I think everyone can ignore this post! Sorry. >>> >>> cheers, >>> Scott. >>> >>> On 14/01/2014, at 8:57 AM, Jeff Coughlin <[email protected]> wrote: >>> >>>> I haven't seen this, but I'm still on FC 6.3 at the moment (except for >>>> some testing I'm doing in 7). Until a fix is found, one thing you can do >>>> is force it to save the title to the label by creating a custom >>>> beforeSave() method inside the custom type. If anything it will at least >>>> give you a temporary workaround. But if this is a bug in FC7, then it's a >>>> pretty serious one (IMO). >>>> >>>> Example: >>>> >>>> <cffunction name="beforeSave" access="public" output="false" >>>> returntype="struct"> >>>> <cfargument name="stProperties" required="true" type="struct" /> >>>> <cfargument name="stFields" required="true" type="struct" /> >>>> <cfargument name="stFormPost" required="false" type="struct" /> >>>> >>>> <cfset arguments.stProperties.label = arguments.stProperties.title /> >>>> >>>> <cfreturn super.beforeSave(argumentCollection=arguments) /> >>>> </cffunction> >>>> >>>> If you feel comfortable sharing your CFC, go ahead and post it (maybe >>>> we'll see something you're missing - sometimes it just takes a second set >>>> of eyes to see something we've overlooked). If not, you are welcome to >>>> send it to just me and I'll be happy to see if something jumps out. >>>> >>>> -- >>>> Jeff Coughlin >>>> >>>> On Jan 13, 2014, at 5:12 PM, Scott Mebberson <[email protected]> >>>> wrote: >>>> >>>>> Has anyone run into a problem in which labels for a custom object type >>>>> aren't working on all occasions. Works for some instances of the object >>>>> type, but not others. The type does have a title property, which always >>>>> contains the real value. >>>>> >>>>> I've tried the following: >>>>> updateApp (a few dozen times, and after each of the following) >>>>> added bLabel to the title cfproperty tag >>>>> added displayLabel.cfm to the type folder in webskins and am outputting >>>>> stObj.title >>>>> But still, no label. The label doesn't exist in the stObj that is passed >>>>> around to the various webskins. >>>>> >>>>> Any ideas? >>>>> >>>>> I'm running FarCry 7 (I did git pull on both core and farcrycms this >>>>> morning so they're fresh!). >>>>> >>>>> Thanks, >>>>> Scott. -- You received this message cos you are subscribed to "farcry-dev" Google group. To post, email: [email protected] To unsubscribe, email: [email protected] For more options: http://groups.google.com/group/farcry-dev -------------------------------- Follow us on Twitter: http://twitter.com/farcry --- You received this message because you are subscribed to the Google Groups "farcry-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
