That can works too, but you should be sure to consider whether documents [N,N+1000) at the time of the spawn are guaranteed to still be [N,N +1000] when the task gets around to executing. If you're adding, removing, or modifying documents, you'll get different document ordering over time.
Wayne. On Wed, 2010-04-07 at 07:53 -0700, Geert Josten wrote: > > Another approach is to have an initial query create a list of > > documents to process and cut it into chunks (say 1000 > > documents each) that are each handed off to a spawned task. > > With this, the configured number of threads in the task queue > > will run in parallel giving you higher overall throughput. > > > That is what I more or less did. However, I decided to pass in a start > and end count, to have the function find out for itself which > documents that should be. > > And one other thing I did was divide the total range into a number of > segments equal to the number of parallel threads I would want to run, > which like this can be smaller than the total number of threads > allowed on the task server, giving more room for normal traffic. So > instead of spawning all tasks at once (cluttering the queue as well), > each task creates the next one. A simple try catch makes sure the > recursive spawning isn't terminated before the end.. > > Kind regards, > Geert
_______________________________________________ General mailing list [email protected] http://xqzone.com/mailman/listinfo/general
