Hey all, The int(time.time()/30) part of the task name is to prevent queue stalls. When memcache gets evicted the work index counter will be reset to zero. That means new fork-join work items may insert tasks that are named the same as tasks that were already inserted. By including a time window of ~30 seconds in the task name, we ensure that this problem can only last for about thirty seconds. This is also why you should raise an exception when you see a TombstonedTaskError exception.
Worst-case scenario if the clocks are wonky is that two tasks are run to do the fan-in work instead of just one, which is an acceptable trade-off in many cases and a fundamental possibility when using the task queue API. This can be mitigated using pigeon-hole acknowledgment entities, like I use in my materialized view example. Hope that helps, -Brett On Mon, Jun 7, 2010 at 2:14 PM, Tristan <tristan.slomin...@gmail.com> wrote: > not a python guy but, the purpose of int (now / 30) will be to come up > with the same name for a span of time (30 milliseconds?). > > notice that int(1/30) = 0 int (3/30) = 0 int (29/30) = 0 and > int(32/30) = 1. this is a way to come up with that task name > uniquely. > > although now i'm confused because doesn't he say later on that time is > a bad thing to use for synchronization and sequence numbers should be > used instead? > > On Jun 7, 2:40 am, Jaroslav Záruba <jaroslav.zar...@gmail.com> wrote: > > Also if someone knew what is the purpose of "now / 30" in the task name, > > please:http://www.youtube.com/watch?v=zSDC_TU7rtc#t=41m35 > > > > Regards > > J. Záruba > > > > 2010/6/7 Jaroslav Záruba <jaroslav.zar...@gmail.com> > > > > > > > > > Hello > > > > > I'm reading through the PDF that Brett Slatkin has published for %subj > > > %. > > >http://tinyurl.com/3523mej > > > > > In the video (the Fan-in part) Brett says that the work_index has to > > > be a hash, so that 'you distribute the load across the BigTable' > > >http://www.youtube.com/watch?v=zSDC_TU7rtc#t=48m44 > > > > > And this is how work_index is created: > > > work_index = '%s-%d' % (sum_name, knuth_hash(index)) > > > ...which I guess creates something like 'votesMovieXYZ-54657651321987' > > > > > My question is why only one half of work_index is hashed? Is it > > > important? > > > Would it be bad to do md5('%s-%d' % (sum_name, index)) so that the > > > hash would be like '6gw8....hq6'? > > > > > Regards > > > J. Záruba > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appeng...@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com> > . > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
