On Thu, Jun 14, 2012 at 6:15 PM, Dannon Baker <dannonba...@me.com> wrote: > On Jun 14, 2012, at 12:48 PM, Peter Cock wrote: > >> In a separate example with 33 sub-tasks, there were two of these >> inversions, while in yet another example with 33 sub-tasks there was >> a trio submitted out of order. This non-deterministic behavior is a >> little surprising, but in itself not an immediate problem. > > You're correct in that submission order shouldn't matter at all, > but I'll take a look and see if I can come up with an explanation for why.
Great. >> In what appears to be a separate (and more concerning) loss of order, >> after merging the output file order appears randomized. I would expect >> the output from task_0, then task_1, ..., finally task_16. I haven't yet >> worked out what order I am getting, but it isn't this, and neither is it >> the order from the SGE job numbers (e.g. correct bar one pair >> switched round). > > This would be happening in the merge. It looks like change set > c959d32f2405 might be the culprit for this -- it doesn't explicitly > reorder by task number in the merge method, which would lead > to (I'm guessing) an alphanumeric sort. I'll test and fix this. I hope you're right - I've not stuck any debugging code in to try and pin point this yet (maybe tomorrow). >> [*] P.S. I would like to see an upper bound on the sleep_time in method >> run_job, say half an hour? Otherwise with a group of long running jobs >> it seems Galaxy may end up waiting a very long time between checks >> for their completion since it just doubles the wait at each point. I had >> sometimes noticed a delay between the sub-jobs finishing according >> to the cluster and Galaxy doing anything about merging it - this is >> probably why. > > This sleep time should currently cap at 8 seconds. Doh. Yes, that makes sense. Peter ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
