[
https://issues.apache.org/jira/browse/CRUNCH-281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Wills updated CRUNCH-281:
------------------------------
Attachment: CRUNCH-281.patch
The patch for this. The fix ended up being pretty simple, so I'm going to
commit it this evening if there aren't any objections.
> Optimize job dependency logic
> -----------------------------
>
> Key: CRUNCH-281
> URL: https://issues.apache.org/jira/browse/CRUNCH-281
> Project: Crunch
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Reporter: Josh Wills
> Assignee: Josh Wills
> Attachments: CRUNCH-281.patch
>
>
> In CRUNCH-237, we added a fix to handle a case where jobs could be executed
> out of dependency order if a dependency was specified via ParallelDoOptions.
> The fix for that issue worked, but was relatively naiive: it forced all of
> the earlier jobs in a pipeline run to finish before any subsequent jobs could
> start, without checking to see if there were implicit/explicit dependencies
> between the jobs in those phases. This could increase the overall runtime of
> a pipeline if the first phase of a pipeline run had one quick job and one
> long job and the second phase of the pipeline had a long running job that
> only depended on the short-running job, we would want to kick off the second
> long running job immediately and not wait for the first long running job to
> finish.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
