Josh Wills created CRUNCH-281:
---------------------------------
Summary: Optimize job dependency logic
Key: CRUNCH-281
URL: https://issues.apache.org/jira/browse/CRUNCH-281
Project: Crunch
Issue Type: Bug
Components: Core
Affects Versions: 0.7.0
Reporter: Josh Wills
Assignee: Josh Wills
Attachments: CRUNCH-281.patch
In CRUNCH-237, we added a fix to handle a case where jobs could be executed out
of dependency order if a dependency was specified via ParallelDoOptions. The
fix for that issue worked, but was relatively naiive: it forced all of the
earlier jobs in a pipeline run to finish before any subsequent jobs could
start, without checking to see if there were implicit/explicit dependencies
between the jobs in those phases. This could increase the overall runtime of a
pipeline if the first phase of a pipeline run had one quick job and one long
job and the second phase of the pipeline had a long running job that only
depended on the short-running job, we would want to kick off the second long
running job immediately and not wait for the first long running job to finish.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
