[GitHub] [helix] richardstartin opened a new issue, #2064: `RuntimeJobDag` construction time depends on ParallelJobs, not the number of jobs to execute

Wed, 27 Apr 2022 04:19:54 -0700 


richardstartin opened a new issue, #2064:
URL: https://github.com/apache/helix/issues/2064

   The following code is executed every time a `RuntimeJobDag` is constructed:
   
   ```java
     public void generateJobList() {
       resetJobListAndDependencyMaps();
       computeIndependentNodes();
       _readyJobList.addAll(_independentNodes);
       if (_isJobQueue && _readyJobList.size() > 0) {
         // For job queue, only get number of parallel jobs to run in the ready 
list.
         for (int i = 1; i < _numParallelJobs; i++) {
           if (_parentsToChildren.containsKey(_readyJobList.peekLast())) {
             
_readyJobList.offer(_parentsToChildren.get(_readyJobList.peekLast()).iterator().next());
           }
         }
       }
       _hasDagChanged = false;
     }
   ```
   
   This code looks suspicious because it only ever puts the first child onto 
the ready job list, but also iterates beyond the number of jobs to execute, for 
instance if `_numParallelJobs` is set to `Integer.MAX_VALUE` as it is in Apache 
Pinot 
[here](https://github.com/apache/pinot/blob/6eff17754a0b0ffdfb8670b54b5ada38961b7917/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/minion/PinotHelixTaskResourceManager.java#L116).
   
   This shows up by making integration tests slow, dominating profiles of tests.
   <img width="1309" alt="Screenshot 2022-04-27 at 12 15 50" 
src="https://user-images.githubusercontent.com/16439049/165506541-9f3a772d-c6b8-4413-aa87-d16dc0327cec.png";>
   Tracing `RuntimeJobDag` reveals that with `_numParallelJobs` equal to 
`Integer.MAX_VALUE` construction can take up to 20 seconds to produce very 
small ready job lists.
   
   <img width="1215" alt="Screenshot 2022-04-27 at 12 17 26" 
src="https://user-images.githubusercontent.com/16439049/165506765-a051983f-d1a9-4596-9be4-69e95c02e53d.png";>
   
   Setting the parallelism to 1 
[here](https://github.com/apache/pinot/pull/8603) halves integration test time, 
but prevents parallelism. 
   
   `RuntimeJobDag. generateJobList ` should terminate once all the jobs have 
been added to the ready list, or the maximum parallelism has been reached, 
whichever happens first. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to