[jira] [Created] (TEZ-3126) Auto-Reduce Parallelism: Vertex not re-configured when reduced by less than half.

[Jonathan Eagles (JIRA)]

Jonathan Eagles created TEZ-3126:
------------------------------------

             Summary: Auto-Reduce Parallelism: Vertex not re-configured when 
reduced by less than half.
                 Key: TEZ-3126
                 URL: https://issues.apache.org/jira/browse/TEZ-3126
             Project: Apache Tez
          Issue Type: Bug
            Reporter: Jonathan Eagles
            Assignee: Jonathan Eagles
            Priority: Critical


For example, when reducing parallelism from 36 to 22. The basePartitionRange 
will be 1 and will not re-configure the vertex.

{code:java|title=ShuffleVertexManager#determineParallelismAndApply|borderStyle=dashed|bgColor=lightgrey}
    int desiredTaskParallelism = 
        (int)(
            (expectedTotalSourceTasksOutputSize+desiredTaskInputDataSize-1)/
            desiredTaskInputDataSize);
    if(desiredTaskParallelism < minTaskParallelism) {
      desiredTaskParallelism = minTaskParallelism;
    }
    
    if(desiredTaskParallelism >= currentParallelism) {
      return true;
    }
    
    // most shufflers will be assigned this range
    basePartitionRange = currentParallelism/desiredTaskParallelism;
    
    if (basePartitionRange <= 1) {
      // nothing to do if range is equal 1 partition. shuffler does it by 
default
      return true;
    }
{code}




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)