katty he created TEZ-4512: ----------------------------- Summary: One-to-One edge Parallelism Key: TEZ-4512 URL: https://issues.apache.org/jira/browse/TEZ-4512 Project: Apache Tez Issue Type: Improvement Affects Versions: 0.10.2 Reporter: katty he
Currently, we check parallelism for One-one edge, if inputVertex's parallelism doesn't equal outputVertex or -1, tez will report error:" Dag submit failed due to 1-1 Edge. Destination vertex parallelism must match source vertex. Vertex: Reducer x does not match vertex: Reducer x", why shouldn't we change the parallelism with inputVertex and outputVertex instead of reporting error? maybe the code can be like: for (Edge e : edges) { Vertex inputVertex = e.getInputVertex(); Vertex outputVertex = e.getOutputVertex(); if (e.getEdgeProperty().getDataMovementType() == DataMovementType.ONE_TO_ONE) { if (inputVertex.getParallelism() != outputVertex.getParallelism()) { // both should be equal or equal to -1. if (outputVertex.getParallelism() != -1) { int parallelism = Math.min(inputVertex.getParallelism(),outputVertex.getParallelism()); inputVertex.setParallelism(parallelism); outputVertex.setParallelism(parallelism); } } } } -- This message was sent by Atlassian Jira (v8.20.10#820010)