reswqa commented on code in PR #23944:
URL: https://github.com/apache/flink/pull/23944#discussion_r1429533482
##########
flink-optimizer/src/main/java/org/apache/flink/optimizer/plantranslate/JobGraphGenerator.java:
##########
@@ -153,6 +154,8 @@ public class JobGraphGenerator implements Visitor<PlanNode>
{
private final boolean useLargeRecordHandler;
+ private final Map<String, String> parallelismOverrides;
Review Comment:
IIRC, This(`JobGraphGenerator`) should only be used in `DataSet` API. I
wonder do we really need to support this configuration for it, since this part
of APIs was deprecated since 1.18.
##########
flink-optimizer/src/main/java/org/apache/flink/optimizer/plantranslate/JobGraphGenerator.java:
##########
@@ -153,6 +154,8 @@ public class JobGraphGenerator implements Visitor<PlanNode>
{
private final boolean useLargeRecordHandler;
+ private final Map<String, String> parallelismOverrides;
Review Comment:
Another question is: Have we already done the same support for DataStream
API(from `StreamJobGraphGenerator`)?
##########
flink-runtime/src/test/java/org/apache/flink/runtime/dispatcher/DispatcherTest.java:
##########
@@ -1151,6 +1151,43 @@ public void testOverridingJobVertexParallelisms() throws
Exception {
Assert.assertEquals(jobGraph.findVertexByID(v3.getID()).getParallelism(), 42);
}
+ @Test
+ public void testOverridingJobVertexParallelismsWithJobSubmission() throws
Exception {
+ JobVertex v1 = new JobVertex("v1");
+ v1.setParallelism(1);
+ JobVertex v2 = new JobVertex("v2");
+ v2.setParallelism(2);
+ JobVertex v3 = new JobVertex("v3");
+ v3.setParallelism(3);
+ jobGraph = new JobGraph(jobGraph.getJobID(), "job", v1, v2, v3);
+ jobGraph.getJobConfiguration()
Review Comment:
It seems that config through the `JobGraph` has a higher priority than the
`Dispatcher` itself, and we may not need to introduce new tests. Just let
`testOverridingJobVertexParallelisms` test the two configurations at the same
time, it also can test the priority of both configuration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
