zhuzhurk commented on code in PR #24186:
URL: https://github.com/apache/flink/pull/24186#discussion_r1464840583
##########
flink-connectors/flink-connector-files/src/test/java/org/apache/flink/connector/file/src/FileSourceTextLinesITCase.java:
##########
@@ -126,8 +127,9 @@ private void testBoundedTextFileSource(
.build();
final StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
+ env.setRuntimeMode(RuntimeExecutionMode.BATCH);
Review Comment:
Can we add a new test to verify dynamic parallelism inference for file
sources?
We may use `miniCluster.getExecutionGraph(jobId)` to get the execution graph
of the job and check the parallelism of the source.
##########
flink-connectors/flink-connector-files/src/test/java/org/apache/flink/connector/file/src/FileSourceTextLinesITCase.java:
##########
@@ -126,8 +127,9 @@ private void testBoundedTextFileSource(
.build();
final StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
+ env.setRuntimeMode(RuntimeExecutionMode.BATCH);
env.setParallelism(PARALLELISM);
- env.setRestartStrategy(RestartStrategies.fixedDelayRestart(1, 0));
+
env.setRestartStrategy(RestartStrategies.fixedDelayRestart(PARALLELISM, 0));
Review Comment:
What would happen if we do not increase it?
##########
flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/src/AbstractFileSource.java:
##########
@@ -177,6 +180,24 @@ public TypeInformation<T> getProducedType() {
return readerFormat.getProducedType();
}
+ @Override
+ public int inferParallelism(Context dynamicParallelismContext) {
Review Comment:
`AbstractFileSource` is a base for both `FileSource` and `HiveSource`. If we
only target `FileSource` in the PR, why not directly modify `FileSource`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
