yunfengzhou-hub commented on code in PR #24272:
URL: https://github.com/apache/flink/pull/24272#discussion_r1497209330
##########
flink-streaming-java/src/test/java/org/apache/flink/streaming/api/graph/StreamingJobGraphGeneratorTest.java:
##########
@@ -2303,6 +2307,102 @@ void
testOutputFormatSupportConcurrentExecutionAttempts() {
new TestingOutputFormatSupportConcurrentExecutionAttempts<>(),
true);
}
+ @Test
+ void testOutputOnlyAfterEndOfStream() {
+ final StreamExecutionEnvironment env =
+ StreamExecutionEnvironment.getExecutionEnvironment(new
Configuration());
+
+ final DataStream<Integer> source = env.fromData(1, 2,
3).name("source");
+ source.keyBy(x -> x)
+ .transform(
+ "map",
+ Types.INT,
+ new StreamOperatorWithConfigurableOperatorAttributes<>(
+ x -> x,
+ new OperatorAttributesBuilder()
+ .setOutputOnlyAfterEndOfStream(true)
+ .build()))
+ .sinkTo(new DiscardingSink<>())
+ .name("sink");
+
+ final StreamGraph streamGraph = env.getStreamGraph(false);
+ Map<String, StreamNode> nodeMap = new HashMap<>();
+ for (StreamNode node : streamGraph.getStreamNodes()) {
+ nodeMap.put(node.getOperatorName(), node);
+ }
+ assertThat(nodeMap).hasSize(3);
+ assertThat(nodeMap.get("Source:
source").isOutputOnlyAfterEndOfStream()).isFalse();
+ assertThat(nodeMap.get("map").isOutputOnlyAfterEndOfStream()).isTrue();
+ assertThat(nodeMap.get("sink:
Writer").isOutputOnlyAfterEndOfStream()).isFalse();
+
+ assertThat(nodeMap.get("Source:
source").getManagedMemoryOperatorScopeUseCaseWeights())
+ .isEmpty();
+
assertThat(nodeMap.get("map").getManagedMemoryOperatorScopeUseCaseWeights()).hasSize(1);
+ assertThat(nodeMap.get("sink:
Writer").getManagedMemoryOperatorScopeUseCaseWeights())
+ .isEmpty();
+
+ JobGraph jobGraph =
StreamingJobGraphGenerator.createJobGraph(streamGraph);
+ Map<String, JobVertex> vertexMap = new HashMap<>();
+ for (JobVertex vertex : jobGraph.getVertices()) {
+ vertexMap.put(vertex.getName(), vertex);
+ }
+ assertThat(vertexMap).hasSize(2);
+ assertHasOutputPartitionType(
+ vertexMap.get("Source: source"),
ResultPartitionType.PIPELINED_BOUNDED);
Review Comment:
Tests have been added right after this line, disabling operator chaining and
checking that the output edge of
StreamOperatorWithConfigurableOperatorAttributes is blocking. Besides, there
will be only one output edge(to be exact, IntermediateDataSet) no matter how
many downstream operators there are, so this test case should be enough to
verify that "all" downstream edges are blocking.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
