curcur commented on a change in pull request #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#discussion_r366830175
##########
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/ExecutionTest.java
##########
@@ -540,6 +553,46 @@ public void testSlotReleaseAtomicallyReleasesExecution()
throws Exception {
});
}
+ /**
+ * Tests that producedPartitions are registered synchronously under an
asynchronous interface.
+ */
+ @Test(expected = IllegalStateException.class)
+ public void testSynchronousRegisterProducedPartitions() throws
Exception {
+ final ScheduledExecutorService executor = new
DirectScheduledExecutorService();
+ final ShuffleMaster<ShuffleDescriptor> shuffleMaster = new
TestingShuffleMaster();
+
+ final ExecutionJobVertex sourceJobVertex =
getExecutionVertex(new JobVertexID(), executor, EAGER, shuffleMaster);
Review comment:
I am kind of thinking it is an overkill to use the
TestingExecutionGraphBuilder, which translate a JobGraph -> ExecutionGraph.
In this particular case, I need to test two ExecutionVertex with one having
partition registered and verify if the registration is done in an asynchronous
way, throw an exception otherwise. It is not testing the entire workflow, so
whether to construct a complete graph does not matter that much.
More specifically, I need to retain the topological relationship between
source and target for ExecutionVertex. But in an ExecutionGraph, I only have
access to ExecutionVertex through `Iterable<ExecutionVertex>
getAllExecutionVertices()`. Then I have to reconstruct the topological relation.
Down to code level, by using `TestingExecutionGraphBuilder`, which builds
ExecutionGraph from JobGraph similar like this:
```
JobGraph jobGraph = new JobGraph("The graph");
JobVertex source = new JobVertex("source");
JobVertex target = new JobVertex("target");
target.connectNewDataSetAsInput(source, POINTWISE, PIPELINED);
jobGraph.addVertex(source);
jobGraph.addVertex(target);
ExecutionGraph eg = TestingExecutionGraphBuilder
.newBuilder()
.setJobGraph(jobGraph)
.setShuffleMaster(shuffleMaster)
.build();
```
Then I have to identify which one is the source and which one is the target,
and register with responding Execution.
The previous implementation looks more clean to me. What do you think? Or
you have a better way to do this.
> I think we should create an ExecutionGraph and get the vertices from it
rather than using this `getExecutionVertex(...)` to create vertices. Otherwise
these 2 vertices are actually from 2 different ExecutionGraphs.
> In this way you do not need to create edges manually. You can leverage
TestingExecutionGraphBuilder to build an ExecutionGraph and set the wanted
shuffle master.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
