[
https://issues.apache.org/jira/browse/FLINK-3315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15159232#comment-15159232
]
ASF GitHub Bot commented on FLINK-3315:
---------------------------------------
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/1641#discussion_r53816021
--- Diff:
flink-streaming-java/src/test/java/org/apache/flink/streaming/api/graph/SlotAllocationTest.java
---
@@ -40,17 +48,142 @@ public void test() {
public boolean filter(Long value) { return false; }
};
- env.generateSequence(1,
10).filter(dummyFilter).isolateResources().filter(dummyFilter)
-
.disableChaining().filter(dummyFilter).startNewResourceGroup().filter(dummyFilter)
- .startNewChain().print();
+ env.generateSequence(1, 10)
+ .filter(dummyFilter).slotSharingGroup("isolated")
+
.filter(dummyFilter).slotSharingGroup("default").disableChaining()
+ .filter(dummyFilter).slotSharingGroup("group 1")
+ .filter(dummyFilter).startNewChain()
+ .print().disableChaining();
+
+ // verify that a second pipeline does not inherit the groups
from the first pipeline
+ env.generateSequence(1, 10)
+
.filter(dummyFilter).slotSharingGroup("isolated-2")
+
.filter(dummyFilter).slotSharingGroup("default").disableChaining()
+ .filter(dummyFilter).slotSharingGroup("group 2")
+ .filter(dummyFilter).startNewChain()
+ .print().disableChaining();
JobGraph jobGraph = env.getStreamGraph().getJobGraph();
List<JobVertex> vertices =
jobGraph.getVerticesSortedTopologicallyFromSources();
- assertEquals(vertices.get(0).getSlotSharingGroup(),
vertices.get(2).getSlotSharingGroup());
+ assertEquals(vertices.get(0).getSlotSharingGroup(),
vertices.get(3).getSlotSharingGroup());
+ assertNotEquals(vertices.get(0).getSlotSharingGroup(),
vertices.get(2).getSlotSharingGroup());
+ assertNotEquals(vertices.get(3).getSlotSharingGroup(),
vertices.get(4).getSlotSharingGroup());
+ assertEquals(vertices.get(4).getSlotSharingGroup(),
vertices.get(5).getSlotSharingGroup());
+ assertEquals(vertices.get(5).getSlotSharingGroup(),
vertices.get(6).getSlotSharingGroup());
--- End diff --
These conditions are actually not so easy to verify since you have to know
how the vertices are internally ordered.
> Fix Slot Sharing in Streaming API
> ---------------------------------
>
> Key: FLINK-3315
> URL: https://issues.apache.org/jira/browse/FLINK-3315
> Project: Flink
> Issue Type: Improvement
> Components: Streaming
> Affects Versions: 1.0.0
> Reporter: Aljoscha Krettek
> Assignee: Aljoscha Krettek
> Priority: Blocker
>
> Right now, the slot sharing/resource group logic is a bit "nebulous". The
> slot sharing group that operators are put in depends on the order in which
> operations are created. For example, in this case:
> {code}
> Source a = env.source()
> Source b = env.source()
> a.map().startNewResourceGroup().sink()
> b.map().sink()
> {code}
> We end up with two resource groups:
> - group 1: source a
> - group 2: map(), sink(), source b, map(), sink()
> The reason is that the slot sharing id is incremented when transforming the
> {{startNewResouceGroup()}} call and all operators that are transformed
> afterwards in graph traversal get that new slot sharing id.
> (There is also {{isolateResources()}} which can be used to isolate an
> operator.)
> What I propose is to remove {{startNewResourceGroup()}} and
> {{isolateResouces()}} and replace it with {{slotSharingGroup(String)}}. By
> default, operations would be in slot sharing group "default". This allows
> very fine grained control over what operators end up in which slot sharing
> group. For example, I could have this topology:
> {code}
> Source a = env.source().slotSharingGroup("sources")
> Source b = env.source().slotSharingGroup("sources")
> a.map().slotSharingGroup("heavy a").sink().slotSharingGroup("sinks")
> b.map().slotSharingGroup("heavy b").sink().slotSharingGroup("sinks")
> {code}
> Which would isolate the lightweight sources and sinks in a group and put
> heavy operations inside their own slot groups.
> This is a bit more low level than the previous API and requires more calls
> than a simple {{startNewResourceGroup()}} but I think not many people would
> use this feature and this design makes it very clear what operations end up
> in the same group.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
