[
https://issues.apache.org/jira/browse/FLINK-14870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhu Zhu updated FLINK-14870:
----------------------------
Description:
In runtime, a null slot sharing group means the vertex should be run in an
individual slot which is not shared with other vertices.
However, at the moment, there is no way to set the slot sharing group of a
vertex to null in production. This is because {{StreamGraphGenerator}} would
always assign a non-null slot sharing group to an operator({{StreamNode}}), and
that would be the group of the generated {{JobVertex}}. (In the case of
{{DataSet}}, all vertices are in the "default" group.)
Moreover, currently users are already able to force an operator to run in an
individual slot by specify a different slot sharing group key to that operator.
So looks that supporting null slot sharing groups in runtime does not make much
sense. And it's making things more complex, with codes like {{@Nullable}}
annotations for slot sharing group, task required resources calculations, and
single task slot allocation.
To make runtime code base and future development simpler, I'd propose to
introduce the assumption that task slot sharing group should not be null.
The detailed proposal is:
1. In {{StreamingJobGraphGenerator}}, we check to ensure the slot sharing group
key of a {{StreamNode}} is non-null, so that we always set a non-null
{{SlotSharingGroup}} to a JobVertex
2. Remove all the {{@Nullable}} annotations for {{SlotSharingGroup}} and
{{SlotSharingGroupID}} in several classes
-3. Always use the slot sharing group resources as the physical slot resources
for a task-
4. Remove {{SchedulerImp#allocateSingleSlot}} and the related code paths
cc [~chesnay] [~gjy] [~trohrmann]
was:
In runtime, a null slot sharing group means the vertex should be run in an
individual slot which is not shared with other vertices.
However, at the moment, there is no way to set the slot sharing group of a
vertex to null in production. This is because {{StreamGraphGenerator}} would
always assign a non-null slot sharing group to an operator({{StreamNode}}), and
that would be the group of the generated {{JobVertex}}. (In the case of
{{DataSet}}, all vertices are in the "default" group.)
Moreover, currently users are already able to force an operator to run in an
individual slot by specify a different slot sharing group key to that operator.
So looks that supporting null slot sharing groups in runtime does not make much
sense. And it's making things more complex, with codes like {{@Nullable}}
annotations for slot sharing group, task required resources calculations, and
single task slot allocation.
To make runtime code base and future development simpler, I'd propose to
introduce the assumption that task slot sharing group should not be null.
The detailed proposal is:
1. In {{StreamingJobGraphGenerator}}, we check to ensure the slot sharing group
key of a {{StreamNode}} is non-null, so that we always set a non-null
{{SlotSharingGroup}} to a JobVertex
2. Remove all the {{@Nullable}} annotations for {{SlotSharingGroup}} and
{{SlotSharingGroupID}} in several classes
3. Always use the slot sharing group resources as the physical slot resources
for a task
4. Remove {{SchedulerImp#allocateSingleSlot}} and the related code paths
cc [~chesnay] [~gjy] [~trohrmann]
> Introducing non-nullable assumption for task slot sharing group
> ---------------------------------------------------------------
>
> Key: FLINK-14870
> URL: https://issues.apache.org/jira/browse/FLINK-14870
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
> Affects Versions: 1.10.0
> Reporter: Zhu Zhu
> Priority: Major
>
> In runtime, a null slot sharing group means the vertex should be run in an
> individual slot which is not shared with other vertices.
> However, at the moment, there is no way to set the slot sharing group of a
> vertex to null in production. This is because {{StreamGraphGenerator}} would
> always assign a non-null slot sharing group to an operator({{StreamNode}}),
> and that would be the group of the generated {{JobVertex}}. (In the case of
> {{DataSet}}, all vertices are in the "default" group.)
> Moreover, currently users are already able to force an operator to run in an
> individual slot by specify a different slot sharing group key to that
> operator.
> So looks that supporting null slot sharing groups in runtime does not make
> much sense. And it's making things more complex, with codes like
> {{@Nullable}} annotations for slot sharing group, task required resources
> calculations, and single task slot allocation.
> To make runtime code base and future development simpler, I'd propose to
> introduce the assumption that task slot sharing group should not be null.
> The detailed proposal is:
> 1. In {{StreamingJobGraphGenerator}}, we check to ensure the slot sharing
> group key of a {{StreamNode}} is non-null, so that we always set a non-null
> {{SlotSharingGroup}} to a JobVertex
> 2. Remove all the {{@Nullable}} annotations for {{SlotSharingGroup}} and
> {{SlotSharingGroupID}} in several classes
> -3. Always use the slot sharing group resources as the physical slot
> resources for a task-
> 4. Remove {{SchedulerImp#allocateSingleSlot}} and the related code paths
> cc [~chesnay] [~gjy] [~trohrmann]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
