[jira] [Commented] (FLINK-13478) Decouple two different release strategies in BoundedBlockingSubpartition
[
https://issues.apache.org/jira/browse/FLINK-13478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897807#comment-16897807
]
Andrey Zagrebin commented on FLINK-13478:
-
Thanks for the clarification [~zjwang]!
> Decouple two different release strategies in BoundedBlockingSubpartition
>
>
> Key: FLINK-13478
> URL: https://issues.apache.org/jira/browse/FLINK-13478
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination, Runtime / Network
>Reporter: zhijiang
>Assignee: zhijiang
>Priority: Minor
>
> We have two basic release strategies atm. One is based on consumption via
> network notification from consumer. The other is based on notification via
> RPC from JM/scheduler.
> But in current implementation of {{BoundedBlockingSubpartition}}, these two
> ways are a bit coupled with each other. If the JM decides to release
> partition and send the release RPC call, it has to wait all the reader views
> really released before finally closing the data file. So the JM-RPC-based
> release strategy still relies on the consumption confirmation via network to
> some extent.
> In order to make these two release strategies independent, if the release
> call is from JM/scheduler RPC, we could immediately release all the view
> readers and then close the data file as well. If the release is based on
> consumption confirmation, after all the view readers for one subpartition are
> released, the subpartition could further notify the parent
> {{ResultPartition}} which decides whether to release the whole partition or
> not.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (FLINK-13478) Decouple two different release strategies in BoundedBlockingSubpartition
[
https://issues.apache.org/jira/browse/FLINK-13478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897482#comment-16897482
]
zhijiang commented on FLINK-13478:
--
[~azagrebin], thanks for the consideration.
I think it would no implicit consequences for the consumer side. Just like the
current way of `PipelinedSubpartition#release`, it only release the view and
would not trigger the connection close. The connection close is normally
initiated by consumer/client side.
> Decouple two different release strategies in BoundedBlockingSubpartition
>
>
> Key: FLINK-13478
> URL: https://issues.apache.org/jira/browse/FLINK-13478
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination, Runtime / Network
>Reporter: zhijiang
>Assignee: zhijiang
>Priority: Minor
>
> We have two basic release strategies atm. One is based on consumption via
> network notification from consumer. The other is based on notification via
> RPC from JM/scheduler.
> But in current implementation of {{BoundedBlockingSubpartition}}, these two
> ways are a bit coupled with each other. If the JM decides to release
> partition and send the release RPC call, it has to wait all the reader views
> really released before finally closing the data file. So the JM-RPC-based
> release strategy still relies on the consumption confirmation via network to
> some extent.
> In order to make these two release strategies independent, if the release
> call is from JM/scheduler RPC, we could immediately release all the view
> readers and then close the data file as well. If the release is based on
> consumption confirmation, after all the view readers for one subpartition are
> released, the subpartition could further notify the parent
> {{ResultPartition}} which decides whether to release the whole partition or
> not.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (FLINK-13478) Decouple two different release strategies in BoundedBlockingSubpartition
[
https://issues.apache.org/jira/browse/FLINK-13478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16896824#comment-16896824
]
Andrey Zagrebin commented on FLINK-13478:
-
As I understood, the idea was to rely on consumption confirmation in this
particular case, because it can be considered as reliable once the connection
has been established (reader exists), as [~zjwang] already pointed out.
In general, I would agree that it might simplify things if the forced external
release does it fully and fast. We should probably make sure that there are no
implicit consequences for the consumer because of abruptly closed connection.
> Decouple two different release strategies in BoundedBlockingSubpartition
>
>
> Key: FLINK-13478
> URL: https://issues.apache.org/jira/browse/FLINK-13478
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination, Runtime / Network
>Reporter: zhijiang
>Assignee: zhijiang
>Priority: Minor
>
> We have two basic release strategies atm. One is based on consumption via
> network notification from consumer. The other is based on notification via
> RPC from JM/scheduler.
> But in current implementation of {{BoundedBlockingSubpartition}}, these two
> ways are a bit coupled with each other. If the JM decides to release
> partition and send the release RPC call, it has to wait all the reader views
> really released before finally closing the data file. So the JM-RPC-based
> release strategy still relies on the consumption confirmation via network to
> some extent.
> In order to make these two release strategies independent, if the release
> call is from JM/scheduler RPC, we could immediately release all the view
> readers and then close the data file as well. If the release is based on
> consumption confirmation, after all the view readers for one subpartition are
> released, the subpartition could further notify the parent
> {{ResultPartition}} which decides whether to release the whole partition or
> not.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (FLINK-13478) Decouple two different release strategies in BoundedBlockingSubpartition
[
https://issues.apache.org/jira/browse/FLINK-13478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16896163#comment-16896163
]
zhijiang commented on FLINK-13478:
--
I think it might belong to a pure refactoring.
>From behavior aspect, this fix would only speed the release of partition
>resource if the release call from JM is already reaching on TM side, no need
>to wait for consumers' network confirmation. But it seems no matter to release
>partitions a bit delay based on the assumption that the partition
>canceled/consumed would be finally confirmed via network once establishment.
It mainly brings a bit confusing to understand the partition release strategy.
If the strategy of partition release is based on JM's decision which is always
reliable, then it should not be blocked/delayed by network notification.
We could focus on it if necessary in 1.10 release.
> Decouple two different release strategies in BoundedBlockingSubpartition
>
>
> Key: FLINK-13478
> URL: https://issues.apache.org/jira/browse/FLINK-13478
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination, Runtime / Network
>Reporter: zhijiang
>Assignee: zhijiang
>Priority: Minor
>
> We have two basic release strategies atm. One is based on consumption via
> network notification from consumer. The other is based on notification via
> RPC from JM/scheduler.
> But in current implementation of {{BoundedBlockingSubpartition}}, these two
> ways are coupled with each other. In detail, the network consumption
> notification could only close data file after the release RPC was triggered
> from JM/scheduler. Also for the release RPC it has to wait all the reader
> views really released before closing the data file. So the release RPC still
> relies on network notification to some extent.
> In order to make these two release strategies independent, if the release
> call is from JM/scheduler RPC, we could immediately release all the view
> readers and then close the data file as well. If the release is based on
> consumption notification, after all the view readers for one subpartition are
> released, the subpartition could further notify the parent
> {{ResultPartition}} which decides whether to release the whole partition or
> not.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (FLINK-13478) Decouple two different release strategies in BoundedBlockingSubpartition
[
https://issues.apache.org/jira/browse/FLINK-13478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16896149#comment-16896149
]
zhijiang commented on FLINK-13478:
--
I think it might belong to a pure refactoring.
>From behavior aspect, this fix would only speed the release of partition
>resource if the release call from JM is already reaching on TM side, no need
>to wait for consumers' network confirmation. But it seems no matter to release
>partitions a bit delay based on the assumption that the partition
>canceled/consumed would be finally confirmed via network once establishment.
It mainly brings a bit confusing to understand the partition release strategy.
If the strategy of partition release is based on JM's decision which is always
reliable, then it should not be blocked/delayed by network notification.
We could focus on it if necessary in 1.10 release.
> Decouple two different release strategies in BoundedBlockingSubpartition
>
>
> Key: FLINK-13478
> URL: https://issues.apache.org/jira/browse/FLINK-13478
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination, Runtime / Network
>Reporter: zhijiang
>Assignee: zhijiang
>Priority: Minor
>
> We have two basic release strategies atm. One is based on consumption via
> network notification from consumer. The other is based on notification via
> RPC from JM/scheduler.
> But in current implementation of {{BoundedBlockingSubpartition}}, these two
> ways are coupled with each other. In detail, the network consumption
> notification could only close data file after the release RPC was triggered
> from JM/scheduler. Also for the release RPC it has to wait all the reader
> views really released before closing the data file. So the release RPC still
> relies on network notification to some extent.
> In order to make these two release strategies independent, if the release
> call is from JM/scheduler RPC, we could immediately release all the view
> readers and then close the data file as well. If the release is based on
> consumption notification, after all the view readers for one subpartition are
> released, the subpartition could further notify the parent
> {{ResultPartition}} which decides whether to release the whole partition or
> not.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (FLINK-13478) Decouple two different release strategies in BoundedBlockingSubpartition
[
https://issues.apache.org/jira/browse/FLINK-13478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16896123#comment-16896123
]
Piotr Nowojski commented on FLINK-13478:
It sounds fine for me (modulo my lack of expertise for the batch), however is
this fixing some behaviour or just a pure refactoring? If the second one maybe
it would be better to leave it as it is if the current implementation/code is
just simpler?
> Decouple two different release strategies in BoundedBlockingSubpartition
>
>
> Key: FLINK-13478
> URL: https://issues.apache.org/jira/browse/FLINK-13478
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination, Runtime / Network
>Reporter: zhijiang
>Assignee: zhijiang
>Priority: Minor
>
> We have two basic release strategies atm. One is based on consumption via
> network notification from consumer. The other is based on notification via
> RPC from JM/scheduler.
> But in current implementation of {{BoundedBlockingSubpartition}}, these two
> ways are coupled with each other. In detail, the network consumption
> notification could only close data file after the release RPC was triggered
> from JM/scheduler. Also for the release RPC it has to wait all the reader
> views really released before closing the data file. So the release RPC still
> relies on network notification to some extent.
> In order to make these two release strategies independent, if the release
> call is from JM/scheduler RPC, we could immediately release all the view
> readers and then close the data file as well. If the release is based on
> consumption notification, after all the view readers for one subpartition are
> released, the subpartition could further notify the parent
> {{ResultPartition}} which decides whether to release the whole partition or
> not.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (FLINK-13478) Decouple two different release strategies in BoundedBlockingSubpartition
[
https://issues.apache.org/jira/browse/FLINK-13478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16895832#comment-16895832
]
zhijiang commented on FLINK-13478:
--
[~StephanEwen] [~pnowojski] [~azagrebin] What do you think this issue?
> Decouple two different release strategies in BoundedBlockingSubpartition
>
>
> Key: FLINK-13478
> URL: https://issues.apache.org/jira/browse/FLINK-13478
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination, Runtime / Network
>Reporter: zhijiang
>Assignee: zhijiang
>Priority: Minor
>
> We have two basic release strategies atm. One is based on consumption via
> network notification from consumer. The other is based on notification via
> RPC from JM/scheduler.
> But in current implementation of {{BoundedBlockingSubpartition}}, these two
> ways are coupled with each other. In detail, the network consumption
> notification could only close data file after the release RPC was triggered
> from JM/scheduler. Also for the release RPC it has to wait all the reader
> views really released before closing the data file. So the release RPC still
> relies on network notification to some extent.
> In order to make these two release strategies independent, if the release
> call is from JM/scheduler RPC, we could immediately release all the view
> readers and then close the data file as well. If the release is based on
> consumption notification, after all the view readers for one subpartition are
> released, the subpartition could further notify the parent
> {{ResultPartition}} which decides whether to release the whole partition or
> not.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
