Andrey Zagrebin created FLINK-13435:
---------------------------------------
Summary: Remove ShuffleDescriptor.ReleaseType and make release
semantics fixed per partition type
Key: FLINK-13435
URL: https://issues.apache.org/jira/browse/FLINK-13435
Project: Flink
Issue Type: Sub-task
Components: Runtime / Coordination, Runtime / Network
Affects Versions: 1.9.0
Reporter: Andrey Zagrebin
Fix For: 1.9.0, 1.10.0
In a long term we do not need auto-release semantics for blocking (persistent)
partition. We expect them always to be released externally by JM and assume
they can be consumed multiple times.
The pipelined partitions have always only one consumer and one consumption
attempt. Afterwards they can be always released automatically.
ShuffleDescriptor.ReleaseType was introduced to make release semantics more
flexible but it is not needed in a long term.
FORCE_PARTITION_RELEASE_ON_CONSUMPTION was introduced as a safety net to be
able to fallback to 1.8 behaviour without the partition tracker and JM taking
care about blocking partition release. We can make this option specific for
NettyShuffleEnvironment which was the only existing shuffle service before. If
it is activated then the blocking partition is also auto-released on a
consumption attempt as it was before. The fine-grained recovery will just not
find the partition after the job restart in this case and will restart the
producer.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
