[
https://issues.apache.org/jira/browse/FLINK-12070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16869404#comment-16869404
]
Yingjie Cao commented on FLINK-12070:
-------------------------------------
I have appended test results of the new implementation (FILE_MMAP mode) in
[https://docs.google.com/document/d/1Uimt-cU7tqEBKGCrJ5H6g1KyJp_NTG0WW2hQ9qKh7uo/edit?usp=sharing|https://docs.google.com/document/d/1Uimt-cU7tqEBKGCrJ5H6g1KyJp_NTG0WW2hQ9qKh7uo/edit?usp=sharing.],
overall both the stability and performance of the FILE_MMAP mode are good
enough. However, the MMAP mode still incurs the previous problem (to save time,
only several cases are tested) and unfortunately, the there seems bugs with the
FILE mode, the FileBufferReader throws exception when nextBuffer method is
call, I guess this is because the writeAndFlush method of netty channel dose
not work synchronous so will not recycle the memory segment immediately.
> Make blocking result partitions consumable multiple times
> ---------------------------------------------------------
>
> Key: FLINK-12070
> URL: https://issues.apache.org/jira/browse/FLINK-12070
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Network
> Affects Versions: 1.9.0
> Reporter: Till Rohrmann
> Assignee: Stephan Ewen
> Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.9.0
>
> Attachments: image-2019-04-18-17-38-24-949.png
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> In order to avoid writing produced results multiple times for multiple
> consumers and in order to speed up batch recoveries, we should make the
> blocking result partitions to be consumable multiple times. At the moment a
> blocking result partition will be released once the consumers has processed
> all data. Instead the result partition should be released once the next
> blocking result has been produced and all consumers of a blocking result
> partition have terminated. Moreover, blocking results should not hold on slot
> resources like network buffers or memory as it is currently the case with
> {{SpillableSubpartitions}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
