[
https://issues.apache.org/jira/browse/FLINK-12070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16863881#comment-16863881
]
Stephan Ewen commented on FLINK-12070:
--------------------------------------
FYI, here are some alternative implementations to the pure mmap file approach:
https://github.com/StephanEwen/incubator-flink/tree/bounded
The type can be changed by changing BoundedBlockingResultpartitionType in
https://github.com/StephanEwen/incubator-flink/commit/2e6e5509fd9a7789ad5b5a652564093e0487a404#diff-097e48ca7d6306f84764b90955d45c79R159
- FILE: Writes and reads from file
- MMAP: Like before
- FILE_MMAP: Write to file (buffers are persisted already) mmap only for read
(no lazy writes when evicting pages).
> Make blocking result partitions consumable multiple times
> ---------------------------------------------------------
>
> Key: FLINK-12070
> URL: https://issues.apache.org/jira/browse/FLINK-12070
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Network
> Affects Versions: 1.9.0
> Reporter: Till Rohrmann
> Assignee: Stephan Ewen
> Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.9.0
>
> Attachments: image-2019-04-18-17-38-24-949.png
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> In order to avoid writing produced results multiple times for multiple
> consumers and in order to speed up batch recoveries, we should make the
> blocking result partitions to be consumable multiple times. At the moment a
> blocking result partition will be released once the consumers has processed
> all data. Instead the result partition should be released once the next
> blocking result has been produced and all consumers of a blocking result
> partition have terminated. Moreover, blocking results should not hold on slot
> resources like network buffers or memory as it is currently the case with
> {{SpillableSubpartitions}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
