[
https://issues.apache.org/jira/browse/FLINK-12070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819778#comment-16819778
]
Yingjie Cao commented on FLINK-12070:
-------------------------------------
I'd like to perform a test to compare the performance of old and new
(mmappartition) blocking resultpartition implementation this week, the
following scenarios will be covered:
# large number of subparitions and small data volume per subpartition (small
enough and can be fit into memory of SpillableSubpartition or not small enough
and must be spill to disk).
# large number of subparitions and large data volume per subpartition.
# snall number of subparitions and small data volume per subpartition (small
enough and can be fit into memory of SpillableSubpartition or not small enough
and must be spill to disk).
# small number of subparitions and large data volume per subpartition.
All the scenarios will be test both on SATA and SSD. The test application will
contains two vertices, the source vertex is responsible for generating data and
pushing to downstream vertex, and different complexities of downstream vertex
will be used.
> Make blocking result partitions consumable multiple times
> ---------------------------------------------------------
>
> Key: FLINK-12070
> URL: https://issues.apache.org/jira/browse/FLINK-12070
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Network
> Reporter: Till Rohrmann
> Assignee: BoWang
> Priority: Major
>
> In order to avoid writing produced results multiple times for multiple
> consumers and in order to speed up batch recoveries, we should make the
> blocking result partitions to be consumable multiple times. At the moment a
> blocking result partition will be released once the consumers has processed
> all data. Instead the result partition should be released once the next
> blocking result has been produced and all consumers of a blocking result
> partition have terminated. Moreover, blocking results should not hold on slot
> resources like network buffers or memory as it is currently the case with
> {{SpillableSubpartitions}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
