[
https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16983552#comment-16983552
]
Piotr Nowojski commented on FLINK-14845:
----------------------------------------
Regarding the org.lz4:lz4-java dependency. This one is very lightweight, I do
not see any transitive dependencies, which is good. So the only potential
issues might arise if user's code depends on the same library in a different
version. This should be resolved by user-first class loader, so I think it's
fine to not make it pluggable in the first version. However probably this
should be changed and moved to plugin sooner or later.
{quote}We have to consider how to handle the condition when there is no free
buffers in the buffer pool, and there is may deadlock problem if we choose to
wait in a blocking way. {quote}
I guess you are right [~kevin.cyj]. We could safely accumulate and not recycle
multiple buffers only when a reader would be polling the buffers from the
{{InputGate}} in a non blocking way and wouldn't depend on more buffers to
arrive before recycling them. It's hard for me to imagine a use case with such
contract.
In that case we would need to document this expected behaviour (previously
returned buffers should be recycled immediately before asking for new ones)
some where in the {{InputGate#getNext}} and {{InputChannel#getNextBuffer}}.
> Introduce data compression to blocking shuffle.
> -----------------------------------------------
>
> Key: FLINK-14845
> URL: https://issues.apache.org/jira/browse/FLINK-14845
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Network
> Reporter: Yingjie Cao
> Assignee: Yingjie Cao
> Priority: Major
>
> Currently, blocking shuffle writer writes raw output data to disk without
> compression. For IO bounded scenario, this can be optimized by compressing
> the output data. It is better to introduce a compression mechanism and offer
> users a config option to let the user decide whether to compress the shuffle
> data. Actually, we hava implemented compression in our inner Flink version
> and here are some key points:
> 1. Where to compress/decompress?
> Compressing at upstream and decompressing at downstream.
> 2. Which thread do compress/decompress?
> Task threads do compress/decompress.
> 3. Data compression granularity.
> Per buffer.
> 4. How to handle that when data size become even bigger after compression?
> Give up compression in this case and introduce an extra flag to identify if
> the data was compressed, that is, the output may be a mixture of compressed
> and uncompressed data.
>
> We'd like to introduce blocking shuffle data compression to Flink if there
> are interests.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
