Amol Umbarkar created ARROW-8845:
------------------------------------
Summary: Selective compression on the wire
Key: ARROW-8845
URL: https://issues.apache.org/jira/browse/ARROW-8845
Project: Apache Arrow
Issue Type: Improvement
Components: FlightRPC
Reporter: Amol Umbarkar
Dask seems to be selectively do compression if it is found to be useful. They
sort of pick 10kb of sample upfront to calculate compression and if the results
are good then the whole batch is compressed. This seems to save de-compression
effort on receiver side.
Please take a look at
[https://blog.dask.org/2016/04/14/dask-distributed-optimizing-protocol#problem-3-unwanted-compression]
Thought this could be relevant to arrow batch transfers as well.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
