Zoltán Borók-Nagy created IMPALA-15525:
------------------------------------------
Summary: Make LZ4 compression of exchange row batches adaptive or
configurable
Key: IMPALA-15525
URL: https://issues.apache.org/jira/browse/IMPALA-15525
Project: IMPALA
Issue Type: Improvement
Components: Backend
Reporter: Zoltán Borók-Nagy
Every row batch sent to a remote receiver is LZ4-compressed on the sender's
fragment thread (OutboundRowBatch::PrepareForSend() -> TryCompress(),
outbound-row-batch.cc:31-51). IMPALA-12430 skips compression only for channels
to the local process, and there is no option to turn it off.
*Data* (TPC-DS SF100, 5 executors on AWS m6id.4xlarge, up to 12.5 Gbps network,
master 1f34ea915e):
* perf: LZ4_compress is 4.9% and LZ4_decompress 1.5% of executor CPU samples,
about 447 CPU-s per pass of the workload. SerializeBatchTime of hash senders
adds up to 440 s.
* The achieved ratio is only 1.5-2.1x, and the network averages about 0.5 Gbps
per host.
* Once senders are no longer limited by round trips (bigger buffers,
IMPALA-15512; more RPCs in flight, IMPALA-XXXXX), LZ4 becomes their bottleneck.
In q72 at MT_DOP=8, the main hash sender spends 5.9 s of its 10.2 s compressing
(3.0 GB -> 1.71 GB per instance).
*Measured with a prototype* (104 queries, MT_DOP=0, total of best-of-2):
* Compression off: -6.6% (-5.0% without q72 and q78). Network bytes grow about
1.85x.
* Adaptive (stop compressing for 64 batches when a sender's ratio is below 2,
then try again): -4.5%.
* At MT_DOP=8, adaptive was slightly better than off.
*Proposal:* add an EXCHANGE_COMPRESSION option (or flag) with values LZ4, NONE
and AUTO. AUTO tracks the achieved ratio per sender and skips compression while
it is below a threshold, re-probing periodically. It could keep LZ4 for
broadcast exchanges, where one compression serves every receiver.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
