[
https://issues.apache.org/jira/browse/IMPALA-12430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17765782#comment-17765782
]
ASF subversion and git services commented on IMPALA-12430:
----------------------------------------------------------
Commit fb2d2b27641a95f51b6789639fab73b60abd7bc5 in impala's branch
refs/heads/branch-4.3.0 from Csaba Ringhofer
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=fb2d2b276 ]
IMPALA-12430: Skip compression when sending row batches within same process
LZ4 compression doesn't seem useful when the RowBatch is sent to a
fragment instance within the same process instead of a remote host.
After this change KrpcDataStreamSender skips compression for channels
where the destination is in the same process.
Other changes:
- OutboundRowBatch is moved to a separate file to make the commonly
included row-batch.h lighter.
- TestObservability.test_global_exchange_counters had to be changed
as skipping compression changed metric ExchangeScanRatio. Also added
a sleep to the test query because it was flaky on my machine (it
doesn't seem flaky in jenkins runs, probably my CPU is faster).
See the Jira for more details on tasks that could be skipped in
intra process RowBatch transfer. From these compression is both
the most expensive and easiest to avoid.
Note that it may also make sense to skip compression if the target
is not the in same process but resides on the same host. This setup is
not typical in production environment AFAIK and it would complicate
testing compression as impalad processes often run on the
same host during tests. For these reasons it seems better to only
implement this if both the host and port are the same.
TPCH benchmark shows significant improvement but it uses only 3
impalad processes so 1/3 of exchanges are affected - in bigger
clusters the change should be much smaller.
+----------+-----------------------+---------+------------+------------+----------------+
| Workload | File Format | Avg (s) | Delta(Avg) | GeoMean(s) |
Delta(GeoMean) |
+----------+-----------------------+---------+------------+------------+----------------+
| TPCH(42) | parquet / none / none | 3.59 | -4.95% | 2.37 | -2.51%
|
+----------+-----------------------+---------+------------+------------+----------------+
Change-Id: I7ea23fd1f0f10f72f3dbd8594f3def3ee190230a
Reviewed-on: http://gerrit.cloudera.org:8080/20462
Tested-by: Impala Public Jenkins <[email protected]>
Reviewed-by: Daniel Becker <[email protected]>
> Optimize sending rows within the same process
> ---------------------------------------------
>
> Key: IMPALA-12430
> URL: https://issues.apache.org/jira/browse/IMPALA-12430
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
> Reporter: Csaba Ringhofer
> Priority: Major
> Labels: performance
> Fix For: Impala 4.3.0
>
>
> Currently sending row batches to exchange nodes always goes through KRPC even
> if the sender and receiver are within the same process.
> This means that the following work is done without actually being necessary:
> sender:
> 1. serialize RowBatch to a single buffer
> 2. compress the buffer with LZ4
> 3. send the buffer as a sidecar in KRPC
> receiver:
> 4. fetch buffer from KRPC
> 5. decompress the buffer
> 6. convert the buffer to RowBatch
> Ideally a single deep copy from the sender's RowBatch to the destination's
> RowBatch is enough (this is needed to cleanup the memory referenced in the
> original RowBatch during send).
> The most expensive part is 2, the compression with LZ4 (decompression is much
> faster) and can be avoided with minimal changes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
