Csaba Ringhofer created IMPALA-12430:
----------------------------------------
Summary: Optimize sending rows within the same process
Key: IMPALA-12430
URL: https://issues.apache.org/jira/browse/IMPALA-12430
Project: IMPALA
Issue Type: Improvement
Components: Backend
Reporter: Csaba Ringhofer
Currently sending row batches to exchange nodes always goes through KRPC even
if the sender and receiver are within the same process.
This means that the following work is done without actually being necessary:
sender:
1. serialize RowBatch to a single buffer
2. compress the buffer with LZ4
3. send the buffer as a sidecar in KRPC
receiver:
4. fetch buffer from KRPC
5. decompress the buffer
6. convert the buffer to RowBatch
Ideally a single deep copy from the sender's RowBatch to the destination's
RowBatch is enough (this is needed to cleanup the memory referenced in the
original RowBatch during send).
The most expensive part is 2, the compression with LZ4 (decompression is much
faster) and can be avoided with minimal changes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
