[
https://issues.apache.org/jira/browse/IMPALA-12433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819175#comment-17819175
]
ASF subversion and git services commented on IMPALA-12433:
----------------------------------------------------------
Commit 2f14fd29c0b47fc2c170a7f0eb1cecaf6b9704f4 in impala's branch
refs/heads/master from Csaba Ringhofer
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=2f14fd29c ]
IMPALA-12433: Share buffers among channels in KrpcDataStreamSender
Before this patch each KrpcDataStreamSender::Channel had 2
OutboundRowBatch with its own serialization and compression buffers.
This patch switches to use a single buffer per channel. This is
enough to store the in-flight data in KRPC, while other buffers
are only used during serialization and compression which is done for
just a single channel at a time, so can be shared among channels.
Memory estimates in the planner are not changed because the existing
calculation has several issues (see IMPALA-12594).
Change-Id: I64854a350a9dae8bf3af11c871882ea4750e60b3
Reviewed-on: http://gerrit.cloudera.org:8080/20719
Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Reviewed-by: Kurt Deschler <kdesc...@cloudera.com>
Reviewed-by: Zihao Ye <eyiz...@163.com>
Reviewed-by: Michael Smith <michael.sm...@cloudera.com>
> KrpcDataStreamSender could share some buffers between channels
> --------------------------------------------------------------
>
> Key: IMPALA-12433
> URL: https://issues.apache.org/jira/browse/IMPALA-12433
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
> Reporter: Csaba Ringhofer
> Priority: Major
> Labels: memory-saving, performance
>
> Currently each channel has two outbound row batches and each of those have 2
> buffers, one for serialization and another for compression.
> https://github.com/apache/impala/blob/0f55e551bc98843c79a9ec82582ddca237aa4fe9/be/src/runtime/row-batch.h#L100
> https://github.com/apache/impala/blob/0f55e551bc98843c79a9ec82582ddca237aa4fe9/be/src/runtime/krpc-data-stream-sender.cc#L236
> https://github.com/apache/impala/blob/0f55e551bc98843c79a9ec82582ddca237aa4fe9/fe/src/main/java/org/apache/impala/planner/DataStreamSink.java#L81
> As serialization + compression is always done from the fragment instance
> thread only one compression is done at a time, so a single compression buffer
> could be shared between channels. If this buffer is sent via KRPC then it
> could be swapped with the per channel buffer.
> As far as I understand at least one buffer per channel is needed because
> async KRPC calls can use it from another thread (this is done to avoid an
> extra copy of the buffer before RPCs). We can only reuse that buffer after
> getting a callback from KRPC.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
