[
https://issues.apache.org/jira/browse/IMPALA-13475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18084122#comment-18084122
]
ASF subversion and git services commented on IMPALA-13475:
----------------------------------------------------------
Commit 536f7c6ce8e64307d4da2bc8be1c41e3aab9f0b1 in impala's branch
refs/heads/master from Csaba Ringhofer
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=536f7c6ce ]
IMPALA-13475: Mem limit deferred RPC processing parallelism
Before this patch KrpcDataStreamRecvr::SenderQueue::GetBatch() didn't
consider the mem requirement of the batches when initiating the
deserialization of deferred RCPs (EnqueueDeserializeTask) and tried
to deserialize as many batches in parallel as possible, based
on FLAGS_datastream_service_num_deserialization_threads.
This meant that some of the awakened threads could be rejected due to
hitting mem limit. In this case the deferred RPC was put back in queue.
Besides wasting resources this could also lead to starving bigger RPCs
by rejecting them and letting smaller RPCs take over.
The patch introduces the concept of "pending deferred RPCs", which
are RPCs where processing was requested, but the deserialization
thread didn't pick up the RPCs yet. The deserialized size of pending
deferred RPCs is considered already taken and only N deserializations
are started if the first N RPC's payload would fit to memory (with
the exception of empty queue where at least 1 is started).
The state flow of deferred RPCs became the following where each
transition should be able to succeed:
1. no deserialization thread started (in deferred_rpcs_)
2. deserialization thread started (in pending_deferred_rpcs_)
3. currently deserialized (tracked in num_pending_enqueue_)
4. enqueued
Note that it is still possible for batches to take over in 3
if their deserialization is faster.
Merging exchanges need special handling as those had multiple
queues with shared mem limit before this patch. The patch splits
the mem limit evenly among the queues to allow them enforcing
the "respect mem limit, with the exception when empty" rule
individually, as at least one batch in queue per sender is
critical to allow the merge sort to progress. Mem limit of closed
queues is reused by counting the active queues in atomic int
'num_active_queues_' and using this to calculate the share of total
mem limit a queue can use. There is still some risk of perf regression
in skewed merging exchanges where one sender may "need" the queue
space more than others.
Some parts are generated by copilot + Claude Sonnet 4.6
Change-Id: I4d8e2b07a19e4d30210d7903c01e75fba632a1de
Reviewed-on: http://gerrit.cloudera.org:8080/22217
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Consider byte size when enqueuing deferred RPCs in KrpcDataStreamRecvr
> ----------------------------------------------------------------------
>
> Key: IMPALA-13475
> URL: https://issues.apache.org/jira/browse/IMPALA-13475
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
> Reporter: Csaba Ringhofer
> Priority: Critical
>
> KrpcDataStreamRecvr::SenderQueue::ProcessDeferredRpc() can fail to process
> the deferred RCP if batch_queue is not empty and the batch queue + the
> currently processed batch would consume too much memory (see
> KrpcDataStreamRecvr::CanEnqueue for details). The deferred RPC is moved back
> to the queue in this case.
> Meanwhile KrpcDataStreamRecvr::SenderQueue::GetBatch() doesn't consider the
> mem requirement of the batches when initiating the deserialization of
> deferred RCPs ( EnqueueDeserializeTask) and tries to deserialize as much
> batches in parallel as possible
> (FLAGS_datastream_service_num_deserialization_threads,
> https://github.com/apache/impala/blob/c83e5d97693fd3035b33622512d1584a5e56ce8b/be/src/runtime/krpc-data-stream-recvr.cc#L281).
> This means that several threads may start ProcessDeferredRpc() even if
> GetBatch() could have predicted that most will fail due to the memory limit.
> While this ProcessDeferredRpc() will fail early in this case and won't do
> much work, these extra failed attempts make
> lock contention worse in KrpcDataStreamRecvr::SenderQueue. In the worst case
> when only 1 batch fits to memory this can lead to
> O(FLAGS_datastream_service_num_deserialization_threads * num_batches) wasted
> ProcessDeferredRpc attempts.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
