Hello Quanlong Huang, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24248
to look at the new patch set (#2).
Change subject: IMPALA-14847: Add diagnostics for large ExecQueryFInstances RPCs
......................................................................
IMPALA-14847: Add diagnostics for large ExecQueryFInstances RPCs
For some complex queries, the ExecQueryFInstances RPCs can
be very large, sometimes hundreds of MBs. This can cause
high utilization of a coordinator's NIC and slow other
queries. This adds a few different diagnostics to help
track down the queries that have large ExecQueryFInstances
RPCs:
1. It prints a warning about ExecQueryFInstances RPCs that
exceed a threshold (set by --large_execquery_rpc_threshold_bytes,
which defaults to 25MB). The warning provides information
about the different pieces of the message as well as
the number of fragments and scan ranges. Here are sample
outputs for different scenarios (with a 512KB threshold):
TPC-DS Q67 mt_dop=4
W20260501 11:44:23.116255 3332252 coordinator-backend-state.cc:367]
5b4d96c7f2203c2f:09d20eb900000000] Large ExecQueryFInstances RPC:
Total: 533.48 KB Shared TQueryCtx: 393.91 KB (390.82 KB from descriptor
table)
TExecPlanFragmentInfo: 75.46 KB ExecQueryFInstancesRequestPB: 64.11 KB
num fragments: 6 num fragment instances: 15 num scan ranges: 608
Insert with large VALUES clause (about 166KB of strings inserted)
W20260501 11:53:01.178194 3334079 coordinator-backend-state.cc:367]
424ab3ed64824bfa:e7425a3900000000] Large ExecQueryFInstances RPC:
Total: 632.54 KB Shared TQueryCtx: 372.08 KB (633.00 B from descriptor
table)
TExecPlanFragmentInfo: 260.43 KB ExecQueryFInstancesRequestPB: 29.00 B
num fragments: 1 num fragment instances: 1 num scan ranges: 0
Query with lots of expressions from query_test/test_exprs.py TestExprLimit
W20260501 15:54:28.689013 4094389 coordinator-backend-state.cc:376]
754bb6df6e866a6c:e308911300000000] Large ExecQueryFInstances RPC:
Total: 33.94 MB Shared TQueryCtx: 3.72 MB (7.84 KB from descriptor table)
TExecPlanFragmentInfo: 30.22 MB ExecQueryFInstancesRequestPB: 967.00 B
num fragments: 2 num fragment instances: 2 num scan ranges: 8
2. It adds a ExecQueryRPCSizes summary stats counter to the
profile. This provides output like:
ExecQueryRPCSizes: (Avg: 534.65 KB (547486) ; Min: 533.48 KB (546281) ;
Max: 537.00 KB (549890) ; Sum: 1.57 MB (1642460) ; Number of samples: 3)
3. This also includes the RPC size in the JSON used for
the WebUI's /query_backends page.
Change-Id: I10e1fd50bf2fec2b8969585f7fcec2ca2487e181
---
M be/src/rpc/sidecar-util.h
M be/src/runtime/coordinator-backend-state.cc
M be/src/runtime/coordinator-backend-state.h
M be/src/runtime/coordinator.cc
M be/src/runtime/coordinator.h
M be/src/scheduling/admission-control-service.cc
M tests/webserver/test_web_pages.py
7 files changed, 86 insertions(+), 14 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/48/24248/2
--
To view, visit http://gerrit.cloudera.org:8080/24248
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I10e1fd50bf2fec2b8969585f7fcec2ca2487e181
Gerrit-Change-Number: 24248
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Joe McDonnell <[email protected]>
Gerrit-Reviewer: Quanlong Huang <[email protected]>
