Hello Quanlong Huang, Csaba Ringhofer, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24248
to look at the new patch set (#6).
Change subject: IMPALA-14847: Add diagnostics for large ExecQueryFInstances RPCs
......................................................................
IMPALA-14847: Add diagnostics for large ExecQueryFInstances RPCs
For some complex queries, the ExecQueryFInstances RPCs can
be very large, sometimes hundreds of MBs. This can cause
high utilization of a coordinator's NIC and slow other
queries. This adds a few different diagnostics to help
track down the queries that have large ExecQueryFInstances
RPCs:
1. It prints a warning when a query has ExecQueryFInstances RPCs
that exceed a threshold (set by --large_execquery_rpc_threshold_bytes,
which defaults to 25MB). The warning includes a query level
summary of the number of RPCs that exceeded the threshold and
the average size of the ExecQueryFInstances RPCs for this query.
It also includes information about the size of the shared TQueryCtx
piece of the message.
It also prints information about the top three RPCs to backends
for this query. This contains detail about the sizes of
backend-specific pieces of the message as well as the number
of fragments and scan ranges.
Here are some outputs for different scenarios (with a 512KB threshold):
TPC-DS Q67 mt_dop=4:
ExecQueryFInstances RPCs for 3 out of 3 exceed the warning threshold
of 512.00 KB. Average RPC size: 534.64 KB Shared TQueryCtx: 393.91 KB
(390.82 KB from descriptor table)
Additional information about the 3 largest RPCs:
Destination: joemcdonnell-22743:27000 Total size: 536.98 KB
Backend-specific size: 143.08 KB TExecPlanFragmentInfo: 77.81 KB
ExecQueryFInstancesRequestPB: 65.27 KB num fragments: 10
num fragment instances: 19 num scan ranges: 611
...
Insert with large VALUES clause (about 166KB of strings inserted)
W20260501 11:44:23.116255 3332252 coordinator-backend-state.cc:367]
ExecQueryFInstances RPCs for 1 out of 1 exceed the warning threshold
of 512.00 KB. Average RPC size: 632.54 KB Shared TQueryCtx: 372.08 KB
(633.00 B from descriptor table)
Additional information about the 1 largest RPCs:
Destination: joemcdonnell-22743:27000 Total size: 632.54 KB
Backend-specific size: 260.46 KB TExecPlanFragmentInfo: 260.43 KB
ExecQueryFInstancesRequestPB: 29.00 B num fragments: 1
num fragment instances: 1 num scan ranges: 0
Query with lots of expressions from query_test/test_exprs.py TestExprLimit
ExecQueryFInstances RPCs for 3 out of 3 exceed the warning threshold
of 512.00 KB. Average RPC size: 13.79 MB Shared TQueryCtx: 3.71 MB
(7.84 KB from descriptor table)
Additional information about the 3 largest RPCs:
Destination: joemcdonnell-22743:27000 Total size: 33.94 MB
Backend-specific size: 30.22 MB TExecPlanFragmentInfo: 30.22 MB
ExecQueryFInstancesRequestPB: 965.00 B num fragments: 2
num fragment instances: 2 num scan ranges: 8
...
2. It adds a ExecQueryRPCSizes summary stats counter to the
profile. This provides output like:
ExecQueryRPCSizes: (Avg: 534.65 KB (547486) ; Min: 533.48 KB (546281) ;
Max: 537.00 KB (549890) ; Sum: 1.57 MB (1642460) ; Number of samples: 3)
3. This also includes the RPC size in the JSON used for
the WebUI's /query_backends page.
Change-Id: I10e1fd50bf2fec2b8969585f7fcec2ca2487e181
---
M be/src/rpc/sidecar-util.h
M be/src/runtime/coordinator-backend-state.cc
M be/src/runtime/coordinator-backend-state.h
M be/src/runtime/coordinator.cc
M be/src/runtime/coordinator.h
M be/src/scheduling/admission-control-service.cc
M tests/webserver/test_web_pages.py
7 files changed, 195 insertions(+), 14 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/48/24248/6
--
To view, visit http://gerrit.cloudera.org:8080/24248
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I10e1fd50bf2fec2b8969585f7fcec2ca2487e181
Gerrit-Change-Number: 24248
Gerrit-PatchSet: 6
Gerrit-Owner: Joe McDonnell <[email protected]>
Gerrit-Reviewer: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Joe McDonnell <[email protected]>
Gerrit-Reviewer: Michael Smith <[email protected]>
Gerrit-Reviewer: Quanlong Huang <[email protected]>
