Zoltán Borók-Nagy created IMPALA-15677:
------------------------------------------
Summary: Coordinator waits for every backend's final report before
returning the last row batch
Key: IMPALA-15677
URL: https://issues.apache.org/jira/browse/IMPALA-15677
Project: IMPALA
Issue Type: Improvement
Components: Backend
Reporter: Zoltán Borók-Nagy
When the coordinator's GetNext() reaches eos, it transitions to
RETURNED_RESULTS before returning. Coordinator::HandleExecStateTransition()
then cancels the backends with fire_and_forget=false, calls WaitForBackends()
for their final status reports, releases admission resources and calls
ComputeQuerySummary() (master 1a26e1f778). The client's last fetch waits for
all of that; the code already has "TODO: should move this off of the query
execution path?".
The cost grows with the instance count. At TPC-DS SF3000 on 8 executors
(MT_DOP=8), it is 211 ms for q77's 2,488 fragment instances, and the final
reports add about 1.8 s over the short queries of the workload.
*Prototype:* for RETURNED_RESULTS, the transition starts a thread that runs
CancelBackends(false), WaitForBackends(),
ReleaseQueryAdmissionControlResources(), the end time update and
ComputeQuerySummary(), and sets finalized_. The client gets eos immediately;
the Coordinator's destructor joins the thread.
Measured on the 103 TPC-DS queries at SF3000, against the same configuration
without it: 590.1 s -> 579.3 s (geomean 0.962). The 46 short queries got 7%
faster, and results were identical.
*Open points:*
* A client that asks for the profile right after eos must still get the
complete profile, so profile retrieval has to wait for finalization.
* Admission resources are released later relative to eos, so a client that
submits the next query right away sees more of IMPALA-?.
* DML and other statements that need the final reports before returning must
keep the synchronous path.
Prototype: https://gerrit.cloudera.org/#/c/25096/ (flag
--async_query_finalization, default true; queries only)
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
