Supposed to be a draft; will post the final patch in a short while. On 11 October 2016 at 16:44, Henry Robinson (Code Review) < [email protected]> wrote:
> Henry Robinson has uploaded a new patch set (#11). > > Change subject: IMPALA-2905: Handle coordinator fragment lifecycle like > all others > ...................................................................... > > IMPALA-2905: Handle coordinator fragment lifecycle like all others > > The plan-root fragment instance that runs on the coordinator should be > handled like all others: started via RPC and run asynchronously. Without > this, the fragment requires special-case code throughout the > coordinator, and does not show up in system metrics etc. > > This patch adds a new sink type, PlanRootSink, to the root fragment > instance so that the coordinator can pull row batches that are pushed by > the root instance. The coordinator signals completion to the fragment > instance via closing the consumer side of the sink, whereupon the > instance is free to complete. > > Since the root instance now runs asynchronously wrt to the coordinator, > we add several coordination methods to allow the coordinator to wait for > a point in the instance's execution to be hit - e.g. to wait until the > instance has been opened. > > Done in this patch: > > * Add PlanRootSink > * Add coordination to PFE to allow coordinator to observe lifecycle > * Make FragmentMgr a singleton > * Removed dead code from Coordinator::Wait() and elsewhere. > * Moved result output exprs out of QES and into PlanRootSink. > * Remove special-case limit-based teardown of coordinator fragment, and > supporting functions in PlanFragmentExecutor. > * Simplified lifecycle of PlanFragmentExecutor by separating Open() into > OpenPlan() and Exec(), the latter of which drives the sink by reading > rows from the plan tree. > * Add child profile to PlanFragmentExecutor to measure time spent in > each lifecycle phase. > * Removed dependency between InitExecProfiles() and starting root fragment. > > Not yet done: > > * Fix planner tests to reflect new sink added at root. > > Change-Id: Ibb0064ec2f085fa3a5598ea80894fb489a01e4df > --- > M be/src/exec/CMakeLists.txt > M be/src/exec/data-sink.cc > A be/src/exec/plan-root-sink.cc > A be/src/exec/plan-root-sink.h > M be/src/runtime/coordinator.cc > M be/src/runtime/coordinator.h > M be/src/runtime/exec-env.cc > M be/src/runtime/exec-env.h > M be/src/runtime/plan-fragment-executor.cc > M be/src/runtime/plan-fragment-executor.h > M be/src/scheduling/query-schedule.cc > M be/src/scheduling/query-schedule.h > M be/src/scheduling/simple-scheduler.cc > M be/src/service/fragment-exec-state.cc > M be/src/service/fragment-exec-state.h > M be/src/service/fragment-mgr.cc > M be/src/service/impala-beeswax-server.cc > M be/src/service/impala-hs2-server.cc > M be/src/service/impala-internal-service.h > M be/src/service/impala-server.cc > M be/src/service/impala-server.h > M be/src/service/query-exec-state.cc > M be/src/service/query-exec-state.h > A be/src/service/query-result-set.h > M be/src/testutil/in-process-servers.cc > M common/thrift/DataSinks.thrift > M fe/src/main/java/org/apache/impala/analysis/QueryStmt.java > A fe/src/main/java/org/apache/impala/planner/PlanRootSink.java > M fe/src/main/java/org/apache/impala/planner/Planner.java > M fe/src/main/java/org/apache/impala/planner/PlannerContext.java > M tests/failure/test_failpoints.py > M tests/hs2/test_json_endpoints.py > 32 files changed, 1,011 insertions(+), 898 deletions(-) > > > git pull ssh://gerrit.cloudera.org:29418/Impala-ASF > refs/changes/02/4402/11 > -- > To view, visit http://gerrit.cloudera.org:8080/4402 > To unsubscribe, visit http://gerrit.cloudera.org:8080/settings > > Gerrit-MessageType: newpatchset > Gerrit-Change-Id: Ibb0064ec2f085fa3a5598ea80894fb489a01e4df > Gerrit-PatchSet: 11 > Gerrit-Project: Impala-ASF > Gerrit-Branch: master > Gerrit-Owner: Henry Robinson <[email protected]> > Gerrit-Reviewer: Alex Behm <[email protected]> > Gerrit-Reviewer: Henry Robinson <[email protected]> > Gerrit-Reviewer: Marcel Kornacker <[email protected]> > Gerrit-Reviewer: Tim Armstrong <[email protected]> > > -- > You received this message because you are subscribed to the Google Groups > "impala-cr" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/a/cloudera.com/d/optout. > -- Henry Robinson Software Engineer Cloudera 415-994-6679
