[ https://issues.apache.org/jira/browse/IMPALA-8791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16905625#comment-16905625 ]
ASF subversion and git services commented on IMPALA-8791: --------------------------------------------------------- Commit 760169edcbca438c5964380a604b6c271c6bd1a3 in impala's branch refs/heads/master from Bikramjeet Vig [ https://gitbox.apache.org/repos/asf?p=impala.git;h=760169e ] IMPALA-8791: Handle the case where there is no fragment scheduled on the coordinator This patch fixes a bug where if an insert or CTAS query has no fragments scheduled on the coordinator and a mem limit is to be enforced on the query (either through query option or automatically through estimates) then the same limit is also applied to the coordinator backend even though it does not execute anything. Highlights: - coord_backend_mem_to_admit_/mem_limit will always refer to the memory to admit/limit for the coordinator regardless of which fragments are scheduled on it. - There will always be a BackendExecParams added for the coordinator because coordinator always spawns a QueryState object with a mem_tracker for tracking runtime filter mem and the result set cache. For the case where this BackendExecParams is empty (no instances scheduled) it would ensure that some minimal amount of memory is accounted for by the admission controller and the right mem limit is applied to the QueryState spawned by the coordinator - added changes to Coordinator and Coordinator::BackendState classes to handle an empty BackendExecParams object Testing: The following cases need to be tested where the kind of fragments schduled on the coordinator backend are: 1. Coordinator fragment + other exec fragments 2. Coordinator fragment only 3. other exec fragments only (eg. insert into values OR insert into select 1) 4. No fragments, but coordinator still creates a QueryState Case 1 is covered by tests working with non-dedicated coordinators. Rest are covered by test_mem_limit_dedicated_coordinator in test_admission_controller.py Change-Id: If5631fa1490d9612ffac3c4c4715348de47d6df2 Reviewed-on: http://gerrit.cloudera.org:8080/13992 Reviewed-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> > Handle the case where there is no fragment scheduled on the coordinator for a > query > ----------------------------------------------------------------------------------- > > Key: IMPALA-8791 > URL: https://issues.apache.org/jira/browse/IMPALA-8791 > Project: IMPALA > Issue Type: Bug > Affects Versions: Product Backlog > Reporter: Bikramjeet Vig > Assignee: Bikramjeet Vig > Priority: Major > > For insert statements executed on a dedicated coord, the fragments get > schduled only on executors but a query state object still gets started up on > the coord host with the coord_mem_limit. we end up with a situation where the > mem admitted is zero for the coord but the mem_reserved is non-zero which > would affect other admission decisions. > There is also a case where there is no coordinator fragment but the execution > fragment gets scheduled on the coord (eg. insert into <tbl> values....) for > this case, the mem admitted is per_backend_mem_limit_ but the mem limit > applied to the coord query state is coord_backend_mem_limit_ which again > causes a inconsistency. -- This message was sent by Atlassian JIRA (v7.6.14#76016) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org