Henry Robinson has uploaded a new change for review. http://gerrit.cloudera.org:8080/3618
Change subject: IMPALA-3610: Account for memory used by filters in the coordinator ...................................................................... IMPALA-3610: Account for memory used by filters in the coordinator Before this patch, Impala would not account for the memory used to aggregate runtime filters together in the coordinator. Impala's memory could therefore be silently overcommitted. This patch accounts for aggregated filter memory in the coordinator's query_mem_tracker(). If the query memory limit is exceeded when a filter update arrives, that update is discarded. If the filter is from a partitioned join, the entire filter can therefore be discarded immediately (to alleviate memory pressure) and a dummy 'always true' filter is sent to backends to unblock them. If the filter is from a broadcast join, the filter is not sent since an update from some other backend might arrive at a time of less memory pressure and allow the filter to be aggregated. The exception is when handling the last update to a broadcast join filter, when the filter should again be discarded. Memory that is added to a memtracker must always be released. To do this, we need to signal to the coordinator that it is finished, and that there is no point trying to process any future updates that might arrive concurrently. This patch adds Coordinator::Done() which is called from QueryExecState::Done(), and which releases memory from all in-process runtime filters. Finally, this patch increases the upper limit for runtime filters to 512MB. This allows testing on very large datasets. The default maximum is still 16MB, per RUNTIME_FILTER_MAX_SIZE. Testing: Added a new test that triggers the OOM condition on the coordinator. All existing runtime filter tests pass. Change-Id: I3c52c8a1c2e79ef370c77bf264885fc859678d1b --- M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h M be/src/runtime/mem-tracker.h M be/src/runtime/runtime-filter-bank.h M be/src/service/impala-server.h M be/src/service/query-exec-state.cc M testdata/workloads/functional-query/queries/QueryTest/runtime_row_filters_phj.test 7 files changed, 113 insertions(+), 26 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/18/3618/1 -- To view, visit http://gerrit.cloudera.org:8080/3618 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I3c52c8a1c2e79ef370c77bf264885fc859678d1b Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: cdh5-trunk Gerrit-Owner: Henry Robinson <[email protected]>
