Tim Armstrong has submitted this change and it was merged. Change subject: IMPALA-3238: avoid log spam for very large hash tables ......................................................................
IMPALA-3238: avoid log spam for very large hash tables Before this change, every time the limit was hit in ConsumeMemory() was hit, a warning message with stack trace was logged. The streaming preagg calls this once per partition per batch, resulting in an enormous volume of logging. The fix is to log the warning only once for each client (i.e. each exec node). We need a very large data set to reproduce this error. To test it manually I lowered the threshold to 1 << 19, ran 'select distinct l_orderkey from tpch_20_parquet', and confirmed that only two warning messages per query appeared in the logs (one per agg node). Change-Id: I366ce1148e83ac2eb71535d552bc68cb08388eec Reviewed-on: http://gerrit.cloudera.org:8080/2659 Reviewed-by: Tim Armstrong <[email protected]> Tested-by: Tim Armstrong <[email protected]> --- M be/src/runtime/buffered-block-mgr.cc 1 file changed, 13 insertions(+), 3 deletions(-) Approvals: Tim Armstrong: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/2659 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: merged Gerrit-Change-Id: I366ce1148e83ac2eb71535d552bc68cb08388eec Gerrit-PatchSet: 3 Gerrit-Project: Impala Gerrit-Branch: cdh5-trunk Gerrit-Owner: Tim Armstrong <[email protected]> Gerrit-Reviewer: Dan Hecht <[email protected]> Gerrit-Reviewer: Henry Robinson <[email protected]> Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Tim Armstrong <[email protected]> Gerrit-Reviewer: Todd Lipcon <[email protected]>
