Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/16723 )
Change subject: IMPALA-10314: Optimize planning time for simple limits ...................................................................... Patch Set 16: > Patch Set 15: > > > Patch Set 9: > > > > > Patch Set 8: > > > > > > > Patch Set 8: > > > > > > > > > Patch Set 8: > > > > > > > > > > > Patch Set 8: Verified-1 > > > > > > > > > > > > Build failed: > > > > > > https://jenkins.impala.io/job/gerrit-verify-dryrun/6687/ > > > > > > > > > > There's 1 failure: > > > > > [gw1] FAILED > > > > > query_test/test_exprs.py::TestExprLimits::test_statement_expression_limit > > > > > > > > > > However, on my desktop I ran query_test/test_exprs.py and all tests > > > > > under it passed. > > > > > > > > On Jenkins this test hit an OOM GC overhead limit exceeded: > > > > gw1] linux2 -- Python 2.7.16 > > > > /home/ubuntu/Impala/bin/../infra/python/env-gcc7.5.0/bin/python > > > > query_test/test_exprs.py:176: in test_statement_expression_limit > > > > assert re.search(expected_err_re, str(err)) > > > > E assert None > > > > E + where None = <function search at 0x7fc9ff90e848>('Exceeded the > > > > statement expression limit \\(250000\\)\nStatement has .* > > > > expressions.', "ImpalaBeeswaxException:\n INNER EXCEPTION: <class > > > > 'beeswaxd.ttypes.BeeswaxException'>\n MESSAGE: OutOfMemoryError: GC > > > > overhead limit exceeded\n") > > > > E + where <function search at 0x7fc9ff90e848> = re.search > > > > E + and "ImpalaBeeswaxException:\n INNER EXCEPTION: <class > > > > 'beeswaxd.ttypes.BeeswaxException'>\n MESSAGE: OutOfMemoryError: GC > > > > overhead limit exceeded\n" = str(ImpalaBeeswaxException()) > > > > > > Oh wait, there's actually another failure in FE: > > > > > > 23:47:52 [INFO] Running org.apache.impala.analysis.ParserTest > > > 23:47:52 [ERROR] Tests run: 98, Failures: 1, Errors: 0, Skipped: 0, Time > > > elapsed: 0.829 s <<< FAILURE! - in org.apache.impala.analysis.ParserTest > > > 23:47:52 [ERROR] TestGetErrorMsg(org.apache.impala.analysis.ParserTest) > > > Time elapsed: 0.005 s <<< FAILURE! > > > at > > > org.apache.impala.analysis.ParserTest.ParserError(ParserTest.java:77) > > > at > > > org.apache.impala.analysis.ParserTest.TestGetErrorMsg(ParserTest.java:3475) > > > > > > Will need to look into this one. > > > > PS9 fixes the ParserTest issue..just had to update the list of expected > > keywords in the test after a WHERE clause. Not yet sure about the other > > OOM failure since I cannot repro locally. > > One difference between my machine's execution vs the one on Jenkins is that > the Jenkins one runs it under a dockerised container and the JVM Max heap is > 1.78GB which is smaller than the 2GB on my machine. > > From ubuntu-16.04-dockerised-tests-3539/logs/ee_tests/impalad_coord_exec-0: > impalad.4069f4f171ef.ubuntu.log.INFO.20201126-001441.1: JVM: max heap size: > Total=1.78 GB > impalad.4069f4f171ef.ubuntu.log.INFO.20201126-001441.1: JVM: non-heap > committed: Total=114.50 MB > > I looked closer at the 'GC overhead limit exceeded' issue with the > test_statement_expression_limit() test and I suspect it happens because in my > original patch the default construction of the Expr object also allocates an > empty ArrayList for hints (this was consistent with how we handle other types > of hints). However, this is not really necessary and it can be done on > demand only when an actual WHERE clause hint is supplied. This particular > test creates a huge number of Exprs and considering that the Dockerised run > has a slightly smaller max heap, it might cause the GC to run more often > without reclaiming sufficient free memory. > > In PS 15 I have made the change to allocate the arraylist only when a > predicate hint was supplied. Hopefully it addresses the issue. It looks like the test_statement_expression_limit passed in the latest run on PS 15. There were couple of load failures which seem like existing flaky test issue especially https://issues.apache.org/jira/browse/IMPALA-10344 (although the same error occurs for a different table). I will start another run with a +2 carry-over. -- To view, visit http://gerrit.cloudera.org:8080/16723 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9d6a79263bc092e0f3e9a1d72da5618f3cc35574 Gerrit-Change-Number: 16723 Gerrit-PatchSet: 16 Gerrit-Owner: Aman Sinha <amsi...@cloudera.com> Gerrit-Reviewer: Aman Sinha <amsi...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com> Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com> Gerrit-Reviewer: Tim Armstrong <tarmstr...@cloudera.com> Gerrit-Comment-Date: Sat, 28 Nov 2020 01:48:47 +0000 Gerrit-HasComments: No
