[Impala-ASF-CR] IMPALA-7614: [DOCS] Document the New Invalidate Options
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/11809 ) Change subject: IMPALA-7614: [DOCS] Document the New Invalidate Options .. Patch Set 3: (3 comments) http://gerrit.cloudera.org:8080/#/c/11809/3/docs/topics/impala_config_options.xml File docs/topics/impala_config_options.xml: http://gerrit.cloudera.org:8080/#/c/11809/3/docs/topics/impala_config_options.xml@355 PS3, Line 355: small bounded http://gerrit.cloudera.org:8080/#/c/11809/3/docs/topics/impala_config_options.xml@365 PS3, Line 365: Java garbage collection-based simplify to: Memory-based http://gerrit.cloudera.org:8080/#/c/11809/3/docs/topics/impala_config_options.xml@373 PS3, Line 373: but the feature could potentially : cause performance risks do we have standardized phrasing around memory knobs? something like, "may require tuning". -- To view, visit http://gerrit.cloudera.org:8080/11809 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I40c552eeaee81ee6528d9f725bd416b51d8ab837 Gerrit-Change-Number: 11809 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Adrian Ng (389) Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 31 Oct 2018 05:57:49 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7786: Start HMS and Sentry in debug mode in the minicluster
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11826 ) Change subject: IMPALA-7786: Start HMS and Sentry in debug mode in the minicluster .. Patch Set 5: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/3384/ -- To view, visit http://gerrit.cloudera.org:8080/11826 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I29b025cbde36ef398ea36fbe69eff26e27d93e48 Gerrit-Change-Number: 11826 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Wed, 31 Oct 2018 05:56:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7794: Rewrite flaky ownership authorization tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11837 ) Change subject: IMPALA-7794: Rewrite flaky ownership authorization tests .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1226/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11837 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic98f8dbec41360261fd0339d835f3ce6b504ee29 Gerrit-Change-Number: 11837 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 31 Oct 2018 04:43:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7363: Fix buffer usage in ScannerContext::Stream::ReadVLong()
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11828 ) Change subject: IMPALA-7363: Fix buffer usage in ScannerContext::Stream::ReadVLong() .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3385/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/11828 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic2b2ffe4b1d67c63cfcea2baedbeff48e65ca417 Gerrit-Change-Number: 11828 Gerrit-PatchSet: 4 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 31 Oct 2018 04:21:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7363: Fix buffer usage in ScannerContext::Stream::ReadVLong()
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11828 ) Change subject: IMPALA-7363: Fix buffer usage in ScannerContext::Stream::ReadVLong() .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11828 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic2b2ffe4b1d67c63cfcea2baedbeff48e65ca417 Gerrit-Change-Number: 11828 Gerrit-PatchSet: 4 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 31 Oct 2018 04:21:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7794: Rewrite flaky ownership authorization tests
Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11837 Change subject: IMPALA-7794: Rewrite flaky ownership authorization tests .. IMPALA-7794: Rewrite flaky ownership authorization tests This patch rewrites the ownership authorization tests to not depend on delay and timeout, which can help to avoid the flakiness. The patch also refactors some tests to reuse Sentry and Impala instances without having to restart them, which can speed up the tests. To keep the same test coverage, no tests were removed. Testing: - Ran all authorization E2E tests multiple times. Change-Id: Ic98f8dbec41360261fd0339d835f3ce6b504ee29 --- M tests/authorization/test_grant_revoke.py M tests/authorization/test_owner_privileges.py M tests/common/sentry_cache_test_suite.py 3 files changed, 298 insertions(+), 412 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/37/11837/3 -- To view, visit http://gerrit.cloudera.org:8080/11837 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ic98f8dbec41360261fd0339d835f3ce6b504ee29 Gerrit-Change-Number: 11837 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya
[Impala-ASF-CR] IMPALA-7363: Fix buffer usage in ScannerContext::Stream::ReadVLong()
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11828 ) Change subject: IMPALA-7363: Fix buffer usage in ScannerContext::Stream::ReadVLong() .. Patch Set 3: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/3382/ -- To view, visit http://gerrit.cloudera.org:8080/11828 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic2b2ffe4b1d67c63cfcea2baedbeff48e65ca417 Gerrit-Change-Number: 11828 Gerrit-PatchSet: 3 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 31 Oct 2018 04:04:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7655: Rewrite if, isnull, coalesce to use CASE
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11760 ) Change subject: IMPALA-7655: Rewrite if, isnull, coalesce to use CASE .. Patch Set 10: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1225/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11760 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I526654d8546e17b2545c42cc59dab66d9fe1b163 Gerrit-Change-Number: 11760 Gerrit-PatchSet: 10 Gerrit-Owner: Paul Rogers Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 31 Oct 2018 02:53:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7655: Rewrite if, isnull, coalesce to use CASE
Paul Rogers has posted comments on this change. ( http://gerrit.cloudera.org:8080/11760 ) Change subject: IMPALA-7655: Rewrite if, isnull, coalesce to use CASE .. Patch Set 7: Turns out that there is a BE bug that means CASE is not equivalent to coalesce(). Disabled all coalesce() rewrites and tests, restoring original behavior from master. Coalesce() can be revisited when IMPALA-7793 is fixed. -- To view, visit http://gerrit.cloudera.org:8080/11760 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I526654d8546e17b2545c42cc59dab66d9fe1b163 Gerrit-Change-Number: 11760 Gerrit-PatchSet: 7 Gerrit-Owner: Paul Rogers Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 31 Oct 2018 02:24:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7655: Rewrite if, isnull, coalesce to use CASE
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11760 ) Change subject: IMPALA-7655: Rewrite if, isnull, coalesce to use CASE .. Patch Set 10: (2 comments) http://gerrit.cloudera.org:8080/#/c/11760/10/fe/src/main/java/org/apache/impala/rewrite/SimplifyConditionalsRule.java File fe/src/main/java/org/apache/impala/rewrite/SimplifyConditionalsRule.java: http://gerrit.cloudera.org:8080/#/c/11760/10/fe/src/main/java/org/apache/impala/rewrite/SimplifyConditionalsRule.java@213 PS10, Line 213:* Simplify COALESCE by skipping leading nulls and applying the following transformations: line too long (92 > 90) http://gerrit.cloudera.org:8080/#/c/11760/10/fe/src/main/java/org/apache/impala/rewrite/SimplifyConditionalsRule.java@234 PS10, Line 234: List newChildren = Lists.newArrayList(expr.getChildren().subList(i, numChildren)); line too long (96 > 90) -- To view, visit http://gerrit.cloudera.org:8080/11760 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I526654d8546e17b2545c42cc59dab66d9fe1b163 Gerrit-Change-Number: 11760 Gerrit-PatchSet: 10 Gerrit-Owner: Paul Rogers Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 31 Oct 2018 02:22:52 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7655: Rewrite if, isnull, coalesce to use CASE
Hello Bharath Vissapragada, Philip Zeyliger, Tim Armstrong, Csaba Ringhofer, Impala Public Jenkins, Vuk Ercegovac, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11760 to look at the new patch set (#10). Change subject: IMPALA-7655: Rewrite if, isnull, coalesce to use CASE .. IMPALA-7655: Rewrite if, isnull, coalesce to use CASE See IMPALA-7655 for backgound. Tim found that the current interpreted forms of if, isnull and coalesce are slow compared to the code-generated CASE statement. This patch rewrites the above functions into the equivalent CASE structure. The rewrite engine has many bugs that are beyond the scope of this change to fix. This change codes around those bugs. The result is that the conditional rewrite happens some of the time, and sometimes produces less-than perfect optimizations. Conditionals in the top-level ORDER BY clause are not rewritten (IMPALA-7753), but those one or more levels down are. Some expressions involving NULL are not simplified (IMPALA-7769). Several hacks were used to work around the fact that the rewrite engine ignores unanalyzed expressions, yet the rewrite engine does not, in general, re-analyze the expressions it produces, causing simplifications to be skipped (IMPALA-7754). And so on. As a result, the BE retains the original interpreted forms that are still used in two cases: 1) top-level conditions in the ORDER BY clause, and 2) if the user disables rewrites. Further, code generation does not occur for CASE statements in the SELECT clause when it is in the root fragment (the most common case in simple tests.) This is another known bug (IMPALA-4356). One possible performance regression is that the new form of the code evaluates some expressions twice, where the original interpreted code evaluated the argument once. E.g. coalesce(id, 10) is rewritten to CASE WHEN id IS NULL THEN id ELSE 10 END. Here, id is evaluated twice. If the "id" were replaced by a complex sub-expression, the gain from compilation could be offset by doing work twice. (IMPALA-7737) Still, the fix provides most of what the JIRA ticket requested within the limitations of the existing code. Conditional function rewrites are moved into a new class, RewriteConditionalsRule in order to keep things simple. Most functions use the simplest possible rewrite, relying on the existing rewrite rules for further simplification. The one exception is coalesce(): the existing code relies on the semantics of the function and so was retained and slightly improved. The code was extended to produce a CASE statement directly, retaining existing simplifications. Tests for conditional functions were in one large function along with other rewrite tests. Moved them into a new file, then broke up the tests by function to allow much easier debugging of each function one-by-one. This required moving the common test mechanims into a new common base class. Existing tests focus on one or two rules at a time. The conditional function rewrite, however, relies on the entire set of rules being applied repeatedly. So, added a new FullRewriteTest case to verify this behavior. This class contains several commented-out tests that cannot pass due to existing rewrite bugs noted above. Changing the rewrite cause the PlannerTest to produce different plans than previously. Changed the expected results file to match the new rewrite rules. Change-Id: I526654d8546e17b2545c42cc59dab66d9fe1b163 --- M be/src/exprs/conditional-functions.h M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java A fe/src/main/java/org/apache/impala/rewrite/RewriteConditionalFnsRule.java M fe/src/main/java/org/apache/impala/rewrite/SimplifyConditionalsRule.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeExprsTest.java A fe/src/test/java/org/apache/impala/analysis/BaseRewriteRulesTest.java M fe/src/test/java/org/apache/impala/analysis/ExprRewriteRulesTest.java A fe/src/test/java/org/apache/impala/analysis/FullRewriteTest.java A fe/src/test/java/org/apache/impala/analysis/RewriteConditionalFnsRuleTest.java M testdata/workloads/functional-planner/queries/PlannerTest/aggregation.test M testdata/workloads/functional-planner/queries/PlannerTest/analytic-fns.test M testdata/workloads/functional-planner/queries/PlannerTest/implicit-joins.test M testdata/workloads/functional-planner/queries/PlannerTest/joins.test M testdata/workloads/functional-planner/queries/PlannerTest/nested-collections.test M testdata/workloads/functional-planner/queries/PlannerTest/outer-joins.test M testdata/workloads/functional-planner/queries/PlannerTest/predicate-propagation.test M testdata/workloads/functional-planner/queries/PlannerTest/runtime-filter-propagation.test 18 files changed, 1,014 insertions(+), 282 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF
[Impala-ASF-CR] IMPALA-7786: Start HMS and Sentry in debug mode in the minicluster
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11826 ) Change subject: IMPALA-7786: Start HMS and Sentry in debug mode in the minicluster .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11826 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I29b025cbde36ef398ea36fbe69eff26e27d93e48 Gerrit-Change-Number: 11826 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Wed, 31 Oct 2018 01:54:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7786: Start HMS and Sentry in debug mode in the minicluster
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11826 ) Change subject: IMPALA-7786: Start HMS and Sentry in debug mode in the minicluster .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3384/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/11826 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I29b025cbde36ef398ea36fbe69eff26e27d93e48 Gerrit-Change-Number: 11826 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Wed, 31 Oct 2018 01:54:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7786: Start HMS and Sentry in debug mode in the minicluster
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/11826 ) Change subject: IMPALA-7786: Start HMS and Sentry in debug mode in the minicluster .. Patch Set 4: I'm not sure what happened here: 00:55:59 ] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.18:test (default-test) on project impala-frontend: Execution default-test of goal org.apache.maven.plugins:maven-surefire-plugin:2.18:test failed: The forked VM terminated without properly saying goodbye. VM crash or System.exit called? 00:55:59 ] [ERROR] Command was /bin/sh -c cd /home/ubuntu/Impala/fe && /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Djava.library.path=/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64:/usr/lib/jvm/java-8-openjdk-amd64/jre/../lib/amd64:/usr/lib/x86_64-linux-gnu/:/usr/lib/x86_64-linux-gnu/:/usr/lib/x86_64-linux-gnu/:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server:/home/ubuntu/Impala/toolchain/cdh_components-663310/hadoop-3.0.0-cdh6.x-SNAPSHOT//lib/native:/home/ubuntu/Impala/toolchain/snappy-1.1.4/lib:/home/ubuntu/Impala/../Impala-lzo/build:/home/ubuntu/Impala/toolchain/gcc-4.9.2/lib64:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server:/home/ubuntu/Impala/toolchain/cdh_components-663310/hadoop-3.0.0-cdh6.x-SNAPSHOT//lib/native:/home/ubuntu/Impala/toolchain/snappy-1.1.4/lib:/home/ubuntu/Impala/../Impala-lzo/build:/home/ubuntu/Impala/toolchain/gcc-4.9.2/lib64:/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib:/home/ubuntu/Impala/be/build/debug/service:/home/ubuntu/Impala/be/build/release/service -jar /home/ubuntu/Impala/logs/surefire/surefirebooter574563636434827.jar /home/ubuntu/Impala/logs/surefire/surefire7967668628670205081tmp /home/ubuntu/Impala/logs/surefire/surefire_07807361286211485039tmp Going to retry. -- To view, visit http://gerrit.cloudera.org:8080/11826 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I29b025cbde36ef398ea36fbe69eff26e27d93e48 Gerrit-Change-Number: 11826 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Wed, 31 Oct 2018 01:54:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7614: [DOCS] Document the New Invalidate Options
Adrian Ng has posted comments on this change. ( http://gerrit.cloudera.org:8080/11809 ) Change subject: IMPALA-7614: [DOCS] Document the New Invalidate Options .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/11809/3/docs/topics/impala_config_options.xml File docs/topics/impala_config_options.xml: http://gerrit.cloudera.org:8080/#/c/11809/3/docs/topics/impala_config_options.xml@363 PS3, Line 363: specified time period. T in seconds. -- To view, visit http://gerrit.cloudera.org:8080/11809 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I40c552eeaee81ee6528d9f725bd416b51d8ab837 Gerrit-Change-Number: 11809 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Adrian Ng (389) Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Comment-Date: Wed, 31 Oct 2018 01:23:54 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7614: [DOCS] Document the New Invalidate Options
Adrian Ng has posted comments on this change. ( http://gerrit.cloudera.org:8080/11809 ) Change subject: IMPALA-7614: [DOCS] Document the New Invalidate Options .. Patch Set 3: (2 comments) http://gerrit.cloudera.org:8080/#/c/11809/2/docs/topics/impala_config_options.xml File docs/topics/impala_config_options.xml: http://gerrit.cloudera.org:8080/#/c/11809/2/docs/topics/impala_config_options.xml@367 PS2, Line 367: is high We can say "reaches 60% of JVM heap size" to be specific. http://gerrit.cloudera.org:8080/#/c/11809/2/docs/topics/impala_config_options.xml@368 PS2, Line 368: certain We can say invalidates 10% of the least recently used tables. -- To view, visit http://gerrit.cloudera.org:8080/11809 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I40c552eeaee81ee6528d9f725bd416b51d8ab837 Gerrit-Change-Number: 11809 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Adrian Ng (389) Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Comment-Date: Wed, 31 Oct 2018 01:14:59 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7655: Rewrite if, isnull, coalesce to use CASE
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11760 ) Change subject: IMPALA-7655: Rewrite if, isnull, coalesce to use CASE .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1224/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11760 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I526654d8546e17b2545c42cc59dab66d9fe1b163 Gerrit-Change-Number: 11760 Gerrit-PatchSet: 9 Gerrit-Owner: Paul Rogers Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 31 Oct 2018 01:10:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7614: [DOCS] Document the New Invalidate Options
Alex Rodoni has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11809 ) Change subject: IMPALA-7614: [DOCS] Document the New Invalidate Options .. IMPALA-7614: [DOCS] Document the New Invalidate Options --invalidate_tables_timeout_s --invalidate_tables_on_memory_pressure Change-Id: I40c552eeaee81ee6528d9f725bd416b51d8ab837 Reviewed-on: http://gerrit.cloudera.org:8080/11809 Reviewed-by: Tianyi Wang Tested-by: Impala Public Jenkins --- M docs/topics/impala_config_options.xml 1 file changed, 35 insertions(+), 34 deletions(-) Approvals: Tianyi Wang: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/11809 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I40c552eeaee81ee6528d9f725bd416b51d8ab837 Gerrit-Change-Number: 11809 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Adrian Ng (389) Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang
[Impala-ASF-CR] IMPALA-7765: [DOCS] Document IMPALA MAX MEM ESTIMATE FOR ADMISSION option
Alex Rodoni has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11804 ) Change subject: IMPALA-7765: [DOCS] Document IMPALA_MAX_MEM_ESTIMATE_FOR_ADMISSION option .. IMPALA-7765: [DOCS] Document IMPALA_MAX_MEM_ESTIMATE_FOR_ADMISSION option Change-Id: Ibef89c98530c6974dc791666cc51c1ded52e7910 Reviewed-on: http://gerrit.cloudera.org:8080/11804 Tested-by: Impala Public Jenkins Reviewed-by: Tim Armstrong --- M docs/impala.ditamap A docs/topics/impala_max_mem_estimate_for_admission.xml 2 files changed, 90 insertions(+), 0 deletions(-) Approvals: Impala Public Jenkins: Verified Tim Armstrong: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/11804 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ibef89c98530c6974dc791666cc51c1ded52e7910 Gerrit-Change-Number: 11804 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-2566: Remove ability to access impala builtin cast functions directly
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11830 ) Change subject: IMPALA-2566: Remove ability to access impala builtin cast functions directly .. Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/11830/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11830/1//COMMIT_MSG@15 PS1, Line 15: It's weird that we're still showing all these functions in _impala_builtins but you can't call any of them. I.e. it doesn't feel like the right way to fix it. I get that's probably a more invasive change but it feels like trading one kind of wonkiness for another - maybe it's better just to fix the narrow casttochar() issue for now? I guess maybe this is easier for now than adding tests for all of the callable casttochar* variants (which would be the right thing to do if we left them in there) but we should file a follow-on to fix the wonkiness at least. http://gerrit.cloudera.org:8080/#/c/11830/1/fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java File fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java: http://gerrit.cloudera.org:8080/#/c/11830/1/fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java@500 PS1, Line 500: Impala builtin cast functions as reserved for internal use only > Open to suggestions for a error message Maybe something like: "_the_function() is reserved for internal use only. Use 'cast(expr AS type)' instead." -- To view, visit http://gerrit.cloudera.org:8080/11830 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6f562607aaaf728fc417066565811bbd269baaa8 Gerrit-Change-Number: 11830 Gerrit-PatchSet: 1 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 31 Oct 2018 01:03:17 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7614: [DOCS] Document the New Invalidate Options
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11809 ) Change subject: IMPALA-7614: [DOCS] Document the New Invalidate Options .. Patch Set 2: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/126/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/11809 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I40c552eeaee81ee6528d9f725bd416b51d8ab837 Gerrit-Change-Number: 11809 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Adrian Ng (389) Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Comment-Date: Wed, 31 Oct 2018 00:59:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7786: Start HMS and Sentry in debug mode in the minicluster
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11826 ) Change subject: IMPALA-7786: Start HMS and Sentry in debug mode in the minicluster .. Patch Set 4: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/3380/ -- To view, visit http://gerrit.cloudera.org:8080/11826 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I29b025cbde36ef398ea36fbe69eff26e27d93e48 Gerrit-Change-Number: 11826 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Wed, 31 Oct 2018 00:56:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7765: [DOCS] Document IMPALA MAX MEM ESTIMATE FOR ADMISSION option
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11804 ) Change subject: IMPALA-7765: [DOCS] Document IMPALA_MAX_MEM_ESTIMATE_FOR_ADMISSION option .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11804 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibef89c98530c6974dc791666cc51c1ded52e7910 Gerrit-Change-Number: 11804 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 31 Oct 2018 00:52:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7614: [DOCS] Document the New Invalidate Options
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/11809 ) Change subject: IMPALA-7614: [DOCS] Document the New Invalidate Options .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11809 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I40c552eeaee81ee6528d9f725bd416b51d8ab837 Gerrit-Change-Number: 11809 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Adrian Ng (389) Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Comment-Date: Wed, 31 Oct 2018 00:49:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7614: [DOCS] Document the New Invalidate Options
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11809 ) Change subject: IMPALA-7614: [DOCS] Document the New Invalidate Options .. Patch Set 2: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/126/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/11809 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I40c552eeaee81ee6528d9f725bd416b51d8ab837 Gerrit-Change-Number: 11809 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Adrian Ng (389) Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Comment-Date: Wed, 31 Oct 2018 00:48:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7614: [DOCS] Document the New Invalidate Options
Hello Tianyi Wang, Adrian Ng, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11809 to look at the new patch set (#2). Change subject: IMPALA-7614: [DOCS] Document the New Invalidate Options .. IMPALA-7614: [DOCS] Document the New Invalidate Options --invalidate_tables_timeout_s --invalidate_tables_on_memory_pressure Change-Id: I40c552eeaee81ee6528d9f725bd416b51d8ab837 --- M docs/topics/impala_config_options.xml 1 file changed, 35 insertions(+), 34 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/09/11809/2 -- To view, visit http://gerrit.cloudera.org:8080/11809 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I40c552eeaee81ee6528d9f725bd416b51d8ab837 Gerrit-Change-Number: 11809 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Adrian Ng (389) Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang
[Impala-ASF-CR] IMPALA-7614: [DOCS] Document the New Invalidate Options
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/11809 ) Change subject: IMPALA-7614: [DOCS] Document the New Invalidate Options .. Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/11809/1/docs/topics/impala_config_options.xml File docs/topics/impala_config_options.xml: http://gerrit.cloudera.org:8080/#/c/11809/1/docs/topics/impala_config_options.xml@356 PS1, Line 356: invalidate > invalidates Done http://gerrit.cloudera.org:8080/#/c/11809/1/docs/topics/impala_config_options.xml@366 PS1, Line 366: When : the size of the garbage collection of catalogd : exceeds a certain threshold, Impala invalidates a certain fraction of : the least recently used tables. > "size of the garbage collection" isn't accurate, how about: Done -- To view, visit http://gerrit.cloudera.org:8080/11809 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I40c552eeaee81ee6528d9f725bd416b51d8ab837 Gerrit-Change-Number: 11809 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Adrian Ng (389) Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Comment-Date: Wed, 31 Oct 2018 00:48:18 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7363: Fix buffer usage in ScannerContext::Stream::ReadVLong()
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11828 ) Change subject: IMPALA-7363: Fix buffer usage in ScannerContext::Stream::ReadVLong() .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1223/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11828 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic2b2ffe4b1d67c63cfcea2baedbeff48e65ca417 Gerrit-Change-Number: 11828 Gerrit-PatchSet: 2 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 31 Oct 2018 00:43:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7614: [DOCS] Document the New Invalidate Options
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/11809 ) Change subject: IMPALA-7614: [DOCS] Document the New Invalidate Options .. Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/11809/1/docs/topics/impala_config_options.xml File docs/topics/impala_config_options.xml: http://gerrit.cloudera.org:8080/#/c/11809/1/docs/topics/impala_config_options.xml@356 PS1, Line 356: invalidate invalidates http://gerrit.cloudera.org:8080/#/c/11809/1/docs/topics/impala_config_options.xml@366 PS1, Line 366: When : the size of the garbage collection of catalogd : exceeds a certain threshold, Impala invalidates a certain fraction of : the least recently used tables. "size of the garbage collection" isn't accurate, how about: "When the memory pressure is high after a Java garbage collection in catalogd, Impala invalidates ... " -- To view, visit http://gerrit.cloudera.org:8080/11809 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I40c552eeaee81ee6528d9f725bd416b51d8ab837 Gerrit-Change-Number: 11809 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Adrian Ng (389) Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Comment-Date: Wed, 31 Oct 2018 00:42:31 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7655: Rewrite if, isnull, coalesce to use CASE
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/11760 ) Change subject: IMPALA-7655: Rewrite if, isnull, coalesce to use CASE .. Patch Set 9: (10 comments) still reviewing the tests. http://gerrit.cloudera.org:8080/#/c/11760/8//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11760/8//COMMIT_MSG@43 PS8, Line 43: (IMPALA-7737) are there examples of these fns in our benchmarks to quantify the regression? if so, would be useful to see the effect. http://gerrit.cloudera.org:8080/#/c/11760/8//COMMIT_MSG@45 PS8, Line 45: Still, the fix provides most of what the JIRA ticket requested I'd skip these next three paragraphs. http://gerrit.cloudera.org:8080/#/c/11760/8//COMMIT_MSG@57 PS8, Line 57: pls make a section for this called "Testing" so its easier to jump to. also, pls condense these so that they're easier to skim. for example: - split up tests for conditional functions to make them easier to test - added unit tests for end-to-end rewrite rule interactions - updated existing planner tests due to rewrites http://gerrit.cloudera.org:8080/#/c/11760/8/be/src/exprs/conditional-functions.h File be/src/exprs/conditional-functions.h: http://gerrit.cloudera.org:8080/#/c/11760/8/be/src/exprs/conditional-functions.h@76 PS8, Line 76: since : /// various bugs mean that this implementation is still sometimes used. But : /// the goal is to remove these classes at some point. simpler: "until their use is eliminated by the frontend". http://gerrit.cloudera.org:8080/#/c/11760/8/fe/src/main/java/org/apache/impala/rewrite/RewriteConditionalFnsRule.java File fe/src/main/java/org/apache/impala/rewrite/RewriteConditionalFnsRule.java: http://gerrit.cloudera.org:8080/#/c/11760/8/fe/src/main/java/org/apache/impala/rewrite/RewriteConditionalFnsRule.java@37 PS8, Line 37: vanish is this accurate given the comments in the commit message about order by? http://gerrit.cloudera.org:8080/#/c/11760/8/fe/src/main/java/org/apache/impala/rewrite/RewriteConditionalFnsRule.java@48 PS8, Line 48: planner runs the rule to simplify CASE : * after this rule. Where that other rule can perform simplifications, : * those simplifications are omitted here simplify and use the specific rule name for concreteness. http://gerrit.cloudera.org:8080/#/c/11760/8/fe/src/main/java/org/apache/impala/rewrite/RewriteConditionalFnsRule.java@106 PS8, Line 106: return rewriteIfNullFn(expr) clarify whether you think this happens after the rewrite or before. If its after, then I expect the example on L109,110 to be in terms of CASE. I'm also fine with omitting the example since its assumed that these rules compose. http://gerrit.cloudera.org:8080/#/c/11760/8/fe/src/main/java/org/apache/impala/rewrite/RewriteConditionalFnsRule.java@130 PS8, Line 130:expr.get isn't this all that's done here (the most general case) and we'll depend on other rewrites for further simplifications? http://gerrit.cloudera.org:8080/#/c/11760/8/fe/src/main/java/org/apache/impala/rewrite/RewriteConditionalFnsRule.java@172 PS8, Line 172: The simplest rewrite here would be to not look at the child exprs for the various scenarios listed above and instead simply translate naively to a case statement. From there, we'd get constant folding and case simplification which will find the first when clause that evals to true. preceding when clauses that remain unknown will be retained, but this transform will need to retain them as well. Aggregate handling will result in a brute-force roll-back of the rewrite in case simplification, which will result in falling back to the case rewrite here. Might want to handle that situation by retaining the coalesce for now. So besides that issue, what else do we miss by doing the simple thing and rely on case simplification? http://gerrit.cloudera.org:8080/#/c/11760/8/testdata/workloads/functional-planner/queries/PlannerTest/predicate-propagation.test File testdata/workloads/functional-planner/queries/PlannerTest/predicate-propagation.test: http://gerrit.cloudera.org:8080/#/c/11760/8/testdata/workloads/functional-planner/queries/PlannerTest/predicate-propagation.test@1137 PS8, Line 1137: | other predicates: functional.alltypestiny.tinyint_col + functional.alltypestiny.smallint_col + functional.alltypestiny.int_col > 10, CASE WHEN functional.alltypestiny.tinyint_col + functional.alltypestiny.bigint_col IS NULL THEN 1 ELSE functional.alltypestiny.tinyint_col + functional.alltypestiny.bigint_col END = 1 so this is the example of the performance regression (same work on multiple when clauses)? -- To view, visit http://gerrit.cloudera.org:8080/11760 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I526654d8546e17b2545c42cc59dab66d9fe1b163
[Impala-ASF-CR] IMPALA-7655: Rewrite if, isnull, coalesce to use CASE
Hello Bharath Vissapragada, Philip Zeyliger, Tim Armstrong, Csaba Ringhofer, Impala Public Jenkins, Vuk Ercegovac, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11760 to look at the new patch set (#9). Change subject: IMPALA-7655: Rewrite if, isnull, coalesce to use CASE .. IMPALA-7655: Rewrite if, isnull, coalesce to use CASE See IMPALA-7655 for backgound. Tim found that the current interpreted forms of if, isnull and coalesce are slow compared to the code-generated CASE statement. This patch rewrites the above functions into the equivalent CASE structure. The rewrite engine has many bugs that are beyond the scope of this change to fix. This change codes around those bugs. The result is that the conditional rewrite happens some of the time, and sometimes produces less-than perfect optimizations. Conditionals in the top-level ORDER BY clause are not rewritten (IMPALA-7753), but those one or more levels down are. Some expressions involving NULL are not simplified (IMPALA-7769). Several hacks were used to work around the fact that the rewrite engine ignores unanalyzed expressions, yet the rewrite engine does not, in general, re-analyze the expressions it produces, causing simplifications to be skipped (IMPALA-7754). And so on. As a result, the BE retains the original interpreted forms that are still used in two cases: 1) top-level conditions in the ORDER BY clause, and 2) if the user disables rewrites. Further, code generation does not occur for CASE statements in the SELECT clause when it is in the root fragment (the most common case in simple tests.) This is another known bug (IMPALA-4356). One possible performance regression is that the new form of the code evaluates some expressions twice, where the original interpreted code evaluated the argument once. E.g. coalesce(id, 10) is rewritten to CASE WHEN id IS NULL THEN id ELSE 10 END. Here, id is evaluated twice. If the "id" were replaced by a complex sub-expression, the gain from compilation could be offset by doing work twice. (IMPALA-7737) Still, the fix provides most of what the JIRA ticket requested within the limitations of the existing code. Conditional function rewrites are moved into a new class, RewriteConditionalsRule in order to keep things simple. Most functions use the simplest possible rewrite, relying on the existing rewrite rules for further simplification. The one exception is coalesce(): the existing code relies on the semantics of the function and so was retained and slightly improved. The code was extended to produce a CASE statement directly, retaining existing simplifications. Tests for conditional functions were in one large function along with other rewrite tests. Moved them into a new file, then broke up the tests by function to allow much easier debugging of each function one-by-one. This required moving the common test mechanims into a new common base class. Existing tests focus on one or two rules at a time. The conditional function rewrite, however, relies on the entire set of rules being applied repeatedly. So, added a new FullRewriteTest case to verify this behavior. This class contains several commented-out tests that cannot pass due to existing rewrite bugs noted above. Changing the rewrite cause the PlannerTest to produce different plans than previously. Changed the expected results file to match the new rewrite rules. Change-Id: I526654d8546e17b2545c42cc59dab66d9fe1b163 --- M be/src/exprs/conditional-functions.h M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java A fe/src/main/java/org/apache/impala/rewrite/RewriteConditionalFnsRule.java M fe/src/main/java/org/apache/impala/rewrite/SimplifyConditionalsRule.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeExprsTest.java A fe/src/test/java/org/apache/impala/analysis/BaseRewriteRulesTest.java M fe/src/test/java/org/apache/impala/analysis/ExprRewriteRulesTest.java A fe/src/test/java/org/apache/impala/analysis/FullRewriteTest.java A fe/src/test/java/org/apache/impala/analysis/RewriteConditionalFnsRuleTest.java M testdata/workloads/functional-planner/queries/PlannerTest/aggregation.test M testdata/workloads/functional-planner/queries/PlannerTest/analytic-fns.test M testdata/workloads/functional-planner/queries/PlannerTest/implicit-joins.test M testdata/workloads/functional-planner/queries/PlannerTest/joins.test M testdata/workloads/functional-planner/queries/PlannerTest/nested-collections.test M testdata/workloads/functional-planner/queries/PlannerTest/outer-joins.test M testdata/workloads/functional-planner/queries/PlannerTest/predicate-propagation.test M testdata/workloads/functional-planner/queries/PlannerTest/runtime-filter-propagation.test 18 files changed, 946 insertions(+), 313 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF
[Impala-ASF-CR] IMPALA-7743: [DOCS] A new option to load incremental statistics from catalog
Alex Rodoni has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11790 ) Change subject: IMPALA-7743: [DOCS] A new option to load incremental statistics from catalog .. IMPALA-7743: [DOCS] A new option to load incremental statistics from catalog --pull_incremental_statistics described in the Incremental Stats section. Change-Id: I8fd9b88138350406065df2f39a48043178759949 Reviewed-on: http://gerrit.cloudera.org:8080/11790 Reviewed-by: Greg Rahn Tested-by: Impala Public Jenkins --- M docs/shared/impala_common.xml M docs/topics/impala_perf_stats.xml 2 files changed, 74 insertions(+), 46 deletions(-) Approvals: Greg Rahn: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/11790 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I8fd9b88138350406065df2f39a48043178759949 Gerrit-Change-Number: 11790 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Adrian Ng (389) Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7363: Fix buffer usage in ScannerContext::Stream::ReadVLong()
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11828 ) Change subject: IMPALA-7363: Fix buffer usage in ScannerContext::Stream::ReadVLong() .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3382/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/11828 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic2b2ffe4b1d67c63cfcea2baedbeff48e65ca417 Gerrit-Change-Number: 11828 Gerrit-PatchSet: 3 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 31 Oct 2018 00:10:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7363: Fix buffer usage in ScannerContext::Stream::ReadVLong()
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11828 ) Change subject: IMPALA-7363: Fix buffer usage in ScannerContext::Stream::ReadVLong() .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11828 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic2b2ffe4b1d67c63cfcea2baedbeff48e65ca417 Gerrit-Change-Number: 11828 Gerrit-PatchSet: 2 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 31 Oct 2018 00:10:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7363: Fix buffer usage in ScannerContext::Stream::ReadVLong()
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11828 ) Change subject: IMPALA-7363: Fix buffer usage in ScannerContext::Stream::ReadVLong() .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11828 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic2b2ffe4b1d67c63cfcea2baedbeff48e65ca417 Gerrit-Change-Number: 11828 Gerrit-PatchSet: 3 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 31 Oct 2018 00:10:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7687: [DOCS] Support for multiple DISTINCT in a query
Alex Rodoni has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11823 ) Change subject: IMPALA-7687: [DOCS] Support for multiple DISTINCT in a query .. IMPALA-7687: [DOCS] Support for multiple DISTINCT in a query - Removed notes about the single DISTINCT restriction. - Rewrote the description for the APPX_COUNT_DISTINCT query option. Change-Id: I3a6e664b016e9408a3ff809f1811253a91764481 Reviewed-on: http://gerrit.cloudera.org:8080/11823 Tested-by: Impala Public Jenkins Reviewed-by: Thomas Marshall --- M docs/shared/impala_common.xml M docs/topics/impala_appx_count_distinct.xml M docs/topics/impala_count.xml M docs/topics/impala_distinct.xml M docs/topics/impala_langref_unsupported.xml M docs/topics/impala_select.xml 6 files changed, 33 insertions(+), 111 deletions(-) Approvals: Impala Public Jenkins: Verified Thomas Marshall: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/11823 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I3a6e664b016e9408a3ff809f1811253a91764481 Gerrit-Change-Number: 11823 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7363: Fix buffer usage in ScannerContext::Stream::ReadVLong()
Pooja Nilangekar has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/11828 ) Change subject: IMPALA-7363: Fix buffer usage in ScannerContext::Stream::ReadVLong() .. IMPALA-7363: Fix buffer usage in ScannerContext::Stream::ReadVLong() Previously, the ReadVLong() function would read a byte and use it after subsequent calls to ReadBytes() function. As a result, the value was invalidated depending on the contents of the boundary_buffer_. This change ensures that all the functions on the buffer are invoked before subsequent calls to ReadBytes(). Testing: Enabled test_tpch_scan_ranges() for sequence files and ran it multiple times to ensure it always produces the desired result. Manually verified the buffer usage of other calls to ReadBytes() and GetBytes(). Change-Id: Ic2b2ffe4b1d67c63cfcea2baedbeff48e65ca417 --- M be/src/exec/scanner-context.h M be/src/exec/scanner-context.inline.h M tests/query_test/test_scanners.py 3 files changed, 11 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/28/11828/2 -- To view, visit http://gerrit.cloudera.org:8080/11828 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic2b2ffe4b1d67c63cfcea2baedbeff48e65ca417 Gerrit-Change-Number: 11828 Gerrit-PatchSet: 2 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7363: Fix buffer usage in ScannerContext::Stream::ReadVLong()
Pooja Nilangekar has posted comments on this change. ( http://gerrit.cloudera.org:8080/11828 ) Change subject: IMPALA-7363: Fix buffer usage in ScannerContext::Stream::ReadVLong() .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/11828/1/be/src/exec/scanner-context.inline.h File be/src/exec/scanner-context.inline.h: http://gerrit.cloudera.org:8080/#/c/11828/1/be/src/exec/scanner-context.inline.h@169 PS1, Line 169: if (is_negative) *value = *value ^ (static_cast(-1)); > nit: could put conditional on one line. Done -- To view, visit http://gerrit.cloudera.org:8080/11828 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic2b2ffe4b1d67c63cfcea2baedbeff48e65ca417 Gerrit-Change-Number: 11828 Gerrit-PatchSet: 2 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 30 Oct 2018 23:54:20 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7363: Fix buffer usage in ScannerContext::Stream::ReadVLong()
Tim Armstrong has posted comments on this change. (
http://gerrit.cloudera.org:8080/11828 )
Change subject: IMPALA-7363: Fix buffer usage in
ScannerContext::Stream::ReadVLong()
..
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/11828/1/be/src/exec/scanner-context.inline.h
File be/src/exec/scanner-context.inline.h:
http://gerrit.cloudera.org:8080/#/c/11828/1/be/src/exec/scanner-context.inline.h@169
PS1, Line 169: if (is_negative) {
nit: could put conditional on one line.
--
To view, visit http://gerrit.cloudera.org:8080/11828
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2b2ffe4b1d67c63cfcea2baedbeff48e65ca417
Gerrit-Change-Number: 11828
Gerrit-PatchSet: 1
Gerrit-Owner: Pooja Nilangekar
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Pooja Nilangekar
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Tue, 30 Oct 2018 23:47:59 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6658: improve Parquet RLE for low bit widths
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11582 ) Change subject: IMPALA-6658: improve Parquet RLE for low bit widths .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1222/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11582 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I191a581d3f699b6669e48ac9dc39c76ed77c4a76 Gerrit-Change-Number: 11582 Gerrit-PatchSet: 7 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 30 Oct 2018 23:16:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6658: improve Parquet RLE for low bit widths
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11582 ) Change subject: IMPALA-6658: improve Parquet RLE for low bit widths .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1221/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11582 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I191a581d3f699b6669e48ac9dc39c76ed77c4a76 Gerrit-Change-Number: 11582 Gerrit-PatchSet: 6 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 30 Oct 2018 23:11:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6658: improve Parquet RLE for low bit widths
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11582 ) Change subject: IMPALA-6658: improve Parquet RLE for low bit widths .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1220/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11582 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I191a581d3f699b6669e48ac9dc39c76ed77c4a76 Gerrit-Change-Number: 11582 Gerrit-PatchSet: 5 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 30 Oct 2018 23:05:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-2566: Remove ability to access impala builtin cast functions directly
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11830 ) Change subject: IMPALA-2566: Remove ability to access impala builtin cast functions directly .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1219/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11830 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6f562607aaaf728fc417066565811bbd269baaa8 Gerrit-Change-Number: 11830 Gerrit-PatchSet: 1 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 30 Oct 2018 22:58:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7687: [DOCS] Support for multiple DISTINCT in a query
Thomas Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/11823 ) Change subject: IMPALA-7687: [DOCS] Support for multiple DISTINCT in a query .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11823 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3a6e664b016e9408a3ff809f1811253a91764481 Gerrit-Change-Number: 11823 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Tue, 30 Oct 2018 22:57:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7363: Fix buffer usage in ScannerContext::Stream::ReadVLong()
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11828 ) Change subject: IMPALA-7363: Fix buffer usage in ScannerContext::Stream::ReadVLong() .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1218/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11828 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic2b2ffe4b1d67c63cfcea2baedbeff48e65ca417 Gerrit-Change-Number: 11828 Gerrit-PatchSet: 1 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 30 Oct 2018 22:51:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6658: improve Parquet RLE for low bit widths
Andrew Sherman has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/11582 ) Change subject: IMPALA-6658: improve Parquet RLE for low bit widths .. IMPALA-6658: improve Parquet RLE for low bit widths RleEncoder buffers values in its own cache to detect run lengths that can be efficiently encoded. When a run is detected it is written with an indicator byte which encodes the length of the run. So an encoded run always has an overhead of at least one byte. This means that for single bit values, encoding 8 values as a run is inefficient. Change RleEncoder to have the ability to use run lengths other than 8. A new parameter to the constructor (min_run_length) allows test callers (only) to set the minimum run length. By default RleEncoder will now use run length encoding for runs of length 16 for single bit values. All other bit widths will use the existing length 8 runs. Internally RleEncoder must buffer more values so that the longer runs can be detected. The internal buffer “buffered_values_” is larger and is now a circular buffer so that the first 8 bytes of the buffer can be separately flushed to BitWriter. Testing: All end-to-end and unit tests pass The unit test rle-test is enhanced to run all tests against RleEncoders using all possible values of min_run_length. In Addition, rle-test is refactored so that the Rle tests are in a class that inherits from ::testing::Test so that a SetUp() method can be used. The Overflow test is enhanced to be more exhaustive (while still completing in a second or two). Change-Id: I191a581d3f699b6669e48ac9dc39c76ed77c4a76 --- M be/src/util/rle-encoding.h M be/src/util/rle-test.cc 2 files changed, 499 insertions(+), 255 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/11582/7 -- To view, visit http://gerrit.cloudera.org:8080/11582 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I191a581d3f699b6669e48ac9dc39c76ed77c4a76 Gerrit-Change-Number: 11582 Gerrit-PatchSet: 7 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall
[Impala-ASF-CR] IMPALA-6658: improve Parquet RLE for low bit widths
Andrew Sherman has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/11582 ) Change subject: IMPALA-6658: improve Parquet RLE for low bit widths .. IMPALA-6658: improve Parquet RLE for low bit widths RleEncoder buffers values in its own cache to detect run lengths that can be efficiently encoded. When a run is detected it is written with an indicator byte which encodes the length of the run. So an encoded run always has an overhead of at least one byte. This means that for single bit values, encoding 8 values as a run is inefficient. Change RleEncoder to have the ability to use run lengths other than 8. A new parameter to the constructor (min_run_length) allows test callers (only) to set the minimum run length. By default RleEncoder will now use run length encoding for runs of length 16 for single bit values. All other bit widths will use the existing length 8 runs. Internally RleEncoder must buffer more values so that the longer runs can be detected. The internal buffer “buffered_values_” is larger and is now a circular buffer so that the first 8 bytes of the buffer can be separately flushed to BitWriter. Testing: All end-to-end and unit tests pass The unit test rle-test is enhanced to run all tests against RleEncoders using all possible values of min_run_length. In Addition, rle-test is refactored so that the Rle tests are in a class that inherits from ::testing::Test so that a SetUp() method can be used. The Overflow test is enhanced to be more exhaustive (while still completing in a second or two). Change-Id: I191a581d3f699b6669e48ac9dc39c76ed77c4a76 --- M be/src/util/rle-encoding.h M be/src/util/rle-test.cc 2 files changed, 484 insertions(+), 256 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/11582/6 -- To view, visit http://gerrit.cloudera.org:8080/11582 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I191a581d3f699b6669e48ac9dc39c76ed77c4a76 Gerrit-Change-Number: 11582 Gerrit-PatchSet: 6 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall
[Impala-ASF-CR] IMPALA-6658: improve Parquet RLE for low bit widths
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/11582 ) Change subject: IMPALA-6658: improve Parquet RLE for low bit widths .. Patch Set 2: (4 comments) Thanks Csaba http://gerrit.cloudera.org:8080/#/c/11582/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11582/2//COMMIT_MSG@20 PS2, Line 20: By default RleEncoder will now use run length encoding for runs of : length 24 for single bit values, and of length 16 for 2 bit wide values. : All other bit widths will use the existing length 8 runs. > Ok, the biggest win (50% size for alternating runs of 8 in the 1 bit case) So you want me to write something like this in the jira? If you know the structure of the data then better encodings are possible. For example with bit_width=1, using min_run_length=24 is better in the case where we avoid interrupting a literal run. Using the notation of 'RXX' for a repeated run of length XX (so R16 is a run of lngth 16), and 'LYY' for a literal run of length YY. L24 R16 L24 R16 L24 min_run_length 8 4 2 4 2 4 min_run_length 16 (new default) 4 2 4 2 4 min_run_length 244 2 3 2 3 (one long literal run) So it is possible to optimize by detecting this situation and avoiding breaking a long literal run for a run of length 16. http://gerrit.cloudera.org:8080/#/c/11582/4/be/src/util/rle-encoding.h File be/src/util/rle-encoding.h: http://gerrit.cloudera.org:8080/#/c/11582/4/be/src/util/rle-encoding.h@250 PS4, Line 250: iter > Can you simplify this expression? MAX_VALUES_PER_LITERAL_RUN must be divisi Good idea http://gerrit.cloudera.org:8080/#/c/11582/4/be/src/util/rle-encoding.h@264 PS4, Line 264: insertin > Aren't we double counting the indicator byte here? My assumption is that th I think this is correct as is but I am changing it to the clearer literal_max_size = num_runs * (1 + bytes_per_run) -- 1 is the indicator -- bytes_per_run is the encoded bytes http://gerrit.cloudera.org:8080/#/c/11582/4/be/src/util/rle-encoding.h@265 PS4, Line 265: / 1 byte for the encoded run length > This seems to assume that a single big literal run is the worsts case - can I think before this change a long literal was not always the worst case, but now it is. -- To view, visit http://gerrit.cloudera.org:8080/11582 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I191a581d3f699b6669e48ac9dc39c76ed77c4a76 Gerrit-Change-Number: 11582 Gerrit-PatchSet: 2 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 30 Oct 2018 22:22:19 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6658: improve Parquet RLE for low bit widths
Andrew Sherman has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/11582 ) Change subject: IMPALA-6658: improve Parquet RLE for low bit widths .. IMPALA-6658: improve Parquet RLE for low bit widths RleEncoder buffers values in its own cache to detect run lengths that can be efficiently encoded. When a run is detected it is written with an indicator byte which encodes the length of the run. So an encoded run always has an overhead of at least one byte. This means that for single bit values, encoding 8 values as a run is inefficient. Change RleEncoder to have the ability to use run lengths other than 8. A new parameter to the constructor (min_run_length) allows test callers (only) to set the minimum run length. By default RleEncoder will now use run length encoding for runs of length 16 for single bit values. All other bit widths will use the existing length 8 runs. Internally RleEncoder must buffer more values so that the longer runs can be detected. The internal buffer “buffered_values_” is larger and is now a circular buffer so that the first 8 bytes of the buffer can be separately flushed to BitWriter. Testing: All end-to-end and unit tests pass The unit test rle-test is enhanced to run all tests against RleEncoders using all possible values of min_run_length. In Addition, rle-test is refactored so that the Rle tests are in a class that inherits from ::testing::Test so that a SetUp() method can be used. The Overflow test is enhanced to be more exhaustive (while still completing in a second or two). Change-Id: I191a581d3f699b6669e48ac9dc39c76ed77c4a76 --- M be/src/util/rle-encoding.h M be/src/util/rle-test.cc 2 files changed, 510 insertions(+), 255 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/11582/5 -- To view, visit http://gerrit.cloudera.org:8080/11582 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I191a581d3f699b6669e48ac9dc39c76ed77c4a76 Gerrit-Change-Number: 11582 Gerrit-PatchSet: 5 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall
[Impala-ASF-CR] IMPALA-2566: Remove ability to access impala builtin cast functions directly
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11830 ) Change subject: IMPALA-2566: Remove ability to access impala builtin cast functions directly .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/11830/1/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java File fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java: http://gerrit.cloudera.org:8080/#/c/11830/1/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java@3962 PS1, Line 3962: String expectedError = "Impala builtin cast functions as reserved for internal use only"; line too long (93 > 90) -- To view, visit http://gerrit.cloudera.org:8080/11830 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6f562607aaaf728fc417066565811bbd269baaa8 Gerrit-Change-Number: 11830 Gerrit-PatchSet: 1 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 30 Oct 2018 22:20:30 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-2566: Remove ability to access impala builtin cast functions directly
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/11830 ) Change subject: IMPALA-2566: Remove ability to access impala builtin cast functions directly .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/11830/1/fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java File fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java: http://gerrit.cloudera.org:8080/#/c/11830/1/fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java@500 PS1, Line 500: Impala builtin cast functions as reserved for internal use only Open to suggestions for a error message -- To view, visit http://gerrit.cloudera.org:8080/11830 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6f562607aaaf728fc417066565811bbd269baaa8 Gerrit-Change-Number: 11830 Gerrit-PatchSet: 1 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Comment-Date: Tue, 30 Oct 2018 22:20:15 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-2566: Remove ability to access impala builtin cast functions directly
Bikramjeet Vig has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11830 Change subject: IMPALA-2566: Remove ability to access impala builtin cast functions directly .. IMPALA-2566: Remove ability to access impala builtin cast functions directly Currently, all impala builtin cast functions are exposed to the user and can be used directly in sql statements. These methods are used internally for casting and sometimes require context not available when used directly. This patch removes direct access to those methods by throwing an analysis exception. Testing: Added relevant frontend tests. Change-Id: I6f562607aaaf728fc417066565811bbd269baaa8 --- M fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java 2 files changed, 26 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/30/11830/1 -- To view, visit http://gerrit.cloudera.org:8080/11830 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I6f562607aaaf728fc417066565811bbd269baaa8 Gerrit-Change-Number: 11830 Gerrit-PatchSet: 1 Gerrit-Owner: Bikramjeet Vig
[Impala-ASF-CR] IMPALA-6374: fix handling of commas in .test files
Tim Armstrong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11800 ) Change subject: IMPALA-6374: fix handling of commas in .test files .. IMPALA-6374: fix handling of commas in .test files The .test file parser implemented an unconventional method for parsing single-quoted strings in comma-separated value format. This didn't handle trailing commas in the string correctly. This commit switches to using a conventional method for parsing comma-separated value format: * Commas enclosed by single quotes are not treated as field separators * Single quotes can be escaped within a string by doubling them. I looked into using Python's .csv module for this, but it wouldn't work without modifying the test file format more because it automatically discards the quotes during parsing, which are actually semantically important in .test files. E.g. without the quotes we can't distinguish between the literal string 'regex:...' and the regex regex: Testing: Ran exhaustive tests and fixed .test files that required modifications. Will rerun before merging. Added a couple of tests to exercise edge cases in the test file parser. Change-Id: I18ddcb0440490ddf8184be66d3681038a1615dd9 Reviewed-on: http://gerrit.cloudera.org:8080/11800 Reviewed-by: Impala Public Jenkins Tested-by: Tim Armstrong --- M testdata/workloads/functional-query/queries/QueryTest/functions-ddl.test M testdata/workloads/functional-query/queries/QueryTest/grant_revoke.test M testdata/workloads/functional-query/queries/QueryTest/hbase-inserts.test M testdata/workloads/functional-query/queries/QueryTest/misc.test A testdata/workloads/functional-query/queries/QueryTest/special-strings.test M testdata/workloads/functional-query/queries/QueryTest/stats-extrapolation.test M testdata/workloads/tpcds/queries/tpcds-decimal_v2-q98.test M testdata/workloads/tpcds/queries/tpcds-q98.test M tests/common/test_result_verifier.py M tests/query_test/test_exprs.py 10 files changed, 157 insertions(+), 112 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved Tim Armstrong: Verified -- To view, visit http://gerrit.cloudera.org:8080/11800 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I18ddcb0440490ddf8184be66d3681038a1615dd9 Gerrit-Change-Number: 11800 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-6374: fix handling of commas in .test files
Tim Armstrong has removed a vote on this change. Change subject: IMPALA-6374: fix handling of commas in .test files .. Removed Verified-1 by Impala Public Jenkins -- To view, visit http://gerrit.cloudera.org:8080/11800 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: deleteVote Gerrit-Change-Id: I18ddcb0440490ddf8184be66d3681038a1615dd9 Gerrit-Change-Number: 11800 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-6374: fix handling of commas in .test files
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11800 ) Change subject: IMPALA-6374: fix handling of commas in .test files .. Patch Set 5: Verified+1 Reran the flaky test: https://jenkins.impala.io/job/python26-incompatibility-check/538/ -- To view, visit http://gerrit.cloudera.org:8080/11800 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I18ddcb0440490ddf8184be66d3681038a1615dd9 Gerrit-Change-Number: 11800 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 30 Oct 2018 22:17:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7363: Fix buffer usage in ScannerContext::Stream::ReadVLong()
Pooja Nilangekar has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11828 Change subject: IMPALA-7363: Fix buffer usage in ScannerContext::Stream::ReadVLong() .. IMPALA-7363: Fix buffer usage in ScannerContext::Stream::ReadVLong() Previously, the ReadVLong() function would read a byte and use it after subsequent calls to ReadBytes() function. As a result, the value was invalidated depending on the contents of the boundary_buffer_. This change ensures that all the functions on the buffer are invoked before subsequent calls to ReadBytes(). Testing: Enabled test_tpch_scan_ranges() for sequence files and ran it multiple times to ensure it always produces the desired result. Manually verified the buffer usage of other calls to ReadBytes() and GetBytes(). Change-Id: Ic2b2ffe4b1d67c63cfcea2baedbeff48e65ca417 --- M be/src/exec/scanner-context.h M be/src/exec/scanner-context.inline.h M tests/query_test/test_scanners.py 3 files changed, 10 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/28/11828/1 -- To view, visit http://gerrit.cloudera.org:8080/11828 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ic2b2ffe4b1d67c63cfcea2baedbeff48e65ca417 Gerrit-Change-Number: 11828 Gerrit-PatchSet: 1 Gerrit-Owner: Pooja Nilangekar
[Impala-ASF-CR] IMPALA-6742: Profiles of running queries should include execution summary.
Yongjun Zhang has posted comments on this change. ( http://gerrit.cloudera.org:8080/11591 ) Change subject: IMPALA-6742: Profiles of running queries should include execution summary. .. Patch Set 2: (3 comments) http://gerrit.cloudera.org:8080/#/c/11591/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11591/1//COMMIT_MSG@9 PS1, Line 9: Currently execution summary is not included in the profiles of running : queries, and it's only reported when the query is finished. This jira makes : the execution summary to the profile reported wh > Use shorter lines. My recommendation is to wrap at 70 characters. Thanks, addressed in new rev. http://gerrit.cloudera.org:8080/#/c/11591/1/be/src/service/impala-server.h File be/src/service/impala-server.h: http://gerrit.cloudera.org:8080/#/c/11591/1/be/src/service/impala-server.h@637 PS1, Line 637: void UpdateE > You don't want [[noreturn]] Thanks Joe, my misunderstanding of this annotation. Fixed in new rev. http://gerrit.cloudera.org:8080/#/c/11591/1/be/src/service/impala-server.cc File be/src/service/impala-server.cc: http://gerrit.cloudera.org:8080/#/c/11591/1/be/src/service/impala-server.cc@1128 PS1, Line 1128:<< PrettyPrinter::Print(cpu_limit_ > I'm thinking this might not hold for some calls to GetRuntimeProfileStr() t Good catch Joe. Indeed. Fixed in new rev by adding a check. I tried to run test_observability both locally and at jenkins, the former had some failures, however, the latter is clean. Looking into why it failed locally. -- To view, visit http://gerrit.cloudera.org:8080/11591 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idc7f714c9427d4b26d4e78cf27ceca2b0b336699 Gerrit-Change-Number: 11591 Gerrit-PatchSet: 2 Gerrit-Owner: Yongjun Zhang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongjun Zhang Gerrit-Comment-Date: Tue, 30 Oct 2018 21:51:06 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7786: Start HMS and Sentry in debug mode in the minicluster
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11826 ) Change subject: IMPALA-7786: Start HMS and Sentry in debug mode in the minicluster .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1217/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11826 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I29b025cbde36ef398ea36fbe69eff26e27d93e48 Gerrit-Change-Number: 11826 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Tue, 30 Oct 2018 21:32:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7586: fix predicate pushdown of escaped strings
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/11814 ) Change subject: IMPALA-7586: fix predicate pushdown of escaped strings .. Patch Set 6: Code-Review+1 (3 comments) http://gerrit.cloudera.org:8080/#/c/11814/5/fe/src/main/java/org/apache/impala/planner/KuduScanNode.java File fe/src/main/java/org/apache/impala/planner/KuduScanNode.java: http://gerrit.cloudera.org:8080/#/c/11814/5/fe/src/main/java/org/apache/impala/planner/KuduScanNode.java@319 PS5, Line 319: getExplainString > The behaviour here seems ok to me - it calls toSql(), which consistently es The behavior overall seems ok to me as well. But in this case, it diverged from the original conjunct that was used to push down. http://gerrit.cloudera.org:8080/#/c/11814/5/testdata/data/strings_with_quotes.csv File testdata/data/strings_with_quotes.csv: http://gerrit.cloudera.org:8080/#/c/11814/5/testdata/data/strings_with_quotes.csv@11 PS5, Line 11: foo\\"bar,11 > Done. I assume you meant that you wanted the value in the table in include Yep. http://gerrit.cloudera.org:8080/#/c/11814/5/testdata/workloads/functional-query/queries/QueryTest/string-escaping-rcfile-bug.test File testdata/workloads/functional-query/queries/QueryTest/string-escaping-rcfile-bug.test: http://gerrit.cloudera.org:8080/#/c/11814/5/testdata/workloads/functional-query/queries/QueryTest/string-escaping-rcfile-bug.test@4 PS5, Line 4: > I think it's better to exercise and track the known-bad behaviour than leav Agreed. -- To view, visit http://gerrit.cloudera.org:8080/11814 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I53d6e20dd48ab6837ddd325db8a9d49ee04fed28 Gerrit-Change-Number: 11814 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 30 Oct 2018 21:24:44 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7786: Start HMS and Sentry in debug mode in the minicluster
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11826 ) Change subject: IMPALA-7786: Start HMS and Sentry in debug mode in the minicluster .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1216/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11826 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I29b025cbde36ef398ea36fbe69eff26e27d93e48 Gerrit-Change-Number: 11826 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Tue, 30 Oct 2018 21:21:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5821: Add query with implicit casts to extended explain output.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11719 ) Change subject: IMPALA-5821: Add query with implicit casts to extended explain output. .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1215/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11719 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I55c3bdacc295137f66b2316a912fc347da30d6b0 Gerrit-Change-Number: 11719 Gerrit-PatchSet: 4 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 30 Oct 2018 21:09:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7786: Start HMS and Sentry in debug mode in the minicluster
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11826 ) Change subject: IMPALA-7786: Start HMS and Sentry in debug mode in the minicluster .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11826 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I29b025cbde36ef398ea36fbe69eff26e27d93e48 Gerrit-Change-Number: 11826 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Tue, 30 Oct 2018 21:06:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7786: Start HMS and Sentry in debug mode in the minicluster
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11826 ) Change subject: IMPALA-7786: Start HMS and Sentry in debug mode in the minicluster .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3380/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/11826 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I29b025cbde36ef398ea36fbe69eff26e27d93e48 Gerrit-Change-Number: 11826 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Tue, 30 Oct 2018 21:06:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7786: Start HMS and Sentry in debug mode in the minicluster
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/11826 ) Change subject: IMPALA-7786: Start HMS and Sentry in debug mode in the minicluster .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11826 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I29b025cbde36ef398ea36fbe69eff26e27d93e48 Gerrit-Change-Number: 11826 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Tue, 30 Oct 2018 21:04:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7786: Start HMS and Sentry in debug mode in the minicluster
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11826 ) Change subject: IMPALA-7786: Start HMS and Sentry in debug mode in the minicluster .. Patch Set 3: (2 comments) http://gerrit.cloudera.org:8080/#/c/11826/3/testdata/bin/run-hive-server.sh File testdata/bin/run-hive-server.sh: http://gerrit.cloudera.org:8080/#/c/11826/3/testdata/bin/run-hive-server.sh@66 PS3, Line 66: export HIVE_METASTORE_HADOOP_OPTS="-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=30010" line too long (106 > 90) http://gerrit.cloudera.org:8080/#/c/11826/3/testdata/bin/run-sentry-service.sh File testdata/bin/run-sentry-service.sh: http://gerrit.cloudera.org:8080/#/c/11826/3/testdata/bin/run-sentry-service.sh@46 PS3, Line 46: export HADOOP_OPTS="-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=30020" line too long (91 > 90) -- To view, visit http://gerrit.cloudera.org:8080/11826 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I29b025cbde36ef398ea36fbe69eff26e27d93e48 Gerrit-Change-Number: 11826 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Tue, 30 Oct 2018 21:03:35 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7786: Start HMS and Sentry in debug mode in the minicluster
Fredy Wijaya has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/11826 ) Change subject: IMPALA-7786: Start HMS and Sentry in debug mode in the minicluster .. IMPALA-7786: Start HMS and Sentry in debug mode in the minicluster This patch updates the HMS and Sentry run scripts to start HMS and Sentry in debug mode in the minicluster to make it easier to debug issues related to HMS and Sentry. HMS debug port: 30010 Sentry debug port: 30020 Testing: - Connected the debugger to both HMS and Sentry. Change-Id: I29b025cbde36ef398ea36fbe69eff26e27d93e48 --- M testdata/bin/run-hive-server.sh M testdata/bin/run-sentry-service.sh 2 files changed, 4 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/26/11826/3 -- To view, visit http://gerrit.cloudera.org:8080/11826 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I29b025cbde36ef398ea36fbe69eff26e27d93e48 Gerrit-Change-Number: 11826 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger
[Impala-ASF-CR] IMPALA-7786: Start HMS and Sentry in debug mode in the minicluster
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/11826 ) Change subject: IMPALA-7786: Start HMS and Sentry in debug mode in the minicluster .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/11826/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11826/2//COMMIT_MSG@13 PS2, Line 13: HMS debug port: 30010 > Watch out for ports above 32k. These are often configured as "ephemeral" po Changed the ports. Done. -- To view, visit http://gerrit.cloudera.org:8080/11826 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I29b025cbde36ef398ea36fbe69eff26e27d93e48 Gerrit-Change-Number: 11826 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Tue, 30 Oct 2018 21:03:00 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7761: Add multiple DISTINCT to targeted stress and perf
Thomas Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/11805 ) Change subject: IMPALA-7761: Add multiple DISTINCT to targeted stress and perf .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/11805/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11805/1//COMMIT_MSG@16 PS1, Line 16: Testing: : - Ran the test file locally. > The queries are fine, but what does it mean to run these locally? I'm not t By "run locally" I mean that I put together a python test that run the files with run_test_case(), like we would do for an equivalent file in the functional-query workload. The idea was to verify that I had the RESULTS/TYPES/LABELS sections correct. The targeted-perf workload is used by single_node_perf_run.py. I believe its something that Mostafa also used to run to check for regressions between releases. Even if no one is running it currently, it seems reasonable to me to add these queries for the future when there is someone paying closer attention to perf issues. The targeted-stress workload seems to be pointless. Its only references in tests/stress/test_mini_stress.py and all of the tests there have been xfailed for years. Probably we should remove targeted-stress, or even better move the queries in workloads/tpch/ that aren't actually regular tpch queries to targeted-stress and modify the stress test to use them. It would also be great to re-enable the tests in test_mini_stress.py, even if just to ensure that the RESULTS/TYPES/LABELS sections stay up to date. That's all (hopefully) outside the scope of this patch. Yes, the intention is for these to be run as part of downstream larger-scale testing. Is there any way to accomplish that currently? It seems that concurrent_select won't find them even if I add them to eg. testdata/workloads/tpch/queries/tpch-aggregations.test, as it only checks for test files of the form 'tpch-q*.test' -- To view, visit http://gerrit.cloudera.org:8080/11805 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I400aaf6b6620b4001895eafff785956bffb312c9 Gerrit-Change-Number: 11805 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 30 Oct 2018 20:55:23 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7786: Start HMS and Sentry in debug mode in the minicluster
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/11826 ) Change subject: IMPALA-7786: Start HMS and Sentry in debug mode in the minicluster .. Patch Set 2: (1 comment) Seems fine except for your port choice. http://gerrit.cloudera.org:8080/#/c/11826/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11826/2//COMMIT_MSG@13 PS2, Line 13: HMS debug port: 4 Watch out for ports above 32k. These are often configured as "ephemeral" ports in linux which will produce somewhat random flakies. $sysctl net.ipv4.ip_local_port_range net.ipv4.ip_local_port_range = 3276860999 -- To view, visit http://gerrit.cloudera.org:8080/11826 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I29b025cbde36ef398ea36fbe69eff26e27d93e48 Gerrit-Change-Number: 11826 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Tue, 30 Oct 2018 20:52:42 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7786: Start HMS and Sentry in debug mode in the minicluster
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11826 ) Change subject: IMPALA-7786: Start HMS and Sentry in debug mode in the minicluster .. Patch Set 2: (2 comments) http://gerrit.cloudera.org:8080/#/c/11826/2/testdata/bin/run-hive-server.sh File testdata/bin/run-hive-server.sh: http://gerrit.cloudera.org:8080/#/c/11826/2/testdata/bin/run-hive-server.sh@66 PS2, Line 66: export HIVE_METASTORE_HADOOP_OPTS="-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=4" line too long (106 > 90) http://gerrit.cloudera.org:8080/#/c/11826/2/testdata/bin/run-sentry-service.sh File testdata/bin/run-sentry-service.sh: http://gerrit.cloudera.org:8080/#/c/11826/2/testdata/bin/run-sentry-service.sh@46 PS2, Line 46: export HADOOP_OPTS="-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5" line too long (91 > 90) -- To view, visit http://gerrit.cloudera.org:8080/11826 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I29b025cbde36ef398ea36fbe69eff26e27d93e48 Gerrit-Change-Number: 11826 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Tue, 30 Oct 2018 20:50:19 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7786: Start HMS and Sentry in debug mode in the minicluster
Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11826 Change subject: IMPALA-7786: Start HMS and Sentry in debug mode in the minicluster .. IMPALA-7786: Start HMS and Sentry in debug mode in the minicluster This patch updates the HMS and Sentry run scripts to start HMS and Sentry in debug mode in the minicluster to make it easier to debug issues related to HMS and Sentry. HMS debug port: 4 Sentry debug port: 5 Testing: - Connected the debugger to both HMS and Sentry. Change-Id: I29b025cbde36ef398ea36fbe69eff26e27d93e48 --- M testdata/bin/run-hive-server.sh M testdata/bin/run-sentry-service.sh 2 files changed, 4 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/26/11826/2 -- To view, visit http://gerrit.cloudera.org:8080/11826 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I29b025cbde36ef398ea36fbe69eff26e27d93e48 Gerrit-Change-Number: 11826 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya
[Impala-ASF-CR] IMPALA-5821: Add query with implicit casts to extended explain output.
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/11719 ) Change subject: IMPALA-5821: Add query with implicit casts to extended explain output. .. Patch Set 4: I added a new patch because rebase was required -- To view, visit http://gerrit.cloudera.org:8080/11719 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I55c3bdacc295137f66b2316a912fc347da30d6b0 Gerrit-Change-Number: 11719 Gerrit-PatchSet: 4 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 30 Oct 2018 20:34:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5821: Add query with implicit casts to extended explain output.
Andrew Sherman has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/11719 ) Change subject: IMPALA-5821: Add query with implicit casts to extended explain output. .. IMPALA-5821: Add query with implicit casts to extended explain output. If explain_level is at 'extended' level or higher, then enhance the output from the explain command. (1) Show the analyzed sql in the explain header, this is the rewritten sql, which includes implicit casts, and literals are printed with a cast so that their type is visible. (2) When predicates are shown in the plan these are shown in the same format. The toSql() method can be called on a ParseNode tree to return the sql corresponding ot the tree. In the past toSQl() has been enhanced to print rewritten sql by partially overloading toSql() [with toSql(boolean)]. This current change requires changing toSQl() in many places as NumericLiteral can appear at different points in ia parse tree. To avoid many new fragile overloads of toSql() I added toSql(ToSqlOptions), where ToSqlOptions is an enum which controls the form of the Sql that is returned. This changes many files but is safer and means that any future options to toSql() can be added painlessly. If SHOW_IMPLICIT_CASTS is passed to toSql() then - in CastExpr print the implicit cast - in NumericLiteral print the literal with a cast to show the type Add a PlannerTestOption directive that will force the query text showing implicit casts to be included in the PLAN section of a .test file. The analyzed query text is wrapped at 80 characters. Note that the analyzed query cannot always be executed as queries rewritten to use LEFT SEMI JOIN are not legal sql. In addition some space characters may be removed from the query for prettier display. Documentation of this change will be done as IMPALA-7718 EXAMPLE OUTPUT: [localhost:21000] default> set explain_level=2; EXPLAIN_LEVEL set to 2 [localhost:21000] default> explain select * from functional_kudu.alltypestiny where bigint_col < 1000 / 100; Query: explain select * from functional_kudu.alltypestiny where bigint_col < 1000 / 100 Max Per-Host Resource Reservation: Memory=0B Threads=2 Per-Host Resource Estimates: Memory=10MB Codegen disabled by planner Analyzed query: SELECT * FROM functional_kudu.alltypestiny WHERE CAST(bigint_col AS DOUBLE) < CAST(10 AS DOUBLE) "" F00:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1 | Per-Host Resources: mem-estimate=4.88MB mem-reservation=0B thread-reservation=2 PLAN-ROOT SINK | mem-estimate=0B mem-reservation=0B thread-reservation=0 | 00:SCAN KUDU [functional_kudu.alltypestiny] predicates: CAST(bigint_col AS DOUBLE) < CAST(10 AS DOUBLE) mem-estimate=4.88MB mem-reservation=0B thread-reservation=1 tuple-ids=0 row-size=97B cardinality=1 in pipelines: 00(GETNEXT) Fetched 16 row(s) in 0.03s TESTING: All end-to-end tests pass. Added a new test in ExprRewriterTest which prints sql with implict casts for some interesting queries. Add a unit test for the code which wraps text at 80 characters. The output of some Planner Tests in .test files has been updated to include the Analyzed sql that is printed when explain_level is at at least 'extended' level. Change-Id: I55c3bdacc295137f66b2316a912fc347da30d6b0 --- M fe/src/main/java/org/apache/impala/analysis/AdminFnStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableAddDropRangePartitionStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableAddPartitionStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableDropPartitionStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterViewStmt.java M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/analysis/AnalyticExpr.java M fe/src/main/java/org/apache/impala/analysis/AnalyticWindow.java M fe/src/main/java/org/apache/impala/analysis/ArithmeticExpr.java M fe/src/main/java/org/apache/impala/analysis/BaseTableRef.java M fe/src/main/java/org/apache/impala/analysis/BetweenPredicate.java M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java M fe/src/main/java/org/apache/impala/analysis/BoolLiteral.java M fe/src/main/java/org/apache/impala/analysis/CaseExpr.java M fe/src/main/java/org/apache/impala/analysis/CastExpr.java M fe/src/main/java/org/apache/impala/analysis/CompoundPredicate.java M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateDataSrcStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateDbStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateDropRoleStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateOrAlterViewStmtBase.java M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeFileStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeStmt.java M
[Impala-ASF-CR] IMPALA-7586: fix predicate pushdown of escaped strings
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11814 ) Change subject: IMPALA-7586: fix predicate pushdown of escaped strings .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1214/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11814 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I53d6e20dd48ab6837ddd325db8a9d49ee04fed28 Gerrit-Change-Number: 11814 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 30 Oct 2018 20:02:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7727: Fix TStatusCode to TErrorCode mapping
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11778 ) Change subject: IMPALA-7727: Fix TStatusCode to TErrorCode mapping .. IMPALA-7727: Fix TStatusCode to TErrorCode mapping - Uses a "GENERAL" TErrorCode type for all non-OK statuses. - Detailed regression root cause description in the jira IMPALA-7727. - Added a regression test. Change-Id: Ie62527734aa73c1524c731773638590bdac9e789 Reviewed-on: http://gerrit.cloudera.org:8080/11778 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/common/status.cc M be/src/common/status.h M be/src/service/child-query.cc M be/src/util/error-util.cc M be/src/util/error-util.h M testdata/workloads/functional-query/queries/QueryTest/compute-stats.test 6 files changed, 37 insertions(+), 27 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/11778 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ie62527734aa73c1524c731773638590bdac9e789 Gerrit-Change-Number: 11778 Gerrit-PatchSet: 6 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7727: Fix TStatusCode to TErrorCode mapping
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11778 ) Change subject: IMPALA-7727: Fix TStatusCode to TErrorCode mapping .. Patch Set 5: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/11778 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie62527734aa73c1524c731773638590bdac9e789 Gerrit-Change-Number: 11778 Gerrit-PatchSet: 5 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 30 Oct 2018 19:55:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7783: Skip test default timezone when testing a real cluster.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11820 ) Change subject: IMPALA-7783: Skip test_default_timezone when testing a real cluster. .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/11820 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia4d4c503d2c77136cedd8f3fd830b6ce70d4457f Gerrit-Change-Number: 11820 Gerrit-PatchSet: 4 Gerrit-Owner: David Knupp Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Comment-Date: Tue, 30 Oct 2018 19:47:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7783: Skip test default timezone when testing a real cluster.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11820 ) Change subject: IMPALA-7783: Skip test_default_timezone when testing a real cluster. .. IMPALA-7783: Skip test_default_timezone when testing a real cluster. test_shell_commandline.py::test_default_timezone assumes that the cluster is running on the same platform as the test process, but that's only guaranteed when the testing a local minicluster. When run against a real cluster, the test executor can be a completely different OS. Change-Id: Ia4d4c503d2c77136cedd8f3fd830b6ce70d4457f Reviewed-on: http://gerrit.cloudera.org:8080/11820 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M tests/shell/test_shell_commandline.py 1 file changed, 6 insertions(+), 0 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/11820 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ia4d4c503d2c77136cedd8f3fd830b6ce70d4457f Gerrit-Change-Number: 11820 Gerrit-PatchSet: 5 Gerrit-Owner: David Knupp Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown
[Impala-ASF-CR] IMPALA-6374: fix handling of commas in .test files
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11800 ) Change subject: IMPALA-6374: fix handling of commas in .test files .. Patch Set 5: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/3377/ -- To view, visit http://gerrit.cloudera.org:8080/11800 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I18ddcb0440490ddf8184be66d3681038a1615dd9 Gerrit-Change-Number: 11800 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 30 Oct 2018 19:43:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7586: fix predicate pushdown of escaped strings
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11814 ) Change subject: IMPALA-7586: fix predicate pushdown of escaped strings .. Patch Set 5: (6 comments) http://gerrit.cloudera.org:8080/#/c/11814/5//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11814/5//COMMIT_MSG@9 PS5, Line 9: This fixes a class of bugs where the planner incorrectly uses the raw : string from the parser instead of the unescaped string. > fyi, I have found a similar issue in partition pruning, see IMPALA-7784. ack http://gerrit.cloudera.org:8080/#/c/11814/5//COMMIT_MSG@26 PS5, Line 26: Added regression test that tests handling of backslash escapes on all file : formats. I did not add a regression test for the data source bug since it > nit: long lines, please wrap at 72 Done http://gerrit.cloudera.org:8080/#/c/11814/5/fe/src/main/java/org/apache/impala/planner/KuduScanNode.java File fe/src/main/java/org/apache/impala/planner/KuduScanNode.java: http://gerrit.cloudera.org:8080/#/c/11814/5/fe/src/main/java/org/apache/impala/planner/KuduScanNode.java@319 PS5, Line 319: getExplainString > It looks like this always prints the normalized/unescaped string even thoug The behaviour here seems ok to me - it calls toSql(), which consistently escapes things. If we didn't escape things, then the expressions in the explain output might not be valid expressions in our SQL expressions, which seems inconvenient. http://gerrit.cloudera.org:8080/#/c/11814/5/testdata/data/strings_with_quotes.csv File testdata/data/strings_with_quotes.csv: http://gerrit.cloudera.org:8080/#/c/11814/5/testdata/data/strings_with_quotes.csv@11 PS5, Line 11: > add foo\"bar, 11 too? That shouldn't be returned with ..where s = "foo\"bar Done. I assume you meant that you wanted the value in the table in include the backslash, which requires a double backslash here. http://gerrit.cloudera.org:8080/#/c/11814/5/testdata/workloads/functional-query/queries/QueryTest/string-escaping-rcfile-bug.test File testdata/workloads/functional-query/queries/QueryTest/string-escaping-rcfile-bug.test: http://gerrit.cloudera.org:8080/#/c/11814/5/testdata/workloads/functional-query/queries/QueryTest/string-escaping-rcfile-bug.test@1 PS5, Line 1: > Can you add an extra comment to make it even clearer that this file contain Done http://gerrit.cloudera.org:8080/#/c/11814/5/testdata/workloads/functional-query/queries/QueryTest/string-escaping-rcfile-bug.test@4 PS5, Line 4: # IMPALA-7778: escapes are ignored so output is incorrect > Can't we just xfail if format == 'rc' instead of doing this? or is it becau I think it's better to exercise and track the known-bad behaviour than leave it unexercised. Also, honestly, I don't know if there is enough usage of RC at this point that anyone will be motivated to fix it. -- To view, visit http://gerrit.cloudera.org:8080/11814 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I53d6e20dd48ab6837ddd325db8a9d49ee04fed28 Gerrit-Change-Number: 11814 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 30 Oct 2018 19:28:43 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7586: fix predicate pushdown of escaped strings
Hello Bharath Vissapragada, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11814 to look at the new patch set (#6). Change subject: IMPALA-7586: fix predicate pushdown of escaped strings .. IMPALA-7586: fix predicate pushdown of escaped strings This fixes a class of bugs where the planner incorrectly uses the raw string from the parser instead of the unescaped string. This occurs in several places that push predicates down to the storage layer: * Kudu scans * HBase scans * Data source scans There are some more complex issues with escapes and the LIKE predicate that are tracked separately by IMPALA-2422. This also uncovered a different issue with RCFiles that is tracked by IMPALA-7778 and is worked around by the tests added. In order to make bugs like this more obvious in future, I renamed getValue() to getValueWithOriginalEscapes(). Testing: Added regression test that tests handling of backslash escapes on all file formats. I did not add a regression test for the data source bug since it seems to require some major modification of the data source test infrastructure. Change-Id: I53d6e20dd48ab6837ddd325db8a9d49ee04fed28 --- M fe/src/main/java/org/apache/impala/analysis/AdminFnStmt.java M fe/src/main/java/org/apache/impala/analysis/Expr.java M fe/src/main/java/org/apache/impala/analysis/ExtractFromExpr.java M fe/src/main/java/org/apache/impala/analysis/LikePredicate.java M fe/src/main/java/org/apache/impala/analysis/StringLiteral.java M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java M fe/src/main/java/org/apache/impala/planner/HBaseScanNode.java M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java M testdata/data/README A testdata/data/strings_with_quotes.csv M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv A testdata/workloads/functional-query/queries/QueryTest/string-escaping-rcfile-bug.test A testdata/workloads/functional-query/queries/QueryTest/string-escaping.test M tests/query_test/test_scanners.py 15 files changed, 199 insertions(+), 14 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/14/11814/6 -- To view, visit http://gerrit.cloudera.org:8080/11814 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I53d6e20dd48ab6837ddd325db8a9d49ee04fed28 Gerrit-Change-Number: 11814 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-6742: Profiles of running queries should include execution summary.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11591 ) Change subject: IMPALA-6742: Profiles of running queries should include execution summary. .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1213/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11591 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idc7f714c9427d4b26d4e78cf27ceca2b0b336699 Gerrit-Change-Number: 11591 Gerrit-PatchSet: 2 Gerrit-Owner: Yongjun Zhang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongjun Zhang Gerrit-Comment-Date: Tue, 30 Oct 2018 18:06:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7586: fix predicate pushdown of escaped strings
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/11814 ) Change subject: IMPALA-7586: fix predicate pushdown of escaped strings .. Patch Set 5: (3 comments) http://gerrit.cloudera.org:8080/#/c/11814/5//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11814/5//COMMIT_MSG@9 PS5, Line 9: This fixes a class of bugs where the planner incorrectly uses the raw : string from the parser instead of the unescaped string. fyi, I have found a similar issue in partition pruning, see IMPALA-7784. http://gerrit.cloudera.org:8080/#/c/11814/5//COMMIT_MSG@26 PS5, Line 26: Added regression test that tests handling of backslash escapes on all file : formats. I did not add a regression test for the data source bug since it nit: long lines, please wrap at 72 http://gerrit.cloudera.org:8080/#/c/11814/5/testdata/workloads/functional-query/queries/QueryTest/string-escaping-rcfile-bug.test File testdata/workloads/functional-query/queries/QueryTest/string-escaping-rcfile-bug.test: http://gerrit.cloudera.org:8080/#/c/11814/5/testdata/workloads/functional-query/queries/QueryTest/string-escaping-rcfile-bug.test@1 PS5, Line 1: Can you add an extra comment to make it even clearer that this file contains buggy results that we plan to fix later? -- To view, visit http://gerrit.cloudera.org:8080/11814 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I53d6e20dd48ab6837ddd325db8a9d49ee04fed28 Gerrit-Change-Number: 11814 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 30 Oct 2018 17:55:01 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6742: Profiles of running queries should include execution summary.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11591 ) Change subject: IMPALA-6742: Profiles of running queries should include execution summary. .. Patch Set 1: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/1212/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/11591 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idc7f714c9427d4b26d4e78cf27ceca2b0b336699 Gerrit-Change-Number: 11591 Gerrit-PatchSet: 1 Gerrit-Owner: Yongjun Zhang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongjun Zhang Gerrit-Comment-Date: Tue, 30 Oct 2018 17:45:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7586: fix predicate pushdown of escaped strings
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/11814 ) Change subject: IMPALA-7586: fix predicate pushdown of escaped strings .. Patch Set 5: Code-Review+1 (3 comments) A bunch of minor comments. Fix lgtm. http://gerrit.cloudera.org:8080/#/c/11814/5/fe/src/main/java/org/apache/impala/planner/KuduScanNode.java File fe/src/main/java/org/apache/impala/planner/KuduScanNode.java: http://gerrit.cloudera.org:8080/#/c/11814/5/fe/src/main/java/org/apache/impala/planner/KuduScanNode.java@319 PS5, Line 319: getExplainString It looks like this always prints the normalized/unescaped string even though we used the original string for predicates in the scan nodes. We could've probably diagnosed the issue faster if this was right. http://gerrit.cloudera.org:8080/#/c/11814/5/testdata/data/strings_with_quotes.csv File testdata/data/strings_with_quotes.csv: http://gerrit.cloudera.org:8080/#/c/11814/5/testdata/data/strings_with_quotes.csv@11 PS5, Line 11: add foo\"bar, 11 too? That shouldn't be returned with ..where s = "foo\"bar".. http://gerrit.cloudera.org:8080/#/c/11814/5/testdata/workloads/functional-query/queries/QueryTest/string-escaping-rcfile-bug.test File testdata/workloads/functional-query/queries/QueryTest/string-escaping-rcfile-bug.test: http://gerrit.cloudera.org:8080/#/c/11814/5/testdata/workloads/functional-query/queries/QueryTest/string-escaping-rcfile-bug.test@4 PS5, Line 4: # IMPALA-7778: escapes are ignored so output is incorrect Can't we just xfail if format == 'rc' instead of doing this? or is it because if someone fixes it in the future, this test starts failing and they know it right away? -- To view, visit http://gerrit.cloudera.org:8080/11814 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I53d6e20dd48ab6837ddd325db8a9d49ee04fed28 Gerrit-Change-Number: 11814 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 30 Oct 2018 17:38:19 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6742: Profiles of running queries should include execution summary.
Yongjun Zhang has posted comments on this change. ( http://gerrit.cloudera.org:8080/11591 ) Change subject: IMPALA-6742: Profiles of running queries should include execution summary. .. Patch Set 2: Hi Joe, thanks a lot for the very good review and sorry for late update. I just uploaded a new rev. Interestingly, some tests in test_observability failed locally but all is clean in jenkins. One question about your comment about line 637, I saw other places included [[noreturn]], what's the guideline for having it or not? thanks. -- To view, visit http://gerrit.cloudera.org:8080/11591 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idc7f714c9427d4b26d4e78cf27ceca2b0b336699 Gerrit-Change-Number: 11591 Gerrit-PatchSet: 2 Gerrit-Owner: Yongjun Zhang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongjun Zhang Gerrit-Comment-Date: Tue, 30 Oct 2018 17:36:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6742: Profiles of running queries should include execution summary.
Hello Tim Armstrong, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11591 to look at the new patch set (#2). Change subject: IMPALA-6742: Profiles of running queries should include execution summary. .. IMPALA-6742: Profiles of running queries should include execution summary. Currently execution summary is not included in the profiles of running queries, and it's only reported when the query is finished. This jira makes the execution summary to the profile reported when queries are still running. Testing: Done with real cluster. Change-Id: Idc7f714c9427d4b26d4e78cf27ceca2b0b336699 --- M be/src/service/impala-server.cc M be/src/service/impala-server.h 2 files changed, 19 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/91/11591/2 -- To view, visit http://gerrit.cloudera.org:8080/11591 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Idc7f714c9427d4b26d4e78cf27ceca2b0b336699 Gerrit-Change-Number: 11591 Gerrit-PatchSet: 2 Gerrit-Owner: Yongjun Zhang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-6742: Profiles of running queries should include execution summary
Yongjun Zhang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11591 Change subject: IMPALA-6742: Profiles of running queries should include execution summary .. IMPALA-6742: Profiles of running queries should include execution summary Currently execution summary is not included in the profiles of running queries, and it's only reported when the query is finished. This jira makes the execution summary to the profile reported when queries are still running. Testing: This is a draft, tests are yet to be done. Change-Id: Idc7f714c9427d4b26d4e78cf27ceca2b0b336699 --- M be/src/service/impala-server.cc M be/src/service/impala-server.h 2 files changed, 15 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/91/11591/1 -- To view, visit http://gerrit.cloudera.org:8080/11591 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Idc7f714c9427d4b26d4e78cf27ceca2b0b336699 Gerrit-Change-Number: 11591 Gerrit-PatchSet: 1 Gerrit-Owner: Yongjun Zhang Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7586: fix predicate pushdown of escaped strings
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11814 ) Change subject: IMPALA-7586: fix predicate pushdown of escaped strings .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1211/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11814 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I53d6e20dd48ab6837ddd325db8a9d49ee04fed28 Gerrit-Change-Number: 11814 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 30 Oct 2018 16:52:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5050: Add support to read TIMESTAMP MILLIS and TIMESTAMP MICROS from Parquet
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/11057 ) Change subject: IMPALA-5050: Add support to read TIMESTAMP_MILLIS and TIMESTAMP_MICROS from Parquet .. Patch Set 18: > Hi Csaba, should I look at this now or is Zoltan still completing > his review? Hi Tim! I think that Zoltan will not continue the review this week, so it would be great if you could look at it. -- To view, visit http://gerrit.cloudera.org:8080/11057 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4c7c01fffa31b3d2ca3480adf6ff851137dadac3 Gerrit-Change-Number: 11057 Gerrit-PatchSet: 18 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 30 Oct 2018 16:33:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7727: Fix TStatusCode to TErrorCode mapping
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11778 ) Change subject: IMPALA-7727: Fix TStatusCode to TErrorCode mapping .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11778 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie62527734aa73c1524c731773638590bdac9e789 Gerrit-Change-Number: 11778 Gerrit-PatchSet: 5 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 30 Oct 2018 16:02:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7586: fix predicate pushdown of escaped strings
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11814 ) Change subject: IMPALA-7586: fix predicate pushdown of escaped strings .. Patch Set 5: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/1210/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/11814 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I53d6e20dd48ab6837ddd325db8a9d49ee04fed28 Gerrit-Change-Number: 11814 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 30 Oct 2018 16:01:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7727: Fix TStatusCode to TErrorCode mapping
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11778 ) Change subject: IMPALA-7727: Fix TStatusCode to TErrorCode mapping .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3379/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/11778 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie62527734aa73c1524c731773638590bdac9e789 Gerrit-Change-Number: 11778 Gerrit-PatchSet: 5 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 30 Oct 2018 16:02:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7783: Skip test default timezone when testing a real cluster.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11820 ) Change subject: IMPALA-7783: Skip test_default_timezone when testing a real cluster. .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3378/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/11820 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia4d4c503d2c77136cedd8f3fd830b6ce70d4457f Gerrit-Change-Number: 11820 Gerrit-PatchSet: 4 Gerrit-Owner: David Knupp Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Comment-Date: Tue, 30 Oct 2018 15:57:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7783: Skip test default timezone when testing a real cluster.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11820 ) Change subject: IMPALA-7783: Skip test_default_timezone when testing a real cluster. .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11820 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia4d4c503d2c77136cedd8f3fd830b6ce70d4457f Gerrit-Change-Number: 11820 Gerrit-PatchSet: 4 Gerrit-Owner: David Knupp Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Comment-Date: Tue, 30 Oct 2018 15:57:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6658: improve Parquet RLE for low bit widths
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/11582 ) Change subject: IMPALA-6658: improve Parquet RLE for low bit widths .. Patch Set 4: (4 comments) I went through code again and my impression is that MaxBufferSize does not do its job properly. http://gerrit.cloudera.org:8080/#/c/11582/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11582/2//COMMIT_MSG@20 PS2, Line 20: length 16 for single bit values. All other bit widths will use the : existing length 8 runs. : > I agree that this might be a better way, but I am happy with the simple imp Ok, the biggest win (50% size for alternating runs of 8 in the 1 bit case) is already gained with this change. Please reflect in the Jira that there is still some potential in improving the RLE encoding. http://gerrit.cloudera.org:8080/#/c/11582/4/be/src/util/rle-encoding.h File be/src/util/rle-encoding.h: http://gerrit.cloudera.org:8080/#/c/11582/4/be/src/util/rle-encoding.h@250 PS4, Line 250: 1 + Can you simplify this expression? MAX_VALUES_PER_LITERAL_RUN must be divisible by 8 - this could be checked by an assert, and the Ceil + static cast could be removed. http://gerrit.cloudera.org:8080/#/c/11582/4/be/src/util/rle-encoding.h@264 PS4, Line 264: num_runs Aren't we double counting the indicator byte here? My assumption is that this num_runs is meant to add the +1 byte/ literal run. http://gerrit.cloudera.org:8080/#/c/11582/4/be/src/util/rle-encoding.h@265 PS4, Line 265: return std::max(MinBufferSize(bit_width), literal_max_size); This seems to assume that a single big literal run is the worsts case - can't this underestimate the number of bytes needed due to IMPALA-6658? -- To view, visit http://gerrit.cloudera.org:8080/11582 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I191a581d3f699b6669e48ac9dc39c76ed77c4a76 Gerrit-Change-Number: 11582 Gerrit-PatchSet: 4 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 30 Oct 2018 15:55:19 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7783: Skip test default timezone when testing a real cluster.
Michael Brown has posted comments on this change. ( http://gerrit.cloudera.org:8080/11820 ) Change subject: IMPALA-7783: Skip test_default_timezone when testing a real cluster. .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11820 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia4d4c503d2c77136cedd8f3fd830b6ce70d4457f Gerrit-Change-Number: 11820 Gerrit-PatchSet: 3 Gerrit-Owner: David Knupp Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Comment-Date: Tue, 30 Oct 2018 15:54:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7783: Skip test default timezone when testing a real cluster.
David Knupp has posted comments on this change. ( http://gerrit.cloudera.org:8080/11820 ) Change subject: IMPALA-7783: Skip test_default_timezone when testing a real cluster. .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/11820/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11820/2//COMMIT_MSG@7 PS2, Line 7: IMPALA-77 > We need an upstream Jira here, not one for Cloudera. Doh -- thanks. I just spaced on this. -- To view, visit http://gerrit.cloudera.org:8080/11820 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia4d4c503d2c77136cedd8f3fd830b6ce70d4457f Gerrit-Change-Number: 11820 Gerrit-PatchSet: 3 Gerrit-Owner: David Knupp Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Comment-Date: Tue, 30 Oct 2018 15:50:43 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7783: Skip test default timezone when testing a real cluster.
Hello Michael Brown, Attila Jeges, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11820 to look at the new patch set (#3). Change subject: IMPALA-7783: Skip test_default_timezone when testing a real cluster. .. IMPALA-7783: Skip test_default_timezone when testing a real cluster. test_shell_commandline.py::test_default_timezone assumes that the cluster is running on the same platform as the test process, but that's only guaranteed when the testing a local minicluster. When run against a real cluster, the test executor can be a completely different OS. Change-Id: Ia4d4c503d2c77136cedd8f3fd830b6ce70d4457f --- M tests/shell/test_shell_commandline.py 1 file changed, 6 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/20/11820/3 -- To view, visit http://gerrit.cloudera.org:8080/11820 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia4d4c503d2c77136cedd8f3fd830b6ce70d4457f Gerrit-Change-Number: 11820 Gerrit-PatchSet: 3 Gerrit-Owner: David Knupp Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown
[Impala-ASF-CR] IMPALA-6374: fix handling of commas in .test files
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11800 ) Change subject: IMPALA-6374: fix handling of commas in .test files .. Patch Set 5: Exhaustive tests passed overnight -- To view, visit http://gerrit.cloudera.org:8080/11800 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I18ddcb0440490ddf8184be66d3681038a1615dd9 Gerrit-Change-Number: 11800 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 30 Oct 2018 15:48:32 + Gerrit-HasComments: No
