[Impala-ASF-CR] IMPALA-7477: Batch-oriented query set construction
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11879 ) Change subject: IMPALA-7477: Batch-oriented query set construction .. Patch Set 1: Yeah that's a good idea for next time :) -- To view, visit http://gerrit.cloudera.org:8080/11879 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I764fa302842438902cd5db2551ec6e3cb77b6874 Gerrit-Change-Number: 11879 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 06 Nov 2018 17:26:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7477: Batch-oriented query set construction
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11879 ) Change subject: IMPALA-7477: Batch-oriented query set construction .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3431/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/11879 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I764fa302842438902cd5db2551ec6e3cb77b6874 Gerrit-Change-Number: 11879 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 06 Nov 2018 17:26:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7477: Batch-oriented query set construction
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11879 ) Change subject: IMPALA-7477: Batch-oriented query set construction .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11879 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I764fa302842438902cd5db2551ec6e3cb77b6874 Gerrit-Change-Number: 11879 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 06 Nov 2018 17:26:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7808: Refactor Analyzer for easier debugging
Csaba Ringhofer has posted comments on this change. (
http://gerrit.cloudera.org:8080/11883 )
Change subject: IMPALA-7808: Refactor Analyzer for easier debugging
..
Patch Set 1:
(4 comments)
Some comments related to coding style in Impala, I like the change otherwise.
http://gerrit.cloudera.org:8080/#/c/11883/1/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
File fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java:
http://gerrit.cloudera.org:8080/#/c/11883/1/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java@461
PS1, Line 461: { return; }
We generally do not use braces in one line "if (cond) stmt;".
http://gerrit.cloudera.org:8080/#/c/11883/1/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java@461
PS1, Line 461:
nit: extra space
http://gerrit.cloudera.org:8080/#/c/11883/1/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
File fe/src/main/java/org/apache/impala/analysis/SelectStmt.java:
http://gerrit.cloudera.org:8080/#/c/11883/1/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@219
PS1, Line 219:
here and at many other functions: in Impala we generally do not add an empty
line at the beginning of functions.
http://gerrit.cloudera.org:8080/#/c/11883/1/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@526
PS1, Line 526:
nit: extra space
--
To view, visit http://gerrit.cloudera.org:8080/11883
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I576c80c4c7a974df226fc91d8903db275069ed52
Gerrit-Change-Number: 11883
Gerrit-PatchSet: 1
Gerrit-Owner: Paul Rogers
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Comment-Date: Tue, 06 Nov 2018 16:38:50 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7808: Refactor Analyzer for easier debugging
Vuk Ercegovac has posted comments on this change. (
http://gerrit.cloudera.org:8080/11883 )
Change subject: IMPALA-7808: Refactor Analyzer for easier debugging
..
Patch Set 1:
(8 comments)
thanks for the change and making it easier to review. mostly minor comments
from my end.
http://gerrit.cloudera.org:8080/#/c/11883/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/11883/1//COMMIT_MSG@37
PS1, Line 37: Testing
pls move this testing section above the Change-Id
http://gerrit.cloudera.org:8080/#/c/11883/1/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
File fe/src/main/java/org/apache/impala/analysis/SelectStmt.java:
http://gerrit.cloudera.org:8080/#/c/11883/1/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@176
PS1, Line 176: private final Analyzer analyzer;
these members should be suffixed with "_"
http://gerrit.cloudera.org:8080/#/c/11883/1/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@186
PS1, Line 186: Note tem
note, you can always include such review-level comments as comments in gerrit.
no need for source changes to communicate these things.
http://gerrit.cloudera.org:8080/#/c/11883/1/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@389
PS1, Line 389: protected
did some other class need this protected?
http://gerrit.cloudera.org:8080/#/c/11883/1/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@617
PS1, Line 617: public
why public?
http://gerrit.cloudera.org:8080/#/c/11883/1/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@659
PS1, Line 659: public
why public?
http://gerrit.cloudera.org:8080/#/c/11883/1/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@757
PS1, Line 757: public void verifyAggregation() throws AnalysisException {
why public?
http://gerrit.cloudera.org:8080/#/c/11883/1/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@894
PS1, Line 894: }
just to remind myelf... end of inner class.
--
To view, visit http://gerrit.cloudera.org:8080/11883
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I576c80c4c7a974df226fc91d8903db275069ed52
Gerrit-Change-Number: 11883
Gerrit-PatchSet: 1
Gerrit-Owner: Paul Rogers
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Vuk Ercegovac
Gerrit-Comment-Date: Tue, 06 Nov 2018 17:58:40 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7788: [DOCS] Impala supports ADLS Gen 2 (ABFS)
Alex Rodoni has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11853 ) Change subject: IMPALA-7788: [DOCS] Impala supports ADLS Gen 2 (ABFS) .. IMPALA-7788: [DOCS] Impala supports ADLS Gen 2 (ABFS) Change-Id: Ic06d9ac92ed78b9092369e211de8a81db1d7ce90 Reviewed-on: http://gerrit.cloudera.org:8080/11853 Tested-by: Impala Public Jenkins Reviewed-by: Joe McDonnell Reviewed-by: Jim Apple --- M docs/shared/impala_common.xml M docs/topics/impala_adls.xml M docs/topics/impala_insert.xml M docs/topics/impala_load_data.xml 4 files changed, 127 insertions(+), 97 deletions(-) Approvals: Impala Public Jenkins: Verified Joe McDonnell: Looks good to me, but someone else must approve Jim Apple: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/11853 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ic06d9ac92ed78b9092369e211de8a81db1d7ce90 Gerrit-Change-Number: 11853 Gerrit-PatchSet: 5 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Joe McDonnell
[Impala-ASF-CR] IMPALA-7819: add per-node storage wait time to profile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11894 ) Change subject: IMPALA-7819: add per-node storage wait time to profile .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/11894/1/be/src/util/runtime-profile-counters.h File be/src/util/runtime-profile-counters.h: http://gerrit.cloudera.org:8080/#/c/11894/1/be/src/util/runtime-profile-counters.h@59 PS1, Line 59: ScopedTimer MACRO_CONCAT(SCOPED_TIMER_COUNTER, __COUNTER__)(c1, c2) line too long (93 > 90) -- To view, visit http://gerrit.cloudera.org:8080/11894 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I74a496efdbe9e804db779c9753b00a303a270580 Gerrit-Change-Number: 11894 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 07 Nov 2018 00:04:56 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7789: [DOCS] Admission status in Impala Shell
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/11895 ) Change subject: IMPALA-7789: [DOCS] Admission status in Impala Shell .. Patch Set 1: Tim, The LIVE_PROGRESS and LIVE_SUMMARY are already mentioned in impala_shell_options.html. -- To view, visit http://gerrit.cloudera.org:8080/11895 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I17d788eb716c6a2f7a144ee2d81bbe823f74d16a Gerrit-Change-Number: 11895 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 07 Nov 2018 01:29:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7819: add per-node storage wait time to profile
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11894 ) Change subject: IMPALA-7819: add per-node storage wait time to profile .. Patch Set 1: Lars, some of the observability work reminded me of this counter that I've wished we had a few times. -- To view, visit http://gerrit.cloudera.org:8080/11894 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I74a496efdbe9e804db779c9753b00a303a270580 Gerrit-Change-Number: 11894 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 07 Nov 2018 00:04:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7819: add per-node storage wait time to profile
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11894 Change subject: IMPALA-7819: add per-node storage wait time to profile .. IMPALA-7819: add per-node storage wait time to profile Adds ScannerIoWaitTime and KuduClientTime to measure time spent waiting for the I/O manager and time spent in the Kudu client, respectively. To minimise the additional overhead, add support to the timer infrastructure to update multiple counters with one stopwatch. Apply this in some other places too. Testing: Ran core tests. Ran some queries manually with and without mt_dop and sanity-checked the profiles. Change-Id: I74a496efdbe9e804db779c9753b00a303a270580 --- M be/src/exec/hdfs-orc-scanner.cc M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/hdfs-scan-node-base.h M be/src/exec/kudu-scan-node-base.cc M be/src/exec/kudu-scan-node-base.h M be/src/exec/kudu-scanner.cc M be/src/exec/scanner-context.cc M be/src/runtime/fragment-instance-state.cc M be/src/runtime/krpc-data-stream-recvr.cc M be/src/util/runtime-profile-counters.h 10 files changed, 84 insertions(+), 43 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/94/11894/1 -- To view, visit http://gerrit.cloudera.org:8080/11894 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I74a496efdbe9e804db779c9753b00a303a270580 Gerrit-Change-Number: 11894 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] IMPALA-3652: Fix resource transfer in subplans with limits
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11852 ) Change subject: IMPALA-3652: Fix resource transfer in subplans with limits .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11852 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3968a379fcbb5d30fcec304995d3e44933dbbc77 Gerrit-Change-Number: 11852 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 07 Nov 2018 00:28:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7819: add per-node storage wait time to profile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11894 ) Change subject: IMPALA-7819: add per-node storage wait time to profile .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1304/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11894 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I74a496efdbe9e804db779c9753b00a303a270580 Gerrit-Change-Number: 11894 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 07 Nov 2018 00:52:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5004: Switch to sorting node for large TopN queries
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11698 ) Change subject: IMPALA-5004: Switch to sorting node for large TopN queries .. Patch Set 10: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11698 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I34c9db33c9302b55e9978f53f9c7061f2806c8a9 Gerrit-Change-Number: 11698 Gerrit-PatchSet: 10 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 07 Nov 2018 01:17:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5004: Switch to sorting node for large TopN queries
Tim Armstrong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11698 ) Change subject: IMPALA-5004: Switch to sorting node for large TopN queries .. IMPALA-5004: Switch to sorting node for large TopN queries Adds a new query option 'topn_bytes_limit' that places a limit on the number of estimated bytes that a TopN operator can process. If the Impala planner estimates that a TopN operator will process more bytes than this limit, it will replace the TopN operator with a sort operator. Since the TopN operator cannot spill to disk, it has to buffer everything in memory. This can cause frequent OOM issues when running with a large limit + offset. Switching to a sort operator allows Impala to spill to disk. We prefer to use the TopN operator when possible as it has better performance than the sort operator for 'order by limit [offset]' queries. The default limit is set to 512MB and is based on micro-benchmarking the topn vs. sort operator for various limits (see the JIRA for full details). The default is set to an intentionally high value in order to avoid performance regressions. Testing: * Added a new planner test to fuctional-planner/ to validate that 'topn_bytes_limit' properly switches between topn and sort operators. Change-Id: I34c9db33c9302b55e9978f53f9c7061f2806c8a9 Reviewed-on: http://gerrit.cloudera.org:8080/11698 Tested-by: Impala Public Jenkins Reviewed-by: Tim Armstrong --- M be/src/service/query-options-test.cc M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M fe/src/main/java/org/apache/impala/analysis/SortInfo.java M fe/src/main/java/org/apache/impala/planner/AggregationNode.java M fe/src/main/java/org/apache/impala/planner/AnalyticEvalNode.java M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java M fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java M fe/src/main/java/org/apache/impala/planner/HBaseScanNode.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/planner/JoinNode.java M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java M fe/src/main/java/org/apache/impala/planner/PlanNode.java M fe/src/main/java/org/apache/impala/planner/SelectNode.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/main/java/org/apache/impala/planner/SortNode.java M fe/src/main/java/org/apache/impala/planner/SubplanNode.java M fe/src/main/java/org/apache/impala/planner/UnionNode.java M fe/src/main/java/org/apache/impala/planner/UnnestNode.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java A testdata/workloads/functional-planner/queries/PlannerTest/topn-bytes-limit-small.test A testdata/workloads/functional-planner/queries/PlannerTest/topn-bytes-limit.test M testdata/workloads/functional-query/queries/QueryTest/spilling-no-debug-action.test 25 files changed, 205 insertions(+), 38 deletions(-) Approvals: Impala Public Jenkins: Verified Tim Armstrong: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/11698 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I34c9db33c9302b55e9978f53f9c7061f2806c8a9 Gerrit-Change-Number: 11698 Gerrit-PatchSet: 11 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7822: handle overflows in repeat() builtin
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11889 ) Change subject: IMPALA-7822: handle overflows in repeat() builtin .. Patch Set 2: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/3432/ -- To view, visit http://gerrit.cloudera.org:8080/11889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I872ce77bc2cb29116881c27ca2a5216f722cdb2a Gerrit-Change-Number: 11889 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Wed, 07 Nov 2018 01:53:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable
Fredy Wijaya has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/11897 ) Change subject: IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable .. IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable Before this patch, running INVALIDATE METADATA when Sentry is unavailable could causue Impala to hang. PolicyReader thread in SentryProxy is used by two use cases, one as a background thread that periodically refreshes Sentry policy and another one as a synchronous operation for INVALIDATE METADATA. For the background thread, we need to swallow any exception thrown while refreshing the Sentry policy in order to not kill the background thread. For a synchronous reset operation, such as INVALIDATE METADATA, swallowing an exception cause the Impala catalog to wait indefinitely for authorization catalog objects that never get processed due to Sentry being unavailable. The patch updates the code by not swallowing any exception in INVALIDATE METADATA and return the exception to the caller. Testing: - Added a new E2E test - Ran all E2E authorization tests Change-Id: Icff987a6184f62a338faadfdc1a0d349d912fc37 --- M fe/src/main/java/org/apache/impala/util/SentryProxy.java M tests/authorization/test_authorization.py M tests/common/custom_cluster_test_suite.py 3 files changed, 51 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/97/11897/5 -- To view, visit http://gerrit.cloudera.org:8080/11897 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Icff987a6184f62a338faadfdc1a0d349d912fc37 Gerrit-Change-Number: 11897 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7808: Refactor Analyzer for easier debugging
Paul Rogers has posted comments on this change. (
http://gerrit.cloudera.org:8080/11883 )
Change subject: IMPALA-7808: Refactor Analyzer for easier debugging
..
Patch Set 2:
(12 comments)
Thanks for the reviews. Addressed the comments.
If you are ready, I'll go ahead and shift the indent of the methods in the
inner class, which will mark a zillion likes as changed. In its current form,
you can more easily see the actual changes.
http://gerrit.cloudera.org:8080/#/c/11883/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/11883/1//COMMIT_MSG@37
PS1, Line 37: Reran a
> pls move this testing section above the Change-Id
Done
http://gerrit.cloudera.org:8080/#/c/11883/1/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
File fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java:
http://gerrit.cloudera.org:8080/#/c/11883/1/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java@461
PS1, Line 461: r
> nit: extra space
Done
http://gerrit.cloudera.org:8080/#/c/11883/1/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java@461
PS1, Line 461: eturn;
> We generally do not use braces in one line "if (cond) stmt;".
Thanks. Every project has its quirks; I 'm still learning Impala's preferences.
Fixed.
http://gerrit.cloudera.org:8080/#/c/11883/1/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
File fe/src/main/java/org/apache/impala/analysis/SelectStmt.java:
http://gerrit.cloudera.org:8080/#/c/11883/1/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@176
PS1, Line 176: private final Analyzer analyzer_;
> these members should be suffixed with "_"
Done
http://gerrit.cloudera.org:8080/#/c/11883/1/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@186
PS1, Line 186: Note tem
> note, you can always include such review-level comments as comments in gerr
Agreed. The comment is here to rather force the need to come back and shift
indentation before the final +2. At that point, the comment will be removed.
http://gerrit.cloudera.org:8080/#/c/11883/1/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@219
PS1, Line 219: if (multiAggInfo_ != null &&
multiAggInfo_.hasAggregateExprs()) {
> here and at many other functions: in Impala we generally do not add an empt
Done
http://gerrit.cloudera.org:8080/#/c/11883/1/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@389
PS1, Line 389: if (t
> did some other class need this protected?
Tried to minimize extra changes. It works to make this private, so went ahead
and did so.
http://gerrit.cloudera.org:8080/#/c/11883/1/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@526
PS1, Line 526: c
> nit: extra space
Done
http://gerrit.cloudera.org:8080/#/c/11883/1/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@617
PS1, Line 617: su
> why public?
Done
http://gerrit.cloudera.org:8080/#/c/11883/1/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@659
PS1, Line 659:
> why public?
Done
http://gerrit.cloudera.org:8080/#/c/11883/1/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@757
PS1, Line 757: }
> why public?
Done
http://gerrit.cloudera.org:8080/#/c/11883/1/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@894
PS1, Line 894:
> just to remind myelf... end of inner class.
Yes. When you are satisfied that nothing changed other than the inner class,
I'll go ahead and indent the whole mess.
--
To view, visit http://gerrit.cloudera.org:8080/11883
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I576c80c4c7a974df226fc91d8903db275069ed52
Gerrit-Change-Number: 11883
Gerrit-PatchSet: 2
Gerrit-Owner: Paul Rogers
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Paul Rogers
Gerrit-Reviewer: Vuk Ercegovac
Gerrit-Comment-Date: Tue, 06 Nov 2018 20:44:33 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7477: Batch-oriented query set construction
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11879 ) Change subject: IMPALA-7477: Batch-oriented query set construction .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/11879 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I764fa302842438902cd5db2551ec6e3cb77b6874 Gerrit-Change-Number: 11879 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 06 Nov 2018 21:27:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7808: Refactor Analyzer for easier debugging
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11883 ) Change subject: IMPALA-7808: Refactor Analyzer for easier debugging .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1295/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11883 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I576c80c4c7a974df226fc91d8903db275069ed52 Gerrit-Change-Number: 11883 Gerrit-PatchSet: 2 Gerrit-Owner: Paul Rogers Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Tue, 06 Nov 2018 21:27:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5004: Switch to sorting node for large TopN queries
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11698 ) Change subject: IMPALA-5004: Switch to sorting node for large TopN queries .. Patch Set 10: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/11698 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I34c9db33c9302b55e9978f53f9c7061f2806c8a9 Gerrit-Change-Number: 11698 Gerrit-PatchSet: 10 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Tue, 06 Nov 2018 20:02:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7795: Implement REFRESH AUTHORIZATION statement
Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11888 Change subject: IMPALA-7795: Implement REFRESH AUTHORIZATION statement .. IMPALA-7795: Implement REFRESH AUTHORIZATION statement This patch implements REFRESH AUTHORIZATION statement to explicitly refresh authorization metadata. This statement is useful to force Impala to refresh its authorization metadata when there is an external update to authorization metadata without having to wait for the Sentry polling or call INVALIDATE METADATA. Some tests were updated to use REFRESH AUTHORIZATION instead of INVALIDATE METADATA to make the tests run faster. Syntax: REFRESH AUTHORIZATION (authorization must be enabled to execute this statement) Testing: - Added new FE tests - Added new E2E authorization tests - Ran all FE tests - Ran all E2E authorization tests Change-Id: I5459e1c97b12dee307e0cf85b94a9f66fd9d9a8c --- M common/thrift/CatalogService.thrift M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/ResetMetadataStmt.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/jflex/sql-scanner.flex M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java M fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M fe/src/test/java/org/apache/impala/analysis/StmtMetadataLoaderTest.java M fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java M tests/authorization/test_authorization.py M tests/authorization/test_grant_revoke.py M tests/authorization/test_owner_privileges.py M tests/common/sentry_cache_test_suite.py 15 files changed, 259 insertions(+), 86 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/88/11888/3 -- To view, visit http://gerrit.cloudera.org:8080/11888 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I5459e1c97b12dee307e0cf85b94a9f66fd9d9a8c Gerrit-Change-Number: 11888 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya
[Impala-ASF-CR] IMPALA-7795: Implement REFRESH AUTHORIZATION statement
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11888 ) Change subject: IMPALA-7795: Implement REFRESH AUTHORIZATION statement .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/11888/3/tests/common/sentry_cache_test_suite.py File tests/common/sentry_cache_test_suite.py: http://gerrit.cloudera.org:8080/#/c/11888/3/tests/common/sentry_cache_test_suite.py@66 PS3, Line 66: n flake8: E501 line too long (92 > 90 characters) -- To view, visit http://gerrit.cloudera.org:8080/11888 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5459e1c97b12dee307e0cf85b94a9f66fd9d9a8c Gerrit-Change-Number: 11888 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Tue, 06 Nov 2018 20:31:41 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-3652: Fix resource transfer in subplans with limits
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11852 ) Change subject: IMPALA-3652: Fix resource transfer in subplans with limits .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1297/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11852 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3968a379fcbb5d30fcec304995d3e44933dbbc77 Gerrit-Change-Number: 11852 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 06 Nov 2018 21:47:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7808: Refactor Analyzer for easier debugging
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/11883 )
Change subject: IMPALA-7808: Refactor Analyzer for easier debugging
..
Patch Set 3:
(11 comments)
http://gerrit.cloudera.org:8080/#/c/11883/3/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
File fe/src/main/java/org/apache/impala/analysis/SelectStmt.java:
http://gerrit.cloudera.org:8080/#/c/11883/3/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@219
PS3, Line 219: // When there is a LIMIT clause in conjunction with an
ORDER BY, the ordering exprs
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/11883/3/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@274
PS3, Line 274: "Star exprs only expand to scalar-typed columns
because complex-typed exprs " +
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/11883/3/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@326
PS3, Line 326: * Generates and registers !empty() predicates to filter out
empty collections directly
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/11883/3/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@341
PS3, Line 341: * are required
so are checked for !empty.
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/11883/3/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@347
PS3, Line 347: * TODO: In some cases, it is possible to generate !empty()
predicates for a correlated
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/11883/3/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@366
PS3, Line 366: if
(analyzer_.isOuterJoined(ref.getResolvedPath().getRootDesc().getId())) continue;
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/11883/3/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@425
PS3, Line 425: throw new AnalysisException("'*' expression in select
list requires FROM clause.");
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/11883/3/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@665
PS3, Line 665: List substAggExprs =
Expr.substituteList(aggExprs_, ndvSmap_, analyzer_, false);
line too long (94 > 90)
http://gerrit.cloudera.org:8080/#/c/11883/3/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@681
PS3, Line 681: // However the original COUNT(c) should have returned 0
instead of NULL in this case.
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/11883/3/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@756
PS3, Line 756: if
(!sortInfo_.getSortExprs().get(i).isBound(multiAggInfo_.getResultTupleId())) {
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/11883/3/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@860
PS3, Line 860: // If 'exprRewritten' is true, we have to compose the new
smap with the existing one.
line too long (91 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/11883
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I576c80c4c7a974df226fc91d8903db275069ed52
Gerrit-Change-Number: 11883
Gerrit-PatchSet: 3
Gerrit-Owner: Paul Rogers
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Paul Rogers
Gerrit-Reviewer: Vuk Ercegovac
Gerrit-Comment-Date: Tue, 06 Nov 2018 22:20:29 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7808: Refactor Analyzer for easier debugging
Hello Csaba Ringhofer, Impala Public Jenkins, Vuk Ercegovac, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11883 to look at the new patch set (#3). Change subject: IMPALA-7808: Refactor Analyzer for easier debugging .. IMPALA-7808: Refactor Analyzer for easier debugging Changes two blocks of code to make debugging easier. No functional changes occur; changes are pure refactoring. A trivial change in AnalyzerContext removes a nested conditional clause. A larger change in SelectStmt takes the large analysis function and breaks it into a series of smaller functions. The functions were large because they shared state: variables created near the top are used much later near the bottom. To solve this, moved the code into an "algorithm" class whose only job is to hold onto the temporary state so that the big function can be broken into smaller pieces, with the temporary class fields used in place of the former local variables. For the most part, the existign code was simply split into functions and indented. One block of code had to be moved below the inner class since it is not part of the analysis process. Testing: No functional change, changes are purely structure. Reran all tests, which passed. Change-Id: I576c80c4c7a974df226fc91d8903db275069ed52 --- M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java 2 files changed, 751 insertions(+), 688 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/11883/3 -- To view, visit http://gerrit.cloudera.org:8080/11883 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I576c80c4c7a974df226fc91d8903db275069ed52 Gerrit-Change-Number: 11883 Gerrit-PatchSet: 3 Gerrit-Owner: Paul Rogers Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7788: [DOCS] Impala supports ADLS Gen 2 (ABFS)
Jim Apple has posted comments on this change. ( http://gerrit.cloudera.org:8080/11853 ) Change subject: IMPALA-7788: [DOCS] Impala supports ADLS Gen 2 (ABFS) .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11853 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic06d9ac92ed78b9092369e211de8a81db1d7ce90 Gerrit-Change-Number: 11853 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Tue, 06 Nov 2018 23:54:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7819: add per-node storage wait time to profile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11894 ) Change subject: IMPALA-7819: add per-node storage wait time to profile .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1303/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11894 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I74a496efdbe9e804db779c9753b00a303a270580 Gerrit-Change-Number: 11894 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 07 Nov 2018 00:41:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7789: [DOCS] Admission status in Impala Shell
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11895 ) Change subject: IMPALA-7789: [DOCS] Admission status in Impala Shell .. Patch Set 1: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/136/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/11895 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I17d788eb716c6a2f7a144ee2d81bbe823f74d16a Gerrit-Change-Number: 11895 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 07 Nov 2018 01:25:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7789: [DOCS] Admission status in Impala Shell
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11895 ) Change subject: IMPALA-7789: [DOCS] Admission status in Impala Shell .. Patch Set 1: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/136/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/11895 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I17d788eb716c6a2f7a144ee2d81bbe823f74d16a Gerrit-Change-Number: 11895 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 07 Nov 2018 01:35:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5821: Add query with implicit casts to extended explain output.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11719 ) Change subject: IMPALA-5821: Add query with implicit casts to extended explain output. .. Patch Set 7: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/11719 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I55c3bdacc295137f66b2316a912fc347da30d6b0 Gerrit-Change-Number: 11719 Gerrit-PatchSet: 7 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 07 Nov 2018 03:28:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7819: add per-node storage wait time to profile
Hello Lars Volker, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11894 to look at the new patch set (#2). Change subject: IMPALA-7819: add per-node storage wait time to profile .. IMPALA-7819: add per-node storage wait time to profile Adds ScannerIoWaitTime and KuduClientTime to measure time spent waiting for the I/O manager and time spent in the Kudu client, respectively. To minimise the additional overhead, add support to the timer infrastructure to update multiple counters with one stopwatch. Apply this in some other places too. Testing: Ran core tests. Ran some queries manually with and without mt_dop and sanity-checked the profiles. Change-Id: I74a496efdbe9e804db779c9753b00a303a270580 --- M be/src/exec/hdfs-orc-scanner.cc M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/hdfs-scan-node-base.h M be/src/exec/kudu-scan-node-base.cc M be/src/exec/kudu-scan-node-base.h M be/src/exec/kudu-scanner.cc M be/src/exec/scanner-context.cc M be/src/runtime/fragment-instance-state.cc M be/src/runtime/krpc-data-stream-recvr.cc M be/src/util/runtime-profile-counters.h 10 files changed, 85 insertions(+), 43 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/94/11894/2 -- To view, visit http://gerrit.cloudera.org:8080/11894 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I74a496efdbe9e804db779c9753b00a303a270580 Gerrit-Change-Number: 11894 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7819: add per-node storage wait time to profile
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11894 ) Change subject: IMPALA-7819: add per-node storage wait time to profile .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/11894/1/be/src/util/runtime-profile-counters.h File be/src/util/runtime-profile-counters.h: http://gerrit.cloudera.org:8080/#/c/11894/1/be/src/util/runtime-profile-counters.h@59 PS1, Line 59: ScopedTimer MACRO_CONCAT(SCOPED_TIMER_COUNTER, __COUNTER__)(c1, c2) > line too long (93 > 90) Done -- To view, visit http://gerrit.cloudera.org:8080/11894 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I74a496efdbe9e804db779c9753b00a303a270580 Gerrit-Change-Number: 11894 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 07 Nov 2018 00:17:02 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7789: [DOCS] Admission status in Impala Shell
Alex Rodoni has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11895 Change subject: IMPALA-7789: [DOCS] Admission status in Impala Shell .. IMPALA-7789: [DOCS] Admission status in Impala Shell Change-Id: I17d788eb716c6a2f7a144ee2d81bbe823f74d16a --- M docs/topics/impala_admission.xml M docs/topics/impala_live_progress.xml M docs/topics/impala_live_summary.xml 3 files changed, 97 insertions(+), 113 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/95/11895/1 -- To view, visit http://gerrit.cloudera.org:8080/11895 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I17d788eb716c6a2f7a144ee2d81bbe823f74d16a Gerrit-Change-Number: 11895 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni
[Impala-ASF-CR] IMPALA-7791: Compute AggregationNode's estimated rows using # instances
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11854 ) Change subject: IMPALA-7791: Compute AggregationNode's estimated rows using # instances .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11854 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2cb9746fafa3e5952e28caa952837e285bcc22ac Gerrit-Change-Number: 11854 Gerrit-PatchSet: 2 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 07 Nov 2018 01:37:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7808: Refactor Analyzer for easier debugging
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11883 ) Change subject: IMPALA-7808: Refactor Analyzer for easier debugging .. IMPALA-7808: Refactor Analyzer for easier debugging Changes two blocks of code to make debugging easier. No functional changes occur; changes are pure refactoring. A trivial change in AnalyzerContext removes a nested conditional clause. A larger change in SelectStmt takes the large analysis function and breaks it into a series of smaller functions. The functions were large because they shared state: variables created near the top are used much later near the bottom. To solve this, moved the code into an "algorithm" class whose only job is to hold onto the temporary state so that the big function can be broken into smaller pieces, with the temporary class fields used in place of the former local variables. For the most part, the existign code was simply split into functions and indented. One block of code had to be moved below the inner class since it is not part of the analysis process. Testing: No functional change, changes are purely structure. Reran all tests, which passed. Change-Id: I576c80c4c7a974df226fc91d8903db275069ed52 Reviewed-on: http://gerrit.cloudera.org:8080/11883 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java 2 files changed, 767 insertions(+), 688 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/11883 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I576c80c4c7a974df226fc91d8903db275069ed52 Gerrit-Change-Number: 11883 Gerrit-PatchSet: 6 Gerrit-Owner: Paul Rogers Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7808: Refactor Analyzer for easier debugging
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11883 ) Change subject: IMPALA-7808: Refactor Analyzer for easier debugging .. Patch Set 5: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/11883 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I576c80c4c7a974df226fc91d8903db275069ed52 Gerrit-Change-Number: 11883 Gerrit-PatchSet: 5 Gerrit-Owner: Paul Rogers Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 07 Nov 2018 02:56:44 + Gerrit-HasComments: No
[Impala-ASF-CR] Bump CDH BUILD VERSION to include Hive fix for RELY constraint
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11891 ) Change subject: Bump CDH_BUILD_VERSION to include Hive fix for RELY constraint .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/11891 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie626882044fea98b81b07caa950b480e16df122d Gerrit-Change-Number: 11891 Gerrit-PatchSet: 1 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Wed, 07 Nov 2018 02:46:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/11897 ) Change subject: IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/11897/4/tests/authorization/test_authorization.py File tests/authorization/test_authorization.py: http://gerrit.cloudera.org:8080/#/c/11897/4/tests/authorization/test_authorization.py@500 PS4, Line 500: # > flake8: E303 too many blank lines (2) Done -- To view, visit http://gerrit.cloudera.org:8080/11897 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icff987a6184f62a338faadfdc1a0d349d912fc37 Gerrit-Change-Number: 11897 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 07 Nov 2018 04:19:37 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable
Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11897 Change subject: IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable .. IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable Before this patch, running INVALIDATE METADATA when Sentry is unavailable could causue Impala to hang. PolicyReader thread in SentryProxy is used by two use cases, one as a background thread that periodically refreshes Sentry policy and another one as a synchronous operation for INVALIDATE METADATA. For the background thread, we need to swallow any exception thrown while refreshing the Sentry policy in order to not kill the background thread. For a synchronous reset operation, such as INVALIDATE METADATA, swallowing an exception cause the Impala catalog to wait indefinitely for authorization catalog objects that never get processed due to Sentry being unavailable. The patch updates the code by not swallowing any exception in INVALIDATE METADATA and return the exception to the caller. Testing: - Added a new E2E test - Ran all E2E authorization tests Change-Id: Icff987a6184f62a338faadfdc1a0d349d912fc37 --- M fe/src/main/java/org/apache/impala/util/SentryProxy.java M tests/authorization/test_authorization.py M tests/common/custom_cluster_test_suite.py 3 files changed, 52 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/97/11897/4 -- To view, visit http://gerrit.cloudera.org:8080/11897 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Icff987a6184f62a338faadfdc1a0d349d912fc37 Gerrit-Change-Number: 11897 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya
[Impala-ASF-CR] IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11897 ) Change subject: IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/11897/4/tests/authorization/test_authorization.py File tests/authorization/test_authorization.py: http://gerrit.cloudera.org:8080/#/c/11897/4/tests/authorization/test_authorization.py@500 PS4, Line 500: s flake8: E303 too many blank lines (2) -- To view, visit http://gerrit.cloudera.org:8080/11897 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icff987a6184f62a338faadfdc1a0d349d912fc37 Gerrit-Change-Number: 11897 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 07 Nov 2018 04:18:30 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11897 ) Change subject: IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1306/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11897 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icff987a6184f62a338faadfdc1a0d349d912fc37 Gerrit-Change-Number: 11897 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 07 Nov 2018 04:59:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11897 ) Change subject: IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1305/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11897 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icff987a6184f62a338faadfdc1a0d349d912fc37 Gerrit-Change-Number: 11897 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 07 Nov 2018 04:55:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7761: Add multiple DISTINCT to targeted perf and stress test
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11805 ) Change subject: IMPALA-7761: Add multiple DISTINCT to targeted perf and stress test .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1292/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11805 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I400aaf6b6620b4001895eafff785956bffb312c9 Gerrit-Change-Number: 11805 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 06 Nov 2018 21:01:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7477: Batch-oriented query set construction
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11879 ) Change subject: IMPALA-7477: Batch-oriented query set construction .. IMPALA-7477: Batch-oriented query set construction Rework the row-by-row construction of query result sets in PlanRootSink so that it materialises an output column at a time. Make some minor optimisations like preallocating output vectors and initialising strings more efficiently. My intent is both to make this faster and to make the QueryResultSet interface better before IMPALA-4268 does a bunch of surgery on this part of the code. Testing: Ran core tests. Perf: Downloaded tpch_parquet.orders via JDBC driver. Before: 3.01s, After: 2.57s. Downloaded l_orderkey from tpch_parquet.lineitem. Before: 1.21s, After: 1.08s. Change-Id: I764fa302842438902cd5db2551ec6e3cb77b6874 Reviewed-on: http://gerrit.cloudera.org:8080/11879 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/exec/plan-root-sink.cc M be/src/exec/plan-root-sink.h M be/src/service/hs2-util.cc M be/src/service/hs2-util.h M be/src/service/query-result-set.cc M be/src/service/query-result-set.h 6 files changed, 331 insertions(+), 161 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/11879 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I764fa302842438902cd5db2551ec6e3cb77b6874 Gerrit-Change-Number: 11879 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-7822: handle overflows in repeat() builtin
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11889 ) Change subject: IMPALA-7822: handle overflows in repeat() builtin .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1298/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I872ce77bc2cb29116881c27ca2a5216f722cdb2a Gerrit-Change-Number: 11889 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 06 Nov 2018 22:33:43 + Gerrit-HasComments: No
[Impala-ASF-CR] Bump CDH BUILD VERSION to include Hive fix for RELY constraint
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11891 ) Change subject: Bump CDH_BUILD_VERSION to include Hive fix for RELY constraint .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1302/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11891 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie626882044fea98b81b07caa950b480e16df122d Gerrit-Change-Number: 11891 Gerrit-PatchSet: 1 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 06 Nov 2018 23:16:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7808: Refactor Analyzer for easier debugging
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11883 ) Change subject: IMPALA-7808: Refactor Analyzer for easier debugging .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/11883/2/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java File fe/src/main/java/org/apache/impala/analysis/SelectStmt.java: http://gerrit.cloudera.org:8080/#/c/11883/2/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@669 PS2, Line 669: List substAggExprs = Expr.substituteList(aggExprs_, ndvSmap_, analyzer_, false); line too long (92 > 90) -- To view, visit http://gerrit.cloudera.org:8080/11883 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I576c80c4c7a974df226fc91d8903db275069ed52 Gerrit-Change-Number: 11883 Gerrit-PatchSet: 2 Gerrit-Owner: Paul Rogers Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Tue, 06 Nov 2018 20:42:51 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7808: Refactor Analyzer for easier debugging
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11883 ) Change subject: IMPALA-7808: Refactor Analyzer for easier debugging .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3434/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/11883 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I576c80c4c7a974df226fc91d8903db275069ed52 Gerrit-Change-Number: 11883 Gerrit-PatchSet: 5 Gerrit-Owner: Paul Rogers Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Tue, 06 Nov 2018 23:04:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7808: Refactor Analyzer for easier debugging
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11883 ) Change subject: IMPALA-7808: Refactor Analyzer for easier debugging .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11883 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I576c80c4c7a974df226fc91d8903db275069ed52 Gerrit-Change-Number: 11883 Gerrit-PatchSet: 5 Gerrit-Owner: Paul Rogers Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Tue, 06 Nov 2018 23:04:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7808: Refactor Analyzer for easier debugging
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11883 ) Change subject: IMPALA-7808: Refactor Analyzer for easier debugging .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1300/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11883 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I576c80c4c7a974df226fc91d8903db275069ed52 Gerrit-Change-Number: 11883 Gerrit-PatchSet: 3 Gerrit-Owner: Paul Rogers Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Tue, 06 Nov 2018 23:04:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5821: Add query with implicit casts to extended explain output.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11719 ) Change subject: IMPALA-5821: Add query with implicit casts to extended explain output. .. Patch Set 7: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3435/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/11719 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I55c3bdacc295137f66b2316a912fc347da30d6b0 Gerrit-Change-Number: 11719 Gerrit-PatchSet: 7 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Tue, 06 Nov 2018 23:31:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5004: Switch to sorting node for large TopN queries
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/11698 ) Change subject: IMPALA-5004: Switch to sorting node for large TopN queries .. Patch Set 10: Tests are passing now. So this patch should be good to merge. The first set of failures in https://jenkins.impala.io/job/gerrit-verify-dryrun/3373/ was due to a change that I have now reverted. The change attempted to dynamically switch from TopN to Sort if the input cardinality of the TopN operator exceeded the value of the limit, after thinking through this change some more and doing some more profiling I realized it does not make sense to implement. The second set of failures in https://jenkins.impala.io/job/gerrit-verify-dryrun/3413/ seems to be due to flaky test failures. -- To view, visit http://gerrit.cloudera.org:8080/11698 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I34c9db33c9302b55e9978f53f9c7061f2806c8a9 Gerrit-Change-Number: 11698 Gerrit-PatchSet: 10 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Tue, 06 Nov 2018 20:23:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7808: Refactor Analyzer for easier debugging
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/11883 ) Change subject: IMPALA-7808: Refactor Analyzer for easier debugging .. Patch Set 2: Code-Review+2 go ahead and apply the correct indentation. -- To view, visit http://gerrit.cloudera.org:8080/11883 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I576c80c4c7a974df226fc91d8903db275069ed52 Gerrit-Change-Number: 11883 Gerrit-PatchSet: 2 Gerrit-Owner: Paul Rogers Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Tue, 06 Nov 2018 21:20:36 + Gerrit-HasComments: No
[Impala-ASF-CR] Bump CDH BUILD VERSION to include Hive fix for RELY constraint
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11891 ) Change subject: Bump CDH_BUILD_VERSION to include Hive fix for RELY constraint .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3433/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/11891 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie626882044fea98b81b07caa950b480e16df122d Gerrit-Change-Number: 11891 Gerrit-PatchSet: 1 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 06 Nov 2018 22:52:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7821: Refactor ExprRewriteRulesTest for easier debugging
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11890 ) Change subject: IMPALA-7821: Refactor ExprRewriteRulesTest for easier debugging .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1299/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11890 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I99a28b170bc0132424041e51f61ebe5c848c4083 Gerrit-Change-Number: 11890 Gerrit-PatchSet: 1 Gerrit-Owner: Paul Rogers Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 06 Nov 2018 22:57:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5821: Add query with implicit casts to extended explain output.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11719 ) Change subject: IMPALA-5821: Add query with implicit casts to extended explain output. .. Patch Set 7: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11719 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I55c3bdacc295137f66b2316a912fc347da30d6b0 Gerrit-Change-Number: 11719 Gerrit-PatchSet: 7 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Tue, 06 Nov 2018 23:31:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5821: Add query with implicit casts to extended explain output.
Thomas Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/11719 ) Change subject: IMPALA-5821: Add query with implicit casts to extended explain output. .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11719 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I55c3bdacc295137f66b2316a912fc347da30d6b0 Gerrit-Change-Number: 11719 Gerrit-PatchSet: 6 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Tue, 06 Nov 2018 23:31:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7761: Add multiple DISTINCT to targeted stress and perf
Hello Michael Brown, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11805 to look at the new patch set (#2). Change subject: IMPALA-7761: Add multiple DISTINCT to targeted stress and perf .. IMPALA-7761: Add multiple DISTINCT to targeted stress and perf IMPALA-110 added support for queries with multiple DISTINCT aggregates in a single select list. This patch adds queries to test this functionality to our targeted-stress and targeted-perf workloads. Also fixes some incorrect return types in another targeted-perf aggregation query. Testing: - Ran the test file locally. Change-Id: I400aaf6b6620b4001895eafff785956bffb312c9 --- M testdata/workloads/targeted-perf/queries/aggregation.test M testdata/workloads/tpcds/queries/tpcds-q39-1.test M testdata/workloads/tpcds/queries/tpcds-q39-2.test A testdata/workloads/tpch/queries/tpch-stress-agg1.test M tests/infra/test_stress_infra.py M tests/util/parse_util.py M tests/util/test_file_parser.py 7 files changed, 109 insertions(+), 13 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/11805/2 -- To view, visit http://gerrit.cloudera.org:8080/11805 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I400aaf6b6620b4001895eafff785956bffb312c9 Gerrit-Change-Number: 11805 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-3652: Fix resource transfer in subplans with limits
Hello Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11852 to look at the new patch set (#2). Change subject: IMPALA-3652: Fix resource transfer in subplans with limits .. IMPALA-3652: Fix resource transfer in subplans with limits Impala assumes that when Reset() is called on an ExecNode, all of the memory returned from that node by GetNext() has been attached to the output RowBatch. In a query with a LIMIT on the subplan, such that some nodes don't reach 'eos', this may not be the case. The solution is to have Reset() take a RowBatch that any such memory can be attached to. I examined all ExecNodes for resources being transferred on 'eos' and added transferring of those resources in Resst(). Testing: - Added e2e tests that repro the issue for hash and nested loop joins. Change-Id: I3968a379fcbb5d30fcec304995d3e44933dbbc77 --- M be/src/exec/aggregation-node-base.cc M be/src/exec/aggregation-node-base.h M be/src/exec/aggregator.h M be/src/exec/analytic-eval-node.cc M be/src/exec/analytic-eval-node.h M be/src/exec/blocking-join-node.cc M be/src/exec/blocking-join-node.h M be/src/exec/cardinality-check-node.cc M be/src/exec/cardinality-check-node.h M be/src/exec/data-source-scan-node.cc M be/src/exec/data-source-scan-node.h M be/src/exec/exchange-node.cc M be/src/exec/exchange-node.h M be/src/exec/exec-node.cc M be/src/exec/exec-node.h M be/src/exec/grouping-aggregator.cc M be/src/exec/grouping-aggregator.h M be/src/exec/hbase-scan-node.cc M be/src/exec/hbase-scan-node.h M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/hdfs-scan-node-base.h M be/src/exec/nested-loop-join-node.cc M be/src/exec/nested-loop-join-node.h M be/src/exec/non-grouping-aggregator.h M be/src/exec/partial-sort-node.cc M be/src/exec/partial-sort-node.h M be/src/exec/partitioned-hash-join-builder.cc M be/src/exec/partitioned-hash-join-builder.h M be/src/exec/partitioned-hash-join-node.cc M be/src/exec/partitioned-hash-join-node.h M be/src/exec/select-node.cc M be/src/exec/select-node.h M be/src/exec/sort-node.cc M be/src/exec/sort-node.h M be/src/exec/streaming-aggregation-node.cc M be/src/exec/streaming-aggregation-node.h M be/src/exec/subplan-node.cc M be/src/exec/subplan-node.h M be/src/exec/topn-node.cc M be/src/exec/topn-node.h M be/src/exec/union-node.cc M be/src/exec/union-node.h M be/src/exec/unnest-node.cc M be/src/exec/unnest-node.h A testdata/workloads/functional-query/queries/QueryTest/nested-types-tpch-limit.test M tests/query_test/test_nested_types.py 46 files changed, 149 insertions(+), 86 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/11852/2 -- To view, visit http://gerrit.cloudera.org:8080/11852 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3968a379fcbb5d30fcec304995d3e44933dbbc77 Gerrit-Change-Number: 11852 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-3652: Fix resource transfer in subplans with limits
Thomas Marshall has posted comments on this change. (
http://gerrit.cloudera.org:8080/11852 )
Change subject: IMPALA-3652: Fix resource transfer in subplans with limits
..
Patch Set 2:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/11852/1/be/src/exec/aggregation-node-base.cc
File be/src/exec/aggregation-node-base.cc:
http://gerrit.cloudera.org:8080/#/c/11852/1/be/src/exec/aggregation-node-base.cc@77
PS1, Line 77: for (auto& agg : aggs_) RETURN_IF_ERROR(agg->Reset(state,
row_batch));
> I think there are cases where previous batches reference memory owned by th
Done
http://gerrit.cloudera.org:8080/#/c/11852/1/be/src/exec/analytic-eval-node.cc
File be/src/exec/analytic-eval-node.cc:
http://gerrit.cloudera.org:8080/#/c/11852/1/be/src/exec/analytic-eval-node.cc@812
PS1, Line 812: Status AnalyticEvalNode::Reset(RuntimeState* state, RowBatch*
row_batch) {
> I feel like we need some of the logic from the eos branch on line 783 here
Done
http://gerrit.cloudera.org:8080/#/c/11852/1/be/src/exec/partitioned-hash-join-node.cc
File be/src/exec/partitioned-hash-join-node.cc:
http://gerrit.cloudera.org:8080/#/c/11852/1/be/src/exec/partitioned-hash-join-node.cc@200
PS1, Line 200: Status PartitionedHashJoinNode::Reset(RuntimeState* state,
RowBatch* row_batch) {
> I think it's possible that returned rows could reference memory from in-mem
Done
http://gerrit.cloudera.org:8080/#/c/11852/1/be/src/exec/partitioned-hash-join-node.cc@211
PS1, Line 211: if (output_unmatched_batch_ != nullptr) {
> I don't understand why this is true - couldn't the previous GetNext() call
Done
http://gerrit.cloudera.org:8080/#/c/11852/1/be/src/exec/subplan-node.cc
File be/src/exec/subplan-node.cc:
http://gerrit.cloudera.org:8080/#/c/11852/1/be/src/exec/subplan-node.cc@91
PS1, Line 91: transferre
> nit: transferred
Done
http://gerrit.cloudera.org:8080/#/c/11852/1/be/src/exec/topn-node.cc
File be/src/exec/topn-node.cc:
http://gerrit.cloudera.org:8080/#/c/11852/1/be/src/exec/topn-node.cc@219
PS1, Line 219: row_batch->tuple_data_pool()->AcquireData(tuple_pool_.get(),
false);
> I think we'd need some very weird query shape with limits on subqueries at
Done
http://gerrit.cloudera.org:8080/#/c/11852/1/testdata/workloads/functional-query/queries/QueryTest/nested-types-tpch.test
File
testdata/workloads/functional-query/queries/QueryTest/nested-types-tpch.test:
http://gerrit.cloudera.org:8080/#/c/11852/1/testdata/workloads/functional-query/queries/QueryTest/nested-types-tpch.test@195
PS1, Line 195:
> Do these options take effect? If batch_size is part of the test vector that
Done
--
To view, visit http://gerrit.cloudera.org:8080/11852
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3968a379fcbb5d30fcec304995d3e44933dbbc77
Gerrit-Change-Number: 11852
Gerrit-PatchSet: 2
Gerrit-Owner: Thomas Marshall
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Thomas Marshall
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Tue, 06 Nov 2018 21:25:17 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7761: Add multiple DISTINCT to targeted perf and stress test
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11805 ) Change subject: IMPALA-7761: Add multiple DISTINCT to targeted perf and stress test .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1296/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11805 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I400aaf6b6620b4001895eafff785956bffb312c9 Gerrit-Change-Number: 11805 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 06 Nov 2018 21:33:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7808: Refactor Analyzer for easier debugging
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/11883 ) Change subject: IMPALA-7808: Refactor Analyzer for easier debugging .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11883 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I576c80c4c7a974df226fc91d8903db275069ed52 Gerrit-Change-Number: 11883 Gerrit-PatchSet: 4 Gerrit-Owner: Paul Rogers Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Tue, 06 Nov 2018 23:03:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7809: support Kudu 1.9 in test concurrent schema change
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11882 ) Change subject: IMPALA-7809: support Kudu 1.9 in test_concurrent_schema_change .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/11882 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7fc24bb6a18aecc0cb726b8d66f0aeccf56bbb9b Gerrit-Change-Number: 11882 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Brown Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 06 Nov 2018 20:08:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7809: support Kudu 1.9 in test concurrent schema change
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11882 ) Change subject: IMPALA-7809: support Kudu 1.9 in test_concurrent_schema_change .. IMPALA-7809: support Kudu 1.9 in test_concurrent_schema_change This patch extends the match of allowable errors in test_concurrent_schema_change to work with Kudu 1.9. Testing: Local with a Kudu 1.9 minicluster environment. Change-Id: I7fc24bb6a18aecc0cb726b8d66f0aeccf56bbb9b Reviewed-on: http://gerrit.cloudera.org:8080/11882 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M tests/query_test/test_kudu.py 1 file changed, 2 insertions(+), 1 deletion(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/11882 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I7fc24bb6a18aecc0cb726b8d66f0aeccf56bbb9b Gerrit-Change-Number: 11882 Gerrit-PatchSet: 3 Gerrit-Owner: Michael Brown Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall
[Impala-ASF-CR] IMPALA-7761: Add multiple DISTINCT to targeted stress and perf
Thomas Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/11805 ) Change subject: IMPALA-7761: Add multiple DISTINCT to targeted stress and perf .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/11805/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11805/1//COMMIT_MSG@16 PS1, Line 16: Testing: : - Ran the test file locally. > > One problem with putting these queries in the tpch workload and > changing the regex is that there are already a number of query > files in the tpch workload other than the regular tpch queries > which presumably weren't written with the idea of being part of the > stress test and aren't necessarily interesting for it. > > How is this a problem? The regex could opt in to the queries you > add without including the ones we're not interested in. I don't > personally see anything wrong with the stress- idea. This is what I ended up going with -- To view, visit http://gerrit.cloudera.org:8080/11805 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I400aaf6b6620b4001895eafff785956bffb312c9 Gerrit-Change-Number: 11805 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 06 Nov 2018 20:26:49 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7795: Implement REFRESH AUTHORIZATION statement
Fredy Wijaya has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/11888 ) Change subject: IMPALA-7795: Implement REFRESH AUTHORIZATION statement .. IMPALA-7795: Implement REFRESH AUTHORIZATION statement This patch implements REFRESH AUTHORIZATION statement to explicitly refresh authorization metadata. This statement is useful to force Impala to refresh its authorization metadata when there is an external update to authorization metadata without having to wait for the Sentry polling or call INVALIDATE METADATA. Some tests were updated to use REFRESH AUTHORIZATION instead of INVALIDATE METADATA to make the tests run faster. Syntax: REFRESH AUTHORIZATION (authorization must be enabled to execute this statement) Testing: - Added new FE tests - Added new E2E authorization tests - Ran all FE tests - Ran all E2E authorization tests Change-Id: I5459e1c97b12dee307e0cf85b94a9f66fd9d9a8c --- M common/thrift/CatalogService.thrift M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/ResetMetadataStmt.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/jflex/sql-scanner.flex M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java M fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M fe/src/test/java/org/apache/impala/analysis/StmtMetadataLoaderTest.java M fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java M tests/authorization/test_authorization.py M tests/authorization/test_grant_revoke.py M tests/authorization/test_owner_privileges.py M tests/common/sentry_cache_test_suite.py 15 files changed, 260 insertions(+), 86 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/88/11888/4 -- To view, visit http://gerrit.cloudera.org:8080/11888 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5459e1c97b12dee307e0cf85b94a9f66fd9d9a8c Gerrit-Change-Number: 11888 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7795: Implement REFRESH AUTHORIZATION statement
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/11888 ) Change subject: IMPALA-7795: Implement REFRESH AUTHORIZATION statement .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/11888/3/tests/common/sentry_cache_test_suite.py File tests/common/sentry_cache_test_suite.py: http://gerrit.cloudera.org:8080/#/c/11888/3/tests/common/sentry_cache_test_suite.py@66 PS3, Line 66: > flake8: E501 line too long (92 > 90 characters) Done -- To view, visit http://gerrit.cloudera.org:8080/11888 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5459e1c97b12dee307e0cf85b94a9f66fd9d9a8c Gerrit-Change-Number: 11888 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Tue, 06 Nov 2018 20:33:54 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7761: Add multiple DISTINCT to targeted perf and stress test
Hello Michael Brown, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11805 to look at the new patch set (#3). Change subject: IMPALA-7761: Add multiple DISTINCT to targeted perf and stress test .. IMPALA-7761: Add multiple DISTINCT to targeted perf and stress test IMPALA-110 added support for queries with multiple DISTINCT aggregates in a single select list. This patch adds queries to test this functionality to our targeted-perf workloads and fixes some incorrect return types in another targeted-perf aggregation query. It also adds some targeted queries to the stress test by extending the regex for stress test files to accept files of the form 'tpch-stress-*' and to allow for multiple tests per file. Testing: - Added an e2e test that runs the stress test file. Change-Id: I400aaf6b6620b4001895eafff785956bffb312c9 --- M testdata/workloads/targeted-perf/queries/aggregation.test M testdata/workloads/tpcds/queries/tpcds-q39-1.test M testdata/workloads/tpcds/queries/tpcds-q39-2.test A testdata/workloads/tpch/queries/tpch-stress-aggregations.test M tests/infra/test_stress_infra.py M tests/query_test/test_aggregation.py M tests/util/parse_util.py M tests/util/test_file_parser.py 8 files changed, 112 insertions(+), 13 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/11805/3 -- To view, visit http://gerrit.cloudera.org:8080/11805 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I400aaf6b6620b4001895eafff785956bffb312c9 Gerrit-Change-Number: 11805 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7795: Implement REFRESH AUTHORIZATION statement
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11888 ) Change subject: IMPALA-7795: Implement REFRESH AUTHORIZATION statement .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1293/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11888 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5459e1c97b12dee307e0cf85b94a9f66fd9d9a8c Gerrit-Change-Number: 11888 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Tue, 06 Nov 2018 21:12:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7795: Implement REFRESH AUTHORIZATION statement
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11888 ) Change subject: IMPALA-7795: Implement REFRESH AUTHORIZATION statement .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1294/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11888 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5459e1c97b12dee307e0cf85b94a9f66fd9d9a8c Gerrit-Change-Number: 11888 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Tue, 06 Nov 2018 21:22:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7822: handle overflows in repeat() builtin
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11889 ) Change subject: IMPALA-7822: handle overflows in repeat() builtin .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3432/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/11889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I872ce77bc2cb29116881c27ca2a5216f722cdb2a Gerrit-Change-Number: 11889 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 06 Nov 2018 21:58:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7822: handle overflows in repeat() builtin
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11889 Change subject: IMPALA-7822: handle overflows in repeat() builtin .. IMPALA-7822: handle overflows in repeat() builtin We need to carefully check that the intermediate value fits in an int64_t and the final size fits in an int. If they don't we raise an error and fail the query. Testing: Added a couple of backend tests to exercise the overflow check code paths. Change-Id: I872ce77bc2cb29116881c27ca2a5216f722cdb2a --- M be/src/exprs/expr-test.cc M be/src/exprs/string-functions-ir.cc 2 files changed, 21 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/11889/1 -- To view, visit http://gerrit.cloudera.org:8080/11889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I872ce77bc2cb29116881c27ca2a5216f722cdb2a Gerrit-Change-Number: 11889 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] IMPALA-7821: Refactor ExprRewriteRulesTest for easier debugging
Paul Rogers has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11890 Change subject: IMPALA-7821: Refactor ExprRewriteRulesTest for easier debugging .. IMPALA-7821: Refactor ExprRewriteRulesTest for easier debugging Impala provides a single test class ExprRewriteRulesTest for all rewrite rules. Each rule class has an associated test function. Th SimplifyConditionalRule class is complex, and the result test function is quite large. When doing work to modify a particular conditional rewrite, it became clear that debugging would be much easier if each detailed rewrite had its own test rather than using one huge test function. This ticket asks to break up the big function. (JUnit does not care about small vs. large functions.) This patch splits the function separate from later work that will modify the tests themselves. Also unified test function naming, using the standard "javaStyle" rather than the unusual "CppStyle" that was used. Added a few new tests, including some that do not pass (commented out) to motivate the need for the refactoring. Testing: this is a test, ran the test to ensure it still passes. Change-Id: I99a28b170bc0132424041e51f61ebe5c848c4083 --- M fe/src/test/java/org/apache/impala/analysis/ExprRewriteRulesTest.java 1 file changed, 112 insertions(+), 24 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/11890/1 -- To view, visit http://gerrit.cloudera.org:8080/11890 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I99a28b170bc0132424041e51f61ebe5c848c4083 Gerrit-Change-Number: 11890 Gerrit-PatchSet: 1 Gerrit-Owner: Paul Rogers
[Impala-ASF-CR] IMPALA-7808: Refactor Analyzer for easier debugging
Hello Csaba Ringhofer, Impala Public Jenkins, Vuk Ercegovac, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11883 to look at the new patch set (#4). Change subject: IMPALA-7808: Refactor Analyzer for easier debugging .. IMPALA-7808: Refactor Analyzer for easier debugging Changes two blocks of code to make debugging easier. No functional changes occur; changes are pure refactoring. A trivial change in AnalyzerContext removes a nested conditional clause. A larger change in SelectStmt takes the large analysis function and breaks it into a series of smaller functions. The functions were large because they shared state: variables created near the top are used much later near the bottom. To solve this, moved the code into an "algorithm" class whose only job is to hold onto the temporary state so that the big function can be broken into smaller pieces, with the temporary class fields used in place of the former local variables. For the most part, the existign code was simply split into functions and indented. One block of code had to be moved below the inner class since it is not part of the analysis process. Testing: No functional change, changes are purely structure. Reran all tests, which passed. Change-Id: I576c80c4c7a974df226fc91d8903db275069ed52 --- M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java 2 files changed, 767 insertions(+), 688 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/11883/4 -- To view, visit http://gerrit.cloudera.org:8080/11883 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I576c80c4c7a974df226fc91d8903db275069ed52 Gerrit-Change-Number: 11883 Gerrit-PatchSet: 4 Gerrit-Owner: Paul Rogers Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] Bump CDH BUILD VERSION to include Hive fix for RELY constraint
Bharath Vissapragada has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11891 Change subject: Bump CDH_BUILD_VERSION to include Hive fix for RELY constraint .. Bump CDH_BUILD_VERSION to include Hive fix for RELY constraint This change bumps the CDH_BUILD_VERSION to a version that includes the fix for a bug that does not correctly propagate "RELY" constraints from SQL statements in Hive. This is needed for the upcoming patch for IMPALA-3531. This new CDH_BUILD_VERSION pulls in newer version of Kudu binaries (v1.9). Kudu version variables in the impala-config are updated accordingly. Also, includes a newer toolchain build that has these v1.9 bits. Testing: DEBUG/Core tests passed. Change-Id: Ie626882044fea98b81b07caa950b480e16df122d --- M bin/impala-config.sh 1 file changed, 5 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/91/11891/1 -- To view, visit http://gerrit.cloudera.org:8080/11891 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ie626882044fea98b81b07caa950b480e16df122d Gerrit-Change-Number: 11891 Gerrit-PatchSet: 1 Gerrit-Owner: Bharath Vissapragada
[Impala-ASF-CR] IMPALA-7808: Refactor Analyzer for easier debugging
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11883 ) Change subject: IMPALA-7808: Refactor Analyzer for easier debugging .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1301/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11883 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I576c80c4c7a974df226fc91d8903db275069ed52 Gerrit-Change-Number: 11883 Gerrit-PatchSet: 4 Gerrit-Owner: Paul Rogers Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Tue, 06 Nov 2018 23:01:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7477: Batch-oriented query set construction
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/11879 ) Change subject: IMPALA-7477: Batch-oriented query set construction .. Patch Set 1: Code-Review+2 Thanks for the comment that points to the fix. What we found useful in such cases is to have PS1 contain the original commit that was reverted and PS2 contain the fix. -- To view, visit http://gerrit.cloudera.org:8080/11879 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I764fa302842438902cd5db2551ec6e3cb77b6874 Gerrit-Change-Number: 11879 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 06 Nov 2018 10:25:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5050: Add support to read TIMESTAMP MILLIS and TIMESTAMP MICROS from Parquet
Hello Zoltan Borok-Nagy, Attila Jeges, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11057 to look at the new patch set (#21). Change subject: IMPALA-5050: Add support to read TIMESTAMP_MILLIS and TIMESTAMP_MICROS from Parquet .. IMPALA-5050: Add support to read TIMESTAMP_MILLIS and TIMESTAMP_MICROS from Parquet Changes: - parquet.thrift is updated to a newer version which contains the timestamp logical type. - INT64 columns with converted types TIMESTAMP_MILLIS and TIMESTAMP_MICROS can be read as TIMESTAMP. - If the logical type is timestamp, then the type will contain the information whether the UTC->local conversion is necessary. This feature is only supported for the new timestamp types, so INT96 timestamps must still use flag convert_legacy_hive_parquet_utc_timestamps. - Min/max stat filtering is enabled again for columns that need UTC->local conversion. This was disabled in IMPALA-7559 because it could incorrectly drop column chunks. - CREATE TABLE LIKE PARQUET converts these columns to TIMESTAMP - before the change, an error was returned instead. - Bulk of the Parquet column stat logic was moved to a new class called "ColumnStatsReader". Testing: - Added unit tests for timezone conversion (this needed a new public function in timezone_db.h and adding CET to tzdb_tiny). - Added parquet files (created with parquet-mr) with int64 timestamp columns. Change-Id: I4c7c01fffa31b3d2ca3480adf6ff851137dadac3 --- M be/src/exec/hdfs-parquet-scanner.cc M be/src/exec/hdfs-parquet-scanner.h M be/src/exec/parquet-column-readers.cc M be/src/exec/parquet-column-readers.h M be/src/exec/parquet-column-stats.cc M be/src/exec/parquet-column-stats.h M be/src/exec/parquet-column-stats.inline.h M be/src/exec/parquet-common.cc M be/src/exec/parquet-common.h M be/src/exec/parquet-metadata-utils.cc M be/src/exprs/timezone_db.h M be/src/runtime/timestamp-test.cc M be/src/runtime/timestamp-value.cc M be/src/runtime/timestamp-value.h M be/src/util/dict-encoding.h M common/thrift/parquet.thrift M fe/src/main/java/org/apache/impala/analysis/ParquetHelper.java M testdata/data/README A testdata/data/int64_timestamps_at_dst_changes.parquet A testdata/data/int64_timestamps_dict.parquet A testdata/data/int64_timestamps_plain.parquet A testdata/tzdb_tiny/CET A testdata/workloads/functional-query/queries/QueryTest/parquet-int64-timestamps.test M tests/query_test/test_scanners.py 24 files changed, 927 insertions(+), 209 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/11057/21 -- To view, visit http://gerrit.cloudera.org:8080/11057 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I4c7c01fffa31b3d2ca3480adf6ff851137dadac3 Gerrit-Change-Number: 11057 Gerrit-PatchSet: 21 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-5050: Add support to read TIMESTAMP MILLIS and TIMESTAMP MICROS from Parquet
Zoltan Borok-Nagy has posted comments on this change. (
http://gerrit.cloudera.org:8080/11057 )
Change subject: IMPALA-5050: Add support to read TIMESTAMP_MILLIS and
TIMESTAMP_MICROS from Parquet
..
Patch Set 20: Code-Review+1
(3 comments)
http://gerrit.cloudera.org:8080/#/c/11057/16/be/src/exec/hdfs-parquet-scanner.cc
File be/src/exec/hdfs-parquet-scanner.cc:
http://gerrit.cloudera.org:8080/#/c/11057/16/be/src/exec/hdfs-parquet-scanner.cc@518
PS16, Line 518: string& fn_name = eval->root().function_name();
: ColumnStatsReader::StatsField stats_field;
: if (fn_name == "lt" || fn_name == "le") {
: // We need to get min stats.
: stats_field = ColumnStatsReader::StatsField::MIN;
: } else if (fn_name == "gt" || fn_name == "ge") {
: // We need to get max stats.
: s
> I have changed ColumnStatsReader to be a non-static class to reduce the num
Thanks for refactoring this, I think it looks much nicer now!
http://gerrit.cloudera.org:8080/#/c/11057/20/be/src/exec/parquet-common.h
File be/src/exec/parquet-common.h:
http://gerrit.cloudera.org:8080/#/c/11057/20/be/src/exec/parquet-common.h@465
PS20, Line 465: the
nit: that
http://gerrit.cloudera.org:8080/#/c/11057/20/be/src/exec/parquet-common.h@469
PS20, Line 469: the
nit: that
--
To view, visit http://gerrit.cloudera.org:8080/11057
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4c7c01fffa31b3d2ca3480adf6ff851137dadac3
Gerrit-Change-Number: 11057
Gerrit-PatchSet: 20
Gerrit-Owner: Csaba Ringhofer
Gerrit-Reviewer: Attila Jeges
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Tue, 06 Nov 2018 11:57:46 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5050: Add support to read TIMESTAMP MILLIS and TIMESTAMP MICROS from Parquet
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11057 ) Change subject: IMPALA-5050: Add support to read TIMESTAMP_MILLIS and TIMESTAMP_MICROS from Parquet .. Patch Set 21: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1290/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11057 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4c7c01fffa31b3d2ca3480adf6ff851137dadac3 Gerrit-Change-Number: 11057 Gerrit-PatchSet: 21 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 06 Nov 2018 13:01:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5050: Add support to read TIMESTAMP MILLIS and TIMESTAMP MICROS from Parquet
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/11057 ) Change subject: IMPALA-5050: Add support to read TIMESTAMP_MILLIS and TIMESTAMP_MICROS from Parquet .. Patch Set 21: Code-Review+1 (2 comments) Carry +1 http://gerrit.cloudera.org:8080/#/c/11057/20/be/src/exec/parquet-common.h File be/src/exec/parquet-common.h: http://gerrit.cloudera.org:8080/#/c/11057/20/be/src/exec/parquet-common.h@465 PS20, Line 465: tha > nit: that Done http://gerrit.cloudera.org:8080/#/c/11057/20/be/src/exec/parquet-common.h@469 PS20, Line 469: tha > nit: that Done -- To view, visit http://gerrit.cloudera.org:8080/11057 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4c7c01fffa31b3d2ca3480adf6ff851137dadac3 Gerrit-Change-Number: 11057 Gerrit-PatchSet: 21 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 06 Nov 2018 13:12:11 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5050: Add support to read TIMESTAMP MILLIS and TIMESTAMP MICROS from Parquet
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/11057 ) Change subject: IMPALA-5050: Add support to read TIMESTAMP_MILLIS and TIMESTAMP_MICROS from Parquet .. Patch Set 22: Code-Review+1 Patch set 22 is only rebase + trivial conflict resolution. Carry +1 -- To view, visit http://gerrit.cloudera.org:8080/11057 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4c7c01fffa31b3d2ca3480adf6ff851137dadac3 Gerrit-Change-Number: 11057 Gerrit-PatchSet: 22 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 06 Nov 2018 15:42:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5050: Add support to read TIMESTAMP MILLIS and TIMESTAMP MICROS from Parquet
Hello Zoltan Borok-Nagy, Attila Jeges, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11057 to look at the new patch set (#22). Change subject: IMPALA-5050: Add support to read TIMESTAMP_MILLIS and TIMESTAMP_MICROS from Parquet .. IMPALA-5050: Add support to read TIMESTAMP_MILLIS and TIMESTAMP_MICROS from Parquet Changes: - parquet.thrift is updated to a newer version which contains the timestamp logical type. - INT64 columns with converted types TIMESTAMP_MILLIS and TIMESTAMP_MICROS can be read as TIMESTAMP. - If the logical type is timestamp, then the type will contain the information whether the UTC->local conversion is necessary. This feature is only supported for the new timestamp types, so INT96 timestamps must still use flag convert_legacy_hive_parquet_utc_timestamps. - Min/max stat filtering is enabled again for columns that need UTC->local conversion. This was disabled in IMPALA-7559 because it could incorrectly drop column chunks. - CREATE TABLE LIKE PARQUET converts these columns to TIMESTAMP - before the change, an error was returned instead. - Bulk of the Parquet column stat logic was moved to a new class called "ColumnStatsReader". Testing: - Added unit tests for timezone conversion (this needed a new public function in timezone_db.h and adding CET to tzdb_tiny). - Added parquet files (created with parquet-mr) with int64 timestamp columns. Change-Id: I4c7c01fffa31b3d2ca3480adf6ff851137dadac3 --- M be/src/exec/hdfs-parquet-scanner.cc M be/src/exec/hdfs-parquet-scanner.h M be/src/exec/parquet-column-readers.cc M be/src/exec/parquet-column-readers.h M be/src/exec/parquet-column-stats.cc M be/src/exec/parquet-column-stats.h M be/src/exec/parquet-column-stats.inline.h M be/src/exec/parquet-common.cc M be/src/exec/parquet-common.h M be/src/exec/parquet-metadata-utils.cc M be/src/exprs/timezone_db.h M be/src/runtime/timestamp-test.cc M be/src/runtime/timestamp-value.cc M be/src/runtime/timestamp-value.h M be/src/util/dict-encoding.h M common/thrift/parquet.thrift M fe/src/main/java/org/apache/impala/analysis/ParquetHelper.java M testdata/data/README A testdata/data/int64_timestamps_at_dst_changes.parquet A testdata/data/int64_timestamps_dict.parquet A testdata/data/int64_timestamps_plain.parquet A testdata/tzdb_tiny/CET A testdata/workloads/functional-query/queries/QueryTest/parquet-int64-timestamps.test M tests/query_test/test_scanners.py 24 files changed, 927 insertions(+), 209 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/11057/22 -- To view, visit http://gerrit.cloudera.org:8080/11057 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I4c7c01fffa31b3d2ca3480adf6ff851137dadac3 Gerrit-Change-Number: 11057 Gerrit-PatchSet: 22 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-5004: Switch to sorting node for large TopN queries
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11698 ) Change subject: IMPALA-5004: Switch to sorting node for large TopN queries .. Patch Set 10: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3429/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/11698 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I34c9db33c9302b55e9978f53f9c7061f2806c8a9 Gerrit-Change-Number: 11698 Gerrit-PatchSet: 10 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Tue, 06 Nov 2018 16:04:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7809: support Kudu 1.9 in test concurrent schema change
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11882 ) Change subject: IMPALA-7809: support Kudu 1.9 in test_concurrent_schema_change .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11882 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7fc24bb6a18aecc0cb726b8d66f0aeccf56bbb9b Gerrit-Change-Number: 11882 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Brown Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 06 Nov 2018 16:08:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7809: support Kudu 1.9 in test concurrent schema change
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11882 ) Change subject: IMPALA-7809: support Kudu 1.9 in test_concurrent_schema_change .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3430/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/11882 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7fc24bb6a18aecc0cb726b8d66f0aeccf56bbb9b Gerrit-Change-Number: 11882 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Brown Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 06 Nov 2018 16:08:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5050: Add support to read TIMESTAMP MILLIS and TIMESTAMP MICROS from Parquet
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11057 ) Change subject: IMPALA-5050: Add support to read TIMESTAMP_MILLIS and TIMESTAMP_MICROS from Parquet .. Patch Set 22: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1291/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11057 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4c7c01fffa31b3d2ca3480adf6ff851137dadac3 Gerrit-Change-Number: 11057 Gerrit-PatchSet: 22 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 06 Nov 2018 16:18:04 + Gerrit-HasComments: No
