[Impala-ASF-CR] IMPALA-9943,IMPALA-4974: INTERSECT/EXCEPT [DISTINCT]
Tim Armstrong has posted comments on this change. (
http://gerrit.cloudera.org:8080/16123 )
Change subject: IMPALA-9943,IMPALA-4974: INTERSECT/EXCEPT [DISTINCT]
..
Patch Set 8:
(15 comments)
It's obviously a big change but I'm getting comfortable with it. I need to look
at the e2e tests again. I had some pretty minor comments at this point.
http://gerrit.cloudera.org:8080/#/c/16123/8/fe/src/main/cup/sql-parser.cup
File fe/src/main/cup/sql-parser.cup:
http://gerrit.cloudera.org:8080/#/c/16123/8/fe/src/main/cup/sql-parser.cup@2544
PS8, Line 2544: // We rely on the left precedence of KW_ORDER, KW_BY, and
KW_LIMIT,
Not your change, but maybe drop a reference to IMPALA-4741 in here.
http://gerrit.cloudera.org:8080/#/c/16123/8/fe/src/main/java/org/apache/impala/analysis/SetOperationStmt.java
File fe/src/main/java/org/apache/impala/analysis/SetOperationStmt.java:
http://gerrit.cloudera.org:8080/#/c/16123/8/fe/src/main/java/org/apache/impala/analysis/SetOperationStmt.java@47
PS8, Line 47: union
set operation?
http://gerrit.cloudera.org:8080/#/c/16123/8/fe/src/main/java/org/apache/impala/analysis/SetOperationStmt.java@166
PS8, Line 166: DISTINCT
should this be union distinct ops?
http://gerrit.cloudera.org:8080/#/c/16123/8/fe/src/main/java/org/apache/impala/analysis/SetOperationStmt.java@449
PS8, Line 449: public void rewriteExprs(ExprRewriter rewriter) throws
AnalysisException {
This is pre-existing, but can you mention that distinctAggInfo_ doesn't need to
be rewritten because the exprs are always simple SlotRefs? I had to read
through the code to convince myself that we weren't missing something here.
http://gerrit.cloudera.org:8080/#/c/16123/8/fe/src/main/java/org/apache/impala/analysis/SetOperationStmt.java@511
PS8, Line 511: parenthesis
nit: parentheses
http://gerrit.cloudera.org:8080/#/c/16123/8/fe/src/main/java/org/apache/impala/analysis/SetOperationStmt.java@816
PS8, Line 816: kmaps
maps?
http://gerrit.cloudera.org:8080/#/c/16123/8/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java
File fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java:
http://gerrit.cloudera.org:8080/#/c/16123/8/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java@201
PS8, Line 201: SelectStmt uSelect = null, eiSelect = null;
Can you briefly comment on uSelect/eiSelect, since they're pretty central? The
method comment explains it at a high level pretty well. Something like
After each iteration of the loop below, exactly one of uSelect and eiSelect
is non-null.
If the last operand processed was a union, uSelect is the current select
statement that has unionStmt nested inside, which in turn contains preceding
union operands. If the last operator processed was an except or intersect,
eiSelect is the current select statement containing preceding except or
intersect operands in the from clause.
http://gerrit.cloudera.org:8080/#/c/16123/8/fe/src/test/java/org/apache/impala/planner/PlannerTest.java
File fe/src/test/java/org/apache/impala/planner/PlannerTest.java:
http://gerrit.cloudera.org:8080/#/c/16123/8/fe/src/test/java/org/apache/impala/planner/PlannerTest.java@60
PS8, Line 60: /*
I understand the frustration :)
We should uncomment this though...
http://gerrit.cloudera.org:8080/#/c/16123/8/testdata/workloads/functional-planner/queries/PlannerTest/setoperation-rewrite.test
File
testdata/workloads/functional-planner/queries/PlannerTest/setoperation-rewrite.test:
http://gerrit.cloudera.org:8080/#/c/16123/8/testdata/workloads/functional-planner/queries/PlannerTest/setoperation-rewrite.test@361
PS8, Line 361: # except nested, doesn't get unnest compare to plan above
I didn't quite see what this comment was getting at.
http://gerrit.cloudera.org:8080/#/c/16123/8/testdata/workloads/functional-query/queries/QueryTest/except.test
File testdata/workloads/functional-query/queries/QueryTest/except.test:
PS8:
Can we add a token query or two that use the MINUS and EXCEPT DISTINCT
alternatives just to prove that they work end-to-end?
http://gerrit.cloudera.org:8080/#/c/16123/8/testdata/workloads/functional-query/queries/QueryTest/except.test@153
PS8, Line 153: (select 10 except select 11) union all select 10
This is a repeat of the one just above.
http://gerrit.cloudera.org:8080/#/c/16123/8/testdata/workloads/functional-query/queries/QueryTest/except.test@166
PS8, Line 166: select 10 union all select 11 except select 10
Would be good to have something like
select 10 union all select 11 union all select 11 except select 10
to show that it will deduplicate the union,
http://gerrit.cloudera.org:8080/#/c/16123/8/testdata/workloads/functional-query/queries/QueryTest/except.test@356
PS8, Line 356: abosrt
absorb?
http://gerrit.cloudera.org:8080/#/c/16123/8/testdata/workloads/functional-query/queries/QueryTest/intersect.test
File
[Impala-ASF-CR] IMPALA-9980: Remove jersey* jars from maven exclusions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16225 ) Change subject: IMPALA-9980: Remove jersey* jars from maven exclusions .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6676/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16225 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I796a9ff4c1beff776147266c2f2649c2d02a8dd8 Gerrit-Change-Number: 16225 Gerrit-PatchSet: 2 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 21 Jul 2020 05:10:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9980: Remove jersey* jars from maven exclusions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16225 ) Change subject: IMPALA-9980: Remove jersey* jars from maven exclusions .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6151/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/16225 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I796a9ff4c1beff776147266c2f2649c2d02a8dd8 Gerrit-Change-Number: 16225 Gerrit-PatchSet: 2 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 21 Jul 2020 05:09:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9980: Remove jersey* jars from maven exclusions
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/16225 ) Change subject: IMPALA-9980: Remove jersey* jars from maven exclusions .. Patch Set 2: Makes sense. Thanks. -- To view, visit http://gerrit.cloudera.org:8080/16225 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I796a9ff4c1beff776147266c2f2649c2d02a8dd8 Gerrit-Change-Number: 16225 Gerrit-PatchSet: 2 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 21 Jul 2020 05:09:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9980: Remove jersey* jars from maven exclusions
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16225 ) Change subject: IMPALA-9980: Remove jersey* jars from maven exclusions .. Patch Set 2: I think it would be better to just include jersey-bundle if that's what Ranger is depending on - that way at least the version will line up with what ranger is expecting. -- To view, visit http://gerrit.cloudera.org:8080/16225 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I796a9ff4c1beff776147266c2f2649c2d02a8dd8 Gerrit-Change-Number: 16225 Gerrit-PatchSet: 2 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 21 Jul 2020 04:54:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9980: Remove jersey* jars from maven exclusions
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/16225 ) Change subject: IMPALA-9980: Remove jersey* jars from maven exclusions .. Patch Set 2: Could we just add jersey-server.jar and jersey-servlet.jar since these were the missing jars instead of the jersey-bundle which seems to be a large jar? -- To view, visit http://gerrit.cloudera.org:8080/16225 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I796a9ff4c1beff776147266c2f2649c2d02a8dd8 Gerrit-Change-Number: 16225 Gerrit-PatchSet: 2 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 21 Jul 2020 04:53:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9980: Remove jersey* jars from maven exclusions
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/16225 ) Change subject: IMPALA-9980: Remove jersey* jars from maven exclusions .. Patch Set 2: (1 comment) Instead of removing all jersey* exclusions, just removing the jersey-bundle dependency from ranger worked in my local testing. I will look into why our dockerized tests did not catch this. http://gerrit.cloudera.org:8080/#/c/16225/1/fe/pom.xml File fe/pom.xml: http://gerrit.cloudera.org:8080/#/c/16225/1/fe/pom.xml@189 PS1, Line 189: rocksdbjni > This doesn't make a lot of sense to me - if the problem is with Ranger, why The original idea was to revert parts of IMPALA-9679 but I missed this exclusion in Ranger dependency. Instead of reverting all jeresey* changes, just removing this exclusion also worked in my local testing. Do you think it is a good idea? -- To view, visit http://gerrit.cloudera.org:8080/16225 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I796a9ff4c1beff776147266c2f2649c2d02a8dd8 Gerrit-Change-Number: 16225 Gerrit-PatchSet: 2 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 21 Jul 2020 04:49:16 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9980: Remove jersey* jars from maven exclusions
Anurag Mantripragada has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/16225 ) Change subject: IMPALA-9980: Remove jersey* jars from maven exclusions .. IMPALA-9980: Remove jersey* jars from maven exclusions IMPALA-9679 added jersey* jars to maven exclusions. These jars are required by Impala Ranger plugin to instantiate RuntimeDelegateImpl As a result of the exclusions, ClassNotFound exceptions are thrown in Impala docker containers when ranger plugin is enabled. This change removes jersey-bundle exclusion from Ranger dependency. Testing: - Built and ran Impala containers locally with ranger enabled. - Ran dockerized tests in precommit. Change-Id: I796a9ff4c1beff776147266c2f2649c2d02a8dd8 --- M fe/pom.xml 1 file changed, 0 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/16225/2 -- To view, visit http://gerrit.cloudera.org:8080/16225 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I796a9ff4c1beff776147266c2f2649c2d02a8dd8 Gerrit-Change-Number: 16225 Gerrit-PatchSet: 2 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] WIP CDPD-8989 Improve admission control pool stats logging to be more explicit
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16220 ) Change subject: WIP CDPD-8989 Improve admission control pool stats logging to be more explicit .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6675/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16220 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id995a9d044082c3b8f044e1ec25bb4c64347f781 Gerrit-Change-Number: 16220 Gerrit-PatchSet: 3 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 21 Jul 2020 03:41:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9925 cast(pow(2, 31) as int) return 2147483647 on aarch64
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16174 ) Change subject: IMPALA-9925 cast(pow(2, 31) as int) return 2147483647 on aarch64 .. Patch Set 4: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/6672/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/16174 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I58ab52acebb9bcddbf298efa886fd30ce35f68bf Gerrit-Change-Number: 16174 Gerrit-PatchSet: 4 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 21 Jul 2020 02:23:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9980: Remove jersey* jars from maven exclusions
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16225 ) Change subject: IMPALA-9980: Remove jersey* jars from maven exclusions .. Patch Set 1: (1 comment) What was the test gap? Why didn't our ranger tests catch this? How are we meant to prevent this regressing in future? http://gerrit.cloudera.org:8080/#/c/16225/1/fe/pom.xml File fe/pom.xml: http://gerrit.cloudera.org:8080/#/c/16225/1/fe/pom.xml@189 PS1, Line 189: jersey-bundle This doesn't make a lot of sense to me - if the problem is with Ranger, why are we removing exclusions from non-Ranger dependencies and leaving the exclusion for ranger? -- To view, visit http://gerrit.cloudera.org:8080/16225 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I796a9ff4c1beff776147266c2f2649c2d02a8dd8 Gerrit-Change-Number: 16225 Gerrit-PatchSet: 1 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 21 Jul 2020 03:20:36 + Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP CDPD-8989 Improve admission control pool stats logging to be more explicit
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16220 ) Change subject: WIP CDPD-8989 Improve admission control pool stats logging to be more explicit .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/16220/3/be/src/scheduling/admission-controller.cc File be/src/scheduling/admission-controller.cc: http://gerrit.cloudera.org:8080/#/c/16220/3/be/src/scheduling/admission-controller.cc@1331 PS3, Line 1331: // the total memory consumption, and the number of all queries running on this line has trailing whitespace -- To view, visit http://gerrit.cloudera.org:8080/16220 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id995a9d044082c3b8f044e1ec25bb4c64347f781 Gerrit-Change-Number: 16220 Gerrit-PatchSet: 3 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 21 Jul 2020 03:20:32 + Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP CDPD-8989 Improve admission control pool stats logging to be more explicit
Qifan Chen has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/16220 ) Change subject: WIP CDPD-8989 Improve admission control pool stats logging to be more explicit .. WIP CDPD-8989 Improve admission control pool stats logging to be more explicit This work addresses the current limitation in admission controller by appending the last known memory consumption statistics about a host to the existing host memory exhaustion message. The message is logged in impalad.INFO when a query is queued or timed out due to memory pressure at the host. This new memory consumption statistics covers the following content: num_running: the total number of queries running top_queries: a list of query Ids for up to 5 queries with top memory consumptions min: the minimal memory consumption of all running queries max: the maximal memory consumption of all running queries total: the total memory consumption of all running queries average: the average memory consumption of all running queries One example of the statistics is as follows. Memory consumed: num_running=2, top_queries=[ dc4fd356433812be:6902546f, c54d5ab3f4773ee8:8fb6a628], min=12.30 MB, max=12.37 MB, total=24.67 MB, average=12.34 MB Testing: TBD Change-Id: Id995a9d044082c3b8f044e1ec25bb4c64347f781 --- M be/src/runtime/mem-tracker.cc M be/src/runtime/mem-tracker.h M be/src/scheduling/admission-controller.cc M common/thrift/StatestoreService.thrift 4 files changed, 190 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/20/16220/3 -- To view, visit http://gerrit.cloudera.org:8080/16220 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id995a9d044082c3b8f044e1ec25bb4c64347f781 Gerrit-Change-Number: 16220 Gerrit-PatchSet: 3 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9980: Remove jersey* jars from maven exclusions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16225 ) Change subject: IMPALA-9980: Remove jersey* jars from maven exclusions .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6674/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16225 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I796a9ff4c1beff776147266c2f2649c2d02a8dd8 Gerrit-Change-Number: 16225 Gerrit-PatchSet: 1 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Tue, 21 Jul 2020 03:16:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6788: Abort ExecFInstance() RPC loop early after query failure
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16192 ) Change subject: IMPALA-6788: Abort ExecFInstance() RPC loop early after query failure .. IMPALA-6788: Abort ExecFInstance() RPC loop early after query failure Stops issuing ExecQueryFInstance rpcs and cancels any inflight when backend reports failure. Adds new debug action CONSTRUCT_QUERY_STATE_REPORT that runs when constructing a query state report. Adds a new test case for handling errors reported from query state. Testing: - Ran following command for new test case and verified that the code working as expected: ./bin/impala-py.test tests/custom_cluster/test_rpc_exception.py\ ::TestRPCException::test_state_report_error \ --workload_exploration_strategy=functional-query:exhaustive - Passed exhaustive tests. Change-Id: I034788f7720fc97c25c54f006ff72dce6cb199c3 Reviewed-on: http://gerrit.cloudera.org:8080/16192 Reviewed-by: Thomas Tauber-Marshall Tested-by: Impala Public Jenkins --- M be/src/runtime/coordinator.cc M be/src/runtime/query-state.cc M tests/custom_cluster/test_rpc_exception.py 3 files changed, 40 insertions(+), 3 deletions(-) Approvals: Thomas Tauber-Marshall: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/16192 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I034788f7720fc97c25c54f006ff72dce6cb199c3 Gerrit-Change-Number: 16192 Gerrit-PatchSet: 5 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-6788: Abort ExecFInstance() RPC loop early after query failure
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16192 ) Change subject: IMPALA-6788: Abort ExecFInstance() RPC loop early after query failure .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16192 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I034788f7720fc97c25c54f006ff72dce6cb199c3 Gerrit-Change-Number: 16192 Gerrit-PatchSet: 4 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 21 Jul 2020 02:43:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9889: Fixed flaky test runtime filters on Kudu table
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16191 ) Change subject: IMPALA-9889: Fixed flaky test_runtime_filters on Kudu table .. IMPALA-9889: Fixed flaky test_runtime_filters on Kudu table Test cases in test_runtime_filters failed occasionally in ASAN builds due to runtime filters not arriving scan nodes in time. Query profiles showed that codegen took 2 to 4 minutes for one fragment when this issue happened. This caused hash join nodes waiting long time to generate and publish runtime filters, hence arrival delay on scan nodes. To avoid the delay, turn on ASYNC_CODEGEN for test_runtime_filters when test runs for slow build like ASAN, TSAN, UBSAN, etc. Testing: - Passed core test for regular debug build and ASAN build. Change-Id: I94a08e272f0870c04c96563fa614e3416fb5379b Reviewed-on: http://gerrit.cloudera.org:8080/16191 Reviewed-by: Thomas Tauber-Marshall Tested-by: Impala Public Jenkins --- M tests/query_test/test_runtime_filters.py 1 file changed, 23 insertions(+), 0 deletions(-) Approvals: Thomas Tauber-Marshall: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/16191 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I94a08e272f0870c04c96563fa614e3416fb5379b Gerrit-Change-Number: 16191 Gerrit-PatchSet: 6 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-9889: Fixed flaky test runtime filters on Kudu table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16191 ) Change subject: IMPALA-9889: Fixed flaky test_runtime_filters on Kudu table .. Patch Set 5: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16191 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I94a08e272f0870c04c96563fa614e3416fb5379b Gerrit-Change-Number: 16191 Gerrit-PatchSet: 5 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 21 Jul 2020 03:05:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9926 base64decode % will not return error when in newer OS
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16175 ) Change subject: IMPALA-9926 base64decode % will not return error when in newer OS .. Patch Set 4: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/6673/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/16175 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib9bd9e03d5f744c18c957cdaf2064fa918086004 Gerrit-Change-Number: 16175 Gerrit-PatchSet: 4 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 21 Jul 2020 02:34:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9980: Remove jersey* jars from maven exclusions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16225 ) Change subject: IMPALA-9980: Remove jersey* jars from maven exclusions .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6150/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/16225 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I796a9ff4c1beff776147266c2f2649c2d02a8dd8 Gerrit-Change-Number: 16225 Gerrit-PatchSet: 1 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Tue, 21 Jul 2020 02:49:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9980: Remove jersey* jars from maven exclusions
Anurag Mantripragada has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16225 Change subject: IMPALA-9980: Remove jersey* jars from maven exclusions .. IMPALA-9980: Remove jersey* jars from maven exclusions IMPALA-9679 added jersey* jars to maven exclusions. These jars are required by Impala Ranger plugin to instantiate RuntimeDelegateImpl As a result of the exclusions, ClassNotFound exceptions are thrown in Impala docker containers when ranger plugin is enabled. This change removes jersey-server.jar and jersey-servlet.jar from exclusions. Testing: Ran dockerized tests in precommit. Change-Id: I796a9ff4c1beff776147266c2f2649c2d02a8dd8 --- M fe/pom.xml 1 file changed, 0 insertions(+), 27 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/16225/1 -- To view, visit http://gerrit.cloudera.org:8080/16225 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I796a9ff4c1beff776147266c2f2649c2d02a8dd8 Gerrit-Change-Number: 16225 Gerrit-PatchSet: 1 Gerrit-Owner: Anurag Mantripragada
[Impala-ASF-CR] IMPALA-9906 Fix thread-pool-test failed case on aarch64
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16173 ) Change subject: IMPALA-9906 Fix thread-pool-test failed case on aarch64 .. Patch Set 4: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/6671/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/16173 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7b353f7eb9662995d9a8ae460bb1631933873d5d Gerrit-Change-Number: 16173 Gerrit-PatchSet: 4 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 21 Jul 2020 02:27:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9904 Fix bad cipher test failed case on aarch64
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16172 ) Change subject: IMPALA-9904 Fix bad cipher test failed case on aarch64 .. Patch Set 4: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/6670/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/16172 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I19751b6bf1045fd6d901c5a67f74e8bdd6bf65d3 Gerrit-Change-Number: 16172 Gerrit-PatchSet: 4 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 21 Jul 2020 02:10:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9544 Replace Intel's SSE instructions with ARM's NEON instructions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15531 ) Change subject: IMPALA-9544 Replace Intel's SSE instructions with ARM's NEON instructions .. Patch Set 38: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/6668/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/15531 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id7dfe17125b2910ece54e7dd18b4e4b25d7de8b9 Gerrit-Change-Number: 15531 Gerrit-PatchSet: 38 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 21 Jul 2020 02:10:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9645 Port LLVM codegen to adapt aarch64
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15718 ) Change subject: IMPALA-9645 Port LLVM codegen to adapt aarch64 .. Patch Set 17: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/6667/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/15718 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3f30ee84ea9bf5245da88154632bb69079103d11 Gerrit-Change-Number: 15718 Gerrit-PatchSet: 17 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 21 Jul 2020 02:10:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9676 Add aarch64 compile options for clang
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15755 ) Change subject: IMPALA-9676 Add aarch64 compile options for clang .. Patch Set 15: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/6669/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/15755 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I69a5ff64bbd4427dd87ec6e884251e76d6a73122 Gerrit-Change-Number: 15755 Gerrit-PatchSet: 15 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 21 Jul 2020 02:10:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9544 Replace Intel's SSE instructions with ARM's NEON instructions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15531 ) Change subject: IMPALA-9544 Replace Intel's SSE instructions with ARM's NEON instructions .. Patch Set 38: (8 comments) http://gerrit.cloudera.org:8080/#/c/15531/38/be/src/util/sse2neon.h File be/src/util/sse2neon.h: http://gerrit.cloudera.org:8080/#/c/15531/38/be/src/util/sse2neon.h@213 PS38, Line 213: // https://msdn.microsoft.com/en-us/library/bb514059%28v=vs.120%29.aspx?f=255=-2147217396 line too long (99 > 90) http://gerrit.cloudera.org:8080/#/c/15531/38/be/src/util/sse2neon.h@406 PS38, Line 406: // https://docs.microsoft.com/en-us/previous-versions/visualstudio/visual-studio-2010/whtfzhzk(v=vs.100) line too long (104 > 90) http://gerrit.cloudera.org:8080/#/c/15531/38/be/src/util/sse2neon.h@413 PS38, Line 413: // https://software.intel.com/sites/landingpage/IntrinsicsGuide/#text=_mm_set1_epi64x=4961 line too long (97 > 90) http://gerrit.cloudera.org:8080/#/c/15531/38/be/src/util/sse2neon.h@1054 PS38, Line 1054: // https://software.intel.com/sites/landingpage/IntrinsicsGuide/#text=_mm_shuffle_epi8=5146 line too long (98 > 90) http://gerrit.cloudera.org:8080/#/c/15531/38/be/src/util/sse2neon.h@1199 PS38, Line 1199: // https://docs.microsoft.com/en-us/previous-versions/visualstudio/visual-studio-2010/y41dkk37(v=vs.100) line too long (104 > 90) http://gerrit.cloudera.org:8080/#/c/15531/38/be/src/util/sse2neon.h@1645 PS38, Line 1645: // https://software.intel.com/sites/landingpage/IntrinsicsGuide/#text=_mm_test_all_zeros=5871 line too long (100 > 90) http://gerrit.cloudera.org:8080/#/c/15531/38/be/src/util/sse2neon.h@3581 PS38, Line 3581: // https://github.com/ColinIanKing/linux-next-mirror/blob/b5f466091e130caaf0735976648f72bd5e09aa84/crypto/aegis128-neon-inner.c#L52 line too long (131 > 90) http://gerrit.cloudera.org:8080/#/c/15531/38/be/src/util/sse2neon.h@3681 PS38, Line 3681: // cpp-compiler-developer-guide-and-reference-allocating-and-freeing-aligned-memory-blocks line too long (98 > 90) -- To view, visit http://gerrit.cloudera.org:8080/15531 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id7dfe17125b2910ece54e7dd18b4e4b25d7de8b9 Gerrit-Change-Number: 15531 Gerrit-PatchSet: 38 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 21 Jul 2020 01:44:14 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9906 Fix thread-pool-test failed case on aarch64
zhaoren...@hotmail.com has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/16173 ) Change subject: IMPALA-9906 Fix thread-pool-test failed case on aarch64 .. IMPALA-9906 Fix thread-pool-test failed case on aarch64 Threads switch frequency is not so fast as x86. So here change the sleep task time from 100ms to 500ms Change-Id: I7b353f7eb9662995d9a8ae460bb1631933873d5d --- M be/src/util/thread-pool-test.cc 1 file changed, 9 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/73/16173/4 -- To view, visit http://gerrit.cloudera.org:8080/16173 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7b353f7eb9662995d9a8ae460bb1631933873d5d Gerrit-Change-Number: 16173 Gerrit-PatchSet: 4 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-9925 cast(pow(2, 31) as int) return 2147483647 on aarch64
zhaoren...@hotmail.com has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/16174 ) Change subject: IMPALA-9925 cast(pow(2, 31) as int) return 2147483647 on aarch64 .. IMPALA-9925 cast(pow(2, 31) as int) return 2147483647 on aarch64 cast(pow(2, 31) as int) return 2147483647 on aarch64 but return 2147483648 on x86 I think aarch64 is correct. So here I will not convert it, just use aarch64's value Change-Id: I58ab52acebb9bcddbf298efa886fd30ce35f68bf --- M be/src/exprs/expr-test.cc 1 file changed, 5 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/74/16174/4 -- To view, visit http://gerrit.cloudera.org:8080/16174 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I58ab52acebb9bcddbf298efa886fd30ce35f68bf Gerrit-Change-Number: 16174 Gerrit-PatchSet: 4 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-9544 Replace Intel's SSE instructions with ARM's NEON instructions
zhaoren...@hotmail.com has uploaded a new patch set (#38). ( http://gerrit.cloudera.org:8080/15531 ) Change subject: IMPALA-9544 Replace Intel's SSE instructions with ARM's NEON instructions .. IMPALA-9544 Replace Intel's SSE instructions with ARM's NEON instructions Replace Intel's SSE instructions with ARM's NEON instructions Replace Intel's crc32 instructions with ARM's instructions Replace Intel's popcntq instruction with ARM's mechanism Replace Intel's pcmpestri and pcmpestrm instructions with ARM mechanism Change-Id: Id7dfe17125b2910ece54e7dd18b4e4b25d7de8b9 --- M CMakeLists.txt M be/CMakeLists.txt M be/src/benchmarks/bswap-benchmark.cc M be/src/benchmarks/int-hash-benchmark.cc M be/src/codegen/CMakeLists.txt M be/src/codegen/gen_ir_descriptions.py M be/src/codegen/llvm-codegen-test.cc M be/src/codegen/llvm-codegen.cc M be/src/exec/delimited-text-parser.inline.h M be/src/kudu/util/block_bloom_filter.cc M be/src/kudu/util/group_varint-inl.h M be/src/kudu/util/group_varint-test.cc A be/src/kudu/util/sse2neon.h M be/src/util/bit-util-test.cc M be/src/util/bit-util.cc M be/src/util/bit-util.h M be/src/util/bloom-filter.cc M be/src/util/bloom-filter.h M be/src/util/cpu-info.cc M be/src/util/hash-util-ir.cc M be/src/util/hash-util.h M be/src/util/sse-util.h A be/src/util/sse2neon.h 23 files changed, 4,024 insertions(+), 25 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/15531/38 -- To view, visit http://gerrit.cloudera.org:8080/15531 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id7dfe17125b2910ece54e7dd18b4e4b25d7de8b9 Gerrit-Change-Number: 15531 Gerrit-PatchSet: 38 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9926 base64decode % will not return error when in newer OS
zhaoren...@hotmail.com has uploaded a new patch set (#4). (
http://gerrit.cloudera.org:8080/16175 )
Change subject: IMPALA-9926 base64decode % will not return error when in
newer OS
..
IMPALA-9926 base64decode % will not return error when in newer OS
for example, base64decode('YWxwaGE%') will return
'alpha\377' in newer os which has newer sasl library.
I tested it on Ubuntu 18.04 aarch64 version.
Change-Id: Ib9bd9e03d5f744c18c957cdaf2064fa918086004
---
M be/src/exprs/expr-test.cc
1 file changed, 4 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/16175/4
--
To view, visit http://gerrit.cloudera.org:8080/16175
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib9bd9e03d5f744c18c957cdaf2064fa918086004
Gerrit-Change-Number: 16175
Gerrit-PatchSet: 4
Gerrit-Owner: Anonymous Coward
Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-9904 Fix bad cipher test failed case on aarch64
zhaoren...@hotmail.com has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/16172 ) Change subject: IMPALA-9904 Fix bad cipher test failed case on aarch64 .. IMPALA-9904 Fix bad cipher test failed case on aarch64 On aarch64 and os ubuntu 18.04, the openssl version is 1.1.1, the server which used openssl can start successfully even ciphers is bad. So here just don't test bad ciphers cases on aarch64. On x86, the server cannot start successfully because the lower openssl version, not because the bad cipher. Change-Id: I19751b6bf1045fd6d901c5a67f74e8bdd6bf65d3 --- M be/src/rpc/rpc-mgr-test.cc M be/src/rpc/thrift-server-test.cc M be/src/util/webserver-test.cc 3 files changed, 10 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/72/16172/4 -- To view, visit http://gerrit.cloudera.org:8080/16172 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I19751b6bf1045fd6d901c5a67f74e8bdd6bf65d3 Gerrit-Change-Number: 16172 Gerrit-PatchSet: 4 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-9645 Port LLVM codegen to adapt aarch64
zhaoren...@hotmail.com has uploaded a new patch set (#17). (
http://gerrit.cloudera.org:8080/15718 )
Change subject: IMPALA-9645 Port LLVM codegen to adapt aarch64
..
IMPALA-9645 Port LLVM codegen to adapt aarch64
On aarch64, the Lowered type of struct {bool, int128} is form
{ {i8}, {i128} }. No padding add. This is different with x86-64,
which is { {i8}, {15*i8}, {i128} } with padding add automatically.
And here also add some type conversion between x86 and aarch64 data types.
And also add some aarch64 cpu's feature.
Change-Id: I3f30ee84ea9bf5245da88154632bb69079103d11
---
M be/src/codegen/codegen-anyval.cc
M be/src/codegen/llvm-codegen.cc
M be/src/exec/text-converter.cc
M be/src/exprs/scalar-fn-call.cc
4 files changed, 175 insertions(+), 11 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/15718/17
--
To view, visit http://gerrit.cloudera.org:8080/15718
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I3f30ee84ea9bf5245da88154632bb69079103d11
Gerrit-Change-Number: 15718
Gerrit-PatchSet: 17
Gerrit-Owner: Anonymous Coward
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tamas Mate
Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9676 Add aarch64 compile options for clang
zhaoren...@hotmail.com has uploaded a new patch set (#15). ( http://gerrit.cloudera.org:8080/15755 ) Change subject: IMPALA-9676 Add aarch64 compile options for clang .. IMPALA-9676 Add aarch64 compile options for clang Add signed-char and armv8a and crc compile options to clang Change-Id: I69a5ff64bbd4427dd87ec6e884251e76d6a73122 --- M be/CMakeLists.txt 1 file changed, 7 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/55/15755/15 -- To view, visit http://gerrit.cloudera.org:8080/15755 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I69a5ff64bbd4427dd87ec6e884251e76d6a73122 Gerrit-Change-Number: 15755 Gerrit-PatchSet: 15 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9478: Profiles should indicate if custom UDFs are being used
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16188 ) Change subject: IMPALA-9478: Profiles should indicate if custom UDFs are being used .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16188 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I79122e6cc74fd5a62c76962289a1615fbac2f345 Gerrit-Change-Number: 16188 Gerrit-PatchSet: 3 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 21 Jul 2020 00:52:06 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP CDPD-8989 Improve admission control pool stats logging to be more explicit
Bikramjeet Vig has posted comments on this change. (
http://gerrit.cloudera.org:8080/16220 )
Change subject: WIP CDPD-8989 Improve admission control pool stats logging to
be more explicit
..
Patch Set 1:
(8 comments)
http://gerrit.cloudera.org:8080/#/c/16220/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/16220/1//COMMIT_MSG@7
PS1, Line 7: WIP CDPD-8989
Can you file an upstream JIRA and add that here instead
http://gerrit.cloudera.org:8080/#/c/16220/1/be/src/runtime/mem-tracker.h
File be/src/runtime/mem-tracker.h:
http://gerrit.cloudera.org:8080/#/c/16220/1/be/src/runtime/mem-tracker.h@362
PS1, Line 362: UpdatePoolStatsForQueries
I think only this needs to be public, rest can be private if they are only
helper functions
http://gerrit.cloudera.org:8080/#/c/16220/1/be/src/runtime/mem-tracker.cc
File be/src/runtime/mem-tracker.cc:
http://gerrit.cloudera.org:8080/#/c/16220/1/be/src/runtime/mem-tracker.cc@568
PS1, Line 568:
: // Recursively append info about this memory tracker and all its
children to ss.
: void MemTracker::GetAllMemTracker(std::stringstream& ss, int
indent) {
: ss << std::string(indent, ' ') << " MemTracker: label=" <<
label_;
:
: if (!pool_name_.empty()) {
: ss << ", pool_name =" << pool_name_;
: }
:
: if (is_query_mem_tracker_) {
: ss << ", qid=" << PrintId(query_id_);
: }
:
: ss << std::endl;
: indent += 3;
: for (MemTracker* child : child_trackers_) {
: child->GetAllMemTracker(ss, indent);
: }
: }
:
: // Return a debug string for all memory trackers reachable from
the root memory
: // tracker reachable from this.
: string MemTracker::GetAllMemTrackers() {
: lock_guard l(child_trackers_lock_);
: std::stringstream ss;
: GetRootMemTracker()->GetAllMemTracker(ss, 0);
: return ss.str();
: }
leftover code from debugging perhaps?
http://gerrit.cloudera.org:8080/#/c/16220/1/be/src/scheduling/admission-controller.cc
File be/src/scheduling/admission-controller.cc:
http://gerrit.cloudera.org:8080/#/c/16220/1/be/src/scheduling/admission-controller.cc@556
PS1, Line 556: DebugPoolStatsForConsumedMemory
Here the check is at the host level, but the log line is produced using pool
level stats which is misleading. You will either have to merge the results from
all pools like we do for updating HostStats (which might get unwieldy when
aggregating the top 5 queries among pools) or add another statestore update
that updates host level stats
http://gerrit.cloudera.org:8080/#/c/16220/1/be/src/scheduling/admission-controller.cc@1328
PS1, Line 1328: if (tracker) {
: // update local_stats_ with the query Ids of the top N
queries, plus the min, the max,
: // the total memory consumption, and the number of all
queries in this pool.
: tracker->UpdatePoolStatsForQueries(5 /*limit*/,
this->local_stats_);
:
: }
This is being called for the pool level memtracker, which will only give up the
top 5 queries in the pool.
http://gerrit.cloudera.org:8080/#/c/16220/1/common/thrift/StatestoreService.thrift
File common/thrift/StatestoreService.thrift:
http://gerrit.cloudera.org:8080/#/c/16220/1/common/thrift/StatestoreService.thrift@52
PS1, Line 52: 5: required i64 min_memory_consumed;
:
: // Max memory consumption among all queries.
: 6: required i64 max_memory_consumed;
Now that i think of this, I am not sure how helpful the min/max memory consumed
will be. I think it will be more helpful to add the current consumption of the
top K queries and print them instead.
http://gerrit.cloudera.org:8080/#/c/16220/1/common/thrift/StatestoreService.thrift@58
PS1, Line 58: 7: required i64 total_memory_consumed;
fyi: this can be fetched using the consumption of the parent tracker instead of
adding all the queries.
Also, seems like this is already tracked locally using
metrics_.local_backend_mem_usage at the pool level
http://gerrit.cloudera.org:8080/#/c/16220/1/common/thrift/StatestoreService.thrift@60
PS1, Line 60: The current number of requests admitted by any admission
controllers
: // that are currently running. This is an instantaneous value
(as opposed to a
: // cumulative sum).
I think a more appropriate description will be that this is the num of queries
that have live fragments taking up memory on the host.
--
To view, visit http://gerrit.cloudera.org:8080/16220
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
[Impala-ASF-CR] IMPALA-7655: Implement codegen for conditional functions (if, isnull, coalesce)
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16208 ) Change subject: IMPALA-7655: Implement codegen for conditional functions (if, isnull, coalesce) .. Patch Set 6: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16208 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I11f617a9148492ccafb46112ce0af103a10090f8 Gerrit-Change-Number: 16208 Gerrit-PatchSet: 6 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 20 Jul 2020 23:37:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7655: Implement codegen for conditional functions (if, isnull, coalesce)
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16208 ) Change subject: IMPALA-7655: Implement codegen for conditional functions (if, isnull, coalesce) .. IMPALA-7655: Implement codegen for conditional functions (if, isnull, coalesce) Implement proper codegen for conditional functions (if, isnull, coalesce) instead of simply calling into interpreted code. We use IRBuilder to generate hand-crafted code. Change-Id: I11f617a9148492ccafb46112ce0af103a10090f8 Reviewed-on: http://gerrit.cloudera.org:8080/16208 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/exprs/case-expr.cc M be/src/exprs/conditional-functions-ir.cc M be/src/exprs/conditional-functions.cc 3 files changed, 289 insertions(+), 85 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/16208 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I11f617a9148492ccafb46112ce0af103a10090f8 Gerrit-Change-Number: 16208 Gerrit-PatchSet: 7 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9953: Shell should continue fetching even when 0 rows are returned
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16222 ) Change subject: IMPALA-9953: Shell should continue fetching even when 0 rows are returned .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16222 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5f8527aea9e433f8cf426435c0ba41355bbf9d88 Gerrit-Change-Number: 16222 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 20 Jul 2020 23:34:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9953: Shell should continue fetching even when 0 rows are returned
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16222 ) Change subject: IMPALA-9953: Shell should continue fetching even when 0 rows are returned .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks// : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16222 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5f8527aea9e433f8cf426435c0ba41355bbf9d88 Gerrit-Change-Number: 16222 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 20 Jul 2020 23:26:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9953: Shell should continue fetching even when 0 rows are returned
Sahil Takiar has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16222 Change subject: IMPALA-9953: Shell should continue fetching even when 0 rows are returned .. IMPALA-9953: Shell should continue fetching even when 0 rows are returned The Impala shell stops fetching rows if it receives a batch that contains 0 rows. This is incorrect because a batch with 0 rows can be returned if the fetch request hits a timeout. Instead, the shell should rely on the value of has_rows / hasMoreRows to determine when to stop issuing fetch requests. Tests: * Added a regression test to test_shell_commandline.py * Ran all shell tests Change-Id: I5f8527aea9e433f8cf426435c0ba41355bbf9d88 --- M shell/impala_shell.py M tests/shell/test_shell_commandline.py 2 files changed, 17 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/22/16222/1 -- To view, visit http://gerrit.cloudera.org:8080/16222 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I5f8527aea9e433f8cf426435c0ba41355bbf9d88 Gerrit-Change-Number: 16222 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar
[Impala-ASF-CR] IMPALA-8547: get json object fails to get value for numeric key
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14905 ) Change subject: IMPALA-8547: get_json_object fails to get value for numeric key .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6665/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14905 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7df037ccf2c79da0ba86a46df1dd28ab0e9a45f4 Gerrit-Change-Number: 14905 Gerrit-PatchSet: 3 Gerrit-Owner: Eugene Zimichev Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 20 Jul 2020 22:22:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8547: get json object fails to get value for numeric key
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14905 ) Change subject: IMPALA-8547: get_json_object fails to get value for numeric key .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6664/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14905 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7df037ccf2c79da0ba86a46df1dd28ab0e9a45f4 Gerrit-Change-Number: 14905 Gerrit-PatchSet: 2 Gerrit-Owner: Eugene Zimichev Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 20 Jul 2020 22:13:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9889: Fixed flaky test runtime filters on Kudu table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16191 ) Change subject: IMPALA-9889: Fixed flaky test_runtime_filters on Kudu table .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6149/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16191 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I94a08e272f0870c04c96563fa614e3416fb5379b Gerrit-Change-Number: 16191 Gerrit-PatchSet: 5 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 20 Jul 2020 21:56:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8547: get json object fails to get value for numeric key
Sahil Takiar has uploaded a new patch set (#3) to the change originally created
by Eugene Zimichev. ( http://gerrit.cloudera.org:8080/14905 )
Change subject: IMPALA-8547: get_json_object fails to get value for numeric key
..
IMPALA-8547: get_json_object fails to get value for numeric key
Allows numeric keys for JSON objects in get_json_object. This patch
makes Impala consistent with Hive and Postgres behavior for
get_json_object.
Queries such as "select get_json_object('{"1": 5}', '$.1');"
would fail before this patch. Now the query will return '5'.
Testing:
* Added tests to expr-test
Change-Id: I7df037ccf2c79da0ba86a46df1dd28ab0e9a45f4
---
M be/src/exprs/expr-test.cc
M be/src/util/string-util.cc
M be/src/util/string-util.h
3 files changed, 6 insertions(+), 4 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/14905/3
--
To view, visit http://gerrit.cloudera.org:8080/14905
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7df037ccf2c79da0ba86a46df1dd28ab0e9a45f4
Gerrit-Change-Number: 14905
Gerrit-PatchSet: 3
Gerrit-Owner: Eugene Zimichev
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Sahil Takiar
Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8547: get json object fails to get value for numeric key
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/14905 ) Change subject: IMPALA-8547: get_json_object fails to get value for numeric key .. Patch Set 2: @Tim - cleaned up the patch a bit, should be good to review. -- To view, visit http://gerrit.cloudera.org:8080/14905 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7df037ccf2c79da0ba86a46df1dd28ab0e9a45f4 Gerrit-Change-Number: 14905 Gerrit-PatchSet: 2 Gerrit-Owner: Eugene Zimichev Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 20 Jul 2020 21:52:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8547: get json object fails to get value for numeric key
Sahil Takiar has uploaded a new patch set (#2) to the change originally created
by Eugene Zimichev. ( http://gerrit.cloudera.org:8080/14905 )
Change subject: IMPALA-8547: get_json_object fails to get value for numeric key
..
IMPALA-8547: get_json_object fails to get value for numeric key
Allows numeric keys for JSON objects in get_json_object. This patch
makes Impala consistent with Hive and Postgres behavior for
get_json_object.
Queries such as "select get_json_object('{"1": 5}', '$.1');"
would fail before this patch. Now the query will return '5'.
Testing:
* Added tests to expr-test
Change-Id: I7df037ccf2c79da0ba86a46df1dd28ab0e9a45f4
---
M be/src/exprs/expr-test.cc
M be/src/util/string-util.cc
2 files changed, 3 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/14905/2
--
To view, visit http://gerrit.cloudera.org:8080/14905
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7df037ccf2c79da0ba86a46df1dd28ab0e9a45f4
Gerrit-Change-Number: 14905
Gerrit-PatchSet: 2
Gerrit-Owner: Eugene Zimichev
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Sahil Takiar
Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9889: Fixed flaky test runtime filters on Kudu table
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/16191 ) Change subject: IMPALA-9889: Fixed flaky test_runtime_filters on Kudu table .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16191 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I94a08e272f0870c04c96563fa614e3416fb5379b Gerrit-Change-Number: 16191 Gerrit-PatchSet: 5 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 20 Jul 2020 21:41:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9478: Profiles should indicate if custom UDFs are being used
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16188 ) Change subject: IMPALA-9478: Profiles should indicate if custom UDFs are being used .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6663/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16188 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I79122e6cc74fd5a62c76962289a1615fbac2f345 Gerrit-Change-Number: 16188 Gerrit-PatchSet: 3 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 20 Jul 2020 21:31:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6788: Abort ExecFInstance() RPC loop early after query failure
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16192 ) Change subject: IMPALA-6788: Abort ExecFInstance() RPC loop early after query failure .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6148/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16192 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I034788f7720fc97c25c54f006ff72dce6cb199c3 Gerrit-Change-Number: 16192 Gerrit-PatchSet: 4 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 20 Jul 2020 21:29:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9478: Profiles should indicate if custom UDFs are being used
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16188 ) Change subject: IMPALA-9478: Profiles should indicate if custom UDFs are being used .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6662/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16188 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I79122e6cc74fd5a62c76962289a1615fbac2f345 Gerrit-Change-Number: 16188 Gerrit-PatchSet: 2 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 20 Jul 2020 21:25:37 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP CDPD-8989 Improve admission control pool stats logging to be more explicit
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16220 ) Change subject: WIP CDPD-8989 Improve admission control pool stats logging to be more explicit .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6661/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16220 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id995a9d044082c3b8f044e1ec25bb4c64347f781 Gerrit-Change-Number: 16220 Gerrit-PatchSet: 1 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 20 Jul 2020 21:25:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9478: Profiles should indicate if custom UDFs are being used
Hello Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16188 to look at the new patch set (#3). Change subject: IMPALA-9478: Profiles should indicate if custom UDFs are being used .. IMPALA-9478: Profiles should indicate if custom UDFs are being used Adds a marker to runtime profiles and explain plans indicating if custom (e.g. non-built in) user-defined functions are being used. For explain plans, a SQL-style comment is added after any function call. For runtime profiles, a new Frontend entry called "User Defined Functions (UDFs)" lists out all UDFs analyzed during planning. Take the following example: create function hive_lower(string) returns string location '/test-warehouse/hive-exec.jar' symbol='org.apache.hadoop.hive.ql.udf.UDFLower'; set explain_level=3; explain select * from functional.alltypes order by hive_lower(string_col); 01:SORT order by: default.hive_lower(string_col) /* JAVA UDF */ ASC materialized: default.hive_lower(string_col) /* JAVA UDF */ This shows up in the runtime profile as well. When the above query is actually run, the runtime profile includes the following entry: Frontend User Defined Functions (UDFs): default.hive_lower Testing: * Added tests to test_udfs.py * Ran core tests Change-Id: I79122e6cc74fd5a62c76962289a1615fbac2f345 --- M fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java M fe/src/main/java/org/apache/impala/service/FrontendProfile.java M tests/query_test/test_udfs.py 3 files changed, 59 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/88/16188/3 -- To view, visit http://gerrit.cloudera.org:8080/16188 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I79122e6cc74fd5a62c76962289a1615fbac2f345 Gerrit-Change-Number: 16188 Gerrit-PatchSet: 3 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9478: Profiles should indicate if custom UDFs are being used
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/16188 ) Change subject: IMPALA-9478: Profiles should indicate if custom UDFs are being used .. Patch Set 2: (2 comments) http://gerrit.cloudera.org:8080/#/c/16188/2/tests/query_test/test_udfs.py File tests/query_test/test_udfs.py: http://gerrit.cloudera.org:8080/#/c/16188/2/tests/query_test/test_udfs.py@625 PS2, Line 625: ; > flake8: E703 statement ends with a semicolon Done http://gerrit.cloudera.org:8080/#/c/16188/2/tests/query_test/test_udfs.py@632 PS2, Line 632: t > flake8: E501 line too long (92 > 90 characters) Done -- To view, visit http://gerrit.cloudera.org:8080/16188 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I79122e6cc74fd5a62c76962289a1615fbac2f345 Gerrit-Change-Number: 16188 Gerrit-PatchSet: 2 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 20 Jul 2020 21:05:13 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9478: Profiles should indicate if custom UDFs are being used
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16188 ) Change subject: IMPALA-9478: Profiles should indicate if custom UDFs are being used .. Patch Set 2: (2 comments) http://gerrit.cloudera.org:8080/#/c/16188/2/tests/query_test/test_udfs.py File tests/query_test/test_udfs.py: http://gerrit.cloudera.org:8080/#/c/16188/2/tests/query_test/test_udfs.py@625 PS2, Line 625: ; flake8: E703 statement ends with a semicolon http://gerrit.cloudera.org:8080/#/c/16188/2/tests/query_test/test_udfs.py@632 PS2, Line 632: t flake8: E501 line too long (92 > 90 characters) -- To view, visit http://gerrit.cloudera.org:8080/16188 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I79122e6cc74fd5a62c76962289a1615fbac2f345 Gerrit-Change-Number: 16188 Gerrit-PatchSet: 2 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 20 Jul 2020 21:03:52 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9478: Profiles should indicate if custom UDFs are being used
Hello Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16188 to look at the new patch set (#2). Change subject: IMPALA-9478: Profiles should indicate if custom UDFs are being used .. IMPALA-9478: Profiles should indicate if custom UDFs are being used Adds a marker to runtime profiles and explain plans indicating if custom (e.g. non-built in) user-defined functions are being used. For explain plans, a SQL-style comment is added after any function call. For runtime profiles, a new Frontend entry called "User Defined Functions (UDFs)" lists out all UDFs analyzed during planning. Take the following example: create function hive_lower(string) returns string location '/test-warehouse/hive-exec.jar' symbol='org.apache.hadoop.hive.ql.udf.UDFLower'; set explain_level=3; explain select * from functional.alltypes order by hive_lower(string_col); 01:SORT order by: default.hive_lower(string_col) /* JAVA UDF */ ASC materialized: default.hive_lower(string_col) /* JAVA UDF */ This shows up in the runtime profile as well. When the above query is actually run, the runtime profile includes the following entry: Frontend User Defined Functions (UDFs): default.hive_lower Testing: * Added tests to test_udfs.py * Ran core tests Change-Id: I79122e6cc74fd5a62c76962289a1615fbac2f345 --- M fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java M fe/src/main/java/org/apache/impala/service/FrontendProfile.java M tests/query_test/test_udfs.py 3 files changed, 58 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/88/16188/2 -- To view, visit http://gerrit.cloudera.org:8080/16188 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I79122e6cc74fd5a62c76962289a1615fbac2f345 Gerrit-Change-Number: 16188 Gerrit-PatchSet: 2 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9478: Profiles should indicate if custom UDFs are being used
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/16188 ) Change subject: IMPALA-9478: Profiles should indicate if custom UDFs are being used .. Patch Set 1: (1 comment) Addressed comments. Added a test as well. http://gerrit.cloudera.org:8080/#/c/16188/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/16188/1//COMMIT_MSG@33 PS1, Line 33: User Defined Functions (UDFs): default.hive_lower, default.hive_lower > Should we deduplicate this list? It could get pretty huge for bigger plans Done -- To view, visit http://gerrit.cloudera.org:8080/16188 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I79122e6cc74fd5a62c76962289a1615fbac2f345 Gerrit-Change-Number: 16188 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 20 Jul 2020 21:03:12 + Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP CDPD-8989 Improve admission control pool stats logging to be more explicit
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/16220 )
Change subject: WIP CDPD-8989 Improve admission control pool stats logging to
be more explicit
..
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/16220/1/be/src/scheduling/admission-controller.cc
File be/src/scheduling/admission-controller.cc:
http://gerrit.cloudera.org:8080/#/c/16220/1/be/src/scheduling/admission-controller.cc@560
PS1, Line 560: PrintBytes(admit_mem_limit), mem_consumed_by_bost,
GetStalenessDetailLocked(" "));
line too long (96 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/16220
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id995a9d044082c3b8f044e1ec25bb4c64347f781
Gerrit-Change-Number: 16220
Gerrit-PatchSet: 1
Gerrit-Owner: Qifan Chen
Gerrit-Reviewer: Bikramjeet Vig
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Sahil Takiar
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Mon, 20 Jul 2020 20:57:48 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP CDPD-8989 Improve admission control pool stats logging to be more explicit
Qifan Chen has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16220 Change subject: WIP CDPD-8989 Improve admission control pool stats logging to be more explicit .. WIP CDPD-8989 Improve admission control pool stats logging to be more explicit This work addresses the current limitation in admission controller by appending the last known memory consumption statistics about a host to the existing host memory exhaustion message. The message is logged in impalad.INFO when a query is queued or timed out due to memory exhaustion at the host. This new memory consumption statistics covers the following content: num_running: the total number of queries running top_queries: a list of query Ids for up to 5 queries with top memory consumptions min: the minimal memory consumption of all running queries max: the maximal memory consumption of all running queries total: the total memory consumption of all running queries average: the average memory consumption of all running queries One example of the appended string is as follows. Memory consumed: num_running=2, top_queries=[ dc4fd356433812be:6902546f, c54d5ab3f4773ee8:8fb6a628], min=12.30 MB, max=12.37 MB, total=24.67 MB, average=12.34 MB Testing: TBD Change-Id: Id995a9d044082c3b8f044e1ec25bb4c64347f781 --- M be/src/runtime/mem-tracker.cc M be/src/runtime/mem-tracker.h M be/src/scheduling/admission-controller.cc M common/thrift/StatestoreService.thrift 4 files changed, 191 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/20/16220/1 -- To view, visit http://gerrit.cloudera.org:8080/16220 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Id995a9d044082c3b8f044e1ec25bb4c64347f781 Gerrit-Change-Number: 16220 Gerrit-PatchSet: 1 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9799: Add retries to TestFetchFirst get num in flight queries calls
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16218 ) Change subject: IMPALA-9799: Add retries to TestFetchFirst get_num_in_flight_queries calls .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6660/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16218 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I349f861e8219e62311e8d4e0bfbd8f3618f0fa46 Gerrit-Change-Number: 16218 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 20 Jul 2020 20:30:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9799: Add retries to TestFetchFirst get num in flight queries calls
Sahil Takiar has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16218 Change subject: IMPALA-9799: Add retries to TestFetchFirst get_num_in_flight_queries calls .. IMPALA-9799: Add retries to TestFetchFirst get_num_in_flight_queries calls The calls to get_num_in_flight_queries in TestFetchFirst are flaky because they expect the number of in flight queries to drop to 0 immediately. This might not always be true, especially in ASAN builds where Impala is generally slower. This patch wraps to call to get_num_in_flight_queries in ImpalaTestSuite.assert_eventually, which adds retries to the calls to get_num_in_flight_queries. Testing: * Ran tests/hs2/test_fetch_first.py locally Change-Id: I349f861e8219e62311e8d4e0bfbd8f3618f0fa46 --- M tests/hs2/test_fetch_first.py 1 file changed, 6 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/16218/1 -- To view, visit http://gerrit.cloudera.org:8080/16218 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I349f861e8219e62311e8d4e0bfbd8f3618f0fa46 Gerrit-Change-Number: 16218 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar
[Impala-ASF-CR] IMPALA-6788: Abort ExecFInstance() RPC loop early after query failure
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16192 ) Change subject: IMPALA-6788: Abort ExecFInstance() RPC loop early after query failure .. Patch Set 4: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6146/ -- To view, visit http://gerrit.cloudera.org:8080/16192 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I034788f7720fc97c25c54f006ff72dce6cb199c3 Gerrit-Change-Number: 16192 Gerrit-PatchSet: 4 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 20 Jul 2020 20:01:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9889: Fixed flaky test runtime filters on Kudu table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16191 ) Change subject: IMPALA-9889: Fixed flaky test_runtime_filters on Kudu table .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6659/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16191 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I94a08e272f0870c04c96563fa614e3416fb5379b Gerrit-Change-Number: 16191 Gerrit-PatchSet: 5 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 20 Jul 2020 19:48:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9903: Reduce Kudu openTable calls per query
Vihang Karajgaonkar has posted comments on this change. (
http://gerrit.cloudera.org:8080/16120 )
Change subject: IMPALA-9903: Reduce Kudu openTable calls per query
..
Patch Set 2:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/16120/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/16120/2//COMMIT_MSG@9
PS2, Line 9: This patch reduces the number of Kudu openTable calls for a query
: by storing the KuduTable object in the FeKuduTable and using it
: in places openTable was previously used.
I think it would be good to be more specific here. Looks like currently we get
the KuduTable object every time in the KuduScanNode which can possibly mean
multiple times per query.
Do we want to tie this with the invalidate/refresh table mechanism? Or do we
want it at once per query granularity?
http://gerrit.cloudera.org:8080/#/c/16120/2/fe/src/main/java/org/apache/impala/catalog/KuduTable.java
File fe/src/main/java/org/apache/impala/catalog/KuduTable.java:
http://gerrit.cloudera.org:8080/#/c/16120/2/fe/src/main/java/org/apache/impala/catalog/KuduTable.java@185
PS2, Line 185: if (kuduTable_ == null) {
: KuduClient client =
KuduUtil.getKuduClient(getKuduMasterHosts());
: kuduTable_ = client.openTable(getKuduTableName());
: }
: return kuduTable_;
This would mean that once kuduTable_ is initialized, it never gets refreshed.
Even after a explicit refresh command which would call the loadSchemaFromKudu().
http://gerrit.cloudera.org:8080/#/c/16120/2/fe/src/main/java/org/apache/impala/catalog/KuduTable.java@298
PS2, Line 298: kuduTable = getKuduTable();
This probably should be kept as is otherwise we won't see a updated Kudu schema
once the table is loaded.
--
To view, visit http://gerrit.cloudera.org:8080/16120
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iec12a5be9b30e19a123142af5453a91bd4300b63
Gerrit-Change-Number: 16120
Gerrit-PatchSet: 2
Gerrit-Owner: Grant Henke
Gerrit-Reviewer: Grant Henke
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Comment-Date: Mon, 20 Jul 2020 19:40:54 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9889: Fixed flaky test runtime filters on Kudu table
Wenzhe Zhou has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/16191 ) Change subject: IMPALA-9889: Fixed flaky test_runtime_filters on Kudu table .. IMPALA-9889: Fixed flaky test_runtime_filters on Kudu table Test cases in test_runtime_filters failed occasionally in ASAN builds due to runtime filters not arriving scan nodes in time. Query profiles showed that codegen took 2 to 4 minutes for one fragment when this issue happened. This caused hash join nodes waiting long time to generate and publish runtime filters, hence arrival delay on scan nodes. To avoid the delay, turn on ASYNC_CODEGEN for test_runtime_filters when test runs for slow build like ASAN, TSAN, UBSAN, etc. Testing: - Passed core test for regular debug build and ASAN build. Change-Id: I94a08e272f0870c04c96563fa614e3416fb5379b --- M tests/query_test/test_runtime_filters.py 1 file changed, 23 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/91/16191/5 -- To view, visit http://gerrit.cloudera.org:8080/16191 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I94a08e272f0870c04c96563fa614e3416fb5379b Gerrit-Change-Number: 16191 Gerrit-PatchSet: 5 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-9889: Fixed flaky test runtime filters on Kudu table
Wenzhe Zhou has posted comments on this change. (
http://gerrit.cloudera.org:8080/16191 )
Change subject: IMPALA-9889: Fixed flaky test_runtime_filters on Kudu table
..
Patch Set 4:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/16191/4/tests/query_test/test_runtime_filters.py
File tests/query_test/test_runtime_filters.py:
http://gerrit.cloudera.org:8080/#/c/16191/4/tests/query_test/test_runtime_filters.py@41
PS4, Line 41: is_asan
> Might be good to use runs_slowly() here instead, to ensure we don't see thi
Agree. Will apply the change for all slow build.
http://gerrit.cloudera.org:8080/#/c/16191/4/tests/query_test/test_runtime_filters.py@76
PS4, Line 76: self.execute_query("SET ASYNC_CODEGEN=TRUE")
> I think rather than doing this for each test case, you can use 'add_exec_op
Will fix it as suggested.
--
To view, visit http://gerrit.cloudera.org:8080/16191
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I94a08e272f0870c04c96563fa614e3416fb5379b
Gerrit-Change-Number: 16191
Gerrit-PatchSet: 4
Gerrit-Owner: Wenzhe Zhou
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Thomas Tauber-Marshall
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Mon, 20 Jul 2020 19:15:37 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9964: Fill file descriptors properly in setFileMetadataFromFS
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16206 ) Change subject: IMPALA-9964: Fill file descriptors properly in setFileMetadataFromFS .. Patch Set 5: Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6145/ -- To view, visit http://gerrit.cloudera.org:8080/16206 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2927171cf426597c86766fb83d565c5e57025c73 Gerrit-Change-Number: 16206 Gerrit-PatchSet: 5 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 20 Jul 2020 19:05:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6692: Trigger sort node run before hitting memory limit.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15963 ) Change subject: IMPALA-6692: Trigger sort node run before hitting memory limit. .. Patch Set 17: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6658/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15963 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2a0ba7c4bae4f1d300d4d9d7f594f63ced06a240 Gerrit-Change-Number: 15963 Gerrit-PatchSet: 17 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 20 Jul 2020 19:02:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7655: Implement codegen for conditional functions (if, isnull, coalesce)
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16208 ) Change subject: IMPALA-7655: Implement codegen for conditional functions (if, isnull, coalesce) .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6657/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16208 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I11f617a9148492ccafb46112ce0af103a10090f8 Gerrit-Change-Number: 16208 Gerrit-PatchSet: 5 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 20 Jul 2020 18:48:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6692: Trigger sort node run before hitting memory limit.
Hello David Rorke, Tim Armstrong, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15963 to look at the new patch set (#17). Change subject: IMPALA-6692: Trigger sort node run before hitting memory limit. .. IMPALA-6692: Trigger sort node run before hitting memory limit. Sorter node works by adding row batches to a sort run. After all batches are added to current unsorted run or memory limit is hit, sorter will immediately start the run. If the latter case happens, sorter will spill the sorted run to disk after sort complete, create new unsorted run object, and continue to add the next row batches, and so on. This algorithm tries to fit as much rows into memory before start sorting. However, in the case of partitioned sort with large number of row batches, fitting too much rows into memory will cause the sort to be slow and block the sorter node for a long time before it can release some memory and continue accepting the next row batch from exchange node. One slow sorter node can block exchange node from sending row batches to other sorter node that is free. This patch speeds up the decision to start the sort without waiting it to hit memory limit first by capping the intermediary quicksort run to lower memory limit, determined by query option 'sort_run_bytes_limit'. If the total used reservation of quicksort has exceeded sort_run_bytes_limit, current unsorted_run_ will be wrapped up, sorted, and then spilled. Thus, overlapping the next sort run with spill from previous sort run. To reduce regression for cases where total input size of sort node might be fully fit into available memory, sort_run_bytes_limit will not be enforced for the first sort run. However, it will stay limited by sort_run_bytes_limit if planner estimates hint that spill is inevitably will happen. We also add new summary counter 'AddBatchTime' to get summary of how much time spent in Sorter::AddBatch. Max of 'AddBatchTime' indicate the longest time spent in Sorter::AddBatch, presumably busy doing intermediary sort. Testing: - Add new e2e test TestQueryFullSort::test_multiple_sort_run_bytes_limits - Run core tests - Run data loading of 3 largest TPC-DS facts table of 300GB scale into real cluster using 5 backends, and 4GB mem_limit. sort_run_bytes_limit is varied between unspecified (not limited) vs 512 MB. The performance result is summarized in the following table. +---+-+--+---+-+ | Insert table | #Rows | Avg | no limit| 512 MB limit | | | | SortDataSize ++--+-+---+ | | | per Node | Query | Max | Query | Max | | | | | Time | AddBatchTime | Time | AddBatchTime | +---+-+--++--+-+---+ | store_sales | 864.00M | 15.29 GB | 30m18s | 53s311ms | 20m | 5s634ms | +---+-+--++--+-+---+ | catalog_sales | 431.97M | 11.34 GB | 23m24s | 31s212ms | 15m27s | 3s603ms | +---+-+--++--+-+---+ | web_sales | 216.01M | 5.67 GB | 8m16s | 29s250ms | 6m41s | 3s856ms | +---+-+--++--+-+---+ Change-Id: I2a0ba7c4bae4f1d300d4d9d7f594f63ced06a240 --- M be/src/exec/sort-node.cc M be/src/exec/sort-node.h M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/query-state.cc M be/src/runtime/query-state.h M be/src/runtime/sorter.cc M be/src/runtime/sorter.h M be/src/service/query-options-test.cc M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M common/thrift/PlanNodes.thrift M fe/src/main/java/org/apache/impala/planner/SortNode.java M tests/query_test/test_sort.py 15 files changed, 224 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/15963/17 -- To view, visit http://gerrit.cloudera.org:8080/15963 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2a0ba7c4bae4f1d300d4d9d7f594f63ced06a240 Gerrit-Change-Number: 15963 Gerrit-PatchSet: 17 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7655: Implement codegen for conditional functions (if, isnull, coalesce)
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16208 ) Change subject: IMPALA-7655: Implement codegen for conditional functions (if, isnull, coalesce) .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16208 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I11f617a9148492ccafb46112ce0af103a10090f8 Gerrit-Change-Number: 16208 Gerrit-PatchSet: 5 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 20 Jul 2020 18:27:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7655: Implement codegen for conditional functions (if, isnull, coalesce)
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16208 ) Change subject: IMPALA-7655: Implement codegen for conditional functions (if, isnull, coalesce) .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6147/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16208 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I11f617a9148492ccafb46112ce0af103a10090f8 Gerrit-Change-Number: 16208 Gerrit-PatchSet: 6 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 20 Jul 2020 18:27:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7655: Implement codegen for conditional functions (if, isnull, coalesce)
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16208 ) Change subject: IMPALA-7655: Implement codegen for conditional functions (if, isnull, coalesce) .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16208 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I11f617a9148492ccafb46112ce0af103a10090f8 Gerrit-Change-Number: 16208 Gerrit-PatchSet: 6 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 20 Jul 2020 18:27:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9964: Fill file descriptors properly in setFileMetadataFromFS
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/16206 ) Change subject: IMPALA-9964: Fill file descriptors properly in setFileMetadataFromFS .. Patch Set 5: Code-Review+2 Thanks for making the suggested changes. -- To view, visit http://gerrit.cloudera.org:8080/16206 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2927171cf426597c86766fb83d565c5e57025c73 Gerrit-Change-Number: 16206 Gerrit-PatchSet: 5 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 20 Jul 2020 18:21:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7655: Implement codegen for conditional functions (if, isnull, coalesce)
Daniel Becker has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/16208 ) Change subject: IMPALA-7655: Implement codegen for conditional functions (if, isnull, coalesce) .. IMPALA-7655: Implement codegen for conditional functions (if, isnull, coalesce) Implement proper codegen for conditional functions (if, isnull, coalesce) instead of simply calling into interpreted code. We use IRBuilder to generate hand-crafted code. Change-Id: I11f617a9148492ccafb46112ce0af103a10090f8 --- M be/src/exprs/case-expr.cc M be/src/exprs/conditional-functions-ir.cc M be/src/exprs/conditional-functions.cc 3 files changed, 289 insertions(+), 85 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/08/16208/5 -- To view, visit http://gerrit.cloudera.org:8080/16208 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I11f617a9148492ccafb46112ce0af103a10090f8 Gerrit-Change-Number: 16208 Gerrit-PatchSet: 5 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 ) Change subject: IMPALA-9741: Support querying Iceberg table by impala .. Patch Set 15: Oh and please do a rebase on top of master because gerrit says that this PS currently has a merge conflict. -- To view, visit http://gerrit.cloudera.org:8080/16143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006 Gerrit-Change-Number: 16143 Gerrit-PatchSet: 15 Gerrit-Owner: wangsheng Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Mon, 20 Jul 2020 17:46:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 ) Change subject: IMPALA-9741: Support querying Iceberg table by impala .. Patch Set 15: Code-Review+1 (8 comments) Thank you very much for your quick and great work! Only found a few nits, other than that LGTM! http://gerrit.cloudera.org:8080/#/c/16143/15/fe/src/main/java/org/apache/impala/analysis/TruncateStmt.java File fe/src/main/java/org/apache/impala/analysis/TruncateStmt.java: http://gerrit.cloudera.org:8080/#/c/16143/15/fe/src/main/java/org/apache/impala/analysis/TruncateStmt.java@66 PS15, Line 66: iceberg table not supported truncate. nit: "we also cannot truncate Iceberg tables." http://gerrit.cloudera.org:8080/#/c/16143/15/testdata/data/README File testdata/data/README: http://gerrit.cloudera.org:8080/#/c/16143/15/testdata/data/README@503 PS15, Line 503: Icebreg nit: Iceberg http://gerrit.cloudera.org:8080/#/c/16143/15/testdata/data/README@507 PS15, Line 507: including table metadata nit: "which contains table metadata" http://gerrit.cloudera.org:8080/#/c/16143/15/testdata/data/README@507 PS15, Line 507: managerd managed http://gerrit.cloudera.org:8080/#/c/16143/15/testdata/data/README@508 PS15, Line 508: including data files nit: which contains the data files. http://gerrit.cloudera.org:8080/#/c/16143/15/testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test File testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test: http://gerrit.cloudera.org:8080/#/c/16143/15/testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test@19 PS15, Line 19: ImpalaRuntimeException: Error making 'createTable' RPC to Hive Metastore: : CAUSED BY: IllegalArgumentException: Cannot find source column: event_time Can we catch this error during analysis, and throw an AnalysisException instead of ImpalaRuntimeException? http://gerrit.cloudera.org:8080/#/c/16143/15/testdata/workloads/functional-query/queries/QueryTest/iceberg-profile.test File testdata/workloads/functional-query/queries/QueryTest/iceberg-profile.test: http://gerrit.cloudera.org:8080/#/c/16143/15/testdata/workloads/functional-query/queries/QueryTest/iceberg-profile.test@7 PS15, Line 7: row_regex: .* PARQUET/GZIP:4 : row_regex: .* PARQUET/GZIP:2 I'm just wondering how deterministic is it. Do the test consistently pass if you run it multiple times? The alternative is to use SUM(aggregation, RowsRead) http://gerrit.cloudera.org:8080/#/c/16143/13/testdata/workloads/functional-query/queries/QueryTest/iceberg_query.test File testdata/workloads/functional-query/queries/QueryTest/iceberg_query.test: http://gerrit.cloudera.org:8080/#/c/16143/13/testdata/workloads/functional-query/queries/QueryTest/iceberg_query.test@77 PS13, Line 77: > 1. I've already add non-count queries in iceberg-query.test The generated test files usually contain blank lines which needs to be removed. You can remove the blank lines in vim with the following: :g/^$/d Other than that it's up to you if you just copy over the original iceberg-query.test file, or just copy-paste some parts of it. But before committing the file don't forget to check that you only modified/added tests that you intended. -- To view, visit http://gerrit.cloudera.org:8080/16143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006 Gerrit-Change-Number: 16143 Gerrit-PatchSet: 15 Gerrit-Owner: wangsheng Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Mon, 20 Jul 2020 17:44:18 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-3127: Support incremental metadata updates in partition level
Vihang Karajgaonkar has posted comments on this change. (
http://gerrit.cloudera.org:8080/16159 )
Change subject: IMPALA-3127: Support incremental metadata updates in partition
level
..
Patch Set 3:
(13 comments)
Sorry for the delay in the review. I took another pass and overall the approach
makes sense to me. I think it would be useful if we could have an exhaustive
test (may be in a separate jira) to make sure that we are not leaking
partitions in statestore. The test could add/drop partitions along with
multiple add/invalidate/drop table commands and make sure that the number of
partition keys in the statestore is as per our expectation.
http://gerrit.cloudera.org:8080/#/c/16159/3/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:
http://gerrit.cloudera.org:8080/#/c/16159/3/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@712
PS3, Line 712: if (!delete && obj.type == HDFS_PARTITION) return;
Its unclear to me that when we generate the minimalObject when delete flag is
false, we set the prev_id in the partObject.setId on line 746. However, that
object is never sent as per this line. How do we invalidate the updated
partitions when topicMode is minimal?
Also, would be good to add a comment here (or somewhere else if more
appropriate) explaining what we send in case of partitions in both v1 and v2
modes since it is not trivial to understand the subtle differences.
For instance, as I understand:
1. In v1 mode (topicMode = full), we only send the partitionIds in the thrift
table which represents the current list of the partitions. Additionally, for
each newly added/removed partition we send a THdfsPartition in the same topic
update. However, coordinators detect the removal of any partitions by absence
of a id partitionIds in the table object.
2. In v2 mode (topicMode = minimal), local coordinators only load what they
need and hence we only send deleted partitionIds. Updated partitions are also
treated as a special case of deleted partitions by sending the previous
partitionId for such partitions so that local coordinators invalidate them
proactively.
http://gerrit.cloudera.org:8080/#/c/16159/3/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@742
PS3, Line 742: if (isDelete) {
: partObject.setId(obj.hdfs_partition.id);
: } else if (obj.hdfs_partition.prev_id !=
HdfsPartition.INITIAL_PARTITION_ID - 1) {
: // For updates, coordinators can invalidate the old
partition instance.
: partObject.setId(obj.hdfs_partition.prev_id);
: }
This looks a bit hacky to me. Do you think it would be more readable by adding
a explicit field to THdfsPartition called prev_id or a boolean indicating that
the id is previous id and let local coordinator decide how to process the
partition. This implementation on the catalogd side assumes the knowledge of
how local catalog invalidates stuff and I think it would cleaner to keep them
independent as much as possible.
If you decide to have prev_id field in the THdfsPartition you can keep a
default value of field to -1 so that its not serialized unless its set.
http://gerrit.cloudera.org:8080/#/c/16159/3/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@1295
PS3, Line 1295: catalogTbl.setTable(((HdfsTable)
tbl).toThriftWithPartitionIds());
wouldn't this line be called for both fullUpdate and a incremental update? The
javadoc of the method toThriftWithPartitionIds suggests this is called for
incremental updates which seems confusing.
http://gerrit.cloudera.org:8080/#/c/16159/3/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@1321
PS3, Line 1321: if (maxSentId < catalogPart.getHdfs_partition().getId()) {
: maxSentId = catalogPart.getHdfs_partition().getId();
: }
nit, perhaps this is more readable?
maxSentId = Math.max(maxSentId, catalogPart.getHdfs_partition().getId());
http://gerrit.cloudera.org:8080/#/c/16159/3/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@1329
PS3, Line 1329: f (!ctx.isFullUpdate())
Can you clarify why this is needed only in case of incremental updates? What
happens on the coordinators when a statestore is restarted and then it requests
for a full-update?
http://gerrit.cloudera.org:8080/#/c/16159/3/fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java
File fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java:
http://gerrit.cloudera.org:8080/#/c/16159/3/fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java@90
PS3, Line 90: HdfsPartition
I think it is worth documenting that even though this extends
CatalogObjectImpl, we don't explicitly have a
[Impala-ASF-CR] IMPALA-6788: Abort ExecFInstance() RPC loop early after query failure
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16192 ) Change subject: IMPALA-6788: Abort ExecFInstance() RPC loop early after query failure .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6656/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16192 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I034788f7720fc97c25c54f006ff72dce6cb199c3 Gerrit-Change-Number: 16192 Gerrit-PatchSet: 4 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 20 Jul 2020 16:42:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6788: Abort ExecFInstance() RPC loop early after query failure
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16192 ) Change subject: IMPALA-6788: Abort ExecFInstance() RPC loop early after query failure .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6146/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16192 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I034788f7720fc97c25c54f006ff72dce6cb199c3 Gerrit-Change-Number: 16192 Gerrit-PatchSet: 4 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 20 Jul 2020 16:15:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6788: Abort ExecFInstance() RPC loop early after query failure
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/16192 ) Change subject: IMPALA-6788: Abort ExecFInstance() RPC loop early after query failure .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16192 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I034788f7720fc97c25c54f006ff72dce6cb199c3 Gerrit-Change-Number: 16192 Gerrit-PatchSet: 4 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 20 Jul 2020 16:14:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6788: Abort ExecFInstance() RPC loop early after query failure
Wenzhe Zhou has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/16192 ) Change subject: IMPALA-6788: Abort ExecFInstance() RPC loop early after query failure .. IMPALA-6788: Abort ExecFInstance() RPC loop early after query failure Stops issuing ExecQueryFInstance rpcs and cancels any inflight when backend reports failure. Adds new debug action CONSTRUCT_QUERY_STATE_REPORT that runs when constructing a query state report. Adds a new test case for handling errors reported from query state. Testing: - Ran following command for new test case and verified that the code working as expected: ./bin/impala-py.test tests/custom_cluster/test_rpc_exception.py\ ::TestRPCException::test_state_report_error \ --workload_exploration_strategy=functional-query:exhaustive - Passed exhaustive tests. Change-Id: I034788f7720fc97c25c54f006ff72dce6cb199c3 --- M be/src/runtime/coordinator.cc M be/src/runtime/query-state.cc M tests/custom_cluster/test_rpc_exception.py 3 files changed, 40 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/92/16192/4 -- To view, visit http://gerrit.cloudera.org:8080/16192 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I034788f7720fc97c25c54f006ff72dce6cb199c3 Gerrit-Change-Number: 16192 Gerrit-PatchSet: 4 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-6788: Abort ExecFInstance() RPC loop early after query failure
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/16192 ) Change subject: IMPALA-6788: Abort ExecFInstance() RPC loop early after query failure .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/16192/3/be/src/runtime/coordinator.cc File be/src/runtime/coordinator.cc: http://gerrit.cloudera.org:8080/#/c/16192/3/be/src/runtime/coordinator.cc@949 PS3, Line 949: IMPALA-6788 > You can remove this. Fixed it. -- To view, visit http://gerrit.cloudera.org:8080/16192 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I034788f7720fc97c25c54f006ff72dce6cb199c3 Gerrit-Change-Number: 16192 Gerrit-PatchSet: 3 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 20 Jul 2020 16:10:35 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6788: Abort ExecFInstance() RPC loop early after query failure
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/16192 ) Change subject: IMPALA-6788: Abort ExecFInstance() RPC loop early after query failure .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/16192/3/be/src/runtime/coordinator.cc File be/src/runtime/coordinator.cc: http://gerrit.cloudera.org:8080/#/c/16192/3/be/src/runtime/coordinator.cc@949 PS3, Line 949: IMPALA-6788 You can remove this. -- To view, visit http://gerrit.cloudera.org:8080/16192 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I034788f7720fc97c25c54f006ff72dce6cb199c3 Gerrit-Change-Number: 16192 Gerrit-PatchSet: 3 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Mon, 20 Jul 2020 15:58:20 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9889: Fixed flaky test runtime filters on Kudu table
Thomas Tauber-Marshall has posted comments on this change. (
http://gerrit.cloudera.org:8080/16191 )
Change subject: IMPALA-9889: Fixed flaky test_runtime_filters on Kudu table
..
Patch Set 4:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/16191/4/tests/query_test/test_runtime_filters.py
File tests/query_test/test_runtime_filters.py:
http://gerrit.cloudera.org:8080/#/c/16191/4/tests/query_test/test_runtime_filters.py@41
PS4, Line 41: is_asan
Might be good to use runs_slowly() here instead, to ensure we don't see this
same flakiness on ubsan, tsan, etc.
http://gerrit.cloudera.org:8080/#/c/16191/4/tests/query_test/test_runtime_filters.py@76
PS4, Line 76: self.execute_query("SET ASYNC_CODEGEN=TRUE")
I think rather than doing this for each test case, you can use
'add_exec_option_dimension' in add_test_dimensions above, which will ensure
every test in the class is covered.
--
To view, visit http://gerrit.cloudera.org:8080/16191
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I94a08e272f0870c04c96563fa614e3416fb5379b
Gerrit-Change-Number: 16191
Gerrit-PatchSet: 4
Gerrit-Owner: Wenzhe Zhou
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Thomas Tauber-Marshall
Gerrit-Comment-Date: Mon, 20 Jul 2020 15:33:19 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7655: Implement codegen for conditional functions (if, isnull, coalesce)
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16208 ) Change subject: IMPALA-7655: Implement codegen for conditional functions (if, isnull, coalesce) .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6655/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16208 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I11f617a9148492ccafb46112ce0af103a10090f8 Gerrit-Change-Number: 16208 Gerrit-PatchSet: 4 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 20 Jul 2020 15:27:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7655: Implement codegen for conditional functions (if, isnull, coalesce)
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16208 ) Change subject: IMPALA-7655: Implement codegen for conditional functions (if, isnull, coalesce) .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/16208/4/be/src/exprs/conditional-functions.cc File be/src/exprs/conditional-functions.cc: http://gerrit.cloudera.org:8080/#/c/16208/4/be/src/exprs/conditional-functions.cc@23 PS4, Line 23: // TODO: Do we need this comment? > Do you think we should have this comment? Let's remove the comment. The behaviour it's describing is the expected behaviour of functions marked inline. -- To view, visit http://gerrit.cloudera.org:8080/16208 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I11f617a9148492ccafb46112ce0af103a10090f8 Gerrit-Change-Number: 16208 Gerrit-PatchSet: 4 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 20 Jul 2020 15:06:00 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7655: Implement codegen for conditional functions (if, isnull, coalesce)
Daniel Becker has posted comments on this change. ( http://gerrit.cloudera.org:8080/16208 ) Change subject: IMPALA-7655: Implement codegen for conditional functions (if, isnull, coalesce) .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/16208/4/be/src/exprs/conditional-functions.cc File be/src/exprs/conditional-functions.cc: http://gerrit.cloudera.org:8080/#/c/16208/4/be/src/exprs/conditional-functions.cc@23 PS4, Line 23: // TODO: Do we need this comment? Do you think we should have this comment? -- To view, visit http://gerrit.cloudera.org:8080/16208 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I11f617a9148492ccafb46112ce0af103a10090f8 Gerrit-Change-Number: 16208 Gerrit-PatchSet: 4 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 20 Jul 2020 15:00:02 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7655: Implement codegen for conditional functions (if, isnull, coalesce)
Daniel Becker has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/16208 ) Change subject: IMPALA-7655: Implement codegen for conditional functions (if, isnull, coalesce) .. IMPALA-7655: Implement codegen for conditional functions (if, isnull, coalesce) Implement proper codegen for conditional functions (if, isnull, coalesce) instead of simply calling into interpreted code. We use IRBuilder to generate hand-crafted code. Change-Id: I11f617a9148492ccafb46112ce0af103a10090f8 --- M be/src/exprs/case-expr.cc M be/src/exprs/conditional-functions-ir.cc M be/src/exprs/conditional-functions.cc 3 files changed, 295 insertions(+), 85 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/08/16208/4 -- To view, visit http://gerrit.cloudera.org:8080/16208 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I11f617a9148492ccafb46112ce0af103a10090f8 Gerrit-Change-Number: 16208 Gerrit-PatchSet: 4 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9964: Fill file descriptors properly in setFileMetadataFromFS
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/16206 ) Change subject: IMPALA-9964: Fill file descriptors properly in setFileMetadataFromFS .. Patch Set 5: The earlier verification job failed by IMPALA-9923. -- To view, visit http://gerrit.cloudera.org:8080/16206 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2927171cf426597c86766fb83d565c5e57025c73 Gerrit-Change-Number: 16206 Gerrit-PatchSet: 5 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 20 Jul 2020 13:57:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9964: Fill file descriptors properly in setFileMetadataFromFS
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16206 ) Change subject: IMPALA-9964: Fill file descriptors properly in setFileMetadataFromFS .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6145/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/16206 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2927171cf426597c86766fb83d565c5e57025c73 Gerrit-Change-Number: 16206 Gerrit-PatchSet: 5 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 20 Jul 2020 13:57:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 ) Change subject: IMPALA-9741: Support querying Iceberg table by impala .. Patch Set 15: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6654/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006 Gerrit-Change-Number: 16143 Gerrit-PatchSet: 15 Gerrit-Owner: wangsheng Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Mon, 20 Jul 2020 11:58:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9964: Fill file descriptors properly in setFileMetadataFromFS
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16206 ) Change subject: IMPALA-9964: Fill file descriptors properly in setFileMetadataFromFS .. Patch Set 5: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6144/ -- To view, visit http://gerrit.cloudera.org:8080/16206 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2927171cf426597c86766fb83d565c5e57025c73 Gerrit-Change-Number: 16206 Gerrit-PatchSet: 5 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 20 Jul 2020 11:57:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 ) Change subject: IMPALA-9741: Support querying Iceberg table by impala .. Patch Set 14: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6653/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006 Gerrit-Change-Number: 16143 Gerrit-PatchSet: 14 Gerrit-Owner: wangsheng Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Mon, 20 Jul 2020 11:55:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
wangsheng has uploaded a new patch set (#15). (
http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
..
IMPALA-9741: Support querying Iceberg table by impala
This patch mainly realizes the querying of iceberg table through impala,
we can use the following sql to create an external iceberg table:
CREATE EXTERNAL TABLE default.iceberg_test (
level string,
event_time timestamp,
message string,
)
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
Or just including table name and location like this:
CREATE EXTERNAL TABLE default.iceberg_test
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
'iceberg_file_format' is the file format in iceberg, currently only
support PARQUET, other format would be supported in the future. And
if you don't specify this property in your SQL, default file format
is PARQUET.
We achieved this function by treating the iceberg table as normal
unpartitioned hdfs table. When querying iceberg table, we pushdown
partition column predicates to iceberg to decide which data files
need to be scanned, and then transfer this information to BE to
do the real scan operation.
Testing:
- Unit test for Iceberg in FileMetadataLoaderTest
- Create table tests in functional_schema_template.sql
- Iceberg table query test in test_scanners.py
Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
---
M be/src/runtime/descriptors.cc
M bin/rat_exclude_files.txt
M common/thrift/CatalogObjects.thrift
M fe/pom.xml
M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionSpec.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/analysis/TruncateStmt.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsPartition.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
A fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java
M testdata/data/README
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/1-100-e1a80ed6-1064-494d-9cdd-c4a30c1ab8dc-0.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/3-102-511427f2-85f0-43ae-9b39-a456f8dc57b6-0.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/4-103-00fc55e1-6ef7-4241-ace2-6d075b9737fc-0.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/6-105-ef9e76d5-c060-4040-8aa1-b7c275610daa-0.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/7-106-c09c9c8d-9478-44f9-8501-f85f53112bc3-0.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/9-108-3b4f06ac-dca3-4f4e-be60-bf42d9927b5b-0.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00011-110-1e653ccf-0963-4fb0-941c-32c9de13268b-0.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00012-111-dfa70658-eb4b-4fa0-9ffa-b892cf90d6ac-0.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00014-113-2d16e751-e2a4-4856-ab89-145996e3815e-0.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00015-114-0f710621-cbbf-4509-a93d-b58808978e2e-0.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00017-116-0b666c79-53df-4507-906c-542e65a83443-0.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00019-118-1bc6bc6e-e061-4da3-9d1e-a427a306c471-0.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00020-119-ae7b2c67-1538-4429-8246-4998960e3817-0.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00022-121-8db0f1e1-d88c-4aad-a8b3-24fd07329cdb-0.parquet
A
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 ) Change subject: IMPALA-9741: Support querying Iceberg table by impala .. Patch Set 14: (1 comment) http://gerrit.cloudera.org:8080/#/c/16143/14/tests/query_test/test_iceberg.py File tests/query_test/test_iceberg.py: http://gerrit.cloudera.org:8080/#/c/16143/14/tests/query_test/test_iceberg.py@18 PS14, Line 18: import pytest flake8: F401 'pytest' imported but unused -- To view, visit http://gerrit.cloudera.org:8080/16143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006 Gerrit-Change-Number: 16143 Gerrit-PatchSet: 14 Gerrit-Owner: wangsheng Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Mon, 20 Jul 2020 11:27:22 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
wangsheng has uploaded a new patch set (#14). (
http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
..
IMPALA-9741: Support querying Iceberg table by impala
This patch mainly realizes the querying of iceberg table through impala,
we can use the following sql to create an external iceberg table:
CREATE EXTERNAL TABLE default.iceberg_test (
level string,
event_time timestamp,
message string,
)
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
Or just including table name and location like this:
CREATE EXTERNAL TABLE default.iceberg_test
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
'iceberg_file_format' is the file format in iceberg, currently only
support PARQUET, other format would be supported in the future. And
if you don't specify this property in your SQL, default file format
is PARQUET.
We achieved this function by treating the iceberg table as normal
unpartitioned hdfs table. When querying iceberg table, we pushdown
partition column predicates to iceberg to decide which data files
need to be scanned, and then transfer this information to BE to
do the real scan operation.
Testing:
- Unit test for Iceberg in FileMetadataLoaderTest
- Create table tests in functional_schema_template.sql
- Iceberg table query test in test_scanners.py
Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
---
M be/src/runtime/descriptors.cc
M bin/rat_exclude_files.txt
M common/thrift/CatalogObjects.thrift
M fe/pom.xml
M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionSpec.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/analysis/TruncateStmt.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsPartition.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
A fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java
M testdata/data/README
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/1-100-e1a80ed6-1064-494d-9cdd-c4a30c1ab8dc-0.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/3-102-511427f2-85f0-43ae-9b39-a456f8dc57b6-0.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/4-103-00fc55e1-6ef7-4241-ace2-6d075b9737fc-0.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/6-105-ef9e76d5-c060-4040-8aa1-b7c275610daa-0.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/7-106-c09c9c8d-9478-44f9-8501-f85f53112bc3-0.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/9-108-3b4f06ac-dca3-4f4e-be60-bf42d9927b5b-0.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00011-110-1e653ccf-0963-4fb0-941c-32c9de13268b-0.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00012-111-dfa70658-eb4b-4fa0-9ffa-b892cf90d6ac-0.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00014-113-2d16e751-e2a4-4856-ab89-145996e3815e-0.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00015-114-0f710621-cbbf-4509-a93d-b58808978e2e-0.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00017-116-0b666c79-53df-4507-906c-542e65a83443-0.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00019-118-1bc6bc6e-e061-4da3-9d1e-a427a306c471-0.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00020-119-ae7b2c67-1538-4429-8246-4998960e3817-0.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00022-121-8db0f1e1-d88c-4aad-a8b3-24fd07329cdb-0.parquet
A
[Impala-ASF-CR] IMPALA-9964: Fill file descriptors properly in setFileMetadataFromFS
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16206 ) Change subject: IMPALA-9964: Fill file descriptors properly in setFileMetadataFromFS .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6652/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16206 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2927171cf426597c86766fb83d565c5e57025c73 Gerrit-Change-Number: 16206 Gerrit-PatchSet: 5 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 20 Jul 2020 08:34:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9964: Fill file descriptors properly in setFileMetadataFromFS
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16206 ) Change subject: IMPALA-9964: Fill file descriptors properly in setFileMetadataFromFS .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6144/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/16206 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2927171cf426597c86766fb83d565c5e57025c73 Gerrit-Change-Number: 16206 Gerrit-PatchSet: 5 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 20 Jul 2020 08:08:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9964: Fill file descriptors properly in setFileMetadataFromFS
Zoltan Borok-Nagy has posted comments on this change. (
http://gerrit.cloudera.org:8080/16206 )
Change subject: IMPALA-9964: Fill file descriptors properly in
setFileMetadataFromFS
..
Patch Set 5:
(9 comments)
Thanks for the comments!
http://gerrit.cloudera.org:8080/#/c/16206/4/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:
http://gerrit.cloudera.org:8080/#/c/16206/4/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@3150
PS4, Line 3150: load();
> Ah, it looks we changed this recently in IMPALA-9778. In that case, I think
Done
http://gerrit.cloudera.org:8080/#/c/16206/4/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@3152
PS4, Line 3152: // Let's retrieve the original partition instance from
builder because this is
> I think it would be more readable if you can you add a comment here which s
Done
http://gerrit.cloudera.org:8080/#/c/16206/4/fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java
File fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java:
http://gerrit.cloudera.org:8080/#/c/16206/4/fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java@1005
PS4, Line 1005: partitionKeyValues_ = partition.partitionKeyValues_;
> Does this need to copy the encoded insert and delete FileDescriptors as wel
Yes, thanks. This would probably cause a bug after using ALTER TABLE statements
(once we start supporting such statements for ACID tables).
http://gerrit.cloudera.org:8080/#/c/16206/4/fe/src/main/java/org/apache/impala/catalog/ParallelFileMetadataLoader.java
File fe/src/main/java/org/apache/impala/catalog/ParallelFileMetadataLoader.java:
http://gerrit.cloudera.org:8080/#/c/16206/4/fe/src/main/java/org/apache/impala/catalog/ParallelFileMetadataLoader.java@115
PS4, Line 115: HdfsPartition.
> nit, please change this to HdfsPartition.Builders
Done
http://gerrit.cloudera.org:8080/#/c/16206/4/fe/src/main/java/org/apache/impala/catalog/ParallelFileMetadataLoader.java@144
PS4, Line 144: load
> may be rename this to loadInternal and loadAndSet to load()
Done
http://gerrit.cloudera.org:8080/#/c/16206/4/fe/src/test/java/org/apache/impala/catalog/PartialCatalogInfoWriteIdTest.java
File
fe/src/test/java/org/apache/impala/catalog/PartialCatalogInfoWriteIdTest.java:
http://gerrit.cloudera.org:8080/#/c/16206/4/fe/src/test/java/org/apache/impala/catalog/PartialCatalogInfoWriteIdTest.java@563
PS4, Line 563: + " where id % 2 = 0");
: catalog_.reset();
: Table tbl = catalog_.getOrLoadTable(testDbName,
testAcidTblName, "test", null);
: Assert.assertFalse("Table mus
> Its a little weird that we execute hiveSql in the try catch block for Impal
Done
http://gerrit.cloudera.org:8080/#/c/16206/4/fe/src/test/java/org/apache/impala/catalog/PartialCatalogInfoWriteIdTest.java@572
PS4, Line 572: + " partition(year=2010,month=10) compact '
> not used
Done
http://gerrit.cloudera.org:8080/#/c/16206/4/fe/src/test/java/org/apache/impala/catalog/PartialCatalogInfoWriteIdTest.java@576
PS4, Line 576: .db(testDbName)
: .tbl(testAcidTblName)
> this insert is not needed
Done
http://gerrit.cloudera.org:8080/#/c/16206/4/fe/src/test/java/org/apache/impala/catalog/PartialCatalogInfoWriteIdTest.java@590
PS4, Line 590: Assert.assertEquals(0,
part.delete_file_descriptors.size());
> I think it would be useful to add the following asserts here to make sure t
Done
--
To view, visit http://gerrit.cloudera.org:8080/16206
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2927171cf426597c86766fb83d565c5e57025c73
Gerrit-Change-Number: 16206
Gerrit-PatchSet: 5
Gerrit-Owner: Zoltan Borok-Nagy
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Mon, 20 Jul 2020 08:07:56 +
Gerrit-HasComments: Yes
