[Impala-ASF-CR] IMPALA-10377: Improve the accuracy of resource estimation
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16842 ) Change subject: IMPALA-10377: Improve the accuracy of resource estimation .. Patch Set 19: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8259/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16842 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic01db168ff2c6d6de33ee553a8175599f035d7a1 Gerrit-Change-Number: 16842 Gerrit-PatchSet: 19 Gerrit-Owner: liuyao Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: liuyao Gerrit-Comment-Date: Sat, 27 Feb 2021 06:08:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10377: Improve the accuracy of resource estimation
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16842 ) Change subject: IMPALA-10377: Improve the accuracy of resource estimation .. Patch Set 19: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6925/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/16842 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic01db168ff2c6d6de33ee553a8175599f035d7a1 Gerrit-Change-Number: 16842 Gerrit-PatchSet: 19 Gerrit-Owner: liuyao Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: liuyao Gerrit-Comment-Date: Sat, 27 Feb 2021 05:51:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10377: Improve the accuracy of resource estimation
Hello Aman Sinha, Zoltan Borok-Nagy, Tim Armstrong, Bikramjeet Vig, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16842 to look at the new patch set (#19). Change subject: IMPALA-10377: Improve the accuracy of resource estimation .. IMPALA-10377: Improve the accuracy of resource estimation PlanNode does not consider some factors when estimating memory, this will cause a large error rate AggregationNode 1.MemoryEstimate = Ndv * (AvgRowSize + SizeOfBucket) 2.When estimating the Ndv of merge aggregation, Ndv should be divided only once. 3.If there is no grouping exprs, MemoryEstimate = MIN_PLAIN_AGG_MEM SortNode 1.MemoryEstimate = Cardinality * AvgRowSize. Memory used when there is enough memory HashJoinNode 1.MemoryEstimate= DataRows + Buckets + DuplicateNodes, DataRows = RightTableCardinality * AvgRowSize, Buckets= roundUpToPowerOf2(RightTableCardinality) * SizeOfBucket, DuplicateNodes = (RightTableCardinality - RightNdv) * SizeOfDuplicateNode KuduScanNode 1.MemoryEstimate = Columns * BytesPerColumn * MaxScannerThreads, Columns are scanned in query, not all the columns of the table UnitTest 1.CardinalityTest adds test cases to test memory estimation. Modify existing test cases related to memory estimation Change-Id: Ic01db168ff2c6d6de33ee553a8175599f035d7a1 --- M fe/src/main/java/org/apache/impala/planner/AggregationNode.java M fe/src/main/java/org/apache/impala/planner/HashJoinNode.java M fe/src/main/java/org/apache/impala/planner/JoinNode.java M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java M fe/src/main/java/org/apache/impala/planner/PlanFragment.java M fe/src/main/java/org/apache/impala/planner/PlanNode.java M fe/src/main/java/org/apache/impala/planner/PlannerContext.java M fe/src/main/java/org/apache/impala/planner/SortNode.java M fe/src/test/java/org/apache/impala/planner/CardinalityTest.java M testdata/workloads/functional-planner/queries/PlannerTest/bloom-filter-assignment.test M testdata/workloads/functional-planner/queries/PlannerTest/constant-folding.test M testdata/workloads/functional-planner/queries/PlannerTest/constant-propagation.test M testdata/workloads/functional-planner/queries/PlannerTest/disable-codegen.test M testdata/workloads/functional-planner/queries/PlannerTest/fk-pk-join-detection-hdfs-num-rows-est-enabled.test M testdata/workloads/functional-planner/queries/PlannerTest/fk-pk-join-detection.test M testdata/workloads/functional-planner/queries/PlannerTest/max-row-size.test M testdata/workloads/functional-planner/queries/PlannerTest/min-max-runtime-filters-hdfs-num-rows-est-enabled.test M testdata/workloads/functional-planner/queries/PlannerTest/min-max-runtime-filters.test M testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test M testdata/workloads/functional-planner/queries/PlannerTest/parquet-filtering-disabled.test M testdata/workloads/functional-planner/queries/PlannerTest/parquet-filtering.test M testdata/workloads/functional-planner/queries/PlannerTest/partition-pruning.test M testdata/workloads/functional-planner/queries/PlannerTest/preagg-bytes-limit.test M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test M testdata/workloads/functional-planner/queries/PlannerTest/result-spooling.test M testdata/workloads/functional-planner/queries/PlannerTest/runtime-filter-query-options.test M testdata/workloads/functional-planner/queries/PlannerTest/sort-expr-materialization.test M testdata/workloads/functional-planner/queries/PlannerTest/spillable-buffer-sizing.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q01.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q02.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q04.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q05.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q06.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q07.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q08.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q09.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q10a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q11.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q12.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q13.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q14a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q14b.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q15.test M
[Impala-ASF-CR] IMPALA-10492: Lower default MAX CNF EXPRS query option
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/17132 ) Change subject: IMPALA-10492: Lower default MAX_CNF_EXPRS query option .. Patch Set 2: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/17132/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/17132/2//COMMIT_MSG@15 PS2, Line 15: CFN nit: CNF -- To view, visit http://gerrit.cloudera.org:8080/17132 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7ca3d0e094ac01c24a046c25d6a1b56bf134faa8 Gerrit-Change-Number: 17132 Gerrit-PatchSet: 2 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Sat, 27 Feb 2021 04:04:33 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10377: Improve the accuracy of resource estimation
liuyao has posted comments on this change. ( http://gerrit.cloudera.org:8080/16842 ) Change subject: IMPALA-10377: Improve the accuracy of resource estimation .. Patch Set 18: (1 comment) http://gerrit.cloudera.org:8080/#/c/16842/18/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java File fe/src/main/java/org/apache/impala/planner/HashJoinNode.java: http://gerrit.cloudera.org:8080/#/c/16842/18/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@238 PS18, Line 238: ndv of the right child column > nit: probably 'NDVs of the columns of the right (build) side?' Yes, it's the right child's NDV. The table on the right will be put into the hash table, but not on the left. So just think about the one on the right. -- To view, visit http://gerrit.cloudera.org:8080/16842 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic01db168ff2c6d6de33ee553a8175599f035d7a1 Gerrit-Change-Number: 16842 Gerrit-PatchSet: 18 Gerrit-Owner: liuyao Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: liuyao Gerrit-Comment-Date: Sat, 27 Feb 2021 02:31:25 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10550: Add External Frontend service port
John Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/17125 ) Change subject: IMPALA-10550: Add External Frontend service port .. Patch Set 5: For clarity - for now it'll be the responsibility of external frontends to ensure they work with impala and not vice-versa until we get more stable. We should take due care not to regress any of the current impala functionality and near term unit test what we can - such as not exposing ExecutePlannedStatement via hs2_port and so forth. -- To view, visit http://gerrit.cloudera.org:8080/17125 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Gerrit-Change-Number: 17125 Gerrit-PatchSet: 5 Gerrit-Owner: John Sherman Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Sat, 27 Feb 2021 01:29:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10492: Lower default MAX CNF EXPRS query option
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17132 ) Change subject: IMPALA-10492: Lower default MAX_CNF_EXPRS query option .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8258/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17132 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7ca3d0e094ac01c24a046c25d6a1b56bf134faa8 Gerrit-Change-Number: 17132 Gerrit-PatchSet: 1 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Sat, 27 Feb 2021 01:26:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10492: Lower default MAX CNF EXPRS query option
Hello Aman Sinha, David Rorke, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17132 to look at the new patch set (#2). Change subject: IMPALA-10492: Lower default MAX_CNF_EXPRS query option .. IMPALA-10492: Lower default MAX_CNF_EXPRS query option MAX_CNF_EXPRS was set to unlimited by default. The CNF rewrite can lead to significant frontend memory usage and eventually OutOfMemory for a complex query that contain many predicates. We need to lower the default value to avoid this memory problem while maintaining performance for our TPC-DS and TPC-H workloads. We investigate the maximum number of CFN expressions in TPC-DS and TPC-H by printing out the final value of 'numCnfExprs_' from ConvertToCNFRule.java to the query profile. We found 5 queries that applies CNF rewrite rules as follow: | Query | numCnfExprs_ | |---+--| | TPCDS-Q13 | 168 | | TPCDS-Q85 | 100 | | TPCDS-Q48 | 34 | | TPCH-Q19 | 124 | | TPCH-Q7 |3 | This patch lower the default value from unlimited to 200 based on the result above. Testing: - Manually verify that MAX_CNF_EXPRS 200 is enough for our TPC-DS and TPC-H worloads. - Pass core tests. Change-Id: I7ca3d0e094ac01c24a046c25d6a1b56bf134faa8 --- M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift 2 files changed, 2 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/32/17132/2 -- To view, visit http://gerrit.cloudera.org:8080/17132 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7ca3d0e094ac01c24a046c25d6a1b56bf134faa8 Gerrit-Change-Number: 17132 Gerrit-PatchSet: 2 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR] IMPALA-10492: Lower default MAX CNF EXPRS query option
Riza Suminto has uploaded this change for review. ( http://gerrit.cloudera.org:8080/17132 Change subject: IMPALA-10492: Lower default MAX_CNF_EXPRS query option .. IMPALA-10492: Lower default MAX_CNF_EXPRS query option MAX_CNF_EXPRS was set to unlimited by default. A complex query that container many predicates the CNF rewrite can lead to significant frontend memory usage and eventually OutOfMemory. We need to lower the default value to avoid the memory problem while maintaining performance for our TPC-DS and TPC-H workloads. We investigate the maximum number of CFN expressions in TPC-DS and TPC-H by printing out the final value of 'numCnfExprs_' from ConvertToCNFRule.java to the query profile. We found 5 queries that applies CNF rewrite rules as follow: | Query | numCnfExprs_ | |---+--| | TPCDS-Q13 | 168 | | TPCDS-Q85 | 100 | | TPCDS-Q48 | 34 | | TPCH-Q19 | 124 | | TPCH-Q7 |3 | This patch lower the default value from unlimited to 200 based on the result above. Testing: - Manually verify that MAX_CNF_EXPRS 200 is enough for our TPC-DS and TPC-H worloads. - Pass core tests. Change-Id: I7ca3d0e094ac01c24a046c25d6a1b56bf134faa8 --- M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift 2 files changed, 2 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/32/17132/1 -- To view, visit http://gerrit.cloudera.org:8080/17132 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I7ca3d0e094ac01c24a046c25d6a1b56bf134faa8 Gerrit-Change-Number: 17132 Gerrit-PatchSet: 1 Gerrit-Owner: Riza Suminto
[Impala-ASF-CR] IMPALA-10550: Add External Frontend service port
j...@cloudera.com has posted comments on this change. ( http://gerrit.cloudera.org:8080/17125 ) Change subject: IMPALA-10550: Add External Frontend service port .. Patch Set 5: > (2 comments) > > Are there any existing hs2 methods that it might make sense to > block for the "external frontend" server? eg. we might want to > return an error for ExecuteStatement() from it if the external > frontend will never need to call it. > > That would give us a little more safety in case people accidentally > expose this port to the outside world (of course, it would still be > possible for bad actors to use the ExecutePlannedStatement > interface, but its probably a lot harder to put together a valid > TExecRequest to use it than it is to put together a SQL string like > ExecuteStatement takes) > > Also out of curiosity - what's the long run testing plan here? Are > we going to have an actual external FE running in the minicluster > that can exercise this stuff? So the current implementation of external frontend does utilize the ExecuteStatement functionality (for things like COMPUTE STATS). I do agree with your assessment that it would be nice to reduce the surface area in the future. The long term plan would also likely include enabling similar protections that intra-impalad communication use between nodes (that prevent people connecting easily to the backend port and pretending to be a coordinator). One option I considered based on your comment was to add a 2nd flag that would be named something like: external_fe_allow_unsafe which defaulted to false and disallowed ExecuteStatement via the external_fe_port. So a user would have to enable external_fe_port AND set external_fe_allow_unsafe to true to be able to call ExecuteStatement. But if someone is enabling the external_fe_port - it is somewhat assumed they know what they are doing so I'm not 100% convinced this approach is worth it. I am open to suggestions (or if you like the idea of the 2nd flag). As for testing - I do believe once the various external FE commits land we should focus on: 1) auditing and shoring up what we can build unit tests around 2) And, yes, it is my understanding that we will eventually be including an external frontend in the minicluster for more end-to-end testing. Otherwise, we will need to mock up some sort of "send pre-made exec request" and "check response" test framework but I suspect that might be not fun to implement cleanly. -- To view, visit http://gerrit.cloudera.org:8080/17125 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Gerrit-Change-Number: 17125 Gerrit-PatchSet: 5 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Sat, 27 Feb 2021 01:00:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10550: Add External Frontend service port
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17125 ) Change subject: IMPALA-10550: Add External Frontend service port .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8257/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17125 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Gerrit-Change-Number: 17125 Gerrit-PatchSet: 5 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Sat, 27 Feb 2021 00:49:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10550: Add External Frontend service port
Hello Aman Sinha, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17125 to look at the new patch set (#5). Change subject: IMPALA-10550: Add External Frontend service port .. IMPALA-10550: Add External Frontend service port - If external_fe_port flag is >0, spins up a new HS2 compatible service port - start-impala-cluster.py starts default clusters with external_fe_port on 21150-21152 - Modify impalad_coordinator Dockerfile to expose external frontend port at 21150 - The intent of this commit is to separate external frontend connections from normal hs2 connections - This allows different security policy to be applied to each type of connection. The external_fe_port should be considered a privileged service and should only be exposed to an external frontend that does user authentication and does authorization checks on generated plans Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Reviewed-by: Aman Sinha --- M be/src/rpc/authentication.cc M be/src/rpc/authentication.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/service/impalad-main.cc M be/src/testutil/in-process-servers.cc M bin/start-impala-cluster.py M common/thrift/metrics.json M docker/impalad_coordinator/Dockerfile M tests/common/impala_cluster.py 10 files changed, 134 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/17125/5 -- To view, visit http://gerrit.cloudera.org:8080/17125 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Gerrit-Change-Number: 17125 Gerrit-PatchSet: 5 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-10547: Restore TPC-DS "reason" table missing from Kudu schema
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/17117 ) Change subject: IMPALA-10547: Restore TPC-DS "reason" table missing from Kudu schema .. IMPALA-10547: Restore TPC-DS "reason" table missing from Kudu schema Restore the missing "reason" table that was inadvertantly removed from the Kudu TPC-DS schema by a previous commit. Testing: - Manually run load-tpc-kudu.py and verify that it correctly creates the reason table. Manually run TPC-DS query 9 which references the reason table. Change-Id: Ic9c83ecbb8ca07c082f6407842ca67ef81194751 Reviewed-on: http://gerrit.cloudera.org:8080/17117 Reviewed-by: Impala Public Jenkins Reviewed-by: Grant Henke Tested-by: Impala Public Jenkins --- M testdata/datasets/tpcds/tpcds_kudu_template.sql 1 file changed, 12 insertions(+), 0 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified Grant Henke: Looks good to me, but someone else must approve -- To view, visit http://gerrit.cloudera.org:8080/17117 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ic9c83ecbb8ca07c082f6407842ca67ef81194751 Gerrit-Change-Number: 17117 Gerrit-PatchSet: 3 Gerrit-Owner: David Rorke Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-10547: Restore TPC-DS "reason" table missing from Kudu schema
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17117 ) Change subject: IMPALA-10547: Restore TPC-DS "reason" table missing from Kudu schema .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/17117 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic9c83ecbb8ca07c082f6407842ca67ef81194751 Gerrit-Change-Number: 17117 Gerrit-PatchSet: 2 Gerrit-Owner: David Rorke Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Fri, 26 Feb 2021 23:44:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10532 TestOverlapMinMaxFilters.test overlap min max filters seems flaky
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/17111 ) Change subject: IMPALA-10532 TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky .. IMPALA-10532 TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky This patch addresses the flakiness seen with a particular test within overlap_min_max_filters by allowing the sum of NumRuntimeFilteredPages to be greater than an expected value. Previously, such a sum can only be equal to the expected value and is not sufficient for various test conditions in which the scan of the parquet data files can start before the arrival of a runtime filter. The extension in test_result_verifier.py allows '>' and '<' condition to be expressed for aggregation(SUM, ), such as aggregation(SUM, NumRuntimeFilteredPages)> 80. Testing: - Ran TestOverlapMinMaxFilters. Change-Id: I93940a104bfb2d68cb1d41d7e303348190fd5972 Reviewed-on: http://gerrit.cloudera.org:8080/17111 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M testdata/workloads/functional-query/queries/QueryTest/overlap_min_max_filters.test M tests/common/test_result_verifier.py 2 files changed, 52 insertions(+), 13 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/17111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I93940a104bfb2d68cb1d41d7e303348190fd5972 Gerrit-Change-Number: 17111 Gerrit-PatchSet: 7 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-10532 TestOverlapMinMaxFilters.test overlap min max filters seems flaky
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17111 ) Change subject: IMPALA-10532 TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky .. Patch Set 6: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/17111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93940a104bfb2d68cb1d41d7e303348190fd5972 Gerrit-Change-Number: 17111 Gerrit-PatchSet: 6 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 26 Feb 2021 22:52:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10550: Add External Frontend service port
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/17125 ) Change subject: IMPALA-10550: Add External Frontend service port .. Patch Set 4: (2 comments) Are there any existing hs2 methods that it might make sense to block for the "external frontend" server? eg. we might want to return an error for ExecuteStatement() from it if the external frontend will never need to call it. That would give us a little more safety in case people accidentally expose this port to the outside world (of course, it would still be possible for bad actors to use the ExecutePlannedStatement interface, but its probably a lot harder to put together a valid TExecRequest to use it than it is to put together a SQL string like ExecuteStatement takes) Also out of curiosity - what's the long run testing plan here? Are we going to have an actual external FE running in the minicluster that can exercise this stuff? http://gerrit.cloudera.org:8080/#/c/17125/4/be/src/rpc/authentication.h File be/src/rpc/authentication.h: http://gerrit.cloudera.org:8080/#/c/17125/4/be/src/rpc/authentication.h@73 PS4, Line 73: AuthProvider* GetExternalFrontendAuthProvider(); Might mention that this is always a NoAuthProvider, or null if external_fe_port is 0 http://gerrit.cloudera.org:8080/#/c/17125/4/be/src/service/impala-server.cc File be/src/service/impala-server.cc: http://gerrit.cloudera.org:8080/#/c/17125/4/be/src/service/impala-server.cc@151 PS4, Line 151: "If 0 or less, the External Frontend server is not started."); You should mention that this server is never protected with authentication so users should be careful to not expose this port if they enable it. -- To view, visit http://gerrit.cloudera.org:8080/17125 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Gerrit-Change-Number: 17125 Gerrit-PatchSet: 4 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Fri, 26 Feb 2021 21:19:50 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10546: Add ImpalaServer interface to retrieve BackendConfig from impalad
Thomas Tauber-Marshall has posted comments on this change. (
http://gerrit.cloudera.org:8080/17116 )
Change subject: IMPALA-10546: Add ImpalaServer interface to retrieve
BackendConfig from impalad
..
Patch Set 9:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/17116/9/be/src/rpc/hs2-http-test.cc
File be/src/rpc/hs2-http-test.cc:
http://gerrit.cloudera.org:8080/#/c/17116/9/be/src/rpc/hs2-http-test.cc@53
PS9, Line 53: return_val
nit: we've used '_return' for all of the other functions here
http://gerrit.cloudera.org:8080/#/c/17116/9/be/src/service/impala-hs2-server.cc
File be/src/service/impala-hs2-server.cc:
http://gerrit.cloudera.org:8080/#/c/17116/9/be/src/service/impala-hs2-server.cc@1193
PS9, Line 1193: shared_ptr session;
I don't think the session is actually used for anything here, we're basically
just checking that it exists. Any reason to not just leave 'sessionHandle' out
of the request entirely and save ourselves some extra work?
http://gerrit.cloudera.org:8080/#/c/17116/9/be/src/util/backend-gflag-util.h
File be/src/util/backend-gflag-util.h:
http://gerrit.cloudera.org:8080/#/c/17116/9/be/src/util/backend-gflag-util.h@27
PS9, Line 27: /// Builds the TBackendGflags object to pass to JNI. This is used
to pass the gflag
: /// configs to the Frontend and the Catalog.
It would be cleaner to put this comment directly above the version of the
function that it really applies to, i.e. GetThriftBackendGflags
http://gerrit.cloudera.org:8080/#/c/17116/9/be/src/util/backend-gflag-util.h@29
PS9, Line 29: class TBackendGflags;
Not a big deal, but its pretty standard in Impala to put all the forward
declaration together at the top, i.e. in this case directly after the
"namespace impala {" line above
http://gerrit.cloudera.org:8080/#/c/17116/9/be/src/util/backend-gflag-util.h@31
PS9, Line 31: GetThriftBackendGflags
Might be nice to rename this, eg. to GetThriftBackendGFlagsForJNI, since the
difference between GetThriftBackendGflags and PopulateThriftBackendGflags isn't
very clear
http://gerrit.cloudera.org:8080/#/c/17116/9/common/thrift/ImpalaService.thrift
File common/thrift/ImpalaService.thrift:
http://gerrit.cloudera.org:8080/#/c/17116/9/common/thrift/ImpalaService.thrift@857
PS9, Line 857: // Returns the current TBackendGflags
Maybe mention that this is only supported for the "external fe" server
http://gerrit.cloudera.org:8080/#/c/17116/9/tests/hs2/test_hs2.py
File tests/hs2/test_hs2.py:
http://gerrit.cloudera.org:8080/#/c/17116/9/tests/hs2/test_hs2.py@738
PS9, Line 738: hs2_client
I'm not sure how this test would work, since I would assune that 'hs2_client'
here would point at the normal hs2 port, not the "external frontend" port, so
shouldn't we hit the "Unsupported operation" error.
Of course, in addition to the case where it works that it tested here, it would
be nice to include a test that checks that it gets the error in the cases where
it should.
--
To view, visit http://gerrit.cloudera.org:8080/17116
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I14a3cee29f1fc91f4431b7ea89053bb3fbfa5e69
Gerrit-Change-Number: 17116
Gerrit-PatchSet: 9
Gerrit-Owner: Kurt Deschler
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Kurt Deschler
Gerrit-Reviewer: Thomas Tauber-Marshall
Gerrit-Comment-Date: Fri, 26 Feb 2021 20:41:17 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10547: Restore TPC-DS "reason" table missing from Kudu schema
Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/17117 ) Change subject: IMPALA-10547: Restore TPC-DS "reason" table missing from Kudu schema .. Patch Set 2: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/17117 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic9c83ecbb8ca07c082f6407842ca67ef81194751 Gerrit-Change-Number: 17117 Gerrit-PatchSet: 2 Gerrit-Owner: David Rorke Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Fri, 26 Feb 2021 20:10:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10546: Add ImpalaServer interface to retrieve BackendConfig from impalad
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17116 ) Change subject: IMPALA-10546: Add ImpalaServer interface to retrieve BackendConfig from impalad .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8255/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17116 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I14a3cee29f1fc91f4431b7ea89053bb3fbfa5e69 Gerrit-Change-Number: 17116 Gerrit-PatchSet: 9 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Fri, 26 Feb 2021 19:56:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10524: Changes to HdfsPartition for third party extensions.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17092 ) Change subject: IMPALA-10524: Changes to HdfsPartition for third party extensions. .. Patch Set 7: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/8256/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/17092 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5a792642f27228118ac8f2e8ef98e8ba7aee4a46 Gerrit-Change-Number: 17092 Gerrit-PatchSet: 7 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Steve Carlin Gerrit-Comment-Date: Fri, 26 Feb 2021 19:49:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10538: [DOCS] Document the newly added argument
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17131 ) Change subject: IMPALA-10538: [DOCS] Document the newly added argument .. Patch Set 1: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/622/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/17131 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iec8007b79afac59cdfb3984bb111806213c21c77 Gerrit-Change-Number: 17131 Gerrit-PatchSet: 1 Gerrit-Owner: Shajini Thayasingh Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 26 Feb 2021 19:47:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10538: [DOCS] Document the newly added argument
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17131 ) Change subject: IMPALA-10538: [DOCS] Document the newly added argument .. Patch Set 1: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/622/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/17131 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iec8007b79afac59cdfb3984bb111806213c21c77 Gerrit-Change-Number: 17131 Gerrit-PatchSet: 1 Gerrit-Owner: Shajini Thayasingh Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 26 Feb 2021 19:40:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10538: [DOCS] Document the newly added argument
Shajini Thayasingh has uploaded this change for review. ( http://gerrit.cloudera.org:8080/17131 Change subject: IMPALA-10538: [DOCS] Document the newly added argument .. IMPALA-10538: [DOCS] Document the newly added argument discussed the newly added scale argument of ndv function elaborated how this argument maps to a precision used by the HLL algorithm Change-Id: Iec8007b79afac59cdfb3984bb111806213c21c77 --- M docs/topics/impala_ndv.xml 1 file changed, 24 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/17131/1 -- To view, visit http://gerrit.cloudera.org:8080/17131 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Iec8007b79afac59cdfb3984bb111806213c21c77 Gerrit-Change-Number: 17131 Gerrit-PatchSet: 1 Gerrit-Owner: Shajini Thayasingh
[Impala-ASF-CR] IMPALA-10524: Changes to HdfsPartition for third party extensions.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17092 ) Change subject: IMPALA-10524: Changes to HdfsPartition for third party extensions. .. Patch Set 7: (1 comment) http://gerrit.cloudera.org:8080/#/c/17092/7/fe/src/main/java/org/apache/impala/analysis/StatementBase.java File fe/src/main/java/org/apache/impala/analysis/StatementBase.java: http://gerrit.cloudera.org:8080/#/c/17092/7/fe/src/main/java/org/apache/impala/analysis/StatementBase.java@203 PS7, Line 203: public static Expr checkTypeCompatibility(String dstTableName, Column dstCol, Expr srcExpr, line too long (93 > 90) -- To view, visit http://gerrit.cloudera.org:8080/17092 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5a792642f27228118ac8f2e8ef98e8ba7aee4a46 Gerrit-Change-Number: 17092 Gerrit-PatchSet: 7 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Steve Carlin Gerrit-Comment-Date: Fri, 26 Feb 2021 19:38:51 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10524: Changes to HdfsPartition for third party extensions.
Steve Carlin has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/17092 ) Change subject: IMPALA-10524: Changes to HdfsPartition for third party extensions. .. IMPALA-10524: Changes to HdfsPartition for third party extensions. Some changes are needed to HdfsPartition and other related classes to allow for third party extensions. These changes include: - A protected constructor which will allow a subclass to instantiate HdfsPartition using its own Builder. - Various changes of permissions to methods and variables to allow third party extension visibility. - Creation of the getHostIndex() method to allow the subclass to override how the hostIndexes are retrieved. - Added a new default method "getFileSystem()" to FeFsPartition which will allow the third party extension to override how the filesystem is obtained from the partition object. Change-Id: I5a792642f27228118ac8f2e8ef98e8ba7aee4a46 --- M be/src/rpc/hs2-http-test.cc M fe/src/main/java/org/apache/impala/analysis/AnalyticWindow.java M fe/src/main/java/org/apache/impala/analysis/StatementBase.java M fe/src/main/java/org/apache/impala/catalog/FeFsPartition.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartitionLocationCompressor.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalFsPartition.java M fe/src/main/java/org/apache/impala/planner/CardinalityCheckNode.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/planner/Planner.java 11 files changed, 78 insertions(+), 19 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/92/17092/7 -- To view, visit http://gerrit.cloudera.org:8080/17092 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5a792642f27228118ac8f2e8ef98e8ba7aee4a46 Gerrit-Change-Number: 17092 Gerrit-PatchSet: 7 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Steve Carlin
[Impala-ASF-CR] IMPALA-10546: Add ImpalaServer interface to retrieve BackendConfig from impalad
Hello Thomas Tauber-Marshall, Joe McDonnell, j...@cloudera.com, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17116 to look at the new patch set (#9). Change subject: IMPALA-10546: Add ImpalaServer interface to retrieve BackendConfig from impalad .. IMPALA-10546: Add ImpalaServer interface to retrieve BackendConfig from impalad This patch add a new interface ImpalaServer::GetBackendConfig() that returns the current TBackendGflags from impalad. Testing: Called new interface from external frontend. Verified that TBackendGflags were populated correctly. Reviewed-by: John Sherman Change-Id: I14a3cee29f1fc91f4431b7ea89053bb3fbfa5e69 --- M be/src/rpc/hs2-http-test.cc M be/src/service/impala-hs2-server.cc M be/src/service/impala-server.h M be/src/util/backend-gflag-util.cc M be/src/util/backend-gflag-util.h M common/thrift/ImpalaService.thrift M tests/hs2/test_hs2.py 7 files changed, 73 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/16/17116/9 -- To view, visit http://gerrit.cloudera.org:8080/17116 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I14a3cee29f1fc91f4431b7ea89053bb3fbfa5e69 Gerrit-Change-Number: 17116 Gerrit-PatchSet: 9 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-10524: Changes to HdfsPartition for third party extensions.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17092 ) Change subject: IMPALA-10524: Changes to HdfsPartition for third party extensions. .. Patch Set 6: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/8254/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/17092 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5a792642f27228118ac8f2e8ef98e8ba7aee4a46 Gerrit-Change-Number: 17092 Gerrit-PatchSet: 6 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Steve Carlin Gerrit-Comment-Date: Fri, 26 Feb 2021 18:28:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10524: Changes to HdfsPartition for third party extensions.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17092 ) Change subject: IMPALA-10524: Changes to HdfsPartition for third party extensions. .. Patch Set 6: (1 comment) http://gerrit.cloudera.org:8080/#/c/17092/6/fe/src/main/java/org/apache/impala/analysis/StatementBase.java File fe/src/main/java/org/apache/impala/analysis/StatementBase.java: http://gerrit.cloudera.org:8080/#/c/17092/6/fe/src/main/java/org/apache/impala/analysis/StatementBase.java@203 PS6, Line 203: public static Expr checkTypeCompatibility(String dstTableName, Column dstCol, Expr srcExpr, line too long (93 > 90) -- To view, visit http://gerrit.cloudera.org:8080/17092 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5a792642f27228118ac8f2e8ef98e8ba7aee4a46 Gerrit-Change-Number: 17092 Gerrit-PatchSet: 6 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Steve Carlin Gerrit-Comment-Date: Fri, 26 Feb 2021 18:18:30 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10524: Changes to HdfsPartition for third party extensions.
Steve Carlin has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/17092 ) Change subject: IMPALA-10524: Changes to HdfsPartition for third party extensions. .. IMPALA-10524: Changes to HdfsPartition for third party extensions. Some changes are needed to HdfsPartition and other related classes to allow for third party extensions. These changes include: - A protected constructor which will allow a subclass to instantiate HdfsPartition using its own Builder. - Various changes of permissions to methods and variables to allow third party extension visibility. - Creation of the getHostIndex() method to allow the subclass to override how the hostIndexes are retrieved. - Added a new default method "getFileSystem()" to FeFsPartition which will allow the third party extension to override how the filesystem is obtained from the partition object. Change-Id: I5a792642f27228118ac8f2e8ef98e8ba7aee4a46 --- M be/src/rpc/hs2-http-test.cc M fe/src/main/java/org/apache/impala/analysis/AnalyticWindow.java M fe/src/main/java/org/apache/impala/analysis/StatementBase.java M fe/src/main/java/org/apache/impala/catalog/FeFsPartition.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartitionLocationCompressor.java M fe/src/main/java/org/apache/impala/catalog/local/LocalFsPartition.java M fe/src/main/java/org/apache/impala/planner/CardinalityCheckNode.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/planner/Planner.java 10 files changed, 77 insertions(+), 19 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/92/17092/6 -- To view, visit http://gerrit.cloudera.org:8080/17092 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5a792642f27228118ac8f2e8ef98e8ba7aee4a46 Gerrit-Change-Number: 17092 Gerrit-PatchSet: 6 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Steve Carlin
[Impala-ASF-CR] IMPALA-10550: Add External Frontend service port
j...@cloudera.com has posted comments on this change. ( http://gerrit.cloudera.org:8080/17125 ) Change subject: IMPALA-10550: Add External Frontend service port .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/17125/3/be/src/service/impala-server.cc File be/src/service/impala-server.cc: http://gerrit.cloudera.org:8080/#/c/17125/3/be/src/service/impala-server.cc@2877 PS3, Line 2877: << external_fe_server_->port(); > tab used for whitespace I'll fix with the review comments. -- To view, visit http://gerrit.cloudera.org:8080/17125 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Gerrit-Change-Number: 17125 Gerrit-PatchSet: 3 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Fri, 26 Feb 2021 18:08:07 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10547: Restore TPC-DS "reason" table missing from Kudu schema
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17117 ) Change subject: IMPALA-10547: Restore TPC-DS "reason" table missing from Kudu schema .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6924/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/17117 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic9c83ecbb8ca07c082f6407842ca67ef81194751 Gerrit-Change-Number: 17117 Gerrit-PatchSet: 2 Gerrit-Owner: David Rorke Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Fri, 26 Feb 2021 18:05:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10547: Restore TPC-DS "reason" table missing from Kudu schema
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17117 ) Change subject: IMPALA-10547: Restore TPC-DS "reason" table missing from Kudu schema .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/17117 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic9c83ecbb8ca07c082f6407842ca67ef81194751 Gerrit-Change-Number: 17117 Gerrit-PatchSet: 2 Gerrit-Owner: David Rorke Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Fri, 26 Feb 2021 18:05:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10222: CREATE TABLE AS SELECT for Iceberg tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17130 ) Change subject: IMPALA-10222: CREATE TABLE AS SELECT for Iceberg tables .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8253/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17130 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I81d2084e401b9fa74d5ad161b51fd3e2aa3fcc67 Gerrit-Change-Number: 17130 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Fri, 26 Feb 2021 17:26:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10222: CREATE TABLE AS SELECT for Iceberg tables
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/17130 )
Change subject: IMPALA-10222: CREATE TABLE AS SELECT for Iceberg tables
..
Patch Set 1:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/17130/1/fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
File fe/src/main/java/org/apache/impala/analysis/InsertStmt.java:
http://gerrit.cloudera.org:8080/#/c/17130/1/fe/src/main/java/org/apache/impala/analysis/InsertStmt.java@502
PS1, Line 502: IcebergPartitionSpec partSpec =
((FeIcebergTable)table_).getDefaultPartitionSpec();
line too long (93 > 90)
http://gerrit.cloudera.org:8080/#/c/17130/1/fe/src/main/java/org/apache/impala/analysis/InsertStmt.java@504
PS1, Line 504: throw new AnalysisException("PARTITION clause is
only valid for INSERT into " +
line too long (91 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/17130
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I81d2084e401b9fa74d5ad161b51fd3e2aa3fcc67
Gerrit-Change-Number: 17130
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Gabor Kaszab
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: wangsheng
Gerrit-Comment-Date: Fri, 26 Feb 2021 17:08:17 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10532 TestOverlapMinMaxFilters.test overlap min max filters seems flaky
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17111 ) Change subject: IMPALA-10532 TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/17111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93940a104bfb2d68cb1d41d7e303348190fd5972 Gerrit-Change-Number: 17111 Gerrit-PatchSet: 6 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 26 Feb 2021 17:07:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10532 TestOverlapMinMaxFilters.test overlap min max filters seems flaky
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17111 ) Change subject: IMPALA-10532 TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6923/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/17111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93940a104bfb2d68cb1d41d7e303348190fd5972 Gerrit-Change-Number: 17111 Gerrit-PatchSet: 6 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 26 Feb 2021 17:07:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10222: CREATE TABLE AS SELECT for Iceberg tables
Zoltan Borok-Nagy has uploaded this change for review. ( http://gerrit.cloudera.org:8080/17130 Change subject: IMPALA-10222: CREATE TABLE AS SELECT for Iceberg tables .. IMPALA-10222: CREATE TABLE AS SELECT for Iceberg tables This patch adds support for CREATE TABLE AS SELECT statements for Iceberg tables. CTAS statements work like the following in Impala: 1. Analysis of the whole CTAS statement 2. Divide CTAS to CREATE stmt and INSERT stmt 3. Create temporary in-memory target table from the CREATE stmt 4. Analyse the INSERT statement by using the temporary target table 5. If everything is OK so far, create the target table 6. Execute the INSERT query For Iceberg tables the non-trivial thing was to create the temporary target table without actually creating it via Iceberg API. I've created a new class 'IcebergCtasTarget' that mimics an FeIceberg table. It can be used with catalog V1 and V2 as well. Testing * e2e CTAS tests in iceberg-ctas.test * SHOW CREATE TABLE stmts in show-create-table.test Change-Id: I81d2084e401b9fa74d5ad161b51fd3e2aa3fcc67 --- M be/src/exec/hdfs-table-sink.cc M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java A fe/src/main/java/org/apache/impala/catalog/CtasTargetTable.java M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java A fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergCtasTarget.java M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java M fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/IcebergSchemaConverter.java M fe/src/main/java/org/apache/impala/util/IcebergUtil.java A testdata/workloads/functional-query/queries/QueryTest/iceberg-ctas.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test M testdata/workloads/functional-query/queries/QueryTest/show-create-table.test M tests/metadata/test_show_create_table.py M tests/query_test/test_iceberg.py 18 files changed, 698 insertions(+), 44 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/30/17130/1 -- To view, visit http://gerrit.cloudera.org:8080/17130 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I81d2084e401b9fa74d5ad161b51fd3e2aa3fcc67 Gerrit-Change-Number: 17130 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17104 ) Change subject: IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8252/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa Gerrit-Change-Number: 17104 Gerrit-PatchSet: 7 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Fri, 26 Feb 2021 15:21:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements
Hello Thomas Tauber-Marshall, Joe McDonnell, j...@cloudera.com, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17104 to look at the new patch set (#7). Change subject: IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements .. IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements The ExecutePlannedStatement interface allows an externally supplied TExecRequest to be executed by impalad. The TExecRequest must be fully populated and will be sent directly to the backend for execution. The following fields in the TExecRequest are updated by the coordinator: - Hostname - KRPC address - Local Timezone In order to add the interface to ImpalaInternalService.thrift, several of the thrift classes were moved to Query.thrift to avoid a circular dependency with Frontend.thrift. Added functionality to format and dump TExecRequest structures to path specified in debug flag dump_exec_request_path. A start timestamp field has been added to TExecRequest to represent the interval in the query profile between when the request was sent by the external frontend and handled by the backend. A local timestamp field has been added to the Ping result struct to return the current backend timestamp. This is used by the external to frontend to populate the start timestamp. Also included is a change to avoid generating silent AnalysisExceptions during table resolution. Tested with TExecRequest structures populated by external frontend. Local timezone change tested withe INT64 TIMESTAMP datatype Reviewed-by: John Sherman Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa --- M be/generated-sources/gen-cpp/CMakeLists.txt M be/src/rpc/hs2-http-test.cc M be/src/runtime/debug-options.h M be/src/runtime/query-driver.cc M be/src/runtime/query-driver.h M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/impala-beeswax-server.cc M be/src/service/impala-hs2-server.cc M be/src/service/impala-server.cc M be/src/service/impala-server.h M common/thrift/CMakeLists.txt M common/thrift/Frontend.thrift M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift A common/thrift/Query.thrift M fe/pom.xml M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/PrivilegeSpec.java M fe/src/main/java/org/apache/impala/analysis/ResetMetadataStmt.java M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java 21 files changed, 986 insertions(+), 755 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/17104/7 -- To view, visit http://gerrit.cloudera.org:8080/17104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa Gerrit-Change-Number: 17104 Gerrit-PatchSet: 7 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-10549: Register transactions from external frontend DML
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17122 ) Change subject: IMPALA-10549: Register transactions from external frontend DML .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8251/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17122 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia8863b8d9d281a5d164f10de9c5ee52cf3be63db Gerrit-Change-Number: 17122 Gerrit-PatchSet: 5 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Fri, 26 Feb 2021 14:54:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17104 ) Change subject: IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8250/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa Gerrit-Change-Number: 17104 Gerrit-PatchSet: 6 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Fri, 26 Feb 2021 14:54:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10549: Register transactions from external frontend DML
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/17122 )
Change subject: IMPALA-10549: Register transactions from external frontend DML
..
Patch Set 5:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/17122/5/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
http://gerrit.cloudera.org:8080/#/c/17122/5/be/src/service/impala-server.cc@1212
PS5, Line 1212:
RETURN_IF_ERROR(exec_env_->frontend()->addTransaction(exec_request->query_exec_request.query_ctx));
line too long (107 > 90)
http://gerrit.cloudera.org:8080/#/c/17122/5/fe/src/main/java/org/apache/impala/service/JniFrontend.java
File fe/src/main/java/org/apache/impala/service/JniFrontend.java:
http://gerrit.cloudera.org:8080/#/c/17122/5/fe/src/main/java/org/apache/impala/service/JniFrontend.java@695
PS5, Line 695: public void addTransaction(byte[] thriftQueryContext) throws
TransactionException, ImpalaException {
line too long (102 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/17122
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia8863b8d9d281a5d164f10de9c5ee52cf3be63db
Gerrit-Change-Number: 17122
Gerrit-PatchSet: 5
Gerrit-Owner: Kurt Deschler
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Kurt Deschler
Gerrit-Reviewer: Thomas Tauber-Marshall
Gerrit-Comment-Date: Fri, 26 Feb 2021 14:35:36 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17104 ) Change subject: IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements .. Patch Set 6: (1 comment) http://gerrit.cloudera.org:8080/#/c/17104/6/be/src/service/client-request-state.h File be/src/service/client-request-state.h: http://gerrit.cloudera.org:8080/#/c/17104/6/be/src/service/client-request-state.h@245 PS6, Line 245: /// This helper method updates the local time zone of client's TExecRequest. The intended line too long (91 > 90) -- To view, visit http://gerrit.cloudera.org:8080/17104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa Gerrit-Change-Number: 17104 Gerrit-PatchSet: 6 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Fri, 26 Feb 2021 14:35:00 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements
Hello Thomas Tauber-Marshall, Joe McDonnell, j...@cloudera.com, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17104 to look at the new patch set (#6). Change subject: IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements .. IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements The ExecutePlannedStatement interface allows an externally supplied TExecRequest to be executed by impalad. The TExecRequest must be fully populated and will be sent directly to the backend for execution. The following fields in the TExecRequest are updated by the coordinator: - Hostname - KRPC address - Local Timezone In order to add the interface to ImpalaInternalService.thrift, several of the thrift classes were moved to Query.thrift to avoid a circular dependency with Frontend.thrift. Added functionality to format and dump TExecRequest structures to path specified in debug flag dump_exec_request_path. A start timestamp field has been added to TExecRequest to represent the interval in the query profile between when the request was sent by the external frontend and handled by the backend. A local timestamp field has been added to the Ping result struct to return the current backend timestamp. This is used by the external to frontend to populate the start timestamp. Also included is a change to avoid generating silent AnalysisExceptions during table resolution. Tested with TExecRequest structures populated by external frontend. Local timezone change tested withe INT64 TIMESTAMP datatype Reviewed-by: John Sherman Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa --- M be/generated-sources/gen-cpp/CMakeLists.txt M be/src/rpc/hs2-http-test.cc M be/src/runtime/debug-options.h M be/src/runtime/query-driver.cc M be/src/runtime/query-driver.h M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/impala-beeswax-server.cc M be/src/service/impala-hs2-server.cc M be/src/service/impala-server.cc M be/src/service/impala-server.h M common/thrift/CMakeLists.txt M common/thrift/Frontend.thrift M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift A common/thrift/Query.thrift M fe/pom.xml M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/PrivilegeSpec.java M fe/src/main/java/org/apache/impala/analysis/ResetMetadataStmt.java M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java 21 files changed, 985 insertions(+), 755 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/17104/6 -- To view, visit http://gerrit.cloudera.org:8080/17104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa Gerrit-Change-Number: 17104 Gerrit-PatchSet: 6 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-10549: Register transactions from external frontend DML
Hello Thomas Tauber-Marshall, Joe McDonnell, j...@cloudera.com, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17122 to look at the new patch set (#5). Change subject: IMPALA-10549: Register transactions from external frontend DML .. IMPALA-10549: Register transactions from external frontend DML This change registers transactions that were started by an external frontend so that coordinator keepalive can track them properly. Testing: manually tested using DMLs from external frontend Reviewed-by: John Sherman Change-Id: Ia8863b8d9d281a5d164f10de9c5ee52cf3be63db --- M be/src/service/frontend.cc M be/src/service/frontend.h M be/src/service/impala-server.cc M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java 5 files changed, 37 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/22/17122/5 -- To view, visit http://gerrit.cloudera.org:8080/17122 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia8863b8d9d281a5d164f10de9c5ee52cf3be63db Gerrit-Change-Number: 17122 Gerrit-PatchSet: 5 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-10377: Improve the accuracy of resource estimation
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/16842 ) Change subject: IMPALA-10377: Improve the accuracy of resource estimation .. Patch Set 18: Code-Review+1 (1 comment) Thanks for applying the changes. The change looks good to me. Could you please resolve the merge conflict? http://gerrit.cloudera.org:8080/#/c/16842/18/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java File fe/src/main/java/org/apache/impala/planner/HashJoinNode.java: http://gerrit.cloudera.org:8080/#/c/16842/18/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@238 PS18, Line 238: ndv of the right child column nit: probably 'NDVs of the columns of the right (build) side?' -- To view, visit http://gerrit.cloudera.org:8080/16842 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic01db168ff2c6d6de33ee553a8175599f035d7a1 Gerrit-Change-Number: 16842 Gerrit-PatchSet: 18 Gerrit-Owner: liuyao Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: liuyao Gerrit-Comment-Date: Fri, 26 Feb 2021 13:40:25 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [WIP] IMPALA-7712: Support Google Cloud Storage
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17121 ) Change subject: [WIP] IMPALA-7712: Support Google Cloud Storage .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8249/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17121 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia91ec956de3b620cccf6a1244b56b7da7a45b32b Gerrit-Change-Number: 17121 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 26 Feb 2021 13:30:55 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-7712: Support Google Cloud Storage
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17121 ) Change subject: [WIP] IMPALA-7712: Support Google Cloud Storage .. Patch Set 2: (2 comments) http://gerrit.cloudera.org:8080/#/c/17121/2/tests/custom_cluster/test_hive_text_codec_interop.py File tests/custom_cluster/test_hive_text_codec_interop.py: http://gerrit.cloudera.org:8080/#/c/17121/2/tests/custom_cluster/test_hive_text_codec_interop.py@24 PS2, Line 24: from tests.common.skip import SkipIfS3, SkipGCS flake8: F401 'tests.common.skip.SkipGCS' imported but unused http://gerrit.cloudera.org:8080/#/c/17121/2/tests/custom_cluster/test_hive_text_codec_interop.py@55 PS2, Line 55: S flake8: F821 undefined name 'SkipIfGCS' -- To view, visit http://gerrit.cloudera.org:8080/17121 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia91ec956de3b620cccf6a1244b56b7da7a45b32b Gerrit-Change-Number: 17121 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 26 Feb 2021 13:12:14 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [WIP] IMPALA-7712: Support Google Cloud Storage
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17121 to look at the new patch set (#2). Change subject: [WIP] IMPALA-7712: Support Google Cloud Storage .. [WIP] IMPALA-7712: Support Google Cloud Storage This patch adds support for GCS(Google Cloud Storage). TODO: fix hanging when loading table/partition on nonexisting location (e.g. test_create_alter_bulk_partition) TODO: fix crash in spilling when default fs is GCS. (e.g. test_queries.py::TestQueries::test_analytic_fns) TODO: Skip more tests that are skiped on non-hdfs storage. Test steps: - Compile and create test data on a GCE instance. - Upload test data to a GCS bucket. - Modify the filesystem prefix of all locations in HMS DB to point to the GCS bucket. Remove some hdfs caching params. - TODO: Run CORE tests. Change-Id: Ia91ec956de3b620cccf6a1244b56b7da7a45b32b --- M be/src/exec/hdfs-table-sink.cc M be/src/runtime/io/disk-io-mgr-test.cc M be/src/runtime/io/disk-io-mgr.cc M be/src/runtime/io/disk-io-mgr.h M be/src/util/hdfs-util.cc M be/src/util/hdfs-util.h M bin/impala-config.sh M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M java/executor-deps/pom.xml M java/pom.xml M testdata/bin/load-test-warehouse-snapshot.sh M testdata/bin/run-all.sh M tests/authorization/test_ranger.py M tests/common/impala_test_suite.py M tests/common/skip.py M tests/custom_cluster/test_event_processing.py M tests/custom_cluster/test_hive_parquet_codec_interop.py M tests/custom_cluster/test_hive_text_codec_interop.py M tests/custom_cluster/test_local_catalog.py M tests/custom_cluster/test_metadata_replicas.py M tests/custom_cluster/test_parquet_max_page_header.py M tests/custom_cluster/test_permanent_udfs.py M tests/metadata/test_compute_stats.py M tests/metadata/test_ddl.py M tests/metadata/test_hms_integration.py M tests/metadata/test_metadata_query_statements.py M tests/metadata/test_partition_metadata.py M tests/metadata/test_refresh_partition.py M tests/metadata/test_reset_metadata.py M tests/metadata/test_views_compatibility.py M tests/query_test/test_acid.py M tests/query_test/test_hbase_queries.py M tests/query_test/test_insert_parquet.py M tests/query_test/test_nested_types.py M tests/query_test/test_partitioning.py M tests/query_test/test_scanners.py M tests/stress/test_acid_stress.py M tests/util/filesystem_utils.py 39 files changed, 187 insertions(+), 27 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/21/17121/2 -- To view, visit http://gerrit.cloudera.org:8080/17121 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia91ec956de3b620cccf6a1244b56b7da7a45b32b Gerrit-Change-Number: 17121 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins
