[Impala-ASF-CR] IMPALA-7002: Throw AuthorizationException when user accesses non-existent table/database in CTE without required privileges.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15123 ) Change subject: IMPALA-7002: Throw AuthorizationException when user accesses non-existent table/database in CTE without required privileges. .. IMPALA-7002: Throw AuthorizationException when user accesses non-existent table/database in CTE without required privileges. Currently if a user without required privileges tries to access a non-existent database or table, then impala returns an analysis exception instead of authorization exception. This happens because during analysis of the with clause, the authorization request does not get registered due to analysis exception being thrown before it. This patch makes sure that those requests get registered regardless. Testing: - Manual test: - ran CTE with non-existent database/table in impala-shell without required privilege, verified that it results in AuthorizationException. - ran CTE with non-existent database/table in impala-shell with the required privilege, verified that it results in AnalysisException. - Added CTE test cases for non-existent database/table/column in AuthorizationStmtTest. - Passed all FE tests. - Passed all core tests. Change-Id: Ia6b657a7147a136198a9a97a679c9131ee814577 Reviewed-on: http://gerrit.cloudera.org:8080/15123 Reviewed-by: Bikramjeet Vig Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/analysis/WithClause.java M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java 2 files changed, 41 insertions(+), 16 deletions(-) Approvals: Bikramjeet Vig: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/15123 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ia6b657a7147a136198a9a97a679c9131ee814577 Gerrit-Change-Number: 15123 Gerrit-PatchSet: 14 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-7002: Throw AuthorizationException when user accesses non-existent table/database in CTE without required privileges.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15123 ) Change subject: IMPALA-7002: Throw AuthorizationException when user accesses non-existent table/database in CTE without required privileges. .. Patch Set 13: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/15123 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia6b657a7147a136198a9a97a679c9131ee814577 Gerrit-Change-Number: 15123 Gerrit-PatchSet: 13 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 13 Feb 2020 03:10:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8870: Bump up guava version from 14.0.1 to 28.1-jre
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15214 ) Change subject: IMPALA-8870: Bump up guava version from 14.0.1 to 28.1-jre .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5215/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15214 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9690a926953a8d3c3872277680b4be0551546c68 Gerrit-Change-Number: 15214 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 13 Feb 2020 02:38:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8870: Bump up guava version from 14.0.1 to 28.1-jre
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15214 ) Change subject: IMPALA-8870: Bump up guava version from 14.0.1 to 28.1-jre .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/15214/1/fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java File fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java: http://gerrit.cloudera.org:8080/#/c/15214/1/fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java@714 PS1, Line 714: profile.addToCounter(STATS_FETCH_TIME, TUnit.TIME_MS, stopwatch.elapsed(MILLISECONDS)); line too long (91 > 90) -- To view, visit http://gerrit.cloudera.org:8080/15214 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9690a926953a8d3c3872277680b4be0551546c68 Gerrit-Change-Number: 15214 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 13 Feb 2020 01:52:18 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8870: Bump up guava version from 14.0.1 to 28.1-jre
Fang-Yu Rao has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15214 Change subject: IMPALA-8870: Bump up guava version from 14.0.1 to 28.1-jre .. IMPALA-8870: Bump up guava version from 14.0.1 to 28.1-jre This patch bumps up the guava version from 14.0.1 to 28.1-jre. Due to some changes in guava's API's, we modify the corresponding call sites accordingly. Change-Id: I9690a926953a8d3c3872277680b4be0551546c68 --- M fe/src/main/java/org/apache/impala/analysis/AggregateInfo.java M fe/src/main/java/org/apache/impala/analysis/AggregateInfoBase.java M fe/src/main/java/org/apache/impala/analysis/AnalyticExpr.java M fe/src/main/java/org/apache/impala/analysis/AnalyticInfo.java M fe/src/main/java/org/apache/impala/analysis/ArithmeticExpr.java M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java M fe/src/main/java/org/apache/impala/analysis/BoolLiteral.java M fe/src/main/java/org/apache/impala/analysis/CaseExpr.java M fe/src/main/java/org/apache/impala/analysis/CastExpr.java M fe/src/main/java/org/apache/impala/analysis/ColumnLineageGraph.java M fe/src/main/java/org/apache/impala/analysis/CompoundPredicate.java M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java M fe/src/main/java/org/apache/impala/analysis/Expr.java M fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java M fe/src/main/java/org/apache/impala/analysis/IsNullPredicate.java M fe/src/main/java/org/apache/impala/analysis/NullLiteral.java M fe/src/main/java/org/apache/impala/analysis/NumericLiteral.java M fe/src/main/java/org/apache/impala/analysis/Path.java M fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java M fe/src/main/java/org/apache/impala/analysis/SlotRef.java M fe/src/main/java/org/apache/impala/analysis/StringLiteral.java M fe/src/main/java/org/apache/impala/analysis/TupleDescriptor.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/Column.java M fe/src/main/java/org/apache/impala/catalog/ColumnStats.java M fe/src/main/java/org/apache/impala/catalog/DataSource.java M fe/src/main/java/org/apache/impala/catalog/FileMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/ParallelFileMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/TableLoader.java M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java M fe/src/main/java/org/apache/impala/planner/AnalyticEvalNode.java M fe/src/main/java/org/apache/impala/planner/DataPartition.java M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java M fe/src/main/java/org/apache/impala/planner/HBaseScanNode.java M fe/src/main/java/org/apache/impala/planner/HashJoinNode.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/planner/NestedLoopJoinNode.java M fe/src/main/java/org/apache/impala/planner/ScanNode.java M fe/src/main/java/org/apache/impala/planner/SortNode.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/util/JvmPauseMonitor.java M fe/src/test/java/org/apache/impala/catalog/local/CatalogdMetaProviderTest.java M impala-parent/pom.xml 46 files changed, 112 insertions(+), 92 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/14/15214/1 -- To view, visit http://gerrit.cloudera.org:8080/15214 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I9690a926953a8d3c3872277680b4be0551546c68 Gerrit-Change-Number: 15214 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-9256: Refactor constraint information into a separate class.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15213 ) Change subject: IMPALA-9256: Refactor constraint information into a separate class. .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5214/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15213 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3f1c441c24df84d2d0791ffe94dff60d039a3341 Gerrit-Change-Number: 15213 Gerrit-PatchSet: 1 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 13 Feb 2020 01:01:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9304: Support starting Hive with Ranger in minicluster
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15189 ) Change subject: IMPALA-9304: Support starting Hive with Ranger in minicluster .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15189 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I01e3a195b00a98388244a922a1a79e65146cec42 Gerrit-Change-Number: 15189 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 13 Feb 2020 00:28:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9304: Support starting Hive with Ranger in minicluster
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15189 ) Change subject: IMPALA-9304: Support starting Hive with Ranger in minicluster .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5328/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15189 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I01e3a195b00a98388244a922a1a79e65146cec42 Gerrit-Change-Number: 15189 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 13 Feb 2020 00:28:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9304: Support starting Hive with Ranger in minicluster
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/15189 ) Change subject: IMPALA-9304: Support starting Hive with Ranger in minicluster .. Patch Set 3: Code-Review+2 This looks good to me. -- To view, visit http://gerrit.cloudera.org:8080/15189 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I01e3a195b00a98388244a922a1a79e65146cec42 Gerrit-Change-Number: 15189 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 13 Feb 2020 00:24:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9256: Refactor constraint information into a separate class.
Anurag Mantripragada has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15213 Change subject: IMPALA-9256: Refactor constraint information into a separate class. .. IMPALA-9256: Refactor constraint information into a separate class. This change refactors the primary keys and foreign keys into a SqlConstraints class since they are almost always used together. This work also helps extend the constraints class to include other constraints we may support in the future. (Ex: Unique constraints.) This patch also: - Fixes a bug in foreign key constraint name generation that was causing foreign keys corresponding to a composite primary key get different foreign key constraint names instead of the same name. - Introduces a canonical representation for foreign keys to prevent bugs like IMPALA-9372 which can occur due to HMS returning results in inconsistent ways. Testing: - Fixed the tests to work with the new behavior. - Ran all the PK/FK tests. Change-Id: I3f1c441c24df84d2d0791ffe94dff60d039a3341 --- M be/generated-sources/gen-cpp/CMakeLists.txt M common/thrift/CMakeLists.txt M common/thrift/CatalogObjects.thrift M common/thrift/CatalogService.thrift A common/thrift/SqlConstraints.thrift M fe/src/main/java/org/apache/impala/analysis/TableDef.java M fe/src/main/java/org/apache/impala/catalog/FeFsTable.java M fe/src/main/java/org/apache/impala/catalog/FeTable.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java A fe/src/main/java/org/apache/impala/catalog/SqlConstraints.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/MetadataOp.java M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java M fe/src/test/java/org/apache/impala/catalog/CatalogTest.java M fe/src/test/java/org/apache/impala/catalog/PartialCatalogInfoTest.java M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java M fe/src/test/java/org/apache/impala/service/JdbcTest.java M tests/hs2/test_hs2.py 24 files changed, 382 insertions(+), 342 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/13/15213/1 -- To view, visit http://gerrit.cloudera.org:8080/15213 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I3f1c441c24df84d2d0791ffe94dff60d039a3341 Gerrit-Change-Number: 15213 Gerrit-PatchSet: 1 Gerrit-Owner: Anurag Mantripragada
[Impala-ASF-CR] IMPALA-7002: Throw AuthorizationException when user accesses non-existent table/database in CTE without required privileges.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15123 ) Change subject: IMPALA-7002: Throw AuthorizationException when user accesses non-existent table/database in CTE without required privileges. .. Patch Set 13: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5213/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15123 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia6b657a7147a136198a9a97a679c9131ee814577 Gerrit-Change-Number: 15123 Gerrit-PatchSet: 13 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 12 Feb 2020 23:06:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9287: Add support for embedded HMS in CDP builds
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15057 ) Change subject: IMPALA-9287: Add support for embedded HMS in CDP builds .. IMPALA-9287: Add support for embedded HMS in CDP builds In some situations, an embedded HMS is enough for catalogd server. And we've already implemented this in IMPALA-8974. But after setting USE_CDP_HIVE=true and rebuilt impala, the custom cluster test case test_kudu_table_create_without_hms would failed due to lacking of related jars. The solution is to add related maven dependency in $IMPALA_HOME/fe/pom.xml and $IMPALA_HOME/shaded-deps/pom.xml. Tests: * Ran test_kudu_table_create_without_hms.py by setting USE_CDP_HIVE=true locally Change-Id: Ibc7d7e30cd560d43bb707dec54f4494355809f66 Reviewed-on: http://gerrit.cloudera.org:8080/15057 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M bin/bootstrap_system.sh M fe/pom.xml M fe/src/test/resources/hive-site.xml.py M tests/custom_cluster/test_kudu_table_create_without_hms.py 4 files changed, 36 insertions(+), 4 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/15057 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ibc7d7e30cd560d43bb707dec54f4494355809f66 Gerrit-Change-Number: 15057 Gerrit-PatchSet: 15 Gerrit-Owner: wangsheng Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: wangsheng
[Impala-ASF-CR] IMPALA-9287: Add support for embedded HMS in CDP builds
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15057 ) Change subject: IMPALA-9287: Add support for embedded HMS in CDP builds .. Patch Set 14: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/15057 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibc7d7e30cd560d43bb707dec54f4494355809f66 Gerrit-Change-Number: 15057 Gerrit-PatchSet: 14 Gerrit-Owner: wangsheng Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Wed, 12 Feb 2020 22:54:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7002: Throw AuthorizationException when user accesses non-existent table/database in CTE without required privileges.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15123 ) Change subject: IMPALA-7002: Throw AuthorizationException when user accesses non-existent table/database in CTE without required privileges. .. Patch Set 12: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5212/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15123 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia6b657a7147a136198a9a97a679c9131ee814577 Gerrit-Change-Number: 15123 Gerrit-PatchSet: 12 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 12 Feb 2020 22:46:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7002: Throw AuthorizationException when user accesses non-existent table/database in CTE without required privileges.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15123 ) Change subject: IMPALA-7002: Throw AuthorizationException when user accesses non-existent table/database in CTE without required privileges. .. Patch Set 13: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5327/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15123 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia6b657a7147a136198a9a97a679c9131ee814577 Gerrit-Change-Number: 15123 Gerrit-PatchSet: 13 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 12 Feb 2020 22:23:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7002: Throw AuthorizationException when user accesses non-existent table/database in CTE without required privileges.
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/15123 ) Change subject: IMPALA-7002: Throw AuthorizationException when user accesses non-existent table/database in CTE without required privileges. .. Patch Set 13: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15123 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia6b657a7147a136198a9a97a679c9131ee814577 Gerrit-Change-Number: 15123 Gerrit-PatchSet: 13 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 12 Feb 2020 22:22:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7002: Throw AuthorizationException when user accesses non-existent table/database in CTE without required privileges.
Wenzhe Zhou has uploaded a new patch set (#13). ( http://gerrit.cloudera.org:8080/15123 ) Change subject: IMPALA-7002: Throw AuthorizationException when user accesses non-existent table/database in CTE without required privileges. .. IMPALA-7002: Throw AuthorizationException when user accesses non-existent table/database in CTE without required privileges. Currently if a user without required privileges tries to access a non-existent database or table, then impala returns an analysis exception instead of authorization exception. This happens because during analysis of the with clause, the authorization request does not get registered due to analysis exception being thrown before it. This patch makes sure that those requests get registered regardless. Testing: - Manual test: - ran CTE with non-existent database/table in impala-shell without required privilege, verified that it results in AuthorizationException. - ran CTE with non-existent database/table in impala-shell with the required privilege, verified that it results in AnalysisException. - Added CTE test cases for non-existent database/table/column in AuthorizationStmtTest. - Passed all FE tests. - Passed all core tests. Change-Id: Ia6b657a7147a136198a9a97a679c9131ee814577 --- M fe/src/main/java/org/apache/impala/analysis/WithClause.java M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java 2 files changed, 41 insertions(+), 16 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/15123/13 -- To view, visit http://gerrit.cloudera.org:8080/15123 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia6b657a7147a136198a9a97a679c9131ee814577 Gerrit-Change-Number: 15123 Gerrit-PatchSet: 13 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-7002: Throw AuthorizationException when user accesses non-existent table/database in CTE without required privileges.
Wenzhe Zhou has posted comments on this change. (
http://gerrit.cloudera.org:8080/15123 )
Change subject: IMPALA-7002: Throw AuthorizationException when user accesses
non-existent table/database in CTE without required privileges.
..
Patch Set 12:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/15123/12/fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java
File
fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java:
http://gerrit.cloudera.org:8080/#/c/15123/12/fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java@606
PS12, Line 606: authorize("with t as (select nocol from
functional.alltypes) select * from t")
: .error(selectError("functional.alltypes"));
:
: // Select from non-existent column in CTE with required
privileges
: // for all existent columns
: authorize("with t as (select nocol from
functional.alltypes) select * from t")
: .error(selectError("functional.alltypes"),
onColumn("functional", "alltypes",
: ALLTYPES_COLUMNS, TPrivilegeLevel.SELECT));
> nit: you can merge these two like :
will merge the code
--
To view, visit http://gerrit.cloudera.org:8080/15123
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia6b657a7147a136198a9a97a679c9131ee814577
Gerrit-Change-Number: 15123
Gerrit-PatchSet: 12
Gerrit-Owner: Wenzhe Zhou
Gerrit-Reviewer: Bikramjeet Vig
Gerrit-Reviewer: Fang-Yu Rao
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Wed, 12 Feb 2020 22:08:56 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7002: Throw AuthorizationException when user accesses non-existent table/database in CTE without required privileges.
Bikramjeet Vig has posted comments on this change. (
http://gerrit.cloudera.org:8080/15123 )
Change subject: IMPALA-7002: Throw AuthorizationException when user accesses
non-existent table/database in CTE without required privileges.
..
Patch Set 12:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/15123/12/fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java
File
fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java:
http://gerrit.cloudera.org:8080/#/c/15123/12/fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java@606
PS12, Line 606: authorize("with t as (select nocol from
functional.alltypes) select * from t")
: .error(selectError("functional.alltypes"));
:
: // Select from non-existent column in CTE with required
privileges
: // for all existent columns
: authorize("with t as (select nocol from
functional.alltypes) select * from t")
: .error(selectError("functional.alltypes"),
onColumn("functional", "alltypes",
: ALLTYPES_COLUMNS, TPrivilegeLevel.SELECT));
nit: you can merge these two like :
authorize("with t as (select nocol from functional.alltypes) select * from
t").error(selectError("functional.alltypes")).error(selectError("functional.alltypes"),
onColumn("functional", "alltypes", ALLTYPES_COLUMNS, TPrivilegeLevel.SELECT));
--
To view, visit http://gerrit.cloudera.org:8080/15123
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia6b657a7147a136198a9a97a679c9131ee814577
Gerrit-Change-Number: 15123
Gerrit-PatchSet: 12
Gerrit-Owner: Wenzhe Zhou
Gerrit-Reviewer: Bikramjeet Vig
Gerrit-Reviewer: Fang-Yu Rao
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Wed, 12 Feb 2020 22:07:09 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7002: Throw AuthorizationException when user accesses non-existent table/database in CTE without required privileges.
Wenzhe Zhou has uploaded a new patch set (#12). ( http://gerrit.cloudera.org:8080/15123 ) Change subject: IMPALA-7002: Throw AuthorizationException when user accesses non-existent table/database in CTE without required privileges. .. IMPALA-7002: Throw AuthorizationException when user accesses non-existent table/database in CTE without required privileges. Currently if a user without required privileges tries to access a non-existent database or table, then impala returns an analysis exception instead of authorization exception. This happens because during analysis of the with clause, the authorization request does not get registered due to analysis exception being thrown before it. This patch makes sure that those requests get registered regardless. Testing: - Manual test: - ran CTE with non-existent database/table in impala-shell without required privilege, verified that it results in AuthorizationException. - ran CTE with non-existent database/table in impala-shell with the required privilege, verified that it results in AnalysisException. - Added CTE test cases for non-existent database/table/column in AuthorizationStmtTest. - Passed all FE tests. - Passed all core tests. Change-Id: Ia6b657a7147a136198a9a97a679c9131ee814577 --- M fe/src/main/java/org/apache/impala/analysis/WithClause.java M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java 2 files changed, 45 insertions(+), 16 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/15123/12 -- To view, visit http://gerrit.cloudera.org:8080/15123 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia6b657a7147a136198a9a97a679c9131ee814577 Gerrit-Change-Number: 15123 Gerrit-PatchSet: 12 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-7002: Throw AuthorizationException when user accesses non-existent table/database in CTE without required privileges.
Wenzhe Zhou has posted comments on this change. (
http://gerrit.cloudera.org:8080/15123 )
Change subject: IMPALA-7002: Throw AuthorizationException when user accesses
non-existent table/database in CTE without required privileges.
..
Patch Set 11:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/15123/11/fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java
File
fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java:
http://gerrit.cloudera.org:8080/#/c/15123/11/fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java@606
PS11, Line 606: authorize("with t as (select nocol from functional.alltypes)
select * from t")
: .error(selectError("functional.alltypes"));
> also run this with privileges on all columns except for the non existing co
will add a new test case with required privileges for all existing columns.
http://gerrit.cloudera.org:8080/#/c/15123/11/fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java@609
PS11, Line 609: // In the following query, the number of explicit column
labels in With clause
: // is greater than the number of columns in the view
statement. It will trigger
: // analysis exception when register local view during
analysis. This test case
: // verify that authorization exception will be thrown if
user don't have privilege
: // to execute SELECT on the column.
> // With clause column labels exceeding the number of columns in the query.
will fix the comments
--
To view, visit http://gerrit.cloudera.org:8080/15123
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia6b657a7147a136198a9a97a679c9131ee814577
Gerrit-Change-Number: 15123
Gerrit-PatchSet: 11
Gerrit-Owner: Wenzhe Zhou
Gerrit-Reviewer: Bikramjeet Vig
Gerrit-Reviewer: Fang-Yu Rao
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Wed, 12 Feb 2020 21:58:07 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9340: fix bug where max missed heartbeats is off by one
Impala Public Jenkins has submitted this change and it was merged. (
http://gerrit.cloudera.org:8080/15201 )
Change subject: IMPALA-9340: fix bug where max missed heartbeats is off by one
..
IMPALA-9340: fix bug where max missed heartbeats is off by one
Max missed heartbeats is off by one due to greater than sign ('>')
used in comparison against statestore_max_missed_heartbeats flag in
failure-detector.cc. This commit change the sign to greater than or
equal ('>=').
Testing:
* Manual test by running impala mini cluster, kill one of impalad, and
verify in statestored log that the killed impalad is declared as
failed exactly at statestore_max_missed_heartbeats
* Run and pass core test
Change-Id: I19f6bfa7e08d231896665d85299302a17959fb6f
Reviewed-on: http://gerrit.cloudera.org:8080/15201
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
---
M be/src/statestore/failure-detector.cc
1 file changed, 2 insertions(+), 2 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/15201
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I19f6bfa7e08d231896665d85299302a17959fb6f
Gerrit-Change-Number: 15201
Gerrit-PatchSet: 4
Gerrit-Owner: Riza Suminto
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Riza Suminto
Gerrit-Reviewer: Sahil Takiar
[Impala-ASF-CR] IMPALA-9340: fix bug where max missed heartbeats is off by one
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15201 ) Change subject: IMPALA-9340: fix bug where max missed heartbeats is off by one .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/15201 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I19f6bfa7e08d231896665d85299302a17959fb6f Gerrit-Change-Number: 15201 Gerrit-PatchSet: 3 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Wed, 12 Feb 2020 21:54:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8712: Make ExecQueryFInstances async
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15154 ) Change subject: IMPALA-8712: Make ExecQueryFInstances async .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5211/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15154 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I33ec96e5885af094c294cd3a76c242995263ba32 Gerrit-Change-Number: 15154 Gerrit-PatchSet: 5 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Wed, 12 Feb 2020 20:21:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8712: Make ExecQueryFInstances async
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15154 ) Change subject: IMPALA-8712: Make ExecQueryFInstances async .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5210/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15154 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I33ec96e5885af094c294cd3a76c242995263ba32 Gerrit-Change-Number: 15154 Gerrit-PatchSet: 4 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Wed, 12 Feb 2020 19:44:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8712: Make ExecQueryFInstances async
Hello Sahil Takiar, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15154 to look at the new patch set (#5). Change subject: IMPALA-8712: Make ExecQueryFInstances async .. IMPALA-8712: Make ExecQueryFInstances async This patch refactors the ExecQueryFInstances rpc to be asychronous. Previously, Impala would issue all the Exec()s, wait for all of them to complete, and then check if any of them resulted in an error. We now stop issuing Exec()s and cancel any that are still in flight as soon as an error occurs. It also performs some cleanup around the thread safety of Coordinator::BackendState, including adding comments and DCHECKS. === Exec RPC Thread Pool === This patch also removes the 'exec_rpc_thread_pool_' from ExecEnv. This thread pool was used to partially simulate async Exec() prior to the switch to KRPC, which provides built-in async rpc capabilities. Removing this thread pool has potential performance implications, as it means that the Exec() parameters are serialized in serialize rather than in parallel (with the level of parallelism determined by the size of the thread pool, which was configurable by an Advanced flag and defaulted to 12). To ensure we don't regress query startup times, I did some performance testing. All tests were done on a 10 node cluster. The baseline used for the tests did not include IMPALA-9181, a perf optimization for query startup done to facilitate this work. I ran TPCH 100 at concurrency levels of 1, 4, and 8 and extracted the query startup times from the profiles. For each concurrency level, the average regression in query startup time was < 2ms. Because query e2e running time was much longer than this, there was no noticable change in total query time. I also ran a 'worst case scenario' with a table with 10,000 pertitions to create a very large Exec() payload to serialize (~1.21MB vs. ~10KB-30KB for TPCH 100). Again, change in query startup time was neglible. Testing: - Added a e2e test that verifies that a query where an Exec() fails doesn't wait for all Exec()s to complete before cancelling and returning the error to the client. Change-Id: I33ec96e5885af094c294cd3a76c242995263ba32 --- M be/src/common/global-flags.cc M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator-backend-state.h M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/runtime/query-state.cc M be/src/util/counting-barrier.h M tests/custom_cluster/test_rpc_exception.py M tests/failure/test_failpoints.py 11 files changed, 444 insertions(+), 266 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/54/15154/5 -- To view, visit http://gerrit.cloudera.org:8080/15154 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I33ec96e5885af094c294cd3a76c242995263ba32 Gerrit-Change-Number: 15154 Gerrit-PatchSet: 5 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-8712: Make ExecQueryFInstances async
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/15154 ) Change subject: IMPALA-8712: Make ExecQueryFInstances async .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/15154/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15154/3//COMMIT_MSG@46 PS3, Line 46: Testing: : - Added a e2e test that verifies that a query where an Exec() > this can be rebased now, right? Done -- To view, visit http://gerrit.cloudera.org:8080/15154 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I33ec96e5885af094c294cd3a76c242995263ba32 Gerrit-Change-Number: 15154 Gerrit-PatchSet: 5 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Wed, 12 Feb 2020 19:35:37 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8712: Make ExecQueryFInstances async
Thomas Tauber-Marshall has posted comments on this change. (
http://gerrit.cloudera.org:8080/15154 )
Change subject: IMPALA-8712: Make ExecQueryFInstances async
..
Patch Set 4:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/15154/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/15154/2//COMMIT_MSG@23
PS2, Line 23: Removing this thread pool has potential performance implications,
as
: it means that the Exec() parameters are serialized in serialize
rather
: than in parallel (with the level of parallelism determined by the
size
: of the thread pool, which was configurable by an Advanced flag and
: defaulted to 12).
> right, a more fine grained counter around all the calls to the ThriftSerial
Done
http://gerrit.cloudera.org:8080/#/c/15154/3/be/src/runtime/coordinator-backend-state.h
File be/src/runtime/coordinator-backend-state.h:
http://gerrit.cloudera.org:8080/#/c/15154/3/be/src/runtime/coordinator-backend-state.h@406
PS3, Line 406:
> nit: single quote
Done
http://gerrit.cloudera.org:8080/#/c/15154/3/be/src/runtime/coordinator-backend-state.cc
File be/src/runtime/coordinator-backend-state.cc:
http://gerrit.cloudera.org:8080/#/c/15154/3/be/src/runtime/coordinator-backend-state.cc@310
PS3, Line 310:
> nit: better name might be 'error' since this is only invoked on error
Its also invoked in the case of IsEmptyBackend() being true, which isn't an
error.
http://gerrit.cloudera.org:8080/#/c/15154/3/be/src/runtime/coordinator-backend-state.cc@549
PS3, Line 549: (!exec_rpc_sent_) {
> can this be re-factored as well so that notify is called after releasing th
Obviously its sort of tricky. I could do the same thing as in the other
functions (put everything in a block with a goto label after it) but that's
ugly and I hate to do it for only even one branch.
I suppose there's no reason that I can't just call 'unlock()' on the
unique_lock, other than the fact that we don't usually do that. Should be fine
as long as I leave a comment explaining why we're doing it.
http://gerrit.cloudera.org:8080/#/c/15154/3/be/src/runtime/coordinator-backend-state.cc@900
PS3, Line 900: ess
> nit: the 'For' seems extraneous and doesn't always make sense in the contex
Done
http://gerrit.cloudera.org:8080/#/c/15154/3/be/src/runtime/coordinator-backend-state.cc@901
PS3, Line 901: Coordin
> nit: maybe 'target backend' to make it clear that the Coordinator is attemp
Done
http://gerrit.cloudera.org:8080/#/c/15154/3/tests/custom_cluster/test_rpc_exception.py
File tests/custom_cluster/test_rpc_exception.py:
http://gerrit.cloudera.org:8080/#/c/15154/3/tests/custom_cluster/test_rpc_exception.py@28
PS3, Line 28: def _get_rpc_debug_action(rpc, action, port=KRPC_PORT):
: """Returns a debug action that causes rpcs with the name 'rpc'
that are sent to the
: impalad at 'port' to execute the debug aciton 'action'."""
: return "IMPALA_SERVICE_POOL:127.0.0.1:{port}:{rpc}:{action}" \
: .format(rpc=rpc, port=port, action=action)
:
:
: def _get_fail_action(rpc, error=None, port=KRPC_PORT, p=0.1):
: """Returns a debug action that causes rpcs with the name 'rpc'
that are sent to the
: impalad at 'port' to FAIL with probability 'p' and return
'error' if specified."""
: action = "FAIL@%s" % p
: if error is not None:
: action += "@" + error
: return _get_rpc_debug_action(rpc, action, port=port
> do we expect to use these methods outside of the TestRPCException class?
No, I moved them outside of the class because I couldn't figure out how to have
one call the other given that they're called from the function decorators where
there isn't a 'this' available and python's support for 'static' type stuff is
weird. Probably someone who understand python better than me could make it work.
I added '_' to the start of both functions names to make it clearer they're not
intended for use outside this file.
--
To view, visit http://gerrit.cloudera.org:8080/15154
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I33ec96e5885af094c294cd3a76c242995263ba32
Gerrit-Change-Number: 15154
Gerrit-PatchSet: 4
Gerrit-Owner: Thomas Tauber-Marshall
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Sahil Takiar
Gerrit-Reviewer: Thomas Tauber-Marshall
Gerrit-Comment-Date: Wed, 12 Feb 2020 19:00:28 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8712: Make ExecQueryFInstances async
Hello Sahil Takiar, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15154 to look at the new patch set (#4). Change subject: IMPALA-8712: Make ExecQueryFInstances async .. IMPALA-8712: Make ExecQueryFInstances async This patch refactors the ExecQueryFInstances rpc to be asychronous. Previously, Impala would issue all the Exec()s, wait for all of them to complete, and then check if any of them resulted in an error. We now stop issuing Exec()s and cancel any that are still in flight as soon as an error occurs. It also performs some cleanup around the thread safety of Coordinator::BackendState, including adding comments and DCHECKS. === Exec RPC Thread Pool === This patch also removes the 'exec_rpc_thread_pool_' from ExecEnv. This thread pool was used to partially simulate async Exec() prior to the switch to KRPC, which provides built-in async rpc capabilities. Removing this thread pool has potential performance implications, as it means that the Exec() parameters are serialized in serialize rather than in parallel (with the level of parallelism determined by the size of the thread pool, which was configurable by an Advanced flag and defaulted to 12). To ensure we don't regress query startup times, I did some performance testing. All tests were done on a 10 node cluster. The baseline used for the tests did not include IMPALA-9181, a perf optimization for query startup done to facilitate this work. I ran TPCH 100 at concurrency levels of 1, 4, and 8 and extracted the query startup times from the profiles. For each concurrency level, the average regression in query startup time was < 2ms. Because query e2e running time was much longer than this, there was no noticable change in total query time. I also ran a 'worst case scenario' with a table with 10,000 pertitions to create a very large Exec() payload to serialize (~1.21MB vs. ~10KB-30KB for TPCH 100). Again, change in query startup time was neglible. TODO: once IMPALA-9335 (krpc rebase) goes in, the change in be/src/kudu/rpc/connection.cc can be removed from this patch. Testing: - Added a e2e test that verifies that a query where an Exec() fails doesn't wait for all Exec()s to complete before cancelling and returning the error to the client. Change-Id: I33ec96e5885af094c294cd3a76c242995263ba32 --- M be/src/common/global-flags.cc M be/src/kudu/rpc/connection.cc M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator-backend-state.h M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/runtime/query-state.cc M be/src/util/counting-barrier.h M tests/custom_cluster/test_rpc_exception.py M tests/failure/test_failpoints.py 12 files changed, 446 insertions(+), 261 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/54/15154/4 -- To view, visit http://gerrit.cloudera.org:8080/15154 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I33ec96e5885af094c294cd3a76c242995263ba32 Gerrit-Change-Number: 15154 Gerrit-PatchSet: 4 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-4224: execute separate join builds fragments
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14859 ) Change subject: IMPALA-4224: execute separate join builds fragments .. Patch Set 40: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5209/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14859 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4403c8e62d9c13854e7830602ee613f8efc80c58 Gerrit-Change-Number: 14859 Gerrit-PatchSet: 40 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 12 Feb 2020 18:46:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9287: Add support for embedded HMS in CDP builds
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/15057 ) Change subject: IMPALA-9287: Add support for embedded HMS in CDP builds .. Patch Set 13: Code-Review+2 Thanks for seeing the patch through. Looks good to me. -- To view, visit http://gerrit.cloudera.org:8080/15057 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibc7d7e30cd560d43bb707dec54f4494355809f66 Gerrit-Change-Number: 15057 Gerrit-PatchSet: 13 Gerrit-Owner: wangsheng Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Wed, 12 Feb 2020 18:02:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9287: Add support for embedded HMS in CDP builds
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15057 ) Change subject: IMPALA-9287: Add support for embedded HMS in CDP builds .. Patch Set 14: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5326/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15057 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibc7d7e30cd560d43bb707dec54f4494355809f66 Gerrit-Change-Number: 15057 Gerrit-PatchSet: 14 Gerrit-Owner: wangsheng Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Wed, 12 Feb 2020 18:02:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9287: Add support for embedded HMS in CDP builds
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15057 ) Change subject: IMPALA-9287: Add support for embedded HMS in CDP builds .. Patch Set 14: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15057 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibc7d7e30cd560d43bb707dec54f4494355809f66 Gerrit-Change-Number: 15057 Gerrit-PatchSet: 14 Gerrit-Owner: wangsheng Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Wed, 12 Feb 2020 18:02:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4224: execute separate join builds fragments
Hello Zoltan Borok-Nagy, Csaba Ringhofer, Bikramjeet Vig, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14859 to look at the new patch set (#40). Change subject: IMPALA-4224: execute separate join builds fragments .. IMPALA-4224: execute separate join builds fragments This enables parallel plans with the join build in a separate fragment and fixes all of the ensuing fallout. After this change, mt_dop plans with joins have separate build fragments. There is still a 1:1 relationship between join nodes and builders, so the builders are only accessed by the join node's thread after it is handed off. This lets us defer the work required to make PhjBuilder and NljBuilder safe to be shared between nodes. Planner changes: * Combined the parallel and distributed planning code paths. * Misc fixes to generate reasonable thrift structures in the query exec requests, i.e. containing the right nodes. * Fixes to resource calculations for the separate build plans. ** Calculate separate join/build resource consumption. ** Simplified the resource estimation by calculating resource consumption for each fragment separately, and assuming that all fragments hit their peak resource consumption at the same time. IMPALA-9255 is the follow-on to make the resource estimation more accurate. Scheduler changes: * Various fixes to handle multiple TPlanExecInfos correctly, which are generated by the planner for the different cohorts. * Add logic to colocate build fragments with parent fragments. Runtime filter changes: * Build sinks now produce runtime filters, which required planner and coordinator fixes to handle. DataSink changes: * Close the input plan tree before calling FlushFinal() to release resources. This depends on Send() not holding onto references to input batches, which was true except for NljBuilder. This invariant is documented. Join builder changes: * Add a common base class for PhjBuilder and NljBuilder with functions to handle synchronisation with the join node. * Close plan tree earlier in FragmentInstanceState::Exec() so that peak resource requirements are lower. * The NLJ always copies input batches, so that it can close its input tree. JoinNode changes: * Join node blocks waiting for build-side to be ready, then eventually signals that it's done, allowing the builder to be cleaned up. * NLJ and PHJ nodes handle both the integrated builder and the external builder. There is a 1:1 relationship between the node and the builder, so we don't deal with thread safety yet. * Buffer reservations are transferred between the builder and join node when running with the separate builder. This is not really necessary right now, since it is all single-threaded, but will be important for the shared broadcast. - The builder transfers memory for probe buffers to the join node at the end of each build phase. - At end of each probe phase, reservation needs to be handed back to builder (or released). ExecSummary changes: * The summary logic was modified to handle connecting fragments via join builds. The logic is an extension of what was used for exchanges. Testing: * Enable --unlock_mt_dop for end-to-end tests * Migrate some tests to run as part of end-to-end tests instead of custom cluster. * Add mt_dop dimension to various end-to-end tests to provide coverage of join queries, spill-to-disk and cancellation. * Ran a single node TPC-H and TPC-DS stress test with mt_dop=0 and mt_dop=4. Perf: * Ran TPC-H scale factor 30 locally with mt_dop=0. No significant change. Change-Id: I4403c8e62d9c13854e7830602ee613f8efc80c58 --- M be/src/exec/CMakeLists.txt M be/src/exec/blocking-join-node.cc M be/src/exec/blocking-join-node.h M be/src/exec/data-sink.cc M be/src/exec/data-sink.h M be/src/exec/exec-node.h A be/src/exec/join-builder.cc A be/src/exec/join-builder.h M be/src/exec/nested-loop-join-builder.cc M be/src/exec/nested-loop-join-builder.h M be/src/exec/nested-loop-join-node.cc M be/src/exec/nested-loop-join-node.h M be/src/exec/partitioned-hash-join-builder.cc M be/src/exec/partitioned-hash-join-builder.h M be/src/exec/partitioned-hash-join-node.cc M be/src/exec/partitioned-hash-join-node.h M be/src/runtime/bufferpool/buffer-pool-internal.h M be/src/runtime/bufferpool/buffer-pool-test.cc M be/src/runtime/bufferpool/buffer-pool.cc M be/src/runtime/bufferpool/buffer-pool.h M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator.cc M be/src/runtime/fragment-instance-state.cc M be/src/runtime/fragment-instance-state.h M be/src/runtime/initial-reservations.cc M be/src/runtime/row-batch.cc M be/src/runtime/runtime-state.cc M be/src/runtime/runtime-state.h M be/src/runtime/spillable-row-batch-queue.h M be/src/util/summary-util.cc M bin/run-all-tests.sh M common/thrift/DataSinks.thrift M
[Impala-ASF-CR] IMPALA-4224: execute separate join builds fragments
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14859 ) Change subject: IMPALA-4224: execute separate join builds fragments .. Patch Set 33: (1 comment) http://gerrit.cloudera.org:8080/#/c/14859/33/be/src/exec/partitioned-hash-join-node.cc File be/src/exec/partitioned-hash-join-node.cc: http://gerrit.cloudera.org:8080/#/c/14859/33/be/src/exec/partitioned-hash-join-node.cc@78 PS33, Line 78: RETURN_IF_ERROR(PhjBuilderConfig::CreateConfig(state, tnode_->node_id, : tnode_->hash_join_node.join_op, _row_desc(), eq_join_conjuncts, : tnode_->runtime_filters, tnode_->hash_join_node.hash_seed, _builder_config)); > Done. Had to move IsSeparateBuild() into the plan node. I had to revert this, since the containing subplan isn't initialized in Init() (when I ran exhaustive tests, it crashed some nested types tests) -- To view, visit http://gerrit.cloudera.org:8080/14859 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4403c8e62d9c13854e7830602ee613f8efc80c58 Gerrit-Change-Number: 14859 Gerrit-PatchSet: 33 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 12 Feb 2020 18:01:40 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7002: Throw AuthorizationException when user accesses non-existent table/database in CTE without required privileges.
Bikramjeet Vig has posted comments on this change. (
http://gerrit.cloudera.org:8080/15123 )
Change subject: IMPALA-7002: Throw AuthorizationException when user accesses
non-existent table/database in CTE without required privileges.
..
Patch Set 11:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/15123/11/fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java
File
fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java:
http://gerrit.cloudera.org:8080/#/c/15123/11/fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java@606
PS11, Line 606: authorize("with t as (select nocol from functional.alltypes)
select * from t")
: .error(selectError("functional.alltypes"));
also run this with privileges on all columns except for the non existing
column(checkout other tests to see how you can do that).
http://gerrit.cloudera.org:8080/#/c/15123/11/fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java@609
PS11, Line 609: // In the following query, the number of explicit column
labels in With clause
: // is greater than the number of columns in the view
statement. It will trigger
: // analysis exception when register local view during
analysis. This test case
: // verify that authorization exception will be thrown if
user don't have privilege
: // to execute SELECT on the column.
// With clause column labels exceeding the number of columns in the query.
--
To view, visit http://gerrit.cloudera.org:8080/15123
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia6b657a7147a136198a9a97a679c9131ee814577
Gerrit-Change-Number: 15123
Gerrit-PatchSet: 11
Gerrit-Owner: Wenzhe Zhou
Gerrit-Reviewer: Bikramjeet Vig
Gerrit-Reviewer: Fang-Yu Rao
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Wed, 12 Feb 2020 17:55:51 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9340: fix bug where max missed heartbeats is off by one
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/15201 ) Change subject: IMPALA-9340: fix bug where max missed heartbeats is off by one .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15201 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I19f6bfa7e08d231896665d85299302a17959fb6f Gerrit-Change-Number: 15201 Gerrit-PatchSet: 2 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Wed, 12 Feb 2020 17:03:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9340: fix bug where max missed heartbeats is off by one
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15201 ) Change subject: IMPALA-9340: fix bug where max missed heartbeats is off by one .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5325/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15201 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I19f6bfa7e08d231896665d85299302a17959fb6f Gerrit-Change-Number: 15201 Gerrit-PatchSet: 3 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Wed, 12 Feb 2020 17:03:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9340: fix bug where max missed heartbeats is off by one
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15201 ) Change subject: IMPALA-9340: fix bug where max missed heartbeats is off by one .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15201 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I19f6bfa7e08d231896665d85299302a17959fb6f Gerrit-Change-Number: 15201 Gerrit-PatchSet: 3 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Wed, 12 Feb 2020 17:03:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9242: Filter privileges before returning them to Sentry
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15068 ) Change subject: IMPALA-9242: Filter privileges before returning them to Sentry .. Patch Set 14: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/15068 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iecd4281368d1c9fe88cfe850ea725cd68895712e Gerrit-Change-Number: 15068 Gerrit-PatchSet: 14 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Anonymous Coward (498) Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 12 Feb 2020 15:02:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9242: Filter privileges before returning them to Sentry
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15068 ) Change subject: IMPALA-9242: Filter privileges before returning them to Sentry .. IMPALA-9242: Filter privileges before returning them to Sentry This change implements the new FilteredPrivilegeCache, which adds functions for filtering privileges based on the authorizable and for returning Privileges directly instead of their String form. The filtering is based on server + db + table (or just server in case of URI privileges) to filter out the bulk of unrelated privileges. Efficient filtering is done by a new class PrincipalPrivilegeTree. It was tempting to reuse Sentry's TreePrivilegeCache, which has a very similar role, but it lacks a "remove" function that is needed to keep this index in sync with the CatalogObjectCache in Principal. I am also a bit concerned about the possible side effect of Sentry's interning of names in privileges - we try to avoid using String.intern() on massive amount of names in Impala. Other Changes: - Add the Sentry privilege name as member to PrincipalPrivileges. Note that the name was a member of TPrivilege till IMPALA-7616. Storing the name shouldn't consume much extra memory, as it is already stored as the key of the PrincipalPrivilege in CatalogObjectCache. Testing: - added unit tests based on Sentry / TestTreePrivilegeCache Change-Id: Iecd4281368d1c9fe88cfe850ea725cd68895712e Reviewed-on: http://gerrit.cloudera.org:8080/15068 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationPolicy.java M fe/src/main/java/org/apache/impala/catalog/Principal.java M fe/src/main/java/org/apache/impala/catalog/PrincipalPrivilege.java A fe/src/main/java/org/apache/impala/catalog/PrincipalPrivilegeTree.java A fe/src/test/java/org/apache/impala/catalog/PrincipalPrivilegeTreeTest.java 5 files changed, 513 insertions(+), 9 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/15068 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Iecd4281368d1c9fe88cfe850ea725cd68895712e Gerrit-Change-Number: 15068 Gerrit-PatchSet: 15 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Anonymous Coward (498) Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-9279: Update the Kudu version to include VARCHAR support
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15134 ) Change subject: IMPALA-9279: Update the Kudu version to include VARCHAR support .. Patch Set 9: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/15134 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iafe56342d43cb63e35c0bbb1b4a99327dda0a44a Gerrit-Change-Number: 15134 Gerrit-PatchSet: 9 Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Comment-Date: Wed, 12 Feb 2020 13:27:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9279: Update the Kudu version to include VARCHAR support
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15134 ) Change subject: IMPALA-9279: Update the Kudu version to include VARCHAR support .. IMPALA-9279: Update the Kudu version to include VARCHAR support Before this change the preferred way of getting Kudu was to pull it in from the specified CDH build (even if USE_CDP_HIVE was set to true). Optionally by setting USE_CDH_KUDU to false, one could force Impala to use the native toolchain Kudu. But even then, the Kudu Java artifacts would be downloaded from CDH. Since Kudu VARCHAR support won't be backported to CDH, this behavior blocks the Impala side of the Kudu/Impala VARCHAR integration. With this change: 1. Using the native toolchain Kudu (including the Java artifacts) is the default behavior. From now on USE_CDH_KUDU will be set to false by default. Impala can be forced to fall back on using the CDH Kudu by explicitly setting USE_CDH_KUDU to true. 2. Kudu version is updated to include the VARCHAR support. Testing: Ran exhaustive tests with USE_CDH_KUDU=true and USE_CDH_KUDU=false. Change-Id: Iafe56342d43cb63e35c0bbb1b4a99327dda0a44a Reviewed-on: http://gerrit.cloudera.org:8080/15134 Reviewed-by: Attila Jeges Tested-by: Impala Public Jenkins --- M bin/bootstrap_toolchain.py M bin/impala-config.sh M impala-parent/pom.xml 3 files changed, 42 insertions(+), 25 deletions(-) Approvals: Attila Jeges: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/15134 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Iafe56342d43cb63e35c0bbb1b4a99327dda0a44a Gerrit-Change-Number: 15134 Gerrit-PatchSet: 10 Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal
[Impala-ASF-CR] WIP: Asynchronous code generation
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/15105 ) Change subject: WIP: Asynchronous code generation .. Patch Set 8: (1 comment) http://gerrit.cloudera.org:8080/#/c/15105/6/be/src/exec/hdfs-scanner.h File be/src/exec/hdfs-scanner.h: http://gerrit.cloudera.org:8080/#/c/15105/6/be/src/exec/hdfs-scanner.h@462 PS6, Line 462: auto invoke(ThisT This, CodegenFnPtrBase* ato > Also auto works, I'll use that. Since this is a wrapper function I think decltype(auto) should be preferred because you want the exact same return type as your delegated functions. -- To view, visit http://gerrit.cloudera.org:8080/15105 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia7cbfa7c6734dcf03641629429057d6a4194aa6b Gerrit-Change-Number: 15105 Gerrit-PatchSet: 8 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 12 Feb 2020 11:47:45 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9242: Filter privileges before returning them to Sentry
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15068 ) Change subject: IMPALA-9242: Filter privileges before returning them to Sentry .. Patch Set 14: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15068 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iecd4281368d1c9fe88cfe850ea725cd68895712e Gerrit-Change-Number: 15068 Gerrit-PatchSet: 14 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Anonymous Coward (498) Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 12 Feb 2020 10:10:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9242: Filter privileges before returning them to Sentry
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15068 ) Change subject: IMPALA-9242: Filter privileges before returning them to Sentry .. Patch Set 14: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5324/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15068 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iecd4281368d1c9fe88cfe850ea725cd68895712e Gerrit-Change-Number: 15068 Gerrit-PatchSet: 14 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Anonymous Coward (498) Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 12 Feb 2020 10:10:50 + Gerrit-HasComments: No
[native-toolchain-CR] Bump boost to 1.61.0-p2 for kudu building.
huangtianhua...@gmail.com has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15211 Change subject: Bump boost to 1.61.0-p2 for kudu building. .. Bump boost to 1.61.0-p2 for kudu building. Change-Id: I6525c255f8aa64759a0480b44855a158fdd0cce6 --- M buildall.sh 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/11/15211/1 -- To view, visit http://gerrit.cloudera.org:8080/15211 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I6525c255f8aa64759a0480b44855a158fdd0cce6 Gerrit-Change-Number: 15211 Gerrit-PatchSet: 1 Gerrit-Owner: Anonymous Coward
[Impala-ASF-CR] IMPALA-9279: Update the Kudu version to include VARCHAR support
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15134 ) Change subject: IMPALA-9279: Update the Kudu version to include VARCHAR support .. Patch Set 9: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5323/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15134 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iafe56342d43cb63e35c0bbb1b4a99327dda0a44a Gerrit-Change-Number: 15134 Gerrit-PatchSet: 9 Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Comment-Date: Wed, 12 Feb 2020 08:34:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9279: Update the Kudu version to include VARCHAR support
Attila Jeges has posted comments on this change. ( http://gerrit.cloudera.org:8080/15134 ) Change subject: IMPALA-9279: Update the Kudu version to include VARCHAR support .. Patch Set 9: Code-Review+2 Carry +2 -- To view, visit http://gerrit.cloudera.org:8080/15134 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iafe56342d43cb63e35c0bbb1b4a99327dda0a44a Gerrit-Change-Number: 15134 Gerrit-PatchSet: 9 Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Comment-Date: Wed, 12 Feb 2020 08:33:43 + Gerrit-HasComments: No
