[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/13005 )
Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive
3.1.0
..
Patch Set 20:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/13005/20/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/13005/20/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@283
PS20, Line 283: if (currentChar == '\\' && (i + 6 <
stringLiteral.length()) && stringLiteral.charAt(i + 1) == 'u') {
line too long (108 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 20
Gerrit-Owner: Vihang Karajgaonkar
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Philip Zeyliger
Gerrit-Reviewer: Sudhanshu Arora
Gerrit-Reviewer: Todd Lipcon
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Tue, 30 Apr 2019 05:54:26 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8280, IMPALA-8281: Add support for show grant user/group with Ranger
Austin Nobis has posted comments on this change. (
http://gerrit.cloudera.org:8080/13074 )
Change subject: IMPALA-8280, IMPALA-8281: Add support for show grant user/group
with Ranger
..
Patch Set 8:
(20 comments)
http://gerrit.cloudera.org:8080/#/c/13074/7//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/13074/7//COMMIT_MSG@9
PS7, Line 9: Add support for SHOW GRANT statements for Apache Ranger. This
patch also
: adds the RangerImpaladAuthorizationManager as the show grant
statement
: is called from impalad.
> mention the list of new syntax and also mention that "show grant" without "
Done
http://gerrit.cloudera.org:8080/#/c/13074/7/fe/src/main/java/org/apache/impala/authorization/AuthorizationPolicy.java
File fe/src/main/java/org/apache/impala/authorization/AuthorizationPolicy.java:
http://gerrit.cloudera.org:8080/#/c/13074/7/fe/src/main/java/org/apache/impala/authorization/AuthorizationPolicy.java@526
PS7, Line 526: Li
> does this need to be public?
Done
http://gerrit.cloudera.org:8080/#/c/13074/7/fe/src/main/java/org/apache/impala/authorization/AuthorizationPolicy.java@529
PS7, Line 529: oThrift();
> This is an unordered set, will this be a problem like the output of the row
Done
http://gerrit.cloudera.org:8080/#/c/13074/7/fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java
File
fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java:
http://gerrit.cloudera.org:8080/#/c/13074/7/fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java@260
PS7, Line 260: resource.put(RangerImpalaResourceBuilder.DATABASE,
getOrAll(privilege.getDb_name()));
: resource.put(RangerImpalaResourceBuilder.TABLE,
getOrAll(privilege.getTable_name()));
: resource.put(RangerImpalaResourceBuilder.COLUMN,
: getOrAll(privilege.getColumn_name()));
:
: return resource;
: }
:
: public static Map createUriResource(TPrivilege
privilege) {
: Map resource = new HashMap<>();
: String uri = privilege.getUri();
: resource.put(RangerImpalaResourceBuilder.URL, uri == null ?
"*" : uri);
:
: return resource;
: }
:
: public static Map
createFunctionResource(TPrivilege privilege) {
: Map resource = new HashMap<>();
:
: resource.put(RangerImpalaResourceBuilder.DATABASE,
getOrAll(privilege.getDb_name()));
: resource.put(RangerImpalaResourceBuilder.UDF, "*");
:
: return resource;
: }
:
: private static String getOrAll(String resource) {
:
> can we make these private now?
They are used in the RangerImpaladAuthorizationManager as well.
http://gerrit.cloudera.org:8080/#/c/13074/7/fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java
File
fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java:
http://gerrit.cloudera.org:8080/#/c/13074/7/fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java@162
PS7, Line 162: upInformation ugi = UserGroupInforma
> let's try to use the one from JDK instead, i.e. new HashSet<>(ugi.getGroupN
The JDK HashSet constructor doesn't take a String[]. The code would have to be:
`return new HashSet<>(Arrays.asList(ugi.getGroupNames());`
http://gerrit.cloudera.org:8080/#/c/13074/7/fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java@236
PS7, Line 236:
: switch (privilege.getScope()) {
: case COLUMN:
: if (!column.isPresent() || column.get().equals("*"))
return null;
: case TABLE:
: if (!table.isPresent() || table.get().equals("*"))
return null;
: case DATABASE:
: if (!database.isPresent() || database.get().equals("*"))
return null;
: break;
: case URI:
: if (!uri.isPresent() || uri.get().equals("*")) return
null;
:
> add default case to to make the compiler happy
Done
http://gerrit.cloudera.org:8080/#/c/13074/7/fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java@259
PS7, Line 259: vate
> since L257 is sa throw. We can make it just an if instead of else if.
Done
http://gerrit.cloudera.org:8080/#/c/13074/7/fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java@276
PS7, Line 276: // Server is used by column, function, and URI
[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
Vihang Karajgaonkar has uploaded a new patch set (#20). ( http://gerrit.cloudera.org:8080/13005 ) Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0 .. IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0 This change adds a compatibility shim in fe so that Impala can interoperate with Hive 3.1.0. It moves the existing Metastoreshim class to a compat-hive-2 directory and adds a new Metastoreshim class under compat-hive-3 directory. These shim classes implement method which are different in hive-2 v/s hive-3 and are used by front end code. At the build time, based on the environment variable IMPALA_HIVE_MAJOR_VERSION one of the two shims is added to as source using the fe/pom.xml build plugin. Additionally, in order to reduce the dependencies footprint of Hive in the front end code, this patch also introduces a new module called shaded-deps. This module using shade plugin to include only the source files from hive-exec which are need by the fe code. For hive-2 build path, no changes are done with respect to hive dependencies to minimize the risk of destabilizing the master branch on the default build option of using Hive-2. The different set of dependencies are activated using maven profiles. The activation of each profile is automatic based on the IMPALA_HIVE_MAJOR_VERSION. Testing: 1. Code compiles and runs against both HMS-3 and HMS-2 2. Ran full-suite of tests using the private jenkins job against HMS-2 3. Running full-tests against HMS-3 will need more work like supporting Tez in the mini-cluster (for dataloading) and HMS transaction support since HMS3 create transactional tables by default. THis will be on-going effort and test failures on Hive-3 will be fixed in additional sub-tasks. Notes: 1. Patch uses a custom build of Hive to be deployed in mini-cluster. This build has the fixes for HIVE-21596. This hack will be removed when the patches are available in official CDP Hive builds. 2. Some of the existing tests rely on the fact the UDFs implement the UDF interface in Hive (UDFLength, UDFHour, UDFYear). These built-in hive functions have been moved to use GenericUDF interface in Hive 3. Impala currently only supports UDFExecutor. In order to have a full compatibility with all the functions in Hive 2.x we should support GenericUDFs too. That would be taken up as a separate patch. 3. Sentry dependencies bring a lot of transitive hive dependencies. The patch excludes such dependencies since they create problems while building against Hive-3. Since these hive-2 dependencies are already included when building against hive-2 this should not be a problem. Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436 --- M CMakeLists.txt M README.md M bin/bootstrap_toolchain.py M bin/impala-config.sh M common/thrift/.gitignore M common/thrift/CMakeLists.txt M fe/CMakeLists.txt M fe/pom.xml A fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java A fe/src/compat-hive-3/java/org/apache/impala/compat/HiveMetadataFormatUtils.java A fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/analysis/StringLiteral.java M fe/src/main/java/org/apache/impala/catalog/FeHBaseTable.java M fe/src/main/java/org/apache/impala/catalog/TableLoader.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java D fe/src/main/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/hive/executor/UdfExecutor.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java M fe/src/main/java/org/apache/impala/service/MetadataOp.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java M fe/src/test/java/org/apache/impala/hive/executor/UdfExecutorTest.java M fe/src/test/java/org/apache/impala/testutil/EmbeddedMetastoreClientPool.java M impala-parent/pom.xml A shaded-deps/.gitignore A shaded-deps/CMakeLists.txt A shaded-deps/pom.xml M testdata/bin/run-hive-server.sh M tests/custom_cluster/test_permanent_udfs.py 31 files changed, 1,791 insertions(+), 456 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/13005/20 -- To view, visit http://gerrit.cloudera.org:8080/13005 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436 Gerrit-Change-Number: 13005 Gerrit-PatchSet: 20 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer:
[Impala-ASF-CR] IMPALA-8280, IMPALA-8281: Add support for show grant user/group with Ranger
Austin Nobis has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/13074 ) Change subject: IMPALA-8280, IMPALA-8281: Add support for show grant user/group with Ranger .. IMPALA-8280, IMPALA-8281: Add support for show grant user/group with Ranger Add support for SHOW GRANT statements for Apache Ranger. This patch also adds the RangerImpaladAuthorizationManager as the show grant statement is called from impalad. The new supported syntax is: SHOW GRANT USER/GROUP ON The following syntax is valid SQL, but is not supported currently by the Apache Ranger integration with Impala: SHOW GRANT USER/GROUP Testing: - Ran all FE unit tests - Ran authorization E2E tests - Updated test_ranger to use show grant statement for verification of granted privileges Change-Id: Ic46fb9fc36c9e11ec78d5840d22eb0668150c2a4 --- M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/ShowGrantPrincipalStmt.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java A fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java M fe/src/main/java/org/apache/impala/catalog/Principal.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeAuthStmtsTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M tests/authorization/test_ranger.py 9 files changed, 730 insertions(+), 88 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/74/13074/8 -- To view, visit http://gerrit.cloudera.org:8080/13074 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic46fb9fc36c9e11ec78d5840d22eb0668150c2a4 Gerrit-Change-Number: 13074 Gerrit-PatchSet: 8 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-7290: part 1: clean up shell tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13083 ) Change subject: IMPALA-7290: part 1: clean up shell tests .. Patch Set 9: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13083 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe5ab7f4817e690b7d3be08d71f8f14364b84412 Gerrit-Change-Number: 13083 Gerrit-PatchSet: 9 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 30 Apr 2019 05:51:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7290: part 1: clean up shell tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13083 ) Change subject: IMPALA-7290: part 1: clean up shell tests .. Patch Set 9: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4113/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13083 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe5ab7f4817e690b7d3be08d71f8f14364b84412 Gerrit-Change-Number: 13083 Gerrit-PatchSet: 9 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 30 Apr 2019 05:51:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8465: fix TestJsonEndpoints for remote clusters
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13136 ) Change subject: IMPALA-8465: fix TestJsonEndpoints for remote clusters .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13136 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8cf6ecfb72ec18be0b5274e360825fb3f203dd20 Gerrit-Change-Number: 13136 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 30 Apr 2019 05:50:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8463: fix skip.header.line.count on local catalog
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13131 ) Change subject: IMPALA-8463: fix skip.header.line.count on local catalog .. Patch Set 7: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4112/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13131 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3957b3661574c356a14197e8829abbed92028a1a Gerrit-Change-Number: 13131 Gerrit-PatchSet: 7 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 30 Apr 2019 05:51:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8463: fix skip.header.line.count on local catalog
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13131 ) Change subject: IMPALA-8463: fix skip.header.line.count on local catalog .. Patch Set 7: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13131 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3957b3661574c356a14197e8829abbed92028a1a Gerrit-Change-Number: 13131 Gerrit-PatchSet: 7 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 30 Apr 2019 05:51:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8465: fix TestJsonEndpoints for remote clusters
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13136 ) Change subject: IMPALA-8465: fix TestJsonEndpoints for remote clusters .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4111/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13136 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8cf6ecfb72ec18be0b5274e360825fb3f203dd20 Gerrit-Change-Number: 13136 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 30 Apr 2019 05:50:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 ) Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0 .. Patch Set 19: (1 comment) http://gerrit.cloudera.org:8080/#/c/13005/19/bin/impala-config.sh File bin/impala-config.sh: http://gerrit.cloudera.org:8080/#/c/13005/19/bin/impala-config.sh@175 PS19, Line 175: export CDP_HIVE_VERSION=3.1.0.6.0.99.0-38-0e7f6337a50 Nitpicking: add a comment here that this is a custom Hive build -- To view, visit http://gerrit.cloudera.org:8080/13005 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436 Gerrit-Change-Number: 13005 Gerrit-PatchSet: 19 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 30 Apr 2019 05:12:40 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 ) Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0 .. Patch Set 19: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/13005 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436 Gerrit-Change-Number: 13005 Gerrit-PatchSet: 19 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 30 Apr 2019 04:31:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8454 (part 3): enable recursive file listing by default
Todd Lipcon has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13127 ) Change subject: IMPALA-8454 (part 3): enable recursive file listing by default .. IMPALA-8454 (part 3): enable recursive file listing by default This enables recursive listing of files within partition directories by default. This is a behavior change, but in fact makes Impala consistent with modern versions of Hive, Spark, Presto, etc. In fact, this is necessary for querying certain Hive tables which have been written out by a query containing a UNION ALL clause if that query is executed by Tez (see HIVE-12812 for example). Technically, this is an incompatible change. Although it's unlikely people were relying on the non-recursive listing, this patch offers two escape hatches: - an individual table may be marked with the 'impala.disable.recursive.listing' property - impala may be globally configured with --recursively_list_partitions=false Given that we know this behavior is inconsistent with other SQL engines, and that there is no performance benefit to not recursing in the common case that there _are_ no subdirectories, I made the flag "hidden" and did not document the new table property. These are only "chicken bit" flags. Change-Id: Ib30e2bcaf820210f2faa8f159d1af2f947a4d0e8 Reviewed-on: http://gerrit.cloudera.org:8080/13127 Reviewed-by: Bharath Vissapragada Tested-by: Impala Public Jenkins --- M be/src/common/global-flags.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/catalog/FeFsTable.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java A tests/metadata/test_recursive_listing.py 8 files changed, 151 insertions(+), 2 deletions(-) Approvals: Bharath Vissapragada: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/13127 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ib30e2bcaf820210f2faa8f159d1af2f947a4d0e8 Gerrit-Change-Number: 13127 Gerrit-PatchSet: 4 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 ) Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0 .. Patch Set 19: (3 comments) I'm basically ready to approve this. One comment and a couple nitpicking things. http://gerrit.cloudera.org:8080/#/c/13005/19//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/13005/19//COMMIT_MSG@40 PS19, Line 40: Patch uses a custom build of Hive Curious: is this true? http://gerrit.cloudera.org:8080/#/c/13005/19/bin/impala-config.sh File bin/impala-config.sh: http://gerrit.cloudera.org:8080/#/c/13005/19/bin/impala-config.sh@a264 PS19, Line 264: Nitpicking: restore this http://gerrit.cloudera.org:8080/#/c/13005/19/fe/pom.xml File fe/pom.xml: http://gerrit.cloudera.org:8080/#/c/13005/19/fe/pom.xml@a316 PS19, Line 316: : : : What happened to the exclusions for calcite? Here and for hive-exec and hive-hbase-handler. Do we need them for the Hive 2 profile? -- To view, visit http://gerrit.cloudera.org:8080/13005 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436 Gerrit-Change-Number: 13005 Gerrit-PatchSet: 19 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 30 Apr 2019 04:35:30 + Gerrit-HasComments: Yes
[Impala-ASF-CR] fe: set classpath using maven dependency resolution
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13185 ) Change subject: fe: set classpath using maven dependency resolution .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2992/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13185 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I103a1da10a54c7525ba7fb584d942ba1cb9fcb94 Gerrit-Change-Number: 13185 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 30 Apr 2019 04:33:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8454 (part 3): enable recursive file listing by default
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13127 ) Change subject: IMPALA-8454 (part 3): enable recursive file listing by default .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13127 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib30e2bcaf820210f2faa8f159d1af2f947a4d0e8 Gerrit-Change-Number: 13127 Gerrit-PatchSet: 3 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 30 Apr 2019 04:25:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7971 (follow-up). Fix compilation error
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13184 ) Change subject: IMPALA-7971 (follow-up). Fix compilation error .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2991/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13184 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8071b421f0dcbae2d303bd9f1e1f6f64657c49c0 Gerrit-Change-Number: 13184 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 30 Apr 2019 04:13:00 + Gerrit-HasComments: No
[Impala-ASF-CR] fe: set classpath using maven dependency resolution
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13185 ) Change subject: fe: set classpath using maven dependency resolution .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4110/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/13185 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I103a1da10a54c7525ba7fb584d942ba1cb9fcb94 Gerrit-Change-Number: 13185 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 30 Apr 2019 04:12:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5351: Support storing column comment of kudu table
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/12977 ) Change subject: IMPALA-5351: Support storing column comment of kudu table .. Patch Set 9: The compilation error fix has been merged. You can rebase it again. Thanks for the patience! -- To view, visit http://gerrit.cloudera.org:8080/12977 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifb3b37eed364f12bdb3c1d7ef5be128f1475936c Gerrit-Change-Number: 12977 Gerrit-PatchSet: 9 Gerrit-Owner: helifu Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: helifu Gerrit-Comment-Date: Tue, 30 Apr 2019 04:03:03 + Gerrit-HasComments: No
[Impala-ASF-CR] Revert "IMPALA-7971: Add support for insert events in event processor."
Fredy Wijaya has abandoned this change. ( http://gerrit.cloudera.org:8080/13182 ) Change subject: Revert "IMPALA-7971: Add support for insert events in event processor." .. Abandoned Abandoning this CR since the fix has been merged instead: https://gerrit.cloudera.org/c/13184/ -- To view, visit http://gerrit.cloudera.org:8080/13182 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: Iec886d5d0d66320e89f12a5032dbea1b1cca92b3 Gerrit-Change-Number: 13182 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] fe: set classpath using maven dependency resolution
Hello Tim Armstrong, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/13185 to review the following change. Change subject: fe: set classpath using maven dependency resolution .. fe: set classpath using maven dependency resolution This changes the FE pom to generate a build classpath file in the target/ directory. Then, bin/set-classpath.sh uses this file to generate the classpath to start the cluster. This replaces the former approach of including all of the jars found in target/dependency/ The advantage of this is that a clean build is no longer required when switching artifact versions. Prior to this patch, if you changed an artifact version and rebuilt, both the old and new artifact would be left in the target/dependency/ directory and pollute the classpath. This doesn't fully remove the target/dependency/ directory, because its existence is likely important for downstream packaging of Impala. We can likely assume that such packaging always does a clean build. This also changes the set-classpath script to no longer load jars from testdata/target/dependency/ since it appears that directory doesn't actually get created during the build. Change-Id: I103a1da10a54c7525ba7fb584d942ba1cb9fcb94 --- M bin/set-classpath.sh M docker/setup_build_context.py M fe/pom.xml 3 files changed, 29 insertions(+), 17 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/85/13185/1 -- To view, visit http://gerrit.cloudera.org:8080/13185 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I103a1da10a54c7525ba7fb584d942ba1cb9fcb94 Gerrit-Change-Number: 13185 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7971 (follow-up). Fix compilation error
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/13184 ) Change subject: IMPALA-7971 (follow-up). Fix compilation error .. Patch Set 1: Verified+1 Code-Review+2 Skipping GVO to unbreak the master build. -- To view, visit http://gerrit.cloudera.org:8080/13184 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8071b421f0dcbae2d303bd9f1e1f6f64657c49c0 Gerrit-Change-Number: 13184 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 30 Apr 2019 03:15:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7971 (follow-up). Fix compilation error
Todd Lipcon has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13184 ) Change subject: IMPALA-7971 (follow-up). Fix compilation error .. IMPALA-7971 (follow-up). Fix compilation error 15a33d1ba was committed at about the same time as 5ced9160bd6, which renamed one of the methods of the FileDescriptor class, causing a compilation error. This follow-up mixes the semantic conflict. Tested tests/custom_cluster/test_event_processing.py manually. Change-Id: I8071b421f0dcbae2d303bd9f1e1f6f64657c49c0 Reviewed-on: http://gerrit.cloudera.org:8080/13184 Reviewed-by: Todd Lipcon Tested-by: Todd Lipcon --- M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: Todd Lipcon: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/13184 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I8071b421f0dcbae2d303bd9f1e1f6f64657c49c0 Gerrit-Change-Number: 13184 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-7971 (follow-up). Fix compilation error
Todd Lipcon has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13184 Change subject: IMPALA-7971 (follow-up). Fix compilation error .. IMPALA-7971 (follow-up). Fix compilation error 15a33d1ba was committed at about the same time as 5ced9160bd6, which renamed one of the methods of the FileDescriptor class, causing a compilation error. This follow-up mixes the semantic conflict. Tested tests/custom_cluster/test_event_processing.py manually. Change-Id: I8071b421f0dcbae2d303bd9f1e1f6f64657c49c0 --- M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/84/13184/1 -- To view, visit http://gerrit.cloudera.org:8080/13184 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I8071b421f0dcbae2d303bd9f1e1f6f64657c49c0 Gerrit-Change-Number: 13184 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon
[Impala-ASF-CR] IMPALA-5351: Support storing column comment of kudu table
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/12977 ) Change subject: IMPALA-5351: Support storing column comment of kudu table .. Patch Set 9: > Patch Set 9: > > It seems there is something wrong with hdfs package, below is my local > compilation error: > > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-compiler-plugin:3.3:compile (default-compile) > on project impala-frontend: Compilation failure > [ERROR] > /mnt/ceph/impala/fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java:[868,51] > cannot find symbol > [ERROR] symbol: method getFileName() > > By the way, how to quote my reply? :) Yeah we're having a compilation error right due to 2 concurrent merges. There's a CR to fix it: https://gerrit.cloudera.org/c/13182/. It should be fixed in few hours as soon as the CR is merged. You may have to rebase again. Sorry :( You can just click on the Reply button in my reply in Gerrit. -- To view, visit http://gerrit.cloudera.org:8080/12977 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifb3b37eed364f12bdb3c1d7ef5be128f1475936c Gerrit-Change-Number: 12977 Gerrit-PatchSet: 9 Gerrit-Owner: helifu Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: helifu Gerrit-Comment-Date: Tue, 30 Apr 2019 03:08:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 ) Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0 .. Patch Set 19: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2990/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13005 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436 Gerrit-Change-Number: 13005 Gerrit-PatchSet: 19 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 30 Apr 2019 02:48:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5351: Support storing column comment of kudu table
helifu has posted comments on this change. ( http://gerrit.cloudera.org:8080/12977 ) Change subject: IMPALA-5351: Support storing column comment of kudu table .. Patch Set 9: It seems there is something wrong with hdfs package, below is my local compilation error: [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.3:compile (default-compile) on project impala-frontend: Compilation failure [ERROR] /mnt/ceph/impala/fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java:[868,51] cannot find symbol [ERROR] symbol: method getFileName() By the way, how to quote my reply? :) -- To view, visit http://gerrit.cloudera.org:8080/12977 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifb3b37eed364f12bdb3c1d7ef5be128f1475936c Gerrit-Change-Number: 12977 Gerrit-PatchSet: 9 Gerrit-Owner: helifu Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: helifu Gerrit-Comment-Date: Tue, 30 Apr 2019 02:47:45 + Gerrit-HasComments: No
[Impala-ASF-CR] Revert "IMPALA-7971: Add support for insert events in event processor."
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/13182 ) Change subject: Revert "IMPALA-7971: Add support for insert events in event processor." .. Patch Set 1: Maybe we should just substitute 'getRelativePath' -> 'getFileName' in the added code to fix the compile error? I'll put up a gerrit for that. -- To view, visit http://gerrit.cloudera.org:8080/13182 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iec886d5d0d66320e89f12a5032dbea1b1cca92b3 Gerrit-Change-Number: 13182 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 30 Apr 2019 02:47:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 ) Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0 .. Patch Set 18: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/2989/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13005 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436 Gerrit-Change-Number: 13005 Gerrit-PatchSet: 18 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 30 Apr 2019 01:44:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/13005 )
Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive
3.1.0
..
Patch Set 19:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/13005/19/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/13005/19/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@283
PS19, Line 283: if (currentChar == '\\' && (i + 6 <
stringLiteral.length()) && stringLiteral.charAt(i + 1) == 'u') {
line too long (108 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 19
Gerrit-Owner: Vihang Karajgaonkar
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Philip Zeyliger
Gerrit-Reviewer: Sudhanshu Arora
Gerrit-Reviewer: Todd Lipcon
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Tue, 30 Apr 2019 02:02:37 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
Vihang Karajgaonkar has uploaded a new patch set (#19). ( http://gerrit.cloudera.org:8080/13005 ) Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0 .. IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0 This change adds a compatibility shim in fe so that Impala can interoperate with Hive 3.1.0. It moves the existing Metastoreshim class to a compat-hive-2 directory and adds a new Metastoreshim class under compat-hive-3 directory. These shim classes implement method which are different in hive-2 v/s hive-3 and are used by front end code. At the build time, based on the environment variable IMPALA_HIVE_MAJOR_VERSION one of the two shims is added to as source using the fe/pom.xml build plugin. Additionally, in order to reduce the dependencies footprint of Hive in the front end code, this patch also introduces a new module called shaded-deps. This module using shade plugin to include only the source files from hive-exec which are need by the fe code. For hive-2 build path, no changes are done with respect to hive dependencies to minimize the risk of destabilizing the master branch on the default build option of using Hive-2. The different set of dependencies are activated using maven profiles. The activation of each profile is automatic based on the IMPALA_HIVE_MAJOR_VERSION. Testing: 1. Code compiles and runs against both HMS-3 and HMS-2 2. Ran full-suite of tests using the private jenkins job against HMS-2 3. Running full-tests against HMS-3 will need more work like supporting Tez in the mini-cluster (for dataloading) and HMS transaction support since HMS3 create transactional tables by default. THis will be on-going effort and test failures on Hive-3 will be fixed in additional sub-tasks. Notes: 1. Patch uses a custom build of Hive to be deployed in mini-cluster. This build has the fixes for HIVE-21596. This hack will be removed when the patches are available in official CDP Hive builds. 2. Some of the existing tests rely on the fact the UDFs implement the UDF interface in Hive (UDFLength, UDFHour, UDFYear). These built-in hive functions have been moved to use GenericUDF interface in Hive 3. Impala currently only supports UDFExecutor. In order to have a full compatibility with all the functions in Hive 2.x we should support GenericUDFs too. That would be taken up as a separate patch. 3. Sentry dependencies bring a lot of transitive hive dependencies. The patch excludes such dependencies since they create problems while building against Hive-3. Since these hive-2 dependencies are already included when building against hive-2 this should not be a problem. Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436 --- M CMakeLists.txt M README.md M bin/bootstrap_toolchain.py M bin/impala-config.sh M common/thrift/.gitignore M common/thrift/CMakeLists.txt M fe/CMakeLists.txt M fe/pom.xml A fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java A fe/src/compat-hive-3/java/org/apache/impala/compat/HiveMetadataFormatUtils.java A fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/analysis/StringLiteral.java M fe/src/main/java/org/apache/impala/catalog/FeHBaseTable.java M fe/src/main/java/org/apache/impala/catalog/TableLoader.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java D fe/src/main/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/hive/executor/UdfExecutor.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java M fe/src/main/java/org/apache/impala/service/MetadataOp.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java M fe/src/test/java/org/apache/impala/hive/executor/UdfExecutorTest.java M fe/src/test/java/org/apache/impala/testutil/EmbeddedMetastoreClientPool.java M impala-parent/pom.xml A shaded-deps/.gitignore A shaded-deps/CMakeLists.txt A shaded-deps/pom.xml M testdata/bin/run-hive-server.sh M tests/custom_cluster/test_permanent_udfs.py 31 files changed, 1,777 insertions(+), 457 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/13005/19 -- To view, visit http://gerrit.cloudera.org:8080/13005 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436 Gerrit-Change-Number: 13005 Gerrit-PatchSet: 19 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer:
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13111 ) Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events. .. Patch Set 6: (8 comments) http://gerrit.cloudera.org:8080/#/c/13111/6/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java: http://gerrit.cloudera.org:8080/#/c/13111/6/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2113 PS6, Line 2113: Throws CatalogException if partition reload is unsuccessful. Throws :* DatabaseNotFoundException if Db doesn't exist We should use @throws CatalogException and @throws DatabaseNotFoundException javadoc instead. http://gerrit.cloudera.org:8080/#/c/13111/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java: http://gerrit.cloudera.org:8080/#/c/13111/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1252 PS6, Line 1252: /** nit: add a new line after L1251 http://gerrit.cloudera.org:8080/#/c/13111/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1255 PS6, Line 1255: getTPartSpecFromHmsPart I think it's better to not shorten partition as part since it can be confusing. http://gerrit.cloudera.org:8080/#/c/13111/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1272 PS6, Line 1272: constructPartStringFromTpart nit: constructPartitionStringFromTPartition http://gerrit.cloudera.org:8080/#/c/13111/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1288 PS6, Line 1288: List addedPartitions_; can this be private final? http://gerrit.cloudera.org:8080/#/c/13111/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1463 PS6, Line 1463: private final List> droppedPartitions_; nit: add a new empty line after this variable declaration http://gerrit.cloudera.org:8080/#/c/13111/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1523 PS6, Line 1523: a typo: an http://gerrit.cloudera.org:8080/#/c/13111/6/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java: http://gerrit.cloudera.org:8080/#/c/13111/6/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@1573 PS6, Line 1573: Collection partsAfterAdd = We don't have to do it in this CR, but the method here is getting too big. We should have a separate helper method for each event. It' makes the code easier to read. -- To view, visit http://gerrit.cloudera.org:8080/13111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5 Gerrit-Change-Number: 13111 Gerrit-PatchSet: 6 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 30 Apr 2019 00:54:18 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/13005 )
Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive
3.1.0
..
Patch Set 18:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/13005/18/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/13005/18/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@283
PS18, Line 283: if (currentChar == '\\' && (i + 6 <
stringLiteral.length()) && stringLiteral.charAt(i + 1) == 'u') {
line too long (108 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 18
Gerrit-Owner: Vihang Karajgaonkar
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Philip Zeyliger
Gerrit-Reviewer: Sudhanshu Arora
Gerrit-Reviewer: Todd Lipcon
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Tue, 30 Apr 2019 01:29:27 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
Vihang Karajgaonkar has uploaded a new patch set (#18). ( http://gerrit.cloudera.org:8080/13005 ) Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0 .. IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0 This change adds a compatibility shim in fe so that Impala can interoperate with Hive 3.1.0. It moves the existing Metastoreshim class to a compat-hive-2 directory and adds a new Metastoreshim class under compat-hive-3 directory. These shim classes implement method which are different in hive-2 v/s hive-3 and are used by front end code. At the build time, based on the environment variable IMPALA_HIVE_MAJOR_VERSION one of the two shims is added to as source using the fe/pom.xml build plugin. Additionally, in order to reduce the dependencies footprint of Hive in the front end code, this patch also introduces a new module called shaded-deps. This module using shade plugin to include only the source files from hive-exec which are need by the fe code. For hive-2 build path, no changes are done with respect to hive dependencies to minimize the risk of destabilizing the master branch on the default build option of using Hive-2. The different set of dependencies are activated using maven profiles. The activation of each profile is automatic based on the IMPALA_HIVE_MAJOR_VERSION. Testing: 1. Code compiles and runs against both HMS-3 and HMS-2 2. Ran full-suite of tests using the private jenkins job against HMS-2 3. Running full-tests against HMS-3 will need more work like supporting Tez in the mini-cluster (for dataloading) and HMS transaction support since HMS3 create transactional tables by default. THis will be on-going effort and test failures on Hive-3 will be fixed in additional sub-tasks. Notes: 1. Patch uses a custom build of Hive to be deployed in mini-cluster. This build has the fixes for HIVE-21596. This hack will be removed when the patches are available in official CDP Hive builds. 2. Some of the existing tests rely on the fact the UDFs implement the UDF interface in Hive (UDFLength, UDFHour, UDFYear). These built-in hive functions have been moved to use GenericUDF interface in Hive 3. Impala currently only supports UDFExecutor. In order to have a full compatibility with all the functions in Hive 2.x we should support GenericUDFs too. That would be taken up as a separate patch. 3. Sentry dependencies bring a lot of transitive hive dependencies. The patch excludes such dependencies since they create problems while building against Hive-3. Since these hive-2 dependencies are already included when building against hive-2 this should not be a problem. Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436 --- M CMakeLists.txt M README.md M bin/bootstrap_toolchain.py M bin/impala-config.sh M common/thrift/.gitignore M common/thrift/CMakeLists.txt M fe/CMakeLists.txt M fe/pom.xml A fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java A fe/src/compat-hive-3/java/org/apache/impala/compat/HiveMetadataFormatUtils.java A fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/analysis/StringLiteral.java M fe/src/main/java/org/apache/impala/catalog/FeHBaseTable.java M fe/src/main/java/org/apache/impala/catalog/TableLoader.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java D fe/src/main/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/hive/executor/UdfExecutor.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java M fe/src/main/java/org/apache/impala/service/MetadataOp.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java M fe/src/test/java/org/apache/impala/hive/executor/UdfExecutorTest.java M fe/src/test/java/org/apache/impala/testutil/EmbeddedMetastoreClientPool.java M impala-parent/pom.xml A shaded-deps/.gitignore A shaded-deps/CMakeLists.txt A shaded-deps/pom.xml M testdata/bin/run-hive-server.sh M tests/custom_cluster/test_permanent_udfs.py 31 files changed, 1,777 insertions(+), 457 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/13005/18 -- To view, visit http://gerrit.cloudera.org:8080/13005 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436 Gerrit-Change-Number: 13005 Gerrit-PatchSet: 18 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer:
[Impala-ASF-CR] IMPALA-8463: fix skip.header.line.count on local catalog
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13131 ) Change subject: IMPALA-8463: fix skip.header.line.count on local catalog .. Patch Set 6: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4108/ -- To view, visit http://gerrit.cloudera.org:8080/13131 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3957b3661574c356a14197e8829abbed92028a1a Gerrit-Change-Number: 13131 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 30 Apr 2019 01:25:52 + Gerrit-HasComments: No
[Impala-ASF-CR] Revert "IMPALA-7971: Add support for insert events in event processor."
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13182 ) Change subject: Revert "IMPALA-7971: Add support for insert events in event processor." .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2987/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13182 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iec886d5d0d66320e89f12a5032dbea1b1cca92b3 Gerrit-Change-Number: 13182 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 30 Apr 2019 01:14:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 ) Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0 .. Patch Set 16: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/2988/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13005 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436 Gerrit-Change-Number: 13005 Gerrit-PatchSet: 16 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 30 Apr 2019 01:14:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
Vihang Karajgaonkar has uploaded a new patch set (#16). ( http://gerrit.cloudera.org:8080/13005 ) Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0 .. IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0 This change adds a compatibility shim in fe so that Impala can interoperate with Hive 3.1.0. It moves the existing Metastoreshim class to a compat-hive-2 directory and adds a new Metastoreshim class under compat-hive-3 directory. These shim classes implement method which are different in hive-2 v/s hive-3 and are used by front end code. At the build time, based on the environment variable IMPALA_HIVE_MAJOR_VERSION one of the two shims is added to as source using the fe/pom.xml build plugin. Additionally, in order to reduce the dependencies footprint of Hive in the front end code, this patch also introduces a new module called shaded-deps. This module using shade plugin to include only the source files from hive-exec which are need by the fe code. For hive-2 build path, no changes are done with respect to hive dependencies to minimize the risk of destabilizing the master branch on the default build option of using Hive-2. The different set of dependencies are activated using maven profiles. The activation of each profile is automatic based on the IMPALA_HIVE_MAJOR_VERSION. Testing: 1. Code compiles and runs against both HMS-3 and HMS-2 2. Ran full-suite of tests using the private jenkins job against HMS-2 3. Running full-tests against HMS-3 will need more work like supporting Tez in the mini-cluster (for dataloading) and HMS transaction support since HMS3 create transactional tables by default. THis will be on-going effort and test failures on Hive-3 will be fixed in additional sub-tasks. Notes: 1. Patch uses a custom build of Hive to be deployed in mini-cluster. This build has the fixes for HIVE-21596. This hack will be removed when the patches are available in official CDP Hive builds. 2. Some of the existing tests rely on the fact the UDFs implement the UDF interface in Hive (UDFLength, UDFHour, UDFYear). These built-in hive functions have been moved to use GenericUDF interface in Hive 3. Impala currently only supports UDFExecutor. In order to have a full compatibility with all the functions in Hive 2.x we should support GenericUDFs too. That would be taken up as a separate patch. 3. Sentry dependencies bring a lot of transitive hive dependencies. The patch excludes such dependencies since they create problems while building against Hive-3. Since these hive-2 dependencies are already included when building against hive-2 this should not be a problem. Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436 --- M CMakeLists.txt M README.md M bin/bootstrap_toolchain.py M bin/impala-config.sh M common/thrift/.gitignore M common/thrift/CMakeLists.txt M fe/CMakeLists.txt M fe/pom.xml A fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java A fe/src/compat-hive-3/java/org/apache/impala/compat/HiveMetadataFormatUtils.java A fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/analysis/StringLiteral.java M fe/src/main/java/org/apache/impala/catalog/FeHBaseTable.java M fe/src/main/java/org/apache/impala/catalog/TableLoader.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java D fe/src/main/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/hive/executor/UdfExecutor.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java M fe/src/main/java/org/apache/impala/service/MetadataOp.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java M fe/src/test/java/org/apache/impala/hive/executor/UdfExecutorTest.java M fe/src/test/java/org/apache/impala/testutil/EmbeddedMetastoreClientPool.java M impala-parent/pom.xml A shaded-deps/.gitignore A shaded-deps/CMakeLists.txt A shaded-deps/pom.xml M testdata/bin/run-hive-server.sh M tests/custom_cluster/test_permanent_udfs.py 31 files changed, 1,775 insertions(+), 455 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/13005/16 -- To view, visit http://gerrit.cloudera.org:8080/13005 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436 Gerrit-Change-Number: 13005 Gerrit-PatchSet: 16 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer:
[Impala-ASF-CR] IMPALA-8465: fix TestJsonEndpoints for remote clusters
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13136 ) Change subject: IMPALA-8465: fix TestJsonEndpoints for remote clusters .. Patch Set 4: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4107/ -- To view, visit http://gerrit.cloudera.org:8080/13136 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8cf6ecfb72ec18be0b5274e360825fb3f203dd20 Gerrit-Change-Number: 13136 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 30 Apr 2019 01:09:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-2990: timeout unresponsive queries in coordinator
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12299 ) Change subject: IMPALA-2990: timeout unresponsive queries in coordinator .. Patch Set 12: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12299 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I196c8c6a5633b1960e2c3a3884777be9b3824987 Gerrit-Change-Number: 12299 Gerrit-PatchSet: 12 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 30 Apr 2019 00:53:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/13005 )
Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive
3.1.0
..
Patch Set 16:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/13005/16/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/13005/16/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@283
PS16, Line 283: if (currentChar == '\\' && (i + 6 <
stringLiteral.length()) && stringLiteral.charAt(i + 1) == 'u') {
line too long (108 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 16
Gerrit-Owner: Vihang Karajgaonkar
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Philip Zeyliger
Gerrit-Reviewer: Sudhanshu Arora
Gerrit-Reviewer: Todd Lipcon
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Tue, 30 Apr 2019 01:04:53 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] Revert "IMPALA-7971: Add support for insert events in event processor."
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13182 ) Change subject: Revert "IMPALA-7971: Add support for insert events in event processor." .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4109/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13182 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iec886d5d0d66320e89f12a5032dbea1b1cca92b3 Gerrit-Change-Number: 13182 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Tue, 30 Apr 2019 00:53:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8469: admit mem limit for dedicated coordinator
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13180 ) Change subject: IMPALA-8469: admit_mem_limit for dedicated coordinator .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2986/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13180 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaceb21b753b9b021bedc4187c0d44aaa6a626521 Gerrit-Change-Number: 13180 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 30 Apr 2019 01:00:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-2990: timeout unresponsive queries in coordinator
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12299 ) Change subject: IMPALA-2990: timeout unresponsive queries in coordinator .. IMPALA-2990: timeout unresponsive queries in coordinator The coordinator currently waits indefinitely if it does not receive a status report from a backend. This could cause a query to hang indefinitely in certain situations, for example if the backend decides to cancel itself as a result of failed status report rpcs. This patch adds a thread to ImpalaServer which periodically iterates over all queries for which that server is the coordinator and cancels any that haven't had a report from a backend in a certain amount of time. This patch adds two flags: --status_report_max_retry_s: the maximum number of seconds a backend will attempt to send status reports before giving up. This is used in place of --status_report_max_retries which is now deprecated. --status_report_cancellation_padding: the coordinator will wait --status_report_max_retry_s * (1 + --status_report_cancellation_padding / 100) before concluding a backend is not responding and cancelling the query. Testing: - Added a functional test that runs a query that is cancelled through the new mechanism. - Passed a full set of exhaustive tests. Ran tests on a 10 node cluster loaded with tpch 500: - Ran the stress test for 1000 queries with the debug actions: 'REPORT_EXEC_STATUS_DELAY:JITTER@1000' Prior to this patch, this setup results in hanging queries. With this patch, no hangs were observed. - Ran perf tests with 4 concurrent streams, 3 iterations per query. Found no change in performance. Change-Id: I196c8c6a5633b1960e2c3a3884777be9b3824987 Reviewed-on: http://gerrit.cloudera.org:8080/12299 Reviewed-by: Thomas Marshall Tested-by: Impala Public Jenkins --- M be/src/common/global-flags.cc M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator-backend-state.h M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h M be/src/runtime/query-state.cc M be/src/runtime/query-state.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M common/thrift/ImpalaInternalService.thrift M common/thrift/generate_error_codes.py M tests/custom_cluster/test_rpc_timeout.py 12 files changed, 191 insertions(+), 47 deletions(-) Approvals: Thomas Marshall: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/12299 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I196c8c6a5633b1960e2c3a3884777be9b3824987 Gerrit-Change-Number: 12299 Gerrit-PatchSet: 13 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] Revert "IMPALA-7971: Add support for insert events in event processor."
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/13182 ) Change subject: Revert "IMPALA-7971: Add support for insert events in event processor." .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13182 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iec886d5d0d66320e89f12a5032dbea1b1cca92b3 Gerrit-Change-Number: 13182 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Tue, 30 Apr 2019 00:52:34 + Gerrit-HasComments: No
[Impala-ASF-CR] Revert "IMPALA-7971: Add support for insert events in event processor."
Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13182 Change subject: Revert "IMPALA-7971: Add support for insert events in event processor." .. Revert "IMPALA-7971: Add support for insert events in event processor." This reverts commit 15a33d1baaf4723794e52825a5c4082ff7487507. Change-Id: Iec886d5d0d66320e89f12a5032dbea1b1cca92b3 --- M be/src/service/client-request-state.cc M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java M fe/src/test/resources/hive-site.xml.py D tests/custom_cluster/test_event_processing.py 10 files changed, 13 insertions(+), 592 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/13182/1 -- To view, visit http://gerrit.cloudera.org:8080/13182 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Iec886d5d0d66320e89f12a5032dbea1b1cca92b3 Gerrit-Change-Number: 13182 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya
[Impala-ASF-CR] IMPALA-7665: Fix unwarranted query cancellation on statestore restart
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13061 ) Change subject: IMPALA-7665: Fix unwarranted query cancellation on statestore restart .. Patch Set 2: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/2985/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13061 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I30b68bd8bde4bf589d58d42d6f683afb166de959 Gerrit-Change-Number: 13061 Gerrit-PatchSet: 2 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 30 Apr 2019 00:49:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7665: Fix unwarranted query cancellation on statestore restart
Hello Lars Volker, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13061 to look at the new patch set (#2). Change subject: IMPALA-7665: Fix unwarranted query cancellation on statestore restart .. IMPALA-7665: Fix unwarranted query cancellation on statestore restart Currently, if the statestore restarts and disseminates an inconsistent view of cluster membership to the coordinators, then they might believe that the backends no longer in the membership update are down and would start canceling queries that are running or scheduled to run on those allegedly failed backends. This patch adds a grace period after statestore recovery/successful registration that give it enough time to gather a consistent state of the cluster. Testing: - Added an e2e test. - Did manual stress testing using concurrent_select.py with statestore_subscriber_timeout_seconds set to 2 secs and failed_backends_query_cancellation_grace_period_ms set to 5 seconds, and the statestore being restarted every 15 seconds. To avoid other effects fo statestore restarts cropping up, I used a local catalog (catalog v2) and ignored query errors caused due to scheduler having an incomplete view of the cluster(no backends). Change-Id: I30b68bd8bde4bf589d58d42d6f683afb166de959 --- M be/src/service/impala-server.cc M be/src/statestore/statestore-subscriber.cc M be/src/statestore/statestore-subscriber.h M tests/custom_cluster/test_restart_services.py 4 files changed, 89 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/61/13061/2 -- To view, visit http://gerrit.cloudera.org:8080/13061 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I30b68bd8bde4bf589d58d42d6f683afb166de959 Gerrit-Change-Number: 13061 Gerrit-PatchSet: 2 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7665: Fix unwarranted query cancellation on statestore restart
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13061 ) Change subject: IMPALA-7665: Fix unwarranted query cancellation on statestore restart .. Patch Set 2: (2 comments) http://gerrit.cloudera.org:8080/#/c/13061/2/tests/custom_cluster/test_restart_services.py File tests/custom_cluster/test_restart_services.py: http://gerrit.cloudera.org:8080/#/c/13061/2/tests/custom_cluster/test_restart_services.py@97 PS2, Line 97: . flake8: E131 continuation line unaligned for hanging indent http://gerrit.cloudera.org:8080/#/c/13061/2/tests/custom_cluster/test_restart_services.py@100 PS2, Line 100: t flake8: E122 continuation line missing indentation or outdented -- To view, visit http://gerrit.cloudera.org:8080/13061 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I30b68bd8bde4bf589d58d42d6f683afb166de959 Gerrit-Change-Number: 13061 Gerrit-PatchSet: 2 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 30 Apr 2019 00:27:03 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8293 (Part 2): Add support for Ranger cache invalidation
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13134 ) Change subject: IMPALA-8293 (Part 2): Add support for Ranger cache invalidation .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2984/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13134 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia7160c082298e0b8cc2742dd3facbd4978581288 Gerrit-Change-Number: 13134 Gerrit-PatchSet: 7 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 30 Apr 2019 00:25:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8469: admit mem limit for dedicated coordinator
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13180 ) Change subject: IMPALA-8469: admit_mem_limit for dedicated coordinator .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2983/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13180 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaceb21b753b9b021bedc4187c0d44aaa6a626521 Gerrit-Change-Number: 13180 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 30 Apr 2019 00:31:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8469: admit mem limit for dedicated coordinator
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13180 ) Change subject: IMPALA-8469: admit_mem_limit for dedicated coordinator .. Patch Set 6: Code-Review+1 Carry +1. Still will wait for exhaustive tests -- To view, visit http://gerrit.cloudera.org:8080/13180 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaceb21b753b9b021bedc4187c0d44aaa6a626521 Gerrit-Change-Number: 13180 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 30 Apr 2019 00:28:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8469: admit mem limit for dedicated coordinator
Hello Lars Volker, Bikramjeet Vig, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13180 to look at the new patch set (#6). Change subject: IMPALA-8469: admit_mem_limit for dedicated coordinator .. IMPALA-8469: admit_mem_limit for dedicated coordinator Refactored to avoid the code duplication that resulted in this bug: * admit_mem_limit is calculated once in ExecEnv * The local backend descriptor is always constructed with a static helper: Scheduler::BuildLocalBackendDescriptor() I chose to factor it in this way, in part, to avoid invasive changes to scheduler-test, which currently doesn't depend on ExecEnv or ImpalaServer. Testing: Added basic test that reproduces the bug. Change-Id: Iaceb21b753b9b021bedc4187c0d44aaa6a626521 --- M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/scheduling/scheduler-test-util.cc M be/src/scheduling/scheduler.cc M be/src/scheduling/scheduler.h M be/src/service/impala-server.cc M tests/common/custom_cluster_test_suite.py M tests/custom_cluster/test_admission_controller.py 8 files changed, 102 insertions(+), 55 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/80/13180/6 -- To view, visit http://gerrit.cloudera.org:8080/13180 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iaceb21b753b9b021bedc4187c0d44aaa6a626521 Gerrit-Change-Number: 13180 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7665: Fix unwarranted query cancellation on statestore restart
Bikramjeet Vig has posted comments on this change. (
http://gerrit.cloudera.org:8080/13061 )
Change subject: IMPALA-7665: Fix unwarranted query cancellation on statestore
restart
..
Patch Set 2:
(15 comments)
http://gerrit.cloudera.org:8080/#/c/13061/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/13061/1//COMMIT_MSG@17
PS1, Line 17: Testing:
> Just to confirm, this test failed before your change?
yup, fails after the first statestore restart
http://gerrit.cloudera.org:8080/#/c/13061/1/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
http://gerrit.cloudera.org:8080/#/c/13061/1/be/src/service/impala-server.cc@235
PS1, Line 235: running on backends
> running on
Done
http://gerrit.cloudera.org:8080/#/c/13061/1/be/src/service/impala-server.cc@236
PS1, Line 236: ould be large enough to g
> This value should be large enough to give the statestore...
Done
http://gerrit.cloudera.org:8080/#/c/13061/1/be/src/statestore/statestore-subscriber.h
File be/src/statestore/statestore-subscriber.h:
http://gerrit.cloudera.org:8080/#/c/13061/1/be/src/statestore/statestore-subscriber.h@130
PS1, Line 130: int64
> int64_t is the standard one, I guess int64 comes from some header we've pul
Done
http://gerrit.cloudera.org:8080/#/c/13061/1/be/src/statestore/statestore-subscriber.h@215
PS1, Line 215: AtomicInt64 last_registration_ms_{0};
> I think last_registration_ms_ would also convey that it's a timestamp, but
Done
http://gerrit.cloudera.org:8080/#/c/13061/1/tests/custom_cluster/test_restart_services.py
File tests/custom_cluster/test_restart_services.py:
http://gerrit.cloudera.org:8080/#/c/13061/1/tests/custom_cluster/test_restart_services.py@26
PS1, Line 26:
> flake8: F811 redefinition of unused 'time' from line 23
Done
http://gerrit.cloudera.org:8080/#/c/13061/1/tests/custom_cluster/test_restart_services.py@98
PS1, Line 98:
> flake8: E131 continuation line unaligned for hanging indent
Done
http://gerrit.cloudera.org:8080/#/c/13061/1/tests/custom_cluster/test_restart_services.py@100
PS1, Line 100: timeout_s=SUBSCRIBER_TIMEOUT_S))
> If you didn't already, can you loop this test for a while, maybe with 'stre
done. Also updated commit message accordingly
http://gerrit.cloudera.org:8080/#/c/13061/1/tests/custom_cluster/test_restart_services.py@103
PS1, Line 103: l not r
> nit: backend or impalad (lowercase d)
Done
http://gerrit.cloudera.org:8080/#/c/13061/1/tests/custom_cluster/test_restart_services.py@112
PS1, Line 112: sure query starts running.
> Does this make sure that the query is actually running, or is there a small
This would work too, but using wait_for_state would also help eliminate the
assert under this
http://gerrit.cloudera.org:8080/#/c/13061/1/tests/custom_cluster/test_restart_services.py@117
PS1, Line 117:
> Might be slightly cleaner to also default CANCELLATION_GRACE_PERIOD_S above
Done
http://gerrit.cloudera.org:8080/#/c/13061/1/tests/custom_cluster/test_restart_services.py@117
PS1, Line 117:
> flake8: E226 missing whitespace around arithmetic operator
Done
http://gerrit.cloudera.org:8080/#/c/13061/1/tests/custom_cluster/test_restart_services.py@119
PS1, Line 119: # Now restart statestore and kill a backend while it is
down, and make sure the
> We might need to reconcile this with IMPALA-2990, which adds another mechan
Yup, after IMPALA-2990 we can probably just skip cancelling running queries and
just reschedule the ones getting scheduled or in queue.
http://gerrit.cloudera.org:8080/#/c/13061/1/tests/custom_cluster/test_restart_services.py@122
PS1, Line 122: self.cluster.statestored.kill()
: self.cluster.impalads[1].kill()
: self.cluster.statestored.start()
> I think it would be interesting to test the case where the statestore comes
done.The second one is implicitly verified since the statestore is restarted 3
times in this test and the time elapsed is verified. Let me know if you think I
should add a separate test for that.
http://gerrit.cloudera.org:8080/#/c/13061/1/tests/custom_cluster/test_restart_services.py@130
PS1, Line 130:
> flake8: E226 missing whitespace around arithmetic operator
Done
--
To view, visit http://gerrit.cloudera.org:8080/13061
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I30b68bd8bde4bf589d58d42d6f683afb166de959
Gerrit-Change-Number: 13061
Gerrit-PatchSet: 2
Gerrit-Owner: Bikramjeet Vig
Gerrit-Reviewer: Bikramjeet Vig
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Tue, 30 Apr 2019 00:26:17 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7892 IMPALA-8416: [DOCS] Described the new network and disk info in query profiles
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13006 ) Change subject: IMPALA-7892 IMPALA-8416: [DOCS] Described the new network and disk info in query profiles .. Patch Set 3: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/308/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/13006 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I25b128bc23f418347b400ca9e694d9d591935592 Gerrit-Change-Number: 13006 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Tue, 30 Apr 2019 00:26:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7892 IMPALA-8416: [DOCS] Described the new network and disk info in query profiles
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13006 ) Change subject: IMPALA-7892 IMPALA-8416: [DOCS] Described the new network and disk info in query profiles .. Patch Set 3: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/308/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/13006 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I25b128bc23f418347b400ca9e694d9d591935592 Gerrit-Change-Number: 13006 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Tue, 30 Apr 2019 00:20:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7892 IMPALA-8416: [DOCS] Described the new network and disk info in query profiles
Hello Lars Volker, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13006 to look at the new patch set (#3). Change subject: IMPALA-7892 IMPALA-8416: [DOCS] Described the new network and disk info in query profiles .. IMPALA-7892 IMPALA-8416: [DOCS] Described the new network and disk info in query profiles - HostDiskReadThroughput - HostDiskWriteThroughput - HostNetworkRx - HostNetworkTx Change-Id: I25b128bc23f418347b400ca9e694d9d591935592 --- M docs/topics/impala_explain_plan.xml 1 file changed, 30 insertions(+), 13 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/06/13006/3 -- To view, visit http://gerrit.cloudera.org:8080/13006 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I25b128bc23f418347b400ca9e694d9d591935592 Gerrit-Change-Number: 13006 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker
[Impala-ASF-CR] IMPALA-8419 : Validate event processing related configurations
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13019 ) Change subject: IMPALA-8419 : Validate event processing related configurations .. Patch Set 14: > Patch Set 14: Verified-1 > > Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4105/ This code seems to have compilation error: 23:16:58 [ERROR] COMPILATION ERROR : 23:16:58 [ERROR] /home/ubuntu/Impala/fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java:[868,51] cannot find symbol 23:16:58 [INFO] BUILD FAILURE 23:16:58 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.3:compile (default-compile) on project impala-frontend: Compilation failure 23:16:58 [ERROR] /home/ubuntu/Impala/fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java:[868,51] cannot find symbol 23:16:58 [ERROR] symbol: method getFileName() 23:16:58 [ERROR] location: variable fd of type org.apache.impala.catalog.HdfsPartition.FileDescriptor 23:16:58 [ERROR] -> [Help 1] 23:16:58 [ERROR] 23:16:58 [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. 23:16:58 [ERROR] Re-run Maven using the -X switch to enable full debug logging. 23:16:58 [ERROR] 23:16:58 [ERROR] For more information about the errors and possible solutions, please read the following articles: 23:16:58 [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException -- To view, visit http://gerrit.cloudera.org:8080/13019 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I94c2783e36287a65122003aa55d8075a806bc606 Gerrit-Change-Number: 13019 Gerrit-PatchSet: 14 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 30 Apr 2019 00:17:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8280, IMPALA-8281: Add support for show grant user/group with Ranger
Fredy Wijaya has posted comments on this change. (
http://gerrit.cloudera.org:8080/13074 )
Change subject: IMPALA-8280, IMPALA-8281: Add support for show grant user/group
with Ranger
..
Patch Set 7:
(20 comments)
http://gerrit.cloudera.org:8080/#/c/13074/7//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/13074/7//COMMIT_MSG@9
PS7, Line 9: Add support for SHOW GRANT statements for Apache Ranger. This
patch also
: adds the RangerImpaladAuthorizationManager as the show grant
statement
: is called from impalad.
mention the list of new syntax and also mention that "show grant" without "on"
is not supported.
http://gerrit.cloudera.org:8080/#/c/13074/7/fe/src/main/java/org/apache/impala/authorization/AuthorizationPolicy.java
File fe/src/main/java/org/apache/impala/authorization/AuthorizationPolicy.java:
http://gerrit.cloudera.org:8080/#/c/13074/7/fe/src/main/java/org/apache/impala/authorization/AuthorizationPolicy.java@526
PS7, Line 526: public
does this need to be public?
http://gerrit.cloudera.org:8080/#/c/13074/7/fe/src/main/java/org/apache/impala/authorization/AuthorizationPolicy.java@529
PS7, Line 529: new HashSet<>()
This is an unordered set, will this be a problem like the output of the row
will be indeterministic? Maybe we should use LinkedHashSet instead.
http://gerrit.cloudera.org:8080/#/c/13074/7/fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java
File
fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java:
http://gerrit.cloudera.org:8080/#/c/13074/7/fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java@260
PS7, Line 260: public static Map
createColumnResource(TPrivilege privilege) {
: Map resource = new HashMap<>();
:
: resource.put(RangerImpalaResourceBuilder.DATABASE,
getOrAll(privilege.getDb_name()));
: resource.put(RangerImpalaResourceBuilder.TABLE,
getOrAll(privilege.getTable_name()));
: resource.put(RangerImpalaResourceBuilder.COLUMN,
: getOrAll(privilege.getColumn_name()));
:
: return resource;
: }
:
: public static Map createUriResource(TPrivilege
privilege) {
: Map resource = new HashMap<>();
: String uri = privilege.getUri();
: resource.put(RangerImpalaResourceBuilder.URL, uri == null ?
"*" : uri);
:
: return resource;
: }
:
: public static Map
createFunctionResource(TPrivilege privilege) {
: Map resource = new HashMap<>();
:
: resource.put(RangerImpalaResourceBuilder.DATABASE,
getOrAll(privilege.getDb_name()));
: resource.put(RangerImpalaResourceBuilder.UDF, "*");
:
: return resource;
: }
can we make these private now?
http://gerrit.cloudera.org:8080/#/c/13074/7/fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java
File
fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java:
http://gerrit.cloudera.org:8080/#/c/13074/7/fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java@162
PS7, Line 162: Sets.newHashSet(ugi.getGroupNames())
let's try to use the one from JDK instead, i.e. new
HashSet<>(ugi.getGroupNames())
http://gerrit.cloudera.org:8080/#/c/13074/7/fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java@236
PS7, Line 236: switch (privilege.getScope()) {
: case COLUMN:
: if (!column.isPresent() || column.get().equals("*"))
return null;
: case TABLE:
: if (!table.isPresent() || table.get().equals("*"))
return null;
: case DATABASE:
: if (!database.isPresent() || database.get().equals("*"))
return null;
: break;
: case URI:
: if (!uri.isPresent() || uri.get().equals("*")) return
null;
: break;
: }
add default case to to make the compiler happy
http://gerrit.cloudera.org:8080/#/c/13074/7/fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java@259
PS7, Line 259: else
since L257 is sa throw. We can make it just an if instead of else if.
http://gerrit.cloudera.org:8080/#/c/13074/7/fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java@276
PS7, Line 276: Map tmpResource = new HashMap<>(resource);
instead of making a copy, we should just make List>
resources
[Impala-ASF-CR] IMPALA-8419 : Validate event processing related configurations
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13019 ) Change subject: IMPALA-8419 : Validate event processing related configurations .. Patch Set 14: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4105/ -- To view, visit http://gerrit.cloudera.org:8080/13019 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I94c2783e36287a65122003aa55d8075a806bc606 Gerrit-Change-Number: 13019 Gerrit-PatchSet: 14 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 30 Apr 2019 00:04:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8293 (Part 2): Add support for Ranger cache invalidation
Fredy Wijaya has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/13134 ) Change subject: IMPALA-8293 (Part 2): Add support for Ranger cache invalidation .. IMPALA-8293 (Part 2): Add support for Ranger cache invalidation This patch adds support for Ranger cache invalidation via INVALIDATE METADATA and REFRESH AUTHORIZATION. This patch introduces a new catalog object type called AUTHZ_REFRESH to allow broadcasting messages from Catalogd to Impalads to update their local Ranger caches. For better user experience, every GRANT/REVOKE statement perform an authorization refresh. Testing: - Replaced the sleep in test_ranger.py with INVALIDATE METADATA or REFRESH AUTHORIZATION - Ran all FE tests - Ran all E2E authorization tests Change-Id: Ia7160c082298e0b8cc2742dd3facbd4978581288 --- M be/src/catalog/catalog-util.cc M common/thrift/CatalogObjects.thrift M fe/src/main/java/org/apache/impala/authorization/AuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/NoopAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationChecker.java A fe/src/main/java/org/apache/impala/catalog/AuthzCacheInvalidation.java M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java M fe/src/main/java/org/apache/impala/service/FeCatalogManager.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java M fe/src/test/java/org/apache/impala/testutil/ImpaladTestCatalog.java M fe/src/test/resources/ranger-hive-security.xml M tests/authorization/test_ranger.py 20 files changed, 378 insertions(+), 97 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/34/13134/7 -- To view, visit http://gerrit.cloudera.org:8080/13134 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia7160c082298e0b8cc2742dd3facbd4978581288 Gerrit-Change-Number: 13134 Gerrit-PatchSet: 7 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8465: fix TestJsonEndpoints for remote clusters
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13136 ) Change subject: IMPALA-8465: fix TestJsonEndpoints for remote clusters .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4107/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13136 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8cf6ecfb72ec18be0b5274e360825fb3f203dd20 Gerrit-Change-Number: 13136 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 29 Apr 2019 23:47:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8469: admit mem limit for dedicated coordinator
Tim Armstrong has posted comments on this change. (
http://gerrit.cloudera.org:8080/13180 )
Change subject: IMPALA-8469: admit_mem_limit for dedicated coordinator
..
Patch Set 5:
(1 comment)
I'm still running exhaustive tests on this one but initial tests succeeded.
http://gerrit.cloudera.org:8080/#/c/13180/3/tests/custom_cluster/test_admission_controller.py
File tests/custom_cluster/test_admission_controller.py:
http://gerrit.cloudera.org:8080/#/c/13180/3/tests/custom_cluster/test_admission_controller.py@507
PS3, Line 507: ("Rejected query from pool default-pool: request memory needed "
: "1.10 GB per node is greater than memory available
for admission 1.00 GB" in
: str(ex)), str
> ping
Done
--
To view, visit http://gerrit.cloudera.org:8080/13180
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaceb21b753b9b021bedc4187c0d44aaa6a626521
Gerrit-Change-Number: 13180
Gerrit-PatchSet: 5
Gerrit-Owner: Tim Armstrong
Gerrit-Reviewer: Bikramjeet Vig
Gerrit-Reviewer: Lars Volker
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Mon, 29 Apr 2019 23:46:43 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8463: fix skip.header.line.count on local catalog
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13131 ) Change subject: IMPALA-8463: fix skip.header.line.count on local catalog .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13131 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3957b3661574c356a14197e8829abbed92028a1a Gerrit-Change-Number: 13131 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 29 Apr 2019 23:47:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8463: fix skip.header.line.count on local catalog
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13131 ) Change subject: IMPALA-8463: fix skip.header.line.count on local catalog .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4108/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13131 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3957b3661574c356a14197e8829abbed92028a1a Gerrit-Change-Number: 13131 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 29 Apr 2019 23:47:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8465: fix TestJsonEndpoints for remote clusters
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13136 ) Change subject: IMPALA-8465: fix TestJsonEndpoints for remote clusters .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13136 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8cf6ecfb72ec18be0b5274e360825fb3f203dd20 Gerrit-Change-Number: 13136 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 29 Apr 2019 23:47:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8469: admit mem limit for dedicated coordinator
Hello Lars Volker, Bikramjeet Vig, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13180 to look at the new patch set (#5). Change subject: IMPALA-8469: admit_mem_limit for dedicated coordinator .. IMPALA-8469: admit_mem_limit for dedicated coordinator Refactored to avoid the code duplication that resulted in this bug: * admit_mem_limit is calculated once in ExecEnv * The local backend descriptor is always constructed with a static helper: Scheduler::BuildLocalBackendDescriptor() I chose to factor it in this way, in part, to avoid invasive changes to scheduler-test, which currently doesn't depend on ExecEnv or ImpalaServer. Testing: Added basic test that reproduces the bug. Change-Id: Iaceb21b753b9b021bedc4187c0d44aaa6a626521 --- M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/scheduling/scheduler-test-util.cc M be/src/scheduling/scheduler.cc M be/src/scheduling/scheduler.h M be/src/service/impala-server.cc M tests/common/custom_cluster_test_suite.py M tests/custom_cluster/test_admission_controller.py 8 files changed, 102 insertions(+), 55 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/80/13180/5 -- To view, visit http://gerrit.cloudera.org:8080/13180 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iaceb21b753b9b021bedc4187c0d44aaa6a626521 Gerrit-Change-Number: 13180 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8463: fix skip.header.line.count on local catalog
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/13131 ) Change subject: IMPALA-8463: fix skip.header.line.count on local catalog .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13131 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3957b3661574c356a14197e8829abbed92028a1a Gerrit-Change-Number: 13131 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 29 Apr 2019 23:46:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8467: ParquetPlainEncoder::Decode leads to multiple test failures in ASAN builds
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13176 ) Change subject: IMPALA-8467: ParquetPlainEncoder::Decode leads to multiple test failures in ASAN builds .. IMPALA-8467: ParquetPlainEncoder::Decode leads to multiple test failures in ASAN builds Fixed the buffer overflow failure. Testing: All BE tests passed with ASAN. Change-Id: I2ac77f5f135f063bf6b8f7406d5b41535190d5a2 Reviewed-on: http://gerrit.cloudera.org:8080/13176 Reviewed-by: Csaba Ringhofer Tested-by: Impala Public Jenkins --- M be/src/exec/parquet/parquet-common.h 1 file changed, 13 insertions(+), 1 deletion(-) Approvals: Csaba Ringhofer: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/13176 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I2ac77f5f135f063bf6b8f7406d5b41535190d5a2 Gerrit-Change-Number: 13176 Gerrit-PatchSet: 4 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8467: ParquetPlainEncoder::Decode leads to multiple test failures in ASAN builds
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13176 ) Change subject: IMPALA-8467: ParquetPlainEncoder::Decode leads to multiple test failures in ASAN builds .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13176 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2ac77f5f135f063bf6b8f7406d5b41535190d5a2 Gerrit-Change-Number: 13176 Gerrit-PatchSet: 3 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 29 Apr 2019 23:41:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8419 : Validate event processing related configurations
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13019 ) Change subject: IMPALA-8419 : Validate event processing related configurations .. Patch Set 14: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/2982/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13019 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I94c2783e36287a65122003aa55d8075a806bc606 Gerrit-Change-Number: 13019 Gerrit-PatchSet: 14 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 29 Apr 2019 23:35:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 ) Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0 .. Patch Set 12: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13005 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436 Gerrit-Change-Number: 13005 Gerrit-PatchSet: 12 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 29 Apr 2019 23:24:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13111 ) Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events. .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2981/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5 Gerrit-Change-Number: 13111 Gerrit-PatchSet: 6 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 29 Apr 2019 23:28:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8466: disable test caching ddl in dockerized cluster
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13181 ) Change subject: IMPALA-8466: disable test_caching_ddl in dockerized cluster .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2980/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13181 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If4e67cf7f5d993c15cb5e1114beb099d0dfdc325 Gerrit-Change-Number: 13181 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 29 Apr 2019 23:26:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8341: Data cache for remote reads
Todd Lipcon has posted comments on this change. (
http://gerrit.cloudera.org:8080/12987 )
Change subject: IMPALA-8341: Data cache for remote reads
..
Patch Set 6:
(24 comments)
http://gerrit.cloudera.org:8080/#/c/12987/6/be/src/runtime/io/data-cache.h
File be/src/runtime/io/data-cache.h:
http://gerrit.cloudera.org:8080/#/c/12987/6/be/src/runtime/io/data-cache.h@112
PS6, Line 112: /// 'config' is the configuration string which specifies a
list of :
: /// tuples, delimited by comma.
per the commit message, we've moved to a single quota rather than per-directory
quotas, right? or is that a typo in the commit message?
http://gerrit.cloudera.org:8080/#/c/12987/6/be/src/runtime/io/data-cache.h@261
PS6, Line 261: int oldest_opened_file_ = 0;
perhaps init to -1?
http://gerrit.cloudera.org:8080/#/c/12987/6/be/src/runtime/io/data-cache.cc
File be/src/runtime/io/data-cache.cc:
http://gerrit.cloudera.org:8080/#/c/12987/6/be/src/runtime/io/data-cache.cc@64
PS6, Line 64: DEFINE_int64(data_cache_file_max_size, 4L << 40,
- can you add a comment here like /* 4TB */?
- have we tested that 4TB actually works in a long-running cluster? Now that
you have the deletion support in, maybe 1TB is a safer default if we're not
sure about full FS support?
- can you rename to _max_size_bytes?
http://gerrit.cloudera.org:8080/#/c/12987/6/be/src/runtime/io/data-cache.cc@68
PS6, Line 68: "(Advanced) The maximum number of allowed opened files per
partition.");
Setting this per-partition creates a dependency between this and the number of
partitions. I think it would better to have this be a total, and then auto-set
the per-partition limit by dividing the capacity among the partitions.
Otherwise it's likely people will have to set this to keep fd limit in check,
right? Or do we generally assume that ulimit -n is boosted super high for
impala?
http://gerrit.cloudera.org:8080/#/c/12987/6/be/src/runtime/io/data-cache.cc@70
PS6, Line 70: "(Advanced) Number of concurrent threads allowed to insert
into the cache");
is this per-partition? should be, right?
http://gerrit.cloudera.org:8080/#/c/12987/6/be/src/runtime/io/data-cache.cc@117
PS6, Line 117: KUDU_RETURN_IF_ERROR(kudu::Env::Default()->NewRWFile(path,
_file->file_),
why not pass the RWFile into the CacheFile constructor vs creating an empty one
and callign NewRWFile here?
http://gerrit.cloudera.org:8080/#/c/12987/6/be/src/runtime/io/data-cache.cc@141
PS6, Line 141: kudu::Status status =
kudu::Env::Default()->DeleteFile(path_);
WARN_NOT_OK could be used here
(i think WARN is more appropriate than ERROR since no data is lost, etc)
http://gerrit.cloudera.org:8080/#/c/12987/6/be/src/runtime/io/data-cache.cc@153
PS6, Line 153: inline
'inline' here and elsewhere isn't necessary since you've defined them inline
inside the class anyway
http://gerrit.cloudera.org:8080/#/c/12987/6/be/src/runtime/io/data-cache.cc@174
PS6, Line 174: if (UNLIKELY(!file_)) return false;
worth a DCHECK that offset + bytes_to_read <= current_offset_
http://gerrit.cloudera.org:8080/#/c/12987/6/be/src/runtime/io/data-cache.cc@192
PS6, Line 192: kudu::Status status = file_->Write(offset, Slice(buffer,
buffer_len));
same DCHECK suggested above
http://gerrit.cloudera.org:8080/#/c/12987/6/be/src/runtime/io/data-cache.cc@205
PS6, Line 205: if (UNLIKELY(!file_)) return;
same
http://gerrit.cloudera.org:8080/#/c/12987/6/be/src/runtime/io/data-cache.cc@228
PS6, Line 228: used for synchronization
instead of just saying used for synchronozation" I think best to say "taken in
write mode during deletion, and shared mode everywhere else"
http://gerrit.cloudera.org:8080/#/c/12987/6/be/src/runtime/io/data-cache.cc@281
PS6, Line 281: Status DataCache::Partition::CreateCacheFile() {
can you DCHECK that lock_ is held by the current thread here? (same elsewhere
in functions that require the lock to be held on entry)
http://gerrit.cloudera.org:8080/#/c/12987/6/be/src/runtime/io/data-cache.cc@347
PS6, Line 347: KUDU_RETURN_IF_ERROR(env->GetFileSizeOnDisk(file->path(),
_sz),
: "CloseAndVerifyFileSizes()");
is this the right method call? seems the same as above
http://gerrit.cloudera.org:8080/#/c/12987/6/be/src/runtime/io/data-cache.cc@352
PS6, Line 352: resize(0);
.clear()
http://gerrit.cloudera.org:8080/#/c/12987/6/be/src/runtime/io/data-cache.cc@360
PS6, Line 360: void DataCache::Partition::Close() {
dcheck the lock is held?
http://gerrit.cloudera.org:8080/#/c/12987/6/be/src/runtime/io/data-cache.cc@380
PS6, Line 380: const CacheEntry* entry = reinterpret_cast(value_slice.data());
I think this pattern is used pretty widely in Impala but it's moderately
sketchy, since it assumes that value_slice's allocation has the same alignment
requirements as CacheEntry. That's likely but not really guaranteed by
anything, and we've had crashes in the past
[Impala-ASF-CR] IMPALA-8465: fix TestJsonEndpoints for remote clusters
David Knupp has posted comments on this change. ( http://gerrit.cloudera.org:8080/13136 ) Change subject: IMPALA-8465: fix TestJsonEndpoints for remote clusters .. Patch Set 3: Code-Review+2 Thanks for the quick fix! -- To view, visit http://gerrit.cloudera.org:8080/13136 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8cf6ecfb72ec18be0b5274e360825fb3f203dd20 Gerrit-Change-Number: 13136 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 29 Apr 2019 22:58:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
Joe McDonnell has posted comments on this change. (
http://gerrit.cloudera.org:8080/13005 )
Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive
3.1.0
..
Patch Set 14:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/13005/14/bin/impala-config.sh
File bin/impala-config.sh:
http://gerrit.cloudera.org:8080/#/c/13005/14/bin/impala-config.sh@a747
PS14, Line 747:
I think we should retain this line for now. Some build systems pass in
DOWNLOAD_CDH_COMPONENTS=false, and it is good to know about it.
http://gerrit.cloudera.org:8080/#/c/13005/14/bin/impala-config.sh@226
PS14, Line 226: export
HIVE_HOME="$IMPALA_TOOLCHAIN/cdh_components-${CDH_BUILD_NUMBER}/hive-\
: ${IMPALA_HIVE_VERSION}"
: export HIVE_METASTORE_THRIFT_DIR=$CDH_COMPONENTS_HOME/hive-\
: ${IMPALA_HIVE_VERSION}/src/metastore/if
Please retain a way to build with HIVE_SRC_DIR_OVERRIDE, as some build systems
rely on it to pass in a local Hive directory. It looks like it would have
different meanings for Hive 2 vs Hive 3, but it might be ok to assume Hive 2
has the version we expected before and Hive 3 is pointing at the top level now.
i.e.
Hive 2:
HIVE_METASTORE_THRIFT_DIR = HIVE_SRC_DIR/metastore/if
Hive 3:
HIVE_METASTORE_THRIFT_DIR = HIVE_SRC_DIR/standalone-metastore/src/main/thrift
Alternatively, we could limit HIVE_SRC_DIR_OVERRIDE to Hive 2 or use separate
variables for Hive 2 vs Hive 3, but both are going to be useful to build
systems. We could do something like a HIVE_METASTORE_THRIFT_DIR_OVERRIDE (but
we would still want to allow HIVE_SRC_DIR_OVERRIDE to allow an easy transition).
http://gerrit.cloudera.org:8080/#/c/13005/14/tests/custom_cluster/test_permanent_udfs.py
File tests/custom_cluster/test_permanent_udfs.py:
http://gerrit.cloudera.org:8080/#/c/13005/14/tests/custom_cluster/test_permanent_udfs.py@502
PS14, Line 502: # Sample java udfs from hive-exec.jar. Function name to
symbol class mapping
: SAMPLE_JAVA_UDFS = [
: ('udfpi', 'org.apache.hadoop.hive.ql.udf.UDFPI'),
: ('udfbin', 'org.apache.hadoop.hive.ql.udf.UDFBin'),
: ('udfhex', 'org.apache.hadoop.hive.ql.udf.UDFHex'),
: ('udfconv', 'org.apache.hadoop.hive.ql.udf.UDFConv'),
: # TODO UDFHour was moved from UDF to GenericUDF in Hive 3
: # Add support for GenericUDFs to handle such cases
: # ('udfhour', 'org.apache.hadoop.hive.ql.udf.UDFHour'),
: ('udflike', 'org.apache.hadoop.hive.ql.udf.UDFLike'),
: ('udfsign', 'org.apache.hadoop.hive.ql.udf.UDFSign'),
: # TODO UDFYear moved to GenericUDF in Hive 3
: # Add support for GenericUDFs
: # ('udfyear', 'org.apache.hadoop.hive.ql.udf.UDFYear'),
: ('udfascii','org.apache.hadoop.hive.ql.udf.UDFAscii')
: ]
If these UDF tests work on Hive 2, then I'd like to keep them enabled. Either
leave this unchanged (knowing that Hive 3 will need some modification later) or
look at the environment variable and tailor this to Hive 2 vs Hive 3.
--
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 14
Gerrit-Owner: Vihang Karajgaonkar
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Philip Zeyliger
Gerrit-Reviewer: Sudhanshu Arora
Gerrit-Reviewer: Todd Lipcon
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Mon, 29 Apr 2019 22:53:49 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 ) Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0 .. Patch Set 14: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2979/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13005 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436 Gerrit-Change-Number: 13005 Gerrit-PatchSet: 14 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 29 Apr 2019 22:51:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8419 : Validate event processing related configurations
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13019 ) Change subject: IMPALA-8419 : Validate event processing related configurations .. Patch Set 14: (7 comments) http://gerrit.cloudera.org:8080/#/c/13019/14/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java: http://gerrit.cloudera.org:8080/#/c/13019/14/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@20 PS14, Line 20: import static org.apache.impala.catalog.events.EventProcessorConfigValidator.hasValidMetastoreConfigs; line too long (102 > 90) http://gerrit.cloudera.org:8080/#/c/13019/14/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@21 PS14, Line 21: import static org.apache.impala.catalog.events.EventProcessorConfigValidator.verifyParametersNotFiltered; line too long (105 > 90) http://gerrit.cloudera.org:8080/#/c/13019/14/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java: http://gerrit.cloudera.org:8080/#/c/13019/14/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@21 PS14, Line 21: import static org.apache.impala.catalog.events.EventProcessorConfigValidator.DEFAULT_METASTORE_CONFIG_VALUE; line too long (108 > 90) http://gerrit.cloudera.org:8080/#/c/13019/14/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@22 PS14, Line 22: import static org.apache.impala.catalog.events.EventProcessorConfigValidator.METASTORE_PARAMETER_EXCLUDE_PATTERNS; line too long (114 > 90) http://gerrit.cloudera.org:8080/#/c/13019/14/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@23 PS14, Line 23: import static org.apache.impala.catalog.events.EventProcessorConfigValidator.validateMetastoreConfigs; line too long (102 > 90) http://gerrit.cloudera.org:8080/#/c/13019/14/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@24 PS14, Line 24: import static org.apache.impala.catalog.events.EventProcessorConfigValidator.validateMetastoreEventParameters; line too long (110 > 90) http://gerrit.cloudera.org:8080/#/c/13019/14/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@81 PS14, Line 81: import org.apache.impala.catalog.events.EventProcessorConfigValidator.MetastoreEventConfigsToValidate; line too long (102 > 90) -- To view, visit http://gerrit.cloudera.org:8080/13019 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I94c2783e36287a65122003aa55d8075a806bc606 Gerrit-Change-Number: 13019 Gerrit-PatchSet: 14 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 29 Apr 2019 22:40:29 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8419 : Validate event processing related configurations
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13019 ) Change subject: IMPALA-8419 : Validate event processing related configurations .. Patch Set 14: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4105/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/13019 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I94c2783e36287a65122003aa55d8075a806bc606 Gerrit-Change-Number: 13019 Gerrit-PatchSet: 14 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 29 Apr 2019 22:39:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8419 : Validate event processing related configurations
Bharath Krishna has uploaded a new patch set (#14). ( http://gerrit.cloudera.org:8080/13019 ) Change subject: IMPALA-8419 : Validate event processing related configurations .. IMPALA-8419 : Validate event processing related configurations Using the Metastore API to get the configuration values, verify that the configurations needed for event processing are set correctly. Also check that the parameters required for event processing is not filtered out by the Hive config METASTORE_PARAMETER_EXCLUDE_PATTERNS. This validation is done while creating the event processor and throws CatalogException if the configuration is incorrect. Testing - Added unit tests Change-Id: I94c2783e36287a65122003aa55d8075a806bc606 --- A fe/src/main/java/org/apache/impala/catalog/events/EventProcessorConfigValidator.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java M fe/src/test/java/org/apache/impala/catalog/events/SynchronousHMSEventProcessorForTests.java 7 files changed, 530 insertions(+), 67 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/13019/14 -- To view, visit http://gerrit.cloudera.org:8080/13019 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I94c2783e36287a65122003aa55d8075a806bc606 Gerrit-Change-Number: 13019 Gerrit-PatchSet: 14 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13111 ) Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events. .. Patch Set 5: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/2978/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5 Gerrit-Change-Number: 13111 Gerrit-PatchSet: 5 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 29 Apr 2019 22:23:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13111 ) Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events. .. Patch Set 6: Resolved merge conflicts with IMPALA-8454. -- To view, visit http://gerrit.cloudera.org:8080/13111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5 Gerrit-Change-Number: 13111 Gerrit-PatchSet: 6 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 29 Apr 2019 22:31:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Anurag Mantripragada has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/13111 ) Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events. .. IMPALA-7973: Add support for fine grained events processing for partition level HMS events. This patch adds support for fine grained updates for add/drop/alter partition events. Currently, partition events invalidate the table. This can be expensive for large tables. Here, we refresh affected partitions in case of add/drop/alter partition events. HMS processes add/drop partitions in a transaction, which means there may be multiple partitions affected in a single add/drop event. We try to refresh all these partitions in a loop. If any of the partition refresh fails, we throw MetastoreNotificationNeedsInvalidateException to mandate a manual invalidate for event processing to continue. Testing: Modified pre-existing tests for partition events to instead test if partitions are added/dropped/altered when event processing is enabled. Change-Id: I213401329f3965dd81055197792ccf8a05368af5 --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 4 files changed, 201 insertions(+), 45 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/11/13111/6 -- To view, visit http://gerrit.cloudera.org:8080/13111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5 Gerrit-Change-Number: 13111 Gerrit-PatchSet: 6 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8466: disable test caching ddl in dockerized cluster
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13181 Change subject: IMPALA-8466: disable test_caching_ddl in dockerized cluster .. IMPALA-8466: disable test_caching_ddl in dockerized cluster The test creates partitions with file:// URLs pointing at the host filesystem, which isn't accessible from within the containers. The only reason the test passed earlier was because of a bug fixed by the IMPALA-8454 patches which suppressed the error. Change-Id: If4e67cf7f5d993c15cb5e1114beb099d0dfdc325 --- M tests/query_test/test_hdfs_caching.py 1 file changed, 3 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/81/13181/1 -- To view, visit http://gerrit.cloudera.org:8080/13181 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: If4e67cf7f5d993c15cb5e1114beb099d0dfdc325 Gerrit-Change-Number: 13181 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] IMPALA-8468: buildall.sh should warn that asan/ubsan/... are exclusive
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13177 ) Change subject: IMPALA-8468: buildall.sh should warn that asan/ubsan/... are exclusive .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13177 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iecaf1e1987a92dd911b30cf11b6e762320bdc4e5 Gerrit-Change-Number: 13177 Gerrit-PatchSet: 3 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 29 Apr 2019 21:49:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/13005 )
Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive
3.1.0
..
Patch Set 14:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/13005/14/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/HiveShim.java:
http://gerrit.cloudera.org:8080/#/c/13005/14/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveShim.java@62
PS14, Line 62: if (currentChar == '\\' && (i + 6 <
stringLiteral.length()) && stringLiteral.charAt(i + 1) == 'u') {
line too long (108 > 90)
http://gerrit.cloudera.org:8080/#/c/13005/14/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/13005/14/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1072
PS14, Line 1072: MetastoreEventsProcessor.getMessageDeserializer()
line has trailing whitespace
--
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 14
Gerrit-Owner: Vihang Karajgaonkar
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Philip Zeyliger
Gerrit-Reviewer: Sudhanshu Arora
Gerrit-Reviewer: Todd Lipcon
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Mon, 29 Apr 2019 21:58:58 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
Vihang Karajgaonkar has uploaded a new patch set (#14). ( http://gerrit.cloudera.org:8080/13005 ) Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0 .. IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0 This change adds a compatibility shim in fe so that Impala can interoperate with Hive 3.1.0. It moves the existing Metastoreshim class to a compat-hive-2 directory and adds a new Metastoreshim class under compat-hive-3 directory. These shim classes implement method which are different in hive-2 v/s hive-3 and are used by front end code. At the build time, based on the environment variable IMPALA_HIVE_MAJOR_VERSION one of the two shims is added to as source using the fe/pom.xml build plugin. Additionally, in order to reduce the dependencies footprint of Hive in the front end code, this patch also introduces a new module called shaded-deps. This module using shade plugin to include only the source files from hive-exec which are need by the fe code. For hive-2 build path, no changes are done with respect to hive dependencies to minimize the risk of destabilizing the master branch on the default build option of using Hive-2. The different set of dependencies are activated using maven profiles. The activation of each profile is automatic based on the IMPALA_HIVE_MAJOR_VERSION. Testing: 1. Code compiles and runs against both HMS-3 and HMS-2 2. Ran full-suite of tests using the private jenkins job against HMS-2 3. Running full-tests against HMS-3 will need more work like supporting Tez in the mini-cluster (for dataloading) and HMS transaction support since HMS3 create transactional tables by default. THis will be on-going effort and test failures on Hive-3 will be fixed in additional sub-tasks. Notes: 1. Patch uses a custom build of Hive to be deployed in mini-cluster. This build has the fixes for HIVE-21596. This hack will be removed when the patches are available in official CDP Hive builds. 2. Some of the existing tests rely on the fact the UDFs implement the UDF interface in Hive (UDFLength, UDFHour, UDFYear). These built-in hive functions have been moved to use GenericUDF interface in Hive 3. Impala currently only supports UDFExecutor. In order to have a full compatibility with all the functions in Hive 2.x we should support GenericUDFs too. That would be taken up as a separate patch. 3. Sentry dependencies bring a lot of transitive hive dependencies. The patch excludes such dependencies since they create problems while building against Hive-3. Since these hive-2 dependencies are already included when building against hive-2 this should not be a problem. Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436 --- M CMakeLists.txt M README.md M bin/bootstrap_toolchain.py M bin/impala-config.sh M bin/set-classpath.sh M common/thrift/.gitignore M common/thrift/CMakeLists.txt M fe/CMakeLists.txt M fe/pom.xml A fe/src/compat-hive-2/java/org/apache/impala/compat/HiveShim.java A fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java A fe/src/compat-hive-3/java/org/apache/impala/compat/HiveMetadataFormatUtils.java A fe/src/compat-hive-3/java/org/apache/impala/compat/HiveShim.java A fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/analysis/StringLiteral.java M fe/src/main/java/org/apache/impala/catalog/FeHBaseTable.java M fe/src/main/java/org/apache/impala/catalog/TableLoader.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java D fe/src/main/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/hive/executor/UdfExecutor.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java M fe/src/main/java/org/apache/impala/service/MetadataOp.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java M fe/src/test/java/org/apache/impala/hive/executor/UdfExecutorTest.java M fe/src/test/java/org/apache/impala/testutil/EmbeddedMetastoreClientPool.java M impala-parent/pom.xml A shaded-deps/.gitignore A shaded-deps/CMakeLists.txt A shaded-deps/pom.xml M testdata/bin/run-hive-server.sh M tests/custom_cluster/test_permanent_udfs.py 34 files changed, 1,885 insertions(+), 465 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/13005/14 -- To view, visit http://gerrit.cloudera.org:8080/13005 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436 Gerrit-Change-Number: 13005 Gerrit-PatchSet: 14 Gerrit-Owner: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8293 (Part 2): Add support for Ranger cache invalidation
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13134 ) Change subject: IMPALA-8293 (Part 2): Add support for Ranger cache invalidation .. Patch Set 5: (11 comments) http://gerrit.cloudera.org:8080/#/c/13134/5//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/13134/5//COMMIT_MSG@11 PS5, Line 11: object type called AUTHZ_REFRESH to allow broadcasting messages from : Catalogd to Impalads to update their local Ranger caches. Can you mention the granularity of invalidation? Is everything refreshed for every grant/revoke? If so, it is a performance problem? http://gerrit.cloudera.org:8080/#/c/13134/5/common/thrift/CatalogObjects.thrift File common/thrift/CatalogObjects.thrift: http://gerrit.cloudera.org:8080/#/c/13134/5/common/thrift/CatalogObjects.thrift@42 PS5, Line 42: AUTHZ_REFRESH nit: something like AUTHZ_CACHE_INVALIDATION? Also, given this is a 'special' kind of Catalog object, document what it does? http://gerrit.cloudera.org:8080/#/c/13134/5/common/thrift/CatalogObjects.thrift@591 PS5, Line 591: TAuthzRefresh nit: same comment on naming. My point is that "refresh" is already confusing enough in Impala's context. http://gerrit.cloudera.org:8080/#/c/13134/5/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java File fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java: http://gerrit.cloudera.org:8080/#/c/13134/5/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java@179 PS5, Line 179: plugin_.refreshPoliciesAndTags(); thread-safe? What happens with authz requests in flight? http://gerrit.cloudera.org:8080/#/c/13134/5/fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java File fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java: http://gerrit.cloudera.org:8080/#/c/13134/5/fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java@225 PS5, Line 225: response.result.setRemoved_catalog_objects(authzDelta.getCatalogObjectsRemoved()); Isn't this a no-op? http://gerrit.cloudera.org:8080/#/c/13134/5/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java: http://gerrit.cloudera.org:8080/#/c/13134/5/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@689 PS5, Line 689: getAllAuthzRefreshes should we assert this of size 1? http://gerrit.cloudera.org:8080/#/c/13134/5/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2357 PS5, Line 2357: removeAuthzRefresh do we ever need to remove this? http://gerrit.cloudera.org:8080/#/c/13134/5/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2362 PS5, Line 2362: authzRefresh.setCatalogVersion(incrementAndGetCatalogVersion()); Is this needed? http://gerrit.cloudera.org:8080/#/c/13134/5/fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java File fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java: http://gerrit.cloudera.org:8080/#/c/13134/5/fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java@371 PS5, Line 371: case AUTHZ_REFRESH: : removeAuthzRefresh(catalogObject.getAuthz_refresh(), dropCatalogVersion); like I commented elsewhere, do we ever need to drop this? http://gerrit.cloudera.org:8080/#/c/13134/5/fe/src/main/java/org/apache/impala/service/Frontend.java File fe/src/main/java/org/apache/impala/service/Frontend.java: http://gerrit.cloudera.org:8080/#/c/13134/5/fe/src/main/java/org/apache/impala/service/Frontend.java@266 PS5, Line 266: catalogManager_.setAuthzChecker(authzChecker_); : authzManager_ = authzFactory.newAuthorizationManager(catalogManager_, : authzChecker_::get); Curious if AuthzChecker can be a part of AuthzManager? http://gerrit.cloudera.org:8080/#/c/13134/5/fe/src/test/resources/ranger-hive-security.xml File fe/src/test/resources/ranger-hive-security.xml: http://gerrit.cloudera.org:8080/#/c/13134/5/fe/src/test/resources/ranger-hive-security.xml@47 PS5, Line 47: 3 seems high, any particular reason to override? -- To view, visit http://gerrit.cloudera.org:8080/13134 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia7160c082298e0b8cc2742dd3facbd4978581288 Gerrit-Change-Number: 13134 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 29 Apr 2019 21:20:24 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8341: Data cache for remote reads
Lars Volker has posted comments on this change. (
http://gerrit.cloudera.org:8080/12987 )
Change subject: IMPALA-8341: Data cache for remote reads
..
Patch Set 5:
(17 comments)
http://gerrit.cloudera.org:8080/#/c/12987/5/be/src/runtime/io/data-cache-test.cc
File be/src/runtime/io/data-cache-test.cc:
http://gerrit.cloudera.org:8080/#/c/12987/5/be/src/runtime/io/data-cache-test.cc@273
PS5, Line 273: FLAGS_data_cache_file_max_size = 1024 * 1024;
I just found out we have ScopedFlagSetter in scoped-flag-setter.h, I think it
fits here and in the other tests.
http://gerrit.cloudera.org:8080/#/c/12987/5/be/src/runtime/io/data-cache.h
File be/src/runtime/io/data-cache.h:
http://gerrit.cloudera.org:8080/#/c/12987/5/be/src/runtime/io/data-cache.h@215
PS5, Line 215: too_many_files
'start_reclaim'?
http://gerrit.cloudera.org:8080/#/c/12987/5/be/src/runtime/io/data-cache.h@337
PS5, Line 337: std::unique_ptr> file_deleter_pool_;
Can you mention in the comment that the pool has only 1 thread and why you're
using a pool? I think it's because the pool makes handling the thread's
lifetime easier, but I'm not sure that's correct.
http://gerrit.cloudera.org:8080/#/c/12987/5/be/src/runtime/io/data-cache.h@341
PS5, Line 341: void CloseOldFiles(uint32_t thread_id, int partition_idx);
Some functions around deleting files are called "Close...". We should point out
in the comments somewhere that closing now also deletes. We could also rename
the thread pool to file_closing_pool or rename the methods to "DeleteOldFiles"
for consistency. I think I prefer the latter, since deletion implies closing,
but the contraposition is not obvious.
http://gerrit.cloudera.org:8080/#/c/12987/4/be/src/runtime/io/data-cache.cc
File be/src/runtime/io/data-cache.cc:
http://gerrit.cloudera.org:8080/#/c/12987/4/be/src/runtime/io/data-cache.cc@72
PS4, Line 72: "(Advanced) Enable checksumming for the cached buffer.");
> This is actually a static class member of DataCache.
Sry for missing that.
http://gerrit.cloudera.org:8080/#/c/12987/4/be/src/runtime/io/data-cache.cc@187
PS4, Line 187: inline
> Not sure which one you are referring to ? Isn't it in #include "common/name
Yeah, I think we commonly omit the explicit include for vector
http://gerrit.cloudera.org:8080/#/c/12987/5/be/src/runtime/io/data-cache.cc
File be/src/runtime/io/data-cache.cc:
http://gerrit.cloudera.org:8080/#/c/12987/5/be/src/runtime/io/data-cache.cc@95
PS5, Line 95: file deleter thread
switch to single thread, or mention pool here
http://gerrit.cloudera.org:8080/#/c/12987/5/be/src/runtime/io/data-cache.cc@112
PS5, Line 112: RetireFile
Can we call this DeleteFile? Otherwise there's a third thing to keep track of
(Close, Delete, Retire) and the differences are subtle. I feel it's clear
enough that DeleteFile would make sure it's closed.
http://gerrit.cloudera.org:8080/#/c/12987/5/be/src/runtime/io/data-cache.cc@125
PS5, Line 125: percpu_rwlock
It's not obvious to me why we only need a percpu_rwlock here. Can you add a
comment?
http://gerrit.cloudera.org:8080/#/c/12987/5/be/src/runtime/io/data-cache.cc@208
PS5, Line 208: holes
nit: singular
http://gerrit.cloudera.org:8080/#/c/12987/5/be/src/runtime/io/data-cache.cc@335
PS5, Line 335: CloseAndVerifyFileSizes
Similar to other comments, I'd call this "VerifySizeAndDeleteFiles", I think
that captures well what's going on and the caller can expect the files to get
closed. I don't feel strongly about that one though.
http://gerrit.cloudera.org:8080/#/c/12987/5/be/src/runtime/io/data-cache.cc@395
PS5, Line 395: meta_cache_->Erase(key);
Will this handle hole punching through the eviction logic?
http://gerrit.cloudera.org:8080/#/c/12987/5/be/src/runtime/io/data-cache.cc@436
PS5, Line 436: VLOG(2) << Substitute("Storing file $0 offset $1 len $2
checksum $3 ",
nit: only append the "checksum $3" part if checksumming is enabled? I don't
feel strongly about it though.
http://gerrit.cloudera.org:8080/#/c/12987/5/be/src/runtime/io/data-cache.cc@457
PS5, Line 457: too_many_files
start_reclaim?
http://gerrit.cloudera.org:8080/#/c/12987/5/be/src/runtime/io/data-cache.cc@633
PS5, Line 633: too_many_files
start_reclaim?
http://gerrit.cloudera.org:8080/#/c/12987/5/be/src/runtime/io/hdfs-file-reader.cc
File be/src/runtime/io/hdfs-file-reader.cc:
http://gerrit.cloudera.org:8080/#/c/12987/5/be/src/runtime/io/hdfs-file-reader.cc@37
PS5, Line 37:
nit: trailing space
http://gerrit.cloudera.org:8080/#/c/12987/5/tests/custom_cluster/test_data_cache.py
File tests/custom_cluster/test_data_cache.py:
http://gerrit.cloudera.org:8080/#/c/12987/5/tests/custom_cluster/test_data_cache.py@23
PS5, Line 23: cache hit and miss counts
: in the runtime profile are as expected.
It actually seems to check the metrics, not the profile counters.
--
To view, visit http://gerrit.cloudera.org:8080/12987
To unsubscribe, visit
[Impala-ASF-CR] IMPALA-8463: fix skip.header.line.count on local catalog
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13131 ) Change subject: IMPALA-8463: fix skip.header.line.count on local catalog .. Patch Set 5: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/13131 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3957b3661574c356a14197e8829abbed92028a1a Gerrit-Change-Number: 13131 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 29 Apr 2019 21:38:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13111 ) Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events. .. Patch Set 5: Rebased after insert events patch was merged. -- To view, visit http://gerrit.cloudera.org:8080/13111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5 Gerrit-Change-Number: 13111 Gerrit-PatchSet: 5 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 29 Apr 2019 21:37:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Anurag Mantripragada has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/13111 ) Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events. .. IMPALA-7973: Add support for fine grained events processing for partition level HMS events. This patch adds support for fine grained updates for add/drop/alter partition events. Currently, partition events invalidate the table. This can be expensive for large tables. Here, we refresh affected partitions in case of add/drop/alter partition events. HMS processes add/drop partitions in a transaction, which means there may be multiple partitions affected in a single add/drop event. We try to refresh all these partitions in a loop. If any of the partition refresh fails, we throw MetastoreNotificationNeedsInvalidateException to mandate a manual invalidate for event processing to continue. Testing: Modified pre-existing tests for partition events to instead test if partitions are added/dropped/altered when event processing is enabled. Change-Id: I213401329f3965dd81055197792ccf8a05368af5 --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 3 files changed, 200 insertions(+), 44 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/11/13111/5 -- To view, visit http://gerrit.cloudera.org:8080/13111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5 Gerrit-Change-Number: 13111 Gerrit-PatchSet: 5 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8454 (part 3): enable recursive file listing by default
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13127 ) Change subject: IMPALA-8454 (part 3): enable recursive file listing by default .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13127 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib30e2bcaf820210f2faa8f159d1af2f947a4d0e8 Gerrit-Change-Number: 13127 Gerrit-PatchSet: 3 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 29 Apr 2019 21:31:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7971: Add support for insert events in event processor.
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/12889 ) Change subject: IMPALA-7971: Add support for insert events in event processor. .. Patch Set 24: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 Gerrit-Change-Number: 12889 Gerrit-PatchSet: 24 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 29 Apr 2019 21:24:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7971: Add support for insert events in event processor.
Bharath Vissapragada has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12889 ) Change subject: IMPALA-7971: Add support for insert events in event processor. .. IMPALA-7971: Add support for insert events in event processor. This patch adds support for detecting and processing insert events triggered by impala as well as external engines (eg.Hive). Inserts from Impala will fire an insert event notification. Using this event, event-processor will refresh table/partition. Both insert into and overwrite are supported for tables/partitions. Known Issues: 1. Inserts into tables from Hive are ignored by the event processor as these inserts create an ALTER event first followed by an INSERT event. The alter will invalidate table making the refresh a no-op. Insert into partitions from hive will create an INSERT event first followed by an ALTER event. In this case, there is an unnecessary table invalidate after a refresh. 2. Existing self-events logic cannot be used for insert events since firing insert event does not allow us to modify table parameters in HMS. This means we cannot get the CatalogServiceIdentifiers in insert events. Therefore, the event-processor will also refresh the tables for which insert operation is performed through Impala. Testing: 1. Added new custom cluster tests to run different insert commands from hive and verified new data is available in Impala without invalidate metadata. 2. Added a test in MetastoreEventsProcessor for testing insert events. Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 Reviewed-on: http://gerrit.cloudera.org:8080/12889 Tested-by: Impala Public Jenkins Reviewed-by: Bharath Vissapragada --- M be/src/service/client-request-state.cc M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java M fe/src/test/resources/hive-site.xml.py A tests/custom_cluster/test_event_processing.py 10 files changed, 592 insertions(+), 13 deletions(-) Approvals: Impala Public Jenkins: Verified Bharath Vissapragada: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/12889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 Gerrit-Change-Number: 12889 Gerrit-PatchSet: 25 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
Vihang Karajgaonkar has posted comments on this change. (
http://gerrit.cloudera.org:8080/13005 )
Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive
3.1.0
..
Patch Set 12:
(11 comments)
Addressed review comments
http://gerrit.cloudera.org:8080/#/c/13005/9//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/13005/9//COMMIT_MSG@58
PS9, Line 58:
> Is this flakiness specific to the hive 3 config? Or the races were already
I checked and confirmed that the notification listener is working is generating
the events as expected but there may be more to it. I will dig more. Even if
there is a some problem with the event data, these races are still present can
show up when Sentry is slow for any reason to update its entries.
http://gerrit.cloudera.org:8080/#/c/13005/9//COMMIT_MSG@65
PS9, Line 65:
> maybe we shoudl disable these tests when running with hive 3 since we don't
So far, I would expect these tests work without any modifications on hive-2
builds (results pending for the last job I triggered). When we turn on cdp for
jobs we should re-investigate how to fix these tests.
http://gerrit.cloudera.org:8080/#/c/13005/3/bin/impala-config.sh
File bin/impala-config.sh:
http://gerrit.cloudera.org:8080/#/c/13005/3/bin/impala-config.sh@167
PS3, Line 167: export IMPALA_RANGER_VERSION=1.2.0.6.0.99.0-45
> This number looks like a CDH_BUILD_NUMBER, and is probably from the same na
I was not aware of this.. Most of the pending patches are merged into official
builds so we may not need this anymore.
http://gerrit.cloudera.org:8080/#/c/13005/3/bin/impala-config.sh@199
PS3, Line 199: # TODO(todd) switch to an official build.
> line too long (91 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/13005/3/bin/impala-config.sh@203
PS3, Line 203: # CDH hive version is used to build and deploy in minicluster
when USE_CDP_HIVE is
> line too long (92 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/13005/3/bin/impala-config.sh@212
PS3, Line 212: fi
> line too long (106 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/13005/9/bin/set-classpath.sh
File bin/set-classpath.sh:
http://gerrit.cloudera.org:8080/#/c/13005/9/bin/set-classpath.sh@30
PS9, Line 30:
#"$IMPALA_HOME"/shaded-deps/target/impala-shaded-deps-0.1-SNAPSHOT.jar:\
> why is this necessary? shouldn't the shaded-deps dependency also end up in
You are right, this is not necessary. removed it.
http://gerrit.cloudera.org:8080/#/c/13005/9/fe/src/main/java/org/apache/impala/catalog/MetaStoreClientPool.java
File fe/src/main/java/org/apache/impala/catalog/MetaStoreClientPool.java:
http://gerrit.cloudera.org:8080/#/c/13005/9/fe/src/main/java/org/apache/impala/catalog/MetaStoreClientPool.java@68
PS9, Line 68: throws MetaException {
: return null;
: }
: };
:
: /**
> Are these changes actually used right now? I think this stuff ended up bein
Yeah. reverted the changes to this file in the latest patch
http://gerrit.cloudera.org:8080/#/c/13005/9/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
File
fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java:
http://gerrit.cloudera.org:8080/#/c/13005/9/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@173
PS9, Line 173: private static MetastoreEventsProcessor instance;
> Can you explain what's going on with this part of the change? Did Sentry mo
Sentry did not move to the JSONMessageFactory (and I believe this may be the
reason why the test_ownership.py fails). However, in the shims approach we use
the ExtendedJsonFactory for hive-2 builds and in hive-3 build we don't expect
Sentry to be there. I will keep investigating while I debug cdp jobs.
http://gerrit.cloudera.org:8080/#/c/13005/9/testdata/bin/run-hive-server.sh
File testdata/bin/run-hive-server.sh:
http://gerrit.cloudera.org:8080/#/c/13005/9/testdata/bin/run-hive-server.sh@66
PS9, Line 66: export HIVE_METASTORE_HADOOP_OPTS="-Xdebug
-Xrunjdwp:transport=dt_socket,server=y,\
> probably remove this
Done
http://gerrit.cloudera.org:8080/#/c/13005/9/testdata/bin/run-hive-server.sh@93
PS9, Line 93: if [ ${ONLY_METASTORE} -eq 0 ]; then
> this is in another patch- guess we can rebase this on top of that one to pi
Rebased my patch.
--
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 12
Gerrit-Owner: Vihang Karajgaonkar
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Philip Zeyliger
Gerrit-Reviewer: Sudhanshu Arora
Gerrit-Reviewer: Todd Lipcon
Gerrit-Reviewer:
[Impala-ASF-CR] IMPALA-7971: Add support for insert events in event processor.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12889 ) Change subject: IMPALA-7971: Add support for insert events in event processor. .. Patch Set 24: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 Gerrit-Change-Number: 12889 Gerrit-PatchSet: 24 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 29 Apr 2019 20:59:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8280, IMPALA-8281: Add support for show grant user/group with Ranger
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13074 ) Change subject: IMPALA-8280, IMPALA-8281: Add support for show grant user/group with Ranger .. Patch Set 7: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/2977/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13074 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic46fb9fc36c9e11ec78d5840d22eb0668150c2a4 Gerrit-Change-Number: 13074 Gerrit-PatchSet: 7 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 29 Apr 2019 20:46:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
Vihang Karajgaonkar has posted comments on this change. (
http://gerrit.cloudera.org:8080/13005 )
Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive
3.1.0
..
Patch Set 3:
(15 comments)
> (1 comment)
>
> Nice work.
>
> When I did the transitional work between Hive 1 and Hive 2, I
> introduced a variable that switched at build time between the two
> worlds. (See a203733fac3e1e37df8abeee39a88d187153a8c5 for the
> revert and "git log --grep IMPALA_MINICLUSTER_PROFILE")
>
> If I'm understanding right, the approach here is to produce a
> single "binary" that works for both worlds? Or at run time do the
> "original" Hive jars get run? I think both approaches are
> plausible; just want to make sure we're clear about it.
>
> (Is the shading slow? I've seen maven-shade-plugin be very slow...)
The original approach was to always build against hive-3, but we switched the
approach to use a compatibility shim based approach as you suggested above.
Shading is not super slow. It task about 4 sec to build the jar if I remember
correctly
http://gerrit.cloudera.org:8080/#/c/13005/2/bin/impala-config.sh
File bin/impala-config.sh:
http://gerrit.cloudera.org:8080/#/c/13005/2/bin/impala-config.sh@199
PS2, Line 199: # When USE_CDP_HIVE is set we use the latest hive version
available to deply in minicluster
> line too long (91 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/13005/2/bin/impala-config.sh@203
PS2, Line 203: # TODO(Vihang) we should repackage the tarballs so that the
src and binaries are extracted
> line too long (92 > 90)
This line was removed later in the patch
http://gerrit.cloudera.org:8080/#/c/13005/2/bin/impala-config.sh@212
PS2, Line 212: export
HIVE_HOME="$IMPALA_TOOLCHAIN/cdh_components-${CDH_BUILD_NUMBER}/hive-${MINICLUSTER_HIVE_VERSION}"
> line too long (106 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/13005/2/bin/impala-config.sh@230
PS2, Line 230: . "$IMPALA_HOME/bin/impala-config-branch.sh"
: if [ -f "$IMPALA_HOME/bin/impala-config-local.sh" ]; then
: . "$IMPALA_HOME/bin/impala-config-local.sh"
: fi
> We need to be careful about which variables are assigned before this and wh
Good point. Moved all the new variable assignment logic post this block.
http://gerrit.cloudera.org:8080/#/c/13005/2/bin/impala-config.sh@546
PS2, Line 546: export
HIVE_METASTORE_THRIFT_DIR=$CDP_COMPONENTS_HOME/apache-hive-${IMPALA_HIVE_VERSION}-bin/src/standalone-metastore/src/main/thrift
> line too long (133 > 90)
This line was moved later in the patch and the line length is under the 90 now.
http://gerrit.cloudera.org:8080/#/c/13005/2/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java
File fe/src/main/java/org/apache/impala/analysis/StringLiteral.java:
http://gerrit.cloudera.org:8080/#/c/13005/2/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java@105
PS2, Line 105: public static String unescapeSQLString(String b) {
> Do we have any plans to create a public classes in Hive through which we ca
Yes, I think we may be able to do that add that. Will do it as a followup item.
http://gerrit.cloudera.org:8080/#/c/13005/3/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java
File fe/src/main/java/org/apache/impala/analysis/StringLiteral.java:
http://gerrit.cloudera.org:8080/#/c/13005/3/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java@96
PS3, Line 96: /**
:* Copied from Apache Hive's BaseSemanticAnalyzer. This method
has not changed
:* since last several years so hoping that it is fairly stable
by now. Sourcing it from
:* the Hive's code without copying brings along with it a lot
of other unnecessary
:* dependencies
:* @param b
:* @return
:*/
> Can you move the parts copied from Hive to a separate file/directory? It wi
The copied code was moved into HiveShims class in the compat-3 directory
http://gerrit.cloudera.org:8080/#/c/13005/3/fe/src/main/java/org/apache/impala/catalog/FeHBaseTable.java
File fe/src/main/java/org/apache/impala/catalog/FeHBaseTable.java:
http://gerrit.cloudera.org:8080/#/c/13005/3/fe/src/main/java/org/apache/impala/catalog/FeHBaseTable.java@122
PS3, Line 122: //TODO (Vihang) this pulls in hive-hbase-handler dependency
which brings all the
: // other stuff we don't need. Perhaps we just need to copy
the constants and mark
: // them public API in Hive source code
: private static final String HBASE_COLUMNS_MAPPING =
"hbase.columns.mapping";
: private static final String HBASE_TABLE_DEFAULT_STORAGE_TYPE
= "hbase.table.default"
: + ".storage.type";
: private static final String HBASE_KEY_COL = ":key";
: private static final String
