[Impala-ASF-CR] IMPALA-8149 : Add support for alter database events
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13049 ) Change subject: IMPALA-8149 : Add support for alter_database events .. Patch Set 6: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4092/ -- To view, visit http://gerrit.cloudera.org:8080/13049 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaf020e85cae04163bf32e31363eb4119d624640b Gerrit-Change-Number: 13049 Gerrit-PatchSet: 6 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Sat, 27 Apr 2019 05:28:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8309: add user authorization provider flag
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12901 ) Change subject: IMPALA-8309: add user authorization_provider flag .. Patch Set 23: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4094/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/12901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 Gerrit-Change-Number: 12901 Gerrit-PatchSet: 23 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Sat, 27 Apr 2019 04:42:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8465: fix TestJsonEndpoints for remote clusters
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13136 ) Change subject: IMPALA-8465: fix TestJsonEndpoints for remote clusters .. Patch Set 2: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4091/ -- To view, visit http://gerrit.cloudera.org:8080/13136 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8cf6ecfb72ec18be0b5274e360825fb3f203dd20 Gerrit-Change-Number: 13136 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Sat, 27 Apr 2019 04:33:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8309: add user authorization provider flag
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12901 ) Change subject: IMPALA-8309: add user authorization_provider flag .. Patch Set 21: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4089/ -- To view, visit http://gerrit.cloudera.org:8080/12901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 Gerrit-Change-Number: 12901 Gerrit-PatchSet: 21 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Sat, 27 Apr 2019 03:14:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 ) Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0 .. Patch Set 11: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/2947/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13005 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436 Gerrit-Change-Number: 13005 Gerrit-PatchSet: 11 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Sat, 27 Apr 2019 02:59:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/13005 )
Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive
3.1.0
..
Patch Set 11:
(26 comments)
http://gerrit.cloudera.org:8080/#/c/13005/11/bin/impala-config.sh
File bin/impala-config.sh:
http://gerrit.cloudera.org:8080/#/c/13005/11/bin/impala-config.sh@181
PS11, Line 181: # When USE_CDP_HIVE is set we use the latest hive version
available to deply in minicluster
line too long (93 > 90)
http://gerrit.cloudera.org:8080/#/c/13005/11/bin/impala-config.sh@185
PS11, Line 185: export
HIVE_METASTORE_THRIFT_DIR=$CDP_COMPONENTS_HOME/apache-hive-${IMPALA_HIVE_VERSION}-src/standalone-metastore/src/main/thrift
line too long (131 > 90)
http://gerrit.cloudera.org:8080/#/c/13005/11/bin/impala-config.sh@204
PS11, Line 204: export
HIVE_HOME="$IMPALA_TOOLCHAIN/cdh_components-${CDH_BUILD_NUMBER}/hive-${IMPALA_HIVE_VERSION}"
line too long (101 > 90)
http://gerrit.cloudera.org:8080/#/c/13005/11/bin/impala-config.sh@205
PS11, Line 205: export
HIVE_METASTORE_THRIFT_DIR=$CDH_COMPONENTS_HOME/hive-${IMPALA_HIVE_VERSION}/src/metastore/if
line too long (100 > 90)
http://gerrit.cloudera.org:8080/#/c/13005/11/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/13005/11/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1073
PS11, Line 1073:
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/13005/11/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1169
PS11, Line 1169:
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/13005/11/fe/src/main/java/org/apache/impala/service/MetadataOp.java
File fe/src/main/java/org/apache/impala/service/MetadataOp.java:
http://gerrit.cloudera.org:8080/#/c/13005/11/fe/src/main/java/org/apache/impala/service/MetadataOp.java@319
PS11, Line 319:
MetastoreShim.mapToInternalTableType(table.getMetaStoreTable().getTableType());
line too long (97 > 90)
http://gerrit.cloudera.org:8080/#/c/13005/11/testdata/bin/run-hive-server.sh
File testdata/bin/run-hive-server.sh:
http://gerrit.cloudera.org:8080/#/c/13005/11/testdata/bin/run-hive-server.sh@66
PS11, Line 66: export HIVE_METASTORE_HADOOP_OPTS="-verbose:class -Xdebug
-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=30010"
line too long (121 > 90)
http://gerrit.cloudera.org:8080/#/c/13005/11/tests/authorization/test_owner_privileges.py
File tests/authorization/test_owner_privileges.py:
http://gerrit.cloudera.org:8080/#/c/13005/11/tests/authorization/test_owner_privileges.py@103
PS11, Line 103: )
flake8: E501 line too long (91 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/11/tests/authorization/test_owner_privileges.py@179
PS11, Line 179: =
flake8: E501 line too long (93 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/11/tests/authorization/test_owner_privileges.py@182
PS11, Line 182: =
flake8: E501 line too long (93 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/11/tests/authorization/test_owner_privileges.py@186
PS11, Line 186: )
flake8: E501 line too long (91 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/11/tests/authorization/test_owner_privileges.py@193
PS11, Line 193: ,
flake8: E501 line too long (91 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/11/tests/authorization/test_owner_privileges.py@213
PS11, Line 213: )
flake8: E501 line too long (91 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/11/tests/authorization/test_owner_privileges.py@226
PS11, Line 226: )
flake8: E501 line too long (91 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/11/tests/authorization/test_owner_privileges.py@236
PS11, Line 236: )
flake8: E501 line too long (91 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/11/tests/authorization/test_owner_privileges.py@282
PS11, Line 282: _
flake8: E501 line too long (100 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/11/tests/authorization/test_owner_privileges.py@287
PS11, Line 287: t
flake8: E501 line too long (101 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/11/tests/authorization/test_owner_privileges.py@293
PS11, Line 293: 2
flake8: E501 line too long (92 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/11/tests/authorization/test_owner_privileges.py@355
PS11, Line 355: _
flake8: E501 line too long (100 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/11/tests/authorization/test_owner_privileges.py@360
PS11, Line 360: t
flake8: E501 line too long (101 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/11/tests/authorization/test_owner_privileges.py@364
PS11, Line 364: 2
flake8: E501 line too long (92 > 90 characters)
[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 ) Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0 .. Patch Set 11: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4093/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/13005 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436 Gerrit-Change-Number: 13005 Gerrit-PatchSet: 11 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Sat, 27 Apr 2019 02:16:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
Vihang Karajgaonkar has uploaded a new patch set (#11). ( http://gerrit.cloudera.org:8080/13005 ) Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0 .. IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0 This change upgrades the hive dependencies of Impala to use Hive 3.1.0 based binaries. Most of the changes in this patch are based off patches provided by Todd (links available in JIRA). Upgrading the dependencies allows us to work with both Hive 3.1.0 and Hive 2.1.0 in the same code line. In order to do this, the patch trims down a lot of unnecessary hive dependencies of the front end code by creating a shaded-deps module. The pom.xml of shaded-deps includes only the files from Hive source which Impala depends for compilation. Additionally, it also uses a custom build of Hive which is based of Hive 3.1.0. This custom build includes patches for HIVE-21596 and HIVE-21586 which are needed by Impala so that it can compile against Hive-3 libraries and be able to talk to HMS-2.x metastore. Once these patches are merged we can get rid of this custom build and rely on more official sources of Hive builds. The patch also changes impala-config.sh so that it always downloads the Hive-3 libraries from the toolchain. The code is always built using Hive-3 jars. However, based on the value of USE_CDP_HIVE, the minicluster is deployed using Hive-3 or Hive-2 binaries. Since Impala implements HiveServer2's TCLIService.thrift interface, it requires us to use the existing mechanism of copying the hive-2/api TCLIService.thrift to hive-3/api. It also adds a few environment variables which point to the metastore's thrift file and the CDH Hive version. Testing: 1. Code compiles and runs against both HMS-3 and HMS-2 2. Ran full-suite of tests using the private jenkins job against HMS-2 3. Running full-tests against HMS-3 will need more work like supporting Tez in the mini-cluster (for dataloading) and HMS transaction support since HMS3 create transactional tables by default. This will be taken up in subsequent change. Notes: 1. Patch uses a custom build of Hive to be deployed in mini-cluster. This build has the fixes for HIVE-21596 2. The Hive 3.1 does not include some of the patches like HIVE-21077. Some of the recent code in Impala depends on that patch to exist in hive. We will need to port HIVE-21077 to Hive 3.1.0 in order to compile against Hive 3.1.0 jars. I will continue working on getting this merged along with HIVE-21586 3. Some of the existing tests rely on the fact the UDFs implement the UDF interface in Hive (UDFLength, UDFHour, UDFYear). These built-in hive functions have been moved to use GenericUDF interface in Hive 3. Impala currently only supports UDFExecutor. In order to have a full compatibility with all the functions in Hive 2.x we should support GenericUDFs too. That would be taken up as a separate patch. 4. The Sentry object ownership tests are flaky. There are race-conditions in that code which get exposed for some by this patch. For example, when a database is created, the object privileges are immediately updated in the catalog cache. If Sentry has not been updated and there is a refresh authorization call in between it can clear the privilege from the cache causing the subsequent statement using that privilege to fail. The patch adds sleep statements in the test to work-around these issues. Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436 --- M CMakeLists.txt M README.md M bin/bootstrap_toolchain.py M bin/impala-config.sh M bin/set-classpath.sh M common/thrift/.gitignore M common/thrift/CMakeLists.txt M fe/CMakeLists.txt M fe/pom.xml A fe/src/compat-hive-2/java/org/apache/impala/compat/HiveShims.java R fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java A fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/analysis/StringLiteral.java M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java M fe/src/main/java/org/apache/impala/catalog/FeHBaseTable.java M fe/src/main/java/org/apache/impala/catalog/TableLoader.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java M fe/src/main/java/org/apache/impala/hive/executor/UdfExecutor.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java M fe/src/main/java/org/apache/impala/service/MetadataOp.java A fe/src/main/java/org/apache/impala/util/MetadataFormatUtils.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java M fe/src/test/java/org/apache/impala/hive/executor/UdfExecutorTest.java M
[Impala-ASF-CR] IMPALA-7957: Fix slot equivalences may be enforced multiple times
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/13051 ) Change subject: IMPALA-7957: Fix slot equivalences may be enforced multiple times .. Patch Set 3: Oh! Thanks for the missing case! I'll refactor the patch later to handle this problem. Just be busy on some other stuff these days... -- To view, visit http://gerrit.cloudera.org:8080/13051 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ida2d5d8149b217e18ebae61e136848162503653e Gerrit-Change-Number: 13051 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Sat, 27 Apr 2019 02:10:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8149 : Add support for alter database events
xiaom...@cloudera.com has posted comments on this change. ( http://gerrit.cloudera.org:8080/13049 ) Change subject: IMPALA-8149 : Add support for alter_database events .. Patch Set 4: Thanks a lot for reviewing Fredy and Bharath! -- To view, visit http://gerrit.cloudera.org:8080/13049 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaf020e85cae04163bf32e31363eb4119d624640b Gerrit-Change-Number: 13049 Gerrit-PatchSet: 4 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Sat, 27 Apr 2019 00:55:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8149 : Add support for alter database events
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13049 ) Change subject: IMPALA-8149 : Add support for alter_database events .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13049 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaf020e85cae04163bf32e31363eb4119d624640b Gerrit-Change-Number: 13049 Gerrit-PatchSet: 5 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Sat, 27 Apr 2019 00:30:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8149 : Add support for alter database events
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13049 ) Change subject: IMPALA-8149 : Add support for alter_database events .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4092/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13049 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaf020e85cae04163bf32e31363eb4119d624640b Gerrit-Change-Number: 13049 Gerrit-PatchSet: 6 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Sat, 27 Apr 2019 00:30:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8149 : Add support for alter database events
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13049 ) Change subject: IMPALA-8149 : Add support for alter_database events .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13049 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaf020e85cae04163bf32e31363eb4119d624640b Gerrit-Change-Number: 13049 Gerrit-PatchSet: 6 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Sat, 27 Apr 2019 00:30:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8149 : Add support for alter database events
Bharath Krishna has posted comments on this change. ( http://gerrit.cloudera.org:8080/13049 ) Change subject: IMPALA-8149 : Add support for alter_database events .. Patch Set 5: Code-Review+1 Thanks Xiaomeng for the updates in patch. LGTM. -- To view, visit http://gerrit.cloudera.org:8080/13049 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaf020e85cae04163bf32e31363eb4119d624640b Gerrit-Change-Number: 13049 Gerrit-PatchSet: 5 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Sat, 27 Apr 2019 00:03:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8149 : Add support for alter database events
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13049 ) Change subject: IMPALA-8149 : Add support for alter_database events .. Patch Set 5: Code-Review+1 (1 comment) I will give a chance for Bharath K take a look at it, too. I can promote to a +2 afterwards. http://gerrit.cloudera.org:8080/#/c/13049/4/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java: http://gerrit.cloudera.org:8080/#/c/13049/4/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@343 PS4, Line 343: est > I was using clang tool do formatting, I guess it takes each argument as ind I don't want to be a stickler. I'm cool with this. -- To view, visit http://gerrit.cloudera.org:8080/13049 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaf020e85cae04163bf32e31363eb4119d624640b Gerrit-Change-Number: 13049 Gerrit-PatchSet: 5 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 26 Apr 2019 23:59:12 + Gerrit-HasComments: Yes
[Impala-ASF-CR] Configure Hive 3's HS2 to execute queries using Tez local mode
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/12931 ) Change subject: Configure Hive 3's HS2 to execute queries using Tez local mode .. Patch Set 6: Code-Review+2 I think this makes sense, and it mostly impacts Hive 3 configurations, so let's get this in. -- To view, visit http://gerrit.cloudera.org:8080/12931 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I76e47fbd1d6ff5103d81a8de430d5465dba284cd Gerrit-Change-Number: 12931 Gerrit-PatchSet: 6 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 26 Apr 2019 23:39:19 + Gerrit-HasComments: No
[Impala-ASF-CR] Configure Hive 3's HS2 to execute queries using Tez local mode
Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12931 ) Change subject: Configure Hive 3's HS2 to execute queries using Tez local mode .. Configure Hive 3's HS2 to execute queries using Tez local mode Hive 3 no longer supports MR execution, so this sets up the appropriate configuration and classpath so that HS2 can run queries using Tez. The bulk of this patch is toolchain changes to download Tez itself. The Tez tarball is slightly odd in that it has no top-level directory, so the patch changes around bootstrap_toolchain a bit to support creating its own top-level directory for a component. The remainder of the patch is some classpath setup and hive-site changes when Hive 3 is enabled. So far I tested this manually by setting up a metastore and impala-config with USE_CDP_HIVE=true, and then connecting to HS2 using hive beeline -u 'jdbc:hive2://localhost:11050' I was able to insert and query data, and was able to verify that queries like 'select count(*)' were executing via Tez local mode. NOTE: this patch relies on a custom build of Tez, based on a private branch. I've submitted a PR to Tez upstream, referenced in the commits here. Will remove this hack once the PR is accepted and makes its way into an official build. Change-Id: I76e47fbd1d6ff5103d81a8de430d5465dba284cd Reviewed-on: http://gerrit.cloudera.org:8080/12931 Tested-by: Impala Public Jenkins Reviewed-by: Joe McDonnell --- M bin/bootstrap_toolchain.py M bin/impala-config.sh M fe/pom.xml M fe/src/test/resources/hive-site.xml.py M testdata/bin/run-hive-server.sh 5 files changed, 90 insertions(+), 10 deletions(-) Approvals: Impala Public Jenkins: Verified Joe McDonnell: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/12931 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I76e47fbd1d6ff5103d81a8de430d5465dba284cd Gerrit-Change-Number: 12931 Gerrit-PatchSet: 7 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8465: fix TestJsonEndpoints for remote clusters
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13136 ) Change subject: IMPALA-8465: fix TestJsonEndpoints for remote clusters .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2946/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13136 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8cf6ecfb72ec18be0b5274e360825fb3f203dd20 Gerrit-Change-Number: 13136 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 26 Apr 2019 23:33:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8149 : Add support for alter database events
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13049 ) Change subject: IMPALA-8149 : Add support for alter_database events .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2945/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13049 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaf020e85cae04163bf32e31363eb4119d624640b Gerrit-Change-Number: 13049 Gerrit-PatchSet: 5 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 26 Apr 2019 23:38:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 ) Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0 .. Patch Set 10: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4088/ -- To view, visit http://gerrit.cloudera.org:8080/13005 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436 Gerrit-Change-Number: 13005 Gerrit-PatchSet: 10 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 26 Apr 2019 23:29:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8454 (part 3): enable recursive file listing by default
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13127 ) Change subject: IMPALA-8454 (part 3): enable recursive file listing by default .. Patch Set 2: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4087/ -- To view, visit http://gerrit.cloudera.org:8080/13127 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib30e2bcaf820210f2faa8f159d1af2f947a4d0e8 Gerrit-Change-Number: 13127 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 26 Apr 2019 23:21:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8309: add user authorization provider flag
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12901 ) Change subject: IMPALA-8309: add user authorization_provider flag .. Patch Set 22: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2944/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 Gerrit-Change-Number: 12901 Gerrit-PatchSet: 22 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Fri, 26 Apr 2019 23:14:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8454 (part 2): Initial support for recursive file listing within a partition
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12991 ) Change subject: IMPALA-8454 (part 2): Initial support for recursive file listing within a partition .. Patch Set 4: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4086/ -- To view, visit http://gerrit.cloudera.org:8080/12991 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9b151d7abb8443c0d9de0a0d82a9f13e07ad5109 Gerrit-Change-Number: 12991 Gerrit-PatchSet: 4 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 26 Apr 2019 23:11:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8465: fix TestJsonEndpoints for remote clusters
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13136 ) Change subject: IMPALA-8465: fix TestJsonEndpoints for remote clusters .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4091/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13136 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8cf6ecfb72ec18be0b5274e360825fb3f203dd20 Gerrit-Change-Number: 13136 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 26 Apr 2019 23:09:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8465: fix TestJsonEndpoints for remote clusters
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13136 ) Change subject: IMPALA-8465: fix TestJsonEndpoints for remote clusters .. Patch Set 1: The precommit should test both code paths. -- To view, visit http://gerrit.cloudera.org:8080/13136 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8cf6ecfb72ec18be0b5274e360825fb3f203dd20 Gerrit-Change-Number: 13136 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 26 Apr 2019 23:09:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8465: fix TestJsonEndpoints for remote clusters
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13136 Change subject: IMPALA-8465: fix TestJsonEndpoints for remote clusters .. IMPALA-8465: fix TestJsonEndpoints for remote clusters ImpalaCluster does not work for remote clusters, and the patch introduced a dependency on it for a test that previously didn't depend on it. This reverts to the old logic for non-dockerized clusters. Change-Id: I8cf6ecfb72ec18be0b5274e360825fb3f203dd20 --- M tests/hs2/test_json_endpoints.py 1 file changed, 9 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/36/13136/1 -- To view, visit http://gerrit.cloudera.org:8080/13136 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I8cf6ecfb72ec18be0b5274e360825fb3f203dd20 Gerrit-Change-Number: 13136 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] IMPALA-8309: add user authorization provider flag
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12901 ) Change subject: IMPALA-8309: add user authorization_provider flag .. Patch Set 21: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2943/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 Gerrit-Change-Number: 12901 Gerrit-PatchSet: 21 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Fri, 26 Apr 2019 23:02:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8463: fix skip.header.line.count on local catalog
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13131 ) Change subject: IMPALA-8463: fix skip.header.line.count on local catalog .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2942/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13131 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3957b3661574c356a14197e8829abbed92028a1a Gerrit-Change-Number: 13131 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Fri, 26 Apr 2019 22:58:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8149 : Add support for alter database events
xiaom...@cloudera.com has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/13049 ) Change subject: IMPALA-8149 : Add support for alter_database events .. IMPALA-8149 : Add support for alter_database events This change adds support for alter_database events in two parts: One is adding catalogServiceId and catalogVersion in db parameters when alter database. The other is adding alter database event, check if it's self event during process, if true do nothing, if false replace caralog cached db with event db. Testing: Enabled testAlterDisableFlagFromDb in MetastoreEventsProcessorTest. Change-Id: Iaf020e85cae04163bf32e31363eb4119d624640b --- M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/Db.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 6 files changed, 337 insertions(+), 111 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/13049/5 -- To view, visit http://gerrit.cloudera.org:8080/13049 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iaf020e85cae04163bf32e31363eb4119d624640b Gerrit-Change-Number: 13049 Gerrit-PatchSet: 5 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8149 : Add support for alter database events
xiaom...@cloudera.com has posted comments on this change. (
http://gerrit.cloudera.org:8080/13049 )
Change subject: IMPALA-8149 : Add support for alter_database events
..
Patch Set 5:
(8 comments)
http://gerrit.cloudera.org:8080/#/c/13049/4/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:
http://gerrit.cloudera.org:8080/#/c/13049/4/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@757
PS4, Line 757:* If tblName is null, removes version number from database.
> We can say:
Done
http://gerrit.cloudera.org:8080/#/c/13049/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/13049/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@445
PS1, Line 445: otected static Stri
> We can just leave it for now. Removing the check is wrong if params can be
Done
http://gerrit.cloudera.org:8080/#/c/13049/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/13049/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3805
PS1, Line 3805: if (db == null) {
> I think removing is fine. Probably we can change the Preconditions.checkSta
Done
http://gerrit.cloudera.org:8080/#/c/13049/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3812
PS1, Line 3812: default:
> Moved it under set_owner which lock db first.
Done
http://gerrit.cloudera.org:8080/#/c/13049/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3813
PS1, Line 3813: throw new UnsupportedOperationException(
> I copied the format in alterTable method which update catalogServiceIdentif
Done
http://gerrit.cloudera.org:8080/#/c/13049/4/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
File
fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java:
http://gerrit.cloudera.org:8080/#/c/13049/4/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@343
PS4, Line 343: est
> nit: fix indentation
I was using clang tool do formatting, I guess it takes each argument as indent
4, the inside argument break indented 4 more spaces on original.
>From Google: "two continuation lines use the same indentation level if and
>only if they begin with syntactically parallel elements."
So I think it make sense to indent this way.
What do you think?
http://gerrit.cloudera.org:8080/#/c/13049/4/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@345
PS4, Line 345: eventsProcessor_.processEvents();
> I feel we should add an additional test, here we are testing if the Notific
Done
http://gerrit.cloudera.org:8080/#/c/13049/4/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@1481
PS4, Line 1481: }
> Can we check if possible that the counter MetastoreEventsProcessor.EVENTS_S
Done
--
To view, visit http://gerrit.cloudera.org:8080/13049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf020e85cae04163bf32e31363eb4119d624640b
Gerrit-Change-Number: 13049
Gerrit-PatchSet: 5
Gerrit-Owner: Anonymous Coward
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Anurag Mantripragada
Gerrit-Reviewer: Bharath Krishna
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Comment-Date: Fri, 26 Apr 2019 22:47:44 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8309: add user authorization provider flag
Fredy Wijaya has uploaded a new patch set (#22) to the change originally created by radford nguyen. ( http://gerrit.cloudera.org:8080/12901 ) Change subject: IMPALA-8309: add user authorization_provider flag .. IMPALA-8309: add user authorization_provider flag This commit adds a `authorization_provider` user-facing flag in order to provide a more human-readable alternative to the `authorization_factory_class` for internally-provided authorization strategies. The `authorization_factory_class` flag is retained, but no longer takes a default value if not specified. The default for `authorization_provider` is "sentry" in order to retain backwards-compatibility. If specified, `authorization_factory_class` will take precedence. Testing: - Manually started minicluster with each of following flags and verified correct authorization strategy chosen: - provider='' factory='' => sentry - provider=sentry factory='' => sentry - provider=ranger factory='' => ranger - provider='' factory=sentry => sentry - provider='' factory=ranger => ranger - provider=sentry factory=sentry => sentry - provider=ranger factory=sentry => sentry - provider=sentry factory=ranger => ranger - provider=ranger factory=ranger => ranger - Wrote unit tests to capture above assertions - Ran fe unit and e2e tests - Wrote e2e test to verify new flag behavior Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 --- M be/src/service/frontend.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/authorization/AuthorizationConfig.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationProvider.java M fe/src/main/java/org/apache/impala/authorization/NoopAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationConfig.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationConfig.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java A fe/src/main/java/org/apache/impala/util/AuthorizationUtil.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java A fe/src/test/java/org/apache/impala/util/AuthorizationUtilTest.java A tests/authorization/test_provider.py M tests/authorization/test_ranger.py 16 files changed, 340 insertions(+), 62 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/12901/22 -- To view, visit http://gerrit.cloudera.org:8080/12901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 Gerrit-Change-Number: 12901 Gerrit-PatchSet: 22 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: radford nguyen
[Impala-ASF-CR] IMPALA-8309: add user authorization provider flag
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/12901 ) Change subject: IMPALA-8309: add user authorization_provider flag .. Patch Set 22: (1 comment) Running a dry-run first. http://gerrit.cloudera.org:8080/#/c/12901/21/tests/authorization/test_provider.py File tests/authorization/test_provider.py: http://gerrit.cloudera.org:8080/#/c/12901/21/tests/authorization/test_provider.py@60 PS21, Line 60: > flake8: E124 closing bracket does not match visual indentation Done -- To view, visit http://gerrit.cloudera.org:8080/12901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 Gerrit-Change-Number: 12901 Gerrit-PatchSet: 22 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Fri, 26 Apr 2019 22:18:18 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8309: add user authorization provider flag
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12901 ) Change subject: IMPALA-8309: add user authorization_provider flag .. Patch Set 21: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4089/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/12901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 Gerrit-Change-Number: 12901 Gerrit-PatchSet: 21 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Fri, 26 Apr 2019 22:16:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8309: add user authorization provider flag
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12901 ) Change subject: IMPALA-8309: add user authorization_provider flag .. Patch Set 21: (1 comment) http://gerrit.cloudera.org:8080/#/c/12901/21/tests/authorization/test_provider.py File tests/authorization/test_provider.py: http://gerrit.cloudera.org:8080/#/c/12901/21/tests/authorization/test_provider.py@60 PS21, Line 60: ) flake8: E124 closing bracket does not match visual indentation -- To view, visit http://gerrit.cloudera.org:8080/12901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 Gerrit-Change-Number: 12901 Gerrit-PatchSet: 21 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Fri, 26 Apr 2019 22:15:57 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8463: fix skip.header.line.count on local catalog
Hello Bharath Vissapragada, Todd Lipcon, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13131 to look at the new patch set (#5). Change subject: IMPALA-8463: fix skip.header.line.count on local catalog .. IMPALA-8463: fix skip.header.line.count on local catalog This moves the logic to FeFsTable and calls it from LocalFsTable. Testing: Added a unit test that repros the problem. End-to-end tests run against local catalog will catch this but aren't enabled. Change-Id: I3957b3661574c356a14197e8829abbed92028a1a --- M fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java M fe/src/main/java/org/apache/impala/catalog/FeFsTable.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java 5 files changed, 57 insertions(+), 51 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/13131/5 -- To view, visit http://gerrit.cloudera.org:8080/13131 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3957b3661574c356a14197e8829abbed92028a1a Gerrit-Change-Number: 13131 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8463: fix skip.header.line.count on local catalog
Tim Armstrong has posted comments on this change. (
http://gerrit.cloudera.org:8080/13131 )
Change subject: IMPALA-8463: fix skip.header.line.count on local catalog
..
Patch Set 5:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/13131/4/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/13131/4/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@1122
PS4, Line 1122: org.apache.hadoop.hive.metastore.api.Table msTbl) throws
Exception {
> can this copy of the function be removed now?
Done
--
To view, visit http://gerrit.cloudera.org:8080/13131
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3957b3661574c356a14197e8829abbed92028a1a
Gerrit-Change-Number: 13131
Gerrit-PatchSet: 5
Gerrit-Owner: Tim Armstrong
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Todd Lipcon
Gerrit-Comment-Date: Fri, 26 Apr 2019 22:06:46 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8293 (Part 2): Add support for Ranger cache invalidation
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13134 ) Change subject: IMPALA-8293 (Part 2): Add support for Ranger cache invalidation .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2941/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13134 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia7160c082298e0b8cc2742dd3facbd4978581288 Gerrit-Change-Number: 13134 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 26 Apr 2019 22:05:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8463: fix skip.header.line.count on local catalog
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13131 ) Change subject: IMPALA-8463: fix skip.header.line.count on local catalog .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2940/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13131 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3957b3661574c356a14197e8829abbed92028a1a Gerrit-Change-Number: 13131 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Fri, 26 Apr 2019 21:33:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8463: fix skip.header.line.count on local catalog
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13131 ) Change subject: IMPALA-8463: fix skip.header.line.count on local catalog .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2939/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13131 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3957b3661574c356a14197e8829abbed92028a1a Gerrit-Change-Number: 13131 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Fri, 26 Apr 2019 21:26:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8293 (Part 2): Add support for Ranger cache invalidation
Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13134 Change subject: IMPALA-8293 (Part 2): Add support for Ranger cache invalidation .. IMPALA-8293 (Part 2): Add support for Ranger cache invalidation This patch adds support for Ranger cache invalidation via INVALIDATE METADATA and REFRESH AUTHORIZATION. This patch introduces a new catalog object type called AUTHZ_REFRESH to allow broadcasting messages from Catalogd to Impalads to update their local Ranger caches. For better user experience, every GRANT/REVOKE statement perform an authorization refresh. Testing: - Replaced the sleep in test_ranger.py with INVALIDATE METADATA or REFRESH AUTHORIZATION - Ran all FE tests - Ran all E2E authorization tests Change-Id: Ia7160c082298e0b8cc2742dd3facbd4978581288 --- M common/thrift/CatalogObjects.thrift M fe/src/main/java/org/apache/impala/authorization/AuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/NoopAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationChecker.java A fe/src/main/java/org/apache/impala/catalog/AuthzRefresh.java M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java M fe/src/main/java/org/apache/impala/service/FeCatalogManager.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java M fe/src/test/java/org/apache/impala/testutil/ImpaladTestCatalog.java M fe/src/test/resources/ranger-hive-security.xml M tests/authorization/test_ranger.py 19 files changed, 389 insertions(+), 97 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/34/13134/5 -- To view, visit http://gerrit.cloudera.org:8080/13134 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ia7160c082298e0b8cc2742dd3facbd4978581288 Gerrit-Change-Number: 13134 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya
[Impala-ASF-CR] IMPALA-8309: add user authorization provider flag
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12901 ) Change subject: IMPALA-8309: add user authorization_provider flag .. Patch Set 20: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4082/ -- To view, visit http://gerrit.cloudera.org:8080/12901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 Gerrit-Change-Number: 12901 Gerrit-PatchSet: 20 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Fri, 26 Apr 2019 21:13:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8444: Fix performance regression when building privilege name
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13095 ) Change subject: IMPALA-8444: Fix performance regression when building privilege name .. IMPALA-8444: Fix performance regression when building privilege name This patch fixes the performance regression when building privilege name by rewriting PrincipalPrivilege.buildPrivilegeName() with a simple string concatentation instead of using a list that gets converted into a string. Below is the result of running a benchmark using JMH comparing the old and new implementations: Result "org.apache.impala.BuildPrivilegeNameBenchmark.fast": 0.344 ±(99.9%) 0.004 us/op [Average] (min, avg, max) = (0.336, 0.344, 0.355), stdev = 0.005 CI (99.9%): [0.339, 0.348] (assumes normal distribution) Result "org.apache.impala.BuildPrivilegeNameBenchmark.slow": 0.831 ±(99.9%) 0.011 us/op [Average] (min, avg, max) = (0.807, 0.831, 0.856), stdev = 0.015 CI (99.9%): [0.820, 0.842] (assumes normal distribution) Benchmark Mode Cnt Score Error Units BuildPrivilegeNameBenchmark.fast avgt 25 0.344 ± 0.004 us/op BuildPrivilegeNameBenchmark.slow avgt 25 0.831 ± 0.011 us/op This patch also updates SentryAuthorizationPolicy.listPrivileges() to reuse the privilege names that have already been built instead of building them again. While fixing this, I found a bug where Principal stores the PrincipalPrivilege in a case insensitive way. This is true for all privilege scopes, except URI. This patch fixes the issue by making privilege name to be case sensitive instead. This patch removes incorrect synchronization in SentryAuthorizationPolicy.listPrivileges() that can cause the operation to run in serial in a highly concurrent workload. Testing: - Ran all FE tests - Ran all E2E authorization tests - Added E2E test for privilege name case sensitivity bug Change-Id: I942d9b55f07c8972f69e532567d9b7d80fceb6e5 Reviewed-on: http://gerrit.cloudera.org:8080/13095 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/authorization/AuthorizationPolicy.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationPolicy.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryProxy.java M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/CatalogObjectCache.java M fe/src/main/java/org/apache/impala/catalog/Principal.java M fe/src/main/java/org/apache/impala/catalog/PrincipalPrivilege.java M tests/authorization/test_grant_revoke.py 8 files changed, 141 insertions(+), 126 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/13095 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I942d9b55f07c8972f69e532567d9b7d80fceb6e5 Gerrit-Change-Number: 13095 Gerrit-PatchSet: 12 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8444: Fix performance regression when building privilege name
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13095 ) Change subject: IMPALA-8444: Fix performance regression when building privilege name .. Patch Set 11: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13095 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I942d9b55f07c8972f69e532567d9b7d80fceb6e5 Gerrit-Change-Number: 13095 Gerrit-PatchSet: 11 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 26 Apr 2019 21:11:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8381: Optimize ParquetPlainEncoder::DecodeBatch() for simple types
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12985 ) Change subject: IMPALA-8381: Optimize ParquetPlainEncoder::DecodeBatch() for simple types .. IMPALA-8381: Optimize ParquetPlainEncoder::DecodeBatch() for simple types Refactored the ParquetPlainEncoder::Decode() and ParquetPlainEncoder::DecodeBatch() methods to increase performance in batch decoding. The `Decode` and `DecodeBatch` methods retain their behaviour and outward interface, but the internal structure changes. We change how we split up the `Decode` template specialisations. The generic unspecialised template is used for numerical parquet types (INT32, INT64, INT96, FLOAT and DOUBLE) and various specialisations are used for BYTE_ARRAY and FIXED_LEN_BYTE_ARRAY. We add a new method template, DecodeNoCheck, which does the actual decoding without bounds checking. It is called by the generic Decode method template internally. For all parquet types except for BYTE_ARRAY, DecodeBatch performs the bounds check once for the whole batch at the same time and calls DecodeNoCheck, so we save the cost of bounds checking for every decoded value. For BYTE_ARRAY, this cannot be done and we have to perform the checks for every value. In the non-BYTE_ARRAY version of DecodeBatch, we explicitly unroll the loop in batches of 8 to increase performance. The overall performance increase is up to 2x for small strides (8 bytes, INT32) but decreases as the stride increases, and disappears from around 40 bytes. With bigger strides, there is no performance difference from the previous implementation. Testing: Added tests to parquet-plain-test.cc to test the `Decode` and the `DecodeBatch` methods both in single-value decoding and batch decoding. Change-Id: I57b7d2573bb6dfd038e581acb3bd8ea1565aa20d Reviewed-on: http://gerrit.cloudera.org:8080/12985 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/exec/parquet/parquet-common.h M be/src/exec/parquet/parquet-plain-test.cc A be/src/testutil/random-vector-generators.h 3 files changed, 468 insertions(+), 95 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/12985 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I57b7d2573bb6dfd038e581acb3bd8ea1565aa20d Gerrit-Change-Number: 12985 Gerrit-PatchSet: 17 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-8381: Optimize ParquetPlainEncoder::DecodeBatch() for simple types
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12985 ) Change subject: IMPALA-8381: Optimize ParquetPlainEncoder::DecodeBatch() for simple types .. Patch Set 16: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12985 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I57b7d2573bb6dfd038e581acb3bd8ea1565aa20d Gerrit-Change-Number: 12985 Gerrit-PatchSet: 16 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 26 Apr 2019 21:08:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8463: fix skip.header.line.count on local catalog
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/13131 ) Change subject: IMPALA-8463: fix skip.header.line.count on local catalog .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/13131/4/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java: http://gerrit.cloudera.org:8080/#/c/13131/4/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@1122 PS4, Line 1122: public static int parseSkipHeaderLineCount(Map tblProperties, can this copy of the function be removed now? -- To view, visit http://gerrit.cloudera.org:8080/13131 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3957b3661574c356a14197e8829abbed92028a1a Gerrit-Change-Number: 13131 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Fri, 26 Apr 2019 21:03:41 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8463: fix skip.header.line.count on local catalog
Hello Bharath Vissapragada, Todd Lipcon, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13131 to look at the new patch set (#4). Change subject: IMPALA-8463: fix skip.header.line.count on local catalog .. IMPALA-8463: fix skip.header.line.count on local catalog This moves the logic to FeFsTable and calls it from LocalFsTable. Testing: Added a unit test that repros the problem. End-to-end tests run against local catalog will catch this but aren't enabled. Change-Id: I3957b3661574c356a14197e8829abbed92028a1a --- M fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java M fe/src/main/java/org/apache/impala/catalog/FeFsTable.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java 5 files changed, 63 insertions(+), 30 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/13131/4 -- To view, visit http://gerrit.cloudera.org:8080/13131 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3957b3661574c356a14197e8829abbed92028a1a Gerrit-Change-Number: 13131 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8463: fix skip.header.line.count on local catalog
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13131 ) Change subject: IMPALA-8463: fix skip.header.line.count on local catalog .. Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/13131/2/fe/src/main/java/org/apache/impala/catalog/FeFsTable.java File fe/src/main/java/org/apache/impala/catalog/FeFsTable.java: http://gerrit.cloudera.org:8080/#/c/13131/2/fe/src/main/java/org/apache/impala/catalog/FeFsTable.java@501 PS2, Line 501: > line too long (91 > 90) Done http://gerrit.cloudera.org:8080/#/c/13131/1/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java File fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java: http://gerrit.cloudera.org:8080/#/c/13131/1/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java@524 PS1, Line 524: org.apache.hadoop.hive.metastore.api.Table msTbl = getMetaStoreTable(); > Can we just move the code from HdfsTable.java into FeFsTable.Utils and call Done -- To view, visit http://gerrit.cloudera.org:8080/13131 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3957b3661574c356a14197e8829abbed92028a1a Gerrit-Change-Number: 13131 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Fri, 26 Apr 2019 20:43:50 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8463: fix skip.header.line.count on local catalog
Hello Bharath Vissapragada, Todd Lipcon, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13131 to look at the new patch set (#3). Change subject: IMPALA-8463: fix skip.header.line.count on local catalog .. IMPALA-8463: fix skip.header.line.count on local catalog This adds missing logic based on the existing implementation for the traditional catalog tables. Testing: Added a unit test that repros the problem. End-to-end tests run against local catalog will catch this but aren't enabled. Change-Id: I3957b3661574c356a14197e8829abbed92028a1a --- M fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java M fe/src/main/java/org/apache/impala/catalog/FeFsTable.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java 5 files changed, 63 insertions(+), 30 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/13131/3 -- To view, visit http://gerrit.cloudera.org:8080/13131 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3957b3661574c356a14197e8829abbed92028a1a Gerrit-Change-Number: 13131 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8463: fix skip.header.line.count on local catalog
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13131 ) Change subject: IMPALA-8463: fix skip.header.line.count on local catalog .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/13131/2/fe/src/main/java/org/apache/impala/catalog/FeFsTable.java File fe/src/main/java/org/apache/impala/catalog/FeFsTable.java: http://gerrit.cloudera.org:8080/#/c/13131/2/fe/src/main/java/org/apache/impala/catalog/FeFsTable.java@501 PS2, Line 501: Preconditions.checkState(tblProperties.containsKey(TBL_PROP_SKIP_HEADER_LINE_COUNT)); line too long (91 > 90) -- To view, visit http://gerrit.cloudera.org:8080/13131 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3957b3661574c356a14197e8829abbed92028a1a Gerrit-Change-Number: 13131 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Fri, 26 Apr 2019 20:39:53 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8463: fix skip.header.line.count on local catalog
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/13131 ) Change subject: IMPALA-8463: fix skip.header.line.count on local catalog .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/13131/1/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java File fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java: http://gerrit.cloudera.org:8080/#/c/13131/1/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java@524 PS1, Line 524: org.apache.hadoop.hive.metastore.api.Table msTbl = getMetaStoreTable(); Can we just move the code from HdfsTable.java into FeFsTable.Utils and call it from here? Or, now that we are Java8-capable, move it into FeFsTable as a default interface method? -- To view, visit http://gerrit.cloudera.org:8080/13131 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3957b3661574c356a14197e8829abbed92028a1a Gerrit-Change-Number: 13131 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Fri, 26 Apr 2019 19:31:40 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 ) Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0 .. Patch Set 10: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2938/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13005 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436 Gerrit-Change-Number: 13005 Gerrit-PatchSet: 10 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 26 Apr 2019 19:23:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 ) Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0 .. Patch Set 10: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4088/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/13005 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436 Gerrit-Change-Number: 13005 Gerrit-PatchSet: 10 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 26 Apr 2019 18:29:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/13005 )
Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive
3.1.0
..
Patch Set 10:
(25 comments)
http://gerrit.cloudera.org:8080/#/c/13005/10/bin/impala-config.sh
File bin/impala-config.sh:
http://gerrit.cloudera.org:8080/#/c/13005/10/bin/impala-config.sh@196
PS10, Line 196: # When USE_CDP_HIVE is set we use the latest hive version
available to deply in minicluster
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/13005/10/bin/impala-config.sh@200
PS10, Line 200: # TODO(Vihang) we should repackage the tarballs so that the
src and binaries are extracted
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/13005/10/bin/impala-config.sh@209
PS10, Line 209: export
HIVE_HOME="$IMPALA_TOOLCHAIN/cdh_components-${CDH_BUILD_NUMBER}/hive-${MINICLUSTER_HIVE_VERSION}"
line too long (106 > 90)
http://gerrit.cloudera.org:8080/#/c/13005/10/bin/impala-config.sh@546
PS10, Line 546: export
HIVE_METASTORE_THRIFT_DIR=$CDP_COMPONENTS_HOME/apache-hive-${IMPALA_HIVE_VERSION}-src/standalone-metastore/src/main/thrift
line too long (129 > 90)
http://gerrit.cloudera.org:8080/#/c/13005/10/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/13005/10/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1073
PS10, Line 1073:
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/13005/10/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1169
PS10, Line 1169:
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/13005/10/testdata/bin/run-hive-server.sh
File testdata/bin/run-hive-server.sh:
http://gerrit.cloudera.org:8080/#/c/13005/10/testdata/bin/run-hive-server.sh@66
PS10, Line 66: export HIVE_METASTORE_HADOOP_OPTS="-verbose:class -Xdebug
-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=30010"
line too long (121 > 90)
http://gerrit.cloudera.org:8080/#/c/13005/10/tests/authorization/test_owner_privileges.py
File tests/authorization/test_owner_privileges.py:
http://gerrit.cloudera.org:8080/#/c/13005/10/tests/authorization/test_owner_privileges.py@103
PS10, Line 103: )
flake8: E501 line too long (91 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/10/tests/authorization/test_owner_privileges.py@179
PS10, Line 179: =
flake8: E501 line too long (93 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/10/tests/authorization/test_owner_privileges.py@182
PS10, Line 182: =
flake8: E501 line too long (93 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/10/tests/authorization/test_owner_privileges.py@186
PS10, Line 186: )
flake8: E501 line too long (91 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/10/tests/authorization/test_owner_privileges.py@193
PS10, Line 193: ,
flake8: E501 line too long (91 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/10/tests/authorization/test_owner_privileges.py@213
PS10, Line 213: )
flake8: E501 line too long (91 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/10/tests/authorization/test_owner_privileges.py@226
PS10, Line 226: )
flake8: E501 line too long (91 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/10/tests/authorization/test_owner_privileges.py@236
PS10, Line 236: )
flake8: E501 line too long (91 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/10/tests/authorization/test_owner_privileges.py@282
PS10, Line 282: _
flake8: E501 line too long (100 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/10/tests/authorization/test_owner_privileges.py@287
PS10, Line 287: t
flake8: E501 line too long (101 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/10/tests/authorization/test_owner_privileges.py@293
PS10, Line 293: 2
flake8: E501 line too long (92 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/10/tests/authorization/test_owner_privileges.py@355
PS10, Line 355: _
flake8: E501 line too long (100 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/10/tests/authorization/test_owner_privileges.py@360
PS10, Line 360: t
flake8: E501 line too long (101 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/10/tests/authorization/test_owner_privileges.py@364
PS10, Line 364: 2
flake8: E501 line too long (92 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/10/tests/authorization/test_owner_privileges.py@368
PS10, Line 368: )
flake8: E501 line too long (91 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/10/tests/authorization/test_owner_privileges.py@412
PS10, Line 412: t
flake8: E501 line too long (101 > 90 characters)
[Impala-ASF-CR] IMPALA-8454 (part 3): enable recursive file listing by default
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13127 ) Change subject: IMPALA-8454 (part 3): enable recursive file listing by default .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2936/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13127 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib30e2bcaf820210f2faa8f159d1af2f947a4d0e8 Gerrit-Change-Number: 13127 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 26 Apr 2019 18:21:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8463: fix skip.header.line.count on local catalog
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13131 ) Change subject: IMPALA-8463: fix skip.header.line.count on local catalog .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2937/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13131 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3957b3661574c356a14197e8829abbed92028a1a Gerrit-Change-Number: 13131 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Fri, 26 Apr 2019 18:22:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
Vihang Karajgaonkar has uploaded a new patch set (#10). ( http://gerrit.cloudera.org:8080/13005 ) Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0 .. IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0 This change upgrades the hive dependencies of Impala to use Hive 3.1.0 based binaries. Most of the changes in this patch are based off patches provided by Todd (links available in JIRA). Upgrading the dependencies allows us to work with both Hive 3.1.0 and Hive 2.1.0 in the same code line. In order to do this, the patch trims down a lot of unnecessary hive dependencies of the front end code by creating a shaded-deps module. The pom.xml of shaded-deps includes only the files from Hive source which Impala depends for compilation. Additionally, it also uses a custom build of Hive which is based of Hive 3.1.0. This custom build includes patches for HIVE-21596 and HIVE-21586 which are needed by Impala so that it can compile against Hive-3 libraries and be able to talk to HMS-2.x metastore. Once these patches are merged we can get rid of this custom build and rely on more official sources of Hive builds. The patch also changes impala-config.sh so that it always downloads the Hive-3 libraries from the toolchain. The code is always built using Hive-3 jars. However, based on the value of USE_CDP_HIVE, the minicluster is deployed using Hive-3 or Hive-2 binaries. Since Impala implements HiveServer2's TCLIService.thrift interface, it requires us to use the existing mechanism of copying the hive-2/api TCLIService.thrift to hive-3/api. It also adds a few environment variables which point to the metastore's thrift file and the CDH Hive version. Testing: 1. Code compiles and runs against both HMS-3 and HMS-2 2. Ran full-suite of tests using the private jenkins job against HMS-2 3. Running full-tests against HMS-3 will need more work like supporting Tez in the mini-cluster (for dataloading) and HMS transaction support since HMS3 create transactional tables by default. This will be taken up in subsequent change. Notes: 1. Patch uses a custom build of Hive to be deployed in mini-cluster. This build has the fixes for HIVE-21596 2. The Hive 3.1 does not include some of the patches like HIVE-21077. Some of the recent code in Impala depends on that patch to exist in hive. We will need to port HIVE-21077 to Hive 3.1.0 in order to compile against Hive 3.1.0 jars. I will continue working on getting this merged along with HIVE-21586 3. Some of the existing tests rely on the fact the UDFs implement the UDF interface in Hive (UDFLength, UDFHour, UDFYear). These built-in hive functions have been moved to use GenericUDF interface in Hive 3. Impala currently only supports UDFExecutor. In order to have a full compatibility with all the functions in Hive 2.x we should support GenericUDFs too. That would be taken up as a separate patch. 4. The Sentry object ownership tests are flaky. There are race-conditions in that code which get exposed for some by this patch. For example, when a database is created, the object privileges are immediately updated in the catalog cache. If Sentry has not been updated and there is a refresh authorization call in between it can clear the privilege from the cache causing the subsequent statement using that privilege to fail. The patch adds sleep statements in the test to work-around these issues. Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436 --- M CMakeLists.txt M README.md M bin/bootstrap_toolchain.py M bin/impala-config.sh M bin/set-classpath.sh M common/thrift/.gitignore M common/thrift/CMakeLists.txt M fe/CMakeLists.txt M fe/pom.xml M fe/src/main/java/org/apache/impala/analysis/StringLiteral.java M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java M fe/src/main/java/org/apache/impala/catalog/FeHBaseTable.java M fe/src/main/java/org/apache/impala/catalog/MetaStoreClientPool.java M fe/src/main/java/org/apache/impala/catalog/TableLoader.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java M fe/src/main/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/hive/executor/UdfExecutor.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java M fe/src/main/java/org/apache/impala/service/MetadataOp.java A fe/src/main/java/org/apache/impala/util/MetadataFormatUtils.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java M fe/src/test/java/org/apache/impala/hive/executor/UdfExecutorTest.java M fe/src/test/java/org/apache/impala/testutil/EmbeddedMetastoreClientPool.java M impala-parent/pom.xml A shaded-deps/.gitignore A
[Impala-ASF-CR] IMPALA-8454 (part 3): enable recursive file listing by default
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13127 ) Change subject: IMPALA-8454 (part 3): enable recursive file listing by default .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4087/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/13127 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib30e2bcaf820210f2faa8f159d1af2f947a4d0e8 Gerrit-Change-Number: 13127 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 26 Apr 2019 17:53:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8454 (part 2): Initial support for recursive file listing within a partition
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12991 ) Change subject: IMPALA-8454 (part 2): Initial support for recursive file listing within a partition .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4086/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/12991 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9b151d7abb8443c0d9de0a0d82a9f13e07ad5109 Gerrit-Change-Number: 12991 Gerrit-PatchSet: 4 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 26 Apr 2019 17:53:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8149 : Add support for alter database events
Fredy Wijaya has posted comments on this change. (
http://gerrit.cloudera.org:8080/13049 )
Change subject: IMPALA-8149 : Add support for alter_database events
..
Patch Set 4:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/13049/4/fe/src/main/java/org/apache/impala/catalog/Db.java
File fe/src/main/java/org/apache/impala/catalog/Db.java:
http://gerrit.cloudera.org:8080/#/c/13049/4/fe/src/main/java/org/apache/impala/catalog/Db.java@523
PS4, Line 523: + " its max capacity %d. Ignoring add request for
version number %d. This "
: + "could cause unnecessary database invalidation
when the event is "
:
nit: fix indentation, 4 spaces for continued indentation
http://gerrit.cloudera.org:8080/#/c/13049/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/13049/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@439
PS1, Line 439: protected void initSelfEventIdentifiersFromEvent() {
: throw new UnsupportedOperationException("Please override
this method in subclass");
: }
> I tried to make it abstract, but some subclasses don't need call initSelfEv
In general, it's always a good idea to force the implementers to make a
conscious decision whether or not to override for compile-time safety rather
than forgetting to override the method and get a runtime exception. However, I
have no string opinion on this since the subclasses are isolated. Perhaps, we
can rename the exception message to use:
throw new UnsupportedOperationException(String.format(
"%s is not supported", ClassUtil.getMethodName()));
http://gerrit.cloudera.org:8080/#/c/13049/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@445
PS1, Line 445: return params.getOr
> So shall I remove this check? As empty case is handled by next line getOrDe
We can just leave it for now. Removing the check is wrong if params can be null
and L446 will throw an NPE.
http://gerrit.cloudera.org:8080/#/c/13049/4/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
File
fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java:
http://gerrit.cloudera.org:8080/#/c/13049/4/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@343
PS4, Line 343:
nit: fix indentation
--
To view, visit http://gerrit.cloudera.org:8080/13049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf020e85cae04163bf32e31363eb4119d624640b
Gerrit-Change-Number: 13049
Gerrit-PatchSet: 4
Gerrit-Owner: Anonymous Coward
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Anurag Mantripragada
Gerrit-Reviewer: Bharath Krishna
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Comment-Date: Fri, 26 Apr 2019 17:36:37 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8463: fix skip.header.line.count on local catalog
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13131 Change subject: IMPALA-8463: fix skip.header.line.count on local catalog .. IMPALA-8463: fix skip.header.line.count on local catalog This adds missing logic based on the existing implementation for the traditional catalog tables. Testing: Added a unit test that repros the problem. End-to-end tests run against local catalog will catch this but aren't enabled. Change-Id: I3957b3661574c356a14197e8829abbed92028a1a --- M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java 2 files changed, 24 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/13131/1 -- To view, visit http://gerrit.cloudera.org:8080/13131 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I3957b3661574c356a14197e8829abbed92028a1a Gerrit-Change-Number: 13131 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] IMPALA-8454 (part 3): enable recursive file listing by default
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/13127 ) Change subject: IMPALA-8454 (part 3): enable recursive file listing by default .. Patch Set 1: (1 comment) > Patch Set 1: Code-Review+2 > > (1 comment) > > Looks pretty good to me. http://gerrit.cloudera.org:8080/#/c/13127/1/tests/metadata/test_recursive_listing.py File tests/metadata/test_recursive_listing.py: http://gerrit.cloudera.org:8080/#/c/13127/1/tests/metadata/test_recursive_listing.py@59 PS1, Line 59: execute > nit: execute_query_expect_success in multiple places? (does an extra assert Done -- To view, visit http://gerrit.cloudera.org:8080/13127 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib30e2bcaf820210f2faa8f159d1af2f947a4d0e8 Gerrit-Change-Number: 13127 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 26 Apr 2019 17:17:49 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8454 (part 3): enable recursive file listing by default
Hello Bharath Vissapragada, Vihang Karajgaonkar, Sudhanshu Arora, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13127 to look at the new patch set (#2). Change subject: IMPALA-8454 (part 3): enable recursive file listing by default .. IMPALA-8454 (part 3): enable recursive file listing by default This enables recursive listing of files within partition directories by default. This is a behavior change, but in fact makes Impala consistent with modern versions of Hive, Spark, Presto, etc. In fact, this is necessary for querying certain Hive tables which have been written out by a query containing a UNION ALL clause if that query is executed by Tez (see HIVE-12812 for example). Technically, this is an incompatible change. Although it's unlikely people were relying on the non-recursive listing, this patch offers two escape hatches: - an individual table may be marked with the 'impala.disable.recursive.listing' property - impala may be globally configured with --recursively_list_partitions=false Given that we know this behavior is inconsistent with other SQL engines, and that there is no performance benefit to not recursing in the common case that there _are_ no subdirectories, I made the flag "hidden" and did not document the new table property. These are only "chicken bit" flags. Change-Id: Ib30e2bcaf820210f2faa8f159d1af2f947a4d0e8 --- M be/src/common/global-flags.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/catalog/FeFsTable.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java A tests/metadata/test_recursive_listing.py 8 files changed, 151 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/27/13127/2 -- To view, visit http://gerrit.cloudera.org:8080/13127 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib30e2bcaf820210f2faa8f159d1af2f947a4d0e8 Gerrit-Change-Number: 13127 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-7957: Fix slot equivalences may be enforced multiple times
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13051 ) Change subject: IMPALA-7957: Fix slot equivalences may be enforced multiple times .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/13051/3/fe/src/main/java/org/apache/impala/analysis/Analyzer.java File fe/src/main/java/org/apache/impala/analysis/Analyzer.java: http://gerrit.cloudera.org:8080/#/c/13051/3/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@221 PS3, Line 221: public final Map, ExprId> inferredConjunctsBySlots = > Can you document the invariant that both orders of every pair must be added Maybe this isn't relevant any more if we change the approach -- To view, visit http://gerrit.cloudera.org:8080/13051 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ida2d5d8149b217e18ebae61e136848162503653e Gerrit-Change-Number: 13051 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 26 Apr 2019 16:50:07 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8444: Fix performance regression when building privilege name
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13095 ) Change subject: IMPALA-8444: Fix performance regression when building privilege name .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2934/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13095 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I942d9b55f07c8972f69e532567d9b7d80fceb6e5 Gerrit-Change-Number: 13095 Gerrit-PatchSet: 9 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 26 Apr 2019 16:52:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7957: Fix slot equivalences may be enforced multiple times
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13051 ) Change subject: IMPALA-7957: Fix slot equivalences may be enforced multiple times .. Patch Set 3: (4 comments) I'm afraid I found a case that this doesn't handle, i.e. the solution is incomplete. I think the issue with the current approach is that it tracks the exact set of predicates that were placed, but createEquivConjuncts() can enumerate predicates derived from any pair of slots in the equivalence class. I think this is on the right track, but I think maybe we need to take the set of predicates that were already assigned for pairs of slots within the tuple, then union the sets in partialEquivSlots() based on those predicates. SELECT t.id, t2.id FROM functional.alltypestiny t LEFT JOIN (SELECT id, int_col, smallint_col FROM functional.alltypestiny WHERE int_col = id and smallint_col = id and tinyint_col = id) t2 ON (t.id = t2.id) UNION ALL VALUES (NULL, NULL) http://gerrit.cloudera.org:8080/#/c/13051/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/13051/3//COMMIT_MSG@43 PS3, Line 43: * Run all tests locally in CORE exploration strategy Can you also add the repro query to an end-to-end inline-view test, just to confirm that it produces the expected results. http://gerrit.cloudera.org:8080/#/c/13051/3/fe/src/main/java/org/apache/impala/analysis/Analyzer.java File fe/src/main/java/org/apache/impala/analysis/Analyzer.java: http://gerrit.cloudera.org:8080/#/c/13051/3/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@220 PS3, Line 220: infereed inferred http://gerrit.cloudera.org:8080/#/c/13051/3/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@221 PS3, Line 221: public final Map, ExprId> inferredConjunctsBySlots = Can you document the invariant that both orders of every pair must be added, i.e. inferredConjunctsBySlots[(a, b)] = c <=> inferredConjunctsBySlots[(b, a)] = c? Or actually, maybe it's better to normalise the pairs, e.g. so that the lower SlotId is always the first one in the pair. http://gerrit.cloudera.org:8080/#/c/13051/3/testdata/workloads/functional-planner/queries/PlannerTest/inline-view.test File testdata/workloads/functional-planner/queries/PlannerTest/inline-view.test: http://gerrit.cloudera.org:8080/#/c/13051/3/testdata/workloads/functional-planner/queries/PlannerTest/inline-view.test@1833 PS3, Line 1833: Some more test cases: # Add the same predicate *outside* the left join, it should still be enforced. SELECT t.id FROM functional.alltypestiny t LEFT JOIN (SELECT id, int_col FROM functional.alltypestiny WHERE int_col = id) t2 ON (t.id = t2.id) where t2.id = t2.int_col UNION ALL VALUES (NULL); # Same thing except with a predicate on a different column (i.e. adding it to the equivalence class) SELECT t.id, t2.id FROM functional.alltypestiny t LEFT JOIN (SELECT id, int_col, smallint_col FROM functional.alltypestiny WHERE int_col = id) t2 ON (t.id = t2.id) where t2.int_col = t2.smallint_col UNION ALL VALUES (NULL, NULL) # Multiple predicates that must not be placed above the join SELECT t.id, t2.id FROM functional.alltypestiny t LEFT JOIN (SELECT id, int_col, smallint_col FROM functional.alltypestiny WHERE int_col = id and smallint_col = id) t2 ON (t.id = t2.id) UNION ALL VALUES (NULL, NULL); -- To view, visit http://gerrit.cloudera.org:8080/13051 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ida2d5d8149b217e18ebae61e136848162503653e Gerrit-Change-Number: 13051 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 26 Apr 2019 16:49:29 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5351: Support storing column comment of kudu table
Fredy Wijaya has posted comments on this change. (
http://gerrit.cloudera.org:8080/12977 )
Change subject: IMPALA-5351: Support storing column comment of kudu table
..
Patch Set 7:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/12977/7/fe/src/main/java/org/apache/impala/catalog/KuduColumn.java
File fe/src/main/java/org/apache/impala/catalog/KuduColumn.java:
http://gerrit.cloudera.org:8080/#/c/12977/7/fe/src/main/java/org/apache/impala/catalog/KuduColumn.java@116
PS7, Line 116:
nit: remove two extra spaces
http://gerrit.cloudera.org:8080/#/c/12977/7/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
File fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java:
http://gerrit.cloudera.org:8080/#/c/12977/7/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java@2607
PS7, Line 2607:
nit: remove one extra space for the indentation
http://gerrit.cloudera.org:8080/#/c/12977/7/tests/metadata/test_ddl.py
File tests/metadata/test_ddl.py:
http://gerrit.cloudera.org:8080/#/c/12977/7/tests/metadata/test_ddl.py@733
PS7, Line 733: create table {0} (i int PRIMARY KEY) STORED AS KUDU
nit: use lower case to be consistent with the style in this file
http://gerrit.cloudera.org:8080/#/c/12977/7/tests/metadata/test_ddl.py@735
PS7, Line 735: comment = self._get_column_comment(table, 'i')
can we have similar test case for "create table {0} (x int comment 'x', primary
key x) stored as kudu"?
http://gerrit.cloudera.org:8080/#/c/12977/7/tests/metadata/test_ddl_base.py
File tests/metadata/test_ddl_base.py:
http://gerrit.cloudera.org:8080/#/c/12977/7/tests/metadata/test_ddl_base.py@122
PS7, Line 122: or len(cols) == 9
if len(cols) <= 9 is better since it will work on both kudu and on kudu tables
--
To view, visit http://gerrit.cloudera.org:8080/12977
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifb3b37eed364f12bdb3c1d7ef5be128f1475936c
Gerrit-Change-Number: 12977
Gerrit-PatchSet: 7
Gerrit-Owner: helifu
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Thomas Marshall
Gerrit-Reviewer: helifu
Gerrit-Comment-Date: Fri, 26 Apr 2019 16:13:34 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8444: Fix performance regression when building privilege name
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13095 ) Change subject: IMPALA-8444: Fix performance regression when building privilege name .. Patch Set 11: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4084/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13095 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I942d9b55f07c8972f69e532567d9b7d80fceb6e5 Gerrit-Change-Number: 13095 Gerrit-PatchSet: 11 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 26 Apr 2019 16:07:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8444: Fix performance regression when building privilege name
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13095 ) Change subject: IMPALA-8444: Fix performance regression when building privilege name .. Patch Set 11: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13095 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I942d9b55f07c8972f69e532567d9b7d80fceb6e5 Gerrit-Change-Number: 13095 Gerrit-PatchSet: 11 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 26 Apr 2019 16:07:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8444: Fix performance regression when building privilege name
Fredy Wijaya has uploaded a new patch set (#10). ( http://gerrit.cloudera.org:8080/13095 ) Change subject: IMPALA-8444: Fix performance regression when building privilege name .. IMPALA-8444: Fix performance regression when building privilege name This patch fixes the performance regression when building privilege name by rewriting PrincipalPrivilege.buildPrivilegeName() with a simple string concatentation instead of using a list that gets converted into a string. Below is the result of running a benchmark using JMH comparing the old and new implementations: Result "org.apache.impala.BuildPrivilegeNameBenchmark.fast": 0.344 ±(99.9%) 0.004 us/op [Average] (min, avg, max) = (0.336, 0.344, 0.355), stdev = 0.005 CI (99.9%): [0.339, 0.348] (assumes normal distribution) Result "org.apache.impala.BuildPrivilegeNameBenchmark.slow": 0.831 ±(99.9%) 0.011 us/op [Average] (min, avg, max) = (0.807, 0.831, 0.856), stdev = 0.015 CI (99.9%): [0.820, 0.842] (assumes normal distribution) Benchmark Mode Cnt Score Error Units BuildPrivilegeNameBenchmark.fast avgt 25 0.344 ± 0.004 us/op BuildPrivilegeNameBenchmark.slow avgt 25 0.831 ± 0.011 us/op This patch also updates SentryAuthorizationPolicy.listPrivileges() to reuse the privilege names that have already been built instead of building them again. While fixing this, I found a bug where Principal stores the PrincipalPrivilege in a case insensitive way. This is true for all privilege scopes, except URI. This patch fixes the issue by making privilege name to be case sensitive instead. This patch removes incorrect synchronization in SentryAuthorizationPolicy.listPrivileges() that can cause the operation to run in serial in a highly concurrent workload. Testing: - Ran all FE tests - Ran all E2E authorization tests - Added E2E test for privilege name case sensitivity bug Change-Id: I942d9b55f07c8972f69e532567d9b7d80fceb6e5 --- M fe/src/main/java/org/apache/impala/authorization/AuthorizationPolicy.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationPolicy.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryProxy.java M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/CatalogObjectCache.java M fe/src/main/java/org/apache/impala/catalog/Principal.java M fe/src/main/java/org/apache/impala/catalog/PrincipalPrivilege.java M tests/authorization/test_grant_revoke.py 8 files changed, 141 insertions(+), 126 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/95/13095/10 -- To view, visit http://gerrit.cloudera.org:8080/13095 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I942d9b55f07c8972f69e532567d9b7d80fceb6e5 Gerrit-Change-Number: 13095 Gerrit-PatchSet: 10 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8444: Fix performance regression when building privilege name
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13095 ) Change subject: IMPALA-8444: Fix performance regression when building privilege name .. Patch Set 10: Code-Review+2 (1 comment) Carry Csaba's +2 http://gerrit.cloudera.org:8080/#/c/13095/9//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/13095/9//COMMIT_MSG@45 PS9, Line 45: sensitivity > nit: probably "case sensitivity" would be better Done -- To view, visit http://gerrit.cloudera.org:8080/13095 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I942d9b55f07c8972f69e532567d9b7d80fceb6e5 Gerrit-Change-Number: 13095 Gerrit-PatchSet: 10 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 26 Apr 2019 16:06:58 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8444: Fix performance regression when building privilege name
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/13095 ) Change subject: IMPALA-8444: Fix performance regression when building privilege name .. Patch Set 9: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/13095/9//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/13095/9//COMMIT_MSG@45 PS9, Line 45: insensitive nit: probably "case sensitivity" would be better -- To view, visit http://gerrit.cloudera.org:8080/13095 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I942d9b55f07c8972f69e532567d9b7d80fceb6e5 Gerrit-Change-Number: 13095 Gerrit-PatchSet: 9 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 26 Apr 2019 16:04:45 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8444: Fix performance regression when building privilege name
Fredy Wijaya has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/13095 ) Change subject: IMPALA-8444: Fix performance regression when building privilege name .. IMPALA-8444: Fix performance regression when building privilege name This patch fixes the performance regression when building privilege name by rewriting PrincipalPrivilege.buildPrivilegeName() with a simple string concatentation instead of using a list that gets converted into a string. Below is the result of running a benchmark using JMH comparing the old and new implementations: Result "org.apache.impala.BuildPrivilegeNameBenchmark.fast": 0.344 ±(99.9%) 0.004 us/op [Average] (min, avg, max) = (0.336, 0.344, 0.355), stdev = 0.005 CI (99.9%): [0.339, 0.348] (assumes normal distribution) Result "org.apache.impala.BuildPrivilegeNameBenchmark.slow": 0.831 ±(99.9%) 0.011 us/op [Average] (min, avg, max) = (0.807, 0.831, 0.856), stdev = 0.015 CI (99.9%): [0.820, 0.842] (assumes normal distribution) Benchmark Mode Cnt Score Error Units BuildPrivilegeNameBenchmark.fast avgt 25 0.344 ± 0.004 us/op BuildPrivilegeNameBenchmark.slow avgt 25 0.831 ± 0.011 us/op This patch also updates SentryAuthorizationPolicy.listPrivileges() to reuse the privilege names that have already been built instead of building them again. While fixing this, I found a bug where Principal stores the PrincipalPrivilege in a case insensitive way. This is true for all privilege scopes, except URI. This patch fixes the issue by making privilege name to be case sensitive instead. This patch removes incorrect synchronization in SentryAuthorizationPolicy.listPrivileges() that can cause the operation to run in serial in a highly concurrent workload. Testing: - Ran all FE tests - Ran all E2E authorization tests - Added E2E test for privilege name case insensitive bug Change-Id: I942d9b55f07c8972f69e532567d9b7d80fceb6e5 --- M fe/src/main/java/org/apache/impala/authorization/AuthorizationPolicy.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationPolicy.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryProxy.java M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/CatalogObjectCache.java M fe/src/main/java/org/apache/impala/catalog/Principal.java M fe/src/main/java/org/apache/impala/catalog/PrincipalPrivilege.java M tests/authorization/test_grant_revoke.py 8 files changed, 141 insertions(+), 126 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/95/13095/9 -- To view, visit http://gerrit.cloudera.org:8080/13095 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I942d9b55f07c8972f69e532567d9b7d80fceb6e5 Gerrit-Change-Number: 13095 Gerrit-PatchSet: 9 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8444: Fix performance regression when building privilege name
Fredy Wijaya has posted comments on this change. (
http://gerrit.cloudera.org:8080/13095 )
Change subject: IMPALA-8444: Fix performance regression when building privilege
name
..
Patch Set 9:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/13095/8//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/13095/8//COMMIT_MSG@8
PS8, Line 8:
> The commit message could also reflect that there are also some fixes relate
Done
http://gerrit.cloudera.org:8080/#/c/13095/8//COMMIT_MSG@33
PS8, Line 33: While fixing this, I found a bug where Principal
: stores the PrincipalPrivilege in a case insensitive way. This is
true
: for all privilege scopes, except URI. This patch fixes the issue
by
: making privilege name to be case sensitiv
> Is this still correct?
Comment updated. Done.
http://gerrit.cloudera.org:8080/#/c/13095/8/fe/src/main/java/org/apache/impala/catalog/CatalogObjectCache.java
File fe/src/main/java/org/apache/impala/catalog/CatalogObjectCache.java:
http://gerrit.cloudera.org:8080/#/c/13095/8/fe/src/main/java/org/apache/impala/catalog/CatalogObjectCache.java@158
PS8, Line 158: }
> Is there some kind of whitespace change here?
I think this is a gerrit bug. I've seen this before. There should not be any
change in this file.
http://gerrit.cloudera.org:8080/#/c/13095/2/fe/src/main/java/org/apache/impala/catalog/Principal.java
File fe/src/main/java/org/apache/impala/catalog/Principal.java:
http://gerrit.cloudera.org:8080/#/c/13095/2/fe/src/main/java/org/apache/impala/catalog/Principal.java@63
PS2, Line 63:*/
: public boolean addPrivilege(PrincipalPrivilege privilege) {
: return principalPrivileges_.add(privilege);
> Can you add tests for the bugs that were fixed?
Done
--
To view, visit http://gerrit.cloudera.org:8080/13095
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I942d9b55f07c8972f69e532567d9b7d80fceb6e5
Gerrit-Change-Number: 13095
Gerrit-PatchSet: 9
Gerrit-Owner: Fredy Wijaya
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Comment-Date: Fri, 26 Apr 2019 15:52:54 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8381: Optimize ParquetPlainEncoder::DecodeBatch() for simple types
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12985 ) Change subject: IMPALA-8381: Optimize ParquetPlainEncoder::DecodeBatch() for simple types .. Patch Set 16: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12985 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I57b7d2573bb6dfd038e581acb3bd8ea1565aa20d Gerrit-Change-Number: 12985 Gerrit-PatchSet: 16 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 26 Apr 2019 15:45:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8381: Optimize ParquetPlainEncoder::DecodeBatch() for simple types
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12985 ) Change subject: IMPALA-8381: Optimize ParquetPlainEncoder::DecodeBatch() for simple types .. Patch Set 16: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4083/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/12985 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I57b7d2573bb6dfd038e581acb3bd8ea1565aa20d Gerrit-Change-Number: 12985 Gerrit-PatchSet: 16 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 26 Apr 2019 15:45:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8309: add user authorization provider flag
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12901 ) Change subject: IMPALA-8309: add user authorization_provider flag .. Patch Set 20: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 Gerrit-Change-Number: 12901 Gerrit-PatchSet: 20 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Fri, 26 Apr 2019 15:21:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8309: add user authorization provider flag
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12901 ) Change subject: IMPALA-8309: add user authorization_provider flag .. Patch Set 20: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4082/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/12901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 Gerrit-Change-Number: 12901 Gerrit-PatchSet: 20 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Fri, 26 Apr 2019 15:21:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8309: add user authorization provider flag
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/12901 ) Change subject: IMPALA-8309: add user authorization_provider flag .. Patch Set 19: Code-Review+2 > Patch Set 19: Verified-1 > > Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4068/ No tests failed and towards the end after jenkins/finalize.sh, it got this error in Jenkins: 00:26:46 Process leaked file descriptors. See https://jenkins.io/redirect/troubleshooting/process-leaked-file-descriptors for more information I suspect something unstable with Jenkins. Will retry. -- To view, visit http://gerrit.cloudera.org:8080/12901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 Gerrit-Change-Number: 12901 Gerrit-PatchSet: 19 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Fri, 26 Apr 2019 15:20:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8309: add user authorization provider flag
Fredy Wijaya has removed Todd Lipcon from this change. ( http://gerrit.cloudera.org:8080/12901 ) Change subject: IMPALA-8309: add user authorization_provider flag .. Removed reviewer Todd Lipcon. -- To view, visit http://gerrit.cloudera.org:8080/12901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: deleteReviewer Gerrit-Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 Gerrit-Change-Number: 12901 Gerrit-PatchSet: 19 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: radford nguyen
[Impala-ASF-CR] IMPALA-8381: Optimize ParquetPlainEncoder::DecodeBatch() for simple types
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/12985 ) Change subject: IMPALA-8381: Optimize ParquetPlainEncoder::DecodeBatch() for simple types .. Patch Set 15: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/12985/14/be/src/exec/parquet/parquet-common.h File be/src/exec/parquet/parquet-common.h: http://gerrit.cloudera.org:8080/#/c/12985/14/be/src/exec/parquet/parquet-common.h@210 PS14, Line 210: s fun > No, I think that we shouldn't check in release, DCHECK's are enough. The co I see, thanks for the explanation. Yeah, I also think that this DCHECK is enough. -- To view, visit http://gerrit.cloudera.org:8080/12985 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I57b7d2573bb6dfd038e581acb3bd8ea1565aa20d Gerrit-Change-Number: 12985 Gerrit-PatchSet: 15 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 26 Apr 2019 14:56:02 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8253: Draft - Parquet delta encoding and decoding.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12621 ) Change subject: IMPALA-8253: Draft - Parquet delta encoding and decoding. .. Patch Set 7: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/2933/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/12621 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie7378ac1a490a6c89a0a4349aae86cbc0fbc80f8 Gerrit-Change-Number: 12621 Gerrit-PatchSet: 7 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 26 Apr 2019 13:26:56 + Gerrit-HasComments: No
[Impala-ASF-CR] Make infra/python compatible with both Python 2 & 3
Akshesh Doshi has posted comments on this change. ( http://gerrit.cloudera.org:8080/13070 ) Change subject: Make infra/python compatible with both Python 2 & 3 .. Patch Set 5: Yayy.. thanks Tim! :) -- To view, visit http://gerrit.cloudera.org:8080/13070 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If4285a021bb581f88425daa52ef8a3f844017d82 Gerrit-Change-Number: 13070 Gerrit-PatchSet: 5 Gerrit-Owner: Akshesh Doshi Gerrit-Reviewer: Akshesh Doshi Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 26 Apr 2019 12:41:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8253: Draft - Parquet delta encoding and decoding.
Hello Gabor Kaszab, Zoltan Borok-Nagy, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12621 to look at the new patch set (#7). Change subject: IMPALA-8253: Draft - Parquet delta encoding and decoding. .. IMPALA-8253: Draft - Parquet delta encoding and decoding. Implemented an encoder and decoder for the Parquet delta encoding (see https://github.com/apache/parquet-format/blob/master/Encodings.md). The coders are not integrated with Impala yet, they provide an interface that Impala could use. TODO: Currently the delta coders only support 32-bit integers. For 64 bit integers, we have to extend the functionality of BitWriter and BatchedBitReader. Added new methods to BitWriter and BatchedBitReader handling Uleb and ZigZag integers for 64 bits. Testing: - Added new tests for the encoder and decoder - Tests covering the additions in BitWriter and BatchedBitReader. Change-Id: Ie7378ac1a490a6c89a0a4349aae86cbc0fbc80f8 --- M be/src/benchmarks/CMakeLists.txt A be/src/benchmarks/parquet-delta-benchmark.cc M be/src/exec/parquet/CMakeLists.txt M be/src/exec/parquet/parquet-common.h A be/src/exec/parquet/parquet-delta-coder-test-data.h A be/src/exec/parquet/parquet-delta-coder-test.cc A be/src/exec/parquet/parquet-delta-decoder.h A be/src/exec/parquet/parquet-delta-encoder.h M be/src/util/CMakeLists.txt M be/src/util/bit-packing-test.cc M be/src/util/bit-packing.h M be/src/util/bit-packing.inline.h M be/src/util/bit-stream-utils-test.cc M be/src/util/bit-stream-utils.h M be/src/util/bit-stream-utils.inline.h 15 files changed, 4,239 insertions(+), 56 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/21/12621/7 -- To view, visit http://gerrit.cloudera.org:8080/12621 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie7378ac1a490a6c89a0a4349aae86cbc0fbc80f8 Gerrit-Change-Number: 12621 Gerrit-PatchSet: 7 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-8253: Draft - Parquet delta encoding and decoding.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12621 ) Change subject: IMPALA-8253: Draft - Parquet delta encoding and decoding. .. Patch Set 6: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/2932/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/12621 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie7378ac1a490a6c89a0a4349aae86cbc0fbc80f8 Gerrit-Change-Number: 12621 Gerrit-PatchSet: 6 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 26 Apr 2019 12:21:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8253: Draft - Parquet delta encoding and decoding.
Daniel Becker has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12621 Change subject: IMPALA-8253: Draft - Parquet delta encoding and decoding. .. IMPALA-8253: Draft - Parquet delta encoding and decoding. Implemented an encoder and decoder for the Parquet delta encoding (see https://github.com/apache/parquet-format/blob/master/Encodings.md). The coders are not integrated with Impala yet, they provide an interface that Impala could use. TODO: Currently the delta coders only support 32-bit integers. For 64 bit integers, we have to extend the functionality of BitWriter and BatchedBitReader. Added new methods to BitWriter and BatchedBitReader handling Uleb and ZigZag integers for 64 bits. Testing: - Added new tests for the encoder and decoder - Tests covering the additions in BitWriter and BatchedBitReader. Change-Id: Ie7378ac1a490a6c89a0a4349aae86cbc0fbc80f8 --- M be/src/benchmarks/CMakeLists.txt A be/src/benchmarks/parquet-delta-benchmark.cc M be/src/exec/parquet/CMakeLists.txt M be/src/exec/parquet/parquet-common.h A be/src/exec/parquet/parquet-delta-coder-test-data.h A be/src/exec/parquet/parquet-delta-coder-test.cc A be/src/exec/parquet/parquet-delta-decoder.h A be/src/exec/parquet/parquet-delta-encoder.h M be/src/util/CMakeLists.txt M be/src/util/bit-packing-test.cc M be/src/util/bit-packing.h M be/src/util/bit-packing.inline.h M be/src/util/bit-stream-utils-test.cc M be/src/util/bit-stream-utils.h M be/src/util/bit-stream-utils.inline.h 15 files changed, 4,238 insertions(+), 56 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/21/12621/6 -- To view, visit http://gerrit.cloudera.org:8080/12621 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ie7378ac1a490a6c89a0a4349aae86cbc0fbc80f8 Gerrit-Change-Number: 12621 Gerrit-PatchSet: 6 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-8454 (part 3): enable recursive file listing by default
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13127 ) Change subject: IMPALA-8454 (part 3): enable recursive file listing by default .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13127 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib30e2bcaf820210f2faa8f159d1af2f947a4d0e8 Gerrit-Change-Number: 13127 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 26 Apr 2019 11:43:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8253: Draft - Parquet delta encoding and decoding.
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/12621 )
Change subject: IMPALA-8253: Draft - Parquet delta encoding and decoding.
..
Patch Set 6:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/12621/6/be/src/exec/parquet/parquet-delta-coder-test-data.h
File be/src/exec/parquet/parquet-delta-coder-test-data.h:
http://gerrit.cloudera.org:8080/#/c/12621/6/be/src/exec/parquet/parquet-delta-coder-test-data.h@454
PS6, Line 454: const std::vector values_are_the_same_plain = {3, 3, 3,
3, 3, 3, 3, 3, 3, 3, 3, 3,
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/12621/6/be/src/exec/parquet/parquet-delta-coder-test-data.h@471
PS6, Line 471: const std::vector delta_is_zero_for_each_block_plain =
{0, 0, 0, 0, 0, 0, 0, 0, 0,
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/12621/6/be/src/exec/parquet/parquet-delta-encoder.h
File be/src/exec/parquet/parquet-delta-encoder.h:
http://gerrit.cloudera.org:8080/#/c/12621/6/be/src/exec/parquet/parquet-delta-encoder.h@47
PS6, Line 47:
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/12621/6/be/src/exec/parquet/parquet-delta-encoder.h@115
PS6, Line 115: const int header_size =
HeaderSize(most_negative_first_value, MAX_TOTAL_VALUE_COUNT);
line too long (91 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/12621
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie7378ac1a490a6c89a0a4349aae86cbc0fbc80f8
Gerrit-Change-Number: 12621
Gerrit-PatchSet: 6
Gerrit-Owner: Daniel Becker
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Daniel Becker
Gerrit-Reviewer: Gabor Kaszab
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Fri, 26 Apr 2019 11:31:40 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] Configure Hive 3's HS2 to execute queries using Tez local mode
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12931 ) Change subject: Configure Hive 3's HS2 to execute queries using Tez local mode .. Patch Set 6: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12931 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I76e47fbd1d6ff5103d81a8de430d5465dba284cd Gerrit-Change-Number: 12931 Gerrit-PatchSet: 6 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 26 Apr 2019 11:19:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8454 (part 1): Refactor file descriptor loading code
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12950 ) Change subject: IMPALA-8454 (part 1): Refactor file descriptor loading code .. Patch Set 7: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12950 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I59edf493b9ba38be5f556b4795a7684d9c9e3a07 Gerrit-Change-Number: 12950 Gerrit-PatchSet: 7 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Fri, 26 Apr 2019 10:54:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8454 (part 1): Refactor file descriptor loading code
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12950 ) Change subject: IMPALA-8454 (part 1): Refactor file descriptor loading code .. IMPALA-8454 (part 1): Refactor file descriptor loading code This refactors various file-descriptor loading code out of HdfsTable into new standalone classes. In order to support ACID tables, we'll need to make various changes to these bits of code, and having them extracted and cleaned up will make that easier. This consolidates all of the places in which we list partition directories into one method which does the appropriate thing regardless of situation. This has a small behavior change related to IMPALA-8406: previously, we had a bug where, while refreshing a table, if one or more partitions failed to refresh, the other partitions might still get refreshed despite an error being returned. Those other partitions wouldn't be available immediately until some other operation caused the table's catalog version number to increase. This was buggy behavior. Rather than tackle that problem in this "refactor" patch, this patch just slightly improves the behavior: we'll either atomically update or not update all partitions, but we might still add new partitions noticed by the REFRESH, and might still update other HMS metadata. This patch may end up slightly improving various other code paths that refresh file descriptor lists. We used to have slightly different ways of doing this in three different places, with different sets of optimizations. Now we do it all in one place, and we pull all the known tricks. Change-Id: I59edf493b9ba38be5f556b4795a7684d9c9e3a07 Reviewed-on: http://gerrit.cloudera.org:8080/12950 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- A fe/src/main/java/org/apache/impala/catalog/FileMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java A fe/src/main/java/org/apache/impala/catalog/ParallelFileMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M fe/src/test/java/org/apache/impala/catalog/CatalogTest.java M fe/src/test/java/org/apache/impala/catalog/HdfsPartitionTest.java M fe/src/test/java/org/apache/impala/planner/TestCaseLoaderTest.java 10 files changed, 478 insertions(+), 465 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/12950 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I59edf493b9ba38be5f556b4795a7684d9c9e3a07 Gerrit-Change-Number: 12950 Gerrit-PatchSet: 8 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen
[Impala-ASF-CR] IMPALA-8454 (part 1): Refactor file descriptor loading code
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12950 ) Change subject: IMPALA-8454 (part 1): Refactor file descriptor loading code .. Patch Set 6: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12950 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I59edf493b9ba38be5f556b4795a7684d9c9e3a07 Gerrit-Change-Number: 12950 Gerrit-PatchSet: 6 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Fri, 26 Apr 2019 10:01:14 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) IMPALA-7140 (part 7): small fixes to enable most queries on HDFS tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13114 ) Change subject: IMPALA-7140 (part 7): small fixes to enable most queries on HDFS tables .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13114 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I6f603e62b7a013c148c0905ebdec2f4303f9c4e5 Gerrit-Change-Number: 13114 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Fri, 26 Apr 2019 09:57:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8454 (part 2): Initial support for recursive file listing within a partition
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12991 ) Change subject: IMPALA-8454 (part 2): Initial support for recursive file listing within a partition .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12991 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9b151d7abb8443c0d9de0a0d82a9f13e07ad5109 Gerrit-Change-Number: 12991 Gerrit-PatchSet: 3 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 26 Apr 2019 09:56:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 ) Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0 .. Patch Set 9: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4075/ -- To view, visit http://gerrit.cloudera.org:8080/13005 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436 Gerrit-Change-Number: 13005 Gerrit-PatchSet: 9 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 26 Apr 2019 09:12:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-966: Type errors are attributed to wrong expression with insert
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13050 ) Change subject: IMPALA-966: Type errors are attributed to wrong expression with insert .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2931/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13050 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I88718fc2cbe1a492165435a542fd2d91d8693a39 Gerrit-Change-Number: 13050 Gerrit-PatchSet: 3 Gerrit-Owner: Alice Fan Gerrit-Reviewer: Alice Fan Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Comment-Date: Fri, 26 Apr 2019 09:00:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-966: Type errors are attributed to wrong expression with insert
Hello Bharath Vissapragada, Paul Rogers, xiaom...@cloudera.com, Bikramjeet Vig, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13050 to look at the new patch set (#3). Change subject: IMPALA-966: Type errors are attributed to wrong expression with insert .. IMPALA-966: Type errors are attributed to wrong expression with insert When insert a union statement with incompatible types, we should make error message blame on correct expression. If there are multiple incompatible types for a single column, error message should print the widest data type expression. Testing: - Added tests to AnalyzeStmtsTest.java Change-Id: I88718fc2cbe1a492165435a542fd2d91d8693a39 --- M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/analysis/QueryStmt.java M fe/src/main/java/org/apache/impala/analysis/StatementBase.java M fe/src/main/java/org/apache/impala/analysis/UnionStmt.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java 6 files changed, 64 insertions(+), 14 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/13050/3 -- To view, visit http://gerrit.cloudera.org:8080/13050 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I88718fc2cbe1a492165435a542fd2d91d8693a39 Gerrit-Change-Number: 13050 Gerrit-PatchSet: 3 Gerrit-Owner: Alice Fan Gerrit-Reviewer: Alice Fan Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers
[Impala-ASF-CR] Configure Hive 3's HS2 to execute queries using Tez local mode
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12931 ) Change subject: Configure Hive 3's HS2 to execute queries using Tez local mode .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2930/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12931 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I76e47fbd1d6ff5103d81a8de430d5465dba284cd Gerrit-Change-Number: 12931 Gerrit-PatchSet: 6 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 26 Apr 2019 07:03:24 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) IMPALA-7140 (part 7): small fixes to enable most queries on HDFS tables
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/13114 ) Change subject: IMPALA-7140 (part 7): small fixes to enable most queries on HDFS tables .. Patch Set 1: The previous build failure is due to the lake of bin/jenkins/dockerized-impala-bootstrap-and-test.sh in 2.x branch. Rerun it after the job ubuntu-16.04-dockerised-tests is disabled. -- To view, visit http://gerrit.cloudera.org:8080/13114 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I6f603e62b7a013c148c0905ebdec2f4303f9c4e5 Gerrit-Change-Number: 13114 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Fri, 26 Apr 2019 07:02:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 ) Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0 .. Patch Set 9: (7 comments) Took a look through this. I'll see if I can get some time tomorrow to try out a "shims" approach as well for comparison. It looks like the majority of the changes here, though, are pretty innocuous (eg copying some stuff into our code where we were relying on hive-internal stuff) and could be made either way. http://gerrit.cloudera.org:8080/#/c/13005/9//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/13005/9//COMMIT_MSG@58 PS9, Line 58: 4. The Sentry object ownership tests are flaky. There are Is this flakiness specific to the hive 3 config? Or the races were already there? Is the problem here that we don't have the notification listener working properly so we're relying on some periodic sync instead of notifications? http://gerrit.cloudera.org:8080/#/c/13005/9//COMMIT_MSG@65 PS9, Line 65: work-around these issues. maybe we shoudl disable these tests when running with hive 3 since we don't anticipate using Sentry with Hive 3 anyway http://gerrit.cloudera.org:8080/#/c/13005/9/bin/set-classpath.sh File bin/set-classpath.sh: http://gerrit.cloudera.org:8080/#/c/13005/9/bin/set-classpath.sh@30 PS9, Line 30: "$IMPALA_HOME"/shaded-deps/target/impala-shaded-deps-0.1-SNAPSHOT.jar:\ why is this necessary? shouldn't the shaded-deps dependency also end up in target/dependency/ like the others handled on line 35? http://gerrit.cloudera.org:8080/#/c/13005/9/fe/src/main/java/org/apache/impala/catalog/MetaStoreClientPool.java File fe/src/main/java/org/apache/impala/catalog/MetaStoreClientPool.java: http://gerrit.cloudera.org:8080/#/c/13005/9/fe/src/main/java/org/apache/impala/catalog/MetaStoreClientPool.java@68 PS9, Line 68: private List serverVersion = null; : : private static final Ordering> VERSION_ORDERING = : Ordering.natural().lexicographical().nullsLast(); : private static final ImmutableList CATALOGS_VERSION = : ImmutableList.of(3, 0, 0); Are these changes actually used right now? I think this stuff ended up being in the Hive impl rather than being doable in this wrapper http://gerrit.cloudera.org:8080/#/c/13005/9/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java: http://gerrit.cloudera.org:8080/#/c/13005/9/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@173 PS9, Line 173: MessageFactory.getDefaultInstance(new HiveConf()).getDeserializer(); Can you explain what's going on with this part of the change? Did Sentry move to the normal JSONMessageFactory? http://gerrit.cloudera.org:8080/#/c/13005/9/testdata/bin/run-hive-server.sh File testdata/bin/run-hive-server.sh: http://gerrit.cloudera.org:8080/#/c/13005/9/testdata/bin/run-hive-server.sh@66 PS9, Line 66: export HIVE_METASTORE_HADOOP_OPTS="-verbose:class -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=30010" probably remove this http://gerrit.cloudera.org:8080/#/c/13005/9/testdata/bin/run-hive-server.sh@93 PS9, Line 93: # For Hive 3, we use Tez for execution. We have to add it to the HS2 classpath. this is in another patch- guess we can rebase this on top of that one to pick it up -- To view, visit http://gerrit.cloudera.org:8080/13005 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436 Gerrit-Change-Number: 13005 Gerrit-PatchSet: 9 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 26 Apr 2019 06:51:52 + Gerrit-HasComments: Yes
