[Impala-ASF-CR] IMPALA-6326: part 2: remove fetch thread in stress test
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12681 ) Change subject: IMPALA-6326: part 2: remove fetch thread in stress test .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2570/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12681 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If9afd74e1408823a9e5c0f2628ec9f8aafdcec69 Gerrit-Change-Number: 12681 Gerrit-PatchSet: 7 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 28 Mar 2019 03:11:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6326: part 2: remove fetch thread in stress test
Hello Thomas Marshall, David Knupp, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12681 to look at the new patch set (#7). Change subject: IMPALA-6326: part 2: remove fetch thread in stress test .. IMPALA-6326: part 2: remove fetch thread in stress test Ensure cursor is only accessed from a single thread. The means reworking the code so that we check the time limit between fetch calls. Use EXEC_TIME_LIMIT_S as an alternative to the previous multi-threaded cancellation logic - it allows queries to be cancelled even when the client is blocked or slow. This is implemented with the concept of a CancelMechanism that determines *how* a query should be cancelled. Query timeouts (where we want to cancel queries that run longer than expected) are implemented using both cancel mechanisms, in case the client is stuck in fetch or similar. Expected cancellations are implemented with a random mechanism so that both code paths get covered. Testing: Ran a cluster stress test. Ran a couple of single-node stress tests with TPC-H and random queries. Change-Id: If9afd74e1408823a9e5c0f2628ec9f8aafdcec69 --- M tests/stress/concurrent_select.py M tests/stress/query_runner.py 2 files changed, 128 insertions(+), 90 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/81/12681/7 -- To view, visit http://gerrit.cloudera.org:8080/12681 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If9afd74e1408823a9e5c0f2628ec9f8aafdcec69 Gerrit-Change-Number: 12681 Gerrit-PatchSet: 7 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-6326: part 2: remove fetch thread in stress test
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12681 ) Change subject: IMPALA-6326: part 2: remove fetch thread in stress test .. Patch Set 8: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12681 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If9afd74e1408823a9e5c0f2628ec9f8aafdcec69 Gerrit-Change-Number: 12681 Gerrit-PatchSet: 8 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 28 Mar 2019 02:36:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6326: part 2: remove fetch thread in stress test
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12681 ) Change subject: IMPALA-6326: part 2: remove fetch thread in stress test .. Patch Set 6: (2 comments) http://gerrit.cloudera.org:8080/#/c/12681/6/tests/stress/query_runner.py File tests/stress/query_runner.py: http://gerrit.cloudera.org:8080/#/c/12681/6/tests/stress/query_runner.py@201 PS6, Line 201: IMPALA-8288 > this JIRA has already been resolved Removed workaround http://gerrit.cloudera.org:8080/#/c/12681/6/tests/stress/query_runner.py@253 PS6, Line 253: from > to? Done -- To view, visit http://gerrit.cloudera.org:8080/12681 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If9afd74e1408823a9e5c0f2628ec9f8aafdcec69 Gerrit-Change-Number: 12681 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 28 Mar 2019 02:35:52 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6326: part 2: remove fetch thread in stress test
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12681 ) Change subject: IMPALA-6326: part 2: remove fetch thread in stress test .. Patch Set 8: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3963/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/12681 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If9afd74e1408823a9e5c0f2628ec9f8aafdcec69 Gerrit-Change-Number: 12681 Gerrit-PatchSet: 8 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 28 Mar 2019 02:36:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8293: Support for Ranger cache invalidation
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12748 ) Change subject: IMPALA-8293: Support for Ranger cache invalidation .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2568/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12748 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I524ee93d09077dd4ff3d18fe517739b7776d01d7 Gerrit-Change-Number: 12748 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 28 Mar 2019 01:54:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8330: Impala shell config file should support flag names
Jim Apple has posted comments on this change. ( http://gerrit.cloudera.org:8080/12823 ) Change subject: IMPALA-8330: Impala shell config file should support flag names .. Patch Set 6: Code-Review+1 (2 comments) http://gerrit.cloudera.org:8080/#/c/12823/6/shell/option_parser.py File shell/option_parser.py: http://gerrit.cloudera.org:8080/#/c/12823/6/shell/option_parser.py@74 PS6, Line 74: option_dests[option.dest] = option You might want to check that option1.dest is never equal to any option2._long_opts unless option1 == option2. http://gerrit.cloudera.org:8080/#/c/12823/6/tests/shell/good_impalarc File tests/shell/good_impalarc: http://gerrit.cloudera.org:8080/#/c/12823/6/tests/shell/good_impalarc@4 PS6, Line 4: ' nit: stray character? -- To view, visit http://gerrit.cloudera.org:8080/12823 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic43603c1b538af08fddcab1b2c1f6ad1af1a6cb9 Gerrit-Change-Number: 12823 Gerrit-PatchSet: 6 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Comment-Date: Thu, 28 Mar 2019 02:00:09 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8345 : Add option to set up minicluster to use Hive 3
Impala Public Jenkins has submitted this change and it was merged. (
http://gerrit.cloudera.org:8080/12846 )
Change subject: IMPALA-8345 : Add option to set up minicluster to use Hive 3
..
IMPALA-8345 : Add option to set up minicluster to use Hive 3
As a first step to integrate Impala with Hive 3.1.0 this patch modifies
the minicluster scripts to optionally use Hive 3.1.0 instead of
CDH Hive 2.1.1.
In order to make sure that existing setups don't break this is
enabled via a environment variable override to bin/impala-config.sh.
When the environment variable USE_CDP_HIVE is set to true the
bootstrap_toolchain script downloads Hive 3.1.0 tarballs and extracts it
in the toolchain directory. These binaries are used to start the Hive
services (Hiveserver2 and metastore). The default is still CDH Hive 2.1.1
Also, since Hive 3.1.0 uses a upgraded metastore schema, this patch
makes use of a different database name so that it is easy to switch from
working from one environment which uses Hive 2.1.1 metastore to another
which usese Hive 3.1.0 metastore.
In order to start a minicluster which uses Hive 3.1.0 users should
follow the steps below:
1. Make sure that minicluster, if running, is stopped
before you run the following commands.
2. Open a new terminal and run following commands.
> export USE_CDP_HIVE=true
> source bin/impala-config.sh
> bin/bootstrap_toolchain.py
The above command downloads the Hive 3.1.0 tarballs and extracts them
in toolchain/cdp_components-${CDP_BUILD_NUMBER} directory. This is a
no-op if the CDP_BUILD_NUMBER has not changed and if the cdp_components
are already downloaded by a previous invocation of the script.
> source bin/create-test-configuration.sh -create-metastore
The above step should provide "-create-metastore" only the first time
so that a new metastore db is created and the Hive 3.1.0 schema is
initialized. For all subsequent invocations, the "-create-metastore"
argument can be skipped. We should still source this script since the
hive-site.xml of Hive 3.1.0 is different than Hive 2.1.0 and
needs to be regenerated.
> testdata/bin/run-all.sh
Note that the testing was performed locally by downloading the Hive 3.1
binaries into
toolchain/cdp_components-976603/apache-hive-3.1.0.6.0.99.0-9-bin. Once
the binaries are available in S3 bucket, the bootstrap_toolchain script
should automatically do this for you.
Testing Done:
1. Made sure that the cluster comes up with Hive 3.1 when the steps
above are performed.
2. Made sure that existing scripts work as they do currently when
argument is not provided.
3. Impala cluster comes and connects to HMS 3.1.0 (Note that Impala
still uses Hive 2.1.1 client. Upgrading client libraries in Impala will
be done as a separate change)
Change-Id: Icfed856c1f5429ed45fd3d9cb08a5d1bb96a9605
Reviewed-on: http://gerrit.cloudera.org:8080/12846
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
---
M bin/bootstrap_toolchain.py
M bin/create-test-configuration.sh
M bin/impala-config.sh
A fe/src/test/resources/postgresql-hive-site.xml.cdp.template
M testdata/bin/run-hive-server.sh
5 files changed, 284 insertions(+), 11 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/12846
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Icfed856c1f5429ed45fd3d9cb08a5d1bb96a9605
Gerrit-Change-Number: 12846
Gerrit-PatchSet: 12
Gerrit-Owner: Vihang Karajgaonkar
Gerrit-Reviewer: Andrew Sherman
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Todd Lipcon
Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8345 : Add option to set up minicluster to use Hive 3
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12846 ) Change subject: IMPALA-8345 : Add option to set up minicluster to use Hive 3 .. Patch Set 11: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12846 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icfed856c1f5429ed45fd3d9cb08a5d1bb96a9605 Gerrit-Change-Number: 12846 Gerrit-PatchSet: 11 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 28 Mar 2019 01:52:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8293: Support for Ranger cache invalidation
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12748 ) Change subject: IMPALA-8293: Support for Ranger cache invalidation .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2569/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12748 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I524ee93d09077dd4ff3d18fe517739b7776d01d7 Gerrit-Change-Number: 12748 Gerrit-PatchSet: 6 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 28 Mar 2019 01:51:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8293: Support for Ranger cache invalidation
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12748 ) Change subject: IMPALA-8293: Support for Ranger cache invalidation .. Patch Set 5: (5 comments) http://gerrit.cloudera.org:8080/#/c/12748/5/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java: http://gerrit.cloudera.org:8080/#/c/12748/5/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@49 PS5, Line 49: import org.apache.impala.authorization.NoneAuthorizationFactory.NoneAuthorizationRefresher; line too long (91 > 90) http://gerrit.cloudera.org:8080/#/c/12748/5/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java File fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java: http://gerrit.cloudera.org:8080/#/c/12748/5/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java@38 PS5, Line 38: import org.apache.impala.authorization.NoneAuthorizationFactory.NoneAuthorizationRefresher; line too long (91 > 90) http://gerrit.cloudera.org:8080/#/c/12748/5/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java: http://gerrit.cloudera.org:8080/#/c/12748/5/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@56 PS5, Line 56: import org.apache.impala.authorization.NoneAuthorizationFactory.NoneAuthorizationRefresher; line too long (91 > 90) http://gerrit.cloudera.org:8080/#/c/12748/5/fe/src/test/java/org/apache/impala/common/FrontendTestBase.java File fe/src/test/java/org/apache/impala/common/FrontendTestBase.java: http://gerrit.cloudera.org:8080/#/c/12748/5/fe/src/test/java/org/apache/impala/common/FrontendTestBase.java@43 PS5, Line 43: import org.apache.impala.authorization.NoneAuthorizationFactory.NoneAuthorizationRefresher; line too long (91 > 90) http://gerrit.cloudera.org:8080/#/c/12748/5/fe/src/test/java/org/apache/impala/testutil/PlannerTestCaseLoader.java File fe/src/test/java/org/apache/impala/testutil/PlannerTestCaseLoader.java: http://gerrit.cloudera.org:8080/#/c/12748/5/fe/src/test/java/org/apache/impala/testutil/PlannerTestCaseLoader.java@21 PS5, Line 21: import org.apache.impala.authorization.NoneAuthorizationFactory.NoneAuthorizationRefresher; line too long (91 > 90) -- To view, visit http://gerrit.cloudera.org:8080/12748 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I524ee93d09077dd4ff3d18fe517739b7776d01d7 Gerrit-Change-Number: 12748 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 28 Mar 2019 01:09:58 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8293: Support for Ranger cache invalidation
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12748 ) Change subject: IMPALA-8293: Support for Ranger cache invalidation .. Patch Set 6: (5 comments) http://gerrit.cloudera.org:8080/#/c/12748/6/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java: http://gerrit.cloudera.org:8080/#/c/12748/6/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@49 PS6, Line 49: import org.apache.impala.authorization.NoneAuthorizationFactory.NoneAuthorizationRefresher; line too long (91 > 90) http://gerrit.cloudera.org:8080/#/c/12748/6/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java File fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java: http://gerrit.cloudera.org:8080/#/c/12748/6/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java@38 PS6, Line 38: import org.apache.impala.authorization.NoneAuthorizationFactory.NoneAuthorizationRefresher; line too long (91 > 90) http://gerrit.cloudera.org:8080/#/c/12748/6/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java: http://gerrit.cloudera.org:8080/#/c/12748/6/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@56 PS6, Line 56: import org.apache.impala.authorization.NoneAuthorizationFactory.NoneAuthorizationRefresher; line too long (91 > 90) http://gerrit.cloudera.org:8080/#/c/12748/6/fe/src/test/java/org/apache/impala/common/FrontendTestBase.java File fe/src/test/java/org/apache/impala/common/FrontendTestBase.java: http://gerrit.cloudera.org:8080/#/c/12748/6/fe/src/test/java/org/apache/impala/common/FrontendTestBase.java@43 PS6, Line 43: import org.apache.impala.authorization.NoneAuthorizationFactory.NoneAuthorizationRefresher; line too long (91 > 90) http://gerrit.cloudera.org:8080/#/c/12748/6/fe/src/test/java/org/apache/impala/testutil/PlannerTestCaseLoader.java File fe/src/test/java/org/apache/impala/testutil/PlannerTestCaseLoader.java: http://gerrit.cloudera.org:8080/#/c/12748/6/fe/src/test/java/org/apache/impala/testutil/PlannerTestCaseLoader.java@21 PS6, Line 21: import org.apache.impala.authorization.NoneAuthorizationFactory.NoneAuthorizationRefresher; line too long (91 > 90) -- To view, visit http://gerrit.cloudera.org:8080/12748 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I524ee93d09077dd4ff3d18fe517739b7776d01d7 Gerrit-Change-Number: 12748 Gerrit-PatchSet: 6 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 28 Mar 2019 01:17:55 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8293: Support for Ranger cache invalidation
Fredy Wijaya has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/12748 ) Change subject: IMPALA-8293: Support for Ranger cache invalidation .. IMPALA-8293: Support for Ranger cache invalidation This patch adds support for Ranger cache invalidation via INVALIDATE METADATA and REFRESH AUTHORIZATION. This patch introduces a new catalog object type called AUTHZ_CACHE to allow Catalogd broadcasting messages to Impalads to update their local Ranger caches. For beter user experience, every GRANT/REVOKE statement will cause Catalogd to send AUTHZ_CACHE catalog object to Impalads. This patch uses a hacked way of invalidating Ranger caches and it will be updated as soon the API in RANGER-2349 is available in the Ranger minicluster. This patch further decouples SentryProxy out of CatalogServiceCatalog and introduces AuthorizationRefresher that can be optionally use as a mechanism to update authorization cache. Testing: - Added new E2E authorization tests - Ran all FE tests - Ran all E2E authorization tests Change-Id: I524ee93d09077dd4ff3d18fe517739b7776d01d7 --- M be/src/catalog/catalog-util.cc M common/thrift/CatalogObjects.thrift M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/authorization/AuthorizationChecker.java A fe/src/main/java/org/apache/impala/authorization/AuthorizationDelta.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationFactory.java A fe/src/main/java/org/apache/impala/authorization/AuthorizationRefresher.java M fe/src/main/java/org/apache/impala/authorization/NoneAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java A fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationRefresher.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java A fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationFactory.java A fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationRefresher.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryCatalogdAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryProxy.java A fe/src/main/java/org/apache/impala/catalog/AuthzCache.java M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/FeCatalogManager.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M fe/src/test/java/org/apache/impala/analysis/AuditingTest.java M fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java M fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java M fe/src/test/java/org/apache/impala/authorization/sentry/SentryProxyTest.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java M fe/src/test/java/org/apache/impala/testutil/CatalogServiceTestCatalog.java M fe/src/test/java/org/apache/impala/testutil/ImpaladTestCatalog.java M fe/src/test/java/org/apache/impala/testutil/PlannerTestCaseLoader.java M tests/authorization/test_ranger.py 36 files changed, 801 insertions(+), 184 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/48/12748/6 -- To view, visit http://gerrit.cloudera.org:8080/12748 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I524ee93d09077dd4ff3d18fe517739b7776d01d7 Gerrit-Change-Number: 12748 Gerrit-PatchSet: 6 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8293: Support for Ranger cache invalidation
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/12748 ) Change subject: IMPALA-8293: Support for Ranger cache invalidation .. Patch Set 5: Todd/Bharath, this is ready for review. -- To view, visit http://gerrit.cloudera.org:8080/12748 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I524ee93d09077dd4ff3d18fe517739b7776d01d7 Gerrit-Change-Number: 12748 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 28 Mar 2019 01:09:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8293: Support for Ranger cache invalidation
Fredy Wijaya has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/12748 ) Change subject: IMPALA-8293: Support for Ranger cache invalidation .. IMPALA-8293: Support for Ranger cache invalidation This patch adds support for Ranger cache invalidation via INVALIDATE METADATA and REFRESH AUTHORIZATION. This patch introduces a new catalog object type called AUTHZ_CACHE to allow Catalogd broadcasting messages to Impalads to update their local Ranger caches. For beter user experience, every GRANT/REVOKE statement will cause Catalogd to send AUTHZ_CACHE catalog object to Impalads. This patch uses a hacked way of invalidating Ranger caches and it will be updated as soon the API in RANGER-2349 is available in the Ranger minicluster. This patch further decouples SentryProxy out of CatalogServiceCatalog and introduces AuthorizationRefresher that can be optionally use as a mechanism to update authorization cache. Testing: - Added new E2E authorization tests - Ran all FE tests - Ran all E2E authorization tests Change-Id: I524ee93d09077dd4ff3d18fe517739b7776d01d7 --- M be/src/catalog/catalog-util.cc M common/thrift/CatalogObjects.thrift M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/authorization/AuthorizationChecker.java A fe/src/main/java/org/apache/impala/authorization/AuthorizationDelta.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationFactory.java A fe/src/main/java/org/apache/impala/authorization/AuthorizationRefresher.java M fe/src/main/java/org/apache/impala/authorization/NoneAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java A fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationRefresher.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java A fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationFactory.java A fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationRefresher.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryCatalogdAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryProxy.java A fe/src/main/java/org/apache/impala/catalog/AuthzCache.java M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/FeCatalogManager.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M fe/src/test/java/org/apache/impala/analysis/AuditingTest.java M fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java M fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java M fe/src/test/java/org/apache/impala/authorization/sentry/SentryProxyTest.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java M fe/src/test/java/org/apache/impala/testutil/CatalogServiceTestCatalog.java M fe/src/test/java/org/apache/impala/testutil/ImpaladTestCatalog.java M fe/src/test/java/org/apache/impala/testutil/PlannerTestCaseLoader.java M tests/authorization/test_ranger.py 36 files changed, 802 insertions(+), 185 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/48/12748/5 -- To view, visit http://gerrit.cloudera.org:8080/12748 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I524ee93d09077dd4ff3d18fe517739b7776d01d7 Gerrit-Change-Number: 12748 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8143: Enhance DoRpcWithRetry().
Michael Ho has posted comments on this change. (
http://gerrit.cloudera.org:8080/12672 )
Change subject: IMPALA-8143: Enhance DoRpcWithRetry().
..
Patch Set 2:
(19 comments)
http://gerrit.cloudera.org:8080/#/c/12672/5/be/src/rpc/rpc-mgr-test.h
File be/src/rpc/rpc-mgr-test.h:
http://gerrit.cloudera.org:8080/#/c/12672/5/be/src/rpc/rpc-mgr-test.h@296
PS5, Line 296: ERR
typo
http://gerrit.cloudera.org:8080/#/c/12672/5/be/src/rpc/rpc-mgr-test.cc
File be/src/rpc/rpc-mgr-test.cc:
http://gerrit.cloudera.org:8080/#/c/12672/5/be/src/rpc/rpc-mgr-test.cc@214
PS5, Line 214: ASSERT_OK(rpc_mgr_.RegisterService(10, 10, scan_mem_impl,
: static_cast<
Not sure if it's output of clang-tidy or something but I find it a bit hard to
ready to split it like this. Can't we keep these 2 lines unchanged ?
http://gerrit.cloudera.org:8080/#/c/12672/5/be/src/rpc/rpc-mgr-test.cc@263
PS5, Line 263: ueue f
typo
http://gerrit.cloudera.org:8080/#/c/12672/5/be/src/rpc/rpc-mgr-test.cc@291
PS5, Line 291:
nullptr;
Also do you want to assert that it's zero ?
http://gerrit.cloudera.org:8080/#/c/12672/5/be/src/rpc/rpc-mgr-test.cc@298
PS5, Line 298: // Configure the service
not used ?
http://gerrit.cloudera.org:8080/#/c/12672/5/be/src/rpc/rpc-mgr-test.cc@316
PS5, Line 316: ASSERT_OK(ping_
Instead of sharing "sleep_ms" between RPCs, it may be cleaner to pass the sleep
time as RPC argument instead.
http://gerrit.cloudera.org:8080/#/c/12672/5/be/src/rpc/rpc-mgr-test.cc@322
PS5, Line 322: // A lambda t
proxy->
http://gerrit.cloudera.org:8080/#/c/12672/5/be/src/rpc/rpc-mgr-test.cc@333
PS5, Line 333:
: // wait for async call to be running
: sleep_started.Wait();
: ASSERT_EQ(0, get_current_queue_size(rpc_mgr_));
:
: // Do a second async call that will fill up the queue
: RpcController async_controller2;
: CountingBarrier async2_done(1);
: // Reset the sleep time
: sleep_ms = 100;
: // Set the deadline to something reasonable otherwise the
pings from DoRpcWithRetry
: // will replace this async call in the queue.
: MonoTime deadline = MonoTime::Now() +
MonoDelta::FromSeconds(10);
: async_controller2.set_deadline(deadline);
: ResponseCallback async2_callback = [&]() {
async2_done.Notify(); };
: proxy.get()->PingAsync(request, , _controller2,
async2_callback);
:
: // Wait for second async to get queued
: SleepForMs(300);
: // Check the queue size
: ASSERT_EQ(1, get_current_queue_size(
These set of tests seem rather timing dependent. For instance, it may occur
that the first time DoRpcWithRetry() is called will succeed because the async
call somehow finishes by then due to some weird scheduling order. May be better
to rethink a better way to test it.
http://gerrit.cloudera.org:8080/#/c/12672/5/be/src/rpc/rpc-mgr-test.cc@378
PS5, Line 378: ASSERT_FALSE(async2_done.pending());
:
: // Test injection of failures with DebugAction.
:
query_ctx.client_request.query_options.__set_debug_action("DoRpcWithRetry:FAIL");
: Status inject_status = RpcMgr::DoRpcWithRetry(ping_rpc,
query_ctx, "ping failed", 1,
: MonoDelta::FromSeconds(100), 0, "DoRpcWithRetry");
: ASSERT_FALSE(inject_status.ok());
: EXPECT_ERROR(inject_status, TErrorCo
Please see comments in rpc-mgr.inline.h that we probably should only retry if
the remote server is busy. For all other errors, we probably should report it
right away to callers. We can consider transparently handling other types of
errors in the future.
http://gerrit.cloudera.org:8080/#/c/12672/5/be/src/rpc/rpc-mgr.h
File be/src/rpc/rpc-mgr.h:
http://gerrit.cloudera.org:8080/#/c/12672/5/be/src/rpc/rpc-mgr.h@178
PS5, Line 178: void ToJson(rapidjson::Document* document);
:
: /// Retry the Rpc 'rpc_call' up to 'times_to_try' times.
: /// Each Rpc has a timeout of 'timeout'.
: /// If the service is busy then sleep 'se
Not needed as discussed in person.
http://gerrit.cloudera.org:8080/#/c/12672/5/be/src/rpc/rpc-mgr.h@189
PS5, Line 189: the R
typename
http://gerrit.cloudera.org:8080/#/c/12672/5/be/src/rpc/rpc-mgr.h@191
PS5, Line 191: n the 'l
It's important to highlight that this only works iff rpc_call is idempotent in
the comments above. Any non-idempotent rpc_call using this interface may have
undesired consequence.
http://gerrit.cloudera.org:8080/#/c/12672/5/be/src/rpc/rpc-mgr.h@193
PS5, Line 193: ebugAction() to potentially inject errors.
: template
It seems inconsistent to pass the timeout in as MonoDelta while passing the
backoff time
[Impala-ASF-CR] IMPALA-8312 : Alter database operations have race condition
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12789 ) Change subject: IMPALA-8312 : Alter database operations have race condition .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2567/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12789 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32c8c96a6029bf9d9db37ea8315f6c9603b5a2fc Gerrit-Change-Number: 12789 Gerrit-PatchSet: 8 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 27 Mar 2019 23:07:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7800: Time out new connections after --fe service threads
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/12579 ) Change subject: IMPALA-7800: Time out new connections after --fe_service_threads .. Patch Set 11: (2 comments) Looking good. Please address the last two items and I can +2 it. http://gerrit.cloudera.org:8080/#/c/12579/11/be/src/rpc/TAcceptQueueServer.cpp File be/src/rpc/TAcceptQueueServer.cpp: http://gerrit.cloudera.org:8080/#/c/12579/11/be/src/rpc/TAcceptQueueServer.cpp@130 PS11, Line 130: timeout nit: timeout_ms http://gerrit.cloudera.org:8080/#/c/12579/10/tests/custom_cluster/test_frontend_connection_limit.py File tests/custom_cluster/test_frontend_connection_limit.py: http://gerrit.cloudera.org:8080/#/c/12579/10/tests/custom_cluster/test_frontend_connection_limit.py@90 PS10, Line 90: except Exception as e: > The test will be marked as failed in that case (XFAIL). Added an else block My understanding is that xfail will just mark it as "unexpected pass" without flagging any other problem if the test happens to pass. Would it be more robust to set a flag in the exception handling block and assert that the flag is set at the end of the test ? In this way, there is no need to mark the test as xfail. -- To view, visit http://gerrit.cloudera.org:8080/12579 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idb345c1d84cc2f691f54ded467f253e758f87e64 Gerrit-Change-Number: 12579 Gerrit-PatchSet: 11 Gerrit-Owner: Zoram Thanga Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Zoram Thanga Gerrit-Comment-Date: Wed, 27 Mar 2019 22:35:07 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8312 : Alter database operations have race condition
Vihang Karajgaonkar has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/12789 ) Change subject: IMPALA-8312 : Alter database operations have race condition .. IMPALA-8312 : Alter database operations have race condition This patch fixes a race condition in the alter database implementation in the catalogOpExecutor. The original implementation did a in-place modification of the metastore database object in the Db. This can lead to partially updated database object becoming visible to a reading thread causing potential problems. In order to fix the race, the patch makes a copy of the existing database object, modifies the copy and then atomically switches the actual database object with the modified copy. This is done while holding the metastoreddlLock, and then taking the write lock on the catalog version object which makes it consistent with the other catalog write operations. Added a test which consistently reproduces the race. The test creating many reader threads and a writer thread which continuously keeps changing the owner name and its type by issuing a alter database operation. The test fails without the patch. After the patch the test passes. The race also applies to the alter database set comment operation, although its hard to write a test for that code-path. Change-Id: I32c8c96a6029bf9d9db37ea8315f6c9603b5a2fc --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/Db.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java A fe/src/test/java/org/apache/impala/catalog/AlterDatabaseTest.java 4 files changed, 314 insertions(+), 41 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/12789/8 -- To view, visit http://gerrit.cloudera.org:8080/12789 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I32c8c96a6029bf9d9db37ea8315f6c9603b5a2fc Gerrit-Change-Number: 12789 Gerrit-PatchSet: 8 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8225: Add Ranger support for grant/revoke privilege to/from user
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12769 ) Change subject: IMPALA-8225: Add Ranger support for grant/revoke privilege to/from user .. IMPALA-8225: Add Ranger support for grant/revoke privilege to/from user This patch adds support for GRANT privilege statements to USER and REVOKE privilege statements from USER. The RangerAuthorizationManager class has been created and will throw UnsupportedOperationException when an unimplemented method is called. The grammar has been updated to support FROM USER and TO USER for GRANT/REVOKE statements. Previously, privileges could be granted to a ROLE via GRANT/REVOKE statements even when the ROLE keyword was omitted, i.e: GRANT ON TO This is still the case for ROLE based authorization to preserve backward compatibility, but Ranger will throw an exception when a GRANT/REVOKE statement excludes the USER keyword. The syntax for the new statement is: GRANT ON TO USER REVOKE ON FROM USER Sentry does not support grant/revoke to/from user. Testing: - An additional end to end test, test_ranger.py, was added. A single test was added that grants and revokes for a user and asserts permissions on a table. The test uses sleep statements to work with Ranger's polling interval for policy changes. More end to end tests will be added in the future when the refresh authorization statement works properly with Ranger. - AuthorizationStmtTest has been refactored to use the new RangerCatalogdAuthorizationManager grant/revoke methods for better test coverage. - Ran all FE tests - Ran all E2E authorization tests Change-Id: I6ee97bf41546d63385026c0e2b19545565402462 Reviewed-on: http://gerrit.cloudera.org:8080/12769 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M common/thrift/JniCatalog.thrift M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/GrantRevokePrivStmt.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/NoneAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java A fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryCatalogdAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryImpaladAuthorizationManager.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java M fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java A tests/authorization/test_ranger.py 13 files changed, 605 insertions(+), 224 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/12769 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I6ee97bf41546d63385026c0e2b19545565402462 Gerrit-Change-Number: 12769 Gerrit-PatchSet: 12 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8225: Add Ranger support for grant/revoke privilege to/from user
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12769 ) Change subject: IMPALA-8225: Add Ranger support for grant/revoke privilege to/from user .. Patch Set 11: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12769 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6ee97bf41546d63385026c0e2b19545565402462 Gerrit-Change-Number: 12769 Gerrit-PatchSet: 11 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 27 Mar 2019 21:57:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8345 : Add option to set up minicluster to use Hive 3
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12846 ) Change subject: IMPALA-8345 : Add option to set up minicluster to use Hive 3 .. Patch Set 10: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2566/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12846 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icfed856c1f5429ed45fd3d9cb08a5d1bb96a9605 Gerrit-Change-Number: 12846 Gerrit-PatchSet: 10 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 27 Mar 2019 21:45:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8345 : Add option to set up minicluster to use Hive 3
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12846 ) Change subject: IMPALA-8345 : Add option to set up minicluster to use Hive 3 .. Patch Set 11: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3962/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/12846 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icfed856c1f5429ed45fd3d9cb08a5d1bb96a9605 Gerrit-Change-Number: 12846 Gerrit-PatchSet: 11 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 27 Mar 2019 21:06:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8345 : Add option to set up minicluster to use Hive 3
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12846 ) Change subject: IMPALA-8345 : Add option to set up minicluster to use Hive 3 .. Patch Set 11: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12846 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icfed856c1f5429ed45fd3d9cb08a5d1bb96a9605 Gerrit-Change-Number: 12846 Gerrit-PatchSet: 11 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 27 Mar 2019 21:06:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8345 : Add option to set up minicluster to use Hive 3
Fredy Wijaya has posted comments on this change. (
http://gerrit.cloudera.org:8080/12846 )
Change subject: IMPALA-8345 : Add option to set up minicluster to use Hive 3
..
Patch Set 10: Code-Review+2
(1 comment)
I saw few +1s from Tim and Andrew. I'm going to promote it to +2. This is a
good start. We can always iterate it again to improve it further.
http://gerrit.cloudera.org:8080/#/c/12846/9/bin/create-test-configuration.sh
File bin/create-test-configuration.sh:
http://gerrit.cloudera.org:8080/#/c/12846/9/bin/create-test-configuration.sh@149
PS9, Line 149: 1>${IMPALA_CLUSTER_LOGS_DIR}/schematool.log 2>&1
> schematool has a problem which prints bunch of new lines on the stdout afte
Sounds good.
--
To view, visit http://gerrit.cloudera.org:8080/12846
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icfed856c1f5429ed45fd3d9cb08a5d1bb96a9605
Gerrit-Change-Number: 12846
Gerrit-PatchSet: 10
Gerrit-Owner: Vihang Karajgaonkar
Gerrit-Reviewer: Andrew Sherman
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Todd Lipcon
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Comment-Date: Wed, 27 Mar 2019 21:05:15 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8345 : Add option to set up minicluster to use Hive 3
Vihang Karajgaonkar has uploaded a new patch set (#10). (
http://gerrit.cloudera.org:8080/12846 )
Change subject: IMPALA-8345 : Add option to set up minicluster to use Hive 3
..
IMPALA-8345 : Add option to set up minicluster to use Hive 3
As a first step to integrate Impala with Hive 3.1.0 this patch modifies
the minicluster scripts to optionally use Hive 3.1.0 instead of
CDH Hive 2.1.1.
In order to make sure that existing setups don't break this is
enabled via a environment variable override to bin/impala-config.sh.
When the environment variable USE_CDP_HIVE is set to true the
bootstrap_toolchain script downloads Hive 3.1.0 tarballs and extracts it
in the toolchain directory. These binaries are used to start the Hive
services (Hiveserver2 and metastore). The default is still CDH Hive 2.1.1
Also, since Hive 3.1.0 uses a upgraded metastore schema, this patch
makes use of a different database name so that it is easy to switch from
working from one environment which uses Hive 2.1.1 metastore to another
which usese Hive 3.1.0 metastore.
In order to start a minicluster which uses Hive 3.1.0 users should
follow the steps below:
1. Make sure that minicluster, if running, is stopped
before you run the following commands.
2. Open a new terminal and run following commands.
> export USE_CDP_HIVE=true
> source bin/impala-config.sh
> bin/bootstrap_toolchain.py
The above command downloads the Hive 3.1.0 tarballs and extracts them
in toolchain/cdp_components-${CDP_BUILD_NUMBER} directory. This is a
no-op if the CDP_BUILD_NUMBER has not changed and if the cdp_components
are already downloaded by a previous invocation of the script.
> source bin/create-test-configuration.sh -create-metastore
The above step should provide "-create-metastore" only the first time
so that a new metastore db is created and the Hive 3.1.0 schema is
initialized. For all subsequent invocations, the "-create-metastore"
argument can be skipped. We should still source this script since the
hive-site.xml of Hive 3.1.0 is different than Hive 2.1.0 and
needs to be regenerated.
> testdata/bin/run-all.sh
Note that the testing was performed locally by downloading the Hive 3.1
binaries into
toolchain/cdp_components-976603/apache-hive-3.1.0.6.0.99.0-9-bin. Once
the binaries are available in S3 bucket, the bootstrap_toolchain script
should automatically do this for you.
Testing Done:
1. Made sure that the cluster comes up with Hive 3.1 when the steps
above are performed.
2. Made sure that existing scripts work as they do currently when
argument is not provided.
3. Impala cluster comes and connects to HMS 3.1.0 (Note that Impala
still uses Hive 2.1.1 client. Upgrading client libraries in Impala will
be done as a separate change)
Change-Id: Icfed856c1f5429ed45fd3d9cb08a5d1bb96a9605
---
M bin/bootstrap_toolchain.py
M bin/create-test-configuration.sh
M bin/impala-config.sh
A fe/src/test/resources/postgresql-hive-site.xml.cdp.template
M testdata/bin/run-hive-server.sh
5 files changed, 284 insertions(+), 11 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/46/12846/10
--
To view, visit http://gerrit.cloudera.org:8080/12846
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Icfed856c1f5429ed45fd3d9cb08a5d1bb96a9605
Gerrit-Change-Number: 12846
Gerrit-PatchSet: 10
Gerrit-Owner: Vihang Karajgaonkar
Gerrit-Reviewer: Andrew Sherman
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Todd Lipcon
Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] [DOCS] DECIMAL V2 default value is now TRUE
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12870 ) Change subject: [DOCS] DECIMAL_V2 default value is now TRUE .. Patch Set 2: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/292/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/12870 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5eaa144a49e1d72a20982e0b8cf4f4a36142c578 Gerrit-Change-Number: 12870 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 27 Mar 2019 20:02:52 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] DECIMAL V2 default value is now TRUE
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12870 ) Change subject: [DOCS] DECIMAL_V2 default value is now TRUE .. Patch Set 1: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/291/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/12870 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5eaa144a49e1d72a20982e0b8cf4f4a36142c578 Gerrit-Change-Number: 12870 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 27 Mar 2019 20:01:14 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] DECIMAL V2 default value is now TRUE
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12870 to look at the new patch set (#2). Change subject: [DOCS] DECIMAL_V2 default value is now TRUE .. [DOCS] DECIMAL_V2 default value is now TRUE Change-Id: I5eaa144a49e1d72a20982e0b8cf4f4a36142c578 --- M docs/topics/impala_decimal_v2.xml 1 file changed, 7 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/70/12870/2 -- To view, visit http://gerrit.cloudera.org:8080/12870 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5eaa144a49e1d72a20982e0b8cf4f4a36142c578 Gerrit-Change-Number: 12870 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] [DOCS] DECIMAL V2 default value is now TRUE
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/12870 ) Change subject: [DOCS] DECIMAL_V2 default value is now TRUE .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12870 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5eaa144a49e1d72a20982e0b8cf4f4a36142c578 Gerrit-Change-Number: 12870 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 27 Mar 2019 19:41:13 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] DECIMAL V2 default value is now TRUE
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12870 ) Change subject: [DOCS] DECIMAL_V2 default value is now TRUE .. Patch Set 2: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/292/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/12870 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5eaa144a49e1d72a20982e0b8cf4f4a36142c578 Gerrit-Change-Number: 12870 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 27 Mar 2019 19:46:15 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] DECIMAL V2 default value is now TRUE
Alex Rodoni has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12870 Change subject: [DOCS] DECIMAL_V2 default value is now TRUE .. [DOCS] DECIMAL_V2 default value is now TRUE Change-Id: I5eaa144a49e1d72a20982e0b8cf4f4a36142c578 --- M docs/topics/impala_decimal_v2.xml 1 file changed, 3 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/70/12870/1 -- To view, visit http://gerrit.cloudera.org:8080/12870 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I5eaa144a49e1d72a20982e0b8cf4f4a36142c578 Gerrit-Change-Number: 12870 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni
[Impala-ASF-CR] [DOCS] DECIMAL V2 default value is now TRUE
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12870 ) Change subject: [DOCS] DECIMAL_V2 default value is now TRUE .. Patch Set 1: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/291/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/12870 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5eaa144a49e1d72a20982e0b8cf4f4a36142c578 Gerrit-Change-Number: 12870 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 27 Mar 2019 19:40:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7982: Add host network usage to profile
John Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/12747 ) Change subject: IMPALA-7982: Add host network usage to profile .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/12747/2/be/src/util/system-state-info.cc File be/src/util/system-state-info.cc: http://gerrit.cloudera.org:8080/#/c/12747/2/be/src/util/system-state-info.cc@73 PS2, Line 73: LOG(WARNING) << "Could not open " << path << ": " << GetStrErrMsg() << endl; Should you just print the error from the status? or is the errno still correctly preserved for GetStrErrMsg to determine the error msg? (Same applies to ReadCurrentProcNetDev) -- To view, visit http://gerrit.cloudera.org:8080/12747 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2cc74f87374080a74a13b7fb6e4da44a11d828ef Gerrit-Change-Number: 12747 Gerrit-PatchSet: 2 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 27 Mar 2019 19:20:26 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8330: Impala shell config file should support flag names
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/12823 ) Change subject: IMPALA-8330: Impala shell config file should support flag names .. Patch Set 6: TBH, the original approach wasn't usable at all and I don't mind deprecating it. I highly doubt if any serious user is relying on it. Users need to know the internal variables in the shell code to configure their rc files, ouch! -- To view, visit http://gerrit.cloudera.org:8080/12823 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic43603c1b538af08fddcab1b2c1f6ad1af1a6cb9 Gerrit-Change-Number: 12823 Gerrit-PatchSet: 6 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Comment-Date: Wed, 27 Mar 2019 19:04:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8330: Impala shell config file should support flag names
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12823 ) Change subject: IMPALA-8330: Impala shell config file should support flag names .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2565/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12823 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic43603c1b538af08fddcab1b2c1f6ad1af1a6cb9 Gerrit-Change-Number: 12823 Gerrit-PatchSet: 6 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Comment-Date: Wed, 27 Mar 2019 18:46:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8330: Impala shell config file should support flag names
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/12823 ) Change subject: IMPALA-8330: Impala shell config file should support flag names .. Patch Set 6: Updated the CR to no longer be a breaking change. -- To view, visit http://gerrit.cloudera.org:8080/12823 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic43603c1b538af08fddcab1b2c1f6ad1af1a6cb9 Gerrit-Change-Number: 12823 Gerrit-PatchSet: 6 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Comment-Date: Wed, 27 Mar 2019 18:04:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8330: Impala shell config file should support flag names
Fredy Wijaya has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/12823 ) Change subject: IMPALA-8330: Impala shell config file should support flag names .. IMPALA-8330: Impala shell config file should support flag names This patch updates the file format in Impala shell config file to accept both short and long flag names in addition to optparse's dest names (variable names to store flag values) for better user experience because dest names are internal to Impala shell. Format: [impala] flag_name=flag_value Example: [impala] ; This is long flag. query=select 1 ; This is short flag. Q=DEFAULT_FILE_FORMAT=parquet ; Flags can be repeated with , var=msg1=hello,var=msg2=world ; The old format using internal variable name is still supported for ; backward compatibility. keyval=msg3=foo,keyval=msg4=bar Testing: - Ran all E2E shell tests on Python 2.6 and 2.7. Change-Id: Ic43603c1b538af08fddcab1b2c1f6ad1af1a6cb9 --- M shell/impala_shell.py M shell/option_parser.py M tests/shell/good_impalarc M tests/shell/test_shell_commandline.py 4 files changed, 44 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/12823/6 -- To view, visit http://gerrit.cloudera.org:8080/12823 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic43603c1b538af08fddcab1b2c1f6ad1af1a6cb9 Gerrit-Change-Number: 12823 Gerrit-PatchSet: 6 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple
[Impala-ASF-CR] IMPALA-7368: Add initial support for DATE type
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12481 ) Change subject: IMPALA-7368: Add initial support for DATE type .. Patch Set 11: (2 comments) http://gerrit.cloudera.org:8080/#/c/12481/14/fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java File fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java: http://gerrit.cloudera.org:8080/#/c/12481/14/fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java@351 PS14, Line 351: return "AVG requires a numeric, timestamp or date parameter: " + toSql(); Needs update http://gerrit.cloudera.org:8080/#/c/12481/11/fe/src/main/java/org/apache/impala/util/FunctionUtils.java File fe/src/main/java/org/apache/impala/util/FunctionUtils.java: http://gerrit.cloudera.org:8080/#/c/12481/11/fe/src/main/java/org/apache/impala/util/FunctionUtils.java@243 PS11, Line 243: return max_func; > That's correct, currently we never raise "ambiguous resolution" error. I don't think this behaviour was a well thought-out decision. The earliest versions of this that I saw was very hacky and actually returned an arbitrary function when there were multiple valid overloads - whichever was listed first in the catalog. I changed that to be a bit more intelligent about choosing functions that don't result in loss of information and also and checking functions in a deterministic order. It's not the way I would have designed it but it was the least disruptive way to evolve it. Introducing an error where there wasn't one before is pretty risky because it will probably break existing queries. Your example seems highly likely to be used in practice. There are some cases where it might not be a breaking change to raise an error, because they didn't occur before, e.g. if one of the inputs is a DATE. But not sure how interesting those cases are. I tend to think that something like this is probably the least of the evils, but the complexity of the behaviour is not ideal. -- To view, visit http://gerrit.cloudera.org:8080/12481 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iea8155ef09557e0afa2f8b2d0b2dc9d0896dc30f Gerrit-Change-Number: 12481 Gerrit-PatchSet: 11 Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 27 Mar 2019 17:59:06 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8225: Add Ranger support for grant/revoke privilege to/from user
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/12769 ) Change subject: IMPALA-8225: Add Ranger support for grant/revoke privilege to/from user .. Patch Set 10: Code-Review+2 Thanks for all the fixes. -- To view, visit http://gerrit.cloudera.org:8080/12769 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6ee97bf41546d63385026c0e2b19545565402462 Gerrit-Change-Number: 12769 Gerrit-PatchSet: 10 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 27 Mar 2019 17:25:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7368: Add initial support for DATE type
Attila Jeges has posted comments on this change. (
http://gerrit.cloudera.org:8080/12481 )
Change subject: IMPALA-7368: Add initial support for DATE type
..
Patch Set 15:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/12481/14/fe/src/main/java/org/apache/impala/analysis/PartitionSpec.java
File fe/src/main/java/org/apache/impala/analysis/PartitionSpec.java:
http://gerrit.cloudera.org:8080/#/c/12481/14/fe/src/main/java/org/apache/impala/analysis/PartitionSpec.java@52
PS14, Line 52: public PartitionSpec(List partitionSpec) {
> Do we ever expect this to get called prior to analysis?
Done
http://gerrit.cloudera.org:8080/#/c/12481/14/fe/src/main/java/org/apache/impala/analysis/PartitionSpec.java@156
PS14, Line 156: }
> same (Precondition on analysis if possible)
Done
http://gerrit.cloudera.org:8080/#/c/12481/11/fe/src/main/java/org/apache/impala/catalog/Function.java
File fe/src/main/java/org/apache/impala/catalog/Function.java:
http://gerrit.cloudera.org:8080/#/c/12481/11/fe/src/main/java/org/apache/impala/catalog/Function.java@191
PS11, Line 191: // Compares this to 'other' for 'mode'.
> hm, I spent some time researching this across multiple databases:
I agree that the function overload resolution algorithm needs to be redesigned
but I don't think we should do it now as it would break backward compatibility.
We can reconsider implicit casting rules and re-implement the resolution logic
(preferably modelled after what Hive does) in a separate change when we are
ready for a breaking change.
What do you think?
http://gerrit.cloudera.org:8080/#/c/12481/11/fe/src/main/java/org/apache/impala/catalog/Function.java@236
PS11, Line 236: private int calcSuperTypeOfMatchScore(Function other, boolean
strict) {
> I think we could simplify this code a lot by adding some utility function l
Done
http://gerrit.cloudera.org:8080/#/c/12481/11/fe/src/main/java/org/apache/impala/util/FunctionUtils.java
File fe/src/main/java/org/apache/impala/util/FunctionUtils.java:
http://gerrit.cloudera.org:8080/#/c/12481/11/fe/src/main/java/org/apache/impala/util/FunctionUtils.java@243
PS11, Line 243: return max_func;
> it's interesting that our function resolution returns the first fit, from t
That's correct, currently we never raise "ambiguous resolution" error.
Adding an error to that effect sounds reasonable but I'm worried that it will
introduce a breaking change.
E.g. year('2011-01-01') resolves to year(TIMESTAMP) right now, but after
introducing year(DATE) users will get an error.
--
To view, visit http://gerrit.cloudera.org:8080/12481
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea8155ef09557e0afa2f8b2d0b2dc9d0896dc30f
Gerrit-Change-Number: 12481
Gerrit-PatchSet: 15
Gerrit-Owner: Attila Jeges
Gerrit-Reviewer: Attila Jeges
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Todd Lipcon
Gerrit-Comment-Date: Wed, 27 Mar 2019 17:42:27 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8359: Fix coverage data generation for impalads
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/12858 ) Change subject: IMPALA-8359: Fix coverage data generation for impalads .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/12858/2/be/src/common/init.cc File be/src/common/init.cc: http://gerrit.cloudera.org:8080/#/c/12858/2/be/src/common/init.cc@199 PS2, Line 199: // We measure code coverage that needs cleanup of resources to flush coverage data. : // In this case we risk calling exit() that does the cleanup unlike _exit(). : exit(0); It's good to track this down. I'm glad we know this is due to the change to an _exit(0) call. There is a function called __gcov_flush() that dumps the coverage data. You can declare it and use it if you link with "-lgcov". I think a common pattern is to call it before _exit(0). I did a quick check, and it seems to work. Kudu also has some code to flush coverage data (see TryFlushCoverage() in be/src/kudu/util/debug-util.cc in our repo). If we link with -lgcov and call kudu::TryFlushCoverage() before _exit(0), then that would work. Kudu uses clang for coverage rather than gcc, but the same code should work. I would prefer one of these two solutions rather than an exit() call, but as a workaround, I would approve this. -- To view, visit http://gerrit.cloudera.org:8080/12858 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9be1e1e73b6cfc3557077f763aee4dbfcc7a2d27 Gerrit-Change-Number: 12858 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 27 Mar 2019 17:34:04 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7368: Add initial support for DATE type
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12481 ) Change subject: IMPALA-7368: Add initial support for DATE type .. Patch Set 15: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2563/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12481 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iea8155ef09557e0afa2f8b2d0b2dc9d0896dc30f Gerrit-Change-Number: 12481 Gerrit-PatchSet: 15 Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 27 Mar 2019 17:32:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8225: Add Ranger support for grant/revoke privilege to/from user
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12769 ) Change subject: IMPALA-8225: Add Ranger support for grant/revoke privilege to/from user .. Patch Set 10: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2564/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12769 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6ee97bf41546d63385026c0e2b19545565402462 Gerrit-Change-Number: 12769 Gerrit-PatchSet: 10 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 27 Mar 2019 17:26:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8225: Add Ranger support for grant/revoke privilege to/from user
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12769 ) Change subject: IMPALA-8225: Add Ranger support for grant/revoke privilege to/from user .. Patch Set 11: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3961/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/12769 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6ee97bf41546d63385026c0e2b19545565402462 Gerrit-Change-Number: 12769 Gerrit-PatchSet: 11 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 27 Mar 2019 17:25:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8225: Add Ranger support for grant/revoke privilege to/from user
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12769 ) Change subject: IMPALA-8225: Add Ranger support for grant/revoke privilege to/from user .. Patch Set 11: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12769 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6ee97bf41546d63385026c0e2b19545565402462 Gerrit-Change-Number: 12769 Gerrit-PatchSet: 11 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 27 Mar 2019 17:25:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8225: Add Ranger support for grant/revoke privilege to/from user
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12769 ) Change subject: IMPALA-8225: Add Ranger support for grant/revoke privilege to/from user .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2562/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12769 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6ee97bf41546d63385026c0e2b19545565402462 Gerrit-Change-Number: 12769 Gerrit-PatchSet: 9 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 27 Mar 2019 17:16:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8225: Add Ranger support for grant/revoke privilege to/from user
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12769 ) Change subject: IMPALA-8225: Add Ranger support for grant/revoke privilege to/from user .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2561/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12769 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6ee97bf41546d63385026c0e2b19545565402462 Gerrit-Change-Number: 12769 Gerrit-PatchSet: 8 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 27 Mar 2019 17:12:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6326: part 2: remove fetch thread in stress test
Thomas Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/12681 ) Change subject: IMPALA-6326: part 2: remove fetch thread in stress test .. Patch Set 6: Code-Review+2 (2 comments) http://gerrit.cloudera.org:8080/#/c/12681/6/tests/stress/query_runner.py File tests/stress/query_runner.py: http://gerrit.cloudera.org:8080/#/c/12681/6/tests/stress/query_runner.py@201 PS6, Line 201: IMPALA-8288 this JIRA has already been resolved http://gerrit.cloudera.org:8080/#/c/12681/6/tests/stress/query_runner.py@253 PS6, Line 253: from to? -- To view, visit http://gerrit.cloudera.org:8080/12681 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If9afd74e1408823a9e5c0f2628ec9f8aafdcec69 Gerrit-Change-Number: 12681 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 27 Mar 2019 17:12:08 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8225: Add Ranger support for grant/revoke privilege to/from user
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12769 ) Change subject: IMPALA-8225: Add Ranger support for grant/revoke privilege to/from user .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2560/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12769 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6ee97bf41546d63385026c0e2b19545565402462 Gerrit-Change-Number: 12769 Gerrit-PatchSet: 7 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 27 Mar 2019 17:09:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7368: Add initial support for DATE type
Attila Jeges has uploaded a new patch set (#15). (
http://gerrit.cloudera.org:8080/12481 )
Change subject: IMPALA-7368: Add initial support for DATE type
..
IMPALA-7368: Add initial support for DATE type
DATE values describe a particular year/month/day in the form
-MM-dd. For example: DATE '2019-02-15'. DATE values do not have a
time of day component. The range of values supported for the DATE type
is -01-01 to -12-31.
This initial DATE type support covers TEXT and HBASE fileformats only.
'DateValue' is used as the internal type to represent DATE values.
The changes are as follows:
- Support for DATE literal syntax.
- Explicit casting between DATE and other types:
- from STRING to DATE. The string value must be formatted as
-MM-dd.
- from DATE to STRING. The resulting string value is formatted as
-MM-dd.
- from TIMESTAMP to DATE. The source timestamp's time of day
component is ignored.
- from DATE to TIMESTAMP. The target timestamp's time of day
component is set to 00:00:00.
- Implicit casting between DATE and other types:
- from STRING to DATE if the source string value is used in a
context where a DATE value is expected.
- from DATE to TIMESTAMP if the source date value is used in a
context where a TIMESTAMP value is expected.
- Since STRING -> DATE, STRING -> TIMESTAMP and DATE -> TIMESTAMP
implicit conversions are now all possible, the existing function
overload resolution logic is not adequate anymore.
For example, it resolves the
if(false, '2011-01-01', DATE '1499-02-02') function call to the
if(BOOLEAN, TIMESTAMP, TIMESTAMP) version of the overloaded
function, instead of the if(BOOLEAN, DATE, DATE) version.
This is clearly wrong, so the function overload resolution logic had
to be changed to resolve function calls to the best-fit overloaded
function definition if there are multiple applicable candidates.
An overloaded function definition is an applicable candidate for a
function call if each actual parameter in the function call either
matches the corresponding formal parameter's type (without casting)
or is implicitly castable to that type.
When looking for the best-fit applicable candidate, a parameter
match score (i.e. the number of actual parameters in the function
call that match their corresponding formal parameter's type without
casting) is calculated and the applicable candidate with the highest
parameter match score is chosen.
There's one more issue that the new resolution logic has to address:
if two applicable candidates have the same parameter match score and
the only difference between the two is that the first one requires a
STRING -> TIMESTAMP implicit cast for some of its parameters while
the second one requires a STRING -> DATE implicit cast for the same
parameters then the first candidate has to be chosen not to break
backward compatibility.
E.g: year('2019-02-15') function call must resolve to
year(TIMESTAMP) instead of year(DATE). Note, that year(DATE) is not
implemented yet, so this is not an issue at the moment but it will
be in the future.
When the resolution algorithm considers overloaded function
definitions, first it orders them lexicographically by the types in
their parameter lists. To ensure the backward compatible behavior
Primitivetype.DATE enum value has to come after
PrimitiveType.TIMESTAMP.
- Codegen infrastructure changes for expression evaluation.
- 'IS [NOT] NULL' and '[NOT] IN' predicates.
- Common comparison operators (including the 'BETWEEN' operator).
- Infrastructure changes for built-in functions.
- Some built-in functions: conditional, aggregate, analytical and
math functions.
- C++ UDF/UDA support.
- Support partitioning and grouping by DATE.
- Beeswax, HiveServer2 support.
These items are tightly coupled and it makes sense to implement them
in one change-set.
Testing:
- A new partitioned TEXT table 'functional.date_tbl' (and the
corresponding HBASE table 'functional_hbase.date_tbl') was
introduced for DATE-related tests.
- BE and FE tests were extended to cover DATE type.
- E2E tests:
- since DATE type is supported for TEXT and HBASE fileformats
only, most DATE tests were implemented separately in
tests/query_test/test_date_queries.py.
Note, that this change-set is not a complete DATE type implementation,
but it lays the foundation for future work:
- Add date support to the random query generator.
- Implement a complete set of built-in functions.
- Add Parquet support.
- Add Kudu support.
- Optionally support Avro and ORC.
For further details, see IMPALA-6169.
Change-Id: Iea8155ef09557e0afa2f8b2d0b2dc9d0896dc30f
---
M be/src/codegen/codegen-anyval.cc
M be/src/codegen/codegen-anyval.h
M be/src/codegen/gen_ir_descriptions.py
M be/src/codegen/llvm-codegen.cc
M be/src/exec/aggregator.cc
M
[Impala-ASF-CR] IMPALA-8225: Add Ranger support for grant/revoke privilege to/from user
Austin Nobis has uploaded a new patch set (#10). ( http://gerrit.cloudera.org:8080/12769 ) Change subject: IMPALA-8225: Add Ranger support for grant/revoke privilege to/from user .. IMPALA-8225: Add Ranger support for grant/revoke privilege to/from user This patch adds support for GRANT privilege statements to USER and REVOKE privilege statements from USER. The RangerAuthorizationManager class has been created and will throw UnsupportedOperationException when an unimplemented method is called. The grammar has been updated to support FROM USER and TO USER for GRANT/REVOKE statements. Previously, privileges could be granted to a ROLE via GRANT/REVOKE statements even when the ROLE keyword was omitted, i.e: GRANT ON TO This is still the case for ROLE based authorization to preserve backward compatibility, but Ranger will throw an exception when a GRANT/REVOKE statement excludes the USER keyword. The syntax for the new statement is: GRANT ON TO USER REVOKE ON FROM USER Sentry does not support grant/revoke to/from user. Testing: - An additional end to end test, test_ranger.py, was added. A single test was added that grants and revokes for a user and asserts permissions on a table. The test uses sleep statements to work with Ranger's polling interval for policy changes. More end to end tests will be added in the future when the refresh authorization statement works properly with Ranger. - AuthorizationStmtTest has been refactored to use the new RangerCatalogdAuthorizationManager grant/revoke methods for better test coverage. - Ran all FE tests - Ran all E2E authorization tests Change-Id: I6ee97bf41546d63385026c0e2b19545565402462 --- M common/thrift/JniCatalog.thrift M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/GrantRevokePrivStmt.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/NoneAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java A fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryCatalogdAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryImpaladAuthorizationManager.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java M fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java A tests/authorization/test_ranger.py 13 files changed, 605 insertions(+), 224 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/12769/10 -- To view, visit http://gerrit.cloudera.org:8080/12769 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6ee97bf41546d63385026c0e2b19545565402462 Gerrit-Change-Number: 12769 Gerrit-PatchSet: 10 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8330: Impala shell config file should use flag names
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/12823 ) Change subject: IMPALA-8330: Impala shell config file should use flag names .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/12823/5/tests/shell/good_impalarc File tests/shell/good_impalarc: PS5: > If this breaks existing .impalarc files, I'd prefer to wait until 4.0. I can make this backward compatible and still support the new feature, too. I'll do that in my next PS. -- To view, visit http://gerrit.cloudera.org:8080/12823 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic43603c1b538af08fddcab1b2c1f6ad1af1a6cb9 Gerrit-Change-Number: 12823 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Comment-Date: Wed, 27 Mar 2019 16:47:03 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8225: Add Ranger support for grant/revoke privilege to/from user
Fredy Wijaya has posted comments on this change. (
http://gerrit.cloudera.org:8080/12769 )
Change subject: IMPALA-8225: Add Ranger support for grant/revoke privilege
to/from user
..
Patch Set 8:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/12769/8/tests/authorization/test_ranger.py
File tests/authorization/test_ranger.py:
http://gerrit.cloudera.org:8080/#/c/12769/8/tests/authorization/test_ranger.py@45
PS8, Line 45:
: admin_client.execute("grant select on database functional to
user {0}".format(user),
: user="admin")
: # TODO: IMPALA-8293 use refresh authorization
: time.sleep(30)
: user_client.execute("show tables in functional", user=user)
: admin_client.execute("revoke select on database functional
from user {0}"
: .format(user), user="admin")
can you use execute_query_expect_success here?
--
To view, visit http://gerrit.cloudera.org:8080/12769
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6ee97bf41546d63385026c0e2b19545565402462
Gerrit-Change-Number: 12769
Gerrit-PatchSet: 8
Gerrit-Owner: Austin Nobis
Gerrit-Reviewer: Austin Nobis
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Comment-Date: Wed, 27 Mar 2019 16:35:54 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8225: Add Ranger support for grant/revoke privilege to/from user
Austin Nobis has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/12769 ) Change subject: IMPALA-8225: Add Ranger support for grant/revoke privilege to/from user .. IMPALA-8225: Add Ranger support for grant/revoke privilege to/from user This patch adds support for GRANT privilege statements to USER and REVOKE privilege statements from USER. The RangerAuthorizationManager class has been created and will throw UnsupportedOperationException when an unimplemented method is called. The grammar has been updated to support FROM USER and TO USER for GRANT/REVOKE statements. Previously, privileges could be granted to a ROLE via GRANT/REVOKE statements even when the ROLE keyword was omitted, i.e: GRANT ON TO This is still the case for ROLE based authorization to preserve backward compatibility, but Ranger will throw an exception when a GRANT/REVOKE statement excludes the USER keyword. The syntax for the new statement is: GRANT ON TO USER REVOKE ON FROM USER Sentry does not support grant/revoke to/from user. Testing: - An additional end to end test, test_ranger.py, was added. A single test was added that grants and revokes for a user and asserts permissions on a table. The test uses sleep statements to work with Ranger's polling interval for policy changes. More end to end tests will be added in the future when the refresh authorization statement works properly with Ranger. - AuthorizationStmtTest has been refactored to use the new RangerCatalogdAuthorizationManager grant/revoke methods for better test coverage. - Ran all FE tests - Ran all E2E authorization tests Change-Id: I6ee97bf41546d63385026c0e2b19545565402462 --- M common/thrift/JniCatalog.thrift M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/GrantRevokePrivStmt.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/NoneAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java A fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryCatalogdAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryImpaladAuthorizationManager.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java M fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java A tests/authorization/test_ranger.py 13 files changed, 598 insertions(+), 224 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/12769/8 -- To view, visit http://gerrit.cloudera.org:8080/12769 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6ee97bf41546d63385026c0e2b19545565402462 Gerrit-Change-Number: 12769 Gerrit-PatchSet: 8 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8225: Add Ranger support for grant/revoke privilege to/from user
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12769 ) Change subject: IMPALA-8225: Add Ranger support for grant/revoke privilege to/from user .. Patch Set 7: (1 comment) http://gerrit.cloudera.org:8080/#/c/12769/7/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java File fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java: http://gerrit.cloudera.org:8080/#/c/12769/7/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java@3005 PS7, Line 3005: RangerCatalogdAuthorizationManager.createGrantRevokeRequests(RANGER_ADMIN.getName(), line too long (96 > 90) -- To view, visit http://gerrit.cloudera.org:8080/12769 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6ee97bf41546d63385026c0e2b19545565402462 Gerrit-Change-Number: 12769 Gerrit-PatchSet: 7 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 27 Mar 2019 16:25:34 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8225: Add Ranger support for grant/revoke privilege to/from user
Austin Nobis has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/12769 ) Change subject: IMPALA-8225: Add Ranger support for grant/revoke privilege to/from user .. IMPALA-8225: Add Ranger support for grant/revoke privilege to/from user This patch adds support for GRANT privilege statements to USER and REVOKE privilege statements from USER. The RangerAuthorizationManager class has been created and will throw UnsupportedOperationException when an unimplemented method is called. The grammar has been updated to support FROM USER and TO USER for GRANT/REVOKE statements. Previously, privileges could be granted to a ROLE via GRANT/REVOKE statements even when the ROLE keyword was omitted, i.e: GRANT ON TO This is still the case for ROLE based authorization to preserve backward compatibility, but Ranger will throw an exception when a GRANT/REVOKE statement excludes the USER keyword. The syntax for the new statement is: GRANT ON TO USER REVOKE ON FROM USER Sentry does not support grant/revoke to/from user. Testing: - An additional end to end test, test_ranger.py, was added. A single test was added that grants and revokes for a user and asserts permissions on a table. The test uses sleep statements to work with Ranger's polling interval for policy changes. More end to end tests will be added in the future when the refresh authorization statement works properly with Ranger. - AuthorizationStmtTest has been refactored to use the new RangerCatalogdAuthorizationManager grant/revoke methods for better test coverage. - Ran all FE tests - Ran all E2E authorization tests Change-Id: I6ee97bf41546d63385026c0e2b19545565402462 --- M common/thrift/JniCatalog.thrift M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/GrantRevokePrivStmt.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/NoneAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java A fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryCatalogdAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryImpaladAuthorizationManager.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java M fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java A tests/authorization/test_ranger.py 13 files changed, 598 insertions(+), 224 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/12769/7 -- To view, visit http://gerrit.cloudera.org:8080/12769 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6ee97bf41546d63385026c0e2b19545565402462 Gerrit-Change-Number: 12769 Gerrit-PatchSet: 7 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR](asf-site) Add 3.2.0 change log
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/12809 ) Change subject: Add 3.2.0 change log .. Patch Set 2: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/12809 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: comment Gerrit-Change-Id: I0592234d9f67677bae0313d65889233e6b9c6210 Gerrit-Change-Number: 12809 Gerrit-PatchSet: 2 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 27 Mar 2019 16:00:08 + Gerrit-HasComments: No
[Impala-ASF-CR](asf-site) Download 3.2.0
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/12808 ) Change subject: Download 3.2.0 .. Patch Set 1: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/12808 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: comment Gerrit-Change-Id: I6b1451baf85a96099116da1a10af8c00a57ee36c Gerrit-Change-Number: 12808 Gerrit-PatchSet: 1 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 27 Mar 2019 15:59:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8314 Set correct permissions for ~/.ssh files
Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12725 ) Change subject: IMPALA-8314 Set correct permissions for ~/.ssh files .. IMPALA-8314 Set correct permissions for ~/.ssh files Without this running bootstrap_system.sh resulted in: ++ chmod go-rwx /home/hacosta/.ssh ++ [[ -f /home/hacosta/.ssh/id_rsa ]] ++ ssh-keygen -t rsa -N '' -q -f /home/hacosta/.ssh/id_rsa ++ cat /home/hacosta/.ssh/id_rsa.pub ++ echo 'NoHostAuthenticationForLocalhost yes' ++ ssh localhost whoami Bad owner or permissions on /home/hacosta/.ssh/config Change-Id: I3dd6b12318dd99c67127bff10a89adb931f44087 Reviewed-on: http://gerrit.cloudera.org:8080/12725 Reviewed-by: Joe McDonnell Tested-by: Impala Public Jenkins --- M bin/bootstrap_system.sh 1 file changed, 2 insertions(+), 2 deletions(-) Approvals: Joe McDonnell: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/12725 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I3dd6b12318dd99c67127bff10a89adb931f44087 Gerrit-Change-Number: 12725 Gerrit-PatchSet: 3 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Joe McDonnell
[Impala-ASF-CR] Add missing sudo call to `service postgresql start`
Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12724 ) Change subject: Add missing sudo call to `service postgresql start` .. Add missing sudo call to `service postgresql start` Change-Id: Ie3859ad1308b7cd1d483d043b9f71d9f95abdf75 Reviewed-on: http://gerrit.cloudera.org:8080/12724 Reviewed-by: Joe McDonnell Tested-by: Impala Public Jenkins --- M bin/bootstrap_system.sh 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: Joe McDonnell: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/12724 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ie3859ad1308b7cd1d483d043b9f71d9f95abdf75 Gerrit-Change-Number: 12724 Gerrit-PatchSet: 3 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal
[Impala-ASF-CR] Use 'wget http://169.254.169.254/' to determine if we're running in aws
Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12727 ) Change subject: Use 'wget http://169.254.169.254/' to determine if we're running in aws .. Use 'wget http://169.254.169.254/' to determine if we're running in aws https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/identify_ec2_instances.html lists the above endpoint as the first choice. In my running instance 'dmidecode -s bios-version' prints out: 1.0 dmidecode --string system-uuid|grep ^ec2 seems like a valid alternative here. I removed the 'grep' assertions since it's possible that images may have already configured ntp here. Change-Id: Iddb2574dbcb3f97cf697095d1777e51ce463b205 Reviewed-on: http://gerrit.cloudera.org:8080/12727 Reviewed-by: Joe McDonnell Tested-by: Impala Public Jenkins --- M bin/bootstrap_system.sh 1 file changed, 3 insertions(+), 3 deletions(-) Approvals: Joe McDonnell: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/12727 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Iddb2574dbcb3f97cf697095d1777e51ce463b205 Gerrit-Change-Number: 12727 Gerrit-PatchSet: 7 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Laszlo Gaal
[Impala-ASF-CR] Move DEBIAN FRONTEND=noninteractive variable to the ubuntu section of the script
Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12723 ) Change subject: Move DEBIAN_FRONTEND=noninteractive variable to the ubuntu section of the script .. Move DEBIAN_FRONTEND=noninteractive variable to the ubuntu section of the script This also changes the behavior to always use DEBIAN_FRONTEND=noninteractive even for interactive terminals. Change-Id: Ifab2f56d3e119e4e73685c3cbd5249760ceb33ac Reviewed-on: http://gerrit.cloudera.org:8080/12723 Reviewed-by: Joe McDonnell Tested-by: Impala Public Jenkins --- M bin/bootstrap_system.sh 1 file changed, 1 insertion(+), 3 deletions(-) Approvals: Joe McDonnell: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/12723 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ifab2f56d3e119e4e73685c3cbd5249760ceb33ac Gerrit-Change-Number: 12723 Gerrit-PatchSet: 3 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell
[Impala-ASF-CR] IMPALA-8307 Use test -x to check for ntp-wait
Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12726 ) Change subject: IMPALA-8307 Use test -x to check for ntp-wait .. IMPALA-8307 Use test -x to check for ntp-wait Running ntp-wait --help can return 0 or 1 depending on the output of ntpq. For example: $ sudo ntp-wait --help 2> /dev/null; echo $? 0 $ sudo killall ntpd $ sudo ntp-wait --help 2> /dev/null; echo $? 1 This commit instead tests whether ntp-wait exists and is executable to determine if it's installed. Change-Id: I53c63dfa651ac242050171da70540d24c4caf32c Reviewed-on: http://gerrit.cloudera.org:8080/12726 Reviewed-by: Joe McDonnell Tested-by: Impala Public Jenkins --- M testdata/cluster/admin 1 file changed, 10 insertions(+), 7 deletions(-) Approvals: Joe McDonnell: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/12726 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I53c63dfa651ac242050171da70540d24c4caf32c Gerrit-Change-Number: 12726 Gerrit-PatchSet: 3 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell
[Impala-ASF-CR] IMPALA-5051: Add INT64 timestamp write support in Parquet
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12247 ) Change subject: IMPALA-5051: Add INT64 timestamp write support in Parquet .. Patch Set 11: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/2559/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/12247 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib41ad532ec902ed5a9a1528513726eac1c11441f Gerrit-Change-Number: 12247 Gerrit-PatchSet: 11 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: Zoltan Ivanfi Gerrit-Comment-Date: Wed, 27 Mar 2019 15:30:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5051: Add INT64 timestamp write support in Parquet
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12247 ) Change subject: IMPALA-5051: Add INT64 timestamp write support in Parquet .. Patch Set 10: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/2558/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/12247 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib41ad532ec902ed5a9a1528513726eac1c11441f Gerrit-Change-Number: 12247 Gerrit-PatchSet: 10 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: Zoltan Ivanfi Gerrit-Comment-Date: Wed, 27 Mar 2019 15:10:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5051: Add INT64 timestamp write support in Parquet
Csaba Ringhofer has posted comments on this change. (
http://gerrit.cloudera.org:8080/12247 )
Change subject: IMPALA-5051: Add INT64 timestamp write support in Parquet
..
Patch Set 11:
(19 comments)
http://gerrit.cloudera.org:8080/#/c/12247/9//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/12247/9//COMMIT_MSG@17
PS9, Line 17: INT96_NANOS
> I think the NANOS suffix is somewhat misleading as it suggests that this is
I am open to a new name, but I want to consult with Hive / Parquet people
first. This is a development query option at the moment, so I think that it is
not a big issue to change the names in the future.
About discouraging people from using INT96_NANOS: currently this is the only
timestamp representation that can be read by other Hadoop components. I would
wait for a Parquet-MR release with logical type support + some cross-component
testing / benchmarking before advising anyone to use the new types.
http://gerrit.cloudera.org:8080/#/c/12247/9//COMMIT_MSG@42
PS9, Line 42: not
> typo
Done
http://gerrit.cloudera.org:8080/#/c/12247/9/be/src/exec/parquet/parquet-metadata-utils.h
File be/src/exec/parquet/parquet-metadata-utils.h:
http://gerrit.cloudera.org:8080/#/c/12247/9/be/src/exec/parquet/parquet-metadata-utils.h@61
PS9, Line 61: that the inpu
> nit: that the input type..
Done
http://gerrit.cloudera.org:8080/#/c/12247/9/be/src/exec/parquet/parquet-metadata-utils.cc
File be/src/exec/parquet/parquet-metadata-utils.cc:
http://gerrit.cloudera.org:8080/#/c/12247/9/be/src/exec/parquet/parquet-metadata-utils.cc@150
PS9, Line 150: parquet::TimeUnit time_unit;
> nit: declare these before using them?
Done
http://gerrit.cloudera.org:8080/#/c/12247/9/be/src/runtime/timestamp-test.cc
File be/src/runtime/timestamp-test.cc:
http://gerrit.cloudera.org:8080/#/c/12247/9/be/src/runtime/timestamp-test.cc@713
PS9, Line 713: // Test padding on double digits
> I'd declare that when you use it
Done
http://gerrit.cloudera.org:8080/#/c/12247/9/be/src/runtime/timestamp-test.cc@721
PS9, Line 721: ASSERT_TRUE(ParseFormatTokens(_ctx)) << "TC: " << i;
> Maybe prefix each of these comments with a comment or assert of the current
I have rewritten min_date assignments to use explicit timestamps.
http://gerrit.cloudera.org:8080/#/c/12247/9/be/src/runtime/timestamp-test.cc@737
PS9, Line 737: TimestampValue min_date = TimestampValue::Parse("1400-01-01");
> This test does not correspond to the comment above, does it? I.e., it is ro
I think that "floor" in the name should make it clear that it rounds toward -.
Note that ToUnixTime() also rounds towards -.
http://gerrit.cloudera.org:8080/#/c/12247/9/be/src/runtime/timestamp-test.cc@740
PS9, Line 740: EQ(MI
> The switch between us and ns confused me for a bit, not sure if there's a g
I have rewritten the code to always reset min_date. I think that it is easier
to understand now, but I can create new variables if you think that it would
further improve readability.
http://gerrit.cloudera.org:8080/#/c/12247/9/be/src/runtime/timestamp-value.h
File be/src/runtime/timestamp-value.h:
http://gerrit.cloudera.org:8080/#/c/12247/9/be/src/runtime/timestamp-value.h@239
PS9, Line 239: 2262
> The famous Impala Year-2262 problem being conceived and I get to be part of
:D
http://gerrit.cloudera.org:8080/#/c/12247/9/be/src/runtime/timestamp-value.h@375
PS9, Line 375:
> nit: 1970-01-01
Done
http://gerrit.cloudera.org:8080/#/c/12247/9/be/src/runtime/timestamp-value.inline.h
File be/src/runtime/timestamp-value.inline.h:
http://gerrit.cloudera.org:8080/#/c/12247/9/be/src/runtime/timestamp-value.inline.h@98
PS9, Line 98: SECONDS_PER_
> While you're here, this could probably be a constant, too, given it's used
Done
http://gerrit.cloudera.org:8080/#/c/12247/9/be/src/runtime/timestamp-value.inline.h@133
PS9, Line 133: return true;
> Nit: I think some of these newlines could go.
Done
http://gerrit.cloudera.org:8080/#/c/12247/9/be/src/runtime/timestamp-value.inline.h@156
PS9, Line 156:
> nit: indent 4 spaces
Done
http://gerrit.cloudera.org:8080/#/c/12247/9/be/src/runtime/timestamp-value.inline.h@160
PS9, Line 160: }
> nit: If it doesn't fit on one line, it should have braces {}
Done
http://gerrit.cloudera.org:8080/#/c/12247/9/tests/query_test/test_insert_parquet.py
File tests/query_test/test_insert_parquet.py:
http://gerrit.cloudera.org:8080/#/c/12247/9/tests/query_test/test_insert_parquet.py@848
PS9, Line 848: insert_stmt = """insert into {0} values
> Can you add a test for the case that values don't fit into 64bit nanos?
Done
http://gerrit.cloudera.org:8080/#/c/12247/10/tests/query_test/test_insert_parquet.py
File tests/query_test/test_insert_parquet.py:
http://gerrit.cloudera.org:8080/#/c/12247/10/tests/query_test/test_insert_parquet.py@824
PS10, Line 824: ("1969-12-31 23:59:59.9"),
This was broken when rounding towards nearest was
[Impala-ASF-CR] IMPALA-5051: Add INT64 timestamp write support in Parquet
Hello Lars Volker, Zoltan Borok-Nagy, Zoltan Ivanfi, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12247 to look at the new patch set (#11). Change subject: IMPALA-5051: Add INT64 timestamp write support in Parquet .. IMPALA-5051: Add INT64 timestamp write support in Parquet Add query option "parquet_timestamp_type" that chooses the Parquet type used when writing TIMESTAMP columns. This is an experimental feature at the moment, because these types are not widely adopted in other Hadoop components yet. For this reason the query option is added as "development" level, and the default behavior is not changed. The following options can be used: INT96_NANOS (default): This is the same as the old behavior, can represent any timestamp that can be handled by Impala. INT64_MILLIS, INT64_MICROS: Can encode the whole [1400..1) range handled by Impala at the cost of reduced precision. Values are rounded towards minus infinity during writing. INT64_NANOS: Can encode a reduced range without losing nanosecond precision: [1677-09-21 00:12:43.145224192 .. 2262-04-11 23:47:16.854775807] Values outside this range are converted to NULLs without warning. The change was done completely in the backend and all TIMESTAMP columns are written using the type set in the query option. An alternative design would have been to implement some parts in the fronted by adding TIMESTAMP->BIGINT conversion functions to the query plan, which would make it easier to add the possibility of per-column setting in the future. I choose the current design because it seemed much simpler and there are no clear plans for the per-column setting. Most of the code will be still useful if we decide to go the other way in the future. All types are written without conversion to UTC (the way Impala always wrote timestamps), and this information is expressed in the new Parquet logical types by setting isAdjustedToUTC to false. The old logical type (converted_type) is not set, because old readers do not read isAdjustedToUTC, and assume that TIMESTAMP_MILLIS and TIMESTAMP_MICROS are written in UTC. These readers can still read int64 timestamp columns as INT_64. Testing: - added unit tests for new TimestampValue->int64 functions - add EE tests for checking values / min-max stats / metadata written for int64 Parquet timestamps - ran core tests Change-Id: Ib41ad532ec902ed5a9a1528513726eac1c11441f --- M be/src/exec/parquet/hdfs-parquet-table-writer.cc M be/src/exec/parquet/hdfs-parquet-table-writer.h M be/src/exec/parquet/parquet-common.cc M be/src/exec/parquet/parquet-common.h M be/src/exec/parquet/parquet-metadata-utils.cc M be/src/exec/parquet/parquet-metadata-utils.h M be/src/runtime/timestamp-test.cc M be/src/runtime/timestamp-value.h M be/src/runtime/timestamp-value.inline.h M be/src/service/query-options.cc M be/src/service/query-options.h M be/src/util/debug-util.cc M be/src/util/debug-util.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M testdata/workloads/functional-query/queries/QueryTest/parquet-int64-timestamps.test M tests/query_test/test_insert_parquet.py M tests/util/get_parquet_metadata.py 18 files changed, 584 insertions(+), 109 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/47/12247/11 -- To view, visit http://gerrit.cloudera.org:8080/12247 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib41ad532ec902ed5a9a1528513726eac1c11441f Gerrit-Change-Number: 12247 Gerrit-PatchSet: 11 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: Zoltan Ivanfi
[Impala-ASF-CR] IMPALA-5051: Add INT64 timestamp write support in Parquet
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12247 ) Change subject: IMPALA-5051: Add INT64 timestamp write support in Parquet .. Patch Set 10: (1 comment) http://gerrit.cloudera.org:8080/#/c/12247/10/tests/util/get_parquet_metadata.py File tests/util/get_parquet_metadata.py: http://gerrit.cloudera.org:8080/#/c/12247/10/tests/util/get_parquet_metadata.py@180 PS10, Line 180: o flake8: E225 missing whitespace around operator -- To view, visit http://gerrit.cloudera.org:8080/12247 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib41ad532ec902ed5a9a1528513726eac1c11441f Gerrit-Change-Number: 12247 Gerrit-PatchSet: 10 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: Zoltan Ivanfi Gerrit-Comment-Date: Wed, 27 Mar 2019 14:28:08 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5051: Add INT64 timestamp write support in Parquet
Hello Lars Volker, Zoltan Borok-Nagy, Zoltan Ivanfi, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12247 to look at the new patch set (#10). Change subject: IMPALA-5051: Add INT64 timestamp write support in Parquet .. IMPALA-5051: Add INT64 timestamp write support in Parquet Add query option "parquet_timestamp_type" that chooses the Parquet type used when writing TIMESTAMP columns. This is an experimental feature at the moment, because these types are not widely adopted in other Hadoop components yet. For this reason the query option is added as "development" level, and the default behavior is not changed. The following options can be used: INT96_NANOS (default): This is the same as the old behavior, can represent any timestamp that can be handled by Impala. INT64_MILLIS, INT64_MICROS: Can encode the whole [1400..1) range handled by Impala at the cost of reduced precision. Values are rounded towards minus infinity during writing. INT64_NANOS: Can encode a reduced range without losing nanosecond precision: [1677-09-21 00:12:43.145224192 .. 2262-04-11 23:47:16.854775807] Values outside this range are converted to NULLs without warning. The change was done completely in the backend and all TIMESTAMP columns are written using the type set in the query option. An alternative design would have been to implement some parts in the fronted by adding TIMESTAMP->BIGINT conversion functions to the query plan, which would make it easier to add the possibility of per-column setting in the future. I choose the current design because it seemed much simpler and there are no clear plans for the per-column setting. Most of the code will be still useful if we decide to go the other way in the future. All types are written without conversion to UTC (the way Impala always wrote timestamps), and this information is expressed in the new Parquet logical types by setting isAdjustedToUTC to false. The old logical type (converted_type) is not set, because old readers do not read isAdjustedToUTC, and assume that TIMESTAMP_MILLIS and TIMESTAMP_MICROS are written in UTC. These readers can still read int64 timestamp columns as INT_64. Testing: - added unit tests for new TimestampValue->int64 functions - add EE tests for checking values / min-max stats / metadata written for int64 Parquet timestamps - ran core tests Change-Id: Ib41ad532ec902ed5a9a1528513726eac1c11441f --- M be/src/exec/parquet/hdfs-parquet-table-writer.cc M be/src/exec/parquet/hdfs-parquet-table-writer.h M be/src/exec/parquet/parquet-common.cc M be/src/exec/parquet/parquet-common.h M be/src/exec/parquet/parquet-metadata-utils.cc M be/src/exec/parquet/parquet-metadata-utils.h M be/src/runtime/timestamp-test.cc M be/src/runtime/timestamp-value.h M be/src/runtime/timestamp-value.inline.h M be/src/service/query-options.cc M be/src/service/query-options.h M be/src/util/debug-util.cc M be/src/util/debug-util.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M testdata/workloads/functional-query/queries/QueryTest/parquet-int64-timestamps.test M tests/query_test/test_insert_parquet.py M tests/util/get_parquet_metadata.py 18 files changed, 585 insertions(+), 109 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/47/12247/10 -- To view, visit http://gerrit.cloudera.org:8080/12247 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib41ad532ec902ed5a9a1528513726eac1c11441f Gerrit-Change-Number: 12247 Gerrit-PatchSet: 10 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: Zoltan Ivanfi
[Impala-ASF-CR](asf-site) Download 3.2.0
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/12808 ) Change subject: Download 3.2.0 .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12808 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: comment Gerrit-Change-Id: I6b1451baf85a96099116da1a10af8c00a57ee36c Gerrit-Change-Number: 12808 Gerrit-PatchSet: 1 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 27 Mar 2019 10:12:41 + Gerrit-HasComments: No
[Impala-ASF-CR](asf-site) Add 3.2.0 change log
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/12809 ) Change subject: Add 3.2.0 change log .. Patch Set 2: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/12809 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: comment Gerrit-Change-Id: I0592234d9f67677bae0313d65889233e6b9c6210 Gerrit-Change-Number: 12809 Gerrit-PatchSet: 2 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 27 Mar 2019 10:11:24 + Gerrit-HasComments: No
[Impala-ASF-CR](asf-site) Add 3.2.0 change log
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12809 ) Change subject: Add 3.2.0 change log .. Patch Set 2: Verified-1 Build Failed https://jenkins.impala.io/job/gerrit-docs-auto-test/290/ : Doc tests failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/12809 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: comment Gerrit-Change-Id: I0592234d9f67677bae0313d65889233e6b9c6210 Gerrit-Change-Number: 12809 Gerrit-PatchSet: 2 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 27 Mar 2019 09:03:32 + Gerrit-HasComments: No
[Impala-ASF-CR](asf-site) Add 3.2.0 change log
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12809 to look at the new patch set (#2). Change subject: Add 3.2.0 change log .. Add 3.2.0 change log Change-Id: I0592234d9f67677bae0313d65889233e6b9c6210 --- A docs/changelog-3.2.html 1 file changed, 541 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/09/12809/2 -- To view, visit http://gerrit.cloudera.org:8080/12809 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0592234d9f67677bae0313d65889233e6b9c6210 Gerrit-Change-Number: 12809 Gerrit-PatchSet: 2 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR](asf-site) Add 3.2.0 change log
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12809 ) Change subject: Add 3.2.0 change log .. Patch Set 2: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/290/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/12809 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: comment Gerrit-Change-Id: I0592234d9f67677bae0313d65889233e6b9c6210 Gerrit-Change-Number: 12809 Gerrit-PatchSet: 2 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 27 Mar 2019 08:46:41 + Gerrit-HasComments: No
[Impala-ASF-CR](asf-site) [DOCS] Impala 3.2 docs to be published
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/12860 ) Change subject: [DOCS] Impala 3.2 docs to be published .. Patch Set 1: Code-Review+2 I think this should wait until changelog makes it into the repo: https://gerrit.cloudera.org/#/c/12809/ (I have also made these changes on impala-docs.html. Let me remove my changes from that review and we can go with yours here.) -- To view, visit http://gerrit.cloudera.org:8080/12860 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: comment Gerrit-Change-Id: I3564270ab8488cd3482c40070f756e6d4b2a056e Gerrit-Change-Number: 12860 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Jim Apple Gerrit-Comment-Date: Wed, 27 Mar 2019 08:43:41 + Gerrit-HasComments: No
