[Impala-ASF-CR] IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11897 ) Change subject: IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable .. Patch Set 10: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/3439/ -- To view, visit http://gerrit.cloudera.org:8080/11897 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icff987a6184f62a338faadfdc1a0d349d912fc37 Gerrit-Change-Number: 11897 Gerrit-PatchSet: 10 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 08 Nov 2018 01:00:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable
Fredy Wijaya has uploaded a new patch set (#11). ( http://gerrit.cloudera.org:8080/11897 ) Change subject: IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable .. IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable Before this patch, running INVALIDATE METADATA when Sentry is unavailable could cause Impala query to hang. PolicyReader thread in SentryProxy is used by two use cases, one as a background thread that periodically refreshes Sentry policy and another one as a synchronous operation for INVALIDATE METADATA. For the background thread, we need to swallow any exception thrown while refreshing the Sentry policy in order to not kill the background thread. For a synchronous reset operation, such as INVALIDATE METADATA, swallowing an exception causes the Impala catalog to wait indefinitely for authorization catalog objects that never get processed due to Sentry being unavailable. The patch updates the code by not swallowing any exception in INVALIDATE METADATA and return the exception to the caller. Testing: - Ran all FE tests - Added a new E2E test - Ran all E2E authorization tests Change-Id: Icff987a6184f62a338faadfdc1a0d349d912fc37 --- A fe/src/main/java/org/apache/impala/common/SentryPolicyReaderException.java A fe/src/main/java/org/apache/impala/common/SentryUnavailableException.java M fe/src/main/java/org/apache/impala/util/SentryProxy.java M fe/src/test/java/org/apache/impala/testutil/ImpaladTestCatalog.java M tests/authorization/test_authorization.py M tests/common/custom_cluster_test_suite.py 6 files changed, 147 insertions(+), 14 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/97/11897/11 -- To view, visit http://gerrit.cloudera.org:8080/11897 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Icff987a6184f62a338faadfdc1a0d349d912fc37 Gerrit-Change-Number: 11897 Gerrit-PatchSet: 11 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/11897 ) Change subject: IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable .. Patch Set 11: Code-Review+2 So apparently ImpalaTestCatalog.java has a bug where FrontendTestBase always creates a CatalogServiceCatalog with an authorization even though authorization is disabled. We never knew about this because the SentryProxy.reset() code used to swallow the exception. Since this is a trivial fix that only affects the tests and not actual code, I'm giving it a +2. -- To view, visit http://gerrit.cloudera.org:8080/11897 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icff987a6184f62a338faadfdc1a0d349d912fc37 Gerrit-Change-Number: 11897 Gerrit-PatchSet: 11 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 08 Nov 2018 03:03:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11897 ) Change subject: IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable .. Patch Set 12: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3444/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/11897 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icff987a6184f62a338faadfdc1a0d349d912fc37 Gerrit-Change-Number: 11897 Gerrit-PatchSet: 12 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 08 Nov 2018 03:03:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11897 ) Change subject: IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable .. Patch Set 11: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1326/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11897 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icff987a6184f62a338faadfdc1a0d349d912fc37 Gerrit-Change-Number: 11897 Gerrit-PatchSet: 11 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 08 Nov 2018 03:36:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11897 ) Change subject: IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable .. Patch Set 12: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11897 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icff987a6184f62a338faadfdc1a0d349d912fc37 Gerrit-Change-Number: 11897 Gerrit-PatchSet: 12 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 08 Nov 2018 03:03:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11897 ) Change subject: IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable .. IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable Before this patch, running INVALIDATE METADATA when Sentry is unavailable could cause Impala query to hang. PolicyReader thread in SentryProxy is used by two use cases, one as a background thread that periodically refreshes Sentry policy and another one as a synchronous operation for INVALIDATE METADATA. For the background thread, we need to swallow any exception thrown while refreshing the Sentry policy in order to not kill the background thread. For a synchronous reset operation, such as INVALIDATE METADATA, swallowing an exception causes the Impala catalog to wait indefinitely for authorization catalog objects that never get processed due to Sentry being unavailable. The patch updates the code by not swallowing any exception in INVALIDATE METADATA and return the exception to the caller. Testing: - Ran all FE tests - Added a new E2E test - Ran all E2E authorization tests Change-Id: Icff987a6184f62a338faadfdc1a0d349d912fc37 Reviewed-on: http://gerrit.cloudera.org:8080/11897 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- A fe/src/main/java/org/apache/impala/common/SentryPolicyReaderException.java A fe/src/main/java/org/apache/impala/common/SentryUnavailableException.java M fe/src/main/java/org/apache/impala/util/SentryProxy.java M fe/src/test/java/org/apache/impala/testutil/ImpaladTestCatalog.java M tests/authorization/test_authorization.py M tests/common/custom_cluster_test_suite.py 6 files changed, 147 insertions(+), 14 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/11897 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Icff987a6184f62a338faadfdc1a0d349d912fc37 Gerrit-Change-Number: 11897 Gerrit-PatchSet: 13 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11897 ) Change subject: IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable .. Patch Set 12: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/11897 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icff987a6184f62a338faadfdc1a0d349d912fc37 Gerrit-Change-Number: 11897 Gerrit-PatchSet: 12 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 08 Nov 2018 07:05:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11897 ) Change subject: IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1315/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11897 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icff987a6184f62a338faadfdc1a0d349d912fc37 Gerrit-Change-Number: 11897 Gerrit-PatchSet: 9 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 07 Nov 2018 21:23:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/11897 ) Change subject: IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable .. Patch Set 10: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/11897 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icff987a6184f62a338faadfdc1a0d349d912fc37 Gerrit-Change-Number: 11897 Gerrit-PatchSet: 10 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 07 Nov 2018 21:07:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11897 ) Change subject: IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable .. Patch Set 10: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3439/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/11897 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icff987a6184f62a338faadfdc1a0d349d912fc37 Gerrit-Change-Number: 11897 Gerrit-PatchSet: 10 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 07 Nov 2018 21:02:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11897 ) Change subject: IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable .. Patch Set 10: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11897 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icff987a6184f62a338faadfdc1a0d349d912fc37 Gerrit-Change-Number: 11897 Gerrit-PatchSet: 10 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 07 Nov 2018 21:02:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/11897 ) Change subject: IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable .. Patch Set 9: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11897 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icff987a6184f62a338faadfdc1a0d349d912fc37 Gerrit-Change-Number: 11897 Gerrit-PatchSet: 9 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 07 Nov 2018 20:59:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable
Fredy Wijaya has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/11897 ) Change subject: IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable .. IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable Before this patch, running INVALIDATE METADATA when Sentry is unavailable could cause Impala query to hang. PolicyReader thread in SentryProxy is used by two use cases, one as a background thread that periodically refreshes Sentry policy and another one as a synchronous operation for INVALIDATE METADATA. For the background thread, we need to swallow any exception thrown while refreshing the Sentry policy in order to not kill the background thread. For a synchronous reset operation, such as INVALIDATE METADATA, swallowing an exception causes the Impala catalog to wait indefinitely for authorization catalog objects that never get processed due to Sentry being unavailable. The patch updates the code by not swallowing any exception in INVALIDATE METADATA and return the exception to the caller. Testing: - Added a new E2E test - Ran all E2E authorization tests Change-Id: Icff987a6184f62a338faadfdc1a0d349d912fc37 --- A fe/src/main/java/org/apache/impala/common/SentryPolicyReaderException.java A fe/src/main/java/org/apache/impala/common/SentryUnavailableException.java M fe/src/main/java/org/apache/impala/util/SentryProxy.java M tests/authorization/test_authorization.py M tests/common/custom_cluster_test_suite.py 5 files changed, 144 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/97/11897/9 -- To view, visit http://gerrit.cloudera.org:8080/11897 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Icff987a6184f62a338faadfdc1a0d349d912fc37 Gerrit-Change-Number: 11897 Gerrit-PatchSet: 9 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable
Fredy Wijaya has posted comments on this change. (
http://gerrit.cloudera.org:8080/11897 )
Change subject: IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is
unavailable
..
Patch Set 9:
(10 comments)
http://gerrit.cloudera.org:8080/#/c/11897/5//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/11897/5//COMMIT_MSG@10
PS5, Line 10: mpala
> query you mean?
Done
http://gerrit.cloudera.org:8080/#/c/11897/5//COMMIT_MSG@10
PS5, Line 10: cause
> typo
Done
http://gerrit.cloudera.org:8080/#/c/11897/5//COMMIT_MSG@17
PS5, Line 17: cause
> causes?
Done
http://gerrit.cloudera.org:8080/#/c/11897/5/fe/src/main/java/org/apache/impala/util/SentryProxy.java
File fe/src/main/java/org/apache/impala/util/SentryProxy.java:
http://gerrit.cloudera.org:8080/#/c/11897/5/fe/src/main/java/org/apache/impala/util/SentryProxy.java@131
PS5, Line 131:* atomically.
> Add a one-liner documentation here?
Done
http://gerrit.cloudera.org:8080/#/c/11897/6/fe/src/main/java/org/apache/impala/util/SentryProxy.java
File fe/src/main/java/org/apache/impala/util/SentryProxy.java:
http://gerrit.cloudera.org:8080/#/c/11897/6/fe/src/main/java/org/apache/impala/util/SentryProxy.java@547
PS6, Line 547:
: /**
> this needs to be updated since we now expect an exception
Done
http://gerrit.cloudera.org:8080/#/c/11897/6/fe/src/main/java/org/apache/impala/util/SentryProxy.java@550
PS6, Line 550: ntimeException if there are a
> can this message be improved, for example, by distinguishing between sentry
Done. I added created two different exceptions.
http://gerrit.cloudera.org:8080/#/c/11897/5/tests/authorization/test_authorization.py
File tests/authorization/test_authorization.py:
http://gerrit.cloudera.org:8080/#/c/11897/5/tests/authorization/test_authorization.py@487
PS5, Line 487:
impala_log_dir=tempfile.mkdtemp(prefix="test_invalidate_metadata_sentry_unavailable_",
: dir=os.getenv("LOG_DIR")))
> whats the use of this?
It just makes debugging easier in case of a flaky test.
http://gerrit.cloudera.org:8080/#/c/11897/5/tests/authorization/test_authorization.py@501
PS5, Line 501: result = self.execute_query_expect_failure(self.client,
"invalidate metadata")
> can we assert the exception?
Done
http://gerrit.cloudera.org:8080/#/c/11897/6/tests/authorization/test_authorization.py
File tests/authorization/test_authorization.py:
http://gerrit.cloudera.org:8080/#/c/11897/6/tests/authorization/test_authorization.py@504
PS6, Line 504:
> this sticks out now since the user will end up doing the retries. if they a
Yeah makes sense.
http://gerrit.cloudera.org:8080/#/c/11897/5/tests/common/custom_cluster_test_suite.py
File tests/common/custom_cluster_test_suite.py:
http://gerrit.cloudera.org:8080/#/c/11897/5/tests/common/custom_cluster_test_suite.py@177
PS5, Line 177: close_fds=True
> Curious what is the significance of this. Doesn't the process kill take car
It's probably to avoid any zombie processes. It seems it's also used in other
places, e.g.
https://github.com/apache/impala/blob/b84075f7f0067c54a340872e5ec1814398310318/tests/catalog_service/test_hms_failure.py#L72
--
To view, visit http://gerrit.cloudera.org:8080/11897
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icff987a6184f62a338faadfdc1a0d349d912fc37
Gerrit-Change-Number: 11897
Gerrit-PatchSet: 9
Gerrit-Owner: Fredy Wijaya
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Vuk Ercegovac
Gerrit-Comment-Date: Wed, 07 Nov 2018 20:45:35 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable
Bharath Vissapragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/11897 )
Change subject: IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is
unavailable
..
Patch Set 5:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/11897/5//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/11897/5//COMMIT_MSG@10
PS5, Line 10: Impala
query you mean?
http://gerrit.cloudera.org:8080/#/c/11897/5//COMMIT_MSG@10
PS5, Line 10: causue
typo
http://gerrit.cloudera.org:8080/#/c/11897/5//COMMIT_MSG@17
PS5, Line 17: cause
causes?
http://gerrit.cloudera.org:8080/#/c/11897/5/fe/src/main/java/org/apache/impala/util/SentryProxy.java
File fe/src/main/java/org/apache/impala/util/SentryProxy.java:
http://gerrit.cloudera.org:8080/#/c/11897/5/fe/src/main/java/org/apache/impala/util/SentryProxy.java@131
PS5, Line 131: private final boolean swallowException_;
Add a one-liner documentation here?
http://gerrit.cloudera.org:8080/#/c/11897/5/tests/authorization/test_authorization.py
File tests/authorization/test_authorization.py:
http://gerrit.cloudera.org:8080/#/c/11897/5/tests/authorization/test_authorization.py@487
PS5, Line 487:
impala_log_dir=tempfile.mkdtemp(prefix="test_invalidate_metadata_sentry_unavailable_",
: dir=os.getenv("LOG_DIR")))
whats the use of this?
http://gerrit.cloudera.org:8080/#/c/11897/5/tests/authorization/test_authorization.py@501
PS5, Line 501: self.execute_query_expect_failure(self.client, "invalidate
metadata")
can we assert the exception?
http://gerrit.cloudera.org:8080/#/c/11897/5/tests/common/custom_cluster_test_suite.py
File tests/common/custom_cluster_test_suite.py:
http://gerrit.cloudera.org:8080/#/c/11897/5/tests/common/custom_cluster_test_suite.py@177
PS5, Line 177: close_fds=True
Curious what is the significance of this. Doesn't the process kill take care of
it?
--
To view, visit http://gerrit.cloudera.org:8080/11897
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icff987a6184f62a338faadfdc1a0d349d912fc37
Gerrit-Change-Number: 11897
Gerrit-PatchSet: 5
Gerrit-Owner: Fredy Wijaya
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Vuk Ercegovac
Gerrit-Comment-Date: Wed, 07 Nov 2018 17:56:23 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11897 ) Change subject: IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1306/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11897 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icff987a6184f62a338faadfdc1a0d349d912fc37 Gerrit-Change-Number: 11897 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 07 Nov 2018 04:59:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11897 ) Change subject: IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1305/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11897 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icff987a6184f62a338faadfdc1a0d349d912fc37 Gerrit-Change-Number: 11897 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 07 Nov 2018 04:55:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable
Fredy Wijaya has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/11897 ) Change subject: IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable .. IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable Before this patch, running INVALIDATE METADATA when Sentry is unavailable could causue Impala to hang. PolicyReader thread in SentryProxy is used by two use cases, one as a background thread that periodically refreshes Sentry policy and another one as a synchronous operation for INVALIDATE METADATA. For the background thread, we need to swallow any exception thrown while refreshing the Sentry policy in order to not kill the background thread. For a synchronous reset operation, such as INVALIDATE METADATA, swallowing an exception cause the Impala catalog to wait indefinitely for authorization catalog objects that never get processed due to Sentry being unavailable. The patch updates the code by not swallowing any exception in INVALIDATE METADATA and return the exception to the caller. Testing: - Added a new E2E test - Ran all E2E authorization tests Change-Id: Icff987a6184f62a338faadfdc1a0d349d912fc37 --- M fe/src/main/java/org/apache/impala/util/SentryProxy.java M tests/authorization/test_authorization.py M tests/common/custom_cluster_test_suite.py 3 files changed, 51 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/97/11897/5 -- To view, visit http://gerrit.cloudera.org:8080/11897 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Icff987a6184f62a338faadfdc1a0d349d912fc37 Gerrit-Change-Number: 11897 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/11897 ) Change subject: IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/11897/4/tests/authorization/test_authorization.py File tests/authorization/test_authorization.py: http://gerrit.cloudera.org:8080/#/c/11897/4/tests/authorization/test_authorization.py@500 PS4, Line 500: # > flake8: E303 too many blank lines (2) Done -- To view, visit http://gerrit.cloudera.org:8080/11897 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icff987a6184f62a338faadfdc1a0d349d912fc37 Gerrit-Change-Number: 11897 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 07 Nov 2018 04:19:37 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11897 ) Change subject: IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/11897/4/tests/authorization/test_authorization.py File tests/authorization/test_authorization.py: http://gerrit.cloudera.org:8080/#/c/11897/4/tests/authorization/test_authorization.py@500 PS4, Line 500: s flake8: E303 too many blank lines (2) -- To view, visit http://gerrit.cloudera.org:8080/11897 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icff987a6184f62a338faadfdc1a0d349d912fc37 Gerrit-Change-Number: 11897 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 07 Nov 2018 04:18:30 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable
Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11897 Change subject: IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable .. IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable Before this patch, running INVALIDATE METADATA when Sentry is unavailable could causue Impala to hang. PolicyReader thread in SentryProxy is used by two use cases, one as a background thread that periodically refreshes Sentry policy and another one as a synchronous operation for INVALIDATE METADATA. For the background thread, we need to swallow any exception thrown while refreshing the Sentry policy in order to not kill the background thread. For a synchronous reset operation, such as INVALIDATE METADATA, swallowing an exception cause the Impala catalog to wait indefinitely for authorization catalog objects that never get processed due to Sentry being unavailable. The patch updates the code by not swallowing any exception in INVALIDATE METADATA and return the exception to the caller. Testing: - Added a new E2E test - Ran all E2E authorization tests Change-Id: Icff987a6184f62a338faadfdc1a0d349d912fc37 --- M fe/src/main/java/org/apache/impala/util/SentryProxy.java M tests/authorization/test_authorization.py M tests/common/custom_cluster_test_suite.py 3 files changed, 52 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/97/11897/4 -- To view, visit http://gerrit.cloudera.org:8080/11897 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Icff987a6184f62a338faadfdc1a0d349d912fc37 Gerrit-Change-Number: 11897 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya
