[Impala-ASF-CR] IMPALA-7633: count user privilege isn't 0 at the end of test owner
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11595 ) Change subject: IMPALA-7633: count_user_privilege isn't 0 at the end of test_owner .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/11595 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifbba0fbd0e24a24b3f2af82ad5209f3fb7fb387b Gerrit-Change-Number: 11595 Gerrit-PatchSet: 1 Gerrit-Owner: Adam Holley Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 05 Oct 2018 20:43:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7633: count user privilege isn't 0 at the end of test owner
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11595 ) Change subject: IMPALA-7633: count_user_privilege isn't 0 at the end of test_owner .. IMPALA-7633: count_user_privilege isn't 0 at the end of test_owner This patch adds a retry loop to validate the count of user privileges in a SHOW GRANT USER statement after a DDL operation. The core of the problem is cache consistency. When a DDL operation is executing, like drop database, HMS is updated with the correct metadata, and Sentry is updated to remove privileges from HMS. However, if a Sentry Refresh happens between when HMS is updated CatalogOpExecutor:1322, and when the local catalog privileges are updated CatalogOpExecutor:1341, then the remove privilege call will fail and a log entry with "User does not exist: foo_user" will be written to the log. The result is that the response back to impalad with catalog updates will not contain the user and privilege updates. Ultimately, when the "SHOW GRANT USER" statement is run, it uses the local Impalad catalog which still contains the privlege because it has not yet been updated from statestore. This is not a security problem because the privilege exists for a maximum of 2s by default, for an object that does not exist. This is the same result as if the database was dropped from Hive, except in that case it can be up to 62s by default that the privilege exists for no object. Testing: - After retry was added, ran tests until log entry appeared and validate test did not fail. Change-Id: Ifbba0fbd0e24a24b3f2af82ad5209f3fb7fb387b Reviewed-on: http://gerrit.cloudera.org:8080/11595 Reviewed-by: Fredy Wijaya Reviewed-by: Vuk Ercegovac Tested-by: Impala Public Jenkins --- M tests/authorization/test_owner_privileges.py 1 file changed, 23 insertions(+), 18 deletions(-) Approvals: Fredy Wijaya: Looks good to me, but someone else must approve Vuk Ercegovac: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/11595 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ifbba0fbd0e24a24b3f2af82ad5209f3fb7fb387b Gerrit-Change-Number: 11595 Gerrit-PatchSet: 2 Gerrit-Owner: Adam Holley Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7633: count user privilege isn't 0 at the end of test owner
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11595 ) Change subject: IMPALA-7633: count_user_privilege isn't 0 at the end of test_owner .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3281/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/11595 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifbba0fbd0e24a24b3f2af82ad5209f3fb7fb387b Gerrit-Change-Number: 11595 Gerrit-PatchSet: 1 Gerrit-Owner: Adam Holley Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 05 Oct 2018 17:05:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7633: count user privilege isn't 0 at the end of test owner
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/11595 ) Change subject: IMPALA-7633: count_user_privilege isn't 0 at the end of test_owner .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11595 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifbba0fbd0e24a24b3f2af82ad5209f3fb7fb387b Gerrit-Change-Number: 11595 Gerrit-PatchSet: 1 Gerrit-Owner: Adam Holley Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 05 Oct 2018 17:00:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7633: count user privilege isn't 0 at the end of test owner
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/11595 ) Change subject: IMPALA-7633: count_user_privilege isn't 0 at the end of test_owner .. Patch Set 1: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/11595 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifbba0fbd0e24a24b3f2af82ad5209f3fb7fb387b Gerrit-Change-Number: 11595 Gerrit-PatchSet: 1 Gerrit-Owner: Adam Holley Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 05 Oct 2018 16:10:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7633: count user privilege isn't 0 at the end of test owner
Adam Holley has posted comments on this change. ( http://gerrit.cloudera.org:8080/11595 ) Change subject: IMPALA-7633: count_user_privilege isn't 0 at the end of test_owner .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/11595/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11595/1//COMMIT_MSG@29 PS1, Line 29: - After retry was added, ran tests until log entry appeared and > I re-ran the cluster tests until the error appeared in the log file. Appro Just to add. increasing the statestore update time will not make the problem more frequent as it relies on a sentry refresh between 1322 and 1341 of CatalogOpExecutor. Increasing the statestore time will just allow a longer window to see the problem when it occurs. -- To view, visit http://gerrit.cloudera.org:8080/11595 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifbba0fbd0e24a24b3f2af82ad5209f3fb7fb387b Gerrit-Change-Number: 11595 Gerrit-PatchSet: 1 Gerrit-Owner: Adam Holley Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 05 Oct 2018 16:02:35 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7633: count user privilege isn't 0 at the end of test owner
Adam Holley has posted comments on this change. ( http://gerrit.cloudera.org:8080/11595 ) Change subject: IMPALA-7633: count_user_privilege isn't 0 at the end of test_owner .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/11595/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11595/1//COMMIT_MSG@29 PS1, Line 29: - After retry was added, ran tests until log entry appeared and > How did you reproduce the issue? Did you increase the statestore update tim I re-ran the cluster tests until the error appeared in the log file. Approximately 12 times. -- To view, visit http://gerrit.cloudera.org:8080/11595 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifbba0fbd0e24a24b3f2af82ad5209f3fb7fb387b Gerrit-Change-Number: 11595 Gerrit-PatchSet: 1 Gerrit-Owner: Adam Holley Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 05 Oct 2018 15:47:08 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7633: count user privilege isn't 0 at the end of test owner
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11595 ) Change subject: IMPALA-7633: count_user_privilege isn't 0 at the end of test_owner .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/961/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11595 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifbba0fbd0e24a24b3f2af82ad5209f3fb7fb387b Gerrit-Change-Number: 11595 Gerrit-PatchSet: 1 Gerrit-Owner: Adam Holley Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 05 Oct 2018 14:44:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7633: count user privilege isn't 0 at the end of test owner
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/11595 ) Change subject: IMPALA-7633: count_user_privilege isn't 0 at the end of test_owner .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/11595/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11595/1//COMMIT_MSG@29 PS1, Line 29: - After retry was added, ran tests until log entry appeared and How did you reproduce the issue? Did you increase the statestore update time to make the problem more frequent? -- To view, visit http://gerrit.cloudera.org:8080/11595 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifbba0fbd0e24a24b3f2af82ad5209f3fb7fb387b Gerrit-Change-Number: 11595 Gerrit-PatchSet: 1 Gerrit-Owner: Adam Holley Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 05 Oct 2018 14:46:52 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7633: count user privilege isn't 0 at the end of test owner
Adam Holley has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11595 Change subject: IMPALA-7633: count_user_privilege isn't 0 at the end of test_owner .. IMPALA-7633: count_user_privilege isn't 0 at the end of test_owner This patch adds a retry loop to validate the count of user privileges in a SHOW GRANT USER statement after a DDL operation. The core of the problem is cache consistency. When a DDL operation is executing, like drop database, HMS is updated with the correct metadata, and Sentry is updated to remove privileges from HMS. However, if a Sentry Refresh happens between when HMS is updated CatalogOpExecutor:1322, and when the local catalog privileges are updated CatalogOpExecutor:1341, then the remove privilege call will fail and a log entry with "User does not exist: foo_user" will be written to the log. The result is that the response back to impalad with catalog updates will not contain the user and privilege updates. Ultimately, when the "SHOW GRANT USER" statement is run, it uses the local Impalad catalog which still contains the privlege because it has not yet been updated from statestore. This is not a security problem because the privilege exists for a maximum of 2s by default, for an object that does not exist. This is the same result as if the database was dropped from Hive, except in that case it can be up to 62s by default that the privilege exists for no object. Testing: - After retry was added, ran tests until log entry appeared and validate test did not fail. Change-Id: Ifbba0fbd0e24a24b3f2af82ad5209f3fb7fb387b --- M tests/authorization/test_owner_privileges.py 1 file changed, 23 insertions(+), 18 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/95/11595/1 -- To view, visit http://gerrit.cloudera.org:8080/11595 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ifbba0fbd0e24a24b3f2af82ad5209f3fb7fb387b Gerrit-Change-Number: 11595 Gerrit-PatchSet: 1 Gerrit-Owner: Adam Holley
[Impala-ASF-CR] IMPALA-7633: count user privilege isn't 0 at the end of test owner
Adam Holley has removed Vuk Ercegovac from this change. ( http://gerrit.cloudera.org:8080/11595 ) Change subject: IMPALA-7633: count_user_privilege isn't 0 at the end of test_owner .. Removed reviewer Vuk Ercegovac. -- To view, visit http://gerrit.cloudera.org:8080/11595 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: deleteReviewer Gerrit-Change-Id: Ifbba0fbd0e24a24b3f2af82ad5209f3fb7fb387b Gerrit-Change-Number: 11595 Gerrit-PatchSet: 1 Gerrit-Owner: Adam Holley Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins
