Hello Anurag Mantripragada, Fang-Yu Rao, Vihang Karajgaonkar, Csaba Ringhofer,
Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15235
to look at the new patch set (#4).
Change subject: IMPALA-9152: Make Impala ranger plugin singleton in FE Tests
......................................................................
IMPALA-9152: Make Impala ranger plugin singleton in FE Tests
Fix the flakiness of Ranger FE tests in AuthorizationStmtTest which is
caused by a row filter policy not being cleanly deleted. There is a bug
in Ranger that policies being deleted in Ranger Admin server but still
exist in Ranger plugins when there are concurrent create policy and get
policy requests (RANGER-2727). It's more possible to hit the bug if we
have more ranger plugins running, since each plugin instance will poll
policies in each 30s regularly.
Impalad and Catalogd servers only initialize one ImpalaRangerPlugin
instance. However, AuthorizationStmtTest has embedded Frontend and
CatalogServiceCatalog objects. It will initialize two ranger plugin
instances totally. What's worse, the JUnit testing framework makes a new
object for each test method run. Currently there are 29 test methods in
AuthorizationStmtTest, which means 29 AuthorizationStmtTest objects will
be created. So we finally have 58 ranger plugin instances running, which
makes RANGER-2727 easy to happen.
The failure can be reproduced by adding the following new test and run
it with all existing tests:
@Test
public void testRangerPolicyRepeatedly() throws Exception {
if (authzProvider_ == AuthorizationProvider.SENTRY) return;
for (int i = 0; i < 100; ++i) {
testRowFilterEnabled();
testColumnMaskEnabled();
}
}
We only explicitly create policies for column masking and row filtering
(other tests are using grant/revoke requests). This test increases the
number of CreatePolicy requests, so increases the possibility of
CreatePolicy requests running concurrently with GetPolicies requests
polling from other ranger plugin instances created by previous tests.
The fix is to make ImpalaRangerPlugin a singleton class so we will have
only one ranger plugin instance, which dramatically reduces the
possibility of hitting RANGER-2727. The thorough fix is bumping CDP
version after RANGER-2727 is resolved. Codes added in the previous patch
(c1244c2f04e629cc07b0830a597c70317be92768) are removed.
Tests:
- Ran AuthorizationStmtTest with the above new test.
Change-Id: I91f2bad1a9ce897b45cfc42f97b192fe2f8c7e06
---
M
fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java
M
fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java
M
fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpalaPlugin.java
M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java
M fe/src/test/java/org/apache/impala/authorization/AuthorizationTestBase.java
5 files changed, 39 insertions(+), 37 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/35/15235/4
--
To view, visit http://gerrit.cloudera.org:8080/15235
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I91f2bad1a9ce897b45cfc42f97b192fe2f8c7e06
Gerrit-Change-Number: 15235
Gerrit-PatchSet: 4
Gerrit-Owner: Quanlong Huang <[email protected]>
Gerrit-Reviewer: Anurag Mantripragada <[email protected]>
Gerrit-Reviewer: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Fang-Yu Rao <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Quanlong Huang <[email protected]>
Gerrit-Reviewer: Vihang Karajgaonkar <[email protected]>
