[jira] [Commented] (HBASE-11039) [VisibilityController] Integration test for labeled data set mixing and filtered excise
[ https://issues.apache.org/jira/browse/HBASE-11039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14060376#comment-14060376 ] ramkrishna.s.vasudevan commented on HBASE-11039: Opened https://issues.apache.org/jira/browse/HBASE-11509 [VisibilityController] Integration test for labeled data set mixing and filtered excise --- Key: HBASE-11039 URL: https://issues.apache.org/jira/browse/HBASE-11039 Project: HBase Issue Type: Test Affects Versions: 0.98.1 Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Priority: Critical Fix For: 0.98.4 Attachments: HBASE-11039_ITBLL_v1.patch, HBASE-11039_v3_0.98.patch, HBASE-11039_v4_0.98.patch, HBASE-11039_v5_0.98.patch Create an integration test for the VisibilityController that: 1. Create several tables of test data 2. Assign a set of auths to each table. Label all entries in the table with appropriate visibility expressions. Insure that some data in every table overlaps with data in other tables at common row/family/qualifier coordinates. Generate data like ITBLL so we can verify all data present later. 3. Mix the data from the different tables into a new common table 4. Verify for each set of auths defined in step #2 that all entries found in the source table can be found in the common table. Like the ITBLL verification step but done N times for each set of auths defined in step #2. 5. Choose one of the source tables. Get its set of auths. Perform a deletion with visibility expression from the common table using those auths. 6. Verify that no data in the common table with the auth set chosen in #5 remains. A simple row count with the set of auths chosen in #5 that should return 0. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11039) [VisibilityController] Integration test for labeled data set mixing and filtered excise
[ https://issues.apache.org/jira/browse/HBASE-11039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14060490#comment-14060490 ] Hudson commented on HBASE-11039: FAILURE: Integrated in HBase-0.98 #394 (See [https://builds.apache.org/job/HBase-0.98/394/]) HBASE-11039 [VisibilityController] Integration test for labeled data set (ramkrishna: rev 0c2ef25ed8be41a0bc5145d5b244457866e659a0) * hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedListWithVisibility.java * hbase-client/src/main/java/org/apache/hadoop/hbase/security/visibility/CellVisibility.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java * hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedList.java * hbase-it/src/test/java/org/apache/hadoop/hbase/IntegrationTestBase.java * hbase-client/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlClient.java [VisibilityController] Integration test for labeled data set mixing and filtered excise --- Key: HBASE-11039 URL: https://issues.apache.org/jira/browse/HBASE-11039 Project: HBase Issue Type: Test Affects Versions: 0.98.1 Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Priority: Critical Fix For: 0.98.4 Attachments: HBASE-11039_ITBLL_v1.patch, HBASE-11039_v3_0.98.patch, HBASE-11039_v4_0.98.patch, HBASE-11039_v5_0.98.patch Create an integration test for the VisibilityController that: 1. Create several tables of test data 2. Assign a set of auths to each table. Label all entries in the table with appropriate visibility expressions. Insure that some data in every table overlaps with data in other tables at common row/family/qualifier coordinates. Generate data like ITBLL so we can verify all data present later. 3. Mix the data from the different tables into a new common table 4. Verify for each set of auths defined in step #2 that all entries found in the source table can be found in the common table. Like the ITBLL verification step but done N times for each set of auths defined in step #2. 5. Choose one of the source tables. Get its set of auths. Perform a deletion with visibility expression from the common table using those auths. 6. Verify that no data in the common table with the auth set chosen in #5 remains. A simple row count with the set of auths chosen in #5 that should return 0. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11039) [VisibilityController] Integration test for labeled data set mixing and filtered excise
[ https://issues.apache.org/jira/browse/HBASE-11039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14060531#comment-14060531 ] Hudson commented on HBASE-11039: SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #374 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/374/]) HBASE-11039 [VisibilityController] Integration test for labeled data set (ramkrishna: rev 0c2ef25ed8be41a0bc5145d5b244457866e659a0) * hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedListWithVisibility.java * hbase-client/src/main/java/org/apache/hadoop/hbase/security/visibility/CellVisibility.java * hbase-client/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlClient.java * hbase-it/src/test/java/org/apache/hadoop/hbase/IntegrationTestBase.java * hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedList.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java [VisibilityController] Integration test for labeled data set mixing and filtered excise --- Key: HBASE-11039 URL: https://issues.apache.org/jira/browse/HBASE-11039 Project: HBase Issue Type: Test Affects Versions: 0.98.1 Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Priority: Critical Fix For: 0.98.4 Attachments: HBASE-11039_ITBLL_v1.patch, HBASE-11039_v3_0.98.patch, HBASE-11039_v4_0.98.patch, HBASE-11039_v5_0.98.patch Create an integration test for the VisibilityController that: 1. Create several tables of test data 2. Assign a set of auths to each table. Label all entries in the table with appropriate visibility expressions. Insure that some data in every table overlaps with data in other tables at common row/family/qualifier coordinates. Generate data like ITBLL so we can verify all data present later. 3. Mix the data from the different tables into a new common table 4. Verify for each set of auths defined in step #2 that all entries found in the source table can be found in the common table. Like the ITBLL verification step but done N times for each set of auths defined in step #2. 5. Choose one of the source tables. Get its set of auths. Perform a deletion with visibility expression from the common table using those auths. 6. Verify that no data in the common table with the auth set chosen in #5 remains. A simple row count with the set of auths chosen in #5 that should return 0. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11039) [VisibilityController] Integration test for labeled data set mixing and filtered excise
[ https://issues.apache.org/jira/browse/HBASE-11039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14060856#comment-14060856 ] Andrew Purtell commented on HBASE-11039: bq. Committed to 0.98. Will commit to trunk and 1.0 after HBASE-11489 is resolved. Will open a follow up JIRA for the same. Please consider committing these changes now, to minimize code deltas between the branches. [VisibilityController] Integration test for labeled data set mixing and filtered excise --- Key: HBASE-11039 URL: https://issues.apache.org/jira/browse/HBASE-11039 Project: HBase Issue Type: Test Affects Versions: 0.98.1 Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Priority: Critical Fix For: 0.98.4 Attachments: HBASE-11039_ITBLL_v1.patch, HBASE-11039_v3_0.98.patch, HBASE-11039_v4_0.98.patch, HBASE-11039_v5_0.98.patch Create an integration test for the VisibilityController that: 1. Create several tables of test data 2. Assign a set of auths to each table. Label all entries in the table with appropriate visibility expressions. Insure that some data in every table overlaps with data in other tables at common row/family/qualifier coordinates. Generate data like ITBLL so we can verify all data present later. 3. Mix the data from the different tables into a new common table 4. Verify for each set of auths defined in step #2 that all entries found in the source table can be found in the common table. Like the ITBLL verification step but done N times for each set of auths defined in step #2. 5. Choose one of the source tables. Get its set of auths. Perform a deletion with visibility expression from the common table using those auths. 6. Verify that no data in the common table with the auth set chosen in #5 remains. A simple row count with the set of auths chosen in #5 that should return 0. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11039) [VisibilityController] Integration test for labeled data set mixing and filtered excise
[ https://issues.apache.org/jira/browse/HBASE-11039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14060895#comment-14060895 ] ramkrishna.s.vasudevan commented on HBASE-11039: [~apurtell] Committed to trunk and branch-1 as part of HBASE-11509. [VisibilityController] Integration test for labeled data set mixing and filtered excise --- Key: HBASE-11039 URL: https://issues.apache.org/jira/browse/HBASE-11039 Project: HBase Issue Type: Test Affects Versions: 0.98.1 Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Priority: Critical Fix For: 0.98.4 Attachments: HBASE-11039_ITBLL_v1.patch, HBASE-11039_v3_0.98.patch, HBASE-11039_v4_0.98.patch, HBASE-11039_v5_0.98.patch Create an integration test for the VisibilityController that: 1. Create several tables of test data 2. Assign a set of auths to each table. Label all entries in the table with appropriate visibility expressions. Insure that some data in every table overlaps with data in other tables at common row/family/qualifier coordinates. Generate data like ITBLL so we can verify all data present later. 3. Mix the data from the different tables into a new common table 4. Verify for each set of auths defined in step #2 that all entries found in the source table can be found in the common table. Like the ITBLL verification step but done N times for each set of auths defined in step #2. 5. Choose one of the source tables. Get its set of auths. Perform a deletion with visibility expression from the common table using those auths. 6. Verify that no data in the common table with the auth set chosen in #5 remains. A simple row count with the set of auths chosen in #5 that should return 0. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11039) [VisibilityController] Integration test for labeled data set mixing and filtered excise
[ https://issues.apache.org/jira/browse/HBASE-11039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14061121#comment-14061121 ] Hudson commented on HBASE-11039: SUCCESS: Integrated in HBase-1.0 #39 (See [https://builds.apache.org/job/HBase-1.0/39/]) HBASE-11509 Forward port HBASE-11039 to trunk and branch-1 after (ramkrishna: rev f0035e683752abfd875a3a6fb8756fb58c29c175) * hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedListWithVisibility.java * hbase-client/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlClient.java * hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedList.java * hbase-client/src/main/java/org/apache/hadoop/hbase/security/visibility/CellVisibility.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java [VisibilityController] Integration test for labeled data set mixing and filtered excise --- Key: HBASE-11039 URL: https://issues.apache.org/jira/browse/HBASE-11039 Project: HBase Issue Type: Test Affects Versions: 0.98.1 Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Priority: Critical Fix For: 0.98.4 Attachments: HBASE-11039_ITBLL_v1.patch, HBASE-11039_v3_0.98.patch, HBASE-11039_v4_0.98.patch, HBASE-11039_v5_0.98.patch Create an integration test for the VisibilityController that: 1. Create several tables of test data 2. Assign a set of auths to each table. Label all entries in the table with appropriate visibility expressions. Insure that some data in every table overlaps with data in other tables at common row/family/qualifier coordinates. Generate data like ITBLL so we can verify all data present later. 3. Mix the data from the different tables into a new common table 4. Verify for each set of auths defined in step #2 that all entries found in the source table can be found in the common table. Like the ITBLL verification step but done N times for each set of auths defined in step #2. 5. Choose one of the source tables. Get its set of auths. Perform a deletion with visibility expression from the common table using those auths. 6. Verify that no data in the common table with the auth set chosen in #5 remains. A simple row count with the set of auths chosen in #5 that should return 0. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11039) [VisibilityController] Integration test for labeled data set mixing and filtered excise
[ https://issues.apache.org/jira/browse/HBASE-11039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14061303#comment-14061303 ] Hudson commented on HBASE-11039: SUCCESS: Integrated in HBase-TRUNK #5301 (See [https://builds.apache.org/job/HBase-TRUNK/5301/]) HBASE-11509 Forward port HBASE-11039 to trunk and branch-1 after (ramkrishna: rev e71d69bf00e792115fdc841cf597ed37b5ae4e2a) * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java * hbase-client/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlClient.java * hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedListWithVisibility.java * hbase-client/src/main/java/org/apache/hadoop/hbase/security/visibility/CellVisibility.java * hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedList.java [VisibilityController] Integration test for labeled data set mixing and filtered excise --- Key: HBASE-11039 URL: https://issues.apache.org/jira/browse/HBASE-11039 Project: HBase Issue Type: Test Affects Versions: 0.98.1 Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Priority: Critical Fix For: 0.98.4 Attachments: HBASE-11039_ITBLL_v1.patch, HBASE-11039_v3_0.98.patch, HBASE-11039_v4_0.98.patch, HBASE-11039_v5_0.98.patch Create an integration test for the VisibilityController that: 1. Create several tables of test data 2. Assign a set of auths to each table. Label all entries in the table with appropriate visibility expressions. Insure that some data in every table overlaps with data in other tables at common row/family/qualifier coordinates. Generate data like ITBLL so we can verify all data present later. 3. Mix the data from the different tables into a new common table 4. Verify for each set of auths defined in step #2 that all entries found in the source table can be found in the common table. Like the ITBLL verification step but done N times for each set of auths defined in step #2. 5. Choose one of the source tables. Get its set of auths. Perform a deletion with visibility expression from the common table using those auths. 6. Verify that no data in the common table with the auth set chosen in #5 remains. A simple row count with the set of auths chosen in #5 that should return 0. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11039) [VisibilityController] Integration test for labeled data set mixing and filtered excise
[ https://issues.apache.org/jira/browse/HBASE-11039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14060343#comment-14060343 ] Andrew Purtell commented on HBASE-11039: bq. Works with mvn verify also. Sets dfs.permissions to false. +1 [VisibilityController] Integration test for labeled data set mixing and filtered excise --- Key: HBASE-11039 URL: https://issues.apache.org/jira/browse/HBASE-11039 Project: HBase Issue Type: Test Affects Versions: 0.98.1 Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Priority: Critical Fix For: 0.99.0, 1.0.0, 0.98.4 Attachments: HBASE-11039_ITBLL_v1.patch, HBASE-11039_v3_0.98.patch, HBASE-11039_v4_0.98.patch, HBASE-11039_v5_0.98.patch Create an integration test for the VisibilityController that: 1. Create several tables of test data 2. Assign a set of auths to each table. Label all entries in the table with appropriate visibility expressions. Insure that some data in every table overlaps with data in other tables at common row/family/qualifier coordinates. Generate data like ITBLL so we can verify all data present later. 3. Mix the data from the different tables into a new common table 4. Verify for each set of auths defined in step #2 that all entries found in the source table can be found in the common table. Like the ITBLL verification step but done N times for each set of auths defined in step #2. 5. Choose one of the source tables. Get its set of auths. Perform a deletion with visibility expression from the common table using those auths. 6. Verify that no data in the common table with the auth set chosen in #5 remains. A simple row count with the set of auths chosen in #5 that should return 0. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11039) [VisibilityController] Integration test for labeled data set mixing and filtered excise
[ https://issues.apache.org/jira/browse/HBASE-11039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14059696#comment-14059696 ] ramkrishna.s.vasudevan commented on HBASE-11039: Thanks for the update Andy. The above steps are the ones needed to run the test in a cluster. Regarding the mvn verfiy, Even specifying an existing user does not solve the MR issue that we get. 1) if the user exists on a real test cluster, Still does not work. and 2) if the environment mapreduce based tests is set up correctly to account for the fake users How to do this? I tried adding ACL CP and VC CP in the setUpCluster() code, still it did not work. [VisibilityController] Integration test for labeled data set mixing and filtered excise --- Key: HBASE-11039 URL: https://issues.apache.org/jira/browse/HBASE-11039 Project: HBase Issue Type: Test Affects Versions: 0.98.1 Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Priority: Critical Fix For: 0.99.0, 1.0.0, 0.98.4 Attachments: HBASE-11039_ITBLL_v1.patch, HBASE-11039_v3_0.98.patch, HBASE-11039_v4_0.98.patch Create an integration test for the VisibilityController that: 1. Create several tables of test data 2. Assign a set of auths to each table. Label all entries in the table with appropriate visibility expressions. Insure that some data in every table overlaps with data in other tables at common row/family/qualifier coordinates. Generate data like ITBLL so we can verify all data present later. 3. Mix the data from the different tables into a new common table 4. Verify for each set of auths defined in step #2 that all entries found in the source table can be found in the common table. Like the ITBLL verification step but done N times for each set of auths defined in step #2. 5. Choose one of the source tables. Get its set of auths. Perform a deletion with visibility expression from the common table using those auths. 6. Verify that no data in the common table with the auth set chosen in #5 remains. A simple row count with the set of auths chosen in #5 that should return 0. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11039) [VisibilityController] Integration test for labeled data set mixing and filtered excise
[ https://issues.apache.org/jira/browse/HBASE-11039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14059698#comment-14059698 ] ramkrishna.s.vasudevan commented on HBASE-11039: I will see what else to be done for that MR related setup. [VisibilityController] Integration test for labeled data set mixing and filtered excise --- Key: HBASE-11039 URL: https://issues.apache.org/jira/browse/HBASE-11039 Project: HBase Issue Type: Test Affects Versions: 0.98.1 Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Priority: Critical Fix For: 0.99.0, 1.0.0, 0.98.4 Attachments: HBASE-11039_ITBLL_v1.patch, HBASE-11039_v3_0.98.patch, HBASE-11039_v4_0.98.patch Create an integration test for the VisibilityController that: 1. Create several tables of test data 2. Assign a set of auths to each table. Label all entries in the table with appropriate visibility expressions. Insure that some data in every table overlaps with data in other tables at common row/family/qualifier coordinates. Generate data like ITBLL so we can verify all data present later. 3. Mix the data from the different tables into a new common table 4. Verify for each set of auths defined in step #2 that all entries found in the source table can be found in the common table. Like the ITBLL verification step but done N times for each set of auths defined in step #2. 5. Choose one of the source tables. Get its set of auths. Perform a deletion with visibility expression from the common table using those auths. 6. Verify that no data in the common table with the auth set chosen in #5 remains. A simple row count with the set of auths chosen in #5 that should return 0. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11039) [VisibilityController] Integration test for labeled data set mixing and filtered excise
[ https://issues.apache.org/jira/browse/HBASE-11039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14059731#comment-14059731 ] ramkrishna.s.vasudevan commented on HBASE-11039: Works with mvn verify also. Sets dfs.permissions to false. [VisibilityController] Integration test for labeled data set mixing and filtered excise --- Key: HBASE-11039 URL: https://issues.apache.org/jira/browse/HBASE-11039 Project: HBase Issue Type: Test Affects Versions: 0.98.1 Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Priority: Critical Fix For: 0.99.0, 1.0.0, 0.98.4 Attachments: HBASE-11039_ITBLL_v1.patch, HBASE-11039_v3_0.98.patch, HBASE-11039_v4_0.98.patch, HBASE-11039_v5_0.98.patch Create an integration test for the VisibilityController that: 1. Create several tables of test data 2. Assign a set of auths to each table. Label all entries in the table with appropriate visibility expressions. Insure that some data in every table overlaps with data in other tables at common row/family/qualifier coordinates. Generate data like ITBLL so we can verify all data present later. 3. Mix the data from the different tables into a new common table 4. Verify for each set of auths defined in step #2 that all entries found in the source table can be found in the common table. Like the ITBLL verification step but done N times for each set of auths defined in step #2. 5. Choose one of the source tables. Get its set of auths. Perform a deletion with visibility expression from the common table using those auths. 6. Verify that no data in the common table with the auth set chosen in #5 remains. A simple row count with the set of auths chosen in #5 that should return 0. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11039) [VisibilityController] Integration test for labeled data set mixing and filtered excise
[ https://issues.apache.org/jira/browse/HBASE-11039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14059012#comment-14059012 ] Andrew Purtell commented on HBASE-11039: 'mvn verify' doesn't work in 0.98? Why? [VisibilityController] Integration test for labeled data set mixing and filtered excise --- Key: HBASE-11039 URL: https://issues.apache.org/jira/browse/HBASE-11039 Project: HBase Issue Type: Test Affects Versions: 0.98.1 Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Priority: Critical Fix For: 0.99.0, 1.0.0, 0.98.4 Attachments: HBASE-11039_ITBLL_v1.patch, HBASE-11039_v3_0.98.patch, HBASE-11039_v4_0.98.patch Create an integration test for the VisibilityController that: 1. Create several tables of test data 2. Assign a set of auths to each table. Label all entries in the table with appropriate visibility expressions. Insure that some data in every table overlaps with data in other tables at common row/family/qualifier coordinates. Generate data like ITBLL so we can verify all data present later. 3. Mix the data from the different tables into a new common table 4. Verify for each set of auths defined in step #2 that all entries found in the source table can be found in the common table. Like the ITBLL verification step but done N times for each set of auths defined in step #2. 5. Choose one of the source tables. Get its set of auths. Perform a deletion with visibility expression from the common table using those auths. 6. Verify that no data in the common table with the auth set chosen in #5 remains. A simple row count with the set of auths chosen in #5 that should return 0. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11039) [VisibilityController] Integration test for labeled data set mixing and filtered excise
[
https://issues.apache.org/jira/browse/HBASE-11039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14059038#comment-14059038
]
Andrew Purtell commented on HBASE-11039:
This issue keeps coming up and affecting tests that we are contributing for
security. It needs to be addressed:
{noformat}
2014-07-11 10:18:09,751 ERROR [IPC Server handler 1 on 41147]
security.UserGroupInformation(1494): PriviledgedActionException as:user1
(auth:SIMPLE) cause:org.apache.hadoop.security.AccessControlException:
Permission denied: user=user1, access=WRITE,
inode=/user/apurtell/test-data/1e64e090-6e55-49b9-81b0-683247560414/IntegrationTestBigLinkedListWithVisibility:apurtell:supergroup:drwxr-xr-x
{noformat}
The tests are creating fake users without consideration for 1) if the user
exists on a real test cluster, and 2) if the job conf for mapreduce based tests
is set up correctly
[VisibilityController] Integration test for labeled data set mixing and
filtered excise
---
Key: HBASE-11039
URL: https://issues.apache.org/jira/browse/HBASE-11039
Project: HBase
Issue Type: Test
Affects Versions: 0.98.1
Reporter: Andrew Purtell
Assignee: ramkrishna.s.vasudevan
Priority: Critical
Fix For: 0.99.0, 1.0.0, 0.98.4
Attachments: HBASE-11039_ITBLL_v1.patch, HBASE-11039_v3_0.98.patch,
HBASE-11039_v4_0.98.patch
Create an integration test for the VisibilityController that:
1. Create several tables of test data
2. Assign a set of auths to each table. Label all entries in the table with
appropriate visibility expressions. Insure that some data in every table
overlaps with data in other tables at common row/family/qualifier
coordinates. Generate data like ITBLL so we can verify all data present later.
3. Mix the data from the different tables into a new common table
4. Verify for each set of auths defined in step #2 that all entries found in
the source table can be found in the common table. Like the ITBLL
verification step but done N times for each set of auths defined in step #2.
5. Choose one of the source tables. Get its set of auths. Perform a deletion
with visibility expression from the common table using those auths.
6. Verify that no data in the common table with the auth set chosen in #5
remains. A simple row count with the set of auths chosen in #5 that should
return 0.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11039) [VisibilityController] Integration test for labeled data set mixing and filtered excise
[
https://issues.apache.org/jira/browse/HBASE-11039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14059584#comment-14059584
]
Andrew Purtell commented on HBASE-11039:
As soon as the 'mvn verify' case can work, we are good to go I think.
Here is some documentation on how the test cluster should be set up and
configured, and how the test should be executed, so it completes successfully.
Relevant part of my hbase-site:
{noformat}
property
namehbase.coprocessor.master.classes/name
valueorg.apache.hadoop.hbase.security.access.AccessController,org.apache.hadoop.hbase.security.visibility.VisibilityController/value
/property
property
namehbase.coprocessor.regionserver.classes/name
valueorg.apache.hadoop.hbase.security.access.AccessController/value
/property
property
namehbase.coprocessor.region.classes/name
valueorg.apache.hadoop.hbase.security.access.AccessController,org.apache.hadoop.hbase.security.visibility.VisibilityController,org.apache.hadoop.hbase.security.token.TokenProvider/value
/property
{noformat}
I'm running the daemons under the 'hbase' user, so the superuser is 'hbase'.
I run the test as the user 'apurtell':
First I grant 'apurtell' all global privs because I don't want to be a problem:
{noformat}
Version 0.98.4-SNAPSHOT, raa7a0ce8395cf9ad5c99028463c619df952e14b8, Fri Jul 11
17:49:11 PDT 2014
hbase(main):001:0 grant 'apurtell', 'RWXCA'
0 row(s) in 1.5130 seconds
{noformat}
Then I run the test, following the example provided in an earlier comment on
this issue:
{noformat}
./bin/hbase
org.apache.hadoop.hbase.test.IntegrationTestBigLinkedListWithVisibility -u
apurtell Loop 1 1 2 /tmp 1 1
{noformat}
[VisibilityController] Integration test for labeled data set mixing and
filtered excise
---
Key: HBASE-11039
URL: https://issues.apache.org/jira/browse/HBASE-11039
Project: HBase
Issue Type: Test
Affects Versions: 0.98.1
Reporter: Andrew Purtell
Assignee: ramkrishna.s.vasudevan
Priority: Critical
Fix For: 0.99.0, 1.0.0, 0.98.4
Attachments: HBASE-11039_ITBLL_v1.patch, HBASE-11039_v3_0.98.patch,
HBASE-11039_v4_0.98.patch
Create an integration test for the VisibilityController that:
1. Create several tables of test data
2. Assign a set of auths to each table. Label all entries in the table with
appropriate visibility expressions. Insure that some data in every table
overlaps with data in other tables at common row/family/qualifier
coordinates. Generate data like ITBLL so we can verify all data present later.
3. Mix the data from the different tables into a new common table
4. Verify for each set of auths defined in step #2 that all entries found in
the source table can be found in the common table. Like the ITBLL
verification step but done N times for each set of auths defined in step #2.
5. Choose one of the source tables. Get its set of auths. Perform a deletion
with visibility expression from the common table using those auths.
6. Verify that no data in the common table with the auth set chosen in #5
remains. A simple row count with the set of auths chosen in #5 that should
return 0.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11039) [VisibilityController] Integration test for labeled data set mixing and filtered excise
[ https://issues.apache.org/jira/browse/HBASE-11039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14057196#comment-14057196 ] ramkrishna.s.vasudevan commented on HBASE-11039: bq.testContinuousIngest(org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList): org/jboss/netty/channel/ChannelFactory I am getting this error in the trunk while trying to run this bq.mvn verify -Dit.test=IntegrationTestBigLinkedList -Dtest.output.tofile=false. Not sure why. bq.Looks like the table name for the test isn't overridden in all places: Not sure what to do there. So should i override the getTableName() and return which of the 4 tables here- the common one or one of the 3 smaller tables? [VisibilityController] Integration test for labeled data set mixing and filtered excise --- Key: HBASE-11039 URL: https://issues.apache.org/jira/browse/HBASE-11039 Project: HBase Issue Type: Test Affects Versions: 0.98.1 Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Priority: Critical Fix For: 0.99.0, 1.0.0, 0.98.4 Attachments: HBASE-11039_ITBLL_v1.patch Create an integration test for the VisibilityController that: 1. Create several tables of test data 2. Assign a set of auths to each table. Label all entries in the table with appropriate visibility expressions. Insure that some data in every table overlaps with data in other tables at common row/family/qualifier coordinates. Generate data like ITBLL so we can verify all data present later. 3. Mix the data from the different tables into a new common table 4. Verify for each set of auths defined in step #2 that all entries found in the source table can be found in the common table. Like the ITBLL verification step but done N times for each set of auths defined in step #2. 5. Choose one of the source tables. Get its set of auths. Perform a deletion with visibility expression from the common table using those auths. 6. Verify that no data in the common table with the auth set chosen in #5 remains. A simple row count with the set of auths chosen in #5 that should return 0. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11039) [VisibilityController] Integration test for labeled data set mixing and filtered excise
[
https://issues.apache.org/jira/browse/HBASE-11039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14057379#comment-14057379
]
ramkrishna.s.vasudevan commented on HBASE-11039:
The IT test cases using MR does not pass when used with
{code}
mvn verify -Dit.test=IntegrationTestBigLinkedListWithVisibility
-Dtest.output.tofile=false.
{code}
because of this issue
{code}
2014-07-10 17:53:16,001 ERROR [IPC Server handler 1 on 41343]
security.UserGroupInformation(1494): PriviledgedActionException as:user1
(auth:SIMPLE) cause:org.apache.hadoop.security.AccessControlException:
Permission denied: user=user1, access=WRITE,
inode=quot;/user/root/test-data/58e79ef0-43c3-4af9-afc7-af6103315ad4/IntegrationTestBigLinkedListWithVisibilityquot;:root:supergroup:drwxr-xr-x
{code}
So in cases where I need to run the MR scan as a non super user then we would
create a user inside the testcase and verify the scan. In such cases the MR job
itself fails.
[VisibilityController] Integration test for labeled data set mixing and
filtered excise
---
Key: HBASE-11039
URL: https://issues.apache.org/jira/browse/HBASE-11039
Project: HBase
Issue Type: Test
Affects Versions: 0.98.1
Reporter: Andrew Purtell
Assignee: ramkrishna.s.vasudevan
Priority: Critical
Fix For: 0.99.0, 1.0.0, 0.98.4
Attachments: HBASE-11039_ITBLL_v1.patch
Create an integration test for the VisibilityController that:
1. Create several tables of test data
2. Assign a set of auths to each table. Label all entries in the table with
appropriate visibility expressions. Insure that some data in every table
overlaps with data in other tables at common row/family/qualifier
coordinates. Generate data like ITBLL so we can verify all data present later.
3. Mix the data from the different tables into a new common table
4. Verify for each set of auths defined in step #2 that all entries found in
the source table can be found in the common table. Like the ITBLL
verification step but done N times for each set of auths defined in step #2.
5. Choose one of the source tables. Get its set of auths. Perform a deletion
with visibility expression from the common table using those auths.
6. Verify that no data in the common table with the auth set chosen in #5
remains. A simple row count with the set of auths chosen in #5 that should
return 0.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11039) [VisibilityController] Integration test for labeled data set mixing and filtered excise
[
https://issues.apache.org/jira/browse/HBASE-11039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14057939#comment-14057939
]
Andrew Purtell commented on HBASE-11039:
Thanks for updating the class Javadoc.
Just about there I think.
{quote}
estContinuousIngest(org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList):
org/jboss/netty/channel/ChannelFactory
I am getting this error in the trunk while trying to run this
{quote}
Yeah. I see you opened HBASE-11489 for that. Let's get this test working in
0.98, and commit the changes that let that happen through trunk, and fix the
Netty related issue on HBASE-11489.
bq. If both VC and ACL are not configured in the site.xml then we may get
errors as ACL table would not be available.
Yes, in this part:
{code}
+private void createTable(HBaseAdmin admin, TableName tableName, boolean
setVersion)
+throws IOException {
+ if (!admin.tableExists(tableName)) {
+HTableDescriptor htd = new HTableDescriptor(tableName);
+HColumnDescriptor family = new HColumnDescriptor(FAMILY_NAME);
+if (setVersion) {
+ family.setMaxVersions(DEFAULT_TABLES_COUNT);
+}
+htd.addFamily(family);
+admin.createTable(htd);
+LOG.info(Granting permissions for user + USER.getShortName());
+AccessControlProtos.Permission.Action[] actions = {
AccessControlProtos.Permission.Action.READ };
+try {
+ AccessControlClient.grant(getConf(), tableName, USER.getShortName(),
null, null, actions);
+} catch (Throwable e) {
+ LOG.fatal(Error in granting permission for the user +
USER.getShortName(), e);
+ throw new IOException(e);
+}
+ }
+}
{code}
Please consider adding a method 'boolean
AccessControlClient#isAccessControllerRunning' that returns true if the AC is
installed and active, false otherwise. Check the return value of the new method
first before attempting any grant. Then we should be ok.
[VisibilityController] Integration test for labeled data set mixing and
filtered excise
---
Key: HBASE-11039
URL: https://issues.apache.org/jira/browse/HBASE-11039
Project: HBase
Issue Type: Test
Affects Versions: 0.98.1
Reporter: Andrew Purtell
Assignee: ramkrishna.s.vasudevan
Priority: Critical
Fix For: 0.99.0, 1.0.0, 0.98.4
Attachments: HBASE-11039_ITBLL_v1.patch, HBASE-11039_v3_0.98.patch
Create an integration test for the VisibilityController that:
1. Create several tables of test data
2. Assign a set of auths to each table. Label all entries in the table with
appropriate visibility expressions. Insure that some data in every table
overlaps with data in other tables at common row/family/qualifier
coordinates. Generate data like ITBLL so we can verify all data present later.
3. Mix the data from the different tables into a new common table
4. Verify for each set of auths defined in step #2 that all entries found in
the source table can be found in the common table. Like the ITBLL
verification step but done N times for each set of auths defined in step #2.
5. Choose one of the source tables. Get its set of auths. Perform a deletion
with visibility expression from the common table using those auths.
6. Verify that no data in the common table with the auth set chosen in #5
remains. A simple row count with the set of auths chosen in #5 that should
return 0.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11039) [VisibilityController] Integration test for labeled data set mixing and filtered excise
[ https://issues.apache.org/jira/browse/HBASE-11039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14056135#comment-14056135 ] ramkrishna.s.vasudevan commented on HBASE-11039: @apurtell bq. ./hbase org.apache.hadoop.hbase.test.IntegrationTestBigLinkedListWithVisibility Loop 1 1 2 /tmp 1 1 Use this command. It works. Internally there is a logic where the num of nodes and the width of each row depending on that the rows are written. This works Andy. [VisibilityController] Integration test for labeled data set mixing and filtered excise --- Key: HBASE-11039 URL: https://issues.apache.org/jira/browse/HBASE-11039 Project: HBase Issue Type: Test Affects Versions: 0.98.1 Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Priority: Critical Fix For: 0.99.0, 1.0.0, 0.98.4 Attachments: HBASE-11039_ITBLL_v1.patch Create an integration test for the VisibilityController that: 1. Create several tables of test data 2. Assign a set of auths to each table. Label all entries in the table with appropriate visibility expressions. Insure that some data in every table overlaps with data in other tables at common row/family/qualifier coordinates. Generate data like ITBLL so we can verify all data present later. 3. Mix the data from the different tables into a new common table 4. Verify for each set of auths defined in step #2 that all entries found in the source table can be found in the common table. Like the ITBLL verification step but done N times for each set of auths defined in step #2. 5. Choose one of the source tables. Get its set of auths. Perform a deletion with visibility expression from the common table using those auths. 6. Verify that no data in the common table with the auth set chosen in #5 remains. A simple row count with the set of auths chosen in #5 that should return 0. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11039) [VisibilityController] Integration test for labeled data set mixing and filtered excise
[ https://issues.apache.org/jira/browse/HBASE-11039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14056602#comment-14056602 ] Andrew Purtell commented on HBASE-11039: bq. Use this command. It works. bq. ./hbase org.apache.hadoop.hbase.test.IntegrationTestBigLinkedListWithVisibility Loop 1 1 2 /tmp 1 1 Thanks. That worked. Please update the test so when it is executed with 'mvn verify' these are the parameters used. Please provide advice and guidance for running the test in the class Javadoc for it. I still need to remove the AccessController from the cluster site configuration and include only the VC. This probably won't be a common configuration for a security enabled QA cluster. Can we include test parameters that allow the test runner to supply the names of real user accounts? Thanks Ram. [VisibilityController] Integration test for labeled data set mixing and filtered excise --- Key: HBASE-11039 URL: https://issues.apache.org/jira/browse/HBASE-11039 Project: HBase Issue Type: Test Affects Versions: 0.98.1 Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Priority: Critical Fix For: 0.99.0, 1.0.0, 0.98.4 Attachments: HBASE-11039_ITBLL_v1.patch Create an integration test for the VisibilityController that: 1. Create several tables of test data 2. Assign a set of auths to each table. Label all entries in the table with appropriate visibility expressions. Insure that some data in every table overlaps with data in other tables at common row/family/qualifier coordinates. Generate data like ITBLL so we can verify all data present later. 3. Mix the data from the different tables into a new common table 4. Verify for each set of auths defined in step #2 that all entries found in the source table can be found in the common table. Like the ITBLL verification step but done N times for each set of auths defined in step #2. 5. Choose one of the source tables. Get its set of auths. Perform a deletion with visibility expression from the common table using those auths. 6. Verify that no data in the common table with the auth set chosen in #5 remains. A simple row count with the set of auths chosen in #5 that should return 0. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11039) [VisibilityController] Integration test for labeled data set mixing and filtered excise
[ https://issues.apache.org/jira/browse/HBASE-11039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14055067#comment-14055067 ] Andrew Purtell commented on HBASE-11039: Sorry for the delay [~ram_krish], I was on holiday. Let me try this today. [VisibilityController] Integration test for labeled data set mixing and filtered excise --- Key: HBASE-11039 URL: https://issues.apache.org/jira/browse/HBASE-11039 Project: HBase Issue Type: Test Affects Versions: 0.98.1 Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Priority: Critical Fix For: 0.99.0, 1.0.0, 0.98.4 Attachments: HBASE-11039_ITBLL_v1.patch Create an integration test for the VisibilityController that: 1. Create several tables of test data 2. Assign a set of auths to each table. Label all entries in the table with appropriate visibility expressions. Insure that some data in every table overlaps with data in other tables at common row/family/qualifier coordinates. Generate data like ITBLL so we can verify all data present later. 3. Mix the data from the different tables into a new common table 4. Verify for each set of auths defined in step #2 that all entries found in the source table can be found in the common table. Like the ITBLL verification step but done N times for each set of auths defined in step #2. 5. Choose one of the source tables. Get its set of auths. Perform a deletion with visibility expression from the common table using those auths. 6. Verify that no data in the common table with the auth set chosen in #5 remains. A simple row count with the set of auths chosen in #5 that should return 0. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11039) [VisibilityController] Integration test for labeled data set mixing and filtered excise
[
https://issues.apache.org/jira/browse/HBASE-11039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14055804#comment-14055804
]
Andrew Purtell commented on HBASE-11039:
I tried running this against a cluster with ./bin/hbase
org.apache.hadoop.hbase.test.IntegrationTestBigLinkedListWithVisibility -m calm
Loop 1 1 1 /tmp 1 and encountered a failure because the test creates users
for testing that do not correspond to user accounts on the cluster. I think we
need command line options for specifying users like with
IntegrationTestIngestWithACL and how to operate this test should be documented.
On the server:
{noformat}
2014-07-08 20:09:32,553 WARN [defaultRpcServer.handler=27,queue=0,port=8120]
security.ShellBasedUnixGroupsMapping: got exception trying to get groups for
user user1
org.apache.hadoop.util.Shell$ExitCodeException: id: user1: no such user
at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
at org.apache.hadoop.util.Shell.run(Shell.java:379)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:678)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:661)
at
org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getUnixGroups(ShellBasedUnixGroupsMapping.java:83)
at
org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getGroups(ShellBasedUnixGroupsMapping.java:52)
at
org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.getGroups(JniBasedUnixGroupsMappingWithFallback.java:50)
at org.apache.hadoop.security.Groups.getGroups(Groups.java:89)
at
org.apache.hadoop.security.UserGroupInformation.getGroupNames(UserGroupInformation.java:1352)
at org.apache.hadoop.hbase.security.User.getGroupNames(User.java:76)
at
org.apache.hadoop.hbase.security.access.TableAuthManager.authorize(TableAuthManager.java:487)
at
org.apache.hadoop.hbase.security.access.TableAuthManager.authorize(TableAuthManager.java:500)
at
org.apache.hadoop.hbase.security.access.AccessController.permissionGranted(AccessController.java:303)
at
org.apache.hadoop.hbase.security.access.AccessController.permissionGranted(AccessController.java:372)
at
org.apache.hadoop.hbase.security.access.AccessController.internalPreRead(AccessController.java:1368)
at
org.apache.hadoop.hbase.security.access.AccessController.preScannerOpen(AccessController.java:1891)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preScannerOpen(RegionCoprocessorHost.java:1784)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3085)
{noformat}
On the client:
{noformat}
2014-07-08 20:09:33,485 ERROR [main] test.IntegrationTestBigLinkedList$Verify:
Expected referenced count does not match with actual referenced count. expected
referenced=1 ,actual=0
{noformat}
[VisibilityController] Integration test for labeled data set mixing and
filtered excise
---
Key: HBASE-11039
URL: https://issues.apache.org/jira/browse/HBASE-11039
Project: HBase
Issue Type: Test
Affects Versions: 0.98.1
Reporter: Andrew Purtell
Assignee: ramkrishna.s.vasudevan
Priority: Critical
Fix For: 0.99.0, 1.0.0, 0.98.4
Attachments: HBASE-11039_ITBLL_v1.patch
Create an integration test for the VisibilityController that:
1. Create several tables of test data
2. Assign a set of auths to each table. Label all entries in the table with
appropriate visibility expressions. Insure that some data in every table
overlaps with data in other tables at common row/family/qualifier
coordinates. Generate data like ITBLL so we can verify all data present later.
3. Mix the data from the different tables into a new common table
4. Verify for each set of auths defined in step #2 that all entries found in
the source table can be found in the common table. Like the ITBLL
verification step but done N times for each set of auths defined in step #2.
5. Choose one of the source tables. Get its set of auths. Perform a deletion
with visibility expression from the common table using those auths.
6. Verify that no data in the common table with the auth set chosen in #5
remains. A simple row count with the set of auths chosen in #5 that should
return 0.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11039) [VisibilityController] Integration test for labeled data set mixing and filtered excise
[
https://issues.apache.org/jira/browse/HBASE-11039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14055812#comment-14055812
]
Andrew Purtell commented on HBASE-11039:
If I run the test with the AccessController removed from the configuration, the
test still fails.
{noformat}
2014-07-08 20:20:49,936 INFO [main]
test.IntegrationTestBigLinkedListWithVisibility$VisibilityLoop: Running Loop
with args:[1, 1, 1, /tmp, 1]
2014-07-08 20:20:49,937 INFO [main]
test.IntegrationTestBigLinkedListWithVisibility$VisibilityLoop: Starting
iteration = 0
2014-07-08 20:20:49,937 INFO [main]
test.IntegrationTestBigLinkedListWithVisibility$VisibilityLoop: Generating data
2014-07-08 20:20:49,938 INFO [main]
test.IntegrationTestBigLinkedList$Generator: Running RandomInputGenerator with
numMappers=1, numNodes=1
2014-07-08 20:20:51,527 INFO [main]
test.IntegrationTestBigLinkedList$Generator: Running Generator with
numMappers=1, numNodes=1
2014-07-08 20:20:51,528 INFO [main]
test.IntegrationTestBigLinkedListWithVisibility$VisibilityGenerator: Creating
tables
2014-07-08 20:20:54,782 INFO [main]
test.IntegrationTestBigLinkedListWithVisibility$VisibilityLoop: Running copier
2014-07-08 20:20:59,782 INFO [main]
test.IntegrationTestBigLinkedListWithVisibility$VisibilityLoop: Running copier
tableName_0
2014-07-08 20:21:04,792 INFO [main]
test.IntegrationTestBigLinkedListWithVisibility$Copier: Running copiers
2014-07-08 20:21:06,886 INFO [main]
test.IntegrationTestBigLinkedListWithVisibility$VisibilityLoop: Running copier
tableName_1
2014-07-08 20:21:11,888 INFO [main]
test.IntegrationTestBigLinkedListWithVisibility$Copier: Running copiers
2014-07-08 20:21:13,805 INFO [main]
test.IntegrationTestBigLinkedListWithVisibility$VisibilityLoop: Running copier
tableName_2
2014-07-08 20:21:18,806 INFO [main]
test.IntegrationTestBigLinkedListWithVisibility$Copier: Running copiers
2014-07-08 20:21:21,080 INFO [main]
test.IntegrationTestBigLinkedListWithVisibility$VisibilityLoop: Verifying
copied data
2014-07-08 20:21:26,080 INFO [main]
test.IntegrationTestBigLinkedListWithVisibility$VisibilityLoop: Verifying table 0
2014-07-08 20:21:31,083 INFO [main]
test.IntegrationTestBigLinkedListWithVisibility$VisibilityVerify: Running
Verify with outputDir=/tmp/a9a6cf46-213b-4c27-bf3a-84e03c99905d, numReducers=1
2014-07-08 20:21:33,084 ERROR [main] test.IntegrationTestBigLinkedList$Verify:
Expected referenced count does not match with actual referenced count. expected
referenced=1 ,actual=0
2014-07-08 20:21:33,100 ERROR [main] util.AbstractHBaseTool: Error running
command-line tool
java.lang.RuntimeException: Verify.verify failed
at
org.apache.hadoop.hbase.test.IntegrationTestBigLinkedListWithVisibility$VisibilityLoop.verify(IntegrationTestBigLinkedListWithVisibility.java:515)
at
org.apache.hadoop.hbase.test.IntegrationTestBigLinkedListWithVisibility$VisibilityLoop.runVerify(IntegrationTestBigLinkedListWithVisibility.java:457)
at
org.apache.hadoop.hbase.test.IntegrationTestBigLinkedListWithVisibility$VisibilityLoop.run(IntegrationTestBigLinkedListWithVisibility.java:558)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at
org.apache.hadoop.hbase.test.IntegrationTestBigLinkedListWithVisibility.runTestFromCommandLine(IntegrationTestBigLinkedListWithVisibility.java:601)
at
org.apache.hadoop.hbase.IntegrationTestBase.doWork(IntegrationTestBase.java:79)
at
org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:112)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at
org.apache.hadoop.hbase.test.IntegrationTestBigLinkedListWithVisibility.main(IntegrationTestBigLinkedListWithVisibility.java:591)
{noformat}
[VisibilityController] Integration test for labeled data set mixing and
filtered excise
---
Key: HBASE-11039
URL: https://issues.apache.org/jira/browse/HBASE-11039
Project: HBase
Issue Type: Test
Affects Versions: 0.98.1
Reporter: Andrew Purtell
Assignee: ramkrishna.s.vasudevan
Priority: Critical
Fix For: 0.99.0, 1.0.0, 0.98.4
Attachments: HBASE-11039_ITBLL_v1.patch
Create an integration test for the VisibilityController that:
1. Create several tables of test data
2. Assign a set of auths to each table. Label all entries in the table with
appropriate visibility expressions. Insure that some data in every table
overlaps with data in other tables at common row/family/qualifier
coordinates. Generate data like ITBLL so we can verify all data present later.
3. Mix the data from the different tables into a new common table
4. Verify for each set of auths defined in step #2 that all
[jira] [Commented] (HBASE-11039) [VisibilityController] Integration test for labeled data set mixing and filtered excise
[
https://issues.apache.org/jira/browse/HBASE-11039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14055814#comment-14055814
]
Andrew Purtell commented on HBASE-11039:
If I run the test as a minicluster
{noformat}
$ mvn verify -Dit.test=IntegrationTestBigLinkedListWithVisibility
-Dtest.output.tofile=false
{noformat}
the defaults for ITBLL mean we wait for a long time for jobs to finish. I
didn't have time to wait for the test to get past generation and copying to the
excise and verification phases. We should override the ITBLL and dial down the
node count to something like 100,000 when running on the minicluster.
Looks like the table name for the test isn't overridden in all places:
{noformat}
2014-07-08 20:29:40,807 WARN [Thread-670]
policies.PeriodicRandomActionPolicy(61): Exception occured during performing
action: org.apache.hadoop.hbase.TableNotFoundException:
IntegrationTestBigLinkedList
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHTableDescriptor(HConnectionManager.java:2686)
at
org.apache.hadoop.hbase.client.HBaseAdmin.getTableDescriptor(HBaseAdmin.java:397)
at
org.apache.hadoop.hbase.client.HBaseAdmin.getTableDescriptor(HBaseAdmin.java:402)
at
org.apache.hadoop.hbase.chaos.actions.ChangeVersionsAction.perform(ChangeVersionsAction.java:55)
at
org.apache.hadoop.hbase.chaos.policies.PeriodicRandomActionPolicy.runOneIteration(PeriodicRandomActionPolicy.java:59)
at
org.apache.hadoop.hbase.chaos.policies.PeriodicPolicy.run(PeriodicPolicy.java:41)
at java.lang.Thread.run(Thread.java:745)
{noformat}
[VisibilityController] Integration test for labeled data set mixing and
filtered excise
---
Key: HBASE-11039
URL: https://issues.apache.org/jira/browse/HBASE-11039
Project: HBase
Issue Type: Test
Affects Versions: 0.98.1
Reporter: Andrew Purtell
Assignee: ramkrishna.s.vasudevan
Priority: Critical
Fix For: 0.99.0, 1.0.0, 0.98.4
Attachments: HBASE-11039_ITBLL_v1.patch
Create an integration test for the VisibilityController that:
1. Create several tables of test data
2. Assign a set of auths to each table. Label all entries in the table with
appropriate visibility expressions. Insure that some data in every table
overlaps with data in other tables at common row/family/qualifier
coordinates. Generate data like ITBLL so we can verify all data present later.
3. Mix the data from the different tables into a new common table
4. Verify for each set of auths defined in step #2 that all entries found in
the source table can be found in the common table. Like the ITBLL
verification step but done N times for each set of auths defined in step #2.
5. Choose one of the source tables. Get its set of auths. Perform a deletion
with visibility expression from the common table using those auths.
6. Verify that no data in the common table with the auth set chosen in #5
remains. A simple row count with the set of auths chosen in #5 that should
return 0.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11039) [VisibilityController] Integration test for labeled data set mixing and filtered excise
[ https://issues.apache.org/jira/browse/HBASE-11039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14055866#comment-14055866 ] ramkrishna.s.vasudevan commented on HBASE-11039: bq. Expected referenced count does not match with actual referenced count. expected referenced=1 ,actual=0 2014-07-08 20:21:33,100 ERROR [main] util.AbstractHBaseTool: Error running command-line tool Is this because the copier does not get completed by the time the verifier runs? Will see once again. How i verified this tool is that i reduced the number of rows to 2 and the row length to 1 and ensure that all worked fine. [VisibilityController] Integration test for labeled data set mixing and filtered excise --- Key: HBASE-11039 URL: https://issues.apache.org/jira/browse/HBASE-11039 Project: HBase Issue Type: Test Affects Versions: 0.98.1 Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Priority: Critical Fix For: 0.99.0, 1.0.0, 0.98.4 Attachments: HBASE-11039_ITBLL_v1.patch Create an integration test for the VisibilityController that: 1. Create several tables of test data 2. Assign a set of auths to each table. Label all entries in the table with appropriate visibility expressions. Insure that some data in every table overlaps with data in other tables at common row/family/qualifier coordinates. Generate data like ITBLL so we can verify all data present later. 3. Mix the data from the different tables into a new common table 4. Verify for each set of auths defined in step #2 that all entries found in the source table can be found in the common table. Like the ITBLL verification step but done N times for each set of auths defined in step #2. 5. Choose one of the source tables. Get its set of auths. Perform a deletion with visibility expression from the common table using those auths. 6. Verify that no data in the common table with the auth set chosen in #5 remains. A simple row count with the set of auths chosen in #5 that should return 0. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11039) [VisibilityController] Integration test for labeled data set mixing and filtered excise
[ https://issues.apache.org/jira/browse/HBASE-11039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14053055#comment-14053055 ] ramkrishna.s.vasudevan commented on HBASE-11039: [~apurtell][~anoop.hbase] Can you give a try at it and see if it is good to go to 0.98.4? [VisibilityController] Integration test for labeled data set mixing and filtered excise --- Key: HBASE-11039 URL: https://issues.apache.org/jira/browse/HBASE-11039 Project: HBase Issue Type: Test Affects Versions: 0.98.1 Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Priority: Critical Fix For: 0.99.0, 1.0.0, 0.98.4 Attachments: HBASE-11039_ITBLL_v1.patch Create an integration test for the VisibilityController that: 1. Create several tables of test data 2. Assign a set of auths to each table. Label all entries in the table with appropriate visibility expressions. Insure that some data in every table overlaps with data in other tables at common row/family/qualifier coordinates. Generate data like ITBLL so we can verify all data present later. 3. Mix the data from the different tables into a new common table 4. Verify for each set of auths defined in step #2 that all entries found in the source table can be found in the common table. Like the ITBLL verification step but done N times for each set of auths defined in step #2. 5. Choose one of the source tables. Get its set of auths. Perform a deletion with visibility expression from the common table using those auths. 6. Verify that no data in the common table with the auth set chosen in #5 remains. A simple row count with the set of auths chosen in #5 that should return 0. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11039) [VisibilityController] Integration test for labeled data set mixing and filtered excise
[ https://issues.apache.org/jira/browse/HBASE-11039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14046242#comment-14046242 ] Andrew Purtell commented on HBASE-11039: I think something based on IntegrationTestBigLinkedList is fine. Any other opinions? Sounds great. Let me port this patch to 0.98 and try it out. [VisibilityController] Integration test for labeled data set mixing and filtered excise --- Key: HBASE-11039 URL: https://issues.apache.org/jira/browse/HBASE-11039 Project: HBase Issue Type: Test Affects Versions: 0.98.1 Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Priority: Critical Fix For: 0.99.0, 0.98.4 Attachments: HBASE-11039_ITBLL_v1.patch Create an integration test for the VisibilityController that: 1. Create several tables of test data 2. Assign a set of auths to each table. Label all entries in the table with appropriate visibility expressions. Insure that some data in every table overlaps with data in other tables at common row/family/qualifier coordinates. Generate data like ITBLL so we can verify all data present later. 3. Mix the data from the different tables into a new common table 4. Verify for each set of auths defined in step #2 that all entries found in the source table can be found in the common table. Like the ITBLL verification step but done N times for each set of auths defined in step #2. 5. Choose one of the source tables. Get its set of auths. Perform a deletion with visibility expression from the common table using those auths. 6. Verify that no data in the common table with the auth set chosen in #5 remains. A simple row count with the set of auths chosen in #5 that should return 0. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11039) [VisibilityController] Integration test for labeled data set mixing and filtered excise
[ https://issues.apache.org/jira/browse/HBASE-11039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14044426#comment-14044426 ] ramkrishna.s.vasudevan commented on HBASE-11039: I have a similar patch that uses LTT. I can raise another JIRA and attach a patch there. That needs some minor tweaks to be done. [VisibilityController] Integration test for labeled data set mixing and filtered excise --- Key: HBASE-11039 URL: https://issues.apache.org/jira/browse/HBASE-11039 Project: HBase Issue Type: Test Affects Versions: 0.98.1 Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Priority: Critical Fix For: 0.99.0, 0.98.4 Attachments: HBASE-11039_ITBLL_v1.patch Create an integration test for the VisibilityController that: 1. Create several tables of test data 2. Assign a set of auths to each table. Label all entries in the table with appropriate visibility expressions. Insure that some data in every table overlaps with data in other tables at common row/family/qualifier coordinates. Generate data like ITBLL so we can verify all data present later. 3. Mix the data from the different tables into a new common table 4. Verify for each set of auths defined in step #2 that all entries found in the source table can be found in the common table. Like the ITBLL verification step but done N times for each set of auths defined in step #2. 5. Choose one of the source tables. Get its set of auths. Perform a deletion with visibility expression from the common table using those auths. 6. Verify that no data in the common table with the auth set chosen in #5 remains. A simple row count with the set of auths chosen in #5 that should return 0. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11039) [VisibilityController] Integration test for labeled data set mixing and filtered excise
[ https://issues.apache.org/jira/browse/HBASE-11039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14042914#comment-14042914 ] Andrew Purtell commented on HBASE-11039: Deleted earlier patch. Pushing to .5 [VisibilityController] Integration test for labeled data set mixing and filtered excise --- Key: HBASE-11039 URL: https://issues.apache.org/jira/browse/HBASE-11039 Project: HBase Issue Type: Test Affects Versions: 0.98.1 Reporter: Andrew Purtell Priority: Critical Fix For: 0.99.0, 0.98.5 Create an integration test for the VisibilityController that: 1. Create several tables of test data 2. Assign a set of auths to each table. Label all entries in the table with appropriate visibility expressions. Insure that some data in every table overlaps with data in other tables at common row/family/qualifier coordinates. Generate data like ITBLL so we can verify all data present later. 3. Mix the data from the different tables into a new common table 4. Verify for each set of auths defined in step #2 that all entries found in the source table can be found in the common table. Like the ITBLL verification step but done N times for each set of auths defined in step #2. 5. Choose one of the source tables. Get its set of auths. Perform a deletion with visibility expression from the common table using those auths. 6. Verify that no data in the common table with the auth set chosen in #5 remains. A simple row count with the set of auths chosen in #5 that should return 0. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11039) [VisibilityController] Integration test for labeled data set mixing and filtered excise
[ https://issues.apache.org/jira/browse/HBASE-11039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14043016#comment-14043016 ] ramkrishna.s.vasudevan commented on HBASE-11039: Andy, I have a patch that works with ITBLL. Lets check it internally once and if fine can try to push it in 0.98.4. [VisibilityController] Integration test for labeled data set mixing and filtered excise --- Key: HBASE-11039 URL: https://issues.apache.org/jira/browse/HBASE-11039 Project: HBase Issue Type: Test Affects Versions: 0.98.1 Reporter: Andrew Purtell Priority: Critical Fix For: 0.99.0, 0.98.5 Create an integration test for the VisibilityController that: 1. Create several tables of test data 2. Assign a set of auths to each table. Label all entries in the table with appropriate visibility expressions. Insure that some data in every table overlaps with data in other tables at common row/family/qualifier coordinates. Generate data like ITBLL so we can verify all data present later. 3. Mix the data from the different tables into a new common table 4. Verify for each set of auths defined in step #2 that all entries found in the source table can be found in the common table. Like the ITBLL verification step but done N times for each set of auths defined in step #2. 5. Choose one of the source tables. Get its set of auths. Perform a deletion with visibility expression from the common table using those auths. 6. Verify that no data in the common table with the auth set chosen in #5 remains. A simple row count with the set of auths chosen in #5 that should return 0. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11039) [VisibilityController] Integration test for labeled data set mixing and filtered excise
[ https://issues.apache.org/jira/browse/HBASE-11039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14009609#comment-14009609 ] ramkrishna.s.vasudevan commented on HBASE-11039: To keep the intent of this JIRA unpolluted may be we can raise another JIRA and attach the patch to that JIRA? [VisibilityController] Integration test for labeled data set mixing and filtered excise --- Key: HBASE-11039 URL: https://issues.apache.org/jira/browse/HBASE-11039 Project: HBase Issue Type: Test Affects Versions: 0.98.1 Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Priority: Critical Fix For: 0.99.0, 0.98.4 Attachments: HBASE-11039_1.patch Create an integration test for the VisibilityController that: 1. Create several tables of test data 2. Assign a set of auths to each table. Label all entries in the table with appropriate visibility expressions. Insure that some data in every table overlaps with data in other tables at common row/family/qualifier coordinates. Generate data like ITBLL so we can verify all data present later. 3. Mix the data from the different tables into a new common table 4. Verify for each set of auths defined in step #2 that all entries found in the source table can be found in the common table. Like the ITBLL verification step but done N times for each set of auths defined in step #2. 5. Choose one of the source tables. Get its set of auths. Perform a deletion with visibility expression from the common table using those auths. 6. Verify that no data in the common table with the auth set chosen in #5 remains. A simple row count with the set of auths chosen in #5 that should return 0. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11039) [VisibilityController] Integration test for labeled data set mixing and filtered excise
[ https://issues.apache.org/jira/browse/HBASE-11039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14009817#comment-14009817 ] Andrew Purtell commented on HBASE-11039: Why attach a patch here that doesn't satisfy what is required by the description. -1 [VisibilityController] Integration test for labeled data set mixing and filtered excise --- Key: HBASE-11039 URL: https://issues.apache.org/jira/browse/HBASE-11039 Project: HBase Issue Type: Test Affects Versions: 0.98.1 Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Priority: Critical Fix For: 0.99.0, 0.98.4 Attachments: HBASE-11039_1.patch Create an integration test for the VisibilityController that: 1. Create several tables of test data 2. Assign a set of auths to each table. Label all entries in the table with appropriate visibility expressions. Insure that some data in every table overlaps with data in other tables at common row/family/qualifier coordinates. Generate data like ITBLL so we can verify all data present later. 3. Mix the data from the different tables into a new common table 4. Verify for each set of auths defined in step #2 that all entries found in the source table can be found in the common table. Like the ITBLL verification step but done N times for each set of auths defined in step #2. 5. Choose one of the source tables. Get its set of auths. Perform a deletion with visibility expression from the common table using those auths. 6. Verify that no data in the common table with the auth set chosen in #5 remains. A simple row count with the set of auths chosen in #5 that should return 0. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11039) [VisibilityController] Integration test for labeled data set mixing and filtered excise
[ https://issues.apache.org/jira/browse/HBASE-11039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004793#comment-14004793 ] ramkrishna.s.vasudevan commented on HBASE-11039: Let me assign to me. Will discuss internally and come up with a patch. [VisibilityController] Integration test for labeled data set mixing and filtered excise --- Key: HBASE-11039 URL: https://issues.apache.org/jira/browse/HBASE-11039 Project: HBase Issue Type: Test Affects Versions: 0.98.1 Reporter: Andrew Purtell Priority: Critical Fix For: 0.99.0, 0.98.3 Create an integration test for the VisibilityController that: 1. Create several tables of test data 2. Assign a set of auths to each table. Label all entries in the table with appropriate visibility expressions. Insure that some data in every table overlaps with data in other tables at common row/family/qualifier coordinates. Generate data like ITBLL so we can verify all data present later. 3. Mix the data from the different tables into a new common table 4. Verify for each set of auths defined in step #2 that all entries found in the source table can be found in the common table. Like the ITBLL verification step but done N times for each set of auths defined in step #2. 5. Choose one of the source tables. Get its set of auths. Perform a deletion with visibility expression from the common table using those auths. 6. Verify that no data in the common table with the auth set chosen in #5 remains. A simple row count with the set of auths chosen in #5 that should return 0. -- This message was sent by Atlassian JIRA (v6.2#6252)
