I ran TestAccessController locally in a loop 100 times without being able to reproduce the failure. Unfortunately surefire did not capture an output log from the test, maybe because it failed in the @BeforeClass setup method? What's weird is that, looking at the code, the first grant() operation succeeded. Only the second failed with an AccessDeniedException.
Looking at TableAuthManager.refreshCacheFromWritable() and TableAuthManager.updateGlobalCache(), it looks like we do have a race condition, where the global caches are initially cleared and then rebuilt, so any authorization checks that come in in the interim will fail. On Thu, Sep 20, 2012 at 4:26 PM, Apache Jenkins Server <[email protected]> wrote: > See <https://builds.apache.org/job/HBase-TRUNK/3360/changes> > > Changes: > > [jdcryans] HBASE-6847 HBASE-6649 broke replication (Devaraj Das via JD) > > ------------------------------------------ > [...truncated 2742 lines...] > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.695 sec > Running org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 44.943 sec > Running org.apache.hadoop.hbase.filter.TestFilterWrapper > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.207 sec > Running org.apache.hadoop.hbase.filter.TestColumnRangeFilter > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.468 sec > Running org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol > Tests run: 45, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.545 sec > Running org.apache.hadoop.hbase.regionserver.wal.TestLogRolling > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 102.532 sec > Running > org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.411 sec > Running org.apache.hadoop.hbase.coprocessor.TestClassLoading > Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 36.999 sec > Running > org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.315 sec > Running org.apache.hadoop.hbase.coprocessor.TestRowProcessorEndpoint > Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 32.452 sec > Running org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.724 sec > Running org.apache.hadoop.hbase.coprocessor.TestWALObserver > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.397 sec > Running org.apache.hadoop.hbase.coprocessor.TestMasterObserver > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.421 sec > Running > org.apache.hadoop.hbase.coprocessor.example.TestZooKeeperScanPolicyObserver > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.855 sec > Running org.apache.hadoop.hbase.coprocessor.example.TestRowCountEndpoint > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.469 sec > Running > org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 23.971 sec > Running > org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.081 sec > Running org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.854 sec > Running org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface > Running org.apache.hadoop.hbase.TestGlobalMemStoreSize > Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 33.787 sec > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.446 sec > Running org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit > Tests run: 30, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 244.026 sec > Running org.apache.hadoop.hbase.mapred.TestTableInputFormat > Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.662 sec > Running org.apache.hadoop.hbase.mapreduce.TestHLogRecordReader > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.42 sec > Running org.apache.hadoop.hbase.fs.TestBlockReorder > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 32.572 sec > Running org.apache.hadoop.hbase.mapred.TestTableMapReduce > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 103.885 sec > Running org.apache.hadoop.hbase.mapreduce.TestWALPlayer > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 58.459 sec > Running > org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery > Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 97.822 sec > Running org.apache.hadoop.hbase.mapreduce.TestTableMapReduce > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 118.348 sec > Running org.apache.hadoop.hbase.mapreduce.TestRowCounter > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 102.634 sec > Running org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 53.322 sec > Running org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 141.1 sec > Running org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles > Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.509 sec > Running org.apache.hadoop.hbase.mapreduce.TestImportTsv > Tests run: 13, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 207.405 sec > Running org.apache.hadoop.hbase.trace.TestHTraceHooks > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.521 sec > Running org.apache.hadoop.hbase.thrift2.TestThriftHBaseServiceHandler > Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.335 sec > Running org.apache.hadoop.hbase.mapreduce.TestImportExport > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 58.392 sec > Running org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.425 sec > Running org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat > Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 306.335 sec > Running org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.438 sec > Running org.apache.hadoop.hbase.catalog.TestMetaReaderEditor > Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.888 sec > Running org.apache.hadoop.hbase.client.TestFromClientSide3 > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 33.125 sec > Running org.apache.hadoop.hbase.TestAcidGuarantees > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 105.967 sec > Running org.apache.hadoop.hbase.client.TestTimestampsFilter > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.979 sec > Running org.apache.hadoop.hbase.catalog.TestMetaMigrationConvertingToPB > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 75.335 sec > Running org.apache.hadoop.hbase.client.TestScannersFromClientSide > Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.919 sec > Running org.apache.hadoop.hbase.client.TestHCM > Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.328 sec > Running org.apache.hadoop.hbase.client.TestShell > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 90.334 sec > Running org.apache.hadoop.hbase.client.TestScannerTimeout > Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 44.174 sec > Running org.apache.hadoop.hbase.client.TestMetaScanner > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.569 sec > Running org.apache.hadoop.hbase.client.TestHTableUtil > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.911 sec > Running org.apache.hadoop.hbase.client.TestHTablePool$TestHTableReusablePool > Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.16 sec > Running > org.apache.hadoop.hbase.client.TestHTablePool$TestHTableThreadLocalPool > Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.423 sec > Running org.apache.hadoop.hbase.client.TestFromClientSide > Tests run: 59, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 168.539 sec > Running org.apache.hadoop.hbase.client.replication.TestReplicationAdmin > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.912 sec > Running org.apache.hadoop.hbase.client.TestMultipleTimestamps > Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.24 sec > Running org.apache.hadoop.hbase.client.TestMultiParallel > Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 39.563 sec > Running org.apache.hadoop.hbase.constraint.TestConstraint > Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.283 sec > Running org.apache.hadoop.hbase.client.TestFromClientSideWithCoprocessor > Tests run: 59, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 168.792 sec > Running org.apache.hadoop.hbase.rest.TestGzipFilter > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.658 sec > Running org.apache.hadoop.hbase.rest.TestMultiRowResource > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.16 sec > Running org.apache.hadoop.hbase.rest.TestScannersWithFilters > Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.708 sec > Running org.apache.hadoop.hbase.rest.TestRowResource > Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 20.01 sec > Running org.apache.hadoop.hbase.rest.client.TestRemoteAdmin > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.584 sec > Running org.apache.hadoop.hbase.rest.client.TestRemoteTable > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.353 sec > Running org.apache.hadoop.hbase.rest.TestVersionResource > Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.435 sec > Running org.apache.hadoop.hbase.client.TestAdmin > Tests run: 41, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 121.647 sec > Running org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan > Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 597.57 sec > Running org.apache.hadoop.hbase.rest.TestStatusResource > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.471 sec > Running org.apache.hadoop.hbase.rest.TestScannerResource > Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.803 sec > Running org.apache.hadoop.hbase.rest.TestSchemaResource > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.203 sec > Running org.apache.hadoop.hbase.rest.TestTableResource > Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 23.552 sec > Running org.apache.hadoop.hbase.io.encoding.TestLoadAndSwitchEncodeOnDisk > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 27.094 sec > Running org.apache.hadoop.hbase.io.encoding.TestEncodedSeekers > Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 33.397 sec > Running org.apache.hadoop.hbase.thrift.TestThriftServerCmdLine > Tests run: 32, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 56.158 sec > Running org.apache.hadoop.hbase.io.encoding.TestBufferedDataBlockEncoder > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.543 sec > Running org.apache.hadoop.hbase.thrift.TestThriftServer > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 83.227 sec > Running org.apache.hadoop.hbase.io.hfile.TestHFileSeek > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.42 sec > Running org.apache.hadoop.hbase.io.encoding.TestUpgradeFromHFileV1ToEncoding > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 36.13 sec > Running org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache > Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.942 sec > Running org.apache.hadoop.hbase.io.hfile.TestHFileBlockIndex > Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.041 sec > Running org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache > Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.604 sec > Running org.apache.hadoop.hbase.io.hfile.TestHFilePerformance > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.782 sec > Running org.apache.hadoop.hbase.io.encoding.TestDataBlockEncoders > Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 117.202 sec > Running org.apache.hadoop.hbase.io.hfile.TestForceCacheImportantBlocks > Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 56.93 sec > Running org.apache.hadoop.hbase.io.hfile.TestScannerSelectionUsingTTL > Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 61.401 sec > Running org.apache.hadoop.hbase.io.encoding.TestChangingEncoding > Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 180.254 sec > Running org.apache.hadoop.hbase.TestClusterBootOrder > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 31.025 sec > Running org.apache.hadoop.hbase.io.hfile.TestHFileBlock > Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 48.861 sec > Running > org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.328 sec > Running org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite > Tests run: 36, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 44.081 sec > Running org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink > Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.071 sec > Running org.apache.hadoop.hbase.replication.TestReplicationZookeeper > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.713 sec > Running org.apache.hadoop.hbase.replication.TestReplicationSource > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.924 sec > Running org.apache.hadoop.hbase.ipc.TestDelayedRpc > Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.357 sec > Running org.apache.hadoop.hbase.ipc.TestProtoBufRpc > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.704 sec > Running org.apache.hadoop.hbase.ipc.TestHBaseClient > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.157 sec > Running org.apache.hadoop.hbase.replication.TestMultiSlaveReplication > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 39.428 sec > Running org.apache.hadoop.hbase.TestZooKeeper > Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 27.73 sec > Running org.apache.hadoop.hbase.replication.TestMasterReplication > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 63.611 sec > Running org.apache.hadoop.hbase.TestHBaseTestingUtility > Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 56.325 sec > > Results : > > Tests in error: > org.apache.hadoop.hbase.security.access.TestAccessController: Error calling > method AccessControlService.grant > > Tests run: 1126, Failures: 0, Errors: 1, Skipped: 10 > > [INFO] > [INFO] > ------------------------------------------------------------------------ > [INFO] Skipping HBase > [INFO] This project has been banned from the build due to previous failures. > [INFO] > ------------------------------------------------------------------------ > [INFO] > [INFO] > ------------------------------------------------------------------------ > [INFO] Skipping HBase > [INFO] This project has been banned from the build due to previous failures. > [INFO] > ------------------------------------------------------------------------ > [INFO] > ------------------------------------------------------------------------ > [INFO] Reactor Summary: > [INFO] > [INFO] HBase ............................................. SUCCESS [6.029s] > [INFO] HBase - Common .................................... SUCCESS [7.146s] > [INFO] HBase - Hadoop Compatibility ...................... SUCCESS [0.645s] > [INFO] HBase - Hadoop One Compatibility .................. SUCCESS [1.274s] > [INFO] HBase - Server .................................... FAILURE > [47:21.690s] > [INFO] HBase - Hadoop Two Compatibility .................. SKIPPED > [INFO] HBase - Integration Tests ......................... SKIPPED > [INFO] > ------------------------------------------------------------------------ > [INFO] BUILD FAILURE > [INFO] > ------------------------------------------------------------------------ > [INFO] Total time: 47:37.431s > [INFO] Finished at: Thu Sep 20 20:26:00 UTC 2012 > [INFO] Final Memory: 74M/823M > [INFO] > ------------------------------------------------------------------------ > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-surefire-plugin:2.12-TRUNK-HBASE-2:test > (secondPartTestsExecution) on project hbase-server: Failure or timeout -> > [Help 1] > [ERROR] > [ERROR] To see the full stack trace of the errors, re-run Maven with the -e > switch. > [ERROR] Re-run Maven using the -X switch to enable full debug logging. > [ERROR] > [ERROR] For more information about the errors and possible solutions, please > read the following articles: > [ERROR] [Help 1] > http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException > [ERROR] > [ERROR] After correcting the problems, you can resume the build with the > command > [ERROR] mvn <goals> -rf :hbase-server > Build step 'Invoke top-level Maven targets' marked build as failure > Archiving artifacts > Recording test results > Updating HBASE-6649 > Updating HBASE-6847
