See <https://builds.apache.org/job/HBase-TRUNK/2674/changes>
Changes: [larsh] HBASE-5526 ^Cnfigurable file and directory based umask (Jesse Yates) [mbautin] [jira] [HBASE-5074] Support checksums in HBase block cache Author: Dhruba Summary: HFile is enhanced to store a checksum for each block. HDFS checksum verification is avoided while reading data into the block cache. On a checksum verification failure, we retry the file system read request with hdfs checksums switched on (thanks Todd). I have a benchmark that shows that it reduces iops on the disk by about 40%. In this experiment, the entire memory on the regionserver is allocated to the regionserver's jvm and the OS buffer cache size is negligible. I also measured negligible (<5%) additional cpu usage while using hbase-level checksums. The salient points of this patch: 1. Each hfile's trailer used to have a 4 byte version number. I enhanced this so that these 4 bytes can be interpreted as a (major version number, minor version). Pre-existing hfiles have a minor version of 0. The new hfile format has a minor version of 1 (thanks Mikhail). The hfile major version remains unchanged at 2. The reason I did not introduce a new major version number is because the code changes needed to store/read checksums do not differ much from existing V2 writers/readers. 2. Introduced a HFileSystem object which is a encapsulates the FileSystem objects needed to access data from hfiles and hlogs. HDFS FileSystem objects already had the ability to switch off checksum verifications for reads. 3. The majority of the code changes are located in hbase.io.hfie package. The retry of a read on an initial checksum failure occurs inside the hbase.io.hfile package itself. The code changes to hbase.regionserver package are minor. 4. The format of a hfileblock is the header followed by the data followed by the checksum(s). Each 16 K (configurable) size of data has a 4 byte checksum. The hfileblock header has two additional fields: a 4 byte value to store the bytesPerChecksum and a 4 byte value to store the size of the user data (excluding the checksum data). This is well explained in the associated javadocs. 5. I added a test to test backward compatibility. I will be writing more unit tests that triggers checksum verification failures aggressively. I have left a few redundant log messages in the code (just for easier debugging) and will remove them in later stage of this patch. I will also be adding metrics on number of checksum verification failures/success in a later version of this diff. 6. By default, hbase-level checksums are switched on and hdfs level checksums are switched off for hfile-reads. No changes to Hlog code path here. Test Plan: The default setting is to switch on hbase checksums for hfile-reads, thus all existing tests actually validate the new code pieces. I will be writing more unit tests for triggering checksum verification failures. Reviewers: mbautin Reviewed By: mbautin CC: JIRA, tedyu, mbautin, dhruba, todd, stack Differential Revision: https://reviews.facebook.net/D1521 [larsh] HBASE-5529 MR test failures becuase MALLOC_ARENA_MAX is not set (Gregory Chanan) [tedyu] HBASE-5515 Add a processRow API that supports atomic multiple reads and writes on a row (Scott Chen) [tedyu] HBASE-5514 Compile against hadoop 0.24-SNAPSHOT (Mingjie) [stack] HBASE-4890 fix possible NPE in HConnectionManager [apurtell] HBASE-5480. Fixups to MultithreadedTableMapper for Hadoop 0.23 ------------------------------------------ [...truncated 2061 lines...] Tests run: 45, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.779 sec Running org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.463 sec Running org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.938 sec Running org.apache.hadoop.hbase.coprocessor.TestMasterObserver Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 37.142 sec Running org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint Tests run: 3, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 40.03 sec Running org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit Tests run: 30, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 262.841 sec Running org.apache.hadoop.hbase.coprocessor.TestWALObserver Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.248 sec Running org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.471 sec Running org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.956 sec Running org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 32.433 sec Running org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.547 sec Running org.apache.hadoop.hbase.TestGlobalMemStoreSize Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.337 sec Running org.apache.hadoop.hbase.mapred.TestTableInputFormat Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 21.776 sec Running org.apache.hadoop.hbase.mapred.TestTableMapReduce Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 68.947 sec Running org.apache.hadoop.hbase.mapreduce.TestMulitthreadedTableMapper Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 73.105 sec Running org.apache.hadoop.hbase.mapreduce.TestTableMapReduce Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 93.577 sec Running org.apache.hadoop.hbase.regionserver.wal.TestLogRolling Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 448.278 sec Running org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 102.89 sec Running org.apache.hadoop.hbase.mapreduce.TestImportTsv Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 83.198 sec Running org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 38.835 sec Running org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 35.613 sec Running org.apache.hadoop.hbase.mapreduce.TestImportExport Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 48.089 sec Running org.apache.hadoop.hbase.thrift2.TestThriftHBaseServiceHandler Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.53 sec Running org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.332 sec Running org.apache.hadoop.hbase.catalog.TestMetaReaderEditor Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 21.387 sec Running org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 219.466 sec Running org.apache.hadoop.hbase.catalog.TestCatalogTracker Tests run: 9, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 6.438 sec Running org.apache.hadoop.hbase.TestAcidGuarantees Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 97.429 sec Running org.apache.hadoop.hbase.client.TestTimestampsFilter Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.322 sec Running org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 80.304 sec Running org.apache.hadoop.hbase.client.TestShell Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 94.089 sec Running org.apache.hadoop.hbase.client.TestFromClientSide Tests run: 54, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 194.092 sec Running org.apache.hadoop.hbase.client.TestHCM Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.065 sec Running org.apache.hadoop.hbase.client.TestInstantSchemaChangeSplit Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 133.429 sec Running org.apache.hadoop.hbase.client.TestInstantSchemaChange Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 221.637 sec Running org.apache.hadoop.hbase.client.TestScannerTimeout Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 52.814 sec Running org.apache.hadoop.hbase.client.TestMetaScanner Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.418 sec Running org.apache.hadoop.hbase.client.TestHTableUtil Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.432 sec Running org.apache.hadoop.hbase.client.TestHTablePool$TestHTableReusablePool Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.13 sec Running org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.97 sec Running org.apache.hadoop.hbase.client.TestHTablePool$TestHTableThreadLocalPool Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 23.873 sec Running org.apache.hadoop.hbase.client.replication.TestReplicationAdmin Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.883 sec Running org.apache.hadoop.hbase.client.TestInstantSchemaChangeFailover Tests run: 4, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 115.879 sec Running org.apache.hadoop.hbase.client.TestMultipleTimestamps Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.868 sec Running org.apache.hadoop.hbase.metrics.TestMetricsMBeanBase Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.142 sec Running org.apache.hadoop.hbase.client.TestMultiParallel Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 36.569 sec Running org.apache.hadoop.hbase.rest.TestGzipFilter Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.538 sec Running org.apache.hadoop.hbase.constraint.TestConstraint Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 45.868 sec Running org.apache.hadoop.hbase.rest.TestMultiRowResource Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 31.164 sec Running org.apache.hadoop.hbase.rest.TestScannersWithFilters Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.517 sec Running org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 593.27 sec Running org.apache.hadoop.hbase.rest.TestRowResource Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.277 sec Running org.apache.hadoop.hbase.rest.client.TestRemoteAdmin Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.565 sec Running org.apache.hadoop.hbase.rest.TestVersionResource Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.896 sec Running org.apache.hadoop.hbase.rest.client.TestRemoteTable Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.342 sec Running org.apache.hadoop.hbase.rest.TestStatusResource Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.225 sec Running org.apache.hadoop.hbase.rest.TestTableResource Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 21.422 sec Running org.apache.hadoop.hbase.rest.TestScannerResource Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.546 sec Running org.apache.hadoop.hbase.rest.TestSchemaResource Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 20.922 sec Running org.apache.hadoop.hbase.io.encoding.TestLoadAndSwitchEncodeOnDisk Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 24.076 sec Running org.apache.hadoop.hbase.thrift.TestThriftServer Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 45.539 sec Running org.apache.hadoop.hbase.client.TestAdmin Tests run: 35, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 190.621 sec Running org.apache.hadoop.hbase.io.encoding.TestUpgradeFromHFileV1ToEncoding Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 84.824 sec Running org.apache.hadoop.hbase.io.hfile.TestHFileSeek Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.269 sec Running org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.616 sec Running org.apache.hadoop.hbase.io.encoding.TestDataBlockEncoders Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 148.743 sec Running org.apache.hadoop.hbase.io.hfile.TestChecksum Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.914 sec Running org.apache.hadoop.hbase.io.hfile.TestHFileBlockIndex Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.275 sec Running org.apache.hadoop.hbase.io.hfile.TestLruBlockCache Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.755 sec Running org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 26.035 sec Running org.apache.hadoop.hbase.io.hfile.TestFixedFileTrailer Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.642 sec Running org.apache.hadoop.hbase.io.hfile.TestHFileBlockCompatibility Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.234 sec Running org.apache.hadoop.hbase.io.hfile.TestHFilePerformance Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.642 sec Running org.apache.hadoop.hbase.io.hfile.TestForceCacheImportantBlocks Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 50.722 sec Running org.apache.hadoop.hbase.io.hfile.TestHFileBlock Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 48.928 sec Running org.apache.hadoop.hbase.io.encoding.TestChangingEncoding Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 246.197 sec Running org.apache.hadoop.hbase.avro.TestAvroServer Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.179 sec Running org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.144 sec Running org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite Tests run: 36, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 32.644 sec Running org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.509 sec Running org.apache.hadoop.hbase.replication.TestReplicationSource Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.768 sec Running org.apache.hadoop.hbase.replication.TestMultiSlaveReplication Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 40.729 sec Running org.apache.hadoop.hbase.replication.TestReplicationPeer Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.635 sec Running org.apache.hadoop.hbase.ipc.TestDelayedRpc Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.669 sec Running org.apache.hadoop.hbase.replication.TestMasterReplication Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 61.942 sec Running org.apache.hadoop.hbase.TestZooKeeper Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 51.945 sec Running org.apache.hadoop.hbase.thrift.TestThriftServerCmdLine Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 395.601 sec Running org.apache.hadoop.hbase.TestHBaseTestingUtility Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 62.131 sec Running org.apache.hadoop.hbase.replication.TestReplication Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 112.993 sec Results : Tests in error: testThreeRSAbort(org.apache.hadoop.hbase.master.TestDistributedLogSplitting): org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=10, exceptions:(..) Tests run: 901, Failures: 0, Errors: 1, Skipped: 10 [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 38:29.309s [INFO] Finished at: Fri Mar 09 00:42:55 UTC 2012 [INFO] Final Memory: 44M/439M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12-TRUNK-HBASE-2:test (secondPartTestsExecution) on project hbase: There are test failures. [ERROR] [ERROR] Please refer to <https://builds.apache.org/job/HBase-TRUNK/ws/trunk/target/surefire-reports> for the individual test results. [ERROR] -> [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12-TRUNK-HBASE-2:test (secondPartTestsExecution) on project hbase: There are test failures. Please refer to <https://builds.apache.org/job/HBase-TRUNK/ws/trunk/target/surefire-reports> for the individual test results. at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:213) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145) at org.apache.maven.lifecycle.internal.MojoExecutor.executeForkedExecutions(MojoExecutor.java:365) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:199) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59) at org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161) at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:319) at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156) at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537) at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196) at org.apache.maven.cli.MavenCli.main(MavenCli.java:141) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290) at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409) at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352) Caused by: org.apache.maven.plugin.MojoFailureException: There are test failures. Please refer to <https://builds.apache.org/job/HBase-TRUNK/ws/trunk/target/surefire-reports> for the individual test results. at org.apache.maven.plugin.surefire.SurefireHelper.reportExecution(SurefireHelper.java:87) at org.apache.maven.plugin.surefire.SurefirePlugin.writeSummary(SurefirePlugin.java:651) at org.apache.maven.plugin.surefire.SurefirePlugin.handleSummary(SurefirePlugin.java:625) at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:137) at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:98) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209) ... 23 more [ERROR] [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException Build step 'Invoke top-level Maven targets' marked build as failure Archiving artifacts Updating HBASE-5515 Updating HBASE-5074 Updating HBASE-4890 Updating HBASE-5480 Updating HBASE-5526 Updating HBASE-5514 Updating HBASE-5529 Recording test results
