See <https://builds.apache.org/job/HBase-TRUNK-security/90/changes>
Changes: [mbautin] [jira] [HBASE-4218] HFile data block encoding framework and delta encoding implementation (Jacek Midgal, Mikhail Bautin) Summary: Adding a framework that allows to "encode" keys in an HFile data block. We support two modes of encoding: (1) both on disk and in cache, and (2) in cache only. This is distinct from compression that is already being done in HBase, e.g. GZ or LZO. When data block encoding is enabled, we store blocks in cache in an uncompressed but encoded form. This allows to fit more blocks in cache and reduce the number of disk reads. The most common example of data block encoding is delta encoding, where we take advantage of the fact that HFile keys are sorted and share a lot of common prefixes, and only store the delta between each pair of consecutive keys. Initial encoding algorithms implemented are DIFF, FAST_DIFF, and PREFIX. This is based on the delta encoding patch developed by Jacek Midgal during his 2011 summer internship at Facebook. The original patch is available here: https://reviews.apache.org/r/2308/diff/. Test Plan: Unit tests. Distributed load test on a five-node cluster. Reviewers: JIRA, tedyu, stack, nspiegelberg, Kannan Reviewed By: Kannan CC: tedyu, todd, mbautin, stack, Kannan, mcorgan, gqchen Differential Revision: https://reviews.facebook.net/D447 [mbautin] adding mbautin as a developer to pom.xml [stack] HBASE-5278 HBase shell script refers to removed 'migrate' functionality [mbautin] HBASE-5230 : ensure that compactions do not cache-on-write data blocks ------------------------------------------ [...truncated 2013 lines...] Running org.apache.hadoop.hbase.mapreduce.TestImportTsv Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 84.563 sec Running org.apache.hadoop.hbase.mapreduce.TestTableMapReduce Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 99.852 sec Running org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 33.386 sec Running org.apache.hadoop.hbase.regionserver.TestStore Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.121 sec Running org.apache.hadoop.hbase.regionserver.wal.TestHLogBench Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.065 sec Running org.apache.hadoop.hbase.regionserver.wal.TestHLog Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 59.218 sec Running org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 202.761 sec Running org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 150.816 sec Running org.apache.hadoop.hbase.regionserver.wal.TestWALReplay Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.865 sec Running org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 483.634 sec Running org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit Tests run: 30, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 242.603 sec Running org.apache.hadoop.hbase.regionserver.TestRegionServerMetrics Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 32.715 sec Running org.apache.hadoop.hbase.regionserver.TestAtomicOperation Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.761 sec Running org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.28 sec Running org.apache.hadoop.hbase.regionserver.TestParallelPut Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.21 sec Running org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.122 sec Running org.apache.hadoop.hbase.regionserver.TestRpcMetrics Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.601 sec Running org.apache.hadoop.hbase.regionserver.handler.TestCloseRegionHandler Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.918 sec Running org.apache.hadoop.hbase.regionserver.TestSeekOptimizations Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.38 sec Running org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.732 sec Running org.apache.hadoop.hbase.regionserver.TestBlocksRead Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.741 sec Running org.apache.hadoop.hbase.regionserver.TestStoreScanner Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.087 sec Running org.apache.hadoop.hbase.regionserver.TestSplitLogWorker Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.995 sec Running org.apache.hadoop.hbase.regionserver.metrics.TestSchemaMetrics Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.823 sec Running org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.984 sec Running org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 39.068 sec Running org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 45.063 sec Running org.apache.hadoop.hbase.regionserver.TestMasterAddressManager Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.476 sec Running org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 27.924 sec Running org.apache.hadoop.hbase.regionserver.TestHRegion Tests run: 61, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 23.357 sec Running org.apache.hadoop.hbase.ipc.TestDelayedRpc Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.2 sec Running org.apache.hadoop.hbase.TestGlobalMemStoreSize Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.162 sec Running org.apache.hadoop.hbase.TestFullLogReconstruction Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 37.647 sec Running org.apache.hadoop.hbase.constraint.TestConstraint Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.793 sec Running org.apache.hadoop.hbase.rest.TestSchemaResource Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.142 sec Running org.apache.hadoop.hbase.rest.client.TestRemoteAdmin Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.441 sec Running org.apache.hadoop.hbase.rest.client.TestRemoteTable Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.361 sec Running org.apache.hadoop.hbase.rest.TestMultiRowResource Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.908 sec Running org.apache.hadoop.hbase.rest.TestTableResource Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.342 sec Running org.apache.hadoop.hbase.rest.TestGzipFilter Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.667 sec Running org.apache.hadoop.hbase.rest.TestRowResource Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.718 sec Running org.apache.hadoop.hbase.rest.TestVersionResource Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.15 sec Running org.apache.hadoop.hbase.rest.TestStatusResource Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.446 sec Running org.apache.hadoop.hbase.rest.TestScannerResource Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.12 sec Running org.apache.hadoop.hbase.metrics.TestMetricsMBeanBase Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.19 sec Running org.apache.hadoop.hbase.io.hfile.TestHFileBlockIndex Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.327 sec Running org.apache.hadoop.hbase.rest.TestScannersWithFilters Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.905 sec Running org.apache.hadoop.hbase.io.hfile.TestLruBlockCache Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.827 sec Running org.apache.hadoop.hbase.io.hfile.TestHFileSeek Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.15 sec Running org.apache.hadoop.hbase.io.hfile.TestFixedFileTrailer Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.393 sec Running org.apache.hadoop.hbase.io.hfile.TestHFilePerformance Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.164 sec Running org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite Tests run: 36, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 30.041 sec Running org.apache.hadoop.hbase.io.hfile.TestHFileBlock Tests run: 16, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 38.599 sec <<< FAILURE! Running org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.753 sec Running org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.444 sec Running org.apache.hadoop.hbase.regionserver.wal.TestLogRolling Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 429.193 sec Running org.apache.hadoop.hbase.io.encoding.TestUpgradeFromHFileV1ToEncoding Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 94.539 sec Running org.apache.hadoop.hbase.io.encoding.TestLoadAndSwitchEncodeOnDisk Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 24.131 sec Running org.apache.hadoop.hbase.TestRegionRebalancing Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 24.971 sec Running org.apache.hadoop.hbase.io.encoding.TestDataBlockEncoders Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 97.429 sec Running org.apache.hadoop.hbase.mapred.TestTableInputFormat Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 20.444 sec Running org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.7 sec Running org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.815 sec Running org.apache.hadoop.hbase.replication.TestReplicationSource Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.232 sec Running org.apache.hadoop.hbase.mapred.TestTableMapReduce Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 60.175 sec Running org.apache.hadoop.hbase.replication.TestReplicationPeer Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.515 sec Running org.apache.hadoop.hbase.io.encoding.TestChangingEncoding Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 244.871 sec Running org.apache.hadoop.hbase.replication.TestMultiSlaveReplication Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 39.162 sec Running org.apache.hadoop.hbase.replication.TestReplication Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 112.908 sec Running org.apache.hadoop.hbase.util.TestFSTableDescriptors Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.728 sec Running org.apache.hadoop.hbase.util.TestIdLock Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.331 sec Running org.apache.hadoop.hbase.replication.TestMasterReplication Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 60.792 sec Running org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.286 sec Running org.apache.hadoop.hbase.util.TestMergeTable Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 27.107 sec Running org.apache.hadoop.hbase.util.TestMiniClusterLoadSequential Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 87.379 sec Running org.apache.hadoop.hbase.util.TestMiniClusterLoadParallel Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 89.054 sec Running org.apache.hadoop.hbase.util.TestHBaseFsck Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 38.444 sec Running org.apache.hadoop.hbase.util.TestPoolMap$TestRoundRobinPoolType Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.005 sec Running org.apache.hadoop.hbase.util.TestPoolMap$TestThreadLocalPoolType Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 sec Running org.apache.hadoop.hbase.util.TestPoolMap$TestReusablePoolType Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.02 sec Running org.apache.hadoop.hbase.util.TestFSUtils Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 20.034 sec Running org.apache.hadoop.hbase.util.TestRegionSplitter Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 33.044 sec Running org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildBase Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.83 sec Running org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildOverlap Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.902 sec Running org.apache.hadoop.hbase.util.TestMiniClusterLoadEncoded Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 66.507 sec Running org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildHole Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.606 sec Running org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.207 sec Running org.apache.hadoop.hbase.TestInfoServers Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.204 sec Running org.apache.hadoop.hbase.util.TestMergeTool Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 331.713 sec Results : Tests in error: testInstantSchemaChangeBlocksDuringLoadBalancerRun(org.apache.hadoop.hbase.client.TestInstantSchemaChange): test timed out after 10000 milliseconds testInstantSchemaJanitor(org.apache.hadoop.hbase.client.TestInstantSchemaChange): Shutting down testPreviousOffset[1](org.apache.hadoop.hbase.io.hfile.TestHFileBlock) testConcurrentReading[1](org.apache.hadoop.hbase.io.hfile.TestHFileBlock): unable to create new native thread Tests run: 877, Failures: 0, Errors: 4, Skipped: 10 [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 43:48.012s [INFO] Finished at: Thu Jan 26 05:45:35 UTC 2012 [INFO] Final Memory: 51M/1020M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12-TRUNK-HBASE-2:test (secondPartTestsExecution) on project hbase: There are test failures. [ERROR] [ERROR] Please refer to /x1/jenkins/jenkins-slave/workspace/HBase-TRUNK-security/trunk/target/surefire-reports for the individual test results. [ERROR] -> [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12-TRUNK-HBASE-2:test (secondPartTestsExecution) on project hbase: There are test failures. Please refer to /x1/jenkins/jenkins-slave/workspace/HBase-TRUNK-security/trunk/target/surefire-reports for the individual test results. at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:213) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145) at org.apache.maven.lifecycle.internal.MojoExecutor.executeForkedExecutions(MojoExecutor.java:365) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:199) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59) at org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161) at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:319) at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156) at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537) at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196) at org.apache.maven.cli.MavenCli.main(MavenCli.java:141) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290) at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409) at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352) Caused by: org.apache.maven.plugin.MojoFailureException: There are test failures. Please refer to /x1/jenkins/jenkins-slave/workspace/HBase-TRUNK-security/trunk/target/surefire-reports for the individual test results. at org.apache.maven.plugin.surefire.SurefireHelper.reportExecution(SurefireHelper.java:87) at org.apache.maven.plugin.surefire.SurefirePlugin.writeSummary(SurefirePlugin.java:651) at org.apache.maven.plugin.surefire.SurefirePlugin.handleSummary(SurefirePlugin.java:625) at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:137) at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:98) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209) ... 23 more [ERROR] [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException Build step 'Invoke top-level Maven targets' marked build as failure Archiving artifacts Updating HBASE-4218 Updating HBASE-5278 Updating HBASE-5230 Recording test results
