See <https://builds.apache.org/job/HBase-TRUNK-security/17/changes>
Changes:
[karthik] HBASE-4463 [jira] Run more aggressive compactions during off peak
hours
Summary:
HBASE-4463 Run more aggressive compactions during off peak hours
Increases the compact selection ratio from 1.3 to 5 at off-peak hours. This
will help utilize the available iops and bandwidth to decrease average num of
files per store. Only one such aggressive compaction is queued per store at any
point.
The number of iops on the disk and the top of the rack bandwidth utilization at
off peak hours is much lower than at peak hours depending on the application
usage pattern. We can utilize this knowledge to improve the performance of the
HBase cluster by increasing the compact selection ratio to a much larger value
during off-peak hours than otherwise - increasing hbase.hstore.compaction.ratio
(1.2 default) to hbase.hstore.compaction.ratio.offpeak (5 default). This will
help reduce the average number of files per store.
Test Plan: Started running the unit tests.
Reviewers: JIRA, Kannan, nspiegelberg, mbautin, stack
Reviewed By: nspiegelberg
CC: nspiegelberg, tedyu, lhofhansl, Karthik
Differential Revision: 471
[karthik] [jira] [HBASE-4867] A tool to merge configuration files
Summary: With our cluster configuration setup it would be good to have a tool
that would merge HBase configuration files so that files appearing later in the
list would override properties specified in earlier files. This way we could
merge an application-specific configuration file with a cluster-specific
configuration file (with the latter overriding the former) and produce a single
HBase configuration file to install on the cluster.
Test Plan: Run the tool on two configuration files (common and
cluster-specific). Use the resulting configuration on a dev cluster.
Reviewers: todd, Karthik, tedyu, stack, JIRA
Reviewed By: Karthik
CC: Karthik, mbautin, todd
Differential Revision: 537
[todd] HBASE-4820. Distributed log splitting coding enhancement to make it
easier to understand, no semantics change. Contributed by Jimmy Xiang.
[dmeil] hbase-4906 book.xml - repatch of MR to RDBMS example. fixed typo
missed in first patch.
[dmeil] hbase-4906. book.xml. added MR summary to RDBMS section in MR chapter.
[dmeil] hbase-4905. book.xml (keyvalue types), ops_mgt.xml (metrics
corrections)
------------------------------------------
[...truncated 1898 lines...]
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.674 sec
Running org.apache.hadoop.hbase.TestKeyValue
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.017 sec
Running org.apache.hadoop.hbase.rest.TestSchemaResource
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.363 sec
Running org.apache.hadoop.hbase.rest.client.TestRemoteAdmin
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.186 sec
Running org.apache.hadoop.hbase.rest.client.TestRemoteTable
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.213 sec
Running org.apache.hadoop.hbase.rest.model.TestVersionModel
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.103 sec
Running org.apache.hadoop.hbase.rest.model.TestTableRegionModel
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.116 sec
Running org.apache.hadoop.hbase.rest.model.TestTableListModel
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.1 sec
Running org.apache.hadoop.hbase.rest.model.TestStorageClusterStatusModel
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.104 sec
Running org.apache.hadoop.hbase.rest.model.TestColumnSchemaModel
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.027 sec
Running org.apache.hadoop.hbase.rest.model.TestCellSetModel
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.054 sec
Running org.apache.hadoop.hbase.rest.model.TestTableSchemaModel
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.108 sec
Running org.apache.hadoop.hbase.rest.model.TestStorageClusterVersionModel
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.037 sec
Running org.apache.hadoop.hbase.rest.model.TestTableInfoModel
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.048 sec
Running org.apache.hadoop.hbase.rest.model.TestCellModel
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.057 sec
Running org.apache.hadoop.hbase.rest.model.TestRowModel
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.033 sec
Running org.apache.hadoop.hbase.rest.model.TestScannerModel
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.057 sec
Running org.apache.hadoop.hbase.rest.TestTableResource
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.945 sec
Running org.apache.hadoop.hbase.rest.TestMultiRowResource
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.305 sec
Running org.apache.hadoop.hbase.rest.TestGzipFilter
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.29 sec
Running org.apache.hadoop.hbase.rest.TestRowResource
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.493 sec
Running org.apache.hadoop.hbase.rest.TestScannerResource
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.266 sec
Running org.apache.hadoop.hbase.rest.TestVersionResource
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.823 sec
Running org.apache.hadoop.hbase.rest.TestStatusResource
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.865 sec
Running org.apache.hadoop.hbase.rest.TestTransform
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.481 sec
Running org.apache.hadoop.hbase.rest.TestScannersWithFilters
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.376 sec
Running org.apache.hadoop.hbase.metrics.TestMetricsMBeanBase
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.082 sec
Running org.apache.hadoop.hbase.io.hfile.TestHFileWriterV2
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.752 sec
Running org.apache.hadoop.hbase.io.hfile.TestHFile
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.479 sec
Running org.apache.hadoop.hbase.io.hfile.TestHFileBlockIndex
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.261 sec
Running org.apache.hadoop.hbase.io.hfile.TestHFileSeek
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.377 sec
Running org.apache.hadoop.hbase.io.hfile.TestLruBlockCache
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.662 sec
Running org.apache.hadoop.hbase.io.hfile.TestFixedFileTrailer
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.438 sec
Running org.apache.hadoop.hbase.io.hfile.TestCachedBlockQueue
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.084 sec
Running org.apache.hadoop.hbase.io.hfile.TestHFilePerformance
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.867 sec
Running org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.436 sec
Running org.apache.hadoop.hbase.io.hfile.TestReseekTo
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.541 sec
Running org.apache.hadoop.hbase.io.hfile.TestSeekTo
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.306 sec
Running org.apache.hadoop.hbase.io.hfile.TestBlockCacheColumnFamilySummary
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 sec
Running org.apache.hadoop.hbase.io.hfile.TestHFileBlock
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 23.787 sec
Running org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.268 sec
Running org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.507 sec
Running org.apache.hadoop.hbase.io.hfile.slab.TestSlab
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.069 sec
Running org.apache.hadoop.hbase.io.hfile.TestHFileReaderV1
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.294 sec
Running org.apache.hadoop.hbase.io.TestHalfStoreFileReader
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.747 sec
Running org.apache.hadoop.hbase.io.TestHbaseObjectWritable
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.352 sec
Running org.apache.hadoop.hbase.io.TestImmutableBytesWritable
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.063 sec
Running org.apache.hadoop.hbase.io.TestHeapSize
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.053 sec
Running org.apache.hadoop.hbase.TestRegionRebalancing
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.764 sec
Running org.apache.hadoop.hbase.mapred.TestTableInputFormat
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.659 sec
Running org.apache.hadoop.hbase.mapred.TestTableMapReduce
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 36.791 sec
Running org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.276 sec
Running
org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.1 sec
Running org.apache.hadoop.hbase.replication.TestReplicationSource
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.551 sec
Running org.apache.hadoop.hbase.replication.TestReplication
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 67.242 sec
Running org.apache.hadoop.hbase.replication.TestReplicationPeer
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.035 sec
Running org.apache.hadoop.hbase.replication.TestMultiSlaveReplication
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 53.58 sec
Running org.apache.hadoop.hbase.replication.TestMasterReplication
Running org.apache.hadoop.hbase.util.TestHBaseFsckComparator
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.07 sec
Running org.apache.hadoop.hbase.util.TestMergeTool
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 338.193 sec
Running org.apache.hadoop.hbase.util.TestFSTableDescriptors
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.749 sec
Running org.apache.hadoop.hbase.util.TestIdLock
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.149 sec
Running org.apache.hadoop.hbase.util.TestCompressionTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.295 sec
Running org.apache.hadoop.hbase.util.TestBytes
Tests run: 13, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.098 sec
Running org.apache.hadoop.hbase.util.TestSortedCopyOnWriteSet
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.014 sec
Running org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.186 sec
Running org.apache.hadoop.hbase.util.TestKeying
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.013 sec
Running org.apache.hadoop.hbase.util.TestByteBloomFilter
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.002 sec
Running org.apache.hadoop.hbase.util.TestRootPath
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec
Running org.apache.hadoop.hbase.util.TestMergeTable
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.628 sec
Running org.apache.hadoop.hbase.util.TestHBaseFsck
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.263 sec
Running org.apache.hadoop.hbase.util.TestFSUtils
Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 18.57 sec <<<
FAILURE!
Running org.apache.hadoop.hbase.util.TestPoolMap$TestRoundRobinPoolType
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.016 sec
Running org.apache.hadoop.hbase.util.TestPoolMap$TestThreadLocalPoolType
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.013 sec
Running org.apache.hadoop.hbase.util.TestPoolMap$TestReusablePoolType
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.004 sec
Running org.apache.hadoop.hbase.util.TestRegionSplitter
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.362 sec
Running org.apache.hadoop.hbase.util.TestThreads
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.01 sec
Running org.apache.hadoop.hbase.util.TestBase64
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.063 sec
Running org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildBase
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 30.4 sec
Running org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildOverlap
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 30.17 sec
Running org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildHole
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.441 sec
Running org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.004 sec
Running org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 sec
Running org.apache.hadoop.hbase.util.TestRegionSplitCalculator
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.044 sec
Running org.apache.hadoop.hbase.TestHRegionLocation
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.068 sec
Running org.apache.hadoop.hbase.migration.TestMigrationFrom090To092
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.184 sec
Running org.apache.hadoop.hbase.TestCompare
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.165 sec
Running org.apache.hadoop.hbase.TestInfoServers
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.247 sec
Running org.apache.hadoop.hbase.TestHServerAddress
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.01 sec
Running org.apache.hadoop.hbase.TestServerName
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.065 sec
Results :
Failed tests:
testcomputeHDFSBlocksDistribution(org.apache.hadoop.hbase.util.TestFSUtils)
Tests run: 1203, Failures: 1, Errors: 0, Skipped: 9
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1:56:51.003s
[INFO] Finished at: Thu Dec 01 06:59:14 UTC 2011
[INFO] Final Memory: 24M/748M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-surefire-plugin:2.11-TRUNK-HBASE-2:test
(default-test) on project hbase: There are test failures.
[ERROR]
[ERROR] Please refer to
/x1/jenkins/jenkins-slave/workspace/HBase-TRUNK-security/trunk/target/surefire-reports
for the individual test results.
[ERROR] -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal
org.apache.maven.plugins:maven-surefire-plugin:2.11-TRUNK-HBASE-2:test
(default-test) on project hbase: There are test failures.
Please refer to
/x1/jenkins/jenkins-slave/workspace/HBase-TRUNK-security/trunk/target/surefire-reports
for the individual test results.
at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:213)
at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at
org.apache.maven.lifecycle.internal.MojoExecutor.executeForkedExecutions(MojoExecutor.java:365)
at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:199)
at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
at
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
at
org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
at
org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:319)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290)
at
org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230)
at
org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
at
org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)
Caused by: org.apache.maven.plugin.MojoFailureException: There are test
failures.
Please refer to
/x1/jenkins/jenkins-slave/workspace/HBase-TRUNK-security/trunk/target/surefire-reports
for the individual test results.
at
org.apache.maven.plugin.surefire.SurefireHelper.reportExecution(SurefireHelper.java:87)
at
org.apache.maven.plugin.surefire.SurefirePlugin.writeSummary(SurefirePlugin.java:641)
at
org.apache.maven.plugin.surefire.SurefirePlugin.handleSummary(SurefirePlugin.java:615)
at
org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:137)
at
org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:98)
at
org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)
... 23 more
[ERROR]
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
Build step 'Invoke top-level Maven targets' marked build as failure
[locks-and-latches] Releasing all the locks
[locks-and-latches] All the locks released
Archiving artifacts
Recording test results
Updating HBASE-4906
Updating HBASE-4905
Updating HBASE-4463
Updating HBASE-4820
Updating HBASE-4867
