[
https://issues.apache.org/jira/browse/HBASE-4709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13247674#comment-13247674
]
Lewis John McGibbney commented on HBASE-4709:
---------------------------------------------
An example of our typical stack trace.
{code}
3287 [main] INFO org.mortbay.log - jetty-6.1.26
3333 [main] INFO org.mortbay.log - Extract
jar:file:/export/home/hudson/hudson-slave/maven-repositories/0/org/apache/hadoop/hadoop-core/1.0.1/hadoop-core-1.0.1.jar!/webapps/hdfs
to /var/tmp/Jetty_localhost_54201_hdfs____txys7y/webapp
3598 [main] INFO org.mortbay.log - Started
SelectChannelConnector@localhost:54201
12/04/05 16:56:06 INFO namenode.NameNode: Web-server up at: localhost:54201
12/04/05 16:56:06 INFO ipc.Server: IPC Server Responder: starting
12/04/05 16:56:06 INFO ipc.Server: IPC Server listener on 54200: starting
12/04/05 16:56:06 INFO ipc.Server: IPC Server handler 0 on 54200: starting
12/04/05 16:56:06 INFO ipc.Server: IPC Server handler 3 on 54200: starting
12/04/05 16:56:07 INFO ipc.Server: IPC Server handler 7 on 54200: starting
12/04/05 16:56:06 INFO ipc.Server: IPC Server handler 2 on 54200: starting
12/04/05 16:56:06 INFO ipc.Server: IPC Server handler 1 on 54200: starting
12/04/05 16:56:07 INFO ipc.Server: IPC Server handler 9 on 54200: starting
12/04/05 16:56:07 INFO ipc.Server: IPC Server handler 8 on 54200: starting
12/04/05 16:56:07 INFO ipc.Server: IPC Server handler 6 on 54200: starting
12/04/05 16:56:06 INFO ipc.Server: IPC Server handler 4 on 54200: starting
12/04/05 16:56:06 INFO ipc.Server: IPC Server handler 5 on 54200: starting
Starting DataNode 0 with dfs.data.dir:
/export/home/hudson/hudson-slave/workspace/gora-trunk/trunk./gora-core/target/test-data/dfs/data/data1,/export/home/hudson/hudson-slave/workspace/gora-trunk/trunk./gora-core/target/test-data/dfs/data/data2
12/04/05 16:56:07 WARN impl.MetricsSystemImpl: Metrics system not started:
Cannot locate configuration: tried hadoop-metrics2-datanode.properties,
hadoop-metrics2.properties
12/04/05 16:56:07 WARN util.MBeans:
Hadoop:service=DataNode,name=MetricsSystem,sub=Control
javax.management.InstanceAlreadyExistsException: MXBean already registered with
name Hadoop:service=NameNode,name=MetricsSystem,sub=Control
at
com.sun.jmx.mbeanserver.MXBeanLookup.addReference(MXBeanLookup.java:120)
at
com.sun.jmx.mbeanserver.MXBeanSupport.register(MXBeanSupport.java:143)
at
com.sun.jmx.mbeanserver.MBeanSupport.preRegister2(MBeanSupport.java:183)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:941)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:917)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:312)
at
com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:482)
at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:56)
at
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.initSystemMBean(MetricsSystemImpl.java:500)
at
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.init(MetricsSystemImpl.java:140)
at
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.init(DefaultMetricsSystem.java:40)
at
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.initialize(DefaultMetricsSystem.java:50)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1520)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1496)
at
org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:417)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:280)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:124)
at
org.apache.hadoop.mapred.HadoopTestCase.setUp(HadoopTestCase.java:148)
at
org.apache.gora.mapreduce.DataStoreMapReduceTestBase.setUp(DataStoreMapReduceTestBase.java:61)
at junit.framework.TestCase.runBare(TestCase.java:132)
at junit.framework.TestResult$1.protect(TestResult.java:110)
at junit.framework.TestResult.runProtected(TestResult.java:128)
at junit.framework.TestResult.run(TestResult.java:113)
at junit.framework.TestCase.run(TestCase.java:124)
at junit.framework.TestSuite.runTest(TestSuite.java:243)
at junit.framework.TestSuite.run(TestSuite.java:238)
at
org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:236)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:113)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at
org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:103)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74)
{code}
You will however notice a difference between the original and my trace, where
this particular one is not using HBaseTestingUtility, however instead relying
on HadoopTestCase which in turn invokes MiniDFSCluster... the problem to that
end is identical. I must state that there is ONE previous 'metrics system not
starting' warning identical to the one you see above. E.g. two in all the 2nd
being followed by the full stack trace.
> Hadoop metrics2 setup in test MiniDFSClusters spewing JMX errors
> ----------------------------------------------------------------
>
> Key: HBASE-4709
> URL: https://issues.apache.org/jira/browse/HBASE-4709
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 0.92.0, 0.94.0
> Reporter: Gary Helmling
> Priority: Minor
>
> Since switching over HBase to build with Hadoop 0.20.205.0, we've been
> getting a lot of metrics related errors in the log files for tests:
> {noformat}
> 2011-10-30 22:00:22,858 INFO [main] log.Slf4jLog(67): jetty-6.1.26
> 2011-10-30 22:00:22,871 INFO [main] log.Slf4jLog(67): Extract
> jar:file:/home/jenkins/.m2/repository/org/apache/hadoop/hadoop-core/0.20.205.0/hadoop-core-0.20.205.0.jar!/webapps/datanode
> to /tmp/Jetty_localhost_55751_datanode____.kw16hy/webapp
> 2011-10-30 22:00:23,048 INFO [main] log.Slf4jLog(67): Started
> SelectChannelConnector@localhost:55751
> Starting DataNode 1 with dfs.data.dir:
> /home/jenkins/jenkins-slave/workspace/HBase-TRUNK/trunk/target/test-data/7ba65a16-03ad-4624-b769-57405945ef58/dfscluster_3775fc23-1b51-4966-8133-205564bae762/dfs/data/data3,/home/jenkins/jenkins-slave/workspace/HBase-TRUNK/trunk/target/test-data/7ba65a16-03ad-4624-b769-57405945ef58/dfscluster_3775fc23-1b51-4966-8133-205564bae762/dfs/data/data4
> 2011-10-30 22:00:23,237 WARN [main] impl.MetricsSystemImpl(137): Metrics
> system not started: Cannot locate configuration: tried
> hadoop-metrics2-datanode.properties, hadoop-metrics2.properties
> 2011-10-30 22:00:23,237 WARN [main] util.MBeans(59):
> Hadoop:service=DataNode,name=MetricsSystem,sub=Control
> javax.management.InstanceAlreadyExistsException: MXBean already registered
> with name Hadoop:service=NameNode,name=MetricsSystem,sub=Control
> at
> com.sun.jmx.mbeanserver.MXBeanLookup.addReference(MXBeanLookup.java:120)
> at
> com.sun.jmx.mbeanserver.MXBeanSupport.register(MXBeanSupport.java:143)
> at
> com.sun.jmx.mbeanserver.MBeanSupport.preRegister2(MBeanSupport.java:183)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:941)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:917)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:312)
> at
> com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:482)
> at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:56)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.initSystemMBean(MetricsSystemImpl.java:500)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.init(MetricsSystemImpl.java:140)
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.init(DefaultMetricsSystem.java:40)
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.initialize(DefaultMetricsSystem.java:50)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1483)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1459)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:417)
> at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:280)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:349)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:518)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:474)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:461)
> {noformat}
> This seems to be due to errors initializing the new hadoop metrics2 code by
> default, when running in a mini cluster. The errors themselves seem to be
> harmless -- they're not breaking any tests -- but we should figure out what
> configuration we need to eliminate them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
