Hi,
I tried getting the test suite running with Hadoop 1.0.3 and HBase
0.90.5 for CRUNCH-16. All tests passed except for the HBase word
count (log attached). I'm no HBase expert, but perhaps someone else
knows what's going wrong here.
To reproduce simply change version numbers (see attached patch) and
run the following command:
mvn -Dit.test=org.apache.crunch.WordCountHBaseIT verify
BTW, HBase is really trying my patience today, it causes a dozen
dependency conflicts with Hadoop, half of which have no clean
resolution.
Thanks in advance,
Matthias
Running org.apache.crunch.WordCountHBaseIT
log4j:WARN No appenders could be found for logger
(org.apache.zookeeper.server.ZooKeeperServer).
log4j:WARN Please initialize the log4j system properly.
1685 [main] WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl - Metrics
system not started: Cannot locate configuration: tried
hadoop-metrics2-namenode.properties, hadoop-metrics2.properties
Starting DataNode 0 with dfs.data.dir:
/tmp/incubator-crunch/crunch/target/test-data/65210a78-e0a6-4435-97fd-a7e27173fafb/dfs/data/data1,/tmp/incubator-crunch/crunch/target/test-data/65210a78-e0a6-4435-97fd-a7e27173fafb/dfs/data/data2
3258 [main] WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl - Metrics
system not started: Cannot locate configuration: tried
hadoop-metrics2-datanode.properties, hadoop-metrics2.properties
3260 [main] WARN org.apache.hadoop.metrics2.util.MBeans -
Hadoop:service=DataNode,name=MetricsSystem,sub=Control
javax.management.InstanceAlreadyExistsException: MXBean already registered with
name Hadoop:service=NameNode,name=MetricsSystem,sub=Control
at com.sun.jmx.mbeanserver.MXBeanLookup.addReference(MXBeanLookup.java:120)
at com.sun.jmx.mbeanserver.MXBeanSupport.register(MXBeanSupport.java:143)
at com.sun.jmx.mbeanserver.MBeanSupport.preRegister2(MBeanSupport.java:183)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:941)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:917)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:312)
at
com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:482)
at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:56)
at
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.initSystemMBean(MetricsSystemImpl.java:500)
at
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.init(MetricsSystemImpl.java:140)
at
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.init(DefaultMetricsSystem.java:40)
at
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.initialize(DefaultMetricsSystem.java:50)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1520)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1496)
at
org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:417)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:280)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:226)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:348)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:293)
at org.apache.crunch.WordCountHBaseIT.setUp(WordCountHBaseIT.java:103)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:46)
at org.junit.rules.RunRules.evaluate(RunRules.java:18)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:236)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:113)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at
org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:103)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74)
3290 [main] WARN org.apache.hadoop.hdfs.server.datanode.DataNode - Invalid
directory in dfs.data.dir: Incorrect permission for
/tmp/incubator-crunch/crunch/target/test-data/65210a78-e0a6-4435-97fd-a7e27173fafb/dfs/data/data1,
expected: rwxr-xr-x, while actual: rwxrwxr-x
3299 [main] WARN org.apache.hadoop.hdfs.server.datanode.DataNode - Invalid
directory in dfs.data.dir: Incorrect permission for
/tmp/incubator-crunch/crunch/target/test-data/65210a78-e0a6-4435-97fd-a7e27173fafb/dfs/data/data2,
expected: rwxr-xr-x, while actual: rwxrwxr-x
3299 [main] ERROR org.apache.hadoop.hdfs.server.datanode.DataNode - All
directories in dfs.data.dir are invalid.
diff --git a/pom.xml b/pom.xml
index 498fc6c..8aa50ef 100644
--- a/pom.xml
+++ b/pom.xml
@@ -46,8 +46,8 @@ under the License.
<java.target.version>1.6</java.target.version>
<scala.version>2.9.2</scala.version>
<avro.version>1.7.0</avro.version>
- <hadoop.version>0.20.2-cdh3u4</hadoop.version>
- <hbase.version>0.90.6-cdh3u4</hbase.version>
+ <hadoop.version>1.0.3</hadoop.version>
+ <hbase.version>0.90.5</hbase.version>
</properties>
<scm>
