Hi,
For Spring Data Hadoop problems, it's best to use the designated forum
[1]. These being said I've tried to reproduce your error but I can't -
I've upgraded the build to CDH 4.1.3 which runs fine against the VM on
the CI (4.1.1).
Maybe you have some other libraries on the client classpath?
From the stacktrace, it looks like the org.apache.hadoop.mapreduce.Job
class has no 'state' or 'info' fields...
Anyway, let's continue the discussion on the forum.
Cheers,
[1] http://forum.springsource.org/forumdisplay.php?87-Hadoop
On 02/12/13 2:51 PM, Christian Schneider wrote:
Hi,
I try to use Spring Data Hadoop with CDH4 to write a Map Reduce Job.
On startup, I get the following exception:
Exception in thread "SimpleAsyncTaskExecutor-1"
java.lang.ExceptionInInitializerError
at
org.springframework.data.hadoop.mapreduce.JobExecutor$2.run(JobExecutor.java:183)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.NullPointerException
at
org.springframework.util.ReflectionUtils.makeAccessible(ReflectionUtils.java:405)
at
org.springframework.data.hadoop.mapreduce.JobUtils.<clinit>(JobUtils.java:123)
... 2 more
I guess there is a problem with my Hadoop related dependencies. I couldn't find
any reference showing how to configure Spring Data together with CDH4. But
Costin showed, he is able to configure it:
https://build.springsource.org/browse/SPRINGDATAHADOOP-CDH4-JOB1
**Maven Setup**
<properties>
<spring.hadoop.version>1.0.0.BUILD-SNAPSHOT</spring.hadoop.version>
<hadoop.version>2.0.0-cdh4.1.3</hadoop.version>
</properties>
<dependencies>
...
<dependency>
<groupId>org.springframework.data</groupId>
<artifactId>spring-data-hadoop</artifactId>
<version>${spring.hadoop.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>${hadoop.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>${hadoop.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-streaming</artifactId>
<version>${hadoop.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-test</artifactId>
<version>2.0.0-mr1-cdh4.1.3</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-tools</artifactId>
<version>2.0.0-mr1-cdh4.1.3</version>
</dependency>
...
</dependencies>
...
<repositories>
<repository>
<id>cloudera</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
<snapshots>
<enabled>false</enabled>
</snapshots>
</repository>
<repository>
<id>spring-snapshot</id>
<name>Spring Maven SNAPSHOT Repository</name>
<url>http://repo.springframework.org/snapshot</url>
</repository>
</repositories>
**Application Context**
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xmlns:hdp="http://www.springframework.org/schema/hadoop";
xmlns:context="http://www.springframework.org/schema/context";
xmlns:hadoop="http://www.springframework.org/schema/hadoop";
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/hadoop
http://www.springframework.org/schema/hadoop/spring-hadoop.xsd
http://www.springframework.org/schema/context/spring-context.xsd
http://www.springframework.org/schema/integration
http://www.springframework.org/schema/context
http://www.springframework.org/schema/context/spring-context-3.1.xsd";>
<context:property-placeholder location="classpath:hadoop.properties" />
<hdp:configuration id="hadoopConfiguration">
fs.defaultFS=${hd.fs}
</hdp:configuration>
<hdp:job id="wordCountJob" input-path="${input.path}"
output-path="${output.path}" mapper="com.example.WordMapper"
reducer="com.example.WordReducer" />
<hdp:job-runner job-ref="wordCountJob" run-at-startup="true"
wait-for-completion="true"/>
</beans>
**Cluster version**
Hadoop 2.0.0-cdh4.1.3
**Note:**
This small Unittest is running fine with the current configuration:
@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(locations = { "classpath:/applicationContext.xml" })
public class Starter {
@Autowired
private Configuration configuration;
@Test
public void shellOps() {
Assert.assertNotNull(this.configuration);
FsShell fsShell = new FsShell(this.configuration);
final Collection<FileStatus> coll = fsShell.ls("/user");
System.out.println(coll);
}
}
It would be nice if someone can give me an example configuration.
Best Regards,
Christian.
--
Costin
