With the help of Costin I got a running Maven configuration. Thank you :).
This is a pom.xml for Spring Data Hadoop and CDH4: <project xmlns="http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> <modelVersion>4.0.0</modelVersion> <groupId>com.example</groupId> <artifactId>com.example.main</artifactId> <version>0.0.1-SNAPSHOT</version> <packaging>jar</packaging> <properties> <java-version>1.7</java-version> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <spring.version>3.2.0.RELEASE</spring.version> <spring.hadoop.version>1.0.0.BUILD-SNAPSHOT</spring.hadoop.version> <hadoop.version.generic>2.0.0-cdh4.1.3</hadoop.version.generic> <hadoop.version.mr1>2.0.0-mr1-cdh4.1.3</hadoop.version.mr1> </properties> <dependencies> <dependency> <groupId>org.springframework</groupId> <artifactId>spring-core</artifactId> <version>${spring.version}</version> <exclusions> <exclusion> <groupId>commons-logging</groupId> <artifactId>commons-logging</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>org.springframework</groupId> <artifactId>spring-context</artifactId> <version>${spring.version}</version> </dependency> <dependency> <groupId>org.springframework.data</groupId> <artifactId>spring-data-hadoop</artifactId> <version>${spring.hadoop.version}</version> <exclusions> <!-- Excluded the Hadoop dependencies to be sure that they are not mixed with them provided by cloudera. --> <exclusion> <artifactId>hadoop-streaming</artifactId> <groupId>org.apache.hadoop</groupId> </exclusion> <exclusion> <artifactId>hadoop-tools</artifactId> <groupId>org.apache.hadoop</groupId> </exclusion> </exclusions> </dependency> <!-- Hadoop Cloudera Dependencies --> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-common</artifactId> <version>${hadoop.version.generic}</version> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-hdfs</artifactId> <version>${hadoop.version.generic}</version> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-tools</artifactId> <version>2.0.0-mr1-cdh4.1.3</version> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-streaming</artifactId> <version>2.0.0-mr1-cdh4.1.3</version> </dependency> </dependencies> <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <configuration> <source>${java-version}</source> <target>${java-version}</target> </configuration> </plugin> </plugins> </build> <repositories> <repository> <id>spring-milestones</id> <url>http://repo.springsource.org/libs-milestone</url> <snapshots> <enabled>false</enabled> </snapshots> </repository> <repository> <id>cloudera</id> <url>https://repository.cloudera.com/artifactory/cloudera-repos/</url> <snapshots> <enabled>false</enabled> </snapshots> </repository> <repository> <id>spring-snapshot</id> <name>Spring Maven SNAPSHOT Repository</name> <url>http://repo.springframework.org/snapshot</url> </repository> </repositories> </project> Best Regards, Christian. -------- Original-Nachricht -------- > Datum: Tue, 12 Feb 2013 16:56:50 +0100 > Von: Christian Schneider <[email protected]> > An: [email protected] > Betreff: Fwd: NullPointerException in Spring Data Hadoop with CDH4 > ---------- Forwarded message ---------- > From: Costin Leau <[email protected]> > Date: 2013/2/12 > Subject: Re: NullPointerException in Spring Data Hadoop with CDH4 > To: [email protected] > > > Hi, > > For Spring Data Hadoop problems, it's best to use the designated forum > [1]. > These being said I've tried to reproduce your error but I can't - I've > upgraded the build to CDH 4.1.3 which runs fine against the VM on the CI > (4.1.1). > Maybe you have some other libraries on the client classpath? > > From the stacktrace, it looks like the org.apache.hadoop.mapreduce.**Job > class has no 'state' or 'info' fields... > > Anyway, let's continue the discussion on the forum. > > Cheers, > [1] > http://forum.springsource.org/**forumdisplay.php?87-Hadoop<http://forum.springsource.org/forumdisplay.php?87-Hadoop> > > > On 02/12/13 2:51 PM, Christian Schneider wrote: > > > Hi, > > I try to use Spring Data Hadoop with CDH4 to write a Map Reduce Job. > > > > On startup, I get the following exception: > > > > Exception in thread "SimpleAsyncTaskExecutor-1" java.lang.** > > ExceptionInInitializerError > > at org.springframework.data.**hadoop.mapreduce.JobExecutor$** > > 2.run(JobExecutor.java:183) > > at java.lang.Thread.run(Thread.**java:722) > > Caused by: java.lang.NullPointerException > > at > org.springframework.util.**ReflectionUtils.**makeAccessible(** > > ReflectionUtils.java:405) > > at org.springframework.data.**hadoop.mapreduce.JobUtils.<** > > clinit>(JobUtils.java:123) > > ... 2 more > > > > I guess there is a problem with my Hadoop related dependencies. I > couldn't > > find any reference showing how to configure Spring Data together with > CDH4. > > But Costin showed, he is able to configure it: > https://build.springsource. > > > **org/browse/SPRINGDATAHADOOP-**CDH4-JOB1<https://build.springsource.org/browse/SPRINGDATAHADOOP-CDH4-JOB1> > > > > > > **Maven Setup** > > > > <properties> > > <spring.hadoop.version>1.0.0.**BUILD-SNAPSHOT</spring.hadoop.** > > version> > > <hadoop.version>2.0.0-cdh4.1.**3</hadoop.version> > > </properties> > > > > <dependencies> > > ... > > <dependency> > > <groupId>org.springframework.**data</groupId> > > <artifactId>spring-data-**hadoop</artifactId> > > <version>${spring.hadoop.**version}</version> > > </dependency> > > > > <dependency> > > <groupId>org.apache.hadoop</**groupId> > > <artifactId>hadoop-common</**artifactId> > > <version>${hadoop.version}</**version> > > </dependency> > > > > <dependency> > > <groupId>org.apache.hadoop</**groupId> > > <artifactId>hadoop-client</**artifactId> > > <version>${hadoop.version}</**version> > > </dependency> > > > > <dependency> > > <groupId>org.apache.hadoop</**groupId> > > <artifactId>hadoop-streaming</**artifactId> > > <version>${hadoop.version}</**version> > > </dependency> > > > > <dependency> > > <groupId>org.apache.hadoop</**groupId> > > <artifactId>hadoop-test</**artifactId> > > <version>2.0.0-mr1-cdh4.1.3</**version> > > </dependency> > > > > <dependency> > > <groupId>org.apache.hadoop</**groupId> > > <artifactId>hadoop-tools</**artifactId> > > <version>2.0.0-mr1-cdh4.1.3</**version> > > </dependency> > > ... > > </dependencies> > > ... > > <repositories> > > <repository> > > <id>cloudera</id> > > <url>https://repository.**cloudera.com/artifactory/** > > > cloudera-repos/<https://repository.cloudera.com/artifactory/cloudera-repos/> > > </url> > > <snapshots> > > <enabled>false</enabled> > > </snapshots> > > </repository> > > > > <repository> > > <id>spring-snapshot</id> > > <name>Spring Maven SNAPSHOT Repository</name> > > > <url>http://repo.**springframework.org/snapshot<http://repo.springframework.org/snapshot> > > </**url> > > </repository> > > </repositories> > > > > **Application Context** > > > > <?xml version="1.0" encoding="UTF-8"?> > > <beans > xmlns="http://www.**springframework.org/schema/**beans<http://www.springframework.org/schema/beans> > > " > > > xmlns:xsi="http://www.w3.org/**2001/XMLSchema-instance<http://www.w3.org/2001/XMLSchema-instance> > > " > > > xmlns:hdp="http://www.**springframework.org/schema/**hadoop<http://www.springframework.org/schema/hadoop>" > > > xmlns:context="http://www.**springframework.org/schema/**context<http://www.springframework.org/schema/context> > > " > > > xmlns:hadoop="http://www.**springframework.org/schema/**hadoop<http://www.springframework.org/schema/hadoop> > > " > > xsi:schemaLocation=" > > > http://www.springframework.**org/schema/beans<http://www.springframework.org/schema/beans> > > > http://www.springframework.**org/schema/beans/spring-beans.**xsd<http://www.springframework.org/schema/beans/spring-beans.xsd> > > > http://www.springframework.**org/schema/hadoop<http://www.springframework.org/schema/hadoop> > > > http://www.springframework.**org/schema/hadoop/spring-**hadoop.xsd<http://www.springframework.org/schema/hadoop/spring-hadoop.xsd> > > > http://www.springframework.**org/schema/context/spring-**context.xsd<http://www.springframework.org/schema/context/spring-context.xsd> > > > http://www.springframework.**org/schema/integration<http://www.springframework.org/schema/integration> > > > http://www.springframework.**org/schema/context<http://www.springframework.org/schema/context> > > > http://www.springframework.**org/schema/context/spring-**context-3.1.xsd<http://www.springframework.org/schema/context/spring-context-3.1.xsd> > > "> > > > > <context:property-placeholder > location="classpath:hadoop.**properties" > > /> > > > > <hdp:configuration id="hadoopConfiguration"> > > fs.defaultFS=${hd.fs} > > </hdp:configuration> > > > > <hdp:job id="wordCountJob" input-path="${input.path}" > > output-path="${output.path}" mapper="com.example.** > > WordMapper" > > reducer="com.example.**WordReducer" /> > > > > <hdp:job-runner job-ref="wordCountJob" run-at-startup="true" > > wait-for-completion="true"/> > > > > </beans> > > > > **Cluster version** > > > > Hadoop 2.0.0-cdh4.1.3 > > > > > > **Note:** > > > > This small Unittest is running fine with the current configuration: > > > > @RunWith(**SpringJUnit4ClassRunner.class) > > @ContextConfiguration(**locations = { > "classpath:/**applicationContext.xml" > > }) > > public class Starter { > > > > @Autowired > > private Configuration configuration; > > > > @Test > > public void shellOps() { > > Assert.assertNotNull(this.**configuration); > > FsShell fsShell = new FsShell(this.configuration); > > final Collection<FileStatus> coll = > fsShell.ls("/user"); > > System.out.println(coll); > > } > > } > > > > > > It would be nice if someone can give me an example configuration. > > > > Best Regards, > > Christian. > > > > > -- > Costin
