So, you should not depend on Hadoop artifacts unless you use them directly. You should mark Hadoop and Spark deps as provided. Then the cluster's version is used at runtime with spark-submit. That's the usual way to do it, which works.
If you need to embed Spark in your app and are running it outside the cluster for some reason, and you have to embed Hadoop and Spark code in your app, the version has to match. You should also use mvn dependency:tree to see all the dependencies coming in. There may be many sources of a Hadoop dep. On Fri, Jan 23, 2015 at 1:05 AM, ey-chih chow <eyc...@hotmail.com> wrote: > Thanks. But after I replace the maven dependence from > > <dependency> > <groupId>org.apache.hadoop</groupId> > <artifactId>hadoop-client</artifactId> > <version>2.5.0-cdh5.2.0</version> > <scope>provided</scope> > <exclusions> > <exclusion> > <groupId>org.mortbay.jetty</groupId> > <artifactId>servlet-api</artifactId> > </exclusion> > <exclusion> > <groupId>javax.servlet</groupId> > <artifactId>servlet-api</artifactId> > </exclusion> > <exclusion> > <groupId>io.netty</groupId> > <artifactId>netty</artifactId> > </exclusion> > </exclusions> > </dependency> > > to > > <dependency> > > <groupId>org.apache.hadoop</groupId> > > <artifactId>hadoop-client</artifactId> > > <version>1.0.4</version> > > <scope>provided</scope> > > <exclusions> > > <exclusion> > > <groupId>org.mortbay.jetty</groupId> > > <artifactId>servlet-api</artifactId> > > </exclusion> > > <exclusion> > > <groupId>javax.servlet</groupId> > > <artifactId>servlet-api</artifactId> > > </exclusion> > > <exclusion> > > <groupId>io.netty</groupId> > > <artifactId>netty</artifactId> > > </exclusion> > > </exclusions> > > </dependency> > > > the warning message is still shown up in the namenode log. Is there any > other thing I need to do? > > > Thanks. > > > Ey-Chih Chow > > --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
