Oh right, you said Spark 0.9. Those profiles won't exist back then. I don't even know if Hadoop 2.6 will work with 0.9 as-is. The profiles were introduced later to fix up some compatibility. Why not use 1.2.1?
On Fri, Feb 13, 2015 at 7:26 PM, Grandl Robert <[email protected]> wrote: > Thanks Sean for your prompt response. > > I was trying to compile as following: > mvn -Phadoop-2.4 -Dhadoop.version=2.6.0 -DskipTests clean package > > but I got a bunch of errors(see below). Hadoop-2.6.0 compiled correctly, and > all hadoop jars are in .m2 repository. > > Do you have any idea what might happens ? > > Robert > > [WARNING] Class com.google.protobuf.Parser not found - continuing with a > stub. > [ERROR] error while loading RpcResponseHeaderProto, class file > '/home/rgrandl/.m2/repository/org/apache/hadoop/hadoop-common/2.6.0/hadoop-common-2.6.0.jar(org/apache/hadoop/ipc/protobuf/RpcHeaderProtos$RpcResponseHeaderProto.class)' > is broken > (class java.lang.NullPointerException/null) > [WARNING] one warning found > [ERROR] one error found > [INFO] > ------------------------------------------------------------------------ > [INFO] Reactor Summary: > [INFO] > [INFO] Spark Project Parent POM .......................... SUCCESS [2.537s] > [INFO] Spark Project Core ................................ FAILURE [25.917s] > [INFO] Spark Project Bagel ............................... SKIPPED > [INFO] Spark Project GraphX .............................. SKIPPED > [INFO] Spark Project ML Library .......................... SKIPPED > [INFO] Spark Project Streaming ........................... SKIPPED > [INFO] Spark Project Tools ............................... SKIPPED > [INFO] Spark Project REPL ................................ SKIPPED > [INFO] Spark Project Assembly ............................ SKIPPED > [INFO] Spark Project External Twitter .................... SKIPPED > [INFO] Spark Project External Kafka ...................... SKIPPED > [INFO] Spark Project External Flume ...................... SKIPPED > [INFO] Spark Project External ZeroMQ ..................... SKIPPED > [INFO] Spark Project External MQTT ....................... SKIPPED > [INFO] Spark Project Examples ............................ SKIPPED > [INFO] > ------------------------------------------------------------------------ > [INFO] BUILD FAILURE > [INFO] > ------------------------------------------------------------------------ > [INFO] Total time: 30.002s > [INFO] Finished at: Fri Feb 13 11:21:36 PST 2015 > [INFO] Final Memory: 49M/1226M > [INFO] > ------------------------------------------------------------------------ > [WARNING] The requested profile "hadoop-2.4" could not be activated because > it does not exist. > [ERROR] Failed to execute goal > net.alchim31.maven:scala-maven-plugin:3.1.5:compile (scala-compile-first) on > project spark-core_2.10: Execution scala-compile-first of goal > net.alchim31.maven:scala-maven-plugin:3.1.5:compile failed. CompileFailed -> > [Help 1] > > > > On Friday, February 13, 2015 11:16 AM, Sean Owen <[email protected]> wrote: > > > If you just need standalone mode, you don't need "-Pyarn". There is no > "-Phadoop-2.6"; you should use "-Phadoop-2.4" for 2.4+. Yes, set > -Dhadoop.version=2.6.0. That should be it. > > If that still doesn't work, define "doesn't succeed". > > On Fri, Feb 13, 2015 at 7:13 PM, Grandl Robert > <[email protected]> wrote: >> Hi guys, >> >> Probably a dummy question. Do you know how to compile Spark 0.9 to easily >> integrate with HDFS 2.6.0 ? >> >> I was trying >> sbt/sbt -Pyarn -Phadoop-2.6 assembly >> or >> mvn -Dhadoop.version=2.6.0 -DskipTests clean package >> >> but none of these approaches succeeded. >> >> Thanks, >> Robert > > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
