I am trying to run BlinkDB(https://github.com/sameeragarwal/blinkdb) which
seems to work only with Spark 0.9. However, if I want to access HDFS I need to
compile Spark against Hadoop version which is running on my cluster(2.6.0).
Hence, the versions problem ...
On Friday, February 13, 2015 11:28 AM, Sean Owen <[email protected]>
wrote:
Oh right, you said Spark 0.9. Those profiles won't exist back then. I
don't even know if Hadoop 2.6 will work with 0.9 as-is. The profiles
were introduced later to fix up some compatibility. Why not use 1.2.1?
On Fri, Feb 13, 2015 at 7:26 PM, Grandl Robert <[email protected]> wrote:
> Thanks Sean for your prompt response.
>
> I was trying to compile as following:
> mvn -Phadoop-2.4 -Dhadoop.version=2.6.0 -DskipTests clean package
>
> but I got a bunch of errors(see below). Hadoop-2.6.0 compiled correctly, and
> all hadoop jars are in .m2 repository.
>
> Do you have any idea what might happens ?
>
> Robert
>
> [WARNING] Class com.google.protobuf.Parser not found - continuing with a
> stub.
> [ERROR] error while loading RpcResponseHeaderProto, class file
> '/home/rgrandl/.m2/repository/org/apache/hadoop/hadoop-common/2.6.0/hadoop-common-2.6.0.jar(org/apache/hadoop/ipc/protobuf/RpcHeaderProtos$RpcResponseHeaderProto.class)'
> is broken
> (class java.lang.NullPointerException/null)
> [WARNING] one warning found
> [ERROR] one error found
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Spark Project Parent POM .......................... SUCCESS [2.537s]
> [INFO] Spark Project Core ................................ FAILURE [25.917s]
> [INFO] Spark Project Bagel ............................... SKIPPED
> [INFO] Spark Project GraphX .............................. SKIPPED
> [INFO] Spark Project ML Library .......................... SKIPPED
> [INFO] Spark Project Streaming ........................... SKIPPED
> [INFO] Spark Project Tools ............................... SKIPPED
> [INFO] Spark Project REPL ................................ SKIPPED
> [INFO] Spark Project Assembly ............................ SKIPPED
> [INFO] Spark Project External Twitter .................... SKIPPED
> [INFO] Spark Project External Kafka ...................... SKIPPED
> [INFO] Spark Project External Flume ...................... SKIPPED
> [INFO] Spark Project External ZeroMQ ..................... SKIPPED
> [INFO] Spark Project External MQTT ....................... SKIPPED
> [INFO] Spark Project Examples ............................ SKIPPED
> [INFO]
> ------------------------------------------------------------------------
> [INFO] BUILD FAILURE
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Total time: 30.002s
> [INFO] Finished at: Fri Feb 13 11:21:36 PST 2015
> [INFO] Final Memory: 49M/1226M
> [INFO]
> ------------------------------------------------------------------------
> [WARNING] The requested profile "hadoop-2.4" could not be activated because
> it does not exist.
> [ERROR] Failed to execute goal
> net.alchim31.maven:scala-maven-plugin:3.1.5:compile (scala-compile-first) on
> project spark-core_2.10: Execution scala-compile-first of goal
> net.alchim31.maven:scala-maven-plugin:3.1.5:compile failed. CompileFailed ->
> [Help 1]
>
>
>
> On Friday, February 13, 2015 11:16 AM, Sean Owen <[email protected]> wrote:
>
>
> If you just need standalone mode, you don't need "-Pyarn". There is no
> "-Phadoop-2.6"; you should use "-Phadoop-2.4" for 2.4+. Yes, set
> -Dhadoop.version=2.6.0. That should be it.
>
> If that still doesn't work, define "doesn't succeed".
>
> On Fri, Feb 13, 2015 at 7:13 PM, Grandl Robert
> <[email protected]> wrote:
>> Hi guys,
>>
>> Probably a dummy question. Do you know how to compile Spark 0.9 to easily
>> integrate with HDFS 2.6.0 ?
>>
>> I was trying
>> sbt/sbt -Pyarn -Phadoop-2.6 assembly
>> or
>> mvn -Dhadoop.version=2.6.0 -DskipTests clean package
>>
>> but none of these approaches succeeded.
>>
>> Thanks,
>> Robert
>
>
On Friday, February 13, 2015 11:28 AM, Sean Owen <[email protected]>
wrote:
Oh right, you said Spark 0.9. Those profiles won't exist back then. I
don't even know if Hadoop 2.6 will work with 0.9 as-is. The profiles
were introduced later to fix up some compatibility. Why not use 1.2.1?
On Fri, Feb 13, 2015 at 7:26 PM, Grandl Robert <[email protected]> wrote:
> Thanks Sean for your prompt response.
>
> I was trying to compile as following:
> mvn -Phadoop-2.4 -Dhadoop.version=2.6.0 -DskipTests clean package
>
> but I got a bunch of errors(see below). Hadoop-2.6.0 compiled correctly, and
> all hadoop jars are in .m2 repository.
>
> Do you have any idea what might happens ?
>
> Robert
>
> [WARNING] Class com.google.protobuf.Parser not found - continuing with a
> stub.
> [ERROR] error while loading RpcResponseHeaderProto, class file
> '/home/rgrandl/.m2/repository/org/apache/hadoop/hadoop-common/2.6.0/hadoop-common-2.6.0.jar(org/apache/hadoop/ipc/protobuf/RpcHeaderProtos$RpcResponseHeaderProto.class)'
> is broken
> (class java.lang.NullPointerException/null)
> [WARNING] one warning found
> [ERROR] one error found
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Spark Project Parent POM .......................... SUCCESS [2.537s]
> [INFO] Spark Project Core ................................ FAILURE [25.917s]
> [INFO] Spark Project Bagel ............................... SKIPPED
> [INFO] Spark Project GraphX .............................. SKIPPED
> [INFO] Spark Project ML Library .......................... SKIPPED
> [INFO] Spark Project Streaming ........................... SKIPPED
> [INFO] Spark Project Tools ............................... SKIPPED
> [INFO] Spark Project REPL ................................ SKIPPED
> [INFO] Spark Project Assembly ............................ SKIPPED
> [INFO] Spark Project External Twitter .................... SKIPPED
> [INFO] Spark Project External Kafka ...................... SKIPPED
> [INFO] Spark Project External Flume ...................... SKIPPED
> [INFO] Spark Project External ZeroMQ ..................... SKIPPED
> [INFO] Spark Project External MQTT ....................... SKIPPED
> [INFO] Spark Project Examples ............................ SKIPPED
> [INFO]
> ------------------------------------------------------------------------
> [INFO] BUILD FAILURE
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Total time: 30.002s
> [INFO] Finished at: Fri Feb 13 11:21:36 PST 2015
> [INFO] Final Memory: 49M/1226M
> [INFO]
> ------------------------------------------------------------------------
> [WARNING] The requested profile "hadoop-2.4" could not be activated because
> it does not exist.
> [ERROR] Failed to execute goal
> net.alchim31.maven:scala-maven-plugin:3.1.5:compile (scala-compile-first) on
> project spark-core_2.10: Execution scala-compile-first of goal
> net.alchim31.maven:scala-maven-plugin:3.1.5:compile failed. CompileFailed ->
> [Help 1]
>
>
>
> On Friday, February 13, 2015 11:16 AM, Sean Owen <[email protected]> wrote:
>
>
> If you just need standalone mode, you don't need "-Pyarn". There is no
> "-Phadoop-2.6"; you should use "-Phadoop-2.4" for 2.4+. Yes, set
> -Dhadoop.version=2.6.0. That should be it.
>
> If that still doesn't work, define "doesn't succeed".
>
> On Fri, Feb 13, 2015 at 7:13 PM, Grandl Robert
> <[email protected]> wrote:
>> Hi guys,
>>
>> Probably a dummy question. Do you know how to compile Spark 0.9 to easily
>> integrate with HDFS 2.6.0 ?
>>
>> I was trying
>> sbt/sbt -Pyarn -Phadoop-2.6 assembly
>> or
>> mvn -Dhadoop.version=2.6.0 -DskipTests clean package
>>
>> but none of these approaches succeeded.
>>
>> Thanks,
>> Robert
>
>
