Hi guys,
Probably a dummy question. Do you know how to compile Spark 0.9 to easily
integrate with HDFS 2.6.0 ?
I was trying
sbt/sbt -Pyarn -Phadoop-2.6 assembly
ormvn -Dhadoop.version=2.6.0 -DskipTests clean package
but none of these approaches succeeded.
Thanks,Robert
I am trying to run BlinkDB(https://github.com/sameeragarwal/blinkdb) which
seems to work only with Spark 0.9. However, if I want to access HDFS I need to
compile Spark against Hadoop version which is running on my cluster(2.6.0).
Hence, the versions problem ...
On Friday, February 13,
If you just need standalone mode, you don't need -Pyarn. There is no
-Phadoop-2.6; you should use -Phadoop-2.4 for 2.4+. Yes, set
-Dhadoop.version=2.6.0. That should be it.
If that still doesn't work, define doesn't succeed.
On Fri, Feb 13, 2015 at 7:13 PM, Grandl Robert
OK, from scanning the pom.xml, I think you would try:
-Pyarn -Dhadoop.version=2.6.0
If it doesn't package or pass tests, then I'd assume it's not supported :(
On Fri, Feb 13, 2015 at 7:33 PM, Grandl Robert rgra...@yahoo.com wrote:
I am trying to run
Thanks Sean for your prompt response.
I was trying to compile as following:
mvn -Phadoop-2.4 -Dhadoop.version=2.6.0 -DskipTests clean package
but I got a bunch of errors(see below). Hadoop-2.6.0 compiled correctly, and
all hadoop jars are in .m2 repository.
Do you have any idea what might
Oh right, you said Spark 0.9. Those profiles won't exist back then. I
don't even know if Hadoop 2.6 will work with 0.9 as-is. The profiles
were introduced later to fix up some compatibility. Why not use 1.2.1?
On Fri, Feb 13, 2015 at 7:26 PM, Grandl Robert rgra...@yahoo.com wrote:
Thanks Sean
