Herman:
For Pre-built with user-provided Hadoop, spark-1.4.1-bin-hadoop2.6.tgz,
e.g., uses hadoop-2.6 profile which defines versions of projects Spark
depends on.
Hadoop cluster is used to provide storage (hdfs) and resource management
(YARN).
For the latter, please see:
https://spark.apache.org/docs/latest/running-on-yarn.html
Cheers
On Thu, Jul 30, 2015 at 1:48 AM, hermansc herman.schis...@gmail.com wrote:
Hi.
I want to run Spark, and more specifically the Pre-build with
user-provided
Hadoop version from the downloads page, but I can't find any documentation
on how to connect the two components together (namely Spark and Hadoop).
I've had some success in settting SPARK_CLASSPATH to my hadoop distribution
lib/ directory, containing jar files such as hadoop-core, hadoop-common
etc.
However, there seems to be many native libraries included in the assembly
jar for Spark versions pre-built for Hadoop distributions (I'm specifically
missing the libsnappy.so files) that are not by default included in
distributions such as Cloudera Hadoop.
Have anyone here actually tried to run Spark without Hadoop included in the
assembly jar and/or have any more resources where I can read about the
proper way of connecting them?
As an aside, the spark-assembly jar in the Spark version pre-built for
user-provided Hadoop distributions is named
spark-assembly-1.4.0-hadoop2.2.0.jar, which doesn't make sense - it should
be called spark-assembly-1.4.0-without-hadoop.jar :)
--
Herman
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Running-Spark-on-user-provided-Hadoop-installation-tp24076.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
