subject:"Re\: Can't run spark on yarn"

Re: Can't run spark on yarn

2015-12-17 Thread Saisai Shao

Please check the Yarn AM log to see why AM is failed to start. That's the
reason why using `sc` will get such complaint.

On Fri, Dec 18, 2015 at 4:25 AM, Eran Witkon  wrote:

> Hi,
> I am trying to install spark 1.5.2 on Apache hadoop 2.6 and Hive and yarn
>
> spark-env.sh
> export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop
>
> bash_profile
> #HADOOP VARIABLES START
> export JAVA_HOME=/usr/lib/jvm/java-8-oracle/
> export HADOOP_INSTALL=/usr/local/hadoop
> export PATH=$PATH:$HADOOP_INSTALL/bin
> export PATH=$PATH:$HADOOP_INSTALL/sbin
> export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
> export HADOOP_COMMON_HOME=$HADOOP_INSTALL
> export HADOOP_HDFS_HOME=$HADOOP_INSTALL
> export YARN_HOME=$HADOOP_INSTALL
> export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_INSTALL/lib/native
> export HADOOP_OPTS="-Djava.library.path=$HADOOP_INSTALL/lib"
> export HADOOP_USER_CLASSPATH_FIRST=true
> export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop
> export YARN_CONF_DIR=/usr/local/hadoop/etc/hadoop
> #HADOOP VARIABLES END
>
> export SPARK_HOME=/usr/local/spark
> export HIVE_HOME=/usr/local/hive
> export PATH=$PATH:$HIVE_HOME/bin
>
>
> When I run spark-shell
> ./bin/spark-shell --master yarn-client
>
> Output:
> 15/12/17 22:22:07 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 15/12/17 22:22:07 INFO spark.SecurityManager: Changing view acls to: hduser
> 15/12/17 22:22:07 INFO spark.SecurityManager: Changing modify acls to:
> hduser
> 15/12/17 22:22:07 INFO spark.SecurityManager: SecurityManager:
> authentication disabled; ui acls disabled; users with view permissions:
> Set(hduser); users with modify permissions: Set(hduser)
> 15/12/17 22:22:07 INFO spark.HttpServer: Starting HTTP Server
> 15/12/17 22:22:07 INFO server.Server: jetty-8.y.z-SNAPSHOT
> 15/12/17 22:22:08 INFO server.AbstractConnector: Started
> SocketConnector@0.0.0.0:38389
> 15/12/17 22:22:08 INFO util.Utils: Successfully started service 'HTTP
> class server' on port 38389.
> Welcome to
>     __
>  / __/__  ___ _/ /__
> _\ \/ _ \/ _ `/ __/  '_/
>/___/ .__/\_,_/_/ /_/\_\   version 1.5.2
>   /_/
>
> Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java
> 1.8.0_66)
> Type in expressions to have them evaluated.
> Type :help for more information.
> 15/12/17 22:22:11 WARN util.Utils: Your hostname, eranw-Lenovo-Yoga-2-Pro
> resolves to a loopback address: 127.0.1.1; using 10.0.0.1 instead (on
> interface wlp1s0)
> 15/12/17 22:22:11 WARN util.Utils: Set SPARK_LOCAL_IP if you need to bind
> to another address
> 15/12/17 22:22:11 INFO spark.SparkContext: Running Spark version 1.5.2
> 15/12/17 22:22:11 INFO spark.SecurityManager: Changing view acls to: hduser
> 15/12/17 22:22:11 INFO spark.SecurityManager: Changing modify acls to:
> hduser
> 15/12/17 22:22:11 INFO spark.SecurityManager: SecurityManager:
> authentication disabled; ui acls disabled; users with view permissions:
> Set(hduser); users with modify permissions: Set(hduser)
> 15/12/17 22:22:11 INFO slf4j.Slf4jLogger: Slf4jLogger started
> 15/12/17 22:22:11 INFO Remoting: Starting remoting
> 15/12/17 22:22:12 INFO util.Utils: Successfully started service
> 'sparkDriver' on port 36381.
> 15/12/17 22:22:12 INFO Remoting: Remoting started; listening on addresses
> :[akka.tcp://sparkDriver@10.0.0.1:36381]
> 15/12/17 22:22:12 INFO spark.SparkEnv: Registering MapOutputTracker
> 15/12/17 22:22:12 INFO spark.SparkEnv: Registering BlockManagerMaster
> 15/12/17 22:22:12 INFO storage.DiskBlockManager: Created local directory
> at /tmp/blockmgr-139fac31-5f21-4c61-9575-3110d5205f7d
> 15/12/17 22:22:12 INFO storage.MemoryStore: MemoryStore started with
> capacity 530.0 MB
> 15/12/17 22:22:12 INFO spark.HttpFileServer: HTTP File server directory is
> /tmp/spark-955ef002-a802-49c6-b440-0656861f737c/httpd-2127cbe1-97d7-40a5-a96f-75216f115f00
> 15/12/17 22:22:12 INFO spark.HttpServer: Starting HTTP Server
> 15/12/17 22:22:12 INFO server.Server: jetty-8.y.z-SNAPSHOT
> 15/12/17 22:22:12 INFO server.AbstractConnector: Started
> SocketConnector@0.0.0.0:36760
> 15/12/17 22:22:12 INFO util.Utils: Successfully started service 'HTTP file
> server' on port 36760.
> 15/12/17 22:22:12 INFO spark.SparkEnv: Registering OutputCommitCoordinator
> 15/12/17 22:22:12 INFO server.Server: jetty-8.y.z-SNAPSHOT
> 15/12/17 22:22:12 INFO server.AbstractConnector: Started
> SelectChannelConnector@0.0.0.0:4040
> 15/12/17 22:22:12 INFO util.Utils: Successfully started service 'SparkUI'
> on port 4040.
> 15/12/17 22:22:12 INFO ui.SparkUI: Started SparkUI at http://10.0.0.1:4040
> 15/12/17 22:22:12 WARN metrics.MetricsSystem: Using default name
> DAGScheduler for source because spark.app.id is not set.
> 15/12/17 22:22:12 INFO client.RMProxy: Connecting to ResourceManager at /
> 0.0.0.0:8032
> 15/12/17 22:22:12 INFO yarn.Client: Requesting a new application from
> cluster with 1 NodeManagers
> 15/12/17 22:22:12

Re: Can't run spark on yarn

2015-12-17 Thread Alexander Pivovarov

Try to start aws EMR 4.2.0 with hadoop and spark applications on spot
instances. Then look at how hadoop and spark configured. Try to configure
your hadoop and spark similar way
On Dec 17, 2015 6:09 PM, "Saisai Shao"  wrote:

> Please check the Yarn AM log to see why AM is failed to start. That's the
> reason why using `sc` will get such complaint.
>
> On Fri, Dec 18, 2015 at 4:25 AM, Eran Witkon  wrote:
>
>> Hi,
>> I am trying to install spark 1.5.2 on Apache hadoop 2.6 and Hive and yarn
>>
>> spark-env.sh
>> export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop
>>
>> bash_profile
>> #HADOOP VARIABLES START
>> export JAVA_HOME=/usr/lib/jvm/java-8-oracle/
>> export HADOOP_INSTALL=/usr/local/hadoop
>> export PATH=$PATH:$HADOOP_INSTALL/bin
>> export PATH=$PATH:$HADOOP_INSTALL/sbin
>> export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
>> export HADOOP_COMMON_HOME=$HADOOP_INSTALL
>> export HADOOP_HDFS_HOME=$HADOOP_INSTALL
>> export YARN_HOME=$HADOOP_INSTALL
>> export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_INSTALL/lib/native
>> export HADOOP_OPTS="-Djava.library.path=$HADOOP_INSTALL/lib"
>> export HADOOP_USER_CLASSPATH_FIRST=true
>> export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop
>> export YARN_CONF_DIR=/usr/local/hadoop/etc/hadoop
>> #HADOOP VARIABLES END
>>
>> export SPARK_HOME=/usr/local/spark
>> export HIVE_HOME=/usr/local/hive
>> export PATH=$PATH:$HIVE_HOME/bin
>>
>>
>> When I run spark-shell
>> ./bin/spark-shell --master yarn-client
>>
>> Output:
>> 15/12/17 22:22:07 WARN util.NativeCodeLoader: Unable to load
>> native-hadoop library for your platform... using builtin-java classes where
>> applicable
>> 15/12/17 22:22:07 INFO spark.SecurityManager: Changing view acls to:
>> hduser
>> 15/12/17 22:22:07 INFO spark.SecurityManager: Changing modify acls to:
>> hduser
>> 15/12/17 22:22:07 INFO spark.SecurityManager: SecurityManager:
>> authentication disabled; ui acls disabled; users with view permissions:
>> Set(hduser); users with modify permissions: Set(hduser)
>> 15/12/17 22:22:07 INFO spark.HttpServer: Starting HTTP Server
>> 15/12/17 22:22:07 INFO server.Server: jetty-8.y.z-SNAPSHOT
>> 15/12/17 22:22:08 INFO server.AbstractConnector: Started
>> SocketConnector@0.0.0.0:38389
>> 15/12/17 22:22:08 INFO util.Utils: Successfully started service 'HTTP
>> class server' on port 38389.
>> Welcome to
>>     __
>>  / __/__  ___ _/ /__
>> _\ \/ _ \/ _ `/ __/  '_/
>>/___/ .__/\_,_/_/ /_/\_\   version 1.5.2
>>   /_/
>>
>> Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java
>> 1.8.0_66)
>> Type in expressions to have them evaluated.
>> Type :help for more information.
>> 15/12/17 22:22:11 WARN util.Utils: Your hostname, eranw-Lenovo-Yoga-2-Pro
>> resolves to a loopback address: 127.0.1.1; using 10.0.0.1 instead (on
>> interface wlp1s0)
>> 15/12/17 22:22:11 WARN util.Utils: Set SPARK_LOCAL_IP if you need to bind
>> to another address
>> 15/12/17 22:22:11 INFO spark.SparkContext: Running Spark version 1.5.2
>> 15/12/17 22:22:11 INFO spark.SecurityManager: Changing view acls to:
>> hduser
>> 15/12/17 22:22:11 INFO spark.SecurityManager: Changing modify acls to:
>> hduser
>> 15/12/17 22:22:11 INFO spark.SecurityManager: SecurityManager:
>> authentication disabled; ui acls disabled; users with view permissions:
>> Set(hduser); users with modify permissions: Set(hduser)
>> 15/12/17 22:22:11 INFO slf4j.Slf4jLogger: Slf4jLogger started
>> 15/12/17 22:22:11 INFO Remoting: Starting remoting
>> 15/12/17 22:22:12 INFO util.Utils: Successfully started service
>> 'sparkDriver' on port 36381.
>> 15/12/17 22:22:12 INFO Remoting: Remoting started; listening on addresses
>> :[akka.tcp://sparkDriver@10.0.0.1:36381]
>> 15/12/17 22:22:12 INFO spark.SparkEnv: Registering MapOutputTracker
>> 15/12/17 22:22:12 INFO spark.SparkEnv: Registering BlockManagerMaster
>> 15/12/17 22:22:12 INFO storage.DiskBlockManager: Created local directory
>> at /tmp/blockmgr-139fac31-5f21-4c61-9575-3110d5205f7d
>> 15/12/17 22:22:12 INFO storage.MemoryStore: MemoryStore started with
>> capacity 530.0 MB
>> 15/12/17 22:22:12 INFO spark.HttpFileServer: HTTP File server directory
>> is
>> /tmp/spark-955ef002-a802-49c6-b440-0656861f737c/httpd-2127cbe1-97d7-40a5-a96f-75216f115f00
>> 15/12/17 22:22:12 INFO spark.HttpServer: Starting HTTP Server
>> 15/12/17 22:22:12 INFO server.Server: jetty-8.y.z-SNAPSHOT
>> 15/12/17 22:22:12 INFO server.AbstractConnector: Started
>> SocketConnector@0.0.0.0:36760
>> 15/12/17 22:22:12 INFO util.Utils: Successfully started service 'HTTP
>> file server' on port 36760.
>> 15/12/17 22:22:12 INFO spark.SparkEnv: Registering OutputCommitCoordinator
>> 15/12/17 22:22:12 INFO server.Server: jetty-8.y.z-SNAPSHOT
>> 15/12/17 22:22:12 INFO server.AbstractConnector: Started
>> SelectChannelConnector@0.0.0.0:4040
>> 15/12/17 22:22:12 INFO util.Utils: Successfully started service 'SparkUI'
>> on port 4040.
>> 15/12/17 22:22:12 INFO ui.SparkUI: Started

Re: Can't run spark on yarn

Re: Can't run spark on yarn

2 matches

Site Navigation

Mail list logo

Footer information