subject:"Can't run spark on yarn"

Can't run spark on yarn

2015-12-17 Thread Eran Witkon

Hi,
I am trying to install spark 1.5.2 on Apache hadoop 2.6 and Hive and yarn

spark-env.sh
export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop

bash_profile
#HADOOP VARIABLES START
export JAVA_HOME=/usr/lib/jvm/java-8-oracle/
export HADOOP_INSTALL=/usr/local/hadoop
export PATH=$PATH:$HADOOP_INSTALL/bin
export PATH=$PATH:$HADOOP_INSTALL/sbin
export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_HOME=$HADOOP_INSTALL
export HADOOP_HDFS_HOME=$HADOOP_INSTALL
export YARN_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_INSTALL/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_INSTALL/lib"
export HADOOP_USER_CLASSPATH_FIRST=true
export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop
export YARN_CONF_DIR=/usr/local/hadoop/etc/hadoop
#HADOOP VARIABLES END

export SPARK_HOME=/usr/local/spark
export HIVE_HOME=/usr/local/hive
export PATH=$PATH:$HIVE_HOME/bin


When I run spark-shell
./bin/spark-shell --master yarn-client

Output:
15/12/17 22:22:07 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
15/12/17 22:22:07 INFO spark.SecurityManager: Changing view acls to: hduser
15/12/17 22:22:07 INFO spark.SecurityManager: Changing modify acls to:
hduser
15/12/17 22:22:07 INFO spark.SecurityManager: SecurityManager:
authentication disabled; ui acls disabled; users with view permissions:
Set(hduser); users with modify permissions: Set(hduser)
15/12/17 22:22:07 INFO spark.HttpServer: Starting HTTP Server
15/12/17 22:22:07 INFO server.Server: jetty-8.y.z-SNAPSHOT
15/12/17 22:22:08 INFO server.AbstractConnector: Started
SocketConnector@0.0.0.0:38389
15/12/17 22:22:08 INFO util.Utils: Successfully started service 'HTTP class
server' on port 38389.
Welcome to
    __
 / __/__  ___ _/ /__
_\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 1.5.2
  /_/

Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java
1.8.0_66)
Type in expressions to have them evaluated.
Type :help for more information.
15/12/17 22:22:11 WARN util.Utils: Your hostname, eranw-Lenovo-Yoga-2-Pro
resolves to a loopback address: 127.0.1.1; using 10.0.0.1 instead (on
interface wlp1s0)
15/12/17 22:22:11 WARN util.Utils: Set SPARK_LOCAL_IP if you need to bind
to another address
15/12/17 22:22:11 INFO spark.SparkContext: Running Spark version 1.5.2
15/12/17 22:22:11 INFO spark.SecurityManager: Changing view acls to: hduser
15/12/17 22:22:11 INFO spark.SecurityManager: Changing modify acls to:
hduser
15/12/17 22:22:11 INFO spark.SecurityManager: SecurityManager:
authentication disabled; ui acls disabled; users with view permissions:
Set(hduser); users with modify permissions: Set(hduser)
15/12/17 22:22:11 INFO slf4j.Slf4jLogger: Slf4jLogger started
15/12/17 22:22:11 INFO Remoting: Starting remoting
15/12/17 22:22:12 INFO util.Utils: Successfully started service
'sparkDriver' on port 36381.
15/12/17 22:22:12 INFO Remoting: Remoting started; listening on addresses
:[akka.tcp://sparkDriver@10.0.0.1:36381]
15/12/17 22:22:12 INFO spark.SparkEnv: Registering MapOutputTracker
15/12/17 22:22:12 INFO spark.SparkEnv: Registering BlockManagerMaster
15/12/17 22:22:12 INFO storage.DiskBlockManager: Created local directory at
/tmp/blockmgr-139fac31-5f21-4c61-9575-3110d5205f7d
15/12/17 22:22:12 INFO storage.MemoryStore: MemoryStore started with
capacity 530.0 MB
15/12/17 22:22:12 INFO spark.HttpFileServer: HTTP File server directory is
/tmp/spark-955ef002-a802-49c6-b440-0656861f737c/httpd-2127cbe1-97d7-40a5-a96f-75216f115f00
15/12/17 22:22:12 INFO spark.HttpServer: Starting HTTP Server
15/12/17 22:22:12 INFO server.Server: jetty-8.y.z-SNAPSHOT
15/12/17 22:22:12 INFO server.AbstractConnector: Started
SocketConnector@0.0.0.0:36760
15/12/17 22:22:12 INFO util.Utils: Successfully started service 'HTTP file
server' on port 36760.
15/12/17 22:22:12 INFO spark.SparkEnv: Registering OutputCommitCoordinator
15/12/17 22:22:12 INFO server.Server: jetty-8.y.z-SNAPSHOT
15/12/17 22:22:12 INFO server.AbstractConnector: Started
SelectChannelConnector@0.0.0.0:4040
15/12/17 22:22:12 INFO util.Utils: Successfully started service 'SparkUI'
on port 4040.
15/12/17 22:22:12 INFO ui.SparkUI: Started SparkUI at http://10.0.0.1:4040
15/12/17 22:22:12 WARN metrics.MetricsSystem: Using default name
DAGScheduler for source because spark.app.id is not set.
15/12/17 22:22:12 INFO client.RMProxy: Connecting to ResourceManager at /
0.0.0.0:8032
15/12/17 22:22:12 INFO yarn.Client: Requesting a new application from
cluster with 1 NodeManagers
15/12/17 22:22:12 INFO yarn.Client: Verifying our application has not
requested more than the maximum memory capability of the cluster (8192 MB
per container)
15/12/17 22:22:12 INFO yarn.Client: Will allocate AM container, with 896 MB
memory including 384 MB overhead
15/12/17 22:22:12 INFO yarn.Client: Setting up container launch context for
our AM
15/12/17 22:22:12 INFO yarn.Client: Setting up the launch

Re: Can't run spark on yarn

2015-12-17 Thread Saisai Shao

Please check the Yarn AM log to see why AM is failed to start. That's the
reason why using `sc` will get such complaint.

On Fri, Dec 18, 2015 at 4:25 AM, Eran Witkon  wrote:

> Hi,
> I am trying to install spark 1.5.2 on Apache hadoop 2.6 and Hive and yarn
>
> spark-env.sh
> export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop
>
> bash_profile
> #HADOOP VARIABLES START
> export JAVA_HOME=/usr/lib/jvm/java-8-oracle/
> export HADOOP_INSTALL=/usr/local/hadoop
> export PATH=$PATH:$HADOOP_INSTALL/bin
> export PATH=$PATH:$HADOOP_INSTALL/sbin
> export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
> export HADOOP_COMMON_HOME=$HADOOP_INSTALL
> export HADOOP_HDFS_HOME=$HADOOP_INSTALL
> export YARN_HOME=$HADOOP_INSTALL
> export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_INSTALL/lib/native
> export HADOOP_OPTS="-Djava.library.path=$HADOOP_INSTALL/lib"
> export HADOOP_USER_CLASSPATH_FIRST=true
> export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop
> export YARN_CONF_DIR=/usr/local/hadoop/etc/hadoop
> #HADOOP VARIABLES END
>
> export SPARK_HOME=/usr/local/spark
> export HIVE_HOME=/usr/local/hive
> export PATH=$PATH:$HIVE_HOME/bin
>
>
> When I run spark-shell
> ./bin/spark-shell --master yarn-client
>
> Output:
> 15/12/17 22:22:07 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 15/12/17 22:22:07 INFO spark.SecurityManager: Changing view acls to: hduser
> 15/12/17 22:22:07 INFO spark.SecurityManager: Changing modify acls to:
> hduser
> 15/12/17 22:22:07 INFO spark.SecurityManager: SecurityManager:
> authentication disabled; ui acls disabled; users with view permissions:
> Set(hduser); users with modify permissions: Set(hduser)
> 15/12/17 22:22:07 INFO spark.HttpServer: Starting HTTP Server
> 15/12/17 22:22:07 INFO server.Server: jetty-8.y.z-SNAPSHOT
> 15/12/17 22:22:08 INFO server.AbstractConnector: Started
> SocketConnector@0.0.0.0:38389
> 15/12/17 22:22:08 INFO util.Utils: Successfully started service 'HTTP
> class server' on port 38389.
> Welcome to
>     __
>  / __/__  ___ _/ /__
> _\ \/ _ \/ _ `/ __/  '_/
>/___/ .__/\_,_/_/ /_/\_\   version 1.5.2
>   /_/
>
> Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java
> 1.8.0_66)
> Type in expressions to have them evaluated.
> Type :help for more information.
> 15/12/17 22:22:11 WARN util.Utils: Your hostname, eranw-Lenovo-Yoga-2-Pro
> resolves to a loopback address: 127.0.1.1; using 10.0.0.1 instead (on
> interface wlp1s0)
> 15/12/17 22:22:11 WARN util.Utils: Set SPARK_LOCAL_IP if you need to bind
> to another address
> 15/12/17 22:22:11 INFO spark.SparkContext: Running Spark version 1.5.2
> 15/12/17 22:22:11 INFO spark.SecurityManager: Changing view acls to: hduser
> 15/12/17 22:22:11 INFO spark.SecurityManager: Changing modify acls to:
> hduser
> 15/12/17 22:22:11 INFO spark.SecurityManager: SecurityManager:
> authentication disabled; ui acls disabled; users with view permissions:
> Set(hduser); users with modify permissions: Set(hduser)
> 15/12/17 22:22:11 INFO slf4j.Slf4jLogger: Slf4jLogger started
> 15/12/17 22:22:11 INFO Remoting: Starting remoting
> 15/12/17 22:22:12 INFO util.Utils: Successfully started service
> 'sparkDriver' on port 36381.
> 15/12/17 22:22:12 INFO Remoting: Remoting started; listening on addresses
> :[akka.tcp://sparkDriver@10.0.0.1:36381]
> 15/12/17 22:22:12 INFO spark.SparkEnv: Registering MapOutputTracker
> 15/12/17 22:22:12 INFO spark.SparkEnv: Registering BlockManagerMaster
> 15/12/17 22:22:12 INFO storage.DiskBlockManager: Created local directory
> at /tmp/blockmgr-139fac31-5f21-4c61-9575-3110d5205f7d
> 15/12/17 22:22:12 INFO storage.MemoryStore: MemoryStore started with
> capacity 530.0 MB
> 15/12/17 22:22:12 INFO spark.HttpFileServer: HTTP File server directory is
> /tmp/spark-955ef002-a802-49c6-b440-0656861f737c/httpd-2127cbe1-97d7-40a5-a96f-75216f115f00
> 15/12/17 22:22:12 INFO spark.HttpServer: Starting HTTP Server
> 15/12/17 22:22:12 INFO server.Server: jetty-8.y.z-SNAPSHOT
> 15/12/17 22:22:12 INFO server.AbstractConnector: Started
> SocketConnector@0.0.0.0:36760
> 15/12/17 22:22:12 INFO util.Utils: Successfully started service 'HTTP file
> server' on port 36760.
> 15/12/17 22:22:12 INFO spark.SparkEnv: Registering OutputCommitCoordinator
> 15/12/17 22:22:12 INFO server.Server: jetty-8.y.z-SNAPSHOT
> 15/12/17 22:22:12 INFO server.AbstractConnector: Started
> SelectChannelConnector@0.0.0.0:4040
> 15/12/17 22:22:12 INFO util.Utils: Successfully started service 'SparkUI'
> on port 4040.
> 15/12/17 22:22:12 INFO ui.SparkUI: Started SparkUI at http://10.0.0.1:4040
> 15/12/17 22:22:12 WARN metrics.MetricsSystem: Using default name
> DAGScheduler for source because spark.app.id is not set.
> 15/12/17 22:22:12 INFO client.RMProxy: Connecting to ResourceManager at /
> 0.0.0.0:8032
> 15/12/17 22:22:12 INFO yarn.Client: Requesting a new application from
> cluster with 1 NodeManagers
> 15/12/17 22:22:12

Re: Can't run spark on yarn

2015-12-17 Thread Alexander Pivovarov

Try to start aws EMR 4.2.0 with hadoop and spark applications on spot
instances. Then look at how hadoop and spark configured. Try to configure
your hadoop and spark similar way
On Dec 17, 2015 6:09 PM, "Saisai Shao"  wrote:

> Please check the Yarn AM log to see why AM is failed to start. That's the
> reason why using `sc` will get such complaint.
>
> On Fri, Dec 18, 2015 at 4:25 AM, Eran Witkon  wrote:
>
>> Hi,
>> I am trying to install spark 1.5.2 on Apache hadoop 2.6 and Hive and yarn
>>
>> spark-env.sh
>> export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop
>>
>> bash_profile
>> #HADOOP VARIABLES START
>> export JAVA_HOME=/usr/lib/jvm/java-8-oracle/
>> export HADOOP_INSTALL=/usr/local/hadoop
>> export PATH=$PATH:$HADOOP_INSTALL/bin
>> export PATH=$PATH:$HADOOP_INSTALL/sbin
>> export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
>> export HADOOP_COMMON_HOME=$HADOOP_INSTALL
>> export HADOOP_HDFS_HOME=$HADOOP_INSTALL
>> export YARN_HOME=$HADOOP_INSTALL
>> export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_INSTALL/lib/native
>> export HADOOP_OPTS="-Djava.library.path=$HADOOP_INSTALL/lib"
>> export HADOOP_USER_CLASSPATH_FIRST=true
>> export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop
>> export YARN_CONF_DIR=/usr/local/hadoop/etc/hadoop
>> #HADOOP VARIABLES END
>>
>> export SPARK_HOME=/usr/local/spark
>> export HIVE_HOME=/usr/local/hive
>> export PATH=$PATH:$HIVE_HOME/bin
>>
>>
>> When I run spark-shell
>> ./bin/spark-shell --master yarn-client
>>
>> Output:
>> 15/12/17 22:22:07 WARN util.NativeCodeLoader: Unable to load
>> native-hadoop library for your platform... using builtin-java classes where
>> applicable
>> 15/12/17 22:22:07 INFO spark.SecurityManager: Changing view acls to:
>> hduser
>> 15/12/17 22:22:07 INFO spark.SecurityManager: Changing modify acls to:
>> hduser
>> 15/12/17 22:22:07 INFO spark.SecurityManager: SecurityManager:
>> authentication disabled; ui acls disabled; users with view permissions:
>> Set(hduser); users with modify permissions: Set(hduser)
>> 15/12/17 22:22:07 INFO spark.HttpServer: Starting HTTP Server
>> 15/12/17 22:22:07 INFO server.Server: jetty-8.y.z-SNAPSHOT
>> 15/12/17 22:22:08 INFO server.AbstractConnector: Started
>> SocketConnector@0.0.0.0:38389
>> 15/12/17 22:22:08 INFO util.Utils: Successfully started service 'HTTP
>> class server' on port 38389.
>> Welcome to
>>     __
>>  / __/__  ___ _/ /__
>> _\ \/ _ \/ _ `/ __/  '_/
>>/___/ .__/\_,_/_/ /_/\_\   version 1.5.2
>>   /_/
>>
>> Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java
>> 1.8.0_66)
>> Type in expressions to have them evaluated.
>> Type :help for more information.
>> 15/12/17 22:22:11 WARN util.Utils: Your hostname, eranw-Lenovo-Yoga-2-Pro
>> resolves to a loopback address: 127.0.1.1; using 10.0.0.1 instead (on
>> interface wlp1s0)
>> 15/12/17 22:22:11 WARN util.Utils: Set SPARK_LOCAL_IP if you need to bind
>> to another address
>> 15/12/17 22:22:11 INFO spark.SparkContext: Running Spark version 1.5.2
>> 15/12/17 22:22:11 INFO spark.SecurityManager: Changing view acls to:
>> hduser
>> 15/12/17 22:22:11 INFO spark.SecurityManager: Changing modify acls to:
>> hduser
>> 15/12/17 22:22:11 INFO spark.SecurityManager: SecurityManager:
>> authentication disabled; ui acls disabled; users with view permissions:
>> Set(hduser); users with modify permissions: Set(hduser)
>> 15/12/17 22:22:11 INFO slf4j.Slf4jLogger: Slf4jLogger started
>> 15/12/17 22:22:11 INFO Remoting: Starting remoting
>> 15/12/17 22:22:12 INFO util.Utils: Successfully started service
>> 'sparkDriver' on port 36381.
>> 15/12/17 22:22:12 INFO Remoting: Remoting started; listening on addresses
>> :[akka.tcp://sparkDriver@10.0.0.1:36381]
>> 15/12/17 22:22:12 INFO spark.SparkEnv: Registering MapOutputTracker
>> 15/12/17 22:22:12 INFO spark.SparkEnv: Registering BlockManagerMaster
>> 15/12/17 22:22:12 INFO storage.DiskBlockManager: Created local directory
>> at /tmp/blockmgr-139fac31-5f21-4c61-9575-3110d5205f7d
>> 15/12/17 22:22:12 INFO storage.MemoryStore: MemoryStore started with
>> capacity 530.0 MB
>> 15/12/17 22:22:12 INFO spark.HttpFileServer: HTTP File server directory
>> is
>> /tmp/spark-955ef002-a802-49c6-b440-0656861f737c/httpd-2127cbe1-97d7-40a5-a96f-75216f115f00
>> 15/12/17 22:22:12 INFO spark.HttpServer: Starting HTTP Server
>> 15/12/17 22:22:12 INFO server.Server: jetty-8.y.z-SNAPSHOT
>> 15/12/17 22:22:12 INFO server.AbstractConnector: Started
>> SocketConnector@0.0.0.0:36760
>> 15/12/17 22:22:12 INFO util.Utils: Successfully started service 'HTTP
>> file server' on port 36760.
>> 15/12/17 22:22:12 INFO spark.SparkEnv: Registering OutputCommitCoordinator
>> 15/12/17 22:22:12 INFO server.Server: jetty-8.y.z-SNAPSHOT
>> 15/12/17 22:22:12 INFO server.AbstractConnector: Started
>> SelectChannelConnector@0.0.0.0:4040
>> 15/12/17 22:22:12 INFO util.Utils: Successfully started service 'SparkUI'
>> on port 4040.
>> 15/12/17 22:22:12 INFO ui.SparkUI: Started

Can't run spark on yarn

Re: Can't run spark on yarn

Re: Can't run spark on yarn

3 matches

Site Navigation

Mail list logo

Footer information