Re: Spark YARN Error - triggering spark-shell
Fixed by adding 2 configurations in yarn-site,xml.
Thanks all!
On Fri, Jun 8, 2018 at 2:44 PM, Aakash Basu
wrote:
> Hi,
>
> I fixed that problem by putting all the Spark JARS in spark-archive.zip
> and putting it in the HDFS (as that problem was happening for that reason) -
>
> But, I'm facing a new issue now, this is the new RPC error I get
> (Stack-Trace below) -
>
>
>
>
> 2018-06-08 14:26:43 WARN NativeCodeLoader:62 - Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> 2018-06-08 14:26:45 INFO SparkContext:54 - Running Spark version 2.3.0
> 2018-06-08 14:26:45 INFO SparkContext:54 - Submitted application:
> EndToEnd_FeatureEngineeringPipeline
> 2018-06-08 14:26:45 INFO SecurityManager:54 - Changing view acls to:
> bblite
> 2018-06-08 14:26:45 INFO SecurityManager:54 - Changing modify acls to:
> bblite
> 2018-06-08 14:26:45 INFO SecurityManager:54 - Changing view acls groups
> to:
> 2018-06-08 14:26:45 INFO SecurityManager:54 - Changing modify acls groups
> to:
> 2018-06-08 14:26:45 INFO SecurityManager:54 - SecurityManager:
> authentication disabled; ui acls disabled; users with view permissions:
> Set(bblite); groups with view permissions: Set(); users with modify
> permissions: Set(bblite); groups with modify permissions: Set()
> 2018-06-08 14:26:45 INFO Utils:54 - Successfully started service
> 'sparkDriver' on port 41957.
> 2018-06-08 14:26:45 INFO SparkEnv:54 - Registering MapOutputTracker
> 2018-06-08 14:26:45 INFO SparkEnv:54 - Registering BlockManagerMaster
> 2018-06-08 14:26:45 INFO BlockManagerMasterEndpoint:54 - Using
> org.apache.spark.storage.DefaultTopologyMapper for getting topology
> information
> 2018-06-08 14:26:45 INFO BlockManagerMasterEndpoint:54 -
> BlockManagerMasterEndpoint up
> 2018-06-08 14:26:45 INFO DiskBlockManager:54 - Created local directory at
> /appdata/spark/tmp/blockmgr-7b035871-a1f7-47ff-aad8-f7a43367836e
> 2018-06-08 14:26:45 INFO MemoryStore:54 - MemoryStore started with
> capacity 366.3 MB
> 2018-06-08 14:26:45 INFO SparkEnv:54 - Registering OutputCommitCoordinator
> 2018-06-08 14:26:45 INFO log:192 - Logging initialized @3659ms
> 2018-06-08 14:26:45 INFO Server:346 - jetty-9.3.z-SNAPSHOT
> 2018-06-08 14:26:45 INFO Server:414 - Started @3733ms
> 2018-06-08 14:26:45 INFO AbstractConnector:278 - Started
> ServerConnector@3080efb7{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
> 2018-06-08 14:26:45 INFO Utils:54 - Successfully started service
> 'SparkUI' on port 4040.
> 2018-06-08 14:26:45 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@2c3409b5{/jobs,null,AVAILABLE,@Spark}
> 2018-06-08 14:26:45 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@7f1ba569{/jobs/json,null,AVAILABLE,@Spark}
> 2018-06-08 14:26:45 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@493631a1{/jobs/job,null,AVAILABLE,@Spark}
> 2018-06-08 14:26:45 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@6b12f33c{/jobs/job/json,null,
> AVAILABLE,@Spark}
> 2018-06-08 14:26:45 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@490023da{/stages,null,AVAILABLE,@Spark}
> 2018-06-08 14:26:45 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@31c3a862{/stages/json,null,AVAILABLE,@Spark}
> 2018-06-08 14:26:45 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@4da2454f{/stages/stage,null,
> AVAILABLE,@Spark}
> 2018-06-08 14:26:45 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@552f182d{/stages/stage/json,
> null,AVAILABLE,@Spark}
> 2018-06-08 14:26:45 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@a78a7fa{/stages/pool,null,AVAILABLE,@Spark}
> 2018-06-08 14:26:45 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@15142105{/stages/pool/json,
> null,AVAILABLE,@Spark}
> 2018-06-08 14:26:45 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@7589c977{/storage,null,AVAILABLE,@Spark}
> 2018-06-08 14:26:45 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@584a599b{/storage/json,null,
> AVAILABLE,@Spark}
> 2018-06-08 14:26:45 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@1742621f{/storage/rdd,null,AVAILABLE,@Spark}
> 2018-06-08 14:26:45 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@23ea75fb{/storage/rdd/json,
> null,AVAILABLE,@Spark}
> 2018-06-08 14:26:45 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@1813d280{/environment,null,AVAILABLE,@Spark}
> 2018-06-08 14:26:45 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@129fc698{/environment/json,
> null,AVAILABLE,@Spark}
> 2018-06-08 14:26:45 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@16c91c4e{/executors,null,AVAILABLE,@Spark}
> 2018-06-08 14:26:45 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@667ce6c1{/executors/json,null,
> AVAILABLE,@Spark}
> 2018-06-08 14:26:45 INFO
Re: Spark YARN Error - triggering spark-shell
Hi,
I fixed that problem by putting all the Spark JARS in spark-archive.zip and
putting it in the HDFS (as that problem was happening for that reason) -
But, I'm facing a new issue now, this is the new RPC error I get
(Stack-Trace below) -
2018-06-08 14:26:43 WARN NativeCodeLoader:62 - Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
2018-06-08 14:26:45 INFO SparkContext:54 - Running Spark version 2.3.0
2018-06-08 14:26:45 INFO SparkContext:54 - Submitted application:
EndToEnd_FeatureEngineeringPipeline
2018-06-08 14:26:45 INFO SecurityManager:54 - Changing view acls to: bblite
2018-06-08 14:26:45 INFO SecurityManager:54 - Changing modify acls to:
bblite
2018-06-08 14:26:45 INFO SecurityManager:54 - Changing view acls groups
to:
2018-06-08 14:26:45 INFO SecurityManager:54 - Changing modify acls groups
to:
2018-06-08 14:26:45 INFO SecurityManager:54 - SecurityManager:
authentication disabled; ui acls disabled; users with view permissions:
Set(bblite); groups with view permissions: Set(); users with modify
permissions: Set(bblite); groups with modify permissions: Set()
2018-06-08 14:26:45 INFO Utils:54 - Successfully started service
'sparkDriver' on port 41957.
2018-06-08 14:26:45 INFO SparkEnv:54 - Registering MapOutputTracker
2018-06-08 14:26:45 INFO SparkEnv:54 - Registering BlockManagerMaster
2018-06-08 14:26:45 INFO BlockManagerMasterEndpoint:54 - Using
org.apache.spark.storage.DefaultTopologyMapper for getting topology
information
2018-06-08 14:26:45 INFO BlockManagerMasterEndpoint:54 -
BlockManagerMasterEndpoint up
2018-06-08 14:26:45 INFO DiskBlockManager:54 - Created local directory at
/appdata/spark/tmp/blockmgr-7b035871-a1f7-47ff-aad8-f7a43367836e
2018-06-08 14:26:45 INFO MemoryStore:54 - MemoryStore started with
capacity 366.3 MB
2018-06-08 14:26:45 INFO SparkEnv:54 - Registering OutputCommitCoordinator
2018-06-08 14:26:45 INFO log:192 - Logging initialized @3659ms
2018-06-08 14:26:45 INFO Server:346 - jetty-9.3.z-SNAPSHOT
2018-06-08 14:26:45 INFO Server:414 - Started @3733ms
2018-06-08 14:26:45 INFO AbstractConnector:278 - Started
ServerConnector@3080efb7{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2018-06-08 14:26:45 INFO Utils:54 - Successfully started service 'SparkUI'
on port 4040.
2018-06-08 14:26:45 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@2c3409b5{/jobs,null,AVAILABLE,@Spark}
2018-06-08 14:26:45 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@7f1ba569{/jobs/json,null,AVAILABLE,@Spark}
2018-06-08 14:26:45 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@493631a1{/jobs/job,null,AVAILABLE,@Spark}
2018-06-08 14:26:45 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@6b12f33c{/jobs/job/json,null,AVAILABLE,@Spark}
2018-06-08 14:26:45 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@490023da{/stages,null,AVAILABLE,@Spark}
2018-06-08 14:26:45 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@31c3a862{/stages/json,null,AVAILABLE,@Spark}
2018-06-08 14:26:45 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@4da2454f{/stages/stage,null,AVAILABLE,@Spark}
2018-06-08 14:26:45 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@552f182d
{/stages/stage/json,null,AVAILABLE,@Spark}
2018-06-08 14:26:45 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@a78a7fa{/stages/pool,null,AVAILABLE,@Spark}
2018-06-08 14:26:45 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@15142105
{/stages/pool/json,null,AVAILABLE,@Spark}
2018-06-08 14:26:45 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@7589c977{/storage,null,AVAILABLE,@Spark}
2018-06-08 14:26:45 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@584a599b{/storage/json,null,AVAILABLE,@Spark}
2018-06-08 14:26:45 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@1742621f{/storage/rdd,null,AVAILABLE,@Spark}
2018-06-08 14:26:45 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@23ea75fb
{/storage/rdd/json,null,AVAILABLE,@Spark}
2018-06-08 14:26:45 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@1813d280{/environment,null,AVAILABLE,@Spark}
2018-06-08 14:26:45 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@129fc698
{/environment/json,null,AVAILABLE,@Spark}
2018-06-08 14:26:45 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@16c91c4e{/executors,null,AVAILABLE,@Spark}
2018-06-08 14:26:45 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@667ce6c1
{/executors/json,null,AVAILABLE,@Spark}
2018-06-08 14:26:45 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@60fdbf5c
{/executors/threadDump,null,AVAILABLE,@Spark}
2018-06-08 14:26:45 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@c3a1edd
{/executors/threadDump/json,null,AVAILABLE,@Spark}
2018-06-08 14:26:45 INFO ContextHandler:781 - Started
Re: Spark YARN Error - triggering spark-shell
It seems, your spark-on-yarn application is not able to get it's application master, org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master. Check once on yarn logs Thanks, Sathish- On Fri, Jun 8, 2018 at 2:22 PM, Jeff Zhang wrote: > > Check the yarn AM log for details. > > > > Aakash Basu 于2018年6月8日周五 下午4:36写道： > >> Hi, >> >> Getting this error when trying to run Spark Shell using YARN - >> >> Command: *spark-shell --master yarn --deploy-mode client* >> >> 2018-06-08 13:39:09 WARN Client:66 - Neither spark.yarn.jars nor >> spark.yarn.archive is set, falling back to uploading libraries under >> SPARK_HOME. >> 2018-06-08 13:39:25 ERROR SparkContext:91 - Error initializing SparkContext. >> org.apache.spark.SparkException: Yarn application has already ended! It >> might have been killed or unable to launch application master. >> >> >> The last half of stack-trace - >> >> 2018-06-08 13:56:11 WARN YarnSchedulerBackend$YarnSchedulerEndpoint:66 - >> Attempted to request executors before the AM has registered! >> 2018-06-08 13:56:11 WARN MetricsSystem:66 - Stopping a MetricsSystem that >> is not running >> org.apache.spark.SparkException: Yarn application has already ended! It >> might have been killed or unable to launch application master. >> at >> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:89) >> at >> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:63) >> at >> org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:164) >> at org.apache.spark.SparkContext.(SparkContext.scala:500) >> at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2486) >> at >> org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:930) >> at >> org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:921) >> at scala.Option.getOrElse(Option.scala:121) >> at >> org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:921) >> at org.apache.spark.repl.Main$.createSparkSession(Main.scala:103) >> ... 55 elided >> :14: error: not found: value spark >>import spark.implicits._ >> ^ >> :14: error: not found: value spark >>import spark.sql >> >> >> Tried putting the *spark-yarn_2.11-2.3.0.jar *in Hadoop yarn, still not >> working, anything else to do? >> >> Thanks, >> Aakash. >> >
Re: Spark YARN Error - triggering spark-shell
Check the yarn AM log for details. Aakash Basu 于2018年6月8日周五 下午4:36写道： > Hi, > > Getting this error when trying to run Spark Shell using YARN - > > Command: *spark-shell --master yarn --deploy-mode client* > > 2018-06-08 13:39:09 WARN Client:66 - Neither spark.yarn.jars nor > spark.yarn.archive is set, falling back to uploading libraries under > SPARK_HOME. > 2018-06-08 13:39:25 ERROR SparkContext:91 - Error initializing SparkContext. > org.apache.spark.SparkException: Yarn application has already ended! It might > have been killed or unable to launch application master. > > > The last half of stack-trace - > > 2018-06-08 13:56:11 WARN YarnSchedulerBackend$YarnSchedulerEndpoint:66 - > Attempted to request executors before the AM has registered! > 2018-06-08 13:56:11 WARN MetricsSystem:66 - Stopping a MetricsSystem that is > not running > org.apache.spark.SparkException: Yarn application has already ended! It might > have been killed or unable to launch application master. > at > org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:89) > at > org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:63) > at > org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:164) > at org.apache.spark.SparkContext.(SparkContext.scala:500) > at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2486) > at > org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:930) > at > org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:921) > at scala.Option.getOrElse(Option.scala:121) > at > org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:921) > at org.apache.spark.repl.Main$.createSparkSession(Main.scala:103) > ... 55 elided > :14: error: not found: value spark >import spark.implicits._ > ^ > :14: error: not found: value spark >import spark.sql > > > Tried putting the *spark-yarn_2.11-2.3.0.jar *in Hadoop yarn, still not > working, anything else to do? > > Thanks, > Aakash. >
Spark YARN Error - triggering spark-shell
Hi, Getting this error when trying to run Spark Shell using YARN - Command: *spark-shell --master yarn --deploy-mode client* 2018-06-08 13:39:09 WARN Client:66 - Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME. 2018-06-08 13:39:25 ERROR SparkContext:91 - Error initializing SparkContext. org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master. The last half of stack-trace - 2018-06-08 13:56:11 WARN YarnSchedulerBackend$YarnSchedulerEndpoint:66 - Attempted to request executors before the AM has registered! 2018-06-08 13:56:11 WARN MetricsSystem:66 - Stopping a MetricsSystem that is not running org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master. at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:89) at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:63) at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:164) at org.apache.spark.SparkContext.(SparkContext.scala:500) at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2486) at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:930) at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:921) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:921) at org.apache.spark.repl.Main$.createSparkSession(Main.scala:103) ... 55 elided :14: error: not found: value spark import spark.implicits._ ^ :14: error: not found: value spark import spark.sql Tried putting the *spark-yarn_2.11-2.3.0.jar *in Hadoop yarn, still not working, anything else to do? Thanks, Aakash.
