[
https://issues.apache.org/jira/browse/SPARK-980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14289139#comment-14289139
]
Sean Owen commented on SPARK-980:
---------------------------------
I believe Nan is correct that this was subsequently resolved. Although the
references to source and PRs aren't live anymore, I see this in Worker.scala:
{code}
new File(sys.env.get("SPARK_HOME").getOrElse("."))
{code}
and this is the only reference to {{SPARK_HOME}}, so it seems to handle the
case where this is missing.
> NullPointerException for single-host setup with S3 URLs
> -------------------------------------------------------
>
> Key: SPARK-980
> URL: https://issues.apache.org/jira/browse/SPARK-980
> Project: Spark
> Issue Type: Bug
> Components: Input/Output
> Affects Versions: 0.8.0
> Reporter: Paul R. Brown
>
> Short version:
> * The use of {{execSparkHome_}} in
> [Worker.scala|https://github.com/apache/incubator-spark/blob/v0.8.0-incubating/core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala#L135]
> should be checked for {{null}} or that value should be defaulted or plumbed
> through.
> * If the {{sparkHome}} argument to {{new SparkContext(...)}} is non-optional,
> then it should not be marked as optional.
> Long version:
> Starting up with {{bin/start-all.sh}} and then connecting from a Scala
> program and attempting to read two S3 URLs results in the following trace in
> the worker log:
> {code}
> 13/12/03 21:50:23 ERROR worker.Worker:
> java.lang.NullPointerException
> at java.io.File.<init>(File.java:277)
> at
> org.apache.spark.deploy.worker.Worker$$anonfun$receive$1.apply(Worker.scala:135)
> at
> org.apache.spark.deploy.worker.Worker$$anonfun$receive$1.apply(Worker.scala:120)
> at akka.actor.Actor$class.apply(Actor.scala:318)
> at org.apache.spark.deploy.worker.Worker.apply(Worker.scala:39)
> at akka.actor.ActorCell.invoke(ActorCell.scala:626)
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:197)
> at akka.dispatch.Mailbox.run(Mailbox.scala:179)
> at
> akka.dispatch.ForkJoinExecutorConfigurator$MailboxExecutionTask.exec(AbstractDispatcher.scala:516)
> at akka.jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:259)
> at akka.jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:975)
> at akka.jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1479)
> at akka.jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)
> {code}
> This is on Mac OS X 10.9, Oracle Java 7u45, and the Hadoop 1 download from
> the incubator.
> Reading the code, this occurs because {{execSparkHome_}} is {{null}}; see
> [Worker.scala#L135|https://github.com/apache/incubator-spark/blob/v0.8.0-incubating/core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala#L135],
> and setting a value explicitly in the Scala driver allows the computation to
> complete.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
