GitHub user zuotingbing opened a pull request:
https://github.com/apache/spark/pull/19398
[SPARK-22174][CORE]Support to automatically create the directory where the
event logs go (`spark.eventLog.dir`)
## What changes were proposed in this pull request?
`2017-09-30 09:47:44,721 ERROR org.apache.spark.SparkContext: Error
initializing SparkContext.
java.io.FileNotFoundException: File file:/tmp/spark-events does not exist
at
org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:611)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:824)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:601)
at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)
at
org.apache.spark.scheduler.EventLoggingListener.start(EventLoggingListener.scala:93)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:516)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2258)
at
org.apache.spark.sql.SparkSession$Builder$$anonfun$8.apply(SparkSession.scala:846)
at
org.apache.spark.sql.SparkSession$Builder$$anonfun$8.apply(SparkSession.scala:838)
at scala.Option.getOrElse(Option.scala:121)
at
org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:838)
at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:31)
at org.apache.spark.examples.SparkPi.main(SparkPi.scala)`
Currently, if our applications are using event logging, the directory where
the event logs go (`spark.eventLog.dir`) should be manually created.
I suggest to create the event log directory automatically in source code,
this will make spark more convenient to use.
## How was this patch tested?
update existing tests
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zuotingbing/spark SPARK-22174
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19398.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19398
----
commit 4c7b646ac1aa5b4ae6e4d594d678e515371a0c9b
Author: zuotingbing <[email protected]>
Date: 2017-09-30T02:05:27Z
[SPARK-22174][CORE]Support to automatically create the directory where the
event logs go (`spark.eventLog.dir`)
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
