yew1eb opened a new pull request, #1609:
URL: https://github.com/apache/auron/pull/1609
# Which issue does this PR close?
Closes #1442 .
# Rationale for this change
The `BuildinfoInSparkUISuite` test failed with
`java.io.FileNotFoundException: File file:/private/tmp/spark-events does not
exist`.
`Utils.createTempDir("/tmp/spark-events")` only creates a random
subdirectory inside the given path, but it does not create the parent directory
`/tmp/spark-events` itself. Spark’s `EventLoggingListener` (started during
`SparkContext` construction) checks `spark.eventLog.dir` and aborts when the
configured directory is missing.
This PR creates a proper temporary directory before the `SparkSession` is
initialized, configures it via `sparkConf`, and cleans it up afterward.
ERROR Log:
```
An exception or error caused a run to abort: File
file:/private/tmp/spark-events does not exist
java.io.FileNotFoundException: File file:/private/tmp/spark-events does not
exist
at
org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:980)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:1301)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:970)
at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:462)
at
org.apache.spark.deploy.history.EventLogFileWriter.requireLogBaseDirAsDirectory(EventLogFileWriters.scala:77)
at
org.apache.spark.deploy.history.SingleEventLogFileWriter.start(EventLogFileWriters.scala:221)
at
org.apache.spark.scheduler.EventLoggingListener.start(EventLoggingListener.scala:83)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:622)
```
# What changes are included in this PR?
# Are there any user-facing changes?
No.
# How was this patch tested?
Verified the test passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
