[
https://issues.apache.org/jira/browse/SPARK-11074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14954007#comment-14954007
]
Marcelo Vanzin commented on SPARK-11074:
----------------------------------------
BTW there have been previous attempts to do this, and I still consider it a bad
idea. For starters, you don't know if the user running the spark app has
permissions to create the directory. The code should also make sure the
directory has the right permissions (which your PR does not do). And it creates
a race (actually more than one) in case multiple applications are starting at
the same time and the directory does not exist.
I consider the log directory part of the infrastructure, and Spark apps should
not be trying to create it.
> Attempt to automatically create event log directory if it doesn't already
> exist
> -------------------------------------------------------------------------------
>
> Key: SPARK-11074
> URL: https://issues.apache.org/jira/browse/SPARK-11074
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 1.5.1
> Reporter: Kay Ousterhout
> Assignee: Kay Ousterhout
> Priority: Minor
>
> Right now, if the log where event logs are to be placed (spark.eventLog.dir)
> doesn't already exist, the SparkContext fails to start. We should try to
> automatically create this directory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
