[
https://issues.apache.org/jira/browse/SPARK-9636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14681403#comment-14681403
]
Philipp Angerer commented on SPARK-9636:
----------------------------------------
OK, great :)
I see why you think my proposal might be to complex, yet I still think that
“log file relative to binary” is much more surprising in an environment where
log files have certain dedicated places.
{{/var/log/}} is something i really expect a system daemon to use for logs.
{{~/.cache/logs}} is merely the best compromise in absence of a dedicated user
log directoy. (e.g. {{$XDG_USER_DATA_DIR}} and {{$XDG_USER_CONFIG_DIR}} are
clear, but there’s no {{$XDG_USER_STATE_DIR}})
i think all this is a consequence of spark not being a good linux citizen. it
has a {{$SPARK_HOME}}, and relies on it, while there should be a way to run it
split up to sensible directories: {{/usr/share/spark/}} for data
{{/usr/lib/spark/}} for shared libraries, {{/usr/lib/pythonx.x/site-packages/}}
for pyspark, {{/usr/bin/}} for binaries and scripts, {{/etc/spark/}} for
configs, and {{/var/log/spark}} for logfiles.
> Treat $SPARK_HOME as write-only
> -------------------------------
>
> Key: SPARK-9636
> URL: https://issues.apache.org/jira/browse/SPARK-9636
> Project: Spark
> Issue Type: Improvement
> Components: Input/Output
> Affects Versions: 1.4.1
> Environment: Linux
> Reporter: Philipp Angerer
> Priority: Minor
> Labels: easyfix
>
> when starting spark scripts as user and it is installed in a directory the
> user has no write permissions on, many things work fine, except for the logs
> (e.g. for {{start-master.sh}})
> logs are per default written to {{$SPARK_LOG_DIR}} or (if unset) to
> {{$SPARK_HOME/logs}}.
> if installed in this way, it should, instead of throwing an error, write logs
> to {{/var/log/spark/}}. that’s easy to fix by simply testing a few log dirs
> in sequence for writability before trying to use one. i suggest using
> {{$SPARK_LOG_DIR}} (if set) → {{/var/log/spark/}} → {{~/.cache/spark-logs/}}
> → {{$SPARK_HOME/logs/}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
