[
https://issues.apache.org/jira/browse/SPARK-30860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun resolved SPARK-30860.
-----------------------------------
Fix Version/s: 3.0.0
Resolution: Fixed
Issue resolved by pull request 27764
[https://github.com/apache/spark/pull/27764]
> Different behavior between rolling and non-rolling event log
> ------------------------------------------------------------
>
> Key: SPARK-30860
> URL: https://issues.apache.org/jira/browse/SPARK-30860
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
> Affects Versions: 3.0.0
> Reporter: Adam Binford
> Assignee: Adam Binford
> Priority: Major
> Fix For: 3.0.0
>
>
> When creating a rolling event log, the application directory is created with
> a call to FileSystem.mkdirs, with the file permission 770. The default
> behavior of HDFS is to set the permission of a file created with
> FileSystem.create or FileSystem.mkdirs to (P & ^umask), where P is the
> permission in the API call and umask is a system value set by
> fs.permissions.umask-mode and defaults to 0022. This means, with default
> settings, any mkdirs call can have at most 755 permissions, which causes
> rolling event log directories to be created with 750 permissions. This causes
> the history server to be unable to prune old applications if they are not run
> by the same user running the history server.
> This is not a problem for non-rolling logs, because it uses
> SparkHadoopUtils.createFile for Hadoop 2 backward compatibility, and then
> calls FileSystem.setPermission with 770 after the file has been created.
> setPermission doesn't have the umask applied to it, so this works fine.
> Obviously this could be fixed by changing fs.permissions.umask-mode, but I'm
> not sure the reason that's set in the first place or if this would hurt
> anything else. The main issue is there is different behavior between rolling
> and non-rolling event logs that might want to be updated in this repo to be
> consistent across each.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
