[
https://issues.apache.org/jira/browse/SPARK-57384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18113658#comment-18113658
]
Rajendra Gujja commented on SPARK-57384:
----------------------------------------
continuing the discussion on this GH issue
https://github.com/apache/spark/issues/55975
I agree that auto-creating event log directories on filesystems with ownership
and permission semantics could conflict with the documented security model.
The current documentation expects spark.eventLog.dir to be manually created
with appropriate ownership and permissions for multi-tenant use. Because of
that, I do not think Spark should broadly auto-create missing event log
directories for HDFS/local filesystems.
I am going to investigate a narrower change for object-store-backed paths only,
where directory ownership semantics are not equivalent to HDFS/POSIX
permissions. If that still seems risky or too special-cased, I can close this
as documentation-backed intended behavior.
Any comments would be appreciated. Thank you!
> Create missing event log directories
> ------------------------------------
>
> Key: SPARK-57384
> URL: https://issues.apache.org/jira/browse/SPARK-57384
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 4.0.2
> Reporter: Rajendra Gujja
> Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
