[
https://issues.apache.org/jira/browse/SPARK-28157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-28157:
----------------------------------
Description:
At Spark 2.4.0/2.3.2/2.2.3, SPARK-24948 delegated access permission checks to
the file system, and maintains a permanent blacklist for all event log files
failed once at reading. Although this reduces a lot of invalid accesses, there
is no way to see this log files back after the permissions are recovered
correctly. The only way has been restarting SHS.
Apache Spark is unable to know the permission recovery. However, we had better
give a second chances for those blacklisted files in a regular manner.
was:
Since Spark 2.4.0, SPARK-24948 delegated access permission checks to the file
system, and maintains a permanent blacklist for all event log files failed once
at reading. Although this reduces a lot of invalid accesses, there is no way to
see this log files back after the permissions are recovered correctly. The only
way has been restarting SHS.
Apache Spark is unable to know the permission recovery. However, we had better
give a second chances for those blacklisted files in a regular manner.
> Make SHS check Spark event log file permission changes
> ------------------------------------------------------
>
> Key: SPARK-28157
> URL: https://issues.apache.org/jira/browse/SPARK-28157
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.4.0, 2.4.1, 2.4.2, 3.0.0, 2.4.3
> Reporter: Dongjoon Hyun
> Priority: Major
>
> At Spark 2.4.0/2.3.2/2.2.3, SPARK-24948 delegated access permission checks to
> the file system, and maintains a permanent blacklist for all event log files
> failed once at reading. Although this reduces a lot of invalid accesses,
> there is no way to see this log files back after the permissions are
> recovered correctly. The only way has been restarting SHS.
> Apache Spark is unable to know the permission recovery. However, we had
> better give a second chances for those blacklisted files in a regular manner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
