[
https://issues.apache.org/jira/browse/SPARK-28157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-28157:
----------------------------------
Description:
At Spark 2.4.0/2.3.2/2.2.3, SPARK-24948 delegated access permission checks to
the file system, and maintains a blacklist for all event log files failed once
at reading. The blacklisted log files are released back after CLEAN_INTERVAL_S .
However, the files whose size don't changes are ignored forever because
shouldReloadLog return false always when the size is the same with the value in
KVStore. This is recovered only via SHS restart.
was:
At Spark 2.4.0/2.3.2/2.2.3, SPARK-24948 delegated access permission checks to
the file system, and maintains a permanent blacklist for all event log files
failed once at reading. Although this reduces a lot of invalid accesses, there
is no way to see this log files back after the permissions are recovered
correctly. The only way has been restarting SHS.
Apache Spark is unable to know the permission recovery. However, we had better
give a second chances for those blacklisted files in a regular manner.
> Make SHS check Spark event log file permission changes
> ------------------------------------------------------
>
> Key: SPARK-28157
> URL: https://issues.apache.org/jira/browse/SPARK-28157
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.4.0, 2.4.1, 2.4.2, 3.0.0, 2.4.3
> Reporter: Dongjoon Hyun
> Priority: Major
>
> At Spark 2.4.0/2.3.2/2.2.3, SPARK-24948 delegated access permission checks to
> the file system, and maintains a blacklist for all event log files failed
> once at reading. The blacklisted log files are released back after
> CLEAN_INTERVAL_S .
> However, the files whose size don't changes are ignored forever because
> shouldReloadLog return false always when the size is the same with the value
> in KVStore. This is recovered only via SHS restart.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
