Yousof Hosny created SPARK-49617:
------------------------------------
Summary: Stop ignoring auth errors when ignoreCorruptFiles=1 in
FileScanRDD
Key: SPARK-49617
URL: https://issues.apache.org/jira/browse/SPARK-49617
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 4.0.0
Reporter: Yousof Hosny
We currently lose data when the ignoreCorruptFiles flag is set to true because
file Authorization Errors are handled as if they were corrupt files. We ignore
these files despite them having no corrupt records. We’d like to fix this bug
by failing the stream when authorization errors are found.
To prevent regressions we will add a SQLConf flag that can be temporarily set
to revert to previous behavior. The idea is to deprecate the flag at a later
major version release.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
