Linhong Liu created SPARK-61585:
-----------------------------------
Summary: spark.sql.files.ignoreCorruptFiles swallows a
SparkUpgradeException and skips the rest of the file instead of raising it
Key: SPARK-61585
URL: https://issues.apache.org/jira/browse/SPARK-61585
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: Linhong Liu
DataSourceUtils.shouldIgnoreCorruptFileException returns true for every
RuntimeException and IOException, which includes SparkUpgradeException (e.g.
INCONSISTENT_BEHAVIOR_CROSS_VERSION from a datetime pattern or value only the
legacy parser accepts). So with spark.sql.files.ignoreCorruptFiles=true, the
file is truncated at the first row that raises the upgrade error and the query
silently returns partial data, even though the parsers rethrow the upgrade
error on purpose rather than turning it into a bad record.
Fix: do not treat a SparkUpgradeException (or a ParquetDecodingException
wrapping one) as file corruption; raise it. A new conf
spark.sql.legacy.files.ignoreCorruptFilesSkipsUpgradeErrors (default false)
restores the previous behavior. Found via correctness analysis.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]