Max Gekk created SPARK-45316:
--------------------------------
Summary: Respect `spark.sql.files.ignoreMissingFiles` in HadoopRDD
Key: SPARK-45316
URL: https://issues.apache.org/jira/browse/SPARK-45316
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.0.0
Reporter: Max Gekk
Assignee: Max Gekk
Currently, the SQL config spark.sql.files.ignoreMissingFiles influences on RDDs
created in Spark SQL such as FileScanRDD but doesn't impact on HadoopRDD and
NewHadoopRDD. The last RDDs have separate core config
spark.files.ignoreMissingFiles. That inconsistency might confuse users if they
don't know implementation details. This ticket aims to eliminate the
inconsistency.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
