rdtr opened a new pull request, #13207:
URL: https://github.com/apache/gluten/pull/13207
## What changes are proposed in this pull request?
Related to #13180.
Spark 4.2 has a new test in ParquetQuerySuite: "SPARK-55857:
Enabling/disabling ignoreMissingFiles". You can set ignoreMissingFiles in two
places:
- the session conf: `spark.sql.files.ignoreMissingFiles`
- the read option: `spark.read.option("ignoreMissingFiles", ...)`
When both are set, Spark uses the read option (see
`FileSourceOptions.ignoreMissingFiles`). Spark works like this since 3.4. So
this is an old problem. The new test only made it visible.
### What happened
I ran the same steps on Gluten:
1. Write two parquet dirs.
2. Create the DataFrame.
3. Delete the files in the first dir.
4. Collect.
| conf | read option | Spark | Gluten (before) |
|---|---|---|---|
| true | (none) | skips the missing file | same |
| false | true | skips the missing file | error: `VeloxRuntimeError ...
FILE_NOT_FOUND` |
| false | (none) | error | same |
| true | false | error | skips the missing file, no error |
The last row is the worst. The user wants an error. But Gluten gives no
error and just returns fewer rows.
### Why
Gluten sends only the session conf to Velox:
- `GlutenConfig.scala`: `spark.sql.files.ignoreMissingFiles` is sent to
native.
- `cpp/velox/utils/ConfigExtractor.cc`: it is set to
`HiveConfig::kIgnoreMissingFilesSession`.
The read option is in `relation.options` (V1) or in `ParquetScan.options` /
`OrcScan.options` (V2). Gluten does not look at it. So Velox always uses the
session conf.
### Fix
In Velox, this setting has one value for the whole query. But one query can
have two scans with different options. So we cannot just send the option to
Velox. For now, this PR falls back the scan:
- `BasicScanExecTransformer`: add `getReadOptions`. It is empty by default.
In `doValidateInternal`, get the value in the same way as Spark: `new
FileSourceOptions(getReadOptions).ignoreMissingFiles`. If it is different from
`SQLConf.get.ignoreMissingFiles`, the validation fails and the scan falls back.
- `FileSourceScanExecTransformer`: `getReadOptions` returns
`relation.options`. Spark gives the same map to `FileScanRDD`.
- `BatchScanExecTransformer`: `getReadOptions` returns the options of
`ParquetScan` / `OrcScan`.
If there is no option, or the option has the same value as the conf, the
scan still runs in native. Other scans (Hive, Iceberg, ...) use the empty map,
so they do not change.
This fallback is a short-term fix. The real fix should be in Velox. Today
`FileSplitReader.cpp` reads ignoreMissingFiles only from the session
properties, so each scan cannot have its own value. If `HiveTableHandle` can
keep a value for each scan, Gluten can send the read option in the ReadRel.
Then we can remove this fallback. I will work on it in a follow-up.
Spark has the same kind of read option for `ignoreCorruptFiles`. This PR
does not change it.
## How was this patch tested?
I added a test "ignoreMissingFiles read option overrides the session conf"
to VeloxScanSuite. It checks the four rows in the table above, for both V1 and
V2 scans. It also checks that the scan runs in native when there is no option,
and falls back when there is an option.
- Spark 4.2 and 4.1 (SPARK_ANSI_SQL_MODE=false): VeloxScanSuite passed (11
tests).
- I also removed the change in gluten-substrait and ran the test again on
Spark 4.2. It failed with `VeloxRuntimeError ... FILE_NOT_FOUND` in the
conf=false / option=true case.
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
