dwsmith1983 commented on code in PR #6116:
URL: https://github.com/apache/datafusion-comet/pull/6116#discussion_r4100379522
##########
native/core/src/parquet/eager_page_index_reader_factory.rs:
##########
@@ -498,6 +544,20 @@ impl AsyncFileReader for EagerPageIndexReader {
}
let metadata = metadata?;
+ // Spark's `ParquetReadSupport` refuses to open a file that
carries no field ids when
+ // the requested schema carries some, unless `ignoreMissing` is
set, and it walks the
+ // raw `MessageType` to decide. The same walk runs here over the
footer's schema. The
+ // error keeps its Spark type through `ParquetError::External`,
which the JNI layer
+ // unwraps, so the JVM sees the same exception Spark raises.
+ if require_file_field_ids
+ && !parquet_schema_has_field_ids(
+ metadata.file_metadata().schema_descr().root_schema(),
+ )
+ {
+ return Err(ParquetError::External(Box::new(
+ SparkError::ParquetMissingFieldIds,
+ )));
+ }
Review Comment:
`ParquetMissingFieldIds` now carries `file_path` from
`object_meta.location`, sent as `filePath` the way `ParquetSchemaConvert` sends
it. The 4.x shim already reads that parameter, so `cannotReadFilesError` names
the file with no shim change. The path is the object store location without a
scheme, the same form the `NotFound` case already sends. The 3.x shims keep
Spark's plain `RuntimeException`, whose message names no file on Spark either.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
