andygrove opened a new issue, #7035:
URL: https://github.com/apache/datafusion-comet/issues/7035
### Describe the bug
The native Parquet scan fails to read a file whose schema contains a nested
encoding that arrow-rs cannot convert, even when the query does not read that
column. Spark clips the file schema to the requested columns before converting
it, so it reads the other columns. arrow-rs converts the whole file schema when
it opens the file.
Two shapes that make `SELECT id` fail:
- A LIST-annotated group that is itself repeated, `repeated group rl (LIST)
{ repeated group list { optional int32 element; } }`. Spark infers `rl` as
`array<array<int>>`. Comet fails with `Parquet error: Arrow: List type cannot
be repeated`.
- A MAP whose `key_value` group has a third field. Spark cannot infer that
column's type, but reads the other columns. Comet fails with `Parquet error:
Arrow: Child of map field must have two children, found 3`.
A related shape fails when it is read: `optional group a (LIST) { repeated
group element { repeated int32 v; } }`, a list whose single element field is
itself repeated. Spark infers `array<int>`, and Comet fails with `column: [a,
list, v], physicalType: UNKNOWN, logicalType: int`. Spark's own answer there is
lossy (a record holding `v = [1, 2]` reads as `[1]`), so this one matters less.
### Steps to reproduce
Write each schema with parquet-mr's `RecordConsumer`, with one record that
sets only `id = 1`, then read it with `spark.read.schema("id
int").parquet(path).collect()`.
On `main` at 66c63bd307, the same on Spark 3.5.9 and 4.1.3:
| File column | Spark | Comet native scan |
| --- | --- | --- |
| `repeated group rl (LIST) { repeated group list { optional int32 element;
} }` | `[1]` | `Parquet error: Arrow: List type cannot be repeated` |
| `optional group m3 (MAP) { repeated group key_value { required binary key
(UTF8); optional int32 value; optional int32 extra; } }` | `[1]` | `Parquet
error: Arrow: Child of map field must have two children, found 3` |
### Expected behavior
A column the query does not read does not stop the native scan from reading
the others. Failing that, the native scan could fall back for such files, but
the file schema is only known when a task opens the file.
### Additional context
Found in a nested-type audit of the native scans. These encodings are
non-standard, and no common writer is known to produce them, so this is low
priority.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
