andygrove opened a new issue, #7207:
URL: https://github.com/apache/datafusion-comet/issues/7207
### Describe the bug
With `spark.sql.parquet.fieldId.read.enabled=true`, the native Parquet scan
never checks the requested type of a root field whose file column another
requested field claims by id. Spark checks it and fails the read.
Take a file with `A` (id 1, `BIGINT`) = 7, read with the schema `[x (id 1)
BIGINT, a (no id) STRUCT<v BIGINT>]`. Spark's `clipParquetGroupFields` resolves
each requested field on its own, so `x` selects `A` by id, `a` selects `A` by
name, and clipping the primitive `A` against the struct `a` fails with
`ClassCastException: optional int64 A = 1 is not a group` in both Spark
readers. Comet gives `A` to `x`, the stronger claim, and reads `a` as a missing
column, so it returns `[7, null]`.
A primitive with a type that Spark can't read the column as behaves the same
way. With `a INT` over `A BIGINT`, Spark's vectorized reader fails with
`SchemaColumnConvertNotSupportedException` once it decodes a row group, because
`VectorizedColumnReader.readBatch` calls `getUpdater` for every requested leaf.
parquet-mr returns NULL, and Comet returns NULL.
Inside a struct, Comet raises both errors on `main`.
### Steps to reproduce
In a `CometTestBase` suite on the default Spark 4.1.3 profile:
```scala
def withId(id: Int) =
new MetadataBuilder().putLong(ParquetUtils.FIELD_ID_METADATA_KEY,
id.toLong).build()
withSQLConf(SQLConf.PARQUET_FIELD_ID_WRITE_ENABLED.key -> "true") {
spark
.createDataFrame(
spark.sparkContext.parallelize(Seq(Row(7L))),
new StructType().add("A", LongType, true, withId(1)))
.write
.parquet(path)
}
val readSchema = new StructType()
.add("x", LongType, true, withId(1))
.add("a", new StructType().add("v", LongType), true)
withSQLConf(SQLConf.PARQUET_FIELD_ID_READ_ENABLED.key -> "true") {
spark.read.schema(readSchema).parquet(path).collect()
}
```
On `main` at 330b29cf10:
| Read schema | Spark vectorized | Spark parquet-mr | Comet native scan |
|---|---|---|---|
| `[x id1 bigint, a struct<v bigint>]` | `ClassCastException: optional int64
A = 1 is not a group` | the same | `[7, null]` |
| `[a struct<v bigint>, x id1 bigint]` | the same | the same | `[7, null]` |
| `[x id1 bigint, a array<struct<v bigint>>]` | the same | the same | `[7,
null]` |
| `[x id1 bigint, a int]` | `SchemaColumnConvertNotSupportedException:
column: [A], physicalType: INT64, logicalType: int` | `[7, null]` | `[7, null]`
|
| `s struct<x id1 bigint, a struct<v bigint>>` | `ClassCastException` |
`ClassCastException` | `SchemaColumnConvertNotSupportedException` at `[s, A]` |
### Expected behavior
Comet should check a losing root field against the file column it selects,
as it does for a nested field: fail at file open where Spark's clipping fails,
and defer a type rejection to a non-empty row group the way `RejectOnNonEmpty`
does.
### Additional context
Found while reviewing #7196, which keeps the struct check for a nested field
that loses its claim. The root reads such a field as missing before any type
check runs. `branch-1.2` has the same code as `main` here.
The shape needs field-id reads and a read schema in which a field without an
id has the name of a file column that another field selects by id, so it should
be rare in practice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
