andygrove opened a new issue, #7030:
URL: https://github.com/apache/datafusion-comet/issues/7030
### Describe the bug
When none of the requested fields under a nested column exist in a Parquet
file, Spark's legacy behavior returns the column as NULL, but the native
Parquet scan returns a non-null array or struct that holds NULLs. Legacy is the
only behavior on Spark 3.4, 3.5 and 4.0, and it is what Spark 4.1+ does with
`spark.sql.legacy.parquet.returnNullStructIfAllFieldsMissing=true`. The shapes
are an array whose element struct has no field in the file, a struct whose only
requested child is such a struct, and a struct whose requested children all
lead to missing fields. Schema evolution that replaced a nested struct's fields
produces them. The results are wrong without an error: `count(col)` and `col IS
NULL` both disagree with Spark.
Spark's `ParquetColumnVector` marks any struct, array or map vector all-null
when every child vector is missing, and it applies this bottom-up (Spark 3.5.9
`ParquetColumnVector.java` lines 111-134). An array whose element struct has no
field in the file is therefore missing itself, and so is a struct whose
requested children are all missing at some depth. Comet's
`parquet_convert_struct_to_struct`
(`native/core/src/parquet/parquet_support.rs` lines 565-576) only nulls a
struct when none of its direct requested fields matches a file field, and the
list conversion keeps the list's own validity. So the list stays non-null with
NULL elements, and a struct whose child exists by name but has no matching
leaves stays non-null.
### Steps to reproduce
In a `CometTestBase` suite:
```scala
spark.sql("""SELECT 1 AS id,
| array(named_struct('x', 1, 'y', 'a')) AS arrs,
| named_struct('a', 1, 'c', named_struct('d', 2)) AS s""".stripMargin)
.coalesce(1).write.parquet(path)
val df = spark.read
.schema("id int, arrs array<struct<w:int>>, s struct<c:struct<n:int>>")
.parquet(path)
df.collect()
df.selectExpr("count(arrs)", "count(s)").collect()
```
On `main` at 66c63bd307:
| Spark version and mode | Query | Spark | Comet native scan |
| --- | --- | --- | --- |
| 3.4.3, 3.5.9 (default) | `SELECT *` | `[1,null,null]` |
`[1,[null],[null]]` |
| 3.4.3, 3.5.9 (default) | `count(arrs), count(s)` | `[0,0]` | `[1,1]` |
| 4.1.3, `returnNullStructIfAllFieldsMissing=true` | `SELECT *` |
`[1,null,null]` | `[1,[null],[null]]` |
| 4.1.3, default | `SELECT *` | `[1,[[null]],[[null]]]` |
`[1,[[null]],[[null]]]` |
Over a wider generated dataset the same holds for `sm struct<sa:
array<struct<z9: int>>>` (Spark NULL, Comet `{[null, ...]}`) and for `sm
struct<mm: map<string,int>, sa: array<struct<z9: int>>>`, where Spark keeps
`sm` but returns `sa` as NULL. A map whose value struct has no field in the
file matches Spark, because the keys are present.
### Expected behavior
With the legacy behavior in effect, a requested nested column, array element
or struct field whose subtree has no field in the file reads as NULL at the
highest level whose whole subtree is missing, as Spark's reader does.
### Additional context
Found in a nested-type audit of the native scans. Spark 4.1's default
behavior (SPARK-53535) matches already. A related detail: on Spark before 4.1,
Comet honors
`spark.sql.legacy.parquet.returnNullStructIfAllFieldsMissing=false` when a
session sets it (`CometNativeScan.populateScanConfFlags`), although Spark 3.x
ignores that 4.1-only key, so setting it there makes even the plain all-missing
struct diverge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
