andygrove opened a new issue, #7029:
URL: https://github.com/apache/datafusion-comet/issues/7029
### Describe the bug
The native Parquet scan returns wrong values for INT96 timestamps inside a
list that is not in the standard three-level `list` / `element` layout,
whenever the value is outside 1677-09-21 to 2262-04-11. That covers files Spark
writes with `spark.sql.parquet.writeLegacyFormat=true` (`repeated group bag {
optional int96 array; }`), Hive's `bag` / `array_element` layout, and the
two-level `repeated int96 array` form. INT96 is still Spark's default
`spark.sql.parquet.outputTimestampType`, so an array of timestamps written in
the legacy format for Hive or Impala compatibility is enough, and sentinel
values such as `9999-12-31` are common in warehouse tables. Nothing fails, so
the wrong values are silent.
DataFusion's `Int96Coercer` (`datafusion-datasource-parquet` 55.2.0,
`schema_coercion.rs` around line 348) builds each leaf's Parquet path by
pushing a hard-coded `".list."` between a list field and its child, and only
coerces a leaf whose path appears among the file's INT96 columns (around line
401). A legacy list's leaf path is `at.bag.array` or `at.array`, never
`at.list.array`, so the leaf is never coerced and stays
`Timestamp(Nanosecond)`. arrow-rs then decodes the INT96 value into a 64-bit
nanosecond count, which wraps outside the range above, and the adapter's later
cast to microseconds keeps the wrapped value. apache/datafusion#15763 fixed
INT96 coercion for nested types in the standard layout only. Maps, structs,
standard lists and top-level columns are not affected, because their paths
match.
### Steps to reproduce
In a `CometTestBase` suite:
```scala
val schema = StructType(Seq(StructField("id", IntegerType),
StructField("at", ArrayType(TimestampType))))
val rows = Seq("9999-12-31 23:59:59", "1600-01-01 00:00:00", "2263-01-01
00:00:00", "2020-01-01 00:00:00")
.zipWithIndex.map { case (s, i) => Row(i,
Seq(java.sql.Timestamp.valueOf(s))) }
withSQLConf(
SQLConf.PARQUET_WRITE_LEGACY_FORMAT.key -> "true",
SQLConf.PARQUET_OUTPUT_TIMESTAMP_TYPE.key -> "INT96",
SQLConf.PARQUET_INT96_REBASE_MODE_IN_WRITE.key -> "CORRECTED",
SQLConf.SESSION_LOCAL_TIMEZONE.key -> "UTC",
CometConf.COMET_ENABLED.key -> "false") {
spark.createDataFrame(spark.sparkContext.parallelize(rows, 1),
schema).write.parquet(path)
}
withSQLConf(
SQLConf.PARQUET_INT96_REBASE_MODE_IN_READ.key -> "CORRECTED",
SQLConf.SESSION_LOCAL_TIMEZONE.key -> "UTC") {
spark.read.parquet(path).select("id", "at").orderBy("id").collect()
}
```
On `main` at 66c63bd307, the same on Spark 3.5.9 and 4.1.3:
| id | Spark | Comet native scan |
| --- | --- | --- |
| 0 | `[9999-12-31 23:59:59.0]` | `[1816-03-30 06:03:09.066278]` |
| 1 | `[1600-01-01 00:00:00.0]` | `[2184-07-21 00:27:31.709551]` |
| 2 | `[2263-01-01 00:00:00.0]` | `[1678-06-12 00:32:28.290449]` |
| 3 | `[2020-01-01 00:00:00.0]` | `[2020-01-01 00:00:00.0]` |
The same file written with `writeLegacyFormat=false` reads correctly, and so
do the map, struct and top-level INT96 columns of the legacy file. A Hive-style
`optional group at (LIST) { repeated group bag { optional int96 array_element;
} }` file and a two-level `optional group at (LIST) { repeated int96 array; }`
file written with parquet-mr give the same wrong values.
### Expected behavior
The native scan returns the same timestamps as Spark. The fix belongs in
DataFusion's `Int96Coercer`, which should match a leaf against the file's
Parquet column paths whatever the list layout, for example by walking the
Parquet schema rather than assuming a `list` group.
### Additional context
Found in a nested-type audit of the native scans. No DataFusion issue covers
the legacy layouts yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
