andygrove opened a new issue, #7047:
URL: https://github.com/apache/datafusion-comet/issues/7047
### Describe the bug
The native Parquet writer ignores
`spark.sql.parquet.datetimeRebaseModeInWrite`, and it writes none of the footer
keys that Spark's readers use to decide whether a file needs rebasing. That
affects dates and timestamps before 1582-10-15, at the top level and inside
structs, arrays and maps, in three ways:
- **Spark 3.4 and 3.5 refuse the write, but the native writer doesn't.** In
the `EXCEPTION` mode those versions default to, Spark fails with
`[INCONSISTENT_BEHAVIOR_CROSS_VERSION.WRITE_ANCIENT_DATETIME]`. The native
writer writes the file.
- **Spark 3.4 and 3.5 then can't read the file back.** The file has no
`org.apache.spark.version` key, so `DataSourceUtils.datetimeRebaseSpec` falls
back to `spark.sql.parquet.datetimeRebaseModeInRead`, also `EXCEPTION` by
default, and the read fails with
`[INCONSISTENT_BEHAVIOR_CROSS_VERSION.READ_ANCIENT_DATETIME]`. A file Spark
writes in `CORRECTED` mode carries the key and reads back fine.
- **`LEGACY` read mode shifts the values silently, on every version.** Spark
treats the native file's proleptic Gregorian values as hybrid-calendar values
and rebases them, so `1000-01-01` reads back as `0999-12-27`. In the other
direction, with `datetimeRebaseModeInWrite=LEGACY` Spark rebases on write and
records `org.apache.spark.legacyDateTime`, while the native writer writes
proleptic values and no key. Readers that expect the hybrid calendar, such as
Spark 2.4 and Hive 2.x, then see shifted dates.
`ParquetWriterExec` sets only the compression on its `WriterProperties`
(`native/core/src/execution/operators/parquet_writer.rs`), and nothing on the
JVM side reads the rebase configs for a native write. Comet's own scan reads
the native files back correctly in every mode, because it never rebases (#5010).
### Steps to reproduce
Write the source with Comet disabled, so it holds proleptic values:
```scala
spark.conf.set("spark.sql.session.timeZone", "UTC")
spark.conf.set("spark.sql.parquet.datetimeRebaseModeInWrite", "CORRECTED")
spark.sql("""SELECT date'1000-01-01' AS d, named_struct('d',
date'1200-03-01') AS s,
array(date'1400-06-01') AS a""").write.parquet("/tmp/src")
spark.conf.unset("spark.sql.parquet.datetimeRebaseModeInWrite")
```
Then write it natively and read the result with Comet disabled:
```scala
spark.conf.set("spark.comet.write.parquet.enabled", "true")
// Spark 4.x. On Spark 3.x the key is
spark.comet.operator.DataWritingCommandExec.allowIncompatible
spark.conf.set("spark.comet.operator.WriteFilesExec.allowIncompatible",
"true")
spark.read.parquet("/tmp/src").write.parquet("/tmp/native")
spark.conf.set("spark.comet.enabled", "false")
spark.read.parquet("/tmp/native").show()
spark.conf.set("spark.sql.parquet.datetimeRebaseModeInRead", "LEGACY")
spark.read.parquet("/tmp/native").show()
```
On `main` at c230bf4e98:
| Step | Spark 3.4.3 and 3.5.9 | Spark 4.1.3 |
| --- | --- | --- |
| Spark writes the source in the default write mode |
`WRITE_ANCIENT_DATETIME` | succeeds |
| Native write in the default write mode | succeeds | succeeds |
| Spark reads the native file in the default read mode |
`READ_ANCIENT_DATETIME` | `1000-01-01`, `{1200-03-01}`, `[1400-06-01]` |
| Spark reads the native file with `datetimeRebaseModeInRead=LEGACY` |
`0999-12-27`, `{1200-02-23}`, `[1400-05-23]` | the same |
| Spark reads a file Spark wrote in `CORRECTED` mode, with `LEGACY` |
`1000-01-01`, `{1200-03-01}`, `[1400-06-01]` | the same |
Timestamp columns shift the same way.
### Expected behavior
The native writer does what Spark's writer does for the effective write
mode, or falls back:
- `CORRECTED`: write `org.apache.spark.version` into the footer (#3427), so
Spark readers treat the file as a Spark 3+ file that needs no rebasing.
- `EXCEPTION`: fail on an ancient value the way Spark does, or fall back for
writes with date or timestamp columns.
- `LEGACY`: fall back, or rebase and write `org.apache.spark.legacyDateTime`
and `org.apache.spark.timeZone` as Spark does.
### Additional context
Found in a nested-type audit of the native writers. #4180 lists
`datetimeRebaseModeInWrite` and `int96RebaseModeInWrite` among the configs to
audit, and #5010 tracks the read side. The native writer never writes INT96, so
`int96RebaseModeInWrite` has no effect on it today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
