voonhous commented on issue #17302:
URL: https://github.com/apache/hudi/issues/17302#issuecomment-5539095717
`hoodie.parquet.outputtimestamptype` has had no effect on any Spark write
path since 1.1.0, bulk insert included.
The Feb 2025 comment above ("looks like it works for BULK_INSERT mode") was
accurate then, on 1.0.x. It stopped being true in #13882 (`4d95b2c2d165`, first
released in 1.1.0), which changed `HoodieRowParquetWriteSupport` from `extends
ParquetWriteSupport` (Spark's, whose `init()` reads
`spark.sql.parquet.outputTimestampType` off the Hadoop conf) to `extends
WriteSupport<InternalRow>` with its own `init()`. The constructor kept setting
that Hadoop conf key, but nothing reads it any more, so the value is inert.
Running the repro from this issue against current master with
`hoodie.parquet.outputtimestamptype=TIMESTAMP_MILLIS` and reading the Parquet
footer back:
| write path | `field3` in the footer |
| --- | --- |
| `bulk_insert` (row writer) | `TIMESTAMP(MICROS,true)` |
| `bulk_insert` with `hoodie.datasource.write.row.writer.enable=false` |
`TIMESTAMP(MICROS,true)` |
| `insert` | `TIMESTAMP(MICROS,true)` |
| `upsert` | `TIMESTAMP(MICROS,true)` |
Setting the Spark session conf
`spark.sql.parquet.outputTimestampType=TIMESTAMP_MILLIS` is ignored on the same
paths.
Both the Spark row writer and the Avro Parquet writer now take the timestamp
unit from the writer schema's logical type, so millis has to be declared there:
```scala
.option("hoodie.write.schema",
"""{"type":"record","name":"rec","fields":[
{"name":"field1","type":"int"},
{"name":"field2","type":["null","string"],"default":null},
{"name":"field3","type":{"type":"long","logicalType":"timestamp-millis"}}]}""")
```
That gives `TIMESTAMP(MILLIS,true)` on bulk insert.
#19815 documents the config as a no-op, marks it `@Deprecated` with
`deprecatedAfter("1.1.0")`, and points at the writer schema. If that is the
resolution here, this issue can be closed once it merges.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
