rangareddy opened a new pull request, #19815: URL: https://github.com/apache/hudi/pull/19815
### Describe the issue this Pull Request addresses Relates to https://issues.apache.org/jira/browse/HUDI-8592 (#17302), which reports that `hoodie.parquet.outputtimestamptype=TIMESTAMP_MILLIS` is ignored and the Parquet file is always written with `timestamp-micros`. The config's own description is a large part of why this gets reported. It currently reads: > Sets spark.sql.parquet.outputTimestampType. Parquet timestamp type to use when Spark writes data to Parquet files. That promises the config governs Spark writes generally. It does not. `HoodieRowParquetWriteSupport` is its only consumer, copying the value into `spark.sql.parquet.outputTimestampType` on the hadoop conf it builds, which covers the Spark row-writer Parquet path. Writes that go through the Avro path instead, such as `insert` and `upsert`, get their Parquet timestamp type from the table's Avro schema via parquet-avro, so the config has no effect on them. That matches the observation in the ticket that the setting appears to work under `BULK_INSERT` and not otherwise. This PR corrects the description only. It does **not** make the config effective on the Avro path: doing that means changing the timestamp logical type in the table's Avro schema, which is a storage-format decision and wants its own discussion rather than a drive-by change. HUDI-8592 should stay open for that. ### Summary and Changelog Users reading the config table on the website (and in the IDE) now see which write paths `hoodie.parquet.outputtimestamptype` actually governs, instead of a description that implies all Spark writes. - `HoodieStorageConfig.PARQUET_OUTPUT_TIMESTAMP_TYPE`: narrowed `withDocumentation(...)` to state that it applies to the Spark row-writer Parquet path (for example bulk insert), and that Avro-path writes derive the Parquet timestamp type from the table's Avro schema and are unaffected. No code was copied. ### Impact No public API or behavior change. The config key, its default (`TIMESTAMP_MICROS`), and its `markAdvanced()` status are untouched, and no code path reads the description. The generated config table on the Hudi website will show the new text. Worth noting for a follow-up, not changed here: `HoodieWriteConfig.parquetOutputTimestampType()` has no callers anywhere in the repo. Removing it is a public method removal, so it does not belong in a documentation change. ### Risk Level none Documentation string only. Verified with `mvn compile checkstyle:check -pl hudi-common`, which passes. ### Documentation Update This PR is the documentation update. The config description is corrected in place, which is what feeds the generated config tables on the Hudi website, so no separate website change is needed. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
