Gabriel39 opened a new pull request, #69157: URL: https://github.com/apache/doris/pull/69157
### What problem does this PR solve? Spark SQL's Hudi `SqlKeyGenerator` always escapes formatted TIMESTAMP partition values, independently of `hoodie.table.url.encode.partitioning`. Doris currently unescapes partition values only when that general flag is enabled. With the flag absent or false, a physical component such as `event_time=2024-01-01 10%3A00%3A00` reaches the BE as an invalid timestamp instead of `2024-01-01 10:00:00`. This breaks `test_hudi_partition_prune` and `test_hudi_runtime_filter_partition_pruning`. Carry timestamp partition-column names from the table schema through Hudi handles and apply exactly one Hive-style unescape to those logical values. Use the same interpretation for HMS/Hudi metadata listing, partition pruning, snapshot pin filtering, and COW/MOR snapshot and incremental scan ranges. Preserve physical file paths. Other types still follow the persisted URL-encoding flag, so a STRING containing literal `%3A`, `%253A`, or `+` remains distinct and unchanged when encoding is disabled. The writer behavior is documented in [Hudi's SqlKeyGenerator](https://github.com/apache/hudi/blob/release-1.0.2/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/command/SqlKeyGenerator.scala). ### Validation - Confirmed the new metadata regression fails before the fix with expected `10:00:00`, actual `10%3A00%3A00`. - `mvn -f fe/pom.xml -pl :fe-connector-hudi -am test -Dmaven.build.cache.enabled=false`: 1,007 tests passed, including 270 Hudi tests; zero failures/errors/skips. - FE Checkstyle/validate and Groovy syntax compilation passed. - Coverage includes real persisted SDK configurations with missing/false/true encoding flags, mixed timestamp/STRING partitions, single-layer decoding, nulls, HMS and metadata pruning, historical listing, a real local COW commit/write-stat mapping, and MOR native/JNI Thrift values. - Extended the existing external regression to compare native/JNI timestamp rows under both partition-listing modes while keeping the existing golden assertion and restoring the session setting. The real Spark writer and full external Doris/HMS/Hudi suites were not run locally; the new external assertion is syntax-checked and remains for CI. ### Release note Fix reads of Spark SQL Hudi TIMESTAMP partitions whose paths are escaped while generic URL partition encoding is disabled. ### Check List (For Author) - [x] Unit Test - [x] Regression test extension - [x] Behavior changed: timestamp partition values are decoded according to Spark/Hudi writer semantics - [x] No documentation needed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
