yihua opened a new pull request, #20166: URL: https://github.com/apache/hudi/pull/20166
### Describe the issue this Pull Request addresses part of #16803 Proof of concept for a DataFrame-native Spark write path, opened as a draft to capture the current state of the work. The design is written up in RFC-114 (link to follow). Not ready for merge: it is gated off by default and still has a parity tail against the legacy writer, listed below. ### Summary and Changelog Today `HoodieSparkSqlWriter` drops the input to `RDD[HoodieRecord]`, caches it, and runs a count-based workload profile before routing. This PoC keeps the input as Dataset/InternalRow operations end to end and only becomes imperative at the record write: - New module `hudi-spark-datasource/hudi-spark-dataframe` with `HoodieDataFrameWriter`: key generation as `withColumn` expressions (simple, complex, timestamp, custom, non-partitioned key generators), precombine as a window dedup with commit-time ordering when no ordering field is set, index tagging as a join (SIMPLE and GLOBAL_SIMPLE against the base-file meta columns, record-level index through thin-key point lookups), a resolve stage that assigns a per-row operation (including `_hoodie_is_deleted` and update-partition-path fan-out), and routing on a prefixed key (`u:` update, `i:` insert, `s:` small-file padding merge). - `DataFrameWriteTask` drives the existing row-native handles per file group: `HoodieRowCreateHandle` for new files, `FileGroupReaderBasedMergeHandle` for copy-on-write merges, and the append handles for merge-on-read. The commit goes through the regular `SparkRDDWriteClient`, so timeline, metadata table, table services and meta sync are unchanged. - Two pluggable insert profilers: `MetadataSizingProfiler` (record size from commit metadata, fill small files first) and a shuffle-stats profiler in `HoodieRoutingShuffle` that materializes only the map stage of the routing shuffle, reads per-block sizes from the `MapOutputTracker`, bin-packs insert reducers, and records shuffle bytes in commit metadata for per-commit calibration. The routing shuffle runs as a physical-plan node under a SQL execution, so the whole write shows up in the Spark UI as one catalyst plan. - `DefaultSource` routes eligible datasource writes into the module behind `hoodie.datasource.write.dataframe.path.enable` (default false) through a reflective call, so `hudi-spark-common` does not depend on the new module. Anything unsupported falls back to the legacy writer with the reason logged: non-append save modes on existing tables, insert dup policies, SQL-internal prepped or merge writes, bucket, global bloom and in-memory indexes, custom index classes, auto keys, dropped partition columns. Known parity tail against the legacy writer, measured by running `TestHoodieSparkSqlWriter`, `TestCOWDataSource` and `TestMORDataSource` with the path force-enabled: timestamp key generator output parity, custom key generator globbing, two option-conflict edge cases, schema evolution wiring (`deduceWriterSchema` is not called yet), hive-style delete, OCC concurrent updates, table-creation fidelity for payload class and merge mode, and UTF-8 key ordering for sorted targets. SQL DML (UPDATE, DELETE, MERGE INTO) bypasses the gate entirely and is not covered. ### Impact No behavior change unless `hoodie.datasource.write.dataframe.path.enable` is set. New module in the reactor. ### Risk Level low: the path is off by default and every unsupported case falls back to the legacy writer before any table state is touched. ### Documentation Update The new config will need a description once the path is ready to be enabled; none for now. ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Enough context is provided in the sections above - [ ] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
