rangareddy commented on issue #15296: URL: https://github.com/apache/hudi/issues/15296#issuecomment-4875078241
This issue was reviewed as part of the JIRA-migrated backlog triage, and I attempted to reproduce it on the latest code. **Setup:** Hudi `1.2.0-SNAPSHOT` (`hudi-flink1.20-bundle`) + Flink `1.20.1`, embedded batch job. COW table, `write.operation=bulk_insert`, `write.bulk_insert.sort_input=true`, wide schema (**4000** data columns + record key + partition). Managed memory deliberately kept small (48 MB) so the external sorter is forced to **spill** — the serialization step the original report blames. **Result: could not reproduce.** For both a 4000×`INT` schema (4000 rows) and a 4000×`VARCHAR` schema (800 rows): - the `bulk_insert` job completed, - Flink `SELECT COUNT(*)` returned the full row count (the report saw an empty result), - every base parquet file has all **4007** columns (5 meta + 4002 data) — no field collapse — and all rows read back correctly. The suspect code path was definitely exercised — logs show `org.apache.hudi.sink.bulk.sort.SortOperator` → `org.apache.flink.table.runtime.operators.sort.BinaryExternalSorter` with real on-disk spills (`here spill the 1th/2th/3th sort buffer data ...`), which is exactly where the report said fields were getting disordered. The output was still intact. This appears to be **fixed on current Hudi** — most likely as part of the Flink `bulk_insert` sort-path rework (e.g. #19131 / HUDI-7818, which replaced the planner `SortCodeGenerator` + `NormalizedKeyComputer` usage with local implementations), plus `BinaryExternalSorter` improvements since the Flink 1.14 / Hudi 0.13 used in the original report. **Caveat:** I tested `INT` and `VARCHAR` columns (variable-length being the classic trigger for binary-row serialization bugs). The original `table.ddl` (JIRA HUDI-4459 attachment) may mix other types. If you can still reproduce this on a recent version, please attach the exact `table.ddl` + `statements.sql` and your Hudi/Flink versions, and I'll rerun against them. Recommend closing as not reproducible on recent versions unless a current-version reproduction can be provided. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
