cshuo opened a new pull request, #19242: URL: https://github.com/apache/hudi/pull/19242
### Describe the issue this Pull Request addresses Flink Lance base-file reader/writer only supported primitive columns, preventing tables containing nested `ROW` and `ARRAY` data from being written or read correctly. Lance native logs were also unnecessarily disabled for table version 10+. ### Summary and Changelog - Add recursive Flink `RowData` ↔ Arrow conversion for `ROW` and `ARRAY`, including nested schemas, null values, and Arrow list offsets. - Enable Lance native logs for write version 10+, while rejecting native Lance column-stats indexing because Lance metadata statistics are unsupported. - Remove the unused `HoodieTableConfig` argument from `CommonClientUtils.shouldWriteNativeLogs` and update common, Flink, and Spark call sites. - Add unit coverage for nested Arrow conversion and MAP rejection. - Add Lance nested-type upsert integration coverage for both COW and MOR tables. - Keep `MAP` unsupported pending Lance Java writer support, tracked by https://github.com/lance-format/lance/issues/3620. ### Impact - **Functional impact**: Flink Lance tables can persist and read nested `ROW` and `ARRAY` columns; Lance tables at write version 10+ use native logs. - **Maintainability**: Native-log selection is now solely version-based and no longer carries an unused table-config dependency. - **Extensibility**: Recursive conversion establishes the structure needed for additional complex types once Lance supports their Arrow encoding. ### Risk Level Medium. The change affects Arrow serialization and native log routing for Lance. Coverage includes nested schema/value round trips, MAP rejection, and COW/MOR upsert integration tests. ### Documentation Update none. ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Enough context is provided in the sections above - [ ] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
