cshuo opened a new pull request, #20750: URL: https://github.com/apache/hudi/pull/20750
### Describe the issue this Pull Request addresses A Spark-created Hudi table can contain both `embedding VECTOR(3)` and `scores ARRAY<FLOAT>`. Flink exposes both columns as arrays. When Hive Metastore lacks `hoodie.vector.columns` and the sink infers its writer schema, it treats `embedding` as an ordinary array. In a copy-on-write table, Flink can read the original row but fail on the next insert when it tries to rewrite the existing Parquet file. ### Summary and Changelog `HoodieHiveCatalog.getTable()` now uses the latest Hudi schema it already reads to populate `hoodie.vector.columns` in the returned catalog options. For the example above, it derives `embedding:3` and leaves `scores` as an ordinary array. Explicit vector settings take precedence, and the derived option stays local to the returned table. The regression test covers multiple vector columns, an ordinary array, and an explicitly configured column order. It also verifies that reading the table does not persist the derived option to Hive Metastore. ### Impact Verified with Spark 3.5.4 and Flink 1.20.1 on a copy-on-write Parquet table. With default write settings, Flink inserted a second row and read both rows back. The new commit retained `embedding` as `VECTOR(3)` and `scores` as `ARRAY<FLOAT>`. All 51 tests in `TestHoodieHiveCatalog` and `TestHoodieTableFactory` passed. Checkstyle passed. ### Risk Level Low. The change uses the existing schema lookup and only supplies the option when it is absent and the stored schema contains top-level vector fields. ### Documentation Update None. ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
