cshuo opened a new pull request, #20750:
URL: https://github.com/apache/hudi/pull/20750

   ### Describe the issue this Pull Request addresses
   
   A Spark-created Hudi table can contain both `embedding VECTOR(3)` and 
`scores ARRAY<FLOAT>`. Flink exposes both columns as arrays. When Hive 
Metastore lacks `hoodie.vector.columns` and the sink infers its writer schema, 
it treats `embedding` as an ordinary array. In a copy-on-write table, Flink can 
read the original row but fail on the next insert when it tries to rewrite the 
existing Parquet file.
   
   ### Summary and Changelog
   
   `HoodieHiveCatalog.getTable()` now uses the latest Hudi schema it already 
reads to populate `hoodie.vector.columns` in the returned catalog options. For 
the example above, it derives `embedding:3` and leaves `scores` as an ordinary 
array. Explicit vector settings take precedence, and the derived option stays 
local to the returned table.
   
   The regression test covers multiple vector columns, an ordinary array, and 
an explicitly configured column order. It also verifies that reading the table 
does not persist the derived option to Hive Metastore.
   
   ### Impact
   
   Verified with Spark 3.5.4 and Flink 1.20.1 on a copy-on-write Parquet table. 
With default write settings, Flink inserted a second row and read both rows 
back. The new commit retained `embedding` as `VECTOR(3)` and `scores` as 
`ARRAY<FLOAT>`.
   
   All 51 tests in `TestHoodieHiveCatalog` and `TestHoodieTableFactory` passed. 
Checkstyle passed.
   
   ### Risk Level
   
   Low. The change uses the existing schema lookup and only supplies the option 
when it is absent and the stored schema contains top-level vector fields.
   
   ### Documentation Update
   
   None.
   
   ### Contributor's checklist
   
   - [ ] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to