yihua opened a new pull request, #868:
URL: https://github.com/apache/hudi-rs/pull/868
## Description
closes #866
An embedding column written by `append` could not be upserted or updated
afterwards: the table schema comes back from Avro with the list item named
`element`, while PyArrow and most Arrow producers name it `item`, and the write
path compared types exactly. `FixedSizeList` columns were rejected outright
because Avro has only `array`.
Each write function now normalizes its batches right after its timeline
reload: fixed-size lists are widened to lists at any depth, and a column that
matches the table's column up to list item names is cast to the table's exact
type. The table schema is read only when a batch carries a list column. Struct
children still have to match by name, so a differently named child is rejected
as before, and the downstream schema checks stay exact.
## How are the changes test-covered
- [ ] N/A
- [x] Automated tests (unit and/or integration tests)
- [ ] Manual tests
- [ ] Details are described below
`table_write_list_columns_tests` covers upsert, update, overwrite and a
two-batch upsert with mismatched item names, fixed-size lists on append and
upsert, merge-on-read upsert and update, a handle that has not seen the latest
commit, a fixed-size list nested in a struct, and rejection of a struct whose
child names differ. All seven fail on `main` and pass here.
https://claude.ai/code/session_01PpvixgKyY2VqVo8UAdtUeF
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
