JingsongLi opened a new pull request, #10958:
URL: https://github.com/apache/paimon/pull/10958

   ### Purpose
   
   Add `MultimodalTable.merge_columns` for importing labels or embeddings by 
key, and `add_columns` for schema-only additions or batched feature 
computation. Both reuse Data Evolution column updates so untouched media files 
are retained.
   
   The local APIs validate source cardinality, output types, lossless key 
conversion, and table mode before publication. Results include row counts and 
schema/snapshot IDs; failures expose their phase and known publication outcome. 
Schema changes and data commits remain separate, and callers must coordinate a 
single writer.
   
   Preserve row counts when projecting newly added columns that have no 
physical provider. Store vectors as physical Parquet lists while preserving 
Arrow metadata, covering ordinary, streaming, and dedicated-vector writers so 
sparse nullable embeddings remain readable.
   
   ### Tests
   
   - Python 3.10 / PyArrow 22: **289 passed, 68 subtests passed, 9 skipped, 1 
deselected** across column enrichment, vector serialization, multimodal tables, 
Data Evolution, shard/row-ID updates, shared-shredding MAPs, and format writes.
   - Added **37 API tests** and **4 streaming Parquet tests**, including media 
file reuse, composite keys, NULLs, deleted rows, batch/file-group alignment, 
response loss, stale table handles, signed zero, lossy key casts, and nested 
metadata.
   - Additional local REST-catalog verification: successful enrichment plus 
nine row-filter, column-mask, and denied-read cases rejected before mutation.
   - Flake8, license headers, diff whitespace, Python 3.6 syntax checks, and 
the executable documentation example passed.
   
   The deselected existing HEVC decoding test requires Torch, which is 
unavailable in this environment. Native execution was not exercised because the 
optional runtime is not installed.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to