JingsongLi commented on PR #9147: URL: https://github.com/apache/paimon/pull/9147#issuecomment-5240605578
Yes, I think `variant_set` would be a good direction, with normal set/upsert semantics: - if the path exists, replace its value; - if the final path component is missing, create it; - an existing-path-only operation should be named `variant_replace`. With that contract, I would not add `create_if_missing` to `variant_set`: creation is part of the meaning of set. For an initial implementation, it would be reasonable to support creation only when the parent container already exists, and explicitly reject missing intermediate containers, incompatible parent types, and undefined array-extension cases until their semantics are specified. I also agree that `variant_get` would make the API more composable and closer to Databricks. For example: ```python current = variant_get(column, "$.velocity.y", pa.float64()) updated = pc.negate(current) result = variant_set(column, "$.velocity.y", updated) ``` Here `variant_set` could accept either an Arrow scalar or an Arrow array. This avoids placing arbitrary Python callbacks in the per-row hot path and lets users build updates from Arrow compute expressions. The existing position-planning code could also be shared by `variant_get` and `variant_set`. The implementation can then choose its strategy independently of the public semantics: - existing path plus equal encoded length: copy-on-write patch, reusing metadata and offsets; - existing path plus different encoded length: rebuild the value and affected offsets; - missing final field: rebuild the value; reuse metadata only if the key is already present in the metadata dictionary, otherwise rebuild metadata as well. If missing-path creation is out of scope for this PR, I would expose the current operation as `variant_replace` first and add the broader `variant_set` once the reconstruction fallback is implemented. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
