JingsongLi commented on PR #9147: URL: https://github.com/apache/paimon/pull/9147#issuecomment-5240424593
One additional implementation idea: the fast path could be based on encoded-length invariance rather than being hard-coded to FLOAT and DOUBLE. For an existing path, after computing and encoding the replacement, if the encoded value has the same length as the old encoded value, the implementation can patch the complete encoded slice directly. The metadata dictionary and all ancestor offset tables remain valid because neither the path nor any byte positions change. This would naturally cover all fixed-width values and potentially variable-width values whose encoded length is unchanged. If the encoded length differs, or the path is missing and must be created, the implementation can fall back to rebuilding the affected value. There is also an Arrow-level optimization opportunity. The current `to_pylist() -> bytearray(value) -> pa.array(values)` path materializes Python objects, copies every row separately, and rebuilds the binary offsets. Since the supported fast path preserves each row length, we could reuse the validity bitmap, offsets, and metadata buffers, allocate/copy the value data buffer once per chunk, and patch the located bytes at their absolute offsets. This should be copy-on-write rather than mutating the input buffer, because Arrow arrays can share buffers and are logically immutable. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
