andygrove opened a new pull request, #1425: URL: https://github.com/apache/arrow-java/pull/1425
## What's Changed `VectorUnloader` wrote a variable-width vector's data buffer from byte 0 up to its last offset, and a list's child vector from element 0. A vector imported through the C Data interface from a sliced array keeps the offsets of the slice, which start past zero, and the bytes or child elements before them, so every such batch was serialized with values no row references, and the k-th slice of an array carried the k-1 slices before it. The Rust and C++ IPC writers re-base offsets in this case; Java's `VectorAppender` was taught to in #1180. `VectorUnloader.appendNodes` now re-bases a variable-width, list or map vector whose first offset is not zero through `splitAndTransfer(0, valueCount)` and writes the result, closing the temporary vector once the record batch has retained its buffers. The copy is limited to the offsets, and to the validity bits when the start is not byte-aligned; the data buffers are shared. A list or map whose offsets start at zero but whose child runs past the last offset, as the first slice of a list array does, keeps its own buffers and only has its child re-based to the covered length. Vectors whose offsets already start at zero are written with their own buffers as before, which a test pins by buffer address. Two things to be aware of: - A list whose offsets point past the end of its child, which is malformed, now fails inside `splitAndTransfer` at unload time instead of producing a stream that fails to load. - The C Data module's `StructVectorUnloader` has a near-copy of `appendNodes` that is left as it is. It only feeds `Data.importIntoVectorSchemaRoot`, where the batch goes straight into a `VectorLoader`, and re-basing there would add an offsets copy to every sliced import whether or not it is ever serialized. Tested with a new `TestVectorUnloadNonZeroStartOffset`, which builds the vectors a C Data import of a slice produces from raw buffers for VarChar, LargeVarChar, List, LargeList, Map, a list whose child runs past its last offset, and a VarChar with a non-zero start nested inside a list that starts at zero. Each case asserts the node lengths and buffer sizes written, that the written offsets start at zero, and that the batch loads back to the right values. Closes #1424. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
