kylebarron commented on PR #489: URL: https://github.com/apache/arrow-js/pull/489#issuecomment-5796982479
I think it's very confusing for users to expose a `valueOffsets` array that has extra padding. Since `Vector` and `Data` are _logically immutable_, there's no reason to maintain the extra padding bytes from the initial allocation. The `subarray` on the typed view is virtually free. --- In the case of pyarrow, `offsets` always respect the _logical_ size of offsets ```py import pyarrow as pa list_arr = pa.array([[0, 1], [2, 3], [4, 5], [6, 7]], type=pa.list_(pa.int32())) list_arr.offsets # <pyarrow.lib.Int32Array object at 0x112711960> # [ # 0, # 2, # 4, # 6, # 8 # ] ``` --- Even if you pass an IPC buffer with extra padding into pyarrow, the pyarrow `offsets` will show the correct (sliced) length according to the spec. <details> <summary>claude-generated reproduction</summary> `[email protected]`, `pyarrow==25.0.1`. ```js // write.mjs import { writeFileSync } from "node:fs"; import { Field, Int32, List, Table, tableToIPC, vectorFromArray } from "apache-arrow"; // 4 rows -> 5 offsets, an odd number of int32s, so the declared IPC buffer // region gets padded up to the next 8 bytes and a 6th offset appears. const table = new Table({ l: vectorFromArray( [[0, 1], [2, 3], [4, 5], [6, 7]], new List(new Field("item", new Int32(), true)), ), }); console.log("arrow-js valueOffsets:", table.getChild("l").data[0].valueOffsets); writeFileSync("padded.arrows", tableToIPC(table, "stream")); ``` ``` arrow-js valueOffsets: Int32Array(16) [ 0, 2, 4, 6, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ] ``` ```py # read.py import pyarrow as pa arr = pa.ipc.open_stream(open("padded.arrows", "rb")).read_all().column("l").chunk(0) print("len(arr) :", len(arr)) print("physical buffer slots :", arr.buffers()[1].size // 4) print("arr.offsets :", arr.offsets.to_pylist()) print("len(arr.offsets) :", len(arr.offsets)) ``` ``` len(arr) : 4 physical buffer slots : 6 arr.offsets : [0, 2, 4, 6, 8] len(arr.offsets) : 5 ``` </details> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
