singhpratech commented on issue #88: URL: https://github.com/apache/arrow-js/issues/88#issuecomment-5579463652
Another consumer that needed this, and one detail worth knowing for whoever implements it. For a Node addon that hands Arrow JS vectors to a native library, I built the export side of the C Data Interface outside Arrow JS (https://github.com/singhpratech/ArrowMetal, `node/src/addon.cc`): the addon fills `ArrowSchema` and `ArrowArray` from a `Vector`'s `Data` (values buffer, validity bitmap, utf8 offsets), pointing every buffer at the V8 backing store's own address so nothing is copied, and keeps a JS reference to each buffer in the struct's `private_data` until the consumer calls the release callback. Import is the reverse. It is tested against apache-arrow 21.2.0 on int32/int64/float64/bool/utf8 with nulls, sliced and doubly sliced. The detail: `Data.slice` advances the numeric values buffer and the utf8 offsets buffer but leaves the validity bitmap alone and keeps the row offset in `Data.offset`, while the C Data Interface applies one `offset` to every buffer. An exporter therefore has to rewind the advanced buffers to their row-0 origin (a typed-array view at a different `byteOffset`, no bytes move) and export `Data.offset` as the C offset; an importer does the opposite. Without that, a sliced vector with nulls exports the wrong validity bits. If a C Data Interface lands in Arrow JS, that mapping is the one non-obvious piece. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
