alberttwong commented on issue #1548: URL: https://github.com/apache/arrow-go/issues/1548#issuecomment-6862538797
We now have a reproducible server-independent C Data/C Stream fixture and measured evidence, merged through redis/redis-adbc-driver#411. Pinned Arrow Go v18.8.0, Go 1.27.3, darwin/arm64. The fixture shares prebuilt payload buffers for 10,000 rows across 105 columns, including nullable mixed integer, UTF-8, Date32, UTC microsecond timestamp and Boolean values. Every value/type/null is checked outside timing. All84 samples across the Arrow and RESP3 fixtures passed; race/checked-allocator tests cover retained batches, moved headers, early release and independent concurrent stream lifetimes. Median Arrow boundary time per 10K total rows: | Contract | Ten 1K batches | One 10K batch | |---|---:|---:| | C Data, schema each, fresh headers |1.243ms|0.124ms| | C Data, schema each, reused empty headers |1.251ms|0.123ms| | C Data, known schema, reused headers |0.715ms|0.071ms| | C Stream, schema once |0.777ms|0.129ms| Header reuse showed no consistent benefit. Known-schema/schema-once cases have different negotiation/lifetime contracts, so they are not transparent replacements for schema-each. Go allocation counters exclude C malloc/calloc and prebuilt shared payload buffers; wall time includes C calls. Network, Redis, ADBC query execution, Python and IPC are excluded. These millisecond observations do not establish Arrow as the bottleneck in multi-second driver writes. We also found a documentation/implementation ownership mismatch in the pinned version: ExportRecordReader's comment says caller Release is unnecessary, but exportStream retains its own producer reference and stream release drops that retained reference. Our counted-reader and checked-allocator tests balance only when the caller also releases its original reference; a private omitted-original-Release negative control fails both assertions. The actual Redis ADBC exporter uses a separate balanced custom path, so this is not evidence of a driver leak. Is clarification of that comment appropriate here, or would you prefer a separate issue/PR? Runnable harness, exact reference-accounting explanation, source hashes and all observations: https://github.com/redis/redis-adbc-driver/tree/03e2c522e5e11538f54015c770675be5c31a4469/benchmarks/upstream-allocation Full report: https://github.com/redis/redis-adbc-driver/blob/03e2c522e5e11538f54015c770675be5c31a4469/benchmarks/upstream-allocation/2026-11-13/report.md -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
