alberttwong commented on issue #1548:
URL: https://github.com/apache/arrow-go/issues/1548#issuecomment-6862538797

   We now have a reproducible server-independent C Data/C Stream fixture and 
measured evidence, merged through redis/redis-adbc-driver#411.
   
   Pinned Arrow Go v18.8.0, Go 1.27.3, darwin/arm64. The fixture shares 
prebuilt payload buffers for 10,000 rows across 105 columns, including nullable 
mixed integer, UTF-8, Date32, UTC microsecond timestamp and Boolean values. 
Every value/type/null is checked outside timing. All84 samples across the Arrow 
and RESP3 fixtures passed; race/checked-allocator tests cover retained batches, 
moved headers, early release and independent concurrent stream lifetimes.
   
   Median Arrow boundary time per 10K total rows:
   
   | Contract | Ten 1K batches | One 10K batch |
   |---|---:|---:|
   | C Data, schema each, fresh headers |1.243ms|0.124ms|
   | C Data, schema each, reused empty headers |1.251ms|0.123ms|
   | C Data, known schema, reused headers |0.715ms|0.071ms|
   | C Stream, schema once |0.777ms|0.129ms|
   
   Header reuse showed no consistent benefit. Known-schema/schema-once cases 
have different negotiation/lifetime contracts, so they are not transparent 
replacements for schema-each. Go allocation counters exclude C malloc/calloc 
and prebuilt shared payload buffers; wall time includes C calls. Network, 
Redis, ADBC query execution, Python and IPC are excluded. These millisecond 
observations do not establish Arrow as the bottleneck in multi-second driver 
writes.
   
   We also found a documentation/implementation ownership mismatch in the 
pinned version: ExportRecordReader's comment says caller Release is 
unnecessary, but exportStream retains its own producer reference and stream 
release drops that retained reference. Our counted-reader and checked-allocator 
tests balance only when the caller also releases its original reference; a 
private omitted-original-Release negative control fails both assertions. The 
actual Redis ADBC exporter uses a separate balanced custom path, so this is not 
evidence of a driver leak. Is clarification of that comment appropriate here, 
or would you prefer a separate issue/PR?
   
   Runnable harness, exact reference-accounting explanation, source hashes and 
all observations: 
https://github.com/redis/redis-adbc-driver/tree/03e2c522e5e11538f54015c770675be5c31a4469/benchmarks/upstream-allocation
   
   Full report: 
https://github.com/redis/redis-adbc-driver/blob/03e2c522e5e11538f54015c770675be5c31a4469/benchmarks/upstream-allocation/2026-11-13/report.md
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to