fallintoplace opened a new pull request, #1321: URL: https://github.com/apache/arrow-go/pull/1321
**What** - Reuse decoder-owned storage while `DELTA_BYTE_ARRAY` values are discarded. - Add correctness tests and discard benchmarks. **Why** - `Discard` currently allocates a new byte slice for every discarded value with a non-empty suffix. - Skipping 65,536 values creates 65,535 allocations only to throw the values away. **Implementation** - Copy the first discarded value into owned scratch because the decoded suffix can alias page data. - Grow the scratch buffer amortized and reuse it across pages. - Keep empty suffixes on the existing zero-copy path. - Benchmark prefix-heavy and low-prefix data at 1,024 and 65,536 values. The benchmark changes from 65,535 allocations to 0 allocations per operation after warm-up, with about 3.3x lower discard time for 65,536 values. Tests: - `go test ./parquet/internal/encoding -count=1` - `go test ./parquet/file -run '^(TestWithEOFReader|TestInvalidHeaders|TestInvalidFooter|TestIncompleteMetadata|TestDeltaLengthByteArrayPackingWithNulls|TestDeltaBinaryPackedMultipleBatches|TestPageStreaming.*|TestPrimitiveReader|TestFullSeekRow|TestSkipEmptyRepeatedRows)$' -count=1` - `go test -race ./parquet/internal/encoding -run 'TestDeltaByteArrayDecoder(DiscardsAllEmptyValues|DiscardCopiesFirstValue|ReusesDiscardScratch|RejectsInvalidPrefixes|KeepsPartialDecodeResults)$' -count=1` - `go vet ./parquet/internal/encoding` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
