hcrosse opened a new pull request, #2445: URL: https://github.com/apache/iceberg-go/pull/2445
Draft until iceberg-go moves to an arrow-go release that includes https://github.com/apache/arrow-go/pull/1444. The scan gain depends on that dictionary reuse. After https://github.com/apache/iceberg-go/pull/2442, dictionaries larger than the batch still take the per-row equality-delete path. I added a memo that keeps dictionary entry results across a file's batches. This PR includes that prerequisite commit until it merges. Only the second commit is this change. ## What For single string or binary keys whose dictionary is larger than the batch, each referenced entry is looked up at most once while batches share the same dictionary object. The memo keeps one byte per entry and resets when a batch brings a different dictionary object. Dictionaries no larger than the batch stay on the prerequisite's bitmap path. Multi-column keys, `makeColEncoder` and delete-file reading are unchanged. The memo compares dictionaries by identity and holds the last one without `Retain`. Holding that reference keeps the old object from being reused, so a different dictionary never compares equal to it. On a match, the current batch keeps the dictionary alive. Reuse relies on Arrow arrays being immutable and on each file's batches being filtered one at a time, as the scanner does. Batches with no positional deletes or deletion vectors in their range keep the dictionary identity. When those deletes remove rows, `compute.Take` or `FilterRecordBatch` rebuilds the dictionary. That batch misses the memo and looks up its entries again, but stays correct. ## Why Data files reach this path today when their embedded `ARROW:schema` records a dictionary type, as it does when PyIceberg writes pyarrow dictionary arrays. The scan option proposed in https://github.com/apache/iceberg-go/issues/2410 would make dictionary columns common. I compared the prerequisite with this change on a Linux amd64 VM with 8 AMD EPYC vCPUs and Go 1.27.1, building each version against arrow-go v18.8.0 and against arrow-go main at [`52023100`](https://github.com/apache/arrow-go/commit/520231008bd28b1f525b116d17716cdcdb4681e0) through a temporary `replace`, interleaved over six rounds. Microbenchmarks used `-cpu=1` and eight batches over a 200,000-entry dictionary: - At 131,072 rows, shared dictionaries took 54% and 61% less time on [arrow-go main](https://github.com/apache/arrow-go/commit/520231008bd28b1f525b116d17716cdcdb4681e0), and 50% and 56% less on v18.8.0, for 10 and 10,000 deletes respectively. The gain holds on both versions because the benchmark builds the shared dictionary itself. At 8,192 rows and with a new dictionary per batch, results were within noise apart from two small gains. No case was significantly slower. The results cost one byte per dictionary entry for each delete set, about 199 KiB here, held until the file is done. Allocations were about 2% to 3% lower. - In an uncommitted scan of a synthetic table with 2 million rows written by PyIceberg, read in 131,072-row batches, the 200,000-entry key took 27.0% less time on [arrow-go main](https://github.com/apache/arrow-go/commit/520231008bd28b1f525b116d17716cdcdb4681e0), from 261.9 to 191.2 ms. There was no significant change on v18.8.0 because each batch gets a new dictionary there. Bytes allocated were flat on both versions, and row counts matched. - With deletes on a 4,096-entry key, the scan was unchanged on both versions because that dictionary stays on the prerequisite's bitmap path. ## Testing `TestProcessEqualityDeletesSingleBytesDictionaryAcrossBatches` covers different dictionaries of the same length, shorter dictionaries, null indices and entries with and without a null delete key, sliced batches and dictionary values, isolation between delete sets, and a bitmap batch between memo batches. `TestDictionaryEntryMemoKeepsResultsForSameDictionary` checks that results survive a second batch with the same dictionary object. Both tests use checked allocators. An uncommitted randomized comparison against the per-row path found no differences across 2,000 scenarios. To reproduce the microbenchmarks: `go test ./table/ -run '^$' -bench '^BenchmarkProcessEqualityDeletesSingleStringDictionaryBatches$' -benchmem -cpu=1 -benchtime=300ms -count=6`. With Go 1.27.1, `gofmt`, `make lint` and `go test -race ./table/` passed on Linux amd64, and `go test ./...` passed on macOS arm64. I did not test remote storage or files from other writers. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
