hcrosse opened a new pull request, #2677: URL: https://github.com/apache/iceberg-go/pull/2677
Closes #2410. Scans expand dictionary columns to plain values, adding time and memory cost when values repeat. I added `WithDictionaryColumns` to return selected string and binary columns as dictionaries. ## What `WithDictionaryColumns(names ...string)` returns the named string and binary columns as `dictionary<int32, utf8>` or `dictionary<int32, binary>` in both the schema and every batch, regardless of `arrow.use_large_types` or stored plain, large, or narrower dictionary layouts. Missing columns, missing parents, and defaults use the same type. Names resolve at read time against the effective snapshot schema, honor `WithCaseSensitive` and time travel renames, and support dotted paths into structs, lists, and map keys or values. Unknown or non string/binary names fail even on empty scans. Valid unprojected names are ignored. Raw and normalized changelog scans preserve the option, while filter only columns and delete files keep their usual types. Dictionaries can change across batches. Row filters on requested columns rely on `arrow-go` v18.9.0 (`apache/arrow-go#1449`, #2632). ## Why In a synthetic single file scan (262,144 rows, one string column, 10 interleaved rounds on Linux amd64, Go 1.26.3, before `arrow-go` v18.9.0 tagged): | Case | Time | Bytes allocated | Allocs | | --- | ---: | ---: | ---: | | Unfiltered (16 / 4,096 distinct) | -39% to -42% | -26% to -29% | +1.5% to +2.2% | | Filtered on column (16 / 4,096 distinct) | ~0% / +5.8% | -10% to -13% | +7.3% to +7.8% | When filtering a requested column, `arrow-go` decodes the dictionary per batch to evaluate the predicate, offsetting the scan time gain. ## Testing I added scan tests covering dictionary, plain, fallback, large and stored dictionary inputs, missing columns and parents, defaults, nested paths, name mapping, split tasks, validation, time travel renames, deletes, raw and normalized changelogs, and row filters (`=`, `IN`, `starts_with`, `IS NULL`, `NOT NULL`, binary `=`, nested `=`). Output type and value checks cover both `use_large_types` modes with a checked allocator. `go test -race ./table/...`, `go vet ./table/...`, and `golangci-lint run ./table/...` pass. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
