hcrosse opened a new pull request, #2677:
URL: https://github.com/apache/iceberg-go/pull/2677

   Closes #2410.
   
   Scans expand dictionary columns to plain values, adding time and memory cost 
when values repeat. I added `WithDictionaryColumns` to return selected string 
and binary columns as dictionaries.
   
   ## What
   
   `WithDictionaryColumns(names ...string)` returns the named string and binary 
columns as `dictionary<int32, utf8>` or `dictionary<int32, binary>` in both the 
schema and every batch, regardless of `arrow.use_large_types` or stored plain, 
large, or narrower dictionary layouts. Missing columns, missing parents, and 
defaults use the same type.
   
   Names resolve at read time against the effective snapshot schema, honor 
`WithCaseSensitive` and time travel renames, and support dotted paths into 
structs, lists, and map keys or values. Unknown or non string/binary names fail 
even on empty scans. Valid unprojected names are ignored.
   
   Raw and normalized changelog scans preserve the option, while filter only 
columns and delete files keep their usual types. Dictionaries can change across 
batches. Row filters on requested columns rely on `arrow-go` v18.9.0 
(`apache/arrow-go#1449`, #2632).
   
   ## Why
   
   In a synthetic single file scan (262,144 rows, one string column, 10 
interleaved rounds on Linux amd64, Go 1.26.3, before `arrow-go` v18.9.0 tagged):
   
   | Case | Time | Bytes allocated | Allocs |
   | --- | ---: | ---: | ---: |
   | Unfiltered (16 / 4,096 distinct) | -39% to -42% | -26% to -29% | +1.5% to 
+2.2% |
   | Filtered on column (16 / 4,096 distinct) | ~0% / +5.8% | -10% to -13% | 
+7.3% to +7.8% |
   
   When filtering a requested column, `arrow-go` decodes the dictionary per 
batch to evaluate the predicate, offsetting the scan time gain.
   
   ## Testing
   
   I added scan tests covering dictionary, plain, fallback, large and stored 
dictionary inputs, missing columns and parents, defaults, nested paths, name 
mapping, split tasks, validation, time travel renames, deletes, raw and 
normalized changelogs, and row filters (`=`, `IN`, `starts_with`, `IS NULL`, 
`NOT NULL`, binary `=`, nested `=`). Output type and value checks cover both 
`use_large_types` modes with a checked allocator. `go test -race ./table/...`, 
`go vet ./table/...`, and `golangci-lint run ./table/...` pass.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to