hcrosse opened a new issue, #2410: URL: https://github.com/apache/iceberg-go/issues/2410
### Feature Request / Improvement When a string column is dictionary-encoded in Parquet, an iceberg-go scan still returns it as a plain Arrow `utf8` column. Expanding the values costs read time and memory. A caller that counts or groups by the column then has to hash every string instead of working on a small dictionary and integer indices. I reproduced this on v0.7.0 and main at `c37ed1a8401c325fbc3291a04360a2686dd939d8`, both with arrow-go v18.8.0. There is no option to request dictionary output for data columns. The Parquet reader enables dictionary output only for `file_path` in position-delete files: https://github.com/apache/iceberg-go/blob/c37ed1a8401c325fbc3291a04360a2686dd939d8/table/internal/parquet_files.go#L2822-L2833. Enabling dictionary output in the reader would not be enough by itself. The scan casts each column to the Arrow type of its Iceberg field in `castIfNeeded`, which turns a dictionary column back into `utf8`: https://github.com/apache/iceberg-go/blob/c37ed1a8401c325fbc3291a04360a2686dd939d8/table/arrow_utils.go#L1100-L1142. **Measured cost** I measured main with Go 1.26.3 on a 4-vCPU Linux VM, using local files and the default 131,072-row batch size. I pinned each process to one core with `taskset` and set `GOMAXPROCS=1`. The synthetic table has 10 million rows, which I wrote with `Table.Append` into four files. The string columns `region` and `category` have 3 and 64 distinct values. The `tenant` column has 4,096. Every data page in those three columns is dictionary-encoded. The other columns are a timestamp, a unique `request_id` string, and an integer, all plain-encoded. I read the same files through the scan and directly with `pqarrow`. I also repeated the direct read with `SetReadDict` on the three string columns. The direct reads use the scanner's reader settings, so only `SetReadDict` differs between them. Each figure is the median of five fresh processes. | Measurement | iceberg-go scan | Direct plain read | Direct read with dictionaries | | --- | ---: | ---: | ---: | | Read all columns | 2,268 ms | 2,419 ms | 1,661 ms | | Read only `tenant` | 431 ms | 559 ms | 146 ms | | Total allocation, all columns | 2,957 MiB | 2,955 MiB | 1,941 MiB | | Count rows per tenant, after reading | 311 ms | 311 ms | 20 ms | The direct plain read is no faster than the scan. It runs more garbage collections, and its allocation is the same. The saving comes from dictionary output. The counting consumer was timed separately from reading and produced identical counts. With 8,192-row batches, the dictionary read still allocated 35% less than the scan. Counting took 151 ms, compared with 20 ms at 131,072 rows, because arrow-go copies the dictionary into every batch. I reported that in https://github.com/apache/arrow-go/issues/1443. I did not modify the scanner. The direct reads skip its schema projection, filtering, and deletes. The test table has no delete files, and the scans used no filter. I did not test remote storage, nulls, or dictionaries larger than 4,096 entries. A caller can read the planned files with `pqarrow` directly, as I did for these measurements. Then the caller has to apply delete files, residual filters, and projection by field ID itself. **Reproduction** This program writes a dictionary-encoded Parquet column through iceberg-go and shows that the scan returns it as `utf8`. It needs these modules: ```text github.com/apache/arrow-go/v18 v18.8.0 github.com/apache/iceberg-go v0.7.0 github.com/uptrace/bun/driver/sqliteshim v1.3.0 ``` ```go package main import ( "context" "fmt" "log" "os" "path/filepath" "strings" "github.com/apache/arrow-go/v18/arrow" "github.com/apache/arrow-go/v18/arrow/array" "github.com/apache/arrow-go/v18/arrow/memory" "github.com/apache/arrow-go/v18/parquet/file" "github.com/apache/iceberg-go" "github.com/apache/iceberg-go/catalog" sqlcat "github.com/apache/iceberg-go/catalog/sql" "github.com/uptrace/bun/driver/sqliteshim" ) func main() { ctx := context.Background() dir, err := os.MkdirTemp("", "dict") check(err) cat, err := catalog.Load(ctx, "local", iceberg.Properties{ "type": "sql", "uri": "file:" + filepath.Join(dir, "catalog.db"), "warehouse": "file://" + dir, sqlcat.DriverKey: sqliteshim.ShimName, sqlcat.DialectKey: string(sqlcat.SQLite), }) check(err) check(cat.CreateNamespace(ctx, catalog.ToIdentifier("db"), nil)) schema := iceberg.NewSchema(0, iceberg.NestedField{ID: 1, Name: "region", Type: iceberg.PrimitiveTypes.String}) tbl, err := cat.CreateTable(ctx, catalog.ToIdentifier("db", "t"), schema) check(err) regions := []string{"us-east-1", "eu-west-1", "ap-southeast-2"} builder := array.NewStringBuilder(memory.DefaultAllocator) for i := range 100_000 { builder.Append(regions[i%len(regions)]) } arrowSchema := arrow.NewSchema([]arrow.Field{{Name: "region", Type: arrow.BinaryTypes.String, Nullable: true}}, nil) batch := array.NewRecordBatch(arrowSchema, []arrow.Array{builder.NewArray()}, -1) reader, err := array.NewRecordReader(arrowSchema, []arrow.RecordBatch{batch}) check(err) tbl, err = tbl.Append(ctx, reader, nil) check(err) tasks, err := tbl.Scan().PlanFiles(ctx) check(err) pf, err := file.OpenParquetFile(strings.TrimPrefix(tasks[0].File.FilePath(), "file://"), false) check(err) column, err := pf.MetaData().RowGroup(0).ColumnChunk(0) check(err) fmt.Printf("data file: dictionary page=%v encodings=%v\n", column.HasDictionaryPage(), column.Encodings()) _, records, err := tbl.Scan().ToArrowRecords(ctx) check(err) for rec, err := range records { check(err) fmt.Printf("scan type: %s\n", rec.Schema().Field(0).Type) break } } func check(err error) { if err != nil { log.Fatal(err) } } ``` Output, the same on v0.7.0 and main: ```text data file: dictionary page=true encodings=[RLE_DICTIONARY PLAIN RLE] scan type: utf8 ``` **Expected behavior** Callers can request dictionary output for selected string columns. The default and unlisted columns stay unchanged. Requested columns retain consistent Arrow types across files, including files with plain-encoded pages or missing columns. Filters, projection, and deletes still apply. **Related work** - https://github.com/apache/iceberg-python/pull/3461 added `dictionary_columns` for Arrow output. - https://github.com/apache/iceberg-python/pull/3595 fixed the streaming path, where a cast to the target schema removed the dictionary encoding. - https://github.com/apache/iceberg-go/pull/931 enabled dictionaries by default when writing, not in scan output. - https://github.com/apache/iceberg-go/pull/2006 and https://github.com/apache/iceberg-go/pull/2231 use dictionaries for row-group pruning and position deletes, not scan output. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
