hcrosse opened a new issue, #1443: URL: https://github.com/apache/arrow-go/issues/1443
### Describe the enhancement requested With `SetReadDict` enabled, `pqarrow` copies a column's whole dictionary into every record batch, even when no values were added since the previous batch. The cost grows with dictionary size times the number of batches. At 1M distinct values and 8,192-row batches, reading one 2M-row row group allocates about 7 GiB on main, against 366 MiB when the dictionary is reused. [`GetBuilderChunks` flushes the dictionary builder](https://github.com/apache/arrow-go/blob/22c9f26df0abbc2066f5d9ae45f31720d29a4300/parquet/file/record_reader.go#L1072-L1084) for each batch. [`newWithDictOffset`](https://github.com/apache/arrow-go/blob/22c9f26df0abbc2066f5d9ae45f31720d29a4300/arrow/array/dictionary.go#L1006-L1039) then calls [`GetDictArrayData`](https://github.com/apache/arrow-go/blob/22c9f26df0abbc2066f5d9ae45f31720d29a4300/arrow/array/util.go#L321-L354), which allocates buffers and copies the dictionary's offsets and values. Because every batch gets its own copy, code that checks dictionary identity misses its fast path. [`array.Concatenate`](https://github.com/apache/arrow-go/blob/22c9f26df0abbc2066f5d9ae45f31720d29a4300/arrow/array/concat.go#L625-L640) and the [IPC writer](https://github.com/apache/arrow-go/blob/22c9f26df0abbc2066f5d9ae45f31720d29a4300/arrow/ipc/writer.go#L262-L276) both compare dictionary values whenever the pointers differ. I have not measured that cost. Dictionary arrays are immutable and reference-counted, so batches can share one. I found nothing in arrow-go that relies on each batch having its own copy. **Measurements** I compared main against a prototype that reuses the last dictionary while the memo table has not grown. The benchmark is a separate program from the reproduction below. Its input is 2 million 25-byte strings drawn from a fixed number of distinct values, in one row group. Every distinct value appears once before the remaining rows are drawn at random. I disabled the writer's cost fallback with `WithDictionaryCostFallbackFor` so that every data page is `RLE_DICTIONARY`. The machine was an 8-vCPU AMD EPYC Linux VM with about 16 GiB RAM, running Go 1.26.10 on amd64. These are medians from six interleaved runs of each version. Each cell gives main → prototype. **ns per row** | Distinct values | 8,192 rows/batch (245 batches) | 65,536 rows/batch (31 batches) | One batch | | --- | --- | --- | --- | | 10k | 21.5 → 5.8 | 7.2 → 5.9 | 6.3 → 7.2 | | 100k | 188 → 12.7 | 35.1 → 12.5 | 13.4 → 13.1 | | 1M | 1,973 → 162 | 363 → 145 | 157 → 138 | **MiB allocated per row-group read** | Distinct values | 8,192 rows/batch (245 batches) | 65,536 rows/batch (31 batches) | One batch | | --- | --- | --- | --- | | 10k | 95 → 26 | 34 → 26 | 41 → 41 | | 100k | 733 → 56 | 139 → 55 | 71 → 71 | | 1M | 7,116 → 366 | 1,195 → 365 | 381 → 381 | Every multi-batch time change in the table is significant (`p=0.002`, benchstat, n=6). The one-batch column and a run with 100 distinct values show no significant time change, and one-batch allocation is identical. Reading without `SetReadDict` is unchanged. At 100k distinct values and 8,192-row batches, 81% of CPU samples are in the `newWithDictOffset` / `GetDictArrayData` call path, most of it in `BinaryMemoTable.CopyOffsetsSubset`. **Reproduction** This program writes 2 million strings with 100k distinct values into one dictionary-encoded row group. For each of three batch sizes, it prints the batch count, how many distinct dictionary arrays the batches carry, the total size of those dictionaries, and the read time. It also prints the encoding used for each page type. Read times vary between runs. I reproduced the same behavior on v18.8.0 and main at `22c9f26df0abbc2066f5d9ae45f31720d29a4300`. Run it in a module that requires `github.com/apache/arrow-go/v18 v18.8.0`. ```go package main import ( "bytes" "context" "fmt" "log" "time" "github.com/apache/arrow-go/v18/arrow" "github.com/apache/arrow-go/v18/arrow/array" "github.com/apache/arrow-go/v18/arrow/memory" "github.com/apache/arrow-go/v18/parquet" "github.com/apache/arrow-go/v18/parquet/file" "github.com/apache/arrow-go/v18/parquet/pqarrow" ) const ( rows = 2_000_000 distinct = 100_000 ) func check(err error) { if err != nil { log.Fatal(err) } } func main() { mem := memory.DefaultAllocator schema := arrow.NewSchema([]arrow.Field{{Name: "s", Type: arrow.BinaryTypes.String}}, nil) builder := array.NewStringBuilder(mem) defer builder.Release() for i := 0; i < rows; i++ { // Runs of 20 keep the writer's cost check from falling back to plain encoding. builder.Append(fmt.Sprintf("value-%018d", i/20%distinct)) } column := builder.NewArray() defer column.Release() record := array.NewRecordBatch(schema, []arrow.Array{column}, rows) defer record.Release() var buf bytes.Buffer writerProps := parquet.NewWriterProperties(parquet.WithDictionaryPageSizeLimit(64 << 20)) writer, err := pqarrow.NewFileWriter(schema, &buf, writerProps, pqarrow.DefaultWriterProps()) check(err) check(writer.Write(record)) check(writer.Close()) pf, err := file.NewParquetReader(bytes.NewReader(buf.Bytes())) check(err) defer pf.Close() chunk, err := pf.MetaData().RowGroup(0).ColumnChunk(0) check(err) fmt.Printf("row groups=%d page encodings=%v\n", pf.NumRowGroups(), chunk.EncodingStats()) for _, batchSize := range []int64{8192, 65536, rows} { props := pqarrow.ArrowReadProperties{BatchSize: batchSize} props.SetReadDict(0, true) fr, err := pqarrow.NewFileReader(pf, props, mem) check(err) rr, err := fr.GetRecordReader(context.Background(), nil, nil) check(err) start := time.Now() batches := 0 seen := map[arrow.ArrayData]bool{} var dictBytes uint64 for rr.Next() { batches++ dict := rr.RecordBatch().Column(0).(*array.Dictionary).Dictionary().Data() if !seen[dict] { seen[dict] = true dictBytes += dict.SizeInBytes() } } check(rr.Err()) rr.Release() fmt.Printf("batch size=%7d batches=%3d distinct dictionaries=%3d dictionary MiB built=%6.1f read=%v\n", batchSize, batches, len(seen), float64(dictBytes)/(1<<20), time.Since(start).Round(time.Millisecond)) } } ``` Output on v18.8.0: ```text row groups=1 page encodings=[{PLAIN DICTIONARY_PAGE} {RLE_DICTIONARY DATA_PAGE}] batch size= 8192 batches=245 distinct dictionaries=245 dictionary MiB built= 654.2 read=407ms batch size= 65536 batches= 31 distinct dictionaries= 31 dictionary MiB built= 82.8 read=61ms batch size=2000000 batches= 1 distinct dictionaries= 1 dictionary MiB built= 2.7 read=26ms ``` Main prints the same counts and sizes. With the prototype, every batch size reports one dictionary and 2.7 MiB, and the 8,192-row read took 28 ms instead of 407 ms. **Expected behavior** Consecutive batches from one column chunk should return the same dictionary `arrow.ArrayData` until new values are added to it. A new dictionary page, such as the next row group's, or a plain-encoded page that adds values still needs a new dictionary. This would not help a column whose pages are mostly plain-encoded, because its dictionary grows with almost every batch. For comparison, arrow-rs [shares the dictionary through `Arc::clone`](https://github.com/apache/arrow-rs/blob/b231cf15efa5dbbbe4ae8503849aa6ab3cfc4e0c/parquet/src/arrow/buffer/dictionary_buffer.rs#L82-L125), rather than copying its values. Related work: - https://github.com/apache/arrow-go/pull/1064 fixed null indexes when exporting a dictionary suffix, not repeated copies of an unchanged dictionary. - https://github.com/apache/arrow-go/pull/1403 made `Concatenate` fast when chunks share dictionary data. Batches from the Parquet reader never share it today, so they miss that path. I have a patch with tests and can open a PR. ### Component(s) Parquet -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
