This is an automated email from the ASF dual-hosted git repository.
zeroshade pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow-go.git
The following commit(s) were added to refs/heads/main by this push:
new b9f4e362 feat(parquet): add WithDictionaryCostFallback writer property
(#1036)
b9f4e362 is described below
commit b9f4e3623dc22d5994cca6f5a37752fbbe11c2d5
Author: Tom Frank <[email protected]>
AuthorDate: Fri Aug 7 18:33:50 2026 +0300
feat(parquet): add WithDictionaryCostFallback writer property (#1036)
### Rationale for this change
The pre-first-page dictionary cost check (`checkDictionarySizeLimit`,
mirroring parquet-mr's `FallbackValuesWriter.shouldFallBack`) compares
**uncompressed** sizes: the dictionary is discarded when `dictSize +
encodedSize >= rawSize`. That comparison only predicts on-disk cost when
pages are stored uncompressed. With a page compressor configured it can
discard dictionaries that clearly win after compression — bit-packed
dictionary indices are near-incompressible but small, while
PLAIN-encoded values often compress far less than the raw-size
comparison assumes. In our benchmarks (ZSTD level 3), a mid-cardinality
`Time64` column written PLAIN after the fallback was 2× larger than the
same column dictionary-encoded (7.1 MB vs 3.6 MB compressed);
mid-cardinality integer columns lost 20%+.
### What changes are included in this PR?
- **The cost fallback is now codec-aware by default**: it only runs for
columns written without page compression, where the uncompressed-size
comparison is sound. For columns with a codec configured the dictionary
is kept (still bounded by the `DictionaryPageSizeLimit` fallback, which
is unaffected). Uncompressed columns keep today's behavior exactly.
- **A per-column override**: `parquet.WithDictionaryCostFallbackFor(path
string, enabled bool)` / `WithDictionaryCostFallbackPath(path
ColumnPath, enabled bool)`, stored as a tri-state
`DictionaryCostFallback *bool` on `ColumnProperties` (nil = codec-based
default). It overrides the default in either direction: `false` keeps an
explicitly requested dictionary for an uncompressed column, `true`
forces the check for a compressed one. Useful for callers that decide
dictionary usage from their own measurements (e.g. trial-encoding a
sample of the actual data with the actual codec).
- `WriterProperties.DictionaryCostFallbackEnabledFor(path string) bool`
resolves the per-column override, falling back to the codec-based
default.
### Are these changes tested?
Yes — `parquet/file/dict_cost_fallback_test.go` covers: (1) an
uncompressed column still discards a non-paying dictionary by default,
(2) global and (3) per-column compression keep the dictionary by
default, (4) a per-column disable keeps an uncompressed column's
dictionary while (5) a disable scoped to a different column leaves the
fallback active, (6) a per-column enable forces the check for a
compressed column, and (7) the `DictionaryPageSizeLimit` fallback still
applies when the cost fallback is off. The existing
`TestDictFallbackDiscardsOrphanDict` snappy variants now force the check
via the per-column option to keep exercising the orphan-discard path
under a codec. Existing `parquet`, `parquet/file`, and
`parquet/internal/encoding` suites pass.
### Are there any user-facing changes?
For columns written with a compression codec, the cost-based dictionary
fallback no longer runs by default — a requested dictionary is kept
unless it overflows `DictionaryPageSizeLimit`. Uncompressed columns are
unchanged. New writer properties: `WithDictionaryCostFallbackFor` /
`WithDictionaryCostFallbackPath`, and a `DictionaryCostFallback *bool`
field on `ColumnProperties`.
---
parquet/file/column_writer_types.gen.go | 72 ++++++++++++----
parquet/file/column_writer_types.gen.go.tmpl | 9 +-
parquet/file/dict_cost_fallback_test.go | 124 +++++++++++++++++++++++++++
parquet/file/dict_fallback_repro_test.go | 9 ++
parquet/writer_properties.go | 71 ++++++++++++---
5 files changed, 257 insertions(+), 28 deletions(-)
diff --git a/parquet/file/column_writer_types.gen.go
b/parquet/file/column_writer_types.gen.go
index d0dc0a5c..bab348a3 100644
--- a/parquet/file/column_writer_types.gen.go
+++ b/parquet/file/column_writer_types.gen.go
@@ -245,8 +245,13 @@ func (w *Int32ColumnChunkWriter)
checkDictionarySizeLimit() {
// plus the encoded indices meet or exceed the raw input bytes, fall
back
// to PLAIN now and discard the dictionary — avoiding the
mid-cardinality
// case where a dict page stays in the file alongside PLAIN pages
without
- // any net compression win.
- if !w.dictPageWritten && len(w.pages) == 0 {
+ // any net compression win. The comparison uses uncompressed sizes,
which
+ // is only meaningful when pages are stored uncompressed: with a codec
+ // configured, the near-incompressible dict indices routinely beat PLAIN
+ // pages that compress well. The check therefore runs only for columns
+ // written without page compression, unless overridden per column with
+ // WithDictionaryCostFallbackFor.
+ if !w.dictPageWritten && len(w.pages) == 0 &&
w.props.DictionaryCostFallbackEnabledFor(w.descr.Path()) {
rawSize := dictEnc.ObservedRawSize()
encodedSize := dictEnc.EstimatedDataEncodedSize()
dictSize := int64(dictEnc.DictEncodedSize())
@@ -507,8 +512,13 @@ func (w *Int64ColumnChunkWriter)
checkDictionarySizeLimit() {
// plus the encoded indices meet or exceed the raw input bytes, fall
back
// to PLAIN now and discard the dictionary — avoiding the
mid-cardinality
// case where a dict page stays in the file alongside PLAIN pages
without
- // any net compression win.
- if !w.dictPageWritten && len(w.pages) == 0 {
+ // any net compression win. The comparison uses uncompressed sizes,
which
+ // is only meaningful when pages are stored uncompressed: with a codec
+ // configured, the near-incompressible dict indices routinely beat PLAIN
+ // pages that compress well. The check therefore runs only for columns
+ // written without page compression, unless overridden per column with
+ // WithDictionaryCostFallbackFor.
+ if !w.dictPageWritten && len(w.pages) == 0 &&
w.props.DictionaryCostFallbackEnabledFor(w.descr.Path()) {
rawSize := dictEnc.ObservedRawSize()
encodedSize := dictEnc.EstimatedDataEncodedSize()
dictSize := int64(dictEnc.DictEncodedSize())
@@ -769,8 +779,13 @@ func (w *Int96ColumnChunkWriter)
checkDictionarySizeLimit() {
// plus the encoded indices meet or exceed the raw input bytes, fall
back
// to PLAIN now and discard the dictionary — avoiding the
mid-cardinality
// case where a dict page stays in the file alongside PLAIN pages
without
- // any net compression win.
- if !w.dictPageWritten && len(w.pages) == 0 {
+ // any net compression win. The comparison uses uncompressed sizes,
which
+ // is only meaningful when pages are stored uncompressed: with a codec
+ // configured, the near-incompressible dict indices routinely beat PLAIN
+ // pages that compress well. The check therefore runs only for columns
+ // written without page compression, unless overridden per column with
+ // WithDictionaryCostFallbackFor.
+ if !w.dictPageWritten && len(w.pages) == 0 &&
w.props.DictionaryCostFallbackEnabledFor(w.descr.Path()) {
rawSize := dictEnc.ObservedRawSize()
encodedSize := dictEnc.EstimatedDataEncodedSize()
dictSize := int64(dictEnc.DictEncodedSize())
@@ -1031,8 +1046,13 @@ func (w *Float32ColumnChunkWriter)
checkDictionarySizeLimit() {
// plus the encoded indices meet or exceed the raw input bytes, fall
back
// to PLAIN now and discard the dictionary — avoiding the
mid-cardinality
// case where a dict page stays in the file alongside PLAIN pages
without
- // any net compression win.
- if !w.dictPageWritten && len(w.pages) == 0 {
+ // any net compression win. The comparison uses uncompressed sizes,
which
+ // is only meaningful when pages are stored uncompressed: with a codec
+ // configured, the near-incompressible dict indices routinely beat PLAIN
+ // pages that compress well. The check therefore runs only for columns
+ // written without page compression, unless overridden per column with
+ // WithDictionaryCostFallbackFor.
+ if !w.dictPageWritten && len(w.pages) == 0 &&
w.props.DictionaryCostFallbackEnabledFor(w.descr.Path()) {
rawSize := dictEnc.ObservedRawSize()
encodedSize := dictEnc.EstimatedDataEncodedSize()
dictSize := int64(dictEnc.DictEncodedSize())
@@ -1293,8 +1313,13 @@ func (w *Float64ColumnChunkWriter)
checkDictionarySizeLimit() {
// plus the encoded indices meet or exceed the raw input bytes, fall
back
// to PLAIN now and discard the dictionary — avoiding the
mid-cardinality
// case where a dict page stays in the file alongside PLAIN pages
without
- // any net compression win.
- if !w.dictPageWritten && len(w.pages) == 0 {
+ // any net compression win. The comparison uses uncompressed sizes,
which
+ // is only meaningful when pages are stored uncompressed: with a codec
+ // configured, the near-incompressible dict indices routinely beat PLAIN
+ // pages that compress well. The check therefore runs only for columns
+ // written without page compression, unless overridden per column with
+ // WithDictionaryCostFallbackFor.
+ if !w.dictPageWritten && len(w.pages) == 0 &&
w.props.DictionaryCostFallbackEnabledFor(w.descr.Path()) {
rawSize := dictEnc.ObservedRawSize()
encodedSize := dictEnc.EstimatedDataEncodedSize()
dictSize := int64(dictEnc.DictEncodedSize())
@@ -1701,8 +1726,13 @@ func (w *BooleanColumnChunkWriter)
checkDictionarySizeLimit() {
// plus the encoded indices meet or exceed the raw input bytes, fall
back
// to PLAIN now and discard the dictionary — avoiding the
mid-cardinality
// case where a dict page stays in the file alongside PLAIN pages
without
- // any net compression win.
- if !w.dictPageWritten && len(w.pages) == 0 {
+ // any net compression win. The comparison uses uncompressed sizes,
which
+ // is only meaningful when pages are stored uncompressed: with a codec
+ // configured, the near-incompressible dict indices routinely beat PLAIN
+ // pages that compress well. The check therefore runs only for columns
+ // written without page compression, unless overridden per column with
+ // WithDictionaryCostFallbackFor.
+ if !w.dictPageWritten && len(w.pages) == 0 &&
w.props.DictionaryCostFallbackEnabledFor(w.descr.Path()) {
rawSize := dictEnc.ObservedRawSize()
encodedSize := dictEnc.EstimatedDataEncodedSize()
dictSize := int64(dictEnc.DictEncodedSize())
@@ -2073,8 +2103,13 @@ func (w *ByteArrayColumnChunkWriter)
checkDictionarySizeLimit() {
// plus the encoded indices meet or exceed the raw input bytes, fall
back
// to PLAIN now and discard the dictionary — avoiding the
mid-cardinality
// case where a dict page stays in the file alongside PLAIN pages
without
- // any net compression win.
- if !w.dictPageWritten && len(w.pages) == 0 {
+ // any net compression win. The comparison uses uncompressed sizes,
which
+ // is only meaningful when pages are stored uncompressed: with a codec
+ // configured, the near-incompressible dict indices routinely beat PLAIN
+ // pages that compress well. The check therefore runs only for columns
+ // written without page compression, unless overridden per column with
+ // WithDictionaryCostFallbackFor.
+ if !w.dictPageWritten && len(w.pages) == 0 &&
w.props.DictionaryCostFallbackEnabledFor(w.descr.Path()) {
rawSize := dictEnc.ObservedRawSize()
encodedSize := dictEnc.EstimatedDataEncodedSize()
dictSize := int64(dictEnc.DictEncodedSize())
@@ -2453,8 +2488,13 @@ func (w *FixedLenByteArrayColumnChunkWriter)
checkDictionarySizeLimit() {
// plus the encoded indices meet or exceed the raw input bytes, fall
back
// to PLAIN now and discard the dictionary — avoiding the
mid-cardinality
// case where a dict page stays in the file alongside PLAIN pages
without
- // any net compression win.
- if !w.dictPageWritten && len(w.pages) == 0 {
+ // any net compression win. The comparison uses uncompressed sizes,
which
+ // is only meaningful when pages are stored uncompressed: with a codec
+ // configured, the near-incompressible dict indices routinely beat PLAIN
+ // pages that compress well. The check therefore runs only for columns
+ // written without page compression, unless overridden per column with
+ // WithDictionaryCostFallbackFor.
+ if !w.dictPageWritten && len(w.pages) == 0 &&
w.props.DictionaryCostFallbackEnabledFor(w.descr.Path()) {
rawSize := dictEnc.ObservedRawSize()
encodedSize := dictEnc.EstimatedDataEncodedSize()
dictSize := int64(dictEnc.DictEncodedSize())
diff --git a/parquet/file/column_writer_types.gen.go.tmpl
b/parquet/file/column_writer_types.gen.go.tmpl
index 30c1e9e3..6988c468 100644
--- a/parquet/file/column_writer_types.gen.go.tmpl
+++ b/parquet/file/column_writer_types.gen.go.tmpl
@@ -574,8 +574,13 @@ func (w *{{.Name}}ColumnChunkWriter)
checkDictionarySizeLimit() {
// plus the encoded indices meet or exceed the raw input bytes, fall back
// to PLAIN now and discard the dictionary — avoiding the mid-cardinality
// case where a dict page stays in the file alongside PLAIN pages without
- // any net compression win.
- if !w.dictPageWritten && len(w.pages) == 0 {
+ // any net compression win. The comparison uses uncompressed sizes, which
+ // is only meaningful when pages are stored uncompressed: with a codec
+ // configured, the near-incompressible dict indices routinely beat PLAIN
+ // pages that compress well. The check therefore runs only for columns
+ // written without page compression, unless overridden per column with
+ // WithDictionaryCostFallbackFor.
+ if !w.dictPageWritten && len(w.pages) == 0 &&
w.props.DictionaryCostFallbackEnabledFor(w.descr.Path()) {
rawSize := dictEnc.ObservedRawSize()
encodedSize := dictEnc.EstimatedDataEncodedSize()
dictSize := int64(dictEnc.DictEncodedSize())
diff --git a/parquet/file/dict_cost_fallback_test.go
b/parquet/file/dict_cost_fallback_test.go
new file mode 100644
index 00000000..90cb0792
--- /dev/null
+++ b/parquet/file/dict_cost_fallback_test.go
@@ -0,0 +1,124 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package file_test
+
+import (
+ "bytes"
+ "testing"
+
+ "github.com/apache/arrow-go/v18/arrow/memory"
+ "github.com/apache/arrow-go/v18/parquet"
+ "github.com/apache/arrow-go/v18/parquet/compress"
+ "github.com/apache/arrow-go/v18/parquet/file"
+ "github.com/apache/arrow-go/v18/parquet/internal/encoding"
+ "github.com/apache/arrow-go/v18/parquet/metadata"
+ "github.com/apache/arrow-go/v18/parquet/schema"
+ "github.com/stretchr/testify/assert"
+ "github.com/stretchr/testify/require"
+)
+
+// writeAllDistinctInt64Chunk writes one buffered row group with a single int64
+// column of all-distinct values, for which the dictionary plus its indices can
+// never beat the raw PLAIN size, and returns the resulting column chunk
metadata.
+func writeAllDistinctInt64Chunk(t *testing.T, props *parquet.WriterProperties)
*metadata.ColumnChunkMetaData {
+ t.Helper()
+
+ sink := encoding.NewBufferWriter(0, memory.DefaultAllocator)
+ fields := schema.FieldList{schema.NewInt64Node("col",
parquet.Repetitions.Required, -1)}
+ sc, err := schema.NewGroupNode("schema", parquet.Repetitions.Required,
fields, -1)
+ require.NoError(t, err)
+
+ writer := file.NewParquetWriter(sink, sc, file.WithWriterProps(props))
+ rgw := writer.AppendBufferedRowGroup()
+ cwr, err := rgw.Column(0)
+ require.NoError(t, err)
+ cw := cwr.(*file.Int64ColumnChunkWriter)
+
+ values := make([]int64, 1000)
+ for i := range values {
+ values[i] = int64(i)*1_000_003 + 7
+ }
+ _, err = cw.WriteBatch(values, nil, nil)
+ require.NoError(t, err)
+ require.NoError(t, rgw.Close())
+ require.NoError(t, writer.Close())
+
+ buffer := sink.Finish()
+ t.Cleanup(buffer.Release)
+ reader, err := file.NewParquetReader(bytes.NewReader(buffer.Bytes()))
+ require.NoError(t, err)
+ t.Cleanup(func() { reader.Close() })
+
+ require.EqualValues(t, 1, reader.NumRowGroups())
+ chunk, err := reader.RowGroup(0).MetaData().ColumnChunk(0)
+ require.NoError(t, err)
+ return chunk
+}
+
+func TestDictionaryCostFallback(t *testing.T) {
+ t.Run("uncompressed discards non-paying dictionary by default", func(t
*testing.T) {
+ chunk := writeAllDistinctInt64Chunk(t,
parquet.NewWriterProperties())
+ assert.False(t, chunk.HasDictionaryPage())
+ })
+
+ t.Run("compressed keeps dictionary by default", func(t *testing.T) {
+ chunk := writeAllDistinctInt64Chunk(t,
parquet.NewWriterProperties(
+ parquet.WithCompression(compress.Codecs.Snappy),
+ ))
+ assert.True(t, chunk.HasDictionaryPage())
+ })
+
+ t.Run("per-column compression keeps that column's dictionary", func(t
*testing.T) {
+ chunk := writeAllDistinctInt64Chunk(t,
parquet.NewWriterProperties(
+ parquet.WithCompressionFor("col",
compress.Codecs.Snappy),
+ ))
+ assert.True(t, chunk.HasDictionaryPage())
+ })
+
+ t.Run("per-column disable keeps an uncompressed column's dictionary",
func(t *testing.T) {
+ chunk := writeAllDistinctInt64Chunk(t,
parquet.NewWriterProperties(
+ parquet.WithDictionaryFor("col", true),
+ parquet.WithDictionaryCostFallbackFor("col", false),
+ ))
+ assert.True(t, chunk.HasDictionaryPage())
+ })
+
+ t.Run("per-column disable for another column leaves the fallback
active", func(t *testing.T) {
+ chunk := writeAllDistinctInt64Chunk(t,
parquet.NewWriterProperties(
+ parquet.WithDictionaryFor("col", true),
+ parquet.WithDictionaryCostFallbackFor("other", false),
+ ))
+ assert.False(t, chunk.HasDictionaryPage())
+ })
+
+ t.Run("per-column enable forces the check for a compressed column",
func(t *testing.T) {
+ chunk := writeAllDistinctInt64Chunk(t,
parquet.NewWriterProperties(
+ parquet.WithCompression(compress.Codecs.Snappy),
+ parquet.WithDictionaryCostFallbackFor("col", true),
+ ))
+ assert.False(t, chunk.HasDictionaryPage())
+ })
+
+ t.Run("page size limit fallback still applies when the cost fallback is
off", func(t *testing.T) {
+ chunk := writeAllDistinctInt64Chunk(t,
parquet.NewWriterProperties(
+ parquet.WithDictionaryFor("col", true),
+ parquet.WithDictionaryCostFallbackFor("col", false),
+ parquet.WithDictionaryPageSizeLimit(512),
+ ))
+ assert.False(t, chunk.HasDictionaryPage())
+ })
+}
diff --git a/parquet/file/dict_fallback_repro_test.go
b/parquet/file/dict_fallback_repro_test.go
index f82b3253..c6ed14cb 100644
--- a/parquet/file/dict_fallback_repro_test.go
+++ b/parquet/file/dict_fallback_repro_test.go
@@ -66,6 +66,11 @@ func runDictFallbackNoDictPageCheck(t *testing.T, version
parquet.Version, codec
dictPageSizeLimit: dictPageSizeLimit,
dataPageSize: dataPageSize,
codec: codec,
+ // The pre-first-page cost check is what discards the
dictionary before
+ // any dict data page is cut here. It only runs by default for
+ // uncompressed columns, so force it for the compressed
variants to keep
+ // exercising the orphan-discard path (parquet-mr parity) under
a codec.
+ forceCostFallback: codec != compress.Codecs.Uncompressed,
}
knobsOn := knobs
@@ -243,6 +248,7 @@ type writerKnobs struct {
dictPageSizeLimit int64
dataPageSize int64
codec compress.Compression
+ forceCostFallback bool
}
type pageInfo struct {
@@ -294,6 +300,9 @@ func writeByteArrayColumn(t *testing.T, values
[]parquet.ByteArray, version parq
if version == parquet.V2_LATEST {
opts = append(opts,
parquet.WithDataPageVersion(parquet.DataPageV2))
}
+ if knobs.forceCostFallback {
+ opts = append(opts, parquet.WithDictionaryCostFallbackFor("v",
true))
+ }
props := parquet.NewWriterProperties(opts...)
var buf bytes.Buffer
diff --git a/parquet/writer_properties.go b/parquet/writer_properties.go
index 4327bcf1..9cedc873 100644
--- a/parquet/writer_properties.go
+++ b/parquet/writer_properties.go
@@ -67,9 +67,14 @@ const (
// ColumnProperties defines the encoding, codec, and so on for a given column.
type ColumnProperties struct {
- Encoding Encoding
- Codec compress.Compression
- DictionaryEnabled bool
+ Encoding Encoding
+ Codec compress.Compression
+ DictionaryEnabled bool
+ // DictionaryCostFallback overrides the codec-based default of the
+ // pre-first-page cost-based dictionary fallback; nil keeps the default
+ // (enabled only for columns written without page compression). See
+ // WithDictionaryCostFallbackFor.
+ DictionaryCostFallback *bool
StatsEnabled bool
PageIndexEnabled bool
MaxStatsSize int64
@@ -121,6 +126,7 @@ type writerPropConfig struct {
codecs map[string]compress.Compression
compressLevel map[string]int
dictEnabled map[string]bool
+ dictCostFallback map[string]bool
statsEnabled map[string]bool
indexEnabled map[string]bool
bloomFilterNDVs map[string]int64
@@ -167,6 +173,29 @@ func WithDictionaryPageSizeLimit(limit int64)
WriterProperty {
}
}
+// WithDictionaryCostFallbackFor controls the cost-based dictionary fallback
for the
+// given column path: before the first data page of a column chunk is cut, the
writer
+// discards the dictionary and falls back to the plain encoding when the
dictionary
+// plus the encoded indices are not smaller than the raw values (mirroring
+// parquet-mr's shouldFallBack). That comparison is done on uncompressed
sizes, so by
+// default it only runs for columns written without page compression — for
columns
+// with a codec configured the dictionary is kept, since near-incompressible
+// dictionary indices routinely beat plain pages that compress well. This
option
+// overrides the codec-based default in either direction: false keeps an
explicitly
+// requested dictionary for an uncompressed column, true forces the check for a
+// compressed one. The DictionaryPageSizeLimit fallback always stays in effect.
+func WithDictionaryCostFallbackFor(path string, enabled bool) WriterProperty {
+ return func(cfg *writerPropConfig) {
+ cfg.dictCostFallback[path] = enabled
+ }
+}
+
+// WithDictionaryCostFallbackPath is like WithDictionaryCostFallbackFor, but
takes
+// a ColumnPath type.
+func WithDictionaryCostFallbackPath(path ColumnPath, enabled bool)
WriterProperty {
+ return WithDictionaryCostFallbackFor(path.String(), enabled)
+}
+
// WithBatchSize specifies the number of rows to use for batch writes to
columns
func WithBatchSize(batch int64) WriterProperty {
return func(cfg *writerPropConfig) {
@@ -560,6 +589,7 @@ func NewWriterProperties(opts ...WriterProperty)
*WriterProperties {
codecs:
make(map[string]compress.Compression),
compressLevel: make(map[string]int),
dictEnabled: make(map[string]bool),
+ dictCostFallback: make(map[string]bool),
statsEnabled: make(map[string]bool),
indexEnabled: make(map[string]bool),
bloomFilterNDVs: make(map[string]int64),
@@ -598,6 +628,10 @@ func NewWriterProperties(opts ...WriterProperty)
*WriterProperties {
get(key).DictionaryEnabled = value
}
+ for key, value := range cfg.dictCostFallback {
+ get(key).DictionaryCostFallback = &value
+ }
+
for key, value := range cfg.statsEnabled {
get(key).StatsEnabled = value
}
@@ -635,13 +669,14 @@ func (w *WriterProperties) FileEncryptionProperties()
*FileEncryptionProperties
return w.encryptionProps
}
-func (w *WriterProperties) Allocator() memory.Allocator { return w.mem }
-func (w *WriterProperties) CreatedBy() string { return
w.createdBy }
-func (w *WriterProperties) RootName() string { return
w.rootName }
-func (w *WriterProperties) RootRepetition() Repetition { return
w.rootRepetition }
-func (w *WriterProperties) WriteBatchSize() int64 { return
w.batchSize }
-func (w *WriterProperties) DataPageSize() int64 { return
w.pageSize }
-func (w *WriterProperties) DictionaryPageSizeLimit() int64 { return
w.dictPagesize }
+func (w *WriterProperties) Allocator() memory.Allocator { return w.mem }
+func (w *WriterProperties) CreatedBy() string { return
w.createdBy }
+func (w *WriterProperties) RootName() string { return w.rootName
}
+func (w *WriterProperties) RootRepetition() Repetition { return
w.rootRepetition }
+func (w *WriterProperties) WriteBatchSize() int64 { return
w.batchSize }
+func (w *WriterProperties) DataPageSize() int64 { return w.pageSize
}
+func (w *WriterProperties) DictionaryPageSizeLimit() int64 { return
w.dictPagesize }
+
func (w *WriterProperties) Version() Version { return
w.parquetVersion }
func (w *WriterProperties) DataPageVersion() DataPageVersion { return
w.dataPageVersion }
func (w *WriterProperties) MaxRowGroupLength() int64 { return
w.maxRowGroupLen }
@@ -732,6 +767,22 @@ func (w *WriterProperties) DictionaryEnabledFor(path
string) bool {
return w.defColumnProps.DictionaryEnabled
}
+// DictionaryCostFallbackEnabledFor returns whether the pre-first-page
cost-based
+// dictionary fallback applies for the given column path: an explicit
per-column
+// WithDictionaryCostFallbackFor setting wins, otherwise the fallback is
enabled
+// only when the column is written without page compression, since the cost
+// comparison is done on uncompressed sizes.
+func (w *WriterProperties) DictionaryCostFallbackEnabledFor(path string) bool {
+ p, ok := w.columnProps[path]
+ if !ok {
+ return w.defColumnProps.Codec == compress.Codecs.Uncompressed
+ }
+ if p.DictionaryCostFallback != nil {
+ return *p.DictionaryCostFallback
+ }
+ return p.Codec == compress.Codecs.Uncompressed
+}
+
// DictionaryEnabledPath is the same as DictionaryEnabledFor but takes a
ColumnPath object.
func (w *WriterProperties) DictionaryEnabledPath(path ColumnPath) bool {
return w.DictionaryEnabledFor(path.String())
