Copilot commented on code in PR #2677:
URL: https://github.com/apache/iceberg-go/pull/2677#discussion_r4646969789
##########
table/internal/parquet_files.go:
##########
@@ -2314,6 +2314,71 @@ func (w wrapPqArrowReader) PrunedSchema(projectedIDs
map[int]struct{}, mapping i
return pruneParquetColumns(w.Manifest, projectedIDs, false, mapping)
}
+// ReadDictionaries rebuilds the schema manifest because pqarrow decides
+// dictionary reads when it builds the manifest, not when it reads. The value
+// receiver is safe: the writes go through the embedded *pqarrow.FileReader.
+func (w wrapPqArrowReader) ReadDictionaries(fieldIDs map[int]struct{}, mapping
iceberg.NameMapping) error {
+ leaves := &dictionaryLeafVisitor{fieldIDs: fieldIDs}
+ if _, err := visitParquetManifest(w.Manifest, leaves,
&iceberg.MappedField{Fields: mapping}); err != nil {
+ return err
+ }
+ if len(leaves.colIndices) == 0 {
+ return nil
+ }
+
+ props := w.Props
+ for _, idx := range leaves.colIndices {
+ props.SetReadDict(idx, true)
+ }
+ meta := w.ParquetReader().MetaData()
+ manifest, err := pqarrow.NewSchemaManifest(meta.Schema,
meta.KeyValueMetadata(), &props)
+ if err != nil {
+ return err
+ }
+ w.Props, w.Manifest = props, manifest
+
+ return nil
+}
Review Comment:
This method mutates `w.Props` and `w.Manifest`, but it has a value receiver,
so those assignments won’t persist back to the caller’s `wrapPqArrowReader`
instance. As a result, the subsequent `PrunedSchema`/`GetRecords` calls may
still use the old manifest/properties. Fix by switching `ReadDictionaries` to a
pointer receiver (`func (w *wrapPqArrowReader) ...`) and updating the comment
(the current note about the value receiver being safe is incorrect for the
`Props`/`Manifest` fields).
##########
table/internal/interfaces.go:
##########
@@ -75,6 +75,15 @@ type FileReader interface {
ReadTable(context.Context) (arrow.Table, error)
}
+// DictionaryReader is a FileReader that can return selected string and binary
+// leaf columns as Arrow dictionaries.
+type DictionaryReader interface {
+ // ReadDictionaries requests dictionary reads for the leaves with the
given
+ // field IDs. Call it before PrunedSchema or GetRecords. Readers may
still
+ // return plain arrays when a file's stored Arrow schema requires them.
+ ReadDictionaries(fieldIDs map[int]struct{}, mapping
iceberg.NameMapping) error
+}
Review Comment:
The comment says `DictionaryReader` is a `FileReader`, but the interface
doesn’t embed `FileReader` (or otherwise require `ReadTable`). Either embed
`FileReader` (e.g., `type DictionaryReader interface { FileReader;
ReadDictionaries(...) error }`) or adjust the comment to avoid implying it
includes the `FileReader` contract.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]