zeroshade commented on code in PR #2445:
URL: https://github.com/apache/iceberg-go/pull/2445#discussion_r4645476715
##########
table/equality_delete_reader.go:
##########
@@ -1657,6 +1657,97 @@ func (m *equalityDeleteMask) applySingleBytesDictionary(
return true
}
+const (
+ dictionaryEntryUnknown uint8 = iota
+ dictionaryEntryKept
+ dictionaryEntryDeleted
+)
+
+// dictionaryEntryMemo caches one delete set's results across a file's
+// batches while they share a dictionary object. Batches must be filtered one
+// at a time. Holding the dictionary without Retain still prevents its address
+// from being reused, and on a match the current batch keeps it alive. Reuse
+// relies on Arrow arrays being immutable.
+type dictionaryEntryMemo struct {
+ dict arrow.ArrayData
+ results []uint8
+}
+
+// apply looks up each dictionary entry that a row references at most once
+// per dictionary. It returns false, leaving m unchanged, for value types other
+// than string and binary.
+func (c *dictionaryEntryMemo) apply(
+ m *equalityDeleteMask,
+ column *array.Dictionary,
+ deleteSet *singleBytesEqualityDeleteSet,
+ mem memory.Allocator,
+) bool {
+ dict := column.Dictionary()
+ if data := column.Data().Dictionary(); c.dict != data {
Review Comment:
nit: the reset runs before the type switch, so a dictionary with an
unsupported value type (e.g. fixed-size binary) still allocates and clears a
`dict.Len()` results buffer, on every batch if dictionaries aren't shared, and
then returns false. Checking the value type first would skip that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]