hcrosse opened a new issue, #2426:
URL: https://github.com/apache/iceberg-go/issues/2426
### Apache Iceberg version
main (development)
### Please describe the bug 🐞
An equality delete on a binary column misses matching rows when the data
file stores that column as an Arrow dictionary. The delete commits without
error, and later scans still return the deleted rows.
I reproduced this on main at `8ef0a092` and on v0.7.0. Both use arrow-go
v18.8.0. The data was written with PyIceberg 0.12.0 and pyarrow 26.0.1 to a
local SQLite catalog.
**Reproduction**
This script appends three rows whose binary column `b` uses
`dictionary_encode()`. The Parquet file keeps the dictionary type in
`ARROW:schema`:
```python
import os
import pyarrow as pa
import pyarrow.parquet as pq
from pyiceberg.catalog.sql import SqlCatalog
WAREHOUSE = "/tmp/iceberg-eqdelete"
os.makedirs(WAREHOUSE)
cat = SqlCatalog("default", uri=f"sqlite:///{WAREHOUSE}/catalog.db",
warehouse=f"file://{WAREHOUSE}")
cat.create_namespace("db")
b = pa.array([b"apple", b"bob", b"apple"]).dictionary_encode()
data = pa.table({"id": [1, 2, 3], "b": b})
tbl = cat.create_table("db.t", schema=data.schema)
tbl.append(data)
path = tbl.inspect.files()["file_path"][0].as_py().removeprefix("file://")
print("stored b:", pq.read_schema(path).field("b").type)
```
This program commits an equality delete for `b = "apple"` with a plain
binary key, then scans the table:
```go
package main
import (
"context"
"fmt"
"log"
"github.com/apache/arrow-go/v18/arrow"
"github.com/apache/arrow-go/v18/arrow/array"
"github.com/apache/arrow-go/v18/arrow/memory"
"github.com/apache/iceberg-go"
"github.com/apache/iceberg-go/catalog"
sqlcat "github.com/apache/iceberg-go/catalog/sql"
"github.com/uptrace/bun/driver/sqliteshim"
)
const warehouse = "/tmp/iceberg-eqdelete"
func main() {
ctx := context.Background()
cat, err := catalog.Load(ctx, "default", iceberg.Properties{
"uri": "file:" + warehouse + "/catalog.db",
"type": "sql",
sqlcat.DriverKey: sqliteshim.ShimName,
sqlcat.DialectKey: string(sqlcat.SQLite),
})
check(err)
tbl, err := cat.LoadTable(ctx, catalog.ToIdentifier("db", "t"))
check(err)
// Equality delete b = "apple", written by iceberg-go as a plain binary
column.
bb := array.NewBinaryBuilder(memory.DefaultAllocator,
arrow.BinaryTypes.Binary)
bb.Append([]byte("apple"))
sc := arrow.NewSchema([]arrow.Field{{Name: "b", Type:
arrow.BinaryTypes.Binary, Nullable: true}}, nil)
rec := array.NewRecordBatch(sc, []arrow.Array{bb.NewArray()}, 1)
field, _ := tbl.Schema().FindFieldByName("b")
tx := tbl.NewTransaction()
check(tx.CommitEqualityDeletes(ctx, []int{field.ID}, func(yield
func(arrow.RecordBatch, error) bool) {
yield(rec, nil)
}, nil))
tbl, err = tx.Commit(ctx)
check(err)
result, err := tbl.Scan().ToArrowTable(ctx)
check(err)
fmt.Println("expected: rows=1")
fmt.Printf("actual: rows=%d\n", result.NumRows())
}
func check(err error) {
if err != nil {
log.Fatal(err)
}
}
```
Save the script as `setup.py` and the program as `main.go` in an empty
directory, then run the commands below. Delete `/tmp/iceberg-eqdelete` before
running them again.
```sh
pip install "pyiceberg[sql-sqlite,pyarrow]==0.12.0" && python setup.py
go mod init repro && go get github.com/apache/iceberg-go@8ef0a092
go mod tidy && go run .
```
Setup output:
```text
stored b: dictionary<values=binary, indices=int32, ordered=0>
```
Go output at main and at v0.7.0:
```text
expected: rows=1
actual: rows=3
```
The delete should remove ids 1 and 3. It removes neither.
In a larger version of this program, the same binary delete removed both
rows from a table written from plain arrays. An equality delete on a
dictionary-encoded string column also worked on the dictionary table.
**Cause**
The data column reaches the delete filter as an `*array.Dictionary`. On
main, the single-column fast path `applySingleBytes` has no dictionary case, so
the reader builds a key for each row. `makeColEncoder` and `encodeArrowValue`
have no dictionary case either, so the key comes from `ValueStr`. For binary
values that is base64 text. The delete key holds the raw bytes, so equal values
never match. For a string dictionary, `ValueStr` returns the raw string, which
is why the string delete works.
The same logical value should produce the same key whether or not the column
is a dictionary.
**Related links**
-
https://github.com/apache/iceberg-go/blob/8ef0a092fcbaf70f99aa8935481bc95071eb41b1/table/equality_delete_reader.go#L1589-L1610
is the single-column fast path, which also has no dictionary case.
-
https://github.com/apache/iceberg-go/blob/8ef0a092fcbaf70f99aa8935481bc95071eb41b1/table/equality_delete_reader.go#L1294-L1304
encodes plain binary values as raw bytes.
-
https://github.com/apache/iceberg-go/blob/8ef0a092fcbaf70f99aa8935481bc95071eb41b1/table/equality_delete_reader.go#L1330-L1333
sends other array types, including dictionaries, to `encodeArrowValue`.
-
https://github.com/apache/iceberg-go/blob/8ef0a092fcbaf70f99aa8935481bc95071eb41b1/table/equality_delete_reader.go#L1155-L1159
encodes the fallback value through `ValueStr`.
-
https://github.com/apache/arrow-go/blob/72e9a5695337bab9ac01ec1e4d58177bc4949dc3/arrow/array/dictionary.go#L448-L453
delegates `Dictionary.ValueStr` to the underlying value.
-
https://github.com/apache/arrow-go/blob/72e9a5695337bab9ac01ec1e4d58177bc4949dc3/arrow/array/binary.go#L64-L70
returns base64 text for `Binary.ValueStr`.
**Not checked**
I did not test a dictionary-typed delete file against plain data, or whether
`WriteEqualityDeletes` decodes dictionary input. I also did not test a pandas
categorical of bytes, other writers, or remote catalogs and storage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
