karubian opened a new issue, #19055:
URL: https://github.com/apache/iceberg/issues/19055
### Apache Iceberg version
1.12.0 (latest release)
### Query engine
Spark
### Please describe the bug 🐞
On a table partitioned by a binary or fixed column, Iceberg Java before
1.11.0 can write a manifest list partition summary whose bounds leave out the
manifest's own partition values, #15128. #15127 fixed new writes in 1.11.0, but
a snapshot keeps a manifest and its summary until a commit rewrites that
manifest, and current releases still skip manifests by those bounds. #15128
shows a filtered read missing rows. On a delete manifest, a filtered scan plans
the data file without its deletes, so deleted rows come back, and a filtered
write can make that permanent.
On Spark 3.5.3 with a Hadoop catalog, the setup on Iceberg 1.10.1, then the
session restarted on 1.12.0 on the same warehouse:
```sql
-- Iceberg 1.10.1
CREATE TABLE t (id bigint NOT NULL, data binary) USING iceberg PARTITIONED
BY (data)
TBLPROPERTIES ('format-version' = '2', 'write.delete.mode' =
'merge-on-read');
INSERT INTO t VALUES (1, X'e3bcd1'), (2, X'e3bcd1');
INSERT INTO t VALUES (3, X'bcd1'), (4, X'bcd1');
DELETE FROM t WHERE id IN (1, 3); -- one delete manifest, summary [bcd1,
e3bcd1]
DELETE FROM t WHERE data = X'bcd1'; -- a metadata delete rewrites it,
summary [bcd1, bcd1d1]
-- Iceberg 1.12.0, same warehouse
SELECT id FROM t; -- 2
SELECT id FROM t WHERE data = X'e3bcd1'; -- 1, 2
UPDATE t SET id = id + 100 WHERE data = X'e3bcd1' AND id = 2; --
copy-on-write
SELECT id FROM t; -- 1, 102
```
The update's scan did not apply the delete, so it copied row 1 into the new
data file, and row 1 is back for every reader; the commit then drops the delete
as older than every live data file. With the setup on 1.12.0 every statement
gives the right rows; a build of main, 2bd60a0, gives the same results as
1.12.0. Also on 1.12.0:
- On this table, a copy-on-write `DELETE` or `MERGE` with a condition on
`data` and `rewrite_data_files` with `where => "data = X'e3bcd1'"` and `options
=> map('rewrite-all', 'true')` bring row 1 back too, and a merge-on-read
`UPDATE` writes it back as row 101. An unfiltered `rewrite_data_files` with
`rewrite-all` applies the delete.
- On the same table in format version 3, where 1.10.1 wrote the wrong
summary in about half the runs, by the order of the deletion vectors in the
manifest, the filtered read returns row 1, the filtered `rewrite_data_files`
and copy-on-write `UPDATE`, `DELETE` and `MERGE` bring it back, and a
merge-on-read `UPDATE`, `DELETE` or `MERGE` adds a second deletion vector for
the data file, after which unfiltered reads and `rewrite_data_files` fail with
`Can't index multiple DVs`.
- In the data manifest shape of #15128, a filtered `DELETE` or `UPDATE`
succeeds and changes nothing, `MERGE ... WHEN NOT MATCHED THEN INSERT` adds a
duplicate row, and `writeTo(t).overwrite(...)` or a dynamic `INSERT OVERWRITE`
of the partition keeps the old rows.
- PyIceberg 0.12.0 reads the v2 table and the data manifest table the same
wrong way: the summaries are wrong for every reader that prunes by them.
Which tables: every release tested from 1.4.3 to 1.10.2 wrote such summaries
for binary partitions, and 1.9.2 to 1.10.2 for fixed ones; on earlier releases
the test could not commit a fixed partition value. String and uuid partitions,
tested as controls, kept right summaries. A recorded bound takes the bytes of
the next entry read when that value is no longer, so whether a summary goes
wrong depends on the order and lengths of the values. Any commit that copies
entries into a new manifest can write one: a metadata delete, an overwrite, a
commit that removes files, or a manifest merge, which appends and row deltas do
once a table has enough manifests. The partition fields at risk are identity on
a binary or fixed column and truncate on a binary column; bucket on a binary
column has int bounds and is not affected.
The summaries alone cannot tell a wrong bound from a right one; the entries
can. This lists the live entries outside their manifest's bounds for a table
whose first partition field is the binary one:
```sql
SET spark.sql.optimizer.nestedSchemaPruning.enabled = false; -- the
manifests table cannot project one summary field
SELECT m.content, base64(e.data_file.partition.data) AS value,
m.partition_summaries[0].lower_bound AS lower,
m.partition_summaries[0].upper_bound AS upper
FROM (SELECT input_file_name() AS manifest, status, data_file FROM
t.entries) e
JOIN t.manifests m ON e.manifest = m.path
WHERE e.status < 2
AND (e.data_file.partition.data <
unbase64(m.partition_summaries[0].lower_bound)
OR e.data_file.partition.data >
unbase64(m.partition_summaries[0].upper_bound));
```
It flags the v2, v3 and data manifest tables written on 1.10.1, and none
written on 1.12.0. It must not filter on the partition column, since metadata
tables prune by the same summaries.
Where Iceberg Java trusts the bounds, which [Manifest
Lists](https://iceberg.apache.org/spec/#manifest-lists) requires to cover the
manifest's values and [Scan
Planning](https://iceberg.apache.org/spec/#scan-planning) lets readers skip by:
- `ManifestEvaluator`, behind scan planning of data and delete manifests,
[DeleteFileIndex](https://github.com/apache/iceberg/blob/2bd60a093a889a44a9e67033dba3abf4c36895f5/core/src/main/java/org/apache/iceberg/DeleteFileIndex.java#L650-L674)
and
[BaseDistributedDataScan](https://github.com/apache/iceberg/blob/2bd60a093a889a44a9e67033dba3abf4c36895f5/core/src/main/java/org/apache/iceberg/BaseDistributedDataScan.java#L207-L226),
metadata tables including the `data_files` lookup of #19035, row filter
deletes and conflict validation.
-
[ManifestFileUtil.canContainAny](https://github.com/apache/iceberg/blob/2bd60a093a889a44a9e67033dba3abf4c36895f5/core/src/main/java/org/apache/iceberg/util/ManifestFileUtil.java#L57-L91),
behind partition drops and deletes of files that do not name their manifest,
[ManifestFilterManager](https://github.com/apache/iceberg/blob/2bd60a093a889a44a9e67033dba3abf4c36895f5/core/src/main/java/org/apache/iceberg/ManifestFilterManager.java#L401-L445).
Repair today, on 1.11.0 or later: `rewrite_manifests` writes right summaries
for the manifests it rewrites, within the chosen spec. It skips a manifest that
is the only one of its content when the target size asks for one manifest,
[RewriteManifestsSparkAction](https://github.com/apache/iceberg/blob/2bd60a093a889a44a9e67033dba3abf4c36895f5/spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteManifestsSparkAction.java#L234-L237),
as in this table. With writers paused, one run with
`commit.manifest.target-size-bytes` set one byte below the smallest manifest's
`length` rewrote the lone manifests of the v2 and data manifest tables with
right summaries; then restore the property, since every commit reads it, and
run `rewrite_manifests` again to merge what the first run split. The target
applies to every content, so on a table whose manifests differ widely in size
the first run can split the larger content into many manifests until the second
run. The core `RewriteMani
fests` keeps delete manifests in its clustering rewrite. Repairing summaries
cannot undo an incorrect data rewrite or restore deletes removed by its commit,
and a v3 table with two deletion vectors for one data file needs a rollback.
I'd propose that `ManifestEvaluator` and `ManifestFileUtil.canContainAny`
stop comparing values with the bounds of binary and fixed partition fields, and
keep the checks that a field holds only nulls. Tables partitioned by those
types lose manifest pruning on them for every manifest, healthy ones too, while
entry-level filtering and other partition fields still prune; with that change
every case above, on v2 and v3, and #19035 give the right rows, and #19036 is
no longer needed. Engines that plan with Iceberg Java, such as Flink and Trino,
and equality deletes go through the same code, though they were not run here;
PyIceberg and other implementations need a repaired table or their own change.
Doing this only where delete manifests are pruned would keep data manifest
pruning but leave the data manifest cases. A `rewrite_manifests` option to
rewrite every manifest would let users repair tables for every reader, and
could follow. I can open a PR for the first.
### Willingness to contribute
- [x] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]