karubian opened a new issue, #19055:
URL: https://github.com/apache/iceberg/issues/19055

   ### Apache Iceberg version
   
   1.12.0 (latest release)
   
   ### Query engine
   
   Spark
   
   ### Please describe the bug 🐞
   
   On a table partitioned by a binary or fixed column, Iceberg Java before 
1.11.0 can write a manifest list partition summary whose bounds leave out the 
manifest's own partition values, #15128. #15127 fixed new writes in 1.11.0, but 
a snapshot keeps a manifest and its summary until a commit rewrites that 
manifest, and current releases still skip manifests by those bounds. #15128 
shows a filtered read missing rows. On a delete manifest, a filtered scan plans 
the data file without its deletes, so deleted rows come back, and a filtered 
write can make that permanent.
   
   On Spark 3.5.3 with a Hadoop catalog, the setup on Iceberg 1.10.1, then the 
session restarted on 1.12.0 on the same warehouse:
   
   ```sql
   -- Iceberg 1.10.1
   CREATE TABLE t (id bigint NOT NULL, data binary) USING iceberg PARTITIONED 
BY (data)
     TBLPROPERTIES ('format-version' = '2', 'write.delete.mode' = 
'merge-on-read');
   INSERT INTO t VALUES (1, X'e3bcd1'), (2, X'e3bcd1');
   INSERT INTO t VALUES (3, X'bcd1'), (4, X'bcd1');
   DELETE FROM t WHERE id IN (1, 3);   -- one delete manifest, summary [bcd1, 
e3bcd1]
   DELETE FROM t WHERE data = X'bcd1'; -- a metadata delete rewrites it, 
summary [bcd1, bcd1d1]
   
   -- Iceberg 1.12.0, same warehouse
   SELECT id FROM t;                         -- 2
   SELECT id FROM t WHERE data = X'e3bcd1';  -- 1, 2
   UPDATE t SET id = id + 100 WHERE data = X'e3bcd1' AND id = 2;  -- 
copy-on-write
   SELECT id FROM t;                         -- 1, 102
   ```
   
   The update's scan did not apply the delete, so it copied row 1 into the new 
data file, and row 1 is back for every reader; the commit then drops the delete 
as older than every live data file. With the setup on 1.12.0 every statement 
gives the right rows; a build of main, 2bd60a0, gives the same results as 
1.12.0. Also on 1.12.0:
   
   - On this table, a copy-on-write `DELETE` or `MERGE` with a condition on 
`data` and `rewrite_data_files` with `where => "data = X'e3bcd1'"` and `options 
=> map('rewrite-all', 'true')` bring row 1 back too, and a merge-on-read 
`UPDATE` writes it back as row 101. An unfiltered `rewrite_data_files` with 
`rewrite-all` applies the delete.
   - On the same table in format version 3, where 1.10.1 wrote the wrong 
summary in about half the runs, by the order of the deletion vectors in the 
manifest, the filtered read returns row 1, the filtered `rewrite_data_files` 
and copy-on-write `UPDATE`, `DELETE` and `MERGE` bring it back, and a 
merge-on-read `UPDATE`, `DELETE` or `MERGE` adds a second deletion vector for 
the data file, after which unfiltered reads and `rewrite_data_files` fail with 
`Can't index multiple DVs`.
   - In the data manifest shape of #15128, a filtered `DELETE` or `UPDATE` 
succeeds and changes nothing, `MERGE ... WHEN NOT MATCHED THEN INSERT` adds a 
duplicate row, and `writeTo(t).overwrite(...)` or a dynamic `INSERT OVERWRITE` 
of the partition keeps the old rows.
   - PyIceberg 0.12.0 reads the v2 table and the data manifest table the same 
wrong way: the summaries are wrong for every reader that prunes by them.
   
   Which tables: every release tested from 1.4.3 to 1.10.2 wrote such summaries 
for binary partitions, and 1.9.2 to 1.10.2 for fixed ones; on earlier releases 
the test could not commit a fixed partition value. String and uuid partitions, 
tested as controls, kept right summaries. A recorded bound takes the bytes of 
the next entry read when that value is no longer, so whether a summary goes 
wrong depends on the order and lengths of the values. Any commit that copies 
entries into a new manifest can write one: a metadata delete, an overwrite, a 
commit that removes files, or a manifest merge, which appends and row deltas do 
once a table has enough manifests. The partition fields at risk are identity on 
a binary or fixed column and truncate on a binary column; bucket on a binary 
column has int bounds and is not affected.
   
   The summaries alone cannot tell a wrong bound from a right one; the entries 
can. This lists the live entries outside their manifest's bounds for a table 
whose first partition field is the binary one:
   
   ```sql
   SET spark.sql.optimizer.nestedSchemaPruning.enabled = false;  -- the 
manifests table cannot project one summary field
   SELECT m.content, base64(e.data_file.partition.data) AS value,
          m.partition_summaries[0].lower_bound AS lower, 
m.partition_summaries[0].upper_bound AS upper
   FROM (SELECT input_file_name() AS manifest, status, data_file FROM 
t.entries) e
   JOIN t.manifests m ON e.manifest = m.path
   WHERE e.status < 2
     AND (e.data_file.partition.data < 
unbase64(m.partition_summaries[0].lower_bound)
       OR e.data_file.partition.data > 
unbase64(m.partition_summaries[0].upper_bound));
   ```
   
   It flags the v2, v3 and data manifest tables written on 1.10.1, and none 
written on 1.12.0. It must not filter on the partition column, since metadata 
tables prune by the same summaries.
   
   Where Iceberg Java trusts the bounds, which [Manifest 
Lists](https://iceberg.apache.org/spec/#manifest-lists) requires to cover the 
manifest's values and [Scan 
Planning](https://iceberg.apache.org/spec/#scan-planning) lets readers skip by:
   
   - `ManifestEvaluator`, behind scan planning of data and delete manifests, 
[DeleteFileIndex](https://github.com/apache/iceberg/blob/2bd60a093a889a44a9e67033dba3abf4c36895f5/core/src/main/java/org/apache/iceberg/DeleteFileIndex.java#L650-L674)
 and 
[BaseDistributedDataScan](https://github.com/apache/iceberg/blob/2bd60a093a889a44a9e67033dba3abf4c36895f5/core/src/main/java/org/apache/iceberg/BaseDistributedDataScan.java#L207-L226),
 metadata tables including the `data_files` lookup of #19035, row filter 
deletes and conflict validation.
   - 
[ManifestFileUtil.canContainAny](https://github.com/apache/iceberg/blob/2bd60a093a889a44a9e67033dba3abf4c36895f5/core/src/main/java/org/apache/iceberg/util/ManifestFileUtil.java#L57-L91),
 behind partition drops and deletes of files that do not name their manifest, 
[ManifestFilterManager](https://github.com/apache/iceberg/blob/2bd60a093a889a44a9e67033dba3abf4c36895f5/core/src/main/java/org/apache/iceberg/ManifestFilterManager.java#L401-L445).
   
   Repair today, on 1.11.0 or later: `rewrite_manifests` writes right summaries 
for the manifests it rewrites, within the chosen spec. It skips a manifest that 
is the only one of its content when the target size asks for one manifest, 
[RewriteManifestsSparkAction](https://github.com/apache/iceberg/blob/2bd60a093a889a44a9e67033dba3abf4c36895f5/spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteManifestsSparkAction.java#L234-L237),
 as in this table. With writers paused, one run with 
`commit.manifest.target-size-bytes` set one byte below the smallest manifest's 
`length` rewrote the lone manifests of the v2 and data manifest tables with 
right summaries; then restore the property, since every commit reads it, and 
run `rewrite_manifests` again to merge what the first run split. The target 
applies to every content, so on a table whose manifests differ widely in size 
the first run can split the larger content into many manifests until the second 
run. The core `RewriteMani
 fests` keeps delete manifests in its clustering rewrite. Repairing summaries 
cannot undo an incorrect data rewrite or restore deletes removed by its commit, 
and a v3 table with two deletion vectors for one data file needs a rollback.
   
   I'd propose that `ManifestEvaluator` and `ManifestFileUtil.canContainAny` 
stop comparing values with the bounds of binary and fixed partition fields, and 
keep the checks that a field holds only nulls. Tables partitioned by those 
types lose manifest pruning on them for every manifest, healthy ones too, while 
entry-level filtering and other partition fields still prune; with that change 
every case above, on v2 and v3, and #19035 give the right rows, and #19036 is 
no longer needed. Engines that plan with Iceberg Java, such as Flink and Trino, 
and equality deletes go through the same code, though they were not run here; 
PyIceberg and other implementations need a repaired table or their own change. 
Doing this only where delete manifests are pruned would keep data manifest 
pruning but leave the data manifest cases. A `rewrite_manifests` option to 
rewrite every manifest would let users repair tables for every reader, and 
could follow. I can open a PR for the first.
   
   ### Willingness to contribute
   
   - [x] I can contribute a fix for this bug independently
   - [ ] I would be willing to contribute a fix for this bug with guidance from 
the Iceberg community
   - [ ] I cannot contribute a fix for this bug at this time
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to