dina-kar commented on issue #2879:
URL: https://github.com/apache/iceberg-rust/issues/2879#issuecomment-6321007284
I cross-checked the read side against Spark and Trino, and it looks complete
to me.
**Versions:** iceberg-rust `v0.11.0-rc.2` (8590485) and `main` @ 000e791,
against Spark 4.0.4 with Iceberg 1.12.0 and Trino 483. All tables were written
by Spark, through a Lakekeeper v0.13.6 REST catalog.
**Tables** (Spark SQL, all merge-on-read):
```sql
CREATE TABLE v3.lineage (id BIGINT, v STRING) USING iceberg
TBLPROPERTIES ('format-version'='3', 'write.update.mode'='merge-on-read',
'write.delete.mode'='merge-on-read');
INSERT INTO v3.lineage SELECT id, 'a' FROM range(0, 50);
INSERT INTO v3.lineage SELECT id, 'b' FROM range(50, 100);
UPDATE v3.lineage SET v = 'c' WHERE id % 10 = 3;
CREATE TABLE v3.dv (id BIGINT, v STRING) USING iceberg
TBLPROPERTIES ('format-version'='3', 'write.delete.mode'='merge-on-read',
'write.update.mode'='merge-on-read');
INSERT INTO v3.dv SELECT id, concat('v', id) FROM range(0, 1000);
DELETE FROM v3.dv WHERE id % 10 = 0;
DELETE FROM v3.dv WHERE id % 10 = 1;
UPDATE v3.dv SET v = 'u' WHERE id % 10 = 2;
```
I also used a partitioned table with a `DELETE` and a `MERGE` (8 DVs), and a
v2 table with position deletes that was upgraded to v3, appended to and deleted
from again.
**Read:**
```rust
table.scan()
.select(["id", "v", "_row_id", "_last_updated_sequence_number"])
.with_filter(Reference::new("id").greater_than_or_equal_to(Datum::long(500)))
.build()?.to_arrow().await?
```
I ran it with and without the filter. Spark and Trino ran the equivalent
`SELECT count(*), sum(_row_id), sum(_row_id * id),
sum(_last_updated_sequence_number) ... WHERE ...` (Trino uses `$row_id`).
| table, filter | rows | sum(_row_id) | sum(_row_id * id) |
sum(_last_updated_sequence_number) |
|---|---|---|---|---|
| lineage, id >= 40 | 60 | 4170 | 307810 | 117 |
| dv, id >= 500 | 400 | 300200 | 233632200 | 550 |
| partitioned with DVs, id >= 1000 | 1714 | 3106464 | 6837533572 | 3428 |
| partitioned with DVs, no filter | 2571 | 3855750 | 7230336108 | 4798 |
| upgraded v2 to v3, id >= 500 | 490 | 344690 | 268441140 | 670 |
| upgraded v2 to v3, no filter | 890 | 484890 | 311893340 | 1070 |
rc.2, `main`, Spark and Trino print identical values in all six cases.
Updated rows keep their original `_row_id` and carry the new sequence number.
One detail for consumers: `_last_updated_sequence_number` comes back as
`RunEndEncoded(Int32, Int64)`, not `Int64`. That seems fine, but engines that
expect a plain `Int64` column have to cast it.
The full probe and the Spark scripts are at
https://github.com/ostrium-labs/loams/tree/he0-iceberg-v3-spike/spikes/he0-iceberg-v3
(`reader-common/probe.rs`, `spark/`, `results/`).
Can this issue be closed? It looks implemented by #2922, #2966, #2985 and
#3058. If something is still missing, please let me know what it is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
