danhuawang opened a new issue, #10995:
URL: https://github.com/apache/gravitino/issues/10995
### Version
main branch
### Describe what's wrong
When using Spark to delete rows from an Iceberg format-v3 table with
`write.delete.format=deletion-vector` and `write.delete.mode=merge-on-read` via
Gravitino's Iceberg REST Catalog, the deletion vector file is never created.
Instead, Spark silently falls back to copy-on-write, rewriting/dropping the
original data file.
The table properties are correctly stored and returned by the server, but
the deletion vector behavior does not take effect.
### Error message and/or stacktrace
No exception is thrown. The issue is silent. After executing `DELETE`, the
following observations confirm the fallback:
- `SELECT * FROM <table>.delete_files` returns 0 rows — no deletion vector
file was written
- `SELECT file_path, content, record_count FROM <table>.files` shows the
data file containing the deleted row has been removed entirely (copy-on-write
behavior)
- Snapshot summary confirms: `"total-delete-files":"0"`,
`"total-position-deletes":"0"`, `"deleted-data-files":"1"`
The REST API response for `GET /iceberg/v1/namespaces/{ns}/tables/{table}`
is missing the `supported-capabilities` field:
```json
{
"metadata-location": "s3://...",
"metadata": {
"format-version": 3,
"properties": {
"write.delete.format": "deletion-vector",
"write.delete.mode": "merge-on-read"
}
}
// "supported-capabilities" is absent — Spark falls back to copy-on-write
}
### How to reproduce
Gravitino IRC server: main branch (Iceberg 1.10.1)
Spark: 4.1.1 with Iceberg 1.10.1
```
-- 1. Create a format-v3 table via Spark pointing to Gravitino IRC
CREATE TABLE aws.spark_db_v3.test_dv (id BIGINT, data STRING)
USING iceberg
TBLPROPERTIES ('format-version' = '3');
-- 2. Set deletion vector properties
ALTER TABLE aws.spark_db_v3.test_dv SET TBLPROPERTIES (
'write.delete.mode' = 'merge-on-read',
'write.delete.format' = 'deletion-vector'
);
-- 3. Insert rows and delete one
INSERT INTO aws.spark_db_v3.test_dv VALUES (1, 'a'), (2, 'b'), (3, 'c');
DELETE FROM aws.spark_db_v3.test_dv WHERE id = 1;
-- 4. Expected: a deletion vector file exists
SELECT * FROM aws.spark_db_v3.test_dv.delete_files;
-- Actual: 0 rows returned
-- 5. Actual: the data file was dropped (copy-on-write)
SELECT file_path, content, record_count FROM aws.spark_db_v3.test_dv.files;
-- Only 2 files remain; the file for id=1 was removed entirely
```
### Additional context
Analysis from AI(only for reference): IcebergCatalogWrapper.loadTable()
builds the response without supported-capabilities. Per the Iceberg REST
Catalog spec, the server must advertise "supported-capabilities":
["deletion-vector"] for format-v3 tables using deletion vectors. Without it,
Spark assumes the server does not support DVs and falls back to copy-on-write
regardless of table properties.
The fix requires injecting "supported-capabilities": ["deletion-vector"]
into the LoadTableResponse when the table has format-version=3 and
write.delete.format=deletion-vector, either via JSON post-processing in
IcebergTableOperations or a custom serializer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
