aokolnychyi commented on a change in pull request #1499:
URL: https://github.com/apache/iceberg/pull/1499#discussion_r497178102
##########
File path: site/docs/spec.md
##########
@@ -416,25 +530,91 @@ Notes:
### Delete Formats
-This section details how to encode row-level deletes in Iceberg metadata.
Row-level deletes are not supported in the current format version 1. This part
of the spec is not yet complete and will be completed as format version 2.
+This section details how to encode row-level deletes in Iceberg delete files.
Row-level deletes are not supported in v1.
+
+Row-level delete files are valid Iceberg data files: files must use valid
Iceberg formats, schemas, and column projection. It is recommended that delete
files are written using the table's default file format.
+
+Row-level delete files are tracked by manifests, like data files. A separate
set of manifests is used for delete files, but the manifest schemas are
identical.
-#### Position-based Delete Files
+Both position and equality deletes allow encoding deleted row values with a
delete. This can be used to reconstruct a stream of changes to a table.
-Position-based delete files identify rows in one or more data files that have
been deleted.
+
+#### Position Delete Files
+
+Position-based delete files identify deleted rows by file and position in one
or more data files, and may optionally contain the deleted row.
+
+A data row is deleted if there is an entry in a position delete file for the
row's file and position in the data file, starting at 0.
Position-based delete files store `file_position_delete`, a struct with the
following fields:
-| Field id, name | Type | Description
|
-|-------------------------|---------------------------------|--------------------------------------------------------------------------------------------------------------------------|
-| **`1 file_path`** | `required string` | The full URI of a
data file with FS scheme. This must match the `file_path` of the target data
file in a manifest entry. |
-| **`2 position`** | `required long` | The ordinal
position of a deleted row in the target data file identified by `file_path`,
starting at `0`. |
+| Field id, name | Type | Description |
+|-----------------------------|------------------------|-------------|
+| **`2147483546 file_path`** | `string` | Full URI or a data
file with FS scheme. This must match the `file_path` of the target data file in
a manifest entry |
+| **`2147483545 pos`** | `long` | Ordinal position of a
deleted row in the target data file identified by `file_path`, starting at `0` |
+| **`2147483544 row`** | `required struct<...>` | Deleted row values.
Omit the column when not storing deleted rows. |
+
+When the deleted row is present, its schema may be any subset of the table
schema. When the column is present, the deleted row values for every delete
must be stored.
The rows in the delete file must be sorted by `file_path` then `position` to
optimize filtering rows while scanning.
* Sorting by `file_path` allows filter pushdown by file in columnar storage
formats.
* Sorting by `position` allows filtering rows while scanning, to avoid
keeping deletes in memory.
-Though the delete files can be written using any supported data file format in
Iceberg, it is recommended to write delete files with same file format as the
table's file format.
+#### Equality Delete Files
+
+Equality delete files identify deleted rows in a collection of data files by
one or more column values, and may optionally contain additional columns of the
deleted row.
Review comment:
I think Flink guys raised a point about being able to reconstruct a CDC
pipeline from Iceberg table. For example, if someone merges a binlog exported
from MySQL into an Iceberg tables, there may be a need to reconstruct this
binlog from the Iceberg table at a later point in time.
That being said, I think it should be disabled by default as the use case is
very limited and we don't want to increase the size of delete files for no good
reason.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
