aokolnychyi commented on a change in pull request #1499:
URL: https://github.com/apache/iceberg/pull/1499#discussion_r495611020
##########
File path: site/docs/spec.md
##########
@@ -52,14 +52,37 @@ Data files in snapshots are tracked by one or more manifest
files that contain a
The manifests that make up a snapshot are stored in a manifest list file. Each
manifest list stores metadata about manifests, including partition stats and
data file counts. These stats are used to avoid reading manifests that are not
required for an operation.
-#### MVCC and Optimistic Concurrency
+#### Optimistic Concurrency
-An atomic swap of one table metadata file for another provides serializable
isolation. Readers use the snapshot that was current when they load the table
metadata and are not affected by changes until they refresh and pick up a new
metadata location.
+An atomic swap of one table metadata file for another provides the basis for
serializable isolation. Readers use the snapshot that was current when they
load the table metadata and are not affected by changes until they refresh and
pick up a new metadata location.
Writers create table metadata files optimistically, assuming that the current
version will not be changed before the writer's commit. Once a writer has
created an update, it commits by swapping the table’s metadata file pointer
from the base version to the new version.
If the snapshot on which an update is based is no longer current, the writer
must retry the update based on the new current version. Some operations support
retry by re-applying metadata changes and committing, under well-defined
conditions. For example, a change that rewrites files can be applied to a new
table snapshot if all of the rewritten files are still in the table.
+The conditions required by a write to successfully commit determines the
isolation level. Writers can select what to validate and can make different
isolation guarantees.
+
+#### Sequence Numbers
+
+The relative age of data and delete files relies on a sequence number that is
assigned to every successful commit. When a snapshot is created for a commit,
it is optimistically assigned the next sequence number, and it is written into
the snapshot's metadata. If the commit fails and must be retried, the sequence
number is reassigned and written into new snapshot metadata.
+
+All manifests, data files, and delete files created for a snapshot inherit the
snapshot's sequence number. Manifest file metadata in the manifest list stores
a manifest's sequence number. New data and metadata file entries are written
with `null` in place of a sequence number, which is replaced with the
manifest's sequence number at read time. When a data or delete file is written
to a new manifest (as "existing"), the inherited sequence number is written to
ensure it does not change after it is first inherited.
+
+Inheriting the sequence number from manifest metadata allows writing a new
manifest once and reusing it in commit retries. To change a sequence number for
a retry, only the manifest list must be rewritten -- which would be rewritten
anyway with the latest set of manifests.
+
+
+#### Row-level Deletes
+
+Row-level deletes are stored in delete files.
+
+There are two ways to encode a row-level delete:
+
+* [_Position deletes_](#position-delete-files) mark a row deleted by data file
path and the row position in the data file
+* [_Equality deletes_](#equality-delete-files) mark a row deleted by one or
more column values, like `id = 5`
+
+Like data files, delete files are tracked by partition. In general, a delete
file must be applied to older data files with the same partition; see [Scan
Planning](#scan-planning) for details. Column metrics can be used to determine
whether a delete file's rows overlap the contents of a data file or a scan
range.
Review comment:
We don't store partition values in positional deletes just yet, do we?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
