Copilot commented on code in PR #30: URL: https://github.com/apache/iceberg-verification/pull/30#discussion_r4623562104
########## reference-tables/README.md: ########## @@ -112,23 +112,55 @@ Consumers read it; the `description` field is documentation and is ignored. } ``` -A table is written once and not regenerated, so `written-by` is per table: the -corpus holds whatever version was current when each table was added. +`written-by` is per table: the corpus holds whatever version was current when +each table was written. + +A table may be regenerated to cover more of its Scope, or to correct it. + +An addition must keep these invariants: + +- The table's Scope must remain the same: an addition may only cover more of + what it states. +- `written-by` must remain the same: the same release must write the table. +- The Input's existing `commits` must remain the same, in the same order. Review Comment: Rows and column values live inside each Input commit, so keeping each commit unchanged forbids the additions permitted by the next bullet. Preserve existing operations, order, and data while explicitly allowing the row and field additions. ########## reference-tables/README.md: ########## @@ -112,23 +112,55 @@ Consumers read it; the `description` field is documentation and is ignored. } ``` -A table is written once and not regenerated, so `written-by` is per table: the -corpus holds whatever version was current when each table was added. +`written-by` is per table: the corpus holds whatever version was current when +each table was written. + +A table may be regenerated to cover more of its Scope, or to correct it. + +An addition must keep these invariants: + +- The table's Scope must remain the same: an addition may only cover more of + what it states. +- `written-by` must remain the same: the same release must write the table. +- The Input's existing `commits` must remain the same, in the same order. +- The Input may gain columns with new field IDs or rows, and must not lose any. Review Comment: This only forbids losing columns; it does not prevent changing an existing field's definition. For example, renaming `location` in the nested-types schema leaves its field ID, Input commits, expected rows, and stated Scope unchanged, so it qualifies as an addition even though the existing table schema has changed. Please require existing field properties and the partition spec to remain unchanged, allowing only new fields and rows. ########## reference-tables/README.md: ########## @@ -112,23 +112,55 @@ Consumers read it; the `description` field is documentation and is ignored. } ``` -A table is written once and not regenerated, so `written-by` is per table: the -corpus holds whatever version was current when each table was added. +`written-by` is per table: the corpus holds whatever version was current when +each table was written. + +A table may be regenerated to cover more of its Scope, or to correct it. + +An addition must keep these invariants: + +- The table's Scope must remain the same: an addition may only cover more of + what it states. +- `written-by` must remain the same: the same release must write the table. +- The Input's existing `commits` must remain the same, in the same order. +- The Input may gain columns with new field IDs or rows, and must not lose any. +- Restricted to the field IDs the table had before, every row of the previous + `expected/scan.json` must appear in the snapshot that the same commit + produces. Review Comment: Checking only whether each old row appears can hide lost duplicate occurrences and can let a positional deletion shift onto a new row, resurrecting an old one. Require at least the previous occurrence counts and preserve each original row's live/deleted status; appending new rows after existing rows in position-deleted files avoids shifting their positions. ########## reference-tables/v2/simple-append-parquet-data/README.md: ########## @@ -23,6 +23,20 @@ Input to `reference-tables/GenerateTables.java`. The schema and partition spec a the spec's JSON forms. Each commit appends one data file holding its rows, written in the spec's single-value JSON keyed by field ID, with `long` values as strings. +## Scope + +The baseline every reader should pass: two appends to an unpartitioned v2 table, +four rows, with a flat schema of `long`, `string` and `timestamptz`. + +The three types exist in every format version and cover the shapes most columns +take: a fixed-width integer, a variable-length UTF-8 value, and a logical type +carrying an annotation. Two appends give the table two snapshots and two +manifests, so a scan at each snapshot goes through the manifest list rather than +a single file. The one null, in `data` of the last row, checks that an optional +column reads back as null. Partitioning, nested and other primitive types, +deletes and schema changes are left to other tables, so that a failure here +points at basic scan planning. Review Comment: An immutable Scope of exactly four rows forbids the additive rows this PR is meant to allow. Describe required coverage rather than exact totals and positions, including the later wording about the sole null in the last row. The same restriction appears in `reference-tables/v2/nested-types-parquet-data/README.md:28` and, for the column count, `reference-tables/v3/initial-default-parquet-data/README.md:32`. ########## reference-tables/README.md: ########## @@ -112,23 +112,55 @@ Consumers read it; the `description` field is documentation and is ignored. } ``` -A table is written once and not regenerated, so `written-by` is per table: the -corpus holds whatever version was current when each table was added. +`written-by` is per table: the corpus holds whatever version was current when +each table was written. + +A table may be regenerated to cover more of its Scope, or to correct it. + +An addition must keep these invariants: + +- The table's Scope must remain the same: an addition may only cover more of + what it states. +- `written-by` must remain the same: the same release must write the table. +- The Input's existing `commits` must remain the same, in the same order. +- The Input may gain columns with new field IDs or rows, and must not lose any. +- Restricted to the field IDs the table had before, every row of the previous + `expected/scan.json` must appear in the snapshot that the same commit + produces. + +A correction must keep these invariants: + +- The table's Scope must remain the same. +- Every change must be to something wrong: the table does not match its Input, + or the Input does not match the spec. Review Comment: A valid Input can still produce spec-invalid metadata or incorrect expected scan results, but neither fits these two correction cases. Include errors in generated artifacts, expected values, and conformance to the table's Scope so those can be corrected without creating a new table. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
