nssalian opened a new pull request, #28:
URL: https://github.com/apache/iceberg-verification/pull/28

   Closes https://github.com/apache/iceberg-verification/issues/19
   
   The reference-tables have no table whose schema changes, so nothing 
currently checks that a reader fills a column added after data was written from 
its `initial-default`. This adds `v3/initial-default-parquet-data`.
   
   Snapshot 1 appends `a.parquet` with `id` and `data`. Then `quantity int` is 
added as required with `initial-default` 42, `note string` is added as optional 
with no default, and `quantity`'s `write-default` is changed to 10. Snapshot 2 
appends `b.parquet` with all four columns. A scan at snapshot 2 returns the old 
rows with `quantity` 42 and `note` null, per the spec: "`initial-default` is 
used to populate the field's value for all records that were written before the 
field was added to the schema". The two defaults differ, so a reader that uses 
`write-default` for old rows fails.
   
   The table is generated by `GenerateTables.java` from the Input block in its 
README, which gains a `schema-updates` commit. The existing tables generate 
unchanged.
   
   Snapshot 1's expected rows assume a scan at an earlier snapshot uses that 
snapshot's schema, which is what every reader below does. This PR states that 
rule in the README's expected values and Procedure.
   
   Checked at both snapshots with a raw fastavro/pyarrow reader, PyIceberg 
0.12.0, Iceberg Java 1.12.0, iceberg-go (d66ab48) and iceberg-rust 0.10.1 and 
0.11.0. All match `expected/scan.json`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to