sungwy commented on code in PR #6: URL: https://github.com/apache/iceberg-verification/pull/6#discussion_r4559309586
########## reference-tables/README.md: ########## @@ -0,0 +1,186 @@ +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, + ~ software distributed under the License is distributed on an + ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ~ KIND, either express or implied. See the License for the + ~ specific language governing permissions and limitations + ~ under the License. + --> + +# Reference tables + +This directory holds complete Iceberg tables: metadata, manifest lists, +manifests, and Parquet data files. Tables are grouped by format version. + +## Assertion + +``` +scan(table, snapshot) == rows +``` + +For every snapshot a table's metadata lists, a full scan at that snapshot returns +exactly the rows in the table's `expected/scan.json`. Bytes are not compared. + +## Size + +A table should stay under 100 KB. The four here are 16 KB to 29 KB. + +Size tracks commit count, and almost nothing else. A commit costs roughly 12 to +14 KB, over half of it the manifest and manifest list it produces. Rows are close +to free: a Parquet file holding five rows is under 1 KB and is nearly all footer. + +So the limit is about seven commits. A table that needs more should say why in +its pull request. + +## Paths + +Prior to v4 the spec requires every path field to be fully qualified, so each +table carries an absolute prefix in `location`, in every `manifest-list`, and in +every `file_path`. That prefix is the sentinel root: + +``` +file:///tmp/iceberg-verification/reference-tables/<format-version>/<table-name> +``` + +A consumer resolves the prefix by symlinking `/tmp/iceberg-verification` to the +root of its checkout, which leaves every committed byte untouched: + +``` +ln -s "$(pwd)" /tmp/iceberg-verification +``` + +This route is verified on Linux. It is not yet verified on Windows. + +Starting with v4 the spec permits relative paths and makes `location` optional, +so a v4 table needs no symlink. + +Loading a table opens only its metadata file, so a load succeeds without the +symlink. The scan in the assertion opens every manifest list, manifest and data +file, so a passing scan also shows the symlink resolves. + +## What is here + +`index.json` lists every table: its name, format version, the implementation and +release that wrote it, the path to its metadata file, and a one-line description. +Consumers read it; the `description` field is documentation and is ignored. + +```json +{ + "tables": [ + { + "name": "v2/simple-append-parquet-data", + "format-version": 2, + "written-by": "Iceberg Java 1.11.0", + "data-file-format": "parquet", + "metadata": "v2/simple-append-parquet-data/metadata/v3.metadata.json", + "description": "Flat schema of long, string and timestamptz. Two appends, four rows, one null." + } + ] +} +``` + +A table is written once and not regenerated, so `written-by` is per table: the Review Comment: yes, I was working on this - just found the time to get in a commit here :) thanks for following up -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
