Copilot commented on code in PR #6: URL: https://github.com/apache/iceberg-verification/pull/6#discussion_r4342435615
########## dev/.rat-excludes: ########## @@ -26,3 +26,7 @@ node_modules/** # Apache header; they are covered by the repository LICENSE and NOTICE at the root. **/*.json **/*.jsonl + +# Avro container files embed their schema as JSON text in the file header, so RAT +# reads them as text. They are binary fixture data and cannot carry a header either. +**/*.avro Review Comment: Excluding `**/*.avro` globally is overly broad and could unintentionally exempt non-fixture Avro files elsewhere in the repo from license checks. Consider scoping this exclude to the fixture subtree instead (e.g., `reference-tables/**/*.avro`) to minimize the blast radius. ########## reference-tables/README.md: ########## @@ -0,0 +1,186 @@ +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, + ~ software distributed under the License is distributed on an + ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ~ KIND, either express or implied. See the License for the + ~ specific language governing permissions and limitations + ~ under the License. + --> + +# Reference tables + +This directory holds complete Iceberg tables: metadata, manifest lists, +manifests, and Parquet data files. Tables are grouped by format version. + +## Assertion + +``` +scan(table, snapshot) == rows +``` + +For every snapshot a table's metadata lists, a full scan at that snapshot returns +exactly the rows in the table's `expected/scan.json`. Bytes are not compared. + +## Size + +A table should stay under 100 KB. The four here are 16 KB to 29 KB. + +Size tracks commit count, and almost nothing else. A commit costs roughly 12 to +14 KB, over half of it the manifest and manifest list it produces. Rows are close +to free: a Parquet file holding five rows is under 1 KB and is nearly all footer. + +So the limit is about seven commits. A table that needs more should say why in +its pull request. + +## Paths + +Prior to v4 the spec requires every path field to be fully qualified, so each +table carries an absolute prefix in `location`, in every `manifest-list`, and in +every `file_path`. That prefix is the sentinel root: + +``` +file:///tmp/iceberg-verification/reference-tables/<format-version>/<table-name> +``` + +A consumer resolves the prefix by symlinking `/tmp/iceberg-verification` to the +root of its checkout, which leaves every committed byte untouched: + +``` +ln -s "$(pwd)" /tmp/iceberg-verification +``` + +This route is verified on Linux. It is not yet verified on Windows. + +Starting with v4 the spec permits relative paths and makes `location` optional, +so a v4 table needs no symlink. + +Loading a table opens only its metadata file, so a load succeeds without the +symlink. The scan in the assertion opens every manifest list, manifest and data +file, so a passing scan also shows the symlink resolves. + +## What is here + +`index.json` lists every table: its name, format version, the implementation and +release that wrote it, the path to its metadata file, and a one-line description. +Consumers read it; the `description` field is documentation and is ignored. + +```json +{ + "tables": [ + { + "name": "v2/simple-append-parquet-data", + "format-version": 2, + "written-by": "Iceberg Java 1.11.0", + "data-file-format": "parquet", + "metadata": "v2/simple-append-parquet-data/metadata/v3.metadata.json", + "description": "Flat schema of long, string and timestamptz. Two appends, four rows, one null." + } + ] +} +``` + +A table is written once and not regenerated, so `written-by` is per table: the +corpus holds whatever version was current when each table was added. + +## Naming + +A table directory names every choice an implementation has to support in order +to run it. Today that is the format version, which is the parent directory, and +the data file format, which is the suffix: + +``` +reference-tables/<format-version>/<what-it-covers>-<data-file-format>-data +``` + +So `v2/simple-append-parquet-data` is readable without opening anything: a v2 +table whose data files are Parquet. The suffix names the data files specifically, +because the manifests in the same directory are Avro and the metadata is JSON. An +implementation that reads no Parquet skips it by path, without parsing the index. +The same choices appear as fields in `index.json` for consumers that would rather +filter there. + +A later fixture that varies something else, delete file format for instance, +carries that in the name too. + +## Entry point + +Resolve a table's `metadata` path from `index.json` and open it as a static +table. Every implementation takes an explicit metadata file path: + +| | | +| -- | -- | +| Iceberg Java | `new BaseTable(new StaticTableOperations(path, io), name)` | +| PyIceberg | `StaticTable.from_metadata(path)` | +| iceberg-rust | `StaticTable::from_metadata_file(path, ident, io)` | +| iceberg-go | `table.NewFromLocation(..., path, ...)` | +| iceberg-cpp | `TableMetadataUtil::Read(io, path)`, then `StaticTable::Make(ident, metadata, path, io)` | + Review Comment: The markdown table syntax here is invalid (double-leading `||`), so it won’t render as a table in most markdown parsers. Replace it with a standard markdown table (single leading `|` per row) so the implementation entrypoints are readable. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
