sungwy commented on code in PR #6: URL: https://github.com/apache/iceberg-verification/pull/6#discussion_r4341991369
########## reference-tables/README.md: ########## @@ -0,0 +1,142 @@ +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, + ~ software distributed under the License is distributed on an + ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ~ KIND, either express or implied. See the License for the + ~ specific language governing permissions and limitations + ~ under the License. + --> + +# Reference tables + +This directory hosts whole Iceberg tables reference fixtures. +Surfaces reference a table by path and hold their own expected +values, so one table can support several surfaces without being copied. + +Tables are grouped by format version. + +## Size + +A table should stay under 100 KB. The three here are 16 KB to 28 KB. + +Size tracks commit count, and almost nothing else. A commit costs roughly 12 to +14 KB, over half of it the manifest and manifest list it produces. Rows are close +to free: a Parquet file holding five rows is under 1 KB and is nearly all footer. +A deletion vector is 385 bytes. A v3 table runs slightly heavier per commit than +v1 or v2, because a v3 manifest entry carries the deletion vector fields whether +or not a vector exists. + +So the limit is about seven commits. A table that needs more should say why in +its pull request. + +## Paths + +Prior to v4 the spec requires every path field to be fully qualified, so each +table carries an absolute prefix in `location`, in every `manifest-list`, and in +every `file_path`. That prefix is the sentinel root: + +``` +file:///tmp/iceberg-verification/reference-tables/<format-version>/<table-name> +``` + +A consumer resolves it one of two ways. Symlinking `/tmp/iceberg-verification` at +the checkout leaves every committed byte untouched. Substituting the prefix while +loading works where symlinks do not, notably on Windows. Both are valid, because Review Comment: I decided to remove the prefix writing approach as it was error-prone, and kept just the symlink based approach for now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
