moomindani commented on code in PR #9: URL: https://github.com/apache/iceberg-verification/pull/9#discussion_r4076799017
########## table-spec/types/README.md: ########## @@ -0,0 +1,116 @@ +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, + ~ software distributed under the License is distributed on an + ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ~ KIND, either express or implied. See the License for the + ~ specific language governing permissions and limitations + ~ under the License. + --> + +# Type decoding + +Parsing a type string produces the same type in every implementation. This +surface pins each type the spec defines and the parse rules that attach to it. + +## Assertion + +``` +parse(input) == decoded +``` + +`input` is a type string, or a JSON object for a nested type; `decoded` is the +language-neutral shape below. Bytes are not compared - each implementation maps +its own type object to `decoded`, so the comparison does not depend on one +language's representation. + +- `valid: true` - the parser succeeds and the decoded type equals `decoded`. A + type an implementation does not model is UNSUPPORTED, not a failure. If the case + also carries `canonical`, re-serializing the parsed type must equal it byte for + byte (the write direction). +- `valid: false` - the parser must reject `input`. A rejection passes; a + successful parse fails. + +`canonical` is present only where the spec pins one spelling. `decimal` has two +blessed forms (`decimal(9,2)` and `decimal(9, 2)`), so its cases have no Review Comment: The unspaced format column you are citing was fixed a few hours after this review: #18145 merged as `b5f363932` on the 21st, and the row now reads ``` |**`decimal(P, S)`**|`JSON string: "decimal(<P>, <S>)"`|`"decimal(9, 2)"`| ``` one template and one example, both spaced. So there is no Appendix C text blessing the two spellings as co-equal canonical forms: the spaced form is the canonical one, and the unspaced one is only something a reader must accept. `canonical: "decimal(9, 2)"` on both `decimal-9-2` and `decimal-9-2-spaced` is the accurate reading now, and the README's "two blessed forms" line goes with it. For context, that spec change came out of this thread — the decimal row had been left half-applied by #16798, whose sibling #16799 had already spaced the geography row. ########## table-spec/types/primitive/cases.json: ########## @@ -0,0 +1,33 @@ +{ + "cases": [ + { "id": "boolean", "valid": true, "input": "boolean", "decoded": { "type": "boolean" }, "canonical": "boolean", "clause": "Primitive Types: boolean; Appendix C canonical string", "spec_ref": "format/spec.md#appendix-c-json-serialization" }, + { "id": "int", "valid": true, "input": "int", "decoded": { "type": "int" }, "canonical": "int", "clause": "Primitive Types: int; Appendix C canonical string", "spec_ref": "format/spec.md#appendix-c-json-serialization" }, + { "id": "long", "valid": true, "input": "long", "decoded": { "type": "long" }, "canonical": "long", "clause": "Primitive Types: long; Appendix C canonical string", "spec_ref": "format/spec.md#appendix-c-json-serialization" }, + { "id": "float", "valid": true, "input": "float", "decoded": { "type": "float" }, "canonical": "float", "clause": "Primitive Types: float; Appendix C canonical string", "spec_ref": "format/spec.md#appendix-c-json-serialization" }, + { "id": "double", "valid": true, "input": "double", "decoded": { "type": "double" }, "canonical": "double", "clause": "Primitive Types: double; Appendix C canonical string", "spec_ref": "format/spec.md#appendix-c-json-serialization" }, + { "id": "date", "valid": true, "input": "date", "decoded": { "type": "date" }, "canonical": "date", "clause": "Primitive Types: date; Appendix C canonical string", "spec_ref": "format/spec.md#appendix-c-json-serialization" }, + { "id": "time", "valid": true, "input": "time", "decoded": { "type": "time" }, "canonical": "time", "clause": "Primitive Types: time; Appendix C canonical string", "spec_ref": "format/spec.md#appendix-c-json-serialization" }, + { "id": "timestamp", "valid": true, "input": "timestamp", "decoded": { "type": "timestamp" }, "canonical": "timestamp", "clause": "Primitive Types: timestamp; Appendix C canonical string", "spec_ref": "format/spec.md#appendix-c-json-serialization" }, + { "id": "timestamptz", "valid": true, "input": "timestamptz", "decoded": { "type": "timestamptz" }, "canonical": "timestamptz", "clause": "Primitive Types: timestamptz; Appendix C canonical string", "spec_ref": "format/spec.md#appendix-c-json-serialization" }, + { "id": "timestamp_ns", "valid": true, "input": "timestamp_ns", "decoded": { "type": "timestamp_ns" }, "canonical": "timestamp_ns", "clause": "Primitive Types: timestamp_ns added in v3; Appendix C canonical string", "spec_ref": "format/spec.md#primitive-types" }, + { "id": "timestamptz_ns", "valid": true, "input": "timestamptz_ns", "decoded": { "type": "timestamptz_ns" }, "canonical": "timestamptz_ns", "clause": "Primitive Types: timestamptz_ns added in v3; Appendix C canonical string", "spec_ref": "format/spec.md#primitive-types" }, + { "id": "string", "valid": true, "input": "string", "decoded": { "type": "string" }, "canonical": "string", "clause": "Primitive Types: string; Appendix C canonical string", "spec_ref": "format/spec.md#appendix-c-json-serialization" }, + { "id": "uuid", "valid": true, "input": "uuid", "decoded": { "type": "uuid" }, "canonical": "uuid", "clause": "Primitive Types: uuid; Appendix C canonical string", "spec_ref": "format/spec.md#appendix-c-json-serialization" }, + { "id": "binary", "valid": true, "input": "binary", "decoded": { "type": "binary" }, "canonical": "binary", "clause": "Primitive Types: binary; Appendix C canonical string", "spec_ref": "format/spec.md#appendix-c-json-serialization" }, + { "id": "unknown", "valid": true, "input": "unknown", "decoded": { "type": "unknown" }, "canonical": "unknown", "clause": "Primitive Types: unknown added in v3; Appendix C canonical string", "spec_ref": "format/spec.md#primitive-types" }, + { "id": "fixed-1", "valid": true, "input": "fixed[1]", "decoded": { "type": "fixed", "length": 1 }, "canonical": "fixed[1]", "clause": "Appendix C: fixed canonical string is fixed[<L>]", "spec_ref": "format/spec.md#appendix-c-json-serialization" }, + { "id": "fixed-16", "valid": true, "input": "fixed[16]", "decoded": { "type": "fixed", "length": 16 }, "canonical": "fixed[16]", "clause": "Appendix C: fixed canonical string is fixed[<L>]", "spec_ref": "format/spec.md#appendix-c-json-serialization" }, + { "id": "decimal-9-2", "valid": true, "input": "decimal(9,2)", "decoded": { "type": "decimal", "precision": 9, "scale": 2 }, "clause": "Appendix C: both decimal(9,2) and decimal(9, 2) are canonical, so no byte-exact form is pinned", "spec_ref": "format/spec.md#appendix-c-json-serialization" }, + { "id": "decimal-9-2-spaced", "valid": true, "input": "decimal(9, 2)", "decoded": { "type": "decimal", "precision": 9, "scale": 2 }, "clause": "Appendix C: the spaced decimal(9, 2) form parses to the same decimal", "spec_ref": "format/spec.md#appendix-c-json-serialization" }, Review Comment: On defining the states, I'd suggest four rather than three, based on running this surface and the schema surface in #3 against PyIceberg: - `pass` - `fail` — a `must` case that failed. The only state that makes a run nonzero. - `advisory_fail` — a `should` case that failed. Reported, never blocking. - `skip` — the case was not run: the surface is not subscribed, or the feature is absent. `skip` is the one I would not leave implicit. PyIceberg has no variant type at all, so both variant cases on the schema surface cannot run — and a runner that folds "not run" into "passed" reports green for a type the client does not implement, which is the opposite of what this corpus is for. A first-class `skip` also gives a client's own skip list something to name. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
