laskoviymishka commented on code in PR #3: URL: https://github.com/apache/iceberg-verification/pull/3#discussion_r4024716876
########## table-spec/schema/README.md: ########## @@ -0,0 +1,141 @@ +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, + ~ software distributed under the License is distributed on an + ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ~ KIND, either express or implied. See the License for the + ~ specific language governing permissions and limitations + ~ under the License. + --> + +# Schema decoding + +Reading a schema JSON should produce the same schema in every implementation. +The assertions in this surface are intentionally scoped narrowly to catch +serialization issues of type strings (like decimal(P, S)), and allowed fields +within the Iceberg schema and its struct fields. + +## Assertion + +``` +project(parse(input)) == decoded +``` + +Bytes are not compared. The spec fixes no key order in a schema JSON, and does +not say whether an absent optional field is written out, so two byte-different +schema JSONs can be the same schema. `decoded` is the comparable form. + +## Scope + +This surface reads the schema JSON object and nothing around it. Writing a schema +back out is a later phase, per `CONTRIBUTING.md`. Whether a type is legal at a given Review Comment: This links to `CONTRIBUTING.md`, but that file doesn't exist yet — the root README still marks it a TODO, so implementors following the reference hit a 404. I'd drop the link and just say the writer phase is out of scope for this surface. ########## table-spec/schema/README.md: ########## @@ -0,0 +1,141 @@ +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, + ~ software distributed under the License is distributed on an + ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ~ KIND, either express or implied. See the License for the + ~ specific language governing permissions and limitations + ~ under the License. + --> + +# Schema decoding + +Reading a schema JSON should produce the same schema in every implementation. +The assertions in this surface are intentionally scoped narrowly to catch +serialization issues of type strings (like decimal(P, S)), and allowed fields Review Comment: The README names type-string serialization (`decimal(P, S)`) as the primary thing this surface catches, but every `valid: false` case in core/ is an identifier-field violation — nothing exercises a malformed type string. A parser that happily accepts `decimal(abc)`, `decimal(39,2)`, or `fixed[-1]` passes the whole surface. I'd add at least one malformed-decimal and one malformed-fixed reject case so the stated scope actually has teeth. ########## table-spec/schema/unknown/cases.json: ########## @@ -0,0 +1,34 @@ +{ + "cases": [ + { + "id": "unknown-optional", + "valid": true, + "input": { + "type": "struct", + "schema-id": 0, + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "future", + "required": false, Review Comment: This is the only case in the subdirectory and it's the happy path — there's no `valid: false` twin for a required `unknown` field. The spec is explicit that `unknown` must always be optional, so an implementation that accepts `required: true` here passes everything in this surface. I'd add a `reject-unknown-required` case with `{ id, name, required: true, type: "unknown" }`. ########## table-spec/schema/core/cases.json: ########## @@ -0,0 +1,693 @@ +{ + "cases": [ + { + "id": "flat-required-and-optional", + "valid": true, + "input": { + "type": "struct", + "schema-id": 0, + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "data", + "required": false, + "type": "string" + } + ] + }, + "decoded": { + "schema-id": 0, + "identifier-field-ids": [], + "fields": [ + { "id": 1, "name": "id", "parent": null, "required": true, "type": "long", "doc": null }, + { "id": 2, "name": "data", "parent": null, "required": false, "type": "string", "doc": null } + ] + } + }, + { + "id": "fields-not-in-id-order", + "valid": true, + "input": { + "type": "struct", + "schema-id": 0, + "fields": [ + { + "id": 3, + "name": "c", + "required": true, + "type": "int" + }, + { + "id": 1, + "name": "a", + "required": true, + "type": "int" + }, + { + "id": 2, + "name": "b", + "required": false, + "type": "int" + } + ] + }, + "decoded": { + "schema-id": 0, + "identifier-field-ids": [], + "fields": [ + { "id": 3, "name": "c", "parent": null, "required": true, "type": "int", "doc": null }, + { "id": 1, "name": "a", "parent": null, "required": true, "type": "int", "doc": null }, + { "id": 2, "name": "b", "parent": null, "required": false, "type": "int", "doc": null } + ] + } + }, + { + "id": "nested-struct", + "valid": true, + "input": { + "type": "struct", + "schema-id": 0, + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "location", + "required": false, + "type": { + "type": "struct", + "fields": [ + { + "id": 3, + "name": "lat", + "required": true, + "type": "double" + }, + { + "id": 4, + "name": "lon", + "required": true, + "type": "double" + } + ] + } + } + ] + }, + "decoded": { + "schema-id": 0, + "identifier-field-ids": [], + "fields": [ + { "id": 1, "name": "id", "parent": null, "required": true, "type": "long", "doc": null }, + { "id": 2, "name": "location", "parent": null, "required": false, "type": "struct", "doc": null }, + { "id": 3, "name": "lat", "parent": 2, "required": true, "type": "double", "doc": null }, + { "id": 4, "name": "lon", "parent": 2, "required": true, "type": "double", "doc": null } + ] + } + }, + { + "id": "list-of-primitive", + "valid": true, + "input": { + "type": "struct", + "schema-id": 0, + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "tags", + "required": true, + "type": { + "type": "list", + "element-id": 3, + "element-required": false, + "element": "string" + } + } + ] + }, + "decoded": { + "schema-id": 0, + "identifier-field-ids": [], + "fields": [ + { "id": 1, "name": "id", "parent": null, "required": true, "type": "long", "doc": null }, + { "id": 2, "name": "tags", "parent": null, "required": true, "type": "list", "doc": null }, + { "id": 3, "name": "element", "parent": 2, "required": false, "type": "string", "doc": null } + ] + } + }, + { + "id": "map-of-primitive", + "valid": true, + "input": { + "type": "struct", + "schema-id": 0, + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "props", + "required": true, + "type": { + "type": "map", + "key-id": 3, + "key": "string", + "value-id": 4, + "value-required": false, + "value": "int" + } + } + ] + }, + "decoded": { + "schema-id": 0, + "identifier-field-ids": [], + "fields": [ + { "id": 1, "name": "id", "parent": null, "required": true, "type": "long", "doc": null }, + { "id": 2, "name": "props", "parent": null, "required": true, "type": "map", "doc": null }, + { "id": 3, "name": "key", "parent": 2, "required": true, "type": "string", "doc": null }, + { "id": 4, "name": "value", "parent": 2, "required": false, "type": "int", "doc": null } + ] + } + }, + { + "id": "list-of-struct", + "valid": true, + "input": { + "type": "struct", + "schema-id": 0, + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "points", + "required": true, + "type": { + "type": "list", + "element-id": 3, + "element-required": true, + "element": { + "type": "struct", + "fields": [ + { + "id": 4, + "name": "x", + "required": true, + "type": "int" + }, + { + "id": 5, + "name": "y", + "required": false, + "type": "int" + } + ] + } + } + } + ] + }, + "decoded": { + "schema-id": 0, + "identifier-field-ids": [], + "fields": [ + { "id": 1, "name": "id", "parent": null, "required": true, "type": "long", "doc": null }, + { "id": 2, "name": "points", "parent": null, "required": true, "type": "list", "doc": null }, + { "id": 3, "name": "element", "parent": 2, "required": true, "type": "struct", "doc": null }, + { "id": 4, "name": "x", "parent": 3, "required": true, "type": "int", "doc": null }, + { "id": 5, "name": "y", "parent": 3, "required": false, "type": "int", "doc": null } + ] + } + }, + { + "id": "map-with-struct-value", + "valid": true, + "input": { + "type": "struct", + "schema-id": 0, + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "by_name", + "required": false, + "type": { + "type": "map", + "key-id": 3, + "key": "string", + "value-id": 4, + "value-required": false, + "value": { + "type": "struct", + "fields": [ + { + "id": 5, + "name": "count", + "required": true, + "type": "long" + } + ] + } + } + } + ] + }, + "decoded": { + "schema-id": 0, + "identifier-field-ids": [], + "fields": [ + { "id": 1, "name": "id", "parent": null, "required": true, "type": "long", "doc": null }, + { "id": 2, "name": "by_name", "parent": null, "required": false, "type": "map", "doc": null }, + { "id": 3, "name": "key", "parent": 2, "required": true, "type": "string", "doc": null }, + { "id": 4, "name": "value", "parent": 2, "required": false, "type": "struct", "doc": null }, + { "id": 5, "name": "count", "parent": 4, "required": true, "type": "long", "doc": null } + ] + } + }, + { + "id": "schema-id-and-identifier-fields", + "valid": true, + "input": { + "type": "struct", + "schema-id": 7, + "identifier-field-ids": [ + 1, + 2 + ], + "fields": [ + { + "id": 1, + "name": "tenant", + "required": true, + "type": "string" + }, + { + "id": 2, + "name": "key", + "required": true, + "type": "long" + }, + { + "id": 3, + "name": "payload", + "required": false, + "type": "string" + } + ] + }, + "decoded": { + "schema-id": 7, + "identifier-field-ids": [1, 2], + "fields": [ + { "id": 1, "name": "tenant", "parent": null, "required": true, "type": "string", "doc": null }, + { "id": 2, "name": "key", "parent": null, "required": true, "type": "long", "doc": null }, + { "id": 3, "name": "payload", "parent": null, "required": false, "type": "string", "doc": null } + ] + } + }, + { + "id": "field-doc", + "valid": true, + "input": { + "type": "struct", + "schema-id": 0, + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long", + "doc": "primary key" + }, + { + "id": 2, + "name": "data", + "required": false, + "type": "string" + } + ] + }, + "decoded": { + "schema-id": 0, + "identifier-field-ids": [], + "fields": [ + { "id": 1, "name": "id", "parent": null, "required": true, "type": "long", "doc": "primary key" }, + { "id": 2, "name": "data", "parent": null, "required": false, "type": "string", "doc": null } + ] + } + }, + { + "id": "all-primitive-types", + "valid": true, + "input": { + "type": "struct", + "schema-id": 0, + "fields": [ + { + "id": 1, + "name": "f_boolean", + "required": false, + "type": "boolean" + }, + { + "id": 2, + "name": "f_int", + "required": false, + "type": "int" + }, + { + "id": 3, + "name": "f_long", + "required": false, + "type": "long" + }, + { + "id": 4, + "name": "f_float", + "required": false, + "type": "float" + }, + { + "id": 5, + "name": "f_double", + "required": false, + "type": "double" + }, + { + "id": 6, + "name": "f_date", + "required": false, + "type": "date" + }, + { + "id": 7, + "name": "f_time", + "required": false, + "type": "time" + }, + { + "id": 8, + "name": "f_timestamp", + "required": false, + "type": "timestamp" + }, + { + "id": 9, + "name": "f_timestamptz", + "required": false, + "type": "timestamptz" + }, + { + "id": 10, + "name": "f_string", + "required": false, + "type": "string" + }, + { + "id": 11, + "name": "f_uuid", + "required": false, + "type": "uuid" + }, + { + "id": 12, + "name": "f_fixed", + "required": false, + "type": "fixed[16]" + }, + { + "id": 13, + "name": "f_binary", + "required": false, + "type": "binary" + }, + { + "id": 14, + "name": "f_decimal", + "required": false, + "type": "decimal(9,2)" + } + ] + }, + "decoded": { + "schema-id": 0, + "identifier-field-ids": [], + "fields": [ + { "id": 1, "name": "f_boolean", "parent": null, "required": false, "type": "boolean", "doc": null }, + { "id": 2, "name": "f_int", "parent": null, "required": false, "type": "int", "doc": null }, + { "id": 3, "name": "f_long", "parent": null, "required": false, "type": "long", "doc": null }, + { "id": 4, "name": "f_float", "parent": null, "required": false, "type": "float", "doc": null }, + { "id": 5, "name": "f_double", "parent": null, "required": false, "type": "double", "doc": null }, + { "id": 6, "name": "f_date", "parent": null, "required": false, "type": "date", "doc": null }, + { "id": 7, "name": "f_time", "parent": null, "required": false, "type": "time", "doc": null }, + { "id": 8, "name": "f_timestamp", "parent": null, "required": false, "type": "timestamp", "doc": null }, + { "id": 9, "name": "f_timestamptz", "parent": null, "required": false, "type": "timestamptz", "doc": null }, + { "id": 10, "name": "f_string", "parent": null, "required": false, "type": "string", "doc": null }, + { "id": 11, "name": "f_uuid", "parent": null, "required": false, "type": "uuid", "doc": null }, + { "id": 12, "name": "f_fixed", "parent": null, "required": false, "type": { "type": "fixed", "length": 16 }, "doc": null }, + { "id": 13, "name": "f_binary", "parent": null, "required": false, "type": "binary", "doc": null }, + { "id": 14, "name": "f_decimal", "parent": null, "required": false, "type": { "type": "decimal", "precision": 9, "scale": 2 }, "doc": null } + ] + } + }, + { + "id": "decimal-max-precision", + "valid": true, + "input": { + "type": "struct", + "schema-id": 0, + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "amount", + "required": false, + "type": "decimal(38, 10)" Review Comment: `decimal-max-precision` uses `decimal(38, 10)` (with a space) while `all-primitive-types` uses `decimal(9,2)` (no space), both `valid: true`. That quietly makes whitespace tolerance a conformance property the README never mentions — a strict parser fails one of these for a reason nobody documented. Copilot flagged this too. Either normalize both to the canonical `decimal(P, S)` form, or keep the split but pull it into a dedicated, named `decimal-whitespace` case and say so in the README. How strict do we want the surface to be on type-string whitespace — wdyt? ########## table-spec/schema/core/cases.json: ########## @@ -0,0 +1,693 @@ +{ + "cases": [ + { + "id": "flat-required-and-optional", + "valid": true, + "input": { + "type": "struct", + "schema-id": 0, Review Comment: The README lists `schema-id` as something core/ covers, but every valid case carries an explicit `schema-id` and the spec makes it optional (defaults to 0). An implementation that mishandles the absent case never gets caught here. I'd add one case that omits `schema-id` and decodes it to 0. ########## table-spec/schema/geography/cases.json: ########## @@ -0,0 +1,64 @@ +{ + "cases": [ + { + "id": "geography-defaults", + "valid": true, + "input": { + "type": "struct", + "schema-id": 0, + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "geog", + "required": false, + "type": "geography" + } + ] + }, + "decoded": { + "schema-id": 0, + "identifier-field-ids": [], + "fields": [ + { "id": 1, "name": "id", "parent": null, "required": true, "type": "long", "doc": null }, + { "id": 2, "name": "geog", "parent": null, "required": false, "type": { "type": "geography", "crs": "OGC:CRS84", "algorithm": "spherical" }, "doc": null } + ] + } + }, + { + "id": "geography-explicit-algorithm", + "valid": true, + "input": { + "type": "struct", + "schema-id": 0, + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "geog", + "required": false, + "type": "geography(srid:4326, vincenty)" Review Comment: geography has three forms — bare, `geography(C)`, and `geography(C, A)` — and we only test the first and third. The single-argument form (explicit CRS, default algorithm) is exactly where a broken parser drops or mis-defaults the algorithm. I'd add a `geography(srid:4326)` case decoding to `{ crs: srid:4326, algorithm: spherical }`. ########## table-spec/schema/timestamp-ns/cases.json: ########## @@ -0,0 +1,41 @@ +{ + "cases": [ + { + "id": "timestamp-ns-both-zones", + "valid": true, + "input": { + "type": "struct", + "schema-id": 0, + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "event_ts", + "required": false, + "type": "timestamp_ns" Review Comment: Two things here clash with the README. The directory is `timestamp-ns` (hyphen) but the v3 type is `timestamp_ns` (underscore), and the README says each subdirectory is "named for" its type — so the rule points at `timestamp_ns/`. This case also covers two types (`timestamp_ns` and `timestamptz_ns`), so "covers one v3 type" isn't quite accurate, and an implementor opting out of one silently loses the other. I'd either rename to `timestamp_ns/` and split the two types, or loosen the README rule to allow a named group. Copilot raised the naming mismatch too. ########## table-spec/schema/core/cases.json: ########## @@ -0,0 +1,693 @@ +{ + "cases": [ + { + "id": "flat-required-and-optional", + "valid": true, + "input": { + "type": "struct", + "schema-id": 0, + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "data", + "required": false, + "type": "string" + } + ] + }, + "decoded": { + "schema-id": 0, + "identifier-field-ids": [], + "fields": [ + { "id": 1, "name": "id", "parent": null, "required": true, "type": "long", "doc": null }, + { "id": 2, "name": "data", "parent": null, "required": false, "type": "string", "doc": null } + ] + } + }, + { + "id": "fields-not-in-id-order", + "valid": true, + "input": { + "type": "struct", + "schema-id": 0, + "fields": [ + { + "id": 3, + "name": "c", + "required": true, + "type": "int" + }, + { + "id": 1, + "name": "a", + "required": true, + "type": "int" + }, + { + "id": 2, + "name": "b", + "required": false, + "type": "int" + } + ] + }, + "decoded": { + "schema-id": 0, + "identifier-field-ids": [], + "fields": [ + { "id": 3, "name": "c", "parent": null, "required": true, "type": "int", "doc": null }, + { "id": 1, "name": "a", "parent": null, "required": true, "type": "int", "doc": null }, + { "id": 2, "name": "b", "parent": null, "required": false, "type": "int", "doc": null } + ] + } + }, + { + "id": "nested-struct", + "valid": true, + "input": { + "type": "struct", + "schema-id": 0, + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "location", + "required": false, + "type": { + "type": "struct", + "fields": [ + { + "id": 3, + "name": "lat", + "required": true, + "type": "double" + }, + { + "id": 4, + "name": "lon", + "required": true, + "type": "double" + } + ] + } + } + ] + }, + "decoded": { + "schema-id": 0, + "identifier-field-ids": [], + "fields": [ + { "id": 1, "name": "id", "parent": null, "required": true, "type": "long", "doc": null }, + { "id": 2, "name": "location", "parent": null, "required": false, "type": "struct", "doc": null }, + { "id": 3, "name": "lat", "parent": 2, "required": true, "type": "double", "doc": null }, + { "id": 4, "name": "lon", "parent": 2, "required": true, "type": "double", "doc": null } + ] + } + }, + { + "id": "list-of-primitive", + "valid": true, + "input": { + "type": "struct", + "schema-id": 0, + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "tags", + "required": true, + "type": { + "type": "list", + "element-id": 3, + "element-required": false, + "element": "string" + } + } + ] + }, + "decoded": { + "schema-id": 0, + "identifier-field-ids": [], + "fields": [ + { "id": 1, "name": "id", "parent": null, "required": true, "type": "long", "doc": null }, + { "id": 2, "name": "tags", "parent": null, "required": true, "type": "list", "doc": null }, + { "id": 3, "name": "element", "parent": 2, "required": false, "type": "string", "doc": null } + ] + } + }, + { + "id": "map-of-primitive", + "valid": true, + "input": { + "type": "struct", + "schema-id": 0, + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "props", + "required": true, + "type": { + "type": "map", + "key-id": 3, + "key": "string", + "value-id": 4, + "value-required": false, + "value": "int" + } + } + ] + }, + "decoded": { + "schema-id": 0, + "identifier-field-ids": [], + "fields": [ + { "id": 1, "name": "id", "parent": null, "required": true, "type": "long", "doc": null }, + { "id": 2, "name": "props", "parent": null, "required": true, "type": "map", "doc": null }, + { "id": 3, "name": "key", "parent": 2, "required": true, "type": "string", "doc": null }, + { "id": 4, "name": "value", "parent": 2, "required": false, "type": "int", "doc": null } + ] + } + }, + { + "id": "list-of-struct", + "valid": true, + "input": { + "type": "struct", + "schema-id": 0, + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "points", + "required": true, + "type": { + "type": "list", + "element-id": 3, + "element-required": true, + "element": { + "type": "struct", + "fields": [ + { + "id": 4, + "name": "x", + "required": true, + "type": "int" + }, + { + "id": 5, + "name": "y", + "required": false, + "type": "int" + } + ] + } + } + } + ] + }, + "decoded": { + "schema-id": 0, + "identifier-field-ids": [], + "fields": [ + { "id": 1, "name": "id", "parent": null, "required": true, "type": "long", "doc": null }, + { "id": 2, "name": "points", "parent": null, "required": true, "type": "list", "doc": null }, + { "id": 3, "name": "element", "parent": 2, "required": true, "type": "struct", "doc": null }, + { "id": 4, "name": "x", "parent": 3, "required": true, "type": "int", "doc": null }, + { "id": 5, "name": "y", "parent": 3, "required": false, "type": "int", "doc": null } + ] + } + }, + { + "id": "map-with-struct-value", + "valid": true, + "input": { + "type": "struct", + "schema-id": 0, + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "by_name", + "required": false, + "type": { + "type": "map", + "key-id": 3, + "key": "string", + "value-id": 4, + "value-required": false, + "value": { + "type": "struct", + "fields": [ + { + "id": 5, + "name": "count", + "required": true, + "type": "long" + } + ] + } + } + } + ] + }, + "decoded": { + "schema-id": 0, + "identifier-field-ids": [], + "fields": [ + { "id": 1, "name": "id", "parent": null, "required": true, "type": "long", "doc": null }, + { "id": 2, "name": "by_name", "parent": null, "required": false, "type": "map", "doc": null }, + { "id": 3, "name": "key", "parent": 2, "required": true, "type": "string", "doc": null }, + { "id": 4, "name": "value", "parent": 2, "required": false, "type": "struct", "doc": null }, + { "id": 5, "name": "count", "parent": 4, "required": true, "type": "long", "doc": null } + ] + } + }, + { + "id": "schema-id-and-identifier-fields", + "valid": true, + "input": { + "type": "struct", + "schema-id": 7, + "identifier-field-ids": [ + 1, + 2 + ], + "fields": [ + { + "id": 1, + "name": "tenant", + "required": true, + "type": "string" + }, + { + "id": 2, + "name": "key", + "required": true, + "type": "long" + }, + { + "id": 3, + "name": "payload", + "required": false, + "type": "string" + } + ] + }, + "decoded": { + "schema-id": 7, + "identifier-field-ids": [1, 2], + "fields": [ + { "id": 1, "name": "tenant", "parent": null, "required": true, "type": "string", "doc": null }, + { "id": 2, "name": "key", "parent": null, "required": true, "type": "long", "doc": null }, + { "id": 3, "name": "payload", "parent": null, "required": false, "type": "string", "doc": null } + ] + } + }, + { + "id": "field-doc", + "valid": true, + "input": { + "type": "struct", + "schema-id": 0, + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long", + "doc": "primary key" + }, + { + "id": 2, + "name": "data", + "required": false, + "type": "string" + } + ] + }, + "decoded": { + "schema-id": 0, + "identifier-field-ids": [], + "fields": [ + { "id": 1, "name": "id", "parent": null, "required": true, "type": "long", "doc": "primary key" }, + { "id": 2, "name": "data", "parent": null, "required": false, "type": "string", "doc": null } + ] + } + }, + { + "id": "all-primitive-types", + "valid": true, + "input": { + "type": "struct", + "schema-id": 0, + "fields": [ + { + "id": 1, + "name": "f_boolean", + "required": false, + "type": "boolean" + }, + { + "id": 2, + "name": "f_int", + "required": false, + "type": "int" + }, + { + "id": 3, + "name": "f_long", + "required": false, + "type": "long" + }, + { + "id": 4, + "name": "f_float", + "required": false, + "type": "float" + }, + { + "id": 5, + "name": "f_double", + "required": false, + "type": "double" + }, + { + "id": 6, + "name": "f_date", + "required": false, + "type": "date" + }, + { + "id": 7, + "name": "f_time", + "required": false, + "type": "time" + }, + { + "id": 8, + "name": "f_timestamp", + "required": false, + "type": "timestamp" + }, + { + "id": 9, + "name": "f_timestamptz", + "required": false, + "type": "timestamptz" + }, + { + "id": 10, + "name": "f_string", + "required": false, + "type": "string" + }, + { + "id": 11, + "name": "f_uuid", + "required": false, + "type": "uuid" + }, + { + "id": 12, + "name": "f_fixed", + "required": false, + "type": "fixed[16]" + }, + { + "id": 13, + "name": "f_binary", + "required": false, + "type": "binary" + }, + { + "id": 14, + "name": "f_decimal", + "required": false, + "type": "decimal(9,2)" + } + ] + }, + "decoded": { + "schema-id": 0, + "identifier-field-ids": [], + "fields": [ + { "id": 1, "name": "f_boolean", "parent": null, "required": false, "type": "boolean", "doc": null }, + { "id": 2, "name": "f_int", "parent": null, "required": false, "type": "int", "doc": null }, + { "id": 3, "name": "f_long", "parent": null, "required": false, "type": "long", "doc": null }, + { "id": 4, "name": "f_float", "parent": null, "required": false, "type": "float", "doc": null }, + { "id": 5, "name": "f_double", "parent": null, "required": false, "type": "double", "doc": null }, + { "id": 6, "name": "f_date", "parent": null, "required": false, "type": "date", "doc": null }, + { "id": 7, "name": "f_time", "parent": null, "required": false, "type": "time", "doc": null }, + { "id": 8, "name": "f_timestamp", "parent": null, "required": false, "type": "timestamp", "doc": null }, + { "id": 9, "name": "f_timestamptz", "parent": null, "required": false, "type": "timestamptz", "doc": null }, + { "id": 10, "name": "f_string", "parent": null, "required": false, "type": "string", "doc": null }, + { "id": 11, "name": "f_uuid", "parent": null, "required": false, "type": "uuid", "doc": null }, + { "id": 12, "name": "f_fixed", "parent": null, "required": false, "type": { "type": "fixed", "length": 16 }, "doc": null }, + { "id": 13, "name": "f_binary", "parent": null, "required": false, "type": "binary", "doc": null }, + { "id": 14, "name": "f_decimal", "parent": null, "required": false, "type": { "type": "decimal", "precision": 9, "scale": 2 }, "doc": null } + ] + } + }, + { + "id": "decimal-max-precision", + "valid": true, + "input": { + "type": "struct", + "schema-id": 0, + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "amount", + "required": false, + "type": "decimal(38, 10)" + } + ] + }, + "decoded": { + "schema-id": 0, + "identifier-field-ids": [], + "fields": [ + { "id": 1, "name": "id", "parent": null, "required": true, "type": "long", "doc": null }, + { "id": 2, "name": "amount", "parent": null, "required": false, "type": { "type": "decimal", "precision": 38, "scale": 10 }, "doc": null } + ] + } + }, + { + "id": "reject-identifier-field-does-not-exist", + "valid": false, + "input": { + "type": "struct", + "schema-id": 0, + "identifier-field-ids": [ + 999 + ], + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + } + ] + } + }, + { + "id": "reject-identifier-optional-field", + "valid": false, + "input": { + "type": "struct", + "schema-id": 0, + "identifier-field-ids": [ + 2 + ], + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "note", + "required": false, + "type": "string" + } + ] + } + }, + { + "id": "reject-identifier-double-field", + "valid": false, + "input": { + "type": "struct", + "schema-id": 0, + "identifier-field-ids": [ + 2 + ], + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "score", + "required": true, + "type": "double" Review Comment: We reject a `double` identifier field here but not a `float` one, and the spec bars both by the same rule. An impl that only checks for `double` passes. I'd add the `float` twin right alongside this case. ########## table-spec/schema/variant/cases.json: ########## @@ -0,0 +1,75 @@ +{ + "cases": [ + { + "id": "variant-top-level", + "valid": true, + "input": { + "type": "struct", + "schema-id": 0, + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "payload", + "required": false, + "type": "variant" + } + ] + }, + "decoded": { + "schema-id": 0, + "identifier-field-ids": [], + "fields": [ + { "id": 1, "name": "id", "parent": null, "required": true, "type": "long", "doc": null }, + { "id": 2, "name": "payload", "parent": null, "required": false, "type": "variant", "doc": null } + ] + } + }, + { + "id": "variant-in-struct", + "valid": true, + "input": { + "type": "struct", + "schema-id": 0, + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "event", + "required": false, + "type": { + "type": "struct", + "fields": [ + { + "id": 3, + "name": "body", + "required": false, + "type": "variant" Review Comment: Both variant cases put it at the top level or inside a struct, but the spec treats variant as first-class — it can be a list element or a map value. A parser that only special-cases variant in struct position passes both of these. I'd add a `list<variant>` and/or `map<string, variant>` case. ########## table-spec/schema/core/cases.json: ########## @@ -0,0 +1,693 @@ +{ + "cases": [ + { + "id": "flat-required-and-optional", + "valid": true, + "input": { + "type": "struct", + "schema-id": 0, + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "data", + "required": false, + "type": "string" + } + ] + }, + "decoded": { + "schema-id": 0, + "identifier-field-ids": [], + "fields": [ + { "id": 1, "name": "id", "parent": null, "required": true, "type": "long", "doc": null }, + { "id": 2, "name": "data", "parent": null, "required": false, "type": "string", "doc": null } + ] + } + }, + { + "id": "fields-not-in-id-order", + "valid": true, + "input": { + "type": "struct", + "schema-id": 0, + "fields": [ + { + "id": 3, + "name": "c", + "required": true, + "type": "int" + }, + { + "id": 1, + "name": "a", + "required": true, + "type": "int" + }, + { + "id": 2, + "name": "b", + "required": false, + "type": "int" + } + ] + }, + "decoded": { + "schema-id": 0, + "identifier-field-ids": [], + "fields": [ + { "id": 3, "name": "c", "parent": null, "required": true, "type": "int", "doc": null }, + { "id": 1, "name": "a", "parent": null, "required": true, "type": "int", "doc": null }, + { "id": 2, "name": "b", "parent": null, "required": false, "type": "int", "doc": null } + ] + } + }, + { + "id": "nested-struct", + "valid": true, + "input": { + "type": "struct", + "schema-id": 0, + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "location", + "required": false, + "type": { + "type": "struct", + "fields": [ + { + "id": 3, + "name": "lat", + "required": true, + "type": "double" + }, + { + "id": 4, + "name": "lon", + "required": true, + "type": "double" + } + ] + } + } + ] + }, + "decoded": { + "schema-id": 0, + "identifier-field-ids": [], + "fields": [ + { "id": 1, "name": "id", "parent": null, "required": true, "type": "long", "doc": null }, + { "id": 2, "name": "location", "parent": null, "required": false, "type": "struct", "doc": null }, + { "id": 3, "name": "lat", "parent": 2, "required": true, "type": "double", "doc": null }, + { "id": 4, "name": "lon", "parent": 2, "required": true, "type": "double", "doc": null } + ] + } + }, + { + "id": "list-of-primitive", + "valid": true, + "input": { + "type": "struct", + "schema-id": 0, + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "tags", + "required": true, + "type": { + "type": "list", + "element-id": 3, + "element-required": false, + "element": "string" + } + } + ] + }, + "decoded": { + "schema-id": 0, + "identifier-field-ids": [], + "fields": [ + { "id": 1, "name": "id", "parent": null, "required": true, "type": "long", "doc": null }, + { "id": 2, "name": "tags", "parent": null, "required": true, "type": "list", "doc": null }, + { "id": 3, "name": "element", "parent": 2, "required": false, "type": "string", "doc": null } + ] + } + }, + { + "id": "map-of-primitive", + "valid": true, + "input": { + "type": "struct", + "schema-id": 0, + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "props", + "required": true, + "type": { + "type": "map", + "key-id": 3, + "key": "string", + "value-id": 4, + "value-required": false, + "value": "int" + } + } + ] + }, + "decoded": { + "schema-id": 0, + "identifier-field-ids": [], + "fields": [ + { "id": 1, "name": "id", "parent": null, "required": true, "type": "long", "doc": null }, + { "id": 2, "name": "props", "parent": null, "required": true, "type": "map", "doc": null }, + { "id": 3, "name": "key", "parent": 2, "required": true, "type": "string", "doc": null }, + { "id": 4, "name": "value", "parent": 2, "required": false, "type": "int", "doc": null } + ] + } + }, + { + "id": "list-of-struct", + "valid": true, + "input": { + "type": "struct", + "schema-id": 0, + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "points", + "required": true, + "type": { + "type": "list", + "element-id": 3, + "element-required": true, + "element": { + "type": "struct", + "fields": [ + { + "id": 4, + "name": "x", + "required": true, + "type": "int" + }, + { + "id": 5, + "name": "y", + "required": false, + "type": "int" + } + ] + } + } + } + ] + }, + "decoded": { + "schema-id": 0, + "identifier-field-ids": [], + "fields": [ + { "id": 1, "name": "id", "parent": null, "required": true, "type": "long", "doc": null }, + { "id": 2, "name": "points", "parent": null, "required": true, "type": "list", "doc": null }, + { "id": 3, "name": "element", "parent": 2, "required": true, "type": "struct", "doc": null }, + { "id": 4, "name": "x", "parent": 3, "required": true, "type": "int", "doc": null }, + { "id": 5, "name": "y", "parent": 3, "required": false, "type": "int", "doc": null } + ] + } + }, + { + "id": "map-with-struct-value", + "valid": true, + "input": { + "type": "struct", + "schema-id": 0, + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "by_name", + "required": false, + "type": { + "type": "map", + "key-id": 3, + "key": "string", + "value-id": 4, + "value-required": false, + "value": { + "type": "struct", + "fields": [ + { + "id": 5, + "name": "count", + "required": true, + "type": "long" + } + ] + } + } + } + ] + }, + "decoded": { + "schema-id": 0, + "identifier-field-ids": [], + "fields": [ + { "id": 1, "name": "id", "parent": null, "required": true, "type": "long", "doc": null }, + { "id": 2, "name": "by_name", "parent": null, "required": false, "type": "map", "doc": null }, + { "id": 3, "name": "key", "parent": 2, "required": true, "type": "string", "doc": null }, + { "id": 4, "name": "value", "parent": 2, "required": false, "type": "struct", "doc": null }, + { "id": 5, "name": "count", "parent": 4, "required": true, "type": "long", "doc": null } + ] + } + }, + { + "id": "schema-id-and-identifier-fields", + "valid": true, + "input": { + "type": "struct", + "schema-id": 7, + "identifier-field-ids": [ + 1, + 2 + ], + "fields": [ + { + "id": 1, + "name": "tenant", + "required": true, + "type": "string" + }, + { + "id": 2, + "name": "key", + "required": true, + "type": "long" + }, + { + "id": 3, + "name": "payload", + "required": false, + "type": "string" + } + ] + }, + "decoded": { + "schema-id": 7, + "identifier-field-ids": [1, 2], + "fields": [ + { "id": 1, "name": "tenant", "parent": null, "required": true, "type": "string", "doc": null }, + { "id": 2, "name": "key", "parent": null, "required": true, "type": "long", "doc": null }, + { "id": 3, "name": "payload", "parent": null, "required": false, "type": "string", "doc": null } + ] + } + }, + { + "id": "field-doc", + "valid": true, + "input": { + "type": "struct", + "schema-id": 0, + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long", + "doc": "primary key" + }, + { + "id": 2, + "name": "data", + "required": false, + "type": "string" + } + ] + }, + "decoded": { + "schema-id": 0, + "identifier-field-ids": [], + "fields": [ + { "id": 1, "name": "id", "parent": null, "required": true, "type": "long", "doc": "primary key" }, + { "id": 2, "name": "data", "parent": null, "required": false, "type": "string", "doc": null } + ] + } + }, + { + "id": "all-primitive-types", + "valid": true, + "input": { + "type": "struct", + "schema-id": 0, + "fields": [ + { + "id": 1, + "name": "f_boolean", + "required": false, + "type": "boolean" + }, + { + "id": 2, + "name": "f_int", + "required": false, + "type": "int" + }, + { + "id": 3, + "name": "f_long", + "required": false, + "type": "long" + }, + { + "id": 4, + "name": "f_float", + "required": false, + "type": "float" + }, + { + "id": 5, + "name": "f_double", + "required": false, + "type": "double" + }, + { + "id": 6, + "name": "f_date", + "required": false, + "type": "date" + }, + { + "id": 7, + "name": "f_time", + "required": false, + "type": "time" + }, + { + "id": 8, + "name": "f_timestamp", + "required": false, + "type": "timestamp" + }, + { + "id": 9, + "name": "f_timestamptz", + "required": false, + "type": "timestamptz" + }, + { + "id": 10, + "name": "f_string", + "required": false, + "type": "string" + }, + { + "id": 11, + "name": "f_uuid", + "required": false, + "type": "uuid" + }, + { + "id": 12, + "name": "f_fixed", + "required": false, + "type": "fixed[16]" + }, + { + "id": 13, + "name": "f_binary", + "required": false, + "type": "binary" + }, + { + "id": 14, + "name": "f_decimal", + "required": false, + "type": "decimal(9,2)" + } + ] + }, + "decoded": { + "schema-id": 0, + "identifier-field-ids": [], + "fields": [ + { "id": 1, "name": "f_boolean", "parent": null, "required": false, "type": "boolean", "doc": null }, + { "id": 2, "name": "f_int", "parent": null, "required": false, "type": "int", "doc": null }, + { "id": 3, "name": "f_long", "parent": null, "required": false, "type": "long", "doc": null }, + { "id": 4, "name": "f_float", "parent": null, "required": false, "type": "float", "doc": null }, + { "id": 5, "name": "f_double", "parent": null, "required": false, "type": "double", "doc": null }, + { "id": 6, "name": "f_date", "parent": null, "required": false, "type": "date", "doc": null }, + { "id": 7, "name": "f_time", "parent": null, "required": false, "type": "time", "doc": null }, + { "id": 8, "name": "f_timestamp", "parent": null, "required": false, "type": "timestamp", "doc": null }, + { "id": 9, "name": "f_timestamptz", "parent": null, "required": false, "type": "timestamptz", "doc": null }, + { "id": 10, "name": "f_string", "parent": null, "required": false, "type": "string", "doc": null }, + { "id": 11, "name": "f_uuid", "parent": null, "required": false, "type": "uuid", "doc": null }, + { "id": 12, "name": "f_fixed", "parent": null, "required": false, "type": { "type": "fixed", "length": 16 }, "doc": null }, + { "id": 13, "name": "f_binary", "parent": null, "required": false, "type": "binary", "doc": null }, + { "id": 14, "name": "f_decimal", "parent": null, "required": false, "type": { "type": "decimal", "precision": 9, "scale": 2 }, "doc": null } + ] + } + }, + { + "id": "decimal-max-precision", + "valid": true, + "input": { + "type": "struct", + "schema-id": 0, + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "amount", + "required": false, + "type": "decimal(38, 10)" + } + ] + }, + "decoded": { + "schema-id": 0, + "identifier-field-ids": [], + "fields": [ + { "id": 1, "name": "id", "parent": null, "required": true, "type": "long", "doc": null }, + { "id": 2, "name": "amount", "parent": null, "required": false, "type": { "type": "decimal", "precision": 38, "scale": 10 }, "doc": null } + ] + } + }, + { + "id": "reject-identifier-field-does-not-exist", + "valid": false, + "input": { + "type": "struct", + "schema-id": 0, + "identifier-field-ids": [ + 999 + ], + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + } + ] + } + }, + { + "id": "reject-identifier-optional-field", + "valid": false, + "input": { + "type": "struct", + "schema-id": 0, + "identifier-field-ids": [ + 2 + ], + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "note", + "required": false, + "type": "string" + } + ] + } + }, + { + "id": "reject-identifier-double-field", + "valid": false, + "input": { + "type": "struct", + "schema-id": 0, + "identifier-field-ids": [ + 2 + ], + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "score", + "required": true, + "type": "double" + } + ] + } + }, + { + "id": "reject-identifier-non-primitive-field", + "valid": false, + "input": { + "type": "struct", + "schema-id": 0, + "identifier-field-ids": [ + 2 + ], + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "loc", + "required": true, + "type": { + "type": "struct", + "fields": [ + { + "id": 3, + "name": "x", + "required": true, + "type": "long" + } + ] + } + } + ] + } + }, + { + "id": "reject-identifier-nested-in-list", Review Comment: We cover an identifier field nested in a list but not one nested in a map — the spec's containment rule applies to both equally. An impl that only walks list ancestry never fails. I'd add the map twin (identifier field pointing into a map key or value). ########## table-spec/schema/core/cases.json: ########## @@ -0,0 +1,693 @@ +{ + "cases": [ + { + "id": "flat-required-and-optional", + "valid": true, + "input": { + "type": "struct", + "schema-id": 0, + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "data", + "required": false, + "type": "string" + } + ] + }, + "decoded": { + "schema-id": 0, + "identifier-field-ids": [], + "fields": [ + { "id": 1, "name": "id", "parent": null, "required": true, "type": "long", "doc": null }, + { "id": 2, "name": "data", "parent": null, "required": false, "type": "string", "doc": null } + ] + } + }, + { + "id": "fields-not-in-id-order", + "valid": true, + "input": { + "type": "struct", + "schema-id": 0, + "fields": [ + { + "id": 3, + "name": "c", + "required": true, + "type": "int" + }, + { + "id": 1, + "name": "a", + "required": true, + "type": "int" + }, + { + "id": 2, + "name": "b", + "required": false, + "type": "int" + } + ] + }, + "decoded": { + "schema-id": 0, + "identifier-field-ids": [], + "fields": [ + { "id": 3, "name": "c", "parent": null, "required": true, "type": "int", "doc": null }, + { "id": 1, "name": "a", "parent": null, "required": true, "type": "int", "doc": null }, + { "id": 2, "name": "b", "parent": null, "required": false, "type": "int", "doc": null } + ] + } + }, + { + "id": "nested-struct", + "valid": true, + "input": { + "type": "struct", + "schema-id": 0, + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "location", + "required": false, + "type": { + "type": "struct", + "fields": [ + { + "id": 3, + "name": "lat", + "required": true, + "type": "double" + }, + { + "id": 4, + "name": "lon", + "required": true, + "type": "double" + } + ] + } + } + ] + }, + "decoded": { + "schema-id": 0, + "identifier-field-ids": [], + "fields": [ + { "id": 1, "name": "id", "parent": null, "required": true, "type": "long", "doc": null }, + { "id": 2, "name": "location", "parent": null, "required": false, "type": "struct", "doc": null }, + { "id": 3, "name": "lat", "parent": 2, "required": true, "type": "double", "doc": null }, + { "id": 4, "name": "lon", "parent": 2, "required": true, "type": "double", "doc": null } + ] + } + }, + { + "id": "list-of-primitive", + "valid": true, + "input": { + "type": "struct", + "schema-id": 0, + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "tags", + "required": true, + "type": { + "type": "list", + "element-id": 3, + "element-required": false, + "element": "string" + } + } + ] + }, + "decoded": { + "schema-id": 0, + "identifier-field-ids": [], + "fields": [ + { "id": 1, "name": "id", "parent": null, "required": true, "type": "long", "doc": null }, + { "id": 2, "name": "tags", "parent": null, "required": true, "type": "list", "doc": null }, + { "id": 3, "name": "element", "parent": 2, "required": false, "type": "string", "doc": null } + ] + } + }, + { + "id": "map-of-primitive", + "valid": true, + "input": { + "type": "struct", + "schema-id": 0, + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "props", + "required": true, + "type": { + "type": "map", + "key-id": 3, + "key": "string", + "value-id": 4, + "value-required": false, + "value": "int" + } + } + ] + }, + "decoded": { + "schema-id": 0, + "identifier-field-ids": [], + "fields": [ + { "id": 1, "name": "id", "parent": null, "required": true, "type": "long", "doc": null }, + { "id": 2, "name": "props", "parent": null, "required": true, "type": "map", "doc": null }, + { "id": 3, "name": "key", "parent": 2, "required": true, "type": "string", "doc": null }, + { "id": 4, "name": "value", "parent": 2, "required": false, "type": "int", "doc": null } + ] + } + }, + { + "id": "list-of-struct", + "valid": true, + "input": { + "type": "struct", + "schema-id": 0, + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "points", + "required": true, + "type": { + "type": "list", + "element-id": 3, + "element-required": true, + "element": { + "type": "struct", + "fields": [ + { + "id": 4, + "name": "x", + "required": true, + "type": "int" + }, + { + "id": 5, + "name": "y", + "required": false, + "type": "int" + } + ] + } + } + } + ] + }, + "decoded": { + "schema-id": 0, + "identifier-field-ids": [], + "fields": [ + { "id": 1, "name": "id", "parent": null, "required": true, "type": "long", "doc": null }, + { "id": 2, "name": "points", "parent": null, "required": true, "type": "list", "doc": null }, + { "id": 3, "name": "element", "parent": 2, "required": true, "type": "struct", "doc": null }, + { "id": 4, "name": "x", "parent": 3, "required": true, "type": "int", "doc": null }, + { "id": 5, "name": "y", "parent": 3, "required": false, "type": "int", "doc": null } + ] + } + }, + { + "id": "map-with-struct-value", + "valid": true, + "input": { + "type": "struct", + "schema-id": 0, + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "by_name", + "required": false, + "type": { + "type": "map", + "key-id": 3, + "key": "string", + "value-id": 4, + "value-required": false, + "value": { + "type": "struct", + "fields": [ + { + "id": 5, + "name": "count", + "required": true, + "type": "long" + } + ] + } + } + } + ] + }, + "decoded": { + "schema-id": 0, + "identifier-field-ids": [], + "fields": [ + { "id": 1, "name": "id", "parent": null, "required": true, "type": "long", "doc": null }, + { "id": 2, "name": "by_name", "parent": null, "required": false, "type": "map", "doc": null }, + { "id": 3, "name": "key", "parent": 2, "required": true, "type": "string", "doc": null }, + { "id": 4, "name": "value", "parent": 2, "required": false, "type": "struct", "doc": null }, + { "id": 5, "name": "count", "parent": 4, "required": true, "type": "long", "doc": null } + ] + } + }, + { + "id": "schema-id-and-identifier-fields", + "valid": true, + "input": { + "type": "struct", + "schema-id": 7, + "identifier-field-ids": [ + 1, + 2 + ], + "fields": [ + { + "id": 1, + "name": "tenant", + "required": true, + "type": "string" + }, + { + "id": 2, + "name": "key", + "required": true, + "type": "long" + }, + { + "id": 3, + "name": "payload", + "required": false, + "type": "string" + } + ] + }, + "decoded": { + "schema-id": 7, + "identifier-field-ids": [1, 2], + "fields": [ + { "id": 1, "name": "tenant", "parent": null, "required": true, "type": "string", "doc": null }, + { "id": 2, "name": "key", "parent": null, "required": true, "type": "long", "doc": null }, + { "id": 3, "name": "payload", "parent": null, "required": false, "type": "string", "doc": null } + ] + } + }, + { + "id": "field-doc", + "valid": true, + "input": { + "type": "struct", + "schema-id": 0, + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long", + "doc": "primary key" + }, + { + "id": 2, + "name": "data", + "required": false, + "type": "string" + } + ] + }, + "decoded": { + "schema-id": 0, + "identifier-field-ids": [], + "fields": [ + { "id": 1, "name": "id", "parent": null, "required": true, "type": "long", "doc": "primary key" }, + { "id": 2, "name": "data", "parent": null, "required": false, "type": "string", "doc": null } + ] + } + }, + { + "id": "all-primitive-types", + "valid": true, + "input": { + "type": "struct", + "schema-id": 0, + "fields": [ + { + "id": 1, + "name": "f_boolean", + "required": false, + "type": "boolean" + }, + { + "id": 2, + "name": "f_int", + "required": false, + "type": "int" + }, + { + "id": 3, + "name": "f_long", + "required": false, + "type": "long" + }, + { + "id": 4, + "name": "f_float", + "required": false, + "type": "float" + }, + { + "id": 5, + "name": "f_double", + "required": false, + "type": "double" + }, + { + "id": 6, + "name": "f_date", + "required": false, + "type": "date" + }, + { + "id": 7, + "name": "f_time", + "required": false, + "type": "time" + }, + { + "id": 8, + "name": "f_timestamp", + "required": false, + "type": "timestamp" + }, + { + "id": 9, + "name": "f_timestamptz", + "required": false, + "type": "timestamptz" + }, + { + "id": 10, + "name": "f_string", + "required": false, + "type": "string" + }, + { + "id": 11, + "name": "f_uuid", + "required": false, + "type": "uuid" + }, + { + "id": 12, + "name": "f_fixed", + "required": false, + "type": "fixed[16]" + }, + { + "id": 13, + "name": "f_binary", + "required": false, + "type": "binary" + }, + { + "id": 14, + "name": "f_decimal", + "required": false, + "type": "decimal(9,2)" + } + ] + }, + "decoded": { + "schema-id": 0, + "identifier-field-ids": [], + "fields": [ + { "id": 1, "name": "f_boolean", "parent": null, "required": false, "type": "boolean", "doc": null }, + { "id": 2, "name": "f_int", "parent": null, "required": false, "type": "int", "doc": null }, + { "id": 3, "name": "f_long", "parent": null, "required": false, "type": "long", "doc": null }, + { "id": 4, "name": "f_float", "parent": null, "required": false, "type": "float", "doc": null }, + { "id": 5, "name": "f_double", "parent": null, "required": false, "type": "double", "doc": null }, + { "id": 6, "name": "f_date", "parent": null, "required": false, "type": "date", "doc": null }, + { "id": 7, "name": "f_time", "parent": null, "required": false, "type": "time", "doc": null }, + { "id": 8, "name": "f_timestamp", "parent": null, "required": false, "type": "timestamp", "doc": null }, + { "id": 9, "name": "f_timestamptz", "parent": null, "required": false, "type": "timestamptz", "doc": null }, + { "id": 10, "name": "f_string", "parent": null, "required": false, "type": "string", "doc": null }, + { "id": 11, "name": "f_uuid", "parent": null, "required": false, "type": "uuid", "doc": null }, + { "id": 12, "name": "f_fixed", "parent": null, "required": false, "type": { "type": "fixed", "length": 16 }, "doc": null }, + { "id": 13, "name": "f_binary", "parent": null, "required": false, "type": "binary", "doc": null }, + { "id": 14, "name": "f_decimal", "parent": null, "required": false, "type": { "type": "decimal", "precision": 9, "scale": 2 }, "doc": null } + ] + } + }, + { + "id": "decimal-max-precision", + "valid": true, + "input": { + "type": "struct", + "schema-id": 0, + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "amount", + "required": false, + "type": "decimal(38, 10)" + } + ] + }, + "decoded": { + "schema-id": 0, + "identifier-field-ids": [], + "fields": [ + { "id": 1, "name": "id", "parent": null, "required": true, "type": "long", "doc": null }, + { "id": 2, "name": "amount", "parent": null, "required": false, "type": { "type": "decimal", "precision": 38, "scale": 10 }, "doc": null } + ] + } + }, + { + "id": "reject-identifier-field-does-not-exist", + "valid": false, + "input": { + "type": "struct", + "schema-id": 0, + "identifier-field-ids": [ + 999 + ], + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + } + ] + } + }, + { + "id": "reject-identifier-optional-field", + "valid": false, + "input": { + "type": "struct", + "schema-id": 0, + "identifier-field-ids": [ + 2 + ], + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "note", + "required": false, + "type": "string" + } + ] + } + }, + { + "id": "reject-identifier-double-field", + "valid": false, + "input": { + "type": "struct", + "schema-id": 0, + "identifier-field-ids": [ + 2 + ], + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "score", + "required": true, + "type": "double" + } + ] + } + }, + { + "id": "reject-identifier-non-primitive-field", + "valid": false, + "input": { + "type": "struct", + "schema-id": 0, + "identifier-field-ids": [ + 2 + ], + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "loc", + "required": true, + "type": { + "type": "struct", + "fields": [ + { + "id": 3, + "name": "x", + "required": true, + "type": "long" + } + ] + } + } + ] + } + }, + { + "id": "reject-identifier-nested-in-list", + "valid": false, + "input": { + "type": "struct", + "schema-id": 0, + "identifier-field-ids": [ + 4 + ], + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "rows", + "required": true, + "type": { + "type": "list", + "element-id": 3, + "element-required": true, + "element": { + "type": "struct", + "fields": [ + { + "id": 4, + "name": "k", + "required": true, + "type": "string" + } + ] + } + } + } + ] + } + }, + { + "id": "reject-identifier-nested-in-optional-struct", Review Comment: This rejects an identifier field inside an optional struct, which is right, but there's no positive case for an identifier inside a required nested struct (all ancestors required), which the spec allows. So an implementation that over-rejects every nested identifier passes all of these. I'd add one positive nested-identifier case to pin down the accept side. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
