laskoviymishka commented on code in PR #17:
URL:
https://github.com/apache/iceberg-verification/pull/17#discussion_r4472635274
##########
table-spec/partition-transforms/bucket/cases.json:
##########
@@ -0,0 +1,148 @@
+{
+ "cases": [
+ {
+ "id": "bucket-int-100",
+ "valid": true,
+ "input": { "transform": "bucket[100]", "source_type": "int", "values":
[null, 34] },
+ "decoded": { "result_types": ["int"], "values": [null, 79] },
+ "clause": "Bucket Transform Details and Appendix B: int hash reference
vector",
+ "spec_ref": "format/spec.md#appendix-b-32-bit-hash-requirements"
+ },
+ {
+ "id": "bucket-long-100",
+ "valid": true,
+ "input": { "transform": "bucket[100]", "source_type": "long", "values":
[null, "34"] },
+ "decoded": { "result_types": ["int"], "values": [null, 79] },
+ "clause": "Appendix B: integer and long hash results must be identical
for all integer values",
+ "spec_ref": "format/spec.md#appendix-b-32-bit-hash-requirements"
+ },
+ {
+ "id": "bucket-decimal-9-2-100",
+ "valid": true,
+ "input": { "transform": "bucket[100]", "source_type": "decimal(9, 2)",
"values": [null, "14.20"] },
+ "decoded": { "result_types": ["int"], "values": [null, 59] },
+ "clause": "Appendix B: decimal 14.20 hashes from its minimum
two's-complement big-endian unscaled value; bucket discards the sign bit",
+ "spec_ref": "format/spec.md#appendix-b-32-bit-hash-requirements"
+ },
+ {
+ "id": "bucket-decimal-10-3-100",
+ "valid": true,
+ "input": { "transform": "bucket[100]", "source_type": "decimal(10, 3)",
"values": [null, "1.420"] },
+ "decoded": { "result_types": ["int"], "values": [null, 59] },
+ "clause": "Appendix B: decimal hashes use the unscaled value and hash
results are not dependent on decimal scale",
+ "spec_ref": "format/spec.md#appendix-b-32-bit-hash-requirements"
+ },
+ {
+ "id": "bucket-date-100",
+ "valid": true,
+ "input": { "transform": "bucket[100]", "source_type": "date", "values":
[null, "2017-11-16"] },
+ "decoded": { "result_types": ["int"], "values": [null, 26] },
+ "clause": "Appendix B: date hash reference vector; bucket discards the
sign bit",
+ "spec_ref": "format/spec.md#appendix-b-32-bit-hash-requirements"
+ },
+ {
+ "id": "bucket-time-100",
+ "valid": true,
+ "input": { "transform": "bucket[100]", "source_type": "time", "values":
[null, "22:31:08"] },
+ "decoded": { "result_types": ["int"], "values": [null, 59] },
+ "clause": "Appendix B: time hash reference vector; bucket discards the
sign bit",
+ "spec_ref": "format/spec.md#appendix-b-32-bit-hash-requirements"
+ },
+ {
+ "id": "bucket-timestamp-100",
+ "valid": true,
+ "input": {
+ "transform": "bucket[100]",
+ "source_type": "timestamp",
+ "values": [null, "2017-11-16T22:31:08", "2017-11-16T22:31:08.000001"]
+ },
+ "decoded": { "result_types": ["int"], "values": [null, 7, 38] },
+ "clause": "Appendix B: timestamp hash reference vectors at whole-second
and microsecond precision",
+ "spec_ref": "format/spec.md#appendix-b-32-bit-hash-requirements"
+ },
+ {
+ "id": "bucket-timestamptz-100",
+ "valid": true,
+ "input": {
+ "transform": "bucket[100]",
+ "source_type": "timestamptz",
+ "values": [
+ null,
+ "2017-11-16T14:31:08-08:00",
+ "2017-11-16T22:31:08Z",
+ "2017-11-16T14:31:08.000001-08:00",
+ "2017-11-16T22:31:08.000001Z"
+ ]
+ },
+ "decoded": { "result_types": ["int"], "values": [null, 7, 7, 38, 38] },
+ "clause": "Appendix B: timestamptz hashes microseconds from the Unix
epoch, so equivalent instants hash identically",
+ "spec_ref": "format/spec.md#appendix-b-32-bit-hash-requirements"
+ },
+ {
+ "id": "bucket-timestamp-ns-100",
+ "valid": true,
+ "input": {
+ "transform": "bucket[100]",
+ "source_type": "timestamp_ns",
+ "values": [null, "2017-11-16T22:31:08",
"2017-11-16T22:31:08.000001001", "2017-11-16T22:31:08.000001999"]
+ },
+ "decoded": { "result_types": ["int"], "values": [null, 7, 38, 38] },
Review Comment:
Could we add a pre-epoch value to pin which way the ns-to-µs conversion
rounds? Every ns input here is a 2017 instant, and for those floor and
truncation give the same bucket. With a Murmur3 that reproduces the Appendix B
vectors, `1969-12-31T23:59:59.999999999` lands in bucket 12 under floor (-1 µs)
and 76 under truncation (0 µs). The spec only says ["converted to microsecond
precision"](https://github.com/apache/iceberg/blob/ea73769d120dc3819d8cfa2253b6dba4bb394886/format/spec.md?plain=1#L2019)
with no direction, so the 12 below follows Java's
[`nanosToMicros`](https://github.com/apache/iceberg/blob/ea73769d120dc3819d8cfa2253b6dba4bb394886/api/src/main/java/org/apache/iceberg/util/DateTimeUtil.java#L104-L106)
(`Math.floorDiv`), not an explicit rule in the standard. The same value with
`Z` belongs in `bucket-timestamptz-ns-100`.
```suggestion
"values": [
null,
"2017-11-16T22:31:08",
"2017-11-16T22:31:08.000001001",
"2017-11-16T22:31:08.000001999",
"1969-12-31T23:59:59.999999999"
]
},
"decoded": { "result_types": ["int"], "values": [null, 7, 38, 38, 12]
},
```
##########
table-spec/partition-transforms/README.md:
##########
@@ -0,0 +1,165 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+# Partition transforms
+
+Binding a partition transform to a source type and applying it to values
+produces the same logical values and result type in every implementation. The
+Iceberg specification is the source of truth for this surface.
+
+## Assertion
+
+```
+apply(bind(transform, source_type), ordered_inputs) == ordered_expected_values
+result_type(bind(transform, source_type)) is one of allowed_result_types
+```
+
+Bytes are not compared. Each implementation decodes the language-neutral input
+values, binds the transform, applies it in order, and maps its results back to
+the language-neutral representation below.
+
+## Scope
+
+This surface covers the value transform and its result type. It binds every
+standard transform to each source type permitted by the Partition Transforms
+table, subject to the deliberate omissions below. The cases include the spec's
+examples, boundary values, and an explicit `null` input for every
+transform/source-type path. As required by the spec, "All transforms must
return
+`null` for a `null` input value."
+
+Consumers subscribe by subdirectory:
+
+- `identity/` - "Source value, unmodified."
+- `bucket/` - Murmur3 x86 32-bit, seed 0, with the sign bit discarded before
+ taking modulo `N`.
+- `truncate/` - integer, long, decimal, string, and binary truncation.
+- `year/`, `month/`, `day/`, and `hour/` - temporal ordinals from the Unix
+ epoch. `day` expected values are logical dates.
+- `void/` - "Always produces `null`."
+
+The subdirectories are independent subscription boundaries. For example, an
+implementation without nanosecond timestamp types may skip cases for those
+source types while running the other cases in the same directory.
+
+## Case format
+
+Each `cases.json` holds a `cases` array. A case has this shape:
+
+```json
+{
+ "id": "bucket-int-100",
+ "valid": true,
+ "input": {
+ "transform": "bucket[100]",
+ "source_type": "int",
+ "values": [34, null]
+ },
+ "decoded": {
+ "result_types": ["int"],
+ "values": [79, null]
Review Comment:
Could the example match the real `bucket-int-100` case? It reuses that id
but puts `null` last, while `bucket/cases.json` has `[null, 34]` and `[null,
79]`, like every other case in the surface.
```suggestion
"values": [null, 34]
},
"decoded": {
"result_types": ["int"],
"values": [null, 79]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
