nssalian commented on code in PR #25:
URL:
https://github.com/apache/iceberg-verification/pull/25#discussion_r4604585996
##########
table-spec/partition-transforms/README.md:
##########
Review Comment:
nit: the validator also refuses decimal overflow ("does not settle decimal
overflow"), e.g. decimal(9, 2) "-9999999.99" with truncate[10] needs 10 digits,
and Java returns "-10000000.00" there without an error. Could this say "integer
and decimal overflow", with a selftest next to "truncate int overflow"?
Deleting the decimal guard leaves make selftest green today.
##########
Makefile:
##########
@@ -126,6 +126,21 @@ selftest: $(VENV_STAMP) ## Check the validator accepts a
good fixture and reject
assert_partition_transform_reject "zero truncate width"
'{"cases":[{"id":"a","valid":true,"input":{"transform":"truncate[0]","source_type":"int","values":[null,34]},"decoded":{"result_types":["int"],"values":[null,34]},"clause":"c","spec_ref":"s"}]}'
'cases/0/input/transform' truncate; \
assert_partition_transform_reject "zero bucket count outside bucket
directory"
'{"cases":[{"id":"a","valid":true,"input":{"transform":"bucket[0]","source_type":"int","values":[null,34]},"decoded":{"result_types":["int"],"values":[null,0]},"clause":"c","spec_ref":"s"}]}'
'cases/0/input/transform'; \
assert_partition_transform_reject "bucket value checked outside bucket
directory"
'{"cases":[{"id":"a","valid":true,"input":{"transform":"bucket[100]","source_type":"int","values":[null,34]},"decoded":{"result_types":["int"],"values":[null,78]},"clause":"c","spec_ref":"s"}]}'
'expected 79, found 78'; \
+ assert_partition_transform_accept "computed truncate value"
'{"cases":[{"id":"a","valid":true,"input":{"transform":"truncate[10]","source_type":"int","values":[null,-1]},"decoded":{"result_types":["int"],"values":[null,-10]},"clause":"c","spec_ref":"s"}]}'
truncate; \
+ assert_partition_transform_accept "computed pre-epoch hour value"
'{"cases":[{"id":"a","valid":true,"input":{"transform":"hour","source_type":"timestamp","values":[null,"1969-12-31T22:59:59.999999"]},"decoded":{"result_types":["int"],"values":[null,-2]},"clause":"c","spec_ref":"s"}]}'
hour; \
+ assert_partition_transform_reject "truncate int remainder toward zero"
'{"cases":[{"id":"a","valid":true,"input":{"transform":"truncate[10]","source_type":"int","values":[null,-1]},"decoded":{"result_types":["int"],"values":[null,0]},"clause":"c","spec_ref":"s"}]}'
'expected -10, found 0' truncate; \
+ assert_partition_transform_reject "truncate long remainder toward zero"
'{"cases":[{"id":"a","valid":true,"input":{"transform":"truncate[10]","source_type":"long","values":[null,"-1"]},"decoded":{"result_types":["long"],"values":[null,"0"]},"clause":"c","spec_ref":"s"}]}'
'expected "-10", found "0"' truncate; \
+ assert_partition_transform_reject "truncate decimal losing its scale"
'{"cases":[{"id":"a","valid":true,"input":{"transform":"truncate[10]","source_type":"decimal(9,
2)","values":[null,"-0.05"]},"decoded":{"result_types":["decimal(9,
2)"],"values":[null,"-0.1"]},"clause":"c","spec_ref":"s"}]}' 'expected "-0.10",
found "-0.1"' truncate; \
+ assert_partition_transform_reject "truncate string counting UTF-8
bytes"
'{"cases":[{"id":"a","valid":true,"input":{"transform":"truncate[3]","source_type":"string","values":[null,"a\ud83d\ude00bcd"]},"decoded":{"result_types":["string"],"values":[null,"a\ud83d\ude00"]},"clause":"c","spec_ref":"s"}]}'
'cases/0/decoded/values/1' truncate; \
Review Comment:
nit: `😀` is 4 bytes in UTF-8, so a byte-counting truncate[3] gives "a" here,
and with "a😀" as the wrong answer this still passes if truncate_value counts
bytes. Using "a" as the wrong value would pin it:
```suggestion
assert_partition_transform_reject "truncate string counting UTF-8
bytes"
'{"cases":[{"id":"a","valid":true,"input":{"transform":"truncate[3]","source_type":"string","values":[null,"a😀bcd"]},"decoded":{"result_types":["string"],"values":[null,"a"]},"clause":"c","spec_ref":"s"}]}'
'cases/0/decoded/values/1' truncate; \
```
##########
dev/validate-fixtures.py:
##########
@@ -370,28 +389,115 @@ def _bucket_errors(path, doc):
if murmur3_x86_32(data) != expected_hash:
raise AssertionError("Murmur3 does not reproduce an Appendix B
vector")
+
+def bucket_value(count, source_type, value):
+ return (murmur3_x86_32(encode_for_hash(source_type, value)) & 0x7FFFFFFF)
% count
+
+
+def truncate_value(width, source_type, value):
+ """Truncate Transform Details; remainders are positive, as Python's % is
for W > 0."""
+ if source_type in ("int", "long"):
+ if source_type == "int":
+ if type(value) is not int or not -(1 << 31) <= value < (1 << 31):
+ raise ValueError("int must be a 32-bit JSON integer")
+ number, bits = value, 32
+ else:
+ if not isinstance(value, str) or not
re.fullmatch(r"-?(0|[1-9][0-9]*)", value):
+ raise ValueError("long must be a decimal string")
+ number, bits = int(value), 64
+ if not -(1 << 63) <= number < (1 << 63):
+ raise ValueError("long must fit 64 bits")
+ truncated = number - number % width
+ if truncated < -(1 << (bits - 1)):
+ raise ValueError("the specification does not settle integer
overflow")
+ return truncated if source_type == "int" else str(truncated)
+ if source_type.startswith("decimal("):
+ precision, scale, unscaled = parse_decimal(source_type, value)
+ truncated = unscaled - unscaled % width
+ if len(str(abs(truncated))) > precision:
+ raise ValueError("the specification does not settle decimal
overflow")
+ return format_decimal(truncated, scale)
+ if source_type == "string":
+ if not isinstance(value, str):
+ raise ValueError("string must be a JSON string")
+ if any(0xD800 <= ord(char) <= 0xDFFF for char in value):
+ raise ValueError("string must hold Unicode scalar values")
+ return value[:width]
+ if source_type == "binary":
+ if not isinstance(value, str) or not re.fullmatch(r"(?:[0-9A-F]{2})*",
value):
+ raise ValueError("binary must be uppercase hexadecimal bytes")
+ return value[: 2 * width]
+ raise ValueError(f"unsupported truncate source type: {source_type}")
+
+
+def temporal_value(transform, source_type, value):
+ """year, month, day, and hour as ordinals from the Unix epoch, rounded
down."""
+ if not isinstance(value, str):
+ raise ValueError(f"{source_type} must be an ISO string")
+ if source_type == "date":
+ if not re.fullmatch(r"\d{4}-\d{2}-\d{2}", value):
+ raise ValueError("date must use YYYY-MM-DD")
+ day = date.fromisoformat(value)
+ hours = None
+ else:
+ match = TIMESTAMP.fullmatch(value)
+ if not match or bool(match[3]) !=
source_type.startswith("timestamptz"):
+ raise ValueError("timestamp zone must match source type")
+ if match[2] and len(match[2]) > (9 if source_type.endswith("_ns") else
6):
+ raise ValueError("timestamp exceeds source type precision")
+ nanoseconds = epoch_nanos(value)
+ hours = nanoseconds // 3_600_000_000_000
+ day = date(1970, 1, 1) + timedelta(days=nanoseconds //
86_400_000_000_000)
+ if transform == "year":
+ return day.year - 1970
+ if transform == "month":
+ return (day.year - 1970) * 12 + day.month - 1
+ if transform == "day":
+ return day.isoformat()
+ if transform == "hour" and hours is not None:
+ return hours
+ raise ValueError(f"unsupported {transform} source type: {source_type}")
+
+
+def expected_value(transform, source_type, value):
+ """Recompute a transform result from the spec, or None for an unchecked
transform."""
+ if value is None:
+ return None
+ bucket = re.fullmatch(r"bucket\[([1-9][0-9]*)\]", transform)
+ if bucket:
+ return bucket_value(int(bucket[1]), source_type, value)
+ truncate = re.fullmatch(r"truncate\[([1-9][0-9]*)\]", transform)
+ if truncate:
+ return truncate_value(int(truncate[1]), source_type, value)
+ if transform in ("year", "month", "day", "hour"):
+ return temporal_value(transform, source_type, value)
+ return None
+
+
+RECOMPUTED = re.compile(r"^(bucket\[|truncate\[|year$|month$|day$|hour$)")
Review Comment:
One more case your sentinel would close: "truncate[10]\n" passes RECOMPUTED
and the schema pattern, then expected_value returns None, so a fixture with
null outputs validates while plain "truncate[10]" reports "expected 30, found
null".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]