SavicStefan commented on code in PR #59669:
URL: https://github.com/apache/spark/pull/59669#discussion_r4452676139
##########
common/variant/src/main/java/org/apache/spark/types/variant/VariantBuilder.java:
##########
@@ -205,6 +205,79 @@ public static Variant pickAtPaths(Variant v, PickNode
tree) {
return builder.result();
}
+ // Return v in canonical form: only referenced dictionary keys, unique and
sorted by unsigned
+ // UTF-8 bytes; object fields stored in key order; minimal integer, decimal,
and offset widths;
+ // strings of at most 63 UTF-8 bytes use short-string encoding; decimal
trailing zeros are
+ // stripped, and integral decimals become longs only when they fit.
Normalize floating-point
+ // -0.0 to 0.0 and NaNs within each type. Preserve string and key bytes,
including invalid UTF-8.
+ // Return v itself if it is already canonical.
+ public static Variant canonicalize(Variant v) {
+ if (isCanonical(v)) {
+ return v;
+ }
+ return doCanonicalize(v);
+ }
+
+ // Check whether v is already canonically encoded. Return false at the first
+ // non-canonical encoding. Malformed bytes also return false so that the
rebuild decides whether
+ // they fail, but an unreadable header may still throw. Returning false does
not establish
+ // validity.
+ static boolean isCanonical(Variant v) {
Review Comment:
Agreed, will added a Javadoc on canonicalize that lists every rule
`isCanonical` checks and `doCanonicalize` produces, including the metadata
header offset size and the object/array header rules and pointed `isCanonical`
and `doCanonicalize` at it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
