cloud-fan commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1609808967
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -173,26 +174,546 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * Collation ID is defined as 32-bit integer. We specify binary layouts
for different classes of
+ * collations. Classes of collations are differentiated by most
significant 3 bits (bit 31, 30
+ * and 29), bit 31 being most significant and bit 0 being least
significant.
+ * ---
+ * General collation ID binary layout:
+ * bit 31: 1 for INDETERMINATE (requires all other bits to be 1 as
well), 0 otherwise.
+ * bit 30: 0 for predefined, 1 for user-defined.
+ * Following bits are specified for predefined collations:
+ * bit 29: 0 for UTF8_BINARY, 1 for ICU collations.
+ * bit 28-24: Reserved.
+ * bit 23-22: Reserved for version.
+ * bit 21-18: Reserved for space trimming.
+ * bit 17-0: Depend on collation family.
+ * ---
+ * INDETERMINATE collation ID binary layout:
+ * bit 31-0: 1
+ * INDETERMINATE collation ID is equal to -1.
+ * ---
+ * User-defined collation ID binary layout:
+ * bit 31: 0
+ * bit 30: 1
+ * bit 29-0: Undefined, reserved for future use.
+ * ---
+ * UTF8_BINARY collation ID binary layout:
+ * bit 31-24: Zeroes.
+ * bit 23-22: Zeroes, reserved for version.
+ * bit 21-18: Zeroes, reserved for space trimming.
+ * bit 17-3: Zeroes.
+ * bit 2: 0, reserved for accent sensitivity.
+ * bit 1: 0, reserved for uppercase and case-insensitive.
+ * bit 0: 0 = case-sensitive, 1 = lowercase.
+ * ---
+ * ICU collation ID binary layout:
Review Comment:
Do we design the binary layout by ourselves? It looks a bit weird that we
put case-sensitivity and accent-sensitivity bits in different places between
`UTF8_BINARY` and other collations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
