dbatomic commented on code in PR #45290:
URL: https://github.com/apache/spark/pull/45290#discussion_r1509062803
##########
sql/api/src/main/scala/org/apache/spark/sql/types/StringType.scala:
##########
@@ -33,6 +33,14 @@ class StringType private(val collationId: Int) extends
AtomicType with Serializa
*/
def isDefaultCollation: Boolean = collationId ==
StringType.DEFAULT_COLLATION_ID
+ /**
+ * Binary collation implies that strings are considered equal only if they
are
+ * byte for byte equal. E.g. all accent or case-insensitive collations are
considered non-binary.
+ * If this field is true, byte level operations can be used against this
datatype (e.g. for
+ * equality and hashing).
+ */
+ def isBinaryCollation: Boolean =
CollationFactory.fetchCollation(collationId).isBinaryCollation
Review Comment:
Binary collation means that you can do hash and equal (and contains) on
binary representation.
Default collation means that you can do hash, equal and *sort* on binary
representation.
Example of binary collation is UNICODE at identical ICU level. You need to
consult ICU tables to do ordering but hashing/eq should work on byte level.
I dislike naming `default collation` and I think that we should switch to
something more descriptive for UCS_BASIC. After joins, my next PR will be to:
1) Rename `UCS_BASIC` -> `UTF8_BINARY` and `UCS_BASIC_LCASE` to
`UTF_BINARY_LCASE`.
2) Figure out default collation story. E.g. it should probably be something
like "isBinarySortable" or something like that.
But this PR is about aggregates so let's leave this for the next PR :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
