dbatomic commented on code in PR #45592:
URL: https://github.com/apache/spark/pull/45592#discussion_r1531933608
##########
sql/api/src/main/scala/org/apache/spark/sql/types/StringType.scala:
##########
@@ -41,12 +37,22 @@ class StringType private(val collationId: Int) extends
AtomicType with Serializa
*/
def isBinaryCollation: Boolean =
CollationFactory.fetchCollation(collationId).isBinaryCollation
+ /**
+ * Spark internal collation implies that strings are considered equal only
if they are
+ * byte for byte equal. E.g. all accent or case-insensitive collations are
considered non-binary.
+ * Also their comparison does not require ICU library calls, as ordering
follows
+ * spark internal implementation. If this field is true, byte level
operations can be
+ * used against this datatype (e.g. for equality, hashing and sorting).
+ */
+ def isUTF8BinaryCollation: Boolean =
+ collationId == CollationFactory.UTF8_BINARY_COLLATION_ID
+
/**
* Type name that is shown to the customer.
* If this is an UTF8_BINARY collation output is `string` due to backwards
compatibility.
*/
override def typeName: String =
- if (isDefaultCollation) "string"
+ if (isDefaultCollation(collationId)) "string"
Review Comment:
I think that this is not ok. We should keep calling `UTF8_BINARY` as
`string` regardless of default collation.
Typename should remain the same regardless of current collation. Especially
since AFAIK we can serialize this typeName.
This should be `isUTF8BinaryCollation` check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
