dejankrak-db commented on code in PR #49772:
URL: https://github.com/apache/spark/pull/49772#discussion_r1951058776
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDDLCommandStringTypes.scala:
##########
@@ -156,21 +124,15 @@ object ResolveDefaultStringTypes extends
Rule[LogicalPlan] {
private def isDefaultStringType(dataType: DataType): Boolean = {
dataType match {
- case st: StringType =>
- // should only return true for StringType object and not
StringType("UTF8_BINARY")
- st.eq(StringType) || st.isInstanceOf[TemporaryStringType]
+ case st: StringType => st.eq(StringType)
case _ => false
}
}
private def replaceDefaultStringType(dataType: DataType, newType:
StringType): DataType = {
dataType.transformRecursively {
case currentType: StringType if isDefaultStringType(currentType) =>
Review Comment:
Actually, I discussed this with @stefankandic as well, we need to
additionally check if it is the default string type (which is the case for
StringType object, but not for StringType("UTF8_BINARY") as the comparison is
done through reference check).
The reason for this is the following: If table level collation specified is
e.g. UNICODE, but the customer specifies explicit UTF8_BINARY collation (which
is the default collation in case no collation is specified anywhere) for one of
the columns, then this column should preserve UTF8_BINARY collation as opposed
to replacing it with table-level UNICODE collation.
I have added clarifying comments throughout these methods to make sure this
is clear and well-explained, and I have also added further tests in
DefaultCollationTestSuite to specifically cover these cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
