srielau commented on code in PR #58581:
URL: https://github.com/apache/spark/pull/58581#discussion_r4040213584
##########
sql/api/src/main/scala/org/apache/spark/sql/types/DataType.scala:
##########
@@ -425,12 +468,16 @@ object DataType {
("name", JString(name)),
("nullable", JBool(nullable)),
("type", dataType: JValue)) =>
- val collationsMap = getCollationsMap(metadataFields)
- val metadataWithoutCollations =
- JObject(metadataFields.filterNot(_._1 == COLLATIONS_METADATA_KEY))
+ val collationsMap = getCollationsMap(metadataFields,
COLLATIONS_METADATA_KEY)
+ val charVarcharCollationsMap =
+ getCollationsMap(metadataFields, CHAR_VARCHAR_COLLATIONS_METADATA_KEY)
Review Comment:
Addressed in 53e36d7254a. Both JVM UDT object branches now validate
__CHAR_VARCHAR_COLLATIONS against the outer field before UDT dispatch, without
traversing UDT.sqlType. Focused coverage verifies rejection for Scala/Java and
Python UDT JSON forms.
##########
python/pyspark/sql/types.py:
##########
@@ -2429,8 +2493,8 @@ def fromWKB(cls, wkb: bytes, srid: int) -> "Geometry":
"interval": CalendarIntervalType,
}
-_LENGTH_CHAR = re.compile(r"char\(\s*(\d+)\s*\)")
-_LENGTH_VARCHAR = re.compile(r"varchar\(\s*(\d+)\s*\)")
+_LENGTH_CHAR = re.compile(r"char\(\s*(\d+)\s*\)(?:\s+collate\s+(\w+))?")
Review Comment:
Addressed in 53e36d7254a. Python now uses fullmatch for standalone CHAR and
VARCHAR JSON parsing, with focused trailing-token rejection tests for both
types.
##########
python/pyspark/sql/types.py:
##########
@@ -324,47 +324,82 @@ def isUTF8BinaryCollation(self) -> bool:
class CharType(AtomicType):
- """Char data type
+ """Char data type.
+
+ A standalone collated ``CharType`` writes its collation inline in JSON and
therefore requires
Review Comment:
Addressed in 53e36d7254a. The CharType and VarcharType docs now state that
preceding Python readers may accept only the constrained-type prefix and
silently drop the inline collation, so a current reader is required for
preservation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
