srielau commented on code in PR #58581:
URL: https://github.com/apache/spark/pull/58581#discussion_r3969374694
##########
sql/connect/common/src/main/scala/org/apache/spark/sql/connect/SparkSession.scala:
##########
@@ -223,7 +223,9 @@ class SparkSession private[sql] (
/** @inheritdoc */
def createDataFrame(rows: java.util.List[Row], schema: StructType):
DataFrame = {
- createDataset(RowEncoder.encoderFor(schema),
rows.iterator().asScala).toDF()
+ // The client cannot observe the server's CHAR/VARCHAR configuration.
Encode an explicitly
Review Comment:
Updated in e19b586586d. The comment now distinguishes the client process
local SqlApiConf consulted by RowEncoder from the server configuration
observable through SparkSession.conf.
##########
python/pyspark/sql/types.py:
##########
@@ -1366,10 +1390,15 @@ def processDataType(dt: DataType, fieldPath: str) ->
None:
return collationMetadata
def _isCollatedString(self, dt: DataType) -> bool:
- return isinstance(dt, StringType) and not dt.isUTF8BinaryCollation()
+ if isinstance(dt, StringType):
+ return not dt.isUTF8BinaryCollation()
+ if isinstance(dt, (CharType, VarcharType)):
+ return not dt.isUTF8BinaryCollation()
Review Comment:
Addressed in e19b586586d. The constrained types are handled explicitly
without changing their inheritance, and exact JSON-shape coverage now includes
omitted, explicit UTF8_BINARY, and non-binary collations.
##########
python/pyspark/sql/types.py:
##########
@@ -1366,10 +1390,15 @@ def processDataType(dt: DataType, fieldPath: str) ->
None:
return collationMetadata
def _isCollatedString(self, dt: DataType) -> bool:
- return isinstance(dt, StringType) and not dt.isUTF8BinaryCollation()
+ if isinstance(dt, StringType):
Review Comment:
Fixed in e19b586586d. Python now checks CharType/VarcharType before
StringType, and the recursive metadata visitor handles constrained string
leaves directly. I also ordered collation removal so the constrained types
retain their lengths. The focused Python and Scala tests pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
