cloud-fan commented on code in PR #58581:
URL: https://github.com/apache/spark/pull/58581#discussion_r3962398463
##########
python/pyspark/sql/types.py:
##########
@@ -1366,10 +1390,15 @@ def processDataType(dt: DataType, fieldPath: str) ->
None:
return collationMetadata
def _isCollatedString(self, dt: DataType) -> bool:
- return isinstance(dt, StringType) and not dt.isUTF8BinaryCollation()
+ if isinstance(dt, StringType):
Review Comment:
**Blocking (P1):** `CharType` and `VarcharType` inherit directly from
`AtomicType`, so this inner branch can never run under the `StringType` guard.
`StructField.jsonValue()` consequently emits inline `collate` suffixes and no
`__COLLATIONS` metadata; older schema-JSON readers cannot consume that
representation even though the same-version round trip passes. Please keep this
fix local: making them Python `StringType` subclasses would broaden every
`isinstance(..., StringType)` dispatch, and the current JSON-removal branch
checks generic `StringType` first, so it would erase CHAR/VARCHAR lengths.
Check the constrained types explicitly before generic `StringType` and let
direct and nested leaves enter the metadata visitor.
##########
sql/connect/common/src/main/scala/org/apache/spark/sql/connect/SparkSession.scala:
##########
@@ -223,7 +223,9 @@ class SparkSession private[sql] (
/** @inheritdoc */
def createDataFrame(rows: java.util.List[Row], schema: StructType):
DataFrame = {
- createDataset(RowEncoder.encoderFor(schema),
rows.iterator().asScala).toDF()
+ // The client cannot observe the server's CHAR/VARCHAR configuration.
Encode an explicitly
Review Comment:
**Nit (P3):** `SparkSession.conf` is backed by `SparkConnectClient`, and
`RuntimeConfig.get` sends a `ConfigRequest` to the server, so the client can
observe this setting. The actual boundary is that `RowEncoder.encoderFor`
consults the client process's local `SqlApiConf`, while an explicit schema must
be encoded independently of it. Please reword the comment around that
distinction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
