cloud-fan commented on code in PR #58581:
URL: https://github.com/apache/spark/pull/58581#discussion_r4015081820
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala:
##########
@@ -1641,9 +1647,35 @@ class SparkConnectPlanner(
.asInstanceOf[Project]
val proj = UnsafeProjection.create(project.projectList,
project.child.output)
- logical.LocalRelation(
- DataTypeUtils.toAttributes(schema),
- data.map(proj).map(_.copy()).toSeq)
+ def restoreFieldNames(actual: DataType, requested: DataType): DataType
=
+ (actual, requested) match {
+ case (_, requestedUdt: UserDefinedType[_]) => requestedUdt
+ case (StructType(actualFields), StructType(requestedFields)) =>
+ StructType(
+ actualFields.zip(requestedFields).map { case (actualField,
requestedField) =>
+ actualField.copy(
+ name = requestedField.name,
+ dataType = restoreFieldNames(actualField.dataType,
requestedField.dataType))
+ })
+ case (ArrayType(actualElement, containsNull),
ArrayType(requestedElement, _)) =>
+ ArrayType(restoreFieldNames(actualElement, requestedElement),
containsNull)
+ case (
+ MapType(actualKey, actualValue, valueContainsNull),
+ MapType(requestedKey, requestedValue, _)) =>
+ MapType(
+ restoreFieldNames(actualKey, requestedKey),
+ restoreFieldNames(actualValue, requestedValue),
+ valueContainsNull)
+ case _ => actual
+ }
+ val output = project.output.zip(schema.fields).map { case (attribute,
field) =>
+ AttributeReference(
+ field.name,
+ restoreFieldNames(attribute.dataType, field.dataType),
Review Comment:
**Blocking (P1):** For a requested explicit `UTF8_BINARY` `StringType`,
Arrow produces an implicit `StringType`, and `Dataset.to` treats the two as
equal. This fallback then keeps the analyzed leaf, so the returned
local-relation schema silently loses the explicit-collation marker; the
merge-target path instead built output attributes from the requested schema.
Could this retain the requested `StringType` instance when reconciliation has
not selected a meaningfully different leaf, while continuing to keep the actual
type when CHAR/VARCHAR policy changes it?
##########
python/pyspark/sql/tests/connect/test_connect_basic.py:
##########
@@ -491,17 +491,28 @@ def test_schema(self):
self._check_print_schema(query)
def test_char_varchar_result_schema(self):
- # SPARK-58794: Python Connect maps first-class CHAR/VARCHAR the same
as classic.
- query = "SELECT CAST('ab' AS CHAR(4)) AS c, CAST('cd' AS VARCHAR(6))
AS v"
+ # SPARK-59276: Python Connect maps first-class CHAR/VARCHAR the same
as classic.
Review Comment:
This remains present on the pinned head: populated Python local data still
reaches Arrow with CharType/VarcharType and fails before the server can apply
its policy. I am keeping this thread open.
<!-- SPARK_DEV_REVIEW_REPLY
{"feedback_id":"inline:4010878638","thread_id":"inline:4010878638","verdict_sha256":"6e770e1bc8bf3d39d4eabf8c3a66d05e702826f18203eaeb664606fe7d74e73e"}
-->
##########
sql/api/src/main/scala/org/apache/spark/sql/types/StructField.scala:
##########
@@ -137,6 +137,8 @@ case class StructField(
}
private def isCollatedString(dt: DataType): Boolean = dt match {
+ case c: CharType => c.collation.isDefined
Review Comment:
This is still unresolved on the pinned head: the writer emits __COLLATIONS
beside char(n)/varchar(n), which the preceding JVM reader rejects. I am keeping
this compatibility thread open.
<!-- SPARK_DEV_REVIEW_REPLY
{"feedback_id":"inline:4010878643","thread_id":"inline:4010878643","verdict_sha256":"6e770e1bc8bf3d39d4eabf8c3a66d05e702826f18203eaeb664606fe7d74e73e"}
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
