JingsongLi commented on code in PR #9265:
URL: https://github.com/apache/paimon/pull/9265#discussion_r4284314105
##########
paimon-spark/paimon-spark4-common/src/main/scala/org/apache/spark/sql/paimon/shims/Spark4Shim.scala:
##########
@@ -440,6 +491,84 @@ class Spark4Shim extends SparkShim {
parser: org.apache.spark.sql.catalyst.parser.ParserInterface):
Expression =
org.apache.paimon.spark.catalog.functions.SQLFunctionConverter
.toSQLFunctionExpression(funcIdent, function, arguments, parser)
+
+ // Spark 4.2 (SPARK-39660) removed partitionSpec from DescribeRelation;
DESCRIBE ... PARTITION is
+ // a separate DescribeTablePartition plan there.
+ override def createTableLikeParts(plan: LogicalPlan)
+ : Option[(Seq[String], Seq[String], Option[String], Option[String],
Map[String, String], Boolean, Boolean)] =
+ plan match {
+ case c: CreateTableLike =>
+ // Parser-stage rule: the children have not been analyzed yet.
+ (c.name, c.source) match {
+ case (target: UnresolvedIdentifier, source: UnresolvedTableOrView) =>
+ Some(
+ (
+ target.nameParts,
+ source.multipartIdentifier,
+ c.provider,
+ c.location,
+ c.properties,
+ c.ifNotExists,
+ // `STORED AS` lands in `serdeInfo`; Paimon tables cannot
honour Hive storage syntax.
+ c.serdeInfo.isDefined))
+ case _ => None
+ }
+ case _ => None
+ }
+
+ override def describeTablePartition(
+ plan: LogicalPlan): Option[(LogicalPlan, Map[String, String], Boolean,
Seq[Attribute])] =
+ plan match {
+ case d: DescribeTablePartition =>
+ (d.table, d.partitionSpec) match {
+ case (
+ r @ ResolvedTable(_, _, table: SupportsPartitionManagement, _),
+ spec: ResolvedPartitionSpec) =>
+ // `ResolvedPartitionSpec` holds the values as an `InternalRow`,
so read each field by
+ // its declared type and render it. Read the types from
`partitionSchema()`, the same
+ // schema `ResolvePartitionSpec` used to build `names` and
`ident`, so a name can never
+ // be missing. (Char/varchar is the one place the declared type
and the stored value
+ // differ: `convertToPartIdent` casts through
`replaceCharVarcharWithString`, leaving a
+ // plain `UTF8String` under a `CharType(n)` field. Harmless here —
`InternalRow.get`
+ // ignores the type argument for a `GenericInternalRow`,
`Literal`'s validation
+ // dispatches on the physical type, where `CharType` maps to
`PhysicalStringType` and
+ // accepts a `UTF8String`, and `Literal.toString` has no
char/varchar/string branch at
+ // all, so the value falls through to `other.toString`.)
+ //
+ // The rendering is `Literal.toString`, NOT what upstream's own
+ // `DescribeTablePartitionExec` uses (`ToPrettyString(...)` +
`escapePathName`), because
+ // the result is compared for equality against Paimon's
`Partition.spec()` rather than
+ // displayed. Note this rendering and `Partition.spec()`'s own do
not agree for every
+ // type: with `partition.legacy-name`
+ // (the default) Paimon stores `field.toString()`, so a DATE
column holds the epoch day
+ // while this renders `2021-01-01`. That mismatch predates Spark
4.2 — on <= 4.1 the
+ // parser produced the same `2021-01-01` via `Cast(literal,
StringType)`.
+ val partSchema = table.partitionSchema()
+ val values = spec.names.zipWithIndex.map {
+ case (name, i) =>
+ val field = partSchema(name)
+ val value = spec.ident.get(i, field.dataType)
+ name -> Literal(value, field.dataType).toString
Review Comment:
Verified fixed on 386e72db27. I independently rebuilt and ran the
DescribeTable suites on Spark 4.0.3, 4.1.2 and 4.2.0; the small DECIMAL(10,7)
0.0000001 cases pass under both partition.legacy-name settings with
quoted/unquoted specs, and BINARY passes as well. The new 4.2 NULL/legacy-DATE
cases also pass. Resolving this finding.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
