LuciferYang commented on code in PR #9265:
URL: https://github.com/apache/paimon/pull/9265#discussion_r4185039108
##########
paimon-spark/paimon-spark4-common/src/main/scala/org/apache/spark/sql/paimon/shims/Spark4Shim.scala:
##########
@@ -440,6 +491,84 @@ class Spark4Shim extends SparkShim {
parser: org.apache.spark.sql.catalyst.parser.ParserInterface):
Expression =
org.apache.paimon.spark.catalog.functions.SQLFunctionConverter
.toSQLFunctionExpression(funcIdent, function, arguments, parser)
+
+ // Spark 4.2 (SPARK-39660) removed partitionSpec from DescribeRelation;
DESCRIBE ... PARTITION is
+ // a separate DescribeTablePartition plan there.
+ override def createTableLikeParts(plan: LogicalPlan)
+ : Option[(Seq[String], Seq[String], Option[String], Option[String],
Map[String, String], Boolean, Boolean)] =
+ plan match {
+ case c: CreateTableLike =>
+ // Parser-stage rule: the children have not been analyzed yet.
+ (c.name, c.source) match {
+ case (target: UnresolvedIdentifier, source: UnresolvedTableOrView) =>
+ Some(
+ (
+ target.nameParts,
+ source.multipartIdentifier,
+ c.provider,
+ c.location,
+ c.properties,
+ c.ifNotExists,
+ // `STORED AS` lands in `serdeInfo`; Paimon tables cannot
honour Hive storage syntax.
+ c.serdeInfo.isDefined))
+ case _ => None
+ }
+ case _ => None
+ }
+
+ override def describeTablePartition(
+ plan: LogicalPlan): Option[(LogicalPlan, Map[String, String], Boolean,
Seq[Attribute])] =
+ plan match {
+ case d: DescribeTablePartition =>
+ (d.table, d.partitionSpec) match {
+ case (
+ r @ ResolvedTable(_, _, table: SupportsPartitionManagement, _),
+ spec: ResolvedPartitionSpec) =>
+ // `ResolvedPartitionSpec` holds the values as an `InternalRow`,
so read each field by
+ // its declared type and render it. Read the types from
`partitionSchema()`, the same
+ // schema `ResolvePartitionSpec` used to build `names` and
`ident`, so a name can never
+ // be missing. (Char/varchar is the one place the declared type
and the stored value
+ // differ: `convertToPartIdent` casts through
`replaceCharVarcharWithString`, leaving a
+ // plain `UTF8String` under a `CharType(n)` field. Harmless here —
`InternalRow.get`
+ // ignores the type argument for a `GenericInternalRow`,
`Literal`'s validation
+ // dispatches on the physical type, where `CharType` maps to
`PhysicalStringType` and
+ // accepts a `UTF8String`, and `Literal.toString` has no
char/varchar/string branch at
+ // all, so the value falls through to `other.toString`.)
+ //
+ // The rendering is `Literal.toString`, NOT what upstream's own
+ // `DescribeTablePartitionExec` uses (`ToPrettyString(...)` +
`escapePathName`), because
+ // the result is compared for equality against Paimon's
`Partition.spec()` rather than
+ // displayed. Note this rendering and `Partition.spec()`'s own do
not agree for every
+ // type: with `partition.legacy-name`
+ // (the default) Paimon stores `field.toString()`, so a DATE
column holds the epoch day
+ // while this renders `2021-01-01`. That mismatch predates Spark
4.2 — on <= 4.1 the
+ // parser produced the same `2021-01-01` via `Cast(literal,
StringType)`.
+ val partSchema = table.partitionSchema()
+ val values = spec.names.zipWithIndex.map {
+ case (name, i) =>
+ val field = partSchema(name)
+ val value = spec.ident.get(i, field.dataType)
+ name -> Literal(value, field.dataType).toString
Review Comment:
Fixed in 46fe2cc02c. On 4.2 the spec values arrive typed, so
`describeTablePartition` no longer renders them with `Literal.toString`. It
names the partition through `paimonPartitionSpec` instead, which goes through
the same `InternalRowPartitionComputer` path as DROP PARTITION. Besides the
small decimal, this fixes BINARY, NULL and legacy-name DATE partitions, which
failed the same way.
`DescribeTableTestBase` now has a DECIMAL(10,7) case for `0.0000001` under
both `partition.legacy-name` settings, with the spec value quoted and unquoted,
plus a BINARY case. The NULL and legacy DATE cases are in the 4.2
`DescribeTableTest` only, since older versions compare the raw spec string.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
