JingsongLi commented on code in PR #9265:
URL: https://github.com/apache/paimon/pull/9265#discussion_r4183492462
##########
paimon-spark/paimon-spark4-common/src/main/scala/org/apache/spark/sql/paimon/shims/Spark4Shim.scala:
##########
@@ -440,6 +491,84 @@ class Spark4Shim extends SparkShim {
parser: org.apache.spark.sql.catalyst.parser.ParserInterface):
Expression =
org.apache.paimon.spark.catalog.functions.SQLFunctionConverter
.toSQLFunctionExpression(funcIdent, function, arguments, parser)
+
+ // Spark 4.2 (SPARK-39660) removed partitionSpec from DescribeRelation;
DESCRIBE ... PARTITION is
+ // a separate DescribeTablePartition plan there.
+ override def createTableLikeParts(plan: LogicalPlan)
+ : Option[(Seq[String], Seq[String], Option[String], Option[String],
Map[String, String], Boolean, Boolean)] =
+ plan match {
+ case c: CreateTableLike =>
+ // Parser-stage rule: the children have not been analyzed yet.
+ (c.name, c.source) match {
+ case (target: UnresolvedIdentifier, source: UnresolvedTableOrView) =>
+ Some(
+ (
+ target.nameParts,
+ source.multipartIdentifier,
+ c.provider,
+ c.location,
+ c.properties,
+ c.ifNotExists,
+ // `STORED AS` lands in `serdeInfo`; Paimon tables cannot
honour Hive storage syntax.
+ c.serdeInfo.isDefined))
+ case _ => None
+ }
+ case _ => None
+ }
+
+ override def describeTablePartition(
+ plan: LogicalPlan): Option[(LogicalPlan, Map[String, String], Boolean,
Seq[Attribute])] =
+ plan match {
+ case d: DescribeTablePartition =>
+ (d.table, d.partitionSpec) match {
+ case (
+ r @ ResolvedTable(_, _, table: SupportsPartitionManagement, _),
+ spec: ResolvedPartitionSpec) =>
+ // `ResolvedPartitionSpec` holds the values as an `InternalRow`,
so read each field by
+ // its declared type and render it. Read the types from
`partitionSchema()`, the same
+ // schema `ResolvePartitionSpec` used to build `names` and
`ident`, so a name can never
+ // be missing. (Char/varchar is the one place the declared type
and the stored value
+ // differ: `convertToPartIdent` casts through
`replaceCharVarcharWithString`, leaving a
+ // plain `UTF8String` under a `CharType(n)` field. Harmless here —
`InternalRow.get`
+ // ignores the type argument for a `GenericInternalRow`,
`Literal`'s validation
+ // dispatches on the physical type, where `CharType` maps to
`PhysicalStringType` and
+ // accepts a `UTF8String`, and `Literal.toString` has no
char/varchar/string branch at
+ // all, so the value falls through to `other.toString`.)
+ //
+ // The rendering is `Literal.toString`, NOT what upstream's own
+ // `DescribeTablePartitionExec` uses (`ToPrettyString(...)` +
`escapePathName`), because
+ // the result is compared for equality against Paimon's
`Partition.spec()` rather than
+ // displayed. Note this rendering and `Partition.spec()`'s own do
not agree for every
+ // type: with `partition.legacy-name`
+ // (the default) Paimon stores `field.toString()`, so a DATE
column holds the epoch day
+ // while this renders `2021-01-01`. That mismatch predates Spark
4.2 — on <= 4.1 the
+ // parser produced the same `2021-01-01` via `Cast(literal,
StringType)`.
+ val partSchema = table.partitionSchema()
+ val values = spec.names.zipWithIndex.map {
+ case (name, i) =>
+ val field = partSchema(name)
+ val value = spec.ident.get(i, field.dataType)
+ name -> Literal(value, field.dataType).toString
Review Comment:
[P2] Render DECIMAL partition values without scientific notation
`Literal(value, field.dataType).toString` renders DECIMAL(10,7) value
`0.0000001` as `1E-7`, while Paimon stores its partition spec as `0.0000001`.
In a real Parquet-backed partitioned table, INSERT/SELECT/SHOW PARTITIONS
succeed, but `DESCRIBE FORMATTED t PARTITION (p=0.0000001)` now throws `Found 0
matching partitions`. I reproduced this on Spark 4.2.0 with default ANSI=true
and both partition.legacy-name settings; the same public SQL succeeds on Spark
4.1.2, and value `1.0000000` succeeds on both runtimes. The old parser path
used a string cast that preserves plain decimal formatting. Please render this
new typed path consistently with Partition.spec (for decimals, toPlainString),
and add a small-decimal DESCRIBE PARTITION regression case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
