This is an automated email from the ASF dual-hosted git repository.
ulysses-you pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 279946b0811f [SPARK-60859][SQL] Show KeyedPartitioning partition keys
in plan strings
279946b0811f is described below
commit 279946b0811facbf8b89dbe4155f7edc50b29de0
Author: Xiduo You <[email protected]>
AuthorDate: Fri Nov 6 16:28:10 2026 +0800
[SPARK-60859][SQL] Show KeyedPartitioning partition keys in plan strings
### What changes were proposed in this pull request?
- `InternalRowComparableWrapper.toString` renders a key by its type at
every level. Atomic values go through the `Literal(value, dataType)`
constructor, so dates, timestamps, times, decimals and binary print as values,
and a CHAR/VARCHAR key prints its stored value without the write-side check.
Struct, array and map values are walked by their field types, and a UDT is
rendered at its SQL type.
- `KeyedPartitioning` overrides `toString`/`simpleString` to summarize the
partition key list. A keyed partitioning can hold one key per split, so the
keys take the fields that `maxFields` leaves after the expressions and the
trailing `isGrouped`, `isCollapsed` and origin fields; both the key list and
the argument list are rendered with `truncatedString`, so at most one `...`
appears.
- When `maxFields` leaves no room for the key summary, the summary is
dropped and the fields around it print; below that budget the argument list is
truncated from the end.
### Why are the changes needed?
Keyed partitioning plan strings showed no key values, and one key per split
would flood the plan.
### Does this PR introduce _any_ user-facing change?
Yes, physical plan strings change. With
`spark.sql.debug.maxToStringFields=10` and 12 keys:
Before: `keyedpartitioning(x#0, InternalRowComparableWrapper1a2b3c4d, ...)`
(object identities, truncated by the argument count)
After: `keyedpartitioning(x#0, [[1], [2], [3], [4], [5], [6], [7], ... 5
more fields], true, false)`
Nested values inside a complex key render as values too, e.g. a `struct<d:
date, b: binary>` key prints `[2024-01-02,0x0102]`.
No query result or execution behavior changes.
### How was this patch tested?
New unit tests:
- `SPARK-60859: a wrapper prints scalar key types readably` covers BOOLEAN,
BYTE, SHORT, INT, LONG, FLOAT, DOUBLE, DECIMAL, STRING (including a null),
CHAR, VARCHAR, BINARY, DATE, TIMESTAMP, TIMESTAMP_NTZ, the nanosecond timestamp
types and the two interval types, through a generic row and an `UnsafeRow`; it
also covers two collated keys that compare equal and print their own case.
- `SPARK-60859: a wrapper prints CHAR/VARCHAR keys as stored` covers both
first-class char/varchar settings, an unpadded shorter CHAR and an over-length
VARCHAR.
- `SPARK-60859: a wrapper prints complex key types by their field types`
covers struct, array, array-of-struct and map keys with nested DATE, TIMESTAMP
and BINARY values, through generic and Unsafe-backed rows, plus a UDT over an
array.
- `SPARK-60859: a KeyedPartitioning plan string summarizes the partition
keys` covers the key budget with one and several expressions, both origins, an
exact fit, an empty key list, a deserialized partitioning, and the
`simpleString` budget.
```
build/sbt 'catalyst/testOnly *InternalRowComparableWrapperSuite
*DistributionSuite' # 31 passed
build/sbt 'sql/testOnly *GroupPartitionsExecSuite *EnsureRequirementsSuite
*KeyGroupedPartitioningSuite *DataSourceV2CatalystRuntimeFilterSuite
*DSv2PlanMergingSuite *ProjectedOrderingAndPartitioningSuite
*ValidateRequirementsSuite *DataFrameSetOperationsSuite' # 504 passed
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Closes #60078 from ulysses-you/keyed-partitioning-plan-string.
Authored-by: Xiduo You <[email protected]>
Signed-off-by: Xiduo You <[email protected]>
---
.../sql/catalyst/plans/physical/partitioning.scala | 38 ++++++---
.../util/InternalRowComparableWrapper.scala | 45 ++++++++++-
.../spark/sql/catalyst/DistributionSuite.scala | 72 ++++++++++++++++-
.../util/InternalRowComparableWrapperSuite.scala | 93 +++++++++++++++++++++-
4 files changed, 231 insertions(+), 17 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/physical/partitioning.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/physical/partitioning.scala
index f4d77b0b51dc..9346ef295c57 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/physical/partitioning.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/physical/partitioning.scala
@@ -27,7 +27,7 @@ import org.apache.spark.{SparkException,
SparkUnsupportedOperationException}
import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.expressions._
import org.apache.spark.sql.catalyst.plans.QueryPlan
-import org.apache.spark.sql.catalyst.util.InternalRowComparableWrapper
+import org.apache.spark.sql.catalyst.util.{truncatedString,
InternalRowComparableWrapper}
import org.apache.spark.sql.connector.catalog.functions.Reducer
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.types.{DataType, IntegerType}
@@ -748,17 +748,33 @@ case class KeyedPartitioning(
override def dataType: DataType = IntegerType
/**
- * Prints the layout's contents where the value object would print, which
keeps the plan string as
- * it was before the layout held them. The list is curated rather than the
layout's own fields,
- * for two reasons. `partitionKeys` has to be printed as a `Seq` for
`maxFields` to truncate it,
- * and a partitioning can hold one key per split. And `dataTypes` has its
naming erased, so
- * printing it would put a struct field name into a plan that appears
nowhere in the query. A
- * field the layout grows is therefore a decision here, and the default is
to print it: the plan
- * string is the only place a reader sees why a marked partitioning still
shuffles.
+ * Prints the layout as the partition keys followed by the flags and
origins. The keys are
+ * summarized once they exceed the fields that `maxFields` leaves after the
expressions and the
+ * trailing fields; when those fields alone exceed the budget, the argument
list is truncated
+ * from the end as usual. `dataTypes` is left out because its naming is
erased, so printing it
+ * would put a struct field name into a plan that appears nowhere in the
query. The plan string
+ * is the only place a reader sees why a marked partitioning still shuffles.
Keys print as
+ * values, so two keys that `equals` considers equal can print differently,
e.g. `'a'` and `'A'`
+ * under `UTF8_LCASE`.
*/
- override protected def stringArgs: Iterator[Any] =
- Iterator(expressions, partitionKeys, isGrouped, isCollapsed) ++
unknownKeysOrigin ++
- ungroupingOrigin
+ override def toString: String = simpleString(SQLConf.get.maxToStringFields)
+
+ override def simpleString(maxFields: Int): String = {
+ val trailing = Seq[Any](isGrouped, isCollapsed) ++ unknownKeysOrigin ++
ungroupingOrigin
+ // The keys take the fields that the expressions and the trailing flags
leave.
+ val keyBudget = maxFields - expressions.size - trailing.size
+ val keys = if (keyBudget < 1) {
+ Iterator.empty
+ } else if (partitionKeys == null) {
+ Iterator.single("null")
+ } else if (partitionKeys.isEmpty) {
+ Iterator.empty
+ } else {
+ Iterator.single(truncatedString(partitionKeys, "[", ", ", "]",
keyBudget))
+ }
+ val args = expressions.map(_.toString) ++ keys ++ trailing.map(_.toString)
+ s"$prettyName" + truncatedString(args, "(", ", ", ")", maxFields)
+ }
override protected def withNewChildrenInternal(
newChildren: IndexedSeq[Expression]): KeyedPartitioning =
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/InternalRowComparableWrapper.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/InternalRowComparableWrapper.scala
index d6b841f1dafc..eeff4a529764 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/InternalRowComparableWrapper.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/InternalRowComparableWrapper.scala
@@ -18,9 +18,11 @@
package org.apache.spark.sql.catalyst.util
import org.apache.spark.sql.catalyst.InternalRow
-import org.apache.spark.sql.catalyst.expressions.{BaseOrdering, Expression,
Murmur3HashFunction, RowOrdering}
+import org.apache.spark.sql.catalyst.expressions.{
+ BaseOrdering, Expression, Literal, Murmur3HashFunction, RowOrdering}
import org.apache.spark.sql.connector.read.{HasPartitionKey, InputPartition}
-import org.apache.spark.sql.types.{DataType, StructField, StructType}
+import org.apache.spark.sql.types.{
+ ArrayType, DataType, MapType, StructField, StructType, UserDefinedType}
import org.apache.spark.util.NonFateSharingCache
/**
@@ -65,6 +67,11 @@ class InternalRowComparableWrapper private (
_ordering
}
+ override def toString: String =
+ row.toSeq(dataTypes).zip(dataTypes)
+ .map { case (value, dataType) =>
InternalRowComparableWrapper.render(value, dataType) }
+ .mkString("[", ",", "]")
+
override def hashCode(): Int = Murmur3HashFunction.hash(
row,
structType,
@@ -114,6 +121,40 @@ object InternalRowComparableWrapper {
getInternalRowComparableWrapperFactory(partitionExpression.map(_.dataType))(partitionRow)
}
+ /**
+ * Renders a value the way a plan string shows it: by the value's type at
every level. Atomic
+ * values go through the [[Literal]] constructor, whose `toString` formats
dates, timestamps,
+ * decimals and binary; complex values are walked by type, because their own
`toString` shows
+ * internal encodings, hex words or an object identity. The constructor is
used rather than
+ * `Literal.create`, which runs the char/varchar write-side check and can
reject a stored value
+ * while a plan is being printed.
+ */
+ private def render(value: Any, dataType: DataType): String = {
+ if (value == null) {
+ "null"
+ } else {
+ dataType match {
+ case s: StructType =>
+ val r = value.asInstanceOf[InternalRow]
+ s.fields.iterator.zipWithIndex
+ .map { case (field, i) => render(r.get(i, field.dataType),
field.dataType) }
+ .mkString("[", ",", "]")
+ case ArrayType(elementType, _) =>
+ val a = value.asInstanceOf[ArrayData]
+ (0 until a.numElements()).map(i => render(a.get(i, elementType),
elementType))
+ .mkString("[", ",", "]")
+ case MapType(keyType, valueType, _) =>
+ val m = value.asInstanceOf[MapData]
+ (0 until m.numElements()).map { i =>
+ s"${render(m.keyArray().get(i, keyType), keyType)} -> " +
+ render(m.valueArray().get(i, valueType), valueType)
+ }.mkString("{", ",", "}")
+ case udt: UserDefinedType[_] => render(value, udt.sqlType)
+ case _ => Literal(value, dataType).toString
+ }
+ }
+ }
+
/**
* The types a row is compared at, which is the given types with their
naming erased: struct field
* names and every nullability go, and nothing else does.
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/DistributionSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/DistributionSuite.scala
index da20e40e043b..472261c59035 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/DistributionSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/DistributionSuite.scala
@@ -17,7 +17,8 @@
package org.apache.spark.sql.catalyst
-import org.apache.spark.SparkFunSuite
+import org.apache.spark.{SparkConf, SparkFunSuite}
+import org.apache.spark.serializer.JavaSerializer
/* Implicit conversions */
import org.apache.spark.sql.catalyst.dsl.expressions._
import org.apache.spark.sql.catalyst.expressions.{Ascending,
AttributeReference, CollationAwareMurmur3Hash, Expression, Literal, Pmod,
SortOrder, TransformExpression}
@@ -508,6 +509,75 @@ class DistributionSuite extends SparkFunSuite with
SQLHelper {
assert(interned.layout eq kpX.layout, "the whole layout is interned, keys
and all")
}
+ test("SPARK-60859: a KeyedPartitioning plan string summarizes the partition
keys") {
+ val x = AttributeReference("x", IntegerType)()
+ val keys = (1 to 12).map(InternalRow(_))
+
+ withSQLConf(SQLConf.MAX_TO_STRING_FIELDS.key -> "10") {
+ // One expression and the two flags leave the keys seven of the ten
slots.
+ assert(KeyedPartitioning(Seq(x), keys).toString ==
+ s"keyedpartitioning($x, [[1], [2], [3], [4], [5], [6], [7], ... 5 more
fields], " +
+ "true, false)")
+
+ // A list that fits is printed whole.
+ assert(KeyedPartitioning(Seq(x), keys.take(7)).toString ==
+ s"keyedpartitioning($x, [[1], [2], [3], [4], [5], [6], [7]], true,
false)")
+
+ assert(KeyedPartitioning(Seq(x), keys.take(8)).toString ==
+ s"keyedpartitioning($x, [[1], [2], [3], [4], [5], [6], [7], ... 1 more
fields], " +
+ "true, false)")
+
+ // An origin after the flags takes a slot from the keys in turn.
+ val marked = KeyedPartitioning(Seq(x), keys)
+ .withLayout(_.copy(unknownKeysOrigin = Some(HASH_ROUTING)))
+ assert(marked.toString ==
+ s"keyedpartitioning($x, [[1], [2], [3], [4], [5], [6], ... 6 more
fields], " +
+ "true, false, HASH_ROUTING)")
+
+ // Both origins take a slot each.
+ val stamped = marked.withLayout(_.copy(ungroupingOrigin =
Some(SPLIT_FOR_JOIN)))
+ assert(stamped.toString ==
+ s"keyedpartitioning($x, [[1], [2], [3], [4], [5], ... 7 more fields],
" +
+ "true, false, HASH_ROUTING, SPLIT_FOR_JOIN)")
+
+ // The expressions take a slot each too.
+ val y = AttributeReference("y", IntegerType)()
+ val z = AttributeReference("z", IntegerType)()
+ val composite = (1 to 12).map(i => InternalRow(i, i, i))
+ assert(KeyedPartitioning(Seq(x, y, z), composite).toString ==
+ s"keyedpartitioning($x, $y, $z, [[1,1,1], [2,2,2], [3,3,3], [4,4,4],
[5,5,5], " +
+ "... 7 more fields], true, false)")
+
+ // An empty key list prints nothing, as before.
+ assert(KeyedPartitioning(Seq(x), Seq.empty).toString ==
+ s"keyedpartitioning($x, true, false)")
+ }
+
+ // With no slot left for the keys, the fields after them win.
+ withSQLConf(SQLConf.MAX_TO_STRING_FIELDS.key -> "3") {
+ assert(KeyedPartitioning(Seq(x), keys).toString ==
+ s"keyedpartitioning($x, true, false)")
+ }
+
+ // `simpleString` takes the whole field budget from its argument: the
expression and the two
+ // flags come first, and the keys get what is left.
+ val plain = KeyedPartitioning(Seq(x), keys)
+ assert(plain.simpleString(3) == s"keyedpartitioning($x, true, false)")
+ assert(plain.simpleString(4) ==
+ s"keyedpartitioning($x, [[1], ... 11 more fields], true, false)")
+ assert(plain.simpleString(5) ==
+ s"keyedpartitioning($x, [[1], [2], ... 10 more fields], true, false)")
+
+ // Below the fields around the keys, the budget cuts the trailing fields
too.
+ assert(plain.simpleString(2) == s"keyedpartitioning($x, true, ... 1 more
fields)")
+
+ // A partitioning that crossed the wire has no key rows, and printing it
must not throw.
+ val serializer = new JavaSerializer(new SparkConf()).newInstance()
+ val restored = serializer.deserialize[KeyedPartitioning](
+ serializer.serialize(KeyedPartitioning(Seq(x), keys)))
+ assert(restored.toString == s"keyedpartitioning($x, null, true, false)")
+ }
+
test("SPARK-59050: a marked one-partition layout keeps the global ordering
claim") {
// The one-partition exemption inside `keysSatisfy`'s ordered branch: a
single partition
// holds every row, so an out-of-set key cannot break the cross-partition
sequence, while
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/InternalRowComparableWrapperSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/InternalRowComparableWrapperSuite.scala
index d6e64868638f..ca4d86211796 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/InternalRowComparableWrapperSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/InternalRowComparableWrapperSuite.scala
@@ -20,12 +20,14 @@ package org.apache.spark.sql.catalyst.util
import java.util.UUID
import org.apache.spark.SparkFunSuite
-import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.{InternalRow, SQLConfHelper}
+import org.apache.spark.sql.catalyst.expressions.UnsafeProjection
import org.apache.spark.sql.catalyst.plans.physical.KeyedPartitioning
+import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.types._
-import org.apache.spark.unsafe.types.UTF8String
+import org.apache.spark.unsafe.types.{TimestampNanosVal, UTF8String}
-class InternalRowComparableWrapperSuite extends SparkFunSuite {
+class InternalRowComparableWrapperSuite extends SparkFunSuite with
SQLConfHelper {
private val structA = new StructType().add("a", IntegerType)
private val structB = new StructType().add("b", IntegerType)
@@ -153,4 +155,89 @@ class InternalRowComparableWrapperSuite extends
SparkFunSuite {
assert(wrap(IntegerType, InternalRow(1)) !== wrap(LongType,
InternalRow(1L)),
"and a type that decides where a value belongs still tells them apart")
}
+
+ test("SPARK-60859: a wrapper prints scalar key types readably") {
+ val dataTypes: Seq[DataType] = Seq(
+ BooleanType, ByteType, ShortType, IntegerType, LongType, FloatType,
DoubleType,
+ DecimalType(10, 2), StringType, StringType, CharType(5), VarcharType(5),
BinaryType,
+ DateType, TimestampType, TimestampNTZType, TimestampNTZNanosType(9),
+ TimestampLTZNanosType(9), TimeType(), YearMonthIntervalType(),
DayTimeIntervalType())
+ val nanos = new TimestampNanosVal(1704164645000000L, 123.toShort)
+ val row = InternalRow(
+ true, 1.toByte, 2.toShort, 3, 4L, 5.5f, 6.5,
Decimal(BigDecimal("12.34"), 10, 2),
+ UTF8String.fromString("ab"), null, UTF8String.fromString("cd"),
+ UTF8String.fromString("ef"), Array[Byte](1, 2),
+ 19724, 1704164645000000L, 1704164645000000L, nanos, nanos,
11045000000000L,
+ 13, 90061000000L)
+ val expected = "[true,1,2,3,4,5.5,6.5,12.34,ab,null,cd,ef,0x0102," +
+ "2024-01-02,2024-01-02 03:04:05,2024-01-02 03:04:05," +
+ "2024-01-02 03:04:05.000000123,2024-01-02 03:04:05.000000123,03:04:05," +
+ "INTERVAL '1-1' YEAR TO MONTH,INTERVAL '1 01:01:01' DAY TO SECOND]"
+ val factory =
InternalRowComparableWrapper.getInternalRowComparableWrapperFactory(dataTypes)
+ withSQLConf(SQLConf.SESSION_LOCAL_TIMEZONE.key -> "UTC") {
+ assert(factory(row).toString === expected)
+ // UnsafeRow's own toString is a hex dump of its words, so the wrapper
has to read the values.
+ assert(factory(UnsafeProjection.create(dataTypes.toArray)(row)).toString
=== expected)
+ }
+
+ // Two collated keys that compare equal still print their own case.
+ val collated = InternalRowComparableWrapper
+ .getInternalRowComparableWrapperFactory(Seq(StringType("UTF8_LCASE")))
+ val upper = collated(InternalRow(UTF8String.fromString("A")))
+ val lower = collated(InternalRow(UTF8String.fromString("a")))
+ assert(upper === lower)
+ assert(upper.toString === "[A]")
+ assert(lower.toString === "[a]")
+ }
+
+ test("SPARK-60859: a wrapper prints CHAR/VARCHAR keys as stored") {
+ val factory = InternalRowComparableWrapper
+ .getInternalRowComparableWrapperFactory(Seq(CharType(5), VarcharType(5)))
+ // A shorter CHAR and an over-length VARCHAR: printing must neither pad
nor throw.
+ val row = InternalRow(UTF8String.fromString("ab"),
UTF8String.fromString("abcdefgh"))
+ assert(factory(row).toString === "[ab,abcdefgh]")
+ Seq(SQLConf.PRESERVE_CHAR_VARCHAR_TYPE_INFO.key,
+ SQLConf.CHAR_VARCHAR_STANDARD_SEMANTICS.key).foreach { key =>
+ withSQLConf(key -> "true") {
+ assert(factory(row).toString === "[ab,abcdefgh]")
+ }
+ }
+ }
+
+ test("SPARK-60859: a wrapper prints complex key types by their field types")
{
+ val date = 19724 // 2024-01-02
+ val timestamp = 1704164645000000L // 2024-01-02 03:04:05
+ val binary = Array[Byte](1, 2)
+ val struct = new StructType().add("d", DateType).add("t",
TimestampType).add("b", BinaryType)
+ val dataTypes: Seq[DataType] = Seq(
+ struct,
+ ArrayType(IntegerType),
+ ArrayType(struct),
+ MapType(StringType, DateType))
+ val row = InternalRow(
+ InternalRow(date, timestamp, binary),
+ new GenericArrayData(Array[Any](1, 2)),
+ new GenericArrayData(Array[Any](
+ InternalRow(date, timestamp, binary), InternalRow(date + 1, timestamp,
binary))),
+ new ArrayBasedMapData(
+ new GenericArrayData(Array[Any](UTF8String.fromString("a"),
UTF8String.fromString("b"))),
+ new GenericArrayData(Array[Any](date, date + 1))))
+ val expected = "[[2024-01-02,2024-01-02 03:04:05,0x0102],[1,2]," +
+ "[[2024-01-02,2024-01-02 03:04:05,0x0102],[2024-01-03,2024-01-02
03:04:05,0x0102]]," +
+ "{a -> 2024-01-02,b -> 2024-01-03}]"
+ val factory =
InternalRowComparableWrapper.getInternalRowComparableWrapperFactory(dataTypes)
+ withSQLConf(SQLConf.SESSION_LOCAL_TIMEZONE.key -> "UTC") {
+ assert(factory(row).toString === expected)
+ // The same values through an UnsafeRow, UnsafeArrayData and
UnsafeMapData.
+ assert(factory(UnsafeProjection.create(dataTypes.toArray)(row)).toString
=== expected)
+ }
+
+ // A UDT key renders at its SQL type, also through an Unsafe-backed value.
+ val udtFactory = InternalRowComparableWrapper
+ .getInternalRowComparableWrapperFactory(Seq(new
TestUDT.MyDenseVectorUDT))
+ val vector = InternalRow(new GenericArrayData(Array[Any](1.0, 2.0)))
+ assert(udtFactory(vector).toString === "[[1.0,2.0]]")
+ val unsafeVector =
UnsafeProjection.create(Array[DataType](ArrayType(DoubleType)))(vector)
+ assert(udtFactory(unsafeVector).toString === "[[1.0,2.0]]")
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]