szehon-ho commented on code in PR #58412:
URL: https://github.com/apache/spark/pull/58412#discussion_r3888736341
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryBaseTable.scala:
##########
@@ -765,15 +765,60 @@ abstract class InMemoryBaseTable(
/** Predicates recorded by [[filter]], for test assertions only. */
def pushedCatalystPredicates: Seq[CatalystExpression] =
catalystPredicates.toSeq
- /** AttributeReferences matching the partition-key InternalRow field
order. */
- private def partitionAttributes: Seq[AttributeReference] = {
+ /**
+ * The `AttributeReference`s standing for the partition key InternalRow
fields, in its field
+ * order, each paired with the name-part sequence of its partition column.
The parts are kept
+ * unflattened so a quoted top-level column `a.b` (parts `Seq("a.b")`)
stays distinct from a
+ * nested column `a`.`b` (parts `Seq("a", "b")`). Example:
+ * - `PARTITIONED BY (part, s.nested)` -> `(Seq("part"),
AttributeReference(part))`, then
+ * `(Seq("s", "nested"), AttributeReference(s.nested))`
+ */
+ private def partitionAttributes: Seq[(Seq[String], AttributeReference)] = {
partitioning.flatMap(_.references()).flatMap { ref =>
- val name = ref.fieldNames.mkString(".")
- readSchema.find(_.name == name).orElse(tableSchema.find(_.name ==
name)).map { f =>
- AttributeReference(f.name, f.dataType, f.nullable)()
+ val path = ref.fieldNames.toImmutableArraySeq
+ val resolver = SQLConf.get.resolver
+ readSchema.findNestedField(path, resolver = resolver)
+ .orElse(tableSchema.findNestedField(path, resolver = resolver)).map {
+ case (_, f) =>
+ path -> AttributeReference(ref.fieldNames.mkString("."),
f.dataType, f.nullable)()
Review Comment:
Thanks, good catch. The fixture now binds only identity transforms and uses
each transform’s actual partition-key ordinal. Predicates on transformed source
columns remain for residual evaluation. I added a regression covering a nested
truncate transform in slot 0 and a nested identity transform in slot 1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
