Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/10852#discussion_r50768561
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/object.scala
---
@@ -44,18 +53,30 @@ trait ObjectOperator extends LogicalPlan {
def withObjectOutput: LogicalPlan = if
(output.head.dataType.isInstanceOf[ObjectType]) {
this
} else {
- withNewSerializer(outputObject)
+ withNewSerializer(outputObject :: Nil)
}
/** Returns a copy of this operator with a different serializer. */
- def withNewSerializer(newSerializer: NamedExpression): LogicalPlan =
makeCopy {
+ def withNewSerializer(newSerializer: Seq[NamedExpression]): LogicalPlan
= makeCopy {
productIterator.map {
- case c if c == serializer => newSerializer :: Nil
+ case c if c == serializer => newSerializer
case other: AnyRef => other
}.toArray
}
}
+/**
+ * A dummy [[ObjectOperator]] that is used to trigger deserializer
resolution in Analyzer and we
+ * can get back the resolved deserializer later.
+ */
+case class DummyObjectOperator(deserializer: Expression, attributes:
Seq[Attribute])
+ extends LeafNode with ObjectOperator {
+
+ override def inputSet: AttributeSet = AttributeSet(attributes)
+ override def serializer: Seq[NamedExpression] = Nil
+ override def deserializers: Seq[(Expression, Seq[Attribute])] =
Seq(deserializer -> attributes)
+}
--- End diff --
Could we instead wrap the resolution logic in a public function that is
defined on `ResolveReferences` and just call that directly when we need to?
The flow is pretty hard to follow here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
