erikerlandson commented on a change in pull request #28983:
URL: https://github.com/apache/spark/pull/28983#discussion_r449569671
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
##########
@@ -679,7 +679,10 @@ object MapObjects {
elementNullable: Boolean = true,
customCollectionCls: Option[Class[_]] = None): MapObjects = {
val loopVar = LambdaVariable("MapObject", elementType, elementNullable)
- MapObjects(loopVar, function(loopVar), inputData, customCollectionCls)
+ // println(s"\n\n $function, $inputData, $elementType, $elementNullable,
$customCollectionCls \n")
+ val fOfLV = if (function == null) loopVar else function(loopVar)
Review comment:
> I guess that that UnresolvedMapObjects should not be resolved and
evaluated on executors.
As of SPARK-27926 that has changed. This change is the source of the bug.
> this only works if function is just "identity"?
Yes. My main concern with this fix is determining whether "identity" will
work in all cases, and if it will not, what else might be done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
