HeartSaVioR commented on a change in pull request #26173: [SPARK-29503][SQL]
Copy result row from RowWriter in GenerateUnsafeProjection when the expression
is lambdaFunction in MapObject
URL: https://github.com/apache/spark/pull/26173#discussion_r337270114
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
##########
@@ -885,13 +887,15 @@ case class MapObjects private(
)
}
- // Make a copy of the data if it's unsafe-backed
- def makeCopyIfInstanceOf(clazz: Class[_ <: Any], value: String) =
- s"$value instanceof ${clazz.getSimpleName}? ${value}.copy() : $value"
+ // Make a copy of the unsafe data if the result contains any
+ def makeCopyUnsafeData(dataType: DataType, value: String) = {
+ s"""${value}.copyUnsafeData("${dataType.catalogString}")"""
Review comment:
So the approach depends on how long of catalog string we are expecting for
very complex type.
1) If we expect this as not considerable size, it would work as it is. We
might be able to refine a bit as defining local variable and assign from there,
but it may require additional conditional logic.
2) If we expect less than 64k but considerable size and we don't want to let
it consume the length of method limitation, we may need to define it to be one
of static fields.
3) If we expect more than 64k, we should have it to be one of static fields,
and also break catalog string down to chunk, and concat to one of String
instance.
Do we have some huge and complicated but realistic (in production) schema to
test this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
