GitHub user JoshRosen opened a pull request:
https://github.com/apache/spark/pull/7306
[SPARK-8932] Support copy() for UnsafeRows that do not use ObjectPools
We call Rpw.copy() in many places throughout SQL but UnsafeRow currently
throws UnsupportedOperationException when copy() is called.
Supporting copying when ObjectPool is used may be difficult, since we may
need to handle deep-copying of objects in the pool. In addition, this copy()
method needs to produce a self-contained row object which may be passed around
/ buffered by downstream code which does not understand the UnsafeRow format.
In the long run, we'll need to figure out how to handle the ObjectPool
corner cases, but this may be unnecessary if other changes are made. Therefore,
in order to unblock my sort patch (#6444) I propose that we support copy() for
the cases where UnsafeRow does not use an ObjectPool and continue to throw
UnsupportedOperationException when an ObjectPool is used.
This patch accomplishes this by modifying UnsafeRow so that it knows the
size of the row's backing data in order to be able to copy it into a byte array.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JoshRosen/spark SPARK-8932
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/7306.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #7306
----
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
