Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/6444#issuecomment-116967403
I'm in the process of updating this to reflect @davies' recent changes to
support arbitrary column types in UnsafeRow. One bit of trickiness is the fact
that UnsafeExternalSorter, which manages the spilling, doesn't know anything
about SQL and thus can't directly implement the logic for spilling and reading
ObjectPools back during the merge. In the worst case, I guess that the merging
step might need to keep a bunch of ObjectPools in memory at the same time,
which could end up causing us to OOM, but in those cases we would have OOM'd
before, too.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
