Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/18543#discussion_r127672102
--- Diff:
sql/catalyst/src/main/java/org/apache/spark/sql/execution/UnsafeExternalRowSorter.java
---
@@ -211,7 +211,10 @@ public int compare(Object baseObj1, long baseOff1,
Object baseObj2, long baseOff
// TODO: Why are the sizes -1?
row1.pointTo(baseObj1, baseOff1, -1);
row2.pointTo(baseObj2, baseOff2, -1);
- return ordering.compare(row1, row2);
+ int comparison = ordering.compare(row1, row2);
+ row1.pointTo(null, 0L, -1);
+ row2.pointTo(null, 0L, -1);
--- End diff --
Good idea @cloud-fan . Looks like `RowComparator` and `KVComparator` could
be cleaned up in `UnsafeExternalSorter.cleanupResources` and
`UnsafeInMemorySorter.free`. @davies I know this is from a long while back, but
does that make sense? Seems like reasonable places to simply 'flush' the
references, and won't hurt anything.
I am not 100% sure there's not another path to take care of, or if this
frees the ref soon enough to avoid the problem. Thoughts @j-baker ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
