sandeepvinayak commented on PR #36680:
URL: https://github.com/apache/spark/pull/36680#issuecomment-1142682897
@JoshRosen Can you please review this when you get chance. Also, it will be
great, if we can get this fix as part of next release. thanks !
--
This is an automated message from
sandeepvinayak commented on PR #36680:
URL: https://github.com/apache/spark/pull/36680#issuecomment-1140203155
@JoshRosen Just took another look at the code, the fix I made is for the
deadlock b/w `TaskMemoryManager` and `UnsafeExternalSorter.SplittableIterator`
which is what we faced and
sandeepvinayak commented on PR #36680:
URL: https://github.com/apache/spark/pull/36680#issuecomment-1140132804
@JoshRosen Unfortunately, we don't have the server logs at this point, can
definitely try to look in another occurrence of deadlock. I will also try to
take another look based on
sandeepvinayak commented on PR #36680:
URL: https://github.com/apache/spark/pull/36680#issuecomment-1140128160
Good catch @JoshRosen , I believe we can do it without having a local
`inMemSorterToFree` by moving the `inMemSorter.freeMemory` to finally. WDYT ?
```java
finally
sandeepvinayak commented on PR #36680:
URL: https://github.com/apache/spark/pull/36680#issuecomment-1139888789
@cloud-fan jenkins looks good now, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above
sandeepvinayak commented on PR #36680:
URL: https://github.com/apache/spark/pull/36680#issuecomment-1139146525
> deadlock occurred between one thread calling
`UnsafeExternalSorter.cleanupResources()` and another thread which was calling
`spill()` on UnsafeExternalSorter
I believe
