Github user JoshRosen commented on the issue:
https://github.com/apache/spark/pull/15022
This change looks good to me as well, although I wonder whether there's an
assertion or unit test that we can add to detect this bug or keep things from
regressing in case this is ever refactored.
I suppose that one invariant is that `unrollMemoryMap.keys.size` and
`pendingUnrollMemoryMap.keys.size` should both be upper-bounded by the number
of executor cores (since that's the maximum number of concurrent tasks).
I wouldn't necessarily block merging this on the addition of new tests /
asserts right now but it would be great to add them if they're low-cost and
it's easy to do so.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
