Github user mallman commented on the issue:
https://github.com/apache/spark/pull/16499
I looked into simply cleaning up the `StorageUtils.dispose` method to only
dispose memory-mapped buffers. However, I did find legitimate uses of that
method to dispose of direct/non-memory-mapped buffers. So I kept the behavior
of that method as-is. Instead, I added a new methodâunmapâwhich will
dispose of memory-mapped buffers *only*, and added calls to that method where
appropriate. At the end of the day, I only found one case where we specifically
wanted an "unmap" behavior instead of the other broader disposal behavior.
(That case being the one what was causing corruption of replicated blocks in
the first place.)
I also found a new memory management bug in `BlockManager` introduced by
the encryption support. In the original codebase, it disposes of a buffer
unsafely. I think part of the problem is the documentation of the
`ChunkedByteBuffer.toByteBuffer` method uses the word "copy" in describing what
that method does. I expanded and made that method's documentation more precise
to clarify that sometimes that method *does not* return a copy of the data. In
those cases, it is not safe to dispose the returned buffer.
I found that there were no uses of `ByteBufferInputStream` where automatic
buffer disposal was called for. Therefore, I dropped that support from that
class to guard against unsafe usage. If someone _really_ wants to actually use
automatic buffer disposal in `ByteBufferInputStream` they
canâcarefullyâre-add that support. I think that that's generally unsafe.
And, like I said, nothing in the codebase was using it anyway except where it
was used incorrectly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
