Github user aarondav commented on a diff in the pull request:
https://github.com/apache/spark/pull/2330#discussion_r18264123
--- Diff: core/src/main/scala/org/apache/spark/network/ManagedBuffer.scala
---
@@ -34,11 +35,17 @@ import org.apache.spark.util.{ByteBufferInputStream,
Utils}
* This interface provides an immutable view for data in the form of
bytes. The implementation
* should specify how the data is provided:
*
- * - FileSegmentManagedBuffer: data backed by part of a file
- * - NioByteBufferManagedBuffer: data backed by a NIO ByteBuffer
- * - NettyByteBufManagedBuffer: data backed by a Netty ByteBuf
+ * - [[FileSegmentManagedBuffer]]: data backed by part of a file
+ * - [[NioManagedBuffer]]: data backed by a NIO ByteBuffer
+ * - [[NettyManagedBuffer]]: data backed by a Netty ByteBuf
+ *
+ * The concrete buffer implementation might be managed outside the JVM
garbage collector.
+ * For example, in the case of [[NettyManagedBuffer]], the buffers are
reference counted.
+ * In that case, if the buffer is going to be passed around to a different
thread, retain/release
--- End diff --
The comment here suggests that you should only have to retain if your
buffer is a NettyManagedBuffer, which isn't quite true, as you could have an
NioManagedBuffer whose ByteBuffer underlies a Netty ByteBuf, in which case it
is never safe to pass around.
I'm just a little worried about sanitary buffer usage, as misuse of this
API from not copying could lead to nondeterministic data corruption.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
