Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/4857#discussion_r25644640
--- Diff:
network/common/src/main/java/org/apache/spark/network/buffer/NioManagedBuffer.java
---
@@ -41,13 +41,13 @@ public long size() {
}
@Override
- public ByteBuffer nioByteBuffer() throws IOException {
+ public LargeByteBuffer nioByteBuffer() throws IOException {
return buf.duplicate();
}
@Override
public InputStream createInputStream() throws IOException {
- return new ByteBufInputStream(Unpooled.wrappedBuffer(buf));
+ return new
ByteBufInputStream(Unpooled.wrappedBuffer(buf.firstByteBuffer()));
--- End diff --
we could wrap the entire `LargeByteBuffer` in a
`LargeByteBufferInputStream` -- but I don't understand why there is already
this level of indirection here so I thought I'd check. This code is only for
fetching shuffle blocks in any case, so other things will break if there is
over 2gb in any case.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
