johnyangk commented on a change in pull request #222: [NEMO-350] Implement
Off-heap SerializedMemoryStore & [NEMO-384] Implement
DirectByteBufferInputStream for Off-heap SerializedMemoryStore
URL: https://github.com/apache/incubator-nemo/pull/222#discussion_r295136836
##########
File path:
common/src/main/java/org/apache/nemo/common/DirectByteBufferOutputStream.java
##########
@@ -140,38 +140,45 @@ public void write(final byte[] b, final int off, final
int len) {
final int arraySize = pageSize * (dataList.size() - 1) +
lastBuf.position();
final byte[] byteArray = new byte[arraySize];
int start = 0;
- int byteToWrite;
- for (final ByteBuffer temp : dataList) {
+ for (final ByteBuffer buffer : dataList) {
// ByteBuffer has to be shifted to read mode by calling
ByteBuffer.flip(),
// which sets limit to the current position and sets the position to 0.
// Note that capacity remains unchanged.
- temp.flip();
- byteToWrite = temp.remaining();
- temp.get(byteArray, start, byteToWrite);
+ final ByteBuffer dupBuffer = buffer.duplicate();
+ dupBuffer.flip();
+ final int byteToWrite = dupBuffer.remaining();
+ dupBuffer.get(byteArray, start, byteToWrite);
Review comment:
I suppose here the contents of the off-heap bytes are copied to an on-heap
array (i.e., bytesArray). If used in the execution code path, this would reduce
the benefits of the off-heap data management introduced in this PR.
But, I believe `toByteArray()` is being used only in
`DirectByteBufferOutputStreamTest`. This led me to think that this method is
intended to be used only in tests. If so, can you rename the method make this
intention explicit? It'd be good to also remove `public`.
e.g.,
```java
// USED BY TESTS ONLY
@VisibleForTesting
byte[] buildByteArrayForTests() {
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
