johnyangk commented on a change in pull request #222: [NEMO-350] Implement
Off-heap SerializedMemoryStore & [NEMO-384] Implement
DirectByteBufferInputStream for Off-heap SerializedMemoryStore
URL: https://github.com/apache/incubator-nemo/pull/222#discussion_r295144377
##########
File path:
runtime/executor/src/main/java/org/apache/nemo/runtime/executor/data/DataUtil.java
##########
@@ -112,19 +114,19 @@ private static void serializePartition(final
EncoderFactory encoderFactory,
final List<SerializedPartition<K>> serializedPartitions = new
ArrayList<>();
for (final NonSerializedPartition<K> partitionToConvert :
partitionsToConvert) {
try (
- DirectByteArrayOutputStream bytesOutputStream = new
DirectByteArrayOutputStream();
- OutputStream wrappedStream = buildOutputStream(bytesOutputStream,
serializer.getEncodeStreamChainers());
+ DirectByteBufferOutputStream bytesOutputStream = new
DirectByteBufferOutputStream();
+ OutputStream wrappedStream = buildOutputStream(bytesOutputStream,
serializer.getEncodeStreamChainers())
) {
serializePartition(serializer.getEncoderFactory(), partitionToConvert,
wrappedStream);
// We need to close wrappedStream on here, because
DirectByteArrayOutputStream:getBufDirectly() returns
// inner buffer directly, which can be an unfinished(not flushed)
buffer.
wrappedStream.close();
// Note that serializedBytes include invalid bytes.
// So we have to use it with the actualLength by using size() whenever
needed.
- final byte[] serializedBytes = bytesOutputStream.getBufDirectly();
+ final List<ByteBuffer> serializedBufList =
bytesOutputStream.getBufferList();
Review comment:
directByteBufferList?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
