amaliujia commented on code in PR #12077:
URL: https://github.com/apache/ozone/pull/12077#discussion_r4609415636
##########
hadoop-hdds/client/src/test/java/org/apache/hadoop/hdds/scm/storage/TestBlockDataStreamOutput.java:
##########
@@ -246,6 +253,35 @@ void
midStreamPutBlockFailsOnDatanodeWithoutCommandSupport() throws Exception {
assertThrows(IOException.class, stream::close);
}
+ @Test
+ void testCloseReleasesBuffersCommittedByStreamClose() throws Exception {
+ final MockDatanodePipeline pipeline = new
MockDatanodePipeline().withDelayedStreamClose();
+ final OzoneClientConfig config = createConfig();
+ config.setDatastreamPutBlockWithoutRaftEnabled(true);
+ final List<StreamBuffer> bufferList = new ArrayList<>();
+ final BlockDataStreamOutput stream = createStream(pipeline, config,
bufferList);
+ final byte[] data = randomBytes(500);
+ stream.write(ByteBuffer.wrap(data), 0, data.length);
+ stream.close();
+
+ assertEquals(500, stream.getTotalAckDataLength());
+ assertThat(bufferList).isEmpty();
+ }
+
+ @Test
+ void testFailedStreamCloseKeepsDataCommittedByBoundaryPutBlock() throws
Exception {
+ final MockDatanodePipeline pipeline = new
MockDatanodePipeline().failStreamClose().withDelayedStreamClose();
+ final OzoneClientConfig config = createConfig();
+ config.setDatastreamPutBlockWithoutRaftEnabled(true);
+ final BlockDataStreamOutput stream = createStream(pipeline, config);
+ // The boundary PutBlock commits the first 400B; the stream close, which
fails, would commit the last 50B.
+ final byte[] data = randomBytes(450);
+ stream.write(ByteBuffer.wrap(data), 0, data.length);
+
+ assertThrows(IOException.class, stream::close);
+ assertEquals(400, stream.getTotalAckDataLength());
Review Comment:
Can we also check the last 50 bytes:
```Java
// Only the last 50B are left in the buffer list, to be written again by
the retry.
final ByteBuffer unacked = ByteBuffer.allocate(50);
for (StreamBuffer buffer : bufferList) {
final ByteBuffer dup = buffer.duplicate();
dup.flip();
unacked.put(dup);
}
assertArrayEquals(Arrays.copyOfRange(data, 400, 450), unacked.array());
```
##########
hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockDataStreamOutput.java:
##########
@@ -464,6 +467,11 @@ public void executePutBlock(boolean sync, boolean close)
throws IOException {
if (putBlockWithoutRaft) {
// PutBlock is supposed to be committed after the data stream close so
there
// is no need to continue.
+ putBlockCommandFutures.add(dataStreamCloseReply.whenComplete((reply,
e) -> {
+ if (e == null && reply != null && reply.isSuccess()) {
+ addBuffersCommittedWithoutRaft(byteBufferList);
Review Comment:
We already have the callback above in `dataStreamCloseReply`
https://github.com/apache/ozone/pull/12077/changes#diff-64d9809fa04e72e1f2c0332c0f4afe04773d92188279cc7bde422a06c65305a5L454-R466,
you could add `addBuffersCommittedWithoutRaft(byteBufferList);` there
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]