amaliujia commented on code in PR #12077:
URL: https://github.com/apache/ozone/pull/12077#discussion_r4609415636


##########
hadoop-hdds/client/src/test/java/org/apache/hadoop/hdds/scm/storage/TestBlockDataStreamOutput.java:
##########
@@ -246,6 +253,35 @@ void 
midStreamPutBlockFailsOnDatanodeWithoutCommandSupport() throws Exception {
     assertThrows(IOException.class, stream::close);
   }
 
+  @Test
+  void testCloseReleasesBuffersCommittedByStreamClose() throws Exception {
+    final MockDatanodePipeline pipeline = new 
MockDatanodePipeline().withDelayedStreamClose();
+    final OzoneClientConfig config = createConfig();
+    config.setDatastreamPutBlockWithoutRaftEnabled(true);
+    final List<StreamBuffer> bufferList = new ArrayList<>();
+    final BlockDataStreamOutput stream = createStream(pipeline, config, 
bufferList);
+    final byte[] data = randomBytes(500);
+    stream.write(ByteBuffer.wrap(data), 0, data.length);
+    stream.close();
+
+    assertEquals(500, stream.getTotalAckDataLength());
+    assertThat(bufferList).isEmpty();
+  }
+
+  @Test
+  void testFailedStreamCloseKeepsDataCommittedByBoundaryPutBlock() throws 
Exception {
+    final MockDatanodePipeline pipeline = new 
MockDatanodePipeline().failStreamClose().withDelayedStreamClose();
+    final OzoneClientConfig config = createConfig();
+    config.setDatastreamPutBlockWithoutRaftEnabled(true);
+    final BlockDataStreamOutput stream = createStream(pipeline, config);
+    // The boundary PutBlock commits the first 400B; the stream close, which 
fails, would commit the last 50B.
+    final byte[] data = randomBytes(450);
+    stream.write(ByteBuffer.wrap(data), 0, data.length);
+
+    assertThrows(IOException.class, stream::close);
+    assertEquals(400, stream.getTotalAckDataLength());

Review Comment:
   Can we also check the last 50 bytes:
   
   ```Java
     // Only the last 50B are left in the buffer list, to be written again by 
the retry.
     final ByteBuffer unacked = ByteBuffer.allocate(50);
     for (StreamBuffer buffer : bufferList) {
       final ByteBuffer dup = buffer.duplicate();
       dup.flip();
       unacked.put(dup);
     }
     assertArrayEquals(Arrays.copyOfRange(data, 400, 450), unacked.array());
   ```



##########
hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockDataStreamOutput.java:
##########
@@ -464,6 +467,11 @@ public void executePutBlock(boolean sync, boolean close) 
throws IOException {
       if (putBlockWithoutRaft) {
         // PutBlock is supposed to be committed after the data stream close so 
there
         // is no need to continue.
+        putBlockCommandFutures.add(dataStreamCloseReply.whenComplete((reply, 
e) -> {
+          if (e == null && reply != null && reply.isSuccess()) {
+            addBuffersCommittedWithoutRaft(byteBufferList);

Review Comment:
   We already have the callback above in `dataStreamCloseReply` 
https://github.com/apache/ozone/pull/12077/changes#diff-64d9809fa04e72e1f2c0332c0f4afe04773d92188279cc7bde422a06c65305a5L454-R466,
   
   you could add `addBuffersCommittedWithoutRaft(byteBufferList);` there



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to