ss77892 commented on code in PR #11873:
URL: https://github.com/apache/ozone/pull/11873#discussion_r4576773219


##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/io/RandomAccessFileChannel.java:
##########
@@ -56,6 +64,35 @@ public synchronized void open(File file) throws 
FileNotFoundException {
     channel = newChannel;
   }
 
+  /**
+   * @return the {@link BlockData} cached for the given block, or null if none 
is cached for it or the cached one has
+   * a lower BCSID than requested.
+   */
+  public BlockData getCachedBlockData(BlockID blockID) {
+    final BlockData cached = cachedBlockData.get();
+    return cached != null && 
cached.getBlockID().getContainerBlockID().equals(blockID.getContainerBlockID())
+        && cached.getBlockCommitSequenceId() >= 
blockID.getBlockCommitSequenceId() ? cached : null;
+  }

Review Comment:
   Yep, this state belongs with the stream, not the file wrapper. Didn't want 
to add a new parameters through HddsDispatcher/Handler. Let me think how to 
better handle it. 



##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueHandler.java:
##########
@@ -2344,20 +2344,31 @@ private long readBlockImpl(ContainerCommandRequestProto 
request, RandomAccessFil
     BlockUtils.verifyReplicaIdx(kvContainer, blockID);
     BlockUtils.verifyBCSId(kvContainer, blockID);
 
-    final BlockData blockData = getBlockManager().getBlock(kvContainer, 
blockID);
+    if (readBlock.getOffset() < 0 || readBlock.getLength() < 0) {
+      return rejectReadBlock(blockFile, streamObserver, 
Status.INVALID_ARGUMENT.withDescription(
+          "Negative offset " + readBlock.getOffset() + " or length " + 
readBlock.getLength() + " for " + blockID));
+    }
+
+    // Cached per stream; re-resolve when the range reaches the cached size, 
since the block may have grown.
+    BlockData blockData = blockFile.getCachedBlockData(blockID);
+    if (blockData == null || readBlock.getLength() >= blockData.getSize() - 
readBlock.getOffset()) {
+      blockData = getBlockManager().getBlock(kvContainer, blockID);
+      blockFile.cacheBlockData(blockData);
+    }
     if (readBlock.getOffset() >= blockData.getSize()) {
       return rejectReadBlock(blockFile, streamObserver, 
Status.OUT_OF_RANGE.withDescription(
           "Requested offset " + readBlock.getOffset() + " is beyond the end of 
block " + blockID + " with size "
               + blockData.getSize()));
     }
     final BlockReadCursor cursor = new BlockReadCursor(readBlock.getOffset(), 
readBlock.getLength(),
         responseDataSize, blockData.getChunks());
-    final ByteBuffer buffer = ByteBuffer.allocate(cursor.responseDataSize());
     long bytesRead = 0;
     while (cursor.hasNext()) {
       final BlockReadCursor.ReadRange range = cursor.next();
+      // Reused across the stream's requests: gRPC serializes the response 
inside onNext, before the buffer is refilled.
+      final ByteBuffer buffer = blockFile.getReadBuffer(range.length());

Review Comment:
   Thanks @peterxcli for looking into it. I agree the buffer has no business in 
RandomAccessFileChannel. But readBlockImpl can't own it. It runs once per 
request on a KeyValueHandler that all streams share. So a buffer kept across 
requests has to live in something per stream. blockFile was the only per-stream 
object reaching the handler, which is why it ended up there. 
    On #12042: it does overlap. Its min(responseSize, end - offset) sizing is 
what this PR does with getReadBuffer(range.length()), so I'll rebase on it once 
it's merged and drop that part from here. #12042 still allocates buffer of at 
least one checksum interval (64 KiB) per request, though. For large sequential 
requests that's spread over many responses and doesn't matter. For small 
positioned reads (e.g. HBase random gets at tens of thousands per second) it's 
still GB/s of allocation per datanode. I've checked  #12042 alone vs #12042 
plus reuse and I see a better result with reuse. 
    



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to