bshashikant commented on a change in pull request #551: HDDS-2717. Handle chunk
increments in datanode
URL: https://github.com/apache/hadoop-ozone/pull/551#discussion_r387595602
##########
File path:
hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockOutputStream.java
##########
@@ -274,13 +271,16 @@ public void writeOnRetry(long len) throws IOException {
if (len == 0) {
return;
}
- int count = 0;
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Retrying write length {} for blockID {}", len, blockID);
+ }
Preconditions.checkArgument(len <= streamBufferMaxSize);
+ int count = 0;
while (len > 0) {
- long writeLen;
- writeLen = Math.min(chunkSize, len);
- if (writeLen == chunkSize) {
- writeChunk(bufferPool.getBuffer(count));
+ ChunkBuffer buffer = bufferPool.getBuffer(count);
+ long writeLen = Math.min(buffer.position(), len);
+ if (!buffer.hasRemaining()) {
Review comment:
> I think so. By sending each buffer only once, we can simply send the
complete buffer. If we sent it incrementally, we would need to keep track of
the last write end position separately.
>
>
https://github.com/apache/hadoop-ozone/blob/9eb8e0ef28db64e573feec5bb652228e4add522d/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockOutputStream.java#L452
>
> This is not new behavior, previously it checked `writeLen == chunkSize`,
which resulted in the same:
>
>
https://github.com/apache/hadoop-ozone/blob/9eb8e0ef28db64e573feec5bb652228e4add522d/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockOutputStream.java#L282-L283
The previous approach was to ensure we only call writeChunk at the chunk
boundary. With the current code, its driven by the buffer size which may or may
not be equal to the chunk size chosen.
Are we ensuring buffer size will always be equal to chunk size?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
