szetszwo commented on code in PR #10823:
URL: https://github.com/apache/ozone/pull/10823#discussion_r3676243122
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/impl/BlockManagerImpl.java:
##########
@@ -209,6 +209,20 @@ public long persistPutBlock(KeyValueContainer container,
// container to determine whether the blockCount is already incremented
// for this block in the DB or not.
long localID = data.getLocalID();
+ // For the PutBlock that is endOfBlock and meanwhile bscId = 0, it means
+ // this PutBlock comes from data stream close without going through the
+ // Ratis, thus there is no log index. In this case, we should not let
+ // 0 to overwrite previous possible PutBlocks from Ratis log that were
+ // generated during immediate flushes from an active data stream.
Instead,
+ // we should load the latest bscid and reuse that id.
+ if (endOfBlock && bcsId == 0) {
+ BlockData existing = db.getStore().getBlockDataTable()
+ .get(containerData.getBlockKey(localID));
+ if (existing != null) {
+ bcsId = existing.getBlockCommitSequenceId();
Review Comment:
> Right now we cannot convert previous PutBlocks to non-Ratis because we
cannot insert metadata command into the byte stream before stream is close.
After all, append PutBlock to the end of the stream and consumes it is a "hack".
Thanks for explaining the details! For V3, we should eliminate PutBlocks
using Raft. In the followup JIRAs, we may
- add a new DataStreamRequestHeaderProto.Type, say STREAM_COMMAND; and
- send a STREAM_COMMAND packet to trigger PutBlocks when user call
flush/hsync.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
