[
https://issues.apache.org/jira/browse/HDDS-12007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18047593#comment-18047593
]
Ivan Andika edited comment on HDDS-12007 at 12/26/25 9:44 AM:
--------------------------------------------------------------
> link(..) should still be called for recording an entry into the RaftLog. It
> just that link() should not trigger PutBlock.
Got it. I tested it again and `sendForward` invokes the following StateMachine
methods
* ContainerStateMachine#link: This currently set the linked flag, which seems
to prevent LocalStream#cleanUp to close the file, since it's already closed
successfully through DataStreamManagement#close. However, this does not seem to
protect the channel entirely since if the LocalStream#cleanUp is triggered
between the StandardWriteOption.CLOSE and the ContainerStateMachine#link, the
issue might still happen.
* ContainerStateMachine#applyTransaction: This will trigger
KeyValueHandler#streamInit since the BlockDataStreamOutput#setupStream calls
DataStreamApi#stream with ContainerCommandRequestProto with type StreamInit
** The KeyValueStreamHandler#streamInit only returns the chunk file path and
does not seem to be used by the client
** Since isWriteStage, isWriteCommitStage, isCombinedStage are all false, the
applyTransaction does not seem to do anything useful in Ozone context
sendForward will create a new Ratis transaction, but the following
putBlockAsync also triggers a Ratis transaction.
IMO StreamInit does not seem to be useful and we can deprecate / remove it. If
we want to send the last PutBlock in streaming and remove the async
putBlockAsync, one idea is to pass a PutBlockRequest in the closeAsync instead
of in the writeAsync in executePutBlockClose, so it will trigger
StateMachine#link and also do useful works by executing PutBlock in
ContaineStateMachine#applyTransaction (unlike StreamInit). Then we can remove
the following putBlockAsync.
was (Author: JIRAUSER298977):
> link(..) should still be called for recording an entry into the RaftLog. It
> just that link() should not trigger PutBlock.
Got it. I tested it again and `sendForward` invokes the following StateMachine
methods
* ContainerStateMachine#link: This currently set the linked flag, which seems
to prevent LocalStream#cleanUp to close the file, since it's already closed
successfully through DataStreamManagement#close
* ContainerStateMachine#applyTransaction: This will trigger
KeyValueHandler#streamInit since the BlockDataStreamOutput#setupStream calls
DataStreamApi#stream with ContainerCommandRequestProto with type StreamInit
** The KeyValueStreamHandler#streamInit only returns the chunk file path and
does not seem to be used by the client
** Since isWriteStage, isWriteCommitStage, isCombinedStage are all false, the
applyTransaction does not seem to do anything useful in Ozone context
sendForward will create a new Ratis transaction, but the following
putBlockAsync also triggers a Ratis transaction.
IMO StreamInit does not seem to be useful and we can deprecate / remove it. If
we want to send the last PutBlock in streaming and remove the async
putBlockAsync, one idea is to pass a PutBlockRequest in the closeAsync instead
of in the writeAsync in executePutBlockClose, so it will trigger
StateMachine#link and also do useful works by executing PutBlock. Then we can
remove the following putBlockAsync.
> BlockDataStreamOutput should only send one PutBlock during close
> ----------------------------------------------------------------
>
> Key: HDDS-12007
> URL: https://issues.apache.org/jira/browse/HDDS-12007
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: Ozone Datanode
> Reporter: Ivan Andika
> Assignee: Tsz-wo Sze
> Priority: Major
> Labels: pull-request-available
> Fix For: 2.0.0
>
>
> Currently, during close two PutBlock request will be sent
> * executePutBlockClose: This uses the DataStreamOutput#writeAsync with
> StandardWriteOption.CLOSE as part of the HDDS-6500 improvements
> ** This will call sendForward which will trigger ContainerStateMachine#link
> which will be processed like PutBlock
> * putBlockAsync: This is a normal PutBlock request which is executed per
> block boundary (similar to Write Pipeline V1)
> We should only call executePutBlockClose during close. We can use
> ClientProtoUtils#getRaftClientReply to convert from DataStreamReply to
> RaftClientReply which we can use to derive the PutBlock response.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
