[
https://issues.apache.org/jira/browse/HDDS-5330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17361821#comment-17361821
]
Stephen O'Donnell commented on HDDS-5330:
-----------------------------------------
I have not seen any specific problems in practice, but in a discussion about
another topic (I don't recall what) I was led to believe only commands from the
current leader should be executed on the DNs and others should be dropped.
One example where problems may arise is with Replication Manager, as many
commands could be queued on the DNs, and then the new active RM could schedule
duplicates or other commands that result in over replication.
Also, there is logic to discard commands from an older term, so if the command
queue looked like:
Term_1
Term_2
Term_1
Then the 3rd command will be dropped. But if the exact same commands were
somehow queued as:
Term_1
Term_1
Term_2
Then none would be dropped.
I guess this is a bit confusing, so I created this Jira so we could confirm how
things are supposed to work:
1) All commands should be executed on the DNs, no matter which SCM issued them,
in which case is the logic to drop the commands intentional?
2) Or only commands from the current leader should be executed and all old
commands dropped?
3) Or, is it possible for a non leader SCM to send commands to the DNs at any
time, and they need to be ignored?
> Datanode commands are not always invalidated when the SCM leader switches
> --------------------------------------------------------------------------
>
> Key: HDDS-5330
> URL: https://issues.apache.org/jira/browse/HDDS-5330
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Stephen O'Donnell
> Priority: Major
>
> A datanode should only process commands from the SCM which is the leader.
> In StateContext.getNextCommand(), there is logic to update the current leader
> SCM term for each command seen on the DN. It picks the command, and the
> updates the term based on the term stored in the command:
> {code}
> updateTermOfLeaderSCM(command);
> if (command.getTerm() == termOfLeaderSCM.get()) {
> return command;
> }
> {code}
> There are a few problems here:
> 1) If there are commands in the queue with a newer term, then the term stored
> in the DN will not be updated until all the pending commands have been
> processed. Therefore the SCM switch can have happened and the stale commands
> continue to be processed.
> 2) While there is a single command queue, there are further sub-queues. For
> example DeleteContainerCommandHandler places the commands into a executor
> queue. Similar for ReplicateContainerCommandHandler. These queues could be
> quite large and hence stale commands could be processed.
> For (1), I believe the term should be updated when the commands are enqueued,
> not dequeued. That would ensure new commands update the term and invalidate
> the old commands immediately.
> For (2), we should check the DN term prior to executing the command and drop
> the command if it is no longer valid.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
