Siyao Meng created HDDS-17334:
---------------------------------
Summary: Open a Ratis pipeline only after its leader is ready
Key: HDDS-17334
URL: https://issues.apache.org/jira/browse/HDDS-17334
Project: Apache Ozone
Issue Type: Improvement
Components: SCM, Ozone Datanode
Reporter: Siyao Meng
SCM can mark a Ratis pipeline OPEN before its Ratis leader is ready. A write in
that window gets {{LeaderNotReadyException}} for WriteChunk while the leader,
once ready, still accepts the following PutBlock. Applying PutBlock without its
chunk fails with "Chunk file not found", the container is marked UNHEALTHY and
the pipeline is closed. See HDDS-17213 for a test hitting this.
h3. Current behavior
* {{PipelineReportHandler#setPipelineLeaderId}} sets the leader for RATIS/ONE
from any pipeline report ("ONE replica pipeline doesn't have leader flag", from
HDDS-2034). {{Pipeline#isHealthy}} is then true and the pipeline is opened on
the first report after the group is added, before any election. In the
HDDS-17213 run: opened 05:39:58.528, leader elected 05:39:59.584, ready
05:39:59.615.
* For RATIS/THREE, SCM waits for a report with {{isLeader}}. The datanode sets
that flag in {{XceiverServerRatis#handleLeaderChangedNotification}}, i.e. on
election, not when the leader is ready, so a short window remains.
h3. Proposal
* Datanode: set {{isLeader}} in the pipeline report only after
{{StateMachine.LeaderEventApi#notifyLeaderReady}}, and send a pipeline report
then.
* SCM: drop the RATIS/ONE special case in
{{PipelineReportHandler#setPipelineLeaderId}}. Datanodes already report
{{isLeader}} for single node groups.
This covers newly created pipelines only. A re-election on an OPEN pipeline
still has the same window.
Once done, the leader ready wait added to {{TestFinalizeBlock}} in HDDS-17213
can be removed.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
