[
https://issues.apache.org/jira/browse/HDDS-5529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jackson Yao updated HDDS-5529:
------------------------------
Description:
{code:java}
// code placeholder
@Replicate
void updatePipelineState(
HddsProtos.PipelineID pipelineIDProto,
HddsProtos.PipelineState newState
)
throws IOException; @Override
public void updatePipelineState(
HddsProtos.PipelineID pipelineIDProto, HddsProtos.PipelineState newState)
throws IOException {
PipelineID pipelineID = PipelineID.getFromProtobuf(pipelineIDProto);
Pipeline.PipelineState oldState = null;
lock.writeLock().lock();
try {
oldState = getPipeline(pipelineID).getPipelineState();
// null check is here to prevent the case where SCM store
// is closed but the staleNode handlers/pipeline creations
// still try to access it.
if (pipelineStore != null) {
pipelineStateMap.updatePipelineState(pipelineID,
Pipeline.PipelineState.fromProtobuf(newState));
transactionBuffer
.addToBuffer(pipelineStore, pipelineID, getPipeline(pipelineID));
}
} catch (PipelineNotFoundException pnfe) {
LOG.warn("Pipeline {} is not found in the pipeline Map. Pipeline"
+ " may have been deleted already.", pipelineID);
} catch (IOException ex) {
LOG.warn("Pipeline {} state update failed", pipelineID);
// revert back to old state in memory
pipelineStateMap.updatePipelineState(pipelineID, oldState);
} finally {
lock.writeLock().unlock();
}
}
{code}
it is annotated with {{@Replicate}} , so it will be replicated by ratis if HA
is enabled and it is actually called when the raft log is applied. suppose we
have one leader L1 , and two followes F1 and F2. if the rocksdb of L1 is ok ,
and the methoed is excuted successfully at L1. when L1 crashed , F1 is elected
as the new leader, but the rocksdb of F1 got some error(for example , can not
be written) , so when F1 applied the raft log entry, it will get an exception,
and the code of e\{{xception}} will be excuted. so the point here is that ,
current HA implementation seems could not guarantee the consistence of
different SCM, because although different scm apply the same raft logs in the
same order, but the result of {{@Replicate}} functions may be different by
other factors. leader may succeed writting DB. but follower may fail when it is
elected as a new leader , this may cause the inconsistence
was:
{code:java}
// code placeholder
{code}
@Replicate void updatePipelineState( HddsProtos.PipelineID pipelineIDProto,
HddsProtos.PipelineState newState ) throws IOException; @Override public void
updatePipelineState( HddsProtos.PipelineID pipelineIDProto,
HddsProtos.PipelineState newState) throws IOException \{ PipelineID pipelineID
= PipelineID.getFromProtobuf(pipelineIDProto); Pipeline.PipelineState oldState
= null; lock.writeLock().lock(); try { oldState =
getPipeline(pipelineID).getPipelineState(); // null check is here to prevent
the case where SCM store // is closed but the staleNode handlers/pipeline
creations // still try to access it. if (pipelineStore != null) {
pipelineStateMap.updatePipelineState(pipelineID,
Pipeline.PipelineState.fromProtobuf(newState)); transactionBuffer
.addToBuffer(pipelineStore, pipelineID, getPipeline(pipelineID)); } } catch
(PipelineNotFoundException pnfe) \{ LOG.warn("Pipeline {} is not found in the
pipeline Map. Pipeline" + " may have been deleted already.", pipelineID); }
catch (IOException ex) \{ LOG.warn("Pipeline {} state update failed",
pipelineID); // revert back to old state in memory
pipelineStateMap.updatePipelineState(pipelineID, oldState); } finally \{
lock.writeLock().unlock(); } }
it is annotated with {{@Replicate}} , so it will be replicated by ratis if HA
is enabled and it is actually called when the raft log is applied. suppose we
have one leader L1 , and two followes F1 and F2. if the rocksdb of L1 is ok ,
and the methoed is excuted successfully at L1. when L1 crashed , F1 is elected
as the new leader, but the rocksdb of F1 got some error(for example , can not
be written) , so when F1 applied the raft log entry, it will get an exception,
and the code of e\{{xception}} will be excuted. so the point here is that ,
current HA implementation seems could not guarantee the consistence of
different SCM, because although different scm apply the same raft logs in the
same order, but the result of {{@Replicate}} functions may be different by
other factors. leader may succeed writting DB. but follower may fail when it is
elected as a new leader , this may cause the inconsistence
> scm should be terminated when IOException is throw by rocksdb
> -------------------------------------------------------------
>
> Key: HDDS-5529
> URL: https://issues.apache.org/jira/browse/HDDS-5529
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: Jackson Yao
> Assignee: Jackson Yao
> Priority: Major
>
> {code:java}
> // code placeholder
> @Replicate
> void updatePipelineState(
> HddsProtos.PipelineID pipelineIDProto,
> HddsProtos.PipelineState newState
> )
> throws IOException; @Override
> public void updatePipelineState(
> HddsProtos.PipelineID pipelineIDProto, HddsProtos.PipelineState
> newState)
> throws IOException {
> PipelineID pipelineID = PipelineID.getFromProtobuf(pipelineIDProto);
> Pipeline.PipelineState oldState = null;
> lock.writeLock().lock();
> try {
> oldState = getPipeline(pipelineID).getPipelineState();
> // null check is here to prevent the case where SCM store
> // is closed but the staleNode handlers/pipeline creations
> // still try to access it.
> if (pipelineStore != null) {
> pipelineStateMap.updatePipelineState(pipelineID,
> Pipeline.PipelineState.fromProtobuf(newState));
> transactionBuffer
> .addToBuffer(pipelineStore, pipelineID, getPipeline(pipelineID));
> }
> } catch (PipelineNotFoundException pnfe) {
> LOG.warn("Pipeline {} is not found in the pipeline Map. Pipeline"
> + " may have been deleted already.", pipelineID);
> } catch (IOException ex) {
> LOG.warn("Pipeline {} state update failed", pipelineID);
> // revert back to old state in memory
> pipelineStateMap.updatePipelineState(pipelineID, oldState);
> } finally {
> lock.writeLock().unlock();
> }
> }
> {code}
>
>
> it is annotated with {{@Replicate}} , so it will be replicated by ratis if HA
> is enabled and it is actually called when the raft log is applied. suppose we
> have one leader L1 , and two followes F1 and F2. if the rocksdb of L1 is ok ,
> and the methoed is excuted successfully at L1. when L1 crashed , F1 is
> elected as the new leader, but the rocksdb of F1 got some error(for example ,
> can not be written) , so when F1 applied the raft log entry, it will get an
> exception, and the code of e\{{xception}} will be excuted. so the point here
> is that , current HA implementation seems could not guarantee the consistence
> of different SCM, because although different scm apply the same raft logs in
> the same order, but the result of {{@Replicate}} functions may be different
> by other factors. leader may succeed writting DB. but follower may fail when
> it is elected as a new leader , this may cause the inconsistence
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
