[
https://issues.apache.org/jira/browse/HDDS-15783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Doroszlai updated HDDS-15783:
------------------------------------
Description:
During an SCM HA cluster upgrade, SCM fails while replaying the Ratis log with
the following error:
{code:java}
2026-07-05 10:00:12,699 ERROR
[36209121-a632-4067-a36d-a6e0194cd83e@group-65230FEE3C3A-StateMachineUpdater]-org.apache.ratis.statemachine.StateMachine:
Terminating with exit status 1: Failed to resolve
java.lang.Objectorg.apache.ratis.thirdparty.com.google.protobuf.InvalidProtocolBufferException:
Failed to resolve java.lang.Object at
org.apache.hadoop.hdds.scm.ha.io.ScmCodecFactory$ClassResolver.get(ScmCodecFactory.java:177)
at
org.apache.hadoop.hdds.scm.ha.io.ScmCodecFactory$ClassResolver.get(ScmCodecFactory.java:147)
at
org.apache.hadoop.hdds.scm.ha.io.ScmListCodec.deserialize(ScmListCodec.java:72)
at
org.apache.hadoop.hdds.scm.ha.SCMRatisRequest.decode(SCMRatisRequest.java:154)
at
org.apache.hadoop.hdds.scm.ha.SCMStateMachine.applyTransaction(SCMStateMachine.java:146)
at
org.apache.ratis.server.impl.RaftServerImpl.applyLogToStateMachine(RaftServerImpl.java:1888)
at
org.apache.ratis.server.impl.StateMachineUpdater.applyLog(StateMachineUpdater.java:255)
at
org.apache.ratis.server.impl.StateMachineUpdater.run(StateMachineUpdater.java:194)
at java.base/java.lang.Thread.run(Thread.java:833)2026-07-05 10:00:12,702
INFO
[shutdown-hook-0]-org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter:
SHUTDOWN_MSG: {code}
*Root cause*
An empty List/ArrayList was written into the SCM Ratis log, and the new
ScmListCodec cannot deserialize it.On serialize, empty lists are stored with
element type java.lang.Object. On deserialize, the codec always tries to
resolve that type and get an element codec — even when there are zero
elements.This is a regression from the reflection-based codec refactor
(HDDS-14740/HDDS-14846). The old ListCodec had the same Object.class sentinel
on serialize, but on deserialize it only loaded the element codec inside the
value loop
was:
During an SCM HA cluster upgrade, SCM fails while replaying the Ratis log with
the following error:
{code:java}
2026-07-05 10:00:12,699 ERROR
[36209121-a632-4067-a36d-a6e0194cd83e@group-65230FEE3C3A-StateMachineUpdater]-org.apache.ratis.statemachine.StateMachine:
Terminating with exit status 1: Failed to resolve
java.lang.Objectorg.apache.ratis.thirdparty.com.google.protobuf.InvalidProtocolBufferException:
Failed to resolve java.lang.Object at
org.apache.hadoop.hdds.scm.ha.io.ScmCodecFactory$ClassResolver.get(ScmCodecFactory.java:177)
at
org.apache.hadoop.hdds.scm.ha.io.ScmCodecFactory$ClassResolver.get(ScmCodecFactory.java:147)
at
org.apache.hadoop.hdds.scm.ha.io.ScmListCodec.deserialize(ScmListCodec.java:72)
at
org.apache.hadoop.hdds.scm.ha.SCMRatisRequest.decode(SCMRatisRequest.java:154)
at
org.apache.hadoop.hdds.scm.ha.SCMStateMachine.applyTransaction(SCMStateMachine.java:146)
at
org.apache.ratis.server.impl.RaftServerImpl.applyLogToStateMachine(RaftServerImpl.java:1888)
at
org.apache.ratis.server.impl.StateMachineUpdater.applyLog(StateMachineUpdater.java:255)
at
org.apache.ratis.server.impl.StateMachineUpdater.run(StateMachineUpdater.java:194)
at java.base/java.lang.Thread.run(Thread.java:833)2026-07-05 10:00:12,702
INFO
[shutdown-hook-0]-org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter:
SHUTDOWN_MSG: {code}
*Root cause*
An empty List/ArrayList was written into the SCM Ratis log, and the new
ScmListCodec cannot deserialize it.On serialize, empty lists are stored with
element type java.lang.Object. On deserialize, the codec always tries to
resolve that type and get an element codec — even when there are zero
elements.This is a regression from the reflection-based codec refactor
(HDDS-14740/[HDDS-14846|https://issues.apache.org/jira/browse/HDDS-14846]). The
old ListCodec had the same Object.class sentinel on serialize, but on
deserialize it only loaded the element codec inside the value loop
> ScmListCodec fails to deserialize empty lists from Ratis log due to
> java.lang.Object element type regression
> ------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-15783
> URL: https://issues.apache.org/jira/browse/HDDS-15783
> Project: Apache Ozone
> Issue Type: Bug
> Components: SCM, SCM HA
> Reporter: Priyesh K
> Assignee: Priyesh K
> Priority: Major
> Labels: pull-request-available
>
> During an SCM HA cluster upgrade, SCM fails while replaying the Ratis log
> with the following error:
> {code:java}
> 2026-07-05 10:00:12,699 ERROR
> [36209121-a632-4067-a36d-a6e0194cd83e@group-65230FEE3C3A-StateMachineUpdater]-org.apache.ratis.statemachine.StateMachine:
> Terminating with exit status 1: Failed to resolve
> java.lang.Objectorg.apache.ratis.thirdparty.com.google.protobuf.InvalidProtocolBufferException:
> Failed to resolve java.lang.Object at
> org.apache.hadoop.hdds.scm.ha.io.ScmCodecFactory$ClassResolver.get(ScmCodecFactory.java:177)
> at
> org.apache.hadoop.hdds.scm.ha.io.ScmCodecFactory$ClassResolver.get(ScmCodecFactory.java:147)
> at
> org.apache.hadoop.hdds.scm.ha.io.ScmListCodec.deserialize(ScmListCodec.java:72)
> at
> org.apache.hadoop.hdds.scm.ha.SCMRatisRequest.decode(SCMRatisRequest.java:154)
> at
> org.apache.hadoop.hdds.scm.ha.SCMStateMachine.applyTransaction(SCMStateMachine.java:146)
> at
> org.apache.ratis.server.impl.RaftServerImpl.applyLogToStateMachine(RaftServerImpl.java:1888)
> at
> org.apache.ratis.server.impl.StateMachineUpdater.applyLog(StateMachineUpdater.java:255)
> at
> org.apache.ratis.server.impl.StateMachineUpdater.run(StateMachineUpdater.java:194)
> at java.base/java.lang.Thread.run(Thread.java:833)2026-07-05 10:00:12,702
> INFO
> [shutdown-hook-0]-org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter:
> SHUTDOWN_MSG: {code}
> *Root cause*
> An empty List/ArrayList was written into the SCM Ratis log, and the new
> ScmListCodec cannot deserialize it.On serialize, empty lists are stored with
> element type java.lang.Object. On deserialize, the codec always tries to
> resolve that type and get an element codec — even when there are zero
> elements.This is a regression from the reflection-based codec refactor
> (HDDS-14740/HDDS-14846). The old ListCodec had the same Object.class sentinel
> on serialize, but on deserialize it only loaded the element codec inside the
> value loop
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
