Hi, Qpid on production environment failed from RCO_1_FIX_VHN master to STO_1_FIX_VHN node automatically with the below log. Can I know what leads it to fail over on its own?
2019-07-20 04:22:04,035 ERROR [REPLICA RCO_1_FIX_VHN(1)] (c.s.j.r.s.ReplicaFeederSyncup) - RCO_1_FIX_VHN Limited list of transactions truncated for hard recovery: 2019-07-20 04:22:04,041 WARN [REPLICA RCO_1_FIX_VHN(1)] (c.s.j.r.i.n.Replica) - RCO_1_FIX_VHN Exiting inner Replica loop with exception com.sleepycat.je.rep.RollbackException: (JE 7.4.5) RCO_1_FIX_VHN(1):/data/qpid-work1/RCO_1_FIX_VHN/config Node RCO_1_FIX_VHN(1):/data/qpid-work1/RCO_1_FIX_VHN/config must rollback 1 total commits to the earliest point indicated by transaction id=-15655064 time=2019-07-20 04:21:47.942 vlsn=46,846,575 lsn=0x30e/0x6acaa4 durable=false in order to rejoin the replication group. All existing ReplicatedEnvironment handles must be closed and reinstantiated. Log files were truncated to file 0x782, offset 0x6998593, vlsn 46,846,573 HARD_RECOVERY: Rolled back past transaction commit or abort. Must run recovery by re-opening Environment handles Environment is invalid and must be closed. 2019-07-20 04:22:04,042 WARN [UNKNOWN RCO_1_FIX_VHN(-1)] (c.s.j.r.i.n.RepNode) - RCO_1_FIX_VHN Environment needs to be restarted: java.lang.Exception: Stacktrace where exception below was rethrown (com.sleepycat.je.rep.RollbackException) 2019-07-20 04:22:04,043 ERROR [DETACHED RCO_1_FIX_VHN(1)] (c.s.j.r.i.n.RepNode) - RCO_1_FIX_VHN RCO_1_FIX_VHN(1):/data/qpid-work1/RCO_1_FIX_VHN/config:DETACHED RCO_1_FIX_VHN(1) exited unexpectedly with exception com.sleepycat.je.rep.RollbackException: (JE 7.4.5) RCO_1_FIX_VHN(1):/data/qpid-work1/RCO_1_FIX_VHN/config Node RCO_1_FIX_VHN(1):/data/qpid-work1/RCO_1_FIX_VHN/config must rollback 1 total commits to the earliest point indicated by transaction id=-15655064 time=2019-07-20 04:21:47.942 vlsn=46,846,575 lsn=0x30e/0x6acaa4 durable=false in order to rejoin the replication group. All existing ReplicatedEnvironment handles must be closed and reinstantiated. Log files were truncated to file 0x782, offset 0x6998593, vlsn 46,846,573 HARD_RECOVERY: Rolled back past transaction commit or abort. Must run recovery by re-opening Environment handles Environment is invalid and must be closed. Originally thrown by HA thread: REPLICA RCO_1_FIX_VHN(1)com.sleepycat.je.rep.RollbackException: (JE 7.4.5) RCO_1_FIX_VHN(1):/data/qpid-work1/RCO_1_FIX_VHN/config Node RCO_1_FIX_VHN(1):/data/qpid-work1/RCO_1_FIX_VHN/config must rollback 1 total commits to the earliest point indicated by transaction id=-15655064 time=2019-07-20 04:21:47.942 vlsn=46,846,575 lsn=0x30e/0x6acaa4 durable=false in order to rejoin the replication group. All existing ReplicatedEnvironment handles must be closed and reinstantiated. Log files were truncated to file 0x782, offset 0x6998593, vlsn 46,846,573 HARD_RECOVERY: Rolled back past transaction commit or abort. Must run recovery by re-opening Environment handles Environment is invalid and must be closed. Originally thrown by HA thread: REPLICA RCO_1_FIX_VHN(1) 2019-07-20 04:22:04,043 WARN [DETACHED RCO_1_FIX_VHN(1)] (o.a.q.s.s.b.r.ReplicatedEnvironmentFacade) - FIX_VHNG:RCO_1_FIX_VHN has transaction(s) ahead of the current master. These must be discarded to allow this node to rejoin the group. This condition is normally caused by the use of weak durability options. Thanks, Balanna Nadikude Notice: All email and instant messages (including attachments) sent to or from Franklin Templeton Investments (FTI) personnel may be retained, monitored and/or reviewed by FTI and its agents, or authorized law enforcement personnel, without further notice or consent. Under new Data Protection regulations in Europe your personal information may be subject to our EU specific Privacy & Cookies Notice which you can read here, regardless of where the information originates. Depending on your location, other privacy laws and regulations may also apply to you. Please reference http://www.franklintempletonglobal.com/privacy .
