Hi Balanna, The Qpid Broker-J HA solution is based on Oracle BDB JE HA library.
The stack trace indicate that data replication from master to replica ended up in unexpected exception thrown from BDB JE: "RCO_1_FIX_VHN Limited list of transactions truncated for hard recovery" The following log entry suggests that BDB JE file was truncated in some way: "Log files were truncated to file 0x782, offset 0x6998593, vlsn 46,846,573 HARD_RECOVERY: Rolled back past transaction commit or abort". I have never seen this exception before and cannot tell you what has caused it. I would like to suggest to seek for help from BDB JE community [1]. I hope that impacted BDB HA Virtual host node had recovered successfully from the incident. Please, let me know if you have to restart broker in order to recover. Out of curiosity, do you run BDB HA nodes using default synchronization policies? If not, what changes have been made? Kind Regards, Alex [1] https://community.oracle.com/community/groundbreakers/database/berkeley_db_family/berkeley_db_java_edition On Mon, 22 Jul 2019 at 13:57, Nadikude, Balanna < [email protected]> wrote: > Hi, > > Qpid on production environment failed from RCO_1_FIX_VHN master to > STO_1_FIX_VHN node automatically with the below log. Can I know what leads > it to fail over on its own? > > > > 2019-07-20 04:22:04,035 ERROR [REPLICA RCO_1_FIX_VHN(1)] > (c.s.j.r.s.ReplicaFeederSyncup) - RCO_1_FIX_VHN Limited list of > transactions truncated for hard recovery: > > > > 2019-07-20 04:22:04,041 WARN [REPLICA RCO_1_FIX_VHN(1)] > (c.s.j.r.i.n.Replica) - RCO_1_FIX_VHN Exiting inner Replica loop with > exception com.sleepycat.je.rep.RollbackException: (JE 7.4.5) > RCO_1_FIX_VHN(1):/data/qpid-work1/RCO_1_FIX_VHN/config Node > RCO_1_FIX_VHN(1):/data/qpid-work1/RCO_1_FIX_VHN/config must rollback 1 > total commits to the earliest point indicated by transaction id=-15655064 > time=2019-07-20 04:21:47.942 vlsn=46,846,575 lsn=0x30e/0x6acaa4 > durable=false in order to rejoin the replication group. All existing > ReplicatedEnvironment handles must be closed and reinstantiated. Log files > were truncated to file 0x782, offset 0x6998593, vlsn 46,846,573 > HARD_RECOVERY: Rolled back past transaction commit or abort. Must run > recovery by re-opening Environment handles Environment is invalid and must > be closed. > > > > 2019-07-20 04:22:04,042 WARN [UNKNOWN RCO_1_FIX_VHN(-1)] > (c.s.j.r.i.n.RepNode) - RCO_1_FIX_VHN Environment needs to be restarted: > java.lang.Exception: Stacktrace where exception below was rethrown > (com.sleepycat.je.rep.RollbackException) > > > > 2019-07-20 04:22:04,043 ERROR [DETACHED RCO_1_FIX_VHN(1)] > (c.s.j.r.i.n.RepNode) - RCO_1_FIX_VHN > RCO_1_FIX_VHN(1):/data/qpid-work1/RCO_1_FIX_VHN/config:DETACHED > RCO_1_FIX_VHN(1) exited unexpectedly with exception > com.sleepycat.je.rep.RollbackException: (JE 7.4.5) > RCO_1_FIX_VHN(1):/data/qpid-work1/RCO_1_FIX_VHN/config Node > RCO_1_FIX_VHN(1):/data/qpid-work1/RCO_1_FIX_VHN/config must rollback 1 > total commits to the earliest point indicated by transaction id=-15655064 > time=2019-07-20 04:21:47.942 vlsn=46,846,575 lsn=0x30e/0x6acaa4 > durable=false in order to rejoin the replication group. All existing > ReplicatedEnvironment handles must be closed and reinstantiated. Log files > were truncated to file 0x782, offset 0x6998593, vlsn 46,846,573 > HARD_RECOVERY: Rolled back past transaction commit or abort. Must run > recovery by re-opening Environment handles Environment is invalid and must > be closed. Originally thrown by HA thread: REPLICA > RCO_1_FIX_VHN(1)com.sleepycat.je.rep.RollbackException: (JE 7.4.5) > RCO_1_FIX_VHN(1):/data/qpid-work1/RCO_1_FIX_VHN/config Node > RCO_1_FIX_VHN(1):/data/qpid-work1/RCO_1_FIX_VHN/config must rollback 1 > total commits to the earliest point indicated by transaction id=-15655064 > time=2019-07-20 04:21:47.942 vlsn=46,846,575 lsn=0x30e/0x6acaa4 > durable=false in order to rejoin the replication group. All existing > ReplicatedEnvironment handles must be closed and reinstantiated. Log files > were truncated to file 0x782, offset 0x6998593, vlsn 46,846,573 > HARD_RECOVERY: Rolled back past transaction commit or abort. Must run > recovery by re-opening Environment handles Environment is invalid and must > be closed. Originally thrown by HA thread: REPLICA RCO_1_FIX_VHN(1) > > > > 2019-07-20 04:22:04,043 WARN [DETACHED RCO_1_FIX_VHN(1)] > (o.a.q.s.s.b.r.ReplicatedEnvironmentFacade) - FIX_VHNG:RCO_1_FIX_VHN has > transaction(s) ahead of the current master. These must be discarded to > allow this node to rejoin the group. This condition is normally caused by > the use of weak durability options. > > > Thanks, > Balanna Nadikude > > Notice: All email and instant messages (including attachments) sent to or > from Franklin Templeton Investments (FTI) personnel may be retained, > monitored and/or reviewed by FTI and its agents, or authorized law > enforcement personnel, without further notice or consent. > > Under new Data Protection regulations in Europe your personal information > may be subject to our EU specific Privacy & Cookies Notice which you can > read here, regardless of where the information originates. Depending on > your location, other privacy laws and regulations may also apply to you. > Please reference http://www.franklintempletonglobal.com/privacy > > . >
