Hi hdfs-devs, I am reaching out to better understand how to resolve the following scenario where one of the JournalNodes has fallen behind (NameNode is no longer writing to it) and is no longer able to sync to the latest state.
We are considering a couple of options: 1. Shutdown the JN that has fallen behind. Perform a failover of the NN, and then bring the JN back up 2. Perform the NN failover without shutting down the JN and rely on the recovery process during the failover to bring the JN in sync Our goal is to bring all 3 JNs back to a healthy state without downtime and without corrupting Namenode state *Couple of questions:1. What would you recommend?2. We also want to better understand how we may have entered this state where the Namenode has completely stopped writing to the third JN* *Setup** 2 Namenodes - Active/Standby setup * 3 Journal Nodes - QJM setup *Current StateThis is what we observe in the Namenode UI* * The Active NameNode is only writing to two of the Journal Nodes QJM to [10.68.x.x1:8485, 10.68.x.x2:8485] Writing segment beginning at txid 177805797437. 10.68.x.x1:8485 (Written txid 177806213375), 10.68.x.x2:8485 (Written txid 177806213366) Journal ManagerState QJM to [10.68.x.x1:8485, 10.68.x.x2:8485, 10.68.x.x3:8485] open for read* The Journal Node that is not being written to has fallen behind and it keeps logging the following error: *ERROR server.Journal: The endTxId of the temporary file is not less than the last committed transaction id. Aborting move to final file/pay/hadoop/shared/hadoop-yy/current/edits_0000000177752370616-0000000177752706122 ; journal id: hadoop-yy* ** The active and standby NameNodes seem to be in sync. The FS images and md5 hashes are being updated continuously and are an exact match.* Thanks Gaurav
