I don't see a crash log in your snippets. Mind pastebinning the NN crash log up somewhere? Did both NNs go down?
In any case, the log below is due to a client attempting to connect with an older HDFS library. This would log such warns (and also indicate the client IP/attempt port as you notice), but in no case will it cause an NN crash. > 2013-02-13 10:20:02,550 WARN ipc.Server - Incorrect header or version > mismatch from10.232.29.4:49938 got version 4 expected version 7 On Fri, Feb 15, 2013 at 11:58 PM, Dheeren Bebortha <[email protected]> wrote: > > > HI, > In one of our test clusters that has Namenode HA using QJM+ YARN + HBase > 0.94, namenode came down with following logs. I am trying to root cause the > issue. Any help is appreciated. > ================= > 2013-02-13 10:18:27,521 INFO hdfs.StateChange - BLOCK* > NameSystem.fsync: file > /hbase/.logs/datanode-X.sfdomain.com,60020,1360091866476/datanode-X.sfdomain.com%2C60020%2C1360091866476.1360750706694 > for DFSClient_hb_rs_datanode-X.sfdomain.com,60020,1360091866476_470800334_38 > 2013-02-13 10:20:01,861 WARN ipc.Server - Incorrect header or version > mismatch from 10.232.29.4:49933 got version 4 expected version 7 > 2013-02-13 10:20:01,884 WARN ipc.Server - Incorrect header or version > mismatch from 10.232.29.4:49935 got version 4 expected version 7 > 2013-02-13 10:20:02,550 WARN ipc.Server - Incorrect header or version > mismatch from 10.232.29.4:49938 got version 4 expected version 7 > 2013-02-13 10:20:08,210 INFO namenode.FSNamesystem - Roll Edit Log from > 10.232.29.14 > ===== > ===== > ====== > 2013-02-13 12:14:32,879 INFO namenode.FileJournalManager - Finalizing edits > file /data/hdfs/current/edits_inprogress_0000000000000065699 -> > /data/hdfs/current/edits_0000000000000065699-0000000000000065700 > 2013-02-13 12:14:32,879 INFO namenode.FSEditLog - Starting log segment at > 65701 > 2013-02-13 12:15:02,507 INFO namenode.NameNode - FSCK started by sfdc > (auth:SIMPLE) from /10.232.29.4 for path / at Wed Feb 13 12:15:02 > GMT+00:00 2013 > 2013-02-13 12:15:02,663 WARN ipc.Server - Incorrect header or version > mismatch from 10.232.29.4:40025 got version 4 expected version 7 > 2013-02-13 12:15:02,663 WARN ipc.Server - Incorrect header or version > mismatch from 10.232.29.4:40027 got version 4 expected version 7 > 2013-02-13 12:15:03,391 WARN ipc.Server - Incorrect header or version > mismatch from 10.232.29.4:40031 got version 4 expected version 7 > 2013-02-13 12:16:33,181 INFO namenode.FSNamesystem - Roll Edit Log from > 10.232.29.14 > ====================== > ====================== -- Harsh J
