I'm not sure what to make of it, but restarting my cluster running today's checkout of code gives me a terrible state. Some random fraction of my datanodes seem to be showing up as having no blocks, probably because of a NPE during the IPC processing block reports. My logs are riddled with sequences like these:
2007-01-17 10:55:39,716 WARN org.apache.hadoop.dfs.StateChange: BLOCK* NameSystem.addStoredBlock: Redundant addStoredBlock request received for blk_-2540546879629459821 on hostXX:50010 2007-01-17 10:55:39,716 WARN org.apache.hadoop.dfs.StateChange: BLOCK* NameSystem.addStoredBlock: Redundant addStoredBlock request received for blk_-2088395555965573137 on hostXX:50010 2007-01-17 10:55:39,716 WARN org.apache.hadoop.dfs.StateChange: BLOCK* NameSystem.addStoredBlock: Redundant addStoredBlock request received for blk_-1801448787921913203 on hostXX:50010 2007-01-17 10:55:39,716 WARN org.apache.hadoop.dfs.StateChange: BLOCK* NameSystem.addStoredBlock: Redundant addStoredBlock request received for blk_-1732709916964926315 on hostXX:50010 2007-01-17 10:55:39,716 WARN org.apache.hadoop.dfs.StateChange: BLOCK* NameSystem.addStoredBlock: Redundant addStoredBlock request received for blk_-1582025333970910026 on hostXX:50010 2007-01-17 10:55:39,716 WARN org.apache.hadoop.dfs.StateChange: BLOCK* NameSystem.addStoredBlock: Redundant addStoredBlock request received for blk_-869004184333473970 on hostXX:50010 2007-01-17 10:55:39,716 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 9000 call error: java.io.IOException: java.lang.NullPointerException java.io.IOException: java.lang.NullPointerException -- Bryan A. P. Pendleton Ph: (877) geek-1-bp
