In many of my jobs I create intermediate data in HDFS. I keep this around for a number of days for inspection until I delete it in large batches.
If I attempt to delete all of it at once, the flood of delete messages to the datanodes seems to cause starvation as they do not seem to responding to the namenode heartbeat. There is about <5% cpu utilization and the logs just show the deletion of blocks. In the worst case (if I'm deleting a few terabytes, about 50% of total capacity across 10 nodes) this causes the master to expire the datanode leases. Once the datanode finishes deletions, it reports back to master and is added back to the cluster. At this point, its blocks have already started to be reassigned so it then starts as an empty node. In one run, this happened to 8 out of 10 nodes before getting back to a steady state. There were a couple moments during that run that a number of the blocks had replication 1. Obviously I can handle this by deleting less at any one time, but it seems like there might be something wrong. With no CPU utilization, why does the datanode not respond to the namenode? Thanks. Jonathan Gray
