We had another of these delete-related RS hang ups. This time we are getting a different error on the client:
java.io.IOException: Call queue is full, is ipc.server.max.callqueue.size too small? full stack here: http://pastebin.com/uq68Mvhm Looking at the RS log, it appears the RS was working on the batch delete for about 1hr. There are no errors in the RS log during this time. There are several "responseTooSlow" messages. Based on processingtimems values they all lead back to our big batch delete. Any theories on how a big batch of deletes could cause a RS to go unresponsive?
