ivakegg opened a new issue, #3909: URL: https://github.com/apache/accumulo/issues/3909
So I was updating the service executor pools configuration on a system and inadvertantly missed the final bracked on the property. This results in "invalid json". All of the tservers then started spamming, in what appeared to be a very tight loop, this error. Many tserver subsequently ran out of memory and others were left in a bad state and became unresponsive (deemed unreachable but did not halt). Also the monitor got pounded so hard with messages that the node itself became unresponsive. Unfortunately the manager was running on the same node and became unreachable as well. I was able to rectify the property using the property editor and after a while the system calmed down. However I still had to drop the entire cluster to be able to recover all of the tservers that were in a bad state. Basically, setting a json property with invalid json should not take down the system. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
