Hey! I would try to spin up a new node and see if the problem occurs on it. If it happens, I would check the history of changes on the cookbook recipe, if you don't find any problem on the new node you might replace all the nodes having problems one by one with a new one and decommission the affected ones. it would cost some time and money but better than having a node tool not working
Best, Sergio Il giorno dom 26 feb 2023 alle ore 10:51 Abe Ratnofsky <a...@aber.io> ha scritto: > Hey Mitch, > > The security upgrade schedule that your colleague is working on may well > be relevant. Is your entire cluster on 3.11.6 or are the failing hosts > possibly on a newer version? > > Abe > > On Feb 26, 2023, at 10:38, Mitch Gitman <mgit...@gmail.com> wrote: > > > > We're running Cassandra 3.11.6 on AWS EC2 instances. These clusters have > been running for a few years. > > > We're suddenly noticing now that on one of our clusters the nodetool > command is failing on certain nodes but not on others. > > > The failure: > > nodetool: Failed to connect to '...:7199' - SecurityException: 'Access > denied! Invalid access level for requested MBeanServer operation.'. > > > I suspect that this stems from some colleague I'm not in coordination with > recently doing some security upgrades, but that's a bit of an academic > matter for now. > > > I've compared the jmxremote.access and jvm.options files on a host where > nodetool is not working vs. a host where nodetool is working, and no > meaningful differences. > > > Any ideas? The interesting aspect of this problem is that it is occurring > on some nodes in the one cluster but not others. > > > I'll update on this thread if I find any solutions on my end. > >
