Thanks Mike, That was my first reaction but the instance is backed up by puppet and no configuration was updated (i double checked and ran puppet manually as well as automatically after restart), Since the system was operational yesterday, So I think I can rule that out.
For other error, I did see the exact error https://lists.apache.org/thread/bobn2vhkswl6c0pkzpy8n13z087z1s6j, https://github.com/RENCI-NRIG/COMET-Accumulo/issues/14 https://markmail.org/message/bc7ijdsgqmod5p2h but those are for lot older accumulo. and server didn't go out of memory so I think that must have been fixed.. [https://opengraph.githubassets.com/a2a13484b2e7a58170dedb3c7c2ac885281f5a1590788aadd302359400e5f74c/RENCI-NRIG/COMET-Accumulo/issues/14]<https://github.com/RENCI-NRIG/COMET-Accumulo/issues/14> COMET - accumulomaster out of memory issue · Issue #14 · RENCI-NRIG/COMET-Accumulo<https://github.com/RENCI-NRIG/COMET-Accumulo/issues/14> In COMET cluster running in AWS, node running accumulomaster also hosts comet head node. In current deployment, EC2 instance is of type small which has 2GB ram. Issue: Accumulomaster process is kil... github.com -S ________________________________ From: Mike Miller <mmil...@apache.org> Sent: Tuesday, March 15, 2022 8:47 AM To: user@accumulo.apache.org <user@accumulo.apache.org> Subject: [External] Re: odd issue with accumulo 1.10.0 starting up Check your configuration. The log message indicates that there is a problem with the internal system user performing operations. The internal system user uses credentials derived from the configuration (such as the instance.secret field). Make sure your configuration is identical across all nodes in your cluster. On Tue, Mar 15, 2022 at 8:34 AM Ligade, Shailesh [USA] <ligade_shail...@bah.com<mailto:ligade_shail...@bah.com>> wrote: Hello, I am getting little odd issue with accumulo starting up on tserver i am seeing [tserver.TabletServer] ERROR: Caller doesn't have permission to get active scnas ThriftSecurityException(user:!SYSTEM, code:BAD_CREDENTIALS) on the ,aster log i am seeing ERROR: read a frame size of 1195725856, which is bigger than the maximum allowable buffer size for ALL connections. from the shell i can list all the tables but canot scan any. Monitor is shwoing tablet count 0 and unassigned tablet 1 HDFS fsck is all healthy. Any suggestions? Thanks -S
