Thanks Ed/Chris, yes replication is table is hosted only one tablet server and I looked at that tablet server, there are no errors.
I am seeing all this errors in active master server log. zkcli to state the replication table showed its online. I guess I will restart masters, if that doesn't work then restart that just one tserver holding the replication tablet and if that fails then restart entire cluster 🙁 Will remove write grant from root and replication user. Appreciated your help -S ________________________________ From: dev1 <d...@etcoleman.com> Sent: Tuesday, January 4, 2022 1:18 PM To: 'user@accumulo.apache.org' <user@accumulo.apache.org> Subject: [External] RE: replication table offline issue Deleting / recreating the replication table should not be necessary and in any case you very likely cannot delete / create the accumulo.replication table – the shell will error on the delete because it is in the accumulo namespace. Is the replication table hosted on a single tserver? Are there any exceptions in the log for that server? (or any of the tservers host it if hosted across multiple tservers) Have you restarted the client? It looks like the exception fragment has client in the classname. What log is that exception occurring? You can try restarting the master(s) The monitor shows the replication table is online? Can you check in ZooKeeper (using the zkCli.sh) * get /accumulo/[instance id]/tables/+rep/state That should return the text ONLINE If the replication table is on a single tserver, then you might be able to just restart that server rather than needing to do a rolling restart of the cluster. If there a no errors in the tserver log this seems unlikely to help. Ed Coleman From: Ligade, Shailesh [USA] <ligade_shail...@bah.com> Sent: Tuesday, January 4, 2022 12:24 PM To: user@accumulo.apache.org Subject: Re: replication table offline issue Sorry this is for accumulo 1.10.0 I am wondering is there a way to delete and recreate the accumulo.replication table. I know it is bit special table..so Also, will restarting entire cluster solve this? or may be just restarting accumulo master may be? Since rolling restart of tservers is bit lengthy process for us just wanted to make sure it may resolve it or not.. -S ________________________________ From: Ligade, Shailesh [USA] Sent: Tuesday, January 4, 2022 11:27 AM To: user@accumulo.apache.org<mailto:user@accumulo.apache.org> <user@accumulo.apache.org<mailto:user@accumulo.apache.org>> Subject: replication table offline issue Hello, I setup replication and ran 'online accumulo.replication' however i n master i keep on getting error stating accumulo.replication is offline. I can scan accumulo.replication table it has no data at all the error is: ------------------------------------- WARN Failed to write work mutations for replication, will retry clinet.MutationRejectedException: # constraint violations: 0 security codes: {} # server errors 0 # exceptions 6 at xxxclient.impl.TabletServerBatchWriter.checkForFailures .... caused by TableOfflineException: Table accumulo.replication (+rep) is offline --------------------------- There are no constraints that I am using on any table. I added grants for root as well as my replication user for accumulo.replication Tbale.WRITE (there was only Table.READ before) if i run offline accumulo.replication i can see it is offline and then i can bring in online again however i still keep getting error Any suggestion on how to fix this? Thanks -S
