Thanks Sean, thanks Josh. To be precise: we're running Accumulo 1.6.1.2.2.0.0-2041 provided by HDP with a hotfix for ACCUMULO-3312.
I did an md5sum on all my accumulo-site.xml's and they all matched... I have a feeling that I just didn't restart the GC properly. I gave it a good kick, things seem OK now. Just curious Sean, which part of that stack trace diverges from the Apache Accumulo distribution? Regards, -Russ On Fri, Mar 6, 2015 at 10:06 AM, Josh Elser <[email protected]> wrote: > The !SYSTEM user's credentials include a hashing of all properties in > accumulo-site.xml that begin with 'instance.'. This includes > instance.secret and has two goals: 1) prevent rogue servers from > maliciously entering the Accumulo instance. 2) prevent servers with bad > configurations from accidentally entering the Accumulo instance. > > BAD_CREDENTIALS on !SYSTEM likely means that the new host has (or the > processing on that host are using) a different version of accumulo-site.xml > than the other servers. > > Trying `diff`'ing accumulo-site.xml on the host running your GC and one of > the hosts running a TabletServer. If they appear to be accurate, you can > check the print-out of site configuration in the debug.log for the GC and > see if it somehow picked up a different version of accumulo-site.xml than > you expected. > > > Russ Weeks wrote: > >> Hi, folks, >> >> I just finished moving my namenode, following the instructions at [1]. I >> think I got everything just about right but now I see these errors in >> the monitor. They seem to be associated with the GC process. >> >> Unable to scan metadata table >> java.lang.RuntimeException: >> org.apache.accumulo.core.client.AccumuloSecurityException: >> Error BAD_CREDENTIALS for user !SYSTEM on table accumulo.root(ID:+r) - >> Username or Password is Invalid >> at org.apache.accumulo.core.client.impl.ScannerIterator. >> hasNext(ScannerIterator.java:187) >> at org.apache.accumulo.server.util.MetadataTableUtil. >> getLogEntries(MetadataTableUtil.java:554) >> at org.apache.accumulo.server.util.MetadataTableUtil$ >> LogEntryIterator.<init>(MetadataTableUtil.java:622) >> at org.apache.accumulo.server.util.MetadataTableUtil. >> getLogEntries(MetadataTableUtil.java:659) >> at org.apache.accumulo.gc.GarbageCollectWriteAheadLogs. >> removeMetadataEntries(GarbageCollectWriteAheadLogs.java:321) >> at org.apache.accumulo.gc.GarbageCollectWriteAheadLogs.collect( >> GarbageCollectWriteAheadLogs.java:151) >> at org.apache.accumulo.gc.SimpleGarbageCollector.run( >> SimpleGarbageCollector.java:608) >> at org.apache.accumulo.gc.SimpleGarbageCollector.main( >> SimpleGarbageCollector.java:159) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at sun.reflect.NativeMethodAccessorImpl.invoke( >> NativeMethodAccessorImpl.java:57) >> at sun.reflect.DelegatingMethodAccessorImpl.invoke( >> DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:606) >> at org.apache.accumulo.start.Main$1.run(Main.java:141) >> at java.lang.Thread.run(Thread.java:745) >> Caused by: org.apache.accumulo.core.client.AccumuloSecurityException: >> Error BAD_CREDENTIALS for user !SYSTEM on table accumulo.root(ID:+r) - >> Username or Password is Invalid >> at org.apache.accumulo.core.client.impl.ThriftScanner. >> scan(ThriftScanner.java:458) >> at org.apache.accumulo.core.client.impl.ThriftScanner. >> scan(ThriftScanner.java:279) >> at org.apache.accumulo.core.client.impl.ScannerIterator$ >> Reader.run(ScannerIterator.java:84) >> at org.apache.accumulo.core.client.impl.ScannerIterator. >> hasNext(ScannerIterator.java:177) >> ... 13 more >> Caused by: ThriftSecurityException(user:!SYSTEM, >> code:BAD_CREDENTIALS) >> at org.apache.accumulo.core.tabletserver.thrift. >> TabletClientService$startScan_result$startScan_resultStandardScheme.read( >> TabletClientService.java:6684) >> at org.apache.accumulo.core.tabletserver.thrift. >> TabletClientService$startScan_result$startScan_resultStandardScheme.read( >> TabletClientService.java:6661) >> at org.apache.accumulo.core.tabletserver.thrift. >> TabletClientService$startScan_result.read(TabletClientService.java:6584) >> at org.apache.thrift.TServiceClient.receiveBase( >> TServiceClient.java:78) >> at org.apache.accumulo.core.tabletserver.thrift. >> TabletClientService$Client.recv_startScan(TabletClientService.java:232) >> at org.apache.accumulo.core.tabletserver.thrift. >> TabletClientService$Client.startScan(TabletClientService.java:208) >> at org.apache.accumulo.core.client.impl.ThriftScanner. >> scan(ThriftScanner.java:403) >> ... 16 more >> >> >> Any ideas? Everything seems to be working, eg. I can connect using the >> shell and browse the contents of my tables. >> >> 1: >> http://accumulo.apache.org/1.6/accumulo_user_manual.html#_ >> instance_volumes_replacements >> >
