The !SYSTEM user's credentials include a hashing of all properties in
accumulo-site.xml that begin with 'instance.'. This includes
instance.secret and has two goals: 1) prevent rogue servers from
maliciously entering the Accumulo instance. 2) prevent servers with bad
configurations from accidentally entering the Accumulo instance.
BAD_CREDENTIALS on !SYSTEM likely means that the new host has (or the
processing on that host are using) a different version of
accumulo-site.xml than the other servers.
Trying `diff`'ing accumulo-site.xml on the host running your GC and one
of the hosts running a TabletServer. If they appear to be accurate, you
can check the print-out of site configuration in the debug.log for the
GC and see if it somehow picked up a different version of
accumulo-site.xml than you expected.
Russ Weeks wrote:
Hi, folks,
I just finished moving my namenode, following the instructions at [1]. I
think I got everything just about right but now I see these errors in
the monitor. They seem to be associated with the GC process.
Unable to scan metadata table
java.lang.RuntimeException:
org.apache.accumulo.core.client.AccumuloSecurityException: Error
BAD_CREDENTIALS for user !SYSTEM on table accumulo.root(ID:+r) - Username or
Password is Invalid
at
org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:187)
at
org.apache.accumulo.server.util.MetadataTableUtil.getLogEntries(MetadataTableUtil.java:554)
at
org.apache.accumulo.server.util.MetadataTableUtil$LogEntryIterator.<init>(MetadataTableUtil.java:622)
at
org.apache.accumulo.server.util.MetadataTableUtil.getLogEntries(MetadataTableUtil.java:659)
at
org.apache.accumulo.gc.GarbageCollectWriteAheadLogs.removeMetadataEntries(GarbageCollectWriteAheadLogs.java:321)
at
org.apache.accumulo.gc.GarbageCollectWriteAheadLogs.collect(GarbageCollectWriteAheadLogs.java:151)
at
org.apache.accumulo.gc.SimpleGarbageCollector.run(SimpleGarbageCollector.java:608)
at
org.apache.accumulo.gc.SimpleGarbageCollector.main(SimpleGarbageCollector.java:159)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.accumulo.start.Main$1.run(Main.java:141)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.accumulo.core.client.AccumuloSecurityException:
Error BAD_CREDENTIALS for user !SYSTEM on table accumulo.root(ID:+r) - Username
or Password is Invalid
at
org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:458)
at
org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:279)
at
org.apache.accumulo.core.client.impl.ScannerIterator$Reader.run(ScannerIterator.java:84)
at
org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:177)
... 13 more
Caused by: ThriftSecurityException(user:!SYSTEM, code:BAD_CREDENTIALS)
at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$startScan_result$startScan_resultStandardScheme.read(TabletClientService.java:6684)
at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$startScan_result$startScan_resultStandardScheme.read(TabletClientService.java:6661)
at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$startScan_result.read(TabletClientService.java:6584)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_startScan(TabletClientService.java:232)
at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.startScan(TabletClientService.java:208)
at
org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:403)
... 16 more
Any ideas? Everything seems to be working, eg. I can connect using the
shell and browse the contents of my tables.
1:
http://accumulo.apache.org/1.6/accumulo_user_manual.html#_instance_volumes_replacements
