EdColeman commented on issue #1300:
URL: https://github.com/apache/accumulo/issues/1300#issuecomment-1424513293
I think we are protected, but maybe we could do more?. I ran a test, going
from 2.1.1-SNAPSHOT to 3.0.0-SNAPSHOT using uno and a single tserver.
* setup 2.1 and 3.0
* started 2.1
* killed the 2.1 manager, all other services running
* started the 3.0 manager
What happened:
* The upgrade failed because it could not talk to the one tserver and had no
host for the metadata table. It did upgrade ZooKeeper and root but failed at
upgrading metadata (trace below)
* The 2.1 tserver remained running, but the manager could not talk to it
(AccumuloSecurityException: Error BAD_CREDENTIALS for user !SYSTEM - Username
or Password is Invalid) Also, it kept its lock in ZooKeeper
* The gc tried to run, but failed when it could not find replication that
was removed as part of the upgrade.
So without a "new" tserver and the metadata table, the upgrade only
partially succeeded and the system is now in an inconsistent state. If there
had been 3.0 tservers, the upgrade would likely have succeeded and the old
teserver would not be able to talk with the manager.
Unsure about the gc.
It may be possible to read the table_locks in ZooKeeper and then try to get
status from the tservers that are registered there BEFORE starting the upgrade.
If that works then we could a) check that the manager can talk to at least one
tserver, b) fail the upgrade if any tserver with a registered lock fails a
status check, or c) use the min tserver count property to about if there are
fewer tservers than specified
There are potential issues with either a) or b) approaches. Starting a
cluster with one tserver or a without a substantial portion of tservers might
not be the best way to proceed. If ALL tservers are required, then a transient
ZooKeeper error could abort the upgrade unnecessarily.
``
3.0 manager upgrade log:
2023-02-09T15:39:25,298 [manager.EventCoordinator] INFO : State changed from
INITIAL to HAVE_LOCK
2023-02-09T15:39:25,303 [upgrade.PreUpgradeValidation] INFO : Starting
validation on ZooKeeper ACLs
2023-02-09T15:39:25,368 [upgrade.PreUpgradeValidation] INFO : Successfully
completed validation on ZooKeeper ACLs
2023-02-09T15:39:25,375 [upgrade.UpgradeCoordinator] INFO : Upgrading
Zookeeper - current version 10 as step towards target version 11
2023-02-09T15:39:25,375 [upgrade.Upgrader10to11] INFO : upgrade of ZooKeeper
entries
2023-02-09T15:39:25,415 [manager.EventCoordinator] INFO : Upgrade status
changed from INITIAL to UPGRADED_ZOOKEEPER
2023-02-09T15:39:25,415 [metrics.MetricsUtil] INFO : initializing metrics,
enabled:false, class:
2023-02-09T15:39:25,427 [metrics.MetricsUtil] INFO : Metric producer
FateMetrics initialize
2023-02-09T15:39:25,427 [metrics.ManagerMetrics] INFO : Registered FATE
metrics module
2023-02-09T15:39:25,437 [manager.EventCoordinator] INFO : State changed from
HAVE_LOCK to NORMAL
2023-02-09T15:39:25,438 [upgrade.UpgradeCoordinator] INFO : Upgrading Root -
current version 10 as step towards target version 11
2023-02-09T15:39:25,438 [upgrade.Upgrader10to11] INFO : upgrade root -
skipping, nothing to do
2023-02-09T15:39:25,439 [manager.EventCoordinator] INFO : Upgrade status
changed from UPGRADED_ZOOKEEPER to UPGRADED_ROOT
2023-02-09T15:39:25,439 [upgrade.UpgradeCoordinator] INFO : Upgrading
Metadata - current version 10 as step towards target version 11
2023-02-09T15:39:25,439 [upgrade.Upgrader10to11] INFO : upgrade metadata
entries
2023-02-09T15:39:25,446 [manager.Manager] INFO : New servers:
[localhost:9997[100002204fb0003]]
2023-02-09T15:39:25,447 [manager.EventCoordinator] INFO : There are now 1
tablet servers
2023-02-09T15:39:25,447 [manager.Manager] INFO : tserver availability check
disabled, continuing with-1 servers. To enable, set
manager.startup.tserver.avail.min.count
2023-02-09T15:39:25,465 [manager.Manager] ERROR: unable to get tablet server
status localhost:9997[100002204fb0003]
org.apache.thrift.transport.TTransportException: Socket is closed by peer.
2023-02-09T15:39:25,465 [manager.Manager] DEBUG: unable to get tablet server
status localhost:9997[100002204fb0003]
org.apache.thrift.transport.TTransportException: Socket is closed by peer.
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:176)
~[libthrift-0.17.0.jar:0.17.0]
at
org.apache.thrift.transport.TTransport.readAll(TTransport.java:100)
~[libthrift-0.17.0.jar:0.17.0]
at
org.apache.thrift.transport.layered.TFramedTransport.readFrame(TFramedTransport.java:132)
~[libthrift-0.17.0.jar:0.17.0]
at
org.apache.thrift.transport.layered.TFramedTransport.read(TFramedTransport.java:100)
~[libthrift-0.17.0.jar:0.17.0]
at
org.apache.thrift.transport.TTransport.readAll(TTransport.java:100)
~[libthrift-0.17.0.jar:0.17.0]
at
org.apache.thrift.protocol.TCompactProtocol.readByte(TCompactProtocol.java:622)
~[libthrift-0.17.0.jar:0.17.0]
at
org.apache.thrift.protocol.TCompactProtocol.readMessageBegin(TCompactProtocol.java:479)
~[libthrift-0.17.0.jar:0.17.0]
at
org.apache.thrift.protocol.TProtocolDecorator.readMessageBegin(TProtocolDecorator.java:156)
~[libthrift-0.17.0.jar:0.17.0]
at
org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
~[libthrift-0.17.0.jar:0.17.0]
at
org.apache.accumulo.core.tabletserver.thrift.TabletServerClientService$Client.recv_getTabletServerStatus(TabletServerClientService.java:186)
~[accumulo-core-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.accumulo.core.tabletserver.thrift.TabletServerClientService$Client.getTabletServerStatus(TabletServerClientService.java:172)
~[accumulo-core-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.accumulo.server.manager.LiveTServerSet$TServerConnection.getTableMap(LiveTServerSet.java:144)
~[accumulo-server-base-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.accumulo.manager.Manager.lambda$gatherTableInformation$3(Manager.java:981)
~[accumulo-manager-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52)
~[accumulo-core-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
~[?:?]
at
org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52)
~[accumulo-core-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
...
2023-02-09T15:39:25,555 [upgrade.UpgradeCoordinator] ERROR: FATAL: Error
performing upgrade
java.lang.RuntimeException:
org.apache.accumulo.core.client.AccumuloSecurityException: Error
BAD_CREDENTIALS for user !SYSTEM - Username or Password is Invalid
at
org.apache.accumulo.core.clientImpl.ScannerIterator.getNextBatch(ScannerIterator.java:183)
~[accumulo-core-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.accumulo.core.clientImpl.ScannerIterator.hasNext(ScannerIterator.java:107)
~[accumulo-core-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.accumulo.manager.upgrade.Upgrader10to11.readReplFilesFromMetadata(Upgrader10to11.java:121)
~[accumulo-manager-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.accumulo.manager.upgrade.Upgrader10to11.upgradeMetadata(Upgrader10to11.java:111)
~[accumulo-manager-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.accumulo.manager.upgrade.UpgradeCoordinator.lambda$upgradeMetadata$0(UpgradeCoordinator.java:208)
~[accumulo-manager-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.accumulo.core.trace.TraceWrappedCallable.call(TraceWrappedCallable.java:53)
~[accumulo-core-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at
org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52)
~[accumulo-core-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
~[?:?]
at
org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52)
~[accumulo-core-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
Caused by: org.apache.accumulo.core.client.AccumuloSecurityException: Error
BAD_CREDENTIALS for user !SYSTEM - Username or Password is Invalid
at
org.apache.accumulo.core.clientImpl.ThriftScanner.getBatchFromServer(ThriftScanner.java:166)
~[accumulo-core-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.accumulo.core.metadata.MetadataLocationObtainer.lookupTablet(MetadataLocationObtainer.java:108)
~[accumulo-core-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.accumulo.core.clientImpl.TabletLocatorImpl.lookupTabletLocation(TabletLocatorImpl.java:537)
~[accumulo-core-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.accumulo.core.clientImpl.TabletLocatorImpl.lookupTabletLocationAndCheckLock(TabletLocatorImpl.java:716)
~[accumulo-core-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.accumulo.core.clientImpl.TabletLocatorImpl._locateTablet(TabletLocatorImpl.java:701)
~[accumulo-core-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.accumulo.core.clientImpl.TabletLocatorImpl.locateTablet(TabletLocatorImpl.java:505)
~[accumulo-core-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.accumulo.core.clientImpl.ThriftScanner.scan(ThriftScanner.java:316)
~[accumulo-core-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.accumulo.core.clientImpl.ScannerIterator.readBatch(ScannerIterator.java:154)
~[accumulo-core-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.accumulo.core.clientImpl.ScannerIterator.getNextBatch(ScannerIterator.java:172)
~[accumulo-core-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
... 11 more
Caused by: org.apache.accumulo.core.clientImpl.thrift.ThriftSecurityException
at
org.apache.accumulo.core.tabletscan.thrift.TabletScanClientService$startScan_result$startScan_resultStandardScheme.read(TabletScanClientService.java:4583)
~[accumulo-core-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.accumulo.core.tabletscan.thrift.TabletScanClientService$startScan_result$startScan_resultStandardScheme.read(TabletScanClientService.java:4559)
~[accumulo-core-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.accumulo.core.tabletscan.thrift.TabletScanClientService$startScan_result.read(TabletScanClientService.java:4465)
~[accumulo-core-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:93)
~[libthrift-0.17.0.jar:0.17.0]
at
org.apache.accumulo.core.tabletscan.thrift.TabletScanClientService$Client.recv_startScan(TabletScanClientService.java:121)
~[accumulo-core-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.accumulo.core.tabletscan.thrift.TabletScanClientService$Client.startScan(TabletScanClientService.java:92)
~[accumulo-core-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.accumulo.core.clientImpl.ThriftScanner.getBatchFromServer(ThriftScanner.java:137)
~[accumulo-core-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.accumulo.core.metadata.MetadataLocationObtainer.lookupTablet(MetadataLocationObtainer.java:108)
~[accumulo-core-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.accumulo.core.clientImpl.TabletLocatorImpl.lookupTabletLocation(TabletLocatorImpl.java:537)
~[accumulo-core-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.accumulo.core.clientImpl.TabletLocatorImpl.lookupTabletLocationAndCheckLock(TabletLocatorImpl.java:716)
~[accumulo-core-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.accumulo.core.clientImpl.TabletLocatorImpl._locateTablet(TabletLocatorImpl.java:701)
~[accumulo-core-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.accumulo.core.clientImpl.TabletLocatorImpl.locateTablet(TabletLocatorImpl.java:505)
~[accumulo-core-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.accumulo.core.clientImpl.ThriftScanner.scan(ThriftScanner.java:316)
~[accumulo-core-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.accumulo.core.clientImpl.ScannerIterator.readBatch(ScannerIterator.java:154)
~[accumulo-core-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.accumulo.core.clientImpl.ScannerIterator.getNextBatch(ScannerIterator.java:172)
~[accumulo-core-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
... 11 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
