Public bug reported: Ubuntu 16.04 includes etcd 2.2.5+dfsg-1ubuntu1 Ubuntu 18.04 includes etcd 3.2.17+dfsg-1
So an upgrade from 16.04 to 18.04 involves a direct upgrade from etcd 2.2 to 3.2. According to the docs, this is not supported. And indeed it doesn't work. After the upgrade etcd server refuses to start: May 27 13:21:06 hims systemd[1]: Starting etcd - highly-available key value store... May 27 13:21:06 hims etcd[15185]: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=http://999.999.999.233:2379 May 27 13:21:06 hims etcd[15185]: recognized and used environment variable ETCD_DATA_DIR=/var/lib/etcd/default May 27 13:21:06 hims etcd[15185]: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=http://999.999.999.233:2380 May 27 13:21:06 hims etcd[15185]: recognized and used environment variable ETCD_INITIAL_CLUSTER=akran.XXX.YY.ZZ=http://999.999.999.232:2380,hims.XXX.YY. May 27 13:21:06 hims etcd[15185]: recognized and used environment variable ETCD_INITIAL_CLUSTER_STATE=new May 27 13:21:06 hims etcd[15185]: recognized and used environment variable ETCD_INITIAL_CLUSTER_TOKEN=XXXXXXXXXXXX May 27 13:21:06 hims etcd[15185]: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379 May 27 13:21:06 hims etcd[15185]: recognized and used environment variable ETCD_LISTEN_PEER_URLS=http://0.0.0.0:2380 May 27 13:21:06 hims etcd[15185]: recognized and used environment variable ETCD_NAME=hims.XXX.YY.ZZ May 27 13:21:06 hims etcd[15185]: etcd Version: 3.2.17 May 27 13:21:06 hims etcd[15185]: Git SHA: Not provided (use ./build instead of go build) May 27 13:21:06 hims etcd[15185]: Go Version: go1.10 May 27 13:21:06 hims etcd[15185]: Go OS/Arch: linux/amd64 May 27 13:21:06 hims etcd[15185]: setting maximum number of CPUs to 24, total number of available CPUs is 24 May 27 13:21:06 hims etcd[15185]: the server is already initialized as member before, starting as etcd member... May 27 13:21:06 hims etcd[15185]: listening for peers on http://0.0.0.0:2380 May 27 13:21:06 hims etcd[15185]: listening for client requests on 0.0.0.0:2379 May 27 13:21:06 hims etcd[15185]: recovered store from snapshot at index 147977273 May 27 13:21:06 hims etcd[15185]: recovering backend from snapshot error: database snapshot file path error: snap: snapshot file doesn't exist May 27 13:21:06 hims etcd[15185]: panic: recovering backend from snapshot error: database snapshot file path error: snap: snapshot file doesn't exist May 27 13:21:06 hims etcd[15185]: panic: runtime error: invalid memory address or nil pointer dereference May 27 13:21:06 hims etcd[15185]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0xb3ad00] May 27 13:21:06 hims etcd[15185]: goroutine 1 [running]: May 27 13:21:06 hims etcd[15185]: github.com/coreos/etcd/etcdserver.NewServer.func1(0xc4202b6628, 0xc4202b6400) May 27 13:21:06 hims etcd[15185]: /build/etcd-4uSRAK/etcd-3.2.17+dfsg/_build/src/github.com/coreos/etcd/etcdserver/server.go:284 +0x40 May 27 13:21:06 hims etcd[15185]: panic(0xd38d20, 0xc4207960f0) May 27 13:21:06 hims etcd[15185]: /usr/lib/go-1.10/src/runtime/panic.go:505 +0x229 May 27 13:21:06 hims etcd[15185]: github.com/coreos/pkg/capnslog.(*PackageLogger).Panicf(0xc42020d240, 0xf1f30b, 0x2a, 0xc4202b64a0, 0x1, 0x1) May 27 13:21:06 hims etcd[15185]: /build/etcd-4uSRAK/etcd-3.2.17+dfsg/_build/src/github.com/coreos/pkg/capnslog/pkg_logger.go:75 +0x162 May 27 13:21:06 hims etcd[15185]: github.com/coreos/etcd/etcdserver.NewServer(0xc420272780, 0xc420272780, 0xfc6ca0, 0xc420796040) May 27 13:21:06 hims etcd[15185]: /build/etcd-4uSRAK/etcd-3.2.17+dfsg/_build/src/github.com/coreos/etcd/etcdserver/server.go:379 +0x2589 May 27 13:21:06 hims etcd[15185]: github.com/coreos/etcd/embed.StartEtcd(0xc420215c00, 0xc420340000, 0x0, 0x0) May 27 13:21:06 hims etcd[15185]: /build/etcd-4uSRAK/etcd-3.2.17+dfsg/_build/src/github.com/coreos/etcd/embed/etcd.go:157 +0x741 May 27 13:21:06 hims etcd[15185]: github.com/coreos/etcd/etcdmain.startEtcd(0xc420215c00, 0xeff649, 0x6, 0xc4202b7201, 0x2) May 27 13:21:06 hims etcd[15185]: /build/etcd-4uSRAK/etcd-3.2.17+dfsg/_build/src/github.com/coreos/etcd/etcdmain/etcd.go:186 +0x73 May 27 13:21:06 hims etcd[15185]: github.com/coreos/etcd/etcdmain.startEtcdOrProxyV2() May 27 13:21:06 hims etcd[15185]: /build/etcd-4uSRAK/etcd-3.2.17+dfsg/_build/src/github.com/coreos/etcd/etcdmain/etcd.go:103 +0x1369 May 27 13:21:06 hims etcd[15185]: github.com/coreos/etcd/etcdmain.Main() May 27 13:21:06 hims etcd[15185]: /build/etcd-4uSRAK/etcd-3.2.17+dfsg/_build/src/github.com/coreos/etcd/etcdmain/main.go:39 +0x12b May 27 13:21:06 hims etcd[15185]: main.main() May 27 13:21:06 hims etcd[15185]: /build/etcd-4uSRAK/etcd-3.2.17+dfsg/_build/src/github.com/coreos/etcd/main.go:28 +0x20 If I clear out /var/lib/etcd and attempt to re-join the cluster, I get the error message: May 27 15:22:48 hims etcd[38731]: the running cluster version(2.2.0) is lower than the minimal cluster version(3.0.0) supported May 27 15:22:48 hims etcd[38731]: incompatible with current running cluster So it is necessary at least to upgrade the whole cluster from 2.2 to 3.0 (possibly via 2.3) before upgrading nodes to 3.2. ProblemType: Bug DistroRelease: Ubuntu 18.04 Package: etcd-server 3.2.17+dfsg-1 ProcVersionSignature: Ubuntu 4.4.0-148.174-generic 4.4.177 Uname: Linux 4.4.0-148-generic x86_64 ApportVersion: 2.20.9-0ubuntu7.6 Architecture: amd64 Date: Mon May 27 16:09:30 2019 InstallationDate: Installed on 2018-04-04 (418 days ago) InstallationMedia: Ubuntu-Server 16.04.3 LTS "Xenial Xerus" - Release amd64 (20170801) SourcePackage: etcd UpgradeStatus: Upgraded to bionic on 2019-05-27 (0 days ago) modified.conffile..etc.default.etcd: [modified] mtime.conffile..etc.default.etcd: 2019-05-27T15:20:47.342333 ** Affects: etcd (Ubuntu) Importance: Undecided Status: New ** Tags: amd64 apport-bug bionic -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1830624 Title: etcd server fails to start after upgrade from 2.2 to 3.2 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/etcd/+bug/1830624/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
