Re: [lxc-users] "error: LXD still not running after 5 minutes" - failed lxd.migrate - how to recover?
On Wed, Aug 08, 2018 at 11:26:10PM +0200, Tomasz Chmielewski wrote: > On 2018-08-08 22:26, Stéphane Graber wrote: > > > > Not sure how to recover now? The containers seem intact in > > > /var/lib/lxd/ > > > > What do you get if you do "journalctl -u snap.lxd.daemon -n 300" and > > -- Logs begin at Thu 2018-07-12 06:07:13 UTC, end at Wed 2018-08-08 21:07:13 > UTC. -- > Aug 08 18:21:12 b1 systemd[1]: Started Service for snap application > lxd.daemon. > Aug 08 18:21:12 b1 lxd.daemon[12581]: => Preparing the system > Aug 08 18:21:12 b1 lxd.daemon[12581]: ==> Creating missing snap > configuration > Aug 08 18:21:13 b1 lxd.daemon[12581]: ==> Loading snap configuration > Aug 08 18:21:13 b1 lxd.daemon[12581]: ==> Setting up mntns symlink > (mnt:[4026532794]) > Aug 08 18:21:13 b1 lxd.daemon[12581]: ==> Setting up kmod wrapper > Aug 08 18:21:13 b1 lxd.daemon[12581]: ==> Preparing /boot > Aug 08 18:21:13 b1 lxd.daemon[12581]: ==> Preparing a clean copy of /run > Aug 08 18:21:13 b1 lxd.daemon[12581]: ==> Preparing a clean copy of /etc > Aug 08 18:21:13 b1 lxd.daemon[12581]: ==> Setting up ceph configuration > Aug 08 18:21:13 b1 lxd.daemon[12581]: ==> Setting up LVM configuration > Aug 08 18:21:13 b1 lxd.daemon[12581]: ==> Rotating logs > Aug 08 18:21:13 b1 lxd.daemon[12581]: ==> Setting up ZFS (0.7) > Aug 08 18:21:13 b1 lxd.daemon[12581]: ==> Escaping the systemd cgroups > Aug 08 18:21:13 b1 lxd.daemon[12581]: ==> Escaping the systemd process > resource limits > Aug 08 18:21:41 b1 systemd[1]: Stopping Service for snap application > lxd.daemon... > Aug 08 18:21:42 b1 lxd.daemon[13595]: => Stop reason is: host shutdown > Aug 08 18:21:42 b1 lxd.daemon[13595]: => Stopping LXD (with container > shutdown) > Aug 08 18:21:42 b1 lxd.daemon[13595]: => Stopping LXCFS > Aug 08 18:21:43 b1 systemd[1]: Stopped Service for snap application > lxd.daemon. > Aug 08 18:21:44 b1 systemd[1]: Started Service for snap application > lxd.daemon. > Aug 08 18:21:44 b1 lxd.daemon[13676]: => Preparing the system > Aug 08 18:21:44 b1 lxd.daemon[13676]: ==> Loading snap configuration > Aug 08 18:21:44 b1 lxd.daemon[13676]: ==> Setting up mntns symlink > (mnt:[4026532794]) > Aug 08 18:21:44 b1 lxd.daemon[13676]: ==> Setting up kmod wrapper > Aug 08 18:21:44 b1 lxd.daemon[13676]: ==> Preparing /boot > Aug 08 18:21:44 b1 lxd.daemon[13676]: ==> Preparing a clean copy of /run > Aug 08 18:21:44 b1 lxd.daemon[13676]: ==> Preparing a clean copy of /etc > Aug 08 18:21:44 b1 lxd.daemon[13676]: ==> Setting up ceph configuration > Aug 08 18:21:44 b1 lxd.daemon[13676]: ==> Setting up LVM configuration > Aug 08 18:21:44 b1 lxd.daemon[13676]: ==> Rotating logs > Aug 08 18:21:44 b1 lxd.daemon[13676]: ==> Setting up ZFS (0.7) > Aug 08 18:21:44 b1 lxd.daemon[13676]: ==> Escaping the systemd cgroups > Aug 08 18:21:44 b1 lxd.daemon[13676]: ==> Escaping the systemd process > resource limits > Aug 08 18:21:44 b1 lxd.daemon[13676]: => Starting LXCFS > Aug 08 18:21:44 b1 lxd.daemon[13676]: => Starting LXD > Aug 08 18:21:44 b1 lxd.daemon[13676]: lvl=warn msg="AppArmor support has > been disabled because of lack of kernel support" t=2018-08-08T18:21:44+ > Aug 08 18:21:44 b1 lxd.daemon[13676]: lvl=warn msg="CGroup memory swap > accounting is disabled, swap limits will be ignored." > t=2018-08-08T18:21:44+ > Aug 08 18:21:44 b1 lxd.daemon[13676]: mount namespace: 5 > Aug 08 18:21:44 b1 lxd.daemon[13676]: hierarchies: > Aug 08 18:21:44 b1 lxd.daemon[13676]: 0: fd: 6: hugetlb > Aug 08 18:21:44 b1 lxd.daemon[13676]: 1: fd: 7: pids > Aug 08 18:21:44 b1 lxd.daemon[13676]: 2: fd: 8: cpuset > Aug 08 18:21:44 b1 lxd.daemon[13676]: 3: fd: 9: perf_event > Aug 08 18:21:44 b1 lxd.daemon[13676]: 4: fd: 10: freezer > Aug 08 18:21:44 b1 lxd.daemon[13676]: 5: fd: 11: memory > Aug 08 18:21:44 b1 lxd.daemon[13676]: 6: fd: 12: devices > Aug 08 18:21:44 b1 lxd.daemon[13676]: 7: fd: 13: blkio > Aug 08 18:21:44 b1 lxd.daemon[13676]: 8: fd: 14: cpu,cpuacct > Aug 08 18:21:44 b1 lxd.daemon[13676]: 9: fd: 15: net_cls,net_prio > Aug 08 18:21:44 b1 lxd.daemon[13676]: 10: fd: 16: rdma > Aug 08 18:21:44 b1 lxd.daemon[13676]: 11: fd: 17: name=systemd > Aug 08 18:21:44 b1 lxd.daemon[13676]: 12: fd: 18: unified > Aug 08 18:28:07 b1 systemd[1]: Stopping Service for snap application > lxd.daemon... > Aug 08 18:28:07 b1 lxd.daemon[18773]: => Stop reason is: host shutdown > Aug 08 18:28:07 b1 lxd.daemon[18773]: => Stopping LXD (with container > shutdown) > Aug 08 18:37:24 b1 lxd.daemon[18773]: => Stopping LXCFS > Aug 08 18:37:25 b1 systemd[1]: Stopped Service for snap application > lxd.daemon. > -- Reboot -- > Aug 08 18:38:55 b1 systemd[1]: Started Service for snap application > lxd.daemon. > Aug 08 18:38:57 b1 lxd.daemon[2345]: => Preparing the system > Aug 08 18:38:57 b1 lxd.daemon[2345]: ==> Loading snap configuration > Aug 08 18:38:57 b1 lxd.daemon[2345]: ==> Setting up mntns symlink > (mnt:[4026532463]) > Aug 08 18:38:57 b1 lxd.daemon[2345]: ==> Setting up
Re: [lxc-users] "error: LXD still not running after 5 minutes" - failed lxd.migrate - how to recover?
On Wed, Aug 08, 2018 at 09:06:40PM +0200, Tomasz Chmielewski wrote: > I've tried to migrate from deb to snap on Ubuntu 18.04. > > Unfortunately, lxd.migrate failed with "error: LXD still not running after 5 > minutes": > > root@b1 ~ # lxd.migrate > => Connecting to source server > => Connecting to destination server > => Running sanity checks > > === Source server > LXD version: 3.0.1 > LXD PID: 2656 > Resources: > Containers: 6 > Images: 4 > Networks: 1 > Storage pools: 1 > > === Destination server > LXD version: 3.3 > LXD PID: 12791 > Resources: > Containers: 0 > Images: 0 > Networks: 0 > Storage pools: 0 > > The migration process will shut down all your containers then move your data > to the destination LXD. > Once the data is moved, the destination LXD will start and apply any needed > updates. > And finally your containers will be brought back to their previous state, > completing the migration. > > WARNING: /var/lib/lxd is a mountpoint. You will need to update that mount > location after the migration. > > Are you ready to proceed (yes/no) [default=no]? yes > => Shutting down the source LXD > => Stopping the source LXD units > => Stopping the destination LXD unit > => Unmounting source LXD paths > => Unmounting destination LXD paths > => Wiping destination LXD clean > => Backing up the database > => Moving the /var/lib/lxd mountpoint > => Updating the storage backends > => Starting the destination LXD > => Waiting for LXD to come online > > error: LXD still not running after 5 minutes. > > > > root@b1 ~ # lxd.migrate > => Connecting to source server > error: Unable to connect to the source LXD: Get http://unix.socket/1.0: dial > unix /var/lib/lxd/unix.socket: connect: no such file or directory > > > > root@b1 ~ # lxc list > Error: Get http://unix.socket/1.0: dial unix /var/lib/lxd/unix.socket: > connect: no such file or directory > > > > Not sure how to recover now? The containers seem intact in /var/lib/lxd/ What do you get if you do "journalctl -u snap.lxd.daemon -n 300" and anything useful looking in /var/snap/lxd/common/lxd/logs/lxd.log? It's expected that "systemctl start lxd" won't work anymore since the data was moved over to the snap which then likely caused your database to be upgraded, making it unreadable for your older deb version of LXD. I'd recommend you do: - systemctl stop lxd lxd.socket - systemctl mask lxd lxd.socket To prevent any accidental startup of your old LXD until the snap migration is done and it can be safely removed. -- Stéphane Graber Ubuntu developer http://www.ubuntu.com signature.asc Description: PGP signature ___ lxc-users mailing list lxc-users@lists.linuxcontainers.org http://lists.linuxcontainers.org/listinfo/lxc-users
Re: [lxc-users] "error: LXD still not running after 5 minutes" - failed lxd.migrate - how to recover?
On 2018-08-08 21:06, Tomasz Chmielewski wrote: I've tried to migrate from deb to snap on Ubuntu 18.04. Unfortunately, lxd.migrate failed with "error: LXD still not running after 5 minutes": (...) Not sure how to recover now? The containers seem intact in /var/lib/lxd/ It seems it's partially migrated with no clear info on how to continue. Attempting to do "systemctl start lxd" produces: Error: failed to open cluster database: failed to ensure schema: schema version '9' is more recent than expected '7' Attempting to start lxd from the snap results in: # /snap/bin/lxc list Error: Both native and snap packages are installed on this system Run "lxd.migrate" to complete your migration to the snap package # systemctl status lxd ● lxd.service - LXD - main daemon Loaded: loaded (/lib/systemd/system/lxd.service; indirect; vendor preset: enabled) Active: activating (start-post) (Result: exit-code) since Wed 2018-08-08 19:28:57 UTC; 9s ago Docs: man:lxd(1) Process: 6829 ExecStart=/usr/bin/lxd --group lxd --logfile=/var/log/lxd/lxd.log (code=exited, status=1/FAILURE) Process: 6824 ExecStartPre=/usr/lib/x86_64-linux-gnu/lxc/lxc-apparmor-load (code=exited, status=0/SUCCESS) Main PID: 6829 (code=exited, status=1/FAILURE); Control PID: 6831 (lxd) Tasks: 8 CGroup: /system.slice/lxd.service └─6831 /usr/lib/lxd/lxd waitready --timeout=600 Aug 08 19:28:57 b1 systemd[1]: Starting LXD - main daemon... Aug 08 19:28:57 b1 lxd[6829]: lvl=warn msg="AppArmor support has been disabled because of lack of kernel support" t=2018-08-08T19:28:57+ Aug 08 19:28:57 b1 lxd[6829]: lvl=warn msg="CGroup memory swap accounting is disabled, swap limits will be ignored." t=2018-08-08T19:28:57+ Aug 08 19:28:58 b1 lxd[6829]: lvl=eror msg="Failed to start the daemon: failed to open cluster database: failed to ensure schema: schema version '9' is more recent than expected '7'" t=2018-08-08T19:28:58+ Aug 08 19:28:58 b1 lxd[6829]: Error: failed to open cluster database: failed to ensure schema: schema version '9' is more recent than expected '7' Aug 08 19:28:58 b1 systemd[1]: lxd.service: Main process exited, code=exited, status=1/FAILURE # lxd.migrate => Connecting to source server error: Unable to connect to the source LXD: Get http://unix.socket/1.0: EOF # dpkg -l|grep lxd ii lxd 3.0.1-0ubuntu1~18.04.1 amd64Container hypervisor based on LXC - daemon ii lxd-client3.0.1-0ubuntu1~18.04.1 amd64Container hypervisor based on LXC - client # snap list Name VersionRev Tracking Publisher Notes core 16-2.34.3 5145 stablecanonical core lxd 3.38011 stablecanonical - -- Tomasz Chmielewski https://lxadm.com ___ lxc-users mailing list lxc-users@lists.linuxcontainers.org http://lists.linuxcontainers.org/listinfo/lxc-users
[lxc-users] "error: LXD still not running after 5 minutes" - failed lxd.migrate - how to recover?
I've tried to migrate from deb to snap on Ubuntu 18.04. Unfortunately, lxd.migrate failed with "error: LXD still not running after 5 minutes": root@b1 ~ # lxd.migrate => Connecting to source server => Connecting to destination server => Running sanity checks === Source server LXD version: 3.0.1 LXD PID: 2656 Resources: Containers: 6 Images: 4 Networks: 1 Storage pools: 1 === Destination server LXD version: 3.3 LXD PID: 12791 Resources: Containers: 0 Images: 0 Networks: 0 Storage pools: 0 The migration process will shut down all your containers then move your data to the destination LXD. Once the data is moved, the destination LXD will start and apply any needed updates. And finally your containers will be brought back to their previous state, completing the migration. WARNING: /var/lib/lxd is a mountpoint. You will need to update that mount location after the migration. Are you ready to proceed (yes/no) [default=no]? yes => Shutting down the source LXD => Stopping the source LXD units => Stopping the destination LXD unit => Unmounting source LXD paths => Unmounting destination LXD paths => Wiping destination LXD clean => Backing up the database => Moving the /var/lib/lxd mountpoint => Updating the storage backends => Starting the destination LXD => Waiting for LXD to come online error: LXD still not running after 5 minutes. root@b1 ~ # lxd.migrate => Connecting to source server error: Unable to connect to the source LXD: Get http://unix.socket/1.0: dial unix /var/lib/lxd/unix.socket: connect: no such file or directory root@b1 ~ # lxc list Error: Get http://unix.socket/1.0: dial unix /var/lib/lxd/unix.socket: connect: no such file or directory Not sure how to recover now? The containers seem intact in /var/lib/lxd/ Tomasz Chmielewski https://lxadm.com ___ lxc-users mailing list lxc-users@lists.linuxcontainers.org http://lists.linuxcontainers.org/listinfo/lxc-users
