[I should add: these systems are using plain dir backend on ext4: no
zfs, btrfs etc]
> Hmm. This means that either the source of the mount or the mount target
> didn't exist. I would expect LXD to create /var/lib/lxd/shmounts on the
> target host when it starts a migration (you can verify that this happens
> by `inotifywait /var/lib/lxd/shmounts` which should tell you that the
> directory gets created).
[On target system]
root@nuc2:~# inotifywait /var/lib/lxd/shmounts
Setting up watches.
Watches established.
[On source system]
root@nuc1:~# lxc move sample nuc2:sample2
It now just hangs here. I've waited 10+ minutes.
ps auxwww on source shows:
...
root 3830 0.0 0.2 141828 10540 pts/0 Sl+ 06:41 0:00 lxc move
sample nuc2:sample2
root 3842 0.1 1.3 65464 52056 ? S 06:41 0:00 rsync -arvP
--devices --numeric-ids --partial /var/lib/lxd/containers/sample/
localhost:/tmp/foo -e sh -c "nc -U /tmp/lxd_rsync_952127373"
root 3843 0.0 0.0 4508 752 ? S 06:41 0:00 sh -c nc -U
/tmp/lxd_rsync_952127373 localhost rsync --server -vlogDtpre.iLsfx --partial
--numeric-ids . /tmp/foo
root 3844 0.0 0.0 9052 956 ? S 06:41 0:00 nc -U
/tmp/lxd_rsync_952127373
root 4486 0.0 0.0 0 0 ? S 06:44 0:00 [kworker/u4:0]
root 5875 3.5 0.1 22624 5192 pts/2 Ss 06:53 0:00 -bash
root 5888 0.0 0.0 37372 3280 pts/2 R+ 06:53 0:00 ps auxwww
root@nuc1:~# strace -p 3844
strace: Process 3844 attached
restart_syscall(<... resuming interrupted poll ...>
ps auxwww on target shows:
root 3391 0.0 0.0 6532 704 pts/0 S+ 06:41 0:00 inotifywait
/var/lib/lxd/shmounts
root 3402 0.0 0.0 14052 912 ? S 06:41 0:00 rsync --server
-vlogDtpre.iLsfx --numeric-ids --devices --partial . /tmp/lxd_restore_287818406/
root 3403 0.1 0.1 28908 5192 ? S 06:41 0:00 rsync --server
-vlogDtpre.iLsfx --numeric-ids --devices --partial .
/var/lib/lxd/containers/sample2/
root 3407 0.0 0.0 44328 3800 ? S 06:41 0:00 rsync --server
-vlogDtpre.iLsfx --numeric-ids --devices --partial .
/var/lib/lxd/containers/sample2/
root 3591 0.0 0.0 0 0 ? S 06:48 0:00 [kworker/u4:1]
root 3748 0.0 0.0 0 0 ? S 06:54 0:00 [kworker/1:0]
root 3757 7.0 0.1 22608 5140 pts/1 Ss 06:54 0:00 -bash
root 3770 0.0 0.0 37372 3384 pts/1 R+ 06:54 0:00 ps auxwww
root@nuc2:~# strace -p 3407
strace: Process 3407 attached
select(1, [0], [], [0], {55, 2239}
So it appears that the filesystem transfer has simply frozen.
(Note: these systems applied all updates yesterday; this gave a minor
kernel update)
> So, the only other option is that the directory doesn't exist on the
> target filesystem. I'm not really sure how that could be given that it
> was mounted originally, unless the target filesystem didn't get
> transferred correctly somehow. Is there any chance you can poke at the
> filesystem on the target once it gets migrated to see?
I wasn't sure how I was going to do this, but fortunately because of the
hang I can.
root@nuc2:~# ls /var/lib/lxd/containers/sample2/rootfs/dev/
agpgart core kmem loop6 midi02 mixer2 pts ram13 ram5 rmidi1
smpte2 tty0 tty7
audio dsp loop0 loop7 midi03 mixer3 ram ram14 ram6 rmidi2
smpte3 tty1 tty8
audio1 dsp1 loop1 mapper midi1 mpu401data ram0 ram15 ram7 rmidi3
sndstat tty2 tty9
audio2 dsp2 loop2 mem midi2 mpu401stat ram1 ram16 ram8
sequencer stderr tty3 urandom
audio3 dsp3 loop3 midi0 midi3 null ram10 ram2 ram9 shm
stdin tty4 zero
audioctl fd loop4 midi00 mixer port ram11 ram3 random smpte0
stdout tty5
console full loop5 midi01 mixer1 ptmx ram12 ram4 rmidi0 smpte1
tty tty6
root@nuc2:~# ls /var/lib/lxd/containers/sample2/rootfs/dev/.lxd*
ls: cannot access '/var/lib/lxd/containers/sample2/rootfs/dev/.lxd*': No such
file or directory
However the same is true on the source (.../dev exists, .../dev/.lxd*
does not). Maybe that's the problem? But then why is the migration
trying to mount something on /dev/.lxd-mounts ?
Otherwise, I'm not sure what I'm looking for.
I can compare the two filesystems:
$ ssh root@nuc1 'cd /var/lib/lxd/containers/sample; find . -type f | sort' > f1
$ ssh root@nuc2 'cd /var/lib/lxd/containers/sample2; find . -type f | sort' > f2
$ diff -u f1 f2
--- f1 2016-05-17 06:49:39.000000000 +0100
+++ f2 2016-05-17 06:49:49.000000000 +0100
@@ -653,6 +653,7 @@
./rootfs/etc/network/if-up.d/upstart
./rootfs/etc/network/interfaces
./rootfs/etc/network/interfaces.d/50-cloud-init.cfg
+./rootfs/etc/network/interfaces.d/eth0.cfg
./rootfs/etc/networks
./rootfs/etc/newt/palette.original
./rootfs/etc/newt/palette.ubuntu
@@ -21780,6 +21781,7 @@
./rootfs/var/backups/shadow.bak
./rootfs/var/cache/apt/archives/lock
./rootfs/var/cache/apt/pkgcache.bin
+./rootfs/var/cache/apt/.pkgcache.bin.AUblXr
./rootfs/var/cache/apt/srcpkgcache.bin
./rootfs/var/cache/debconf/config.dat
./rootfs/var/cache/debconf/config.dat-old
@@ -21834,120 +21836,38 @@
./rootfs/var/lib/apparmor/profiles/.apparmor.md5sums
./rootfs/var/lib/apt/extended_states
./rootfs/var/lib/apt/keyrings/ubuntu-archive-keyring.gpg
-./rootfs/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial-backports_InRelease
-./rootfs/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial-backports_main_binary-amd64_Packages
-./rootfs/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial-backports_main_i18n_Translation-en
-./rootfs/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial-backports_main_source_Sources
-./rootfs/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial-backports_multiverse_binary-amd64_Packages
-./rootfs/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial-backports_multiverse_i18n_Translation-en
-./rootfs/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial-backports_multiverse_source_Sources
-./rootfs/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial-backports_restricted_binary-amd64_Packages
-./rootfs/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial-backports_restricted_i18n_Translation-en
-./rootfs/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial-backports_restricted_source_Sources
-./rootfs/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial-backports_universe_binary-amd64_Packages
-./rootfs/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial-backports_universe_i18n_Translation-en
-./rootfs/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial-backports_universe_source_Sources
./rootfs/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial_InRelease
./rootfs/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial_main_binary-amd64_Packages
./rootfs/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial_main_i18n_Translation-en
-./rootfs/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial_main_source_Sources
./rootfs/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial_multiverse_binary-amd64_Packages
./rootfs/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial_multiverse_i18n_Translation-en
-./rootfs/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial_multiverse_source_Sources
./rootfs/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial_restricted_binary-amd64_Packages
./rootfs/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial_restricted_i18n_Translation-en
-./rootfs/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial_restricted_source_Sources
./rootfs/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial_universe_binary-amd64_Packages
./rootfs/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial_universe_i18n_Translation-en
-./rootfs/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial_universe_source_Sources
./rootfs/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial-updates_InRelease
./rootfs/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial-updates_main_binary-amd64_Packages
./rootfs/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial-updates_main_i18n_Translation-en
-./rootfs/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial-updates_main_source_Sources
./rootfs/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial-updates_multiverse_binary-amd64_Packages
./rootfs/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial-updates_multiverse_i18n_Translation-en
-./rootfs/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial-updates_multiverse_source_Sources
./rootfs/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial-updates_restricted_binary-amd64_Packages
./rootfs/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial-updates_restricted_i18n_Translation-en
-./rootfs/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial-updates_restricted_source_Sources
./rootfs/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial-updates_universe_binary-amd64_Packages
./rootfs/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial-updates_universe_i18n_Translation-en
-./rootfs/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial-updates_universe_source_Sources
./rootfs/var/lib/apt/lists/lock
./rootfs/var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_xenial-security_InRelease
./rootfs/var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_xenial-security_main_binary-amd64_Packages
./rootfs/var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_xenial-security_main_i18n_Translation-en
-./rootfs/var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_xenial-security_main_source_Sources
./rootfs/var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_xenial-security_multiverse_binary-amd64_Packages
./rootfs/var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_xenial-security_multiverse_i18n_Translation-en
-./rootfs/var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_xenial-security_multiverse_source_Sources
./rootfs/var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_xenial-security_restricted_binary-amd64_Packages
./rootfs/var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_xenial-security_restricted_i18n_Translation-en
-./rootfs/var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_xenial-security_restricted_source_Sources
./rootfs/var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_xenial-security_universe_binary-amd64_Packages
./rootfs/var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_xenial-security_universe_i18n_Translation-en
-./rootfs/var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_xenial-security_universe_source_Sources
-./rootfs/var/lib/apt/periodic/unattended-upgrades-stamp
-./rootfs/var/lib/apt/periodic/update-stamp
./rootfs/var/lib/apt/periodic/update-success-stamp
-./rootfs/var/lib/apt/periodic/upgrade-stamp
-./rootfs/var/lib/cloud/data/instance-id
-./rootfs/var/lib/cloud/data/previous-datasource
-./rootfs/var/lib/cloud/data/previous-hostname
-./rootfs/var/lib/cloud/data/previous-instance-id
-./rootfs/var/lib/cloud/data/result.json
-./rootfs/var/lib/cloud/data/status.json
-./rootfs/var/lib/cloud/instances/sample/boot-finished
-./rootfs/var/lib/cloud/instances/sample/cloud-config.txt
-./rootfs/var/lib/cloud/instances/sample/datasource
-./rootfs/var/lib/cloud/instances/sample/obj.pkl
-./rootfs/var/lib/cloud/instances/sample/sem/config_apt_configure
-./rootfs/var/lib/cloud/instances/sample/sem/config_apt_pipelining
-./rootfs/var/lib/cloud/instances/sample/sem/config_byobu
-./rootfs/var/lib/cloud/instances/sample/sem/config_ca_certs
-./rootfs/var/lib/cloud/instances/sample/sem/config_chef
-./rootfs/var/lib/cloud/instances/sample/sem/config_disk_setup
-./rootfs/var/lib/cloud/instances/sample/sem/config_fan
-./rootfs/var/lib/cloud/instances/sample/sem/config_grub_dpkg
-./rootfs/var/lib/cloud/instances/sample/sem/config_keys_to_console
-./rootfs/var/lib/cloud/instances/sample/sem/config_landscape
-./rootfs/var/lib/cloud/instances/sample/sem/config_locale
-./rootfs/var/lib/cloud/instances/sample/sem/config_lxd
-./rootfs/var/lib/cloud/instances/sample/sem/config_mcollective
-./rootfs/var/lib/cloud/instances/sample/sem/config_mounts
-./rootfs/var/lib/cloud/instances/sample/sem/config_package_update_upgrade_install
-./rootfs/var/lib/cloud/instances/sample/sem/config_phone_home
-./rootfs/var/lib/cloud/instances/sample/sem/config_power_state_change
-./rootfs/var/lib/cloud/instances/sample/sem/config_puppet
-./rootfs/var/lib/cloud/instances/sample/sem/config_rightscale_userdata
-./rootfs/var/lib/cloud/instances/sample/sem/config_rsyslog
-./rootfs/var/lib/cloud/instances/sample/sem/config_runcmd
-./rootfs/var/lib/cloud/instances/sample/sem/config_salt_minion
-./rootfs/var/lib/cloud/instances/sample/sem/config_scripts_per_instance
-./rootfs/var/lib/cloud/instances/sample/sem/config_scripts_user
-./rootfs/var/lib/cloud/instances/sample/sem/config_scripts_vendor
-./rootfs/var/lib/cloud/instances/sample/sem/config_seed_random
-./rootfs/var/lib/cloud/instances/sample/sem/config_set_hostname
-./rootfs/var/lib/cloud/instances/sample/sem/config_set_passwords
-./rootfs/var/lib/cloud/instances/sample/sem/config_snappy
-./rootfs/var/lib/cloud/instances/sample/sem/config_ssh
-./rootfs/var/lib/cloud/instances/sample/sem/config_ssh_authkey_fingerprints
-./rootfs/var/lib/cloud/instances/sample/sem/config_ssh_import_id
-./rootfs/var/lib/cloud/instances/sample/sem/config_timezone
-./rootfs/var/lib/cloud/instances/sample/sem/config_ubuntu_init_switch
-./rootfs/var/lib/cloud/instances/sample/sem/config_users_groups
-./rootfs/var/lib/cloud/instances/sample/sem/config_write_files
-./rootfs/var/lib/cloud/instances/sample/sem/consume_data
-./rootfs/var/lib/cloud/instances/sample/user-data.txt
-./rootfs/var/lib/cloud/instances/sample/user-data.txt.i
-./rootfs/var/lib/cloud/instances/sample/vendor-cloud-config.txt
-./rootfs/var/lib/cloud/instances/sample/vendor-data.txt
-./rootfs/var/lib/cloud/instances/sample/vendor-data.txt.i
./rootfs/var/lib/cloud/seed/nocloud-net/meta-data
./rootfs/var/lib/cloud/seed/nocloud-net/user-data
./rootfs/var/lib/cloud/seed/nocloud-net/vendor-data
-./rootfs/var/lib/cloud/sem/config_scripts_per_once.once
-./rootfs/var/lib/dhcp/dhclient.eth0.leases
./rootfs/var/lib/dpkg/alternatives/awk
./rootfs/var/lib/dpkg/alternatives/builtins.7.gz
./rootfs/var/lib/dpkg/alternatives/editor
@@ -23991,8 +23911,6 @@
./rootfs/var/lib/dpkg/triggers/update-ca-certificates
./rootfs/var/lib/dpkg/triggers/update-ca-certificates-fresh
./rootfs/var/lib/dpkg/triggers/update-initramfs
-./rootfs/var/lib/logrotate/status
-./rootfs/var/lib/lxd/lxd.db
./rootfs/var/lib/man-db/auto-update
./rootfs/var/lib/mdadm/CONF-UNCHECKED
./rootfs/var/lib/mdadm/mdadm.conf-generated
@@ -24083,9 +24001,6 @@
./rootfs/var/lib/systemd/deb-systemd-helper-enabled/ureadahead.service.dsh-also
./rootfs/var/lib/systemd/deb-systemd-helper-enabled/uuidd.service.dsh-also
./rootfs/var/lib/systemd/deb-systemd-helper-enabled/uuidd.socket.dsh-also
-./rootfs/var/lib/systemd/random-seed
-./rootfs/var/lib/systemd/timers/stamp-apt-daily.timer
-./rootfs/var/lib/ubuntu-release-upgrader/release-upgrade-available
./rootfs/var/lib/ucf/cache/:etc:apt:apt.conf.d:20auto-upgrades
./rootfs/var/lib/ucf/cache/:etc:apt:apt.conf.d:50unattended-upgrades
./rootfs/var/lib/ucf/cache/:etc:rsyslog.d:50-default.conf
@@ -24111,29 +24026,18 @@
./rootfs/var/lib/ucf/registry.4
./rootfs/var/lib/ucf/registry.5
./rootfs/var/lib/ucf/registry.6
-./rootfs/var/lib/update-manager/meta-release-lts
./rootfs/var/lib/update-notifier/dpkg-run-stamp
-./rootfs/var/lib/update-notifier/fsck-at-reboot
./rootfs/var/lib/update-notifier/updates-available
./rootfs/var/lib/usbutils/usb.ids
./rootfs/var/lib/xml-core/catalog
./rootfs/var/lib/xml-core/xml-core
./rootfs/var/log/apt/history.log
./rootfs/var/log/apt/term.log
-./rootfs/var/log/auth.log
./rootfs/var/log/btmp
-./rootfs/var/log/cloud-init.log
-./rootfs/var/log/cloud-init-output.log
./rootfs/var/log/dpkg.log
./rootfs/var/log/fsck/checkfs
./rootfs/var/log/fsck/checkroot
-./rootfs/var/log/kern.log
./rootfs/var/log/lastlog
-./rootfs/var/log/syslog
-./rootfs/var/log/syslog.1
-./rootfs/var/log/unattended-upgrades/unattended-upgrades-dpkg.log
-./rootfs/var/log/unattended-upgrades/unattended-upgrades.log
-./rootfs/var/log/unattended-upgrades/unattended-upgrades-shutdown.log
./rootfs/var/log/wtmp
./rootfs/var/spool/cron/atjobs/.SEQ
./templates/cloud-init-meta.tpl
But I don't know what to make of that, other than "some files didn't get
copied".
It's possible the source was in the middle of an unattended upgrade when
the migration started, but it isn't now:
root@sample:~# ps auxwww
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.1 37364 5392 ? Ss 05:39 0:00 /sbin/init
root 44 0.0 0.1 35276 4048 ? Ss 05:39 0:00
/lib/systemd/systemd-journald
root 51 0.0 0.0 41764 3272 ? Ss 05:39 0:00
/lib/systemd/systemd-udevd
syslog 91 0.0 0.0 186900 3360 ? Ssl 05:39 0:00
/usr/sbin/rsyslogd -n
root 92 0.0 0.0 28548 3064 ? Ss 05:39 0:00
/lib/systemd/systemd-logind
daemon 97 0.0 0.0 26044 2196 ? Ss 05:39 0:00 /usr/sbin/atd
-f
message+ 98 0.0 0.0 42908 3880 ? Ss 05:39 0:00
/usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile
--systemd-activation
root 108 0.0 0.1 274488 6264 ? Ssl 05:39 0:00
/usr/lib/accountsservice/accounts-daemon
root 116 0.0 0.0 27728 3100 ? Ss 05:39 0:00 /usr/sbin/cron
-f
root 122 0.0 0.1 277180 6052 ? Ssl 05:39 0:00
/usr/lib/policykit-1/polkitd --no-debug
root 255 0.0 0.0 16120 860 ? Ss 05:39 0:00 /sbin/dhclient
-1 -v -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases -I -df
/var/lib/dhcp/dhclient6.eth0.leases eth0
root 321 0.0 0.1 65612 6172 ? Ss 05:39 0:00 /usr/sbin/sshd
-D
root 344 0.0 0.0 14476 2144 console Ss+ 05:39 0:00 /sbin/agetty
--noclear --keep-baud console 115200 38400 9600 linux
root 373 0.0 0.1 95460 7156 ? Ss 05:40 0:00 sshd:
root@pts/0
root 375 0.0 0.1 36536 4204 ? Ss 05:40 0:00
/lib/systemd/systemd --user
root 376 0.0 0.0 60816 1532 ? S 05:40 0:00 (sd-pam)
root 413 0.2 0.1 21144 5040 pts/0 Ss 05:40 0:01 -bash
root 2184 0.0 0.0 36084 3312 pts/0 R+ 05:51 0:00 ps auxwww
Maybe this was a one-off; I'll abort and retry
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1580765
Title:
Live migration error: Can't mount at ./dev/.lxd-mounts
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/criu/+bug/1580765/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
