Bug#913728: ganeti-2.15: Can not export/import VMs using gnt-backup
Hi Apollo!, El vie., 16 de nov. de 2018 a la(s) 21:18, Apollon Oikonomopoulos ( apoi...@debian.org) escribió: > Control: tags -1 moreinfo > > Hi again, > > On 09:06 Wed 14 Nov , Maximiliano Boscovich wrote: > > root@lisa:~# gnt-backup export -n lisa acme.sf-az2-fe > > Wed Nov 14 08:13:51 2018 Shutting down instance acme.sf-az2-fe > > Wed Nov 14 08:15:53 2018 Creating a snapshot of disk/0 on node lisa > > Wed Nov 14 08:15:53 2018 Starting instance acme.sf-az2-fe > > Wed Nov 14 08:15:54 2018 Exporting snapshot/0 from lisa to lisa > > Wed Nov 14 08:15:57 2018 snapshot/0 is now listening, starting export > > Wed Nov 14 08:16:05 2018 snapshot/0 sent 0M, 0.0 MiB/s > > Wed Nov 14 08:16:55 2018 - WARNING: import > > 'import-disk0-2018-11-14_08_15_54-a5_3EV' on lisa failed: Exited due to > > signal 15 > > Wed Nov 14 08:16:55 2018 snapshot/0 failed to receive data: Exited due to > > signal 15 (recent output: Child process didn't establish connection in > time > > (60s), sending SIGTERM\nsocat: W exiting on signal 15) > > Wed Nov 14 08:16:55 2018 - WARNING: Aborting export > > 'export-disk0-2018-11-14_08_15_59-6wxfk8' on > > b2e076b8-499a-45a0-8a65-1a4d2005708f > > Wed Nov 14 08:16:57 2018 - WARNING: export > > 'export-disk0-2018-11-14_08_15_59-6wxfk8' on lisa failed: Exited due to > > signal 15 > > Wed Nov 14 08:16:57 2018 snapshot/0 failed to send data: Exited due to > > signal 15 (recent output: DUMP: Date of this level 0 dump: Wed Nov 14 > > 08:15:59 2018\n DUMP: Dumping > > > /dev/mapper/vg--ganeti-76049ec2--8232--413d--8151--d263c3efa80e.disk0.snap-1 > > (an unlisted file system) to standard output\n DUMP: Label: none\n > DUMP: > > Writing 10 Kilobyte records\n DUMP: mapping (Pass I) [regular files]\n > > DUMP: mapping (Pass II) [directories]\n DUMP: estimated 247771 blocks.\n > > DUMP: Volume 1 started with block 1 at: Wed Nov 14 08:15:59 2018\n DUMP: > > dumping (Pass III) [directories]\n DUMP: DUMP: The ENTIRE dump is > > aborted.\nSignal on pipe: cannot recover\n DUMP: The ENTIRE dump is > > aborted.\nsocat: W exiting on signal 15) > > Wed Nov 14 08:16:57 2018 Removing snapshot of disk/0 on node lisa > > Wed Nov 14 08:16:57 2018 - WARNING: Some disk exports have failed; there > > may be leftover data for instance acme.sf-az2-fe on node lisa > > Failure: command execution error: > > Export failed, errors in export finalization, disk export: disk(s) 0 > > I can reproduce this when blocking the import/export connections between > nodes. Are you running a firewall on the machine? If so, does it filter > traffic on the loopback interface? Keep in mind that import/export uses > two completely random, high ports for the transfer. > I don't have any firewall on that server, and the export was doing in the same node root@lisa:~# iptables -S -P INPUT ACCEPT -P FORWARD ACCEPT -P OUTPUT ACCEPT This is my loopback interface: root@lisa:~# ip a|grep lo 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo inet 172.21.0.111/24 brd 172.21.0.255 scope global administracion And this is de lisa IP (cluster IP) root@lisa:~# ip a|grep admini 8: vlan300@bond0: mtu 1500 qdisc noqueue master administracion state UP group default qlen 1000 12: administracion: mtu 1500 qdisc noqueue state UP group default qlen 1000 inet 172.21.0.111/24 brd 172.21.0.255 scope global administracion And this is de bridge config root@lisa:~# brctl show bridge name bridge id STP enabled interfaces administracion 8000.6cae8b616faa no vlan300 sf-az1-be 8000.6cae8b616faa no vlan502 sf-az1-fe 8000.6cae8b616faa no vlan500 sf-az1-mw 8000.6cae8b616faa no vlan501 sf-az2-be 8000.6cae8b616faa no vlan505 sf-az2-fe 8000.6cae8b616faa no tap0 vlan503 sf-az2-mw 8000.6cae8b616faa no vlan504 I do a test with socat and its work great (i think so). I ran a server: root@lisa:~# socat -T 1 -d -d tcp-l:10081,reuseaddr,fork,crlf system:"echo -e \"\\\"HTTP/1.0 200 OK\\\nDocumentType: text/html\\\n\\\ndate: \$\(date\)server:\$SOCAT_SOCKADDR:\$SOCAT_SOCKPORTclient: \$SOCAT_PEERADDR:\$SOCAT_PEERPORT\\\n\\\"\"; cat; echo -e \"\\\"\\\n\\\"\""& [1] 20135 root@lisa:~# 2018/11/17 11:33:17 socat[20135] N listening on AF=2 0.0.0.0:10081 Connected to it with localhost IP: root@lisa:~# socat - TCP:localhost:10081 2018/11/17 11:33:24 socat[20135] N accepting connection from AF=2 127.0.0.1:49816 on AF=2 127.0.0.1:10081 2018/11/17 11:33:24 socat[20135] N forked off child process 20137 2018/11/17 11:33:24 socat[20135] N listening on AF=2 0.0.0.0:10081 2018/11/17 11:33:24 socat[20137] N forking off child, using socket for reading and writin
Bug#913728: ganeti-2.15: Can not export/import VMs using gnt-backup
noded pid=27003 INFO 172.21.0.111:58460 POST /impexp_status HTTP/1.1 200 2018-11-14 11:54:00,743: ganeti-noded pid=27005 INFO 172.21.0.111:58462 POST /impexp_status HTTP/1.1 200 2018-11-14 11:54:01,851: ganeti-noded pid=27012 INFO 172.21.0.111:58464 POST /impexp_status HTTP/1.1 200 2018-11-14 11:54:02,960: ganeti-noded pid=27014 INFO 172.21.0.111:58466 POST /impexp_status HTTP/1.1 200 2018-11-14 11:54:04,067: ganeti-noded pid=27016 INFO 172.21.0.111:58468 POST /impexp_status HTTP/1.1 200 2018-11-14 11:54:05,173: ganeti-noded pid=27018 INFO 172.21.0.111:58470 POST /impexp_status HTTP/1.1 200 2018-11-14 11:54:06,282: ganeti-noded pid=27020 INFO 172.21.0.111:58472 POST /impexp_status HTTP/1.1 200 2018-11-14 11:54:07,391: ganeti-noded pid=27022 INFO 172.21.0.111:58474 POST /impexp_status HTTP/1.1 200 2018-11-14 11:54:07,480: ganeti-noded pid=27023 INFO Abort import/export export-disk0-2018-11-14_11_53_10-lAhxhh 2018-11-14 11:54:07,481: ganeti-noded pid=27023 INFO Import/export export-disk0-2018-11-14_11_53_10-lAhxhh is running with PID 26862, sending SIGTERM 2018-11-14 11:54:07,481: ganeti-noded pid=27023 INFO 172.21.0.111:58476 POST /impexp_abort HTTP/1.1 200 2018-11-14 11:54:07,567: ganeti-noded pid=27029 INFO Finalizing import/export import-disk0-2018-11-14_11_53_06-A5knmP 2018-11-14 11:54:07,568: ganeti-noded pid=27029 INFO 172.21.0.111:58478 POST /impexp_cleanup HTTP/1.1 200 2018-11-14 11:54:08,676: ganeti-noded pid=27034 INFO 172.21.0.111:58480 POST /impexp_status HTTP/1.1 200 2018-11-14 11:54:08,765: ganeti-noded pid=27035 INFO RunCmd lvs --noheadings '--separator=|' --units=k --nosuffix -olv_attr,lv_kernel_major,lv_kernel_minor,vg_extent_size,stripes,devices /dev/vg-ganeti/76049ec2-8232-413d-8151-d263c3efa80e.disk0.snap 2018-11-14 11:54:08,786: ganeti-noded pid=27035 INFO RunCmd lvremove -f vg-ganeti/76049ec2-8232-413d-8151-d263c3efa80e.disk0.snap 2018-11-14 11:54:09,037: ganeti-noded pid=27035 INFO 172.21.0.111:58482 POST /blockdev_remove HTTP/1.1 200 2018-11-14 11:54:09,142: ganeti-noded pid=27088 INFO Finalizing import/export export-disk0-2018-11-14_11_53_10-lAhxhh 2018-11-14 11:54:09,143: ganeti-noded pid=27088 INFO 172.21.0.111:58484 POST /impexp_cleanup HTTP/1.1 200 2018-11-14 11:55:04,368: ganeti-noded pid=27147 INFO 172.21.0.111:58488 POST /all_instances_info HTTP/1.1 200 2018-11-14 11:55:04,440: ganeti-noded pid=27148 INFO 172.21.0.111:58490 POST /node_info HTTP/1.1 200 2018-11-14 11:55:05,128: ganeti-noded pid=27152 INFO RunCmd lvs --noheadings --units=m --nosuffix '--separator=|' -ovg_name,lv_name,lv_size,lv_attr 2018-11-14 11:55:05,146: ganeti-noded pid=27152 INFO 172.21.0.111:58492 POST /lv_list HTTP/1.1 200 Regards, Maximiliano. El mié., 14 de nov. de 2018 a la(s) 10:37, Apollon Oikonomopoulos ( apoi...@debian.org) escribió: > Hi, > > On 09:06 Wed 14 Nov , Maximiliano Boscovich wrote: > > I also tried to import a instances exported from another cluster, > > and I > > had the same result. > > I think its a problem related with SOCAT, i saw some other bugs > > reported, but anyone match exactly with this problem and output. > > Thanks for the report! > > Can you please share any relevant information from > /var/log/ganeti/node-daemon.log on lisa? > > Regards, > Apollon >
Bug#913728: ganeti-2.15: Can not export/import VMs using gnt-backup
Package: ganeti-2.15 Version: 2.15.2-7+deb9u3 Severity: normal Dear Maintainer, * What led up to the situation? Try to export a VM using gnt-backup * What exactly did you do (or not do) that was effective (or ineffective)? I created a new instances using debootstrap template's, and then try to export its instances. * What was the outcome of this action? root@lisa:~# gnt-instance add -t plain -n lisa -o debootstrap+default -B memory=1G,vcpus=1 --disk=0:size=5G --net 0:link=sf-az2-fe,mac=00:16:3e:f3:60:02 acme.sf-az2-fe Wed Nov 14 08:10:35 2018 * disk 0, size 5.0G Wed Nov 14 08:10:35 2018 * creating instance disks... Wed Nov 14 08:10:35 2018 adding instance acme.sf-az2-fe to cluster config Wed Nov 14 08:10:35 2018 adding disks to cluster config Wed Nov 14 08:10:35 2018 - INFO: Waiting for instance acme.sf-az2-fe to sync disks Wed Nov 14 08:10:35 2018 - INFO: Instance acme.sf-az2-fe's disks are in sync Wed Nov 14 08:10:35 2018 - INFO: Waiting for instance acme.sf-az2-fe to sync disks Wed Nov 14 08:10:35 2018 - INFO: Instance acme.sf-az2-fe's disks are in sync Wed Nov 14 08:10:35 2018 * running the instance OS create scripts... Wed Nov 14 08:13:30 2018 * starting instance... root@lisa:~# gnt-backup export -n lisa acme.sf-az2-fe Wed Nov 14 08:13:51 2018 Shutting down instance acme.sf-az2-fe Wed Nov 14 08:15:53 2018 Creating a snapshot of disk/0 on node lisa Wed Nov 14 08:15:53 2018 Starting instance acme.sf-az2-fe Wed Nov 14 08:15:54 2018 Exporting snapshot/0 from lisa to lisa Wed Nov 14 08:15:57 2018 snapshot/0 is now listening, starting export Wed Nov 14 08:16:05 2018 snapshot/0 sent 0M, 0.0 MiB/s Wed Nov 14 08:16:55 2018 - WARNING: import 'import-disk0-2018-11-14_08_15_54-a5_3EV' on lisa failed: Exited due to signal 15 Wed Nov 14 08:16:55 2018 snapshot/0 failed to receive data: Exited due to signal 15 (recent output: Child process didn't establish connection in time (60s), sending SIGTERM\nsocat: W exiting on signal 15) Wed Nov 14 08:16:55 2018 - WARNING: Aborting export 'export-disk0-2018-11-14_08_15_59-6wxfk8' on b2e076b8-499a-45a0-8a65-1a4d2005708f Wed Nov 14 08:16:57 2018 - WARNING: export 'export-disk0-2018-11-14_08_15_59-6wxfk8' on lisa failed: Exited due to signal 15 Wed Nov 14 08:16:57 2018 snapshot/0 failed to send data: Exited due to signal 15 (recent output: DUMP: Date of this level 0 dump: Wed Nov 14 08:15:59 2018\n DUMP: Dumping /dev/mapper/vg--ganeti-76049ec2--8232--413d--8151--d263c3efa80e.disk0.snap-1 (an unlisted file system) to standard output\n DUMP: Label: none\n DUMP: Writing 10 Kilobyte records\n DUMP: mapping (Pass I) [regular files]\n DUMP: mapping (Pass II) [directories]\n DUMP: estimated 247771 blocks.\n DUMP: Volume 1 started with block 1 at: Wed Nov 14 08:15:59 2018\n DUMP: dumping (Pass III) [directories]\n DUMP: DUMP: The ENTIRE dump is aborted.\nSignal on pipe: cannot recover\n DUMP: The ENTIRE dump is aborted.\nsocat: W exiting on signal 15) Wed Nov 14 08:16:57 2018 Removing snapshot of disk/0 on node lisa Wed Nov 14 08:16:57 2018 - WARNING: Some disk exports have failed; there may be leftover data for instance acme.sf-az2-fe on node lisa Failure: command execution error: Export failed, errors in export finalization, disk export: disk(s) 0 I also tried to import a instances exported from another cluster, and I had the same result. I think its a problem related with SOCAT, i saw some other bugs reported, but anyone match exactly with this problem and output. -- System Information: Debian Release: 9.6 APT prefers stable-updates APT policy: (500, 'stable-updates'), (500, 'proposed-updates'), (500, 'stable') Architecture: amd64 (x86_64) Kernel: Linux 4.9.0-8-amd64 (SMP w/24 CPU cores) Locale: LANG=es_AR.UTF-8, LC_CTYPE=es_AR.UTF-8 (charmap=UTF-8), LANGUAGE=es_AR:es (charmap=UTF-8) Shell: /bin/sh linked to /bin/dash Init: systemd (via /run/systemd/system) Versions of packages ganeti-2.15 depends on: ii adduser3.115 ii bridge-utils 1.5-13+deb9u1 ii debconf [debconf-2.0] 1.5.61 ii fping 3.15-1 ii iproute1:4.9.0-1+deb9u1 ii iproute2 4.9.0-1+deb9u1 ii iputils-arping 3:20161105-1 ii lvm2 2.02.168-2 ii openssh-client 1:7.4p1-10+deb9u4 ii openssh-server 1:7.4p1-10+deb9u4 ii openssl1.1.0f-3+deb9u2 ii python 2.7.13-2 ii python-bitarray0.8.1-1+b2 ii python-fdsend 0.2.1-2 ii python-ipaddr 2.1.11-2 ii python-openssl 16.2.0-1 ii python-paramiko2.0.0-1 ii python-psutil 5.0.1-1 ii python-pycurl 7.43.0-2 ii python-pyinotify 0.9.6-1 ii python-pyparsing 2.1.10+dfsg1-1 ii python-simplejson 3.10.0-1 ii socat 1.7.3.1-2+deb9u1 ganeti-2.15 recommends no packages. ganeti-2.15 suggests no packages. -- debconf information: ganeti-2.15/abort-removal: true
