Public bug reported:
I'm running ganeti 3.0.1 on Ubuntu 20.04, and running into issues with
moving instances between host nodes. Under the hood, this use socat to
copy the disk data from one node to another. In 20.04, the copy seems
to copy nearly all the data, but then at the very end it fails with:
Wed Jul 14 19:57:00 2021 - WARNING: import
'import-disk0-2021-07-14_19_52_25-8ockgvr6' on gnt1.example.com failed: Exited
with status 1
Wed Jul 14 19:57:00 2021 disk/0 failed to receive data: Exited with status
1 (recent output: dd: warning: partial read (65494 bytes); suggest
iflag=fullblock\nsocat: E SSL_read():Connection reset by peer\n0+980142 records
in\n0+980142 records out\n21976203264 bytes (22 GB, 20 GiB) copied, 273.039 s,
80.5 MB/s)
The copy seems to be almost 2MB short, expected size is 21,978,152,960
the copy above reports it copied 21,976,203,264
I've tried to isolate this down to a reproducable test case that does
not require ganeti, but I don't seem to have the socat chops to make it
happen. I'll list what I have so far down below.
I believe this is an issue with socat version 1.7.3.3-2 because:
- If I install socat_1.7.3.2-2ubuntu2_amd64.deb from 18.04 or
socat_1.7.4.1-3ubuntu1_amd64.deb from 21.04 (just download those packages and
"dpkg -i", no dependency issues are reported), my VM copy will succeed.
- On the socat website ( http://www.dest-unreach.org/socat/ ) it notes that
"Socat version 1.7.4.1 fixes [...] file transfer with OpenSSL.". That seems
like exactly what ganeti is doing, SSL bulk transfers.
The actual commands being run by ganeti are:
bash -o errexit -o pipefail -c { echo -E -n
M=b0f141f7085de052088d687fc70a027e53928be8 &&{ LC_ALL=C dd bs=1048576
<&0 2>&6 & pid=${!}; echo $pid >&8; wait $pid; } } | /usr/bin/socat -ls
-d -d -b1048576 -u stdin OPENSSL:10.1.1.1:38219,connect-
timeout=20,retry=10,intervall=1,keepalive,keepidle=60,keepintvl=10,keepcnt=5,verify=1,cipher=HIGH:-DES:-3DES:-EXPORT:-DH,compress=none,key=/var/lib/ganeti/server.pem,cert=/var/lib/ganeti/server.pem,cafile=/var/run/ganeti/import-
export/export-disk0-2021-07-14_19_52_30-ehngj3q8/ca,pf=ipv4,openssl-
commonname=ganeti.example.com 2>&4
and:
bash -o errexit -o pipefail -c /usr/bin/socat -ls -d -d -b1048576 -u
OPENSSL-
LISTEN:0,reuseaddr,forever,intervall=0.01,keepalive,keepidle=60,keepintvl=10,keepcnt=5,verify=1,cipher=HIGH:-DES:-3DES:-EXPORT:-DH,compress=none,key=/var/lib/ganeti/server.pem,cert=/var/lib/ganeti/server.pem,cafile=/var/run/ganeti/import-
export/import-disk0-2021-07-14_19_52_25-8ockgvr6/ca,pf=ipv4 stdout 2>&4
| { { read -n 42 magic && if test "$magic" !=
M=b0f141f7085de052088d687fc70a027e53928be8; then echo 'Magic value
mismatch' >&2; exit 1;fi; } && { LC_ALL=C dd bs=1048576 <&0 2>&6 &
pid=${!}; echo $pid >&8; wait $pid; } }
I tried to simplify this down to a pair of socat commands like this:
/usr/bin/socat -ls -d -d -b1048576 -u
OPENSSL-LISTEN:12345,reuseaddr,forever,intervall=0.01,keepalive,keepidle=60,keepintvl=10,keepcnt=5,verify=0,cipher=HIGH:-DES:-3DES:-EXPORT:-DH,compress=none,key=/etc/ssl/private/ssl-cert-snakeoil.key,cert=/etc/ssl/certs/ssl-cert-snakeoil.pem,pf=ipv4
stdout | sha1sum &
dd if=/usr/bin/perl bs=1048576 | /usr/bin/socat -ls -d -d -b1048576 -u
stdin
OPENSSL:127.0.0.1:12345,connect-timeout=20,retry=10,intervall=1,keepalive,keepidle=60,keepintvl=10,keepcnt=5,verify=1,cipher=HIGH:-DES:-3DES:-EXPORT:-DH,compress=none,key=/etc/ssl/private/ssl-cert-snakeoil.key,cert=//etc/ssl/certs/ssl-cert-snakeoil.pem
But this fails with: "ioctl(6, IOCTL_VM_SOCKETS_GET_LOCAL_CID, ...):
Inappropriate ioctl for device" and my google searches didn't turn up
what might be the problem.
I'm going to be just installing the 21.04 socat package on my
ganeti+20.04 systems, as so far in my testing that seems to be working
fine and will prevent me from having to pin the 18.04 packages.
** Affects: socat (Ubuntu)
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1936407
Title:
socat bug with SSL "filte transfers"
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/socat/+bug/1936407/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
