Re: [libvirt] virNetSocketNewListenTCP tries just one address
On Wed, Mar 28, Olaf Hering wrote: > > How can libvirt tell whether this is a misconfiguration of DNS or host's > > interfaces? > By simply cycling through the 'runp' list to see if any bind() succeeds? This change fixes /etc/sysconfig/network/ifcfg-br0:BOOTPROTO='dhcp4' for me. Just keep going in virNetSocketNewListenTCP: Apr 11 22:18:02 macintyre-old libvirtd[8017]: 2018-04-11 20:18:02.553+: 8021: error : virNetSocketNewListenTCP:323 : virNetSocketNewListenTCP: macintyre-old.arch.suse.de 49152 0: Success Apr 11 22:18:02 macintyre-old libvirtd[8017]: 2018-04-11 20:18:02.555+: 8021: error : virNetSocketNewListenTCP:394 : virNetSocketNewListenTCP: bind to '10.161.8.197': Ok: Success Apr 11 22:18:02 macintyre-old libvirtd[8017]: 2018-04-11 20:18:02.555+: 8021: error : virNetSocketNewListenTCP:394 : virNetSocketNewListenTCP: bind to '2620:113:80c0:8000:10:161:8:197': Ok: Cannot assign requested address You get the idea. Olaf --- src/rpc/virnetsocket.c | 34 -- 1 file changed, 20 insertions(+), 14 deletions(-) --- a/src/rpc/virnetsocket.c +++ b/src/rpc/virnetsocket.c @@ -320,6 +320,7 @@ int virNetSocketNewListenTCP(const char *nodename, *retsocks = NULL; *nretsocks = 0; +virReportSystemError(errno, "%s: %s %s %d", __func__, nodename, service, family); memset(&hints, 0, sizeof(hints)); hints.ai_family = family; hints.ai_flags = AI_PASSIVE; @@ -383,12 +384,17 @@ int virNetSocketNewListenTCP(const char *nodename, } } #endif - -if (bind(fd, runp->ai_addr, runp->ai_addrlen) < 0) { -if (errno != EADDRINUSE) { -virReportSystemError(errno, "%s", _("Unable to bind to port")); -goto error; -} +e = bind(fd, runp->ai_addr, runp->ai_addrlen); + { +char hostname[123]; +int r, oe = errno; +memset(hostname, 0, sizeof(hostname)); +r = getnameinfo(runp->ai_addr,runp->ai_addrlen,hostname, sizeof(hostname), NULL, 0, NI_NUMERICHOST); +errno = oe; +virReportSystemError(errno, "%s: bind to '%s': %s", __func__, hostname, r ? gai_strerror(r) : "Ok"); + } +if (e < 0) { +if (errno == EADDRINUSE) addrInUse = true; VIR_FORCE_CLOSE(fd); runp = runp->ai_next; @@ -412,14 +418,14 @@ int virNetSocketNewListenTCP(const char *nodename, fd = -1; } -if (nsocks == 0 && familyNotSupported) { -virReportSystemError(EAFNOSUPPORT, "%s", _("Unable to bind to port")); -goto error; -} - -if (nsocks == 0 && -addrInUse) { -virReportSystemError(EADDRINUSE, "%s", _("Unable to bind to port")); +if (nsocks == 0) { + if (familyNotSupported) +errno = EAFNOSUPPORT; + else if(addrInUse) +errno = EADDRINUSE; + else +errno = EDESTADDRREQ; +virReportSystemError(errno, "%s", _("Unable to bind to port")); goto error; } signature.asc Description: PGP signature -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] virNetSocketNewListenTCP tries just one address
On Tue, Mar 27, Ján Tomko wrote: > It cannot, but the admin of the network should be able to control both. The admin must not control my (test) host, nor must I control the DNS server in the network. But there are likely cases where the admin for DNS and libvirtd is the same person. > How can libvirt tell whether this is a misconfiguration of DNS or host's > interfaces? By simply cycling through the 'runp' list to see if any bind() succeeds? Olaf signature.asc Description: PGP signature -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] virNetSocketNewListenTCP tries just one address
On Tue, Mar 27, 2018 at 11:18:10AM +0200, Olaf Hering wrote: On Tue, Mar 27, Ján Tomko wrote: Why does your hostname resolve to an unavailable address? How can the DNS server possibly know how a host has configured itself? It cannot, but the admin of the network should be able to control both. How can libvirt tell whether this is a misconfiguration of DNS or host's interfaces? But we can possibly apply different rules for different callers: virNetServerServiceNewTCP where silently ignoring a failure for one address on daemon/system startup might be hard to catch and libxlDomainMigrationDstPrepare which was called by an API or by data source - whether it was user provided in the migration URI or libvirtd tried to figure it out For QEMU migration, we allow overriding the default listen address via the migrate_host qemu.conf option, this is passed to QEMU and looking at the code, it goes for best-effort and does not report an error as long as binding to one of the addresses succeeds. Anyway, globally ignoring EADDRNOTAVAIL feels too lenient for me. Jano In this case I had BOOTPROTO='dhcp4' instead of 'dhcp' in /etc/sysconfig/network/ifcfg-br0 due to all the migration issues I'm seeing. It turned out they are unrelated to such setting. Olaf -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list signature.asc Description: Digital signature -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] virNetSocketNewListenTCP tries just one address
On Tue, Mar 27, Ján Tomko wrote: > Why does your hostname resolve to an unavailable address? How can the DNS server possibly know how a host has configured itself? In this case I had BOOTPROTO='dhcp4' instead of 'dhcp' in /etc/sysconfig/network/ifcfg-br0 due to all the migration issues I'm seeing. It turned out they are unrelated to such setting. Olaf signature.asc Description: PGP signature -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] virNetSocketNewListenTCP tries just one address
On Tue, Mar 27, 2018 at 09:57:13AM +0200, Olaf Hering wrote: To rescue this bug from the noise in a subthread: If a hostname resolves to more than one address and the host currently configured itself for just IPv4, doing a bind() to some IPv6 address will fail. As a result an error is returned instead of continuing with the next item in 'runp'. Mär 20 09:35:52 macintyre-old libvirtd[4527]: 2018-03-20 08:35:52.521+: 4531: error : virNetSocketNewListenTCP:389 : Unable to bind to port: Cannot assign requested address After further debugging: 27672 16:04:46.774906 socket(PF_INET6, SOCK_STREAM, IPPROTO_TCP) = 35 27672 16:04:46.775041 setsockopt(35, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0 27672 16:04:46.775172 setsockopt(35, SOL_IPV6, IPV6_V6ONLY, [1], 4) = 0 27672 16:04:46.775302 bind(35, {sa_family=AF_INET6, sin6_port=htons(49152), inet_pton(AF_INET6, "2620:113:80c0:8000:10:161:8:197", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = -1 EADDRNOTAVAIL (Cannot assign requested address) 27672 16:04:46.775455 gettid() = 27672 27672 16:04:46.775590 write(4, "2018-03-20 15:04:46.775+: 27672: error : virNetSocketNewListenTCP:389 : Unable to bind to port: Cannot assign requested address\n", 132) = 132 27672 16:04:46.775742 gettid() = 27672 27672 16:04:46.775875 write(4, "2018-03-20 15:04:46.775+: 27672: info : virObjectUnref:350 : OBJECT_UNREF: obj=0x7fa4bc003530\n", 98) = 98 27672 16:04:46.776026 gettid() = 27672 So for some reason libvirtd tries to bind to ipv6 even if the host does not have that ipv6 address at this point, only the link-local address. Not sure if there is a way to detect that within libvirt. Perhaps it should just move on with the runp list and try the next one? The AI_ADDRCONFIG flag to getaddrinfo should have taken care of filtering out IPv6 addresses if you only have the link-local one (fe80::), but 2620:: looks like a legitimate unicast address. Checking for EADDRNOTAVAIL here is intentional to catch configuration errors (i.e. a typo in a literal listen address or misconfigured DNS). Why does your hostname resolve to an unavailable address? Jano signature.asc Description: Digital signature -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] virNetSocketNewListenTCP tries just one address
On Tue, Mar 27, 2018 at 09:57:13AM +0200, Olaf Hering wrote: > To rescue this bug from the noise in a subthread: > > If a hostname resolves to more than one address and the host currently > configured itself for just IPv4, doing a bind() to some IPv6 address > will fail. As a result an error is returned instead of continuing with > the next item in 'runp'. > > > Mär 20 09:35:52 macintyre-old libvirtd[4527]: 2018-03-20 08:35:52.521+: > > 4531: error : virNetSocketNewListenTCP:389 : Unable to bind to port: Cannot > > assign requested address > > After further debugging: > > 27672 16:04:46.774906 socket(PF_INET6, SOCK_STREAM, IPPROTO_TCP) = 35 > 27672 16:04:46.775041 setsockopt(35, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0 > 27672 16:04:46.775172 setsockopt(35, SOL_IPV6, IPV6_V6ONLY, [1], 4) = 0 > 27672 16:04:46.775302 bind(35, {sa_family=AF_INET6, sin6_port=htons(49152), > inet_pton(AF_INET6, "2620:113:80c0:8000:10:161:8:197", &sin6_addr), > sin6_flowinfo=0, sin6_scope_id=0}, 28) = -1 EADDRNOTAVAIL (Cannot assign > requested address) > 27672 16:04:46.775455 gettid() = 27672 > 27672 16:04:46.775590 write(4, "2018-03-20 15:04:46.775+: 27672: error : > virNetSocketNewListenTCP:389 : Unable to bind to port: Cannot assign > requested address\n", 132) = 132 > 27672 16:04:46.775742 gettid() = 27672 > 27672 16:04:46.775875 write(4, "2018-03-20 15:04:46.775+: 27672: info : > virObjectUnref:350 : OBJECT_UNREF: obj=0x7fa4bc003530\n", 98) = 98 > 27672 16:04:46.776026 gettid() = 27672 > > So for some reason libvirtd tries to bind to ipv6 even if the host does > not have that ipv6 address at this point, only the link-local address. > Not sure if there is a way to detect that within libvirt. Perhaps it > should just move on with the runp list and try the next one? It looks like the virNetSocketNewListenTCP only treats EAFNOSUPPORT as an error to continue with - everything else is fatal. At very least we need to add EADDRNOTAVAIL too. Regards, Daniel -- |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o-https://fstop138.berrange.com :| |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :| -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list