An update, A maintenance release on 12.1 available today provided an opportunity to catch this timeout issue again, so we tcpdumped our DNS traffic during the apt upgrade process.
The issue did not occur. All we saw were what appeared to be normal requests for ghcr.io and pkg-containers.githubusercontent.com and we did not experience any timeouts. Will report back if we see this issue again, cheers, Ian 12:06:17.484308 IP 105.244.196.73.37145 > 105.244.196.155.53: 23286+ AAAA? ghcr.io. (25) 12:06:17.484357 IP 105.244.196.73.43760 > 105.244.196.155.53: 14626+ A? ghcr.io. (25) 12:06:17.484516 IP 105.244.196.155.53 > 105.244.196.73.37145: 23286 0/1/0 (109) 12:06:17.484562 IP 105.244.196.155.53 > 105.244.196.73.43760: 14626 1/0/0 A 140.82.114.34 (41) 12:06:17.649272 IP 105.244.196.73.35596 > 105.244.196.155.53: 37537+ A? pkg-containers.githubusercontent.com. (54) 12:06:17.649312 IP 105.244.196.73.59271 > 105.244.196.155.53: 34028+ AAAA? pkg-containers.githubusercontent.com. (54) 12:06:17.649490 IP 105.244.196.155.53 > 105.244.196.73.35596: 37537 4/0/0 A 185.199.111.154, A 185.199.110.154, A 185.199.108.154, A 185.199.109.154 (118) 12:06:17.649562 IP 105.244.196.155.53 > 105.244.196.73.59271: 34028 4/0/0 AAAA 2606:50c0:8000::154, AAAA 2606:50c0:8002::154, AAAA 2606:50c0:8003::154, AAAA 2606:50c0:8001::154 (166) Thu Jan 12 12:02:55 EST 2023 - Images detected: - proxysql - haproxy-portal - pfsso - radiusd-eduroam - httpd.aaa - radiusd-cli - pfconfig - fingerbank-db - pfcmd - radiusd-load-balancer - httpd.admin_dispatcher - radiusd-acct - pfpki - httpd.portal - httpd.dispatcher - pfcron - pfconnector - httpd.webservices - radiusd-auth - haproxy-admin - pfqueue - api-frontend - pfperl-api ghcr.io/inverse-inc/packetfence/proxysql:maintenance-12-1 ghcr.io/inverse-inc/packetfence/haproxy-portal:maintenance-12-1 ghcr.io/inverse-inc/packetfence/pfsso:maintenance-12-1 ghcr.io/inverse-inc/packetfence/radiusd-eduroam:maintenance-12-1 ghcr.io/inverse-inc/packetfence/httpd.aaa:maintenance-12-1 ghcr.io/inverse-inc/packetfence/radiusd-cli:maintenance-12-1 ghcr.io/inverse-inc/packetfence/pfconfig:maintenance-12-1 ghcr.io/inverse-inc/packetfence/fingerbank-db:maintenance-12-1 ghcr.io/inverse-inc/packetfence/pfcmd:maintenance-12-1 ghcr.io/inverse-inc/packetfence/radiusd-load-balancer:maintenance-12-1 ghcr.io/inverse-inc/packetfence/httpd.admin_dispatcher:maintenance-12-1 ghcr.io/inverse-inc/packetfence/radiusd-acct:maintenance-12-1 ghcr.io/inverse-inc/packetfence/pfpki:maintenance-12-1 ghcr.io/inverse-inc/packetfence/httpd.portal:maintenance-12-1 ghcr.io/inverse-inc/packetfence/httpd.dispatcher:maintenance-12-1 ghcr.io/inverse-inc/packetfence/pfcron:maintenance-12-1 ghcr.io/inverse-inc/packetfence/pfconnector:maintenance-12-1 ghcr.io/inverse-inc/packetfence/httpd.webservices:maintenance-12-1 ghcr.io/inverse-inc/packetfence/radiusd-auth:maintenance-12-1 ghcr.io/inverse-inc/packetfence/haproxy-admin:maintenance-12-1 ghcr.io/inverse-inc/packetfence/pfqueue:maintenance-12-1 ghcr.io/inverse-inc/packetfence/api-frontend:maintenance-12-1 ghcr.io/inverse-inc/packetfence/pfperl-api:maintenance-12-1 Thu Jan 12 12:11:47 EST 2023 - Pull of images finished Thu Jan 12 12:11:49 EST 2023 - Tag of images finished On Tue, Jan 10, 2023 at 3:27 PM Ian MacDonald <i...@netstatz.com> wrote: > Hey PF Users, > > For recent versions; I believe 11.1, 12.0 and now 12.1 and possibly 11.0 > (Fairly certain since the images below were downloaded from Inverse repos > all at once during the installation or upgrade process) We have been having > to restart the upgrade process due to timeout related errors. Often 1-5 > times re-executions are required to complete the upgrade process. > > - proxysql > - haproxy-portal > - pfsso > - radiusd-eduroam > - httpd.aaa > - radiusd-cli > - pfconfig > - fingerbank-db > - pfcmd > - radiusd-load-balancer > - httpd.admin_dispatcher > - radiusd-acct > - pfpki > - httpd.portal > - httpd.dispatcher > - pfcron > - pfconnector > - httpd.webservices > - radiusd-auth > - haproxy-admin > - pfqueue > - api-frontend > - pfperl-api > > We do not really understand why, as there does not appear to be any > connectivity or DNS lookup issues that would cause this type of behavior. > Below are some of the output lines captured during our installation > process during a recent upgrade from 11.1 to 12.0 and then again from 12.0 > to 12.1. > > In a minor 12.0 upgrade we saw this one referencing an IPv6 github > address, yet the system is IPv4, so no idea why it is attempting IPv6 > error pulling image configuration: Get " > https://pkg-containers.githubusercontent.com/ghcr1/blobs/sha256:c31d236d97e3beb137f8c2b02bfbe88d0093b5592d9f181935c9c03a0132a142?se=2023-01-10T14%3A40%3A00Z&sig=%2B > HBahj6l0521Bm%2FB40v51MhZmNHztLYxzxBgJlsefEE%3D&sp=r&spr=https&sr=b&sv=2019-12-12": > dial tcp [2606:50c0:8001::154]:443: connect: network is unreachable > > In another 12.0 upgrade attempt, we saw this one, which looks like a > timeout to our DNS recursor, but to which there are no I/O bound or > restrictive conditions we can see. > > error pulling image configuration: Get " > https://ghcr.io/v2/inverse-inc/packetfence/pfcmd/blobs/sha256:5631317df2b6910aa8da1f20a382c04ecc0ffb572aeb7fd3201a18b0bee18633": > dial tcp: lookup ghcr.io on 105.244 > .196.155:53: read udp 10.2.1.2:35975->105.244.196.155:53: i/o timeout > > In 12.1 using the do-upgrade script we saw these similar messages > > Error response from daemon: Get "https://ghcr.io/v2/": dial tcp: lookup > ghcr.io on 105.244.196.155:53: read udp 10.2.1.2:60065->105.244.196.155:53: > i/o timeout > > Error response from daemon: Head " > https://ghcr.io/v2/inverse-inc/packetfence/radiusd-eduroam/manifests/maintenance-12-1": > dial tcp: lookup ghcr.io on 105.244.196.155:53: read udp 10.2.1.2:55582-> > 105.244.196.155:53: i/o timeout > > We just repeated, which seemed like a good time to send this email, and it > worked (3rd time just now on 12.1). > > Tue Jan 10 15:22:56 EST 2023 - Pull of images finished > Tue Jan 10 15:22:58 EST 2023 - Tag of images finished > Tue Jan 10 15:23:45 EST 2023 - Previous images cleaned > > It seems very odd that we get these timeouts when doing the image > downloading. Maybe somebody else has seen this or knows why it may be > occurring during this stage of the installation/upgrade process. > > cheers, > Ian >
_______________________________________________ PacketFence-users mailing list PacketFence-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/packetfence-users
