Public bug reported: Test run here: https://solutions.qa.canonical.com/testruns/testRun/19f7492e-1a8c-4ec3-b0bb-0fc4f2453f7b
Artifacts/Logs/Bundles here: https://oil-jenkins.canonical.com/artifacts/19f7492e-1a8c-4ec3-b0bb-0fc4f2453f7b/index.html Juju Openstack model crashdump here: https://oil-jenkins.canonical.com/artifacts/19f7492e-1a8c-4ec3-b0bb-0fc4f2453f7b/generated/generated/openstack/juju-crashdump-openstack-2021-04-09-19.57.18.tar.gz --- We've seen this in several different environments. Some using Stein- Bionic, some using Ussuri-Bionic. Ussuri-Focal had no issues. Juju ssh-ing to any of these machines is incredibly slow and unresponsive. (an ls takes somewhere around 30-60seconds) As seen below, we're failing to grab lxd images for our instances. Likely due to the diminished networking. On an earlier manual run I was apt installing at 400B/s. --- Machine State DNS Inst id Series AZ Message 0 started <ourIP> duision bionic zone1 Deployed 0/lxd/0 pending pending bionic starting 0/lxd/1 pending pending bionic starting 0/lxd/2 pending pending bionic starting 0/lxd/3 pending pending bionic acquiring LXD image 0/lxd/4 pending pending bionic starting 0/lxd/5 down pending bionic failed to start machine 0/lxd/5 (acquiring LXD image: no matching image found), retrying in 10s (10 more attempts) 0/lxd/6 pending pending bionic Creating container spec 0/lxd/7 pending pending bionic starting 0/lxd/8 pending pending bionic starting 0/lxd/9 pending pending bionic starting 0/lxd/10 pending pending bionic starting 1 started <ourIP> azurill bionic zone1 Deployed 1/lxd/0 pending pending bionic starting 1/lxd/1 down pending bionic failed to start machine 1/lxd/1 (acquiring LXD image: no matching image found), retrying in 10s (10 more attempts) 1/lxd/2 pending pending bionic starting 1/lxd/3 pending pending bionic starting 1/lxd/4 pending pending bionic starting 1/lxd/5 pending pending bionic starting 1/lxd/6 pending pending bionic starting 1/lxd/7 down pending bionic failed to start machine 1/lxd/7 (acquiring LXD image: Failed remote image download: Get https://cloud-images.ubuntu.com/releases/server/releases/bionic/release-20210325/ubuntu-18.04-server-cloudimg-amd64.squashfs: proxyconnect tcp: Unable to connect to: squid.internal), retrying in 10s (10 more attempts) 1/lxd/8 pending pending bionic starting 1/lxd/9 pending pending bionic starting 1/lxd/10 pending pending bionic starting 2 started <ourIP> meowth bionic zone2 Deployed 2/lxd/0 down pending bionic failed to start machine 2/lxd/0 (acquiring LXD image: Failed remote image download: Get https://cloud-images.ubuntu.com/releases/bionic/release-20210325/ubuntu-18.04-server-cloudimg-amd64.squashfs: read tcp: read: connection reset by peer), retrying in 10s (10 more attempts) --- kern.log from one of our machines shows a bunch of this: --- Apr 9 20:52:22 azurill kernel: [ 4196.092358] br-eth0: received packet on eth0 with own address as source address Apr 9 20:52:22 azurill kernel: [ 4196.317426] br-eth0: received packet on eth0 with own address as source address Apr 9 20:52:22 azurill kernel: [ 4196.387884] br-eth0: received packet on eth0 with own address as source address Apr 9 20:52:22 azurill kernel: [ 4196.423968] br-eth0: received packet on eth0 with own address as source address Apr 9 20:52:22 azurill kernel: [ 4196.488808] br-eth0: received packet on eth0 with own address as source address Apr 9 20:52:22 azurill kernel: [ 4196.919973] br-eth0: received packet on eth0 with own address as source address Apr 9 20:52:23 azurill kernel: [ 4197.116675] br-eth0: received packet on eth0 with own address as source address Apr 9 20:52:23 azurill kernel: [ 4197.331899] br-eth0: received packet on eth0 with own address as source address Apr 9 20:52:23 azurill kernel: [ 4197.447991] br-eth0: received packet on eth0 with own address as source address Apr 9 20:52:23 azurill kernel: [ 4197.513159] br-eth0: received packet on eth0 with own address as source address ** Affects: cloud-init Importance: Undecided Status: New ** Tags: cdo-qa cdo-release-blocker foundations-engine -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to cloud-init. https://bugs.launchpad.net/bugs/1923257 Title: diminished networking, packet issues during Bionic openstack deploys Status in cloud-init: New Bug description: Test run here: https://solutions.qa.canonical.com/testruns/testRun/19f7492e-1a8c-4ec3-b0bb-0fc4f2453f7b Artifacts/Logs/Bundles here: https://oil-jenkins.canonical.com/artifacts/19f7492e-1a8c-4ec3-b0bb-0fc4f2453f7b/index.html Juju Openstack model crashdump here: https://oil-jenkins.canonical.com/artifacts/19f7492e-1a8c-4ec3-b0bb-0fc4f2453f7b/generated/generated/openstack/juju-crashdump-openstack-2021-04-09-19.57.18.tar.gz --- We've seen this in several different environments. Some using Stein- Bionic, some using Ussuri-Bionic. Ussuri-Focal had no issues. Juju ssh-ing to any of these machines is incredibly slow and unresponsive. (an ls takes somewhere around 30-60seconds) As seen below, we're failing to grab lxd images for our instances. Likely due to the diminished networking. On an earlier manual run I was apt installing at 400B/s. --- Machine State DNS Inst id Series AZ Message 0 started <ourIP> duision bionic zone1 Deployed 0/lxd/0 pending pending bionic starting 0/lxd/1 pending pending bionic starting 0/lxd/2 pending pending bionic starting 0/lxd/3 pending pending bionic acquiring LXD image 0/lxd/4 pending pending bionic starting 0/lxd/5 down pending bionic failed to start machine 0/lxd/5 (acquiring LXD image: no matching image found), retrying in 10s (10 more attempts) 0/lxd/6 pending pending bionic Creating container spec 0/lxd/7 pending pending bionic starting 0/lxd/8 pending pending bionic starting 0/lxd/9 pending pending bionic starting 0/lxd/10 pending pending bionic starting 1 started <ourIP> azurill bionic zone1 Deployed 1/lxd/0 pending pending bionic starting 1/lxd/1 down pending bionic failed to start machine 1/lxd/1 (acquiring LXD image: no matching image found), retrying in 10s (10 more attempts) 1/lxd/2 pending pending bionic starting 1/lxd/3 pending pending bionic starting 1/lxd/4 pending pending bionic starting 1/lxd/5 pending pending bionic starting 1/lxd/6 pending pending bionic starting 1/lxd/7 down pending bionic failed to start machine 1/lxd/7 (acquiring LXD image: Failed remote image download: Get https://cloud-images.ubuntu.com/releases/server/releases/bionic/release-20210325/ubuntu-18.04-server-cloudimg-amd64.squashfs: proxyconnect tcp: Unable to connect to: squid.internal), retrying in 10s (10 more attempts) 1/lxd/8 pending pending bionic starting 1/lxd/9 pending pending bionic starting 1/lxd/10 pending pending bionic starting 2 started <ourIP> meowth bionic zone2 Deployed 2/lxd/0 down pending bionic failed to start machine 2/lxd/0 (acquiring LXD image: Failed remote image download: Get https://cloud-images.ubuntu.com/releases/bionic/release-20210325/ubuntu-18.04-server-cloudimg-amd64.squashfs: read tcp: read: connection reset by peer), retrying in 10s (10 more attempts) --- kern.log from one of our machines shows a bunch of this: --- Apr 9 20:52:22 azurill kernel: [ 4196.092358] br-eth0: received packet on eth0 with own address as source address Apr 9 20:52:22 azurill kernel: [ 4196.317426] br-eth0: received packet on eth0 with own address as source address Apr 9 20:52:22 azurill kernel: [ 4196.387884] br-eth0: received packet on eth0 with own address as source address Apr 9 20:52:22 azurill kernel: [ 4196.423968] br-eth0: received packet on eth0 with own address as source address Apr 9 20:52:22 azurill kernel: [ 4196.488808] br-eth0: received packet on eth0 with own address as source address Apr 9 20:52:22 azurill kernel: [ 4196.919973] br-eth0: received packet on eth0 with own address as source address Apr 9 20:52:23 azurill kernel: [ 4197.116675] br-eth0: received packet on eth0 with own address as source address Apr 9 20:52:23 azurill kernel: [ 4197.331899] br-eth0: received packet on eth0 with own address as source address Apr 9 20:52:23 azurill kernel: [ 4197.447991] br-eth0: received packet on eth0 with own address as source address Apr 9 20:52:23 azurill kernel: [ 4197.513159] br-eth0: received packet on eth0 with own address as source address To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1923257/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
