[Yahoo-eng-team] [Bug 1834875] Re: cloud-init growpart race with udev
** Project changed: cloud-init => cloud-initramfs-tools ** Changed in: cloud-initramfs-tools Status: Incomplete => Fix Released ** Bug watch removed: github.com/systemd/systemd/issues #12258 https://github.com/systemd/systemd/issues/12258 -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to cloud-init. https://bugs.launchpad.net/bugs/1834875 Title: cloud-init growpart race with udev Status in cloud-initramfs-tools: Fix Released Status in cloud-utils: Fix Committed Status in cloud-initramfs-tools package in Ubuntu: Fix Released Status in cloud-utils package in Ubuntu: Fix Released Status in cloud-initramfs-tools source package in Eoan: Fix Committed Status in cloud-utils source package in Eoan: Fix Committed Bug description: [impact] On Azure, it happens regularly (20-30%), that cloud-init's growpart module fails to extend the partition to full size. Such as in this example: 2019-06-28 12:24:18,666 - util.py[DEBUG]: Running command ['growpart', '--dry-run', '/dev/sda', '1'] with allowed return codes [0] (shell=False, capture=True) 2019-06-28 12:24:19,157 - util.py[DEBUG]: Running command ['growpart', '/dev/sda', '1'] with allowed return codes [0] (shell=False, capture=True) 2019-06-28 12:24:19,726 - util.py[DEBUG]: resize_devices took 1.075 seconds 2019-06-28 12:24:19,726 - handlers.py[DEBUG]: finish: init-network/config-growpart: FAIL: running config-growpart with frequency always 2019-06-28 12:24:19,727 - util.py[WARNING]: Running module growpart () failed 2019-06-28 12:24:19,727 - util.py[DEBUG]: Running module growpart () failed Traceback (most recent call last): File "/usr/lib/python3/dist-packages/cloudinit/stages.py", line 812, in _run_modules freq=freq) File "/usr/lib/python3/dist-packages/cloudinit/cloud.py", line 54, in run return self._runners.run(name, functor, args, freq, clear_on_fail) File "/usr/lib/python3/dist-packages/cloudinit/helpers.py", line 187, in run results = functor(*args) File "/usr/lib/python3/dist-packages/cloudinit/config/cc_growpart.py", line 351, in handle func=resize_devices, args=(resizer, devices)) File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 2521, in log_time ret = func(*args, **kwargs) File "/usr/lib/python3/dist-packages/cloudinit/config/cc_growpart.py", line 298, in resize_devices (old, new) = resizer.resize(disk, ptnum, blockdev) File "/usr/lib/python3/dist-packages/cloudinit/config/cc_growpart.py", line 159, in resize return (before, get_size(partdev)) File "/usr/lib/python3/dist-packages/cloudinit/config/cc_growpart.py", line 198, in get_size fd = os.open(filename, os.O_RDONLY) FileNotFoundError: [Errno 2] No such file or directory: '/dev/disk/by-partuuid/a5f2b49f-abd6-427f-bbc4-ba5559235cf3' @rcj suggested this is a race with udev. This seems to only happen on Cosmic and later. [test case] slightly unfortunately, the only way we've been able to reliably reproduce this is to get the CPC team to run their tests in azure, so we'll have to rely on them for SRU verification too [regression potential] The change is fairly simple and was thoroughly reviewed and tested before it got uploaded to focal, and the backport is trivial so I am confident that the regression potential is slight. To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-initramfs-tools/+bug/1834875/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1878317] [NEW] pep8 error caused by hacking version < 3.0.1
Public bug reported: flake8 new release 3.8.0 added new checks and gate pep8 job start failing. hacking 3.0.1 fix the pinning of flake8 to avoid bringing in a new version with new checks. Though it is fixed in latest hacking but 2.0 and 3.0 has cap for flake8 as <4.0.0 which mean flake8 new version 3.9.0 can also break the pep8 job if new check are added. To avoid similar gate break in future, we need to bump the hacking min version. - http://lists.openstack.org/pipermail/openstack- discuss/2020-May/014828.html https://review.opendev.org/#/c/727347/ changing the hacing version, but some related error in the code, see below: pep8 run-test-pre: PYTHONHASHSEED='1704528896' pep8 finish: run-test-pre after 0.00 seconds pep8 start: run-test pep8 run-test: commands[0] | bash tools/flake8wrap.sh setting PATH=/home/zuul/src/opendev.org/openstack/nova/.tox/shared/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games [4453] /home/zuul/src/opendev.org/openstack/nova$ /bin/bash tools/flake8wrap.sh Running flake8 on all files ./nova/tests/unit/virt/vmwareapi/test_driver_api.py:126:14: H202: assertRaises Exception too broad ./nova/tests/unit/compute/test_compute_mgr.py:6264:9: E306 expected 1 blank line before a nested definition, found 0 ./nova/api/openstack/compute/servers.py:104:23: E741 ambiguous variable name 'l' ./nova/db/sqlalchemy/migrate_repo/versions/330_enforce_mitaka_online_migrations.py:48:35: E711 comparison to None should be 'if cond is None:' ERROR: InvocationError for command /bin/bash tools/flake8wrap.sh (exited with code 1) pep8 finish: run-test after 70.95 seconds pep8 start: run-test-post pep8 finish: run-test-post after 0.00 seconds ___ summary ERROR: pep8: commands failed ** Affects: nova Importance: Low Assignee: Brin Zhang (zhangbailin) Status: In Progress ** Changed in: nova Importance: Undecided => Low ** Changed in: nova Status: New => Confirmed ** Changed in: nova Assignee: (unassigned) => Brin Zhang (zhangbailin) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1878317 Title: pep8 error caused by hacking version < 3.0.1 Status in OpenStack Compute (nova): In Progress Bug description: flake8 new release 3.8.0 added new checks and gate pep8 job start failing. hacking 3.0.1 fix the pinning of flake8 to avoid bringing in a new version with new checks. Though it is fixed in latest hacking but 2.0 and 3.0 has cap for flake8 as <4.0.0 which mean flake8 new version 3.9.0 can also break the pep8 job if new check are added. To avoid similar gate break in future, we need to bump the hacking min version. - http://lists.openstack.org/pipermail/openstack- discuss/2020-May/014828.html https://review.opendev.org/#/c/727347/ changing the hacing version, but some related error in the code, see below: pep8 run-test-pre: PYTHONHASHSEED='1704528896' pep8 finish: run-test-pre after 0.00 seconds pep8 start: run-test pep8 run-test: commands[0] | bash tools/flake8wrap.sh setting PATH=/home/zuul/src/opendev.org/openstack/nova/.tox/shared/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games [4453] /home/zuul/src/opendev.org/openstack/nova$ /bin/bash tools/flake8wrap.sh Running flake8 on all files ./nova/tests/unit/virt/vmwareapi/test_driver_api.py:126:14: H202: assertRaises Exception too broad ./nova/tests/unit/compute/test_compute_mgr.py:6264:9: E306 expected 1 blank line before a nested definition, found 0 ./nova/api/openstack/compute/servers.py:104:23: E741 ambiguous variable name 'l' ./nova/db/sqlalchemy/migrate_repo/versions/330_enforce_mitaka_online_migrations.py:48:35: E711 comparison to None should be 'if cond is None:' ERROR: InvocationError for command /bin/bash tools/flake8wrap.sh (exited with code 1) pep8 finish: run-test after 70.95 seconds pep8 start: run-test-post pep8 finish: run-test-post after 0.00 seconds ___ summary ERROR: pep8: commands failed To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1878317/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1878314] [NEW] Upgrade Rocky milestone revisions has no effective
Public bug reported: Excute command "neutron-db-manage upgrade rocky", but didn't take effect. It seems lacks tag for rocky milestone revisions. [1] https://github.com/openstack/neutron/blob/master/neutron/db/migration/alembic_migrations/versions/rocky/expand/867d39095bf4_port_forwarding.py ** Affects: neutron Importance: Undecided Assignee: Dongcan Ye (hellochosen) Status: New ** Changed in: neutron Assignee: (unassigned) => Dongcan Ye (hellochosen) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1878314 Title: Upgrade Rocky milestone revisions has no effective Status in neutron: New Bug description: Excute command "neutron-db-manage upgrade rocky", but didn't take effect. It seems lacks tag for rocky milestone revisions. [1] https://github.com/openstack/neutron/blob/master/neutron/db/migration/alembic_migrations/versions/rocky/expand/867d39095bf4_port_forwarding.py To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1878314/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1878299] [NEW] [floatingip port_forwarding] changing external port to used value hangs with retriable exception
Public bug reported:
When trying to change external-protocol-port to a value that is already used by
another pf row, there is no error checking and we end up 'stuck' until the api
times out. The neutron db is catching
the improper config, but the validation should likely not allow it to get that
far.
(overcloud) [stack@undercloud-0 ~]$ openstack floating ip port forwarding list
${FIP_UUID}
+--+--+-+---+---+--+
| ID | Internal Port ID |
Internal IP Address | Internal Port | External Port | Protocol |
+--+--+-+---+---+--+
| 5a8515b8-9e03-442f-a8d7-41acd11f59b5 | 63767606-35ea-4c08-b6c8-1216d0c407e8 |
192.168.30.159 |22 | 2021 | tcp |
| 60693ea5-e985-404d-87ce-798dd4f6f4da | d5a31eba-89cb-40a8-ba98-d6ca1a8fffb1 |
192.168.30.84 |22 | 2020 | tcp |
+--+--+-+---+---+--+
(overcloud) [stack@undercloud-0 ~]$
(overcloud) [stack@undercloud-0 ~]$
# now, doing something invalid: try to change external-protocol-port of $PF_ID2
to the port that is being used by $PF_ID (60693ea5-e985-404d-87ce-798dd4f6f4da)
(overcloud) [stack@undercloud-0 ~]$ echo ${PF_ID2}
5a8515b8-9e03-442f-a8d7-41acd11f59b5
(overcloud) [stack@undercloud-0 ~]$ time openstack floating ip port
forwarding set --external-protocol-port 2020 ${FIP_UUID} ${PF_ID2}
HttpException: 504: Server Error for url:
http://10.0.0.125:9696/v2.0/floatingips/aee2c979-31a1-453a-b508-319c71fee9dc/port_forwardings/5a8515b8-9e03
-442f-a8d7-41acd11f59b5, The server didn't respond in time.: 504 Gateway
Time-out
real2m3.552s
user0m1.672s
sys 0m0.727s
=-==-
[root@controller-0 neutron]# tail -F server.log
...
2020-05-12 17:54:15.900 26 DEBUG neutron.api.v2.base
[req-f0726861-9f58-4e35-b3b0-15a8bd4ada5b 251059acfcae468c89fa33c988910832
ea7d486cda284d8fa7f3eaf8351f080d - default default] Request body:
{'port_forwarding': {'external_port': 2020}} prepare_request_body
/usr/lib/python3.6/site-packages/neutron/api/v2/base.py:719
2020-05-12 17:54:16.137 28 DEBUG neutron.wsgi [-] (28) accepted ('172.17.1.38',
57154) server /usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
2020-05-12 17:54:16.370 26 DEBUG neutron_lib.db.api
[req-f0726861-9f58-4e35-b3b0-15a8bd4ada5b 251059acfcae468c89fa33c988910832
ea7d486cda284d8fa7f3eaf8351f080d - default default] Retry wrapper got retriable
exception: (pymysql.err.IntegrityError) (1062, "Duplicate entry
'aee2c979-31a1-453a-b508-319c71fee9dc-2020-tcp' for key
'uniq_port_forwardings0floatingip_id0external_port0protocol'")
[SQL: UPDATE portforwardings SET external_port=%(external_port)s WHERE
portforwardings.id = %(portforwardings_id)s]
[parameters: {'external_port': 2020, 'portforwardings_id':
'5a8515b8-9e03-442f-a8d7-41acd11f59b5'}]
(Background on this error at: http://sqlalche.me/e/gkpj) wrapped
/usr/lib/python3.6/site-packages/neutron_lib/db/api.py:183
2020-05-12 17:54:16.371 26 DEBUG oslo_db.api
[req-f0726861-9f58-4e35-b3b0-15a8bd4ada5b 251059acfcae468c89fa33c988910832
ea7d486cda284d8fa7f3eaf8351f080d - default default] Performing DB retry for
function neutron.api.v2.base.Controller._update wrapper
/usr/lib/python3.6/site-packages/oslo_db/api.py:156
2020-05-12 17:54:16.458 28 DEBUG neutron.wsgi [-] (28) accepted
('172.17.1.141', 48380) server
/usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
2020-05-12 17:54:17.882 28 DEBUG neutron.wsgi [-] (28) accepted ('172.17.1.84',
39608) server /usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
2020-05-12 17:54:17.946 26 DEBUG neutron.api.v2.base
[req-f0726861-9f58-4e35-b3b0-15a8bd4ada5b 251059acfcae468c89fa33c988910832
ea7d486cda284d8fa7f3eaf8351f080d - default default] Request body:
{'port_forwarding': {'external_port': 2020}} prepare_request_body
/usr/lib/python3.6/site-packages/neutron/api/v2/base.py:719
2020-05-12 17:54:18.145 29 DEBUG neutron.wsgi [-] (29) accepted ('172.17.1.38',
57244) server /usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
2020-05-12 17:54:18.292 26 DEBUG neutron_lib.db.api
[req-f0726861-9f58-4e35-b3b0-15a8bd4ada5b 251059acfcae468c89fa33c988910832
ea7d486cda284d8fa7f3eaf8351f080d - default default] Retry wrapper got retriable
exception: (pymysql.err.IntegrityError) (1062, "Duplicate entry
'aee2c979-31a1-453a-b508-319c71fee9dc-2020-tcp' for key
'uniq_port_forwardings0floatingip_id0external_port0protocol'")
[SQL: UPDATE portforwardings SET external_port=%(external_port)s WHERE
portforwardings.id = %(portforwardings_id)s]
[parameters: {'external_port': 2020, 'portforwardings_id':
'5a8515b8-9e03-442f-a8d7-41acd11f59b5'}]
(Background on
[Yahoo-eng-team] [Bug 1877404] Re: Add "qos_policy_id" field to "FloatingIP" OVO
Reviewed: https://review.opendev.org/726208 Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=8cfe41fa6bea0901e4b2d41cc47d745195e9953b Submitter: Zuul Branch:master commit 8cfe41fa6bea0901e4b2d41cc47d745195e9953b Author: Rodolfo Alonso Hernandez Date: Thu May 7 15:49:21 2020 + Add "qos_policy_id" field to "FloatingIP" OVO This new synthetic field is linked to a "QosPolicyFloatingIPBinding" register. This binding register will bind a QoS policy and a floating IP. Now is possible to provide this field in the create/update input parameters. If provided, the "FloatingIP" OVO will create/delete the "QosPolicyFloatingIPBinding" register. The OVO takes this parameter from the DB object. When the DB object is retrieved, the QoS policy binding register is retrieved too due to a backref link in the "QosFIPPolicyBinding" DB model to the "FloatingIP" DB model. Change-Id: Ideb042a71336b110bbe0f9e81ed8e0c21434fc42 Closes-Bug: #1877404 Related-Bug: #1877408 ** Changed in: neutron Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1877404 Title: Add "qos_policy_id" field to "FloatingIP" OVO Status in neutron: Fix Released Bug description: This bug is related to https://bugs.launchpad.net/neutron/+bug/1596611 The aim of this bug is to add a new field in the "FloatingIP" OVO. Currently we have a "FloatingIP" DB object with a "qos_policy_binding" field. It contains, if any, a "QosFIPPolicyBinding". This register binds a QoS policy with a floating IP DB registers. This new synthetic field will contain, if present, the QoS policy ID attached to this floating IP, improving the OVO interface. This new interface will help in the OVN QoS FIP integration (https://bugs.launchpad.net/neutron/+bug/1877408). To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1877404/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1878218] [NEW] "Attach Interface" option is visible to all roles on instances in the Project section
Public bug reported: "Attach Interface" option is visible to all roles on instances in the Project section. Horizon user with all roles has "Attach Interface" action available in the Project -> Compute -> Instances tab. It should be visible only to the roles defined in the Horizon policies. This issue is reproducible on Horizon stein. ** Affects: horizon Importance: Undecided Assignee: Gayathri Devi Kathiri (gayathridevikathiri) Status: In Progress ** Changed in: horizon Assignee: (unassigned) => Gayathri Devi Kathiri (gayathridevikathiri) ** Changed in: horizon Status: New => In Progress -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Dashboard (Horizon). https://bugs.launchpad.net/bugs/1878218 Title: "Attach Interface" option is visible to all roles on instances in the Project section Status in OpenStack Dashboard (Horizon): In Progress Bug description: "Attach Interface" option is visible to all roles on instances in the Project section. Horizon user with all roles has "Attach Interface" action available in the Project -> Compute -> Instances tab. It should be visible only to the roles defined in the Horizon policies. This issue is reproducible on Horizon stein. To manage notifications about this bug go to: https://bugs.launchpad.net/horizon/+bug/1878218/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1877248] Re: Rabbitmq cluster split brain causes l2 population related flows to be cleared when ovs-neutron-agent is restarted
** Changed in: neutron Status: Opinion => In Progress ** Changed in: neutron Assignee: (unassigned) => norman shen (jshen28) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1877248 Title: Rabbitmq cluster split brain causes l2 population related flows to be cleared when ovs-neutron-agent is restarted Status in neutron: In Progress Bug description: Pre-conditions: Rabbitmq split brain, and then restart ovs-neutron- agent results: In normal, when the ovs-neutron-agent restarts, the method add_fdb_entries will be called to refresh the l2 pop related flows. However, under my experimental conditions, the agent cannot receive the rpc to call add_fdb_entries due to the Rabbitmq split brain, so the l2 pop related flows cannot be updated in time, so these old flows will be cleaned as stale due to the different cookie id. However, these flows are actually useful, and deleting them will affect the tenant traffic. Our temporary solution is to not restart the ovs-neutron-agent until the rabbitmq cluster is restored. Is there a better solution? To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1877248/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1875981] Re: Admin deleting servers for other tenants
I can reproduce this and I think the error is in the neutron code, not in designate. ** Also affects: neutron Importance: Undecided Status: New ** Changed in: neutron Status: New => Confirmed -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1875981 Title: Admin deleting servers for other tenants Status in Designate: New Status in neutron: Confirmed Bug description: lets say we have a tenant user_side with zone example.com (dns_domain defined on a shared neutron network) so the user create a server (dns neutron extension enabled) which result a designate recordset below server1.example.com in zone example.com that only exist in the tenant user_side if an admin wants to delete server1 from the tenant user_side it will use openstack server delete server1 which will delete the server but will not delete the designate recordset since the zone example.com does not exist in admin tenant which will leave an orphan record in designate the admin should be able to delete all the resources of server1 including the designate recordset To manage notifications about this bug go to: https://bugs.launchpad.net/designate/+bug/1875981/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1734204] Re: Insufficient free host memory pages available to allocate guest RAM with Open vSwitch DPDK in Newton
Picking this back up again - I'll fold the fix for the regression
introduced by this change into the same SRU so it will consist of two
patches.
** Changed in: nova (Ubuntu Bionic)
Status: Won't Fix => Triaged
** Changed in: cloud-archive/queens
Status: Won't Fix => Triaged
** Changed in: cloud-archive/queens
Assignee: (unassigned) => James Page (james-page)
** Changed in: nova (Ubuntu Bionic)
Assignee: (unassigned) => James Page (james-page)
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1734204
Title:
Insufficient free host memory pages available to allocate guest RAM
with Open vSwitch DPDK in Newton
Status in Ubuntu Cloud Archive:
Invalid
Status in Ubuntu Cloud Archive queens series:
Triaged
Status in OpenStack Compute (nova):
Fix Released
Status in nova package in Ubuntu:
Invalid
Status in nova source package in Bionic:
Triaged
Bug description:
When spawning an instance and scheduling it onto a compute node which still
has sufficient pCPUs for the instance and also sufficient free huge pages for
the instance memory, nova returns:
Raw
[stack@undercloud-4 ~]$ nova show 1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc
(...)
| fault| {"message": "Exceeded maximum number
of retries. Exceeded max scheduling attempts 3 for instance
1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc. Last exception: internal error: process
exited while connecting to monitor: 2017-11-23T19:53:20.311446Z qemu-kvm:
-chardev pty,id=cha", "code": 500, "details": " File
\"/usr/lib/python2.7/site-packages/nova/conductor/manager.py\", line 492, in
build_instances |
| | filter_properties,
instances[0].uuid)
|
| | File
\"/usr/lib/python2.7/site-packages/nova/scheduler/utils.py\", line 184, in
populate_retry
|
| | raise
exception.MaxRetriesExceeded(reason=msg)
|
| | ", "created":
"2017-11-23T19:53:22Z"}
(...)
And /var/log/nova/nova-compute.log on the compute node gives the following
ERROR message:
Raw
2017-11-23 19:53:21.021 153615 ERROR nova.compute.manager
[req-2ad59cdf-4901-4df1-8bd7-ebaea20b9361 5d1785ee87294a6fad5e2b91cc20
8c307c08d2234b339c504bfdd896c13e - - -] [instance:
1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc] Instance failed
to spawn
2017-11-23 19:53:21.021 153615 ERROR nova.compute.manager [instance:
1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc] Traceback (most recent call last):
2017-11-23 19:53:21.021 153615 ERROR nova.compute.manager [instance:
1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc] File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2087, in
_build_resources
2017-11-23 19:53:21.021 153615 ERROR nova.compute.manager [instance:
1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc] yield resources
2017-11-23 19:53:21.021 153615 ERROR nova.compute.manager [instance:
1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc] File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1928, in
_build_and_run_instance
2017-11-23 19:53:21.021 153615 ERROR nova.compute.manager [instance:
1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc] block_device_info=block_device_info)
2017-11-23 19:53:21.021 153615 ERROR nova.compute.manager [instance:
1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc] File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2674, in
spawn
2017-11-23 19:53:21.021 153615 ERROR nova.compute.manager [instance:
1b72e7a1-c298-4c92-8d2c-0a9fe886e9bc]
[Yahoo-eng-team] [Bug 1838309] Re: Live migration might fail when run after revert of previous live migration
Now that the minimum versions for Ussuri are libvirt 4.0.0 are QEMU 2.1,
I think we can close this one unless libvirt 4.0.0 with QEMU 2.5 have
the same issues.
Please open this one again if you see this.
** Changed in: nova
Status: New => Won't Fix
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1838309
Title:
Live migration might fail when run after revert of previous live
migration
Status in OpenStack Compute (nova):
Won't Fix
Bug description:
When migrating an instance between two computes on queens, running two
different qemu versions, first live migration failed and was rolled
back (traceback follows just in case, unrelated to this issue):
2019-07-26 14:39:44.469 1576 ERROR nova.virt.libvirt.driver
[req-26f3a831-8e4f-43a2-83ce-e60645264147 0aa8a4a6ed7d4733871ef79fa0302d43
31ee6aa6bff7498fba21b9807697ec32 - default default] [instance:
b0681d51-2924-44be-a8b7-36db0d86b92f] Live Migration failure: internal error:
qemu unexpectedly closed the monitor: 2019-07-26 14:39:43.479+: Domain
id=16 is tainted: shell-scripts
2019-07-26T14:39:43.630545Z qemu-system-x86_64: -drive
file=rbd:cinder/volume-df3d0060-451c-4b22-8d15-2c579fb47681:id=cinder:auth_supported=cephx\;none:mon_host=192.168.16.14\:6789\;192.168.16.15\:6789\;192.168.16.16\:6789,file.password-secret=virtio-disk2-secret0,format=raw,if=none,id=drive-virtio-disk2,serial=df3d0060-451c-4b22-8d15-2c579fb47681,cache=writeback,discard=unmap:
'serial' is deprecated, please use the corresponding option of '-device'
instead
2019-07-26T14:39:44.075108Z qemu-system-x86_64: VQ 2 size 0x80 <
last_avail_idx 0xedda - used_idx 0xeddd
2019-07-26T14:39:44.075130Z qemu-system-x86_64: Failed to load
virtio-balloon:virtio
2019-07-26T14:39:44.075134Z qemu-system-x86_64: error while loading state for
instance 0x0 of device ':00:07.0/virtio-balloon'
2019-07-26T14:39:44.075582Z qemu-system-x86_64: load of migration failed:
Operation not permitted: libvirtError: internal error: qemu unexpectedly closed
the monitor: 2019-07-26 14:39:43.479+: Domain id=16 is tainted:
shell-scripts
then, after revert, live migration was retried, and now it failed
because of the following problem:
{u'message': u'Requested operation is not valid: cannot undefine transient
domain', u'code': 500, u'details': u' File
"/usr/lib/python2.7/dist-packages/nova/compute/manag
er.py", line 202, in decorated_function\nreturn function(self, context,
*args, **kwargs)\n File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6438, in
_post_live_migration\ndestroy_vifs=destroy_vifs)\n File
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1100, in
cleanup\nself._undefine_domain(instance)\n File
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1012, in
_undefine_domain\ninstance=instance)\n File
"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in
__exit__\nself.force_reraise()\n File
"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in
force_reraise\nsix.reraise(self.type_, self.value, self.tb)\n File
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 999, in
_undefine_domain\nguest.delete_configuration(support_uefi)\n File
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 271, in
delete_configuration\nself._domain.undefine()\n File
"/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit\n
result = proxy_call(self._autowrap, f, *args, **kwargs)\n File
"/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call\n
rv = execute(f, *args, **kwargs)\n File
"/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute\n
six.reraise(c, e, tb)\n File
"/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker\n
rv = meth(*args, **kwargs)\n File
"/usr/lib/python2.7/dist-packages/libvirt.py", line 2701, in undefine\nif
ret == -1: raise libvirtError (\'virDomainUndefine() failed\', dom=self)\n',
u'created': u'2019-07-29T14:39:41Z'}
It seems to happen because a domain was already undefined once on the
first try to live migrate and after that it can not be undefined
second time. We might need to check if the domain is persistent before
undefining it in case of live migrations.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1838309/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1878160] [NEW] [OVN] Functional tests environment is using old OVN
Public bug reported: Environment used for functional tests in Neutron master installs old OVS and OVN instead newest one specified in zuul configuration. 2020-05-11 17:51:44.018628 | controller | + /home/zuul/src/opendev.org/openstack/neutron/tools/configure_for_func_testing.sh:_install_base_deps:111 : source /home/zuul/src/opendev.org/openstack/neutron/devstack/lib/ovs 2020-05-11 17:51:44.021258 | controller | ++ /home/zuul/src/opendev.org/openstack/neutron/devstack/lib/ovs:source:13 : OVS_REPO=https://github.com/openvswitch/ovs.git 2020-05-11 17:51:44.024454 | controller | +++ /home/zuul/src/opendev.org/openstack/neutron/devstack/lib/ovs:source:14 : basename https://github.com/openvswitch/ovs.git 2020-05-11 17:51:44.025329 | controller | +++ /home/zuul/src/opendev.org/openstack/neutron/devstack/lib/ovs:source:14 : cut -f1 -d. 2020-05-11 17:51:44.029281 | controller | ++ /home/zuul/src/opendev.org/openstack/neutron/devstack/lib/ovs:source:14 : OVS_REPO_NAME=ovs 2020-05-11 17:51:44.031459 | controller | ++ /home/zuul/src/opendev.org/openstack/neutron/devstack/lib/ovs:source:15 : OVS_BRANCH=master 2020-05-11 17:51:44.034660 | controller | + /home/zuul/src/opendev.org/openstack/neutron/tools/configure_for_func_testing.sh:_install_base_deps:112 : remove_ovs_packages 2020-05-11 17:51:44.036794 | controller | + /home/zuul/src/opendev.org/openstack/neutron/devstack/lib/ovs:remove_ovs_packages:213 : for package in openvswitch openvswitch-switch openvswitch-common 2020-05-11 17:51:44.039203 | controller | + /home/zuul/src/opendev.org/openstack/neutron/devstack/lib/ovs:remove_ovs_packages:214 : is_package_installed openvswitch 2020-05-11 17:51:44.040999 | controller | + functions-common:is_package_installed:1338 : [[ -z openvswitch ]] 2020-05-11 17:51:44.042712 | controller | + functions-common:is_package_installed:1342 : [[ -z deb ]] 2020-05-11 17:51:44.044782 | controller | + functions-common:is_package_installed:1346 : [[ deb = \d\e\b ]] 2020-05-11 17:51:44.046600 | controller | + functions-common:is_package_installed:1347 : dpkg -s openvswitch 2020-05-11 17:51:44.063710 | controller | + /home/zuul/src/opendev.org/openstack/neutron/devstack/lib/ovs:remove_ovs_packages:213 : for package in openvswitch openvswitch-switch openvswitch-common 2020-05-11 17:51:44.065874 | controller | + /home/zuul/src/opendev.org/openstack/neutron/devstack/lib/ovs:remove_ovs_packages:214 : is_package_installed openvswitch-switch 2020-05-11 17:51:44.068487 | controller | + functions-common:is_package_installed:1338 : [[ -z openvswitch-switch ]] 2020-05-11 17:51:44.070984 | controller | + functions-common:is_package_installed:1342 : [[ -z deb ]] 2020-05-11 17:51:44.073079 | controller | + functions-common:is_package_installed:1346 : [[ deb = \d\e\b ]] 2020-05-11 17:51:44.075080 | controller | + functions-common:is_package_installed:1347 : dpkg -s openvswitch-switch 2020-05-11 17:51:44.093565 | controller | + /home/zuul/src/opendev.org/openstack/neutron/devstack/lib/ovs:remove_ovs_packages:213 : for package in openvswitch openvswitch-switch openvswitch-common 2020-05-11 17:51:44.095882 | controller | + /home/zuul/src/opendev.org/openstack/neutron/devstack/lib/ovs:remove_ovs_packages:214 : is_package_installed openvswitch-common 2020-05-11 17:51:44.098046 | controller | + functions-common:is_package_installed:1338 : [[ -z openvswitch-common ]] 2020-05-11 17:51:44.100093 | controller | + functions-common:is_package_installed:1342 : [[ -z deb ]] 2020-05-11 17:51:44.102364 | controller | + functions-common:is_package_installed:1346 : [[ deb = \d\e\b ]] 2020-05-11 17:51:44.104367 | controller | + functions-common:is_package_installed:1347 : dpkg -s openvswitch-common 2020-05-11 17:51:44.121001 | controller | + /home/zuul/src/opendev.org/openstack/neutron/tools/configure_for_func_testing.sh:_install_base_deps:113 : OVS_BRANCH=v2.12.0 2020-05-11 17:51:44.123399 | controller | + /home/zuul/src/opendev.org/openstack/neutron/tools/configure_for_func_testing.sh:_install_base_deps:114 : compile_ovs False /usr /var 2020-05-11 17:51:57.183697 | controller | + /home/zuul/src/opendev.org/openstack/neutron/devstack/lib/ovs:prepare_for_compilation:43 : cd /home/zuul/src/opendev.org/openstack/ovs 2020-05-11 17:51:57.186069 | controller | + /home/zuul/src/opendev.org/openstack/neutron/devstack/lib/ovs:prepare_for_compilation:44 : git checkout v2.12.0 2020-05-11 17:51:57.477222 | controller | Note: checking out 'v2.12.0'. Example log: https://0b56f229cf5dd2511d5c- 6e1e7a8d8abee2ff0e137b8a66c992cf.ssl.cf2.rackcdn.com/726850/1/check /neutron-functional/2f23d61/job-output.txt ** Affects: neutron Importance: High Status: Confirmed ** Tags: ovn ovn-octavia-provider ** Tags added: ovn-octavia-provider ** Changed in: neutron Importance: Undecided => High ** Changed in: neutron Status: New => Confirmed -- You received this bug
[Yahoo-eng-team] [Bug 1876026] Re: need a support to not allow 0.0.0.0/8 , 0.0.0.0/32 or ::/0
Moving it neutron as its neutron related PR ** Project changed: nova => neutron -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1876026 Title: need a support to not allow 0.0.0.0/8 , 0.0.0.0/32 or ::/0 Status in neutron: New Bug description: With VIO, We also support 0.0.0.0/8 , 0.0.0.0/32 and ::/0 however while sending api , we are making it as 0.0.0.0/0 only as it invalid cidr and not needed from openstack point of view. We can add support so that it wont be alloewd only To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1876026/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
