[Yahoo-eng-team] [Bug 1843801] Re: metadata-proxy process stops listening on port 80
[Expired for neutron because there has been no activity for 60 days.] ** Changed in: neutron Status: Incomplete => Expired -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1843801 Title: metadata-proxy process stops listening on port 80 Status in neutron: Expired Bug description: I'm running a metadata agent on provider network and I see that the metadata service stops listening on port 80 randomly. I see that the process itself is running, but port 80 is not open in the DHCP namespace. There are no logs in neutron-server, neutron- metadata-agent, neutron-dhcp-agent or journalctl. The only way to recover is to kill ns-metadata-proxy and have neutron- metadata-agent restart it at which point, the port is up. In addition to monitoring the process itself, neutron-metadata-agent must watch for port 80 in the namespace as well. ENV: Ubuntu 16.04 running neutron rocky. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1843801/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1852356] [NEW] tox testenv not installing requirements properly
Public bug reported:
I've encountered this recently when building new tox environments:
=
Failures during discovery
=
--- import errors ---
Failed to import test module: glance.tests.unit.api.test_cmd
Traceback (most recent call last):
File
"/home/brosmait/repos/openstack/glance/.tox/py36/lib/python3.6/site-packages/unittest2/loader.py",
line 456, in _find_test_path
module = self._get_module_from_name(name)
File
"/home/brosmait/repos/openstack/glance/.tox/py36/lib/python3.6/site-packages/unittest2/loader.py",
line 395, in _get_module_from_name
__import__(name)
File
"/home/brosmait/repos/openstack/glance/glance/tests/unit/api/test_cmd.py", line
20, in
import glance.cmd.api
File "/home/brosmait/repos/openstack/glance/glance/cmd/api.py", line 42, in
from oslo_reports import guru_meditation_report as gmr
ModuleNotFoundError: No module named 'oslo_reports'
Looking at the current tox.ini, the base testenv sets some env vars and then we
have:
install_command = pip install
-c{env:UPPER_CONSTRAINTS_FILE:https://releases.openstack.org/constraints/upper/master}
{opts} {packages}
deps = -r{toxinidir}/test-requirements.txt
No mention of the requirements at all.
I think we are hitting a bug that Tony Breeds patched in Cinder, Change-
Id: I5b1c285680f7c3256a707ea5973573f1176d4a26
Change the testenv like this:
install_command = pip install {opts} {packages}
deps =
-c{env:UPPER_CONSTRAINTS_FILE:https://releases.openstack.org/constraints/upper/master}
-r{toxinidir}/test-requirements.txt
-r{toxinidir}/requirements.txt
This makes sure that the u-c file is applied to both the test-req and
the requirements, which is what we want.
** Affects: glance
Importance: Medium
Assignee: Brian Rosmaita (brian-rosmaita)
Status: In Progress
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/1852356
Title:
tox testenv not installing requirements properly
Status in Glance:
In Progress
Bug description:
I've encountered this recently when building new tox environments:
=
Failures during discovery
=
--- import errors ---
Failed to import test module: glance.tests.unit.api.test_cmd
Traceback (most recent call last):
File
"/home/brosmait/repos/openstack/glance/.tox/py36/lib/python3.6/site-packages/unittest2/loader.py",
line 456, in _find_test_path
module = self._get_module_from_name(name)
File
"/home/brosmait/repos/openstack/glance/.tox/py36/lib/python3.6/site-packages/unittest2/loader.py",
line 395, in _get_module_from_name
__import__(name)
File
"/home/brosmait/repos/openstack/glance/glance/tests/unit/api/test_cmd.py", line
20, in
import glance.cmd.api
File "/home/brosmait/repos/openstack/glance/glance/cmd/api.py", line 42, in
from oslo_reports import guru_meditation_report as gmr
ModuleNotFoundError: No module named 'oslo_reports'
Looking at the current tox.ini, the base testenv sets some env vars and then
we have:
install_command = pip install
-c{env:UPPER_CONSTRAINTS_FILE:https://releases.openstack.org/constraints/upper/master}
{opts} {packages}
deps = -r{toxinidir}/test-requirements.txt
No mention of the requirements at all.
I think we are hitting a bug that Tony Breeds patched in Cinder,
Change-Id: I5b1c285680f7c3256a707ea5973573f1176d4a26
Change the testenv like this:
install_command = pip install {opts} {packages}
deps =
-c{env:UPPER_CONSTRAINTS_FILE:https://releases.openstack.org/constraints/upper/master}
-r{toxinidir}/test-requirements.txt
-r{toxinidir}/requirements.txt
This makes sure that the u-c file is applied to both the test-req and
the requirements, which is what we want.
To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/1852356/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1852354] [NEW] TestTasksDeserializer class not using FakePolicyEnforcer
Public bug reported: This was uncovered by a patch implementing policy-in-code for Glance: https://review.opendev.org/#/c/693129/ The TestTasksDeserializer class creates a glance.v2.tasks.RequestDeserializer but doesn't pass it a policy_enforcer. As a result, RequestDeserializer uses a real policy enforcer. The default policy in code that's checked by the RequestDeserializer is "tasks_api_access: role:admin", and with that policy in place, all the unit test requests fail. Need to pass the unit test class the FakePolicyEnforcer so that the deserialization can be tested. ** Affects: glance Importance: High Assignee: Brian Rosmaita (brian-rosmaita) Status: Triaged ** Changed in: glance Status: New => Triaged ** Changed in: glance Importance: Undecided => High ** Changed in: glance Assignee: (unassigned) => Brian Rosmaita (brian-rosmaita) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to Glance. https://bugs.launchpad.net/bugs/1852354 Title: TestTasksDeserializer class not using FakePolicyEnforcer Status in Glance: Triaged Bug description: This was uncovered by a patch implementing policy-in-code for Glance: https://review.opendev.org/#/c/693129/ The TestTasksDeserializer class creates a glance.v2.tasks.RequestDeserializer but doesn't pass it a policy_enforcer. As a result, RequestDeserializer uses a real policy enforcer. The default policy in code that's checked by the RequestDeserializer is "tasks_api_access: role:admin", and with that policy in place, all the unit test requests fail. Need to pass the unit test class the FakePolicyEnforcer so that the deserialization can be tested. To manage notifications about this bug go to: https://bugs.launchpad.net/glance/+bug/1852354/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1852175] Re: MessagingTimeout
** Also affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1852175 Title: MessagingTimeout Status in neutron: New Status in oslo.messaging: Incomplete Bug description: Hi All, We have a rocky 18.1.9 branch openstack ansible setup. When trying to launch a vm , it is going into ERROR state saying the vif creation failed. I see the following messages in the openvswitch agent log: --- 2019-11-01 17:29:23.971 6135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [-] Failed reporting state!: MessagingTimeout: Timed out waiting for a reply to messa ge ID ec0806a7312a4e55b090500257cc998a 2019-11-01 17:29:23.971 6135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last): 2019-11-01 17:29:23.971 6135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/openstack/venvs/neutron-18.1.9/lib/python2.7/site-packages/neutron/plugins/m l2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 325, in _report_state 2019-11-01 17:29:23.971 6135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent True) 2019-11-01 17:29:23.971 6135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/openstack/venvs/neutron-18.1.9/lib/python2.7/site-packages/neutron/agent/rpc .py", line 97, in report_state 2019-11-01 17:29:23.971 6135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return method(context, 'report_state', **kwargs) 2019-11-01 17:29:23.971 6135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/openstack/venvs/neutron-18.1.9/lib/python2.7/site-packages/oslo_messaging/rp c/client.py", line 179, in call 2019-11-01 17:29:23.971 6135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent retry=self.retry) 2019-11-01 17:29:23.971 6135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/openstack/venvs/neutron-18.1.9/lib/python2.7/site-packages/oslo_messaging/tr ansport.py", line 133, in _send 2019-11-01 17:29:23.971 6135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent retry=retry) 2019-11-01 17:29:23.971 6135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/openstack/venvs/neutron-18.1.9/lib/python2.7/site-packages/oslo_messaging/_d rivers/amqpdriver.py", line 584, in send 2019-11-01 17:29:23.971 6135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent call_monitor_timeout, retry=retry) 2019-11-01 17:29:23.971 6135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/openstack/venvs/neutron-18.1.9/lib/python2.7/site-packages/oslo_messaging/_d rivers/amqpdriver.py", line 573, in _send 2019-11-01 17:29:23.971 6135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent call_monitor_timeout) 2019-11-01 17:29:23.971 6135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/openstack/venvs/neutron-18.1.9/lib/python2.7/site-packages/oslo_messaging/_d rivers/amqpdriver.py", line 459, in wait 2019-11-01 17:29:23.971 6135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent message = self.waiters.get(msg_id, timeout=timeout) 2019-11-01 17:29:23.971 6135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/openstack/venvs/neutron-18.1.9/lib/python2.7/site-packages/oslo_messaging/_d rivers/amqpdriver.py", line 336, in get 2019-11-01 17:29:23.971 6135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 'to message ID %s' % msg_id) 2019-11-01 17:29:23.971 6135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent MessagingTimeout: Timed out waiting for a reply to message ID ec0806a7312a4e55b090500 257cc998a --- Please help us with the issue. Thanks Kumar To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1852175/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1852207] Re: reschedule ignores requested availability zone
** Changed in: nova Status: New => Invalid ** Tags added: pike-only -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1852207 Title: reschedule ignores requested availability zone Status in OpenStack Compute (nova): Invalid Status in OpenStack Compute (nova) pike series: In Progress Bug description: This is only observed on stable/pike. Reproduce - * Create an aggregate mapped to an availability zone * Add one host to the aggregate * Make sure (e.g. with fault injection) that the nova-compute on the host in the aggregate will fail during the instance_claim process to trigger re-schedule * boot an instance requesting the az Expected Instance boot fails with NoValidHost Actual -- Instance is rescheduled and the scheduler ignores the requested az and therefore instance is booted outside of the requested az. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1852207/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1850639] Re: FloatingIP list bad performance
Ok, so it's not related to sqlalchemy, as I expected it's an issue with
neutron DB object, fixed in Rocky:
https://review.opendev.org/#/c/565358/
** Changed in: neutron
Status: In Progress => Invalid
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1850639
Title:
FloatingIP list bad performance
Status in neutron:
Invalid
Bug description:
Faced on stable/queens but applicable to master too.
On quite a heavy loaded environment it was noticed that simple floatingip
list command takes significant time (~1200 fips) while for example port list is
always faster (>7000 ports).
If enable sqlalchemy debug logs there can be seen lots of:
2019-10-22 21:02:44,977.977 23957 DEBUG sqlalchemy.orm.path_registry
[req-3db31d53-f6b9-408e-b8c7-bf037ef10a1b 1df8a7d5eb
b5414b9e29cf581098681c 10479799101a4fe4ada17daa105707c5 - default default]
set 'memoized_setups' on path 'EntityRegistry(
(,))' to '{}' set
/usr/lib/python2.7/dist-packages/sqlalchemy/orm/path_registry.
py:63
- which basically eats all the time of a request.
As a test I commented 'dns' field in FloatingIP DB object definition and
response time reduced from 14 to 1 second. DNS extension is not configured on
the environment and no external DNS is used.
Also I don't see this field used anywhere in neutron.
Interestingly Port DB object has 'dns' field either (with
corresponding portdnses table in DB, all the same as done for
floatingips), however DB object is not used when listing ports.
The proposal would be to remove 'dns' field from FloatingIP OVO as not
used, until we find performance bottleneck.
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1850639/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1852207] [NEW] reschedule ignores requested availability zone
Public bug reported: This is only observed on stable/pike. Reproduce - * Create an aggregate mapped to an availability zone * Add one host to the aggregate * Make sure (e.g. with fault injection) that the nova-compute on the host in the aggregate will fail during the instance_claim process to trigger re-schedule * boot an instance requesting the az Expected Instance boot fails with NoValidHost Actual -- Instance is rescheduled and the scheduler ignores the requested az and therefore instance is booted outside of the requested az. ** Affects: nova Importance: Undecided Status: New ** Affects: nova/pike Importance: Medium Assignee: Balazs Gibizer (balazs-gibizer) Status: Triaged ** Tags: scheduler ** Also affects: nova/pike Importance: Undecided Status: New ** Changed in: nova/pike Assignee: (unassigned) => Balazs Gibizer (balazs-gibizer) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1852207 Title: reschedule ignores requested availability zone Status in OpenStack Compute (nova): New Status in OpenStack Compute (nova) pike series: Triaged Bug description: This is only observed on stable/pike. Reproduce - * Create an aggregate mapped to an availability zone * Add one host to the aggregate * Make sure (e.g. with fault injection) that the nova-compute on the host in the aggregate will fail during the instance_claim process to trigger re-schedule * boot an instance requesting the az Expected Instance boot fails with NoValidHost Actual -- Instance is rescheduled and the scheduler ignores the requested az and therefore instance is booted outside of the requested az. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1852207/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
