[Yahoo-eng-team] [Bug 2006949] Re: SLAAC and stateless DHCP are not working with stateless SG
It was fixed in OVN with https://github.com/ovn- org/ovn/commit/071cd7385f4aaf6e0e4635aa16a84e174b53d4ef ** Changed in: neutron Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2006949 Title: SLAAC and stateless DHCP are not working with stateless SG Status in neutron: Fix Released Bug description: Bug originally found by Alex Katz and reported in the bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2149731 Description of problem: When a stateless security group is attached to the instance it fails to get an IPv6 address using SLAAC or stateless DHCP. An explicit rule is required to allow ICMPv6 traffic. Checked with the custom security group (only egress traffic is allowed) as well as with the default security group (egress and ingress from the same SG are allowed). Version-Release number of selected component (if applicable): RHOS-17.1-RHEL-9-20221115.n.2 Red Hat Enterprise Linux release 9.1 (Plow) How reproducible: 100% Steps to Reproduce: openstack network create net_dual_slaac openstack subnet create --subnet-range 10.100.1.0/24 --network net_dual_slaac subnet_dual_slaac openstack subnet create --subnet-range 2001:0:0:1::0/64 --ip-version 6 --ipv6-ra-mode slaac --ipv6-address-mode slaac --network net_dual_slaac subnet_dual_slaac_ipv6 openstack router create router_test_boot EXT_NET=`openstack network list --external -f value -c Name` openstack router set --external-gateway $EXT_NET router_test_boot openstack router add subnet router_test_boot subnet_dual_slaac openstack security group create --stateless test_sg openstack server create --image --flavor --network net_dual_slaac --security-group test_sg vm_1 Actual results: only IPv4 address appear on the instance Expected results: IPv6 address is expected Additional info: can be worked around by adding icmpv6 rule: # openstack security group rule create --ingress --protocol icmpv6 test_sg To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2006949/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2045058] [NEW] [RFE] Use call_monitor_timeout of oslo.messaging RPCClient instead of custom backoff mechanism and hardcoded timeouts
Public bug reported: Currently, neutron RPC clients will repeat calls, timeout, back off, repeat again... this logic is implemented in neutron-lib RPCClient itself. This is done to handle requests that take a very long time. Instead of failing, then bumping timeout and hope that it's enough now (and leave the server unaware), we could instead enable active heartbeating with oslo.messaging call_monitor_timeout option. See nova did this for their clients: https://opendev.org/openstack/nova/commit/fe26a52024416ed2d37c2d5027da4b23231dc515 I believe this should replace backoff logic in neutron-lib. ** Affects: neutron Importance: Undecided Status: New ** Tags: rfe -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2045058 Title: [RFE] Use call_monitor_timeout of oslo.messaging RPCClient instead of custom backoff mechanism and hardcoded timeouts Status in neutron: New Bug description: Currently, neutron RPC clients will repeat calls, timeout, back off, repeat again... this logic is implemented in neutron-lib RPCClient itself. This is done to handle requests that take a very long time. Instead of failing, then bumping timeout and hope that it's enough now (and leave the server unaware), we could instead enable active heartbeating with oslo.messaging call_monitor_timeout option. See nova did this for their clients: https://opendev.org/openstack/nova/commit/fe26a52024416ed2d37c2d5027da4b23231dc515 I believe this should replace backoff logic in neutron-lib. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2045058/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2027156] [NEW] nova should use quota_details neutron api to validate available ports before VM creation
Public bug reported: Originally reported at https://bugzilla.redhat.com/show_bug.cgi?id=102 -- When nova creates a VM, it issues a request to fetch all ports for the project, which may take a significant time and even time out. The code in https://github.com/openstack/nova/blob/master/nova/network/neutron.py#L2654 suggests that nova fetches IDs of all project ports just to count the number of ports in db and compare it to quota (to pre-validate that there's enough free ports to allocate for the request). Instead of fetching all IDs via get_ports, nova should attempt to use quota_details API that gives ready access to info on available ports: https://docs.openstack.org/api-ref/network/v2/?expanded=list-quotas-for- projects-with-non-default-quota-values-detail,show-quota-details-for-a- tenant-detail#quotas-details-extension-quota-details Note that this is an API extension and was added later than the original `quota` API, so it may be that special care should be taken to accommodate to neutron-servers with no such extension. Also, the extension seems to be controlled by the cfg.CONF.QUOTAS.quota_driver config option: https://github.com/openstack/neutron/blob/6fa9142ced0eed8de9a89042747438b93de7202d/neutron/extensions/quotasv2_detail.py#L37 Which means that there may be an (upstream) configuration that would end up with quota_details API missing. So a fallback mechanism may be due. Side note: it may be argued that neutron should not expose this config option in the first place. This bz doesn't attempt to address this request. ** Affects: nova Importance: Undecided Status: New ** Tags: neutron ** Tags added: neutron -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/2027156 Title: nova should use quota_details neutron api to validate available ports before VM creation Status in OpenStack Compute (nova): New Bug description: Originally reported at https://bugzilla.redhat.com/show_bug.cgi?id=102 -- When nova creates a VM, it issues a request to fetch all ports for the project, which may take a significant time and even time out. The code in https://github.com/openstack/nova/blob/master/nova/network/neutron.py#L2654 suggests that nova fetches IDs of all project ports just to count the number of ports in db and compare it to quota (to pre-validate that there's enough free ports to allocate for the request). Instead of fetching all IDs via get_ports, nova should attempt to use quota_details API that gives ready access to info on available ports: https://docs.openstack.org/api-ref/network/v2/?expanded=list-quotas- for-projects-with-non-default-quota-values-detail,show-quota-details- for-a-tenant-detail#quotas-details-extension-quota-details Note that this is an API extension and was added later than the original `quota` API, so it may be that special care should be taken to accommodate to neutron-servers with no such extension. Also, the extension seems to be controlled by the cfg.CONF.QUOTAS.quota_driver config option: https://github.com/openstack/neutron/blob/6fa9142ced0eed8de9a89042747438b93de7202d/neutron/extensions/quotasv2_detail.py#L37 Which means that there may be an (upstream) configuration that would end up with quota_details API missing. So a fallback mechanism may be due. Side note: it may be argued that neutron should not expose this config option in the first place. This bz doesn't attempt to address this request. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/2027156/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2024621] [NEW] tempest: add scenario for reverse DNS resolution
Public bug reported: The existing scenario in tempest plugin - https://github.com/openstack/neutron-tempest- plugin/blob/6e4d04071bbe672839c791d0f60ebab038c9a3de/neutron_tempest_plugin/scenario/test_dns_integration.py - does validate that regular resolution (hostname to IP) works, but not vice versa (reverse DNS, using PTR records). This feature is supported by ml2/ovn and ml2/ovs and ideally should be validated end-to-end by the plugin. ** Affects: neutron Importance: Undecided Status: New ** Tags: dns tempest ** Tags added: tempest ** Tags added: dns -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2024621 Title: tempest: add scenario for reverse DNS resolution Status in neutron: New Bug description: The existing scenario in tempest plugin - https://github.com/openstack/neutron-tempest- plugin/blob/6e4d04071bbe672839c791d0f60ebab038c9a3de/neutron_tempest_plugin/scenario/test_dns_integration.py - does validate that regular resolution (hostname to IP) works, but not vice versa (reverse DNS, using PTR records). This feature is supported by ml2/ovn and ml2/ovs and ideally should be validated end-to-end by the plugin. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2024621/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2024502] [NEW] Tempest: add scenario to validate that stateless SG rules are working in presence of Load Balancer attached to the same network
Public bug reported: It was found that in case of ML2/OVN, stateless SG rules stop working when a Octavia Load Balancer is attached to the same network. This was addressed in OVN core project with: https://github.com/ovn- org/ovn/commit/a0f82efdd9dfd3ef2d9606c1890e353df1097a51 I think it may make sense to add a new integration scenario for this case in tempest plugin that would: - create stateless SG - define some rules - start a VM for the SG - check rules work as expected - define a Load Balancer for the network - check the SG rules still work as expected This is a corner case, but since we know it's problematic in some OVN branches and since ML2/OVN is the default implementation, - and since OVN core team is considering adjusting the ACL conntrack behavior in the near future that may affect stateless behavior - it may be wise to implement the scenario nevertheless. ** Affects: neutron Importance: Undecided Status: New ** Tags: ovn ovn-octavia-provider tempest ** Tags added: ovn ovn-octavia-provider tempest ** Bug watch added: Red Hat Bugzilla #2214303 https://bugzilla.redhat.com/show_bug.cgi?id=2214303 -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2024502 Title: Tempest: add scenario to validate that stateless SG rules are working in presence of Load Balancer attached to the same network Status in neutron: New Bug description: It was found that in case of ML2/OVN, stateless SG rules stop working when a Octavia Load Balancer is attached to the same network. This was addressed in OVN core project with: https://github.com/ovn- org/ovn/commit/a0f82efdd9dfd3ef2d9606c1890e353df1097a51 I think it may make sense to add a new integration scenario for this case in tempest plugin that would: - create stateless SG - define some rules - start a VM for the SG - check rules work as expected - define a Load Balancer for the network - check the SG rules still work as expected This is a corner case, but since we know it's problematic in some OVN branches and since ML2/OVN is the default implementation, - and since OVN core team is considering adjusting the ACL conntrack behavior in the near future that may affect stateless behavior - it may be wise to implement the scenario nevertheless. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2024502/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2011600] [NEW] functional test_get_datapath_id fails with neutron.common.utils.WaitTimeout: Timed out after 5 seconds
Public bug reported:
https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_ec1/876659/3/check/neutron-
functional-with-uwsgi/ec1de9e/testr_results.html
2023-03-09 21:37:05.333441 | controller | {3}
neutron.tests.functional.agent.test_ovs_lib.OVSBridgeTestCase.test_get_datapath_id
[5.136944s] ... FAILED
ft1.7:
neutron.tests.functional.agent.test_ovs_lib.OVSBridgeTestCase.test_get_datapath_idtesttools.testresult.real._StringException:
Traceback (most recent call last):
File "/home/zuul/src/opendev.org/openstack/neutron/neutron/common/utils.py",
line 745, in wait_until_true
eventlet.sleep(sleep)
File
"/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/eventlet/greenthread.py",
line 36, in sleep
hub.switch()
File
"/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/eventlet/hubs/hub.py",
line 313, in switch
return self.greenlet.switch()
eventlet.timeout.Timeout: 5 seconds
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py",
line 182, in func
return f(self, *args, **kwargs)
File
"/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/agent/test_ovs_lib.py",
line 228, in test_get_datapath_id
utils.wait_until_true(_check_datapath_id, timeout=5)
File "/home/zuul/src/opendev.org/openstack/neutron/neutron/common/utils.py",
line 750, in wait_until_true
raise WaitTimeout(_("Timed out after %d seconds") % timeout)
neutron.common.utils.WaitTimeout: Timed out after 5 seconds
** Affects: neutron
Importance: Undecided
Status: New
** Tags: functional-tests gate-failure ovs ovs-lib
** Tags added: functional-tests gate-failure ovs ovs-lib
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2011600
Title:
functional test_get_datapath_id fails with
neutron.common.utils.WaitTimeout: Timed out after 5 seconds
Status in neutron:
New
Bug description:
https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_ec1/876659/3/check/neutron-
functional-with-uwsgi/ec1de9e/testr_results.html
2023-03-09 21:37:05.333441 | controller | {3}
neutron.tests.functional.agent.test_ovs_lib.OVSBridgeTestCase.test_get_datapath_id
[5.136944s] ... FAILED
ft1.7:
neutron.tests.functional.agent.test_ovs_lib.OVSBridgeTestCase.test_get_datapath_idtesttools.testresult.real._StringException:
Traceback (most recent call last):
File
"/home/zuul/src/opendev.org/openstack/neutron/neutron/common/utils.py", line
745, in wait_until_true
eventlet.sleep(sleep)
File
"/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/eventlet/greenthread.py",
line 36, in sleep
hub.switch()
File
"/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/eventlet/hubs/hub.py",
line 313, in switch
return self.greenlet.switch()
eventlet.timeout.Timeout: 5 seconds
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py",
line 182, in func
return f(self, *args, **kwargs)
File
"/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/agent/test_ovs_lib.py",
line 228, in test_get_datapath_id
utils.wait_until_true(_check_datapath_id, timeout=5)
File
"/home/zuul/src/opendev.org/openstack/neutron/neutron/common/utils.py", line
750, in wait_until_true
raise WaitTimeout(_("Timed out after %d seconds") % timeout)
neutron.common.utils.WaitTimeout: Timed out after 5 seconds
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2011600/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2009053] [NEW] OVN: default stateless SG blocks metadata traffic
Public bug reported: Bug originally found by Alex Katz and reported in the bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2149713 Description of problem: When a stateless security group is attached to the instance it fails to fetch metadata info. An explicit rule is required to allow metadata traffic from 169.254.169.254. Checked with the custom security group (only egress traffic is allowed) as well as with the default security group (egress and ingress from the same SG are allowed). Version-Release number of selected component (if applicable): RHOS-17.1-RHEL-9-20221115.n.2 Red Hat Enterprise Linux release 9.1 (Plow) How reproducible: 100% Steps to Reproduce: openstack security group create --stateless test_sg openstack server create --image --flavor --network --security-group test_sg vm_1 Actual results: checking http://169.254.169.254/2009-04-04/instance-id failed 1/20: up 21.53. request failed failed 2/20: up 70.89. request failed failed 3/20: up 120.12. request failed failed 4/20: up 169.36. request failed failed 5/20: up 218.81. request failed failed 6/20: up 268.17. request failed Expected results: Metadata is successfully fetched ** Affects: neutron Importance: Undecided Assignee: Ihar Hrachyshka (ihar-hrachyshka) Status: Confirmed ** Tags: ovn sg-fw ** Changed in: neutron Status: New => Confirmed ** Changed in: neutron Assignee: (unassigned) => Ihar Hrachyshka (ihar-hrachyshka) ** Tags added: ovn sg-fw -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2009053 Title: OVN: default stateless SG blocks metadata traffic Status in neutron: Confirmed Bug description: Bug originally found by Alex Katz and reported in the bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2149713 Description of problem: When a stateless security group is attached to the instance it fails to fetch metadata info. An explicit rule is required to allow metadata traffic from 169.254.169.254. Checked with the custom security group (only egress traffic is allowed) as well as with the default security group (egress and ingress from the same SG are allowed). Version-Release number of selected component (if applicable): RHOS-17.1-RHEL-9-20221115.n.2 Red Hat Enterprise Linux release 9.1 (Plow) How reproducible: 100% Steps to Reproduce: openstack security group create --stateless test_sg openstack server create --image --flavor --network --security-group test_sg vm_1 Actual results: checking http://169.254.169.254/2009-04-04/instance-id failed 1/20: up 21.53. request failed failed 2/20: up 70.89. request failed failed 3/20: up 120.12. request failed failed 4/20: up 169.36. request failed failed 5/20: up 218.81. request failed failed 6/20: up 268.17. request failed Expected results: Metadata is successfully fetched To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2009053/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1997262] [NEW] networking-ovn-dsvm-functional-py27 job killed on timeout in stable/train branch
Public bug reported:
Examples:
https://zuul.opendev.org/t/openstack/build/a1d2c87b23e641539cff111abc8c5e4a/log/job-output.txt
https://5b948c35ca2c61dd95cb-b83277dfb85347aca874505fa2930911.ssl.cf2.rackcdn.com/862358/2/check/networking-ovn-dsvm-functional-py27/073519e/job-output.txt
The job takes more than 2h and is eventually killed.
Log example:
2022-11-20 01:04:17.245895 | primary | 2022-11-20 01:04:17.245 | {7}
networking_ovn.tests.functional.test_ovn_db_sync.TestOvnNbSync.test_ovn_nb_sync_off
[67.421953s] ... ok
2022-11-20 01:04:17.247309 | primary | 2022-11-20 01:04:17.246 | {0}
networking_ovn.tests.functional.test_router.TestRouter.test_gateway_chassis_with_bridge_mappings
[12.473030s] ... ok
2022-11-20 01:04:17.248841 | primary | 2022-11-20 01:04:17.248 | {2}
networking_ovn.tests.functional.test_router.TestRouter.test_router_port_ipv6_ra_configs_ipv4
[6.018013s] ... ok
2022-11-20 01:04:17.250444 | primary | 2022-11-20 01:04:17.250 | {0}
networking_ovn.tests.functional.test_router.TestRouter.test_router_port_ipv6_ra_configs_addr_mode_slaac
[5.654530s] ... ok
2022-11-20 01:04:17.252024 | primary | 2022-11-20 01:04:17.251 | {2}
networking_ovn.tests.functional.test_trunk_driver.TestOVNTrunkDriver.test_subport_add
[4.991551s] ... ok
2022-11-20 01:04:17.255232 | primary | 2022-11-20 01:04:17.254 | {0}
networking_ovn.tests.functional.test_trunk_driver.TestOVNTrunkDriver.test_trunk_create_with_subports
[4.687337s] ... ok
2022-11-20 01:04:17.256586 | primary | 2022-11-20 01:04:17.256 | {6}
networking_ovn.tests.functional.test_ovn_db_sync.TestOvnNbSync.test_ovn_nb_sync_repair
[73.895267s] ... ok
2022-11-20 01:04:17.257905 | primary | 2022-11-20 01:04:17.257 | {4}
networking_ovn.tests.functional.test_ovn_db_sync.TestOvnNbSyncOverSsl.test_ovn_nb_sync_off
[64.915188s] ... ok
2022-11-20 01:04:17.263929 | primary | 2022-11-20 01:04:17.259 | {6}
networking_ovn.tests.functional.test_ovsdb_monitor.TestNBDbMonitor.test_distributed_lock
[3.913342s] ... ok
2022-11-20 01:04:17.265641 | primary | 2022-11-20 01:04:17.265 | {6}
networking_ovn.tests.functional.test_ovsdb_monitor.TestNBDbMonitor.test_floatingip_mac_bindings
[11.765796s] ... ok
2022-11-20 01:04:17.267119 | primary | 2022-11-20 01:04:17.266 | {6}
networking_ovn.tests.functional.test_ovsdb_monitor.TestNBDbMonitorOverSsl.test_port_up_down_events
[6.316748s] ... ok
2022-11-20 01:04:17.271742 | primary | 2022-11-20 01:04:17.271 | {6}
networking_ovn.tests.functional.test_qos_driver.TestOVNQosDriver.test_port_qos_options_with_rule
[3.441892s] ... ok
2022-11-20 02:51:16.344359 | primary |
/home/zuul/workspace/devstack-gate/functions.sh: line 1009: 26167 Killed
timeout -s 9 ${REMAINING_TIME}m bash -c "source
$WORKSPACE/devstack-gate/functions.sh && $cmd"
Observation:
both failed runs were executing the following test cases just before the
test runner froze:
2022-11-20 01:04:17.256586 | primary | 2022-11-20 01:04:17.256 | {6}
networking_ovn.tests.functional.test_ovn_db_sync.TestOvnNbSync.test_ovn_nb_sync_repair
[73.895267s] ... ok
2022-11-20 01:04:17.257905 | primary | 2022-11-20 01:04:17.257 | {4}
networking_ovn.tests.functional.test_ovn_db_sync.TestOvnNbSyncOverSsl.test_ovn_nb_sync_off
[64.915188s] ... ok
(I don't know if this is related though)
** Affects: neutron
Importance: Undecided
Status: New
** Tags: functional-tests gate-failure ovn
** Tags added: ovn
** Tags added: gate-failure
** Tags added: functional-tests
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1997262
Title:
networking-ovn-dsvm-functional-py27 job killed on timeout in
stable/train branch
Status in neutron:
New
Bug description:
Examples:
https://zuul.opendev.org/t/openstack/build/a1d2c87b23e641539cff111abc8c5e4a/log/job-output.txt
https://5b948c35ca2c61dd95cb-b83277dfb85347aca874505fa2930911.ssl.cf2.rackcdn.com/862358/2/check/networking-ovn-dsvm-functional-py27/073519e/job-output.txt
The job takes more than 2h and is eventually killed.
Log example:
2022-11-20 01:04:17.245895 | primary | 2022-11-20 01:04:17.245 | {7}
networking_ovn.tests.functional.test_ovn_db_sync.TestOvnNbSync.test_ovn_nb_sync_off
[67.421953s] ... ok
2022-11-20 01:04:17.247309 | primary | 2022-11-20 01:04:17.246 | {0}
networking_ovn.tests.functional.test_router.TestRouter.test_gateway_chassis_with_bridge_mappings
[12.473030s] ... ok
2022-11-20 01:04:17.248841 | primary | 2022-11-20 01:04:17.248 | {2}
networking_ovn.tests.functional.test_router.TestRouter.test_router_port_ipv6_ra_configs_ipv4
[6.018013s] ... ok
2022-11-20 01:04:17.250444 | primary | 2022-11-20 01:04:17.250 | {0}
networking_ovn.tests.functional.test_router.TestRouter.test_router_port_ipv6_ra_configs_addr_mode_slaac
[5.654530s] ... ok
2022-11-20 01:04:17.252024 | primary | 2022-11-20 01:04:17.251 | {2}
[Yahoo-eng-team] [Bug 1997092] [NEW] Metadata service broken after minor neutron update when OVN 21.09+ is used
Public bug reported: Originally reported at: https://bugzilla.redhat.com/show_bug.cgi?id=2093901 Prerequisites: 1. OVN 21.09+ that includes https://github.com/ovn-org/ovn/commit/3ae8470edc648b7401433a22a9f15053cc7e666d 2. Existing metadata namespace created by OVN agent before commit https://review.opendev.org/c/openstack/neutron/+/768462 Steps to reproduce: 1. Neutron OVN metadata agent updated to include the patch from prereq (2). 2. Neutron OVN metadata agent is restarted. It will create a new network namespace to host the metadata vif. It will also remove the old vif. 3. curl http://169.254.169.254/latest/meta-data/ from a VM that is hosted on the same node. It fails. This happens because the agent first creates new vif, then deletes the old vif. Which puts OVN into a situation where 2 interfaces exist in parallel assigned to the same LSP. This scenario is considered invalid by OVN core team. There's a patch up for review for OVN core to handle the situation more gracefully: https://patchwork.ozlabs.org/project/ovn/patch/20221114092437.2807815-1-xsimo...@redhat.com/ This patch will not leave metadata service broken, but it will trigger full recompute in OVN. So we should not rely on its mechanics. Instead Neutron should make sure that no two vifs carry the same iface-id at the same time. The reason why this was not a problem with OVN 21.06 or earlier is because the patch referred in prereq (1) changed the behavior in this invalid / undefined scenario. ** Affects: neutron Importance: Undecided Assignee: Ihar Hrachyshka (ihar-hrachyshka) Status: In Progress -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1997092 Title: Metadata service broken after minor neutron update when OVN 21.09+ is used Status in neutron: In Progress Bug description: Originally reported at: https://bugzilla.redhat.com/show_bug.cgi?id=2093901 Prerequisites: 1. OVN 21.09+ that includes https://github.com/ovn-org/ovn/commit/3ae8470edc648b7401433a22a9f15053cc7e666d 2. Existing metadata namespace created by OVN agent before commit https://review.opendev.org/c/openstack/neutron/+/768462 Steps to reproduce: 1. Neutron OVN metadata agent updated to include the patch from prereq (2). 2. Neutron OVN metadata agent is restarted. It will create a new network namespace to host the metadata vif. It will also remove the old vif. 3. curl http://169.254.169.254/latest/meta-data/ from a VM that is hosted on the same node. It fails. This happens because the agent first creates new vif, then deletes the old vif. Which puts OVN into a situation where 2 interfaces exist in parallel assigned to the same LSP. This scenario is considered invalid by OVN core team. There's a patch up for review for OVN core to handle the situation more gracefully: https://patchwork.ozlabs.org/project/ovn/patch/20221114092437.2807815-1-xsimo...@redhat.com/ This patch will not leave metadata service broken, but it will trigger full recompute in OVN. So we should not rely on its mechanics. Instead Neutron should make sure that no two vifs carry the same iface-id at the same time. The reason why this was not a problem with OVN 21.06 or earlier is because the patch referred in prereq (1) changed the behavior in this invalid / undefined scenario. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1997092/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1952653] [NEW] OVN: stateless field update not reflected in OVN flows
Public bug reported:
Originally reported at:
https://bugzilla.redhat.com/show_bug.cgi?id=2017048
When we switch back and forth between stateful to stateless security
group, the ACL rules in OVN are not updated.
The new ACL rules are created according to the SG mode though.
Steps to Reproduce:
1. The pre created SG is stateful
$ openstack security group show default -c stateful -f value
True
In a controller, we can list the ACLs as "allow-related"
[heat-admin@overcloud-controller-0 ~]$ sudo podman exec -ti ovn_controller
ovn-nbctl list ACL
_uuid : e31f56a6-a373-4f0f-b690-dafd7b964d99
action : allow-related
direction : to-lport
external_ids:
{"neutron:security_group_rule_id"="459deec0-a4b2-4c5a-af5d-fe6c66346fef"}
log : false
match : "outport == @pg_26f1270b_efaa_4948_826d_c227e7808ca5 &&
ip4 && ip4.src == 0.0.0.0/0 && udp"
meter : []
name: []
priority: 1002
severity: []
2. Switch to stateless SG:
$ openstack security group set --stateless default
Yet, the "list ACL" command shows that the rule still uses conntrack
(allow-related).
3. We create a new rule:
openstack security group rule create default --protocol tcp --dst-port 22:22
--remote-ip 0.0.0.0/0
Actual results:
The rules are a mix of allow-related and allow-stateless:
[heat-admin@overcloud-controller-0 ~]$ sudo podman exec -ti
ovn_controller ovn-nbctl list ACL
_uuid : fb970be6-493c-424a-b133-66dcbe3aedee
action : allow-stateless
direction : to-lport
external_ids:
{"neutron:security_group_rule_id"="4ef8bc9c-7450-43c5-9878-c01037b72240"}
log : false
match : "outport == @pg_26f1270b_efaa_4948_826d_c227e7808ca5 &&
ip4 && ip4.src == 0.0.0.0/0 && tcp && tcp.dst == 22"
meter : []
name: []
priority: 1002
severity: []
_uuid : 18cc8207-6489-49d9-bb57-31ed04b04726
action : allow-related
direction : to-lport
external_ids:
{"neutron:security_group_rule_id"="2698c7f8-176e-4bfb-b6c6-7314272e6dd8"}
log : false
match : "outport == @pg_26f1270b_efaa_4948_826d_c227e7808ca5 &&
ip4 && ip4.src == 0.0.0.0/0 && tcp"
meter : []
name: []
priority: 1002
severity: []
Expected results:
All rules should be "allow-stateless" or "allow-related" when we update the
security group to stateless or stateful.
** Affects: neutron
Importance: Undecided
Assignee: Ihar Hrachyshka (ihar-hrachyshka)
Status: In Progress
** Tags: ovn sg-fw
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1952653
Title:
OVN: stateless field update not reflected in OVN flows
Status in neutron:
In Progress
Bug description:
Originally reported at:
https://bugzilla.redhat.com/show_bug.cgi?id=2017048
When we switch back and forth between stateful to stateless security
group, the ACL rules in OVN are not updated.
The new ACL rules are created according to the SG mode though.
Steps to Reproduce:
1. The pre created SG is stateful
$ openstack security group show default -c stateful -f value
True
In a controller, we can list the ACLs as "allow-related"
[heat-admin@overcloud-controller-0 ~]$ sudo podman exec -ti ovn_controller
ovn-nbctl list ACL
_uuid : e31f56a6-a373-4f0f-b690-dafd7b964d99
action : allow-related
direction : to-lport
external_ids:
{"neutron:security_group_rule_id"="459deec0-a4b2-4c5a-af5d-fe6c66346fef"}
log : false
match : "outport == @pg_26f1270b_efaa_4948_826d_c227e7808ca5 &&
ip4 && ip4.src == 0.0.0.0/0 && udp"
meter : []
name: []
priority: 1002
severity: []
2. Switch to stateless SG:
$ openstack security group set --stateless default
Yet, the "list ACL" command shows that the rule still uses conntrack
(allow-related).
3. We create a new rule:
openstack security group rule create default --protocol tcp --dst-port 22:22
--remote-ip 0.0.0.0/0
Actual results:
The rules are a mix of allow-related and allow-stateless:
[heat-admin@overcloud-controller-0 ~]$ sudo podman exec -ti
ovn_controller ovn-nbctl list ACL
_uuid : fb970be6-493c-424a-b133-66dcbe3aedee
action : allow-stateless
direction : to-lport
external_i
[Yahoo-eng-team] [Bug 1946251] [NEW] API: allow to disable anti-spoofing but not SGs
Public bug reported: Right now, port security API - seems to [1] - disable both ACL filtering (SGs) and anti-spoofing (allowed address pairs logic). An argument may be made to allow to disable anti-spoofing but still implement ACL filtering on a port. (This actually happened in one of synthetic NFV test environments in-house.) In this case, the user story would look like as follows: 0. A user creates a SG with TCP blocked. 1. A user creates a port using this SG. 2. A user uses a new API to mark the port to allow MAC spoofing. 3. A user sends TCP traffic using a different MAC through the port and sees it blocked. 4. A user sends UDP traffic using a different MAC through the port and see it's not blocked. Allowed-address-pairs API allows to specify masks for IP addresses, effectively allowing to match against ANY IP address using /0 mask. But MAC address part of the API doesn't support masks or other ways to list groups of addresses. Perhaps the feature request may be fulfilled by extending the API to allow a way to list groups of MAC addresses in anti-spoofing mechanism (either via a hardcoded special value like "ANY" or via a mask). This doesn't necessarily mean it's the optimal way to do it, throwing it here just as an idea to explore. [1] https://bugs.launchpad.net/neutron/+bug/1946250 ** Affects: neutron Importance: Undecided Status: New ** Tags: api rfe sg-fw ** Description changed: Right now, port security API - seems to [1] - disable both ACL filtering (SGs) and anti-spoofing (allowed address pairs logic). An argument may be made to allow to disable anti-spoofing but still implement ACL filtering on a port. (This actually happened in one of synthetic NFV test environments in-house.) In this case, the user story would look like as follows: 0. A user creates a SG with TCP blocked. 1. A user creates a port using this SG. 2. A user uses a new API to mark the port to allow MAC spoofing. - 3. A user sends TCP traffic through the port and sees it blocked. - 4. A user sends UDP traffic through the port and see it's not blocked. + 3. A user sends TCP traffic using a different MAC through the port and sees it blocked. + 4. A user sends UDP traffic using a different MAC through the port and see it's not blocked. Allowed-address-pairs API allows to specify masks for IP addresses, effectively allowing to match against ANY IP address using /0 mask. But MAC address part of the API doesn't support masks or other ways to list groups of addresses. Perhaps the feature request may be fulfilled by extending the API to allow a way to list groups of MAC addresses in anti-spoofing mechanism (either via a hardcoded special value like "ANY" or via a mask). This doesn't necessarily mean it's the optimal way to do it, throwing it here just as an idea to explore. [1] https://bugs.launchpad.net/neutron/+bug/1946250 -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1946251 Title: API: allow to disable anti-spoofing but not SGs Status in neutron: New Bug description: Right now, port security API - seems to [1] - disable both ACL filtering (SGs) and anti-spoofing (allowed address pairs logic). An argument may be made to allow to disable anti-spoofing but still implement ACL filtering on a port. (This actually happened in one of synthetic NFV test environments in-house.) In this case, the user story would look like as follows: 0. A user creates a SG with TCP blocked. 1. A user creates a port using this SG. 2. A user uses a new API to mark the port to allow MAC spoofing. 3. A user sends TCP traffic using a different MAC through the port and sees it blocked. 4. A user sends UDP traffic using a different MAC through the port and see it's not blocked. Allowed-address-pairs API allows to specify masks for IP addresses, effectively allowing to match against ANY IP address using /0 mask. But MAC address part of the API doesn't support masks or other ways to list groups of addresses. Perhaps the feature request may be fulfilled by extending the API to allow a way to list groups of MAC addresses in anti-spoofing mechanism (either via a hardcoded special value like "ANY" or via a mask). This doesn't necessarily mean it's the optimal way to do it, throwing it here just as an idea to explore. [1] https://bugs.launchpad.net/neutron/+bug/1946250 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1946251/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1946250] [NEW] Neutron API reference should explain the intended behavior of port security extension
Public bug reported: https://docs.openstack.org/api-ref/network/v2/#port-security The explanation as of the time of writing is as follows: "The port-security extension adds the port_security_enabled boolean attribute to networks. At the network level, port_security_enabled defines the default value for new ports attached to the network; they will inherit the value of their network’s port_security_enabled unless explicitly set on the port itself. While the default value for port_security_enabled is true, this can be changed by updating the respective network. Note that changing a value of port_security_enabled on a network, does not cascade the value to ports attached to the network." It explains how the attribute behaves and how it's inherited by ports, but there is no explanation of what the attribute DOES. Does it disable anti-spoofing? Or SGs? Or both? Is the fact that - traditionally - port_security_enabled=false disables both the intent of the API, or it's just a historical fact on how drivers - traditionally - implement the API? Same problem as to how port level extension is explained: https://docs.openstack.org/api-ref/network/v2/#id53 "The port-security extension adds the port_security_enabled boolean attribute to ports. If a port-security value is not specified during port creation, a port will inherit the port_security_enabled from the network its connected to." ** Affects: neutron Importance: Undecided Status: New ** Tags: api-ref -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1946250 Title: Neutron API reference should explain the intended behavior of port security extension Status in neutron: New Bug description: https://docs.openstack.org/api-ref/network/v2/#port-security The explanation as of the time of writing is as follows: "The port-security extension adds the port_security_enabled boolean attribute to networks. At the network level, port_security_enabled defines the default value for new ports attached to the network; they will inherit the value of their network’s port_security_enabled unless explicitly set on the port itself. While the default value for port_security_enabled is true, this can be changed by updating the respective network. Note that changing a value of port_security_enabled on a network, does not cascade the value to ports attached to the network." It explains how the attribute behaves and how it's inherited by ports, but there is no explanation of what the attribute DOES. Does it disable anti-spoofing? Or SGs? Or both? Is the fact that - traditionally - port_security_enabled=false disables both the intent of the API, or it's just a historical fact on how drivers - traditionally - implement the API? Same problem as to how port level extension is explained: https://docs.openstack.org/api-ref/network/v2/#id53 "The port-security extension adds the port_security_enabled boolean attribute to ports. If a port-security value is not specified during port creation, a port will inherit the port_security_enabled from the network its connected to." To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1946250/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1744412] Re: Losing access to instances via floating IPs
I don't believe we are going to change any defaults for oslo.db / oslo.service options in upstream, so closing it as Won't Fix. Also, it's not clear if there is any issue in fresh versions of neutron. ** Changed in: neutron Status: Confirmed => Won't Fix -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1744412 Title: Losing access to instances via floating IPs Status in neutron: Won't Fix Bug description: Description of problem: Neutron Floating IPs stop working, instances become unreachable. Version-Release number of selected component (if applicable): Ocata. Neutron-related RPMs: puppet-neutron-10.3.2-0.20180103174737.2e7d298.el7.centos.noarch openstack-neutron-common-10.0.5-0.20180105192920.295c700.el7.centos.noarch openvswitch-ovn-common-2.6.1-10.1.git20161206.el7.x86_64 openstack-neutron-sriov-nic-agent-10.0.5-0.20180105192920.295c700.el7.centos.noarch python-neutron-lib-1.1.0-1.el7.noarch openvswitch-2.6.1-10.1.git20161206.el7.x86_64 python2-neutronclient-6.1.1-1.el7.noarch python-openvswitch-2.6.1-10.1.git20161206.el7.noarch openstack-neutron-lbaas-10.0.2-0.20180104200311.10771af.el7.centos.noarch openvswitch-ovn-host-2.6.1-10.1.git20161206.el7.x86_64 python-neutron-10.0.5-0.20180105192920.295c700.el7.centos.noarch openvswitch-ovn-central-2.6.1-10.1.git20161206.el7.x86_64 openstack-neutron-metering-agent-10.0.5-0.20180105192920.295c700.el7.centos.noarch python-neutron-lbaas-10.0.2-0.20180104200311.10771af.el7.centos.noarch openstack-neutron-openvswitch-10.0.5-0.20180105192920.295c700.el7.centos.noarch openstack-neutron-ml2-10.0.5-0.20180105192920.295c700.el7.centos.noarch openstack-neutron-10.0.5-0.20180105192920.295c700.el7.centos.noarch How reproducible: Not sure. We have noticed over the past day or so several users complaining about unreachable instances. Not all VMs have this issue and it is not clear how connectivity was lost in the first place. Actual results: In some cases, router is active on more than one controller, or router looks in the correct configuration but the qg- interface isn't NAT-ing the traffic to the qr-xxx interface. Iptables look correct. Expected results: VMs reachable via FIP. Additional info: Some ports appear to be stuck in 'BUILD' status, but not sure what is causing it. See this error in the logs: 2018-01-19 18:36:03.930 74015 ERROR neutron.agent.l3.router_info Traceback (most recent call last): 2018-01-19 18:36:03.930 74015 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 256, in call 2018-01-19 18:36:03.930 74015 ERROR neutron.agent.l3.router_info return func(*args, **kwargs) 2018-01-19 18:36:03.930 74015 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 1116, in process 2018-01-19 18:36:03.930 74015 ERROR neutron.agent.l3.router_info self.process_external() 2018-01-19 18:36:03.930 74015 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 910, in process_external 2018-01-19 18:36:03.930 74015 ERROR neutron.agent.l3.router_info self.update_fip_statuses(fip_statuses) 2018-01-19 18:36:03.930 74015 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 926, in update_fip_statuses 2018-01-19 18:36:03.930 74015 ERROR neutron.agent.l3.router_info self.agent.context, self.router_id, fip_statuses) 2018-01-19 18:36:03.930 74015 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 125, in update_floatingip_statuses 2018-01-19 18:36:03.930 74015 ERROR neutron.agent.l3.router_info router_id=router_id, fip_statuses=fip_statuses) 2018-01-19 18:36:03.930 74015 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/common/rpc.py", line 151, in call 2018-01-19 18:36:03.930 74015 ERROR neutron.agent.l3.router_info return self._original_context.call(ctxt, method, **kwargs) 2018-01-19 18:36:03.930 74015 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call 2018-01-19 18:36:03.930 74015 ERROR neutron.agent.l3.router_info retry=self.retry) 2018-01-19 18:36:03.930 74015 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 97, in _send 2018-01-19 18:36:03.930 74015 ERROR neutron.agent.l3.router_info timeout=timeout, retry=retry) 2018-01-19 18:36:03.930 74015 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 566, in send 2018-01-19 18:36:03.930 74015 ERROR
[Yahoo-eng-team] [Bug 1753209] Re: neutron_tempest_plugin.api.admin.test_shared_network_extension.RBACSharedNetworksTest, rbac policy in use across tenants.
@Akihiro, probably yes. Probably will require a separate bug report.
** Changed in: neutron
Status: New => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1753209
Title:
neutron_tempest_plugin.api.admin.test_shared_network_extension.RBACSharedNetworksTest,
rbac policy in use across tenants.
Status in neutron:
Fix Released
Status in tempest:
New
Status in tripleo:
In Progress
Bug description:
neutron_tempest_plugin.api.admin.test_shared_network_extension.RBACSharedNetworksTest
failure
https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-
centos-7-ovb-1ctlr_1comp-featureset020-master/6cec620/tempest.html.gz
Details: {u'message': u'RBAC policy on object 3cfbd0a7-84f2-4e3f-917e-
bf51b5995e20 cannot be removed because other objects depend on
it.\nDetails: Callback
neutron.plugins.ml2.plugin.Ml2Plugin.validate_network_rbac_policy_change
--9223372036850840529 failed with "Unable to reconfigure sharing
settings for network 3cfbd0a7-84f2-4e3f-917e-bf51b5995e20. Multiple
tenants are using it.",Callback
neutron.services.network_ip_availability.plugin.NetworkIPAvailabilityPlugin.validate_network_rbac_policy_change
--9223372036853400817 failed with "Unable to reconfigure sharing
settings for network 3cfbd0a7-84f2-4e3f-917e-bf51b5995e20. Multiple
tenants are using it.",Callback
neutron.services.network_ip_availability.plugin.NetworkIPAvailabilityPlugin.validate_network_rbac_policy_change
--9223372036853463713 failed with "Unable to reconfigure sharing
settings for network 3cfbd0a7-84f2-4e3f-917e-bf51b5995e20. Multiple
tenants are using it."', u'type': u'RbacPolicyInUse', u'detail': u''}
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1753209/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1713163] Re: test_delete_saving_image fails because image hasn't transitioned to SAVING
As far as I understand, this test shouldn't even exist in scope of tempest, because you can't guarantee that backend will be slow to process the request. This kind of interaction would be better served by a test with better execution flow control. I am not sure if glance have those, but in neutron realm it would be a functional test with backend persistence artificially slowed down. ** Also affects: glance Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to Glance. https://bugs.launchpad.net/bugs/1713163 Title: test_delete_saving_image fails because image hasn't transitioned to SAVING Status in Glance: New Status in tempest: Confirmed Bug description: http://logs.openstack.org/33/440933/4/check/gate-tempest-dsvm-neutron- linuxbridge-ubuntu-xenial/ebfa453/logs/testr_results.html.gz Traceback (most recent call last): File "tempest/api/compute/images/test_images.py", line 48, in test_delete_saving_image wait_until='SAVING') File "tempest/api/compute/base.py", line 301, in create_image_from_server image_id, wait_until) File "tempest/common/waiters.py", line 179, in wait_for_image_status raise lib_exc.TimeoutException(message) tempest.lib.exceptions.TimeoutException: Request timed out Details: (ImagesTestJSON:test_delete_saving_image) Image 93090479-409d-49f9-b2d3-2bc11a40f967 failed to reach SAVING state (current state ACTIVE) within the required time (196 s). I briefly looked at the test case, and it seems to me that it may happen when the image snapshot is produced quicker than tempest being able to catch the intermittent transition. If so, the test case may need a refinement to check that a snapshot was produced and not that the state was in SAVING for some time. To manage notifications about this bug go to: https://bugs.launchpad.net/glance/+bug/1713163/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1759345] [NEW] -with-oslo-master job fails with DuplicateOptError: duplicate option: polling_interval
Public bug reported: http://logs.openstack.org/periodic/git.openstack.org/openstack/neutron/master /openstack-tox-py27-with-oslo-master/f65fec8/job-output.txt.gz 2018-03-27 06:16:32.984356 | ubuntu-xenial | --- import errors --- 2018-03-27 06:16:32.984599 | ubuntu-xenial | Failed to import test module: neutron.tests.unit.plugins.ml2.drivers.mech_sriov.agent.common.test_config 2018-03-27 06:16:32.984708 | ubuntu-xenial | Traceback (most recent call last): 2018-03-27 06:16:32.985036 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/neutron/.tox/py27/local/lib/python2.7/site-packages/unittest2/loader.py", line 456, in _find_test_path 2018-03-27 06:16:32.985174 | ubuntu-xenial | module = self._get_module_from_name(name) 2018-03-27 06:16:32.985496 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/neutron/.tox/py27/local/lib/python2.7/site-packages/unittest2/loader.py", line 395, in _get_module_from_name 2018-03-27 06:16:32.985563 | ubuntu-xenial | __import__(name) 2018-03-27 06:16:32.985804 | ubuntu-xenial | File "neutron/tests/unit/plugins/ml2/drivers/mech_sriov/agent/common/test_config.py", line 21, in 2018-03-27 06:16:32.985993 | ubuntu-xenial | from neutron.plugins.ml2.drivers.mech_sriov.agent.common import config 2018-03-27 06:16:32.986193 | ubuntu-xenial | File "neutron/plugins/ml2/drivers/mech_sriov/agent/common/config.py", line 57, in 2018-03-27 06:16:32.986351 | ubuntu-xenial | agent_common_config.register_agent_sriov_nic_opts() 2018-03-27 06:16:32.986588 | ubuntu-xenial | File "neutron/conf/plugins/ml2/drivers/mech_sriov/agent_common.py", line 56, in register_agent_sriov_nic_opts 2018-03-27 06:16:32.986717 | ubuntu-xenial | cfg.register_opts(agent_opts, 'AGENT') 2018-03-27 06:16:32.987017 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/neutron/.tox/py27/local/lib/python2.7/site-packages/oslo_config/cfg.py", line 2416, in __inner 2018-03-27 06:16:32.987129 | ubuntu-xenial | result = f(self, *args, **kwargs) 2018-03-27 06:16:32.987458 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/neutron/.tox/py27/local/lib/python2.7/site-packages/oslo_config/cfg.py", line 2614, in register_opts 2018-03-27 06:16:32.987604 | ubuntu-xenial | self.register_opt(opt, group, clear_cache=False) 2018-03-27 06:16:32.987969 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/neutron/.tox/py27/local/lib/python2.7/site-packages/oslo_config/cfg.py", line 2420, in __inner 2018-03-27 06:16:32.988101 | ubuntu-xenial | return f(self, *args, **kwargs) 2018-03-27 06:16:32.988437 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/neutron/.tox/py27/local/lib/python2.7/site-packages/oslo_config/cfg.py", line 2590, in register_opt 2018-03-27 06:16:32.988546 | ubuntu-xenial | return group._register_opt(opt, cli) 2018-03-27 06:16:32.988847 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/neutron/.tox/py27/local/lib/python2.7/site-packages/oslo_config/cfg.py", line 1886, in _register_opt 2018-03-27 06:16:32.988952 | ubuntu-xenial | if _is_opt_registered(self._opts, opt): 2018-03-27 06:16:32.989257 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/neutron/.tox/py27/local/lib/python2.7/site-packages/oslo_config/cfg.py", line 801, in _is_opt_registered 2018-03-27 06:16:32.989352 | ubuntu-xenial | raise DuplicateOptError(opt.name) 2018-03-27 06:16:32.989475 | ubuntu-xenial | DuplicateOptError: duplicate option: polling_interval It happens because test runner imports two different modules with the same option defined. This is a problem now that oslo.config tracks location of option definitions and consider it a conflict if the same option is declared in different locations. (The trigger is https://review.openstack.org/#/c/537399/) ** Affects: neutron Importance: Medium Assignee: Ihar Hrachyshka (ihar-hrachyshka) Status: In Progress ** Tags: gate-failure sriov-pci-pt unittest ** Changed in: neutron Assignee: (unassigned) => Ihar Hrachyshka (ihar-hrachyshka) ** Changed in: neutron Importance: Undecided => Medium ** Changed in: neutron Status: New => In Progress ** Tags added: gate-failure sriov-pci-pt unittest -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1759345 Title: -with-oslo-master job fails with DuplicateOptError: duplicate option: polling_interval Status in neutron: In Progress Bug description: http://logs.openstack.org/periodic/git.openstack.org/openstack/neutron/master /openstack-tox-py27-with-oslo-master/f65fec8/job-output.txt.gz 2018-03-27 06:16:32.984356 | ubuntu-xenial | --- import
[Yahoo-eng-team] [Bug 1757087] Re: neutron.tests.unit.objects.qos.test_policy unittests fail on stable/pike branch
** Changed in: neutron Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1757087 Title: neutron.tests.unit.objects.qos.test_policy unittests fail on stable/pike branch Status in neutron: Fix Released Bug description: http://logs.openstack.org/13/510013/2/check/openstack-tox- py27/7d3e9b9/testr_results.html.gz To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1757087/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1750334] Re: ovsdb commands timeouts cause fullstack tests failures
*** This bug is a duplicate of bug 1741889 *** https://bugs.launchpad.net/bugs/1741889 ** This bug has been marked a duplicate of bug 1741889 functional: DbAddCommand sometimes times out after 10 seconds -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1750334 Title: ovsdb commands timeouts cause fullstack tests failures Status in neutron: Confirmed Bug description: Quite often tests like neutron.tests.fullstack.test_connectivity.TestUninterruptedConnectivityOnL2AgentRestart.test_l2_agent_restart are failing because of timeout while executing ovsdb commands. Because of this issue test environment is not prepared and test fails. Example issues: - neutron.tests.fullstack.test_connectivity.TestUninterruptedConnectivityOnL2AgentRestart.test_l2_agent_restart(OVS,VLANs,openflow-native): http://logs.openstack.org/81/545681/1/check/neutron-fullstack/8285bf3/logs/testr_results.html.gz - neutron.tests.fullstack.test_connectivity.TestUninterruptedConnectivityOnL2AgentRestart.test_l2_agent_restart(OVS,Flat network,openflow-native): http://logs.openstack.org/79/545679/1/check /neutron-fullstack/3a12865/logs/testr_results.html.gz To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1750334/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1664347] Re: test_volume_boot_pattern failed to get an instance into ACTIVE state
** No longer affects: neutron
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1664347
Title:
test_volume_boot_pattern failed to get an instance into ACTIVE state
Status in OpenStack Compute (nova):
Confirmed
Bug description:
http://logs.openstack.org/53/431753/2/check/gate-grenade-dsvm-neutron-
dvr-multinode-ubuntu-
xenial/8cf8895/logs/grenade.sh.txt.gz#_2017-02-11_01_24_50_334
2017-02-11 01:24:50.334 | Captured traceback:
2017-02-11 01:24:50.334 | ~~~
2017-02-11 01:24:50.334 | Traceback (most recent call last):
2017-02-11 01:24:50.334 | File "tempest/test.py", line 99, in wrapper
2017-02-11 01:24:50.334 | return f(self, *func_args, **func_kwargs)
2017-02-11 01:24:50.334 | File
"tempest/scenario/test_volume_boot_pattern.py", line 168, in
test_volume_boot_pattern
2017-02-11 01:24:50.334 | security_group=security_group))
2017-02-11 01:24:50.334 | File
"tempest/scenario/test_volume_boot_pattern.py", line 72, in
_boot_instance_from_resource
2017-02-11 01:24:50.334 | return self.create_server(image_id='',
**create_kwargs)
2017-02-11 01:24:50.334 | File "tempest/scenario/manager.py", line 208,
in create_server
2017-02-11 01:24:50.334 | image_id=image_id, **kwargs)
2017-02-11 01:24:50.334 | File "tempest/common/compute.py", line 182,
in create_test_server
2017-02-11 01:24:50.334 | server['id'])
2017-02-11 01:24:50.334 | File
"/opt/stack/old/tempest/.tox/tempest/local/lib/python2.7/site-packages/oslo_utils/excutils.py",
line 220, in __exit__
2017-02-11 01:24:50.334 | self.force_reraise()
2017-02-11 01:24:50.334 | File
"/opt/stack/old/tempest/.tox/tempest/local/lib/python2.7/site-packages/oslo_utils/excutils.py",
line 196, in force_reraise
2017-02-11 01:24:50.334 | six.reraise(self.type_, self.value, self.tb)
2017-02-11 01:24:50.334 | File "tempest/common/compute.py", line 164,
in create_test_server
2017-02-11 01:24:50.334 | clients.servers_client, server['id'],
wait_until)
2017-02-11 01:24:50.334 | File "tempest/common/waiters.py", line 96, in
wait_for_server_status
2017-02-11 01:24:50.334 | raise lib_exc.TimeoutException(message)
2017-02-11 01:24:50.334 | tempest.lib.exceptions.TimeoutException:
Request timed out
2017-02-11 01:24:50.334 | Details:
(TestVolumeBootPatternV2:test_volume_boot_pattern) Server
8a628dec-ebe5-401a-9ba9-72358eabca06 failed to reach ACTIVE status and task
state "None" within the required time (196 s). Current status: BUILD. Current
task state: spawning.
The instance was created at:
2017-02-11 01:24:50.304 | 2017-02-11 01:17:58,462 26423 INFO
[tempest.lib.common.rest_client] Request
(TestVolumeBootPatternV2:test_volume_boot_pattern): 202 POST
http://158.69.83.100:8774/v2.1/servers 2.300s
Then the test spins on the instance waiting for it to transition into ACTIVE:
2017-02-11 01:24:50.305 | 2017-02-11 01:17:58,802 26423 INFO
[tempest.lib.common.rest_client] Request
(TestVolumeBootPatternV2:test_volume_boot_pattern): 200 GET
http://158.69.83.100:8774/v2.1/servers/8a628dec-ebe5-401a-9ba9-72358eabca06
0.333s
2017-02-11 01:24:50.305 | 2017-02-11 01:17:58,802 26423 DEBUG
[tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token':
'', 'Accept': 'application/json', 'Content-Type': 'application/json'}
2017-02-11 01:24:50.305 | Body: None
2017-02-11 01:24:50.305 | Response - Headers: {u'connection':
'close', u'vary': 'X-OpenStack-Nova-API-Version', u'content-length': '1093',
u'x-openstack-nova-api-version': '2.1', u'openstack-api-version': 'compute
2.1', 'status': '200', 'content-location':
'http://158.69.83.100:8774/v2.1/servers/8a628dec-ebe5-401a-9ba9-72358eabca06',
u'content-type': 'application/json', u'x-compute-request-id':
'req-3abcfd5a-7e3d-4333-b66b-547e4ce79ce7', u'date': 'Sat, 11 Feb 2017 01:17:58
GMT'}
2017-02-11 01:24:50.305 | Body: {"server":
{"OS-EXT-STS:task_state": "scheduling", "addresses": {}, "links": [{"href":
"http://158.69.83.100:8774/v2.1/servers/8a628dec-ebe5-401a-9ba9-72358eabca06;,
"rel": "self"}, {"href":
"http://158.69.83.100:8774/servers/8a628dec-ebe5-401a-9ba9-72358eabca06;,
"rel": "bookmark"}], "image": "", "OS-EXT-STS:vm_state": "building",
"OS-SRV-USG:launched_at": null, "flavor": {"id": "42", "links": [{"href":
"http://158.69.83.100:8774/flavors/42;, "rel": "bookmark"}]}, "id":
"8a628dec-ebe5-401a-9ba9-72358eabca06", "user_id":
"f4e2fa64f9c54cadb1cac2642a2d757f", "OS-DCF:diskConfig": "MANUAL",
"accessIPv4": "", "accessIPv6": "", "progress": 0, "OS-EXT-STS:power_state": 0,
"OS-EXT-AZ:availability_zone": "", "metadata": {}, "status": "BUILD",
"updated": "2017-02-11T01:17:58Z", "hostId": "", "OS-SRV-USG:terminated_at":
null,
[Yahoo-eng-team] [Bug 1738475] Re: test_notify_port_ready_after_enable_dhcp fails with RuntimeError: 'dhcp_ready_on_ports' not be called
Haven't happened for a long time, probably fixed already. ** Changed in: neutron Status: New => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1738475 Title: test_notify_port_ready_after_enable_dhcp fails with RuntimeError: 'dhcp_ready_on_ports' not be called Status in neutron: Fix Released Bug description: As of recent there appears to be an intermittent failure in neutron.tests.functional.agent.test_dhcp_agent.DHCPAgentOVSTestCase.test_notify_port_ready_after_enable_dhcp; it fails with: RuntimeError: 'dhcp_ready_on_ports' not be called For example [1]. As of right now I only see 2 patches that have failed with this [2], but it also appears to only have recently started occurring. Additional debug is required. [1] http://logs.openstack.org/82/423382/18/check/legacy-neutron-dsvm-functional/9fef974/job-output.txt.gz#_2017-12-15_15_09_15_599225 [2] http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22RuntimeError%3A%20'dhcp_ready_on_ports'%20not%20be%20called%5C%22 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1738475/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1577457] Re: LinuxBridge fullstack tests run with noop firewall
** Changed in: neutron Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1577457 Title: LinuxBridge fullstack tests run with noop firewall Status in neutron: Fix Released Bug description: LinuxBridge agents should be using iptables firewall To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1577457/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1673531] Re: fullstack test_controller_timeout_does_not_break_connectivity_sigkill(GRE and l2pop, openflow-native_ovsdb-cli) failure
We believe it was fixed, probably by https://review.openstack.org/#/c/547345/ ** Changed in: neutron Status: Confirmed => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1673531 Title: fullstack test_controller_timeout_does_not_break_connectivity_sigkill(GRE and l2pop,openflow-native_ovsdb-cli) failure Status in neutron: Fix Released Bug description: Logs for failure: http://logs.openstack.org/98/446598/1/check/gate- neutron-dsvm-fullstack-ubuntu-xenial/2e0f93e/logs/dsvm-fullstack- logs/TestOvsConnectivitySameNetworkOnOvsBridgeControllerStop .test_controller_timeout_does_not_break_connectivity_sigkill_GRE-and- l2pop,openflow-native_ovsdb-cli_/ logstash query: http://logstash.openstack.org/#dashboard/file/logstash.json?query=messge%3A%5C%22in%20test_controller_timeout_does_not_break_connectivity_sigkill%5C%22%20AND%20tags%3Aconsole%20AND%20build_name %3Agate-neutron-dsvm-fullstack-ubuntu-xenial To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1673531/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1741889] Re: functional: DbAddCommand sometimes times out after 10 seconds
After ovsdbapp release, we don't see it happening in functional job anymore. There are some timeout hits in logstash in tempest jobs, but happening at the same time as other actions executed by the agent (e.g. update state RPC call) are timing out too, so probably host overloaded. ** Changed in: neutron Status: New => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1741889 Title: functional: DbAddCommand sometimes times out after 10 seconds Status in neutron: Fix Released Status in ovsdbapp: Fix Released Bug description: Traceback (most recent call last): File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/fixtures/fixture.py", line 197, in setUp self._setUp() File "neutron/tests/common/net_helpers.py", line 721, in _setUp self.bridge = common_base.create_resource(self.prefix, ovs.add_bridge) File "neutron/tests/common/base.py", line 47, in create_resource return creation_func(name, *args, **kwargs) File "neutron/agent/common/ovs_lib.py", line 140, in add_bridge br.create() File "neutron/agent/common/ovs_lib.py", line 255, in create FAILMODE_SECURE)) File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__ self.gen.next() File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/ovsdbapp/api.py", line 94, in transaction self._nested_txn = None File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/ovsdbapp/api.py", line 54, in __exit__ self.result = self.commit() File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 57, in commit timeout=self.timeout) ovsdbapp.exceptions.TimeoutException: Commands [, ] exceeded timeout 10 seconds logstash query: http://logstash.openstack.org/#dashboard/file/logstash.json?query=build_name%3A%5C%22legacy-neutron-dsvm-functional%5C%22%20AND%20message%3A%5C%22ovsdbapp.exceptions.TimeoutException%3A%20Commands%5C%22%20AND%20tags%3Aconsole%20AND%20build_status%3AFAILURE In the logs we can see AddBridgeCommand result returning TRY_AGAIN over and over again: http://logs.openstack.org/70/531070/4/check/legacy-neutron-dsvm-functional/9aa7107/logs/dsvm-functional-logs/neutron.tests.functional.agent.test_dhcp_agent.DHCPAgentOVSTestCase.test_agent_mtu_set_on_interface_driver.txt.gz#_2018-01-08_04_20_06_754 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1741889/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1734356] Re: fullstack: Test runner doesn't wait enough time for env to come up
** Changed in: neutron Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1734356 Title: fullstack: Test runner doesn't wait enough time for env to come up Status in neutron: Fix Released Bug description: As shown in http://logs.openstack.org/71/520371/7/check/legacy- neutron-dsvm-fullstack/ad585a2/logs/dsvm-fullstack- logs/TestOvsConnectivitySameNetwork.test_connectivity_VXLAN,openflow- native_.txt.gz fullstack doesn't wait enough time for agents to report to neutron server. Currently it uses default value of 60 seconds, so once all agent processes are started they have 60 seconds to report back. On busy or weak hardware machines this is not enough. Here comes RCA: Logs: Problematic neutron-openvswitch-agent: http://logs.openstack.org/71/520371/7/check/legacy-neutron-dsvm-fullstack/ad585a2/logs/dsvm-fullstack-logs/TestOvsConnectivitySameNetwork.test_connectivity_VXLAN,openflow-native_/neutron-openvswitch-agent--2017-11-20--21-51-57-666771.txt.gz Neutron-server: http://logs.openstack.org/71/520371/7/check/legacy-neutron-dsvm-fullstack/ad585a2/logs/dsvm-fullstack-logs/TestOvsConnectivitySameNetwork.test_connectivity_VXLAN,openflow-native_/neutron-server--2017-11-20--21-51-23-097309.txt.gz Test runner: http://logs.openstack.org/71/520371/7/check/legacy-neutron-dsvm-fullstack/ad585a2/logs/dsvm-fullstack-logs/TestOvsConnectivitySameNetwork.test_connectivity_VXLAN,openflow-native_.txt.gz RCA: Agent started by test runner at 21:51:57.666: http://logs.openstack.org/71/520371/7/check/legacy-neutron-dsvm-fullstack/ad585a2/logs/dsvm-fullstack-logs/TestOvsConnectivitySameNetwork.test_connectivity_VXLAN,openflow-native_.txt.gz#_2017-11-20_21_51_57_666 Test runner stopped polling server for agents at 21:52:51.506: http://logs.openstack.org/71/520371/7/check/legacy-neutron-dsvm-fullstack/ad585a2/logs/dsvm-fullstack-logs/TestOvsConnectivitySameNetwork.test_connectivity_VXLAN,openflow-native_.txt.gz#_2017-11-20_21_52_51_506 Agent process started at 21:52:23.444, 26 seconds! after async_process claimed process has started Server got report of the agent at 21:52:58.939 - 7 seconds after test runner stopped polling: http://logs.openstack.org/71/520371/7/check/legacy-neutron-dsvm-fullstack/ad585a2/logs/dsvm-fullstack-logs/TestOvsConnectivitySameNetwork.test_connectivity_VXLAN,openflow-native_/neutron-server--2017-11-20--21-51-23-097309.txt.gz#_2017-11-20_21_52_58_939 Summary: On environment with three nodes, running dhcp and ovs agent on each, last agent reported 7 seconds after API polling was finished by test runner. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1734356/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1687074] Re: Sometimes ovsdb fails with "tcp:127.0.0.1:6640: error parsing stream"
Marked neutron part fixed because we don't see gate failure because of the error message lately. It could even be that failures we saw in the past were not related to the error message, and the message was just a red herring. Slawek and others made huge progress lately on stabilizing fullstack job and merged a lot of patches unrelated to the message, and the job is not quite stable. We are closing it for neutron. ovsdbapp folks should decide on their own if they want to pursue it. ** Changed in: neutron Status: Confirmed => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1687074 Title: Sometimes ovsdb fails with "tcp:127.0.0.1:6640: error parsing stream" Status in neutron: Fix Released Status in ovsdbapp: Confirmed Bug description: Example (Ocata): http://logs.openstack.org/67/460867/1/check/gate- neutron-dsvm-functional-ubuntu-xenial/382d800/logs/dsvm-functional- logs/neutron.tests.functional.agent.ovsdb.test_impl_idl.ImplIdlTestCase.test_post_commit_vswitchd_completed_no_failures.txt.gz 2017-04-28 07:59:01.430 11929 WARNING neutron.agent.ovsdb.native.vlog [-] tcp:127.0.0.1:6640: error parsing stream: line 0, column 1, byte 1: syntax error at beginning of input 2017-04-28 07:59:01.431 11929 DEBUG neutron.agent.ovsdb.impl_idl [-] Running txn command(idx=0): AddBridgeCommand(name=test-brc6de03bf, may_exist=True, datapath_type=None) do_commit neutron/agent/ovsdb/impl_idl.py:100 2017-04-28 07:59:01.433 11929 DEBUG neutron.agent.ovsdb.impl_idl [-] OVSDB transaction returned TRY_AGAIN, retrying do_commit neutron/agent/ovsdb/impl_idl.py:111 2017-04-28 07:59:01.433 11929 WARNING neutron.agent.ovsdb.native.vlog [-] tcp:127.0.0.1:6640: connection dropped (Protocol error) If we look at logstash here: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22tcp%3A127.0.0.1%3A6640%3A%20error%20parsing%20stream%5C%22 We see some interesting data points, sometimes it actually logs what's in the buffer, and I see instances of: 2017-04-27 19:02:51.755 [neutron.tests.functional.tests.common.exclusive_resources.test_port.TestExclusivePort.test_port] 3300 WARNING ovsdbapp.backend.ovs_idl.vlog [-] tcp:127.0.0.1:6640: error parsing stream: line 0, column 1355, byte 1355: invalid keyword 'id' 2017-04-27 14:22:02.294 [neutron.tests.functional.agent.linux.test_ovsdb_monitor.TestSimpleInterfaceMonitor.test_get_events_native_] 3433 WARNING ovsdbapp.backend.ovs_idl.vlog [-] tcp:127.0.0.1:6640: error parsing stream: line 0, column 3, byte 3: invalid keyword 'new' 2017-04-27 04:46:17.667 [neutron.tests.functional.agent.test_dhcp_agent.DHCPAgentOVSTestCase.test_bad_address_allocation] 4136 WARNING ovsdbapp.backend.ovs_idl.vlog [-] tcp:127.0.0.1:6640: error parsing stream: line 0, column 3, byte 3: invalid keyword 'ace' 2017-04-26 18:04:59.110 [neutron.tests.functional.agent.linux.test_linuxbridge_arp_protect.LinuxBridgeARPSpoofTestCase.test_arp_correct_protection_allowed_address_pairs] 3477 WARNING ovsdbapp.backend.ovs_idl.vlog [-] tcp:127.0.0.1:6640: error parsing stream: line 0, column 3, byte 3: invalid keyword 'err' 2017-04-25 19:00:01.452 [neutron.tests.functional.agent.test_dhcp_agent.DHCPAgentOVSTestCase.test_agent_mtu_set_on_interface_driver] 4251 WARNING ovsdbapp.backend.ovs_idl.vlog [-] tcp:127.0.0.1:6640: error parsing stream: line 0, column 3, byte 3: invalid keyword 'set' 2017-04-25 16:34:11.355 [neutron.tests.functional.agent.linux.test_linuxbridge_arp_protect.LinuxBridgeARPSpoofTestCase.test_arp_fails_incorrect_mac_protection] 3332 WARNING ovsdbapp.backend.ovs_idl.vlog [-] tcp:127.0.0.1:6640: error parsing stream: line 0, column 5, byte 5: invalid keyword 'tatus' 2017-04-25 03:28:25.858 [neutron.tests.functional.agent.ovsdb.test_impl_idl.ImplIdlTestCase.test_post_commit_vswitchd_completed_no_failures] 4112 WARNING ovsdbapp.backend.ovs_idl.vlog [-] tcp:127.0.0.1:6640: error parsing stream: line 0, column 3, byte 3: invalid keyword 'set' 2017-04-24 21:59:39.094 [neutron.tests.functional.agent.linux.test_linuxbridge_arp_protect.LinuxBridgeARPSpoofTestCase.test_arp_protection_port_security_disabled] 3682 WARNING ovsdbapp.backend.ovs_idl.vlog [-] tcp:127.0.0.1:6640: error parsing stream: line 0, column 5, byte 5: invalid keyword 'rsion' Terry says it doesn't resemble the protocol, but some random crap, potentially from some random place in memory (SCARY!) To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1687074/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1276510] Re: MySQL 2013 lost connection is being raised
** Changed in: oslo-incubator/havana Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1276510 Title: MySQL 2013 lost connection is being raised Status in Ceilometer: Fix Released Status in Cinder: Fix Released Status in Cinder havana series: Fix Released Status in Ironic: Fix Released Status in OpenStack Identity (keystone): Invalid Status in neutron: Fix Released Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) havana series: Fix Released Status in oslo-incubator: Fix Released Status in oslo-incubator havana series: Fix Released Status in Sahara: Fix Released Status in tuskar: Fix Released Bug description: MySQL's 2013 code error is not in the list of connection lost issues. This causes the reconnect loop to raise this error and stop retrying. [database] max_retries = -1 retry_interval = 1 mysql down: ==> scheduler.log <== 2014-02-03 16:51:50.956 16184 CRITICAL cinder [-] (OperationalError) (2013, "Lost connection to MySQL server at 'reading initial communication packet', system error: 0") None None To manage notifications about this bug go to: https://bugs.launchpad.net/ceilometer/+bug/1276510/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1257293] Re: [messaging] QPID broadcast RPC requests to all servers for a given topic
** Changed in: oslo-incubator/havana Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1257293 Title: [messaging] QPID broadcast RPC requests to all servers for a given topic Status in Ceilometer: Fix Released Status in Ceilometer havana series: Fix Released Status in Cinder: Fix Released Status in Cinder havana series: Fix Released Status in OpenStack Heat: Fix Released Status in OpenStack Heat havana series: Fix Released Status in OpenStack Identity (keystone): Fix Released Status in OpenStack Identity (keystone) havana series: Fix Released Status in neutron: Fix Released Status in neutron havana series: Fix Released Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) havana series: Fix Released Status in oslo-incubator: Fix Released Status in oslo-incubator havana series: Fix Released Status in oslo.messaging: Fix Released Bug description: According to the oslo.messaging documentation, when a RPC request is made to a given topic, and there are multiple servers for that topic, only _one_ server should service that RPC request. See http://docs.openstack.org/developer/oslo.messaging/target.html "topic (str) – A name which identifies the set of interfaces exposed by a server. Multiple servers may listen on a topic and messages will be dispatched to one of the servers in a round-robin fashion." In the case of a QPID-based deployment using topology version 2, this is not the case. Instead, each listening server gets a copy of the RPC and will process it. For more detail, see https://bugs.launchpad.net/oslo/+bug/1178375/comments/26 To manage notifications about this bug go to: https://bugs.launchpad.net/ceilometer/+bug/1257293/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1374519] Re: Orphaned queues are not auto-deleted for Qpid
** Changed in: oslo.messaging/juno Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1374519 Title: Orphaned queues are not auto-deleted for Qpid Status in neutron: Fix Released Status in neutron icehouse series: Won't Fix Status in oslo.messaging: Fix Released Status in oslo.messaging juno series: Fix Released Bug description: The following patch incorrectly set auto-delete for Qpid to False: https://github.com/openstack/oslo-incubator/commit/5ff534d1#diff- 372094c4bfc6319d22875a970aa6b730R190 While for RabbitMQ, it's True. This results in queues left on the broker if client dies and does not return back. Red Hat bug for reference: https://bugzilla.redhat.com/show_bug.cgi?id=1099657 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1374519/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1602081] Re: Use oslo.context's policy dict
** Changed in: keystone Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1602081 Title: Use oslo.context's policy dict Status in Cinder: Fix Released Status in Glance: Fix Released Status in OpenStack Heat: Fix Released Status in Ironic: Fix Released Status in OpenStack Identity (keystone): Fix Released Status in OpenStack Compute (nova): Fix Released Bug description: This is a cross project goal to standardize the values available to policy writers and to improve the basic oslo.context object. It is part of the follow up work to bug #1577996 and bug #968696. There has been an ongoing problem for how we define the 'admin' role. Because tokens are project scoped having the 'admin' role on any project granted you the 'admin' role on all of OpenStack. As a solution to this keystone defined an is_admin_project field so that keystone defines a single project that your token must be scoped to to perform admin operations. This has been implemented. The next phase of this is to make all the projects understand the X -Is-Admin-Project header from keystonemiddleware and pass it to oslo_policy. However this pattern of keystone changes something and then goes to every project to fix it has been repeated a number of times now and we would like to make it much more automatic. Ongoing work has enhanced the base oslo.context object to include both the load_from_environ and to_policy_values methods. The load_from_environ classmethod takes an environment dict with all the standard auth_token and oslo middleware headers and loads them into their standard place on the context object. The to_policy_values() then creates a standard credentials dictionary with all the information that should be required to enforce policy from the context. The combination of these two methods means in future when authentication information needs to be passed to policy it can be handled entirely by oslo.context and does not require changes in each individual service. Note that in future a similar pattern will hopefully be employed to simplify passing authentication information over RPC to solve the timeout issues. This is a prerequisite for that work. There are a few common problems in services that are required to make this work: 1. Most service context.__init__ functions take and discard **kwargs. This is so if the context.from_dict receives arguments it doesn't know how to handle (possibly because new things have been added to the base to_dict) it ignores them. Unfortunately to make the load_from_environ method work we need to pass parameters to __init__ that are handled by the base class. To make this work we simply have to do a better job of using from_dict. Instead of passing everything to __init__ and ignoring what we don't know we have from_dict extract only the parameters that context knows how to use and call __init__ with those. 2. The parameters passed to the base context.__init__ are old. Typically they are user and tenant where most services expect user_id and project_id. There is ongoing work to improve this in oslo.context but for now we have to ensure that the subclass correctly sets and uses the right variable names. 3. Some services provide additional information to the policy enforcement method. To continue to make this function we will simply override the to_policy_values method in the subclasses. To manage notifications about this bug go to: https://bugs.launchpad.net/cinder/+bug/1602081/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1527675] Re: Neutron multinode grenade sometimes fails at resource phase create
** Changed in: neutron Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1527675 Title: Neutron multinode grenade sometimes fails at resource phase create Status in neutron: Fix Released Bug description: Mailing list thread: http://lists.openstack.org/pipermail/openstack- dev/2015-November/080503.html http://logs.openstack.org/35/187235/11/experimental/gate-grenade-dsvm-neutron-multinode/a5af283/logs/grenade.sh.txt.gz Pinging the VM works, however when SSH'ing in to verify further, the connection is repeatedly closed. http://logs.openstack.org/35/187235/11/experimental/gate-grenade-dsvm- neutron- multinode/a5af283/logs/grenade.sh.txt.gz#_2015-11-30_20_25_18_391 2015-11-30 20:20:18.283 | + ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i /opt/stack/save/cinder_key.pem cirros@172.24.5.53 'echo '\''I am a teapot'\'' > verify.txt' 2015-11-30 20:25:18.391 | Connection closed by 172.24.5.53 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1527675/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1608691] Re: exception translation for pymysql "Lock wait timeout exceeded" broken
** Changed in: neutron
Status: Fix Committed => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1608691
Title:
exception translation for pymysql "Lock wait timeout exceeded" broken
Status in neutron:
Fix Released
Status in oslo.db:
Fix Released
Bug description:
It looks like the classifier for "lock wait timeout exceeded" as a
DBDeadlock is not correct for pymysql. Snippet of traceback showing it
as the generic DBError:
2016-07-31 00:21:14.938 4145 ERROR neutron.api.v2.resource File
"/usr/local/lib/python2.7/dist-packages/pymysql/err.py", line 120, in
raise_mysql_exception
2016-07-31 00:21:14.938 4145 ERROR neutron.api.v2.resource
_check_mysql_exception(errinfo)
2016-07-31 00:21:14.938 4145 ERROR neutron.api.v2.resource File
"/usr/local/lib/python2.7/dist-packages/pymysql/err.py", line 115, in
_check_mysql_exception
2016-07-31 00:21:14.938 4145 ERROR neutron.api.v2.resource raise
InternalError(errno, errorvalue)
2016-07-31 00:21:14.938 4145 ERROR neutron.api.v2.resource DBError:
(pymysql.err.InternalError) (1205, u'Lock wait timeout exceeded; try restarting
transaction') [SQL: u'SELECT ipavailabilityranges.allocation_pool_id AS
ipavailabilityranges_allocation_pool_id, ipavailabilityranges.first_ip AS
ipavailabilityranges_first_ip, ipavailabilityranges.last_ip AS
ipavailabilityranges_last_ip \nFROM ipavailabilityranges INNER JOIN
ipallocationpools ON ipallocationpools.id =
ipavailabilityranges.allocation_pool_id \nWHERE ipallocationpools.subnet_id =
%(subnet_id_1)s \n LIMIT %(param_1)s FOR UPDATE'] [parameters: {u'param_1': 1,
u'subnet_id_1': u'344485c9-e600-40c0-8acf-d9ba98dacf99'}]
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1608691/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1694769] Re: Nova fails to plug port because of missing ipset when calling iptables-restore
** Changed in: os-vif/newton
Status: In Progress => Opinion
** Changed in: os-vif/ocata
Status: Fix Committed => Fix Released
** Changed in: os-vif/newton
Status: Opinion => Invalid
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1694769
Title:
Nova fails to plug port because of missing ipset when calling
iptables-restore
Status in neutron:
Won't Fix
Status in os-vif:
Fix Released
Status in os-vif newton series:
Invalid
Status in os-vif ocata series:
Fix Released
Bug description:
This is Ocata, linuxbridge.
http://logs.openstack.org/95/466395/3/gate/gate-tempest-dsvm-neutron-
linuxbridge-ubuntu-xenial/e5923b4/logs/testr_results.html.gz
File "tempest/common/compute.py", line 188, in create_test_server
clients.servers_client, server['id'], wait_until)
File "tempest/common/waiters.py", line 76, in wait_for_server_status
server_id=server_id)
tempest.exceptions.BuildErrorException: Server
2a04ac11-2ec6-4a0d-a8f5-c89d129e881d failed to build and is in ERROR status
Details: {u'created': u'2017-05-27T03:00:23Z', u'code': 500, u'message': u'No
valid host was found. There are not enough hosts available.'}
The failure in nova-cpu log:
http://logs.openstack.org/95/466395/3/gate/gate-tempest-dsvm-neutron-
linuxbridge-ubuntu-
xenial/e5923b4/logs/screen-n-cpu.txt.gz#_2017-05-27_03_00_21_716
2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager
[req-06c29149-80d9-4923-b9c4-54591a3f5e7e
tempest-ServerActionsTestJSON-1792219232
tempest-ServerActionsTestJSON-1792219232] [instance:
2a04ac11-2ec6-4a0d-a8f5-c89d129e881d] Instance failed to spawn
2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance:
2a04ac11-2ec6-4a0d-a8f5-c89d129e881d] Traceback (most recent call last):
2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance:
2a04ac11-2ec6-4a0d-a8f5-c89d129e881d] File
"/opt/stack/new/nova/nova/compute/manager.py", line 2124, in _build_resources
2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance:
2a04ac11-2ec6-4a0d-a8f5-c89d129e881d] yield resources
2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance:
2a04ac11-2ec6-4a0d-a8f5-c89d129e881d] File
"/opt/stack/new/nova/nova/compute/manager.py", line 1930, in
_build_and_run_instance
2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance:
2a04ac11-2ec6-4a0d-a8f5-c89d129e881d] block_device_info=block_device_info)
2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance:
2a04ac11-2ec6-4a0d-a8f5-c89d129e881d] File
"/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 2698, in spawn
2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance:
2a04ac11-2ec6-4a0d-a8f5-c89d129e881d] destroy_disks_on_failure=True)
2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance:
2a04ac11-2ec6-4a0d-a8f5-c89d129e881d] File
"/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 5114, in
_create_domain_and_network
2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance:
2a04ac11-2ec6-4a0d-a8f5-c89d129e881d] destroy_disks_on_failure)
2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance:
2a04ac11-2ec6-4a0d-a8f5-c89d129e881d] File
"/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in
__exit__
2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance:
2a04ac11-2ec6-4a0d-a8f5-c89d129e881d] self.force_reraise()
2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance:
2a04ac11-2ec6-4a0d-a8f5-c89d129e881d] File
"/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance:
2a04ac11-2ec6-4a0d-a8f5-c89d129e881d] six.reraise(self.type_, self.value,
self.tb)
2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance:
2a04ac11-2ec6-4a0d-a8f5-c89d129e881d] File
"/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 5077, in
_create_domain_and_network
2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance:
2a04ac11-2ec6-4a0d-a8f5-c89d129e881d] self.plug_vifs(instance, network_info)
2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance:
2a04ac11-2ec6-4a0d-a8f5-c89d129e881d] File
"/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 749, in plug_vifs
2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance:
2a04ac11-2ec6-4a0d-a8f5-c89d129e881d] self.vif_driver.plug(instance, vif)
2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance:
2a04ac11-2ec6-4a0d-a8f5-c89d129e881d] File
"/opt/stack/new/nova/nova/virt/libvirt/vif.py", line 786, in plug
2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance:
2a04ac11-2ec6-4a0d-a8f5-c89d129e881d] self._plug_os_vif(instance, vif_obj)
2017-05-27
[Yahoo-eng-team] [Bug 1627424] Re: FlushError on IPAllocation
** Changed in: neutron Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1627424 Title: FlushError on IPAllocation Status in neutron: Fix Released Bug description: http://logs.openstack.org/49/373249/1/check/gate-tempest-dsvm-neutron- full-ubuntu-xenial/b1312ed/logs/screen-q-svc.txt.gz?level=TRACE http://logs.openstack.org/68/347268/5/gate/gate-tempest-dsvm-neutron- full-ubuntu-xenial/a82f85d/logs/ To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1627424/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1672921] Re: Functional gate job broken due to os-log-merger crashing on null-bytes
** Changed in: os-log-merger Status: Fix Committed => Fix Released ** Changed in: os-log-merger Assignee: (unassigned) => Ihar Hrachyshka (ihar-hrachyshka) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1672921 Title: Functional gate job broken due to os-log-merger crashing on null-bytes Status in neutron: Fix Released Status in os-log-merger: Fix Released Bug description: gate-neutron-dsvm-functional-ubuntu-xenial seems to be failing even though tests are being passed. 1. http://logs.openstack.org/53/319353/3/check/gate-neutron-dsvm-functional-ubuntu-xenial/b36e01e/ 2. http://logs.openstack.org/25/338625/33/check/gate-neutron-dsvm-functional-ubuntu-xenial/fc680a7/ To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1672921/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1742197] Re: linuxbridge scenario job doesn't execute NetworkMtuBaseTest
** Changed in: neutron
Status: Fix Committed => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1742197
Title:
linuxbridge scenario job doesn't execute NetworkMtuBaseTest
Status in neutron:
Fix Released
Bug description:
This is because the job is not configured to access vxlan / gre type
drivers:
2018-01-08 16:40:26.894303 | primary | {1} setUpClass
(neutron_tempest_plugin.scenario.test_mtu.NetworkMtuBaseTest) ...
SKIPPED: GRE or VXLAN type_driver is not enabled
Linuxbridge doesn't support GRE but VXLAN should work.
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1742197/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1734136] Re: Openstack Neutron is not sending complete data during qos-policy-name update
** Changed in: neutron Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1734136 Title: Openstack Neutron is not sending complete data during qos-policy-name update Status in neutron: Fix Released Bug description: 1.create a qos-policy with bandwidth limit rule and dscp marking rule 2.update the qos-policy-name 3.After updating the qos-policy name, bandwidth limit rules and dscp marking rules are missing. Only the qos-policy-name is getting updated. Openstack neutron is not sending bandwidth_limit_rules & dscp_marking_rules along with the updated qos policy name when I have updated the qos-policy-name from Vamsi to Krish. Expected behavior is to have the updated policy name along with bandwidth_limit_rules & dscp_marking_rules same as that in old qos- policy-name. Version: Stable/Newton 192.168.56.102 --> Openstack 192.168.56.1 --> ODL JavaScript Object Notation: application/json Object Member Key: policy Object Member Key: bandwidth_limit_rules Array Key: bandwidth_limit_rules Member Key: name String value: Krish Key: name Member Key: created_at String value: 2017-11-20T19:27:06Z Key: created_at Member Key: updated_at String value: 2017-11-20T19:27:15Z Key: updated_at Member Key: dscp_marking_rules Array Key: dscp_marking_rules Member Key: revision_number Number value: 4 Key: revision_number Member Key: shared False value Key: shared Member Key: project_id String value: eacd04d3ff4b4e1d9dc2f5282edbb9e0 Key: project_id Member Key: id String value: 066fd21e-2041-43f6-956a-6bec0948bf15 Key: id Member Key: description String value: Key: description Key: policy Attached the wireshark cap for this issue. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1734136/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1744359] Re: Neutron haproxy logs are not being collected
** Changed in: neutron Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1744359 Title: Neutron haproxy logs are not being collected Status in devstack: New Status in neutron: Fix Released Status in tripleo: Triaged Bug description: In Neutron, we use haproxy to proxy metadata requests from instances to Nova Metadata service. By default, haproxy logs to /dev/log but in Ubuntu, those requests get redirected by rsyslog to /var/log/haproxy.log which is not being collected. ubuntu@devstack:~$ cat /etc/rsyslog.d/49-haproxy.conf # Create an additional socket in haproxy's chroot in order to allow logging via # /dev/log to chroot'ed HAProxy processes $AddUnixListenSocket /var/lib/haproxy/dev/log # Send HAProxy messages to a dedicated logfile if $programname startswith 'haproxy' then /var/log/haproxy.log &~ Another possibility would be to change the haproxy.cfg file to include the log-tag option so that haproxy uses a different tag [0] and then it'd be dumped into syslog instead but this would break backwards compatibility. [0] https://cbonte.github.io/haproxy-dconv/configuration-1.5.html#3.1 -log-tag To manage notifications about this bug go to: https://bugs.launchpad.net/devstack/+bug/1744359/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1744361] Re: test_server_security_groups failed to reboot with "Domain not found: no domain with matching uuid" because of missing vif event from linuxbridge agent
** Changed in: neutron
Status: Confirmed => Invalid
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1744361
Title:
test_server_security_groups failed to reboot with "Domain not found:
no domain with matching uuid" because of missing vif event from
linuxbridge agent
Status in neutron:
Invalid
Status in OpenStack Compute (nova):
Fix Released
Status in OpenStack Compute (nova) ocata series:
Fix Committed
Status in OpenStack Compute (nova) pike series:
Fix Committed
Status in OpenStack Compute (nova) queens series:
Fix Released
Bug description:
This happened in master (Queens) in tempest-linuxbridge job:
http://logs.openstack.org/56/534456/1/check/neutron-tempest-
linuxbridge/7693ca5/logs/testr_results.html.gz
From tempest log:
Traceback (most recent call last):
File "tempest/api/compute/security_groups/test_security_groups.py", line
105, in test_server_security_groups
'ACTIVE')
File "tempest/common/waiters.py", line 96, in wait_for_server_status
raise lib_exc.TimeoutException(message)
tempest.lib.exceptions.TimeoutException: Request timed out
Details: (SecurityGroupsTestJSON:test_server_security_groups) Server
dd2ec3b0-f909-4104-ae2a-9b878d936ed4 failed to reach ACTIVE status and task
state "None" within the required time (196 s). Current status: HARD_REBOOT.
Current task state: reboot_started_hard.
In nova-compute log, we see this:
Jan 19 10:38:37.869397 ubuntu-xenial-rax-iad-0002010825 nova-compute[28907]:
WARNING nova.virt.libvirt.driver [None req-838e0ebe-afe5-4cca-abcc-4ba116889cba
service nova] [instance: dd2ec3b0-f909-4104-ae2a-9b878d936ed4] Timeout waiting
for vif plugging callback for instance with vm_state active and task_state
reboot_started_hard.: Timeout: 300 seconds
Jan 19 10:38:37.880789 ubuntu-xenial-rax-iad-0002010825 nova-compute[28907]:
DEBUG nova.compute.manager [None req-838e0ebe-afe5-4cca-abcc-4ba116889cba
service nova] [instance: dd2ec3b0-f909-4104-ae2a-9b878d936ed4] Checking state
{{(pid=28907) _get_power_state
/opt/stack/new/nova/nova/compute/manager.py:1183}}
Jan 19 10:38:37.885265 ubuntu-xenial-rax-iad-0002010825 nova-compute[28907]:
ERROR nova.compute.manager [None req-838e0ebe-afe5-4cca-abcc-4ba116889cba
service nova] [instance: dd2ec3b0-f909-4104-ae2a-9b878d936ed4] Cannot reboot
instance: Domain not found: no domain with matching uuid
'dd2ec3b0-f909-4104-ae2a-9b878d936ed4' (instance-0023): libvirtError:
Domain not found: no domain with matching uuid
'dd2ec3b0-f909-4104-ae2a-9b878d936ed4' (instance-0023)
Jan 19 10:38:37.981484 ubuntu-xenial-rax-iad-0002010825 nova-compute[28907]:
DEBUG nova.compute.manager [None req-838e0ebe-afe5-4cca-abcc-4ba116889cba
service nova] [instance: dd2ec3b0-f909-4104-ae2a-9b878d936ed4] Instance has
been destroyed from under us while trying to set it to ERROR {{(pid=28907)
_set_instance_obj_error_state /opt/stack/new/nova/nova/compute/manager.py:586}}
Jan 19 10:38:38.150498 ubuntu-xenial-rax-iad-0002010825 nova-compute[28907]:
ERROR oslo_messaging.rpc.server [None req-838e0ebe-afe5-4cca-abcc-4ba116889cba
service nova] Exception during message handling: libvirtError: Domain not
found: no domain with matching uuid 'dd2ec3b0-f909-4104-ae2a-9b878d936ed4'
(instance-0023)
Jan 19 10:38:38.150813 ubuntu-xenial-rax-iad-0002010825 nova-compute[28907]:
ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Jan 19 10:38:38.151125 ubuntu-xenial-rax-iad-0002010825 nova-compute[28907]:
ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line
163, in _process_incoming
Jan 19 10:38:38.151388 ubuntu-xenial-rax-iad-0002010825 nova-compute[28907]:
ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
Jan 19 10:38:38.151668 ubuntu-xenial-rax-iad-0002010825 nova-compute[28907]:
ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line
220, in dispatch
Jan 19 10:38:38.151939 ubuntu-xenial-rax-iad-0002010825 nova-compute[28907]:
ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method,
ctxt, args)
Jan 19 10:38:38.152205 ubuntu-xenial-rax-iad-0002010825 nova-compute[28907]:
ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line
190, in _do_dispatch
Jan 19 10:38:38.152491 ubuntu-xenial-rax-iad-0002010825 nova-compute[28907]:
ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
Jan 19 10:38:38.152705 ubuntu-xenial-rax-iad-0002010825 nova-compute[28907]:
ERROR oslo_messaging.rpc.server File
"/opt/stack/new/nova/nova/exception_wrapper.py", line 76, in wrapped
Jan 19 10:38:38.152962 ubuntu-xenial-rax-iad-0002010825 nova-compute[28907]:
ERROR oslo_messaging.rpc.server
[Yahoo-eng-team] [Bug 1750890] Re: Neutron db performance at scale
Yes, since we landed a bunch of patches in Ocata+ that should fix performance in several scenarios, incl. switch to subquery type for model attributes, I advise you check if scalability issues affect your fresh setup. If so, please provide more details (charts, measurements), and we will take a closer look. Marking the bug as fixed for the time being. (An alternative would be 'Incomplete', but we had related fixes merged in Ocata+). As for older releases, we no longer support Newton, so we can't provide backports ourselves. ** Changed in: neutron Status: New => Fix Released ** Changed in: neutron Importance: Undecided => High ** Tags added: db -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1750890 Title: Neutron db performance at scale Status in neutron: Fix Released Bug description: OpenStack Neutron (like OpenStack) relies on SQL Alcehmy and its ORM for database support. From our observations, Neutron is not utilizing the ORM models directly, but rather inserting an additional model layer above SQLAlchmeny and manually building these models from a number of underlying DB models. We ran into significant performance issues due to the increased number of queries at large scale. For ports the problem starts here https://github.com/openstack/neutron/blob/master/neutron/db/db_base_plugin_common.py#L202-L219. The base dict is built from a single DB query row and then the processing of all extensions (which is the default behaviour) leads to a sequential series of additional queries per row to augment the dict. In our opinion, this causes issues from a performance perspective, it leads to the classic n+1 query anti-pattern and fundamentally does not scale (an alternate option would be to do a “joined” query with active extensions). This illustrates the type of workarounds that result from this approach https://github.com/openstack/neutron/blob/master/neutron/db/_utils.py#L95-L107. Instead of using native SQL to filter fields from the result the whole result reset has to be iterated to filter out fields, again surely this is an anti-pattern when processing DB objects. With respect to LBaaS support, we removed the intermediate model layer with this (and a couple of previous) commit(s) https://github.com/sapcc/neutron- lbaas/commit/f71867fbf6c8a27df43aaff6046948dce60f3081. This is just an interim change but after implementing this we saw LBAAS API requests going from > 1-5 minutes and degrading with # of objects to a consistent sub second response time. Version: This is/should be present in all versions, but our testing has been done in Mitaka and above. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1750890/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1748658] Re: Restarting Neutron containers which make use of network namespaces doesn't work
I believe the fix belongs to packaging / containers / tripleo and not neutron itself. ** Changed in: neutron Status: New => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1748658 Title: Restarting Neutron containers which make use of network namespaces doesn't work Status in neutron: Invalid Status in tripleo: Fix Released Bug description: When DHCP, L3, Metadata or OVN-Metadata containers are restarted they can't set the previous namespaces: [heat-admin@overcloud-novacompute-0 neutron]$ sudo docker restart 8559f5a7fa45 8559f5a7fa45 [heat-admin@overcloud-novacompute-0 neutron]$ tail -f /var/log/containers/neutron/networking-ovn-metadata-agent.log 2018-02-09 08:34:41.059 5 CRITICAL neutron [-] Unhandled error: ProcessExecutionError: Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Invalid argument 2018-02-09 08:34:41.059 5 ERROR neutron Traceback (most recent call last): 2018-02-09 08:34:41.059 5 ERROR neutron File "/usr/bin/networking-ovn-metadata-agent", line 10, in 2018-02-09 08:34:41.059 5 ERROR neutron sys.exit(main()) 2018-02-09 08:34:41.059 5 ERROR neutron File "/usr/lib/python2.7/site-packages/networking_ovn/cmd/eventlet/agents/metadata.py", line 17, in main 2018-02-09 08:34:41.059 5 ERROR neutron metadata_agent.main() 2018-02-09 08:34:41.059 5 ERROR neutron File "/usr/lib/python2.7/site-packages/networking_ovn/agent/metadata_agent.py", line 38, in main 2018-02-09 08:34:41.059 5 ERROR neutron agt.start() 2018-02-09 08:34:41.059 5 ERROR neutron File "/usr/lib/python2.7/site-packages/networking_ovn/agent/metadata/agent.py", line 147, in start 2018-02-09 08:34:41.059 5 ERROR neutron self.sync() 2018-02-09 08:34:41.059 5 ERROR neutron File "/usr/lib/python2.7/site-packages/networking_ovn/agent/metadata/agent.py", line 56, in wrapped 2018-02-09 08:34:41.059 5 ERROR neutron return f(*args, **kwargs) 2018-02-09 08:34:41.059 5 ERROR neutron File "/usr/lib/python2.7/site-packages/networking_ovn/agent/metadata/agent.py", line 169, in sync 2018-02-09 08:34:41.059 5 ERROR neutron metadata_namespaces = self.ensure_all_networks_provisioned() 2018-02-09 08:34:41.059 5 ERROR neutron File "/usr/lib/python2.7/site-packages/networking_ovn/agent/metadata/agent.py", line 350, in ensure_all_networks_provisioned 2018-02-09 08:34:41.059 5 ERROR neutron netns = self.provision_datapath(datapath) 2018-02-09 08:34:41.059 5 ERROR neutron File "/usr/lib/python2.7/site-packages/networking_ovn/agent/metadata/agent.py", line 294, in provision_datapath 2018-02-09 08:34:41.059 5 ERROR neutron veth_name[0], veth_name[1], namespace) 2018-02-09 08:34:41.059 5 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 182, in add_veth 2018-02-09 08:34:41.059 5 ERROR neutron self._as_root([], 'link', tuple(args)) 2018-02-09 08:34:41.059 5 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 94, in _as_root 2018-02-09 08:34:41.059 5 ERROR neutron namespace=namespace) 2018-02-09 08:34:41.059 5 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 102, in _execute 2018-02-09 08:34:41.059 5 ERROR neutron log_fail_as_error=self.log_fail_as_error) 2018-02-09 08:34:41.059 5 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 151, in execute 2018-02-09 08:34:41.059 5 ERROR neutron raise ProcessExecutionError(msg, returncode=returncode) 2018-02-09 08:34:41.059 5 ERROR neutron ProcessExecutionError: Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Invalid argument 2018-02-09 08:34:41.059 5 ERROR neutron 2018-02-09 08:34:41.059 5 ERROR neutron 2018-02-09 08:34:41.177 21 INFO oslo_service.service [-] Parent process has died unexpectedly, exiting 2018-02-09 08:34:41.178 21 INFO eventlet.wsgi.server [-] (21) wsgi exited, is_accepting=True An easy way to reproduce the bug: [heat-admin@overcloud-novacompute-0 ~]$ sudo docker exec -u root -it 5c5f254a9321bd74b5911f46acb9513574c2cd9a3c59805a85cffd960bcc864d /bin/bash [root@overcloud-novacompute-0 /]# ip netns a my_netns [root@overcloud-novacompute-0 /]# exit [heat-admin@overcloud-novacompute-0 ~]$ sudo ip netns [heat-admin@overcloud-novacompute-0 ~]$ sudo docker restart 5c5f254a9321bd74b5911f46acb9513574c2cd9a3c59805a85cffd960bcc864d 5c5f254a9321bd74b5911f46acb9513574c2cd9a3c59805a85cffd960bcc864d [heat-admin@overcloud-novacompute-0 ~]$ sudo docker exec -u root -it 5c5f254a9321bd74b5911f46acb9513574c2cd9a3c59805a85cffd960bcc864d /bin/bash [root@overcloud-novacompute-0 /]# ip netns RTNETLINK answers: Invalid argument RTNETLINK answers: Invalid argument my_netns
[Yahoo-eng-team] [Bug 1750546] Re: neutron-specs sphinx warning treated as error, <1.6.1 req not taken into account
Fixed in https://review.openstack.org/#/c/529031/ ** Tags added: gate-failure ** Changed in: neutron Status: New => Fix Released ** Changed in: neutron Importance: Undecided => High ** Changed in: neutron Assignee: (unassigned) => zhaobo (zhaobo6) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1750546 Title: neutron-specs sphinx warning treated as error, <1.6.1 req not taken into account Status in neutron: Fix Released Bug description: The neutron-specs docs is currently failing in master, preventing to merge anything (and get nice HTML previews in gerrit changes for new specs): 2018-02-19 13:31:49.216472 | ubuntu-xenial | Warning, treated as error: 2018-02-19 13:31:49.216571 | ubuntu-xenial | /home/zuul/src/git.openstack.org/openstack/neutron-specs/doc/source/specs/kilo/ml2-ovs-portsecurity.rst:370:Citation [ovs_firewall_driver] is not referenced. Sphinx 1.6.x has a warning on ununsed reference. For some reason, sphinx 1.6.5 ends up being used despite the "sphinx>=1.5.1,<1.6.1" line in requirements.txt: 2018-02-19 13:31:35.564024 | 2018-02-19 13:31:35.564265 | LOOP [sphinx : Run sphinx] 2018-02-19 13:31:36.535037 | ubuntu-xenial | Running Sphinx v1.6.5 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1750546/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1628996] Re: subnet service types not working with sqlite3 version 3.7.17
[vagrant@localhost ~]$ yum info sqlite Installed Packages Name: sqlite Arch: x86_64 Version : 3.7.17 Release : 8.el7 Size: 795 k Repo: installed >From repo : anaconda Summary : Library that implements an embeddable SQL database engine URL : http://www.sqlite.org/ License : Public Domain Description : SQLite is a C library that implements an SQL database engine. A large : subset of SQL92 is supported. A complete database is stored in a : single disk file. The API is designed for convenience and ease of use. : Applications that link against SQLite can enjoy the power and : flexibility of an SQL database without the administrative hassles of : supporting a separate database server. Version 2 and version 3 binaries : are named to permit each to be installed on a single host [vagrant@localhost neutron]$ tox -e py27 test_subnet_service_types py27: commands succeeded congratulations :) ** Changed in: neutron Status: New => Invalid ** Tags removed: timeout-abandon -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1628996 Title: subnet service types not working with sqlite3 version 3.7.17 Status in neutron: Invalid Bug description: subnet service types not working with sqlite3 version 3.7.17. But it works from sqlite3 version 3.8.0 and above versions. Because of this, subnet service type unit tests failing in sqlite3 version 3.7.17. Captured traceback: ~~~ Traceback (most recent call last): File "neutron/tests/base.py", line 125, in func return f(self, *args, **kwargs) File "neutron/tests/unit/extensions/test_subnet_service_types.py", line 245, in test_create_port_no_device_owner_no_fallback self.test_create_port_no_device_owner(fallback=False) File "neutron/tests/base.py", line 125, in func return f(self, *args, **kwargs) File "neutron/tests/unit/extensions/test_subnet_service_types.py", line 242, in test_create_port_no_device_owner self._assert_port_res(port, '', subnet, fallback) File "neutron/tests/unit/extensions/test_subnet_service_types.py", line 173, in _assert_port_res self.assertEqual(error, res['NeutronError']['type']) KeyError: 'NeutronError' _query_filter_service_subnets [1] is behaving differently in 3.7.17 and 3.8.0 for these tests [1] https://github.com/openstack/neutron/blob/master/neutron/db/ipam_backend_mixin.py#L597 I have seen this on centos7 setup, which by default uses sqlite3 version 3.7.17. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1628996/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1707003] Re: gate-tempest-dsvm-neutron-dvr-ha-multinode-full-ubuntu-xenial-nv job has a very high failure rate
** Changed in: neutron Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1707003 Title: gate-tempest-dsvm-neutron-dvr-ha-multinode-full-ubuntu-xenial-nv job has a very high failure rate Status in neutron: Fix Released Bug description: Looking at the Neutron Failure Rate dashboard, specifically the tempest jobs: http://grafana.openstack.org/dashboard/db/neutron-failure- rate?panelId=10 One can see the gate-tempest-dsvm-neutron-dvr-ha-multinode-full- ubuntu-xenial-nv job has a very high failure rate, over 90% for the past 5 days. Matt Riedemann did an analysis (which I'll paste below), but the summary is that setup of the 3-node job is failing a lot, and not discovering the third node, leading to a failure when that node is attempted to be used. So the first step is to change devstack-gate (?) code to wait for all the subnodes to show up from a Nova perspective before proceeding. There was a previous attempt at a grenade change in https://review.openstack.org/#/c/426310/ that was abandoned, but that seems like a good start based on the analysis. Matt's comment #1: Looking at the gate-tempest-dsvm-neutron-dvr-ha-multinode-full-ubuntu- xenial-nv job failure, the subnode-2 and subnode-3 all look OK as far as their config. They use the same values for nova-cpu.conf pointing at the nova_cell1 MQ which points at the cell1 conductor and cell1 database. I see that the compute nodes are created for both subnode-2 and subnode-3 *after* discover_hosts runs: 2017-07-25 15:06:55.991684 | + /opt/stack/new/devstack-gate/devstack- vm-gate.sh:main:L777: discover_hosts Jul 25 15:06:58.945371 ubuntu-xenial-3-node-rax-iad-10067333-744503 nova-compute[794]: INFO nova.compute.resource_tracker [None req- f69c76bf-0263-494b-8257-61617c90d799 None None] Compute node record created for ubuntu-xenial-3-node-rax-iad-10067333-744503:ubuntu- xenial-3-node-rax-iad-10067333-744503 with uuid: 1788fe0b-496c-4eda- b03a-2cf4a2733a94 Jul 25 15:07:02.323379 ubuntu-xenial-3-node-rax-iad-10067333-744504 nova-compute[827]: INFO nova.compute.resource_tracker [None req- 95419fec-a2a7-467f-b167-d83755273a7a None None] Compute node record created for ubuntu-xenial-3-node-rax-iad-10067333-744504:ubuntu- xenial-3-node-rax-iad-10067333-744504 with uuid: ae3420a1-20d2-42a1 -909d-fc9cf1b14248 And looking at the discover_hosts output, only subnode-2 is discovered as the unmapped host: http://logs.openstack.org/56/477556/5/experimental/gate-tempest-dsvm- neutron-dvr-ha-multinode-full-ubuntu-xenial-nv/432c235/logs/devstack- gate-discover-hosts.txt.gz The compute node from the primary host is discovered and mapped to cell1 as part of the devstack run on the primary host: http://logs.openstack.org/56/477556/5/experimental/gate-tempest-dsvm- neutron-dvr-ha-multinode-full-ubuntu-xenial- nv/432c235/logs/devstacklog.txt.gz#_2017-07-25_14_50_45_831 So it seems that we are simply getting lucky by discovering the compute node from subnode-2 and mapping it to cell1 but missing the compute node from subnode-3, so it doesn't get mapped and then things fail when Tempest tries to use it. This could be a problem on any 3 node job, and might not just be related to this devstack change. Matt's comment #2: I've gone through the dvr-ha 3-node job failure and it just appears to be a latent issue that we could also hit in 2-node jobs, and I even noticed in a 2-node job that when the subnode compute node is created it actually happens after we start running discover_hosts from the primary via devstack-gate, so it just seems to be a race window, which we already have, and maybe expose more in 3-node jobs if they are slower, or slow down the traffic on the control node. If you look at the cells v2 setup guide, it even says to make sure the computes are created before running discover_hosts: https://docs.openstack.org/nova/latest/user/cells.html "Configure and start your compute hosts. Before step 7, make sure you have compute hosts in the database by running nova service-list --binary nova-compute." Step 7 is running 'nova-manage cell_v2 discover_hosts'. Ideally what we should be doing is have devstack-gate pass a variable to the discover_hosts.sh script in devstack telling it how many compute services we expect (3 in the case of the dvr-ha job) and then have that discover_hosts.sh script run nova-compute service-list --binary nova-compute and count the results until the expected number is hit, or it times out, but then run discover_hosts. That's really what we expect from other deployment tools like triple-o and kolla. But overall I'm not finding anything in this change that's killing these jobs outright, so let's get it in. Matt's comment
[Yahoo-eng-team] [Bug 1742385] Re: test_create_show_delete_security_group_rule_names failure on networking-midonet gate
** Changed in: neutron
Status: Fix Committed => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1742385
Title:
test_create_show_delete_security_group_rule_names failure on
networking-midonet gate
Status in networking-midonet:
Fix Released
Status in neutron:
Fix Released
Bug description:
test_create_show_delete_security_group_rule_names is failing on
networking-midonet gate jobs.
it seems like a neutron-lib version mismatch.
1.11.0 is used by neutron.
on the other hand, 1.12.0 is used by tempest plugins installed via
TEMPEST_PLUGINS, which doesn't honor upper-constraints.
i guess tempest plugins should not use the list of protocols from neutron-lib
because tempest is branchless and supposed to work against old servers (with
older neutron-lib) and other implementations (like midonet) as well.
note: "ipip" is added by neutron-lib 1.12.0
(I18e5e42b687e12b64f5a9c523a912c8dd1afa9d2)
eg. http://logs.openstack.org/91/531091/3/check/networking-midonet-
tempest-multinode-ml2/1503639/logs/testr_results.html.gz
Traceback (most recent call last):
File
"/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/neutron_tempest_plugin/api/test_security_groups.py",
line 80, in test_create_show_delete_security_group_rule_names
ethertype=self.ethertype)
File
"/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/neutron_tempest_plugin/api/base_security_groups.py",
line 88, in _test_create_show_delete_security_group_rule
self._create_security_group_rule(**kwargs)['security_group_rule'])
File
"/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/neutron_tempest_plugin/api/base_security_groups.py",
line 58, in _create_security_group_rule
rule_create_body = self.client.create_security_group_rule(**kwargs)
File
"/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/neutron_tempest_plugin/services/network/json/network_client.py",
line 865, in create_security_group_rule
resp, body = self.post(uri, body)
File "tempest/lib/common/rest_client.py", line 279, in post
return self.request('POST', url, extra_headers, headers, body, chunked)
File "tempest/lib/common/rest_client.py", line 668, in request
self._error_checker(resp, resp_body)
File "tempest/lib/common/rest_client.py", line 779, in _error_checker
raise exceptions.BadRequest(resp_body, resp=resp)
tempest.lib.exceptions.BadRequest: Bad request
Details: {u'type': u'SecurityGroupRuleInvalidProtocol', u'message':
u"Security group rule protocol ipip not supported. Only protocol values [None,
'ah', 'pgm', 'tcp', 'ipv6-encap', 'dccp', 'igmp', 'icmp', 'esp', 'ipv6-icmp',
'vrrp', 'gre', 'sctp', 'rsvp', 'ipv6-route', 'udp', 'ipv6-opts', 'ipv6-nonxt',
'udplite', 'egp', 'icmpv6', 'ipv6-frag', 'ospf'] and integer representations [0
to 255] are supported.", u'detail': u''}
To manage notifications about this bug go to:
https://bugs.launchpad.net/networking-midonet/+bug/1742385/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1748546] Re: ovsfw tempest tests random fails
** Changed in: neutron Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1748546 Title: ovsfw tempest tests random fails Status in neutron: Fix Released Bug description: Some random tests from neutron-tempest-ovsfw job are failing very often. Only common thing in those failing tests is that test fails because instance is not reachable and it looks that it is not reachable because of some missing firewall Openflow rules. Example of failing tests can be found e.g. on: http://logs.openstack.org/57/542257/4/check/neutron-tempest- ovsfw/cf76bcb/logs/testr_results.html.gz or on http://logs.openstack.org/54/537654/4/check/neutron-tempest- ovsfw/5c90b2b/logs/testr_results.html.gz (except hard_reboot test which is different problem and is covered in https://bugs.launchpad.net/neutron/+bug/1747709) To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1748546/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1753209] Re: neutron_tempest_plugin.api.admin.test_shared_network_extension.RBACSharedNetworksTest, rbac policy in use across tenants.
So the reason of failure here is that one test case uses a network that
was temporarily shared with all tenants while running a RBAC test case.
When the network is for a short span of time shared with everyone, the
other test case -
VolumesSnapshotTestJSON:test_snapshot_create_delete_with_volume_in_use -
creates an instance with the following request body:
Body: {"server": {"flavorRef": "e443576a-50cd-4023-88e0-574b1ec1726e",
"name": "tempest-VolumesSnapshotTestJSON-instance-1302082485", "imageRef":
"9a98ccb7-cd30-43da-85a9-5cfd8126c5ae"}}
which, as you can see, doesn't specify network to use, so nova then apparently
picks one of available networks to boot the instance on, and it happens to be
the network that was momentarily shared in RBAC test case.
I don't think that the Volumes test case should rely on nova to pick a
network for them, and instead pre-create a tenant network for this
specific test case and then boot the instance on it. By doing so, we
will avoid the race condition between those test cases b/c Volumes test
case won't touch the shared network and create VIF ports on it.
This is probably a bug for cinder folks to solve since the offending
test case is in VolumesSnapshot class.
One thing to mull over on neutron side though is whether the rbac test
case, as written, could reduce its impact. One thing to consider is that
the network that is momentarily shared with everyone will pop up for all
tenants, incl. those unaware of tempest running in background. Since
some operators execute tempest periodically against their cloud, they
probably don't want their customers to get random networks popping up
for a brief moment. Though the RBAC test case explicitly validates the
wildcard RBAC behavior so not sure if we can easily get rid of it w/o
loosing some coverage.
How does tempest core team look at such cases? Do we forbid any impact
on other cloud users? If so, we could probably shorten the neutron api
case to avoid it, even if it means a particular use case won't be
covered with tempest.
** Also affects: neutron
Importance: Undecided
Status: New
** Also affects: tempest
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1753209
Title:
neutron_tempest_plugin.api.admin.test_shared_network_extension.RBACSharedNetworksTest,
rbac policy in use across tenants.
Status in neutron:
New
Status in tempest:
New
Status in tripleo:
Triaged
Bug description:
neutron_tempest_plugin.api.admin.test_shared_network_extension.RBACSharedNetworksTest
failure
https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-
centos-7-ovb-1ctlr_1comp-featureset020-master/6cec620/tempest.html.gz
Details: {u'message': u'RBAC policy on object 3cfbd0a7-84f2-4e3f-917e-
bf51b5995e20 cannot be removed because other objects depend on
it.\nDetails: Callback
neutron.plugins.ml2.plugin.Ml2Plugin.validate_network_rbac_policy_change
--9223372036850840529 failed with "Unable to reconfigure sharing
settings for network 3cfbd0a7-84f2-4e3f-917e-bf51b5995e20. Multiple
tenants are using it.",Callback
neutron.services.network_ip_availability.plugin.NetworkIPAvailabilityPlugin.validate_network_rbac_policy_change
--9223372036853400817 failed with "Unable to reconfigure sharing
settings for network 3cfbd0a7-84f2-4e3f-917e-bf51b5995e20. Multiple
tenants are using it.",Callback
neutron.services.network_ip_availability.plugin.NetworkIPAvailabilityPlugin.validate_network_rbac_policy_change
--9223372036853463713 failed with "Unable to reconfigure sharing
settings for network 3cfbd0a7-84f2-4e3f-917e-bf51b5995e20. Multiple
tenants are using it."', u'type': u'RbacPolicyInUse', u'detail': u''}
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1753209/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1744983] Re: coverage job fails with timeout
** Changed in: neutron
Status: Confirmed => Fix Released
** Changed in: neutron
Assignee: (unassigned) => Ihar Hrachyshka (ihar-hrachyshka)
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1744983
Title:
coverage job fails with timeout
Status in neutron:
Fix Released
Bug description:
It happened in Ocata: http://logs.openstack.org/41/532941/1/gate
/openstack-tox-cover/5a26283/job-
output.txt.gz#_2018-01-22_23_39_21_734729
2018-01-22 23:02:09.046252 | ubuntu-xenial | load_tests =
test_base.optimize_db_test_loader(__file__)
2018-01-22 23:39:21.734729 | RUN END RESULT_TIMED_OUT: [untrusted :
git.openstack.org/openstack-infra/zuul-jobs/playbooks/tox/run.yaml@master]
It took at least 37 minutes for coverage report to run.
In a good run from the same branch, we can see that the same tox run
took ~18 minutes: http://logs.openstack.org/61/532461/1/gate
/openstack-tox-cover/2ebf0d6/job-
output.txt.gz#_2018-01-17_10_27_36_867873
2018-01-17 10:27:36.867873 | ubuntu-xenial | load_tests =
test_base.optimize_db_test_loader(__file__)
2018-01-17 10:45:02.053109 | ubuntu-xenial |
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
There are no leads in logs. I am also not sure whether the failure
affects other branches.
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1744983/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1745638] Re: OVN Check for router port correctness
** Changed in: neutron Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1745638 Title: OVN Check for router port correctness Status in neutron: Fix Released Bug description: This bug is created to track the merging of https://review.openstack.org/#/c/534423/ during Queens RC1 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1745638/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1745192] [NEW] Can't create a vlan transparent network in mixed driver environment
Public bug reported: If I use several ml2 drivers, f.e. openvswitch and cisco, one that supports vlan-transparent extension and another that doesn't, then I can't create a vlan-transparent network at all, because of check_vlan_transparency ml2 driver manager behavior. What we should do instead of requiring that all drivers support the feature is: 1. always allow to create such networks; 2. fail on port binding attempt for a driver that doesn't support the feature; neutron-server will then try the next driver in line, and hopefully will succeed. There may also be some work on nova scheduler side to make sure that a transparent port lands on an appropriate node, but that's a separate issue to follow up. Note: once the bug is fixed, we should be able to safely revert https://review.openstack.org/#/c/359919/ that arguably was a mistake, since SR-IOV driver doesn't support the feature, and would require additional configuration on libvirt side, as described in https://serverfault.com/questions/805992/is-vlan-tagging-on-guest- allowed-in-sr-iov-on-kvm Another candidate to revert check_vlan_transparency from True to False is linuxbridge driver, that I believe also doesn't support the feature (but someone with more knowledge about the driver should check it first.) ** Affects: neutron Importance: Wishlist Status: Confirmed ** Tags: linuxbridge low-hanging-fruit rfe sriov-pci-pt ** Changed in: neutron Importance: Undecided => Wishlist ** Changed in: neutron Status: New => Confirmed ** Tags added: low-hanging-fruit ** Tags added: linuxbridge sriov-pci-pt ** Tags added: rfe -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1745192 Title: Can't create a vlan transparent network in mixed driver environment Status in neutron: Confirmed Bug description: If I use several ml2 drivers, f.e. openvswitch and cisco, one that supports vlan-transparent extension and another that doesn't, then I can't create a vlan-transparent network at all, because of check_vlan_transparency ml2 driver manager behavior. What we should do instead of requiring that all drivers support the feature is: 1. always allow to create such networks; 2. fail on port binding attempt for a driver that doesn't support the feature; neutron-server will then try the next driver in line, and hopefully will succeed. There may also be some work on nova scheduler side to make sure that a transparent port lands on an appropriate node, but that's a separate issue to follow up. Note: once the bug is fixed, we should be able to safely revert https://review.openstack.org/#/c/359919/ that arguably was a mistake, since SR-IOV driver doesn't support the feature, and would require additional configuration on libvirt side, as described in https://serverfault.com/questions/805992/is-vlan-tagging-on-guest- allowed-in-sr-iov-on-kvm Another candidate to revert check_vlan_transparency from True to False is linuxbridge driver, that I believe also doesn't support the feature (but someone with more knowledge about the driver should check it first.) To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1745192/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1744363] Re: unit test jobs often time out
neutron-lib bumped for Queens. Considering it done. Feel free to reopen if it resurfaces. ** Changed in: neutron Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1744363 Title: unit test jobs often time out Status in neutron: Fix Released Bug description: One suspect is deprecation messages that usually slow down runs and popped up in numbers lately. We should clean those up in attempt to speed up the jobs. If it doesn't work, we may consider bumping time for the job, though it's undesirable. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1744363/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1744983] [NEW] coverage job fails with timeout
Public bug reported:
It happened in Ocata: http://logs.openstack.org/41/532941/1/gate
/openstack-tox-cover/5a26283/job-
output.txt.gz#_2018-01-22_23_39_21_734729
2018-01-22 23:02:09.046252 | ubuntu-xenial | load_tests =
test_base.optimize_db_test_loader(__file__)
2018-01-22 23:39:21.734729 | RUN END RESULT_TIMED_OUT: [untrusted :
git.openstack.org/openstack-infra/zuul-jobs/playbooks/tox/run.yaml@master]
It took at least 37 minutes for coverage report to run.
In a good run from the same branch, we can see that the same tox run
took ~18 minutes: http://logs.openstack.org/61/532461/1/gate/openstack-
tox-cover/2ebf0d6/job-output.txt.gz#_2018-01-17_10_27_36_867873
2018-01-17 10:27:36.867873 | ubuntu-xenial | load_tests =
test_base.optimize_db_test_loader(__file__)
2018-01-17 10:45:02.053109 | ubuntu-xenial |
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
There are no leads in logs. I am also not sure whether the failure
affects other branches.
** Affects: neutron
Importance: Medium
Status: Confirmed
** Tags: gate-failure unittest
** Changed in: neutron
Importance: Undecided => Medium
** Changed in: neutron
Status: New => Confirmed
** Tags added: gate-failure unittest
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1744983
Title:
coverage job fails with timeout
Status in neutron:
Confirmed
Bug description:
It happened in Ocata: http://logs.openstack.org/41/532941/1/gate
/openstack-tox-cover/5a26283/job-
output.txt.gz#_2018-01-22_23_39_21_734729
2018-01-22 23:02:09.046252 | ubuntu-xenial | load_tests =
test_base.optimize_db_test_loader(__file__)
2018-01-22 23:39:21.734729 | RUN END RESULT_TIMED_OUT: [untrusted :
git.openstack.org/openstack-infra/zuul-jobs/playbooks/tox/run.yaml@master]
It took at least 37 minutes for coverage report to run.
In a good run from the same branch, we can see that the same tox run
took ~18 minutes: http://logs.openstack.org/61/532461/1/gate
/openstack-tox-cover/2ebf0d6/job-
output.txt.gz#_2018-01-17_10_27_36_867873
2018-01-17 10:27:36.867873 | ubuntu-xenial | load_tests =
test_base.optimize_db_test_loader(__file__)
2018-01-17 10:45:02.053109 | ubuntu-xenial |
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
There are no leads in logs. I am also not sure whether the failure
affects other branches.
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1744983/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1660612] Re: gate-tempest-dsvm-neutron-linuxbridge-ubuntu-xenial times out on execution
Other jobs, not just linuxbridge, are also affected. For example, DVR/pike job here: http://logs.openstack.org/08/533308/1/check/neutron- tempest-dvr/bfc4761/ara/ ** Summary changed: - gate-tempest-dsvm-neutron-linuxbridge-ubuntu-xenial times out on execution + Tempest full jobs time out on execution ** Tags removed: linuxbridge ** Also affects: tempest Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1660612 Title: Tempest full jobs time out on execution Status in neutron: Confirmed Status in tempest: New Bug description: It took a bit above 1 hour to run tempest with linux bridge agent and then the job was terminated: http://logs.openstack.org/47/416647/4/check/gate-tempest-dsvm-neutron- linuxbridge-ubuntu- xenial/7d1d4e5/console.html#_2017-01-30_17_51_50_343819 e-r-q: http://logstash.openstack.org/#dashboard/file/logstash.json?query=build_name %3Agate-tempest-dsvm-neutron-linuxbridge-ubuntu- xenial%20AND%20message%3A%5C%22Killed%5C%22%20AND%20message%3A%5C%22timeout%20-s%209%5C%22%20AND%20tags%3Aconsole To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1660612/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1744396] [NEW] Split fullstack test_securitygroup test case into small pieces
Public bug reported: This is the description of the test case: """Tests if a security group rules are working, by confirming that 0. traffic is allowed when port security is disabled, 1. connection from outside of allowed security group is blocked 2. connection from allowed security group is permitted 3. traffic not explicitly allowed (eg. ICMP) is blocked, 4. a security group update takes effect, 5. a remote security group member addition works, and 6. an established connection stops by deleting a SG rule. 7. multiple overlapping remote rules work, 8. test other protocol functionality by using SCTP protocol 9. test two vms with same mac on the same host in different networks It shouldn't be like that. Each scenario should live in its own test case. Otherwise whenever something fails, it's hard to assess which step got us into broken state. The test case should be split into pieces. ** Affects: neutron Importance: Wishlist Status: Confirmed ** Tags: fullstack low-hanging-fruit ** Changed in: neutron Importance: Undecided => Wishlist ** Changed in: neutron Status: New => Confirmed ** Tags added: low-hanging-fruit -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1744396 Title: Split fullstack test_securitygroup test case into small pieces Status in neutron: Confirmed Bug description: This is the description of the test case: """Tests if a security group rules are working, by confirming that 0. traffic is allowed when port security is disabled, 1. connection from outside of allowed security group is blocked 2. connection from allowed security group is permitted 3. traffic not explicitly allowed (eg. ICMP) is blocked, 4. a security group update takes effect, 5. a remote security group member addition works, and 6. an established connection stops by deleting a SG rule. 7. multiple overlapping remote rules work, 8. test other protocol functionality by using SCTP protocol 9. test two vms with same mac on the same host in different networks It shouldn't be like that. Each scenario should live in its own test case. Otherwise whenever something fails, it's hard to assess which step got us into broken state. The test case should be split into pieces. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1744396/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1744394] [NEW] test_distributed_port_binding_deleted_by_port_deletion failed due to warnings during port binding delete
Public bug reported: http://logs.openstack.org/56/534456/1/check/openstack-tox- py35/151aefe/testr_results.html.gz (Queens) ft1.6: neutron.tests.unit.plugins.ml2.test_db.Ml2DvrDBTestCase.test_distributed_port_binding_deleted_by_port_deletion_StringException: Traceback (most recent call last): File "/home/zuul/src/git.openstack.org/openstack/neutron/neutron/tests/base.py", line 132, in func return f(self, *args, **kwargs) File "/home/zuul/src/git.openstack.org/openstack/neutron/neutron/tests/unit/plugins/ml2/test_db.py", line 438, in test_distributed_port_binding_deleted_by_port_deletion self.assertEqual([], warning_list) File "/home/zuul/src/git.openstack.org/openstack/neutron/.tox/py35/lib/python3.5/site-packages/testtools/testcase.py", line 411, in assertEqual self.assertThat(observed, matcher, message) File "/home/zuul/src/git.openstack.org/openstack/neutron/.tox/py35/lib/python3.5/site-packages/testtools/testcase.py", line 498, in assertThat raise mismatch_error testtools.matchers._impl.MismatchError: !=: reference = [] actual= [, ] It's not clear which warnings those objects are (we may extend the test to expose those on failure). The failure itself may be related to OVO adoption for port bindings that landed lately: https://review.openstack.org/407868 ** Affects: neutron Importance: High Status: Confirmed ** Tags: gate-failure unittest ** Changed in: neutron Importance: Undecided => High ** Changed in: neutron Status: New => Confirmed -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1744394 Title: test_distributed_port_binding_deleted_by_port_deletion failed due to warnings during port binding delete Status in neutron: Confirmed Bug description: http://logs.openstack.org/56/534456/1/check/openstack-tox- py35/151aefe/testr_results.html.gz (Queens) ft1.6: neutron.tests.unit.plugins.ml2.test_db.Ml2DvrDBTestCase.test_distributed_port_binding_deleted_by_port_deletion_StringException: Traceback (most recent call last): File "/home/zuul/src/git.openstack.org/openstack/neutron/neutron/tests/base.py", line 132, in func return f(self, *args, **kwargs) File "/home/zuul/src/git.openstack.org/openstack/neutron/neutron/tests/unit/plugins/ml2/test_db.py", line 438, in test_distributed_port_binding_deleted_by_port_deletion self.assertEqual([], warning_list) File "/home/zuul/src/git.openstack.org/openstack/neutron/.tox/py35/lib/python3.5/site-packages/testtools/testcase.py", line 411, in assertEqual self.assertThat(observed, matcher, message) File "/home/zuul/src/git.openstack.org/openstack/neutron/.tox/py35/lib/python3.5/site-packages/testtools/testcase.py", line 498, in assertThat raise mismatch_error testtools.matchers._impl.MismatchError: !=: reference = [] actual= [, ] It's not clear which warnings those objects are (we may extend the test to expose those on failure). The failure itself may be related to OVO adoption for port bindings that landed lately: https://review.openstack.org/407868 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1744394/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1744359] Re: Neutron haproxy logs are not being collected
I guess having haproxy prefix is backwards compatible enough to allow it to slip like that. ** Also affects: tripleo Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1744359 Title: Neutron haproxy logs are not being collected Status in devstack: New Status in neutron: In Progress Status in tripleo: New Bug description: In Neutron, we use haproxy to proxy metadata requests from instances to Nova Metadata service. By default, haproxy logs to /dev/log but in Ubuntu, those requests get redirected by rsyslog to /var/log/haproxy.log which is not being collected. ubuntu@devstack:~$ cat /etc/rsyslog.d/49-haproxy.conf # Create an additional socket in haproxy's chroot in order to allow logging via # /dev/log to chroot'ed HAProxy processes $AddUnixListenSocket /var/lib/haproxy/dev/log # Send HAProxy messages to a dedicated logfile if $programname startswith 'haproxy' then /var/log/haproxy.log &~ Another possibility would be to change the haproxy.cfg file to include the log-tag option so that haproxy uses a different tag [0] and then it'd be dumped into syslog instead but this would break backwards compatibility. [0] https://cbonte.github.io/haproxy-dconv/configuration-1.5.html#3.1 -log-tag To manage notifications about this bug go to: https://bugs.launchpad.net/devstack/+bug/1744359/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1744363] [NEW] unit test jobs often time out
Public bug reported: One suspect is deprecation messages that usually slow down runs and popped up in numbers lately. We should clean those up in attempt to speed up the jobs. If it doesn't work, we may consider bumping time for the job, though it's undesirable. ** Affects: neutron Importance: High Status: Confirmed ** Tags: gate-failure unittest ** Tags added: gate-failure unittest ** Changed in: neutron Status: New => Confirmed ** Changed in: neutron Importance: Undecided => High -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1744363 Title: unit test jobs often time out Status in neutron: Confirmed Bug description: One suspect is deprecation messages that usually slow down runs and popped up in numbers lately. We should clean those up in attempt to speed up the jobs. If it doesn't work, we may consider bumping time for the job, though it's undesirable. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1744363/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1744361] [NEW] test_server_security_groups failed to reboot with "Domain not found: no domain with matching uuid" because of missing vif event from linuxbridge agent
Public bug reported:
This happened in master (Queens) in tempest-linuxbridge job:
http://logs.openstack.org/56/534456/1/check/neutron-tempest-
linuxbridge/7693ca5/logs/testr_results.html.gz
>From tempest log:
Traceback (most recent call last):
File "tempest/api/compute/security_groups/test_security_groups.py", line 105,
in test_server_security_groups
'ACTIVE')
File "tempest/common/waiters.py", line 96, in wait_for_server_status
raise lib_exc.TimeoutException(message)
tempest.lib.exceptions.TimeoutException: Request timed out
Details: (SecurityGroupsTestJSON:test_server_security_groups) Server
dd2ec3b0-f909-4104-ae2a-9b878d936ed4 failed to reach ACTIVE status and task
state "None" within the required time (196 s). Current status: HARD_REBOOT.
Current task state: reboot_started_hard.
In nova-compute log, we see this:
Jan 19 10:38:37.869397 ubuntu-xenial-rax-iad-0002010825 nova-compute[28907]:
WARNING nova.virt.libvirt.driver [None req-838e0ebe-afe5-4cca-abcc-4ba116889cba
service nova] [instance: dd2ec3b0-f909-4104-ae2a-9b878d936ed4] Timeout waiting
for vif plugging callback for instance with vm_state active and task_state
reboot_started_hard.: Timeout: 300 seconds
Jan 19 10:38:37.880789 ubuntu-xenial-rax-iad-0002010825 nova-compute[28907]:
DEBUG nova.compute.manager [None req-838e0ebe-afe5-4cca-abcc-4ba116889cba
service nova] [instance: dd2ec3b0-f909-4104-ae2a-9b878d936ed4] Checking state
{{(pid=28907) _get_power_state
/opt/stack/new/nova/nova/compute/manager.py:1183}}
Jan 19 10:38:37.885265 ubuntu-xenial-rax-iad-0002010825 nova-compute[28907]:
ERROR nova.compute.manager [None req-838e0ebe-afe5-4cca-abcc-4ba116889cba
service nova] [instance: dd2ec3b0-f909-4104-ae2a-9b878d936ed4] Cannot reboot
instance: Domain not found: no domain with matching uuid
'dd2ec3b0-f909-4104-ae2a-9b878d936ed4' (instance-0023): libvirtError:
Domain not found: no domain with matching uuid
'dd2ec3b0-f909-4104-ae2a-9b878d936ed4' (instance-0023)
Jan 19 10:38:37.981484 ubuntu-xenial-rax-iad-0002010825 nova-compute[28907]:
DEBUG nova.compute.manager [None req-838e0ebe-afe5-4cca-abcc-4ba116889cba
service nova] [instance: dd2ec3b0-f909-4104-ae2a-9b878d936ed4] Instance has
been destroyed from under us while trying to set it to ERROR {{(pid=28907)
_set_instance_obj_error_state /opt/stack/new/nova/nova/compute/manager.py:586}}
Jan 19 10:38:38.150498 ubuntu-xenial-rax-iad-0002010825 nova-compute[28907]:
ERROR oslo_messaging.rpc.server [None req-838e0ebe-afe5-4cca-abcc-4ba116889cba
service nova] Exception during message handling: libvirtError: Domain not
found: no domain with matching uuid 'dd2ec3b0-f909-4104-ae2a-9b878d936ed4'
(instance-0023)
Jan 19 10:38:38.150813 ubuntu-xenial-rax-iad-0002010825 nova-compute[28907]:
ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Jan 19 10:38:38.151125 ubuntu-xenial-rax-iad-0002010825 nova-compute[28907]:
ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line
163, in _process_incoming
Jan 19 10:38:38.151388 ubuntu-xenial-rax-iad-0002010825 nova-compute[28907]:
ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
Jan 19 10:38:38.151668 ubuntu-xenial-rax-iad-0002010825 nova-compute[28907]:
ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line
220, in dispatch
Jan 19 10:38:38.151939 ubuntu-xenial-rax-iad-0002010825 nova-compute[28907]:
ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method,
ctxt, args)
Jan 19 10:38:38.152205 ubuntu-xenial-rax-iad-0002010825 nova-compute[28907]:
ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line
190, in _do_dispatch
Jan 19 10:38:38.152491 ubuntu-xenial-rax-iad-0002010825 nova-compute[28907]:
ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
Jan 19 10:38:38.152705 ubuntu-xenial-rax-iad-0002010825 nova-compute[28907]:
ERROR oslo_messaging.rpc.server File
"/opt/stack/new/nova/nova/exception_wrapper.py", line 76, in wrapped
Jan 19 10:38:38.152962 ubuntu-xenial-rax-iad-0002010825 nova-compute[28907]:
ERROR oslo_messaging.rpc.server function_name, call_dict, binary)
Jan 19 10:38:38.153268 ubuntu-xenial-rax-iad-0002010825 nova-compute[28907]:
ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in
__exit__
Jan 19 10:38:38.153481 ubuntu-xenial-rax-iad-0002010825 nova-compute[28907]:
ERROR oslo_messaging.rpc.server self.force_reraise()
Jan 19 10:38:38.153710 ubuntu-xenial-rax-iad-0002010825 nova-compute[28907]:
ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in
force_reraise
Jan 19 10:38:38.153942 ubuntu-xenial-rax-iad-0002010825 nova-compute[28907]:
ERROR oslo_messaging.rpc.server
[Yahoo-eng-team] [Bug 1743658] [NEW] SLAAC address incorrectly deallocated from HA router port due to race condition
Public bug reported:
This was originally reported in Red Hat Bugzilla:
https://bugzilla.redhat.com/show_bug.cgi?id=1486324
The issue is triggered when executing
tempest.scenario.test_network_v6.TestGettingAddress tests in a loop with
L3 HA enabled. The failure looks as follows:
Captured traceback:
~~~
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/tempest/common/utils/__init__.py",
line 89, in wrapper
return f(*func_args, **func_kwargs)
File
"/usr/lib/python2.7/site-packages/tempest/scenario/test_network_v6.py", line
258, in test_dualnet_multi_prefix_slaac
dualnet=True)
File
"/usr/lib/python2.7/site-packages/tempest/scenario/test_network_v6.py", line
196, in _prepare_and_test
(ip, srv['id'], ssh.exec_command("ip address")))
File "/usr/lib/python2.7/site-packages/unittest2/case.py", line 666, in
fail
raise self.failureException(msg)
AssertionError: Address 2003::1:f816:3eff:fee0:fbf0 not configured for
instance d89f1b14-20ef-47f7-80a4-9d3173446dbc, ip address output is
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc pfifo_fast qlen
1000
link/ether fa:16:3e:12:f5:ea brd ff:ff:ff:ff:ff:ff
inet 10.100.0.8/28 brd 10.100.0.15 scope global eth0
inet6 fe80::f816:3eff:fe12:f5ea/64 scope link
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen
1000
link/ether fa:16:3e:e0:fb:f0 brd ff:ff:ff:ff:ff:ff
inet6 2003::f816:3eff:fee0:fbf0/64 scope global dynamic
valid_lft 86335sec preferred_lft 14335sec
inet6 fe80::f816:3eff:fee0:fbf0/64 scope link
valid_lft forever preferred_lft forever
The test case creates a network with two ipv6 slaac subnets, then starts
an instance on the network and checks that the instance OS configured
addresses from both prefixes. It fails because an address from the
second prefix is not configured.
When we check l3 agent log, we see that radvd first is correctly
configured for both prefixes, but then something happens that
reconfigures radvd again, now without one of prefixes.
If we trace the event that triggered the second radvd reconfiguration
back to server, we see that the router update happened as a result of
update_routers_states execution (which is itself remotely triggered by
l3 agent).
We see that the update_routers_states call started before the second
subnet was added to the router in question. At the same time, we see
that the call is complete AFTER the subnet is added.
In server log, we see both allocation and deallocation events for router
gateway address:
2018-01-16 19:50:31.459 886987 DEBUG neutron.db.db_base_plugin_common
[req-13cbe23b-ae7f-472e-9049-601e75e04b6a
269a2421f89742b09cfd722dc28aca5c 97e8b9e11107489aad70e7e6d172ddce -
default default] Allocated IP 2003:0:0:1::1
(d7bd8a16-6bf6-4a40-b508-c27d963b3912/fc6a62f5-d1a1-4272-aa67-a70ee8b1ee01/6f4b11b3-4a31-4264
-8c3b-439c8bd903bd) _store_ip_allocation /usr/lib/python2.7/site-
packages/neutron/db/db_base_plugin_common.py:122
2018-01-16 19:50:35.144 883810 DEBUG neutron.db.db_base_plugin_common
[req-445e4d27-16e4-463f-bfce-b7603a739b01 - - - - -] Delete allocated IP
2003:0:0:1::1
(d7bd8a16-6bf6-4a40-b508-c27d963b3912/fc6a62f5-d1a1-4272-aa67-a70ee8b1ee01)
_delete_ip_allocation
/usr/lib/python2.7/site-packages/neutron/db/db_base_plugin_common.py:108
The allocation event belongs to add_router_interface, while deletion is
from update_routers_states.
Code inspection suggests that deallocation happens because
update_routers_states does the following:
1. fetch all router ports;
2. then for each port payload, set host, and pass the payload into update_port.
If add_router_interface happened in between those two steps, then we
risk calling update_port with a port payload that DOESN'T contain a
fixed_ip that was added during add_router_interface call.
I think we should avoid passing the whole port payload into update_port,
instead just pass a dict with a single key of host. This is both
semantically correct, fixes the race condition, and in theory may be
slightly quicker since the core plugin won't need to process fields that
were not changed.
** Affects: neutron
Importance: High
Assignee: Ihar Hrachyshka (ihar-hrachyshka)
Status: Confirmed
** Tags: l3-ha
** Changed in: neutron
Status: New => Confirmed
** Changed in: neutron
Importance: Undecided => High
** Changed in: neutron
Assignee: (unassigned) => Ihar Hrachyshka (ihar-hrachyshka)
** Tags added: l3-ha
--
You received thi
[Yahoo-eng-team] [Bug 1743463] [NEW] linuxbridge scenario job fails with: "AttributeError: 'LinuxbridgeAgentExtensionAPI' object has no attribute 'request_int_br'"
Public bug reported: log extension is not compatible with linuxbridge agent. We should, short term, avoid configuring it there; and long term, make it compatible with the agent. (It should be doable because the agent uses the same iptables firewall). ** Affects: neutron Importance: High Status: Confirmed -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1743463 Title: linuxbridge scenario job fails with: "AttributeError: 'LinuxbridgeAgentExtensionAPI' object has no attribute 'request_int_br'" Status in neutron: Confirmed Bug description: log extension is not compatible with linuxbridge agent. We should, short term, avoid configuring it there; and long term, make it compatible with the agent. (It should be doable because the agent uses the same iptables firewall). To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1743463/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1345341] Re: radvd needs functional tests
We have some functional and fullstack tests that cover the functionality. I think at this point the bug is vague and moot and can be closed. If you have specific ideas on what's missing, feel free to reopen / repurpose the bug / report a new bug. ** Changed in: neutron Status: Confirmed => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1345341 Title: radvd needs functional tests Status in neutron: Fix Released Bug description: See the review comments for https://review.openstack.org/102648 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1345341/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1742200] [NEW] dvr scenario job fails with timeouts
Public bug reported: The job routinely takes almost 3h to execute, which results in job timeout. This may be related to Meltdown bug patched in infra. Anyway, we should strive to speed up the job, because an alternative would be splitting the job into parts, and that's not ideal (more jobs in gate means higher spurious failure rate). We currently use concurrency=2 for the job. We should try to increase it by removing the cap, and see if it helps. ** Affects: neutron Importance: Undecided Status: New ** Tags: gate-failure ** Tags added: gate-failure -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1742200 Title: dvr scenario job fails with timeouts Status in neutron: New Bug description: The job routinely takes almost 3h to execute, which results in job timeout. This may be related to Meltdown bug patched in infra. Anyway, we should strive to speed up the job, because an alternative would be splitting the job into parts, and that's not ideal (more jobs in gate means higher spurious failure rate). We currently use concurrency=2 for the job. We should try to increase it by removing the cap, and see if it helps. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1742200/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1742197] [NEW] linuxbridge scenario job doesn't execute NetworkMtuBaseTest
Public bug reported:
This is because the job is not configured to access vxlan / gre type
drivers:
2018-01-08 16:40:26.894303 | primary | {1} setUpClass
(neutron_tempest_plugin.scenario.test_mtu.NetworkMtuBaseTest) ...
SKIPPED: GRE or VXLAN type_driver is not enabled
Linuxbridge doesn't support GRE but VXLAN should work.
** Affects: neutron
Importance: Medium
Status: Confirmed
** Tags: linuxbridge low-hanging-fruit
** Tags added: linuxbridge
** Tags added: low-hanging-fruit
** Changed in: neutron
Importance: Undecided => Medium
** Changed in: neutron
Status: New => Confirmed
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1742197
Title:
linuxbridge scenario job doesn't execute NetworkMtuBaseTest
Status in neutron:
Confirmed
Bug description:
This is because the job is not configured to access vxlan / gre type
drivers:
2018-01-08 16:40:26.894303 | primary | {1} setUpClass
(neutron_tempest_plugin.scenario.test_mtu.NetworkMtuBaseTest) ...
SKIPPED: GRE or VXLAN type_driver is not enabled
Linuxbridge doesn't support GRE but VXLAN should work.
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1742197/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1741954] [NEW] create_and_list_trunk_subports rally scenario failed with timeouts
Public bug reported:
This happened once in Pike.
http://logs.openstack.org/16/529616/1/check/legacy-rally-dsvm-neutron-
neutron/72b511a/rally-
plot/results.html.gz#/NeutronTrunks.create_and_list_trunk_subports/failures
Traceback (most recent call last):
File "/opt/stack/new/rally/rally/task/runner.py", line 71, in
_run_scenario_once
getattr(scenario_inst, method_name)(**scenario_kwargs)
File "/home/zuul/.rally/plugins/plugins/trunk_scenario.py", line 41, in run
trunk = self._create_trunk(trunk_payload)
File "/opt/stack/new/rally/rally/task/atomic.py", line 91, in
func_atomic_actions
f = func(self, *args, **kwargs)
File "/home/zuul/.rally/plugins/plugins/trunk_scenario.py", line 53, in
_create_trunk
return self.clients("neutron").create_trunk({'trunk': trunk_payload})
File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py",
line 2067, in create_trunk
return self.post(self.trunks_path, body=body)
File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py",
line 357, in post
headers=headers, params=params)
File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py",
line 280, in do_request
resp, replybody = self.httpclient.do_request(action, method, body=body)
File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line
342, in do_request
return self.request(url, method, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line
330, in request
resp = super(SessionClient, self).request(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line
192, in request
return self.session.request(url, method, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/positional/__init__.py", line
108, in inner
return wrapped(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line
703, in request
resp = send(**kwargs)
File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line
768, in _send_request
raise exceptions.ConnectTimeout(msg)
ConnectTimeout: Request to http://104.239.168.166:9696/v2.0/trunks timed out
All four attempts failed the same way. Other scenarios worked just fine.
When you look at q-svc log, you see that all four POST requests for
trunks took a very long time, for example, req-b98d273b-
8c38-4d54-98f5-1d74008795d8 took 3:30 minutes to get from quotas check
to precommit_create notification:
http://logs.openstack.org/16/529616/1/check/legacy-rally-dsvm-neutron-
neutron/72b511a/logs/screen-q-svc.txt.gz?#_Jan_05_19_46_52_971085
The only major code between quota check and this notification is
validation of the request. Requests are, granted, heavy loaded, with 125
subports per trunk, so it may have something to do with it.
** Affects: neutron
Importance: High
Status: Confirmed
** Tags: gate-failure
** Tags added: gate-failure
** Changed in: neutron
Importance: Undecided => High
** Changed in: neutron
Status: New => Confirmed
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1741954
Title:
create_and_list_trunk_subports rally scenario failed with timeouts
Status in neutron:
Confirmed
Bug description:
This happened once in Pike.
http://logs.openstack.org/16/529616/1/check/legacy-rally-dsvm-neutron-
neutron/72b511a/rally-
plot/results.html.gz#/NeutronTrunks.create_and_list_trunk_subports/failures
Traceback (most recent call last):
File "/opt/stack/new/rally/rally/task/runner.py", line 71, in
_run_scenario_once
getattr(scenario_inst, method_name)(**scenario_kwargs)
File "/home/zuul/.rally/plugins/plugins/trunk_scenario.py", line 41, in run
trunk = self._create_trunk(trunk_payload)
File "/opt/stack/new/rally/rally/task/atomic.py", line 91, in
func_atomic_actions
f = func(self, *args, **kwargs)
File "/home/zuul/.rally/plugins/plugins/trunk_scenario.py", line 53, in
_create_trunk
return self.clients("neutron").create_trunk({'trunk': trunk_payload})
File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py",
line 2067, in create_trunk
return self.post(self.trunks_path, body=body)
File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py",
line 357, in post
headers=headers, params=params)
File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py",
line 280, in do_request
resp, replybody = self.httpclient.do_request(action, method, body=body)
File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line
342, in do_request
return self.request(url, method, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line
330, in request
resp = super(SessionClient, self).request(*args, **kwargs)
File
[Yahoo-eng-team] [Bug 1737011] [NEW] ServerActionsTestJSON.test_reboot_server_hard failed to ssh into instance
Public bug reported: http://logs.openstack.org/83/526183/1/gate/legacy-tempest-dsvm- py35/166f0c9/logs/testr_results.html.gz It happened on master (Queens). The last messages in the instance console log are: [3.395758] Freeing unused kernel memory: 928k freed [3.418579] Write protecting the kernel read-only data: 12288k [3.525176] Freeing unused kernel memory: 1596k freed [3.589609] Freeing unused kernel memory: 1184k freed info: initramfs: up at 3.71 So it seems that no getty was started on tty*. Nothing clearly suspicious in nova-cpu service log or syslog. ** Affects: nova Importance: Undecided Status: New ** Tags: gate-failure ** Tags added: gate-failure -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1737011 Title: ServerActionsTestJSON.test_reboot_server_hard failed to ssh into instance Status in OpenStack Compute (nova): New Bug description: http://logs.openstack.org/83/526183/1/gate/legacy-tempest-dsvm- py35/166f0c9/logs/testr_results.html.gz It happened on master (Queens). The last messages in the instance console log are: [3.395758] Freeing unused kernel memory: 928k freed [3.418579] Write protecting the kernel read-only data: 12288k [3.525176] Freeing unused kernel memory: 1596k freed [3.589609] Freeing unused kernel memory: 1184k freed info: initramfs: up at 3.71 So it seems that no getty was started on tty*. Nothing clearly suspicious in nova-cpu service log or syslog. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1737011/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1675910] Re: segment event transaction semantics are wrong
Reopened since we reverted the fix. ** Changed in: neutron Status: Fix Released => Confirmed -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1675910 Title: segment event transaction semantics are wrong Status in neutron: Confirmed Bug description: _delete_segments_for_network is currently being called inside of a transaction, which results in all of the BEFORE/PRECOMMIT/AFTER events for the segments themselves being inside of a transaction. This makes them all effectively PRECOMMIT in the database lifecycle which violates the semantics we've assigned to them. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1675910/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1731948] Re: Wrong OVO classes registered in some cases
Added oslo.versionedobjects to the list of affected projects so that it allows to use custom registries for consuming projects. ** Also affects: oslo.versionedobjects Importance: Undecided Status: New ** Changed in: oslo.versionedobjects Importance: Undecided => Wishlist ** Tags added: gate-failure unittest ** Changed in: neutron Importance: Medium => High -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1731948 Title: Wrong OVO classes registered in some cases Status in neutron: In Progress Status in oslo.versionedobjects: New Bug description: When patch https://review.openstack.org/#/c/321001 was merged some UT in projects like networking-midonet started failing. It is reported on https://bugs.launchpad.net/networking-midonet/+bug/1731623 It looks that reason of this problem is that wrong OVO classes are registered in case when e.g. 2 different projects uses same names of OVO objects. I checked it little bit and it looks that neutron.objects.subnet.Subnet has got registered os_vif.objects.route.Route object instead of neutron.objects.subnet.Route, see my logs from one exampled test: http://paste.openstack.org/show/626170/ To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1731948/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1673531] Re: fullstack test_controller_timeout_does_not_break_connectivity_sigkill(GRE and l2pop, openflow-native_ovsdb-cli) failure
Still fails: http://logs.openstack.org/71/520371/7/check/legacy-neutron- dsvm-fullstack/ad585a2/logs/testr_results.html.gz ** Changed in: neutron Status: Fix Released => Confirmed ** Changed in: neutron Importance: Undecided => High ** Changed in: neutron Assignee: (unassigned) => Ihar Hrachyshka (ihar-hrachyshka) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1673531 Title: fullstack test_controller_timeout_does_not_break_connectivity_sigkill(GRE and l2pop,openflow-native_ovsdb-cli) failure Status in neutron: Confirmed Bug description: Logs for failure: http://logs.openstack.org/98/446598/1/check/gate- neutron-dsvm-fullstack-ubuntu-xenial/2e0f93e/logs/dsvm-fullstack- logs/TestOvsConnectivitySameNetworkOnOvsBridgeControllerStop .test_controller_timeout_does_not_break_connectivity_sigkill_GRE-and- l2pop,openflow-native_ovsdb-cli_/ logstash query: http://logstash.openstack.org/#dashboard/file/logstash.json?query=messge%3A%5C%22in%20test_controller_timeout_does_not_break_connectivity_sigkill%5C%22%20AND%20tags%3Aconsole%20AND%20build_name %3Agate-neutron-dsvm-fullstack-ubuntu-xenial To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1673531/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1706750] Re: FlushError for automatic IPAllocations when subnet create is retried
We still see this failure happening in Red Hat CI with the same symptoms
for OSP12 (Pike), reopening.
** Changed in: neutron
Status: Fix Released => Confirmed
** Bug watch added: Red Hat Bugzilla #1506668
https://bugzilla.redhat.com/show_bug.cgi?id=1506668
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1706750
Title:
FlushError for automatic IPAllocations when subnet create is retried
Status in neutron:
Confirmed
Bug description:
The error is seen in Red Hat CI:
https://bugzilla.redhat.com/show_bug.cgi?id=1470784
There are hits in Kibana too:
http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22FlushError%3A%20New%20instance%20%3CIPAllocation%20at%5C%22
In server.log:
2017-07-13 15:20:51.064 85185 ERROR neutron.pecan_wsgi.hooks.translation
[req-3c97d83e-3beb-4b28-a29f-697fc1ee8bc6 5f2e0d72ef8e4fb18b500759cfe5e503
205f2a3630a0426f8da20b4467ea003b - default default] POST failed.: FlushError:
New instance with identity key (, ('2003::1:f816:3eff:fe3f:aa00',
'8032a1cd-9236-4126-83fa-2f039ed2b890',
'c0542b84-bcd0-48e1-ace1-6da1f50246dc')) conflicts with persistent instance
2017-07-13 15:20:51.064 85185 ERROR neutron.pecan_wsgi.hooks.translation
Traceback (most recent call last):
2017-07-13 15:20:51.064 85185 ERROR neutron.pecan_wsgi.hooks.translation
File "/usr/lib/python2.7/site-packages/pecan/core.py", line 678, in __call__
2017-07-13 15:20:51.064 85185 ERROR neutron.pecan_wsgi.hooks.translation
self.invoke_controller(controller, args, kwargs, state)
2017-07-13 15:20:51.064 85185 ERROR neutron.pecan_wsgi.hooks.translation
File "/usr/lib/python2.7/site-packages/pecan/core.py", line 569, in
invoke_controller
2017-07-13 15:20:51.064 85185 ERROR neutron.pecan_wsgi.hooks.translation
result = controller(*args, **kwargs)
2017-07-13 15:20:51.064 85185 ERROR neutron.pecan_wsgi.hooks.translation
File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 94, in wrapped
2017-07-13 15:20:51.064 85185 ERROR neutron.pecan_wsgi.hooks.translation
setattr(e, '_RETRY_EXCEEDED', True)
2017-07-13 15:20:51.064 85185 ERROR neutron.pecan_wsgi.hooks.translation
File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in
__exit__
2017-07-13 15:20:51.064 85185 ERROR neutron.pecan_wsgi.hooks.translation
self.force_reraise()
2017-07-13 15:20:51.064 85185 ERROR neutron.pecan_wsgi.hooks.translation
File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2017-07-13 15:20:51.064 85185 ERROR neutron.pecan_wsgi.hooks.translation
six.reraise(self.type_, self.value, self.tb)
2017-07-13 15:20:51.064 85185 ERROR neutron.pecan_wsgi.hooks.translation
File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 90, in wrapped
2017-07-13 15:20:51.064 85185 ERROR neutron.pecan_wsgi.hooks.translation
return f(*args, **kwargs)
2017-07-13 15:20:51.064 85185 ERROR neutron.pecan_wsgi.hooks.translation
File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 150, in wrapper
2017-07-13 15:20:51.064 85185 ERROR neutron.pecan_wsgi.hooks.translation
ectxt.value = e.inner_exc
2017-07-13 15:20:51.064 85185 ERROR neutron.pecan_wsgi.hooks.translation
File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in
__exit__
2017-07-13 15:20:51.064 85185 ERROR neutron.pecan_wsgi.hooks.translation
self.force_reraise()
2017-07-13 15:20:51.064 85185 ERROR neutron.pecan_wsgi.hooks.translation
File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2017-07-13 15:20:51.064 85185 ERROR neutron.pecan_wsgi.hooks.translation
six.reraise(self.type_, self.value, self.tb)
2017-07-13 15:20:51.064 85185 ERROR neutron.pecan_wsgi.hooks.translation
File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 138, in wrapper
2017-07-13 15:20:51.064 85185 ERROR neutron.pecan_wsgi.hooks.translation
return f(*args, **kwargs)
2017-07-13 15:20:51.064 85185 ERROR neutron.pecan_wsgi.hooks.translation
File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 129, in wrapped
2017-07-13 15:20:51.064 85185 ERROR neutron.pecan_wsgi.hooks.translation
LOG.debug("Retry wrapper got retriable exception: %s", e)
2017-07-13 15:20:51.064 85185 ERROR neutron.pecan_wsgi.hooks.translation
File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in
__exit__
2017-07-13 15:20:51.064 85185 ERROR neutron.pecan_wsgi.hooks.translation
self.force_reraise()
2017-07-13 15:20:51.064 85185 ERROR neutron.pecan_wsgi.hooks.translation
File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2017-07-13 15:20:51.064 85185 ERROR neutron.pecan_wsgi.hooks.translation
six.reraise(self.type_, self.value, self.tb)
2017-07-13
[Yahoo-eng-team] [Bug 1671634] Re: [RFE] Allow to set MTU for networks
** Changed in: neutron Status: In Progress => Won't Fix ** Changed in: neutron Status: Won't Fix => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1671634 Title: [RFE] Allow to set MTU for networks Status in neutron: Fix Released Bug description: ATM neutron does allow to configure MTU for networks to reflect underlying infrastructure, but only to operators, and only by changing configuration options. Ideally, users would be allowed to modify MTU for networks (to simplify matters, on creation only, though we can also look at resource updates) to accommodate for custom workloads relying on specific MTUs. Or maybe sometimes users want to get consistent MTUs across all their networks instead of having different MTUs based on network type backing their networks. Both of those use cases would be served by allowing PUT for 'mtu' network attribute. I guess it will require a fake extension to signal the change in behavior, even while the implementation may still lay in existing plugin code handling MTUs. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1671634/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1722644] [NEW] TrunkTest fails for OVS/DVR scenario job
Public bug reported:
http://logs.openstack.org/13/474213/7/check/gate-tempest-dsvm-neutron-
dvr-multinode-scenario-ubuntu-xenial-
nv/752030f/logs/testr_results.html.gz
Traceback (most recent call last):
File "/opt/stack/new/neutron/neutron/tests/tempest/scenario/test_trunk.py",
line 91, in _detach_and_delete_trunk
server['id'], trunk['port_id'])
File "tempest/lib/services/compute/interfaces_client.py", line 55, in
delete_interface
port_id))
File "tempest/lib/common/rest_client.py", line 301, in delete
return self.request('DELETE', url, extra_headers, headers, body)
File "tempest/lib/services/compute/base_compute_client.py", line 48, in
request
method, url, extra_headers, headers, body, chunked)
File "tempest/lib/common/rest_client.py", line 659, in request
self._error_checker(resp, resp_body)
File "tempest/lib/common/rest_client.py", line 780, in _error_checker
raise exceptions.Conflict(resp_body, resp=resp)
tempest.lib.exceptions.Conflict: An object with that identifier already exists
Details: {u'code': 409, u'message': u"Cannot 'detach_interface' instance
bc3e8d26-d392-4d66-91c4-b10223e19184 while it is in vm_state building"}
}}}
traceback-2: {{{
Traceback (most recent call last):
File "/opt/stack/new/neutron/neutron/tests/tempest/scenario/test_trunk.py",
line 91, in _detach_and_delete_trunk
server['id'], trunk['port_id'])
File "tempest/lib/services/compute/interfaces_client.py", line 55, in
delete_interface
port_id))
File "tempest/lib/common/rest_client.py", line 301, in delete
return self.request('DELETE', url, extra_headers, headers, body)
File "tempest/lib/services/compute/base_compute_client.py", line 48, in
request
method, url, extra_headers, headers, body, chunked)
File "tempest/lib/common/rest_client.py", line 659, in request
self._error_checker(resp, resp_body)
File "tempest/lib/common/rest_client.py", line 780, in _error_checker
raise exceptions.Conflict(resp_body, resp=resp)
tempest.lib.exceptions.Conflict: An object with that identifier already exists
Details: {u'code': 409, u'message': u"Cannot 'detach_interface' instance
1ba5eedf-c64e-4304-8547-7fdcdff2c1d2 while it is in vm_state error"}
}}}
Traceback (most recent call last):
File "/opt/stack/new/neutron/neutron/tests/tempest/scenario/test_trunk.py",
line 164, in test_trunk_subport_lifecycle
self._wait_for_server(server)
File "/opt/stack/new/neutron/neutron/tests/tempest/scenario/test_trunk.py",
line 143, in _wait_for_server
constants.SERVER_STATUS_ACTIVE)
File "tempest/common/waiters.py", line 76, in wait_for_server_status
server_id=server_id)
tempest.exceptions.BuildErrorException: Server
1ba5eedf-c64e-4304-8547-7fdcdff2c1d2 failed to build and is in ERROR status
Details: {u'created': u'2017-10-10T19:03:25Z', u'code': 500, u'message':
u'Build of instance 1ba5eedf-c64e-4304-8547-7fdcdff2c1d2 aborted: Failed to
allocate the network(s), not rescheduling.'}
** Affects: neutron
Importance: High
Status: Confirmed
** Tags: gate-failure tempest trunk
** Changed in: neutron
Importance: Undecided => Critical
** Changed in: neutron
Importance: Critical => High
** Changed in: neutron
Status: New => Confirmed
** Tags added: gate-failure tempest trunk
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1722644
Title:
TrunkTest fails for OVS/DVR scenario job
Status in neutron:
Confirmed
Bug description:
http://logs.openstack.org/13/474213/7/check/gate-tempest-dsvm-neutron-
dvr-multinode-scenario-ubuntu-xenial-
nv/752030f/logs/testr_results.html.gz
Traceback (most recent call last):
File "/opt/stack/new/neutron/neutron/tests/tempest/scenario/test_trunk.py",
line 91, in _detach_and_delete_trunk
server['id'], trunk['port_id'])
File "tempest/lib/services/compute/interfaces_client.py", line 55, in
delete_interface
port_id))
File "tempest/lib/common/rest_client.py", line 301, in delete
return self.request('DELETE', url, extra_headers, headers, body)
File "tempest/lib/services/compute/base_compute_client.py", line 48, in
request
method, url, extra_headers, headers, body, chunked)
File "tempest/lib/common/rest_client.py", line 659, in request
self._error_checker(resp, resp_body)
File "tempest/lib/common/rest_client.py", line 780, in _error_checker
raise exceptions.Conflict(resp_body, resp=resp)
tempest.lib.exceptions.Conflict: An object with that identifier already exists
Details: {u'code': 409, u'message': u"Cannot 'detach_interface' instance
bc3e8d26-d392-4d66-91c4-b10223e19184 while it is in vm_state building"}
}}}
traceback-2: {{{
Traceback (most recent call last):
File "/opt/stack/new/neutron/neutron/tests/tempest/scenario/test_trunk.py",
line 91, in _detach_and_delete_trunk
server['id'], trunk['port_id'])
[Yahoo-eng-team] [Bug 1719711] [NEW] iptables failed to apply when binding a port with AGENT.debug_iptables_rules enabled
Public bug reported:
http://logs.openstack.org/21/504021/2/check/gate-tempest-dsvm-neutron-
scenario-linuxbridge-ubuntu-xenial-nv/e47a3f3/testr_results.html.gz
Traceback (most recent call last):
File
"/opt/stack/new/neutron/neutron/tests/tempest/scenario/test_security_groups.py",
line 127, in test_two_sec_groups
num_servers=1, security_groups=security_groups_list)
File
"/opt/stack/new/neutron/neutron/tests/tempest/scenario/test_security_groups.py",
line 54, in create_vm_testing_sec_grp
const.SERVER_STATUS_ACTIVE)
File "tempest/common/waiters.py", line 76, in wait_for_server_status
server_id=server_id)
tempest.exceptions.BuildErrorException: Server
e1120d99-f0eb-43eb-a38b-847843a838b5 failed to build and is in ERROR status
Details: {u'message': u'Build of instance e1120d99-f0eb-43eb-a38b-847843a838b5
aborted: Failed to allocate the network(s), not rescheduling.', u'code': 500,
u'created': u'2017-09-26T09:23:42Z'}
In linuxbridge agent log: http://logs.openstack.org/21/504021/2/check
/gate-tempest-dsvm-neutron-scenario-linuxbridge-ubuntu-xenial-
nv/e47a3f3/logs/screen-q-agt.txt.gz?level=TRACE#_Sep_26_09_16_30_623747
Sep 26 09:16:30.623747 ubuntu-xenial-ovh-gra1-11134533
neutron-linuxbridge-agent[24363]: ERROR neutron.agent.linux.iptables_manager
[None req-78fc6bc1-a089-4d5f-91d8-e5191e45978c None None] IPTables Rules did
not converge. Diff: # Generated by iptables_manager
Sep 26 09:16:30.623936 ubuntu-xenial-ovh-gra1-11134533
neutron-linuxbridge-agent[24363]: *filter
Sep 26 09:16:30.624117 ubuntu-xenial-ovh-gra1-11134533
neutron-linuxbridge-agent[24363]: -D neutron-linuxbri-ibc1a22b9-e 6
Sep 26 09:16:30.624316 ubuntu-xenial-ovh-gra1-11134533
neutron-linuxbridge-agent[24363]: -I neutron-linuxbri-ibc1a22b9-e 6 -p 1 -j
RETURN
Sep 26 09:16:30.624482 ubuntu-xenial-ovh-gra1-11134533
neutron-linuxbridge-agent[24363]: COMMIT
Sep 26 09:16:30.624955 ubuntu-xenial-ovh-gra1-11134533
neutron-linuxbridge-agent[24363]: # Completed by iptables_manager
Sep 26 09:16:30.635308 ubuntu-xenial-ovh-gra1-11134533
neutron-linuxbridge-agent[24363]: ERROR
neutron.plugins.ml2.drivers.agent._common_agent [None
req-78fc6bc1-a089-4d5f-91d8-e5191e45978c None None] Error in agent loop.
Devices info: {'current': set(['tapbc1a22b9-ef', 'tapc9488f0f-ae',
'tape2d2e245-96', 'tap93881b27-41', 'tapb265ee77-37', 'tapbadc6b64-69',
'tapa813220a-1d', 'tapa376782a-75', 'tap395ccf4d-c9', 'tapca94a412-e7',
'tap58f740f2-aa', 'tapb2444941-9f']), 'timestamps': {'tap93881b27-41': 56,
'tapc9488f0f-ae': 62, 'tape2d2e245-96': 11, 'tapbc1a22b9-ef': 68,
'tapb265ee77-37': 9, 'tapbadc6b64-69': 55, 'tapa813220a-1d': 66,
'tapa376782a-75': 65, 'tap395ccf4d-c9': 67, 'tapca94a412-e7': 6,
'tap58f740f2-aa': 59, 'tapb2444941-9f': 10}, 'removed': set([]), 'added':
set([]), 'updated': set([])}: IpTablesApplyException: IPTables Rules did not
converge. Diff: # Generated by iptables_manager
Sep 26 09:16:30.636316 ubuntu-xenial-ovh-gra1-11134533
neutron-linuxbridge-agent[24363]: *filter
Sep 26 09:16:30.636510 ubuntu-xenial-ovh-gra1-11134533
neutron-linuxbridge-agent[24363]: -D neutron-linuxbri-ibc1a22b9-e 6
Sep 26 09:16:30.636700 ubuntu-xenial-ovh-gra1-11134533
neutron-linuxbridge-agent[24363]: -I neutron-linuxbri-ibc1a22b9-e 6 -p 1 -j
RETURN
Sep 26 09:16:30.636898 ubuntu-xenial-ovh-gra1-11134533
neutron-linuxbridge-agent[24363]: COMMIT
Sep 26 09:16:30.637075 ubuntu-xenial-ovh-gra1-11134533
neutron-linuxbridge-agent[24363]: # Completed by iptables_manager
Sep 26 09:16:30.637269 ubuntu-xenial-ovh-gra1-11134533
neutron-linuxbridge-agent[24363]: ERROR
neutron.plugins.ml2.drivers.agent._common_agent Traceback (most recent call
last):
Sep 26 09:16:30.637683 ubuntu-xenial-ovh-gra1-11134533
neutron-linuxbridge-agent[24363]: ERROR
neutron.plugins.ml2.drivers.agent._common_agent File
"/opt/stack/new/neutron/neutron/plugins/ml2/drivers/agent/_common_agent.py",
line 453, in daemon_loop
Sep 26 09:16:30.637962 ubuntu-xenial-ovh-gra1-11134533
neutron-linuxbridge-agent[24363]: ERROR
neutron.plugins.ml2.drivers.agent._common_agent sync =
self.process_network_devices(device_info)
Sep 26 09:16:30.638211 ubuntu-xenial-ovh-gra1-11134533
neutron-linuxbridge-agent[24363]: ERROR
neutron.plugins.ml2.drivers.agent._common_agent File
"/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 157, in
wrapper
Sep 26 09:16:30.638373 ubuntu-xenial-ovh-gra1-11134533
neutron-linuxbridge-agent[24363]: ERROR
neutron.plugins.ml2.drivers.agent._common_agent result = f(*args, **kwargs)
Sep 26 09:16:30.638538 ubuntu-xenial-ovh-gra1-11134533
neutron-linuxbridge-agent[24363]: ERROR
neutron.plugins.ml2.drivers.agent._common_agent File
"/opt/stack/new/neutron/neutron/plugins/ml2/drivers/agent/_common_agent.py",
line 200, in process_network_devices
Sep 26 09:16:30.638728 ubuntu-xenial-ovh-gra1-11134533
neutron-linuxbridge-agent[24363]: ERROR
neutron.plugins.ml2.drivers.agent._common_agent
[Yahoo-eng-team] [Bug 1710589] Re: rally sla failure / internal error on load
** Changed in: neutron
Importance: High => Undecided
** Changed in: neutron
Importance: Undecided => Medium
** Changed in: neutron
Importance: Medium => Undecided
** No longer affects: neutron
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1710589
Title:
rally sla failure / internal error on load
Status in networking-midonet:
New
Bug description:
On networking-midonet gate, for following scenarios,
max_seconds_per_iteration validation is failing miserably.
These scenarios are currently disabled for an unrelated issue [1] but there
is an attempt to revive it. [2]
IIRC, these scenarios used to work. I don't remember when it was though.
(Maybe newton timeframe)
NetworkPlugin.create_ports
NetworkPlugin.create_routers
NetworkPlugin.create_subnets
NetworkPlugin.create_subnets_routers_interfaces
[1] bug 1670577
[2] https://review.openstack.org/#/c/492374/
It's probably due to retriable exceptions like the following due to
contentions.
eg. http://logs.openstack.org/74/492374/3/check/gate-networking-
midonet-rally-dsvm-ml2-ubuntu-xenial/d1aa9b9/logs/screen-q-svc.txt.gz
Aug 14 06:08:16.188519 ubuntu-xenial-citycloud-kna1-10421138 neutron-
server[14391]: DEBUG neutron.db.api [None req-
6c986099-0e8d-4181-a488-c73d348f2ef5 c_rally_1a0448a8_y9LQrXNX
c_rally_1a0448a8_QH0dDRH4] Retry wrapper got retriable exception:
UPDATE statement on table 'standardattributes' expected to update 1
row(s); 0 were matched. {{(pid=14479) wrapped
/opt/stack/new/neutron/neutron/db/api.py:129}}
Aug 14 05:46:27.279500 ubuntu-xenial-citycloud-kna1-10421138 neutron-
server[14391]: DEBUG neutron.db.api [None req-6134c462-06e4-493f-
8ff1-9b206dfda3dd c_rally_edf0ce7f_CaBvNcQe c_rally_edf0ce7f_ONZDWSHE]
Retry wrapper got retriable exception: Failed to create a duplicate
IpamAllocation: for attribute(s) ['PRIMARY'] with value(s)
2.224.0.54-29522f59-ea05-4904-b034-fa3555da8ade {{(pid=14479) wrapped
/opt/stack/new/neutron/neutron/db/api.py:129}}
Aug 14 05:39:37.960137 ubuntu-xenial-citycloud-kna1-10421138 neutron-
server[14391]: DEBUG neutron.db.api [None req-f3ecc8f3-2c7a-4b3c-ac8c-
abe2766e36f4 c_rally_6464adb0_pXKwRzJz c_rally_6464adb0_uyMbSZqP]
Retry wrapper got retriable exception: Failed to create a duplicate
DefaultSecurityGroup: for attribute(s) ['PRIMARY'] with value(s)
bc3b6a26a56646e7b098b9419d17c0d1 {{(pid=14480) wrapped
/opt/stack/new/neutron/neutron/db/api.py:129}}
Even without max_seconds_per_iteration, it sometimes times out,
or causes internal error reaching to the retry limit.
eg. http://logs.openstack.org/74/492374/4/check/gate-networking-
midonet-rally-dsvm-ml2-ubuntu-
xenial/5f6c27d/logs/screen-q-svc.txt.gz?level=TRACE
Aug 14 13:54:59.896408 ubuntu-xenial-citycloud-la1-10426313
neutron-server[13990]: ERROR oslo_db.api [None
req-c66e03cb-f36a-494b-9f83-fcb51dd0a420 c_rally_732dd379_PVser3Et
c_rally_732dd379_0ywBv9Oh] DB exceeded retry limit.: StaleDataError: UPDATE
statement on table 'standardattributes' expected to update 1 row(s); 0 were
matched.
Aug 14 13:54:59.896764 ubuntu-xenial-citycloud-la1-10426313
neutron-server[13990]: ERROR oslo_db.api Traceback (most recent call last):
Aug 14 13:54:59.896968 ubuntu-xenial-citycloud-la1-10426313
neutron-server[13990]: ERROR oslo_db.api File
"/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 138, in wrapper
Aug 14 13:54:59.897135 ubuntu-xenial-citycloud-la1-10426313
neutron-server[13990]: ERROR oslo_db.api return f(*args, **kwargs)
Aug 14 13:54:59.897300 ubuntu-xenial-citycloud-la1-10426313
neutron-server[13990]: ERROR oslo_db.api File
"/opt/stack/new/neutron/neutron/db/api.py", line 129, in wrapped
Aug 14 13:54:59.897465 ubuntu-xenial-citycloud-la1-10426313
neutron-server[13990]: ERROR oslo_db.api LOG.debug("Retry wrapper got
retriable exception: %s", e)
Aug 14 13:54:59.897653 ubuntu-xenial-citycloud-la1-10426313
neutron-server[13990]: ERROR oslo_db.api File
"/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in
__exit__
Aug 14 13:54:59.897819 ubuntu-xenial-citycloud-la1-10426313
neutron-server[13990]: ERROR oslo_db.api self.force_reraise()
Aug 14 13:54:59.897970 ubuntu-xenial-citycloud-la1-10426313
neutron-server[13990]: ERROR oslo_db.api File
"/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in
force_reraise
Aug 14 13:54:59.898135 ubuntu-xenial-citycloud-la1-10426313
neutron-server[13990]: ERROR oslo_db.api six.reraise(self.type_,
self.value, self.tb)
Aug 14 13:54:59.898300 ubuntu-xenial-citycloud-la1-10426313
neutron-server[13990]: ERROR oslo_db.api File
"/opt/stack/new/neutron/neutron/db/api.py", line 125, in wrapped
Aug 14 13:54:59.898458 ubuntu-xenial-citycloud-la1-10426313
neutron-server[13990]: ERROR oslo_db.api return
[Yahoo-eng-team] [Bug 1683365] Re: test_rule_update_forbidden_for_regular_tenants_own_policy fails with NotFound
** Changed in: neutron
Status: Confirmed => Invalid
** Tags removed: gate-failure
** Tags removed: qos
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1683365
Title:
test_rule_update_forbidden_for_regular_tenants_own_policy fails with
NotFound
Status in neutron:
Invalid
Bug description:
http://logs.openstack.org/06/440806/5/gate/gate-neutron-dsvm-api-
ubuntu-xenial/6d9b1d2/testr_results.html.gz
pythonlogging:'': {{{
2017-04-17 07:24:06,647 21614 INFO [tempest.lib.common.rest_client]
Request
(QosBandwidthLimitRuleTestJSON:test_rule_update_forbidden_for_regular_tenants_own_policy):
201 POST http://15.184.66.59:9696/v2.0/qos/policies 0.156s
2017-04-17 07:24:06,647 21614 DEBUG[tempest.lib.common.rest_client]
Request - Headers: {'Accept': 'application/json', 'X-Auth-Token': '',
'Content-Type': 'application/json'}
Body: {"policy": {"description": "test policy", "shared": false,
"tenant_id": "dae0db2239d040a7a24a2548e7745010", "name": "test-policy"}}
Response - Headers: {u'connection': 'close', 'status': '201',
u'x-openstack-request-id': 'req-f18fe708-dcc6-4e0a-9ec8-ea1f1e93e409',
u'content-type': 'application/json', 'content-location':
'http://15.184.66.59:9696/v2.0/qos/policies', u'content-length': '318',
u'date': 'Mon, 17 Apr 2017 07:24:06 GMT'}
Body:
{"policy":{"name":"test-policy","rules":[],"tenant_id":"dae0db2239d040a7a24a2548e7745010","created_at":"2017-04-17T07:24:06Z","updated_at":"2017-04-17T07:24:06Z","revision_number":1,"shared":false,"project_id":"dae0db2239d040a7a24a2548e7745010","id":"275ede54-e091-4615-8feb-1171c76c9f86","description":"test
policy"}}
2017-04-17 07:24:07,162 21614 INFO [tempest.lib.common.rest_client]
Request
(QosBandwidthLimitRuleTestJSON:test_rule_update_forbidden_for_regular_tenants_own_policy):
201 POST
http://15.184.66.59:9696/v2.0/qos/policies/275ede54-e091-4615-8feb-1171c76c9f86/bandwidth_limit_rules
0.514s
2017-04-17 07:24:07,163 21614 DEBUG[tempest.lib.common.rest_client]
Request - Headers: {'Accept': 'application/json', 'X-Auth-Token': '',
'Content-Type': 'application/json'}
Body: {"bandwidth_limit_rule": {"max_burst_kbps": 1, "max_kbps": 1}}
Response - Headers: {u'connection': 'close', 'status': '201',
u'x-openstack-request-id': 'req-a81448a4-580f-430d-8612-5b9f49279bde',
u'content-type': 'application/json', 'content-location':
'http://15.184.66.59:9696/v2.0/qos/policies/275ede54-e091-4615-8feb-1171c76c9f86/bandwidth_limit_rules',
u'content-length': '102', u'date': 'Mon, 17 Apr 2017 07:24:07 GMT'}
Body:
{"bandwidth_limit_rule":{"max_kbps":1,"id":"404613bf-d473-48e1-a8b0-9334afe6bf68","max_burst_kbps":1}}
2017-04-17 07:24:07,411 21614 INFO [tempest.lib.common.rest_client]
Request
(QosBandwidthLimitRuleTestJSON:test_rule_update_forbidden_for_regular_tenants_own_policy):
404 PUT
http://15.184.66.59:9696/v2.0/qos/policies/275ede54-e091-4615-8feb-1171c76c9f86/bandwidth_limit_rules/404613bf-d473-48e1-a8b0-9334afe6bf68
0.247s
2017-04-17 07:24:07,412 21614 DEBUG[tempest.lib.common.rest_client]
Request - Headers: {'Accept': 'application/json', 'X-Auth-Token': '',
'Content-Type': 'application/json'}
Body: {"bandwidth_limit_rule": {"max_burst_kbps": 4, "max_kbps": 2}}
Response - Headers: {u'connection': 'close', 'status': '404',
u'x-openstack-request-id': 'req-d02feb71-801a-4554-8d19-dfbfedd1e23b',
u'content-type': 'application/json', 'content-location':
'http://15.184.66.59:9696/v2.0/qos/policies/275ede54-e091-4615-8feb-1171c76c9f86/bandwidth_limit_rules/404613bf-d473-48e1-a8b0-9334afe6bf68',
u'content-length': '103', u'date': 'Mon, 17 Apr 2017 07:24:07 GMT'}
Body: {"NeutronError": {"message": "The resource could not be
found.", "type": "HTTPNotFound", "detail": ""}}
}}}
Traceback (most recent call last):
File "/opt/stack/new/neutron/neutron/tests/tempest/api/test_qos.py", line
478, in test_rule_update_forbidden_for_regular_tenants_own_policy
policy['id'], rule['id'], max_kbps=2, max_burst_kbps=4)
File "/usr/local/lib/python2.7/dist-packages/testtools/testcase.py", line
485, in assertRaises
self.assertThat(our_callable, matcher)
File "/usr/local/lib/python2.7/dist-packages/testtools/testcase.py", line
496, in assertThat
mismatch_error = self._matchHelper(matchee, matcher, message, verbose)
File "/usr/local/lib/python2.7/dist-packages/testtools/testcase.py", line
547, in _matchHelper
mismatch = matcher.match(matchee)
File
"/usr/local/lib/python2.7/dist-packages/testtools/matchers/_exception.py", line
108, in match
mismatch = self.exception_matcher.match(exc_info)
File
"/usr/local/lib/python2.7/dist-packages/testtools/matchers/_higherorder.py",
line 62, in match
mismatch = matcher.match(matchee)
File
[Yahoo-eng-team] [Bug 1718785] [NEW] rally job died with: InvalidTaskException: Task config is invalid: `'module' object has no attribute 'MultipleMatchesFound'`
Public bug reported:
http://logs.openstack.org/55/506355/2/check/gate-rally-dsvm-neutron-
neutron-ubuntu-xenial/c22fbcc/console.html
2017-09-21 21:08:52.213803 | 2017-09-21 21:08:52.213 | 2017-09-21 21:08:51.897
6883 ERROR rally.task.engine [-] 'module' object has no attribute
'MultipleMatchesFound': AttributeError: 'module' object has no attribute
'MultipleMatchesFound'
2017-09-21 21:08:52.215123 | 2017-09-21 21:08:52.214 | 2017-09-21 21:08:51.897
6883 ERROR rally.task.engine Traceback (most recent call last):
2017-09-21 21:08:52.216367 | 2017-09-21 21:08:52.216 | 2017-09-21 21:08:51.897
6883 ERROR rally.task.engine File
"/opt/stack/new/rally/rally/task/engine.py", line 250, in __init__
2017-09-21 21:08:52.217586 | 2017-09-21 21:08:52.217 | 2017-09-21 21:08:51.897
6883 ERROR rally.task.engine self.config = TaskConfig(config)
2017-09-21 21:08:52.219629 | 2017-09-21 21:08:52.218 | 2017-09-21 21:08:51.897
6883 ERROR rally.task.engine File
"/opt/stack/new/rally/rally/task/engine.py", line 712, in __init__
2017-09-21 21:08:52.220123 | 2017-09-21 21:08:52.219 | 2017-09-21 21:08:51.897
6883 ERROR rally.task.engine exceptions.MultipleMatchesFound):
2017-09-21 21:08:52.221429 | 2017-09-21 21:08:52.221 | 2017-09-21 21:08:51.897
6883 ERROR rally.task.engine AttributeError: 'module' object has no attribute
'MultipleMatchesFound'
2017-09-21 21:08:52.223034 | 2017-09-21 21:08:52.222 | 2017-09-21 21:08:51.897
6883 ERROR rally.task.engine
2017-09-21 21:08:52.224314 | 2017-09-21 21:08:52.224 | 2017-09-21 21:08:51.898
6883 ERROR rally.cli.cliutils [-] Task config is invalid: `'module' object has
no attribute 'MultipleMatchesFound'`: InvalidTaskException: Task config is
invalid: `'module' object has no attribute 'MultipleMatchesFound'`
2017-09-21 21:08:52.226689 | 2017-09-21 21:08:52.225 | 2017-09-21 21:08:51.898
6883 ERROR rally.cli.cliutils Traceback (most recent call last):
2017-09-21 21:08:52.227945 | 2017-09-21 21:08:52.227 | 2017-09-21 21:08:51.898
6883 ERROR rally.cli.cliutils File
"/opt/stack/new/rally/rally/cli/cliutils.py", line 660, in run
2017-09-21 21:08:52.229480 | 2017-09-21 21:08:52.229 | 2017-09-21 21:08:51.898
6883 ERROR rally.cli.cliutils ret = fn(*fn_args, **fn_kwargs)
2017-09-21 21:08:52.231150 | 2017-09-21 21:08:52.230 | 2017-09-21 21:08:51.898
6883 ERROR rally.cli.cliutils File "", line 2, in start
2017-09-21 21:08:52.232648 | 2017-09-21 21:08:52.232 | 2017-09-21 21:08:51.898
6883 ERROR rally.cli.cliutils File
"/opt/stack/new/rally/rally/cli/envutils.py", line 68, in default_from_global
2017-09-21 21:08:52.234113 | 2017-09-21 21:08:52.233 | 2017-09-21 21:08:51.898
6883 ERROR rally.cli.cliutils return f(*args, **kwargs)
2017-09-21 21:08:52.235420 | 2017-09-21 21:08:52.235 | 2017-09-21 21:08:51.898
6883 ERROR rally.cli.cliutils File "", line 2, in start
2017-09-21 21:08:52.236955 | 2017-09-21 21:08:52.236 | 2017-09-21 21:08:51.898
6883 ERROR rally.cli.cliutils File
"/opt/stack/new/rally/rally/plugins/__init__.py", line 51, in
ensure_plugins_are_loaded
2017-09-21 21:08:52.238519 | 2017-09-21 21:08:52.238 | 2017-09-21 21:08:51.898
6883 ERROR rally.cli.cliutils return f(*args, **kwargs)
2017-09-21 21:08:52.239774 | 2017-09-21 21:08:52.239 | 2017-09-21 21:08:51.898
6883 ERROR rally.cli.cliutils File
"/opt/stack/new/rally/rally/cli/commands/task.py", line 332, in start
2017-09-21 21:08:52.240985 | 2017-09-21 21:08:52.240 | 2017-09-21 21:08:51.898
6883 ERROR rally.cli.cliutils abort_on_sla_failure=abort_on_sla_failure)
2017-09-21 21:08:52.242361 | 2017-09-21 21:08:52.242 | 2017-09-21 21:08:51.898
6883 ERROR rally.cli.cliutils File "/opt/stack/new/rally/rally/api.py", line
438, in start
2017-09-21 21:08:52.243799 | 2017-09-21 21:08:52.243 | 2017-09-21 21:08:51.898
6883 ERROR rally.cli.cliutils abort_on_sla_failure=abort_on_sla_failure)
2017-09-21 21:08:52.245062 | 2017-09-21 21:08:52.244 | 2017-09-21 21:08:51.898
6883 ERROR rally.cli.cliutils File
"/opt/stack/new/rally/rally/task/engine.py", line 257, in __init__
2017-09-21 21:08:52.246380 | 2017-09-21 21:08:52.246 | 2017-09-21 21:08:51.898
6883 ERROR rally.cli.cliutils raise exceptions.InvalidTaskException(str(e))
2017-09-21 21:08:52.247718 | 2017-09-21 21:08:52.247 | 2017-09-21 21:08:51.898
6883 ERROR rally.cli.cliutils InvalidTaskException: Task config is invalid:
`'module' object has no attribute 'MultipleMatchesFound'`
2017-09-21 21:08:52.248924 | 2017-09-21 21:08:52.248 | 2017-09-21 21:08:51.898
6883 ERROR rally.cli.cliutils
2017-09-21 21:08:52.249489 | +
/home/jenkins/workspace/gate-rally-dsvm-neutron-neutron-ubuntu-xenial/devstack-gate/functions.sh:tsfilter:L103:
return 1
** Affects: neutron
Importance: Undecided
Assignee: Ihar Hrachyshka (ihar-hrachyshka)
Status: Invalid
** Affects: rally
Importance: Undecided
Assignee
[Yahoo-eng-team] [Bug 1718282] [NEW] Port update failed with 500 when trying to recreate default security group
Public bug reported: On port update, default security group may be missing. In this case, port update will first create the group, then proceed to port object. The problem is that when it recreates the group, it uses AFTER_UPDATE event, which contradicts the transactional semantics of _ensure_default_security_group_handler. Logs wise, we get this in neutron-server log: Sep 14 12:03:03.604813 ubuntu-xenial-2-node-rax-dfw-10932230 neutron- server[30503]: WARNING neutron.plugins.ml2.ovo_rpc [None req- 71600acd-c114-4dbd-a599-a9126fae14fb tempest- NetworkDefaultSecGroupTest-1846858447 tempest- NetworkDefaultSecGroupTest-1846858447] This handler is supposed to handle AFTER events, as in 'AFTER it's committed', not BEFORE. Offending resource event: security_group, after_create. Location: And then later: Sep 14 12:03:04.038599 ubuntu-xenial-2-node-rax-dfw-10932230 neutron-server[30503]: ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 1332, in update_port Sep 14 12:03:04.038761 ubuntu-xenial-2-node-rax-dfw-10932230 neutron-server[30503]: ERROR neutron.pecan_wsgi.hooks.translation context.session.expire(port_db) Sep 14 12:03:04.038924 ubuntu-xenial-2-node-rax-dfw-10932230 neutron-server[30503]: ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 1533, in expire Sep 14 12:03:04.039083 ubuntu-xenial-2-node-rax-dfw-10932230 neutron-server[30503]: ERROR neutron.pecan_wsgi.hooks.translation self._expire_state(state, attribute_names) Sep 14 12:03:04.039243 ubuntu-xenial-2-node-rax-dfw-10932230 neutron-server[30503]: ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 1536, in _expire_state Sep 14 12:03:04.039406 ubuntu-xenial-2-node-rax-dfw-10932230 neutron-server[30503]: ERROR neutron.pecan_wsgi.hooks.translation self._validate_persistent(state) Sep 14 12:03:04.041280 ubuntu-xenial-2-node-rax-dfw-10932230 neutron-server[30503]: ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 1976, in _validate_persistent Sep 14 12:03:04.041453 ubuntu-xenial-2-node-rax-dfw-10932230 neutron-server[30503]: ERROR neutron.pecan_wsgi.hooks.translation state_str(state)) Sep 14 12:03:04.041658 ubuntu-xenial-2-node-rax-dfw-10932230 neutron-server[30503]: ERROR neutron.pecan_wsgi.hooks.translation InvalidRequestError: Instance '' is not persistent within this Session Logs can be found in: http://logs.openstack.org/21/504021/1/check/gate- tempest-dsvm-neutron-dvr-multinode-scenario-ubuntu-xenial- nv/c6647c4/logs/screen-q-svc.txt.gz#_Sep_14_12_03_04_041658 ** Affects: neutron Importance: High Assignee: Kevin Benton (kevinbenton) Status: Confirmed ** Tags: db gate-failure sg-fw ** Changed in: neutron Importance: Undecided => High ** Changed in: neutron Status: New => Confirmed ** Changed in: neutron Assignee: (unassigned) => Kevin Benton (kevinbenton) ** Tags added: gate-failure sg-fw ** Tags added: db ** Bug watch added: Red Hat Bugzilla #1493175 https://bugzilla.redhat.com/show_bug.cgi?id=1493175 -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1718282 Title: Port update failed with 500 when trying to recreate default security group Status in neutron: Confirmed Bug description: On port update, default security group may be missing. In this case, port update will first create the group, then proceed to port object. The problem is that when it recreates the group, it uses AFTER_UPDATE event, which contradicts the transactional semantics of _ensure_default_security_group_handler. Logs wise, we get this in neutron-server log: Sep 14 12:03:03.604813 ubuntu-xenial-2-node-rax-dfw-10932230 neutron- server[30503]: WARNING neutron.plugins.ml2.ovo_rpc [None req- 71600acd-c114-4dbd-a599-a9126fae14fb tempest- NetworkDefaultSecGroupTest-1846858447 tempest- NetworkDefaultSecGroupTest-1846858447] This handler is supposed to handle AFTER events, as in 'AFTER it's committed', not BEFORE. Offending resource event: security_group, after_create. Location: And then later: Sep 14 12:03:04.038599 ubuntu-xenial-2-node-rax-dfw-10932230 neutron-server[30503]: ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 1332, in update_port Sep 14 12:03:04.038761 ubuntu-xenial-2-node-rax-dfw-10932230 neutron-server[30503]: ERROR neutron.pecan_wsgi.hooks.translation context.session.expire(port_db) Sep 14 12:03:04.038924 ubuntu-xenial-2-node-rax-dfw-10932230 neutron-server[30503]: ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py",
[Yahoo-eng-team] [Bug 1717046] [NEW] L3HARouterVRIdAllocationDbObjectTestCase.test_delete_objects fails because of duplicate record
cursor.execute(statement, parameters)
oslo_db.exception.DBDuplicateEntry: (sqlite3.IntegrityError) UNIQUE constraint
failed: ha_router_vrid_allocations.network_id, ha_router_vrid_allocations.vr_id
[SQL: 'INSERT INTO ha_router_vrid_allocations (network_id, vr_id) VALUES (?,
?)'] [parameters: ('af5e7d3e-c0d4-4a82-8ef5-2d930cfdd5e6', 566)]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/jenkins/workspace/gate-neutron-python35/neutron/tests/base.py",
line 118, in func
return f(self, *args, **kwargs)
File
"/home/jenkins/workspace/gate-neutron-python35/neutron/tests/unit/objects/test_base.py",
line 1887, in test_delete_objects
self._make_object(fields).create()
File "/home/jenkins/workspace/gate-neutron-python35/neutron/objects/base.py",
line 225, in decorator
res = func(self, *args, **kwargs)
File "/home/jenkins/workspace/gate-neutron-python35/neutron/objects/base.py",
line 636, in create
object_class=self.__class__, db_exception=db_exc)
neutron_lib.objects.exceptions.NeutronDbObjectDuplicateEntry: Failed to create
a duplicate L3HARouterVRIdAllocation: for attribute(s) ['network_id', 'vr_id']
with value(s) None
** Affects: neutron
Importance: High
Assignee: Ihar Hrachyshka (ihar-hrachyshka)
Status: Confirmed
** Tags: gate-failure
** Changed in: neutron
Status: New => Confirmed
** Changed in: neutron
Assignee: (unassigned) => Ihar Hrachyshka (ihar-hrachyshka)
** Changed in: neutron
Importance: Undecided => High
** Tags added: gate-failure
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1717046
Title:
L3HARouterVRIdAllocationDbObjectTestCase.test_delete_objects fails
because of duplicate record
Status in neutron:
Confirmed
Bug description:
http://logs.openstack.org/94/503794/1/check/gate-neutron-
python35/27f478d/testr_results.html.gz
Traceback (most recent call last):
File
"/home/jenkins/workspace/gate-neutron-python35/neutron/objects/base.py", line
633, in create
self.modify_fields_to_db(fields))
File
"/home/jenkins/workspace/gate-neutron-python35/neutron/objects/db/api.py", line
61, in create_object
context.session.add(db_obj)
File
"/home/jenkins/workspace/gate-neutron-python35/.tox/py35/lib/python3.5/site-packages/sqlalchemy/orm/session.py",
line 567, in __exit__
self.rollback()
File
"/home/jenkins/workspace/gate-neutron-python35/.tox/py35/lib/python3.5/site-packages/sqlalchemy/util/langhelpers.py",
line 66, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File
"/home/jenkins/workspace/gate-neutron-python35/.tox/py35/lib/python3.5/site-packages/sqlalchemy/util/compat.py",
line 187, in reraise
raise value
File
"/home/jenkins/workspace/gate-neutron-python35/.tox/py35/lib/python3.5/site-packages/sqlalchemy/orm/session.py",
line 564, in __exit__
self.commit()
File
"/home/jenkins/workspace/gate-neutron-python35/.tox/py35/lib/python3.5/site-packages/sqlalchemy/orm/session.py",
line 461, in commit
self._prepare_impl()
File
"/home/jenkins/workspace/gate-neutron-python35/.tox/py35/lib/python3.5/site-packages/sqlalchemy/orm/session.py",
line 441, in _prepare_impl
self.session.flush()
File
"/home/jenkins/workspace/gate-neutron-python35/.tox/py35/lib/python3.5/site-packages/sqlalchemy/orm/session.py",
line 2177, in flush
self._flush(objects)
File
"/home/jenkins/workspace/gate-neutron-python35/.tox/py35/lib/python3.5/site-packages/sqlalchemy/orm/session.py",
line 2297, in _flush
transaction.rollback(_capture_exception=True)
File
"/home/jenkins/workspace/gate-neutron-python35/.tox/py35/lib/python3.5/site-packages/sqlalchemy/util/langhelpers.py",
line 66, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File
"/home/jenkins/workspace/gate-neutron-python35/.tox/py35/lib/python3.5/site-packages/sqlalchemy/util/compat.py",
line 187, in reraise
raise value
File
"/home/jenkins/workspace/gate-neutron-python35/.tox/py35/lib/python3.5/site-packages/sqlalchemy/orm/session.py",
line 2261, in _flush
flush_context.execute()
File
"/home/jenkins/workspace/gate-neutron-python35/.tox/py35/lib/python3.5/site-packages/sqlalchemy/orm/unitofwork.py",
line 389, in execute
rec.execute(self)
File
"/home/jenkins/workspace/gate-neutron-python35/.tox/py35/lib/python3.5/site-packages/sqlalchemy/orm/unitofwork.py",
line 548, in execute
uow
File
"/home/jenkins/workspace/gate-neutron-python35/.tox/py35/lib/python3.5/site-packages/sqlalchemy/orm/persistence.py",
line 181, in save_obj
map
[Yahoo-eng-team] [Bug 1716746] [NEW] functional job broken by new os-testr
Public bug reported: functional job fails with: 2017-09-12 16:09:20.705975 | 2017-09-12 16:09:20.705 | + /opt/stack/new/neutron/neutron/tests/contrib/post_test_hook.sh:main:L67: testr_exit_code=0 2017-09-12 16:09:20.707372 | 2017-09-12 16:09:20.706 | + /opt/stack/new/neutron/neutron/tests/contrib/post_test_hook.sh:main:L68: set -e 2017-09-12 16:09:20.718005 | 2017-09-12 16:09:20.717 | + /opt/stack/new/neutron/neutron/tests/contrib/post_test_hook.sh:main:L71: generate_testr_results 2017-09-12 16:09:20.719619 | 2017-09-12 16:09:20.719 | + /opt/stack/new/neutron/neutron/tests/contrib/post_test_hook.sh:generate_testr_results:L12: sudo -H -u stack chmod o+rw . 2017-09-12 16:09:20.720974 | 2017-09-12 16:09:20.720 | + /opt/stack/new/neutron/neutron/tests/contrib/post_test_hook.sh:generate_testr_results:L13: sudo -H -u stack chmod o+rw -R .testrepository 2017-09-12 16:09:20.722284 | 2017-09-12 16:09:20.721 | chmod: cannot access '.testrepository': No such file or directory This is because new os-testr switched to stestr that has a different name for the directory (.stestr). ** Affects: neutron Importance: Critical Assignee: Ihar Hrachyshka (ihar-hrachyshka) Status: In Progress ** Tags: gate-failure ** Changed in: neutron Importance: Undecided => Critical ** Changed in: neutron Status: New => Confirmed ** Tags added: gate-failure ** Changed in: neutron Assignee: (unassigned) => Ihar Hrachyshka (ihar-hrachyshka) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1716746 Title: functional job broken by new os-testr Status in neutron: In Progress Bug description: functional job fails with: 2017-09-12 16:09:20.705975 | 2017-09-12 16:09:20.705 | + /opt/stack/new/neutron/neutron/tests/contrib/post_test_hook.sh:main:L67: testr_exit_code=0 2017-09-12 16:09:20.707372 | 2017-09-12 16:09:20.706 | + /opt/stack/new/neutron/neutron/tests/contrib/post_test_hook.sh:main:L68: set -e 2017-09-12 16:09:20.718005 | 2017-09-12 16:09:20.717 | + /opt/stack/new/neutron/neutron/tests/contrib/post_test_hook.sh:main:L71: generate_testr_results 2017-09-12 16:09:20.719619 | 2017-09-12 16:09:20.719 | + /opt/stack/new/neutron/neutron/tests/contrib/post_test_hook.sh:generate_testr_results:L12: sudo -H -u stack chmod o+rw . 2017-09-12 16:09:20.720974 | 2017-09-12 16:09:20.720 | + /opt/stack/new/neutron/neutron/tests/contrib/post_test_hook.sh:generate_testr_results:L13: sudo -H -u stack chmod o+rw -R .testrepository 2017-09-12 16:09:20.722284 | 2017-09-12 16:09:20.721 | chmod: cannot access '.testrepository': No such file or directory This is because new os-testr switched to stestr that has a different name for the directory (.stestr). To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1716746/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1576840] Re: fullstack OVS agent in native openflow mode sometimes fails to bind socket
It's long fixed. ** Changed in: neutron Assignee: sudhakar kumar srivastava (sudhakar.srivastava) => (unassigned) ** Changed in: neutron Status: Confirmed => Won't Fix ** Changed in: neutron Status: Won't Fix => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1576840 Title: fullstack OVS agent in native openflow mode sometimes fails to bind socket Status in neutron: Fix Released Bug description: Plenty of hits in the last few days, currently the top issue affecting fullstack stability. Example paste: http://paste.openstack.org/show/495797/ Example logs: http://logs.openstack.org/18/276018/21/check/gate-neutron-dsvm-fullstack/c0761dc/logs/TestOvsConnectivitySameNetwork.test_connectivity_VLANs,openflow-native_ovsdb-native_/ To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1576840/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1602567] Re: fullstack doesn't work with branch-2.5 ovs
We currently compile v2.6.1 from sources for fullstack. I consider it's fixed. ** Changed in: neutron Status: Confirmed => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1602567 Title: fullstack doesn't work with branch-2.5 ovs Status in neutron: Fix Released Bug description: In order to test OVSFirewall, we need a newer version of ovs than one comes with Trusty. But the combination fails on VXLAN connectivity tests. See fullstack log of https://review.openstack.org/#/c/341328/1 for a example. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1602567/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1627106] Re: TimeoutException while executing tests adding bridge using OVSDB native
Added ovsdbapp to the list of affected projects because I believe this error comes from the library. ** Changed in: neutron Milestone: pike-2 => None ** Also affects: ovsdbapp Importance: Undecided Status: New ** Changed in: ovsdbapp Status: New => Confirmed -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1627106 Title: TimeoutException while executing tests adding bridge using OVSDB native Status in neutron: Confirmed Status in ovsdbapp: Confirmed Bug description: http://logs.openstack.org/91/366291/12/check/gate-neutron-dsvm- functional-ubuntu-trusty/a23c816/testr_results.html.gz Traceback (most recent call last): File "neutron/tests/base.py", line 125, in func return f(self, *args, **kwargs) File "neutron/tests/functional/agent/ovsdb/test_impl_idl.py", line 62, in test_post_commit_vswitchd_completed_no_failures self._add_br_and_test() File "neutron/tests/functional/agent/ovsdb/test_impl_idl.py", line 56, in _add_br_and_test self._add_br() File "neutron/tests/functional/agent/ovsdb/test_impl_idl.py", line 52, in _add_br tr.add(ovsdb.add_br(self.brname)) File "neutron/agent/ovsdb/api.py", line 76, in __exit__ self.result = self.commit() File "neutron/agent/ovsdb/impl_idl.py", line 72, in commit 'timeout': self.timeout}) neutron.agent.ovsdb.api.TimeoutException: Commands [AddBridgeCommand(name=test-br6925d8e2, datapath_type=None, may_exist=True)] exceeded timeout 10 seconds I suspect this one may hit us because we finally made timeout working with Icd745514adc14730b9179fa7a6dd5c115f5e87a5. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1627106/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1673531] Re: fullstack test_controller_timeout_does_not_break_connectivity_sigkill(GRE and l2pop, openflow-native_ovsdb-cli) failure
No hits in 7 days. I claim it's fixed. If not, reopen. ** Changed in: neutron Status: Confirmed => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1673531 Title: fullstack test_controller_timeout_does_not_break_connectivity_sigkill(GRE and l2pop,openflow-native_ovsdb-cli) failure Status in neutron: Fix Released Bug description: Logs for failure: http://logs.openstack.org/98/446598/1/check/gate- neutron-dsvm-fullstack-ubuntu-xenial/2e0f93e/logs/dsvm-fullstack- logs/TestOvsConnectivitySameNetworkOnOvsBridgeControllerStop .test_controller_timeout_does_not_break_connectivity_sigkill_GRE-and- l2pop,openflow-native_ovsdb-cli_/ logstash query: http://logstash.openstack.org/#dashboard/file/logstash.json?query=messge%3A%5C%22in%20test_controller_timeout_does_not_break_connectivity_sigkill%5C%22%20AND%20tags%3Aconsole%20AND%20build_name %3Agate-neutron-dsvm-fullstack-ubuntu-xenial To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1673531/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1713163] [NEW] test_delete_saving_image fails because image hasn't transitioned to SAVING
Public bug reported: http://logs.openstack.org/33/440933/4/check/gate-tempest-dsvm-neutron- linuxbridge-ubuntu-xenial/ebfa453/logs/testr_results.html.gz Traceback (most recent call last): File "tempest/api/compute/images/test_images.py", line 48, in test_delete_saving_image wait_until='SAVING') File "tempest/api/compute/base.py", line 301, in create_image_from_server image_id, wait_until) File "tempest/common/waiters.py", line 179, in wait_for_image_status raise lib_exc.TimeoutException(message) tempest.lib.exceptions.TimeoutException: Request timed out Details: (ImagesTestJSON:test_delete_saving_image) Image 93090479-409d-49f9-b2d3-2bc11a40f967 failed to reach SAVING state (current state ACTIVE) within the required time (196 s). I briefly looked at the test case, and it seems to me that it may happen when the image snapshot is produced quicker than tempest being able to catch the intermittent transition. If so, the test case may need a refinement to check that a snapshot was produced and not that the state was in SAVING for some time. ** Affects: nova Importance: Undecided Status: New ** Tags: gate-failure ** Tags added: gate-failure -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1713163 Title: test_delete_saving_image fails because image hasn't transitioned to SAVING Status in OpenStack Compute (nova): New Bug description: http://logs.openstack.org/33/440933/4/check/gate-tempest-dsvm-neutron- linuxbridge-ubuntu-xenial/ebfa453/logs/testr_results.html.gz Traceback (most recent call last): File "tempest/api/compute/images/test_images.py", line 48, in test_delete_saving_image wait_until='SAVING') File "tempest/api/compute/base.py", line 301, in create_image_from_server image_id, wait_until) File "tempest/common/waiters.py", line 179, in wait_for_image_status raise lib_exc.TimeoutException(message) tempest.lib.exceptions.TimeoutException: Request timed out Details: (ImagesTestJSON:test_delete_saving_image) Image 93090479-409d-49f9-b2d3-2bc11a40f967 failed to reach SAVING state (current state ACTIVE) within the required time (196 s). I briefly looked at the test case, and it seems to me that it may happen when the image snapshot is produced quicker than tempest being able to catch the intermittent transition. If so, the test case may need a refinement to check that a snapshot was produced and not that the state was in SAVING for some time. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1713163/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1713162] [NEW] Kernel panic when configuring APIC timers in an instance
Public bug reported: I am not sure Nova is the best candidate for this, but reporting nevertheless. Feel free to move to another project that is a better fit. This happened in gate in Queens. http://logs.openstack.org/74/495974/4/check/gate-tempest-dsvm-neutron- full-ubuntu-xenial/c61f703/logs/testr_results.html.gz In instance console log, we can see: [0.732045] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [0.736045] ..MP-BIOS bug: 8254 timer not connected to IO-APIC [0.736045] ...trying to set up timer (IRQ0) through the 8259A ... [0.736045] . (found apic 0 pin 2) ... [0.736045] ... failed. [0.736045] ...trying to set up timer as Virtual Wire IRQ... [0.744045] . failed. [0.744045] ...trying to set up timer as ExtINT IRQ... [0.752046] . failed :(. [0.752046] Kernel panic - not syncing: IO-APIC + timer doesn't work! Boot with apic=debug and send a report. Then try booting with the 'noapic' option. [0.752046] [0.752046] Pid: 1, comm: swapper/0 Not tainted 3.2.0-80-virtual #116-Ubuntu [0.752046] Call Trace: [0.752046] [] panic+0x91/0x1a4 [0.752046] [] setup_IO_APIC+0x651/0x693 [0.752046] [] native_smp_prepare_cpus+0x1c4/0x207 [0.752046] [] kernel_init+0x8c/0x169 [0.752046] [] kernel_thread_helper+0x4/0x10 [0.752046] [] ? start_kernel+0x3c7/0x3c7 [0.752046] [] ? gs_change+0x13/0x13 ** Affects: nova Importance: Undecided Status: New ** Tags: gate-failure ** Tags added: gate-failure -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1713162 Title: Kernel panic when configuring APIC timers in an instance Status in OpenStack Compute (nova): New Bug description: I am not sure Nova is the best candidate for this, but reporting nevertheless. Feel free to move to another project that is a better fit. This happened in gate in Queens. http://logs.openstack.org/74/495974/4/check/gate-tempest-dsvm-neutron- full-ubuntu-xenial/c61f703/logs/testr_results.html.gz In instance console log, we can see: [0.732045] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [0.736045] ..MP-BIOS bug: 8254 timer not connected to IO-APIC [0.736045] ...trying to set up timer (IRQ0) through the 8259A ... [0.736045] . (found apic 0 pin 2) ... [0.736045] ... failed. [0.736045] ...trying to set up timer as Virtual Wire IRQ... [0.744045] . failed. [0.744045] ...trying to set up timer as ExtINT IRQ... [0.752046] . failed :(. [0.752046] Kernel panic - not syncing: IO-APIC + timer doesn't work! Boot with apic=debug and send a report. Then try booting with the 'noapic' option. [0.752046] [0.752046] Pid: 1, comm: swapper/0 Not tainted 3.2.0-80-virtual #116-Ubuntu [0.752046] Call Trace: [0.752046] [] panic+0x91/0x1a4 [0.752046] [] setup_IO_APIC+0x651/0x693 [0.752046] [] native_smp_prepare_cpus+0x1c4/0x207 [0.752046] [] kernel_init+0x8c/0x169 [0.752046] [] kernel_thread_helper+0x4/0x10 [0.752046] [] ? start_kernel+0x3c7/0x3c7 [0.752046] [] ? gs_change+0x13/0x13 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1713162/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1712412] [NEW] DVR external port setup fails with KeyError: 'host'
Public bug reported: http://logs.openstack.org/28/481928/9/check/gate-tempest-dsvm-neutron- dvr-multinode-scenario-ubuntu-xenial- nv/8fb7c92/logs/subnode-2/screen-q-l3.txt.gz?level=TRACE Aug 22 07:55:55.697082 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent [-] Failed to process compatible router: 02fc503f-72c9-49bd-a4f8-30305a7efb90: KeyError: 'host' Aug 22 07:55:55.697264 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent Traceback (most recent call last): Aug 22 07:55:55.697446 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent File "/opt/stack/new/neutron/neutron/agent/l3/agent.py", line 538, in _process_router_update Aug 22 07:55:55.697665 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent self._process_router_if_compatible(router) Aug 22 07:55:55.697850 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent File "/opt/stack/new/neutron/neutron/agent/l3/agent.py", line 475, in _process_router_if_compatible Aug 22 07:55:55.698048 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent self._process_updated_router(router) Aug 22 07:55:55.698229 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent File "/opt/stack/new/neutron/neutron/agent/l3/agent.py", line 490, in _process_updated_router Aug 22 07:55:55.698405 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent ri.process() Aug 22 07:55:55.698581 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent File "/opt/stack/new/neutron/neutron/agent/l3/dvr_local_router.py", line 737, in process Aug 22 07:55:55.698756 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent super(DvrLocalRouter, self).process() Aug 22 07:55:55.698936 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent File "/opt/stack/new/neutron/neutron/agent/l3/dvr_router_base.py", line 29, in process Aug 22 07:55:55.699114 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent super(DvrRouterBase, self).process() Aug 22 07:55:55.699289 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent File "/opt/stack/new/neutron/neutron/agent/l3/ha_router.py", line 436, in process Aug 22 07:55:55.699469 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent super(HaRouter, self).process() Aug 22 07:55:55.699654 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent File "/opt/stack/new/neutron/neutron/common/utils.py", line 189, in call Aug 22 07:55:55.699903 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent self.logger(e) Aug 22 07:55:55.700084 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ Aug 22 07:55:55.700260 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent self.force_reraise() Aug 22 07:55:55.700457 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise Aug 22 07:55:55.700635 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent six.reraise(self.type_, self.value, self.tb) Aug 22 07:55:55.700816 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent File "/opt/stack/new/neutron/neutron/common/utils.py", line 186, in call Aug 22 07:55:55.700995 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent return func(*args, **kwargs) Aug 22 07:55:55.701180 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent File "/opt/stack/new/neutron/neutron/agent/l3/router_info.py", line 1120, in process Aug 22 07:55:55.701357 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent self.process_external() Aug 22 07:55:55.701537 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent File "/opt/stack/new/neutron/neutron/agent/l3/dvr_local_router.py", line 560, in process_external Aug 22 07:55:55.701713 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent
[Yahoo-eng-team] [Bug 1666731] Re: ofctl timeout error in gate functional tests
It doesn't affect functional gate anymore. We see some hits in
fullstack, but there, agents properly recover from those errors. I am
closing the bug.
** Changed in: neutron
Status: Confirmed => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1666731
Title:
ofctl timeout error in gate functional tests
Status in neutron:
Fix Released
Bug description:
Functional test failure in test_install_flood_to_tun
RuntimeError: ofctl request
version=0x4,msg_type=0xe,msg_len=None,xid=0xc073a211,OFPFlowMod(buffer_id=4294967295,command=0,cookie=10613959233739808590L,cookie_mask=0,flags=0,hard_timeout=0,idle_timeout=0,instructions=[OFPInstructionActions(actions=[OFPActionPopVlan(len=8,type=18),
OFPActionSetField(tunnel_id=888),
OFPActionOutput(len=16,max_len=0,port=-1,type=0)],type=4)],match=OFPMatch(oxm_fields={'vlan_vid':
4873}),out_group=0,out_port=0,priority=1,table_id=22) timed out
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1666731/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1669805] Re: rally job failing in gate due to "Quota for tenant X could not be found" Error
After latest rally changes that tweaked SLA and number of parallel test runs, the job is very stable. I don't think we've seen this error for a while. Please reopen if we still experience the issue. ** Changed in: neutron Status: Confirmed => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1669805 Title: rally job failing in gate due to "Quota for tenant X could not be found" Error Status in neutron: Fix Released Bug description: Rally job is failing in the gate due to the following error during cleanup [0]: 2017-03-03 13:14:56.897549 | 2017-03-03 13:14:56.897 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager rutils.retry(resource._max_attempts, resource.delete) 2017-03-03 13:14:56.899015 | 2017-03-03 13:14:56.898 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager File "/opt/stack/new/rally/rally/common/utils.py", line 223, in retry 2017-03-03 13:14:56.900375 | 2017-03-03 13:14:56.900 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager return func(*args, **kwargs) 2017-03-03 13:14:56.901935 | 2017-03-03 13:14:56.901 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager File "/opt/stack/new/rally/rally/plugins/openstack/cleanup/resources.py", line 472, in delete 2017-03-03 13:14:56.903254 | 2017-03-03 13:14:56.902 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager self._manager().delete_quota(self.tenant_uuid) 2017-03-03 13:14:56.904746 | 2017-03-03 13:14:56.904 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager File "/usr/local/lib/python2.7/dist-packages/debtcollector/renames.py", line 43, in decorator 2017-03-03 13:14:56.906156 | 2017-03-03 13:14:56.905 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager return wrapped(*args, **kwargs) 2017-03-03 13:14:56.907779 | 2017-03-03 13:14:56.907 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 742, in delete_quota 2017-03-03 13:14:56.909166 | 2017-03-03 13:14:56.908 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager return self.delete(self.quota_path % (project_id)) 2017-03-03 13:14:56.910455 | 2017-03-03 13:14:56.910 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 357, in delete 2017-03-03 13:14:56.911722 | 2017-03-03 13:14:56.911 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager headers=headers, params=params) 2017-03-03 13:14:56.913068 | 2017-03-03 13:14:56.912 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 338, in retry_request 2017-03-03 13:14:56.914760 | 2017-03-03 13:14:56.914 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager headers=headers, params=params) 2017-03-03 13:14:56.916094 | 2017-03-03 13:14:56.915 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 301, in do_request 2017-03-03 13:14:56.917732 | 2017-03-03 13:14:56.917 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager self._handle_fault_response(status_code, replybody, resp) 2017-03-03 13:14:56.919266 | 2017-03-03 13:14:56.918 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 276, in _handle_fault_response 2017-03-03 13:14:56.920699 | 2017-03-03 13:14:56.920 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager exception_handler_v20(status_code, error_body) 2017-03-03 13:14:56.922342 | 2017-03-03 13:14:56.922 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 92, in exception_handler_v20 2017-03-03 13:14:56.923773 | 2017-03-03 13:14:56.923 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager request_ids=request_ids) 2017-03-03 13:14:56.925720 | 2017-03-03 13:14:56.925 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager NotFound: Quota for tenant 82ee0ba1b6534f958d1acd2f717b5c3d could not be found. Seems like we're hitting this errors from 1AM today (03/03/2017) [1] [0] http://logs.openstack.org/91/431691/30/check/gate-rally-dsvm-neutron-neutron-ubuntu-xenial/ab3471c/console.html#_2017-03-03_13_14_56_925720 [1]
[Yahoo-eng-team] [Bug 1712185] [NEW] iptables-restore calls fail acquiring 'xlock' with iptables from master
Public bug reported: This happens when you use iptables that includes https://git.netfilter.org/iptables/commit/?id=999eaa241212d3952ddff39a99d0d55a74e3639e (f.e. the one from latest RHEL repos) neutron.tests.functional.agent.test_firewall.FirewallTestCase.test_established_connection_is_cut(IptablesFirewallDriver,without ipset) -- Captured traceback: ~~~ Traceback (most recent call last): File "neutron/tests/functional/agent/test_firewall.py", line 113, in setUp self.firewall.prepare_port_filter(self.src_port_desc) File "neutron/agent/linux/iptables_firewall.py", line 204, in prepare_port_filter return self.iptables.apply() File "neutron/agent/linux/iptables_manager.py", line 432, in apply return self._apply() File "neutron/agent/linux/iptables_manager.py", line 440, in _apply first = self._apply_synchronized() File "neutron/agent/linux/iptables_manager.py", line 539, in _apply_synchronized ' '.join(log_lines)) File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ self.force_reraise() File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise six.reraise(self.type_, self.value, self.tb) File "neutron/agent/linux/iptables_manager.py", line 518, in _apply_synchronized run_as_root=True) File "neutron/agent/linux/utils.py", line 156, in execute raise ProcessExecutionError(msg, returncode=returncode) neutron.agent.linux.utils.ProcessExecutionError: Exit code: 4; Stdin: # Generated by iptables_manager *filter :neutron-filter-top - [0:0] :run.py-FORWARD - [0:0] :run.py-INPUT - [0:0] :run.py-OUTPUT - [0:0] :run.py-it-veth0bc5 - [0:0] :run.py-local - [0:0] :run.py-ot-veth0bc5 - [0:0] :run.py-sg-chain - [0:0] :run.py-sg-fallback - [0:0] -I FORWARD 1 -j neutron-filter-top -I FORWARD 2 -j run.py-FORWARD -I INPUT 1 -j run.py-INPUT -I OUTPUT 1 -j neutron-filter-top -I OUTPUT 2 -j run.py-OUTPUT -I neutron-filter-top 1 -j run.py-local -I run.py-FORWARD 1 -m physdev --physdev-out test-veth0bc5b8 --physdev-is-bridged -m comment --comment "Direct traffic from the VM interface to the security group chain." -j run.py-sg-chain -I run.py-FORWARD 2 -m physdev --physdev-in test-veth0bc5b8 --physdev-is-bridged -m comment --comment "Direct traffic from the VM interface to the security group chain." -j run.py-sg-chain -I run.py-INPUT 1 -m physdev --physdev-in test-veth0bc5b8 --physdev-is-bridged -m comment --comment "Direct incoming traffic from VM to the security group chain." -j run.py-ot-veth0bc5 -I run.py-it-veth0bc5 1 -p ipv6-icmp -m icmp6 --icmpv6-type 130 -j RETURN -I run.py-it-veth0bc5 2 -p ipv6-icmp -m icmp6 --icmpv6-type 134 -j RETURN -I run.py-it-veth0bc5 3 -p ipv6-icmp -m icmp6 --icmpv6-type 135 -j RETURN -I run.py-it-veth0bc5 4 -p ipv6-icmp -m icmp6 --icmpv6-type 136 -j RETURN -I run.py-it-veth0bc5 5 -m state --state RELATED,ESTABLISHED -m comment --comment "Direct packets associated with a known session to the RETURN chain." -j RETURN -I run.py-it-veth0bc5 6 -m state --state INVALID -m comment --comment "Drop packets that appear related to an existing connection (e.g. TCP ACK/FIN) but do not have an entry in conntrack." -j DROP -I run.py-it-veth0bc5 7 -m comment --comment "Send unmatched traffic to the fallback chain." -j run.py-sg-fallback -I run.py-ot-veth0bc5 1 -s ::/128 -d ff02::/16 -p ipv6-icmp -m icmp6 --icmpv6-type 131 -m comment --comment "Allow IPv6 ICMP traffic." -j RETURN -I run.py-ot-veth0bc5 2 -s ::/128 -d ff02::/16 -p ipv6-icmp -m icmp6 --icmpv6-type 135 -m comment --comment "Allow IPv6 ICMP traffic." -j RETURN -I run.py-ot-veth0bc5 3 -s ::/128 -d ff02::/16 -p ipv6-icmp -m icmp6 --icmpv6-type 143 -m comment --comment "Allow IPv6 ICMP traffic." -j RETURN -I run.py-ot-veth0bc5 4 -p ipv6-icmp -m icmp6 --icmpv6-type 134 -m comment --comment "Drop IPv6 Router Advts from VM Instance." -j DROP -I run.py-ot-veth0bc5 5 -p ipv6-icmp -m comment --comment "Allow IPv6 ICMP traffic." -j RETURN -I run.py-ot-veth0bc5 6 -p udp -m udp --sport 546 --dport 547 -m comment --comment "Allow DHCP client traffic." -j RETURN -I run.py-ot-veth0bc5 7 -p udp -m udp --sport 547 --dport 546 -m comment --comment "Prevent DHCP Spoofing by VM." -j DROP -I run.py-ot-veth0bc5 8 -m state --state RELATED,ESTABLISHED -m comment --comment "Direct packets associated with a known session to the RETURN chain." -j RETURN -I run.py-ot-veth0bc5 9 -m state --state INVALID -m comment --comment "Drop packets that appear related to an existing connection (e.g. TCP ACK/FIN) but do not have an entry in conntrack." -j DROP -I run.py-ot-veth0bc5
[Yahoo-eng-team] [Bug 1711463] [NEW] TestNetworkBasicOps.test_network_basic_ops failed with "Timed out waiting for 172.24.5.8 to become reachable"
Public bug reported:
http://logs.openstack.org/45/493945/2/gate/gate-grenade-dsvm-neutron-
dvr-multinode-ubuntu-xenial/c101360/logs/testr_results.html.gz
2017-08-17 19:34:47,447 1906 DEBUG[tempest.scenario.manager] checking
network connections to IP 172.24.5.8 with user: cirros
2017-08-17 19:34:47,448 1906 DEBUG[tempest.scenario.manager]
TestNetworkBasicOps:test_network_basic_ops begins to ping 172.24.5.8 in 120 sec
and the expected result is reachable
2017-08-17 19:36:47,871 1906 DEBUG[tempest.scenario.manager]
TestNetworkBasicOps:test_network_basic_ops finishes ping 172.24.5.8 in 120 sec
and the ping result is unexpected
2017-08-17 19:36:47,872 1906 ERROR[tempest.scenario.manager] Public network
connectivity check failed: after re-associate floating ip
2017-08-17 19:36:47.872 1906 ERROR tempest.scenario.manager Traceback (most
recent call last):
2017-08-17 19:36:47.872 1906 ERROR tempest.scenario.manager File
"tempest/scenario/manager.py", line 609, in check_public_network_connectivity
2017-08-17 19:36:47.872 1906 ERROR tempest.scenario.manager mtu=mtu)
2017-08-17 19:36:47.872 1906 ERROR tempest.scenario.manager File
"tempest/scenario/manager.py", line 592, in check_vm_connectivity
2017-08-17 19:36:47.872 1906 ERROR tempest.scenario.manager msg=msg)
2017-08-17 19:36:47.872 1906 ERROR tempest.scenario.manager File
"/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/unittest2/case.py",
line 702, in assertTrue
2017-08-17 19:36:47.872 1906 ERROR tempest.scenario.manager raise
self.failureException(msg)
2017-08-17 19:36:47.872 1906 ERROR tempest.scenario.manager AssertionError:
False is not true : Timed out waiting for 172.24.5.8 to become reachable
In l3 agent log, we see the address is configured:
Aug 17 19:33:40.782930 ubuntu-xenial-2-node-citycloud-kna1-10495241
neutron-l3-agent[21093]: DEBUG neutron.agent.linux.utils [-] Running command
(rootwrap daemon): ['ip', 'netns', 'exec',
'fip-8df87dc6-670f-4f85-861e-3f041537e632', 'ip', '-4', 'route', 'replace',
'172.24.5.8/32', 'via', '169.254.95.212', 'dev', 'fpr-d2baa122-5']
{{(pid=21093) execute_rootwrap_daemon
/opt/stack/new/neutron/neutron/agent/linux/utils.py:108}}
Aug 17 19:33:40.857528 ubuntu-xenial-2-node-citycloud-kna1-10495241
neutron-l3-agent[21093]: DEBUG neutron.agent.linux.utils [-] Running command
(rootwrap daemon): ['ip', 'netns', 'exec',
'fip-8df87dc6-670f-4f85-861e-3f041537e632', 'arping', '-U', '-I',
'fg-1090ccce-f5', '-c', '1', '-w', '1.5', '172.24.5.8'] {{(pid=21093)
execute_rootwrap_daemon
/opt/stack/new/neutron/neutron/agent/linux/utils.py:108}}
No traces in q-agt or q-l3 for the port, or at all. It will probably be
hard to debug the failure, but reporting nevertheless, if anything for
tracking purposes.
The failure is in Q (current master), but since it's grenade, it's
probably Pike code.
** Affects: neutron
Importance: High
Status: Confirmed
** Tags: gate-failure l3-dvr-backlog
** Changed in: neutron
Importance: Undecided => High
** Tags added: gate-failure l3-dvr-backlog
** Changed in: neutron
Status: New => Confirmed
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1711463
Title:
TestNetworkBasicOps.test_network_basic_ops failed with "Timed out
waiting for 172.24.5.8 to become reachable"
Status in neutron:
Confirmed
Bug description:
http://logs.openstack.org/45/493945/2/gate/gate-grenade-dsvm-neutron-
dvr-multinode-ubuntu-xenial/c101360/logs/testr_results.html.gz
2017-08-17 19:34:47,447 1906 DEBUG[tempest.scenario.manager] checking
network connections to IP 172.24.5.8 with user: cirros
2017-08-17 19:34:47,448 1906 DEBUG[tempest.scenario.manager]
TestNetworkBasicOps:test_network_basic_ops begins to ping 172.24.5.8 in 120 sec
and the expected result is reachable
2017-08-17 19:36:47,871 1906 DEBUG[tempest.scenario.manager]
TestNetworkBasicOps:test_network_basic_ops finishes ping 172.24.5.8 in 120 sec
and the ping result is unexpected
2017-08-17 19:36:47,872 1906 ERROR[tempest.scenario.manager] Public
network connectivity check failed: after re-associate floating ip
2017-08-17 19:36:47.872 1906 ERROR tempest.scenario.manager Traceback (most
recent call last):
2017-08-17 19:36:47.872 1906 ERROR tempest.scenario.manager File
"tempest/scenario/manager.py", line 609, in check_public_network_connectivity
2017-08-17 19:36:47.872 1906 ERROR tempest.scenario.manager mtu=mtu)
2017-08-17 19:36:47.872 1906 ERROR tempest.scenario.manager File
"tempest/scenario/manager.py", line 592, in check_vm_connectivity
2017-08-17 19:36:47.872 1906 ERROR tempest.scenario.manager msg=msg)
2017-08-17 19:36:47.872 1906 ERROR tempest.scenario.manager File
"/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/unittest2/case.py",
line 702, in assertTrue
[Yahoo-eng-team] [Bug 1707160] Re: test_create_port_in_allowed_allocation_pools test fails on ironic grenade
Adding oslo to the list of affected projects. The fix is here:
https://review.openstack.org/#/c/486706/
** Also affects: oslo.messaging
Importance: Undecided
Status: New
** Changed in: oslo.messaging
Assignee: (unassigned) => Ken Giusti (kgiusti)
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1707160
Title:
test_create_port_in_allowed_allocation_pools test fails on ironic
grenade
Status in neutron:
Fix Released
Status in OpenStack Compute (nova):
Incomplete
Status in oslo.messaging:
New
Bug description:
Here is an example of a job at
http://logs.openstack.org/58/487458/6/check/gate-grenade-dsvm-ironic-
ubuntu-xenial/d8f187e/console.html#_2017-07-28_09_33_52_031224
2017-07-28 09:33:52.027473 | Captured pythonlogging:
2017-07-28 09:33:52.027484 | ~~~
2017-07-28 09:33:52.027539 | 2017-07-28 09:15:48,746 9778 INFO
[tempest.lib.common.rest_client] Request
(PortsTestJSON:test_create_port_in_allowed_allocation_pools): 201 POST
http://149.202.183.40:9696/v2.0/networks 0.342s
2017-07-28 09:33:52.027604 | 2017-07-28 09:15:48,746 9778 DEBUG
[tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token':
'', 'Accept': 'application/json', 'Content-Type': 'application/json'}
2017-07-28 09:33:52.027633 | Body: {"network": {"name":
"tempest-PortsTestJSON-test-network-1596805013"}}
2017-07-28 09:33:52.027728 | Response - Headers: {u'date': 'Fri, 28
Jul 2017 09:15:48 GMT', u'x-openstack-request-id':
'req-0502025a-db49-4f1f-b30d-c38b8098b79e', u'content-type':
'application/json', u'content-length': '582', 'content-location':
'http://149.202.183.40:9696/v2.0/networks', 'status': '201', u'connection':
'close'}
2017-07-28 09:33:52.027880 | Body:
{"network":{"status":"ACTIVE","router:external":false,"availability_zone_hints":[],"availability_zones":[],"description":"","subnets":[],"shared":false,"tenant_id":"5c851bb85bef4b008714ef04d1fe3671","created_at":"2017-07-28T09:15:48Z","tags":[],"ipv6_address_scope":null,"mtu":1450,"updated_at":"2017-07-28T09:15:48Z","admin_state_up":true,"revision_number":2,"ipv4_address_scope":null,"is_default":false,"port_security_enabled":true,"project_id":"5c851bb85bef4b008714ef04d1fe3671","id":"b8a3fb1c-86a4-4518-8c3a-dd12db585659","name":"tempest-PortsTestJSON-test-network-1596805013"}}
2017-07-28 09:33:52.027936 | 2017-07-28 09:15:49,430 9778 INFO
[tempest.lib.common.rest_client] Request
(PortsTestJSON:test_create_port_in_allowed_allocation_pools): 201 POST
http://149.202.183.40:9696/v2.0/subnets 0.682s
2017-07-28 09:33:52.027998 | 2017-07-28 09:15:49,431 9778 DEBUG
[tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token':
'', 'Accept': 'application/json', 'Content-Type': 'application/json'}
2017-07-28 09:33:52.028054 | Body: {"subnet": {"ip_version": 4,
"allocation_pools": [{"end": "10.1.0.14", "start": "10.1.0.2"}], "network_id":
"b8a3fb1c-86a4-4518-8c3a-dd12db585659", "gateway_ip": "10.1.0.1", "cidr":
"10.1.0.0/28"}}
2017-07-28 09:33:52.028135 | Response - Headers: {u'date': 'Fri, 28
Jul 2017 09:15:49 GMT', u'x-openstack-request-id':
'req-1a50b739-8683-4aaa-ba4a-6e9daf73f1c8', u'content-type':
'application/json', u'content-length': '594', 'content-location':
'http://149.202.183.40:9696/v2.0/subnets', 'status': '201', u'connection':
'close'}
2017-07-28 09:33:52.030085 | Body:
{"subnet":{"service_types":[],"description":"","enable_dhcp":true,"tags":[],"network_id":"b8a3fb1c-86a4-4518-8c3a-dd12db585659","tenant_id":"5c851bb85bef4b008714ef04d1fe3671","created_at":"2017-07-28T09:15:49Z","dns_nameservers":[],"updated_at":"2017-07-28T09:15:49Z","gateway_ip":"10.1.0.1","ipv6_ra_mode":null,"allocation_pools":[{"start":"10.1.0.2","end":"10.1.0.14"}],"host_routes":[],"revision_number":0,"ip_version":4,"ipv6_address_mode":null,"cidr":"10.1.0.0/28","project_id":"5c851bb85bef4b008714ef04d1fe3671","id":"be974b50-e56b-44a8-86a9-6bcc345f9d55","subnetpool_id":null,"name":""}}
2017-07-28 09:33:52.030176 | 2017-07-28 09:15:50,616 9778 INFO
[tempest.lib.common.rest_client] Request
(PortsTestJSON:test_create_port_in_allowed_allocation_pools): 201 POST
http://149.202.183.40:9696/v2.0/ports 1.185s
2017-07-28 09:33:52.030232 | 2017-07-28 09:15:50,617 9778 DEBUG
[tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token':
'', 'Accept': 'application/json', 'Content-Type': 'application/json'}
2017-07-28 09:33:52.030259 | Body: {"port": {"network_id":
"b8a3fb1c-86a4-4518-8c3a-dd12db585659"}}
2017-07-28 09:33:52.030369 | Response - Headers: {u'date': 'Fri, 28
Jul 2017 09:15:50 GMT', u'x-openstack-request-id':
'req-6b57ff81-c874-4e97-8183-bd57c7e8de81', u'content-type':
'application/json', u'content-length': '691', 'content-location':
[Yahoo-eng-team] [Bug 1707160] Re: test_create_port_in_allowed_allocation_pools test fails on ironic grenade
For Neutron, a workaround was committed as:
https://review.openstack.org/#/c/490305/
** Changed in: neutron
Status: Confirmed => Fix Released
** Changed in: neutron
Assignee: (unassigned) => Kevin Benton (kevinbenton)
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1707160
Title:
test_create_port_in_allowed_allocation_pools test fails on ironic
grenade
Status in neutron:
Fix Released
Status in OpenStack Compute (nova):
Incomplete
Status in oslo.messaging:
New
Bug description:
Here is an example of a job at
http://logs.openstack.org/58/487458/6/check/gate-grenade-dsvm-ironic-
ubuntu-xenial/d8f187e/console.html#_2017-07-28_09_33_52_031224
2017-07-28 09:33:52.027473 | Captured pythonlogging:
2017-07-28 09:33:52.027484 | ~~~
2017-07-28 09:33:52.027539 | 2017-07-28 09:15:48,746 9778 INFO
[tempest.lib.common.rest_client] Request
(PortsTestJSON:test_create_port_in_allowed_allocation_pools): 201 POST
http://149.202.183.40:9696/v2.0/networks 0.342s
2017-07-28 09:33:52.027604 | 2017-07-28 09:15:48,746 9778 DEBUG
[tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token':
'', 'Accept': 'application/json', 'Content-Type': 'application/json'}
2017-07-28 09:33:52.027633 | Body: {"network": {"name":
"tempest-PortsTestJSON-test-network-1596805013"}}
2017-07-28 09:33:52.027728 | Response - Headers: {u'date': 'Fri, 28
Jul 2017 09:15:48 GMT', u'x-openstack-request-id':
'req-0502025a-db49-4f1f-b30d-c38b8098b79e', u'content-type':
'application/json', u'content-length': '582', 'content-location':
'http://149.202.183.40:9696/v2.0/networks', 'status': '201', u'connection':
'close'}
2017-07-28 09:33:52.027880 | Body:
{"network":{"status":"ACTIVE","router:external":false,"availability_zone_hints":[],"availability_zones":[],"description":"","subnets":[],"shared":false,"tenant_id":"5c851bb85bef4b008714ef04d1fe3671","created_at":"2017-07-28T09:15:48Z","tags":[],"ipv6_address_scope":null,"mtu":1450,"updated_at":"2017-07-28T09:15:48Z","admin_state_up":true,"revision_number":2,"ipv4_address_scope":null,"is_default":false,"port_security_enabled":true,"project_id":"5c851bb85bef4b008714ef04d1fe3671","id":"b8a3fb1c-86a4-4518-8c3a-dd12db585659","name":"tempest-PortsTestJSON-test-network-1596805013"}}
2017-07-28 09:33:52.027936 | 2017-07-28 09:15:49,430 9778 INFO
[tempest.lib.common.rest_client] Request
(PortsTestJSON:test_create_port_in_allowed_allocation_pools): 201 POST
http://149.202.183.40:9696/v2.0/subnets 0.682s
2017-07-28 09:33:52.027998 | 2017-07-28 09:15:49,431 9778 DEBUG
[tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token':
'', 'Accept': 'application/json', 'Content-Type': 'application/json'}
2017-07-28 09:33:52.028054 | Body: {"subnet": {"ip_version": 4,
"allocation_pools": [{"end": "10.1.0.14", "start": "10.1.0.2"}], "network_id":
"b8a3fb1c-86a4-4518-8c3a-dd12db585659", "gateway_ip": "10.1.0.1", "cidr":
"10.1.0.0/28"}}
2017-07-28 09:33:52.028135 | Response - Headers: {u'date': 'Fri, 28
Jul 2017 09:15:49 GMT', u'x-openstack-request-id':
'req-1a50b739-8683-4aaa-ba4a-6e9daf73f1c8', u'content-type':
'application/json', u'content-length': '594', 'content-location':
'http://149.202.183.40:9696/v2.0/subnets', 'status': '201', u'connection':
'close'}
2017-07-28 09:33:52.030085 | Body:
{"subnet":{"service_types":[],"description":"","enable_dhcp":true,"tags":[],"network_id":"b8a3fb1c-86a4-4518-8c3a-dd12db585659","tenant_id":"5c851bb85bef4b008714ef04d1fe3671","created_at":"2017-07-28T09:15:49Z","dns_nameservers":[],"updated_at":"2017-07-28T09:15:49Z","gateway_ip":"10.1.0.1","ipv6_ra_mode":null,"allocation_pools":[{"start":"10.1.0.2","end":"10.1.0.14"}],"host_routes":[],"revision_number":0,"ip_version":4,"ipv6_address_mode":null,"cidr":"10.1.0.0/28","project_id":"5c851bb85bef4b008714ef04d1fe3671","id":"be974b50-e56b-44a8-86a9-6bcc345f9d55","subnetpool_id":null,"name":""}}
2017-07-28 09:33:52.030176 | 2017-07-28 09:15:50,616 9778 INFO
[tempest.lib.common.rest_client] Request
(PortsTestJSON:test_create_port_in_allowed_allocation_pools): 201 POST
http://149.202.183.40:9696/v2.0/ports 1.185s
2017-07-28 09:33:52.030232 | 2017-07-28 09:15:50,617 9778 DEBUG
[tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token':
'', 'Accept': 'application/json', 'Content-Type': 'application/json'}
2017-07-28 09:33:52.030259 | Body: {"port": {"network_id":
"b8a3fb1c-86a4-4518-8c3a-dd12db585659"}}
2017-07-28 09:33:52.030369 | Response - Headers: {u'date': 'Fri, 28
Jul 2017 09:15:50 GMT', u'x-openstack-request-id':
'req-6b57ff81-c874-4e97-8183-bd57c7e8de81', u'content-type':
'application/json', u'content-length': '691', 'content-location':
[Yahoo-eng-team] [Bug 1707160] Re: test_create_port_in_allowed_allocation_pools test fails on ironic grenade
OK, I still see this failing on a neutron api request here:
http://logs.openstack.org/01/489701/1/check/gate-grenade-dsvm-ironic-
ubuntu-xenial/50785e5/logs/testr_results.html.gz
In neutron-server log, we see first NotFound ERROR message from the
missing fanout exchange (that's not an issue per se, and I sent a patch
to downgrade the message to DEBUG [1]); then the request handler
attempts to fanout another event to agents; then request is locked for
~115 seconds to end with:
2017-08-01 22:50:59.696 1787 ERROR oslo.messaging._drivers.impl_rabbit
[req-765ebd21-ee44-4db2-85f8-f6b7e52cdf19 tempest-RoutersTest-175057974
tempest-RoutersTest-175057974] [fdb10a5b-b3c8-436c-ba74-193c721329bb] AMQP
server on 104.130.216.130:5672 is unreachable: timed out. Trying again in 1
seconds. Client port: None: timeout: timed out
2017-08-01 22:51:00.715 1787 INFO oslo.messaging._drivers.impl_rabbit
[req-765ebd21-ee44-4db2-85f8-f6b7e52cdf19 tempest-RoutersTest-175057974
tempest-RoutersTest-175057974] [fdb10a5b-b3c8-436c-ba74-193c721329bb]
Reconnected to AMQP server on 104.130.216.130:5672 via [amqp] client with port
51878.
When we look at rabbitmq logs around the time of failure we see:
=ERROR REPORT 1-Aug-2017::22:49:05 ===
Channel error on connection <0.3221.0> (104.130.216.130:43578 ->
104.130.216.130:5672, vhost: '/', user: 'stackrabbit'), channel 1:
{amqp_error,not_found,
"no exchange 'q-agent-notifier-port-delete_fanout' in vhost '/'",
'basic.publish'}
=WARNING REPORT 1-Aug-2017::22:50:59 ===
closing AMQP connection <0.3221.0> (104.130.216.130:43578 ->
104.130.216.130:5672):
connection_closed_abruptly
I suspect that after NotFound is raised, oslo.messaging channels get
into a state that doesn't allow to send any more messages, and only
connection reset restores connectivity.
[1] https://review.openstack.org/#/c/490139/
** Changed in: neutron
Status: Fix Released => Confirmed
** Changed in: neutron
Milestone: None => pike-3
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1707160
Title:
test_create_port_in_allowed_allocation_pools test fails on ironic
grenade
Status in neutron:
Confirmed
Status in OpenStack Compute (nova):
New
Bug description:
Here is an example of a job at
http://logs.openstack.org/58/487458/6/check/gate-grenade-dsvm-ironic-
ubuntu-xenial/d8f187e/console.html#_2017-07-28_09_33_52_031224
2017-07-28 09:33:52.027473 | Captured pythonlogging:
2017-07-28 09:33:52.027484 | ~~~
2017-07-28 09:33:52.027539 | 2017-07-28 09:15:48,746 9778 INFO
[tempest.lib.common.rest_client] Request
(PortsTestJSON:test_create_port_in_allowed_allocation_pools): 201 POST
http://149.202.183.40:9696/v2.0/networks 0.342s
2017-07-28 09:33:52.027604 | 2017-07-28 09:15:48,746 9778 DEBUG
[tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token':
'', 'Accept': 'application/json', 'Content-Type': 'application/json'}
2017-07-28 09:33:52.027633 | Body: {"network": {"name":
"tempest-PortsTestJSON-test-network-1596805013"}}
2017-07-28 09:33:52.027728 | Response - Headers: {u'date': 'Fri, 28
Jul 2017 09:15:48 GMT', u'x-openstack-request-id':
'req-0502025a-db49-4f1f-b30d-c38b8098b79e', u'content-type':
'application/json', u'content-length': '582', 'content-location':
'http://149.202.183.40:9696/v2.0/networks', 'status': '201', u'connection':
'close'}
2017-07-28 09:33:52.027880 | Body:
{"network":{"status":"ACTIVE","router:external":false,"availability_zone_hints":[],"availability_zones":[],"description":"","subnets":[],"shared":false,"tenant_id":"5c851bb85bef4b008714ef04d1fe3671","created_at":"2017-07-28T09:15:48Z","tags":[],"ipv6_address_scope":null,"mtu":1450,"updated_at":"2017-07-28T09:15:48Z","admin_state_up":true,"revision_number":2,"ipv4_address_scope":null,"is_default":false,"port_security_enabled":true,"project_id":"5c851bb85bef4b008714ef04d1fe3671","id":"b8a3fb1c-86a4-4518-8c3a-dd12db585659","name":"tempest-PortsTestJSON-test-network-1596805013"}}
2017-07-28 09:33:52.027936 | 2017-07-28 09:15:49,430 9778 INFO
[tempest.lib.common.rest_client] Request
(PortsTestJSON:test_create_port_in_allowed_allocation_pools): 201 POST
http://149.202.183.40:9696/v2.0/subnets 0.682s
2017-07-28 09:33:52.027998 | 2017-07-28 09:15:49,431 9778 DEBUG
[tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token':
'', 'Accept': 'application/json', 'Content-Type': 'application/json'}
2017-07-28 09:33:52.028054 | Body: {"subnet": {"ip_version": 4,
"allocation_pools": [{"end": "10.1.0.14", "start": "10.1.0.2"}], "network_id":
"b8a3fb1c-86a4-4518-8c3a-dd12db585659", "gateway_ip": "10.1.0.1", "cidr":
"10.1.0.0/28"}}
2017-07-28 09:33:52.028135 | Response - Headers: {u'date': 'Fri, 28
Jul 2017 09:15:49 GMT',
[Yahoo-eng-team] [Bug 1707160] Re: test_create_port_in_allowed_allocation_pools test fails on ironic grenade
In the last failure, it's not neutron API but nova API that times out:
http://logs.openstack.org/99/489699/1/check/gate-grenade-dsvm-ironic-
ubuntu-xenial/2e8b6fa/logs/testr_results.html.gz
The relevant log messages for the Nova API request start at:
http://logs.openstack.org/99/489699/1/check/gate-grenade-dsvm-ironic-
ubuntu-
xenial/2e8b6fa/logs/new/screen-n-api.txt.gz#_2017-08-02_17_40_21_213
2017-08-02 17:40:21.149 30615 DEBUG nova.osapi_compute.wsgi.server
[req-84752f96-9ca4-43dd-86ed-64620eb7d8ca - -] (30615) accepted ('10.0.1.169',
52578) server /usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py:883
2017-08-02 17:40:21.213 30615 DEBUG nova.api.openstack.wsgi
[req-04a0710f-c696-415e-a2a7-1381bf0174b1 tempest-TestServerBasicOps-512068857
tempest-TestServerBasicOps-512068857] Action: 'create', calling method:
, body: {"security_group": {"name":
"tempest-TestServerBasicOps-722492525", "description":
"tempest-TestServerBasicOps-722492525 description"}} _process_stack
/opt/stack/new/nova/nova/api/openstack/wsgi.py:609
2017-08-02 17:40:21.619 30615 INFO nova.osapi_compute.wsgi.server
[req-04a0710f-c696-415e-a2a7-1381bf0174b1 tempest-TestServerBasicOps-512068857
tempest-TestServerBasicOps-512068857] 10.0.1.169 "POST /v2.1/os-security-groups
HTTP/1.1" status: 200 len: 637 time: 0.4696331
2017-08-02 17:40:21.623 30615 DEBUG nova.osapi_compute.wsgi.server
[req-84752f96-9ca4-43dd-86ed-64620eb7d8ca - -] (30615) accepted ('10.0.1.169',
52582) server /usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py:883
2017-08-02 17:40:21.651 30615 DEBUG nova.api.openstack.wsgi
[req-983ceec8-1c1e-4d6e-b560-9f4aa358bf6a tempest-TestServerBasicOps-512068857
tempest-TestServerBasicOps-512068857] Action: 'create', calling method:
, body: {"security_group_rule":
{"from_port": 22, "to_port": 22, "ip_protocol": "tcp", "cidr": "0.0.0.0/0",
"parent_group_id": "1da383d4-6d09-451b-8e87-1d708803ae02"}} _process_stack
/opt/stack/new/nova/nova/api/openstack/wsgi.py:609
There seems to be no completion for the request.
** Also affects: nova
Importance: Undecided
Status: New
** Changed in: neutron
Status: In Progress => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1707160
Title:
test_create_port_in_allowed_allocation_pools test fails on ironic
grenade
Status in neutron:
Fix Released
Status in OpenStack Compute (nova):
New
Bug description:
Here is an example of a job at
http://logs.openstack.org/58/487458/6/check/gate-grenade-dsvm-ironic-
ubuntu-xenial/d8f187e/console.html#_2017-07-28_09_33_52_031224
2017-07-28 09:33:52.027473 | Captured pythonlogging:
2017-07-28 09:33:52.027484 | ~~~
2017-07-28 09:33:52.027539 | 2017-07-28 09:15:48,746 9778 INFO
[tempest.lib.common.rest_client] Request
(PortsTestJSON:test_create_port_in_allowed_allocation_pools): 201 POST
http://149.202.183.40:9696/v2.0/networks 0.342s
2017-07-28 09:33:52.027604 | 2017-07-28 09:15:48,746 9778 DEBUG
[tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token':
'', 'Accept': 'application/json', 'Content-Type': 'application/json'}
2017-07-28 09:33:52.027633 | Body: {"network": {"name":
"tempest-PortsTestJSON-test-network-1596805013"}}
2017-07-28 09:33:52.027728 | Response - Headers: {u'date': 'Fri, 28
Jul 2017 09:15:48 GMT', u'x-openstack-request-id':
'req-0502025a-db49-4f1f-b30d-c38b8098b79e', u'content-type':
'application/json', u'content-length': '582', 'content-location':
'http://149.202.183.40:9696/v2.0/networks', 'status': '201', u'connection':
'close'}
2017-07-28 09:33:52.027880 | Body:
{"network":{"status":"ACTIVE","router:external":false,"availability_zone_hints":[],"availability_zones":[],"description":"","subnets":[],"shared":false,"tenant_id":"5c851bb85bef4b008714ef04d1fe3671","created_at":"2017-07-28T09:15:48Z","tags":[],"ipv6_address_scope":null,"mtu":1450,"updated_at":"2017-07-28T09:15:48Z","admin_state_up":true,"revision_number":2,"ipv4_address_scope":null,"is_default":false,"port_security_enabled":true,"project_id":"5c851bb85bef4b008714ef04d1fe3671","id":"b8a3fb1c-86a4-4518-8c3a-dd12db585659","name":"tempest-PortsTestJSON-test-network-1596805013"}}
2017-07-28 09:33:52.027936 | 2017-07-28 09:15:49,430 9778 INFO
[tempest.lib.common.rest_client] Request
(PortsTestJSON:test_create_port_in_allowed_allocation_pools): 201 POST
http://149.202.183.40:9696/v2.0/subnets 0.682s
2017-07-28 09:33:52.027998 | 2017-07-28 09:15:49,431 9778 DEBUG
[tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token':
'', 'Accept': 'application/json', 'Content-Type': 'application/json'}
2017-07-28 09:33:52.028054 | Body: {"subnet": {"ip_version": 4,
"allocation_pools": [{"end": "10.1.0.14", "start": "10.1.0.2"}], "network_id":
[Yahoo-eng-team] [Bug 1707933] [NEW] functional tests timeout after a test worker killed
Public bug reported: Grafana failure rate at 30%; triage of July gate failures for the job suggested that most of them come from the same root cause: some test workers killed during test execution: https://etherpad.openstack.org/p/neutron-functional-gate-failures-july When it happens, we see the following line in console log: 2017-07-13 17:54:12.742494 | 2017-07-13 17:54:12.742 | Killed Most of the times it happens during FirewallTestCase execution, which is suspicious. Kuba sent a test patch to collect information about callers to 'kill' CLI: https://review.openstack.org/#/c/487065/ but it hasn't revealed much. ** Affects: neutron Importance: Critical Status: Confirmed ** Tags: functional-tests gate-failure ** Changed in: neutron Importance: Undecided => Critical ** Changed in: neutron Status: New => Confirmed ** Tags added: gate ** Tags removed: gate ** Tags added: functional-tests gate-failure -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1707933 Title: functional tests timeout after a test worker killed Status in neutron: Confirmed Bug description: Grafana failure rate at 30%; triage of July gate failures for the job suggested that most of them come from the same root cause: some test workers killed during test execution: https://etherpad.openstack.org/p/neutron-functional-gate-failures-july When it happens, we see the following line in console log: 2017-07-13 17:54:12.742494 | 2017-07-13 17:54:12.742 | Killed Most of the times it happens during FirewallTestCase execution, which is suspicious. Kuba sent a test patch to collect information about callers to 'kill' CLI: https://review.openstack.org/#/c/487065/ but it hasn't revealed much. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1707933/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1704000] Re: Sometimes OVO unit tests clash on non-unique attributes
I believe this is fixed with https://review.openstack.org/485082
** Changed in: neutron
Status: In Progress => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1704000
Title:
Sometimes OVO unit tests clash on non-unique attributes
Status in neutron:
Fix Released
Bug description:
ft1.22:
neutron.tests.unit.objects.test_l3agent.RouterL3AgentBindingDbObjTestCase.test_update_objects_StringException:
Traceback (most recent call last):
File "neutron/tests/base.py", line 118, in func
return f(self, *args, **kwargs)
File "neutron/tests/unit/objects/test_base.py", line 1848, in
test_update_objects
self.context, new_values, **keys)
File "neutron/objects/base.py", line 494, in update_objects
**cls.modify_fields_to_db(kwargs))
File "neutron/objects/db/api.py", line 103, in update_objects
return q.update(values, synchronize_session=False)
File
"/home/jenkins/workspace/gate-neutron-python27-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py",
line 3345, in update
update_op.exec_()
File
"/home/jenkins/workspace/gate-neutron-python27-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py",
line 1179, in exec_
self._do_exec()
File
"/home/jenkins/workspace/gate-neutron-python27-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py",
line 1334, in _do_exec
mapper=self.mapper)
File
"/home/jenkins/workspace/gate-neutron-python27-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py",
line 1139, in execute
bind, close_with_result=True).execute(clause, params or {})
File
"/home/jenkins/workspace/gate-neutron-python27-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
line 945, in execute
return meth(self, multiparams, params)
File
"/home/jenkins/workspace/gate-neutron-python27-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/sqlalchemy/sql/elements.py",
line 263, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File
"/home/jenkins/workspace/gate-neutron-python27-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
line 1053, in _execute_clauseelement
compiled_sql, distilled_params
File
"/home/jenkins/workspace/gate-neutron-python27-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
line 1189, in _execute_context
context)
File
"/home/jenkins/workspace/gate-neutron-python27-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
line 1398, in _handle_dbapi_exception
util.raise_from_cause(newraise, exc_info)
File
"/home/jenkins/workspace/gate-neutron-python27-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py",
line 203, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File
"/home/jenkins/workspace/gate-neutron-python27-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
line 1182, in _execute_context
context)
File
"/home/jenkins/workspace/gate-neutron-python27-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py",
line 470, in do_execute
cursor.execute(statement, parameters)
oslo_db.exception.DBDuplicateEntry: (sqlite3.IntegrityError) UNIQUE
constraint failed: routerl3agentbindings.router_id,
routerl3agentbindings.binding_index [SQL: u'UPDATE routerl3agentbindings SET
binding_index=? WHERE routerl3agentbindings.router_id IN (?) AND
routerl3agentbindings.l3_agent_id IN (?)'] [parameters: (2,
'2a0036c0-dfc8-4dee-b6e8-ae7039abb5e0', '98fa8bc4-d5be-422f-88a2-cfcd78a7f2d6')]
http://logs.openstack.org/73/304873/45/check/gate-neutron-python27
-ubuntu-xenial/c6512d6/testr_results.html.gz
This is because of self.update_obj_fields used in several test classes
not updating unique_tracker in the base test class that makes sure no
non-unique values are generated by get_random_object_fields. We
probably need to make those two cooperate.
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1704000/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1707688] [NEW] "error: [Errno 32] Broken pipe" when plugging OVS VIF into an instance
Public bug reported: http://logs.openstack.org/85/486585/1/gate/gate-tempest-dsvm-neutron- dvr-ubuntu- xenial/acc8282/logs/screen-n-cpu.txt.gz#_2017-07-28_08_35_03_006 2017-07-28 08:35:03.006 1542 ERROR os_vif [req-398df22e-6d9b-4203-b2a9-1d7bc2011939 tempest-TestVolumeBootPattern-1767728258 tempest-TestVolumeBootPattern-1767728258] Failed to plug vif VIFBridge(active=False,address=fa:16:3e:fa:58:82,bridge_name='qbrd3b698dd-61',has_traffic_filtering=True,id=d3b698dd-61b1-4cf6-8967-5f57b204bcc1,network=Network(4cec1154-d5c3-4b79-9172-a20f55ea2c03),plugin='ovs',port_profile=VIFPortProfileBase,preserve_on_delete=False,vif_name='tapd3b698dd-61') 2017-07-28 08:35:03.006 1542 ERROR os_vif Traceback (most recent call last): 2017-07-28 08:35:03.006 1542 ERROR os_vif File "/usr/local/lib/python2.7/dist-packages/os_vif/__init__.py", line 80, in plug 2017-07-28 08:35:03.006 1542 ERROR os_vif plugin.plug(vif, instance_info) 2017-07-28 08:35:03.006 1542 ERROR os_vif File "/usr/local/lib/python2.7/dist-packages/vif_plug_ovs/ovs.py", line 170, in plug 2017-07-28 08:35:03.006 1542 ERROR os_vif self._plug_bridge(vif, instance_info) 2017-07-28 08:35:03.006 1542 ERROR os_vif File "/usr/local/lib/python2.7/dist-packages/vif_plug_ovs/ovs.py", line 133, in _plug_bridge 2017-07-28 08:35:03.006 1542 ERROR os_vif linux_net.ensure_bridge(vif.bridge_name) 2017-07-28 08:35:03.006 1542 ERROR os_vif File "/usr/local/lib/python2.7/dist-packages/oslo_privsep/priv_context.py", line 205, in _wrap 2017-07-28 08:35:03.006 1542 ERROR os_vif return self.channel.remote_call(name, args, kwargs) 2017-07-28 08:35:03.006 1542 ERROR os_vif File "/usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py", line 177, in remote_call 2017-07-28 08:35:03.006 1542 ERROR os_vif result = self.send_recv((Message.CALL.value, name, args, kwargs)) 2017-07-28 08:35:03.006 1542 ERROR os_vif File "/usr/local/lib/python2.7/dist-packages/oslo_privsep/comm.py", line 160, in send_recv 2017-07-28 08:35:03.006 1542 ERROR os_vif self.writer.send((myid, msg)) 2017-07-28 08:35:03.006 1542 ERROR os_vif File "/usr/local/lib/python2.7/dist-packages/oslo_privsep/comm.py", line 55, in send 2017-07-28 08:35:03.006 1542 ERROR os_vif self.writesock.sendall(buf) 2017-07-28 08:35:03.006 1542 ERROR os_vif File "/usr/local/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 385, in sendall 2017-07-28 08:35:03.006 1542 ERROR os_vif tail = self.send(data, flags) 2017-07-28 08:35:03.006 1542 ERROR os_vif File "/usr/local/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 379, in send 2017-07-28 08:35:03.006 1542 ERROR os_vif return self._send_loop(self.fd.send, data, flags) 2017-07-28 08:35:03.006 1542 ERROR os_vif File "/usr/local/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 366, in _send_loop 2017-07-28 08:35:03.006 1542 ERROR os_vif return send_method(data, *args) 2017-07-28 08:35:03.006 1542 ERROR os_vif error: [Errno 32] Broken pipe 2017-07-28 08:35:03.006 1542 ERROR os_vif 2017-07-28 08:35:03.007 1542 ERROR nova.virt.libvirt.driver [req-398df22e-6d9b-4203-b2a9-1d7bc2011939 tempest-TestVolumeBootPattern-1767728258 tempest-TestVolumeBootPattern-1767728258] [instance: dffc7798-8ac8-4de6-9991-4930c5805208] Failed to start libvirt guest Looking at libvirtd log, there is nothing that would immediately suggest what happened, but for the reference, that's what I see there: 2017-07-28 08:35:03.689+: 11804: debug : virThreadJobSet:96 : Thread 11804 (virNetServerHandleJob) is now running job remoteDispatchConnectGetLibVersion 2017-07-28 08:35:03.689+: 11804: debug : virThreadJobClear:121 : Thread 11804 (virNetServerHandleJob) finished job remoteDispatchConnectGetLibVersion with ret=0 2017-07-28 08:35:03.689+: 11807: debug : virThreadJobSet:96 : Thread 11807 (virNetServerHandleJob) is now running job remoteDispatchDomainLookupByName 2017-07-28 08:35:03.689+: 11807: debug : qemuDomainLookupByName:1595 : Domain not found: no domain with matching name 'instance-0081' 2017-07-28 08:35:03.689+: 11807: debug : virThreadJobClear:121 : Thread 11807 (virNetServerHandleJob) finished job remoteDispatchDomainLookupByName with ret=-1 2017-07-28 08:35:03.697+: 11802: debug : virThreadJobSet:96 : Thread 11802 (virNetServerHandleJob) is now running job remoteDispatchConnectGetLibVersion 2017-07-28 08:35:03.697+: 11802: debug : virThreadJobClear:121 : Thread 11802 (virNetServerHandleJob) finished job remoteDispatchConnectGetLibVersion with ret=0 2017-07-28 08:35:03.698+: 11806: debug : virThreadJobSet:96 : Thread 11806 (virNetServerHandleJob) is now running job remoteDispatchDomainLookupByName 2017-07-28 08:35:03.698+: 11806: debug : qemuDomainLookupByName:1595 : Domain not found: no domain with matching name 'instance-0081' 2017-07-28 08:35:03.698+: 11806: debug : virThreadJobClear:121 : Thread 11806
[Yahoo-eng-team] [Bug 1706750] [NEW] FlushError for automatic IPAllocations when subnet create is retried
eutron/objects/base.py", line 632, in create
2017-07-13 15:20:51.064 85185 ERROR neutron.pecan_wsgi.hooks.translation
self.modify_fields_to_db(fields))
2017-07-13 15:20:51.064 85185 ERROR neutron.pecan_wsgi.hooks.translation File
"/usr/lib/python2.7/site-packages/neutron/objects/db/api.py", line 61, in
create_object
2017-07-13 15:20:51.064 85185 ERROR neutron.pecan_wsgi.hooks.translation
context.session.add(db_obj)
2017-07-13 15:20:51.064 85185 ERROR neutron.pecan_wsgi.hooks.translation File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 567, in
__exit__
2017-07-13 15:20:51.064 85185 ERROR neutron.pecan_wsgi.hooks.translation
self.rollback()
2017-07-13 15:20:51.064 85185 ERROR neutron.pecan_wsgi.hooks.translation File
"/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66,
in __exit__
2017-07-13 15:20:51.064 85185 ERROR neutron.pecan_wsgi.hooks.translation
compat.reraise(exc_type, exc_value, exc_tb)
2017-07-13 15:20:51.064 85185 ERROR neutron.pecan_wsgi.hooks.translation File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 564, in
__exit__
2017-07-13 15:20:51.064 85185 ERROR neutron.pecan_wsgi.hooks.translation
self.commit()
2017-07-13 15:20:51.064 85185 ERROR neutron.pecan_wsgi.hooks.translation File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 461, in
commit
2017-07-13 15:20:51.064 85185 ERROR neutron.pecan_wsgi.hooks.translation
self._prepare_impl()
2017-07-13 15:20:51.064 85185 ERROR neutron.pecan_wsgi.hooks.translation File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 441, in
_prepare_impl
2017-07-13 15:20:51.064 85185 ERROR neutron.pecan_wsgi.hooks.translation
self.session.flush()
2017-07-13 15:20:51.064 85185 ERROR neutron.pecan_wsgi.hooks.translation File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2171, in
flush
2017-07-13 15:20:51.064 85185 ERROR neutron.pecan_wsgi.hooks.translation
self._flush(objects)
2017-07-13 15:20:51.064 85185 ERROR neutron.pecan_wsgi.hooks.translation File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2291, in
_flush
2017-07-13 15:20:51.064 85185 ERROR neutron.pecan_wsgi.hooks.translation
transaction.rollback(_capture_exception=True)
2017-07-13 15:20:51.064 85185 ERROR neutron.pecan_wsgi.hooks.translation File
"/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66,
in __exit__
2017-07-13 15:20:51.064 85185 ERROR neutron.pecan_wsgi.hooks.translation
compat.reraise(exc_type, exc_value, exc_tb)
2017-07-13 15:20:51.064 85185 ERROR neutron.pecan_wsgi.hooks.translation File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2255, in
_flush
2017-07-13 15:20:51.064 85185 ERROR neutron.pecan_wsgi.hooks.translation
flush_context.execute()
2017-07-13 15:20:51.064 85185 ERROR neutron.pecan_wsgi.hooks.translation File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 389, in
execute
2017-07-13 15:20:51.064 85185 ERROR neutron.pecan_wsgi.hooks.translation
rec.execute(self)
2017-07-13 15:20:51.064 85185 ERROR neutron.pecan_wsgi.hooks.translation File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 548, in
execute
2017-07-13 15:20:51.064 85185 ERROR neutron.pecan_wsgi.hooks.translation uow
2017-07-13 15:20:51.064 85185 ERROR neutron.pecan_wsgi.hooks.translation File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 156,
in save_obj
2017-07-13 15:20:51.064 85185 ERROR neutron.pecan_wsgi.hooks.translation
base_mapper, states, uowtransaction
2017-07-13 15:20:51.064 85185 ERROR neutron.pecan_wsgi.hooks.translation File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 312,
in _organize_states_for_save
2017-07-13 15:20:51.064 85185 ERROR neutron.pecan_wsgi.hooks.translation
state_str(existing)))
2017-07-13 15:20:51.064 85185 ERROR neutron.pecan_wsgi.hooks.translation
FlushError: New instance with identity key (, ('2003::1:f816:3eff:fe3f:aa00',
'8032a1cd-9236-4126-83fa-2f039ed2b890',
'c0542b84-bcd0-48e1-ace1-6da1f50246dc')) conflicts with persistent instance
2017-07-13 15:20:51.064 85185 ERROR neutron.pecan_wsgi.hooks.translation
Just before the traceback, we see retry triggered by StaleDataError:
2017-07-13 15:20:50.668 85185 DEBUG neutron.db.api
[req-3c97d83e-3beb-4b28-a29f-697fc1ee8bc6 5f2e0d72ef8e4fb18b500759cfe5e503
205f2a3630a0426f8da20b4467ea003b - default default] Retry wrapper got retriable
exception: UPDATE statement on table 'standardattributes' expected to update 1
row(s); 0 were matched. wrapped
/usr/lib/python2.7/site-packages/neutron/db/api.py:129
2017-07-13 15:20:50.669 85185 DEBUG oslo_db.api
[req-3c97d83e-
[Yahoo-eng-team] [Bug 1705567] [NEW] gate-tempest-dsvm-neutron-linuxbridge-ubuntu-xenial failed with timeout
Public bug reported:
http://logs.openstack.org/85/385085/30/gate/gate-tempest-dsvm-neutron-
linuxbridge-ubuntu-xenial/75e271d/console.html
2017-07-20 19:18:32.693567 | full runtests: commands[2] | tempest run --combine
--serial --regex (?!.*\[.*\bslow\b.*\])(^tempest\.scenario) --concurrency=4
2017-07-20 19:18:38.317292 | running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
2017-07-20 19:18:38.317404 | OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
2017-07-20 19:18:38.317449 | OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
2017-07-20 19:18:38.317501 |
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
2017-07-20 19:18:38.317574 | ${PYTHON:-python} -m subunit.run discover -t
${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --list
2017-07-20 19:18:38.317621 | running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
2017-07-20 19:18:38.317657 | OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
2017-07-20 19:18:38.317692 | OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
2017-07-20 19:18:38.317733 |
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
2017-07-20 19:18:38.317814 | ${PYTHON:-python} -m subunit.run discover -t
${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --load-list
/tmp/tmpqrNAOh
2017-07-20 19:20:27.479411 | {0}
tempest.scenario.test_minimum_basic.TestMinimumBasicScenario.test_minimum_basic_scenario
[101.261843s] ... ok
2017-07-20 19:20:27.479460 |
2017-07-20 19:20:27.479481 | Captured stderr:
2017-07-20 19:20:27.479498 |
2017-07-20 19:20:27.479565 |
/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/cryptography/hazmat/backends/openssl/rsa.py:477:
DeprecationWarning: signer and verifier have been deprecated. Please use sign
and verify instead.
2017-07-20 19:20:27.479587 | _warn_sign_verify_deprecated()
2017-07-20 19:20:27.479600 |
2017-07-20 19:22:09.799157 | {0}
tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_reboot
[93.151292s] ... ok
2017-07-20 19:23:37.650731 | {0}
tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_hotplug_nic
[84.459821s] ... ok
2017-07-20 19:25:34.994829 | {0}
tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops
[117.306077s] ... ok
2017-07-20 19:28:02.218494 | {0}
tempest.scenario.test_network_v6.TestGettingAddress.test_dualnet_multi_prefix_slaac
[143.769198s] ... ok
2017-07-20 19:28:05.681801 | {0}
tempest.scenario.test_object_storage_basic_ops.TestObjectStorageBasicOps.test_swift_basic_ops
[0.381328s] ... ok
2017-07-20 19:30:17.429097 | {0}
tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_cross_tenant_traffic
[128.538566s] ... ok
2017-07-20 19:31:43.217495 | {0}
tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_in_tenant_traffic
[85.784122s] ... ok
2017-07-20 19:32:41.287100 | {0}
tempest.scenario.test_server_basic_ops.TestServerBasicOps.test_server_basic_ops
[49.211398s] ... ok
2017-07-20 19:32:48.257653 | {0} setUpClass
(tempest.scenario.test_server_multinode.TestServerMultinode) ... SKIPPED: Less
than 2 compute nodes, skipping multinode tests.
2017-07-20 19:33:30.275095 | {0}
tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_boot_server_from_encrypted_volume_luks
[30.113681s] ... ok
2017-07-20 19:34:42.975903 | {0}
tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_create_ebs_image_and_check_boot
[72.662936s] ... ok
2017-07-20 19:37:50.719757 | {0}
tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_volume_boot_pattern
[187.710856s] ... ok
2017-07-20 19:38:03.835802 |
/home/jenkins/workspace/gate-tempest-dsvm-neutron-linuxbridge-ubuntu-xenial/devstack-gate/functions.sh:
line 1127: 12175 Killed timeout -s 9 ${REMAINING_TIME}m bash
-c "source $WORKSPACE/devstack-gate/functions.sh && $cmd"
2017-07-20 19:38:03.835887 | ERROR: the main setup script run by this job
failed - exit code: 137
2017-07-20 19:38:03.845801 | please look at the relevant log files to
determine the root cause
It's not fully clear why despite --concurrency=4, it looks like a single
test thread is running. That being said, the slow tests had only 20
minutes to run before being abrupted, so it may as well be an issue in
the previous phase of the job.
** Affects: neutron
Importance: High
Status: New
** Tags: gate-failure linuxbridge tempest
** Changed in: neutron
Importance: Undecided => High
** Tags added: gate-failure linuxbridge
** Tags added: tempest
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1705567
Title:
gate-tempest-dsvm-neutron-linuxbridge-ubuntu-xenial failed with
timeout
Status in neutron:
New
Bug description:
http://logs.openstack.org/85/385085/30/gate/gate-tempest-dsvm-neutron-
[Yahoo-eng-team] [Bug 1704000] [NEW] Sometimes OVO unit tests clash on non-unique attributes
Public bug reported:
ft1.22:
neutron.tests.unit.objects.test_l3agent.RouterL3AgentBindingDbObjTestCase.test_update_objects_StringException:
Traceback (most recent call last):
File "neutron/tests/base.py", line 118, in func
return f(self, *args, **kwargs)
File "neutron/tests/unit/objects/test_base.py", line 1848, in
test_update_objects
self.context, new_values, **keys)
File "neutron/objects/base.py", line 494, in update_objects
**cls.modify_fields_to_db(kwargs))
File "neutron/objects/db/api.py", line 103, in update_objects
return q.update(values, synchronize_session=False)
File
"/home/jenkins/workspace/gate-neutron-python27-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py",
line 3345, in update
update_op.exec_()
File
"/home/jenkins/workspace/gate-neutron-python27-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py",
line 1179, in exec_
self._do_exec()
File
"/home/jenkins/workspace/gate-neutron-python27-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py",
line 1334, in _do_exec
mapper=self.mapper)
File
"/home/jenkins/workspace/gate-neutron-python27-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py",
line 1139, in execute
bind, close_with_result=True).execute(clause, params or {})
File
"/home/jenkins/workspace/gate-neutron-python27-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
line 945, in execute
return meth(self, multiparams, params)
File
"/home/jenkins/workspace/gate-neutron-python27-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/sqlalchemy/sql/elements.py",
line 263, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File
"/home/jenkins/workspace/gate-neutron-python27-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
line 1053, in _execute_clauseelement
compiled_sql, distilled_params
File
"/home/jenkins/workspace/gate-neutron-python27-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
line 1189, in _execute_context
context)
File
"/home/jenkins/workspace/gate-neutron-python27-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
line 1398, in _handle_dbapi_exception
util.raise_from_cause(newraise, exc_info)
File
"/home/jenkins/workspace/gate-neutron-python27-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py",
line 203, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File
"/home/jenkins/workspace/gate-neutron-python27-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
line 1182, in _execute_context
context)
File
"/home/jenkins/workspace/gate-neutron-python27-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py",
line 470, in do_execute
cursor.execute(statement, parameters)
oslo_db.exception.DBDuplicateEntry: (sqlite3.IntegrityError) UNIQUE constraint
failed: routerl3agentbindings.router_id, routerl3agentbindings.binding_index
[SQL: u'UPDATE routerl3agentbindings SET binding_index=? WHERE
routerl3agentbindings.router_id IN (?) AND routerl3agentbindings.l3_agent_id IN
(?)'] [parameters: (2, '2a0036c0-dfc8-4dee-b6e8-ae7039abb5e0',
'98fa8bc4-d5be-422f-88a2-cfcd78a7f2d6')]
http://logs.openstack.org/73/304873/45/check/gate-neutron-python27
-ubuntu-xenial/c6512d6/testr_results.html.gz
This is because of self.update_obj_fields used in several test classes
not updating unique_tracker in the base test class that makes sure no
non-unique values are generated by get_random_object_fields. We probably
need to make those two cooperate.
** Affects: neutron
Importance: High
Status: Confirmed
** Tags: gate-failure unittest
** Changed in: neutron
Importance: Undecided => High
** Changed in: neutron
Status: New => Confirmed
** Tags added: gate-failure unittest
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1704000
Title:
Sometimes OVO unit tests clash on non-unique attributes
Status in neutron:
Confirmed
Bug description:
ft1.22:
neutron.tests.unit.objects.test_l3agent.RouterL3AgentBindingDbObjTestCase.test_update_objects_StringException:
Traceback (most recent call last):
File "neutron/tests/base.py", line 118, in func
return f(self, *args, **kwargs)
File "neutron/tests/unit/objects/test_base.py", line 1848, in
test_update_objects
self.context, new_values, **keys)
File "neutron/objects/base.py", line 494, in update_objects
**cls.modify_fields_to_db(kwargs))
File "neutron/objects/db/api.py", line 103, in update_objects
return q.update(values,
[Yahoo-eng-team] [Bug 1450067] Re: Server with ML2 & L3 service plugin exposes dvr extension even if OVS agent is unused
Now that we have a config knob for this, I believe we can claim it's fixed from neutron side. I understand there are some hard feelings about the solution, and we can revisit the way we tackle it later, but from short sight it seems like we can close the bug. ** Changed in: neutron Assignee: (unassigned) => Ihar Hrachyshka (ihar-hrachyshka) ** Changed in: neutron Status: Confirmed => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1450067 Title: Server with ML2 & L3 service plugin exposes dvr extension even if OVS agent is unused Status in neutron: Fix Released Bug description: In a deployment using the L3 service plugin, the DVR extension is always declared as available, this is even if the ML2 OVS mech driver is not configured. Deployments could be using the LB mech driver or others. Not only is the extension declared, DVR router creation is not blocked. We should not rely only on documentation, but additionally provide expected behavior (Fail gracefully, not expose the extension). To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1450067/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1703938] [NEW] AttributeError: 'PortContext' object has no attribute 'session' in l3_hamode_db
Public bug reported: Jul 11 20:08:35.720679 ubuntu-xenial-3-node-osic-cloud1-s3500-9770546 neutron-server[27121]: ERROR neutron.plugins.ml2.managers [None req-13c07cf3-201f-4b86-9e92-8f51bd141c6c admin admin] Mechanism driver 'l2population' failed in delete_port_postcommit: AttributeError: 'PortContext' object has no attribute 'session' Jul 11 20:08:35.720775 ubuntu-xenial-3-node-osic-cloud1-s3500-9770546 neutron-server[27121]: ERROR neutron.plugins.ml2.managers Traceback (most recent call last): Jul 11 20:08:35.720895 ubuntu-xenial-3-node-osic-cloud1-s3500-9770546 neutron-server[27121]: ERROR neutron.plugins.ml2.managers File "/opt/stack/new/neutron/neutron/plugins/ml2/managers.py", line 426, in _call_on_drivers Jul 11 20:08:35.720971 ubuntu-xenial-3-node-osic-cloud1-s3500-9770546 neutron-server[27121]: ERROR neutron.plugins.ml2.managers getattr(driver.obj, method_name)(context) Jul 11 20:08:35.721056 ubuntu-xenial-3-node-osic-cloud1-s3500-9770546 neutron-server[27121]: ERROR neutron.plugins.ml2.managers File "/opt/stack/new/neutron/neutron/plugins/ml2/drivers/l2pop/mech_driver.py", line 79, in delete_port_postcommit Jul 11 20:08:35.721134 ubuntu-xenial-3-node-osic-cloud1-s3500-9770546 neutron-server[27121]: ERROR neutron.plugins.ml2.managers context, port['device_owner'], port['device_id']): Jul 11 20:08:35.721206 ubuntu-xenial-3-node-osic-cloud1-s3500-9770546 neutron-server[27121]: ERROR neutron.plugins.ml2.managers File "/opt/stack/new/neutron/neutron/db/l3_hamode_db.py", line 726, in is_ha_router_port Jul 11 20:08:35.721283 ubuntu-xenial-3-node-osic-cloud1-s3500-9770546 neutron-server[27121]: ERROR neutron.plugins.ml2.managers context, router_id=router_id, ha=True) Jul 11 20:08:35.721369 ubuntu-xenial-3-node-osic-cloud1-s3500-9770546 neutron-server[27121]: ERROR neutron.plugins.ml2.managers File "/opt/stack/new/neutron/neutron/objects/base.py", line 712, in objects_exist Jul 11 20:08:35.721447 ubuntu-xenial-3-node-osic-cloud1-s3500-9770546 neutron-server[27121]: ERROR neutron.plugins.ml2.managers context, cls.db_model, **cls.modify_fields_to_db(kwargs)) Jul 11 20:08:35.721526 ubuntu-xenial-3-node-osic-cloud1-s3500-9770546 neutron-server[27121]: ERROR neutron.plugins.ml2.managers File "/opt/stack/new/neutron/neutron/objects/db/api.py", line 32, in get_object Jul 11 20:08:35.721610 ubuntu-xenial-3-node-osic-cloud1-s3500-9770546 neutron-server[27121]: ERROR neutron.plugins.ml2.managers return _get_filter_query(context, model, **kwargs).first() Jul 11 20:08:35.721725 ubuntu-xenial-3-node-osic-cloud1-s3500-9770546 neutron-server[27121]: ERROR neutron.plugins.ml2.managers File "/opt/stack/new/neutron/neutron/objects/db/api.py", line 25, in _get_filter_query Jul 11 20:08:35.721802 ubuntu-xenial-3-node-osic-cloud1-s3500-9770546 neutron-server[27121]: ERROR neutron.plugins.ml2.managers with context.session.begin(subtransactions=True): Jul 11 20:08:35.721877 ubuntu-xenial-3-node-osic-cloud1-s3500-9770546 neutron-server[27121]: ERROR neutron.plugins.ml2.managers AttributeError: 'PortContext' object has no attribute 'session' Jul 11 20:08:35.721952 ubuntu-xenial-3-node-osic-cloud1-s3500-9770546 neutron-server[27121]: ERROR neutron.plugins.ml2.managers Example: http://logs.openstack.org/73/304873/44/check/gate-tempest-dsvm- neutron-dvr-ha-multinode-full-ubuntu-xenial- nv/586400d/logs/screen-q-svc.txt.gz?level=TRACE#_Jul_11_20_08_35_720679 ** Affects: neutron Importance: Undecided Status: New ** Tags: db l3-ha ** Tags added: db l3-ha -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1703938 Title: AttributeError: 'PortContext' object has no attribute 'session' in l3_hamode_db Status in neutron: New Bug description: Jul 11 20:08:35.720679 ubuntu-xenial-3-node-osic-cloud1-s3500-9770546 neutron-server[27121]: ERROR neutron.plugins.ml2.managers [None req-13c07cf3-201f-4b86-9e92-8f51bd141c6c admin admin] Mechanism driver 'l2population' failed in delete_port_postcommit: AttributeError: 'PortContext' object has no attribute 'session' Jul 11 20:08:35.720775 ubuntu-xenial-3-node-osic-cloud1-s3500-9770546 neutron-server[27121]: ERROR neutron.plugins.ml2.managers Traceback (most recent call last): Jul 11 20:08:35.720895 ubuntu-xenial-3-node-osic-cloud1-s3500-9770546 neutron-server[27121]: ERROR neutron.plugins.ml2.managers File "/opt/stack/new/neutron/neutron/plugins/ml2/managers.py", line 426, in _call_on_drivers Jul 11 20:08:35.720971 ubuntu-xenial-3-node-osic-cloud1-s3500-9770546 neutron-server[27121]: ERROR neutron.plugins.ml2.managers getattr(driver.obj, method_name)(context) Jul 11 20:08:35.721056 ubuntu-xenial-3-node-osic-cloud1-s3500-9770546 neutron-server[27121]: ERROR neutron.plugins.ml2.managers File "/opt/stack/new/neutron/neutron/plugins/ml2/drivers/l2pop/mech_driver.py",
