[Yahoo-eng-team] [Bug 1710589] Re: rally sla failure / internal error on load
** Changed in: neutron
Importance: High => Undecided
** Changed in: neutron
Importance: Undecided => Medium
** Changed in: neutron
Importance: Medium => Undecided
** No longer affects: neutron
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1710589
Title:
rally sla failure / internal error on load
Status in networking-midonet:
New
Bug description:
On networking-midonet gate, for following scenarios,
max_seconds_per_iteration validation is failing miserably.
These scenarios are currently disabled for an unrelated issue [1] but there
is an attempt to revive it. [2]
IIRC, these scenarios used to work. I don't remember when it was though.
(Maybe newton timeframe)
NetworkPlugin.create_ports
NetworkPlugin.create_routers
NetworkPlugin.create_subnets
NetworkPlugin.create_subnets_routers_interfaces
[1] bug 1670577
[2] https://review.openstack.org/#/c/492374/
It's probably due to retriable exceptions like the following due to
contentions.
eg. http://logs.openstack.org/74/492374/3/check/gate-networking-
midonet-rally-dsvm-ml2-ubuntu-xenial/d1aa9b9/logs/screen-q-svc.txt.gz
Aug 14 06:08:16.188519 ubuntu-xenial-citycloud-kna1-10421138 neutron-
server[14391]: DEBUG neutron.db.api [None req-
6c986099-0e8d-4181-a488-c73d348f2ef5 c_rally_1a0448a8_y9LQrXNX
c_rally_1a0448a8_QH0dDRH4] Retry wrapper got retriable exception:
UPDATE statement on table 'standardattributes' expected to update 1
row(s); 0 were matched. {{(pid=14479) wrapped
/opt/stack/new/neutron/neutron/db/api.py:129}}
Aug 14 05:46:27.279500 ubuntu-xenial-citycloud-kna1-10421138 neutron-
server[14391]: DEBUG neutron.db.api [None req-6134c462-06e4-493f-
8ff1-9b206dfda3dd c_rally_edf0ce7f_CaBvNcQe c_rally_edf0ce7f_ONZDWSHE]
Retry wrapper got retriable exception: Failed to create a duplicate
IpamAllocation: for attribute(s) ['PRIMARY'] with value(s)
2.224.0.54-29522f59-ea05-4904-b034-fa3555da8ade {{(pid=14479) wrapped
/opt/stack/new/neutron/neutron/db/api.py:129}}
Aug 14 05:39:37.960137 ubuntu-xenial-citycloud-kna1-10421138 neutron-
server[14391]: DEBUG neutron.db.api [None req-f3ecc8f3-2c7a-4b3c-ac8c-
abe2766e36f4 c_rally_6464adb0_pXKwRzJz c_rally_6464adb0_uyMbSZqP]
Retry wrapper got retriable exception: Failed to create a duplicate
DefaultSecurityGroup: for attribute(s) ['PRIMARY'] with value(s)
bc3b6a26a56646e7b098b9419d17c0d1 {{(pid=14480) wrapped
/opt/stack/new/neutron/neutron/db/api.py:129}}
Even without max_seconds_per_iteration, it sometimes times out,
or causes internal error reaching to the retry limit.
eg. http://logs.openstack.org/74/492374/4/check/gate-networking-
midonet-rally-dsvm-ml2-ubuntu-
xenial/5f6c27d/logs/screen-q-svc.txt.gz?level=TRACE
Aug 14 13:54:59.896408 ubuntu-xenial-citycloud-la1-10426313
neutron-server[13990]: ERROR oslo_db.api [None
req-c66e03cb-f36a-494b-9f83-fcb51dd0a420 c_rally_732dd379_PVser3Et
c_rally_732dd379_0ywBv9Oh] DB exceeded retry limit.: StaleDataError: UPDATE
statement on table 'standardattributes' expected to update 1 row(s); 0 were
matched.
Aug 14 13:54:59.896764 ubuntu-xenial-citycloud-la1-10426313
neutron-server[13990]: ERROR oslo_db.api Traceback (most recent call last):
Aug 14 13:54:59.896968 ubuntu-xenial-citycloud-la1-10426313
neutron-server[13990]: ERROR oslo_db.api File
"/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 138, in wrapper
Aug 14 13:54:59.897135 ubuntu-xenial-citycloud-la1-10426313
neutron-server[13990]: ERROR oslo_db.api return f(*args, **kwargs)
Aug 14 13:54:59.897300 ubuntu-xenial-citycloud-la1-10426313
neutron-server[13990]: ERROR oslo_db.api File
"/opt/stack/new/neutron/neutron/db/api.py", line 129, in wrapped
Aug 14 13:54:59.897465 ubuntu-xenial-citycloud-la1-10426313
neutron-server[13990]: ERROR oslo_db.api LOG.debug("Retry wrapper got
retriable exception: %s", e)
Aug 14 13:54:59.897653 ubuntu-xenial-citycloud-la1-10426313
neutron-server[13990]: ERROR oslo_db.api File
"/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in
__exit__
Aug 14 13:54:59.897819 ubuntu-xenial-citycloud-la1-10426313
neutron-server[13990]: ERROR oslo_db.api self.force_reraise()
Aug 14 13:54:59.897970 ubuntu-xenial-citycloud-la1-10426313
neutron-server[13990]: ERROR oslo_db.api File
"/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in
force_reraise
Aug 14 13:54:59.898135 ubuntu-xenial-citycloud-la1-10426313
neutron-server[13990]: ERROR oslo_db.api six.reraise(self.type_,
self.value, self.tb)
Aug 14 13:54:59.898300 ubuntu-xenial-citycloud-la1-10426313
neutron-server[13990]: ERROR oslo_db.api File
"/opt/stack/new/neutron/neutron/db/api.py", line 125, in wrapped
Aug 14 13:54:59.898458 ubuntu-xenial-citycloud-la1-10426313
neutron-server[13990]: ERROR oslo_db.api return
[Yahoo-eng-team] [Bug 1710589] Re: rally sla failure
Added neutron as it doesn't seem networking-midonet specific
** Description changed:
For following scenarios, max_seconds_per_iteration validation is failing
miserably.
IIRC, these scenarios used to work. I don't remember when it was though.
(Maybe newton timeframe)
NetworkPlugin.create_ports
NetworkPlugin.create_routers
NetworkPlugin.create_subnets
NetworkPlugin.create_subnets_routers_interfaces
It's probably due to retriable exceptions like the following due to
contentions.
eg. http://logs.openstack.org/74/492374/3/check/gate-networking-midonet-
rally-dsvm-ml2-ubuntu-xenial/d1aa9b9/logs/screen-q-svc.txt.gz
Aug 14 06:08:16.188519 ubuntu-xenial-citycloud-kna1-10421138 neutron-
server[14391]: DEBUG neutron.db.api [None req-
6c986099-0e8d-4181-a488-c73d348f2ef5 c_rally_1a0448a8_y9LQrXNX
c_rally_1a0448a8_QH0dDRH4] Retry wrapper got retriable exception: UPDATE
statement on table 'standardattributes' expected to update 1 row(s); 0
were matched. {{(pid=14479) wrapped
/opt/stack/new/neutron/neutron/db/api.py:129}}
Aug 14 05:46:27.279500 ubuntu-xenial-citycloud-kna1-10421138 neutron-
server[14391]: DEBUG neutron.db.api [None req-6134c462-06e4-493f-
8ff1-9b206dfda3dd c_rally_edf0ce7f_CaBvNcQe c_rally_edf0ce7f_ONZDWSHE]
Retry wrapper got retriable exception: Failed to create a duplicate
IpamAllocation: for attribute(s) ['PRIMARY'] with value(s)
2.224.0.54-29522f59-ea05-4904-b034-fa3555da8ade {{(pid=14479) wrapped
/opt/stack/new/neutron/neutron/db/api.py:129}}
Aug 14 05:39:37.960137 ubuntu-xenial-citycloud-kna1-10421138 neutron-
server[14391]: DEBUG neutron.db.api [None req-f3ecc8f3-2c7a-4b3c-ac8c-
abe2766e36f4 c_rally_6464adb0_pXKwRzJz c_rally_6464adb0_uyMbSZqP] Retry
wrapper got retriable exception: Failed to create a duplicate
DefaultSecurityGroup: for attribute(s) ['PRIMARY'] with value(s)
bc3b6a26a56646e7b098b9419d17c0d1 {{(pid=14480) wrapped
/opt/stack/new/neutron/neutron/db/api.py:129}}
+
+
+ Even without max_seconds_per_iteration, it sometimes times out,
+ or causes internal error reaching to the retry limit.
+
+ eg. http://logs.openstack.org/74/492374/4/check/gate-networking-midonet-
+ rally-dsvm-ml2-ubuntu-
+ xenial/5f6c27d/logs/screen-q-svc.txt.gz?level=TRACE
+
+ Aug 14 13:54:59.896408 ubuntu-xenial-citycloud-la1-10426313
neutron-server[13990]: ERROR oslo_db.api [None
req-c66e03cb-f36a-494b-9f83-fcb51dd0a420 c_rally_732dd379_PVser3Et
c_rally_732dd379_0ywBv9Oh] DB exceeded retry limit.: StaleDataError: UPDATE
statement on table 'standardattributes' expected to update 1 row(s); 0 were
matched.
+ Aug 14 13:54:59.896764 ubuntu-xenial-citycloud-la1-10426313
neutron-server[13990]: ERROR oslo_db.api Traceback (most recent call last):
+ Aug 14 13:54:59.896968 ubuntu-xenial-citycloud-la1-10426313
neutron-server[13990]: ERROR oslo_db.api File
"/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 138, in wrapper
+ Aug 14 13:54:59.897135 ubuntu-xenial-citycloud-la1-10426313
neutron-server[13990]: ERROR oslo_db.api return f(*args, **kwargs)
+ Aug 14 13:54:59.897300 ubuntu-xenial-citycloud-la1-10426313
neutron-server[13990]: ERROR oslo_db.api File
"/opt/stack/new/neutron/neutron/db/api.py", line 129, in wrapped
+ Aug 14 13:54:59.897465 ubuntu-xenial-citycloud-la1-10426313
neutron-server[13990]: ERROR oslo_db.api LOG.debug("Retry wrapper got
retriable exception: %s", e)
+ Aug 14 13:54:59.897653 ubuntu-xenial-citycloud-la1-10426313
neutron-server[13990]: ERROR oslo_db.api File
"/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in
__exit__
+ Aug 14 13:54:59.897819 ubuntu-xenial-citycloud-la1-10426313
neutron-server[13990]: ERROR oslo_db.api self.force_reraise()
+ Aug 14 13:54:59.897970 ubuntu-xenial-citycloud-la1-10426313
neutron-server[13990]: ERROR oslo_db.api File
"/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in
force_reraise
+ Aug 14 13:54:59.898135 ubuntu-xenial-citycloud-la1-10426313
neutron-server[13990]: ERROR oslo_db.api six.reraise(self.type_,
self.value, self.tb)
+ Aug 14 13:54:59.898300 ubuntu-xenial-citycloud-la1-10426313
neutron-server[13990]: ERROR oslo_db.api File
"/opt/stack/new/neutron/neutron/db/api.py", line 125, in wrapped
+ Aug 14 13:54:59.898458 ubuntu-xenial-citycloud-la1-10426313
neutron-server[13990]: ERROR oslo_db.api return f(*dup_args, **dup_kwargs)
+ Aug 14 13:54:59.898621 ubuntu-xenial-citycloud-la1-10426313
neutron-server[13990]: ERROR oslo_db.api File
"/opt/stack/new/neutron/neutron/db/db_base_plugin_v2.py", line 1013, in
delete_subnet
+ Aug 14 13:54:59.898792 ubuntu-xenial-citycloud-la1-10426313
neutron-server[13990]: ERROR oslo_db.api self.ipam.delete_subnet(context,
id)
+ Aug 14 13:54:59.898948 ubuntu-xenial-citycloud-la1-10426313
neutron-server[13990]: ERROR oslo_db.api File
"/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 162, in
delete_subnet
+ Aug
