Public bug reported:
When trying to change external-protocol-port to a value that is already used by
another pf row, there is no error checking and we end up 'stuck' until the api
times out. The neutron db is catching
the improper config, but the validation should likely not allow it to get that
far.
(overcloud) [stack@undercloud-0 ~]$ openstack floating ip port forwarding list
${FIP_UUID}
+--------------------------------------+--------------------------------------+---------------------+---------------+---------------+----------+
| ID | Internal Port ID |
Internal IP Address | Internal Port | External Port | Protocol |
+--------------------------------------+--------------------------------------+---------------------+---------------+---------------+----------+
| 5a8515b8-9e03-442f-a8d7-41acd11f59b5 | 63767606-35ea-4c08-b6c8-1216d0c407e8 |
192.168.30.159 | 22 | 2021 | tcp |
| 60693ea5-e985-404d-87ce-798dd4f6f4da | d5a31eba-89cb-40a8-ba98-d6ca1a8fffb1 |
192.168.30.84 | 22 | 2020 | tcp |
+--------------------------------------+--------------------------------------+---------------------+---------------+---------------+----------+
(overcloud) [stack@undercloud-0 ~]$
(overcloud) [stack@undercloud-0 ~]$
# now, doing something invalid: try to change external-protocol-port of $PF_ID2
to the port that is being used by $PF_ID (60693ea5-e985-404d-87ce-798dd4f6f4da)
(overcloud) [stack@undercloud-0 ~]$ echo ${PF_ID2}
5a8515b8-9e03-442f-a8d7-41acd11f59b5
(overcloud) [stack@undercloud-0 ~]$ time openstack floating ip port
forwarding set --external-protocol-port 2020 ${FIP_UUID} ${PF_ID2}
HttpException: 504: Server Error for url:
http://10.0.0.125:9696/v2.0/floatingips/aee2c979-31a1-453a-b508-319c71fee9dc/port_forwardings/5a8515b8-9e03
-442f-a8d7-41acd11f59b5, The server didn't respond in time.: 504 Gateway
Time-out
real 2m3.552s
user 0m1.672s
sys 0m0.727s
=-==-
[root@controller-0 neutron]# tail -F server.log
...
2020-05-12 17:54:15.900 26 DEBUG neutron.api.v2.base
[req-f0726861-9f58-4e35-b3b0-15a8bd4ada5b 251059acfcae468c89fa33c988910832
ea7d486cda284d8fa7f3eaf8351f080d - default default] Request body:
{'port_forwarding': {'external_port': 2020}} prepare_request_body
/usr/lib/python3.6/site-packages/neutron/api/v2/base.py:719
2020-05-12 17:54:16.137 28 DEBUG neutron.wsgi [-] (28) accepted ('172.17.1.38',
57154) server /usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
2020-05-12 17:54:16.370 26 DEBUG neutron_lib.db.api
[req-f0726861-9f58-4e35-b3b0-15a8bd4ada5b 251059acfcae468c89fa33c988910832
ea7d486cda284d8fa7f3eaf8351f080d - default default] Retry wrapper got retriable
exception: (pymysql.err.IntegrityError) (1062, "Duplicate entry
'aee2c979-31a1-453a-b508-319c71fee9dc-2020-tcp' for key
'uniq_port_forwardings0floatingip_id0external_port0protocol'")
[SQL: UPDATE portforwardings SET external_port=%(external_port)s WHERE
portforwardings.id = %(portforwardings_id)s]
[parameters: {'external_port': 2020, 'portforwardings_id':
'5a8515b8-9e03-442f-a8d7-41acd11f59b5'}]
(Background on this error at: http://sqlalche.me/e/gkpj) wrapped
/usr/lib/python3.6/site-packages/neutron_lib/db/api.py:183
2020-05-12 17:54:16.371 26 DEBUG oslo_db.api
[req-f0726861-9f58-4e35-b3b0-15a8bd4ada5b 251059acfcae468c89fa33c988910832
ea7d486cda284d8fa7f3eaf8351f080d - default default] Performing DB retry for
function neutron.api.v2.base.Controller._update wrapper
/usr/lib/python3.6/site-packages/oslo_db/api.py:156
2020-05-12 17:54:16.458 28 DEBUG neutron.wsgi [-] (28) accepted
('172.17.1.141', 48380) server
/usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
2020-05-12 17:54:17.882 28 DEBUG neutron.wsgi [-] (28) accepted ('172.17.1.84',
39608) server /usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
2020-05-12 17:54:17.946 26 DEBUG neutron.api.v2.base
[req-f0726861-9f58-4e35-b3b0-15a8bd4ada5b 251059acfcae468c89fa33c988910832
ea7d486cda284d8fa7f3eaf8351f080d - default default] Request body:
{'port_forwarding': {'external_port': 2020}} prepare_request_body
/usr/lib/python3.6/site-packages/neutron/api/v2/base.py:719
2020-05-12 17:54:18.145 29 DEBUG neutron.wsgi [-] (29) accepted ('172.17.1.38',
57244) server /usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
2020-05-12 17:54:18.292 26 DEBUG neutron_lib.db.api
[req-f0726861-9f58-4e35-b3b0-15a8bd4ada5b 251059acfcae468c89fa33c988910832
ea7d486cda284d8fa7f3eaf8351f080d - default default] Retry wrapper got retriable
exception: (pymysql.err.IntegrityError) (1062, "Duplicate entry
'aee2c979-31a1-453a-b508-319c71fee9dc-2020-tcp' for key
'uniq_port_forwardings0floatingip_id0external_port0protocol'")
[SQL: UPDATE portforwardings SET external_port=%(external_port)s WHERE
portforwardings.id = %(portforwardings_id)s]
[parameters: {'external_port': 2020, 'portforwardings_id':
'5a8515b8-9e03-442f-a8d7-41acd11f59b5'}]
(Background on this error at: http://sqlalche.me/e/gkpj) wrapped
/usr/lib/python3.6/site-packages/neutron_lib/db/api.py:183
2020-05-12 17:54:18.292 26 DEBUG oslo_db.api
[req-f0726861-9f58-4e35-b3b0-15a8bd4ada5b 251059acfcae468c89fa33c988910832
ea7d486cda284d8fa7f3eaf8351f080d - default default] Performing DB retry for
function neutron.api.v2.base.Controller._update wrapper
/usr/lib/python3.6/site-packages/oslo_db/api.py:156
2020-05-12 17:54:20.474 28 DEBUG neutron.wsgi [-] (28) accepted
('172.17.1.141', 48566) server
/usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
2020-05-12 17:54:22.027 27 DEBUG neutron.pecan_wsgi.hooks.policy_enforcement
[req-8005a4d6-5356-4472-aa28-6debbf9f1175 b4758ff03a5e44b8b23e758ca548d08d
8b966122dec44bed80b5c4fd53919a0d - default default] Attributes excluded by
policy engine: ['shared'] _exclude_attributes_by_policy
/usr/lib/python3.6/site-packages/neutron/pecan_wsgi/hooks/policy_enforcement.py:256
2020-05-12 17:54:22.166 28 DEBUG neutron.wsgi [-] (28) accepted ('172.17.1.38',
57438) server /usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
2020-05-12 17:54:22.481 29 DEBUG neutron.wsgi [-] (29) accepted
('172.17.1.141', 48664) server
/usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
2020-05-12 17:54:23.903 28 DEBUG neutron.wsgi [-] (28) accepted ('172.17.1.84',
39922) server /usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
2020-05-12 17:54:24.175 28 DEBUG neutron.wsgi [-] (28) accepted ('172.17.1.38',
57534) server /usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
2020-05-12 17:54:24.489 29 DEBUG neutron.wsgi [-] (29) accepted
('172.17.1.141', 48760) server
/usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
2020-05-12 17:54:25.910 29 DEBUG neutron.wsgi [-] (29) accepted ('172.17.1.84',
40016) server /usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
2020-05-12 17:54:26.183 29 DEBUG neutron.wsgi [-] (29) accepted ('172.17.1.38',
57626) server /usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
2020-05-12 17:54:26.494 29 DEBUG neutron.wsgi [-] (29) accepted
('172.17.1.141', 48854) server
/usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
2020-05-12 17:54:28.195 28 DEBUG neutron.wsgi [-] (28) accepted ('172.17.1.38',
57722) server /usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
...
** Affects: neutron
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1878299
Title:
[floatingip port_forwarding] changing external port to used value
hangs with retriable exception
Status in neutron:
New
Bug description:
When trying to change external-protocol-port to a value that is already used
by another pf row, there is no error checking and we end up 'stuck' until the
api times out. The neutron db is catching
the improper config, but the validation should likely not allow it to get
that far.
(overcloud) [stack@undercloud-0 ~]$ openstack floating ip port forwarding
list ${FIP_UUID}
+--------------------------------------+--------------------------------------+---------------------+---------------+---------------+----------+
| ID | Internal Port ID
| Internal IP Address | Internal Port | External Port | Protocol |
+--------------------------------------+--------------------------------------+---------------------+---------------+---------------+----------+
| 5a8515b8-9e03-442f-a8d7-41acd11f59b5 | 63767606-35ea-4c08-b6c8-1216d0c407e8
| 192.168.30.159 | 22 | 2021 | tcp |
| 60693ea5-e985-404d-87ce-798dd4f6f4da | d5a31eba-89cb-40a8-ba98-d6ca1a8fffb1
| 192.168.30.84 | 22 | 2020 | tcp |
+--------------------------------------+--------------------------------------+---------------------+---------------+---------------+----------+
(overcloud) [stack@undercloud-0 ~]$
(overcloud) [stack@undercloud-0 ~]$
# now, doing something invalid: try to change external-protocol-port of
$PF_ID2 to the port that is being used by $PF_ID
(60693ea5-e985-404d-87ce-798dd4f6f4da)
(overcloud) [stack@undercloud-0 ~]$ echo ${PF_ID2}
5a8515b8-9e03-442f-a8d7-41acd11f59b5
(overcloud) [stack@undercloud-0 ~]$ time openstack floating ip port
forwarding set --external-protocol-port 2020 ${FIP_UUID} ${PF_ID2}
HttpException: 504: Server Error for url:
http://10.0.0.125:9696/v2.0/floatingips/aee2c979-31a1-453a-b508-319c71fee9dc/port_forwardings/5a8515b8-9e03
-442f-a8d7-41acd11f59b5, The server didn't respond in time.: 504
Gateway Time-out
real 2m3.552s
user 0m1.672s
sys 0m0.727s
=-==-
[root@controller-0 neutron]# tail -F server.log
...
2020-05-12 17:54:15.900 26 DEBUG neutron.api.v2.base
[req-f0726861-9f58-4e35-b3b0-15a8bd4ada5b 251059acfcae468c89fa33c988910832
ea7d486cda284d8fa7f3eaf8351f080d - default default] Request body:
{'port_forwarding': {'external_port': 2020}} prepare_request_body
/usr/lib/python3.6/site-packages/neutron/api/v2/base.py:719
2020-05-12 17:54:16.137 28 DEBUG neutron.wsgi [-] (28) accepted
('172.17.1.38', 57154) server
/usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
2020-05-12 17:54:16.370 26 DEBUG neutron_lib.db.api
[req-f0726861-9f58-4e35-b3b0-15a8bd4ada5b 251059acfcae468c89fa33c988910832
ea7d486cda284d8fa7f3eaf8351f080d - default default] Retry wrapper got retriable
exception: (pymysql.err.IntegrityError) (1062, "Duplicate entry
'aee2c979-31a1-453a-b508-319c71fee9dc-2020-tcp' for key
'uniq_port_forwardings0floatingip_id0external_port0protocol'")
[SQL: UPDATE portforwardings SET external_port=%(external_port)s WHERE
portforwardings.id = %(portforwardings_id)s]
[parameters: {'external_port': 2020, 'portforwardings_id':
'5a8515b8-9e03-442f-a8d7-41acd11f59b5'}]
(Background on this error at: http://sqlalche.me/e/gkpj) wrapped
/usr/lib/python3.6/site-packages/neutron_lib/db/api.py:183
2020-05-12 17:54:16.371 26 DEBUG oslo_db.api
[req-f0726861-9f58-4e35-b3b0-15a8bd4ada5b 251059acfcae468c89fa33c988910832
ea7d486cda284d8fa7f3eaf8351f080d - default default] Performing DB retry for
function neutron.api.v2.base.Controller._update wrapper
/usr/lib/python3.6/site-packages/oslo_db/api.py:156
2020-05-12 17:54:16.458 28 DEBUG neutron.wsgi [-] (28) accepted
('172.17.1.141', 48380) server
/usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
2020-05-12 17:54:17.882 28 DEBUG neutron.wsgi [-] (28) accepted
('172.17.1.84', 39608) server
/usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
2020-05-12 17:54:17.946 26 DEBUG neutron.api.v2.base
[req-f0726861-9f58-4e35-b3b0-15a8bd4ada5b 251059acfcae468c89fa33c988910832
ea7d486cda284d8fa7f3eaf8351f080d - default default] Request body:
{'port_forwarding': {'external_port': 2020}} prepare_request_body
/usr/lib/python3.6/site-packages/neutron/api/v2/base.py:719
2020-05-12 17:54:18.145 29 DEBUG neutron.wsgi [-] (29) accepted
('172.17.1.38', 57244) server
/usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
2020-05-12 17:54:18.292 26 DEBUG neutron_lib.db.api
[req-f0726861-9f58-4e35-b3b0-15a8bd4ada5b 251059acfcae468c89fa33c988910832
ea7d486cda284d8fa7f3eaf8351f080d - default default] Retry wrapper got retriable
exception: (pymysql.err.IntegrityError) (1062, "Duplicate entry
'aee2c979-31a1-453a-b508-319c71fee9dc-2020-tcp' for key
'uniq_port_forwardings0floatingip_id0external_port0protocol'")
[SQL: UPDATE portforwardings SET external_port=%(external_port)s WHERE
portforwardings.id = %(portforwardings_id)s]
[parameters: {'external_port': 2020, 'portforwardings_id':
'5a8515b8-9e03-442f-a8d7-41acd11f59b5'}]
(Background on this error at: http://sqlalche.me/e/gkpj) wrapped
/usr/lib/python3.6/site-packages/neutron_lib/db/api.py:183
2020-05-12 17:54:18.292 26 DEBUG oslo_db.api
[req-f0726861-9f58-4e35-b3b0-15a8bd4ada5b 251059acfcae468c89fa33c988910832
ea7d486cda284d8fa7f3eaf8351f080d - default default] Performing DB retry for
function neutron.api.v2.base.Controller._update wrapper
/usr/lib/python3.6/site-packages/oslo_db/api.py:156
2020-05-12 17:54:20.474 28 DEBUG neutron.wsgi [-] (28) accepted
('172.17.1.141', 48566) server
/usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
2020-05-12 17:54:22.027 27 DEBUG neutron.pecan_wsgi.hooks.policy_enforcement
[req-8005a4d6-5356-4472-aa28-6debbf9f1175 b4758ff03a5e44b8b23e758ca548d08d
8b966122dec44bed80b5c4fd53919a0d - default default] Attributes excluded by
policy engine: ['shared'] _exclude_attributes_by_policy
/usr/lib/python3.6/site-packages/neutron/pecan_wsgi/hooks/policy_enforcement.py:256
2020-05-12 17:54:22.166 28 DEBUG neutron.wsgi [-] (28) accepted
('172.17.1.38', 57438) server
/usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
2020-05-12 17:54:22.481 29 DEBUG neutron.wsgi [-] (29) accepted
('172.17.1.141', 48664) server
/usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
2020-05-12 17:54:23.903 28 DEBUG neutron.wsgi [-] (28) accepted
('172.17.1.84', 39922) server
/usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
2020-05-12 17:54:24.175 28 DEBUG neutron.wsgi [-] (28) accepted
('172.17.1.38', 57534) server
/usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
2020-05-12 17:54:24.489 29 DEBUG neutron.wsgi [-] (29) accepted
('172.17.1.141', 48760) server
/usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
2020-05-12 17:54:25.910 29 DEBUG neutron.wsgi [-] (29) accepted
('172.17.1.84', 40016) server
/usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
2020-05-12 17:54:26.183 29 DEBUG neutron.wsgi [-] (29) accepted
('172.17.1.38', 57626) server
/usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
2020-05-12 17:54:26.494 29 DEBUG neutron.wsgi [-] (29) accepted
('172.17.1.141', 48854) server
/usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
2020-05-12 17:54:28.195 28 DEBUG neutron.wsgi [-] (28) accepted
('172.17.1.38', 57722) server
/usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
...
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1878299/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
