[Bug 1759918] Re: [dvr] router remove subnet silently fails

2018-03-29 Thread Dmitrii Shcherbakov
The policy rule issue now lives here
https://bugs.launchpad.net/neutron/+bug/1759956

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1759918

Title:
  [dvr] router remove subnet   silently fails

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/neutron/+bug/1759918/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1759918] Re: [dvr] router remove subnet silently fails

2018-03-29 Thread Dmitrii Shcherbakov
"cannot be deleted directly via the port API" is by design it seems:

https://bugs.launchpad.net/neutron/+bug/1425504

However, deleting via a subnet should be possible.

Client debug log revealed the issue (I also had a static route
configured to 192.168.200.0/24 on pubrouter via neutron extra routes
extension)

openstack router remove subnet 4f9ca9ef-303b-4082-abbc-e50782d9b800
6694cc70-7667-4583-8eec-1decb19063c9 --debug

http://10.232.1.207:9696 "PUT 
/v2.0/routers/4f9ca9ef-303b-4082-abbc-e50782d9b800/remove_router_interface 
HTTP/1.1" 409 257
RESP: [409] Content-Type: application/json Content-Length: 257 
X-Openstack-Request-Id: req-afac4d67-d309-4bcd-83d6-bfd08d6a6d9b Date: Thu, 29 
Mar 2018 19:30:24 GMT Connection: keep-alive 
RESP BODY: {"NeutronError": {"message": "Router interface for subnet 
6694cc70-7667-4583-8eec-1decb19063c9 on router 
4f9ca9ef-303b-4082-abbc-e50782d9b800 cannot be deleted, as it is required by 
one or more routes.", "type": "RouterInterfaceInUseByRoute", "detail": ""}}

openstack router show pubrouter -c routes -f value
destination='8.8.8.8/32', gateway='192.168.200.10'

openstack router set --no-route pubrouter

After that a route got deleted

openstack router remove subnet 4f9ca9ef-303b-4082-abbc-e50782d9b800
6694cc70-7667-4583-8eec-1decb19063c9 --debug

PUT call to network for 
http://10.232.1.207:9696/v2.0/routers/4f9ca9ef-303b-4082-abbc-e50782d9b800/remove_router_interface
 used request id req-cf7a0614-5fd5-4842-aef4-422be3ab96b1
Manager RegionOne ran task network.PUT.routers.remove_router_interface in 
4.53609895706s

However, policy rules for 192.168.200.0/24 were kept but were deleted
for 192.168.100.0/24 which should not have happened.

juju run --application 
neutron-gateway-l2,neutron-gateway-l1,neutron-openvswitch-l1,neutron-openvswitch-l2
 'sudo ip netns exec qrouter-4f9ca9ef-303b-4082-abbc-e50782d9b800 ip rule'
- Stdout: "0:\tfrom all lookup local \n32766:\tfrom all lookup main 
\n32767:\tfrom
all lookup default \n8:\tfrom 192.168.200.0/24 lookup 16 \n"
  UnitId: neutron-gateway-l1/0
- Stdout: "0:\tfrom all lookup local \n32766:\tfrom all lookup main 
\n32767:\tfrom
all lookup default \n8:\tfrom 192.168.200.0/24 lookup 16 \n"
  UnitId: neutron-openvswitch-l1/0
- Stdout: "0:\tfrom all lookup local \n32766:\tfrom all lookup main 
\n32767:\tfrom
all lookup default \n8:\tfrom 192.168.200.0/24 lookup 16 \n"
  UnitId: neutron-openvswitch-l1/1
- Stdout: "0:\tfrom all lookup local \n32766:\tfrom all lookup main 
\n32767:\tfrom
all lookup default \n8:\tfrom 192.168.200.0/24 lookup 16 \n"
  UnitId: neutron-openvswitch-l1/2
- Stdout: "0:\tfrom all lookup local \n32766:\tfrom all lookup main 
\n32767:\tfrom
all lookup default \n8:\tfrom 192.168.200.0/24 lookup 16 \n"
  UnitId: neutron-openvswitch-l2/0
- Stdout: "0:\tfrom all lookup local \n32766:\tfrom all lookup main 
\n32767:\tfrom
all lookup default \n8:\tfrom 192.168.200.0/24 lookup 16 \n"
  UnitId: neutron-openvswitch-l2/1
- Stdout: "0:\tfrom all lookup local \n32766:\tfrom all lookup main 
\n32767:\tfrom
all lookup default \n8:\tfrom 192.168.200.0/24 lookup 16 \n"
  UnitId: neutron-gateway-l2/0


Disabling and re-enabling the router brought the right policy rule back:

ip netns exec qrouter-4f9ca9ef-303b-4082-abbc-e50782d9b800 ip rule
0:  from all lookup local 
32766:  from all lookup main 
32767:  from all lookup default 
8:  from 192.168.100.0/24 lookup 16 

And adding 192.168.200.0/24 again only created qr- ports but not policy
rules for 192.168.200.0/24.

The logs for neutron-l3-agent were full of this:
https://paste.ubuntu.com/p/symM9Js9nJ/

So after disabling the router, restarting l3 agent, and enabling the
router everything went back to normal. And removing a subnet from a
router resulted in both interfaces and policy rules being deleted.

There are clearly some idempotency and ordering problems.

I was able to reproduce incorrect policy rule deletion more than twice.
So I will mark this bug as invalid as the original issue with subnet
removal was due to a static route presence but create a new one for
policy rules.

** Changed in: neutron (Ubuntu)
   Status: New => Invalid

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1759918

Title:
  [dvr] router remove subnet   silently fails

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/neutron/+bug/1759918/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1759918] Re: [dvr] router remove subnet silently fails

2018-03-29 Thread Dmitrii Shcherbakov
SELECT 1 failure seems to be OK:
http://docs.sqlalchemy.org/en/latest/core/pooling.html#disconnect-handling-pessimistic

So this fails:

remove_router_interface failed (client error): There was a conflict when
trying to complete your request.

remove_router_interface
https://github.com/openstack/neutron/blob/stable/queens/neutron/db/l3_db.py#L1002-L1005

openstack port list --router pubrouter | grep 168.200
| ad410866-0c90-46db-8cae-eb9f28e336fa |  | fa:16:3e:3a:0b:f8 | 
ip_address='192.168.200.1', subnet_id='6694cc70-7667-4583-8eec-1decb19063c9'  | 
ACTIVE |
| ea140f1d-d006-4f77-b2c1-85d6fba94a4a |  | fa:16:3e:a6:f3:a0 | 
ip_address='192.168.200.12', subnet_id='6694cc70-7667-4583-8eec-1decb19063c9' | 
ACTIVE |

One port is

binding_vif_type  | ovs
device_owner | network:router_centralized_snat 

The other is:

binding_vif_type  | distributed
device_owner  | network:router_interface_distributed

https://paste.ubuntu.com/p/zTP3SzCBxM/

ip netns exec qrouter-4f9ca9ef-303b-4082-abbc-e50782d9b800 ip a s | grep 200
inet 192.168.200.1/24 brd 192.168.200.255 scope global qr-ad410866-0c

>From an instance I can ping 192.168.200.1 but not 192.168.200.12:

https://paste.ubuntu.com/p/Hw8rr4bCFb/

Deleting ports manually is not possible:

https://paste.ubuntu.com/p/Znnb6t4HPP/
cannot be deleted directly via the port API: has device owner 
network:router_centralized_snat
cannot be deleted directly via the port API: has device owner 
network:router_interface_distributed

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1759918

Title:
  [dvr] router remove subnet   silently fails

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/neutron/+bug/1759918/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs