Public bug reported:
I am using kolla-ansible to install OpenStack on 3 servers:
Controller
Compute1
Compute2
Each server has three network interfaces:
1. management (10.0.0.0/24)
2. provider (xxx.xxx.xxx.160/27)
3. server internet access (xxx.xxx.xxx.160/27)
After deployment, I am unable to get internet connectivity on any
instance. The following error is seen in
/var/log/kolla/neutron/neutron-l3-agent.log after instance creation. I
am unsure what changes I can make to resolve this. Network and router
information is below.
LOG:
2022-07-26 18:29:26.061 18 INFO neutron.agent.l3.agent [-] Starting processing
update 4e471427-5bf3-4e72-9e56-cea3c8474b0c, action 3, priority 2, update_id
0f830a94-0c6f-4dff-9063-aabad002173a. Wait time elapsed: 6.087
2022-07-26 18:29:26.061 18 INFO neutron.agent.l3.agent [-] Starting router
update for 4e471427-5bf3-4e72-9e56-cea3c8474b0c, action 3, priority 2,
update_id 0f830a94-0c6f-4dff-9063-aabad002173a. Wait time elapsed: 6.087
2022-07-26 18:29:26.492 18 INFO neutron.agent.linux.interface [-] Device
qg-6a95a75b-9b already exists
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info [-] (101,
'Network is unreachable'): pr2modules.netlink.exceptions.NetlinkError: (101,
'Network is unreachable')
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info Traceback (most
recent call last):
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info File
"/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/common/utils.py", line
174, in call
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info return
func(*args, **kwargs)
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info File
"/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/l3/router_info.py",
line 1312, in process
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info
self.process_external()
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info File
"<decorator-gen-92>", line 2, in process_external
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info File
"/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/common/coordination.py",
line 78, in _synchronized
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info return f(*a,
**k)
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info File
"/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/l3/router_info.py",
line 1046, in process_external
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info
self._process_external_gateway(ex_gw_port)
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info File
"/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/l3/router_info.py",
line 927, in _process_external_gateway
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info
self.external_gateway_added(ex_gw_port, interface_name)
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info File
"/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/l3/router_info.py",
line 880, in external_gateway_added
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info ex_gw_port,
interface_name, self.ns_name, preserve_ips)
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info File
"/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/l3/router_info.py",
line 823, in _external_gateway_added
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info ns_name,
preserve_ips)
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info File
"/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/l3/router_info.py",
line 853, in _external_gateway_settings
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info
device.route.add_gateway(ip)
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info File
"/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py",
line 613, in add_gateway
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info scope=scope)
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info File
"/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py",
line 651, in add_route
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info table=table,
metric=metric, scope=scope, **kwargs)
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info File
"/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py",
line 1525, in add_ip_route
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info
metric=metric, scope=scope, proto=proto, **kwargs)
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info File
"/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/priv_context.py",
line 272, in _wrap
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info
r_call_timeout)
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info File
"/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py", line
215, in remote_call
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info raise
exc_type(*result[2])
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info
pr2modules.netlink.exceptions.NetlinkError: (101, 'Network is unreachable')
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info
2022-07-26 18:29:26.977 18 ERROR neutron.agent.l3.agent [-] Failed to process
compatible router: 4e471427-5bf3-4e72-9e56-cea3c8474b0c:
pr2modules.netlink.exceptions.NetlinkError: (101, 'Network is unreachable')
SUBNETS:
root@deployment:~# openstack subnet list --long
+--------------------------------------+------------------+--------------------------------------+------------------+----------------------------------+------+--------------+-----------------------------+-------------+------------+---------------+---------------+------+
| ID | Name | Network
| Subnet | Project | DHCP |
Name Servers | Allocation Pools | Host Routes | IP Version | Gateway
| Service Types | Tags |
+--------------------------------------+------------------+--------------------------------------+------------------+----------------------------------+------+--------------+-----------------------------+-------------+------------+---------------+---------------+------+
| 76d3d8e6-93c4-407a-af8a-b67c06316644 | private-subnet |
12cd0385-f70d-45f6-afcf-4c835d4039ed | 10.0.0.0/24 |
56648a9748c64ce8a99efab01210ce06 | True | | 10.0.0.2-10.0.0.254
| | 4 | 10.0.0.1 | | |
| fabe73a6-f17e-437c-96d4-59caa9a7e7f3 | public-subnet |
57bb06e9-a4c1-47b8-94de-a5504d910798 | xxx.xxx.xxx.160/27 |
56648a9748c64ce8a99efab01210ce06 | True | xxx.xxx.xxx.22 |
xxx.xxx.xxx.175-xxx.xxx.xxx.190 | | 4 | xxx.xxx.xxx.161 |
| |
+--------------------------------------+------------------+--------------------------------------+------------------+----------------------------------+------+--------------+-----------------------------+-------------+------------+---------------+---------------+------+
ROUTER:
root@deployment:~# openstack router list --long
+--------------------------------------+-------------------+--------+-------+----------------------------------+-------------+-------+--------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------+------+
| ID | Name | Status | State |
Project | Distributed | HA | Routes | External
gateway info
| Availability zones | Tags |
+--------------------------------------+-------------------+--------+-------+----------------------------------+-------------+-------+--------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------+------+
| 21ecb828-a0cf-4e8e-a500-9395222efc4b | private-ns-router | ACTIVE | UP |
56648a9748c64ce8a99efab01210ce06 | False | False | |
{"network_id": "57bb06e9-a4c1-47b8-94de-a5504d910798", "external_fixed_ips":
[{"subnet_id": "fabe73a6-f17e-437c-96d4-59caa9a7e7f3", "ip_address":
"xxx.xxx.xxx.190"}], "enable_snat": true} | nova | |
+--------------------------------------+-------------------+--------+-------+----------------------------------+-------------+-------+--------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------+------+
Thank you in advance for the help.
** Affects: neutron
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1982882
Title:
Failed to process compatible router - network is unreachable
Status in neutron:
New
Bug description:
I am using kolla-ansible to install OpenStack on 3 servers:
Controller
Compute1
Compute2
Each server has three network interfaces:
1. management (10.0.0.0/24)
2. provider (xxx.xxx.xxx.160/27)
3. server internet access (xxx.xxx.xxx.160/27)
After deployment, I am unable to get internet connectivity on any
instance. The following error is seen in
/var/log/kolla/neutron/neutron-l3-agent.log after instance creation. I
am unsure what changes I can make to resolve this. Network and router
information is below.
LOG:
2022-07-26 18:29:26.061 18 INFO neutron.agent.l3.agent [-] Starting
processing update 4e471427-5bf3-4e72-9e56-cea3c8474b0c, action 3, priority 2,
update_id 0f830a94-0c6f-4dff-9063-aabad002173a. Wait time elapsed: 6.087
2022-07-26 18:29:26.061 18 INFO neutron.agent.l3.agent [-] Starting router
update for 4e471427-5bf3-4e72-9e56-cea3c8474b0c, action 3, priority 2,
update_id 0f830a94-0c6f-4dff-9063-aabad002173a. Wait time elapsed: 6.087
2022-07-26 18:29:26.492 18 INFO neutron.agent.linux.interface [-] Device
qg-6a95a75b-9b already exists
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info [-] (101,
'Network is unreachable'): pr2modules.netlink.exceptions.NetlinkError: (101,
'Network is unreachable')
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info Traceback (most
recent call last):
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info File
"/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/common/utils.py", line
174, in call
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info return
func(*args, **kwargs)
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info File
"/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/l3/router_info.py",
line 1312, in process
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info
self.process_external()
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info File
"<decorator-gen-92>", line 2, in process_external
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info File
"/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/common/coordination.py",
line 78, in _synchronized
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info return
f(*a, **k)
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info File
"/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/l3/router_info.py",
line 1046, in process_external
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info
self._process_external_gateway(ex_gw_port)
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info File
"/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/l3/router_info.py",
line 927, in _process_external_gateway
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info
self.external_gateway_added(ex_gw_port, interface_name)
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info File
"/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/l3/router_info.py",
line 880, in external_gateway_added
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info ex_gw_port,
interface_name, self.ns_name, preserve_ips)
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info File
"/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/l3/router_info.py",
line 823, in _external_gateway_added
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info ns_name,
preserve_ips)
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info File
"/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/l3/router_info.py",
line 853, in _external_gateway_settings
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info
device.route.add_gateway(ip)
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info File
"/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py",
line 613, in add_gateway
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info scope=scope)
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info File
"/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py",
line 651, in add_route
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info
table=table, metric=metric, scope=scope, **kwargs)
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info File
"/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py",
line 1525, in add_ip_route
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info
metric=metric, scope=scope, proto=proto, **kwargs)
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info File
"/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/priv_context.py",
line 272, in _wrap
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info
r_call_timeout)
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info File
"/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py", line
215, in remote_call
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info raise
exc_type(*result[2])
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info
pr2modules.netlink.exceptions.NetlinkError: (101, 'Network is unreachable')
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info
2022-07-26 18:29:26.977 18 ERROR neutron.agent.l3.agent [-] Failed to process
compatible router: 4e471427-5bf3-4e72-9e56-cea3c8474b0c:
pr2modules.netlink.exceptions.NetlinkError: (101, 'Network is unreachable')
SUBNETS:
root@deployment:~# openstack subnet list --long
+--------------------------------------+------------------+--------------------------------------+------------------+----------------------------------+------+--------------+-----------------------------+-------------+------------+---------------+---------------+------+
| ID | Name | Network
| Subnet | Project | DHCP
| Name Servers | Allocation Pools | Host Routes | IP Version |
Gateway | Service Types | Tags |
+--------------------------------------+------------------+--------------------------------------+------------------+----------------------------------+------+--------------+-----------------------------+-------------+------------+---------------+---------------+------+
| 76d3d8e6-93c4-407a-af8a-b67c06316644 | private-subnet |
12cd0385-f70d-45f6-afcf-4c835d4039ed | 10.0.0.0/24 |
56648a9748c64ce8a99efab01210ce06 | True | | 10.0.0.2-10.0.0.254
| | 4 | 10.0.0.1 | | |
| fabe73a6-f17e-437c-96d4-59caa9a7e7f3 | public-subnet |
57bb06e9-a4c1-47b8-94de-a5504d910798 | xxx.xxx.xxx.160/27 |
56648a9748c64ce8a99efab01210ce06 | True | xxx.xxx.xxx.22 |
xxx.xxx.xxx.175-xxx.xxx.xxx.190 | | 4 | xxx.xxx.xxx.161 |
| |
+--------------------------------------+------------------+--------------------------------------+------------------+----------------------------------+------+--------------+-----------------------------+-------------+------------+---------------+---------------+------+
ROUTER:
root@deployment:~# openstack router list --long
+--------------------------------------+-------------------+--------+-------+----------------------------------+-------------+-------+--------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------+------+
| ID | Name | Status | State |
Project | Distributed | HA | Routes | External
gateway info
| Availability zones | Tags |
+--------------------------------------+-------------------+--------+-------+----------------------------------+-------------+-------+--------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------+------+
| 21ecb828-a0cf-4e8e-a500-9395222efc4b | private-ns-router | ACTIVE | UP |
56648a9748c64ce8a99efab01210ce06 | False | False | |
{"network_id": "57bb06e9-a4c1-47b8-94de-a5504d910798", "external_fixed_ips":
[{"subnet_id": "fabe73a6-f17e-437c-96d4-59caa9a7e7f3", "ip_address":
"xxx.xxx.xxx.190"}], "enable_snat": true} | nova | |
+--------------------------------------+-------------------+--------+-------+----------------------------------+-------------+-------+--------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------+------+
Thank you in advance for the help.
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1982882/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
