akrasnov-drv commented on issue #11607:
URL: https://github.com/apache/cloudstack/issues/11607#issuecomment-3279969313
First of all I want to confirm, I updated manually mysql raw for the IP (set
it to allocated), and it fixed the issue.
Here is the log for job for router, if you want more than this, please let
me know what filter to use. Full log is quite large.
```
/var/log/cloudstack/management/management-server.log:2025-09-11 00:26:10,712
DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-119:[ctx-f4fedbbb,
job-91161/job-91162, ctx-0423e0ee]) (logid:218f089f) Asking VirtualRouter to
prepare for Nic
{"broadcastUri":"vlan:\/\/untagged","deviceId":0,"iPv4Address":"10.10.126.13","id":30012,"instanceId":29960,"reservationId":"1bb73151-7a01-43aa-9530-19276c9054fa","uuid":"43dae93a-e114-4be1-88f3-7fd0d9c9902d"}
/var/log/cloudstack/management/management-server.log:2025-09-11 00:26:10,730
DEBUG [c.c.a.m.ClusteredAgentAttache] (Work-Job-Executor-119:[ctx-f4fedbbb,
job-91161/job-91162, ctx-0423e0ee]) (logid:218f089f) Seq
48-1086211935126425991: Routed from 179113803344805
/var/log/cloudstack/management/management-server.log:2025-09-11 00:26:10,730
DEBUG [c.c.a.t.Request] (Work-Job-Executor-119:[ctx-f4fedbbb,
job-91161/job-91162, ctx-0423e0ee]) (logid:218f089f) Seq
48-1086211935126425991: Sending { Cmd , MgmtId: 179113803344805, via:
48(kvm110), Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.routing.DhcpEntryCommand":{"vmMac":"1e:00:8b:00:00:7d","vmIpAddress":"10.10.126.13","vmName":"dn-cl-tiny-720","defaultRouter":"10.10.127.254","defaultDns":"10.10.126.13","duid":"00:03:00:01:1e:00:8b:00:00:7d","isDefault":"true","executeInSequence":"false","remove":"false","accessDetails":{"router.name":"r-29959-VM","router.guest.ip":"10.10.126.13","router.ip":"169.254.71.88","zone.network.type":"Basic"},"wait":"0","bypassHostMaintenance":"false"}}]
}
/var/log/cloudstack/management/management-server.log:2025-09-11 00:26:11,591
DEBUG [c.c.a.m.ClusteredAgentAttache] (Work-Job-Executor-119:[ctx-f4fedbbb,
job-91161/job-91162, ctx-0423e0ee]) (logid:218f089f) Seq
48-1086211935126425992: Routed from 179113803344805
/var/log/cloudstack/management/management-server.log:2025-09-11 00:26:11,592
DEBUG [c.c.a.t.Request] (Work-Job-Executor-119:[ctx-f4fedbbb,
job-91161/job-91162, ctx-0423e0ee]) (logid:218f089f) Seq
48-1086211935126425992: Sending { Cmd , MgmtId: 179113803344805, via:
48(kvm110), Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.routing.SavePasswordCommand":{,"vmIpAddress":"10.10.126.13","vmName":"dn-cl-tiny-720","executeInSequence":"false","accessDetails":{"router.name":"r-29959-VM","router.guest.ip":"10.10.126.13","router.ip":"169.254.71.88","zone.network.type":"Basic"},"wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.routing.VmDataCommand":{"vmIpAddress":"10.10.126.13","vmName":"dn-cl-tiny-720","executeInSequence":"false","accessDetails":{"router.name":"r-29959-VM","router.guest.ip":"10.10.126.13","router.ip":"169.254.71.88","zone.network.type":"Basic"},"wait":"0","bypassHostMaintenance":"false"}}]
}
/var/log/cloudstack/management/management-server.log:2025-09-11 00:26:13,117
DEBUG [c.c.a.m.ClusteredAgentAttache] (Work-Job-Executor-119:[ctx-f4fedbbb,
job-91161/job-91162, ctx-0423e0ee]) (logid:218f089f) Seq
48-3149704989392250255: Routed from 179113803344805
/var/log/cloudstack/management/management-server.log:2025-09-11 00:26:31,173
DEBUG [c.c.a.m.ClusteredAgentAttache] (Work-Job-Executor-119:[ctx-f4fedbbb,
job-91161/job-91162, ctx-0423e0ee]) (logid:218f089f) Seq
48-3149704989392250257: Routed from 179113803344805
/var/log/cloudstack/management/management-server.log:2025-09-11 00:28:25,208
DEBUG [c.c.a.m.ClusteredAgentAttache] (Work-Job-Executor-119:[ctx-f4fedbbb,
job-91161/job-91162, ctx-0423e0ee]) (logid:218f089f) Seq
48-3149704989392250260: Routed from 179113803344805
```
As there are a lot of IPs I limited the queue, also, as I wrote above, I
update the raw, originally it was
```
mysql> SELECT * FROM user_ip_address where id=115;
+-----+--------------------------------------+------------+-----------+-------------------+----------------+------------+-----------+------------+----------------+-------+-------+-------------+-------------------+------------+---------------------+-----------+--------+-----------+-------------+---------+---------+---------+------------+--------------+------------+
| id | uuid | account_id | domain_id |
public_ip_address | data_center_id | source_nat | allocated | vlan_db_id |
one_to_one_nat | vm_id | state | mac_address | source_network_id | network_id |
physical_network_id | is_system | vpc_id | dnat_vmip | is_portable | display |
removed | created | rule_state | forsystemvms | for_router |
+-----+--------------------------------------+------------+-----------+-------------------+----------------+------------+-----------+------------+----------------+-------+-------+-------------+-------------------+------------+---------------------+-----------+--------+-----------+-------------+---------+---------+---------+------------+--------------+------------+
| 115 | 47c5f17e-9d44-49ef-8355-730d877a0c77 | NULL | NULL |
10.10.126.13 | 1 | 0 | NULL | 2 |
0 | NULL | Free | 125 | 204 | NULL |
200 | 0 | NULL | NULL | 0 | 1 | NULL
| NULL | NULL | 0 | 0 |
+-----+--------------------------------------+------------+-----------+-------------------+----------------+------------+-----------+------------+----------------+-------+-------+-------------+-------------------+------------+-------------
```
now it's
```
mysql> select count(*) from user_ip_address where public_ip_address like
"10.10.126.%";
+----------+
| count(*) |
+----------+
| 246 |
+----------+
1 row in set (0.00 sec)
mysql> SELECT * FROM user_ip_address WHERE id BETWEEN 111 AND 115;
+-----+--------------------------------------+------------+-----------+-------------------+----------------+------------+---------------------+------------+----------------+-------+-----------+-------------+-------------------+------------+---------------------+-----------+--------+-----------+-------------+---------+---------------------+---------+------------+--------------+------------+
| id | uuid | account_id | domain_id |
public_ip_address | data_center_id | source_nat | allocated |
vlan_db_id | one_to_one_nat | vm_id | state | mac_address |
source_network_id | network_id | physical_network_id | is_system | vpc_id |
dnat_vmip | is_portable | display | removed | created | rule_state
| forsystemvms | for_router |
+-----+--------------------------------------+------------+-----------+-------------------+----------------+------------+---------------------+------------+----------------+-------+-----------+-------------+-------------------+------------+---------------------+-----------+--------+-----------+-------------+---------+---------------------+---------+------------+--------------+------------+
| 111 | 5effad56-c1e5-45e2-a057-f027ee0670c3 | NULL | NULL |
10.10.67.125 | 1 | 0 | NULL |
1 | 0 | NULL | Free | 121 | 204 |
NULL | 200 | 0 | NULL | NULL | 0 |
1 | 2025-08-19 12:51:25 | NULL | NULL | 0 | 0 |
| 112 | 0e6dce6a-0a5b-4bf5-84c0-8ff2f579e8e1 | 1 | 1 |
10.10.126.10 | 1 | 0 | 2025-09-03 15:12:25 |
2 | 0 | NULL | Allocated | 122 | 204 |
NULL | 200 | 0 | NULL | NULL | 0 |
1 | NULL | NULL | NULL | 0 | 0 |
| 113 | 573e095f-ef95-4f52-801e-8a1aa1d8b7de | 1 | 1 |
10.10.126.11 | 1 | 0 | 2025-09-03 15:12:25 |
2 | 0 | NULL | Allocated | 123 | 204 |
NULL | 200 | 0 | NULL | NULL | 0 |
1 | NULL | NULL | NULL | 0 | 0 |
| 114 | 0f17b747-08f5-4b25-a1cf-227bb0549cc5 | 2 | 1 |
10.10.126.12 | 1 | 0 | 2025-09-10 15:51:08 |
2 | 0 | NULL | Allocated | 124 | 204 |
NULL | 200 | 0 | NULL | NULL | 0 |
1 | NULL | NULL | NULL | 0 | 0 |
| 115 | 47c5f17e-9d44-49ef-8355-730d877a0c77 | 2 | 1 |
10.10.126.13 | 1 | 0 | 2025-09-10 15:51:08 |
2 | 0 | NULL | Allocated | 125 | 204 |
NULL | 200 | 0 | NULL | NULL | 0 |
1 | NULL | NULL | NULL | 0 | 0 |
+-----+--------------------------------------+------------+-----------+-------------------+----------------+------------+---------------------+------------+----------------+-------+-----------+-------------+-------------------+------------+---------------------+-----------+--------+-----------+-------------+---------+---------------------+---------+------------+--------------+------------+
5 rows in set (0.00 sec)
mysql> select * from nics where ip4_address ="10.10.126.13";
+-------+--------------------------------------+-------------+-------------------+--------------+---------------+---------------+---------+-----------------+------------+------+--------------+-------------+---------------------------+--------------------------------------+-----------+-------------+---------------+-------------+-------------+--------------+---------------------+---------------------+-------------+----------+--------------+-------------+------+
| id | uuid | instance_id | mac_address
| ip4_address | netmask | gateway | ip_type | broadcast_uri
| network_id | mode | state | strategy | reserver_name |
reservation_id | device_id | update_time | isolation_uri
| ip6_address | default_nic | vm_type | created | removed
| ip6_gateway | ip6_cidr | secondary_ip | display_nic | mtu |
+-------+--------------------------------------+-------------+-------------------+--------------+---------------+---------------+---------+-----------------+------------+------+--------------+-------------+---------------------------+--------------------------------------+-----------+-------------+---------------+-------------+-------------+--------------+---------------------+---------------------+-------------+----------+--------------+-------------+------+
| 23489 | 75e8bb64-e1cc-4f60-a82f-572eb2b3070d | NULL | NULL
| 10.10.126.13 | NULL | NULL | NULL | NULL
| 204 | NULL | Reserved | PlaceHolder | NULL |
NULL | 0 | NULL | NULL
| NULL | 0 | DomainRouter | 2025-08-20 14:17:29 | NULL
| NULL | NULL | 0 | 1 | NULL |
| 29825 | be2444f2-cab2-4319-9cab-ae06539f08cd | 29776 |
1e:00:c4:00:00:7d | 10.10.126.13 | 255.255.240.0 | 10.10.127.254 | Ip4 |
vlan://untagged | 204 | Dhcp | Deallocating | Start |
DirectPodBasedNetworkGuru | 7b6c80cc-bd05-4553-9a4c-e2ebd2561205 | 0 |
NULL | NULL | NULL | 1 | DomainRouter |
2025-09-08 12:37:30 | 2025-09-10 14:57:42 | NULL | NULL |
0 | 1 | 1500 |
| 29996 | 86a20417-e5c6-48b9-aa6d-bac47e66151e | 29946 |
1e:00:c2:00:00:7d | 10.10.126.13 | 255.255.240.0 | 10.10.127.254 | Ip4 |
vlan://untagged | 204 | Dhcp | Deallocating | Start |
DirectPodBasedNetworkGuru | d2679db3-2793-4879-8a58-f8bd6a3fa095 | 0 |
NULL | NULL | NULL | 1 | User |
2025-09-10 10:58:35 | 2025-09-10 11:16:19 | NULL | NULL |
0 | 1 | NULL |
| 29999 | 33498cbf-bd4c-4079-b3cf-e623bf21eb7c | 29949 |
1e:00:dc:00:00:7d | 10.10.126.13 | 255.255.240.0 | 10.10.127.254 | Ip4 |
vlan://untagged | 204 | Dhcp | Deallocating | Start |
DirectPodBasedNetworkGuru | 3ad6f1de-0c4b-430a-8a10-32a627da253d | 0 |
NULL | NULL | NULL | 1 | User |
2025-09-10 12:44:45 | 2025-09-10 13:22:05 | NULL | NULL |
0 | 1 | NULL |
| 30001 | 3884f72d-32f7-4003-b01d-2a82ad46dac1 | 29951 |
1e:00:51:00:00:7d | 10.10.126.13 | 255.255.240.0 | 10.10.127.254 | Ip4 |
vlan://untagged | 204 | Dhcp | Deallocating | Start |
DirectPodBasedNetworkGuru | 8771e24f-39ad-48be-bebe-f12e298eb0b0 | 0 |
NULL | NULL | NULL | 1 | User |
2025-09-10 13:46:27 | 2025-09-10 14:20:41 | NULL | NULL |
0 | 1 | NULL |
| 30003 | 9d4fdd27-02cb-4d5a-8971-f1fc4f75e000 | 29953 |
1e:00:0e:00:00:7d | 10.10.126.13 | 255.255.240.0 | 10.10.127.254 | Ip4 |
vlan://untagged | 204 | Dhcp | Deallocating | Start |
DirectPodBasedNetworkGuru | 93ee6e30-8da1-43a6-a88d-e8a29f9e8c64 | 0 |
NULL | NULL | NULL | 1 | User |
2025-09-10 14:28:52 | 2025-09-10 14:51:17 | NULL | NULL |
0 | 1 | NULL |
| 30006 | 6d4bc286-06ff-4241-a850-9ec7b27f95e3 | 29956 |
1e:00:bf:00:00:7d | 10.10.126.13 | 255.255.240.0 | 10.10.127.254 | Ip4 |
vlan://untagged | 204 | Dhcp | Deallocating | Start |
DirectPodBasedNetworkGuru | 10db5b63-0ca6-48c2-9b02-70aa00202ef5 | 0 |
NULL | NULL | NULL | 1 | DomainRouter |
2025-09-10 15:00:34 | 2025-09-10 15:39:00 | NULL | NULL |
0 | 1 | 1500 |
| 30008 | 8b3fdc44-0ef9-46e3-8829-4ba898e21320 | 29957 |
1e:00:91:00:00:7d | 10.10.126.13 | 255.255.240.0 | 10.10.127.254 | Ip4 |
vlan://untagged | 204 | Dhcp | Deallocating | Start |
DirectPodBasedNetworkGuru | edbc8002-f19e-4f19-a477-7f86223f1e12 | 0 |
NULL | NULL | NULL | 1 | User |
2025-09-10 15:01:30 | 2025-09-10 15:39:00 | NULL | NULL |
0 | 1 | NULL |
| 30010 | a2a033c3-663f-474b-8611-f99467dcf973 | 29959 |
1e:00:50:00:00:7d | 10.10.126.13 | 255.255.240.0 | 10.10.127.254 | Ip4 |
vlan://untagged | 204 | Dhcp | Reserved | Start |
DirectPodBasedNetworkGuru | 3d40d5da-438d-4c0d-8cc6-0688e8cc6d35 | 0 |
NULL | NULL | NULL | 1 | DomainRouter |
2025-09-10 15:51:08 | NULL | NULL | NULL |
0 | 1 | 1500 |
| 30012 | 43dae93a-e114-4be1-88f3-7fd0d9c9902d | 29960 |
1e:00:8b:00:00:7d | 10.10.126.13 | 255.255.240.0 | 10.10.127.254 | Ip4 |
vlan://untagged | 204 | Dhcp | Deallocating | Start |
DirectPodBasedNetworkGuru | 1bb73151-7a01-43aa-9530-19276c9054fa | 0 |
NULL | NULL | NULL | 1 | User |
2025-09-11 00:26:10 | 2025-09-11 00:54:07 | NULL | NULL |
0 | 1 | NULL |
| 30014 | 0093ab0e-9a97-4aaa-86dc-1ee81ceb3567 | 29962 |
1e:00:a6:00:00:7d | 10.10.126.13 | 255.255.240.0 | 10.10.127.254 | Ip4 |
vlan://untagged | 204 | Dhcp | Deallocating | Start |
DirectPodBasedNetworkGuru | d6637bff-c725-4f2b-8b57-c66fc2dd0673 | 0 |
NULL | NULL | NULL | 1 | User |
2025-09-11 04:44:10 | 2025-09-11 05:12:39 | NULL | NULL |
0 | 1 | NULL |
+-------+--------------------------------------+-------------+-------------------+--------------+---------------+---------------+---------+-----------------+------------+------+--------------+-------------+---------------------------+--------------------------------------+-----------+-------------+---------------+-------------+-------------+--------------+---------------------+---------------------+-------------+----------+--------------+-------------+------+
11 rows in set (0.00 sec)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
