MitchDrage opened a new issue, #13720:
URL: https://github.com/apache/cloudstack/issues/13720
### problem
SUMMARY
On a redundant VR, the password server is started with both the VRRP VIP and
the router's own guest IP, but passwd_server_ip.py binds only the first of the
two. Guests derive the password server address from the DHCP server-identifier,
which is the router's own guest IP, so their requests to port 8080 are answered
with a TCP RST. Passwords accumulate uncollected in
/var/cache/cloud/passwords-<vip> and password injection silently fails for
every guest on every isolated network with redundant routers.
This is not to be confused with #12161, which changes the DNS list for
redundant VPC VRs only and does not affect this path.
CODE
I believe this was introduced in #10183 when a second entry became added to
listen address list for the real IP of the VR unconditionally. The password
server script chooses addresses[0], so never selects the real IP in a redundant
VR deployment, but that is what the DHCP server tells the client to use.
First released in 4.19.2.0, still present on main.
Code:
passwd_server_ip.py serve() Lines 179-198 (4.22.0.0) binds addresses[0];
CsDhcp.py configure_server() Lines 145-147 builds the listen-address list.
In the environment below, the guest network is 192.168.0.0/24, the VRRP VIP
(gateway) is 192.168.0.1, and the primary router's own guest IP is 192.168.0.48.
EXPECTED RESULTS
The guest retrieves its password from the address it was given as the DHCP
server-identifier, and the entry is removed from the password cache.
ACTUAL RESULTS
The password server listens on the VIP only:
root@r-3050-VM:~# ss -lntp | grep 8080
LISTEN 0 5 192.168.0.1:8080 0.0.0.0:* users:(("python",pid=3372,fd=3))
Passwords are delivered by the management server and never collected:
root@r-3050-VM:~# cat /var/cache/cloud/passwords-192.168.0.1
192.168.0.82=<redacted>
192.168.0.188=<redacted>
192.168.0.38=<redacted>
Metadata on port 80 works for every guest, because cloudstack binds all
addresses. Both agents complete a full metadata crawl and only the password
request fails. Note the 61 second stall in the cloudbase-init crawl, which is
the password request timing out:
192.168.0.82 - - [.../Jul/2026:03:30:36 +0000] "GET
/latest/meta-data/instance-id HTTP/1.1" 200 291 "-" "python-requests/2.33.1"
192.168.0.82 - - [.../Jul/2026:03:31:37 +0000] "GET
/latest/meta-data/service-offering HTTP/1.1" 200 302 "-"
"python-requests/2.33.1"
192.168.0.38 - - [.../Jul/2026:04:55:56 +0000] "GET
/latest/meta-data/instance-id HTTP/1.1" 200 291 "-"
"Cloud-Init/26.1-0ubuntu1~22.04.1"
192.168.0.38 - - [.../Jul/2026:04:55:56 +0000] "GET /latest/user-data
HTTP/1.1" 200 351 "-" "Cloud-Init/26.1-0ubuntu1~22.04.1"
The guest requests port 8080 at the router's own guest IP (192.168.0.48),
not at the VIP, and is reset because nothing is bound there:
05:03:16.482340 IP 192.168.0.38.43828 > 192.168.0.48.8080: Flags [S], seq
2121724, length 0
05:03:16.482361 IP 192.168.0.48.8080 > 192.168.0.38.43828: Flags [R.], seq
0, ack 2121725, win 0, length 0
05:03:17.484909 IP 192.168.0.38.43838 > 192.168.0.48.8080: Flags [S], seq
2008568524, length 0
05:03:17.484944 IP 192.168.0.48.8080 > 192.168.0.38.43838: Flags [R.], seq
0, ack 2008568525, win 0, length 0
This is not a firewall problem. The rules are present and counting the SYNs
above as accepted:
root@r-3050-VM:~# iptables-save | grep -E 'dport (80|8080)'
-A INPUT -i eth0 -p tcp -m tcp --dport 80 -m state --state NEW -j ACCEPT
-A INPUT -s 192.168.0.0/24 -i eth0 -p tcp -m tcp --dport 80 -m state --state
NEW -j ACCEPT
-A INPUT -s 192.168.0.0/24 -i eth0 -p tcp -m tcp --dport 8080 -m state
--state NEW -j ACCEPT
17 204 10640 ACCEPT 6 -- eth0 * 192.168.0.0/24 0.0.0.0/0 tcp
dpt:8080 state NEW
Confirmed with a DNAT rule on the primary router, after which every pending
password was collected immediately:
iptables -t nat -I PREROUTING -i eth0 -d 192.168.0.48/32 -p tcp --dport 8080
\
-j DNAT --to-destination 192.168.0.1:8080
### versions
ACS: 4.22.1.0
systemvm template: systemvm-kvm-4.22.0-x86_64
Hypervisor: KVM.
Guests tested: Windows Server 2022 (cloudbase-init) and Ubuntu 22.04
(cloud-init 26.1). Same guest templates work correctly on a shared network
served by a single, non-redundant VR.
CONFIGURATION
Advanced zone, KVM, isolated guest networks using a network offering with
redundant routers enabled. UserData/password service provided by VirtualRouter.
### The steps to reproduce the bug
1. Create an isolated network using an offering with redundant VRs enabled
and UserData provided by VirtualRouter.
2. Deploy a guest from any template with a password-enabled image
(cloudbase-init or cloud-init), or reset the password of an existing guest.
3. Observe the guest never receives its password.
### What to do about it?
passwd_server_ip.py: start a listener for every address supplied rather than
only addresses[0]. allowAddresses already contains both, and savepassword.sh
already POSTs to every local IP, so no other change is needed. Keeping
addresses[0] as listeningAddress preserves the existing
/var/cache/cloud/passwords-<vip> path. This is what I am running locally using
custom userdata, verified across save, fetch and acknowledge.
Something like this??
FROM:
(Starting at line 182 of passwd_server_ip.py)
```
global listeningAddress
global allowAddresses
if len(sys.argv) > 1:
addresses = sys.argv[1].split(",")
if len(addresses) > 0:
listeningAddress = addresses[0]
allowAddresses.append(addresses[0])
if len(addresses) > 1:
allowAddresses.append(addresses[1])
server_address = (listeningAddress, 8080)
passwordServer = ServerClass(server_address, HandlerClass)
passwordServer.allow_reuse_address = True
sa = passwordServer.socket.getsockname()
initToken()
loadPasswordFile()
syslog.syslog('serve_password running on %s:%s' % (sa[0], sa[1]))
try:
passwordServer.serve_forever()
```
TO:
```
global listeningAddress
global allowAddresses
addresses = [listeningAddress]
if len(sys.argv) > 1:
addresses = [a.strip() for a in sys.argv[1].split(",") if a.strip()]
listeningAddress = addresses[0] # Need to keep listeningAddress
as-is on the first address as it's global. Used in getPasswordFile()
for addr in addresses:
if addr not in allowAddresses:
allowAddresses.append(addr)
initToken()
loadPasswordFile()
servers = []
for addr in addresses:
server = ServerClass((addr, 8080), HandlerClass)
server.allow_reuse_address = True
servers.append(server)
syslog.syslog('serve_password running on %s:8080' % addr)
for server in servers[1:]: # Add each subsequent server as a thread.
Primary server starts in the try block.
t = threading.Thread(target=server.serve_forever)
t.daemon = True
t.start()
try:
servers[0].serve_forever()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
