Hello CloudStack Community, I am writing to report a console proxy authentication issue we are experiencing in our production CloudStack environment with multiple management servers.
--- Environment: - CloudStack Version: 4.19.3.0 - Hypervisor: KVM - Management Servers: 3 nodes - Console Proxy VM: 1 instance - OS: Ubuntu 22.04 LTS --- Issue Description: In our multi-management-server setup, noVNC console sessions fail intermittently with the error: "Failed to connect to server/access token has expired" After investigation, we identified the following flow causing the failure: 1. User sends a console request via the browser, which is handled by Node1. 2. Node1 generates a session token and stores it in-memory via ConsoleAccessManagerImpl. 3. The AllowConsoleAccessCommand is forwarded to the Console Proxy VM (CPVM). 4. The CPVM sends a ConsoleAccessAuthenticationCommand to verify the session, but it connects to Node2 or Node3 (randomly, as all three MS IPs are listed in the CPVM cmdline). 5. Since the session was stored in-memory on Node1 only and NOT persisted to the database (console_session table), Node2/Node3 cannot find it. 6. Authentication fails with: "External authenticator failed request for vm <uuid> with sid <sid>." Key Observations: - The console_session table in the CloudStack database exists but sessions are NOT being inserted at the time of console requests. - Querying the console_session table immediately after a failed attempt confirms the session UUID is absent. - The CPVM cmdline contains all three MS IPs: host=10.10.11.61,10.10.11.62,10.10.11.63 - The issue is intermittent because authentication succeeds only when the CPVM happens to connect back to the same MS that created the session. -- Regards, Azmir Ahmed Bangladesh Online (BOL) Level 9, SAM Tower, Plot 4, Road 22, Gulshan 1, Dhaka 1212, Bangladesh Tel: +880 9609 000 999, +880 2 5881 5559, Ext: 14193, Fax: +880 2 9895757 Cell: +880 1787680841, Web: www.bol-online.com
