sagb opened a new issue, #9418:
URL: https://github.com/apache/cloudstack/issues/9418
<!--
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release and main branch are affected too.
Always add information AFTER of these HTML comments, but no need to delete
the comments.
-->
##### ISSUE TYPE
<!-- Pick one below and delete the rest -->
Bug Report
##### COMPONENT NAME
<!--
Categorize the issue, e.g. API, VR, VPN, UI, etc.
-->
Kubernetes
##### CLOUDSTACK VERSION
<!--
New line separated list of affected versions, commit ID for issues on main
branch.
-->
4.19.0.2
##### CONFIGURATION
<!--
Information about the configuration if relevant, e.g. basic network,
advanced networking, etc. N/A otherwise
-->
Kubernetes 1.27.3, two control nodes.
##### SUMMARY
<!-- Explain the problem/feature briefly -->
Our K8s certificates expired. Seems like Cloudstack didn't automatically
renew them.
I tried to renew them manually on both control nodes using:
```
kubeadm certs renew all
systemctl restart kubelet
```
This updated the certificates separately for each control node, and both are
recognized by Kubernetes when using /etc/kubernetes/admin.conf as kubeconfig.
However, CloudStack's "Kubernetes access" page only showed the old, expired
certificate.
In attempt to trigger an automatic renewal, I've restored nodes from
snapshot, stopped the k8s cluster from Cloudstack's web UI, and started it.
It doesn't start, and there is an exception in
/var/log/cloudstack/management/management-server.log:
```
2024-07-19 10:19:43,690 WARN [c.c.k.c.u.KubernetesClusterUtil]
(API-Job-Executor-19:ctx-b594d6ec job-4889 ctx-21a09500) (logid:767ef7b5) API
endpoint for Kubernetes cluster : mediatech not available
javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake
at
java.base/sun.security.ssl.SSLSocketImpl.handleEOF(SSLSocketImpl.java:1701)
at
java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1519)
at
java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1421)
at
java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:456)
at
java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:427)
at
java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:580)
at
java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:201)
at
java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1614)
at
java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1542)
at
java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:250)
at
com.cloud.kubernetes.cluster.utils.KubernetesClusterUtil.isKubernetesClusterServerRunning(KubernetesClusterUtil.java:239)
at
com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterStartWorker.startStoppedKubernetesCluster(KubernetesClusterStartWorker.java:590)
at
com.cloud.kubernetes.cluster.KubernetesClusterManagerImpl.startKubernetesCluster(KubernetesClusterManagerImpl.java:1324)
at
org.apache.cloudstack.api.command.user.kubernetes.cluster.StartKubernetesClusterCmd.execute(StartKubernetesClusterCmd.java:113)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
at
com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:112)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.io.EOFException: SSL peer shut down incorrectly
at
java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:489)
at
java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:478)
at
java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:160)
... 28 more
```
The abscence of automatic renewal is probably a bug.
Also, I would be grateful for a hint how to recover from the current
situation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
