nosvalds opened a new issue #390: URL: https://github.com/apache/solr-operator/issues/390
Asked in https://kubernetes.slack.com/archives/CQSNS615F/p1639651250121300, recommended to bring here. I’m seeing an issue where my Solr cloud (3 replicas) doesn’t recover when cert-manager renews the certificate. It seems to be tied to my updateStrategy. What I’m seeing: - Certificate is renewed by cert-manager at 60days (expiry is 90days - default for LetsEncrypt), `restartOnTLSSecretUpdate` setting starts to restart the cluster ```yaml solrTLS: restartOnTLSSecretUpdate: true pkcs12Secret: name: solr-tls key: keystore.p12 keyStorePasswordSecret: name: pkcs12-keystore-password key: password-key ``` - Below settings ensure only 1 pod restarts at a time so we still have 2 active pods to serve requests ```yaml updateStrategy: managed: maxPodsUnavailable: 1 maxShardReplicasUnavailable: 1 ``` After the 1 Pod restarts, its collections are shown as having a “Down” status, in the logs I see ```yaml 2021-12-16 10:15:14.746 ERROR (recoveryExecutor-11-thread-3-processing-n:pod-1.default:443_solr x:transaction_10_shard1_replica_n5 c:transaction_10 s:shard1 r:core_node6) [c:transaction_10 s:shard1 r:core_node6 x:transaction_10_shard1_replica_n5] o.a.s.c.RecoveryStrategy Failed to connect leader https://pod-2.default:443/solr on recovery, try again ``` - The cluster is stuck at this state, as the updateStrategy settings won’t allow the other 2 pods to restart since the 1 pod’s collections are “Down” The error seems to indicate that the pod that has restarted can’t communicate with the pods that haven’t restarted yet, presumably because the certificates. - A (bad) “workaround” I found is that if it set the below, then all 3 pods will restart and come up heathy. But this obviously has the downside of a short downtime. ```yaml updateStrategy: managed: maxPodsUnavailable: 3 maxShardReplicasUnavailable: 3 ``` I don’t think the active certs would have been expired, as Lets Encrypt certs have 90day duration and they renewed at 60days (which is the default for cert-manager of 2/3 of the duration). https://cert-manager.io/docs/usage/certificate/#renewal & https://cert-manager.io/docs/faq/#if-renewbefore-or-duration-is-not-defined-what-will-be-the-default-value I wonder if the act of cert-manager renewing the certificates invalidates the active one though? I can’t find this specifically in the docs. This would be a problem then. It also would explain why I was still seeing this problem when triggering a manual renewal using the cert-manager cmctl https://cert-manager.io/docs/usage/cmctl/#renew cli. If this is the case, we would need `restartOnTLSSecretUpdate` to be able to ignore the `updateStrategy. managed. max*Unavailable` settings. Sort of related issue: https://github.com/jetstack/cert-manager/issues/1168, but Solr has `restartOnTLSSecretUpdate` for our pods. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
