Just to confirm if there's not any misconfiguration at ACS, try doing these and let us know if you get the IP assigned.
$ kubectl run --image=nginx nginx $ kubectl expose pod/nginx --type=LoadBalancer --port=80 For your nginx ingress controller service of type Load Balancer, what does $ kubectl describe svc ingress-nginx-controller say? Thanks, Jayanth ________________________________ From: Eric Green <eric.lee.gr...@gmail.com> Sent: Thursday, December 12, 2024 8:46:12 am To: users@cloudstack.apache.org <users@cloudstack.apache.org> Subject: Kubernetes nginx ingress svc stuck in 'Pending' I am currently successfully using the nginx ingress on AKS. But when I try to deploy it into Cloudstack Kubernetes, the EXTERNAL-PI is stuck in <pending> state. I've tried multiple ways to deploy it and am putting the correct annotations as documented in the Cloudstack Kubernetes provider but I'm still stumped — no IP address showing up. I must have something misconfigured but what it is, eludes me. Any suggestions? Kubernetes versions: :~$ kubectl version Client Version: v1.29.7 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.28.4 How I deploy the nginx ingress: helm install -f nginx-cloudstack.yaml ingress-nginx ingress-nginx --repo https://kubernetes.github.io/ingress-nginx The config file nginx-cloudstack.yaml: ------------- # For deploying nginx load balancer on Cloudstack # Use command: # helm install -f nginx-cloudstack.yaml ingress-nginx ingress-nginx --repo https://kubernetes.github.io/ingress-nginx controller: kind: DaemonSet service: annotations: service.beta.kubernetes.io/cloudstack-load-balancer-proxy-protocol: enabled ingressClassResource: default: true extraArgs: default-ssl-certificate: "default/mycert" config: use-proxy-protocol: "true" ------------- The results: $ kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE ingress-nginx-controller LoadBalancer 10.105.120.100 <pending> 80:32164/TCP,443:32429/TCP 16m ingress-nginx-controller-admission ClusterIP 10.111.151.26 <none> 443/TCP 16m Kubectl get events shows no issues: 19m Normal Scheduled pod/ingress-nginx-admission-create-bfdkq Successfully assigned default/ingress-nginx-admission-create-bfdkq to default-node-193b75945ef 19m Normal Pulling pod/ingress-nginx-admission-create-bfdkq Pulling image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.4@sha256:a9f03b34a3cbfbb26d103a14046ab2c5130a80c3d69d526ff8063d2b37b9fd3f" 19m Normal Pulled pod/ingress-nginx-admission-create-bfdkq Successfully pulled image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.4@sha256:a9f03b34a3cbfbb26d103a14046ab2c5130a80c3d69d526ff8063d2b37b9fd3f" in 2.55s (2.55s including waiting) 19m Normal Created pod/ingress-nginx-admission-create-bfdkq Created container create 19m Normal Started pod/ingress-nginx-admission-create-bfdkq Started container create 19m Normal SuccessfulCreate job/ingress-nginx-admission-create Created pod: ingress-nginx-admission-create-bfdkq 18m Normal Completed job/ingress-nginx-admission-create Job completed 18m Normal Scheduled pod/ingress-nginx-admission-patch-x5wxz Successfully assigned default/ingress-nginx-admission-patch-x5wxz to default-node-193b75945ef 18m Normal Pulled pod/ingress-nginx-admission-patch-x5wxz Container image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.4@sha256:a9f03b34a3cbfbb26d103a14046ab2c5130a80c3d69d526ff8063d2b37b9fd3f" already present on machine 18m Normal Created pod/ingress-nginx-admission-patch-x5wxz Created container patch 18m Normal Started pod/ingress-nginx-admission-patch-x5wxz Started container patch 18m Normal SuccessfulCreate job/ingress-nginx-admission-patch Created pod: ingress-nginx-admission-patch-x5wxz 18m Normal Completed job/ingress-nginx-admission-patch Job completed 18m Normal Scheduled pod/ingress-nginx-controller-lkzt9 Successfully assigned default/ingress-nginx-controller-lkzt9 to default-node-193b759116c 18m Normal Pulling pod/ingress-nginx-controller-lkzt9 Pulling image "registry.k8s.io/ingress-nginx/controller:v1.11.3@sha256:d56f135b6462cfc476447cfe564b83a45e8bb7da2774963b00d12161112270b7" 18m Normal Pulled pod/ingress-nginx-controller-lkzt9 Successfully pulled image "registry.k8s.io/ingress-nginx/controller:v1.11.3@sha256:d56f135b6462cfc476447cfe564b83a45e8bb7da2774963b00d12161112270b7" in 6.128s (6.128s including waiting) 18m Normal Created pod/ingress-nginx-controller-lkzt9 Created container controller 18m Normal Started pod/ingress-nginx-controller-lkzt9 Started container controller 18m Normal RELOAD pod/ingress-nginx-controller-lkzt9 NGINX reload triggered due to a change in configuration 18m Normal Scheduled pod/ingress-nginx-controller-s6zqd Successfully assigned default/ingress-nginx-controller-s6zqd to default-node-193b75945ef 18m Normal Pulling pod/ingress-nginx-controller-s6zqd Pulling image "registry.k8s.io/ingress-nginx/controller:v1.11.3@sha256:d56f135b6462cfc476447cfe564b83a45e8bb7da2774963b00d12161112270b7" 18m Normal Pulled pod/ingress-nginx-controller-s6zqd Successfully pulled image "registry.k8s.io/ingress-nginx/controller:v1.11.3@sha256:d56f135b6462cfc476447cfe564b83a45e8bb7da2774963b00d12161112270b7" in 6.171s (6.171s including waiting) 18m Normal Created pod/ingress-nginx-controller-s6zqd Created container controller 18m Normal Started pod/ingress-nginx-controller-s6zqd Started container controller 18m Normal RELOAD pod/ingress-nginx-controller-s6zqd NGINX reload triggered due to a change in configuration 18m Normal SuccessfulCreate daemonset/ingress-nginx-controller Created pod: ingress-nginx-controller-s6zqd 18m Normal SuccessfulCreate daemonset/ingress-nginx-controller Created pod: ingress-nginx-controller-lkzt9 18m Normal CREATE configmap/ingress-nginx-controller ConfigMap default/ingress-nginx-controller 18m Normal CREATE configmap/ingress-nginx-controller ConfigMap default/ingress-nginx-controller $ Similarly, kubectl logs on the two pods created on the two nodes shows no issues. >From output of 'helm template' the three things needed to hook into Cloudstack >seem to be there, the ConfigMap saying use-proxy-protocol: "true", the Service >saying " service.beta.kubernetes.io/cloudstack-load-balancer-proxy-protocol: "enabled" ", and the inclusion of the ConfigMap into the DaemonSet: --- # Source: ingress-nginx/templates/controller-configmap.yaml apiVersion: v1 kind: ConfigMap metadata: labels: helm.sh/chart: ingress-nginx-4.11.3 app.kubernetes.io/name: ingress-nginx app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/version: "1.11.3" app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/managed-by: Helm app.kubernetes.io/component: controller name: ingress-nginx-controller namespace: default data: allow-snippet-annotations: "false" use-proxy-protocol: "true" --- # Source: ingress-nginx/templates/controller-service.yaml apiVersion: v1 kind: Service metadata: annotations: service.beta.kubernetes.io/cloudstack-load-balancer-proxy-protocol: "enabled" labels: helm.sh/chart: ingress-nginx-4.11.3 app.kubernetes.io/name: ingress-nginx app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/version: "1.11.3" app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/managed-by: Helm app.kubernetes.io/component: controller name: ingress-nginx-controller namespace: default spec: type: LoadBalancer ipFamilyPolicy: SingleStack ipFamilies: - IPv4 ports: - name: http port: 80 protocol: TCP targetPort: http appProtocol: http - name: https port: 443 protocol: TCP targetPort: https appProtocol: https selector: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/component: controller --- # Source: ingress-nginx/templates/controller-daemonset.yaml apiVersion: apps/v1 kind: DaemonSet metadata: labels: helm.sh/chart: ingress-nginx-4.11.3 app.kubernetes.io/name: ingress-nginx app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/version: "1.11.3" app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/managed-by: Helm app.kubernetes.io/component: controller name: ingress-nginx-controller namespace: default spec: selector: matchLabels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/component: controller revisionHistoryLimit: 10 minReadySeconds: 0 template: metadata: labels: helm.sh/chart: ingress-nginx-4.11.3 app.kubernetes.io/name: ingress-nginx app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/version: "1.11.3" app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/managed-by: Helm app.kubernetes.io/component: controller spec: dnsPolicy: ClusterFirst containers: - name: controller image: registry.k8s.io/ingress-nginx/controller:v1.11.3@sha256:d56f135b6462cfc476447cfe564b83a45e8bb7da2774963b00d12161112270b7 imagePullPolicy: IfNotPresent lifecycle: preStop: exec: command: - /wait-shutdown args: - /nginx-ingress-controller - --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller - --election-id=ingress-nginx-leader - --controller-class=k8s.io/ingress-nginx - --ingress-class=nginx - --configmap=$(POD_NAMESPACE)/ingress-nginx-controller - --validating-webhook=:8443 - --validating-webhook-certificate=/usr/local/certificates/cert - --validating-webhook-key=/usr/local/certificates/key - --enable-metrics=false - --default-ssl-certificate=default/mycert securityContext: ... --- Disclaimer *** This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended solely for the use of the addressee(s). If you are not the intended recipient, please notify the sender by e-mail and delete the original message. Further, you are not authorised to copy, disclose, or distribute this e-mail or its contents to any other person and any such actions are unlawful and strictly prohibited. This e-mail may contain viruses. NxtGen Datacenter & Cloud Technologies Private Ltd (“NxtGen”) has taken every reasonable precaution to minimize this risk but is not liable for any damage you may sustain as a result of any virus in this e-mail. You should carry out your own virus checks before opening the e-mail or attachment. NxtGen reserves the right to monitor and review the content of all messages sent to or from this e-mail address. Messages sent to or from this e-mail address may be stored on the NxtGen e-mail system. *** End of Disclaimer ***NXTGEN***
