CentOS PaaS SIG meeting (2018-09-12)
Hello, It's time for our weekly PaaS SIG sync-up meeting Time: 1700 UTC - Wednesdays (date -d "1700 UTC") Date: Tomorrow Wednesday, September 12 2018 Where: IRC- Freenode - #centos-devel Agenda: - OpenShift Current Status -- rpms -- automation - Open Floor Minutes from last meeting: https://www.centos.org/minutes/2018/September/centos-devel.2018-09-05-17.00.log.html -- Ricardo Martinelli de Oliveira Senior Software Engineer ___ users mailing list users@lists.openshift.redhat.com http://lists.openshift.redhat.com/openshiftmm/listinfo/users
Re: 3.10 openshift-ansible install is failing - cni not configured
Thanks for the reply. I was pinning the release only because I was updating a working inventory from 3.9 and forgot that I had pinned that release to avoid upgrading to 3.10. I've updated the inventory to set openshift_release="3.10" and commented out openshift_image_tag and openshift_pkg_version so that the ansible scripts will derive the correct values. I have re-run the installer using a fresh version of the master and minion VMs (CentOS 7.5 with docker installed). I get the same error. The output of systemctl status origin-node on the master is: ● origin-node.service - OpenShift Node Loaded: loaded (/etc/systemd/system/origin-node.service; enabled; vendor preset: disabled) Active: active (running) since Tue 2018-09-11 10:31:51 PDT; 3min 29s ago Docs: https://github.com/openshift/origin Main PID: 21183 (hyperkube) CGroup: /system.slice/origin-node.service └─21183 /usr/bin/hyperkube kubelet --v=2 --address=0.0.0.0 --allow-privileged=true --anonymous-auth=true --authentication-token-webhook=true --authentication-token-webhook-cache-ttl=5m --authorization-mode=Webhook --authorization-webhook-cache-authorized-ttl=5m --authorization-webhook-cache-unauthorized-ttl=5m --bootstrap-kubeconfig=/etc/origin/node/bootstrap.kubeconfig --cadvisor-port=0 --cert-dir=/etc/origin/node/certificates --cgroup-driver=systemd --client-ca-file=/etc/origin/node/client-ca.crt --cluster-dns=10.93.233.126 --cluster-domain=cluster.local --container-runtime-endpoint=/var/run/dockershim.sock --containerized=false --enable-controller-attach-detach=true --experimental-dockershim-root-directory=/var/lib/dockershim --fail-swap-on=false --feature-gates=RotateKubeletClientCertificate=true,RotateKubeletServerCertificate=true --file-check-frequency=0s --healthz-bind-address= --healthz-port=0 --host-ipc-sources=api --host-ipc-sources=file --host-network-sources=api --host-network-sources=file --host-pid-sources=api --host-pid-sources=file --hostname-override= --http-check-frequency=0s --image-service-endpoint=/var/run/dockershim.sock --iptables-masquerade-bit=0 --kubeconfig=/etc/origin/node/node.kubeconfig --max-pods=250 --network-plugin=cni --node-ip= --pod-infra-container-image= docker.io/openshift/origin-pod:v3.10.0 --pod-manifest-path=/etc/origin/node/pods --port=10250 --read-only-port=0 --register-node=true --root-dir=/var/lib/origin/openshift.local.volumes --rotate-certificates=true --tls-cert-file= --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305 --tls-cipher-suites=TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA --tls-cipher-suites=TLS_RSA_WITH_AES_128_GCM_SHA256 --tls-cipher-suites=TLS_RSA_WITH_AES_256_GCM_SHA384 --tls-cipher-suites=TLS_RSA_WITH_AES_128_CBC_SHA --tls-cipher-suites=TLS_RSA_WITH_AES_256_CBC_SHA --tls-min-version=VersionTLS12 --tls-private-key-file= Sep 11 10:35:17 ph67-dev-psh-oso310-master origin-node[21183]: E0911 10:35:17.667696 21183 reflector.go:205] github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://ph67-dev-psh-oso310-master.pdx.hcl.com:8443/api/v1/pods?fieldSelector=spec.nodeName%3Dph67-dev-psh-oso310-master&limit=500&resourceVersion=0: dial tcp 10.93.233.126:8443: getsockopt: connection refused Sep 11 10:35:17 ph67-dev-psh-oso310-master origin-node[21183]: E0911 10:35:17.668264 21183 reflector.go:205] github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubelet/kubelet.go:461: Failed to list *v1.Node: Get https://ph67-dev-psh-oso310-master.pdx.hcl.com:8443/api/v1/nodes?fieldSelector=metadata.name%3Dph67-dev-psh-oso310-master&limit=500&resourceVersion=0: dial tcp 10.93.233.126:8443: getsockopt: connection refused Sep 11 10:35:18 ph67-dev-psh-oso310-master origin-node[21183]: W0911 10:35:18.518516 21183 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d Sep 11 10:35:18 ph67-dev-psh-oso310-master origin-node[21183]: E0911 10:35:18.518716 21183 kubelet.go:2143] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized Sep 11 10:35:18 ph67-dev-psh-oso310-master origin-node[21183]: E0911 10:35:18.667894 21183 reflector.go:205] github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubelet/kubelet.go:452: Failed to list *v1.Service: Get https://ph67-dev-psh-oso310-master.pdx.hcl.com:8443/api/v1/services?limit=500&re
Re: 3.10 openshift-ansible install is failing - cni not configured
Hi Peter,
Is there a reason behind pinning the release, image_tag and pkg_version
variables to this release version? I would recommend you use just 3.10,
this will ensure that you get the latest version of Openshift installed
Futhermore I found several bugreports with this issue:
https://github.com/openshift/openshift-ansible/issues/7967
https://bugzilla.redhat.com/show_bug.cgi?id=1568583
https://bugzilla.redhat.com/show_bug.cgi?id=1568450#c7
Some more logs from the node would help to troubleshoot the problem.
Best regards,
Alexander
On Tue, Sep 11, 2018 at 3:50 PM, Peter Heitman wrote:
> I am attempting to use the openshift-ansible installer for 3.10 to deploy
> openshift on 1 master and 3 minions. I am using the same inventory I have
> been using for 3.9 with the changes shown below. I'm consistently hitting a
> problem with the control plane pods not appearing. Looking in to it, it
> seems that the cni plugin is not being configured properly. From systemctl
> status origin-node, I see the following:
>
> E0911 06:19:25.821170 18922 kubelet.go:2143] Container runtime network
> not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker:
> network plugin is not ready: cni config uninitialized
>
> Is there something I need to add to my 3.10 inventory to address this? Are
> there other workarounds?
>
> - openshift_release=v3.9.0
> + openshift_release=v3.10.0
>
> - openshift_image_tag=v3.9.0
> - openshift_image_tag=v3.10.0
> + openshift_pkg_version=-3.10.0
> + openshift_pkg_version=-3.9.0
>
> - openshift_metrics_image_version=v3.9
> + openshift_metrics_image_version=v3.10
>
> - [masters]
> - openshift_node_labels="{'node-role.kubernetes.io/master':
> 'true', 'node-role.kubernetes.io/infra': 'true'}"
> openshift_schedulable=true
>
> + [masters]
> +
>
> + [masters:vars]
> + #openshift_node_group_name="node-config-master"
> + openshift_node_group_name="node-config-master-infra"
> + openshift_schedulable=true
>
> - [compute-nodes]
> - openshift_node_labels="{'node-role.kubernetes.io/compute':
> 'true'}" openshift_schedulable=true
> - openshift_node_labels="{'node-role.kubernetes.io/compute':
> 'true'}" openshift_schedulable=true
> - openshift_node_labels="{'node-role.kubernetes.io/compute':
> 'true'}" openshift_schedulable=true
>
> + [compute-nodes]
> +
> +
> +
>
> + [compute-nodes:vars]
> + openshift_node_group_name="node-config-compute"
> + openshift_schedulable=true
>
>
> ___
> users mailing list
> users@lists.openshift.redhat.com
> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>
>
--
Alexander Bartilla
IT-Consultant
Cloudwerkstatt GmbH - Lassallestraße 7b – A-1020 Wien
+43-660-8989058
alexander.barti...@cloudwerkstatt.com
*[image: id:image001.png@01D24B57.D1D08F70]*
--
Cloudwerkstatt GmbH - Lassallestraße 7b - A-1020 Wien -
ATU68384759 - FN408516i - Handelsgericht Wien
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users
3.10 openshift-ansible install is failing - cni not configured
I am attempting to use the openshift-ansible installer for 3.10 to deploy
openshift on 1 master and 3 minions. I am using the same inventory I have
been using for 3.9 with the changes shown below. I'm consistently hitting a
problem with the control plane pods not appearing. Looking in to it, it
seems that the cni plugin is not being configured properly. From systemctl
status origin-node, I see the following:
E0911 06:19:25.821170 18922 kubelet.go:2143] Container runtime network
not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker:
network plugin is not ready: cni config uninitialized
Is there something I need to add to my 3.10 inventory to address this? Are
there other workarounds?
- openshift_release=v3.9.0
+ openshift_release=v3.10.0
- openshift_image_tag=v3.9.0
- openshift_image_tag=v3.10.0
+ openshift_pkg_version=-3.10.0
+ openshift_pkg_version=-3.9.0
- openshift_metrics_image_version=v3.9
+ openshift_metrics_image_version=v3.10
- [masters]
- openshift_node_labels="{'node-role.kubernetes.io/master':
'true', 'node-role.kubernetes.io/infra': 'true'}" openshift_schedulable=true
+ [masters]
+
+ [masters:vars]
+ #openshift_node_group_name="node-config-master"
+ openshift_node_group_name="node-config-master-infra"
+ openshift_schedulable=true
- [compute-nodes]
- openshift_node_labels="{'node-role.kubernetes.io/compute':
'true'}" openshift_schedulable=true
- openshift_node_labels="{'node-role.kubernetes.io/compute':
'true'}" openshift_schedulable=true
- openshift_node_labels="{'node-role.kubernetes.io/compute':
'true'}" openshift_schedulable=true
+ [compute-nodes]
+
+
+
+ [compute-nodes:vars]
+ openshift_node_group_name="node-config-compute"
+ openshift_schedulable=true
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users
bad certificates with ansible service broker
We're having problems with the ansible service broker with the etcd rejecting the certificate of the ansible service broker. In the logs of the asb-etcd pod I see this: 2018-09-11 09:13:26.779392 I | embed: rejected connection from "127.0.0.1:50656" (error "tls: failed to verify client's certificate: x509: certificate signed by unknown authority", ServerName "") WARNING: 2018/09/11 09:13:26 Failed to dial 0.0.0.0:2379: connection error: desc = "transport: authentication handshake failed: remote error: tls: bad certificate"; please retry. This results in the asb pod to failing to start. I believe this may have happened after the cluster certificates were updated using the redeploy-certificates.yml playbook. This is using Origin 3.7.2. Any thoughts on how to correct this? ___ users mailing list users@lists.openshift.redhat.com http://lists.openshift.redhat.com/openshiftmm/listinfo/users
