I just tried restarting the controller service a couple more times and those panics dont show anymore and the deployment has started.
On Mon, 3 Dec 2018 at 10:39 Cameron Braid <[email protected]> wrote: > > Yeah, these look like errors : > > *Dec 03 10:28:44 node01-2018.drivenow.com.au > <http://node01-2018.drivenow.com.au> origin-master-controllers[89591]: > E1203 10:28:44.150240 89591 runtime.go:66] Observed a panic: "invalid > memory address or nil pointer dereference" (runtime error: invalid memory > address or nil pointer dereference)* > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: > /builddir/build/BUILD/origin-3.7.0/_output/local/go/src/ > github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72 > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: > /builddir/build/BUILD/origin-3.7.0/_output/local/go/src/ > github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65 > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: > /builddir/build/BUILD/origin-3.7.0/_output/local/go/src/ > github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51 > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: > /usr/lib/golang/src/runtime/asm_amd64.s:514 > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: /usr/lib/golang/src/runtime/panic.go:489 > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: /usr/lib/golang/src/runtime/panic.go:63 > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: > /usr/lib/golang/src/runtime/signal_unix.go:290 > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: > /builddir/build/BUILD/origin-3.7.0/_output/local/go/src/ > github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/daemon/daemoncontroller.go:155 > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: > /builddir/build/BUILD/origin-3.7.0/_output/local/go/src/ > github.com/openshift/origin/vendor/k8s.io/client-go/tools/cache/controller.go:192 > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: <autogenerated>:57 > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: > /builddir/build/BUILD/origin-3.7.0/_output/local/go/src/ > github.com/openshift/origin/vendor/k8s.io/client-go/tools/cache/shared_informer.go:547 > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: > /usr/lib/golang/src/runtime/asm_amd64.s:2197 > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: I1203 10:28:44.230853 89591 > controller_utils.go:1032] Caches are synced for RC controller > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: W1203 10:28:44.300741 89591 > shared_informer.go:298] resyncPeriod 30000000000 is smaller than > resyncCheckPeriod 600000000000 and the informer has already started. > Changing it to 600000000000 > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: I1203 10:28:44.300811 89591 > controllermanager.go:466] Started "statefulset" > *Dec 03 10:28:44 node01-2018.drivenow.com.au > <http://node01-2018.drivenow.com.au> origin-master-controllers[89591]: > E1203 10:28:44.300820 89591 runtime.go:66] Observed a panic: "invalid > memory address or nil pointer dereference" (runtime error: invalid memory > address or nil pointer dereference)* > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: > /builddir/build/BUILD/origin-3.7.0/_output/local/go/src/ > github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72 > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: > /builddir/build/BUILD/origin-3.7.0/_output/local/go/src/ > github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65 > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: > /builddir/build/BUILD/origin-3.7.0/_output/local/go/src/ > github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51 > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: > /usr/lib/golang/src/runtime/asm_amd64.s:514 > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: /usr/lib/golang/src/runtime/panic.go:489 > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: /usr/lib/golang/src/runtime/panic.go:63 > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: > /usr/lib/golang/src/runtime/signal_unix.go:290 > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: > /builddir/build/BUILD/origin-3.7.0/_output/local/go/src/ > github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/statefulset/stateful_set.go:365 > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: > /builddir/build/BUILD/origin-3.7.0/_output/local/go/src/ > github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/statefulset/stateful_set.go:174 > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: > /builddir/build/BUILD/origin-3.7.0/_output/local/go/src/ > github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/statefulset/stateful_set.go:111 > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: > /builddir/build/BUILD/origin-3.7.0/_output/local/go/src/ > github.com/openshift/origin/vendor/k8s.io/client-go/tools/cache/controller.go:192 > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: <autogenerated>:57 > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: > /builddir/build/BUILD/origin-3.7.0/_output/local/go/src/ > github.com/openshift/origin/vendor/k8s.io/client-go/tools/cache/shared_informer.go:547 > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: > /usr/lib/golang/src/runtime/asm_amd64.s:2197 > > > > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: I1203 10:28:44.750941 89591 > controllermanager.go:456] Starting "deployment" > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: I1203 10:28:44.750997 89591 > attach_detach_controller.go:242] Starting attach detach controller > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: I1203 10:28:44.751014 89591 > controller_utils.go:1025] Waiting for caches to sync for attach detach > controller > *Dec 03 10:28:44 node01-2018.drivenow.com.au > <http://node01-2018.drivenow.com.au> origin-master-controllers[89591]: > E1203 10:28:44.900441 89591 runtime.go:66] Observed a panic: "invalid > memory address or nil pointer dereference" (runtime error: invalid memory > address or nil pointer dereference)* > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: > /builddir/build/BUILD/origin-3.7.0/_output/local/go/src/ > github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72 > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: > /builddir/build/BUILD/origin-3.7.0/_output/local/go/src/ > github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65 > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: > /builddir/build/BUILD/origin-3.7.0/_output/local/go/src/ > github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51 > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: > /usr/lib/golang/src/runtime/asm_amd64.s:514 > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: /usr/lib/golang/src/runtime/panic.go:489 > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: /usr/lib/golang/src/runtime/panic.go:63 > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: > /usr/lib/golang/src/runtime/signal_unix.go:290 > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: > /builddir/build/BUILD/origin-3.7.0/_output/local/go/src/ > github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/deployment/deployment_controller.go:169 > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: > /builddir/build/BUILD/origin-3.7.0/_output/local/go/src/ > github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/deployment/deployment_controller.go:121 > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: > /builddir/build/BUILD/origin-3.7.0/_output/local/go/src/ > github.com/openshift/origin/vendor/k8s.io/client-go/tools/cache/controller.go:192 > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: <autogenerated>:57 > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: > /builddir/build/BUILD/origin-3.7.0/_output/local/go/src/ > github.com/openshift/origin/vendor/k8s.io/client-go/tools/cache/shared_informer.go:547 > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: > /usr/lib/golang/src/runtime/asm_amd64.s:2197 > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: I1203 10:28:44.900445 89591 > controllermanager.go:466] Started "deployment" > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: W1203 10:28:44.900490 89591 > controllermanager.go:450] "ttl" is disabled > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: I1203 10:28:44.900454 89591 > deployment_controller.go:152] Starting deployment controller > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: I1203 10:28:44.900765 89591 > controller_utils.go:1025] Waiting for caches to sync for deployment > controller > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: E1203 10:28:44.907405 89591 > actual_state_of_world.go:478] Failed to set statusUpdateNeeded to needed > true because nodeName="node03-2018" does not exist > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: E1203 10:28:44.907431 89591 > actual_state_of_world.go:492] Failed to update statusUpdateNeeded field in > actual state of world: Failed to set statusUpdateNeeded to needed true > because nodeName="node03-2018" does not exist > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: E1203 10:28:44.907467 89591 > actual_state_of_world.go:478] Failed to set statusUpdateNeeded to needed > true because nodeName="node01-2018" does not exist > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: E1203 10:28:44.907478 89591 > actual_state_of_world.go:492] Failed to update statusUpdateNeeded field in > actual state of world: Failed to set statusUpdateNeeded to needed true > because nodeName="node01-2018" does not exist > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: E1203 10:28:44.907521 89591 > actual_state_of_world.go:478] Failed to set statusUpdateNeeded to needed > true because nodeName="node02-2018" does not exist > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: E1203 10:28:44.907535 89591 > actual_state_of_world.go:492] Failed to update statusUpdateNeeded field in > actual state of world: Failed to set statusUpdateNeeded to needed true > because nodeName="node02-2018" does not exist > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: I1203 10:28:44.923823 89591 > controller_utils.go:1032] Caches are synced for replica set controller > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: I1203 10:28:44.943026 89591 > controller_utils.go:1032] Caches are synced for node controller > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: I1203 10:28:44.943076 89591 > taint_controller.go:182] Starting NoExecuteTaintManager > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: I1203 10:28:44.943092 89591 > nodecontroller.go:525] NodeController observed a new Node: "node03-2018" > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: I1203 10:28:44.943113 89591 > controller_utils.go:274] Recording Registered Node node03-2018 in > NodeController event message for node node03-2018 > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: I1203 10:28:44.943145 89591 > nodecontroller.go:542] Initializing eviction metric for zone: > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: I1203 10:28:44.943159 89591 > nodecontroller.go:525] NodeController observed a new Node: "node01-2018" > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: I1203 10:28:44.943167 89591 > controller_utils.go:274] Recording Registered Node node01-2018 in > NodeController event message for node node01-2018 > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: I1203 10:28:44.943177 89591 > nodecontroller.go:525] NodeController observed a new Node: "node02-2018" > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: I1203 10:28:44.943185 89591 > controller_utils.go:274] Recording Registered Node node02-2018 in > NodeController event message for node node02-2018 > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: W1203 10:28:44.943257 89591 > nodecontroller.go:877] Missing timestamp for Node node03-2018. Assuming now > as a timestamp. > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: I1203 10:28:44.943280 89591 > event.go:218] Event(v1.ObjectReference{Kind:"Node", Namespace:"", > Name:"node03-2018", UID:"9d690c08-f0ed-11e7-a024-0cc47a39144c", > APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: > 'RegisteredNode' Node node03-2018 event: Registered Node node03-2018 in > NodeController > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: I1203 10:28:44.943307 89591 > event.go:218] Event(v1.ObjectReference{Kind:"Node", Namespace:"", > Name:"node01-2018", UID:"9d638709-f0ed-11e7-a024-0cc47a39144c", > APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: > 'RegisteredNode' Node node01-2018 event: Registered Node node01-2018 in > NodeController > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: I1203 10:28:44.943319 89591 > event.go:218] Event(v1.ObjectReference{Kind:"Node", Namespace:"", > Name:"node02-2018", UID:"9d6692f6-f0ed-11e7-a024-0cc47a39144c", > APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: > 'RegisteredNode' Node node02-2018 event: Registered Node node02-2018 in > NodeController > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: W1203 10:28:44.943296 89591 > nodecontroller.go:877] Missing timestamp for Node node01-2018. Assuming now > as a timestamp. > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: W1203 10:28:44.943364 89591 > nodecontroller.go:877] Missing timestamp for Node node02-2018. Assuming now > as a timestamp. > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: I1203 10:28:44.943394 89591 > nodecontroller.go:793] NodeController detected that zone is now in state > Normal. > Dec 03 10:28:44 node01-2018.drivenow.com.au > origin-master-controllers[89591]: I1203 10:28:44.950122 89591 > controller_utils.go:1032] Caches are synced for daemon sets controller > > Cameron > > On Mon, 3 Dec 2018 at 02:48 Clayton Coleman <[email protected]> wrote: > >> Are there errors in the controller logs? >> >> On Dec 2, 2018, at 2:42 AM, Cameron Braid <[email protected]> wrote: >> >> Sorry, a typo - its a 3.7 cluster not 3.6 >> >> ~> oc version >> >> oc v3.7.2+282e43f >> >> kubernetes v1.7.6+a08f5eeb62 >> >> features: Basic-Auth >> >> >> Server >> >> openshift v3.7.0+7ed6862 >> >> kubernetes v1.7.6+a08f5eeb62 >> >> On Sun, 2 Dec 2018 at 18:39 Cameron Braid <[email protected]> wrote: >> >>> I have a strange issue in my 3.6 cluster. I create a >>> extensions/v1beta1 Deployment and nothing happens. No pods are >>> created. oc describe shows >>> >>> oc -n drivenow-staging-x describe deployment strimzi-cluster-operator >>> Name: strimzi-cluster-operator >>> Namespace: drivenow-staging-x >>> CreationTimestamp: Sun, 02 Dec 2018 01:33:10 +1100 >>> Labels: app=strimzi >>> Annotations: <none> >>> Selector: name=strimzi-cluster-operator >>> *Replicas: 1 desired | 0 updated | 0 total | 0 available | 0 unavailable* >>> StrategyType: Recreate >>> MinReadySeconds: 0 >>> Pod Template: >>> Labels: name=strimzi-cluster-operator >>> Service Account: strimzi-cluster-operator >>> Containers: >>> strimzi-cluster-operator: >>> Image: strimzi/cluster-operator:0.6.0 >>> Port: <none> >>> Limits: >>> cpu: 1 >>> memory: 256Mi >>> Requests: >>> cpu: 200m >>> memory: 256Mi >>> Liveness: http-get http://:8080/healthy delay=10s timeout=1s >>> period=30s #success=1 #failure=3 >>> Readiness: http-get http://:8080/ready delay=10s timeout=1s >>> period=30s #success=1 #failure=3 >>> Environment: ... >>> Mounts: <none> >>> Volumes: <none> >>> OldReplicaSets: <none> >>> NewReplicaSet: <none> >>> Events: <none> >>> >>> All nodes are ready, schedulable and there doesnt appear to be antyhing >>> in the logs. >>> >>> I've tried restarting origin-node, origin-master-api and >>> origin-master-controllers on all nodes and that has no impact. >>> >>> I'm out of ideas. >>> >>> Cameron >>> >> _______________________________________________ >> users mailing list >> [email protected] >> http://lists.openshift.redhat.com/openshiftmm/listinfo/users >> >>
_______________________________________________ users mailing list [email protected] http://lists.openshift.redhat.com/openshiftmm/listinfo/users
