Re: Origin 3.6 cluster no longer deploying deployemnts

2018-12-02 Thread Cameron Braid 
I just tried restarting the controller service a couple more times and
those panics dont show anymore and the deployment has started.



On Mon, 3 Dec 2018 at 10:39 Cameron Braid  wrote:

>
> Yeah, these look like errors :
>
> *Dec 03 10:28:44 node01-2018.drivenow.com.au
>  origin-master-controllers[89591]:
> E1203 10:28:44.150240   89591 runtime.go:66] Observed a panic: "invalid
> memory address or nil pointer dereference" (runtime error: invalid memory
> address or nil pointer dereference)*
> Dec 03 10:28:44 node01-2018.drivenow.com.au
> origin-master-controllers[89591]:
> /builddir/build/BUILD/origin-3.7.0/_output/local/go/src/
> github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72
> Dec 03 10:28:44 node01-2018.drivenow.com.au
> origin-master-controllers[89591]:
> /builddir/build/BUILD/origin-3.7.0/_output/local/go/src/
> github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
> Dec 03 10:28:44 node01-2018.drivenow.com.au
> origin-master-controllers[89591]:
> /builddir/build/BUILD/origin-3.7.0/_output/local/go/src/
> github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
> Dec 03 10:28:44 node01-2018.drivenow.com.au
> origin-master-controllers[89591]:
> /usr/lib/golang/src/runtime/asm_amd64.s:514
> Dec 03 10:28:44 node01-2018.drivenow.com.au
> origin-master-controllers[89591]: /usr/lib/golang/src/runtime/panic.go:489
> Dec 03 10:28:44 node01-2018.drivenow.com.au
> origin-master-controllers[89591]: /usr/lib/golang/src/runtime/panic.go:63
> Dec 03 10:28:44 node01-2018.drivenow.com.au
> origin-master-controllers[89591]:
> /usr/lib/golang/src/runtime/signal_unix.go:290
> Dec 03 10:28:44 node01-2018.drivenow.com.au
> origin-master-controllers[89591]:
> /builddir/build/BUILD/origin-3.7.0/_output/local/go/src/
> github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/daemon/daemoncontroller.go:155
> Dec 03 10:28:44 node01-2018.drivenow.com.au
> origin-master-controllers[89591]:
> /builddir/build/BUILD/origin-3.7.0/_output/local/go/src/
> github.com/openshift/origin/vendor/k8s.io/client-go/tools/cache/controller.go:192
> Dec 03 10:28:44 node01-2018.drivenow.com.au
> origin-master-controllers[89591]: :57
> Dec 03 10:28:44 node01-2018.drivenow.com.au
> origin-master-controllers[89591]:
> /builddir/build/BUILD/origin-3.7.0/_output/local/go/src/
> github.com/openshift/origin/vendor/k8s.io/client-go/tools/cache/shared_informer.go:547
> Dec 03 10:28:44 node01-2018.drivenow.com.au
> origin-master-controllers[89591]:
> /usr/lib/golang/src/runtime/asm_amd64.s:2197
> Dec 03 10:28:44 node01-2018.drivenow.com.au
> origin-master-controllers[89591]: I1203 10:28:44.230853   89591
> controller_utils.go:1032] Caches are synced for RC controller
> Dec 03 10:28:44 node01-2018.drivenow.com.au
> origin-master-controllers[89591]: W1203 10:28:44.300741   89591
> shared_informer.go:298] resyncPeriod 300 is smaller than
> resyncCheckPeriod 6000 and the informer has already started.
> Changing it to 6000
> Dec 03 10:28:44 node01-2018.drivenow.com.au
> origin-master-controllers[89591]: I1203 10:28:44.300811   89591
> controllermanager.go:466] Started "statefulset"
> *Dec 03 10:28:44 node01-2018.drivenow.com.au
>  origin-master-controllers[89591]:
> E1203 10:28:44.300820   89591 runtime.go:66] Observed a panic: "invalid
> memory address or nil pointer dereference" (runtime error: invalid memory
> address or nil pointer dereference)*
> Dec 03 10:28:44 node01-2018.drivenow.com.au
> origin-master-controllers[89591]:
> /builddir/build/BUILD/origin-3.7.0/_output/local/go/src/
> github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72
> Dec 03 10:28:44 node01-2018.drivenow.com.au
> origin-master-controllers[89591]:
> /builddir/build/BUILD/origin-3.7.0/_output/local/go/src/
> github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
> Dec 03 10:28:44 node01-2018.drivenow.com.au
> origin-master-controllers[89591]:
> /builddir/build/BUILD/origin-3.7.0/_output/local/go/src/
> github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
> Dec 03 10:28:44 node01-2018.drivenow.com.au
> origin-master-controllers[89591]:
> /usr/lib/golang/src/runtime/asm_amd64.s:514
> Dec 03 10:28:44 node01-2018.drivenow.com.au
> origin-master-controllers[89591]: /usr/lib/golang/src/runtime/panic.go:489
> Dec 03 10:28:44 node01-2018.drivenow.com.au
> origin-master-controllers[89591]: /usr/lib/golang/src/runtime/panic.go:63
> Dec 03 10:28:44 node01-2018.drivenow.com.au
> origin-master-controllers[89591]:
> /usr/lib/golang/src/runtime/signal_unix.go:290
> Dec 03 10:28:44 node01-2018.drivenow.com.au
> origin-master-controllers[89591]:
> /builddir/build/BUILD/origin-3.7.0/_output/local/go/src/
> github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/statefulset/stateful_set.go:365

Re: Origin 3.6 cluster no longer deploying deployemnts

2018-12-02 Thread Cameron Braid 
Yeah, these look like errors :

*Dec 03 10:28:44 node01-2018.drivenow.com.au
 origin-master-controllers[89591]:
E1203 10:28:44.150240   89591 runtime.go:66] Observed a panic: "invalid
memory address or nil pointer dereference" (runtime error: invalid memory
address or nil pointer dereference)*
Dec 03 10:28:44 node01-2018.drivenow.com.au
origin-master-controllers[89591]:
/builddir/build/BUILD/origin-3.7.0/_output/local/go/src/
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72
Dec 03 10:28:44 node01-2018.drivenow.com.au
origin-master-controllers[89591]:
/builddir/build/BUILD/origin-3.7.0/_output/local/go/src/
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
Dec 03 10:28:44 node01-2018.drivenow.com.au
origin-master-controllers[89591]:
/builddir/build/BUILD/origin-3.7.0/_output/local/go/src/
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
Dec 03 10:28:44 node01-2018.drivenow.com.au
origin-master-controllers[89591]:
/usr/lib/golang/src/runtime/asm_amd64.s:514
Dec 03 10:28:44 node01-2018.drivenow.com.au
origin-master-controllers[89591]: /usr/lib/golang/src/runtime/panic.go:489
Dec 03 10:28:44 node01-2018.drivenow.com.au
origin-master-controllers[89591]: /usr/lib/golang/src/runtime/panic.go:63
Dec 03 10:28:44 node01-2018.drivenow.com.au
origin-master-controllers[89591]:
/usr/lib/golang/src/runtime/signal_unix.go:290
Dec 03 10:28:44 node01-2018.drivenow.com.au
origin-master-controllers[89591]:
/builddir/build/BUILD/origin-3.7.0/_output/local/go/src/
github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/daemon/daemoncontroller.go:155
Dec 03 10:28:44 node01-2018.drivenow.com.au
origin-master-controllers[89591]:
/builddir/build/BUILD/origin-3.7.0/_output/local/go/src/
github.com/openshift/origin/vendor/k8s.io/client-go/tools/cache/controller.go:192
Dec 03 10:28:44 node01-2018.drivenow.com.au
origin-master-controllers[89591]: :57
Dec 03 10:28:44 node01-2018.drivenow.com.au
origin-master-controllers[89591]:
/builddir/build/BUILD/origin-3.7.0/_output/local/go/src/
github.com/openshift/origin/vendor/k8s.io/client-go/tools/cache/shared_informer.go:547
Dec 03 10:28:44 node01-2018.drivenow.com.au
origin-master-controllers[89591]:
/usr/lib/golang/src/runtime/asm_amd64.s:2197
Dec 03 10:28:44 node01-2018.drivenow.com.au
origin-master-controllers[89591]: I1203 10:28:44.230853   89591
controller_utils.go:1032] Caches are synced for RC controller
Dec 03 10:28:44 node01-2018.drivenow.com.au
origin-master-controllers[89591]: W1203 10:28:44.300741   89591
shared_informer.go:298] resyncPeriod 300 is smaller than
resyncCheckPeriod 6000 and the informer has already started.
Changing it to 6000
Dec 03 10:28:44 node01-2018.drivenow.com.au
origin-master-controllers[89591]: I1203 10:28:44.300811   89591
controllermanager.go:466] Started "statefulset"
*Dec 03 10:28:44 node01-2018.drivenow.com.au
 origin-master-controllers[89591]:
E1203 10:28:44.300820   89591 runtime.go:66] Observed a panic: "invalid
memory address or nil pointer dereference" (runtime error: invalid memory
address or nil pointer dereference)*
Dec 03 10:28:44 node01-2018.drivenow.com.au
origin-master-controllers[89591]:
/builddir/build/BUILD/origin-3.7.0/_output/local/go/src/
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72
Dec 03 10:28:44 node01-2018.drivenow.com.au
origin-master-controllers[89591]:
/builddir/build/BUILD/origin-3.7.0/_output/local/go/src/
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
Dec 03 10:28:44 node01-2018.drivenow.com.au
origin-master-controllers[89591]:
/builddir/build/BUILD/origin-3.7.0/_output/local/go/src/
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
Dec 03 10:28:44 node01-2018.drivenow.com.au
origin-master-controllers[89591]:
/usr/lib/golang/src/runtime/asm_amd64.s:514
Dec 03 10:28:44 node01-2018.drivenow.com.au
origin-master-controllers[89591]: /usr/lib/golang/src/runtime/panic.go:489
Dec 03 10:28:44 node01-2018.drivenow.com.au
origin-master-controllers[89591]: /usr/lib/golang/src/runtime/panic.go:63
Dec 03 10:28:44 node01-2018.drivenow.com.au
origin-master-controllers[89591]:
/usr/lib/golang/src/runtime/signal_unix.go:290
Dec 03 10:28:44 node01-2018.drivenow.com.au
origin-master-controllers[89591]:
/builddir/build/BUILD/origin-3.7.0/_output/local/go/src/
github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/statefulset/stateful_set.go:365
Dec 03 10:28:44 node01-2018.drivenow.com.au
origin-master-controllers[89591]:
/builddir/build/BUILD/origin-3.7.0/_output/local/go/src/
github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/statefulset/stateful_set.go:174
Dec 03 10:28:44 node01-2018.drivenow.com.au
origin-master-controllers[89591]:

Re: Origin 3.6 cluster no longer deploying deployemnts

2018-12-02 Thread Clayton Coleman 
Are there errors in the controller logs?

On Dec 2, 2018, at 2:42 AM, Cameron Braid  wrote:

Sorry, a typo - its a 3.7 cluster not 3.6

 ~> oc version

oc v3.7.2+282e43f

kubernetes v1.7.6+a08f5eeb62

features: Basic-Auth


Server

openshift v3.7.0+7ed6862

kubernetes v1.7.6+a08f5eeb62

On Sun, 2 Dec 2018 at 18:39 Cameron Braid  wrote:

> I have a strange issue in my 3.6 cluster.  I create a extensions/v1beta1 
> Deployment
> and nothing happens.  No pods are created.  oc describe shows
>
>  oc -n drivenow-staging-x describe deployment strimzi-cluster-operator
> Name: strimzi-cluster-operator
> Namespace: drivenow-staging-x
> CreationTimestamp: Sun, 02 Dec 2018 01:33:10 +1100
> Labels: app=strimzi
> Annotations: 
> Selector: name=strimzi-cluster-operator
> *Replicas: 1 desired | 0 updated | 0 total | 0 available | 0 unavailable*
> StrategyType: Recreate
> MinReadySeconds: 0
> Pod Template:
>   Labels: name=strimzi-cluster-operator
>   Service Account: strimzi-cluster-operator
>   Containers:
>strimzi-cluster-operator:
> Image: strimzi/cluster-operator:0.6.0
> Port: 
> Limits:
>   cpu: 1
>   memory: 256Mi
> Requests:
>   cpu: 200m
>   memory: 256Mi
> Liveness: http-get http://:8080/healthy delay=10s timeout=1s
> period=30s #success=1 #failure=3
> Readiness: http-get http://:8080/ready delay=10s timeout=1s
> period=30s #success=1 #failure=3
> Environment: ...
> Mounts: 
>   Volumes: 
> OldReplicaSets: 
> NewReplicaSet: 
> Events: 
>
> All nodes are ready, schedulable and there doesnt appear to be antyhing in
> the logs.
>
> I've tried restarting origin-node, origin-master-api and
> origin-master-controllers on all nodes and that has no impact.
>
> I'm out of ideas.
>
> Cameron
>
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Origin 3.6 cluster no longer deploying deployemnts

2018-12-01 Thread Cameron Braid 
Sorry, a typo - its a 3.7 cluster not 3.6

 ~> oc version

oc v3.7.2+282e43f

kubernetes v1.7.6+a08f5eeb62

features: Basic-Auth


Server

openshift v3.7.0+7ed6862

kubernetes v1.7.6+a08f5eeb62

On Sun, 2 Dec 2018 at 18:39 Cameron Braid  wrote:

> I have a strange issue in my 3.6 cluster.  I create a extensions/v1beta1 
> Deployment
> and nothing happens.  No pods are created.  oc describe shows
>
>  oc -n drivenow-staging-x describe deployment strimzi-cluster-operator
> Name: strimzi-cluster-operator
> Namespace: drivenow-staging-x
> CreationTimestamp: Sun, 02 Dec 2018 01:33:10 +1100
> Labels: app=strimzi
> Annotations: 
> Selector: name=strimzi-cluster-operator
> *Replicas: 1 desired | 0 updated | 0 total | 0 available | 0 unavailable*
> StrategyType: Recreate
> MinReadySeconds: 0
> Pod Template:
>   Labels: name=strimzi-cluster-operator
>   Service Account: strimzi-cluster-operator
>   Containers:
>strimzi-cluster-operator:
> Image: strimzi/cluster-operator:0.6.0
> Port: 
> Limits:
>   cpu: 1
>   memory: 256Mi
> Requests:
>   cpu: 200m
>   memory: 256Mi
> Liveness: http-get http://:8080/healthy delay=10s timeout=1s
> period=30s #success=1 #failure=3
> Readiness: http-get http://:8080/ready delay=10s timeout=1s
> period=30s #success=1 #failure=3
> Environment: ...
> Mounts: 
>   Volumes: 
> OldReplicaSets: 
> NewReplicaSet: 
> Events: 
>
> All nodes are ready, schedulable and there doesnt appear to be antyhing in
> the logs.
>
> I've tried restarting origin-node, origin-master-api and
> origin-master-controllers on all nodes and that has no impact.
>
> I'm out of ideas.
>
> Cameron
>
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Origin 3.6 cluster no longer deploying deployemnts

2018-12-01 Thread Cameron Braid 
I have a strange issue in my 3.6 cluster.  I create a
extensions/v1beta1 Deployment
and nothing happens.  No pods are created.  oc describe shows

 oc -n drivenow-staging-x describe deployment strimzi-cluster-operator
Name: strimzi-cluster-operator
Namespace: drivenow-staging-x
CreationTimestamp: Sun, 02 Dec 2018 01:33:10 +1100
Labels: app=strimzi
Annotations: 
Selector: name=strimzi-cluster-operator
*Replicas: 1 desired | 0 updated | 0 total | 0 available | 0 unavailable*
StrategyType: Recreate
MinReadySeconds: 0
Pod Template:
  Labels: name=strimzi-cluster-operator
  Service Account: strimzi-cluster-operator
  Containers:
   strimzi-cluster-operator:
Image: strimzi/cluster-operator:0.6.0
Port: 
Limits:
  cpu: 1
  memory: 256Mi
Requests:
  cpu: 200m
  memory: 256Mi
Liveness: http-get http://:8080/healthy delay=10s timeout=1s period=30s
#success=1 #failure=3
Readiness: http-get http://:8080/ready delay=10s timeout=1s period=30s
#success=1 #failure=3
Environment: ...
Mounts: 
  Volumes: 
OldReplicaSets: 
NewReplicaSet: 
Events: 

All nodes are ready, schedulable and there doesnt appear to be antyhing in
the logs.

I've tried restarting origin-node, origin-master-api and
origin-master-controllers on all nodes and that has no impact.

I'm out of ideas.

Cameron
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users