date:20181119

OKD 3.9 to 3.10 upgrade failure on CentOS

2018-11-19 Thread Dharmit Shah

Hi,

I'm trying to upgrade my OKD 3.9 cluster to 3.10 using
openshift-ansible. I have already described the problem in detail and
provided logs on the GitHub issue [1].

I could really use some help on this issue!

Regards,
Dharmit

[1] https://github.com/openshift/openshift-ansible/issues/10690

-- 
Dharmit Shah
Red Hat Developer Tools (https://developers.redhat.com/)
irc, mattermost: dharmit
https://dharmitshah.com

___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Regarding Logging

2018-11-19 Thread Kasturi Narra

Hello Everyone,

   I have a setup where i am trying to install logging using ocp3.9+
cns3.11 . I see that logging pods are up and running but when i access the
webconsole i get an error  present at [1]  and i tried the solution
provided at [2] but having no luck. Can some one of you please help me on
resolving this issue ?

[root@dhcp46-170 ~]# oc version
oc v3.9.43
kubernetes v1.9.1+a0ce1bc657
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://dhcp46-170.lab.eng.blr.redhat.com:8443
openshift v3.9.43
kubernetes v1.9.1+a0ce1bc657

[root@dhcp46-170 ~]# oc get pods
NAME  READY STATUSRESTARTS   AGE
logging-curator-1-bgjbj   1/1   Running   0  2h
logging-es-data-master-5gjnm57x-2-5vjq6   2/2   Running   0  2h
logging-kibana-1-872dn2/2   Running   0  2h

[1] Discover: [exception] The index returned an empty result. You can use
the Time Picker to change the time filter or select a higher time interval
[2] https://access.redhat.com/solutions/3352681

Thanks
kasturi
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Regarding Logging

2018-11-19 Thread Jeff Cantrill

It doesn't appear you have any fluentd pods which are responsible for
collecting logs from the other pods.  Are your nodes labeled with
'logging-infra-fluend=true'

On Mon, Nov 19, 2018 at 7:28 AM Kasturi Narra  wrote:

> Hello Everyone,
>
>I have a setup where i am trying to install logging using ocp3.9+
> cns3.11 . I see that logging pods are up and running but when i access the
> webconsole i get an error  present at [1]  and i tried the solution
> provided at [2] but having no luck. Can some one of you please help me on
> resolving this issue ?
>
> [root@dhcp46-170 ~]# oc version
> oc v3.9.43
> kubernetes v1.9.1+a0ce1bc657
> features: Basic-Auth GSSAPI Kerberos SPNEGO
>
> Server https://dhcp46-170.lab.eng.blr.redhat.com:8443
> openshift v3.9.43
> kubernetes v1.9.1+a0ce1bc657
>
> [root@dhcp46-170 ~]# oc get pods
> NAME  READY STATUSRESTARTS
> AGE
> logging-curator-1-bgjbj   1/1   Running   0  2h
> logging-es-data-master-5gjnm57x-2-5vjq6   2/2   Running   0  2h
> logging-kibana-1-872dn2/2   Running   0  2h
>
> [1] Discover: [exception] The index returned an empty result. You can use
> the Time Picker to change the time filter or select a higher time interval
> [2] https://access.redhat.com/solutions/3352681
>
> Thanks
> kasturi
> ___
> users mailing list
> users@lists.openshift.redhat.com
> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>


-- 
--
Jeff Cantrill
Senior Software Engineer, Red Hat Engineering
OpenShift Logging
Red Hat, Inc.
*Office*: 703-748-4420 | 866-546-8970 ext. 8162420
jcant...@redhat.com
http://www.redhat.com
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Regarding Logging

2018-11-19 Thread Kasturi Narra

Hello Jeff,

   yes , i do have it. Here is the output i have got.

dhcp46-68.lab.eng.blr.redhat.comReady 6d
 v1.9.1+a0ce1bc657
beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=dhcp46-68.lab.eng.blr.redhat.com,logging-infra-fluentd=true,region=infra,registry=enabled,role=node,router=enabled

Thanks
kasturi

On Mon, Nov 19, 2018 at 7:16 PM Jeff Cantrill  wrote:

> It doesn't appear you have any fluentd pods which are responsible for
> collecting logs from the other pods.  Are your nodes labeled with
> 'logging-infra-fluend=true'
>
> On Mon, Nov 19, 2018 at 7:28 AM Kasturi Narra  wrote:
>
>> Hello Everyone,
>>
>>I have a setup where i am trying to install logging using ocp3.9+
>> cns3.11 . I see that logging pods are up and running but when i access the
>> webconsole i get an error  present at [1]  and i tried the solution
>> provided at [2] but having no luck. Can some one of you please help me on
>> resolving this issue ?
>>
>> [root@dhcp46-170 ~]# oc version
>> oc v3.9.43
>> kubernetes v1.9.1+a0ce1bc657
>> features: Basic-Auth GSSAPI Kerberos SPNEGO
>>
>> Server https://dhcp46-170.lab.eng.blr.redhat.com:8443
>> openshift v3.9.43
>> kubernetes v1.9.1+a0ce1bc657
>>
>> [root@dhcp46-170 ~]# oc get pods
>> NAME  READY STATUSRESTARTS
>> AGE
>> logging-curator-1-bgjbj   1/1   Running   0
>>  2h
>> logging-es-data-master-5gjnm57x-2-5vjq6   2/2   Running   0
>>  2h
>> logging-kibana-1-872dn2/2   Running   0
>>  2h
>>
>> [1] Discover: [exception] The index returned an empty result. You can
>> use the Time Picker to change the time filter or select a higher time
>> interval
>> [2] https://access.redhat.com/solutions/3352681
>>
>> Thanks
>> kasturi
>> ___
>> users mailing list
>> users@lists.openshift.redhat.com
>> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>>
>
>
> --
> --
> Jeff Cantrill
> Senior Software Engineer, Red Hat Engineering
> OpenShift Logging
> Red Hat, Inc.
> *Office*: 703-748-4420 | 866-546-8970 ext. 8162420
> jcant...@redhat.com
> http://www.redhat.com
>
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Regarding Logging

2018-11-19 Thread Rich Megginson


On 11/19/18 8:32 AM, Kasturi Narra wrote:

Hello Jeff,
   yes , i do have it. Here is the output i have got.

dhcp46-68.lab.eng.blr.redhat.com     Ready         6d        v1.9.1+a0ce1bc657 
beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=dhcp46-68.lab.eng.blr.redhat.com,logging-infra-fluentd=true,region=infra,registry=enabled,role=node,router=enabled 





oc get daemonset

oc describe daemonset logging-fluentd



Thanks
kasturi

On Mon, Nov 19, 2018 at 7:16 PM Jeff Cantrill mailto:jcant...@redhat.com>> wrote:

It doesn't appear you have any fluentd pods which are responsible for 
collecting logs from the other pods.  Are your nodes labeled with 
'logging-infra-fluend=true'

On Mon, Nov 19, 2018 at 7:28 AM Kasturi Narra mailto:kna...@redhat.com>> wrote:

Hello Everyone,

   I have a setup where i am trying to install logging using ocp3.9+ 
cns3.11 . I see that logging pods are up and running but when i access the 
webconsole i get an error  present at
[1]  and i tried the solution provided at [2] but having no luck. Can 
some one of you please help me on resolving this issue ?

[root@dhcp46-170 ~]# oc version
oc v3.9.43
kubernetes v1.9.1+a0ce1bc657
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://dhcp46-170.lab.eng.blr.redhat.com:8443
openshift v3.9.43
kubernetes v1.9.1+a0ce1bc657

[root@dhcp46-170 ~]# oc get pods
NAME  READY     STATUS    RESTARTS   AGE
logging-curator-1-bgjbj 1/1       Running   0          2h
logging-es-data-master-5gjnm57x-2-5vjq6 2/2       Running   0          
2h
logging-kibana-1-872dn  2/2       Running   0          2h

[1] Discover: [exception] The index returned an empty result. You can 
use the Time Picker to change the time filter or select a higher time interval
[2] https://access.redhat.com/solutions/3352681

Thanks
kasturi
___
users mailing list
users@lists.openshift.redhat.com 

http://lists.openshift.redhat.com/openshiftmm/listinfo/users



-- 
--

Jeff Cantrill
Senior Software Engineer, Red Hat Engineering
OpenShift Logging
Red Hat, Inc.
*Office*: 703-748-4420 | 866-546-8970 ext. 8162420
jcant...@redhat.com 
http://www.redhat.com


___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users



___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Regarding Logging

2018-11-19 Thread Kasturi Narra

Hello,

  Please find replies line

On Mon, Nov 19, 2018 at 9:12 PM Rich Megginson  wrote:

> On 11/19/18 8:32 AM, Kasturi Narra wrote:
> > Hello Jeff,
> >yes , i do have it. Here is the output i have got.
> >
> > dhcp46-68.lab.eng.blr.redhat.com <
> http://dhcp46-68.lab.eng.blr.redhat.com>Ready 6d
>  v1.9.1+a0ce1bc657
> >
> beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=dhcp46-68.lab.eng.blr.redhat.com,logging-infra-fluentd=true,region=infra,registry=enabled,role=node,router=enabled
> > <
> http://beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=dhcp46-68.lab.eng.blr.redhat.com,logging-infra-fluentd=true,region=infra,registry=enabled,role=node,router=enabled
> >
> >
>
> oc get daemonset
>

[root@dhcp46-170 ~]# oc get daemonset
NAME  DESIRED   CURRENT   READY UP-TO-DATE   AVAILABLE
NODE SELECTORAGE
logging-fluentd   0 0 0 00
logging-infra-fluentd=true   3m


>
> oc describe daemonset logging-fluentd
>

[root@dhcp46-170 ~]# oc describe daemonset logging-fluentd
Name:   logging-fluentd
Selector:   component=fluentd,provider=openshift
Node-Selector:  logging-infra-fluentd=true
Labels: component=fluentd
logging-infra=fluentd
provider=openshift
Annotations:
Desired Number of Nodes Scheduled: 0
Current Number of Nodes Scheduled: 0
Number of Nodes Scheduled with Up-to-date Pods: 0
Number of Nodes Scheduled with Available Pods: 0
Number of Nodes Misscheduled: 0
Pods Status:  0 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:   component=fluentd
logging-infra=fluentd
provider=openshift
  Service Account:  aggregated-logging-fluentd
  Containers:
   fluentd-elasticsearch:
Image:  registry.access.redhat.com/openshift3/logging-fluentd:v3.9.43
Port:   
Limits:
  memory:  512Mi
Requests:
  cpu: 100m
  memory:  512Mi
Environment:
  K8S_HOST_URL:https://kubernetes.default.svc.cluster.local
  ES_HOST: logging-es
  ES_PORT: 9200
  ES_CLIENT_CERT:  /etc/fluent/keys/cert
  ES_CLIENT_KEY:   /etc/fluent/keys/key
  ES_CA:   /etc/fluent/keys/ca
  OPS_HOST:logging-es
  OPS_PORT:9200
  OPS_CLIENT_CERT: /etc/fluent/keys/ops-cert
  OPS_CLIENT_KEY:  /etc/fluent/keys/ops-key
  OPS_CA:  /etc/fluent/keys/ops-ca
  JOURNAL_SOURCE:
  JOURNAL_READ_FROM_HEAD:
  BUFFER_QUEUE_LIMIT:  32
  BUFFER_SIZE_LIMIT:   8m
  FLUENTD_CPU_LIMIT:   node allocatable (limits.cpu)
  FLUENTD_MEMORY_LIMIT:536870912 (limits.memory)
  FILE_BUFFER_LIMIT:   256Mi
Mounts:
  /etc/docker from dockerdaemoncfg (ro)
  /etc/docker-hostname from dockerhostname (ro)
  /etc/fluent/configs.d/user from config (ro)
  /etc/fluent/keys from certs (ro)
  /etc/localtime from localtime (ro)
  /etc/origin/node from originnodecfg (ro)
  /etc/sysconfig/docker from dockercfg (ro)
  /run/log/journal from runlogjournal (rw)
  /var/lib/docker/containers from varlibdockercontainers (ro)
  /var/lib/fluentd from filebufferstorage (rw)
  /var/log from varlog (rw)
  Volumes:
   runlogjournal:
Type:  HostPath (bare host directory volume)
Path:  /run/log/journal
HostPathType:
   varlog:
Type:  HostPath (bare host directory volume)
Path:  /var/log
HostPathType:
   varlibdockercontainers:
Type:  HostPath (bare host directory volume)
Path:  /var/lib/docker/containers
HostPathType:
   config:
Type:  ConfigMap (a volume populated by a ConfigMap)
Name:  logging-fluentd
Optional:  false
   certs:
Type:Secret (a volume populated by a Secret)
SecretName:  logging-fluentd
Optional:false
   dockerhostname:
Type:  HostPath (bare host directory volume)
Path:  /etc/hostname
HostPathType:
   localtime:
Type:  HostPath (bare host directory volume)
Path:  /etc/localtime
HostPathType:
   dockercfg:
Type:  HostPath (bare host directory volume)
Path:  /etc/sysconfig/docker
HostPathType:
   originnodecfg:
Type:  HostPath (bare host directory volume)
Path:  /etc/origin/node
HostPathType:
   dockerdaemoncfg:
Type:  HostPath (bare host directory volume)
Path:  /etc/docker
HostPathType:
   filebufferstorage:
Type:  HostPath (bare host directory volume)
Path:  /var/lib/fluentd
HostPathType:
Events:


>
>
> > Thanks
> > kasturi
> >
> > On Mon, Nov 19, 2018 at 7:16 PM Jeff Cantrill  > wrote:
> >
> > It

Re: Regarding Logging

2018-11-19 Thread Rich Megginson


Try unlabeling then relabeling the nodes:

oc label node --all logging-infra-fluentd-

wait a minute

oc label node --all logging-infra-fluentd=true

On 11/19/18 8:44 AM, Kasturi Narra wrote:

Hello,

  Please find replies line

On Mon, Nov 19, 2018 at 9:12 PM Rich Megginson mailto:rmegg...@redhat.com>> wrote:

On 11/19/18 8:32 AM, Kasturi Narra wrote:
> Hello Jeff,
>    yes , i do have it. Here is the output i have got.
>
> dhcp46-68.lab.eng.blr.redhat.com  
    Ready         6d        
v1.9.1+a0ce1bc657
>

beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=dhcp46-68.lab.eng.blr.redhat.com,logging-infra-fluentd=true,region=infra,registry=enabled,role=node,router=enabled



>


>

oc get daemonset


[root@dhcp46-170 ~]# oc get daemonset
NAME              DESIRED   CURRENT   READY UP-TO-DATE   AVAILABLE   NODE 
SELECTOR  AGE
logging-fluentd   0         0         0         0        0           
logging-infra-fluentd=true   3m


oc describe daemonset logging-fluentd


[root@dhcp46-170 ~]# oc describe daemonset logging-fluentd
Name:           logging-fluentd
Selector:       component=fluentd,provider=openshift
Node-Selector:  logging-infra-fluentd=true
Labels:         component=fluentd
                logging-infra=fluentd
                provider=openshift
Annotations:    
Desired Number of Nodes Scheduled: 0
Current Number of Nodes Scheduled: 0
Number of Nodes Scheduled with Up-to-date Pods: 0
Number of Nodes Scheduled with Available Pods: 0
Number of Nodes Misscheduled: 0
Pods Status:  0 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:           component=fluentd
                    logging-infra=fluentd
                    provider=openshift
  Service Account:  aggregated-logging-fluentd
  Containers:
   fluentd-elasticsearch:
    Image: registry.access.redhat.com/openshift3/logging-fluentd:v3.9.43 

    Port:   
    Limits:
      memory:  512Mi
    Requests:
      cpu:     100m
      memory:  512Mi
    Environment:
      K8S_HOST_URL: https://kubernetes.default.svc.cluster.local
      ES_HOST:                 logging-es
      ES_PORT:                 9200
      ES_CLIENT_CERT:          /etc/fluent/keys/cert
      ES_CLIENT_KEY:           /etc/fluent/keys/key
      ES_CA:                   /etc/fluent/keys/ca
      OPS_HOST:                logging-es
      OPS_PORT:                9200
      OPS_CLIENT_CERT: /etc/fluent/keys/ops-cert
      OPS_CLIENT_KEY:  /etc/fluent/keys/ops-key
      OPS_CA:  /etc/fluent/keys/ops-ca
      JOURNAL_SOURCE:
      JOURNAL_READ_FROM_HEAD:
      BUFFER_QUEUE_LIMIT:      32
      BUFFER_SIZE_LIMIT:       8m
      FLUENTD_CPU_LIMIT:       node allocatable (limits.cpu)
      FLUENTD_MEMORY_LIMIT:    536870912 (limits.memory)
      FILE_BUFFER_LIMIT:       256Mi
    Mounts:
      /etc/docker from dockerdaemoncfg (ro)
      /etc/docker-hostname from dockerhostname (ro)
      /etc/fluent/configs.d/user from config (ro)
      /etc/fluent/keys from certs (ro)
      /etc/localtime from localtime (ro)
      /etc/origin/node from originnodecfg (ro)
      /etc/sysconfig/docker from dockercfg (ro)
      /run/log/journal from runlogjournal (rw)
      /var/lib/docker/containers from varlibdockercontainers (ro)
      /var/lib/fluentd from filebufferstorage (rw)
      /var/log from varlog (rw)
  Volumes:
   runlogjournal:
    Type:          HostPath (bare host directory volume)
    Path:          /run/log/journal
    HostPathType:
   varlog:
    Type:          HostPath (bare host directory volume)
    Path:          /var/log
    HostPathType:
   varlibdockercontainers:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/docker/containers
    HostPathType:
   config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      logging-fluentd
    Optional:  false
   certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  logging-fluentd
    Optional:    false
   dockerhostname:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/hostname
    HostPathType:
   localtime:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/localtime
    HostPathType:
   dockercfg:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/sysconfig/docker
    HostPathType:
   originnodecfg:
    Type:

OKD 3.9 to 3.10 upgrade failure on CentOS

Regarding Logging

Re: Regarding Logging

Re: Regarding Logging

Re: Regarding Logging

Re: Regarding Logging

Re: Regarding Logging

7 matches

Site Navigation

Mail list logo

Footer information