Re: The cluster-logging pods (Elasticsearch, Kibana, Fluentd) don't start - Openshift 4.1

2019-11-06 Thread Rich Megginson
e the minKubeVersion there, but if that doesn't trigger a redeployment, I'm not sure how to do that. Thank you. -Original Message- From: "Rich Megginson" [rmegg...@redhat.com] Date: 11/06/2019 01:21 PM To: users@lists.openshift.redhat.com Subject: Re: The cluster-logging pods (Elastic

Re: The cluster-logging pods (Elasticsearch, Kibana, Fluentd) don't start - Openshift 4.1

2019-11-06 Thread Rich Megginson
are you running into https://bugzilla.redhat.com/show_bug.cgi?id=1766343 ? On 11/6/19 9:19 AM, Full Name wrote: Hi all, I'm trying to deploy logging on Openshift cluster 4.1.21 using the procedure described in the following link

Re: Elasticsearch Restart taking too long

2019-07-01 Thread Rich Megginson
What version of OKD are you using?  Did you check the Elasticsearch elastic.co docs? On 6/26/19 3:54 PM, bahhooo wrote: Hi all, When the elasticsearch pod within the logging stack is restarted, it takes a considerable amount  of time until the elastic cluster becomes green. The cluster

Re: Redirect logs of a namespace to ops logging

2019-04-29 Thread Rich Megginson
You might try setting the env var OCP_OPERATIONS_PROJECTS in the fluentd daemonset: oc set env OCP_OPERATIONS_PROJECTS="default openshift openshift- kube-" https://github.com/openshift/origin-aggregated-logging/blob/release-3.10/fluentd/run.sh On 4/29/19 7:48 AM, bahhooo wrote: Hi all, In a

Re: Regarding Logging

2018-11-21 Thread Rich Megginson
/* for evidence that fluentd is up and doing something. Thanks kasturi On Mon, Nov 19, 2018 at 9:21 PM Rich Megginson mailto:rmegg...@redhat.com>> wrote: Try unlabeling then relabeling the nodes: oc label node --all logging-infra-fluentd- wait a minute oc label node --all logging

Re: Regarding Logging

2018-11-19 Thread Rich Megginson
Try unlabeling then relabeling the nodes: oc label node --all logging-infra-fluentd- wait a minute oc label node --all logging-infra-fluentd=true On 11/19/18 8:44 AM, Kasturi Narra wrote: Hello,   Please find replies line On Mon, Nov 19, 2018 at 9:12 PM Rich Megginson mailto:rmegg

Re: Regarding Logging

2018-11-19 Thread Rich Megginson
On 11/19/18 8:32 AM, Kasturi Narra wrote: Hello Jeff,    yes , i do have it. Here is the output i have got. dhcp46-68.lab.eng.blr.redhat.com    Ready        6d        v1.9.1+a0ce1bc657

Re: [CentOS-devel] [CentOS PaaS SIG]: Origin v3.11 rpms available for testing

2018-10-18 Thread Rich Megginson
On 10/17/18 3:38 AM, Daniel Comnea wrote: Hi, We would like to announce that OKD v3.11 rpms are available for testing at [1]. As such we are calling for help from community to start testing and let us know if there are issues with the rpms and its dependencies. And in the spirit of

Re: Node names as IPs not hostnames

2018-10-09 Thread Rich Megginson
Are you hitting https://github.com/openshift/openshift-ansible/pull/9598 ? On 10/9/18 11:25 AM, Dan Pungă wrote: Thanks for the reply Scott! I've used the release branches for both 3.9 and 3.10 of the openshift-ansible project, yes. I've initially checked the openshift_facts.py script flow in

Re: Best file system for elasticsearch

2018-10-05 Thread Rich Megginson
The best filesystem for Elasticsearch is local disk.  Our performance team has tested on Gluster block storage and it does work, and it might perform well enough for your particular use case, but you'll have to do some testing to find out if it is suitable for your workload. On 10/5/18 3:06 AM,

Re: Logging / Kibana export Logs

2018-08-16 Thread Rich Megginson
You could expose Elasticsearch externally https://docs.okd.io/latest/install_config/aggregate_logging.html |openshift_logging_es_allow_external| Set to |true| to expose Elasticsearch as a reencrypt route. Set to |false| by default. Except that username/password and token auth is

Re: Openshift centralized logging - add custom container logfiles

2018-08-16 Thread Rich Megginson
On 08/16/2018 05:42 AM, Aleksandar Lazic wrote: Am 16.08.2018 um 12:48 schrieb Aleksandar Kostadinov: Might be real nice to allow pod to request sockets created where different log streams can be sent to central logging without extra containers in the pod. You can run socklog/fluentbit/... in

Re: [logging]

2018-06-01 Thread Rich Megginson
Not sure how logstash chooses which cert/key to use from the truststore.  You might ask on a logstash forum. Or, just use the fluentd cert/key with plain old client cert and key files in pem format, if logstash supports that.  You can dump the fluentd ca, cert, and key using oc extract -n

Re: [logging]

2018-05-21 Thread Rich Megginson
On 05/20/2018 12:31 PM, Himmat Singh wrote: Hi Team, I am using openshift logging image  with below version for provides us centralize logging capability for our openshift cluster and external environment logs. |registry.access.redhat.com/openshift3/logging-fluentd:v3.9

Re: [CentOS-devel] CentOS Origin packages in testing for 3.7, 3.8 and 3.9

2018-04-13 Thread Rich Megginson
On 04/13/2018 09:58 AM, Troy Dawson wrote: On Fri, Apr 13, 2018 at 8:42 AM, Rich Megginson <rmegg...@redhat.com> wrote: On 04/13/2018 09:39 AM, Troy Dawson wrote: On Fri, Apr 13, 2018 at 7:58 AM, Rich Megginson <rmegg...@redhat.com> wrote: On 04/12/2018 07:32 AM, Troy Dawson wro

Re: [CentOS-devel] CentOS Origin packages in testing for 3.7, 3.8 and 3.9

2018-04-13 Thread Rich Megginson
On 04/13/2018 09:39 AM, Troy Dawson wrote: On Fri, Apr 13, 2018 at 7:58 AM, Rich Megginson <rmegg...@redhat.com> wrote: On 04/12/2018 07:32 AM, Troy Dawson wrote: We have origin packages for 3.7.2, 3.8.0 and 3.9.0. We also have the corresponding openshift-ansible packages. They have be

Re: [CentOS-devel] CentOS Origin packages in testing for 3.7, 3.8 and 3.9

2018-04-13 Thread Rich Megginson
On 04/12/2018 07:32 AM, Troy Dawson wrote: We have origin packages for 3.7.2, 3.8.0 and 3.9.0. We also have the corresponding openshift-ansible packages. They have been put in our testing repos. DO NOT USE ORIGIN 3.8.0, IT IS FOR UPGRADE PURPOSES ONLY These will not be released until

Re: Limting disk usage for container logs

2018-02-19 Thread Rich Megginson
On 02/19/2018 07:01 AM, bahhooo wrote: Hi, is this even possible? It would be nice to be able specify disk quotas for projects. Currently one single project can go wild and fill up the whole elasticsearch disk, rendering logging out of operation. Not exactly.  There are a couple of things

Re: Logging seems to be working, but no logs are collected

2017-11-02 Thread Rich Megginson
riko Hosoi wrote: On 11/01/2017 12:56 PM, Rich Megginson wrote: On 11/01/2017 01:18 PM, Tim Dudgeon wrote: More data on this. Just to confirm that the journal on the node is receiving events: sudo journalctl -n 25 -- Logs begin at Wed 2017-11-01 14:24:08 UTC, end at Wed 2017-11-01 19:15:15 UTC.

Re: Logging seems to be working, but no logs are collected

2017-11-01 Thread Rich Megginson
On 11/01/2017 01:56 PM, Rich Megginson wrote: On 11/01/2017 01:18 PM, Tim Dudgeon wrote: More data on this. Just to confirm that the journal on the node is receiving events: sudo journalctl -n 25 -- Logs begin at Wed 2017-11-01 14:24:08 UTC, end at Wed 2017-11-01 19:15:15 UTC. -- Nov 01 19:14

Re: Logging seems to be working, but no logs are collected

2017-11-01 Thread Rich Megginson
em" 2017-11-01 17:14:01 + [warn]: no patterns matched tag="journal.system" 2017-11-01 17:18:06 + [warn]: no patterns matched tag="journal.system" 2017-11-01 17:22:09 + [warn]: no patterns matched tag="journal.system" This is really a basic centos7 im

Re: confusion over storage for logging

2017-11-01 Thread Rich Megginson
o the container that will go away when the pod/container goes away" i.e. not persistent.  You can use local disk storage to achieve persistence without using NFS. On 01/11/2017 17:21, Rich Megginson wrote: On 11/01/2017 10:50 AM, Tim Dudgeon wrote: I am confused over persistent

Re: Logging seems to be working, but no logs are collected

2017-10-31 Thread Rich Megginson
3;i=a54f2;b=93b6daa912044dd9ae9f05521c603efc;m=55116ad995;t=55cdb72d7c92d;x=5a16032caedc4423 On 31/10/2017 17:31, Rich Megginson wrote: # docker info | grep -i log # journalctl -r -n 1 --show-cursor On 10/31/2017 11:12 AM, Tim Dudgeon wrote: Thanks. Those links are useful.

Re: Logging seems to be working, but no logs are collected

2017-10-31 Thread Rich Megginson
# docker info | grep -i log # journalctl -r -n 1 --show-cursor On 10/31/2017 11:12 AM, Tim Dudgeon wrote: Thanks. Those links are useful. It looks to me like its a problem at the fluentd level. This is what I see on on of the fluentd pods: sh-4.2# cat /var/log/es-containers.log.pos cat:

Re: split long log records

2017-06-20 Thread Rich Megginson
On 06/19/2017 10:48 AM, Peter Portante wrote: Hi Andre, This is a hard-coded Docker size. For background see: * https://bugzilla.redhat.com/show_bug.cgi?id=1422008, "[RFE] Fluentd handling of long log lines (> 16KB) split by Docker and indexed into several ES documents" * And the reason

Re: Logging (aggregated, FluentD)

2017-04-19 Thread Rich Megginson
vironment if that would help. Sure. At this point I have no idea what's wrong. On Tue, Apr 18, 2017 at 4:47 PM, Rich Megginson <rmegg...@redhat.com <mailto:rmegg...@redhat.com>> wrote: On 04/18/2017 01:51 PM, Shepp wrote: Hello, I've posted over on the Fl

Re: Logging (aggregated, FluentD)

2017-04-18 Thread Rich Megginson
On 04/18/2017 01:51 PM, Shepp wrote: Hello, I've posted over on the FluentD Google Groups but was directed here. https://groups.google.com/forum/#!topic/fluentd/Uo2E6kQzM5E I've got an OpenShift test lab in AWS, all the

Re: cluster-reader and secrets

2017-01-11 Thread Rich Megginson
On 01/11/2017 08:51 PM, Clayton Coleman wrote: On Wed, Jan 11, 2017 at 4:21 PM, Matt Wringe > wrote: - Original Message - > From: "Clayton Coleman" > > To: "Matt Wringe"

Re: what is meaning of openshift_hosted_logging_hostname

2016-12-15 Thread Rich Megginson
t_hosted_logging_hostname? Not sure. You could try it. *Van:* users-boun...@lists.openshift.redhat.com <users-boun...@lists.openshift.redhat.com> namens Rich Megginson <rmegg...@redhat.com> *Verzonden:* donderda

Re: what is meaning of openshift_hosted_logging_hostname

2016-12-15 Thread Rich Megginson
On 12/15/2016 01:59 PM, Den Cowboy wrote: Hi, I saw this option in the ansible playbook example: https://github.com/openshift/openshift-ansible/blob/master/inventory/byo/hosts.ose.example 1) What is the meaning of this viarable: openshift_hosted_logging_hostname? This is the external

Re: Ansible installation OpenShift origin 1.2.0 failed

2016-11-17 Thread Rich Megginson
On 11/17/2016 10:36 AM, Den Cowboy wrote: Thanks. This could probably be the issue. # yum -y --enablerepo=epel --showduplicates list ansible Failed to set locale, defaulting to C Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirror2.hs-esslingen.de * epel:

Re: Origin Aggregated Logging of default namespace

2016-11-16 Thread Rich Megginson
On 11/15/2016 08:40 PM, Frank Liauw wrote: I have origin aggregated logging enabled on my openshift cluster. However, pods from the default namespaces are not reflected in kibana, and there's no corresponding index for all out-of-the-box namespaces (default, management-infra, openshift,

Re: Managing OpenShift Configuration with Puppet/Ansible… what are your best practices?

2016-10-13 Thread Rich Megginson
On 10/13/2016 07:52 AM, Philippe Lafoucrière wrote: Just to clarify our need here: We want the projects config inside a configuration tool. There's currently nothing preventing from modifying the config of a project (let's say, a DC), and no one will be notified of the change. Do you mean,

Re: Managing OpenShift Configuration with Puppet/Ansible… what are your best practices?

2016-10-12 Thread Rich Megginson
On 10/12/2016 03:15 AM, Stéphane Klein wrote: Hi, I use OpenShift with success but I search a good practice to: * keep all object config history (deployment config history, image stream creation/delete, project creation/delete…) * I use OpenShift template system, sometime I need to update this

Re: clean up elastic search logging project

2016-10-10 Thread Rich Megginson
On 10/10/2016 01:26 AM, Den Cowboy wrote: Hi, We have implemented our logging project:https://docs.openshift.org/latest/install_config/aggregate_logging.html We are using ephemeral storage for ou ES containers. But now is our question how we can set a limit or something: clear db after

Re: logs aggregation

2016-06-22 Thread Rich Megginson
On 06/22/2016 07:05 AM, Philippe Lafoucrière wrote: Hi, Something would be very useful in the console: logs from all replicas aggregated in a single page. This is particularly useful when several web servers are serving the same site, and we need to debug something (ala docker-compose). We

Re: Feedback

2016-06-16 Thread Rich Megginson
On 06/16/2016 02:01 PM, Srinivas Naga Kotaru (skotaru) wrote: Just to share an update Am able to successfully install and configure metrics and logging using real certs, node classification and separate apps and OPS ES clusters. Am happy that this setup finally working and live. Want to

Re: logging-fluentd-template: use own image

2016-06-02 Thread Rich Megginson
On 06/02/2016 06:07 AM, Lorenz Vanthillo wrote: I'm busy with setting up logging on OpenShift Origin 1.1.6. At the moment I have the base setup with the help of templates and the documentation about aggregating logging. This was all fine. But now have edited our fluentd image. Now we try to

Re: CentOS OpsTools (logging, monitoring, etc.) SIG proposal

2016-05-20 Thread Rich Megginson
On 05/20/2016 11:29 AM, Jason DeTiberus wrote: On Fri, May 20, 2016 at 1:22 PM, Rich Megginson <rmegg...@redhat.com <mailto:rmegg...@redhat.com>> wrote: On 05/20/2016 10:54 AM, Jason DeTiberus wrote: On Fri, May 20, 2016 at 10:40 AM, Rich Megginson <rmeg

Re: CentOS OpsTools (logging, monitoring, etc.) SIG proposal

2016-05-20 Thread Rich Megginson
On 05/20/2016 10:54 AM, Jason DeTiberus wrote: On Fri, May 20, 2016 at 10:40 AM, Rich Megginson <rmegg...@redhat.com <mailto:rmegg...@redhat.com>> wrote: We are trying to start up a CentOS OpsTools SIG https://wiki.centos.org/SpecialInterestGroup for logging, mon