Re: logging-es errors: shards failed

2016-07-15 Thread Eric Wolinetz
The logging-ops instance will contain the logs from /var/log/messages* and the "default", "openshift" and "openshift-infra" name spaces only. On Fri, Jul 15, 2016 at 3:28 PM, Alex Wauck wrote: > I also tried to fetch the logs from our logging-ops ES instance. That > also

Re: logging-es errors: shards failed

2016-07-15 Thread Alex Wauck
I also tried to fetch the logs from our logging-ops ES instance. That also met with failure. Searching for "kubernetes_namespace_name: logging" there lead to "No results found". On Fri, Jul 15, 2016 at 2:48 PM, Peter Portante wrote: > Well, we don't send ES logs to

Re: logging-es errors: shards failed

2016-07-15 Thread Luke Meyer
They surely do. Although it would probably be easiest here to just get them from `oc logs` against the ES pod, especially if we can't trust ES storage. On Fri, Jul 15, 2016 at 3:26 PM, Peter Portante wrote: > Eric, Luke, > > Do the logs from the ES instance itself flow into

Re: logging-es errors: shards failed

2016-07-15 Thread Peter Portante
Eric, Luke, Do the logs from the ES instance itself flow into that ES instance? -peter On Fri, Jul 15, 2016 at 12:14 PM, Alex Wauck wrote: > I'm not sure that I can. I clicked the "Archive" link for the logging-es > pod and then changed the query in Kibana to

Re: Atomic Centos, can't upgrade

2016-07-15 Thread Colin Walters
On Mon, Jul 11, 2016, at 09:56 AM, Scott Dodson wrote: > That commit is mostly related to the fact that we cannot > upgrade/downgrade docker on atomic host like can on RHEL so abort the > docker upgrade playbook early. For short term fixes, it is however possible to use `atomic host deploy` to

Re: logging-es errors: shards failed

2016-07-15 Thread Alex Wauck
I'm not sure that I can. I clicked the "Archive" link for the logging-es pod and then changed the query in Kibana to "kubernetes_container_name: logging-es-cycd8veb && kubernetes_namespace_name: logging". I got no results, instead getting this error: - *Index:*

Re: Atomic Centos, can't upgrade

2016-07-15 Thread Philippe Lafoucrière
https://docs.openshift.org/latest/dev_guide/shared_memory.html fixed the issue, but It seems something changed regarding /dev or shm docker mounts between 1.2.0 and 1.2.1. Can someone confirm? ​ ___ users mailing list users@lists.openshift.redhat.com

Re: logging-es errors: shards failed

2016-07-15 Thread Peter Portante
Can you go back further in the logs to the point where the errors started? I am thinking about possible Java HEAP issues, or possibly ES restarting for some reason. -peter On Fri, Jul 15, 2016 at 11:37 AM, Lukáš Vlček wrote: > Also looking at this. > Alex, is it possible to

Re: logging-es errors: shards failed

2016-07-15 Thread Eric Wolinetz
Adding Lukas and Peter On Fri, Jul 15, 2016 at 8:07 AM, Luke Meyer wrote: > I believe the "queue capacity" there is the number of parallel searches > that can be queued while the existing search workers operate. It sounds > like it has plenty of capacity there and it has a

Re: Atomic Centos, can't upgrade

2016-07-15 Thread Philippe Lafoucrière
I confirm: it's fixed :)​ thanks! ___ users mailing list users@lists.openshift.redhat.com http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: logging-es errors: shards failed

2016-07-15 Thread Luke Meyer
I believe the "queue capacity" there is the number of parallel searches that can be queued while the existing search workers operate. It sounds like it has plenty of capacity there and it has a different reason for rejecting the query. I would guess the data requested is missing given it couldn't