Hi All, I followed through the instructions on https://docs.openshift.org/latest/install_config/aggregate_logging.html and have setup a 3 node ES cluster. Fluentd is also deployed on all my nodes.
I am getting kibana logs on the logging project, but all my other projects do not have any logs; kibana shows "No results found", with occasional errors reading "Discover: An error occurred with your request. Reset your inputs and try again." Probing the requests made by kibana, some calls to /elasticsearch/_msearch?timeout=0&ignore_unavailable=true&preference=1471245075265 are failing from time to time. Looking into the ES logs for all 3 cluster pods, I don't see much errors to be concerned, with the last error of 2 nodes similar to the following which seems to be a known issue with Openshift's setup ( https://lists.openshift.redhat.com/openshift-archives/users/2015-December/msg00078.html) and possibly explains the failed requests made by kibana on auto-refresh, but that's a problem for another day: [2016-08-15 06:53:49,130][INFO ][cluster.service ] [Gremlin] added {[Quicksilver][t2l6Oz8uT-WS8Fa7S7jzfQ][logging-es-d7r1t3dm-2-a0cf0][inet[/10.1.3.3:9300]],}, reason: zen-disco-receive(from master [[One Above All][CyFgyTTtS_S85yYRom2wVQ][logging-es-0w45va6n-2-8m85p][inet[/10.1.2.5:9300 ]]]) [2016-08-15 06:59:27,727][ERROR][com.floragunn.searchguard.filter.SearchGuardActionFilter] Error while apply() due to com.floragunn.searchguard.tokeneval.MalformedConfigurationException: no bypass or execute filters at all for action indices:admin/mappings/fields/get com.floragunn.searchguard.tokeneval.MalformedConfigurationException: no bypass or execute filters at all Looking into fluentd logs, one of my nodes is complaining of a "getaddrinfo" error: 2016-08-15 03:45:18 -0400 [error]: unexpected error error="getaddrinfo: Name or service not known" 2016-08-15 03:45:18 -0400 [error]: /usr/share/ruby/net/http.rb:878:in `initialize' 2016-08-15 03:45:18 -0400 [error]: /usr/share/ruby/net/http.rb:878:in `open' 2016-08-15 03:45:18 -0400 [error]: /usr/share/ruby/net/http.rb:878:in `block in connect' 2016-08-15 03:45:18 -0400 [error]: /usr/share/ruby/timeout.rb:52:in `timeout' 2016-08-15 03:45:18 -0400 [error]: /usr/share/ruby/net/http.rb:877:in `connect' 2016-08-15 03:45:18 -0400 [error]: /usr/share/ruby/net/http.rb:862:in `do_start' 2016-08-15 03:45:18 -0400 [error]: /usr/share/ruby/net/http.rb:851:in `start' 2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/rest-client-2.0.0/lib/restclient/request.rb:766:in `transmit' 2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/rest-client-2.0.0/lib/restclient/request.rb:215:in `execute' 2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/rest-client-2.0.0/lib/restclient/request.rb:52:in `execute' 2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/rest-client-2.0.0/lib/restclient/resource.rb:51:in `get' 2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/kubeclient-1.1.4/lib/kubeclient/common.rb:328:in `block in api' 2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/kubeclient-1.1.4/lib/kubeclient/common.rb:58:in `handle_exception' 2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/kubeclient-1.1.4/lib/kubeclient/common.rb:327:in `api' 2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/kubeclient-1.1.4/lib/kubeclient/common.rb:322:in `api_valid?' 2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluent-plugin-kubernetes_metadata_filter-0.24.0/lib/fluent/plugin/filter_kubernetes_metadata.rb:167:in `configure' 2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/agent.rb:144:in `add_filter' 2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/agent.rb:61:in `block in configure' 2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/agent.rb:57:in `each' 2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/agent.rb:57:in `configure' 2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/root_agent.rb:83:in `block in configure' 2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/root_agent.rb:83:in `each' 2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/root_agent.rb:83:in `configure' 2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/engine.rb:129:in `configure' 2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/engine.rb:103:in `run_configure' 2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:483:in `run_configure' 2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:154:in `block in start' 2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:360:in `call' 2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:360:in `main_process' 2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:333:in `block in supervise' 2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:332:in `fork' 2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:332:in `supervise' 2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:150:in `start' 2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/command/fluentd.rb:173:in `<top (required)>' 2016-08-15 03:45:18 -0400 [error]: /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require' 2016-08-15 03:45:18 -0400 [error]: /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require' 2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/bin/fluentd:5:in `<top (required)>' 2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/bin/fluentd:23:in `load' 2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/bin/fluentd:23:in `<main>' 2016-08-15 03:45:18 -0400 [error]: fluentd main process died unexpectedly. restarting. Other fluentd pods are fine: 2016-08-15 02:55:58 -0400 [info]: reading config file path="/etc/fluent/fluent.conf" However, I'm not getting any logs on pods running on nodes that have fluentd deployed properly. Any thoughts will be much appreciated :) Frank
_______________________________________________ users mailing list [email protected] http://lists.openshift.redhat.com/openshiftmm/listinfo/users
