Hi Eric,
Thanks for your help. Attached the requested logs.
ES
[2016-04-20
06:57:08,912][ERROR][io.fabric8.elasticsearch.plugin.acl.DynamicACLFilter]
[Baphomet] Exception encountered when seeding initial ACL
org.elasticsearch.cluster.block.ClusterBlockException: blocked by:
[SERVICE_UNAVAILABLE/1/state not recovered / initialized];
at
org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:151)
at
org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction.checkGlobalBlock(TransportShardSingleOperationAction.java:103)
at
org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction.<init>(TransportShardSingleOperationAction.java:132)
at
org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction.<init>(TransportShardSingleOperationAction.java:116)
at
org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction.doExecute(TransportShardSingleOperationAction.java:89)
at
org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction.doExecute(TransportShardSingleOperationAction.java:55)
at
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167)
at
io.fabric8.elasticsearch.plugin.KibanaUserReindexAction.apply(KibanaUserReindexAction.java:74)
at
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at
com.floragunn.searchguard.filter.SearchGuardActionFilter.apply0(SearchGuardActionFilter.java:145)
at
com.floragunn.searchguard.filter.SearchGuardActionFilter.apply(SearchGuardActionFilter.java:90)
at
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at
com.floragunn.searchguard.filter.AbstractActionFilter.apply(AbstractActionFilter.java:105)
at
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at
com.floragunn.searchguard.filter.AbstractActionFilter.apply(AbstractActionFilter.java:105)
at
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at
com.floragunn.searchguard.filter.AbstractActionFilter.apply(AbstractActionFilter.java:105)
at
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at
org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:64)
at
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at
io.fabric8.elasticsearch.plugin.ActionForbiddenActionFilter.apply(ActionForbiddenActionFilter.java:48)
at
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:82)
at
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:55)
at org.elasticsearch.client.node.NodeClient.execute(NodeClient.java:90)
at org.elasticsearch.client.support.AbstractClient.get(AbstractClient.java:188)
at
io.fabric8.elasticsearch.plugin.acl.DynamicACLFilter.loadAcl(DynamicACLFilter.java:245)
at
io.fabric8.elasticsearch.plugin.acl.DynamicACLFilter.seedInitialACL(DynamicACLFilter.java:297)
at
io.fabric8.elasticsearch.plugin.acl.DynamicACLFilter.onSearchGuardACLActionRequest(DynamicACLFilter.java:118)
at
io.fabric8.elasticsearch.plugin.acl.DefaultACLNotifierService$1.run(DefaultACLNotifierService.java:64)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[2016-04-20 06:57:09,683][WARN ][io.fabric8.kubernetes.api.KubernetesHelper]
Lookup logging-es-cluster result: host not found
[2016-04-20
06:57:09,912][ERROR][io.fabric8.elasticsearch.plugin.acl.DynamicACLFilter]
[Baphomet] Error checking ACL when seeding
org.elasticsearch.cluster.block.ClusterBlockException: blocked by:
[SERVICE_UNAVAILABLE/1/state not recovered / initialized];
at
org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:151)
at
org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction.checkGlobalBlock(TransportShardSingleOperationAction.java:103)
at
org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction.<init>(TransportShardSingleOperationAction.java:132)
at
org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction.<init>(TransportShardSingleOperationAction.java:116)
at
org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction.doExecute(TransportShardSingleOperationAction.java:89)
at
org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction.doExecute(TransportShardSingleOperationAction.java:55)
at
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167)
at
io.fabric8.elasticsearch.plugin.KibanaUserReindexAction.apply(KibanaUserReindexAction.java:74)
at
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at
com.floragunn.searchguard.filter.SearchGuardActionFilter.apply0(SearchGuardActionFilter.java:145)
at
com.floragunn.searchguard.filter.SearchGuardActionFilter.apply(SearchGuardActionFilter.java:90)
at
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at
com.floragunn.searchguard.filter.AbstractActionFilter.apply(AbstractActionFilter.java:105)
at
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at
com.floragunn.searchguard.filter.AbstractActionFilter.apply(AbstractActionFilter.java:105)
at
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at
com.floragunn.searchguard.filter.AbstractActionFilter.apply(AbstractActionFilter.java:105)
at
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at
org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:64)
at
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at
io.fabric8.elasticsearch.plugin.ActionForbiddenActionFilter.apply(ActionForbiddenActionFilter.java:48)
at
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:82)
at
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:55)
at org.elasticsearch.client.node.NodeClient.execute(NodeClient.java:90)
at org.elasticsearch.client.support.AbstractClient.get(AbstractClient.java:188)
at
io.fabric8.elasticsearch.plugin.acl.DynamicACLFilter.loadAcl(DynamicACLFilter.java:245)
at
io.fabric8.elasticsearch.plugin.acl.DynamicACLFilter.seedInitialACL(DynamicACLFilter.java:297)
at
io.fabric8.elasticsearch.plugin.acl.DynamicACLFilter.onSearchGuardACLActionRequest(DynamicACLFilter.java:118)
at
io.fabric8.elasticsearch.plugin.acl.DefaultACLNotifierService$1.run(DefaultACLNotifierService.java:64)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[2016-04-20
06:57:09,913][ERROR][io.fabric8.elasticsearch.plugin.acl.DynamicACLFilter]
[Baphomet] Exception encountered when seeding initial ACL
org.elasticsearch.cluster.block.ClusterBlockException: blocked by:
[SERVICE_UNAVAILABLE/1/state not recovered / initialized];
at
org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:151)
at
org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction.checkGlobalBlock(TransportShardSingleOperationAction.java:103)
at
org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction.<init>(TransportShardSingleOperationAction.java:132)
at
org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction.<init>(TransportShardSingleOperationAction.java:116)
at
org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction.doExecute(TransportShardSingleOperationAction.java:89)
at
org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction.doExecute(TransportShardSingleOperationAction.java:55)
at
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167)
at
io.fabric8.elasticsearch.plugin.KibanaUserReindexAction.apply(KibanaUserReindexAction.java:74)
at
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at
com.floragunn.searchguard.filter.SearchGuardActionFilter.apply0(SearchGuardActionFilter.java:145)
at
com.floragunn.searchguard.filter.SearchGuardActionFilter.apply(SearchGuardActionFilter.java:90)
at
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at
com.floragunn.searchguard.filter.AbstractActionFilter.apply(AbstractActionFilter.java:105)
at
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at
com.floragunn.searchguard.filter.AbstractActionFilter.apply(AbstractActionFilter.java:105)
at
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at
com.floragunn.searchguard.filter.AbstractActionFilter.apply(AbstractActionFilter.java:105)
at
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at
org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:64)
at
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at
io.fabric8.elasticsearch.plugin.ActionForbiddenActionFilter.apply(ActionForbiddenActionFilter.java:48)
at
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:82)
at
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:55)
at org.elasticsearch.client.node.NodeClient.execute(NodeClient.java:90)
at org.elasticsearch.client.support.AbstractClient.get(AbstractClient.java:188)
at
io.fabric8.elasticsearch.plugin.acl.DynamicACLFilter.loadAcl(DynamicACLFilter.java:245)
at
io.fabric8.elasticsearch.plugin.acl.DynamicACLFilter.seedInitialACL(DynamicACLFilter.java:297)
at
io.fabric8.elasticsearch.plugin.acl.DynamicACLFilter.onSearchGuardACLActionRequest(DynamicACLFilter.java:118)
at
io.fabric8.elasticsearch.plugin.acl.DefaultACLNotifierService$1.run(DefaultACLNotifierService.java:64)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[2016-04-20
06:57:10,913][ERROR][io.fabric8.elasticsearch.plugin.acl.DynamicACLFilter]
[Baphomet] Error checking ACL when seeding
org.elasticsearch.cluster.block.ClusterBlockException: blocked by:
[SERVICE_UNAVAILABLE/1/state not recovered / initialized];
at
org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:151)
at
org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction.checkGlobalBlock(TransportShardSingleOperationAction.java:103)
at
org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction.<init>(TransportShardSingleOperationAction.java:132)
at
org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction.<init>(TransportShardSingleOperationAction.java:116)
at
org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction.doExecute(TransportShardSingleOperationAction.java:89)
at
org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction.doExecute(TransportShardSingleOperationAction.java:55)
at
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167)
at
io.fabric8.elasticsearch.plugin.KibanaUserReindexAction.apply(KibanaUserReindexAction.java:74)
at
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at
com.floragunn.searchguard.filter.SearchGuardActionFilter.apply0(SearchGuardActionFilter.java:145)
at
com.floragunn.searchguard.filter.SearchGuardActionFilter.apply(SearchGuardActionFilter.java:90)
at
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at
com.floragunn.searchguard.filter.AbstractActionFilter.apply(AbstractActionFilter.java:105)
at
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at
com.floragunn.searchguard.filter.AbstractActionFilter.apply(AbstractActionFilter.java:105)
at
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at
com.floragunn.searchguard.filter.AbstractActionFilter.apply(AbstractActionFilter.java:105)
at
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at
org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:64)
at
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at
io.fabric8.elasticsearch.plugin.ActionForbiddenActionFilter.apply(ActionForbiddenActionFilter.java:48)
at
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:82)
at
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:55)
at org.elasticsearch.client.node.NodeClient.execute(NodeClient.java:90)
at org.elasticsearch.client.support.AbstractClient.get(AbstractClient.java:188)
at
io.fabric8.elasticsearch.plugin.acl.DynamicACLFilter.loadAcl(DynamicACLFilter.java:245)
at
io.fabric8.elasticsearch.plugin.acl.DynamicACLFilter.seedInitialACL(DynamicACLFilter.java:297)
at
io.fabric8.elasticsearch.plugin.acl.DynamicACLFilter.onSearchGuardACLActionRequest(DynamicACLFilter.java:118)
at
io.fabric8.elasticsearch.plugin.acl.DefaultACLNotifierService$1.run(DefaultACLNotifierService.java:64)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[2016-04-20
06:57:10,913][ERROR][io.fabric8.elasticsearch.plugin.acl.DynamicACLFilter]
[Baphomet] Exception encountered when seeding initial ACL
org.elasticsearch.cluster.block.ClusterBlockException: blocked by:
[SERVICE_UNAVAILABLE/1/state not recovered / initialized];
at
org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:151)
at
org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction.checkGlobalBlock(TransportShardSingleOperationAction.java:103)
at
org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction.<init>(TransportShardSingleOperationAction.java:132)
at
org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction.<init>(TransportShardSingleOperationAction.java:116)
at
org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction.doExecute(TransportShardSingleOperationAction.java:89)
at
org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction.doExecute(TransportShardSingleOperationAction.java:55)
at
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167)
at
io.fabric8.elasticsearch.plugin.KibanaUserReindexAction.apply(KibanaUserReindexAction.java:74)
at
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at
com.floragunn.searchguard.filter.SearchGuardActionFilter.apply0(SearchGuardActionFilter.java:145)
at
com.floragunn.searchguard.filter.SearchGuardActionFilter.apply(SearchGuardActionFilter.java:90)
at
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at
com.floragunn.searchguard.filter.AbstractActionFilter.apply(AbstractActionFilter.java:105)
at
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at
com.floragunn.searchguard.filter.AbstractActionFilter.apply(AbstractActionFilter.java:105)
at
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at
com.floragunn.searchguard.filter.AbstractActionFilter.apply(AbstractActionFilter.java:105)
at
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at
org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:64)
at
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at
io.fabric8.elasticsearch.plugin.ActionForbiddenActionFilter.apply(ActionForbiddenActionFilter.java:48)
at
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:82)
at
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:55)
at org.elasticsearch.client.node.NodeClient.execute(NodeClient.java:90)
at org.elasticsearch.client.support.AbstractClient.get(AbstractClient.java:188)
at
io.fabric8.elasticsearch.plugin.acl.DynamicACLFilter.loadAcl(DynamicACLFilter.java:245)
at
io.fabric8.elasticsearch.plugin.acl.DynamicACLFilter.seedInitialACL(DynamicACLFilter.java:297)
at
io.fabric8.elasticsearch.plugin.acl.DynamicACLFilter.onSearchGuardACLActionRequest(DynamicACLFilter.java:118)
at
io.fabric8.elasticsearch.plugin.acl.DefaultACLNotifierService$1.run(DefaultACLNotifierService.java:64)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[2016-04-20 06:57:11,186][WARN ][io.fabric8.kubernetes.api.KubernetesHelper]
Lookup logging-es-cluster result: host not found
[2016-04-20 06:57:11,197][INFO ][cluster.service ] [Baphomet]
new_master
[Baphomet][TSHXtJg3T864x7lBLTVqvA][logging-es-0g0sypsu-1-tw59i][inet[/x.x.x.x:9300]],
reason: zen-disco-join (elected_as_master)
[2016-04-20 06:57:11,214][INFO ][http ] [Baphomet]
bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address
{inet[/x.x.x.x:9200]}
[2016-04-20 06:57:11,214][INFO ][node ] [Baphomet] started
[2016-04-20 06:57:11,227][INFO ][gateway ] [Baphomet]
recovered [0] indices into cluster_state
[2016-04-20 06:57:12,254][INFO ][cluster.metadata ] [Baphomet]
[.searchguard.logging-es-0g0sypsu-1-tw59i] creating index, cause [auto(index
api)], templates [], shards [5]/[1], mappings [ac]
[2016-04-20 06:57:12,537][INFO ][cluster.metadata ] [Baphomet]
[.searchguard.logging-es-0g0sypsu-1-tw59i] update_mapping [ac] (dynamic)
[2016-04-20 06:58:21,001][INFO ][cluster.metadata ] [Baphomet]
[logging.2016.04.20] creating index, cause [auto(bulk api)], templates [],
shards [5]/[1], mappings [fluentd]
[2016-04-20 06:58:21,120][INFO ][cluster.metadata ] [Baphomet]
[logging.2016.04.20] update_mapping [fluentd] (dynamic)
[2016-04-20 06:59:39,673][INFO ][cluster.metadata ] [Baphomet]
[.operations.2016.04.07] creating index, cause [auto(bulk api)], templates [],
shards [5]/[1], mappings [fluentd]
[2016-04-20 06:59:39,748][INFO ][cluster.metadata ] [Baphomet]
[.operations.2016.04.20] creating index, cause [auto(bulk api)], templates [],
shards [5]/[1], mappings [fluentd]
[2016-04-20 06:59:40,003][INFO ][cluster.metadata ] [Baphomet]
[.operations.2016.04.20] update_mapping [fluentd] (dynamic)
[2016-04-20 06:59:42,509][INFO ][cluster.metadata ] [Baphomet]
[.operations.2016.04.07] update_mapping [fluentd] (dynamic)
[2016-04-20 07:00:32,865][INFO ][cluster.metadata ] [Baphomet]
[.operations.2016.04.20] update_mapping [fluentd] (dynamic)
[2016-04-20 09:19:32,061][INFO ][cluster.metadata ] [Baphomet]
[.operations.2016.04.08] creating index, cause [auto(bulk api)], templates [],
shards [5]/[1], mappings [fluentd]
[2016-04-20 09:19:32,916][INFO ][cluster.metadata ] [Baphomet]
[.operations.2016.04.08] update_mapping [fluentd] (dynamic)
[2016-04-20 10:07:26,159][INFO ][cluster.metadata ] [Baphomet]
[.operations.2016.04.09] creating index, cause [auto(bulk api)], templates [],
shards [5]/[1], mappings [fluentd]
[2016-04-20 10:07:26,432][INFO ][cluster.metadata ] [Baphomet]
[.operations.2016.04.09] update_mapping [fluentd] (dynamic)
[2016-04-20 10:22:17,801][INFO ][cluster.metadata ] [Baphomet]
[.operations.2016.04.10] creating index, cause [auto(bulk api)], templates [],
shards [5]/[1], mappings [fluentd]
[2016-04-20 10:22:18,235][INFO ][cluster.metadata ] [Baphomet]
[.operations.2016.04.10] update_mapping [fluentd] (dynamic)
[2016-04-20 10:40:59,387][INFO ][cluster.metadata ] [Baphomet]
[.operations.2016.04.11] creating index, cause [auto(bulk api)], templates [],
shards [5]/[1], mappings [fluentd]
[2016-04-20 10:40:59,695][INFO ][cluster.metadata ] [Baphomet]
[.operations.2016.04.11] update_mapping [fluentd] (dynamic)
[2016-04-20 10:53:44,022][INFO ][cluster.metadata ] [Baphomet]
[.operations.2016.04.12] creating index, cause [auto(bulk api)], templates [],
shards [5]/[1], mappings [fluentd]
[2016-04-20 10:53:44,218][INFO ][cluster.metadata ] [Baphomet]
[.operations.2016.04.12] update_mapping [fluentd] (dynamic)
[2016-04-20 11:13:14,707][INFO ][cluster.metadata ] [Baphomet]
[.operations.2016.04.13] creating index, cause [auto(bulk api)], templates [],
shards [5]/[1], mappings [fluentd]
[2016-04-20 11:13:14,934][INFO ][cluster.metadata ] [Baphomet]
[.operations.2016.04.13] update_mapping [fluentd] (dynamic)
[2016-04-20 11:29:00,212][INFO ][cluster.metadata ] [Baphomet]
[.operations.2016.04.14] creating index, cause [auto(bulk api)], templates [],
shards [5]/[1], mappings [fluentd]
[2016-04-20 11:29:00,665][INFO ][cluster.metadata ] [Baphomet]
[.operations.2016.04.14] update_mapping [fluentd] (dynamic)
[2016-04-20 11:44:03,826][INFO ][cluster.metadata ] [Baphomet]
[.operations.2016.04.15] creating index, cause [auto(bulk api)], templates [],
shards [5]/[1], mappings [fluentd]
[2016-04-20 11:44:04,612][INFO ][cluster.metadata ] [Baphomet]
[.operations.2016.04.15] update_mapping [fluentd] (dynamic)
[2016-04-20 11:55:34,170][INFO ][cluster.metadata ] [Baphomet]
[.operations.2016.04.16] creating index, cause [auto(bulk api)], templates [],
shards [5]/[1], mappings [fluentd]
[2016-04-20 11:55:34,338][INFO ][cluster.metadata ] [Baphomet]
[.operations.2016.04.16] update_mapping [fluentd] (dynamic)
[2016-04-20 12:13:41,236][INFO ][cluster.metadata ] [Baphomet]
[.operations.2016.04.17] creating index, cause [auto(bulk api)], templates [],
shards [5]/[1], mappings [fluentd]
[2016-04-20 12:13:41,920][INFO ][cluster.metadata ] [Baphomet]
[.operations.2016.04.17] update_mapping [fluentd] (dynamic)
[2016-04-20 12:27:59,827][INFO ][cluster.metadata ] [Baphomet]
[.operations.2016.04.18] creating index, cause [auto(bulk api)], templates [],
shards [5]/[1], mappings [fluentd]
[2016-04-20 12:28:00,589][INFO ][cluster.metadata ] [Baphomet]
[.operations.2016.04.18] update_mapping [fluentd] (dynamic)
[2016-04-20 12:36:39,258][INFO ][cluster.metadata ] [Baphomet]
[.operations.2016.04.19] creating index, cause [auto(bulk api)], templates [],
shards [5]/[1], mappings [fluentd]
[2016-04-20 12:36:39,640][INFO ][cluster.metadata ] [Baphomet]
[.operations.2016.04.19] update_mapping [fluentd] (dynamic)
[2016-04-20 12:45:37,104][INFO ][cluster.metadata ] [Baphomet]
[.operations.2016.04.20] update_mapping [fluentd] (dynamic)
[2016-04-21 00:00:18,301][INFO ][cluster.metadata ] [Baphomet]
[.operations.2016.04.21] creating index, cause [auto(bulk api)], templates [],
shards [5]/[1], mappings [fluentd]
[2016-04-21 00:00:18,393][INFO ][cluster.metadata ] [Baphomet]
[.operations.2016.04.21] update_mapping [fluentd] (dynamic)
[2016-04-21 00:01:43,323][INFO ][cluster.metadata ] [Baphomet]
[.operations.2016.04.21] update_mapping [fluentd] (dynamic)
[2016-04-21 00:03:18,723][INFO ][cluster.metadata ] [Baphomet]
[.operations.2016.04.21] update_mapping [fluentd] (dynamic)
[2016-04-21 00:06:34,760][INFO ][cluster.metadata ] [Baphomet]
[.operations.2016.04.21] update_mapping [fluentd] (dynamic)
[2016-04-21 00:07:21,827][INFO ][cluster.metadata ] [Baphomet]
[logging.2016.04.21] creating index, cause [auto(bulk api)], templates [],
shards [5]/[1], mappings [fluentd]
[2016-04-21 00:07:21,911][INFO ][cluster.metadata ] [Baphomet]
[logging.2016.04.21] update_mapping [fluentd] (dynamic)
Fluentd
Name: logging-fluentd-6c6bt
Namespace: logging
Node: xxxxx/x.x.x.x
Start Time: Wed, 20 Apr 2016 06:37:13 +0000
Labels: component=fluentd,provider=openshift
Status: Running
IP: x.x.x.x
Controllers: DaemonSet/logging-fluentd
Containers:
fluentd-elasticsearch:
Container ID:
docker://2cbe9d5813c3d771c4fc10c92be61136bb43e11f85fc9c117c64ca5e42060627
Image: docker.io/openshift/origin-logging-fluentd:latest
Image ID:
docker://f841fe531e980e970d69b353bf49ec8f69dd13f88dccfdaff578d91f0fa58e63
Port:
QoS Tier:
cpu: Guaranteed
memory: BestEffort
Limits:
cpu: 100m
Requests:
cpu: 100m
State: Running
Started: Wed, 20 Apr 2016 07:43:35 +0000
Ready: True
Restart Count: 0
Environment Variables:
K8S_HOST_URL: https://kubernetes.default.svc.cluster.local
ES_HOST: logging-es
ES_PORT: 9200
ES_CLIENT_CERT: /etc/fluent/keys/cert
ES_CLIENT_KEY: /etc/fluent/keys/key
ES_CA: /etc/fluent/keys/ca
OPS_HOST: logging-es
OPS_PORT: 9200
OPS_CLIENT_CERT: /etc/fluent/keys/cert
OPS_CLIENT_KEY: /etc/fluent/keys/key
OPS_CA: /etc/fluent/keys/ca
ES_COPY: false
ES_COPY_HOST:
ES_COPY_PORT:
ES_COPY_SCHEME: https
ES_COPY_CLIENT_CERT:
ES_COPY_CLIENT_KEY:
ES_COPY_CA:
ES_COPY_USERNAME:
ES_COPY_PASSWORD:
OPS_COPY_HOST:
OPS_COPY_PORT:
OPS_COPY_SCHEME: https
OPS_COPY_CLIENT_CERT:
OPS_COPY_CLIENT_KEY:
OPS_COPY_CA:
OPS_COPY_USERNAME:
OPS_COPY_PASSWORD:
Conditions:
Type Status
Ready True
Volumes:
varlog:
Type: HostPath (bare host directory volume)
Path: /var/log
varlibdockercontainers:
Type: HostPath (bare host directory volume)
Path: /var/lib/docker/containers
certs:
Type: Secret (a volume populated by a Secret)
SecretName: logging-fluentd
dockerhostname:
Type: HostPath (bare host directory volume)
Path: /etc/hostname
aggregated-logging-fluentd-token-f7ysx:
Type: Secret (a volume populated by a Secret)
SecretName: aggregated-logging-fluentd-token-f7ysx
No events.
On 20 Apr 2016, at 9:54 PM, Eric Wolinetz
<[email protected]<mailto:[email protected]>> wrote:
Hi Sebastian,
Your Elasticsearch instance does not seem to have started up completely within
the pod you showed logs for. Kibana will fail to start up if it is unable to
reach its Elasticsearch instance after a certain period of time.
Can you send some more of your Elasticsearch logs? It looks like its currently
recovering/initializing. Do you see any different ERROR messages within there?
Your Fluentd errors looks to be something else. What does the following look
like?
$ oc describe pod -l component=fluentd
On Wed, Apr 20, 2016 at 2:54 AM, Sebastian Wieseler
<[email protected]<mailto:[email protected]>> wrote:
Dear community,
I followed the guide
https://docs.openshift.org/latest/install_config/aggregate_logging.html
NAME READY STATUS RESTARTS AGE
logging-kibana-1-uwob1 1/2 Error 12 43m
$ oc logs logging-kibana-1-uwob1 -c kibana
{"name":"Kibana","hostname":"logging-kibana-1-uwob1","pid":7,"level":50,"err":{"message":"Request
Timeout after 5000ms","name":"Error","stack":"Error: Request Timeout after
5000ms\n at null.<anonymous>
(/opt/app-root/src/src/node_modules/elasticsearch/src/lib/transport.js:282:15)\n
at Timer.listOnTimeout [as ontimeout]
(timers.js:112:15)"},"msg":"","time":"2016-04-20T07:16:15.760Z","v":0}
{"name":"Kibana","hostname":"logging-kibana-1-uwob1","pid":7,"level":60,"err":{"message":"Request
Timeout after 5000ms","name":"Error","stack":"Error: Request Timeout after
5000ms\n at null.<anonymous>
(/opt/app-root/src/src/node_modules/elasticsearch/src/lib/transport.js:282:15)\n
at Timer.listOnTimeout [as ontimeout]
(timers.js:112:15)"},"msg":"","time":"2016-04-20T07:16:15.762Z","v":0}
[root@MRNZ-TS8-OC-MASTER-01 glusterfs]# oc logs logging-kibana-1-uwob1 -c
kibana
{"name":"Kibana","hostname":"logging-kibana-1-uwob1","pid":7,"level":50,"err":{"message":"Request
Timeout after 5000ms","name":"Error","stack":"Error: Request Timeout after
5000ms\n at null.<anonymous>
(/opt/app-root/src/src/node_modules/elasticsearch/src/lib/transport.js:282:15)\n
at Timer.listOnTimeout [as ontimeout]
(timers.js:112:15)"},"msg":"","time":"2016-04-20T07:38:40.789Z","v":0}
{"name":"Kibana","hostname":"logging-kibana-1-uwob1","pid":7,"level":60,"err":{"message":"Request
Timeout after 5000ms","name":"Error","stack":"Error: Request Timeout after
5000ms\n at null.<anonymous>
(/opt/app-root/src/src/node_modules/elasticsearch/src/lib/transport.js:282:15)\n
at Timer.listOnTimeout [as ontimeout]
(timers.js:112:15)"},"msg":"","time":"2016-04-20T07:38:40.790Z","v":0}
Elastic search pod is running, but the log shows:
[2016-04-20
06:57:03,910][ERROR][io.fabric8.elasticsearch.plugin.acl.DynamicACLFilter]
[Baphomet] Exception encountered when seeding initial ACL
org.elasticsearch.cluster.block.ClusterBlockException: blocked by:
[SERVICE_UNAVAILABLE/1/state not recovered / initialized];
at
org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:151)
at
org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction.checkGlobalBlock(TransportShardSingleOperationAction.java:103)
at
org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction.<init>(TransportShardSingleOperationAction.java:132)
at
org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction.<init>(TransportShardSingleOperationAction.java:116)
Fluent pod is running too, but the log shows:
2016-04-20 07:47:18 +0000 [error]: fluentd main process died unexpectedly.
restarting.
2016-04-20 07:47:48 +0000 [error]: unexpected error error="getaddrinfo: Name or
service not known"
2016-04-20 07:47:48 +0000 [error]: /usr/share/ruby/net/http.rb:878:in
`initialize'
2016-04-20 07:47:48 +0000 [error]: /usr/share/ruby/net/http.rb:878:in `open'
2016-04-20 07:47:48 +0000 [error]: /usr/share/ruby/net/http.rb:878:in `block
in connect'
2016-04-20 07:47:48 +0000 [error]: /usr/share/ruby/timeout.rb:52:in `timeout'
2016-04-20 07:47:48 +0000 [error]: /usr/share/ruby/net/http.rb:877:in
`connect'
2016-04-20 07:47:48 +0000 [error]: /usr/share/ruby/net/http.rb:862:in
`do_start'
2016-04-20 07:47:48 +0000 [error]: /usr/share/ruby/net/http.rb:851:in `start'
2016-04-20 07:47:48 +0000 [error]:
/opt/app-root/src/gems/rest-client-1.8.0/lib/restclient/request.rb:413:in
`transmit'
2016-04-20 07:47:48 +0000 [error]:
/opt/app-root/src/gems/rest-client-1.8.0/lib/restclient/request.rb:176:in
`execute'
2016-04-20 07:47:48 +0000 [error]:
/opt/app-root/src/gems/rest-client-1.8.0/lib/restclient/request.rb:41:in
`execute'
2016-04-20 07:47:48 +0000 [error]:
/opt/app-root/src/gems/rest-client-1.8.0/lib/restclient/resource.rb:51:in `get'
2016-04-20 07:47:48 +0000 [error]:
/opt/app-root/src/gems/kubeclient-1.1.2/lib/kubeclient/common.rb:310:in `block
in api'
2016-04-20 07:47:48 +0000 [error]:
/opt/app-root/src/gems/kubeclient-1.1.2/lib/kubeclient/common.rb:51:in
`handle_exception'
2016-04-20 07:47:48 +0000 [error]:
/opt/app-root/src/gems/kubeclient-1.1.2/lib/kubeclient/common.rb:309:in `api'
2016-04-20 07:47:48 +0000 [error]:
/opt/app-root/src/gems/kubeclient-1.1.2/lib/kubeclient/common.rb:304:in
`api_valid?'
2016-04-20 07:47:48 +0000 [error]:
/opt/app-root/src/gems/fluent-plugin-kubernetes_metadata_filter-0.18.0/lib/fluent/plugin/filter_kubernetes_metadata.rb:134:in
`configure’
…
2016-04-20 07:50:42 +0000 [warn]: emit transaction failed:
error_class=Fluent::ConfigError error="Exception encountered fetching metadata
from Kubernetes API endpoint: getaddrinfo: Name or service not known"
tag="kubernetes.var.log.containers.docker-registry-2-9tehj_default_registry-957340d6e4686b63bdcdccf18f1b0b4054d1faab9724097928f8102c3190f312.log"
2016-04-20 07:50:42 +0000 [warn]:
/opt/app-root/src/gems/fluent-plugin-kubernetes_metadata_filter-0.18.0/lib/fluent/plugin/filter_kubernetes_metadata.rb:232:in
`rescue in start_watch'
2016-04-20 07:50:42 +0000 [warn]:
/opt/app-root/src/gems/fluent-plugin-kubernetes_metadata_filter-0.18.0/lib/fluent/plugin/filter_kubernetes_metadata.rb:228:in
`start_watch'
2016-04-20 07:50:42 +0000 [warn]:
/opt/app-root/src/gems/fluent-plugin-kubernetes_metadata_filter-0.18.0/lib/fluent/plugin/filter_kubernetes_metadata.rb:140:in
`block in configure'
2016-04-20 07:51:11 +0000 [warn]: emit transaction failed:
error_class=SocketError error="getaddrinfo: Name or service not known"
tag="kubernetes.var.log.containers.router-1-94cv3_default_router-3e8bc8d6d4ef8a52f8f46a7e83b5a510a4f1c8d63658cc49b3ce506822e84811.log”
Any idea why kibana is not starting properly?
Greetings,
Sebastian
_______________________________________________
users mailing list
[email protected]<mailto:[email protected]>
http://lists.openshift.redhat.com/openshiftmm/listinfo/users
_______________________________________________
users mailing list
[email protected]
http://lists.openshift.redhat.com/openshiftmm/listinfo/users
