So it was pointed out that the hawkular-metrics pod was failing.
I tried restarting it to check this and it don't improve the situation.
The logs for that pod show this:
2017-08-29 14:44:09,639 WARN [com.datastax.driver.core.Cluster]
(cluster3-worker-1) Re-preparing already prepared query is generally
an anti-pattern and will likely affect performance. Consider preparing
the statement only once. Query='SELECT DISTINCT tenant_id, tname FROM
metrics_tags_idx'
2017-08-29 14:44:09,694 INFO
[org.hawkular.metrics.core.util.GCGraceSecondsManager]
(metricsservice-lifecycle-thread) Checking tables in hawkular_metrics
to see if gc_grace_seconds needs to be updated
2017-08-29 14:44:09,727 FATAL
[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
(metricsservice-lifecycle-thread) HAWKMETRICS200006: An error occurred
trying to connect to the Cassandra cluster:
java.lang.NullPointerException
at
org.hawkular.metrics.core.dropwizard.HawkularObjectNameFactory.createName(HawkularObjectNameFactory.java:54)
at
com.codahale.metrics.JmxReporter$JmxListener.createName(JmxReporter.java:656)
at
com.codahale.metrics.JmxReporter$JmxListener.onTimerAdded(JmxReporter.java:633)
at
com.codahale.metrics.MetricRegistry.notifyListenerOfAddedMetric(MetricRegistry.java:356)
at
com.codahale.metrics.MetricRegistry.addListener(MetricRegistry.java:191)
at com.codahale.metrics.JmxReporter.start(JmxReporter.java:715)
at
org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle.startMetricsService(MetricsServiceLifecycle.java:474)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2017-08-29 14:44:09,729 INFO
[org.hawkular.metrics.core.util.GCGraceSecondsManager]
(RxComputationScheduler-4) gc_grace_seconds for tenants_by_time is set
to 864000. Resetting to 0
2017-08-29 14:44:09,839 INFO [org.wildfly.extension.undertow]
(ServerService Thread Pool -- 73) WFLYUT0021: Registered web context:
/hawkular/__alerter-metrics
2017-08-29 14:44:09,876 INFO [org.jboss.as.server] (ServerService
Thread Pool -- 37) WFLYSRV0010: Deployed "hawkular-metrics.ear"
(runtime-name : "hawkular-metrics.ear")
2017-08-29 14:44:09,902 INFO
[org.hawkular.metrics.core.util.GCGraceSecondsManager]
(RxComputationScheduler-3) Finished gc_grace_seconds updates in 206 ms
2017-08-29 14:44:09,995 INFO [org.jboss.as] (Controller Boot Thread)
WFLYSRV0060: Http management interface listening on
http://127.0.0.1:9990/management
2017-08-29 14:44:09,995 INFO [org.jboss.as] (Controller Boot Thread)
WFLYSRV0051: Admin console listening on http://127.0.0.1:9990
2017-08-29 14:44:09,995 INFO [org.jboss.as] (Controller Boot Thread)
WFLYSRV0025: WildFly Full 10.1.0.Final (WildFly Core 2.2.0.Final)
started in 23950ms - Started 1639 of 2091 services (849 services are
lazy, passive or on-demand)
2017-08-29 14:44:13,032 SEVERE
[com.google.common.util.concurrent.ExecutionList]
(cluster3-nio-worker-5) RuntimeException while executing runnable
rx.observable.ListenableFutureObservable$2$1@29128f9b with executor
com.google.common.util.concurrent.MoreExecutors$ListeningDecorator@5d4b9aa2:
java.util.concurrent.RejectedExecutionException: Task
rx.observable.ListenableFutureObservable$2$1@29128f9b rejected from
java.util.concurrent.ThreadPoolExecutor@4095943c[Terminated, pool size
= 0, active threads = 0, queued tasks = 0, completed tasks = 78]
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
at
com.google.common.util.concurrent.MoreExecutors$ListeningDecorator.execute(MoreExecutors.java:550)
at
com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
at
com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
at
com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:185)
at
com.datastax.driver.core.DefaultResultSetFuture.onSet(DefaultResultSetFuture.java:174)
at
com.datastax.driver.core.RequestHandler.setFinalResult(RequestHandler.java:176)
at
com.datastax.driver.core.RequestHandler.access$2400(RequestHandler.java:46)
at
com.datastax.driver.core.RequestHandler$SpeculativeExecution.setFinalResult(RequestHandler.java:792)
at
com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:467)
at
com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1070)
at
com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:993)
at
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321)
at
io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:266)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321)
at
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321)
at
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293)
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1070)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:904)
at
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:411)
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321)
at
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1280)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:890)
at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
at
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:564)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:505)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:419)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:391)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
at
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:145)
at java.lang.Thread.run(Thread.java:748)
2017-08-29 14:44:13,033 SEVERE
[com.google.common.util.concurrent.ExecutionList]
(cluster3-nio-worker-3) RuntimeException while executing runnable
rx.observable.ListenableFutureObservable$2$1@1f22d19a with executor
com.google.common.util.concurrent.MoreExecutors$ListeningDecorator@5d4b9aa2:
java.util.concurrent.RejectedExecutionException: Task
rx.observable.ListenableFutureObservable$2$1@1f22d19a rejected from
java.util.concurrent.ThreadPoolExecutor@4095943c[Terminated, pool size
= 0, active threads = 0, queued tasks = 0, completed tasks = 78]
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
at
com.google.common.util.concurrent.MoreExecutors$ListeningDecorator.execute(MoreExecutors.java:550)
at
com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
at
com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
at
com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:185)
at
com.datastax.driver.core.DefaultResultSetFuture.onSet(DefaultResultSetFuture.java:174)
at
com.datastax.driver.core.RequestHandler.setFinalResult(RequestHandler.java:176)
at
com.datastax.driver.core.RequestHandler.access$2400(RequestHandler.java:46)
at
com.datastax.driver.core.RequestHandler$SpeculativeExecution.setFinalResult(RequestHandler.java:792)
at
com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:467)
at
com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1070)
at
com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:993)
at
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321)
at
io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:266)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321)
at
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321)
at
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293)
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1070)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:904)
at
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:411)
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321)
at
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1280)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:890)
at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
at
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:564)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:505)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:419)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:391)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
at
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:145)
at java.lang.Thread.run(Thread.java:748)
The hawkular-cassandra pod seem to be running fine.
Any thoughts?
On 29/08/2017 15:20, bahhooo wrote:
Hi Tim,
it looks like your hawkular pod is not ready. As a result heapster is
not ready, either.
openshift-infra po/hawkular-metrics-r1cct *0*/1 Running 0 2m
openshift-infra po/heapster-z6wm4 *0*/1 Running 0 2m
Check the logs of the hawkular pod with
$ oc logs -f hawkular-metrics-r1cct -n openshift-infra
and try to find out what's worng.
Sometimes restarting the pod helps.
_______________________________________________
users mailing list
[email protected]
http://lists.openshift.redhat.com/openshiftmm/listinfo/users
