On 06/09/2017 12:33, Aleksandar Lazic wrote:
Hi Tim.
A dump question but do you have any proxy setuped?
No, just a vanilla ansible install running on a machine on EC2. The
inventory file was posted earlier.
on Mittwoch, 06. September 2017 at 12:49 was written:
No joy.
The cassandra pod starts fine but the hawkular on fails to start with
what looks like the same errors as I described before.
One of the interesting par is that hawkular can connect to cas
###
2017-09-05 14:54:48,123 INFO [com.datastax.driver.core.Cluster] (ServerService
Thread Pool -- 64) New Cassandra host hawkular-cassandra/172.30.151.137:9042
added
...
2017-09-05 14:54:48,276 INFO [org.cassalog.core.CassalogImpl]
(metricsservice-lifecycle-thread) Applying ChangeSet
-- version: set-keyspace
USE hawkular_metrics
...
####
and then got you a NullPointerException
###
2017-09-05 14:54:49,163 FATAL
[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
(metricsservice-lifecycle-thread) HAWKMETRICS200006: An error occurred trying
to connect to the Cassandra cluster: java.lang.NullPointerException
at
org.hawkular.metrics.core.dropwizard.HawkularObjectNameFactory.createName(HawkularObjectNameFactory.java:54)
at
com.codahale.metrics.JmxReporter$JmxListener.createName(JmxReporter.java:656)
at
com.codahale.metrics.JmxReporter$JmxListener.onTimerAdded(JmxReporter.java:633)
at
com.codahale.metrics.MetricRegistry.notifyListenerOfAddedMetric(MetricRegistry.java:356)
at
com.codahale.metrics.MetricRegistry.addListener(MetricRegistry.java:191)
at com.codahale.metrics.JmxReporter.start(JmxReporter.java:715)
at
org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle.startMetricsService(MetricsServiceLifecycle.java:474)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
###
Was the Cassandra up when the hawkular started?
Yes.
are you able to curl Cassandra from hawkular pod?
oc debug rc/hawkular-metrics
curl -v telnet://hawkular-cassandra:9042/
Yes:
$ curl -v telnet://hawkular-cassandra:9042/
* About to connect() to hawkular-cassandra port 9042 (#0)
* Trying 172.30.78.190...
* Connected to hawkular-cassandra (172.30.78.190) port 9042 (#0)
^C
Tim
Tim
On 06/09/2017 10:34, Aleksandar Lazic wrote:
Hi Tim.
on Dienstag, 05. September 2017 at 17:10 was written:
Still no joy with this.
I retried with the latest code and still hitting the same problem.
Metrics does not seem to be working with a new Ansible install.
I'm using a minimal setup with an inventory like this:
[OSEv3:children]
[snipp]
When the install completes the openshift-infra project pods ends up like
this:
NAME READY STATUS RESTARTS AGE
hawkular-cassandra-1-4m7lq 1/1 Running 0 16m
hawkular-metrics-0nl1q 0/1 CrashLoopBackOff 7 16m
heapster-cgw0b 0/1 Running 1 16m
The hawkular-metrics pods is failing, and it looks like its because it
can't connect to the cassandra pod.
The full log of the hawkular-metrics pod is here:
https://gist.github.com/tdudgeon/f3099911eed441817369ee03635aad7d
Any help resolving this would be appreciated.
I normally do the following.
oc scale rc/hawkular-metrics --replicas=0
oc scale rc/heapster --replicas=0
oc scale rc/hawkular-cassandra --replicas=0
after all pods are down start the new instaces.
Sometimes I must send a
oc delete pod <HANGING_POD> --grace-period=0
when a pod is hanging.
Now start cas
oc scale rc/hawkular-cassandra --replicas=1
Wait until cas is up without errors!
oc logs -f <NEW_POD>
Now run hawk
oc scale rc/hawkular-metrics --replicas=1
Wait until hawk is up without errors!
oc logs -f <NEW_POD>
Now run heapster
oc scale rc/heapster --replicas=1
oc logs -f <NEW_POD>
Does this help?
Tim
_______________________________________________
users mailing list
[email protected]
http://lists.openshift.redhat.com/openshiftmm/listinfo/users
