Hi Tim. on Mittwoch, 06. September 2017 at 13:43 was written:
> On 06/09/2017 12:33, Aleksandar Lazic wrote: >> Hi Tim. >> >> A dump question but do you have any proxy setuped? > No, just a vanilla ansible install running on a machine on EC2. The > inventory file was posted earlier. >> >> on Mittwoch, 06. September 2017 at 12:49 was written: >> >>> No joy. >>> The cassandra pod starts fine but the hawkular on fails to start with >>> what looks like the same errors as I described before. >> One of the interesting par is that hawkular can connect to cas >> >> ### >> 2017-09-05 14:54:48,123 INFO [com.datastax.driver.core.Cluster] >> (ServerService Thread Pool -- 64) New Cassandra host >> hawkular-cassandra/172.30.151.137:9042 added >> ... >> 2017-09-05 14:54:48,276 INFO [org.cassalog.core.CassalogImpl] >> (metricsservice-lifecycle-thread) Applying ChangeSet >> -- version: set-keyspace >> USE hawkular_metrics >> ... >> #### >> >> and then got you a NullPointerException >> >> ### >> 2017-09-05 14:54:49,163 FATAL >> [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] >> (metricsservice-lifecycle-thread) HAWKMETRICS200006: An error occurred >> trying to connect to the Cassandra cluster: java.lang.NullPointerException >> at >> org.hawkular.metrics.core.dropwizard.HawkularObjectNameFactory.createName(HawkularObjectNameFactory.java:54) >> at >> com.codahale.metrics.JmxReporter$JmxListener.createName(JmxReporter.java:656) >> at >> com.codahale.metrics.JmxReporter$JmxListener.onTimerAdded(JmxReporter.java:633) >> at >> com.codahale.metrics.MetricRegistry.notifyListenerOfAddedMetric(MetricRegistry.java:356) >> at >> com.codahale.metrics.MetricRegistry.addListener(MetricRegistry.java:191) >> at com.codahale.metrics.JmxReporter.start(JmxReporter.java:715) >> at >> org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle.startMetricsService(MetricsServiceLifecycle.java:474) >> at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) >> at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >> at java.lang.Thread.run(Thread.java:748) >> ### >> >> Was the Cassandra up when the hawkular started? > Yes. >> >> are you able to curl Cassandra from hawkular pod? >> >> oc debug rc/hawkular-metrics >> >> curl -v telnet://hawkular-cassandra:9042/ > Yes: > $ curl -v telnet://hawkular-cassandra:9042/ > * About to connect() to hawkular-cassandra port 9042 (#0) > * Trying 172.30.78.190... > * Connected to hawkular-cassandra (172.30.78.190) port 9042 (#0) > ^C that's strange. Please can you try a previous image version of hawkular? > Tim >> >>> Tim >> >>> On 06/09/2017 10:34, Aleksandar Lazic wrote: >>>> Hi Tim. >>>> >>>> on Dienstag, 05. September 2017 at 17:10 was written: >>>> >>>>> Still no joy with this. >>>>> I retried with the latest code and still hitting the same problem. >>>>> Metrics does not seem to be working with a new Ansible install. >>>>> I'm using a minimal setup with an inventory like this: >>>>>> [OSEv3:children] >>>> [snipp] >>>> >>>>> When the install completes the openshift-infra project pods ends up like >>>>> this: >>>>>> NAME READY STATUS RESTARTS AGE >>>>>> hawkular-cassandra-1-4m7lq 1/1 Running 0 16m >>>>>> hawkular-metrics-0nl1q 0/1 CrashLoopBackOff 7 16m >>>>>> heapster-cgw0b 0/1 Running 1 16m >>>>> The hawkular-metrics pods is failing, and it looks like its because it >>>>> can't connect to the cassandra pod. >>>>> The full log of the hawkular-metrics pod is here: >>>>> https://gist.github.com/tdudgeon/f3099911eed441817369ee03635aad7d >>>>> Any help resolving this would be appreciated. >>>> I normally do the following. >>>> >>>> oc scale rc/hawkular-metrics --replicas=0 >>>> oc scale rc/heapster --replicas=0 >>>> oc scale rc/hawkular-cassandra --replicas=0 >>>> >>>> after all pods are down start the new instaces. >>>> >>>> Sometimes I must send a >>>> >>>> oc delete pod <HANGING_POD> --grace-period=0 >>>> >>>> when a pod is hanging. >>>> >>>> Now start cas >>>> >>>> oc scale rc/hawkular-cassandra --replicas=1 >>>> >>>> Wait until cas is up without errors! >>>> >>>> oc logs -f <NEW_POD> >>>> >>>> Now run hawk >>>> >>>> oc scale rc/hawkular-metrics --replicas=1 >>>> >>>> Wait until hawk is up without errors! >>>> >>>> oc logs -f <NEW_POD> >>>> >>>> Now run heapster >>>> >>>> oc scale rc/heapster --replicas=1 >>>> >>>> oc logs -f <NEW_POD> >>>> >>>> Does this help? >>>> >>>>> Tim -- Best Regards Aleks
smime.p7s
Description: S/MIME Cryptographic Signature
_______________________________________________ users mailing list [email protected] http://lists.openshift.redhat.com/openshiftmm/listinfo/users
