I can see internal DNS server also resolvable dig +short @master-01 hawkular-metrics.openshift-infra.svc.cluster.local 172.30.117.176
so all looks good but still seems to be some issues. -- Srinivas Kotaru On 6/14/16, 10:37 AM, "Srinivas Naga Kotaru (skotaru)" <[email protected]> wrote: >I removed readiness probes from both hawkular-cassandra-1 & hawkular-metrics >as both status shows probes failed. > >It looks good now. Both containers looks and running >(hawkular-cassandra-1-kr8ka , hawkular-metrics-vhe3u) however heapster-7yl34 >logs still shows Could not connect to >https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 6. >Status Code 000. > >Are we good or still had issues? > > ># oc get pods >NAME READY STATUS RESTARTS AGE >hawkular-cassandra-1-kr8ka 1/1 Running 0 6m >hawkular-metrics-vhe3u 1/1 Running 2 5m >heapster-7yl34 0/1 Running 2 5m > > > > > >-- >Srinivas Kotaru > >On 6/14/16, 10:07 AM, "Srinivas Naga Kotaru (skotaru)" <[email protected]> >wrote: > >>Matt >> >>Just want to share more info by running describe pod. >> >>It seems to be health probe failing. Do you think it is the issue? >> >> >> >># oc describe pod hawkular-cassandra-1-it5uh >>Name: hawkular-cassandra-1-it5uh >>Namespace: openshift-infra >>Node: l3inpn-id2-003.cisco.com/173.36.96.16 >>Start Time: Tue, 14 Jun 2016 16:36:21 +0000 >>Labels: >>metrics-infra=hawkular-cassandra,name=hawkular-cassandra-1,type=hawkular-cassandra >>Status: Running >>IP: 10.1.9.2 >>Controllers: ReplicationController/hawkular-cassandra-1 >>Containers: >> hawkular-cassandra-1: >> Container ID: >> docker://17a9575eb655145859a9207f5c4bde7456f947e27188a056ff2bd08c4ce6ae5d >> Image: >> registry.access.redhat.com/openshift3/metrics-cassandra:latest >> Image ID: >> docker://ee2117c9848298ca5a0cbbce354fd4adff370435225324ab9d60cd9cd9a95c53 >> Ports: 9042/TCP, 9160/TCP, 7000/TCP, 7001/TCP >> Command: >> /opt/apache-cassandra/bin/cassandra-docker.sh >> --cluster_name=hawkular-metrics >> --data_volume=/cassandra_data >> --internode_encryption=all >> --require_node_auth=true >> --enable_client_encryption=true >> --require_client_auth=true >> --keystore_file=/secret/cassandra.keystore >> --keystore_password_file=/secret/cassandra.keystore.password >> --truststore_file=/secret/cassandra.truststore >> --truststore_password_file=/secret/cassandra.truststore.password >> --cassandra_pem_file=/secret/cassandra.pem >> QoS Tier: >> cpu: BestEffort >> memory: BestEffort >> State: Running >> Started: Tue, 14 Jun 2016 16:37:01 +0000 >> Ready: True >> Restart Count: 0 >> Readiness: exec >> [/opt/apache-cassandra/bin/cassandra-docker-ready.sh] delay=0s timeout=1s >> period=10s #success=1 #failure=3 >> Environment Variables: >> CASSANDRA_MASTER: true >> POD_NAMESPACE: openshift-infra (v1:metadata.namespace) >>Conditions: >> Type Status >> Ready True >>Volumes: >> cassandra-data: >> Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim >> in the same namespace) >> ClaimName: metrics-cassandra-1 >> ReadOnly: false >> hawkular-cassandra-secrets: >> Type: Secret (a volume populated by a Secret) >> SecretName: hawkular-cassandra-secrets >> cassandra-token-4urfd: >> Type: Secret (a volume populated by a Secret) >> SecretName: cassandra-token-4urfd >>Events: >> FirstSeen LastSeen Count From >> SubobjectPath Type Reason >> Message >> --------- -------- ----- ---- >> ------------- -------- ------ >> ------- >> 27m 27m 1 {default-scheduler } >> Normal Scheduled >> Successfully assigned hawkular-cassandra-1-it5uh to l3inpn-id2-003.cisco.com >> 27m 27m 1 {kubelet l3inpn-id2-003.cisco.com} >> spec.containers{hawkular-cassandra-1} Normal Pulling >> pulling image >> "registry.access.redhat.com/openshift3/metrics-cassandra:latest" >> 27m 27m 1 {kubelet l3inpn-id2-003.cisco.com} >> spec.containers{hawkular-cassandra-1} Normal Pulled >> Successfully pulled image >> "registry.access.redhat.com/openshift3/metrics-cassandra:latest" >> 27m 27m 1 {kubelet l3inpn-id2-003.cisco.com} >> spec.containers{hawkular-cassandra-1} Normal Created >> Created container with docker id 17a9575eb655 >> 27m 27m 1 {kubelet l3inpn-id2-003.cisco.com} >> spec.containers{hawkular-cassandra-1} Normal Started >> Started container with docker id 17a9575eb655 >> 27m 26m 3 {kubelet l3inpn-id2-003.cisco.com} >> spec.containers{hawkular-cassandra-1} Warning Unhealthy >> Readiness probe failed: cat: /etc/ld.so.conf.d/*.conf: No such file or >> directory >>nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException: >>'Connection refused'. >>Cassandra not in the up and normal state. Current state is >>/opt/apache-cassandra/bin/cassandra-docker-ready.sh: line 28: [: =: unary >>operator expected >> >> >> >> >> >>-- >>Srinivas Kotaru >> >>On 6/14/16, 10:00 AM, "Srinivas Naga Kotaru (skotaru)" <[email protected]> >>wrote: >> >>>Matt >>> >>>DNS service is working >>> >>> >>>~ dig +short @master-01 kubernetes.default.svc.cluster.local >>>172.30.0.1 >>>~ dig +short @master-01 jenkins.alln-test.svc.cluster.local >>>172.30.85.148 >>>~ dig +short @master-01 cakephp-example.alln-test.svc.cluster.local >>>172.30.31.6 >>> >>>I captured hawkular-metrics, it shows the problem. It seems to be unable to >>>connect Cassandra cluster >>> >>># oc exec hawkular-metrics-qldd8 cat /opt/eap/standalone/log/server.log >>> >>>12:40:58,848 INFO [org.jboss.as.clustering.infinispan] (ServerService >>>Thread Pool -- 31) JBAS010280: Activating Infinispan subsystem. >>>12:40:58,898 INFO [org.jboss.as.webservices] (ServerService Thread Pool -- >>>48) JBAS015537: Activating WebServices Extension >>>12:40:58,901 WARN [org.jboss.as.txn] (ServerService Thread Pool -- 46) >>>JBAS010153: Node identifier property is set to the default value. Please >>>make sure it is unique. >>>12:40:58,902 INFO [org.jboss.as.security] (ServerService Thread Pool -- 44) >>>JBAS013371: Activating Security Subsystem >>>12:40:58,912 INFO [org.jboss.as.naming] (ServerService Thread Pool -- 39) >>>JBAS011800: Activating Naming Subsystem >>>12:40:58,928 INFO [org.jboss.as.jsf] (ServerService Thread Pool -- 37) >>>JBAS012615: Activated the following JSF Implementations: [main, 1.2] >>>12:40:58,939 INFO [org.jboss.as.connector.logging] (MSC service thread 1-2) >>>JBAS010408: Starting JCA Subsystem (IronJacamar 1.0.32.Final-redhat-1) >>>12:40:58,981 INFO [org.jboss.as.security] (MSC service thread 1-2) >>>JBAS013370: Current PicketBox version=4.1.1.Final-redhat-1 >>>12:40:58,961 INFO [org.jboss.as.connector.subsystems.datasources] >>>(ServerService Thread Pool -- 27) JBAS010403: Deploying JDBC-compliant >>>driver class org.h2.Driver (version 1.3) >>>12:40:59,039 INFO [org.jboss.as.naming] (MSC service thread 1-2) >>>JBAS011802: Starting Naming Service >>>12:40:59,052 INFO [org.jboss.as.mail.extension] (MSC service thread 1-4) >>>JBAS015400: Bound mail session [java:jboss/mail/Default] >>>12:40:59,614 INFO [org.apache.coyote.http11.Http11Protocol] (MSC service >>>thread 1-1) JBWEB003001: Coyote HTTP/1.1 initializing on : http-/0.0.0.0:8444 >>>12:40:59,620 INFO [org.apache.coyote.http11.Http11Protocol] (MSC service >>>thread 1-2) JBWEB003001: Coyote HTTP/1.1 initializing on : http-/0.0.0.0:8080 >>>12:40:59,621 INFO [org.apache.coyote.http11.Http11Protocol] (MSC service >>>thread 1-1) JBWEB003000: Coyote HTTP/1.1 starting on: http-/0.0.0.0:8444 >>>12:40:59,627 INFO [org.apache.coyote.http11.Http11Protocol] (MSC service >>>thread 1-2) JBWEB003000: Coyote HTTP/1.1 starting on: http-/0.0.0.0:8080 >>>12:40:59,654 INFO [org.jboss.as.connector.subsystems.datasources] (MSC >>>service thread 1-3) JBAS010400: Bound data source >>>[java:jboss/datasources/ExampleDS] >>>12:40:59,985 INFO [org.jboss.as.server.deployment] (MSC service thread 1-2) >>>JBAS015876: Starting deployment of "activemq-rar.rar" (runtime-name: >>>"activemq-rar.rar") >>>12:40:59,986 INFO [org.jboss.as.server.deployment] (MSC service thread 1-1) >>>JBAS015876: Starting deployment of "hawkular-metrics-api-jaxrs.war" >>>(runtime-name: "hawkular-metrics-api-jaxrs.war") >>>12:40:59,987 INFO [org.jboss.as.server.deployment.scanner] (MSC service >>>thread 1-3) JBAS015012: Started FileSystemDeploymentService for directory >>>/opt/eap/standalone/deployments >>>12:41:00,011 INFO [org.jboss.as.remoting] (MSC service thread 1-2) >>>JBAS017100: Listening on 127.0.0.1:9999 >>>12:41:00,030 WARN [org.jboss.as.domain.http.api] (MSC service thread 1-3) >>>JBAS015102: Unable to load console module for slot eap, disabling console >>>12:41:00,042 INFO [org.jboss.ws.common.management] (MSC service thread 1-4) >>>JBWS022052: Starting JBoss Web Services - Stack CXF Server >>>4.3.5.Final-redhat-3 >>>12:41:00,045 INFO [org.jboss.as.remoting] (MSC service thread 1-4) >>>JBAS017100: Listening on 0.0.0.0:4447 >>>12:41:01,361 INFO [org.jboss.as.connector.deployers.RADeployer] (MSC >>>service thread 1-3) IJ020001: Required license terms for >>>file:/opt/eap/standalone/tmp/vfs/temp/temp912bb9d1372c33d3/activemq-rar.rar-a722089a00658182/contents/ >>>12:41:01,784 WARN [org.jboss.as.dependency.private] (MSC service thread >>>1-1) JBAS015867: Deployment "deployment.hawkular-metrics-api-jaxrs.war" is >>>using a private module ("org.codehaus.jackson.jackson-core-asl:main") which >>>may be changed or removed in future versions without notice. >>>12:41:01,785 WARN [org.jboss.as.dependency.private] (MSC service thread >>>1-1) JBAS015867: Deployment "deployment.hawkular-metrics-api-jaxrs.war" is >>>using a private module ("org.codehaus.jackson.jackson-core-asl:main") which >>>may be changed or removed in future versions without notice. >>>12:41:01,785 WARN [org.jboss.as.dependency.private] (MSC service thread >>>1-1) JBAS015867: Deployment "deployment.hawkular-metrics-api-jaxrs.war" is >>>using a private module ("org.codehaus.jackson.jackson-mapper-asl:main") >>>which may be changed or removed in future versions without notice. >>>12:41:01,785 WARN [org.jboss.as.dependency.private] (MSC service thread >>>1-1) JBAS015867: Deployment "deployment.hawkular-metrics-api-jaxrs.war" is >>>using a private module ("org.codehaus.jackson.jackson-mapper-asl:main") >>>which may be changed or removed in future versions without notice. >>>12:41:01,843 INFO [org.jboss.weld.deployer] (MSC service thread 1-1) >>>JBAS016002: Processing weld deployment hawkular-metrics-api-jaxrs.war >>>12:41:01,941 INFO [org.jboss.weld.deployer] (MSC service thread 1-1) >>>JBAS016005: Starting Services for CDI deployment: >>>hawkular-metrics-api-jaxrs.war >>>12:41:01,986 INFO [org.jboss.weld.Version] (MSC service thread 1-1) >>>WELD-000900 1.1.31 (redhat) >>>12:41:02,013 INFO [org.jboss.weld.deployer] (MSC service thread 1-3) >>>JBAS016008: Starting weld service for deployment >>>hawkular-metrics-api-jaxrs.war >>>12:41:02,820 INFO [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] >>>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics >>>service >>>12:41:02,835 INFO [org.jboss.web] (ServerService Thread Pool -- 51) >>>JBAS018210: Register web context: /hawkular/metrics >>>12:41:02,958 INFO [org.jboss.as.server] (ServerService Thread Pool -- 28) >>>JBAS015859: Deployed "hawkular-metrics-api-jaxrs.war" (runtime-name : >>>"hawkular-metrics-api-jaxrs.war") >>>12:41:02,959 INFO [org.jboss.as.server] (ServerService Thread Pool -- 28) >>>JBAS015859: Deployed "activemq-rar.rar" (runtime-name : "activemq-rar.rar") >>>12:41:02,994 INFO [org.jboss.as] (Controller Boot Thread) JBAS015961: Http >>>management interface listening on http://127.0.0.1:9990/management >>>12:41:02,994 INFO [org.jboss.as] (Controller Boot Thread) JBAS015951: Admin >>>console listening on http://127.0.0.1:9990 >>>12:41:02,994 INFO [org.jboss.as] (Controller Boot Thread) JBAS015874: JBoss >>>EAP 6.4.4.GA (AS 7.5.4.Final-redhat-4) started in 6593ms - Started 267 of >>>303 services (62 services are lazy, passive or on-demand) >>>12:41:02,999 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] >>>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to >>>Cassandra cluster - assuming its not up yet: hawkular-cassandra: unknown >>>error >>>12:41:02,999 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] >>>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [1] Retrying connecting >>>to Cassandra cluster in [1]s... >>>12:41:04,000 INFO [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] >>>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics >>>service >>>12:41:04,001 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] >>>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to >>>Cassandra cluster - assuming its not up yet: hawkular-cassandra >>>12:41:04,001 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] >>>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [2] Retrying connecting >>>to Cassandra cluster in [2]s... >>>12:41:06,002 INFO [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] >>>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics >>>service >>>12:41:06,002 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] >>>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to >>>Cassandra cluster - assuming its not up yet: hawkular-cassandra >>>12:41:06,003 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] >>>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [3] Retrying connecting >>>to Cassandra cluster in [3]s... >>>12:41:08,203 INFO [org.jboss.resteasy.cdi.i18n] (http-/0.0.0.0:8080-2) >>>RESTEASY006050: Found BeanManager at java:comp/BeanManager >>>12:41:08,299 INFO [org.hibernate.validator.internal.util.Version] >>>(http-/0.0.0.0:8080-2) HV000001: Hibernate Validator 4.3.2.Final-redhat-2 >>>12:41:08,418 INFO [org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp] >>>(http-/0.0.0.0:8080-2) HAWKMETRICS200001: Hawkular Metrics starting >>>12:41:08,421 INFO [org.jboss.resteasy.resteasy_jaxrs.i18n] >>>(http-/0.0.0.0:8080-2) RESTEASY000320: Deploying >>>javax.ws.rs.core.Application: class >>>org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp$Proxy$_$$_WeldClientProxy >>>12:41:08,421 INFO [org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp] >>>(http-/0.0.0.0:8080-2) HAWKMETRICS200001: Hawkular Metrics starting >>>12:41:08,422 INFO [org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp] >>>(http-/0.0.0.0:8080-2) Virtual clock is disabled >>>12:41:08,422 INFO [org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp] >>>(http-/0.0.0.0:8080-2) Virtual clock is disabled >>>12:41:09,003 INFO [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] >>>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics >>>service >>>12:41:09,004 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] >>>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to >>>Cassandra cluster - assuming its not up yet: hawkular-cassandra >>>12:41:09,004 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] >>>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [4] Retrying connecting >>>to Cassandra cluster in [4]s... >>>12:41:13,004 INFO [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] >>>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics >>>service >>>12:41:13,123 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] >>>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to >>>Cassandra cluster - assuming its not up yet: hawkular-cassandra: unknown >>>error >>>12:41:13,124 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] >>>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [5] Retrying connecting >>>to Cassandra cluster in [1]s... >>>12:41:14,124 INFO [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] >>>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics >>>service >>>12:41:14,124 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] >>>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to >>>Cassandra cluster - assuming its not up yet: hawkular-cassandra >>>12:41:14,125 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] >>>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [6] Retrying connecting >>>to Cassandra cluster in [2]s... >>>12:41:16,125 INFO [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] >>>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics >>>service >>>12:41:16,126 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] >>>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to >>>Cassandra cluster - assuming its not up yet: hawkular-cassandra >>>12:41:16,126 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] >>>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [7] Retrying connecting >>>to Cassandra cluster in [3]s... >>>12:41:19,126 INFO [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] >>>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics >>>service >>>12:41:19,127 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] >>>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to >>>Cassandra cluster - assuming its not up yet: hawkular-cassandra >>>12:41:19,127 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] >>>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [8] Retrying connecting >>>to Cassandra cluster in [4]s... >>>12:41:23,128 INFO [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] >>>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics >>>service >>>12:41:23,249 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] >>>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to >>>Cassandra cluster - assuming its not up yet: hawkular-cassandra: unknown >>>error >>>12:41:23,249 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] >>>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [9] Retrying connecting >>>to Cassandra cluster in [1]s... >>>12:41:24,250 INFO [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] >>>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics >>>service >>>12:41:24,250 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] >>>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to >>>Cassandra cluster - assuming its not up yet: hawkular-cassandra >>>12:41:24,250 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] >>>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [10] Retrying >>>connecting to Cassandra cluster in [2]s... >>>12:41:26,251 INFO [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] >>>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics >>>service >>>12:41:26,251 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] >>>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to >>>Cassandra cluster - assuming its not up yet: hawkular-cassandra >>>12:41:26,251 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] >>>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [11] Retrying >>>connecting to Cassandra cluster in [3]s... >>>12:41:29,252 INFO [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] >>>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics >>>service >>>12:41:29,252 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] >>>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to >>>Cassandra cluster - assuming its not up yet: hawkular-cassandra >>>12:41:29,253 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] >>>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [12] Retrying >>>connecting to Cassandra cluster in [4]s... >>>12:41:33,253 INFO [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] >>>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics >>>service >>>12:41:33,376 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] >>>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to >>>Cassandra cluster - assuming its not up yet: hawkular-cassandra: unknown >>>error >>>12:41:33,377 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] >>>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [13] Retrying >>>connecting to Cassandra cluster in [1]s... >>>12:41:34,377 INFO [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] >>>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics >>>service >>>12:41:34,378 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] >>>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to >>>Cassandra cluster - assuming its not up yet: hawkular-cassandra >>>12:41:34,378 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] >>>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [14] Retrying >>>connecting to Cassandra cluster in [2]s... >>>12:41:36,378 INFO [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] >>>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics >>>service >>>12:41:36,379 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] >>>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to >>>Cassandra cluster - assuming its not up yet: hawkular-cassandra >>>12:41:36,379 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] >>>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [15] Retrying >>>connecting to Cassandra cluster in [3]s... >>>12:41:39,379 INFO [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] >>>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics >>>service >>>12:41:39,380 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] >>>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to >>>Cassandra cluster - assuming its not up yet: hawkular-cassandra >>>12:41:39,380 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] >>>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [16] Retrying >>>connecting to Cassandra cluster in [4]s... >>>12:41:43,380 INFO [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] >>>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics >>>service >>>12:41:43,503 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] >>>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to >>>Cassandra cluster - assuming its not up yet: hawkular-cassandra: unknown >>>error >>>12:41:43,504 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] >>>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [17] Retrying >>>connecting to Cassandra cluster in [1]s... >>>12:41:44,504 INFO [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] >>>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics >>>service >>>12:41:44,505 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] >>>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to >>>Cassandra cluster - assuming its not up yet: hawkular-cassandra >>>12:41:44,505 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] >>>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [18] Retrying >>>connecting to Cassandra cluster in [2]s... >>> >>>-- >>>Srinivas Kotaru >>> >>>On 6/14/16, 6:06 AM, "Matt Wringe" <[email protected]> wrote: >>> >>>>----- Original Message ----- >>>>> From: "Srinivas Naga Kotaru (skotaru)" <[email protected]> >>>>> To: "Matt Wringe" <[email protected]> >>>>> Cc: [email protected] >>>>> Sent: Monday, June 13, 2016 7:26:06 PM >>>>> Subject: Re: Metrics deployment >>>>> >>>>> Matt >>>>> >>>>> PV issue resolved. Was able to to see PV successfully bounded and Casandra >>>>> container has been running. However, it seems puzzle not fully yet solved. >>>> >>>>Are you sure the OpenShift DNS server is running? >>>> >>>>If you are running OSE 3.1, can you please follow this >>>>https://access.redhat.com/solutions/2329131 and see if you are now seeing >>>>errors in the Hawkular Metrics logs (essentially just run `oc exec >>>>hawkular-metrics-xxxxx cat /opt/eap/standalone/log/server.log`) >>>> >>>>> >>>>> I could see other container(heapster) not coming up, and seeing below >>>>> errors >>>>> >>>>> [skotaru@l3imas-id2-01 metrics]$ oc logs -f heapster-fnkdc >>>>> Endpoint Check in effect. Checking >>>>> https://hawkular-metrics:443/hawkular/metrics/status >>>>> Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. >>>>> Curl exit code: 6. Status Code 000 >>>>> 'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible >>>>> [HTTP status code: 000. Curl exit code 6]. Retrying. >>>>> Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. >>>>> Curl exit code: 6. Status Code 000 >>>>> 'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible >>>>> [HTTP status code: 000. Curl exit code 6]. Retrying. >>>>> >>>>> >>>>> # oc get pv >>>>> pv-5gb-0011 5Gi RWO Bound >>>>> openshift-infra/metrics-cassandra-1 22m >>>>> >>>>> >>>>> $ oc get pods >>>>> NAME READY STATUS RESTARTS AGE >>>>> hawkular-cassandra-1-2pzd7 1/1 Running 0 20m >>>>> hawkular-metrics-mf5qf 0/1 Running 7 20m >>>>> heapster-fnkdc 0/1 Error 6 20m >>>>> metrics-deployer-cvep0 0/1 Completed 0 21m >>>>> >>>>> # oc logs -f hawkular-metrics-mf5qf >>>>> >>>>> 19:20:00,819 INFO [org.xnio] (MSC service thread 1-2) XNIO Version >>>>> 3.0.14.GA-redhat-1 >>>>> 19:20:00,831 INFO [org.jboss.as.server] (Controller Boot Thread) >>>>> JBAS015888: >>>>> Creating http management service using socket-binding (management-http) >>>>> 19:20:00,834 INFO [org.xnio.nio] (MSC service thread 1-2) XNIO NIO >>>>> Implementation Version 3.0.14.GA-redhat-1 >>>>> 19:20:00,844 INFO [org.jboss.remoting] (MSC service thread 1-2) JBoss >>>>> Remoting version 3.3.5.Final-redhat-1 >>>>> >>>>> $ oc logs -f heapster-fnkdc >>>>> Endpoint Check in effect. Checking >>>>> https://hawkular-metrics:443/hawkular/metrics/status >>>>> Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. >>>>> Curl exit code: 6. Status Code 000 >>>>> 'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible >>>>> [HTTP status code: 000. Curl exit code 6]. Retrying. >>>>> Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. >>>>> Curl exit code: 6. Status Code 000 >>>>> 'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible >>>>> [HTTP status code: 000. Curl exit code 6]. Retrying. >>>>> Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. >>>>> Curl exit code: 6. Status Code 000 >>>>> >>>>> $ oc logs -f hawkular-cassandra-1-2pzd7 >>>>> INFO 23:00:24 Starting listening for CQL clients on >>>>> hawkular-cassandra-1-2pzd7/10.1.6.2:9042... >>>>> INFO 23:00:24 Binding thrift service to >>>>> hawkular-cassandra-1-2pzd7/10.1.6.2:9160 >>>>> INFO 23:00:24 enabling encrypted thrift connections between client and >>>>> server >>>>> INFO 23:00:24 Listening for thrift clients... >>>>> INFO 23:00:26 Created default superuser role 'cassandra' >>>>> >>>>> # oc get svc >>>>> NAME CLUSTER-IP EXTERNAL-IP PORT(S) >>>>> AGE >>>>> hawkular-cassandra 172.30.2.13 <none> >>>>> 9042/TCP,9160/TCP,7000/TCP,7001/TCP 25m >>>>> hawkular-cassandra-nodes None <none> >>>>> 9042/TCP,9160/TCP,7000/TCP,7001/TCP 25m >>>>> hawkular-metrics 172.30.117.176 <none> 443/TCP >>>>> 25m >>>>> heapster 172.30.107.135 <none> 80/TCP >>>>> 25m >>>>> >>>>> #curl -I 172.30.117.176:443//hawkular/metrics/status >>>>> >>>>> HTTP/1.1 504 Gateway Timeout >>>>> Mime-Version: 1.0 >>>>> Date: Mon, 13 Jun 2016 23:25:47 GMT >>>>> Content-Type: text/html >>>>> Connection: keep-alive >>>>> Proxy-Connection: keep-alive >>>>> Content-Length: 1572 >>>>> >>>>> -- >>>>> Srinivas Kotaru >>>>> >>>>> On 6/13/16, 2:33 PM, "Srinivas Naga Kotaru (skotaru)" <[email protected]> >>>>> wrote: >>>>> >>>>> >Matt >>>>> > >>>>> >That is good catch. I ran without USE_PERSISTENT_STORAGE=false and >>>>> >working >>>>> > >>>>> >I adjusted PV to 5Gi and reran. Will update progress. >>>>> > >>>>> >Thanks you for your help so far. >>>>> > >>>>> >-- >>>>> >Srinivas Kotaru >>>>> > >>>>> >On 6/13/16, 2:27 PM, "Matt Wringe" <[email protected]> wrote: >>>>> > >>>>> >> >>>>> >> >>>>> >>----- Original Message ----- >>>>> >>> From: "Srinivas Naga Kotaru (skotaru)" <[email protected]> >>>>> >>> To: "Matt Wringe" <[email protected]> >>>>> >>> Cc: [email protected] >>>>> >>> Sent: Monday, June 13, 2016 5:21:01 PM >>>>> >>> Subject: Re: Metrics deployment >>>>> >>> >>>>> >>> Oh ok >>>>> >>> >>>>> >>> Am using PV for metrics >>>>> >>> >>>>> >>> description: "The persistent volume size for each of the Cassandra >>>>> >>> nodes" >>>>> >>> name: CASSANDRA_PV_SIZE >>>>> >>> value: "10Gi" >>>>> >>> >>>>> >>> oc get pv >>>>> >>> NAME CAPACITY ACCESSMODES STATUS CLAIM >>>>> >>> REASON >>>>> >>> AGE >>>>> >>> pv-1gb-001 1Gi RWO Available >>>>> >>> 4d >>>>> >>> pv-1gb-002 1Gi RWO Available >>>>> >>> 4d >>>>> >>> pv-1gb-003 1Gi RWO Available >>>>> >>> 4d >>>>> >>> pv-1gb-004 1Gi RWO Bound thlatt/mongodb >>>>> >>> 4d >>>>> >>> pv-1gb-005 1Gi RWO Available >>>>> >>> 4d >>>>> >>> pv-2gb-0010 2Gi RWO Available >>>>> >>> 4d >>>>> >>> pv-2gb-006 2Gi RWO Available >>>>> >>> 4d >>>>> >>> pv-2gb-007 2Gi RWO Available >>>>> >>> 4d >>>>> >>> pv-2gb-008 2Gi RWO Available >>>>> >>> 4d >>>>> >>> pv-2gb-009 2Gi RWO Available >>>>> >>> 4d >>>>> >>> pv-5gb-0011 5Gi RWO Available >>>>> >>> 4d >>>>> >>> pv-5gb-0012 5Gi RWO Available >>>>> >>> 4d >>>>> >>> pv-5gb-0013 5Gi RWO Available >>>>> >>> 4d >>>>> >>> pv-5gb-0014 5Gi RWO Available >>>>> >>> 4d >>>>> >>> pv-5gb-0015 5Gi RWO Available >>>>> >>> 4d >>>>> >>> >>>>> >>> am running with below command >>>>> >>> >>>>> >>> $ oc new-app -f metrics-deployer.yaml ( hardcoded HOSTNAME, >>>>> >>> MASTER_API >>>>> >>> and >>>>> >>> PV info so not passing any parameters) >>>>> >>> >>>>> >> >>>>> >>I would suspect that Cassandra is blocked because its waiting for 10Gi >>>>> >>PV >>>>> >>to become available, and none of the PV listed above are big enough. >>>>> >> >>>>> >>> >>>>> >>> -- >>>>> >>> Srinivas Kotaru >>>>> >>> >>>>> >>> On 6/13/16, 2:12 PM, "Matt Wringe" <[email protected]> wrote: >>>>> >>> >>>>> >>> >----- Original Message ----- >>>>> >>> >> From: "Srinivas Naga Kotaru (skotaru)" <[email protected]> >>>>> >>> >> To: "Matt Wringe" <[email protected]> >>>>> >>> >> Cc: [email protected] >>>>> >>> >> Sent: Monday, June 13, 2016 4:55:55 PM >>>>> >>> >> Subject: Re: Metrics deployment >>>>> >>> >> >>>>> >>> >> Matt >>>>> >>> >> >>>>> >>> >> Thanks for looking into. I rerun the setup, but had the same issue >>>>> >>> >> >>>>> >>> >> # oc get pods >>>>> >>> >> NAME READY STATUS RESTARTS >>>>> >>> >> AGE >>>>> >>> >> hawkular-cassandra-1-y2egy 0/1 ContainerCreating 0 >>>>> >>> >> 5m >>>>> >>> >> hawkular-metrics-4b16f 0/1 Running 1 >>>>> >>> >> 4m >>>>> >>> >> heapster-x2gj2 0/1 Running 2 >>>>> >>> >> 4m >>>>> >>> >> metrics-deployer-9v7vc 0/1 Completed 0 >>>>> >>> >> 6m >>>>> >>> >> >>>>> >>> >> $ oc logs -f hawkular-cassandra-1-y2egy >>>>> >>> >> Error from server: container "hawkular-cassandra-1" in pod >>>>> >>> >> "hawkular-cassandra-1-y2egy" is waiting to start: ContainerCreating >>>>> >>> > >>>>> >>> >Ok, so it looks like something is blocking the Cassandra pod from >>>>> >>> >starting. >>>>> >>> > >>>>> >>> >If you are using persistent storage, Cassandra will not start until >>>>> >>> >the >>>>> >>> >PV >>>>> >>> >is available. There may be some more information about Cassandra in >>>>> >>> >the >>>>> >>> >pod >>>>> >>> >section of the console under events. >>>>> >>> > >>>>> >>> >What command did you use when deploying the deployer? >>>>> >>> > >>>>> >>> >> >>>>> >>> >> $ oc logs -f hawkular-metrics-4b16f >>>>> >>> >> >>>>> >>> >> 16:54:25,703 DEBUG [org.jboss.as.config] (MSC service thread 1-4) >>>>> >>> >> VM >>>>> >>> >> Arguments: -Duser.home=/home/jboss -Duser.name=jboss -D[Standalone] >>>>> >>> >> -XX:+UseCompressedOops -verbose:gc >>>>> >>> >> -Xloggc:/opt/eap/standalone/log/gc.log >>>>> >>> >> -XX:+PrintGCDetails -XX:+PrintGCDateStamps >>>>> >>> >> -XX:+UseGCLogFileRotation >>>>> >>> >> -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=3M >>>>> >>> >> -XX:-TraceClassUnloading >>>>> >>> >> -Xms1303m -Xmx1303m -XX:MaxPermSize=256m >>>>> >>> >> -Djava.net.preferIPv4Stack=true >>>>> >>> >> -Djboss.modules.system.pkgs=org.jboss.logmanager >>>>> >>> >> -Djava.awt.headless=true >>>>> >>> >> -Djboss.modules.policy-permissions=true >>>>> >>> >> -Xbootclasspath/p:/opt/eap/jboss-modules.jar:/opt/eap/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-1.5.4.Final-redhat-1.jar:/opt/eap/modules/system/layers/base/org/jboss/logmanager/ext/main/javax.json-1.0.4.jar:/opt/eap/modules/system/layers/base/org/jboss/logmanager/ext/main/jboss-logmanager-ext-1.0.0.Alpha2-redhat-1.jar >>>>> >>> >> -Djava.util.logging.manager=org.jboss.logmanager.LogManager >>>>> >>> >> -javaagent:/opt/eap/jolokia.jar=port=8778,protocol=https,caCert=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt,clientPrincipal=cn=system:master-proxy,useSslClientAuthentication=true,extraClientCheck=true,host=0.0.0.0,discoveryEnabled=false >>>>> >>> >> -Djava.security.egd=file:/dev/./urandom >>>>> >>> >> -Dorg.jboss.boot.log.file=/opt/eap/standalone/log/server.log >>>>> >>> >> -Dlogging.configuration=file:/opt/eap/standalone/configuration/logging.properties >>>>> >>> >> 16:54:27,079 INFO [org.xnio] (MSC service thread 1-3) XNIO Version >>>>> >>> >> 3.0.14.GA-redhat-1 >>>>> >>> >> 16:54:27,083 INFO [org.xnio.nio] (MSC service thread 1-3) XNIO NIO >>>>> >>> >> Implementation Version 3.0.14.GA-redhat-1 >>>>> >>> >> 16:54:27,101 INFO [org.jboss.as.server] (Controller Boot Thread) >>>>> >>> >> JBAS015888: >>>>> >>> >> Creating http management service using socket-binding >>>>> >>> >> (management-http) >>>>> >>> >> 16:54:27,104 INFO [org.jboss.remoting] (MSC service thread 1-3) >>>>> >>> >> JBoss >>>>> >>> >> Remoting version 3.3.5.Final-redhat-1 >>>>> >>> >> >>>>> >>> >> $ oc logs -f heapster-x2gj2 >>>>> >>> >> Endpoint Check in effect. Checking >>>>> >>> >> https://hawkular-metrics:443/hawkular/metrics/status >>>>> >>> >> Could not connect to >>>>> >>> >> https://hawkular-metrics:443/hawkular/metrics/status. >>>>> >>> >> Curl exit code: 6. Status Code 000 >>>>> >>> >> 'https://hawkular-metrics:443/hawkular/metrics/status' is not >>>>> >>> >> accessible >>>>> >>> >> [HTTP status code: 000. Curl exit code 6]. Retrying. >>>>> >>> >> Could not connect to >>>>> >>> >> https://hawkular-metrics:443/hawkular/metrics/status. >>>>> >>> >> Curl exit code: 6. Status Code 000 >>>>> >>> >> 'https://hawkular-metrics:443/hawkular/metrics/status' is not >>>>> >>> >> accessible >>>>> >>> >> [HTTP status code: 000. Curl exit code 6]. Retrying. >>>>> >>> >> Could not connect to >>>>> >>> >> https://hawkular-metrics:443/hawkular/metrics/status. >>>>> >>> >> Curl exit code: 6. Status Code 000 >>>>> >>> >> >>>>> >>> >> >>>>> >>> >> $ oc logs -f metrics-deployer-9v7vc >>>>> >>> >> >>>>> >>> >> ++ oc create -f - >>>>> >>> >> serviceaccount "heapster" created >>>>> >>> >> service "heapster" created >>>>> >>> >> replicationcontroller "heapster" created >>>>> >>> >> + echo 'Success!' >>>>> >>> >> Success! >>>>> >>> >> >>>>> >>> >> -- >>>>> >>> >> Srinivas Kotaru >>>>> >>> >> >>>>> >>> >> On 6/13/16, 1:49 PM, "Matt Wringe" <[email protected]> wrote: >>>>> >>> >> >>>>> >>> >> > >>>>> >>> >> > >>>>> >>> >> >----- Original Message ----- >>>>> >>> >> >> From: "Srinivas Naga Kotaru (skotaru)" <[email protected]> >>>>> >>> >> >> To: [email protected] >>>>> >>> >> >> Sent: Monday, June 13, 2016 3:58:12 PM >>>>> >>> >> >> Subject: Metrics deployment >>>>> >>> >> >> >>>>> >>> >> >> >>>>> >>> >> >> >>>>> >>> >> >> Hi >>>>> >>> >> >> >>>>> >>> >> >> >>>>> >>> >> >> >>>>> >>> >> >> Am trying to configure metrics in our newly installed clusters. >>>>> >>> >> >> Am >>>>> >>> >> >> seeing >>>>> >>> >> >> below errors once metrics-deploy script was successful. I used >>>>> >>> >> >> our >>>>> >>> >> >> environment specific HAWKULAR_METRICS_HOSTNAME and MASTER_URL >>>>> >>> >> >> >>>>> >>> >> >> >>>>> >>> >> >> >>>>> >>> >> >> # oc new-app -f metrics-deployer.yaml >>>>> >>> >> >> >>>>> >>> >> >> >>>>> >>> >> >> >>>>> >>> >> >> Note: customized, CASSANDARA PV, MASTER_URL, and >>>>> >>> >> >> HAWKULAR_METRICS_HOSTNAME >>>>> >>> >> >> ( >>>>> >>> >> >> hard coded as values) >>>>> >>> >> >> >>>>> >>> >> >> >>>>> >>> >> >> >>>>> >>> >> >> template "hawkular-heapster" created >>>>> >>> >> >> >>>>> >>> >> >> Deploying the Heapster component >>>>> >>> >> >> >>>>> >>> >> >> ++ echo 'Deploying the Heapster component' >>>>> >>> >> >> >>>>> >>> >> >> ++ '[' -n '' ']' >>>>> >>> >> >> >>>>> >>> >> >> ++ oc create -f - >>>>> >>> >> >> >>>>> >>> >> >> ++ oc process hawkular-heapster -v >>>>> >>> >> >> IMAGE_PREFIX=registry.access.redhat.com/openshift3/,IMAGE_VERSION=latest,MASTER_URL=https://lae3-alln-int-idev01.cisco.com:443,NODE_ID=nodename >>>>> >>> >> >> >>>>> >>> >> >> serviceaccount "heapster" created >>>>> >>> >> >> >>>>> >>> >> >> service "heapster" created >>>>> >>> >> >> >>>>> >>> >> >> replicationcontroller "heapster" created >>>>> >>> >> >> >>>>> >>> >> >> + echo 'Success!' >>>>> >>> >> >> >>>>> >>> >> >> Success! >>>>> >>> >> >> >>>>> >>> >> >> >>>>> >>> >> >> >>>>> >>> >> >> # oc get pods >>>>> >>> >> >> >>>>> >>> >> >> NAME READY STATUS RESTARTS AGE >>>>> >>> >> >> >>>>> >>> >> >> hawkular-cassandra-1-9nzio 0/1 ContainerCreating 0 4m >>>>> >>> >> >> >>>>> >>> >> >> hawkular-metrics-hi7mb 0/1 Running 1 4m >>>>> >>> >> >> >>>>> >>> >> >> heapster-e8gbu 0/1 Running 2 4m >>>>> >>> >> >> >>>>> >>> >> >> metrics-deployer-64703 0/1 ContainerCreating 0 3s >>>>> >>> >> >> >>>>> >>> >> >> metrics-deployer-cd1nf 0/1 Completed 0 5m >>>>> >>> >> >> >>>>> >>> >> > >>>>> >>> >> >It looks like none of your containers are fully up and running >>>>> >>> >> >yet. >>>>> >>> >> > >>>>> >>> >> >Without Cassandra running, Hawkular Metrics will not run, and >>>>> >>> >> >Heapster >>>>> >>> >> >will >>>>> >>> >> >wait until Hawkular Metrics is fully running. >>>>> >>> >> > >>>>> >>> >> >Do you see anything in the Cassandra logs? The first step will be >>>>> >>> >> >to >>>>> >>> >> >get >>>>> >>> >> >Cassandra running properly. >>>>> >>> >> > >>>>> >>> >> >> >>>>> >>> >> >> >>>>> >>> >> >> >>>>> >>> >> >> $ oc logs -f heapster-e8gbu >>>>> >>> >> >> >>>>> >>> >> >> Endpoint Check in effect. Checking >>>>> >>> >> >> https://hawkular-metrics:443/hawkular/metrics/status >>>>> >>> >> >> >>>>> >>> >> >> Could not connect to >>>>> >>> >> >> https://hawkular-metrics:443/hawkular/metrics/status. >>>>> >>> >> >> Curl exit code: 6. Status Code 000 >>>>> >>> >> >> >>>>> >>> >> >> 'https://hawkular-metrics:443/hawkular/metrics/status' is not >>>>> >>> >> >> accessible >>>>> >>> >> >> [HTTP status code: 000. Curl exit code 6]. Retrying. >>>>> >>> >> >> >>>>> >>> >> >> Could not connect to >>>>> >>> >> >> https://hawkular-metrics:443/hawkular/metrics/status. >>>>> >>> >> >> Curl exit code: 6. Status Code 000 >>>>> >>> >> > >>>>> >>> >> >Heapster waits until Hawkular Metrics is started before trying to >>>>> >>> >> >push >>>>> >>> >> >metrics to it. The issue that you are seeing is because Heapster >>>>> >>> >> >could >>>>> >>> >> >not >>>>> >>> >> >properly connect to Hawkular Metrics. Until the Hawkular Metrics >>>>> >>> >> >service >>>>> >>> >> >is >>>>> >>> >> >fully up, Heapster will not be able to connect to it. >>>>> >>> >> > >>>>> >>> >> > >>>>> >>> >> >> >>>>> >>> >> >> >>>>> >>> >> >> >>>>> >>> >> >> >>>>> >>> >> >> What is the wrong? Why it checking just hawkular-metrics rather >>>>> >>> >> >> full >>>>> >>> >> >> routing >>>>> >>> >> >> URL which was provided as HAWKULAR_METRICS_HOSTNAME >>>>> >>> >> > >>>>> >>> >> >The Hawkular Metrics service has two hostnames: the internal >>>>> >>> >> >hostname >>>>> >>> >> >used >>>>> >>> >> >by the internal components (eg 'hawkular-metrics') and the >>>>> >>> >> >external >>>>> >>> >> >hostname (eg what is configured via HAWKULAR_METRICS_HOSTNAME). >>>>> >>> >> >The >>>>> >>> >> >OpenShift dns server will resolve hostnames to the name of >>>>> >>> >> >services, >>>>> >>> >> >which >>>>> >>> >> >is where the internal 'hawkular-metrics' comes from. >>>>> >>> >> > >>>>> >>> >> >> >>>>> >>> >> >> >>>>> >>> >> >> >>>>> >>> >> >> >>>>> >>> >> >> >>>>> >>> >> >> >>>>> >>> >> >> >>>>> >>> >> >> >>>>> >>> >> >> >>>>> >>> >> >> >>>>> >>> >> >> -- >>>>> >>> >> >> >>>>> >>> >> >> >>>>> >>> >> >> Srinivas Kotaru >>>>> >>> >> >> >>>>> >>> >> >> _______________________________________________ >>>>> >>> >> >> users mailing list >>>>> >>> >> >> [email protected] >>>>> >>> >> >> http://lists.openshift.redhat.com/openshiftmm/listinfo/users >>>>> >>> >> >> >>>>> >>> >> >>>>> >>> >> >>>>> >>> >>>>> >>> >>>>> > >>>>> >>>>> >>> >> > _______________________________________________ users mailing list [email protected] http://lists.openshift.redhat.com/openshiftmm/listinfo/users
