Matt
DNS service is working
~ dig +short @master-01 kubernetes.default.svc.cluster.local
172.30.0.1
~ dig +short @master-01 jenkins.alln-test.svc.cluster.local
172.30.85.148
~ dig +short @master-01 cakephp-example.alln-test.svc.cluster.local
172.30.31.6
I captured hawkular-metrics, it shows the problem. It seems to be unable to
connect Cassandra cluster
# oc exec hawkular-metrics-qldd8 cat /opt/eap/standalone/log/server.log
12:40:58,848 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread
Pool -- 31) JBAS010280: Activating Infinispan subsystem.
12:40:58,898 INFO [org.jboss.as.webservices] (ServerService Thread Pool -- 48)
JBAS015537: Activating WebServices Extension
12:40:58,901 WARN [org.jboss.as.txn] (ServerService Thread Pool -- 46)
JBAS010153: Node identifier property is set to the default value. Please make
sure it is unique.
12:40:58,902 INFO [org.jboss.as.security] (ServerService Thread Pool -- 44)
JBAS013371: Activating Security Subsystem
12:40:58,912 INFO [org.jboss.as.naming] (ServerService Thread Pool -- 39)
JBAS011800: Activating Naming Subsystem
12:40:58,928 INFO [org.jboss.as.jsf] (ServerService Thread Pool -- 37)
JBAS012615: Activated the following JSF Implementations: [main, 1.2]
12:40:58,939 INFO [org.jboss.as.connector.logging] (MSC service thread 1-2)
JBAS010408: Starting JCA Subsystem (IronJacamar 1.0.32.Final-redhat-1)
12:40:58,981 INFO [org.jboss.as.security] (MSC service thread 1-2) JBAS013370:
Current PicketBox version=4.1.1.Final-redhat-1
12:40:58,961 INFO [org.jboss.as.connector.subsystems.datasources]
(ServerService Thread Pool -- 27) JBAS010403: Deploying JDBC-compliant driver
class org.h2.Driver (version 1.3)
12:40:59,039 INFO [org.jboss.as.naming] (MSC service thread 1-2) JBAS011802:
Starting Naming Service
12:40:59,052 INFO [org.jboss.as.mail.extension] (MSC service thread 1-4)
JBAS015400: Bound mail session [java:jboss/mail/Default]
12:40:59,614 INFO [org.apache.coyote.http11.Http11Protocol] (MSC service
thread 1-1) JBWEB003001: Coyote HTTP/1.1 initializing on : http-/0.0.0.0:8444
12:40:59,620 INFO [org.apache.coyote.http11.Http11Protocol] (MSC service
thread 1-2) JBWEB003001: Coyote HTTP/1.1 initializing on : http-/0.0.0.0:8080
12:40:59,621 INFO [org.apache.coyote.http11.Http11Protocol] (MSC service
thread 1-1) JBWEB003000: Coyote HTTP/1.1 starting on: http-/0.0.0.0:8444
12:40:59,627 INFO [org.apache.coyote.http11.Http11Protocol] (MSC service
thread 1-2) JBWEB003000: Coyote HTTP/1.1 starting on: http-/0.0.0.0:8080
12:40:59,654 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service
thread 1-3) JBAS010400: Bound data source [java:jboss/datasources/ExampleDS]
12:40:59,985 INFO [org.jboss.as.server.deployment] (MSC service thread 1-2)
JBAS015876: Starting deployment of "activemq-rar.rar" (runtime-name:
"activemq-rar.rar")
12:40:59,986 INFO [org.jboss.as.server.deployment] (MSC service thread 1-1)
JBAS015876: Starting deployment of "hawkular-metrics-api-jaxrs.war"
(runtime-name: "hawkular-metrics-api-jaxrs.war")
12:40:59,987 INFO [org.jboss.as.server.deployment.scanner] (MSC service thread
1-3) JBAS015012: Started FileSystemDeploymentService for directory
/opt/eap/standalone/deployments
12:41:00,011 INFO [org.jboss.as.remoting] (MSC service thread 1-2) JBAS017100:
Listening on 127.0.0.1:9999
12:41:00,030 WARN [org.jboss.as.domain.http.api] (MSC service thread 1-3)
JBAS015102: Unable to load console module for slot eap, disabling console
12:41:00,042 INFO [org.jboss.ws.common.management] (MSC service thread 1-4)
JBWS022052: Starting JBoss Web Services - Stack CXF Server 4.3.5.Final-redhat-3
12:41:00,045 INFO [org.jboss.as.remoting] (MSC service thread 1-4) JBAS017100:
Listening on 0.0.0.0:4447
12:41:01,361 INFO [org.jboss.as.connector.deployers.RADeployer] (MSC service
thread 1-3) IJ020001: Required license terms for
file:/opt/eap/standalone/tmp/vfs/temp/temp912bb9d1372c33d3/activemq-rar.rar-a722089a00658182/contents/
12:41:01,784 WARN [org.jboss.as.dependency.private] (MSC service thread 1-1)
JBAS015867: Deployment "deployment.hawkular-metrics-api-jaxrs.war" is using a
private module ("org.codehaus.jackson.jackson-core-asl:main") which may be
changed or removed in future versions without notice.
12:41:01,785 WARN [org.jboss.as.dependency.private] (MSC service thread 1-1)
JBAS015867: Deployment "deployment.hawkular-metrics-api-jaxrs.war" is using a
private module ("org.codehaus.jackson.jackson-core-asl:main") which may be
changed or removed in future versions without notice.
12:41:01,785 WARN [org.jboss.as.dependency.private] (MSC service thread 1-1)
JBAS015867: Deployment "deployment.hawkular-metrics-api-jaxrs.war" is using a
private module ("org.codehaus.jackson.jackson-mapper-asl:main") which may be
changed or removed in future versions without notice.
12:41:01,785 WARN [org.jboss.as.dependency.private] (MSC service thread 1-1)
JBAS015867: Deployment "deployment.hawkular-metrics-api-jaxrs.war" is using a
private module ("org.codehaus.jackson.jackson-mapper-asl:main") which may be
changed or removed in future versions without notice.
12:41:01,843 INFO [org.jboss.weld.deployer] (MSC service thread 1-1)
JBAS016002: Processing weld deployment hawkular-metrics-api-jaxrs.war
12:41:01,941 INFO [org.jboss.weld.deployer] (MSC service thread 1-1)
JBAS016005: Starting Services for CDI deployment: hawkular-metrics-api-jaxrs.war
12:41:01,986 INFO [org.jboss.weld.Version] (MSC service thread 1-1)
WELD-000900 1.1.31 (redhat)
12:41:02,013 INFO [org.jboss.weld.deployer] (MSC service thread 1-3)
JBAS016008: Starting weld service for deployment hawkular-metrics-api-jaxrs.war
12:41:02,820 INFO [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics
service
12:41:02,835 INFO [org.jboss.web] (ServerService Thread Pool -- 51)
JBAS018210: Register web context: /hawkular/metrics
12:41:02,958 INFO [org.jboss.as.server] (ServerService Thread Pool -- 28)
JBAS015859: Deployed "hawkular-metrics-api-jaxrs.war" (runtime-name :
"hawkular-metrics-api-jaxrs.war")
12:41:02,959 INFO [org.jboss.as.server] (ServerService Thread Pool -- 28)
JBAS015859: Deployed "activemq-rar.rar" (runtime-name : "activemq-rar.rar")
12:41:02,994 INFO [org.jboss.as] (Controller Boot Thread) JBAS015961: Http
management interface listening on http://127.0.0.1:9990/management
12:41:02,994 INFO [org.jboss.as] (Controller Boot Thread) JBAS015951: Admin
console listening on http://127.0.0.1:9990
12:41:02,994 INFO [org.jboss.as] (Controller Boot Thread) JBAS015874: JBoss
EAP 6.4.4.GA (AS 7.5.4.Final-redhat-4) started in 6593ms - Started 267 of 303
services (62 services are lazy, passive or on-demand)
12:41:02,999 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to
Cassandra cluster - assuming its not up yet: hawkular-cassandra: unknown error
12:41:02,999 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
(metricsservice-lifecycle-thread) HAWKMETRICS200004: [1] Retrying connecting to
Cassandra cluster in [1]s...
12:41:04,000 INFO [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics
service
12:41:04,001 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to
Cassandra cluster - assuming its not up yet: hawkular-cassandra
12:41:04,001 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
(metricsservice-lifecycle-thread) HAWKMETRICS200004: [2] Retrying connecting to
Cassandra cluster in [2]s...
12:41:06,002 INFO [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics
service
12:41:06,002 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to
Cassandra cluster - assuming its not up yet: hawkular-cassandra
12:41:06,003 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
(metricsservice-lifecycle-thread) HAWKMETRICS200004: [3] Retrying connecting to
Cassandra cluster in [3]s...
12:41:08,203 INFO [org.jboss.resteasy.cdi.i18n] (http-/0.0.0.0:8080-2)
RESTEASY006050: Found BeanManager at java:comp/BeanManager
12:41:08,299 INFO [org.hibernate.validator.internal.util.Version]
(http-/0.0.0.0:8080-2) HV000001: Hibernate Validator 4.3.2.Final-redhat-2
12:41:08,418 INFO [org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp]
(http-/0.0.0.0:8080-2) HAWKMETRICS200001: Hawkular Metrics starting
12:41:08,421 INFO [org.jboss.resteasy.resteasy_jaxrs.i18n]
(http-/0.0.0.0:8080-2) RESTEASY000320: Deploying javax.ws.rs.core.Application:
class
org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp$Proxy$_$$_WeldClientProxy
12:41:08,421 INFO [org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp]
(http-/0.0.0.0:8080-2) HAWKMETRICS200001: Hawkular Metrics starting
12:41:08,422 INFO [org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp]
(http-/0.0.0.0:8080-2) Virtual clock is disabled
12:41:08,422 INFO [org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp]
(http-/0.0.0.0:8080-2) Virtual clock is disabled
12:41:09,003 INFO [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics
service
12:41:09,004 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to
Cassandra cluster - assuming its not up yet: hawkular-cassandra
12:41:09,004 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
(metricsservice-lifecycle-thread) HAWKMETRICS200004: [4] Retrying connecting to
Cassandra cluster in [4]s...
12:41:13,004 INFO [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics
service
12:41:13,123 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to
Cassandra cluster - assuming its not up yet: hawkular-cassandra: unknown error
12:41:13,124 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
(metricsservice-lifecycle-thread) HAWKMETRICS200004: [5] Retrying connecting to
Cassandra cluster in [1]s...
12:41:14,124 INFO [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics
service
12:41:14,124 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to
Cassandra cluster - assuming its not up yet: hawkular-cassandra
12:41:14,125 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
(metricsservice-lifecycle-thread) HAWKMETRICS200004: [6] Retrying connecting to
Cassandra cluster in [2]s...
12:41:16,125 INFO [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics
service
12:41:16,126 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to
Cassandra cluster - assuming its not up yet: hawkular-cassandra
12:41:16,126 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
(metricsservice-lifecycle-thread) HAWKMETRICS200004: [7] Retrying connecting to
Cassandra cluster in [3]s...
12:41:19,126 INFO [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics
service
12:41:19,127 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to
Cassandra cluster - assuming its not up yet: hawkular-cassandra
12:41:19,127 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
(metricsservice-lifecycle-thread) HAWKMETRICS200004: [8] Retrying connecting to
Cassandra cluster in [4]s...
12:41:23,128 INFO [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics
service
12:41:23,249 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to
Cassandra cluster - assuming its not up yet: hawkular-cassandra: unknown error
12:41:23,249 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
(metricsservice-lifecycle-thread) HAWKMETRICS200004: [9] Retrying connecting to
Cassandra cluster in [1]s...
12:41:24,250 INFO [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics
service
12:41:24,250 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to
Cassandra cluster - assuming its not up yet: hawkular-cassandra
12:41:24,250 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
(metricsservice-lifecycle-thread) HAWKMETRICS200004: [10] Retrying connecting
to Cassandra cluster in [2]s...
12:41:26,251 INFO [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics
service
12:41:26,251 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to
Cassandra cluster - assuming its not up yet: hawkular-cassandra
12:41:26,251 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
(metricsservice-lifecycle-thread) HAWKMETRICS200004: [11] Retrying connecting
to Cassandra cluster in [3]s...
12:41:29,252 INFO [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics
service
12:41:29,252 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to
Cassandra cluster - assuming its not up yet: hawkular-cassandra
12:41:29,253 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
(metricsservice-lifecycle-thread) HAWKMETRICS200004: [12] Retrying connecting
to Cassandra cluster in [4]s...
12:41:33,253 INFO [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics
service
12:41:33,376 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to
Cassandra cluster - assuming its not up yet: hawkular-cassandra: unknown error
12:41:33,377 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
(metricsservice-lifecycle-thread) HAWKMETRICS200004: [13] Retrying connecting
to Cassandra cluster in [1]s...
12:41:34,377 INFO [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics
service
12:41:34,378 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to
Cassandra cluster - assuming its not up yet: hawkular-cassandra
12:41:34,378 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
(metricsservice-lifecycle-thread) HAWKMETRICS200004: [14] Retrying connecting
to Cassandra cluster in [2]s...
12:41:36,378 INFO [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics
service
12:41:36,379 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to
Cassandra cluster - assuming its not up yet: hawkular-cassandra
12:41:36,379 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
(metricsservice-lifecycle-thread) HAWKMETRICS200004: [15] Retrying connecting
to Cassandra cluster in [3]s...
12:41:39,379 INFO [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics
service
12:41:39,380 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to
Cassandra cluster - assuming its not up yet: hawkular-cassandra
12:41:39,380 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
(metricsservice-lifecycle-thread) HAWKMETRICS200004: [16] Retrying connecting
to Cassandra cluster in [4]s...
12:41:43,380 INFO [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics
service
12:41:43,503 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to
Cassandra cluster - assuming its not up yet: hawkular-cassandra: unknown error
12:41:43,504 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
(metricsservice-lifecycle-thread) HAWKMETRICS200004: [17] Retrying connecting
to Cassandra cluster in [1]s...
12:41:44,504 INFO [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics
service
12:41:44,505 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to
Cassandra cluster - assuming its not up yet: hawkular-cassandra
12:41:44,505 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
(metricsservice-lifecycle-thread) HAWKMETRICS200004: [18] Retrying connecting
to Cassandra cluster in [2]s...
--
Srinivas Kotaru
On 6/14/16, 6:06 AM, "Matt Wringe" <[email protected]> wrote:
>----- Original Message -----
>> From: "Srinivas Naga Kotaru (skotaru)" <[email protected]>
>> To: "Matt Wringe" <[email protected]>
>> Cc: [email protected]
>> Sent: Monday, June 13, 2016 7:26:06 PM
>> Subject: Re: Metrics deployment
>>
>> Matt
>>
>> PV issue resolved. Was able to to see PV successfully bounded and Casandra
>> container has been running. However, it seems puzzle not fully yet solved.
>
>Are you sure the OpenShift DNS server is running?
>
>If you are running OSE 3.1, can you please follow this
>https://access.redhat.com/solutions/2329131 and see if you are now seeing
>errors in the Hawkular Metrics logs (essentially just run `oc exec
>hawkular-metrics-xxxxx cat /opt/eap/standalone/log/server.log`)
>
>>
>> I could see other container(heapster) not coming up, and seeing below errors
>>
>> [skotaru@l3imas-id2-01 metrics]$ oc logs -f heapster-fnkdc
>> Endpoint Check in effect. Checking
>> https://hawkular-metrics:443/hawkular/metrics/status
>> Could not connect to https://hawkular-metrics:443/hawkular/metrics/status.
>> Curl exit code: 6. Status Code 000
>> 'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible
>> [HTTP status code: 000. Curl exit code 6]. Retrying.
>> Could not connect to https://hawkular-metrics:443/hawkular/metrics/status.
>> Curl exit code: 6. Status Code 000
>> 'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible
>> [HTTP status code: 000. Curl exit code 6]. Retrying.
>>
>>
>> # oc get pv
>> pv-5gb-0011 5Gi RWO Bound
>> openshift-infra/metrics-cassandra-1 22m
>>
>>
>> $ oc get pods
>> NAME READY STATUS RESTARTS AGE
>> hawkular-cassandra-1-2pzd7 1/1 Running 0 20m
>> hawkular-metrics-mf5qf 0/1 Running 7 20m
>> heapster-fnkdc 0/1 Error 6 20m
>> metrics-deployer-cvep0 0/1 Completed 0 21m
>>
>> # oc logs -f hawkular-metrics-mf5qf
>>
>> 19:20:00,819 INFO [org.xnio] (MSC service thread 1-2) XNIO Version
>> 3.0.14.GA-redhat-1
>> 19:20:00,831 INFO [org.jboss.as.server] (Controller Boot Thread) JBAS015888:
>> Creating http management service using socket-binding (management-http)
>> 19:20:00,834 INFO [org.xnio.nio] (MSC service thread 1-2) XNIO NIO
>> Implementation Version 3.0.14.GA-redhat-1
>> 19:20:00,844 INFO [org.jboss.remoting] (MSC service thread 1-2) JBoss
>> Remoting version 3.3.5.Final-redhat-1
>>
>> $ oc logs -f heapster-fnkdc
>> Endpoint Check in effect. Checking
>> https://hawkular-metrics:443/hawkular/metrics/status
>> Could not connect to https://hawkular-metrics:443/hawkular/metrics/status.
>> Curl exit code: 6. Status Code 000
>> 'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible
>> [HTTP status code: 000. Curl exit code 6]. Retrying.
>> Could not connect to https://hawkular-metrics:443/hawkular/metrics/status.
>> Curl exit code: 6. Status Code 000
>> 'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible
>> [HTTP status code: 000. Curl exit code 6]. Retrying.
>> Could not connect to https://hawkular-metrics:443/hawkular/metrics/status.
>> Curl exit code: 6. Status Code 000
>>
>> $ oc logs -f hawkular-cassandra-1-2pzd7
>> INFO 23:00:24 Starting listening for CQL clients on
>> hawkular-cassandra-1-2pzd7/10.1.6.2:9042...
>> INFO 23:00:24 Binding thrift service to
>> hawkular-cassandra-1-2pzd7/10.1.6.2:9160
>> INFO 23:00:24 enabling encrypted thrift connections between client and
>> server
>> INFO 23:00:24 Listening for thrift clients...
>> INFO 23:00:26 Created default superuser role 'cassandra'
>>
>> # oc get svc
>> NAME CLUSTER-IP EXTERNAL-IP PORT(S)
>> AGE
>> hawkular-cassandra 172.30.2.13 <none>
>> 9042/TCP,9160/TCP,7000/TCP,7001/TCP 25m
>> hawkular-cassandra-nodes None <none>
>> 9042/TCP,9160/TCP,7000/TCP,7001/TCP 25m
>> hawkular-metrics 172.30.117.176 <none> 443/TCP
>> 25m
>> heapster 172.30.107.135 <none> 80/TCP
>> 25m
>>
>> #curl -I 172.30.117.176:443//hawkular/metrics/status
>>
>> HTTP/1.1 504 Gateway Timeout
>> Mime-Version: 1.0
>> Date: Mon, 13 Jun 2016 23:25:47 GMT
>> Content-Type: text/html
>> Connection: keep-alive
>> Proxy-Connection: keep-alive
>> Content-Length: 1572
>>
>> --
>> Srinivas Kotaru
>>
>> On 6/13/16, 2:33 PM, "Srinivas Naga Kotaru (skotaru)" <[email protected]>
>> wrote:
>>
>> >Matt
>> >
>> >That is good catch. I ran without USE_PERSISTENT_STORAGE=false and working
>> >
>> >I adjusted PV to 5Gi and reran. Will update progress.
>> >
>> >Thanks you for your help so far.
>> >
>> >--
>> >Srinivas Kotaru
>> >
>> >On 6/13/16, 2:27 PM, "Matt Wringe" <[email protected]> wrote:
>> >
>> >>
>> >>
>> >>----- Original Message -----
>> >>> From: "Srinivas Naga Kotaru (skotaru)" <[email protected]>
>> >>> To: "Matt Wringe" <[email protected]>
>> >>> Cc: [email protected]
>> >>> Sent: Monday, June 13, 2016 5:21:01 PM
>> >>> Subject: Re: Metrics deployment
>> >>>
>> >>> Oh ok
>> >>>
>> >>> Am using PV for metrics
>> >>>
>> >>> description: "The persistent volume size for each of the Cassandra nodes"
>> >>> name: CASSANDRA_PV_SIZE
>> >>> value: "10Gi"
>> >>>
>> >>> oc get pv
>> >>> NAME CAPACITY ACCESSMODES STATUS CLAIM
>> >>> REASON
>> >>> AGE
>> >>> pv-1gb-001 1Gi RWO Available
>> >>> 4d
>> >>> pv-1gb-002 1Gi RWO Available
>> >>> 4d
>> >>> pv-1gb-003 1Gi RWO Available
>> >>> 4d
>> >>> pv-1gb-004 1Gi RWO Bound thlatt/mongodb
>> >>> 4d
>> >>> pv-1gb-005 1Gi RWO Available
>> >>> 4d
>> >>> pv-2gb-0010 2Gi RWO Available
>> >>> 4d
>> >>> pv-2gb-006 2Gi RWO Available
>> >>> 4d
>> >>> pv-2gb-007 2Gi RWO Available
>> >>> 4d
>> >>> pv-2gb-008 2Gi RWO Available
>> >>> 4d
>> >>> pv-2gb-009 2Gi RWO Available
>> >>> 4d
>> >>> pv-5gb-0011 5Gi RWO Available
>> >>> 4d
>> >>> pv-5gb-0012 5Gi RWO Available
>> >>> 4d
>> >>> pv-5gb-0013 5Gi RWO Available
>> >>> 4d
>> >>> pv-5gb-0014 5Gi RWO Available
>> >>> 4d
>> >>> pv-5gb-0015 5Gi RWO Available
>> >>> 4d
>> >>>
>> >>> am running with below command
>> >>>
>> >>> $ oc new-app -f metrics-deployer.yaml ( hardcoded HOSTNAME, MASTER_API
>> >>> and
>> >>> PV info so not passing any parameters)
>> >>>
>> >>
>> >>I would suspect that Cassandra is blocked because its waiting for 10Gi PV
>> >>to become available, and none of the PV listed above are big enough.
>> >>
>> >>>
>> >>> --
>> >>> Srinivas Kotaru
>> >>>
>> >>> On 6/13/16, 2:12 PM, "Matt Wringe" <[email protected]> wrote:
>> >>>
>> >>> >----- Original Message -----
>> >>> >> From: "Srinivas Naga Kotaru (skotaru)" <[email protected]>
>> >>> >> To: "Matt Wringe" <[email protected]>
>> >>> >> Cc: [email protected]
>> >>> >> Sent: Monday, June 13, 2016 4:55:55 PM
>> >>> >> Subject: Re: Metrics deployment
>> >>> >>
>> >>> >> Matt
>> >>> >>
>> >>> >> Thanks for looking into. I rerun the setup, but had the same issue
>> >>> >>
>> >>> >> # oc get pods
>> >>> >> NAME READY STATUS RESTARTS
>> >>> >> AGE
>> >>> >> hawkular-cassandra-1-y2egy 0/1 ContainerCreating 0
>> >>> >> 5m
>> >>> >> hawkular-metrics-4b16f 0/1 Running 1
>> >>> >> 4m
>> >>> >> heapster-x2gj2 0/1 Running 2
>> >>> >> 4m
>> >>> >> metrics-deployer-9v7vc 0/1 Completed 0
>> >>> >> 6m
>> >>> >>
>> >>> >> $ oc logs -f hawkular-cassandra-1-y2egy
>> >>> >> Error from server: container "hawkular-cassandra-1" in pod
>> >>> >> "hawkular-cassandra-1-y2egy" is waiting to start: ContainerCreating
>> >>> >
>> >>> >Ok, so it looks like something is blocking the Cassandra pod from
>> >>> >starting.
>> >>> >
>> >>> >If you are using persistent storage, Cassandra will not start until the
>> >>> >PV
>> >>> >is available. There may be some more information about Cassandra in the
>> >>> >pod
>> >>> >section of the console under events.
>> >>> >
>> >>> >What command did you use when deploying the deployer?
>> >>> >
>> >>> >>
>> >>> >> $ oc logs -f hawkular-metrics-4b16f
>> >>> >>
>> >>> >> 16:54:25,703 DEBUG [org.jboss.as.config] (MSC service thread 1-4) VM
>> >>> >> Arguments: -Duser.home=/home/jboss -Duser.name=jboss -D[Standalone]
>> >>> >> -XX:+UseCompressedOops -verbose:gc
>> >>> >> -Xloggc:/opt/eap/standalone/log/gc.log
>> >>> >> -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation
>> >>> >> -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=3M -XX:-TraceClassUnloading
>> >>> >> -Xms1303m -Xmx1303m -XX:MaxPermSize=256m
>> >>> >> -Djava.net.preferIPv4Stack=true
>> >>> >> -Djboss.modules.system.pkgs=org.jboss.logmanager
>> >>> >> -Djava.awt.headless=true
>> >>> >> -Djboss.modules.policy-permissions=true
>> >>> >> -Xbootclasspath/p:/opt/eap/jboss-modules.jar:/opt/eap/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-1.5.4.Final-redhat-1.jar:/opt/eap/modules/system/layers/base/org/jboss/logmanager/ext/main/javax.json-1.0.4.jar:/opt/eap/modules/system/layers/base/org/jboss/logmanager/ext/main/jboss-logmanager-ext-1.0.0.Alpha2-redhat-1.jar
>> >>> >> -Djava.util.logging.manager=org.jboss.logmanager.LogManager
>> >>> >> -javaagent:/opt/eap/jolokia.jar=port=8778,protocol=https,caCert=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt,clientPrincipal=cn=system:master-proxy,useSslClientAuthentication=true,extraClientCheck=true,host=0.0.0.0,discoveryEnabled=false
>> >>> >> -Djava.security.egd=file:/dev/./urandom
>> >>> >> -Dorg.jboss.boot.log.file=/opt/eap/standalone/log/server.log
>> >>> >> -Dlogging.configuration=file:/opt/eap/standalone/configuration/logging.properties
>> >>> >> 16:54:27,079 INFO [org.xnio] (MSC service thread 1-3) XNIO Version
>> >>> >> 3.0.14.GA-redhat-1
>> >>> >> 16:54:27,083 INFO [org.xnio.nio] (MSC service thread 1-3) XNIO NIO
>> >>> >> Implementation Version 3.0.14.GA-redhat-1
>> >>> >> 16:54:27,101 INFO [org.jboss.as.server] (Controller Boot Thread)
>> >>> >> JBAS015888:
>> >>> >> Creating http management service using socket-binding
>> >>> >> (management-http)
>> >>> >> 16:54:27,104 INFO [org.jboss.remoting] (MSC service thread 1-3) JBoss
>> >>> >> Remoting version 3.3.5.Final-redhat-1
>> >>> >>
>> >>> >> $ oc logs -f heapster-x2gj2
>> >>> >> Endpoint Check in effect. Checking
>> >>> >> https://hawkular-metrics:443/hawkular/metrics/status
>> >>> >> Could not connect to
>> >>> >> https://hawkular-metrics:443/hawkular/metrics/status.
>> >>> >> Curl exit code: 6. Status Code 000
>> >>> >> 'https://hawkular-metrics:443/hawkular/metrics/status' is not
>> >>> >> accessible
>> >>> >> [HTTP status code: 000. Curl exit code 6]. Retrying.
>> >>> >> Could not connect to
>> >>> >> https://hawkular-metrics:443/hawkular/metrics/status.
>> >>> >> Curl exit code: 6. Status Code 000
>> >>> >> 'https://hawkular-metrics:443/hawkular/metrics/status' is not
>> >>> >> accessible
>> >>> >> [HTTP status code: 000. Curl exit code 6]. Retrying.
>> >>> >> Could not connect to
>> >>> >> https://hawkular-metrics:443/hawkular/metrics/status.
>> >>> >> Curl exit code: 6. Status Code 000
>> >>> >>
>> >>> >>
>> >>> >> $ oc logs -f metrics-deployer-9v7vc
>> >>> >>
>> >>> >> ++ oc create -f -
>> >>> >> serviceaccount "heapster" created
>> >>> >> service "heapster" created
>> >>> >> replicationcontroller "heapster" created
>> >>> >> + echo 'Success!'
>> >>> >> Success!
>> >>> >>
>> >>> >> --
>> >>> >> Srinivas Kotaru
>> >>> >>
>> >>> >> On 6/13/16, 1:49 PM, "Matt Wringe" <[email protected]> wrote:
>> >>> >>
>> >>> >> >
>> >>> >> >
>> >>> >> >----- Original Message -----
>> >>> >> >> From: "Srinivas Naga Kotaru (skotaru)" <[email protected]>
>> >>> >> >> To: [email protected]
>> >>> >> >> Sent: Monday, June 13, 2016 3:58:12 PM
>> >>> >> >> Subject: Metrics deployment
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> Hi
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> Am trying to configure metrics in our newly installed clusters. Am
>> >>> >> >> seeing
>> >>> >> >> below errors once metrics-deploy script was successful. I used our
>> >>> >> >> environment specific HAWKULAR_METRICS_HOSTNAME and MASTER_URL
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> # oc new-app -f metrics-deployer.yaml
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> Note: customized, CASSANDARA PV, MASTER_URL, and
>> >>> >> >> HAWKULAR_METRICS_HOSTNAME
>> >>> >> >> (
>> >>> >> >> hard coded as values)
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> template "hawkular-heapster" created
>> >>> >> >>
>> >>> >> >> Deploying the Heapster component
>> >>> >> >>
>> >>> >> >> ++ echo 'Deploying the Heapster component'
>> >>> >> >>
>> >>> >> >> ++ '[' -n '' ']'
>> >>> >> >>
>> >>> >> >> ++ oc create -f -
>> >>> >> >>
>> >>> >> >> ++ oc process hawkular-heapster -v
>> >>> >> >> IMAGE_PREFIX=registry.access.redhat.com/openshift3/,IMAGE_VERSION=latest,MASTER_URL=https://lae3-alln-int-idev01.cisco.com:443,NODE_ID=nodename
>> >>> >> >>
>> >>> >> >> serviceaccount "heapster" created
>> >>> >> >>
>> >>> >> >> service "heapster" created
>> >>> >> >>
>> >>> >> >> replicationcontroller "heapster" created
>> >>> >> >>
>> >>> >> >> + echo 'Success!'
>> >>> >> >>
>> >>> >> >> Success!
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> # oc get pods
>> >>> >> >>
>> >>> >> >> NAME READY STATUS RESTARTS AGE
>> >>> >> >>
>> >>> >> >> hawkular-cassandra-1-9nzio 0/1 ContainerCreating 0 4m
>> >>> >> >>
>> >>> >> >> hawkular-metrics-hi7mb 0/1 Running 1 4m
>> >>> >> >>
>> >>> >> >> heapster-e8gbu 0/1 Running 2 4m
>> >>> >> >>
>> >>> >> >> metrics-deployer-64703 0/1 ContainerCreating 0 3s
>> >>> >> >>
>> >>> >> >> metrics-deployer-cd1nf 0/1 Completed 0 5m
>> >>> >> >>
>> >>> >> >
>> >>> >> >It looks like none of your containers are fully up and running yet.
>> >>> >> >
>> >>> >> >Without Cassandra running, Hawkular Metrics will not run, and
>> >>> >> >Heapster
>> >>> >> >will
>> >>> >> >wait until Hawkular Metrics is fully running.
>> >>> >> >
>> >>> >> >Do you see anything in the Cassandra logs? The first step will be to
>> >>> >> >get
>> >>> >> >Cassandra running properly.
>> >>> >> >
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> $ oc logs -f heapster-e8gbu
>> >>> >> >>
>> >>> >> >> Endpoint Check in effect. Checking
>> >>> >> >> https://hawkular-metrics:443/hawkular/metrics/status
>> >>> >> >>
>> >>> >> >> Could not connect to
>> >>> >> >> https://hawkular-metrics:443/hawkular/metrics/status.
>> >>> >> >> Curl exit code: 6. Status Code 000
>> >>> >> >>
>> >>> >> >> 'https://hawkular-metrics:443/hawkular/metrics/status' is not
>> >>> >> >> accessible
>> >>> >> >> [HTTP status code: 000. Curl exit code 6]. Retrying.
>> >>> >> >>
>> >>> >> >> Could not connect to
>> >>> >> >> https://hawkular-metrics:443/hawkular/metrics/status.
>> >>> >> >> Curl exit code: 6. Status Code 000
>> >>> >> >
>> >>> >> >Heapster waits until Hawkular Metrics is started before trying to
>> >>> >> >push
>> >>> >> >metrics to it. The issue that you are seeing is because Heapster
>> >>> >> >could
>> >>> >> >not
>> >>> >> >properly connect to Hawkular Metrics. Until the Hawkular Metrics
>> >>> >> >service
>> >>> >> >is
>> >>> >> >fully up, Heapster will not be able to connect to it.
>> >>> >> >
>> >>> >> >
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> What is the wrong? Why it checking just hawkular-metrics rather
>> >>> >> >> full
>> >>> >> >> routing
>> >>> >> >> URL which was provided as HAWKULAR_METRICS_HOSTNAME
>> >>> >> >
>> >>> >> >The Hawkular Metrics service has two hostnames: the internal hostname
>> >>> >> >used
>> >>> >> >by the internal components (eg 'hawkular-metrics') and the external
>> >>> >> >hostname (eg what is configured via HAWKULAR_METRICS_HOSTNAME). The
>> >>> >> >OpenShift dns server will resolve hostnames to the name of services,
>> >>> >> >which
>> >>> >> >is where the internal 'hawkular-metrics' comes from.
>> >>> >> >
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> --
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> Srinivas Kotaru
>> >>> >> >>
>> >>> >> >> _______________________________________________
>> >>> >> >> users mailing list
>> >>> >> >> [email protected]
>> >>> >> >> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>> >>> >> >>
>> >>> >>
>> >>> >>
>> >>>
>> >>>
>> >
>>
>>
_______________________________________________
users mailing list
[email protected]
http://lists.openshift.redhat.com/openshiftmm/listinfo/users
