I’m still struck with this issue. It is kind of chicken and egg problem.
Heapster health probes failing since it is waiting for hawkular to start.
Hawkular health probes are failing since it is unable to connect Casandra.
Cansandra health probe also failing.
@Matt: Internal DNS looks ups working. I’m able to create apps, build and
deploy code. Router and registry components also working as expected
oc get pods
NAME READY STATUS RESTARTS AGE
hawkular-cassandra-1-mxd2m 1/1 Running 0 1h
hawkular-metrics-gvp9k 0/1 Running 4 11m
heapster-uleul 0/1 Running 4 11m
metrics-deployer-2z75w 0/1 Completed 0 1h
heapster events:
------------------------
Events:
FirstSeen LastSeen Count From
SubobjectPath Type Reason Message
--------- -------- ----- ----
------------- -------- ------ -------
1m 1m 1 {default-scheduler }
Normal Scheduled Successfully
assigned heapster-uleul to l3inpn-id2-004.cisco.com
1m 1m 1 {kubelet l3inpn-id2-004.cisco.com}
spec.containers{heapster} Normal Pulling pulling image
"registry.access.redhat.com/openshift3/metrics-heapster:latest"
1m 1m 1 {kubelet l3inpn-id2-004.cisco.com}
spec.containers{heapster} Normal Pulled Successfully
pulled image "registry.access.redhat.com/openshift3/metrics-heapster:latest"
1m 1m 1 {kubelet l3inpn-id2-004.cisco.com}
spec.containers{heapster} Normal Created Created
container with docker id a22bb9a246ca
1m 1m 1 {kubelet l3inpn-id2-004.cisco.com}
spec.containers{heapster} Normal Started Started
container with docker id a22bb9a246ca
1m 5s 10 {kubelet l3inpn-id2-004.cisco.com}
spec.containers{heapster} Warning Unhealthy Readiness probe
failed: The heapster process is not yet started, it is waiting for the Hawkular
Metrics to start.
Hawkular events :
===============
Events:
FirstSeen LastSeen Count From
SubobjectPath Type Reason Message
--------- -------- ----- ----
------------- -------- ------ -------
1m 1m 1 {default-scheduler }
Normal Scheduled
Successfully assigned hawkular-metrics-gvp9k to l3inpn-id2-004.cisco.com
1m 1m 1 {kubelet l3inpn-id2-004.cisco.com}
spec.containers{hawkular-metrics} Normal Pulling pulling
image "registry.access.redhat.com/openshift3/metrics-hawkular-metrics:latest"
1m 1m 1 {kubelet l3inpn-id2-004.cisco.com}
spec.containers{hawkular-metrics} Normal Pulled
Successfully pulled image
"registry.access.redhat.com/openshift3/metrics-hawkular-metrics:latest"
1m 1m 1 {kubelet l3inpn-id2-004.cisco.com}
spec.containers{hawkular-metrics} Normal Created Created
container with docker id 6e32459624d7
1m 1m 1 {kubelet l3inpn-id2-004.cisco.com}
spec.containers{hawkular-metrics} Normal Started Started
container with docker id 6e32459624d7
1m 57s 4 {kubelet l3inpn-id2-004.cisco.com}
spec.containers{hawkular-metrics} Warning Unhealthy
Readiness probe failed: Failed to access the status endpoint : <urlopen error
[Errno 111] Connection refused>.
46s 7s 5 {kubelet l3inpn-id2-004.cisco.com}
spec.containers{hawkular-metrics} Warning Unhealthy Readiness probe
failed:
Casandra events
==============
Events:
FirstSeen LastSeen Count From
SubobjectPath Type Reason Message
--------- -------- ----- ----
------------- -------- ------ -------
1h 1h 1 {default-scheduler }
Normal Scheduled
Successfully assigned hawkular-cassandra-1-mxd2m to l3inpn-id2-003.cisco.com
1h 1h 1 {kubelet l3inpn-id2-003.cisco.com}
spec.containers{hawkular-cassandra-1} Normal Pulling pulling
image "registry.access.redhat.com/openshift3/metrics-cassandra:latest"
1h 1h 1 {kubelet l3inpn-id2-003.cisco.com}
spec.containers{hawkular-cassandra-1} Normal Pulled
Successfully pulled image
"registry.access.redhat.com/openshift3/metrics-cassandra:latest"
1h 1h 1 {kubelet l3inpn-id2-003.cisco.com}
spec.containers{hawkular-cassandra-1} Normal Created Created
container with docker id 2a1390edd4f1
1h 1h 1 {kubelet l3inpn-id2-003.cisco.com}
spec.containers{hawkular-cassandra-1} Normal Started Started
container with docker id 2a1390edd4f1
1h 1h 1 {kubelet l3inpn-id2-003.cisco.com}
spec.containers{hawkular-cassandra-1} Warning Unhealthy
Readiness probe failed: cat: /etc/ld.so.conf.d/*.conf: No such file or directory
nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException: 'Connection
refused'.
/opt/apache-cassandra/bin/cassandra-docker-ready.sh: line 28: [: =: unary
operator expected
Cassandra not in the up and normal state. Current state is
1h 1h 2 {kubelet l3inpn-id2-003.cisco.com}
spec.containers{hawkular-cassandra-1} Warning Unhealthy Readiness probe
failed: cat: /etc/ld.so.conf.d/*.conf: No such file or directory
nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException: 'Connection
refused'.
Cassandra not in the up and normal state. Current state is
/opt/apache-cassandra/bin/cassandra-docker-ready.sh: line 28: [: =: unary
operator expected
Logs:
====
# oc logs -f heapster-uleul
Endpoint Check in effect. Checking
https://hawkular-metrics:443/hawkular/metrics/status
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl
exit code: 6. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP
status code: 000. Curl exit code 6]. Retrying.
# oc logs -f hawkular-metrics-gvp9k
16:42:43,201 INFO [org.jboss.as.server] (Controller Boot Thread) JBAS015888:
Creating http management service using socket-binding (management-http)
16:42:43,202 INFO [org.xnio] (MSC service thread 1-1) XNIO Version
3.0.14.GA-redhat-1
16:42:43,208 INFO [org.xnio.nio] (MSC service thread 1-1) XNIO NIO
Implementation Version 3.0.14.GA-redhat-1
16:42:43,223 INFO [org.jboss.remoting] (MSC service thread 1-1) JBoss Remoting
version 3.3.5.Final-redhat-1
# oc logs -f hawkular-cassandra-1-mxd2m
INFO 20:15:20 Compacted (b7840c40-326c-11e6-a5f7-694864f768a1) 4 sstables to
[/cassandra_data/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/la-5-big,]
to level=0. 1,643 bytes to 1,155 (~70% of original) in 207ms = 0.005321MB/s.
0 total partitions merged to 11. Partition merge counts were {1:11, }
INFO 20:15:20 Compacting (b7a3f050-326c-11e6-a5f7-694864f768a1)
[/cassandra_data/data/system/schema_functions-d1b675fe2b503ca48e49c0f81989dcad/la-4-big-Data.db:level=0,
/cassandra_data/data/system/schema_functions-d1b675fe2b503ca48e49c0f81989dcad/la-3-big-Data.db:level=0,
/cassandra_data/data/system/schema_functions-d1b675fe2b503ca48e49c0f81989dcad/la-2-big-Data.db:level=0,
/cassandra_data/data/system/schema_functions-d1b675fe2b503ca48e49c0f81989dcad/la-1-big-Data.db:level=0,
]
INFO 20:15:20 Writing Memtable-schema_columns@672348153(49.749KiB serialized
bytes, 967 ops, 0%/0% of on/off-heap limit)
INFO 20:15:20 Completed flushing
/cassandra_data/data/system/schema_columns-296e9c049bec3085827dc17d3df2122a/tmp-la-7-big-Data.db
(11.212KiB) for commitlog position ReplayPosition(segmentId=1465931718336,
position=159511)
INFO 20:15:20 Compacted (b797bb50-326c-11e6-a5f7-694864f768a1) 4 sstables to
[/cassandra_data/data/system/schema_triggers-0359bc7171233ee19a4ab9dfb11fc125/la-5-big,]
to level=0. 128 bytes to 32 (~25% of original) in 146ms = 0.000209MB/s. 0
total partitions merged to 1. Partition merge counts were {4:1, }
INFO 20:15:20 Compacting (b7ae77a0-326c-11e6-a5f7-694864f768a1)
[/cassandra_data/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/la-22-big-Data.db:level=0,
/cassandra_data/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/la-21-big-Data.db:level=0,
/cassandra_data/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/la-19-big-Data.db:level=0,
/cassandra_data/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/la-20-big-Data.db:level=0,
]
INFO 20:15:20 Compacted (b7a3f050-326c-11e6-a5f7-694864f768a1) 4 sstables to
[/cassandra_data/data/system/schema_functions-d1b675fe2b503ca48e49c0f81989dcad/la-5-big,]
to level=0. 128 bytes to 32 (~25% of original) in 155ms = 0.000197MB/s. 0
total partitions merged to 1. Partition merge counts were {4:1, }
INFO 20:15:20 Compacting (b7bca870-326c-11e6-a5f7-694864f768a1)
[/cassandra_data/data/system/schema_usertypes-3aa752254f82350b8d5c430fa221fa0a/la-3-big-Data.db:level=0,
/cassandra_data/data/system/schema_usertypes-3aa752254f82350b8d5c430fa221fa0a/la-2-big-Data.db:level=0,
/cassandra_data/data/system/schema_usertypes-3aa752254f82350b8d5c430fa221fa0a/la-1-big-Data.db:level=0,
/cassandra_data/data/system/schema_usertypes-3aa752254f82350b8d5c430fa221fa0a/la-4-big-Data.db:level=0,
]
INFO 20:15:20 Compacted (b7bca870-326c-11e6-a5f7-694864f768a1) 4 sstables to
[/cassandra_data/data/system/schema_usertypes-3aa752254f82350b8d5c430fa221fa0a/la-5-big,]
to level=0. 128 bytes to 32 (~25% of original) in 83ms = 0.000368MB/s. 0
total partitions merged to 1. Partition merge counts were {4:1, }
INFO 20:15:20 Compacted (b7ae77a0-326c-11e6-a5f7-694864f768a1) 4 sstables to
[/cassandra_data/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/la-23-big,]
to level=0. 2,538 bytes to 527 (~20% of original) in 176ms = 0.002856MB/s. 0
total partitions merged to 3. Partition merge counts were {4:3, }
almost struck here. Any points to look for? Any ports need to be open
explicitly ?
--
Srinivas Kotaru
On 6/14/16, 11:27 AM, "Matt Wringe" <[email protected]> wrote:
>----- Original Message -----
>> From: "Srinivas Naga Kotaru (skotaru)" <[email protected]>
>> To: "Matt Wringe" <[email protected]>
>> Cc: [email protected]
>> Sent: Tuesday, June 14, 2016 2:09:49 PM
>> Subject: Re: Metrics deployment
>>
>> Not sure what do you mean by installation. Am just running oc new-app -f
>> metrics-deployer.yaml with default values except HOST_NAME and PV storage.
>
>I would suspect something wrong with your node or cluster installation. Or you
>have firewall rules blocking connections between your nodes so that pod cannot
>connect with each other or access the OpenShift DNS server.
>
>The lifecycle hooks exist to make sure that components only enter the ready
>state when they are fully started and ready.
>
>Can you check the Hawkular Metrics status page and see what that outputs? eg
>https://${HAWKULAR_METRICS_HOSTNAME}/hawkular/metrics/status
>
>>
>> I just deleted entire metrics setup and re running. But not sure this will
>> fix the issue.
>>
>> $ ./delete_metrics-infra.sh
>> replicationcontroller "hawkular-cassandra-1" deleted
>> replicationcontroller "hawkular-metrics" deleted
>> replicationcontroller "heapster" deleted
>> route "hawkular-metrics" deleted
>> service "hawkular-cassandra" deleted
>> service "hawkular-cassandra-nodes" deleted
>> service "hawkular-metrics" deleted
>> service "heapster" deleted
>> pod "heapster-lyf65" deleted
>> serviceaccount "cassandra" deleted
>> serviceaccount "hawkular" deleted
>> serviceaccount "heapster" deleted
>> template "hawkular-cassandra-node-emptydir" deleted
>> template "hawkular-cassandra-node-pv" deleted
>> template "hawkular-cassandra-services" deleted
>> template "hawkular-heapster" deleted
>> template "hawkular-metrics" deleted
>> template "hawkular-support" deleted
>> secret "hawkular-cassandra-certificate" deleted
>> secret "hawkular-cassandra-secrets" deleted
>> secret "hawkular-metrics-account" deleted
>> secret "hawkular-metrics-certificate" deleted
>> secret "hawkular-metrics-secrets" deleted
>> secret "heapster-secrets" deleted
>>
>> --
>> Srinivas Kotaru
>>
>> On 6/14/16, 10:53 AM, "Matt Wringe" <[email protected]> wrote:
>>
>> >----- Original Message -----
>> >> From: "Srinivas Naga Kotaru (skotaru)" <[email protected]>
>> >> To: "Matt Wringe" <[email protected]>
>> >> Cc: [email protected]
>> >> Sent: Tuesday, June 14, 2016 1:37:01 PM
>> >> Subject: Re: Metrics deployment
>> >>
>> >> I removed readiness probes from both hawkular-cassandra-1 &
>> >> hawkular-metrics
>> >> as both status shows probes failed.
>> >
>> >You should not have to remove the probes, this indicates that something is
>> >wrong with your installation.
>> >
>> >>
>> >> It looks good now. Both containers looks and running
>> >> (hawkular-cassandra-1-kr8ka , hawkular-metrics-vhe3u) however
>> >> heapster-7yl34
>> >> logs still shows Could not connect to
>> >> https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 6.
>> >> Status Code 000.
>> >>
>> >> Are we good or still had issues?
>> >>
>> >>
>> >> # oc get pods
>> >> NAME READY STATUS RESTARTS AGE
>> >> hawkular-cassandra-1-kr8ka 1/1 Running 0 6m
>> >> hawkular-metrics-vhe3u 1/1 Running 2 5m
>> >> heapster-7yl34 0/1 Running 2 5m
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> --
>> >> Srinivas Kotaru
>> >>
>> >> On 6/14/16, 10:07 AM, "Srinivas Naga Kotaru (skotaru)" <[email protected]>
>> >> wrote:
>> >>
>> >> >Matt
>> >> >
>> >> >Just want to share more info by running describe pod.
>> >> >
>> >> >It seems to be health probe failing. Do you think it is the issue?
>> >> >
>> >> >
>> >> >
>> >> ># oc describe pod hawkular-cassandra-1-it5uh
>> >> >Name: hawkular-cassandra-1-it5uh
>> >> >Namespace: openshift-infra
>> >> >Node: l3inpn-id2-003.cisco.com/173.36.96.16
>> >> >Start Time: Tue, 14 Jun 2016 16:36:21 +0000
>> >> >Labels:
>> >> >
>> >> > metrics-infra=hawkular-cassandra,name=hawkular-cassandra-1,type=hawkular-cassandra
>> >> >Status: Running
>> >> >IP: 10.1.9.2
>> >> >Controllers: ReplicationController/hawkular-cassandra-1
>> >> >Containers:
>> >> > hawkular-cassandra-1:
>> >> > Container ID:
>> >> >
>> >> > docker://17a9575eb655145859a9207f5c4bde7456f947e27188a056ff2bd08c4ce6ae5d
>> >> > Image:
>> >> >
>> >> > registry.access.redhat.com/openshift3/metrics-cassandra:latest
>> >> > Image ID:
>> >> >
>> >> > docker://ee2117c9848298ca5a0cbbce354fd4adff370435225324ab9d60cd9cd9a95c53
>> >> > Ports: 9042/TCP, 9160/TCP, 7000/TCP, 7001/TCP
>> >> > Command:
>> >> > /opt/apache-cassandra/bin/cassandra-docker.sh
>> >> > --cluster_name=hawkular-metrics
>> >> > --data_volume=/cassandra_data
>> >> > --internode_encryption=all
>> >> > --require_node_auth=true
>> >> > --enable_client_encryption=true
>> >> > --require_client_auth=true
>> >> > --keystore_file=/secret/cassandra.keystore
>> >> > --keystore_password_file=/secret/cassandra.keystore.password
>> >> > --truststore_file=/secret/cassandra.truststore
>> >> > --truststore_password_file=/secret/cassandra.truststore.password
>> >> > --cassandra_pem_file=/secret/cassandra.pem
>> >> > QoS Tier:
>> >> > cpu: BestEffort
>> >> > memory: BestEffort
>> >> > State: Running
>> >> > Started: Tue, 14 Jun 2016 16:37:01 +0000
>> >> > Ready: True
>> >> > Restart Count: 0
>> >> > Readiness: exec
>> >> > [/opt/apache-cassandra/bin/cassandra-docker-ready.sh]
>> >> > delay=0s timeout=1s period=10s #success=1 #failure=3
>> >> > Environment Variables:
>> >> > CASSANDRA_MASTER: true
>> >> > POD_NAMESPACE: openshift-infra (v1:metadata.namespace)
>> >> >Conditions:
>> >> > Type Status
>> >> > Ready True
>> >> >Volumes:
>> >> > cassandra-data:
>> >> > Type: PersistentVolumeClaim (a reference to a
>> >> > PersistentVolumeClaim
>> >> > in
>> >> > the same namespace)
>> >> > ClaimName: metrics-cassandra-1
>> >> > ReadOnly: false
>> >> > hawkular-cassandra-secrets:
>> >> > Type: Secret (a volume populated by a Secret)
>> >> > SecretName: hawkular-cassandra-secrets
>> >> > cassandra-token-4urfd:
>> >> > Type: Secret (a volume populated by a Secret)
>> >> > SecretName: cassandra-token-4urfd
>> >> >Events:
>> >> > FirstSeen LastSeen Count From
>> >> > SubobjectPath Type Reason
>> >> > Message
>> >> > --------- -------- ----- ----
>> >> > ------------- -------- ------
>> >> > -------
>> >> > 27m 27m 1 {default-scheduler }
>> >> > Normal
>> >> > Scheduled Successfully
>> >> > assigned hawkular-cassandra-1-it5uh to l3inpn-id2-003.cisco.com
>> >> > 27m 27m 1 {kubelet
>> >> > l3inpn-id2-003.cisco.com}
>> >> > spec.containers{hawkular-cassandra-1} Normal Pulling
>> >> > pulling image
>> >> > "registry.access.redhat.com/openshift3/metrics-cassandra:latest"
>> >> > 27m 27m 1 {kubelet
>> >> > l3inpn-id2-003.cisco.com}
>> >> > spec.containers{hawkular-cassandra-1} Normal Pulled
>> >> > Successfully
>> >> > pulled
>> >> > image "registry.access.redhat.com/openshift3/metrics-cassandra:latest"
>> >> > 27m 27m 1 {kubelet
>> >> > l3inpn-id2-003.cisco.com}
>> >> > spec.containers{hawkular-cassandra-1} Normal Created
>> >> > Created
>> >> > container
>> >> > with docker id 17a9575eb655
>> >> > 27m 27m 1 {kubelet
>> >> > l3inpn-id2-003.cisco.com}
>> >> > spec.containers{hawkular-cassandra-1} Normal Started
>> >> > Started
>> >> > container
>> >> > with docker id 17a9575eb655
>> >> > 27m 26m 3 {kubelet
>> >> > l3inpn-id2-003.cisco.com}
>> >> > spec.containers{hawkular-cassandra-1} Warning
>> >> > Unhealthy Readiness
>> >> > probe
>> >> > failed: cat: /etc/ld.so.conf.d/*.conf: No such file or directory
>> >> >nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException:
>> >> >'Connection refused'.
>> >> >Cassandra not in the up and normal state. Current state is
>> >> >/opt/apache-cassandra/bin/cassandra-docker-ready.sh: line 28: [: =: unary
>> >> >operator expected
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >--
>> >> >Srinivas Kotaru
>> >> >
>> >> >On 6/14/16, 10:00 AM, "Srinivas Naga Kotaru (skotaru)"
>> >> ><[email protected]>
>> >> >wrote:
>> >> >
>> >> >>Matt
>> >> >>
>> >> >>DNS service is working
>> >> >>
>> >> >>
>> >> >>~ dig +short @master-01 kubernetes.default.svc.cluster.local
>> >> >>172.30.0.1
>> >> >>~ dig +short @master-01 jenkins.alln-test.svc.cluster.local
>> >> >>172.30.85.148
>> >> >>~ dig +short @master-01 cakephp-example.alln-test.svc.cluster.local
>> >> >>172.30.31.6
>> >> >>
>> >> >>I captured hawkular-metrics, it shows the problem. It seems to be unable
>> >> >>to
>> >> >>connect Cassandra cluster
>> >> >>
>> >> >># oc exec hawkular-metrics-qldd8 cat /opt/eap/standalone/log/server.log
>> >> >>
>> >> >>12:40:58,848 INFO [org.jboss.as.clustering.infinispan] (ServerService
>> >> >>Thread Pool -- 31) JBAS010280: Activating Infinispan subsystem.
>> >> >>12:40:58,898 INFO [org.jboss.as.webservices] (ServerService Thread Pool
>> >> >>--
>> >> >>48) JBAS015537: Activating WebServices Extension
>> >> >>12:40:58,901 WARN [org.jboss.as.txn] (ServerService Thread Pool -- 46)
>> >> >>JBAS010153: Node identifier property is set to the default value. Please
>> >> >>make sure it is unique.
>> >> >>12:40:58,902 INFO [org.jboss.as.security] (ServerService Thread Pool --
>> >> >>44) JBAS013371: Activating Security Subsystem
>> >> >>12:40:58,912 INFO [org.jboss.as.naming] (ServerService Thread Pool --
>> >> >>39)
>> >> >>JBAS011800: Activating Naming Subsystem
>> >> >>12:40:58,928 INFO [org.jboss.as.jsf] (ServerService Thread Pool -- 37)
>> >> >>JBAS012615: Activated the following JSF Implementations: [main, 1.2]
>> >> >>12:40:58,939 INFO [org.jboss.as.connector.logging] (MSC service thread
>> >> >>1-2) JBAS010408: Starting JCA Subsystem (IronJacamar
>> >> >>1.0.32.Final-redhat-1)
>> >> >>12:40:58,981 INFO [org.jboss.as.security] (MSC service thread 1-2)
>> >> >>JBAS013370: Current PicketBox version=4.1.1.Final-redhat-1
>> >> >>12:40:58,961 INFO [org.jboss.as.connector.subsystems.datasources]
>> >> >>(ServerService Thread Pool -- 27) JBAS010403: Deploying JDBC-compliant
>> >> >>driver class org.h2.Driver (version 1.3)
>> >> >>12:40:59,039 INFO [org.jboss.as.naming] (MSC service thread 1-2)
>> >> >>JBAS011802: Starting Naming Service
>> >> >>12:40:59,052 INFO [org.jboss.as.mail.extension] (MSC service thread
>> >> >>1-4)
>> >> >>JBAS015400: Bound mail session [java:jboss/mail/Default]
>> >> >>12:40:59,614 INFO [org.apache.coyote.http11.Http11Protocol] (MSC
>> >> >>service
>> >> >>thread 1-1) JBWEB003001: Coyote HTTP/1.1 initializing on :
>> >> >>http-/0.0.0.0:8444
>> >> >>12:40:59,620 INFO [org.apache.coyote.http11.Http11Protocol] (MSC
>> >> >>service
>> >> >>thread 1-2) JBWEB003001: Coyote HTTP/1.1 initializing on :
>> >> >>http-/0.0.0.0:8080
>> >> >>12:40:59,621 INFO [org.apache.coyote.http11.Http11Protocol] (MSC
>> >> >>service
>> >> >>thread 1-1) JBWEB003000: Coyote HTTP/1.1 starting on: http-/0.0.0.0:8444
>> >> >>12:40:59,627 INFO [org.apache.coyote.http11.Http11Protocol] (MSC
>> >> >>service
>> >> >>thread 1-2) JBWEB003000: Coyote HTTP/1.1 starting on: http-/0.0.0.0:8080
>> >> >>12:40:59,654 INFO [org.jboss.as.connector.subsystems.datasources] (MSC
>> >> >>service thread 1-3) JBAS010400: Bound data source
>> >> >>[java:jboss/datasources/ExampleDS]
>> >> >>12:40:59,985 INFO [org.jboss.as.server.deployment] (MSC service thread
>> >> >>1-2) JBAS015876: Starting deployment of "activemq-rar.rar"
>> >> >>(runtime-name:
>> >> >>"activemq-rar.rar")
>> >> >>12:40:59,986 INFO [org.jboss.as.server.deployment] (MSC service thread
>> >> >>1-1) JBAS015876: Starting deployment of "hawkular-metrics-api-jaxrs.war"
>> >> >>(runtime-name: "hawkular-metrics-api-jaxrs.war")
>> >> >>12:40:59,987 INFO [org.jboss.as.server.deployment.scanner] (MSC service
>> >> >>thread 1-3) JBAS015012: Started FileSystemDeploymentService for
>> >> >>directory
>> >> >>/opt/eap/standalone/deployments
>> >> >>12:41:00,011 INFO [org.jboss.as.remoting] (MSC service thread 1-2)
>> >> >>JBAS017100: Listening on 127.0.0.1:9999
>> >> >>12:41:00,030 WARN [org.jboss.as.domain.http.api] (MSC service thread
>> >> >>1-3)
>> >> >>JBAS015102: Unable to load console module for slot eap, disabling
>> >> >>console
>> >> >>12:41:00,042 INFO [org.jboss.ws.common.management] (MSC service thread
>> >> >>1-4) JBWS022052: Starting JBoss Web Services - Stack CXF Server
>> >> >>4.3.5.Final-redhat-3
>> >> >>12:41:00,045 INFO [org.jboss.as.remoting] (MSC service thread 1-4)
>> >> >>JBAS017100: Listening on 0.0.0.0:4447
>> >> >>12:41:01,361 INFO [org.jboss.as.connector.deployers.RADeployer] (MSC
>> >> >>service thread 1-3) IJ020001: Required license terms for
>> >> >>file:/opt/eap/standalone/tmp/vfs/temp/temp912bb9d1372c33d3/activemq-rar.rar-a722089a00658182/contents/
>> >> >>12:41:01,784 WARN [org.jboss.as.dependency.private] (MSC service thread
>> >> >>1-1) JBAS015867: Deployment "deployment.hawkular-metrics-api-jaxrs.war"
>> >> >>is
>> >> >>using a private module ("org.codehaus.jackson.jackson-core-asl:main")
>> >> >>which may be changed or removed in future versions without notice.
>> >> >>12:41:01,785 WARN [org.jboss.as.dependency.private] (MSC service thread
>> >> >>1-1) JBAS015867: Deployment "deployment.hawkular-metrics-api-jaxrs.war"
>> >> >>is
>> >> >>using a private module ("org.codehaus.jackson.jackson-core-asl:main")
>> >> >>which may be changed or removed in future versions without notice.
>> >> >>12:41:01,785 WARN [org.jboss.as.dependency.private] (MSC service thread
>> >> >>1-1) JBAS015867: Deployment "deployment.hawkular-metrics-api-jaxrs.war"
>> >> >>is
>> >> >>using a private module ("org.codehaus.jackson.jackson-mapper-asl:main")
>> >> >>which may be changed or removed in future versions without notice.
>> >> >>12:41:01,785 WARN [org.jboss.as.dependency.private] (MSC service thread
>> >> >>1-1) JBAS015867: Deployment "deployment.hawkular-metrics-api-jaxrs.war"
>> >> >>is
>> >> >>using a private module ("org.codehaus.jackson.jackson-mapper-asl:main")
>> >> >>which may be changed or removed in future versions without notice.
>> >> >>12:41:01,843 INFO [org.jboss.weld.deployer] (MSC service thread 1-1)
>> >> >>JBAS016002: Processing weld deployment hawkular-metrics-api-jaxrs.war
>> >> >>12:41:01,941 INFO [org.jboss.weld.deployer] (MSC service thread 1-1)
>> >> >>JBAS016005: Starting Services for CDI deployment:
>> >> >>hawkular-metrics-api-jaxrs.war
>> >> >>12:41:01,986 INFO [org.jboss.weld.Version] (MSC service thread 1-1)
>> >> >>WELD-000900 1.1.31 (redhat)
>> >> >>12:41:02,013 INFO [org.jboss.weld.deployer] (MSC service thread 1-3)
>> >> >>JBAS016008: Starting weld service for deployment
>> >> >>hawkular-metrics-api-jaxrs.war
>> >> >>12:41:02,820 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>> >> >>metrics
>> >> >>service
>> >> >>12:41:02,835 INFO [org.jboss.web] (ServerService Thread Pool -- 51)
>> >> >>JBAS018210: Register web context: /hawkular/metrics
>> >> >>12:41:02,958 INFO [org.jboss.as.server] (ServerService Thread Pool --
>> >> >>28)
>> >> >>JBAS015859: Deployed "hawkular-metrics-api-jaxrs.war" (runtime-name :
>> >> >>"hawkular-metrics-api-jaxrs.war")
>> >> >>12:41:02,959 INFO [org.jboss.as.server] (ServerService Thread Pool --
>> >> >>28)
>> >> >>JBAS015859: Deployed "activemq-rar.rar" (runtime-name :
>> >> >>"activemq-rar.rar")
>> >> >>12:41:02,994 INFO [org.jboss.as] (Controller Boot Thread) JBAS015961:
>> >> >>Http
>> >> >>management interface listening on http://127.0.0.1:9990/management
>> >> >>12:41:02,994 INFO [org.jboss.as] (Controller Boot Thread) JBAS015951:
>> >> >>Admin console listening on http://127.0.0.1:9990
>> >> >>12:41:02,994 INFO [org.jboss.as] (Controller Boot Thread) JBAS015874:
>> >> >>JBoss EAP 6.4.4.GA (AS 7.5.4.Final-redhat-4) started in 6593ms - Started
>> >> >>267 of 303 services (62 services are lazy, passive or on-demand)
>> >> >>12:41:02,999 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
>> >> >>to
>> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra: unknown
>> >> >>error
>> >> >>12:41:02,999 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [1] Retrying
>> >> >>connecting to Cassandra cluster in [1]s...
>> >> >>12:41:04,000 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>> >> >>metrics
>> >> >>service
>> >> >>12:41:04,001 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
>> >> >>to
>> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>> >> >>12:41:04,001 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [2] Retrying
>> >> >>connecting to Cassandra cluster in [2]s...
>> >> >>12:41:06,002 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>> >> >>metrics
>> >> >>service
>> >> >>12:41:06,002 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
>> >> >>to
>> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>> >> >>12:41:06,003 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [3] Retrying
>> >> >>connecting to Cassandra cluster in [3]s...
>> >> >>12:41:08,203 INFO [org.jboss.resteasy.cdi.i18n] (http-/0.0.0.0:8080-2)
>> >> >>RESTEASY006050: Found BeanManager at java:comp/BeanManager
>> >> >>12:41:08,299 INFO [org.hibernate.validator.internal.util.Version]
>> >> >>(http-/0.0.0.0:8080-2) HV000001: Hibernate Validator
>> >> >>4.3.2.Final-redhat-2
>> >> >>12:41:08,418 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp]
>> >> >>(http-/0.0.0.0:8080-2) HAWKMETRICS200001: Hawkular Metrics starting
>> >> >>12:41:08,421 INFO [org.jboss.resteasy.resteasy_jaxrs.i18n]
>> >> >>(http-/0.0.0.0:8080-2) RESTEASY000320: Deploying
>> >> >>javax.ws.rs.core.Application: class
>> >> >>org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp$Proxy$_$$_WeldClientProxy
>> >> >>12:41:08,421 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp]
>> >> >>(http-/0.0.0.0:8080-2) HAWKMETRICS200001: Hawkular Metrics starting
>> >> >>12:41:08,422 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp]
>> >> >>(http-/0.0.0.0:8080-2) Virtual clock is disabled
>> >> >>12:41:08,422 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp]
>> >> >>(http-/0.0.0.0:8080-2) Virtual clock is disabled
>> >> >>12:41:09,003 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>> >> >>metrics
>> >> >>service
>> >> >>12:41:09,004 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
>> >> >>to
>> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>> >> >>12:41:09,004 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [4] Retrying
>> >> >>connecting to Cassandra cluster in [4]s...
>> >> >>12:41:13,004 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>> >> >>metrics
>> >> >>service
>> >> >>12:41:13,123 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
>> >> >>to
>> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra: unknown
>> >> >>error
>> >> >>12:41:13,124 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [5] Retrying
>> >> >>connecting to Cassandra cluster in [1]s...
>> >> >>12:41:14,124 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>> >> >>metrics
>> >> >>service
>> >> >>12:41:14,124 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
>> >> >>to
>> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>> >> >>12:41:14,125 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [6] Retrying
>> >> >>connecting to Cassandra cluster in [2]s...
>> >> >>12:41:16,125 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>> >> >>metrics
>> >> >>service
>> >> >>12:41:16,126 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
>> >> >>to
>> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>> >> >>12:41:16,126 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [7] Retrying
>> >> >>connecting to Cassandra cluster in [3]s...
>> >> >>12:41:19,126 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>> >> >>metrics
>> >> >>service
>> >> >>12:41:19,127 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
>> >> >>to
>> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>> >> >>12:41:19,127 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [8] Retrying
>> >> >>connecting to Cassandra cluster in [4]s...
>> >> >>12:41:23,128 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>> >> >>metrics
>> >> >>service
>> >> >>12:41:23,249 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
>> >> >>to
>> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra: unknown
>> >> >>error
>> >> >>12:41:23,249 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [9] Retrying
>> >> >>connecting to Cassandra cluster in [1]s...
>> >> >>12:41:24,250 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>> >> >>metrics
>> >> >>service
>> >> >>12:41:24,250 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
>> >> >>to
>> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>> >> >>12:41:24,250 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [10] Retrying
>> >> >>connecting to Cassandra cluster in [2]s...
>> >> >>12:41:26,251 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>> >> >>metrics
>> >> >>service
>> >> >>12:41:26,251 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
>> >> >>to
>> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>> >> >>12:41:26,251 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [11] Retrying
>> >> >>connecting to Cassandra cluster in [3]s...
>> >> >>12:41:29,252 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>> >> >>metrics
>> >> >>service
>> >> >>12:41:29,252 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
>> >> >>to
>> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>> >> >>12:41:29,253 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [12] Retrying
>> >> >>connecting to Cassandra cluster in [4]s...
>> >> >>12:41:33,253 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>> >> >>metrics
>> >> >>service
>> >> >>12:41:33,376 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
>> >> >>to
>> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra: unknown
>> >> >>error
>> >> >>12:41:33,377 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [13] Retrying
>> >> >>connecting to Cassandra cluster in [1]s...
>> >> >>12:41:34,377 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>> >> >>metrics
>> >> >>service
>> >> >>12:41:34,378 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
>> >> >>to
>> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>> >> >>12:41:34,378 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [14] Retrying
>> >> >>connecting to Cassandra cluster in [2]s...
>> >> >>12:41:36,378 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>> >> >>metrics
>> >> >>service
>> >> >>12:41:36,379 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
>> >> >>to
>> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>> >> >>12:41:36,379 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [15] Retrying
>> >> >>connecting to Cassandra cluster in [3]s...
>> >> >>12:41:39,379 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>> >> >>metrics
>> >> >>service
>> >> >>12:41:39,380 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
>> >> >>to
>> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>> >> >>12:41:39,380 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [16] Retrying
>> >> >>connecting to Cassandra cluster in [4]s...
>> >> >>12:41:43,380 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>> >> >>metrics
>> >> >>service
>> >> >>12:41:43,503 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
>> >> >>to
>> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra: unknown
>> >> >>error
>> >> >>12:41:43,504 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [17] Retrying
>> >> >>connecting to Cassandra cluster in [1]s...
>> >> >>12:41:44,504 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>> >> >>metrics
>> >> >>service
>> >> >>12:41:44,505 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
>> >> >>to
>> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>> >> >>12:41:44,505 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [18] Retrying
>> >> >>connecting to Cassandra cluster in [2]s...
>> >> >>
>> >> >>--
>> >> >>Srinivas Kotaru
>> >> >>
>> >> >>On 6/14/16, 6:06 AM, "Matt Wringe" <[email protected]> wrote:
>> >> >>
>> >> >>>----- Original Message -----
>> >> >>>> From: "Srinivas Naga Kotaru (skotaru)" <[email protected]>
>> >> >>>> To: "Matt Wringe" <[email protected]>
>> >> >>>> Cc: [email protected]
>> >> >>>> Sent: Monday, June 13, 2016 7:26:06 PM
>> >> >>>> Subject: Re: Metrics deployment
>> >> >>>>
>> >> >>>> Matt
>> >> >>>>
>> >> >>>> PV issue resolved. Was able to to see PV successfully bounded and
>> >> >>>> Casandra
>> >> >>>> container has been running. However, it seems puzzle not fully yet
>> >> >>>> solved.
>> >> >>>
>> >> >>>Are you sure the OpenShift DNS server is running?
>> >> >>>
>> >> >>>If you are running OSE 3.1, can you please follow this
>> >> >>>https://access.redhat.com/solutions/2329131 and see if you are now
>> >> >>>seeing
>> >> >>>errors in the Hawkular Metrics logs (essentially just run `oc exec
>> >> >>>hawkular-metrics-xxxxx cat /opt/eap/standalone/log/server.log`)
>> >> >>>
>> >> >>>>
>> >> >>>> I could see other container(heapster) not coming up, and seeing below
>> >> >>>> errors
>> >> >>>>
>> >> >>>> [skotaru@l3imas-id2-01 metrics]$ oc logs -f heapster-fnkdc
>> >> >>>> Endpoint Check in effect. Checking
>> >> >>>> https://hawkular-metrics:443/hawkular/metrics/status
>> >> >>>> Could not connect to
>> >> >>>> https://hawkular-metrics:443/hawkular/metrics/status.
>> >> >>>> Curl exit code: 6. Status Code 000
>> >> >>>> 'https://hawkular-metrics:443/hawkular/metrics/status' is not
>> >> >>>> accessible
>> >> >>>> [HTTP status code: 000. Curl exit code 6]. Retrying.
>> >> >>>> Could not connect to
>> >> >>>> https://hawkular-metrics:443/hawkular/metrics/status.
>> >> >>>> Curl exit code: 6. Status Code 000
>> >> >>>> 'https://hawkular-metrics:443/hawkular/metrics/status' is not
>> >> >>>> accessible
>> >> >>>> [HTTP status code: 000. Curl exit code 6]. Retrying.
>> >> >>>>
>> >> >>>>
>> >> >>>> # oc get pv
>> >> >>>> pv-5gb-0011 5Gi RWO Bound
>> >> >>>> openshift-infra/metrics-cassandra-1 22m
>> >> >>>>
>> >> >>>>
>> >> >>>> $ oc get pods
>> >> >>>> NAME READY STATUS RESTARTS AGE
>> >> >>>> hawkular-cassandra-1-2pzd7 1/1 Running 0 20m
>> >> >>>> hawkular-metrics-mf5qf 0/1 Running 7 20m
>> >> >>>> heapster-fnkdc 0/1 Error 6 20m
>> >> >>>> metrics-deployer-cvep0 0/1 Completed 0 21m
>> >> >>>>
>> >> >>>> # oc logs -f hawkular-metrics-mf5qf
>> >> >>>>
>> >> >>>> 19:20:00,819 INFO [org.xnio] (MSC service thread 1-2) XNIO Version
>> >> >>>> 3.0.14.GA-redhat-1
>> >> >>>> 19:20:00,831 INFO [org.jboss.as.server] (Controller Boot Thread)
>> >> >>>> JBAS015888:
>> >> >>>> Creating http management service using socket-binding
>> >> >>>> (management-http)
>> >> >>>> 19:20:00,834 INFO [org.xnio.nio] (MSC service thread 1-2) XNIO NIO
>> >> >>>> Implementation Version 3.0.14.GA-redhat-1
>> >> >>>> 19:20:00,844 INFO [org.jboss.remoting] (MSC service thread 1-2)
>> >> >>>> JBoss
>> >> >>>> Remoting version 3.3.5.Final-redhat-1
>> >> >>>>
>> >> >>>> $ oc logs -f heapster-fnkdc
>> >> >>>> Endpoint Check in effect. Checking
>> >> >>>> https://hawkular-metrics:443/hawkular/metrics/status
>> >> >>>> Could not connect to
>> >> >>>> https://hawkular-metrics:443/hawkular/metrics/status.
>> >> >>>> Curl exit code: 6. Status Code 000
>> >> >>>> 'https://hawkular-metrics:443/hawkular/metrics/status' is not
>> >> >>>> accessible
>> >> >>>> [HTTP status code: 000. Curl exit code 6]. Retrying.
>> >> >>>> Could not connect to
>> >> >>>> https://hawkular-metrics:443/hawkular/metrics/status.
>> >> >>>> Curl exit code: 6. Status Code 000
>> >> >>>> 'https://hawkular-metrics:443/hawkular/metrics/status' is not
>> >> >>>> accessible
>> >> >>>> [HTTP status code: 000. Curl exit code 6]. Retrying.
>> >> >>>> Could not connect to
>> >> >>>> https://hawkular-metrics:443/hawkular/metrics/status.
>> >> >>>> Curl exit code: 6. Status Code 000
>> >> >>>>
>> >> >>>> $ oc logs -f hawkular-cassandra-1-2pzd7
>> >> >>>> INFO 23:00:24 Starting listening for CQL clients on
>> >> >>>> hawkular-cassandra-1-2pzd7/10.1.6.2:9042...
>> >> >>>> INFO 23:00:24 Binding thrift service to
>> >> >>>> hawkular-cassandra-1-2pzd7/10.1.6.2:9160
>> >> >>>> INFO 23:00:24 enabling encrypted thrift connections between client
>> >> >>>> and
>> >> >>>> server
>> >> >>>> INFO 23:00:24 Listening for thrift clients...
>> >> >>>> INFO 23:00:26 Created default superuser role 'cassandra'
>> >> >>>>
>> >> >>>> # oc get svc
>> >> >>>> NAME CLUSTER-IP EXTERNAL-IP PORT(S)
>> >> >>>> AGE
>> >> >>>> hawkular-cassandra 172.30.2.13 <none>
>> >> >>>> 9042/TCP,9160/TCP,7000/TCP,7001/TCP 25m
>> >> >>>> hawkular-cassandra-nodes None <none>
>> >> >>>> 9042/TCP,9160/TCP,7000/TCP,7001/TCP 25m
>> >> >>>> hawkular-metrics 172.30.117.176 <none> 443/TCP
>> >> >>>> 25m
>> >> >>>> heapster 172.30.107.135 <none> 80/TCP
>> >> >>>> 25m
>> >> >>>>
>> >> >>>> #curl -I 172.30.117.176:443//hawkular/metrics/status
>> >> >>>>
>> >> >>>> HTTP/1.1 504 Gateway Timeout
>> >> >>>> Mime-Version: 1.0
>> >> >>>> Date: Mon, 13 Jun 2016 23:25:47 GMT
>> >> >>>> Content-Type: text/html
>> >> >>>> Connection: keep-alive
>> >> >>>> Proxy-Connection: keep-alive
>> >> >>>> Content-Length: 1572
>> >> >>>>
>> >> >>>> --
>> >> >>>> Srinivas Kotaru
>> >> >>>>
>> >> >>>> On 6/13/16, 2:33 PM, "Srinivas Naga Kotaru (skotaru)"
>> >> >>>> <[email protected]>
>> >> >>>> wrote:
>> >> >>>>
>> >> >>>> >Matt
>> >> >>>> >
>> >> >>>> >That is good catch. I ran without USE_PERSISTENT_STORAGE=false and
>> >> >>>> >working
>> >> >>>> >
>> >> >>>> >I adjusted PV to 5Gi and reran. Will update progress.
>> >> >>>> >
>> >> >>>> >Thanks you for your help so far.
>> >> >>>> >
>> >> >>>> >--
>> >> >>>> >Srinivas Kotaru
>> >> >>>> >
>> >> >>>> >On 6/13/16, 2:27 PM, "Matt Wringe" <[email protected]> wrote:
>> >> >>>> >
>> >> >>>> >>
>> >> >>>> >>
>> >> >>>> >>----- Original Message -----
>> >> >>>> >>> From: "Srinivas Naga Kotaru (skotaru)" <[email protected]>
>> >> >>>> >>> To: "Matt Wringe" <[email protected]>
>> >> >>>> >>> Cc: [email protected]
>> >> >>>> >>> Sent: Monday, June 13, 2016 5:21:01 PM
>> >> >>>> >>> Subject: Re: Metrics deployment
>> >> >>>> >>>
>> >> >>>> >>> Oh ok
>> >> >>>> >>>
>> >> >>>> >>> Am using PV for metrics
>> >> >>>> >>>
>> >> >>>> >>> description: "The persistent volume size for each of the
>> >> >>>> >>> Cassandra
>> >> >>>> >>> nodes"
>> >> >>>> >>> name: CASSANDRA_PV_SIZE
>> >> >>>> >>> value: "10Gi"
>> >> >>>> >>>
>> >> >>>> >>> oc get pv
>> >> >>>> >>> NAME CAPACITY ACCESSMODES STATUS CLAIM
>> >> >>>> >>> REASON
>> >> >>>> >>> AGE
>> >> >>>> >>> pv-1gb-001 1Gi RWO Available
>> >> >>>> >>> 4d
>> >> >>>> >>> pv-1gb-002 1Gi RWO Available
>> >> >>>> >>> 4d
>> >> >>>> >>> pv-1gb-003 1Gi RWO Available
>> >> >>>> >>> 4d
>> >> >>>> >>> pv-1gb-004 1Gi RWO Bound thlatt/mongodb
>> >> >>>> >>> 4d
>> >> >>>> >>> pv-1gb-005 1Gi RWO Available
>> >> >>>> >>> 4d
>> >> >>>> >>> pv-2gb-0010 2Gi RWO Available
>> >> >>>> >>> 4d
>> >> >>>> >>> pv-2gb-006 2Gi RWO Available
>> >> >>>> >>> 4d
>> >> >>>> >>> pv-2gb-007 2Gi RWO Available
>> >> >>>> >>> 4d
>> >> >>>> >>> pv-2gb-008 2Gi RWO Available
>> >> >>>> >>> 4d
>> >> >>>> >>> pv-2gb-009 2Gi RWO Available
>> >> >>>> >>> 4d
>> >> >>>> >>> pv-5gb-0011 5Gi RWO Available
>> >> >>>> >>> 4d
>> >> >>>> >>> pv-5gb-0012 5Gi RWO Available
>> >> >>>> >>> 4d
>> >> >>>> >>> pv-5gb-0013 5Gi RWO Available
>> >> >>>> >>> 4d
>> >> >>>> >>> pv-5gb-0014 5Gi RWO Available
>> >> >>>> >>> 4d
>> >> >>>> >>> pv-5gb-0015 5Gi RWO Available
>> >> >>>> >>> 4d
>> >> >>>> >>>
>> >> >>>> >>> am running with below command
>> >> >>>> >>>
>> >> >>>> >>> $ oc new-app -f metrics-deployer.yaml ( hardcoded HOSTNAME,
>> >> >>>> >>> MASTER_API
>> >> >>>> >>> and
>> >> >>>> >>> PV info so not passing any parameters)
>> >> >>>> >>>
>> >> >>>> >>
>> >> >>>> >>I would suspect that Cassandra is blocked because its waiting for
>> >> >>>> >>10Gi
>> >> >>>> >>PV
>> >> >>>> >>to become available, and none of the PV listed above are big
>> >> >>>> >>enough.
>> >> >>>> >>
>> >> >>>> >>>
>> >> >>>> >>> --
>> >> >>>> >>> Srinivas Kotaru
>> >> >>>> >>>
>> >> >>>> >>> On 6/13/16, 2:12 PM, "Matt Wringe" <[email protected]> wrote:
>> >> >>>> >>>
>> >> >>>> >>> >----- Original Message -----
>> >> >>>> >>> >> From: "Srinivas Naga Kotaru (skotaru)" <[email protected]>
>> >> >>>> >>> >> To: "Matt Wringe" <[email protected]>
>> >> >>>> >>> >> Cc: [email protected]
>> >> >>>> >>> >> Sent: Monday, June 13, 2016 4:55:55 PM
>> >> >>>> >>> >> Subject: Re: Metrics deployment
>> >> >>>> >>> >>
>> >> >>>> >>> >> Matt
>> >> >>>> >>> >>
>> >> >>>> >>> >> Thanks for looking into. I rerun the setup, but had the same
>> >> >>>> >>> >> issue
>> >> >>>> >>> >>
>> >> >>>> >>> >> # oc get pods
>> >> >>>> >>> >> NAME READY STATUS
>> >> >>>> >>> >> RESTARTS
>> >> >>>> >>> >> AGE
>> >> >>>> >>> >> hawkular-cassandra-1-y2egy 0/1 ContainerCreating 0
>> >> >>>> >>> >> 5m
>> >> >>>> >>> >> hawkular-metrics-4b16f 0/1 Running 1
>> >> >>>> >>> >> 4m
>> >> >>>> >>> >> heapster-x2gj2 0/1 Running 2
>> >> >>>> >>> >> 4m
>> >> >>>> >>> >> metrics-deployer-9v7vc 0/1 Completed 0
>> >> >>>> >>> >> 6m
>> >> >>>> >>> >>
>> >> >>>> >>> >> $ oc logs -f hawkular-cassandra-1-y2egy
>> >> >>>> >>> >> Error from server: container "hawkular-cassandra-1" in pod
>> >> >>>> >>> >> "hawkular-cassandra-1-y2egy" is waiting to start:
>> >> >>>> >>> >> ContainerCreating
>> >> >>>> >>> >
>> >> >>>> >>> >Ok, so it looks like something is blocking the Cassandra pod
>> >> >>>> >>> >from
>> >> >>>> >>> >starting.
>> >> >>>> >>> >
>> >> >>>> >>> >If you are using persistent storage, Cassandra will not start
>> >> >>>> >>> >until
>> >> >>>> >>> >the
>> >> >>>> >>> >PV
>> >> >>>> >>> >is available. There may be some more information about Cassandra
>> >> >>>> >>> >in
>> >> >>>> >>> >the
>> >> >>>> >>> >pod
>> >> >>>> >>> >section of the console under events.
>> >> >>>> >>> >
>> >> >>>> >>> >What command did you use when deploying the deployer?
>> >> >>>> >>> >
>> >> >>>> >>> >>
>> >> >>>> >>> >> $ oc logs -f hawkular-metrics-4b16f
>> >> >>>> >>> >>
>> >> >>>> >>> >> 16:54:25,703 DEBUG [org.jboss.as.config] (MSC service thread
>> >> >>>> >>> >> 1-4)
>> >> >>>> >>> >> VM
>> >> >>>> >>> >> Arguments: -Duser.home=/home/jboss -Duser.name=jboss
>> >> >>>> >>> >> -D[Standalone]
>> >> >>>> >>> >> -XX:+UseCompressedOops -verbose:gc
>> >> >>>> >>> >> -Xloggc:/opt/eap/standalone/log/gc.log
>> >> >>>> >>> >> -XX:+PrintGCDetails -XX:+PrintGCDateStamps
>> >> >>>> >>> >> -XX:+UseGCLogFileRotation
>> >> >>>> >>> >> -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=3M
>> >> >>>> >>> >> -XX:-TraceClassUnloading
>> >> >>>> >>> >> -Xms1303m -Xmx1303m -XX:MaxPermSize=256m
>> >> >>>> >>> >> -Djava.net.preferIPv4Stack=true
>> >> >>>> >>> >> -Djboss.modules.system.pkgs=org.jboss.logmanager
>> >> >>>> >>> >> -Djava.awt.headless=true
>> >> >>>> >>> >> -Djboss.modules.policy-permissions=true
>> >> >>>> >>> >> -Xbootclasspath/p:/opt/eap/jboss-modules.jar:/opt/eap/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-1.5.4.Final-redhat-1.jar:/opt/eap/modules/system/layers/base/org/jboss/logmanager/ext/main/javax.json-1.0.4.jar:/opt/eap/modules/system/layers/base/org/jboss/logmanager/ext/main/jboss-logmanager-ext-1.0.0.Alpha2-redhat-1.jar
>> >> >>>> >>> >> -Djava.util.logging.manager=org.jboss.logmanager.LogManager
>> >> >>>> >>> >> -javaagent:/opt/eap/jolokia.jar=port=8778,protocol=https,caCert=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt,clientPrincipal=cn=system:master-proxy,useSslClientAuthentication=true,extraClientCheck=true,host=0.0.0.0,discoveryEnabled=false
>> >> >>>> >>> >> -Djava.security.egd=file:/dev/./urandom
>> >> >>>> >>> >> -Dorg.jboss.boot.log.file=/opt/eap/standalone/log/server.log
>> >> >>>> >>> >> -Dlogging.configuration=file:/opt/eap/standalone/configuration/logging.properties
>> >> >>>> >>> >> 16:54:27,079 INFO [org.xnio] (MSC service thread 1-3) XNIO
>> >> >>>> >>> >> Version
>> >> >>>> >>> >> 3.0.14.GA-redhat-1
>> >> >>>> >>> >> 16:54:27,083 INFO [org.xnio.nio] (MSC service thread 1-3)
>> >> >>>> >>> >> XNIO
>> >> >>>> >>> >> NIO
>> >> >>>> >>> >> Implementation Version 3.0.14.GA-redhat-1
>> >> >>>> >>> >> 16:54:27,101 INFO [org.jboss.as.server] (Controller Boot
>> >> >>>> >>> >> Thread)
>> >> >>>> >>> >> JBAS015888:
>> >> >>>> >>> >> Creating http management service using socket-binding
>> >> >>>> >>> >> (management-http)
>> >> >>>> >>> >> 16:54:27,104 INFO [org.jboss.remoting] (MSC service thread
>> >> >>>> >>> >> 1-3)
>> >> >>>> >>> >> JBoss
>> >> >>>> >>> >> Remoting version 3.3.5.Final-redhat-1
>> >> >>>> >>> >>
>> >> >>>> >>> >> $ oc logs -f heapster-x2gj2
>> >> >>>> >>> >> Endpoint Check in effect. Checking
>> >> >>>> >>> >> https://hawkular-metrics:443/hawkular/metrics/status
>> >> >>>> >>> >> Could not connect to
>> >> >>>> >>> >> https://hawkular-metrics:443/hawkular/metrics/status.
>> >> >>>> >>> >> Curl exit code: 6. Status Code 000
>> >> >>>> >>> >> 'https://hawkular-metrics:443/hawkular/metrics/status' is not
>> >> >>>> >>> >> accessible
>> >> >>>> >>> >> [HTTP status code: 000. Curl exit code 6]. Retrying.
>> >> >>>> >>> >> Could not connect to
>> >> >>>> >>> >> https://hawkular-metrics:443/hawkular/metrics/status.
>> >> >>>> >>> >> Curl exit code: 6. Status Code 000
>> >> >>>> >>> >> 'https://hawkular-metrics:443/hawkular/metrics/status' is not
>> >> >>>> >>> >> accessible
>> >> >>>> >>> >> [HTTP status code: 000. Curl exit code 6]. Retrying.
>> >> >>>> >>> >> Could not connect to
>> >> >>>> >>> >> https://hawkular-metrics:443/hawkular/metrics/status.
>> >> >>>> >>> >> Curl exit code: 6. Status Code 000
>> >> >>>> >>> >>
>> >> >>>> >>> >>
>> >> >>>> >>> >> $ oc logs -f metrics-deployer-9v7vc
>> >> >>>> >>> >>
>> >> >>>> >>> >> ++ oc create -f -
>> >> >>>> >>> >> serviceaccount "heapster" created
>> >> >>>> >>> >> service "heapster" created
>> >> >>>> >>> >> replicationcontroller "heapster" created
>> >> >>>> >>> >> + echo 'Success!'
>> >> >>>> >>> >> Success!
>> >> >>>> >>> >>
>> >> >>>> >>> >> --
>> >> >>>> >>> >> Srinivas Kotaru
>> >> >>>> >>> >>
>> >> >>>> >>> >> On 6/13/16, 1:49 PM, "Matt Wringe" <[email protected]> wrote:
>> >> >>>> >>> >>
>> >> >>>> >>> >> >
>> >> >>>> >>> >> >
>> >> >>>> >>> >> >----- Original Message -----
>> >> >>>> >>> >> >> From: "Srinivas Naga Kotaru (skotaru)" <[email protected]>
>> >> >>>> >>> >> >> To: [email protected]
>> >> >>>> >>> >> >> Sent: Monday, June 13, 2016 3:58:12 PM
>> >> >>>> >>> >> >> Subject: Metrics deployment
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> Hi
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> Am trying to configure metrics in our newly installed
>> >> >>>> >>> >> >> clusters. Am
>> >> >>>> >>> >> >> seeing
>> >> >>>> >>> >> >> below errors once metrics-deploy script was successful. I
>> >> >>>> >>> >> >> used
>> >> >>>> >>> >> >> our
>> >> >>>> >>> >> >> environment specific HAWKULAR_METRICS_HOSTNAME and
>> >> >>>> >>> >> >> MASTER_URL
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> # oc new-app -f metrics-deployer.yaml
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> Note: customized, CASSANDARA PV, MASTER_URL, and
>> >> >>>> >>> >> >> HAWKULAR_METRICS_HOSTNAME
>> >> >>>> >>> >> >> (
>> >> >>>> >>> >> >> hard coded as values)
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> template "hawkular-heapster" created
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> Deploying the Heapster component
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> ++ echo 'Deploying the Heapster component'
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> ++ '[' -n '' ']'
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> ++ oc create -f -
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> ++ oc process hawkular-heapster -v
>> >> >>>> >>> >> >> IMAGE_PREFIX=registry.access.redhat.com/openshift3/,IMAGE_VERSION=latest,MASTER_URL=https://lae3-alln-int-idev01.cisco.com:443,NODE_ID=nodename
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> serviceaccount "heapster" created
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> service "heapster" created
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> replicationcontroller "heapster" created
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> + echo 'Success!'
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> Success!
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> # oc get pods
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> NAME READY STATUS RESTARTS AGE
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> hawkular-cassandra-1-9nzio 0/1 ContainerCreating 0 4m
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> hawkular-metrics-hi7mb 0/1 Running 1 4m
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> heapster-e8gbu 0/1 Running 2 4m
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> metrics-deployer-64703 0/1 ContainerCreating 0 3s
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> metrics-deployer-cd1nf 0/1 Completed 0 5m
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >
>> >> >>>> >>> >> >It looks like none of your containers are fully up and
>> >> >>>> >>> >> >running
>> >> >>>> >>> >> >yet.
>> >> >>>> >>> >> >
>> >> >>>> >>> >> >Without Cassandra running, Hawkular Metrics will not run, and
>> >> >>>> >>> >> >Heapster
>> >> >>>> >>> >> >will
>> >> >>>> >>> >> >wait until Hawkular Metrics is fully running.
>> >> >>>> >>> >> >
>> >> >>>> >>> >> >Do you see anything in the Cassandra logs? The first step
>> >> >>>> >>> >> >will
>> >> >>>> >>> >> >be to
>> >> >>>> >>> >> >get
>> >> >>>> >>> >> >Cassandra running properly.
>> >> >>>> >>> >> >
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> $ oc logs -f heapster-e8gbu
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> Endpoint Check in effect. Checking
>> >> >>>> >>> >> >> https://hawkular-metrics:443/hawkular/metrics/status
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> Could not connect to
>> >> >>>> >>> >> >> https://hawkular-metrics:443/hawkular/metrics/status.
>> >> >>>> >>> >> >> Curl exit code: 6. Status Code 000
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> 'https://hawkular-metrics:443/hawkular/metrics/status' is
>> >> >>>> >>> >> >> not
>> >> >>>> >>> >> >> accessible
>> >> >>>> >>> >> >> [HTTP status code: 000. Curl exit code 6]. Retrying.
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> Could not connect to
>> >> >>>> >>> >> >> https://hawkular-metrics:443/hawkular/metrics/status.
>> >> >>>> >>> >> >> Curl exit code: 6. Status Code 000
>> >> >>>> >>> >> >
>> >> >>>> >>> >> >Heapster waits until Hawkular Metrics is started before
>> >> >>>> >>> >> >trying
>> >> >>>> >>> >> >to
>> >> >>>> >>> >> >push
>> >> >>>> >>> >> >metrics to it. The issue that you are seeing is because
>> >> >>>> >>> >> >Heapster
>> >> >>>> >>> >> >could
>> >> >>>> >>> >> >not
>> >> >>>> >>> >> >properly connect to Hawkular Metrics. Until the Hawkular
>> >> >>>> >>> >> >Metrics
>> >> >>>> >>> >> >service
>> >> >>>> >>> >> >is
>> >> >>>> >>> >> >fully up, Heapster will not be able to connect to it.
>> >> >>>> >>> >> >
>> >> >>>> >>> >> >
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> What is the wrong? Why it checking just hawkular-metrics
>> >> >>>> >>> >> >> rather
>> >> >>>> >>> >> >> full
>> >> >>>> >>> >> >> routing
>> >> >>>> >>> >> >> URL which was provided as HAWKULAR_METRICS_HOSTNAME
>> >> >>>> >>> >> >
>> >> >>>> >>> >> >The Hawkular Metrics service has two hostnames: the internal
>> >> >>>> >>> >> >hostname
>> >> >>>> >>> >> >used
>> >> >>>> >>> >> >by the internal components (eg 'hawkular-metrics') and the
>> >> >>>> >>> >> >external
>> >> >>>> >>> >> >hostname (eg what is configured via
>> >> >>>> >>> >> >HAWKULAR_METRICS_HOSTNAME).
>> >> >>>> >>> >> >The
>> >> >>>> >>> >> >OpenShift dns server will resolve hostnames to the name of
>> >> >>>> >>> >> >services,
>> >> >>>> >>> >> >which
>> >> >>>> >>> >> >is where the internal 'hawkular-metrics' comes from.
>> >> >>>> >>> >> >
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> --
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> Srinivas Kotaru
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> _______________________________________________
>> >> >>>> >>> >> >> users mailing list
>> >> >>>> >>> >> >> [email protected]
>> >> >>>> >>> >> >> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>> >> >>>> >>> >> >>
>> >> >>>> >>> >>
>> >> >>>> >>> >>
>> >> >>>> >>>
>> >> >>>> >>>
>> >> >>>> >
>> >> >>>>
>> >> >>>>
>> >> >>
>> >> >
>> >>
>> >>
>>
>>
_______________________________________________
users mailing list
[email protected]
http://lists.openshift.redhat.com/openshiftmm/listinfo/users
