----- Original Message -----
> From: "Srinivas Naga Kotaru (skotaru)" <[email protected]>
> To: "Matt Wringe" <[email protected]>
> Cc: [email protected]
> Sent: Tuesday, June 14, 2016 4:44:04 PM
> Subject: Re: Metrics deployment
>
>
> I’m still struck with this issue. It is kind of chicken and egg problem.
> Heapster health probes failing since it is waiting for hawkular to start.
> Hawkular health probes are failing since it is unable to connect Casandra.
> Cansandra health probe also failing.
The first step is to get Cassandra running, ignore Hawkular Metrics and
Heapster until you have Cassandra running properly. Without Cassandra being
able to run, those other components will not fully start.
>
> @Matt: Internal DNS looks ups working. I’m able to create apps, build and
> deploy code. Router and registry components also working as expected
>
>
> oc get pods
> NAME READY STATUS RESTARTS AGE
> hawkular-cassandra-1-mxd2m 1/1 Running 0 1h
> hawkular-metrics-gvp9k 0/1 Running 4 11m
> heapster-uleul 0/1 Running 4 11m
> metrics-deployer-2z75w 0/1 Completed 0 1h
Cassandra being in "READY 1/1" means that it started up properly. So Cassandra
is running. Why do you think its not running? Things like readiness probes are
expected to fail until the pod is ready. Just because there is a failure in the
events doesn't mean its an error condition.
Hawkular Metrics is not running here. So lets figure out why. Ignore Heapster
until Hawkular Metrics is started.
>
> heapster events:
> ------------------------
>
>
> Events:
> FirstSeen LastSeen Count From
> SubobjectPath Type Reason Message
> --------- -------- ----- ----
> ------------- -------- ------ -------
> 1m 1m 1 {default-scheduler }
> Normal Scheduled Successfully
> assigned heapster-uleul to l3inpn-id2-004.cisco.com
> 1m 1m 1 {kubelet l3inpn-id2-004.cisco.com}
> spec.containers{heapster}
> Normal Pulling pulling image
> "registry.access.redhat.com/openshift3/metrics-heapster:latest"
> 1m 1m 1 {kubelet l3inpn-id2-004.cisco.com}
> spec.containers{heapster}
> Normal Pulled Successfully pulled image
> "registry.access.redhat.com/openshift3/metrics-heapster:latest"
> 1m 1m 1 {kubelet l3inpn-id2-004.cisco.com}
> spec.containers{heapster}
> Normal Created Created container with docker id
> a22bb9a246ca
> 1m 1m 1 {kubelet l3inpn-id2-004.cisco.com}
> spec.containers{heapster}
> Normal Started Started container with docker id
> a22bb9a246ca
> 1m 5s 10 {kubelet l3inpn-id2-004.cisco.com}
> spec.containers{heapster}
> Warning Unhealthy Readiness probe failed: The heapster
> process is not yet
> started, it is waiting for the Hawkular Metrics to start.
Readiness probe failing here is expected, and we know why from the error
message:
"Readiness probe failed: The heapster process is not yet started, it is waiting
for the Hawkular Metrics to start."
Once Hawkular Metrics is running then Heapster should automatically start
functioning.
>
> Hawkular events :
> ===============
>
> Events:
> FirstSeen LastSeen Count From
> SubobjectPath Type Reason
> Message
> --------- -------- ----- ----
> ------------- -------- ------
> -------
> 1m 1m 1 {default-scheduler }
> Normal Scheduled
> Successfully
> assigned hawkular-metrics-gvp9k to l3inpn-id2-004.cisco.com
> 1m 1m 1 {kubelet l3inpn-id2-004.cisco.com}
> spec.containers{hawkular-metrics} Normal Pulling
> pulling image
> "registry.access.redhat.com/openshift3/metrics-hawkular-metrics:latest"
> 1m 1m 1 {kubelet l3inpn-id2-004.cisco.com}
> spec.containers{hawkular-metrics} Normal Pulled
> Successfully pulled
> image
> "registry.access.redhat.com/openshift3/metrics-hawkular-metrics:latest"
> 1m 1m 1 {kubelet l3inpn-id2-004.cisco.com}
> spec.containers{hawkular-metrics} Normal Created
> Created container with
> docker id 6e32459624d7
> 1m 1m 1 {kubelet l3inpn-id2-004.cisco.com}
> spec.containers{hawkular-metrics} Normal Started
> Started container with
> docker id 6e32459624d7
> 1m 57s 4 {kubelet l3inpn-id2-004.cisco.com}
> spec.containers{hawkular-metrics} Warning Unhealthy
> Readiness probe
> failed: Failed to access the status endpoint : <urlopen error [Errno 111]
> Connection refused>.
>
> 46s 7s 5 {kubelet l3inpn-id2-004.cisco.com}
> spec.containers{hawkular-metrics} Warning Unhealthy
> Readiness probe
> failed:
>
>
> Casandra events
> ==============
>
> Events:
> FirstSeen LastSeen Count From
> SubobjectPath Type Reason
> Message
> --------- -------- ----- ----
> ------------- -------- ------
> -------
> 1h 1h 1 {default-scheduler }
> Normal Scheduled
> Successfully
> assigned hawkular-cassandra-1-mxd2m to l3inpn-id2-003.cisco.com
> 1h 1h 1 {kubelet l3inpn-id2-003.cisco.com}
> spec.containers{hawkular-cassandra-1} Normal Pulling
> pulling image
> "registry.access.redhat.com/openshift3/metrics-cassandra:latest"
> 1h 1h 1 {kubelet l3inpn-id2-003.cisco.com}
> spec.containers{hawkular-cassandra-1} Normal Pulled
> Successfully pulled
> image "registry.access.redhat.com/openshift3/metrics-cassandra:latest"
> 1h 1h 1 {kubelet l3inpn-id2-003.cisco.com}
> spec.containers{hawkular-cassandra-1} Normal Created
> Created container
> with docker id 2a1390edd4f1
> 1h 1h 1 {kubelet l3inpn-id2-003.cisco.com}
> spec.containers{hawkular-cassandra-1} Normal Started
> Started container
> with docker id 2a1390edd4f1
> 1h 1h 1 {kubelet l3inpn-id2-003.cisco.com}
> spec.containers{hawkular-cassandra-1} Warning Unhealthy
> Readiness probe
> failed: cat: /etc/ld.so.conf.d/*.conf: No such file or directory
> nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException:
> 'Connection refused'.
> /opt/apache-cassandra/bin/cassandra-docker-ready.sh: line 28: [: =: unary
> operator expected
> Cassandra not in the up and normal state. Current state is
>
> 1h 1h 2 {kubelet l3inpn-id2-003.cisco.com}
> spec.containers{hawkular-cassandra-1} Warning Unhealthy
> Readiness probe
> failed: cat: /etc/ld.so.conf.d/*.conf: No such file or directory
> nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException:
> 'Connection refused'.
> Cassandra not in the up and normal state. Current state is
> /opt/apache-cassandra/bin/cassandra-docker-ready.sh: line 28: [: =: unary
> operator expected
>
>
> Logs:
> ====
>
> # oc logs -f heapster-uleul
> Endpoint Check in effect. Checking
> https://hawkular-metrics:443/hawkular/metrics/status
> Could not connect to https://hawkular-metrics:443/hawkular/metrics/status.
> Curl exit code: 6. Status Code 000
> 'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible
> [HTTP status code: 000. Curl exit code 6]. Retrying.
Heapster is not starting, as expected, the problem is that it cannot access
that URL because it cannot resolve the host (error code 6). This is expected,
that hostname will not be resolvable until Hawkular Metrics starts.
> # oc logs -f hawkular-metrics-gvp9k
>
> 16:42:43,201 INFO [org.jboss.as.server] (Controller Boot Thread) JBAS015888:
> Creating http management service using socket-binding (management-http)
> 16:42:43,202 INFO [org.xnio] (MSC service thread 1-1) XNIO Version
> 3.0.14.GA-redhat-1
> 16:42:43,208 INFO [org.xnio.nio] (MSC service thread 1-1) XNIO NIO
> Implementation Version 3.0.14.GA-redhat-1
> 16:42:43,223 INFO [org.jboss.remoting] (MSC service thread 1-1) JBoss
> Remoting version 3.3.5.Final-redhat-1
I need the full logs for Hawkular Metrics, if you could get me the logs for the
last container that was run that would be helpful (eg oc get logs -p ....)
>
> # oc logs -f hawkular-cassandra-1-mxd2m
>
> INFO 20:15:20 Compacted (b7840c40-326c-11e6-a5f7-694864f768a1) 4 sstables to
> [/cassandra_data/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/la-5-big,]
> to level=0. 1,643 bytes to 1,155 (~70% of original) in 207ms =
> 0.005321MB/s. 0 total partitions merged to 11. Partition merge counts were
> {1:11, }
> INFO 20:15:20 Compacting (b7a3f050-326c-11e6-a5f7-694864f768a1)
> [/cassandra_data/data/system/schema_functions-d1b675fe2b503ca48e49c0f81989dcad/la-4-big-Data.db:level=0,
> /cassandra_data/data/system/schema_functions-d1b675fe2b503ca48e49c0f81989dcad/la-3-big-Data.db:level=0,
> /cassandra_data/data/system/schema_functions-d1b675fe2b503ca48e49c0f81989dcad/la-2-big-Data.db:level=0,
> /cassandra_data/data/system/schema_functions-d1b675fe2b503ca48e49c0f81989dcad/la-1-big-Data.db:level=0,
> ]
> INFO 20:15:20 Writing Memtable-schema_columns@672348153(49.749KiB serialized
> bytes, 967 ops, 0%/0% of on/off-heap limit)
> INFO 20:15:20 Completed flushing
> /cassandra_data/data/system/schema_columns-296e9c049bec3085827dc17d3df2122a/tmp-la-7-big-Data.db
> (11.212KiB) for commitlog position ReplayPosition(segmentId=1465931718336,
> position=159511)
> INFO 20:15:20 Compacted (b797bb50-326c-11e6-a5f7-694864f768a1) 4 sstables to
> [/cassandra_data/data/system/schema_triggers-0359bc7171233ee19a4ab9dfb11fc125/la-5-big,]
> to level=0. 128 bytes to 32 (~25% of original) in 146ms = 0.000209MB/s. 0
> total partitions merged to 1. Partition merge counts were {4:1, }
> INFO 20:15:20 Compacting (b7ae77a0-326c-11e6-a5f7-694864f768a1)
> [/cassandra_data/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/la-22-big-Data.db:level=0,
> /cassandra_data/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/la-21-big-Data.db:level=0,
> /cassandra_data/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/la-19-big-Data.db:level=0,
> /cassandra_data/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/la-20-big-Data.db:level=0,
> ]
> INFO 20:15:20 Compacted (b7a3f050-326c-11e6-a5f7-694864f768a1) 4 sstables to
> [/cassandra_data/data/system/schema_functions-d1b675fe2b503ca48e49c0f81989dcad/la-5-big,]
> to level=0. 128 bytes to 32 (~25% of original) in 155ms = 0.000197MB/s. 0
> total partitions merged to 1. Partition merge counts were {4:1, }
> INFO 20:15:20 Compacting (b7bca870-326c-11e6-a5f7-694864f768a1)
> [/cassandra_data/data/system/schema_usertypes-3aa752254f82350b8d5c430fa221fa0a/la-3-big-Data.db:level=0,
> /cassandra_data/data/system/schema_usertypes-3aa752254f82350b8d5c430fa221fa0a/la-2-big-Data.db:level=0,
> /cassandra_data/data/system/schema_usertypes-3aa752254f82350b8d5c430fa221fa0a/la-1-big-Data.db:level=0,
> /cassandra_data/data/system/schema_usertypes-3aa752254f82350b8d5c430fa221fa0a/la-4-big-Data.db:level=0,
> ]
> INFO 20:15:20 Compacted (b7bca870-326c-11e6-a5f7-694864f768a1) 4 sstables to
> [/cassandra_data/data/system/schema_usertypes-3aa752254f82350b8d5c430fa221fa0a/la-5-big,]
> to level=0. 128 bytes to 32 (~25% of original) in 83ms = 0.000368MB/s. 0
> total partitions merged to 1. Partition merge counts were {4:1, }
> INFO 20:15:20 Compacted (b7ae77a0-326c-11e6-a5f7-694864f768a1) 4 sstables to
> [/cassandra_data/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/la-23-big,]
> to level=0. 2,538 bytes to 527 (~20% of original) in 176ms = 0.002856MB/s.
> 0 total partitions merged to 3. Partition merge counts were {4:3, }
>
>
>
> almost struck here. Any points to look for? Any ports need to be open
> explicitly ?
>
>
>
>
>
>
>
> --
> Srinivas Kotaru
>
> On 6/14/16, 11:27 AM, "Matt Wringe" <[email protected]> wrote:
>
> >----- Original Message -----
> >> From: "Srinivas Naga Kotaru (skotaru)" <[email protected]>
> >> To: "Matt Wringe" <[email protected]>
> >> Cc: [email protected]
> >> Sent: Tuesday, June 14, 2016 2:09:49 PM
> >> Subject: Re: Metrics deployment
> >>
> >> Not sure what do you mean by installation. Am just running oc new-app -f
> >> metrics-deployer.yaml with default values except HOST_NAME and PV storage.
> >
> >I would suspect something wrong with your node or cluster installation. Or
> >you have firewall rules blocking connections between your nodes so that pod
> >cannot connect with each other or access the OpenShift DNS server.
> >
> >The lifecycle hooks exist to make sure that components only enter the ready
> >state when they are fully started and ready.
> >
> >Can you check the Hawkular Metrics status page and see what that outputs? eg
> >https://${HAWKULAR_METRICS_HOSTNAME}/hawkular/metrics/status
> >
> >>
> >> I just deleted entire metrics setup and re running. But not sure this will
> >> fix the issue.
> >>
> >> $ ./delete_metrics-infra.sh
> >> replicationcontroller "hawkular-cassandra-1" deleted
> >> replicationcontroller "hawkular-metrics" deleted
> >> replicationcontroller "heapster" deleted
> >> route "hawkular-metrics" deleted
> >> service "hawkular-cassandra" deleted
> >> service "hawkular-cassandra-nodes" deleted
> >> service "hawkular-metrics" deleted
> >> service "heapster" deleted
> >> pod "heapster-lyf65" deleted
> >> serviceaccount "cassandra" deleted
> >> serviceaccount "hawkular" deleted
> >> serviceaccount "heapster" deleted
> >> template "hawkular-cassandra-node-emptydir" deleted
> >> template "hawkular-cassandra-node-pv" deleted
> >> template "hawkular-cassandra-services" deleted
> >> template "hawkular-heapster" deleted
> >> template "hawkular-metrics" deleted
> >> template "hawkular-support" deleted
> >> secret "hawkular-cassandra-certificate" deleted
> >> secret "hawkular-cassandra-secrets" deleted
> >> secret "hawkular-metrics-account" deleted
> >> secret "hawkular-metrics-certificate" deleted
> >> secret "hawkular-metrics-secrets" deleted
> >> secret "heapster-secrets" deleted
> >>
> >> --
> >> Srinivas Kotaru
> >>
> >> On 6/14/16, 10:53 AM, "Matt Wringe" <[email protected]> wrote:
> >>
> >> >----- Original Message -----
> >> >> From: "Srinivas Naga Kotaru (skotaru)" <[email protected]>
> >> >> To: "Matt Wringe" <[email protected]>
> >> >> Cc: [email protected]
> >> >> Sent: Tuesday, June 14, 2016 1:37:01 PM
> >> >> Subject: Re: Metrics deployment
> >> >>
> >> >> I removed readiness probes from both hawkular-cassandra-1 &
> >> >> hawkular-metrics
> >> >> as both status shows probes failed.
> >> >
> >> >You should not have to remove the probes, this indicates that something
> >> >is
> >> >wrong with your installation.
> >> >
> >> >>
> >> >> It looks good now. Both containers looks and running
> >> >> (hawkular-cassandra-1-kr8ka , hawkular-metrics-vhe3u) however
> >> >> heapster-7yl34
> >> >> logs still shows Could not connect to
> >> >> https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code:
> >> >> 6.
> >> >> Status Code 000.
> >> >>
> >> >> Are we good or still had issues?
> >> >>
> >> >>
> >> >> # oc get pods
> >> >> NAME READY STATUS RESTARTS AGE
> >> >> hawkular-cassandra-1-kr8ka 1/1 Running 0 6m
> >> >> hawkular-metrics-vhe3u 1/1 Running 2 5m
> >> >> heapster-7yl34 0/1 Running 2 5m
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Srinivas Kotaru
> >> >>
> >> >> On 6/14/16, 10:07 AM, "Srinivas Naga Kotaru (skotaru)"
> >> >> <[email protected]>
> >> >> wrote:
> >> >>
> >> >> >Matt
> >> >> >
> >> >> >Just want to share more info by running describe pod.
> >> >> >
> >> >> >It seems to be health probe failing. Do you think it is the issue?
> >> >> >
> >> >> >
> >> >> >
> >> >> ># oc describe pod hawkular-cassandra-1-it5uh
> >> >> >Name: hawkular-cassandra-1-it5uh
> >> >> >Namespace: openshift-infra
> >> >> >Node: l3inpn-id2-003.cisco.com/173.36.96.16
> >> >> >Start Time: Tue, 14 Jun 2016 16:36:21 +0000
> >> >> >Labels:
> >> >> >
> >> >> > metrics-infra=hawkular-cassandra,name=hawkular-cassandra-1,type=hawkular-cassandra
> >> >> >Status: Running
> >> >> >IP: 10.1.9.2
> >> >> >Controllers: ReplicationController/hawkular-cassandra-1
> >> >> >Containers:
> >> >> > hawkular-cassandra-1:
> >> >> > Container ID:
> >> >> >
> >> >> > docker://17a9575eb655145859a9207f5c4bde7456f947e27188a056ff2bd08c4ce6ae5d
> >> >> > Image:
> >> >> >
> >> >> > registry.access.redhat.com/openshift3/metrics-cassandra:latest
> >> >> > Image ID:
> >> >> >
> >> >> > docker://ee2117c9848298ca5a0cbbce354fd4adff370435225324ab9d60cd9cd9a95c53
> >> >> > Ports: 9042/TCP, 9160/TCP, 7000/TCP, 7001/TCP
> >> >> > Command:
> >> >> > /opt/apache-cassandra/bin/cassandra-docker.sh
> >> >> > --cluster_name=hawkular-metrics
> >> >> > --data_volume=/cassandra_data
> >> >> > --internode_encryption=all
> >> >> > --require_node_auth=true
> >> >> > --enable_client_encryption=true
> >> >> > --require_client_auth=true
> >> >> > --keystore_file=/secret/cassandra.keystore
> >> >> > --keystore_password_file=/secret/cassandra.keystore.password
> >> >> > --truststore_file=/secret/cassandra.truststore
> >> >> > --truststore_password_file=/secret/cassandra.truststore.password
> >> >> > --cassandra_pem_file=/secret/cassandra.pem
> >> >> > QoS Tier:
> >> >> > cpu: BestEffort
> >> >> > memory: BestEffort
> >> >> > State: Running
> >> >> > Started: Tue, 14 Jun 2016 16:37:01 +0000
> >> >> > Ready: True
> >> >> > Restart Count: 0
> >> >> > Readiness: exec
> >> >> > [/opt/apache-cassandra/bin/cassandra-docker-ready.sh]
> >> >> > delay=0s timeout=1s period=10s #success=1 #failure=3
> >> >> > Environment Variables:
> >> >> > CASSANDRA_MASTER: true
> >> >> > POD_NAMESPACE: openshift-infra (v1:metadata.namespace)
> >> >> >Conditions:
> >> >> > Type Status
> >> >> > Ready True
> >> >> >Volumes:
> >> >> > cassandra-data:
> >> >> > Type: PersistentVolumeClaim (a reference to a
> >> >> > PersistentVolumeClaim
> >> >> > in
> >> >> > the same namespace)
> >> >> > ClaimName: metrics-cassandra-1
> >> >> > ReadOnly: false
> >> >> > hawkular-cassandra-secrets:
> >> >> > Type: Secret (a volume populated by a Secret)
> >> >> > SecretName: hawkular-cassandra-secrets
> >> >> > cassandra-token-4urfd:
> >> >> > Type: Secret (a volume populated by a Secret)
> >> >> > SecretName: cassandra-token-4urfd
> >> >> >Events:
> >> >> > FirstSeen LastSeen Count From
> >> >> > SubobjectPath Type Reason
> >> >> > Message
> >> >> > --------- -------- ----- ----
> >> >> > ------------- -------- ------
> >> >> > -------
> >> >> > 27m 27m 1 {default-scheduler }
> >> >> > Normal
> >> >> > Scheduled
> >> >> > Successfully
> >> >> > assigned hawkular-cassandra-1-it5uh to l3inpn-id2-003.cisco.com
> >> >> > 27m 27m 1 {kubelet
> >> >> > l3inpn-id2-003.cisco.com}
> >> >> > spec.containers{hawkular-cassandra-1} Normal Pulling
> >> >> > pulling
> >> >> > image
> >> >> > "registry.access.redhat.com/openshift3/metrics-cassandra:latest"
> >> >> > 27m 27m 1 {kubelet
> >> >> > l3inpn-id2-003.cisco.com}
> >> >> > spec.containers{hawkular-cassandra-1} Normal Pulled
> >> >> > Successfully
> >> >> > pulled
> >> >> > image
> >> >> > "registry.access.redhat.com/openshift3/metrics-cassandra:latest"
> >> >> > 27m 27m 1 {kubelet
> >> >> > l3inpn-id2-003.cisco.com}
> >> >> > spec.containers{hawkular-cassandra-1} Normal Created
> >> >> > Created
> >> >> > container
> >> >> > with docker id 17a9575eb655
> >> >> > 27m 27m 1 {kubelet
> >> >> > l3inpn-id2-003.cisco.com}
> >> >> > spec.containers{hawkular-cassandra-1} Normal Started
> >> >> > Started
> >> >> > container
> >> >> > with docker id 17a9575eb655
> >> >> > 27m 26m 3 {kubelet
> >> >> > l3inpn-id2-003.cisco.com}
> >> >> > spec.containers{hawkular-cassandra-1} Warning
> >> >> > Unhealthy Readiness
> >> >> > probe
> >> >> > failed: cat: /etc/ld.so.conf.d/*.conf: No such file or directory
> >> >> >nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException:
> >> >> >'Connection refused'.
> >> >> >Cassandra not in the up and normal state. Current state is
> >> >> >/opt/apache-cassandra/bin/cassandra-docker-ready.sh: line 28: [: =:
> >> >> >unary
> >> >> >operator expected
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> >--
> >> >> >Srinivas Kotaru
> >> >> >
> >> >> >On 6/14/16, 10:00 AM, "Srinivas Naga Kotaru (skotaru)"
> >> >> ><[email protected]>
> >> >> >wrote:
> >> >> >
> >> >> >>Matt
> >> >> >>
> >> >> >>DNS service is working
> >> >> >>
> >> >> >>
> >> >> >>~ dig +short @master-01 kubernetes.default.svc.cluster.local
> >> >> >>172.30.0.1
> >> >> >>~ dig +short @master-01 jenkins.alln-test.svc.cluster.local
> >> >> >>172.30.85.148
> >> >> >>~ dig +short @master-01 cakephp-example.alln-test.svc.cluster.local
> >> >> >>172.30.31.6
> >> >> >>
> >> >> >>I captured hawkular-metrics, it shows the problem. It seems to be
> >> >> >>unable
> >> >> >>to
> >> >> >>connect Cassandra cluster
> >> >> >>
> >> >> >># oc exec hawkular-metrics-qldd8 cat
> >> >> >>/opt/eap/standalone/log/server.log
> >> >> >>
> >> >> >>12:40:58,848 INFO [org.jboss.as.clustering.infinispan]
> >> >> >>(ServerService
> >> >> >>Thread Pool -- 31) JBAS010280: Activating Infinispan subsystem.
> >> >> >>12:40:58,898 INFO [org.jboss.as.webservices] (ServerService Thread
> >> >> >>Pool
> >> >> >>--
> >> >> >>48) JBAS015537: Activating WebServices Extension
> >> >> >>12:40:58,901 WARN [org.jboss.as.txn] (ServerService Thread Pool --
> >> >> >>46)
> >> >> >>JBAS010153: Node identifier property is set to the default value.
> >> >> >>Please
> >> >> >>make sure it is unique.
> >> >> >>12:40:58,902 INFO [org.jboss.as.security] (ServerService Thread Pool
> >> >> >>--
> >> >> >>44) JBAS013371: Activating Security Subsystem
> >> >> >>12:40:58,912 INFO [org.jboss.as.naming] (ServerService Thread Pool
> >> >> >>--
> >> >> >>39)
> >> >> >>JBAS011800: Activating Naming Subsystem
> >> >> >>12:40:58,928 INFO [org.jboss.as.jsf] (ServerService Thread Pool --
> >> >> >>37)
> >> >> >>JBAS012615: Activated the following JSF Implementations: [main, 1.2]
> >> >> >>12:40:58,939 INFO [org.jboss.as.connector.logging] (MSC service
> >> >> >>thread
> >> >> >>1-2) JBAS010408: Starting JCA Subsystem (IronJacamar
> >> >> >>1.0.32.Final-redhat-1)
> >> >> >>12:40:58,981 INFO [org.jboss.as.security] (MSC service thread 1-2)
> >> >> >>JBAS013370: Current PicketBox version=4.1.1.Final-redhat-1
> >> >> >>12:40:58,961 INFO [org.jboss.as.connector.subsystems.datasources]
> >> >> >>(ServerService Thread Pool -- 27) JBAS010403: Deploying
> >> >> >>JDBC-compliant
> >> >> >>driver class org.h2.Driver (version 1.3)
> >> >> >>12:40:59,039 INFO [org.jboss.as.naming] (MSC service thread 1-2)
> >> >> >>JBAS011802: Starting Naming Service
> >> >> >>12:40:59,052 INFO [org.jboss.as.mail.extension] (MSC service thread
> >> >> >>1-4)
> >> >> >>JBAS015400: Bound mail session [java:jboss/mail/Default]
> >> >> >>12:40:59,614 INFO [org.apache.coyote.http11.Http11Protocol] (MSC
> >> >> >>service
> >> >> >>thread 1-1) JBWEB003001: Coyote HTTP/1.1 initializing on :
> >> >> >>http-/0.0.0.0:8444
> >> >> >>12:40:59,620 INFO [org.apache.coyote.http11.Http11Protocol] (MSC
> >> >> >>service
> >> >> >>thread 1-2) JBWEB003001: Coyote HTTP/1.1 initializing on :
> >> >> >>http-/0.0.0.0:8080
> >> >> >>12:40:59,621 INFO [org.apache.coyote.http11.Http11Protocol] (MSC
> >> >> >>service
> >> >> >>thread 1-1) JBWEB003000: Coyote HTTP/1.1 starting on:
> >> >> >>http-/0.0.0.0:8444
> >> >> >>12:40:59,627 INFO [org.apache.coyote.http11.Http11Protocol] (MSC
> >> >> >>service
> >> >> >>thread 1-2) JBWEB003000: Coyote HTTP/1.1 starting on:
> >> >> >>http-/0.0.0.0:8080
> >> >> >>12:40:59,654 INFO [org.jboss.as.connector.subsystems.datasources]
> >> >> >>(MSC
> >> >> >>service thread 1-3) JBAS010400: Bound data source
> >> >> >>[java:jboss/datasources/ExampleDS]
> >> >> >>12:40:59,985 INFO [org.jboss.as.server.deployment] (MSC service
> >> >> >>thread
> >> >> >>1-2) JBAS015876: Starting deployment of "activemq-rar.rar"
> >> >> >>(runtime-name:
> >> >> >>"activemq-rar.rar")
> >> >> >>12:40:59,986 INFO [org.jboss.as.server.deployment] (MSC service
> >> >> >>thread
> >> >> >>1-1) JBAS015876: Starting deployment of
> >> >> >>"hawkular-metrics-api-jaxrs.war"
> >> >> >>(runtime-name: "hawkular-metrics-api-jaxrs.war")
> >> >> >>12:40:59,987 INFO [org.jboss.as.server.deployment.scanner] (MSC
> >> >> >>service
> >> >> >>thread 1-3) JBAS015012: Started FileSystemDeploymentService for
> >> >> >>directory
> >> >> >>/opt/eap/standalone/deployments
> >> >> >>12:41:00,011 INFO [org.jboss.as.remoting] (MSC service thread 1-2)
> >> >> >>JBAS017100: Listening on 127.0.0.1:9999
> >> >> >>12:41:00,030 WARN [org.jboss.as.domain.http.api] (MSC service thread
> >> >> >>1-3)
> >> >> >>JBAS015102: Unable to load console module for slot eap, disabling
> >> >> >>console
> >> >> >>12:41:00,042 INFO [org.jboss.ws.common.management] (MSC service
> >> >> >>thread
> >> >> >>1-4) JBWS022052: Starting JBoss Web Services - Stack CXF Server
> >> >> >>4.3.5.Final-redhat-3
> >> >> >>12:41:00,045 INFO [org.jboss.as.remoting] (MSC service thread 1-4)
> >> >> >>JBAS017100: Listening on 0.0.0.0:4447
> >> >> >>12:41:01,361 INFO [org.jboss.as.connector.deployers.RADeployer] (MSC
> >> >> >>service thread 1-3) IJ020001: Required license terms for
> >> >> >>file:/opt/eap/standalone/tmp/vfs/temp/temp912bb9d1372c33d3/activemq-rar.rar-a722089a00658182/contents/
> >> >> >>12:41:01,784 WARN [org.jboss.as.dependency.private] (MSC service
> >> >> >>thread
> >> >> >>1-1) JBAS015867: Deployment
> >> >> >>"deployment.hawkular-metrics-api-jaxrs.war"
> >> >> >>is
> >> >> >>using a private module ("org.codehaus.jackson.jackson-core-asl:main")
> >> >> >>which may be changed or removed in future versions without notice.
> >> >> >>12:41:01,785 WARN [org.jboss.as.dependency.private] (MSC service
> >> >> >>thread
> >> >> >>1-1) JBAS015867: Deployment
> >> >> >>"deployment.hawkular-metrics-api-jaxrs.war"
> >> >> >>is
> >> >> >>using a private module ("org.codehaus.jackson.jackson-core-asl:main")
> >> >> >>which may be changed or removed in future versions without notice.
> >> >> >>12:41:01,785 WARN [org.jboss.as.dependency.private] (MSC service
> >> >> >>thread
> >> >> >>1-1) JBAS015867: Deployment
> >> >> >>"deployment.hawkular-metrics-api-jaxrs.war"
> >> >> >>is
> >> >> >>using a private module
> >> >> >>("org.codehaus.jackson.jackson-mapper-asl:main")
> >> >> >>which may be changed or removed in future versions without notice.
> >> >> >>12:41:01,785 WARN [org.jboss.as.dependency.private] (MSC service
> >> >> >>thread
> >> >> >>1-1) JBAS015867: Deployment
> >> >> >>"deployment.hawkular-metrics-api-jaxrs.war"
> >> >> >>is
> >> >> >>using a private module
> >> >> >>("org.codehaus.jackson.jackson-mapper-asl:main")
> >> >> >>which may be changed or removed in future versions without notice.
> >> >> >>12:41:01,843 INFO [org.jboss.weld.deployer] (MSC service thread 1-1)
> >> >> >>JBAS016002: Processing weld deployment hawkular-metrics-api-jaxrs.war
> >> >> >>12:41:01,941 INFO [org.jboss.weld.deployer] (MSC service thread 1-1)
> >> >> >>JBAS016005: Starting Services for CDI deployment:
> >> >> >>hawkular-metrics-api-jaxrs.war
> >> >> >>12:41:01,986 INFO [org.jboss.weld.Version] (MSC service thread 1-1)
> >> >> >>WELD-000900 1.1.31 (redhat)
> >> >> >>12:41:02,013 INFO [org.jboss.weld.deployer] (MSC service thread 1-3)
> >> >> >>JBAS016008: Starting weld service for deployment
> >> >> >>hawkular-metrics-api-jaxrs.war
> >> >> >>12:41:02,820 INFO
> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
> >> >> >>metrics
> >> >> >>service
> >> >> >>12:41:02,835 INFO [org.jboss.web] (ServerService Thread Pool -- 51)
> >> >> >>JBAS018210: Register web context: /hawkular/metrics
> >> >> >>12:41:02,958 INFO [org.jboss.as.server] (ServerService Thread Pool
> >> >> >>--
> >> >> >>28)
> >> >> >>JBAS015859: Deployed "hawkular-metrics-api-jaxrs.war" (runtime-name :
> >> >> >>"hawkular-metrics-api-jaxrs.war")
> >> >> >>12:41:02,959 INFO [org.jboss.as.server] (ServerService Thread Pool
> >> >> >>--
> >> >> >>28)
> >> >> >>JBAS015859: Deployed "activemq-rar.rar" (runtime-name :
> >> >> >>"activemq-rar.rar")
> >> >> >>12:41:02,994 INFO [org.jboss.as] (Controller Boot Thread)
> >> >> >>JBAS015961:
> >> >> >>Http
> >> >> >>management interface listening on http://127.0.0.1:9990/management
> >> >> >>12:41:02,994 INFO [org.jboss.as] (Controller Boot Thread)
> >> >> >>JBAS015951:
> >> >> >>Admin console listening on http://127.0.0.1:9990
> >> >> >>12:41:02,994 INFO [org.jboss.as] (Controller Boot Thread)
> >> >> >>JBAS015874:
> >> >> >>JBoss EAP 6.4.4.GA (AS 7.5.4.Final-redhat-4) started in 6593ms -
> >> >> >>Started
> >> >> >>267 of 303 services (62 services are lazy, passive or on-demand)
> >> >> >>12:41:02,999 WARN
> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not
> >> >> >>connect
> >> >> >>to
> >> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra:
> >> >> >>unknown
> >> >> >>error
> >> >> >>12:41:02,999 WARN
> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [1] Retrying
> >> >> >>connecting to Cassandra cluster in [1]s...
> >> >> >>12:41:04,000 INFO
> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
> >> >> >>metrics
> >> >> >>service
> >> >> >>12:41:04,001 WARN
> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not
> >> >> >>connect
> >> >> >>to
> >> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
> >> >> >>12:41:04,001 WARN
> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [2] Retrying
> >> >> >>connecting to Cassandra cluster in [2]s...
> >> >> >>12:41:06,002 INFO
> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
> >> >> >>metrics
> >> >> >>service
> >> >> >>12:41:06,002 WARN
> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not
> >> >> >>connect
> >> >> >>to
> >> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
> >> >> >>12:41:06,003 WARN
> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [3] Retrying
> >> >> >>connecting to Cassandra cluster in [3]s...
> >> >> >>12:41:08,203 INFO [org.jboss.resteasy.cdi.i18n]
> >> >> >>(http-/0.0.0.0:8080-2)
> >> >> >>RESTEASY006050: Found BeanManager at java:comp/BeanManager
> >> >> >>12:41:08,299 INFO [org.hibernate.validator.internal.util.Version]
> >> >> >>(http-/0.0.0.0:8080-2) HV000001: Hibernate Validator
> >> >> >>4.3.2.Final-redhat-2
> >> >> >>12:41:08,418 INFO
> >> >> >>[org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp]
> >> >> >>(http-/0.0.0.0:8080-2) HAWKMETRICS200001: Hawkular Metrics starting
> >> >> >>12:41:08,421 INFO [org.jboss.resteasy.resteasy_jaxrs.i18n]
> >> >> >>(http-/0.0.0.0:8080-2) RESTEASY000320: Deploying
> >> >> >>javax.ws.rs.core.Application: class
> >> >> >>org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp$Proxy$_$$_WeldClientProxy
> >> >> >>12:41:08,421 INFO
> >> >> >>[org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp]
> >> >> >>(http-/0.0.0.0:8080-2) HAWKMETRICS200001: Hawkular Metrics starting
> >> >> >>12:41:08,422 INFO
> >> >> >>[org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp]
> >> >> >>(http-/0.0.0.0:8080-2) Virtual clock is disabled
> >> >> >>12:41:08,422 INFO
> >> >> >>[org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp]
> >> >> >>(http-/0.0.0.0:8080-2) Virtual clock is disabled
> >> >> >>12:41:09,003 INFO
> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
> >> >> >>metrics
> >> >> >>service
> >> >> >>12:41:09,004 WARN
> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not
> >> >> >>connect
> >> >> >>to
> >> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
> >> >> >>12:41:09,004 WARN
> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [4] Retrying
> >> >> >>connecting to Cassandra cluster in [4]s...
> >> >> >>12:41:13,004 INFO
> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
> >> >> >>metrics
> >> >> >>service
> >> >> >>12:41:13,123 WARN
> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not
> >> >> >>connect
> >> >> >>to
> >> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra:
> >> >> >>unknown
> >> >> >>error
> >> >> >>12:41:13,124 WARN
> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [5] Retrying
> >> >> >>connecting to Cassandra cluster in [1]s...
> >> >> >>12:41:14,124 INFO
> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
> >> >> >>metrics
> >> >> >>service
> >> >> >>12:41:14,124 WARN
> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not
> >> >> >>connect
> >> >> >>to
> >> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
> >> >> >>12:41:14,125 WARN
> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [6] Retrying
> >> >> >>connecting to Cassandra cluster in [2]s...
> >> >> >>12:41:16,125 INFO
> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
> >> >> >>metrics
> >> >> >>service
> >> >> >>12:41:16,126 WARN
> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not
> >> >> >>connect
> >> >> >>to
> >> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
> >> >> >>12:41:16,126 WARN
> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [7] Retrying
> >> >> >>connecting to Cassandra cluster in [3]s...
> >> >> >>12:41:19,126 INFO
> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
> >> >> >>metrics
> >> >> >>service
> >> >> >>12:41:19,127 WARN
> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not
> >> >> >>connect
> >> >> >>to
> >> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
> >> >> >>12:41:19,127 WARN
> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [8] Retrying
> >> >> >>connecting to Cassandra cluster in [4]s...
> >> >> >>12:41:23,128 INFO
> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
> >> >> >>metrics
> >> >> >>service
> >> >> >>12:41:23,249 WARN
> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not
> >> >> >>connect
> >> >> >>to
> >> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra:
> >> >> >>unknown
> >> >> >>error
> >> >> >>12:41:23,249 WARN
> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [9] Retrying
> >> >> >>connecting to Cassandra cluster in [1]s...
> >> >> >>12:41:24,250 INFO
> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
> >> >> >>metrics
> >> >> >>service
> >> >> >>12:41:24,250 WARN
> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not
> >> >> >>connect
> >> >> >>to
> >> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
> >> >> >>12:41:24,250 WARN
> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [10] Retrying
> >> >> >>connecting to Cassandra cluster in [2]s...
> >> >> >>12:41:26,251 INFO
> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
> >> >> >>metrics
> >> >> >>service
> >> >> >>12:41:26,251 WARN
> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not
> >> >> >>connect
> >> >> >>to
> >> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
> >> >> >>12:41:26,251 WARN
> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [11] Retrying
> >> >> >>connecting to Cassandra cluster in [3]s...
> >> >> >>12:41:29,252 INFO
> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
> >> >> >>metrics
> >> >> >>service
> >> >> >>12:41:29,252 WARN
> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not
> >> >> >>connect
> >> >> >>to
> >> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
> >> >> >>12:41:29,253 WARN
> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [12] Retrying
> >> >> >>connecting to Cassandra cluster in [4]s...
> >> >> >>12:41:33,253 INFO
> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
> >> >> >>metrics
> >> >> >>service
> >> >> >>12:41:33,376 WARN
> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not
> >> >> >>connect
> >> >> >>to
> >> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra:
> >> >> >>unknown
> >> >> >>error
> >> >> >>12:41:33,377 WARN
> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [13] Retrying
> >> >> >>connecting to Cassandra cluster in [1]s...
> >> >> >>12:41:34,377 INFO
> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
> >> >> >>metrics
> >> >> >>service
> >> >> >>12:41:34,378 WARN
> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not
> >> >> >>connect
> >> >> >>to
> >> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
> >> >> >>12:41:34,378 WARN
> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [14] Retrying
> >> >> >>connecting to Cassandra cluster in [2]s...
> >> >> >>12:41:36,378 INFO
> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
> >> >> >>metrics
> >> >> >>service
> >> >> >>12:41:36,379 WARN
> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not
> >> >> >>connect
> >> >> >>to
> >> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
> >> >> >>12:41:36,379 WARN
> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [15] Retrying
> >> >> >>connecting to Cassandra cluster in [3]s...
> >> >> >>12:41:39,379 INFO
> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
> >> >> >>metrics
> >> >> >>service
> >> >> >>12:41:39,380 WARN
> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not
> >> >> >>connect
> >> >> >>to
> >> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
> >> >> >>12:41:39,380 WARN
> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [16] Retrying
> >> >> >>connecting to Cassandra cluster in [4]s...
> >> >> >>12:41:43,380 INFO
> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
> >> >> >>metrics
> >> >> >>service
> >> >> >>12:41:43,503 WARN
> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not
> >> >> >>connect
> >> >> >>to
> >> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra:
> >> >> >>unknown
> >> >> >>error
> >> >> >>12:41:43,504 WARN
> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [17] Retrying
> >> >> >>connecting to Cassandra cluster in [1]s...
> >> >> >>12:41:44,504 INFO
> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
> >> >> >>metrics
> >> >> >>service
> >> >> >>12:41:44,505 WARN
> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not
> >> >> >>connect
> >> >> >>to
> >> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
> >> >> >>12:41:44,505 WARN
> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [18] Retrying
> >> >> >>connecting to Cassandra cluster in [2]s...
> >> >> >>
> >> >> >>--
> >> >> >>Srinivas Kotaru
> >> >> >>
> >> >> >>On 6/14/16, 6:06 AM, "Matt Wringe" <[email protected]> wrote:
> >> >> >>
> >> >> >>>----- Original Message -----
> >> >> >>>> From: "Srinivas Naga Kotaru (skotaru)" <[email protected]>
> >> >> >>>> To: "Matt Wringe" <[email protected]>
> >> >> >>>> Cc: [email protected]
> >> >> >>>> Sent: Monday, June 13, 2016 7:26:06 PM
> >> >> >>>> Subject: Re: Metrics deployment
> >> >> >>>>
> >> >> >>>> Matt
> >> >> >>>>
> >> >> >>>> PV issue resolved. Was able to to see PV successfully bounded and
> >> >> >>>> Casandra
> >> >> >>>> container has been running. However, it seems puzzle not fully yet
> >> >> >>>> solved.
> >> >> >>>
> >> >> >>>Are you sure the OpenShift DNS server is running?
> >> >> >>>
> >> >> >>>If you are running OSE 3.1, can you please follow this
> >> >> >>>https://access.redhat.com/solutions/2329131 and see if you are now
> >> >> >>>seeing
> >> >> >>>errors in the Hawkular Metrics logs (essentially just run `oc exec
> >> >> >>>hawkular-metrics-xxxxx cat /opt/eap/standalone/log/server.log`)
> >> >> >>>
> >> >> >>>>
> >> >> >>>> I could see other container(heapster) not coming up, and seeing
> >> >> >>>> below
> >> >> >>>> errors
> >> >> >>>>
> >> >> >>>> [skotaru@l3imas-id2-01 metrics]$ oc logs -f heapster-fnkdc
> >> >> >>>> Endpoint Check in effect. Checking
> >> >> >>>> https://hawkular-metrics:443/hawkular/metrics/status
> >> >> >>>> Could not connect to
> >> >> >>>> https://hawkular-metrics:443/hawkular/metrics/status.
> >> >> >>>> Curl exit code: 6. Status Code 000
> >> >> >>>> 'https://hawkular-metrics:443/hawkular/metrics/status' is not
> >> >> >>>> accessible
> >> >> >>>> [HTTP status code: 000. Curl exit code 6]. Retrying.
> >> >> >>>> Could not connect to
> >> >> >>>> https://hawkular-metrics:443/hawkular/metrics/status.
> >> >> >>>> Curl exit code: 6. Status Code 000
> >> >> >>>> 'https://hawkular-metrics:443/hawkular/metrics/status' is not
> >> >> >>>> accessible
> >> >> >>>> [HTTP status code: 000. Curl exit code 6]. Retrying.
> >> >> >>>>
> >> >> >>>>
> >> >> >>>> # oc get pv
> >> >> >>>> pv-5gb-0011 5Gi RWO Bound
> >> >> >>>> openshift-infra/metrics-cassandra-1 22m
> >> >> >>>>
> >> >> >>>>
> >> >> >>>> $ oc get pods
> >> >> >>>> NAME READY STATUS RESTARTS AGE
> >> >> >>>> hawkular-cassandra-1-2pzd7 1/1 Running 0 20m
> >> >> >>>> hawkular-metrics-mf5qf 0/1 Running 7 20m
> >> >> >>>> heapster-fnkdc 0/1 Error 6 20m
> >> >> >>>> metrics-deployer-cvep0 0/1 Completed 0 21m
> >> >> >>>>
> >> >> >>>> # oc logs -f hawkular-metrics-mf5qf
> >> >> >>>>
> >> >> >>>> 19:20:00,819 INFO [org.xnio] (MSC service thread 1-2) XNIO
> >> >> >>>> Version
> >> >> >>>> 3.0.14.GA-redhat-1
> >> >> >>>> 19:20:00,831 INFO [org.jboss.as.server] (Controller Boot Thread)
> >> >> >>>> JBAS015888:
> >> >> >>>> Creating http management service using socket-binding
> >> >> >>>> (management-http)
> >> >> >>>> 19:20:00,834 INFO [org.xnio.nio] (MSC service thread 1-2) XNIO
> >> >> >>>> NIO
> >> >> >>>> Implementation Version 3.0.14.GA-redhat-1
> >> >> >>>> 19:20:00,844 INFO [org.jboss.remoting] (MSC service thread 1-2)
> >> >> >>>> JBoss
> >> >> >>>> Remoting version 3.3.5.Final-redhat-1
> >> >> >>>>
> >> >> >>>> $ oc logs -f heapster-fnkdc
> >> >> >>>> Endpoint Check in effect. Checking
> >> >> >>>> https://hawkular-metrics:443/hawkular/metrics/status
> >> >> >>>> Could not connect to
> >> >> >>>> https://hawkular-metrics:443/hawkular/metrics/status.
> >> >> >>>> Curl exit code: 6. Status Code 000
> >> >> >>>> 'https://hawkular-metrics:443/hawkular/metrics/status' is not
> >> >> >>>> accessible
> >> >> >>>> [HTTP status code: 000. Curl exit code 6]. Retrying.
> >> >> >>>> Could not connect to
> >> >> >>>> https://hawkular-metrics:443/hawkular/metrics/status.
> >> >> >>>> Curl exit code: 6. Status Code 000
> >> >> >>>> 'https://hawkular-metrics:443/hawkular/metrics/status' is not
> >> >> >>>> accessible
> >> >> >>>> [HTTP status code: 000. Curl exit code 6]. Retrying.
> >> >> >>>> Could not connect to
> >> >> >>>> https://hawkular-metrics:443/hawkular/metrics/status.
> >> >> >>>> Curl exit code: 6. Status Code 000
> >> >> >>>>
> >> >> >>>> $ oc logs -f hawkular-cassandra-1-2pzd7
> >> >> >>>> INFO 23:00:24 Starting listening for CQL clients on
> >> >> >>>> hawkular-cassandra-1-2pzd7/10.1.6.2:9042...
> >> >> >>>> INFO 23:00:24 Binding thrift service to
> >> >> >>>> hawkular-cassandra-1-2pzd7/10.1.6.2:9160
> >> >> >>>> INFO 23:00:24 enabling encrypted thrift connections between
> >> >> >>>> client
> >> >> >>>> and
> >> >> >>>> server
> >> >> >>>> INFO 23:00:24 Listening for thrift clients...
> >> >> >>>> INFO 23:00:26 Created default superuser role 'cassandra'
> >> >> >>>>
> >> >> >>>> # oc get svc
> >> >> >>>> NAME CLUSTER-IP EXTERNAL-IP PORT(S)
> >> >> >>>> AGE
> >> >> >>>> hawkular-cassandra 172.30.2.13 <none>
> >> >> >>>> 9042/TCP,9160/TCP,7000/TCP,7001/TCP 25m
> >> >> >>>> hawkular-cassandra-nodes None <none>
> >> >> >>>> 9042/TCP,9160/TCP,7000/TCP,7001/TCP 25m
> >> >> >>>> hawkular-metrics 172.30.117.176 <none> 443/TCP
> >> >> >>>> 25m
> >> >> >>>> heapster 172.30.107.135 <none> 80/TCP
> >> >> >>>> 25m
> >> >> >>>>
> >> >> >>>> #curl -I 172.30.117.176:443//hawkular/metrics/status
> >> >> >>>>
> >> >> >>>> HTTP/1.1 504 Gateway Timeout
> >> >> >>>> Mime-Version: 1.0
> >> >> >>>> Date: Mon, 13 Jun 2016 23:25:47 GMT
> >> >> >>>> Content-Type: text/html
> >> >> >>>> Connection: keep-alive
> >> >> >>>> Proxy-Connection: keep-alive
> >> >> >>>> Content-Length: 1572
> >> >> >>>>
> >> >> >>>> --
> >> >> >>>> Srinivas Kotaru
> >> >> >>>>
> >> >> >>>> On 6/13/16, 2:33 PM, "Srinivas Naga Kotaru (skotaru)"
> >> >> >>>> <[email protected]>
> >> >> >>>> wrote:
> >> >> >>>>
> >> >> >>>> >Matt
> >> >> >>>> >
> >> >> >>>> >That is good catch. I ran without USE_PERSISTENT_STORAGE=false
> >> >> >>>> >and
> >> >> >>>> >working
> >> >> >>>> >
> >> >> >>>> >I adjusted PV to 5Gi and reran. Will update progress.
> >> >> >>>> >
> >> >> >>>> >Thanks you for your help so far.
> >> >> >>>> >
> >> >> >>>> >--
> >> >> >>>> >Srinivas Kotaru
> >> >> >>>> >
> >> >> >>>> >On 6/13/16, 2:27 PM, "Matt Wringe" <[email protected]> wrote:
> >> >> >>>> >
> >> >> >>>> >>
> >> >> >>>> >>
> >> >> >>>> >>----- Original Message -----
> >> >> >>>> >>> From: "Srinivas Naga Kotaru (skotaru)" <[email protected]>
> >> >> >>>> >>> To: "Matt Wringe" <[email protected]>
> >> >> >>>> >>> Cc: [email protected]
> >> >> >>>> >>> Sent: Monday, June 13, 2016 5:21:01 PM
> >> >> >>>> >>> Subject: Re: Metrics deployment
> >> >> >>>> >>>
> >> >> >>>> >>> Oh ok
> >> >> >>>> >>>
> >> >> >>>> >>> Am using PV for metrics
> >> >> >>>> >>>
> >> >> >>>> >>> description: "The persistent volume size for each of the
> >> >> >>>> >>> Cassandra
> >> >> >>>> >>> nodes"
> >> >> >>>> >>> name: CASSANDRA_PV_SIZE
> >> >> >>>> >>> value: "10Gi"
> >> >> >>>> >>>
> >> >> >>>> >>> oc get pv
> >> >> >>>> >>> NAME CAPACITY ACCESSMODES STATUS CLAIM
> >> >> >>>> >>> REASON
> >> >> >>>> >>> AGE
> >> >> >>>> >>> pv-1gb-001 1Gi RWO Available
> >> >> >>>> >>> 4d
> >> >> >>>> >>> pv-1gb-002 1Gi RWO Available
> >> >> >>>> >>> 4d
> >> >> >>>> >>> pv-1gb-003 1Gi RWO Available
> >> >> >>>> >>> 4d
> >> >> >>>> >>> pv-1gb-004 1Gi RWO Bound
> >> >> >>>> >>> thlatt/mongodb
> >> >> >>>> >>> 4d
> >> >> >>>> >>> pv-1gb-005 1Gi RWO Available
> >> >> >>>> >>> 4d
> >> >> >>>> >>> pv-2gb-0010 2Gi RWO Available
> >> >> >>>> >>> 4d
> >> >> >>>> >>> pv-2gb-006 2Gi RWO Available
> >> >> >>>> >>> 4d
> >> >> >>>> >>> pv-2gb-007 2Gi RWO Available
> >> >> >>>> >>> 4d
> >> >> >>>> >>> pv-2gb-008 2Gi RWO Available
> >> >> >>>> >>> 4d
> >> >> >>>> >>> pv-2gb-009 2Gi RWO Available
> >> >> >>>> >>> 4d
> >> >> >>>> >>> pv-5gb-0011 5Gi RWO Available
> >> >> >>>> >>> 4d
> >> >> >>>> >>> pv-5gb-0012 5Gi RWO Available
> >> >> >>>> >>> 4d
> >> >> >>>> >>> pv-5gb-0013 5Gi RWO Available
> >> >> >>>> >>> 4d
> >> >> >>>> >>> pv-5gb-0014 5Gi RWO Available
> >> >> >>>> >>> 4d
> >> >> >>>> >>> pv-5gb-0015 5Gi RWO Available
> >> >> >>>> >>> 4d
> >> >> >>>> >>>
> >> >> >>>> >>> am running with below command
> >> >> >>>> >>>
> >> >> >>>> >>> $ oc new-app -f metrics-deployer.yaml ( hardcoded HOSTNAME,
> >> >> >>>> >>> MASTER_API
> >> >> >>>> >>> and
> >> >> >>>> >>> PV info so not passing any parameters)
> >> >> >>>> >>>
> >> >> >>>> >>
> >> >> >>>> >>I would suspect that Cassandra is blocked because its waiting
> >> >> >>>> >>for
> >> >> >>>> >>10Gi
> >> >> >>>> >>PV
> >> >> >>>> >>to become available, and none of the PV listed above are big
> >> >> >>>> >>enough.
> >> >> >>>> >>
> >> >> >>>> >>>
> >> >> >>>> >>> --
> >> >> >>>> >>> Srinivas Kotaru
> >> >> >>>> >>>
> >> >> >>>> >>> On 6/13/16, 2:12 PM, "Matt Wringe" <[email protected]> wrote:
> >> >> >>>> >>>
> >> >> >>>> >>> >----- Original Message -----
> >> >> >>>> >>> >> From: "Srinivas Naga Kotaru (skotaru)" <[email protected]>
> >> >> >>>> >>> >> To: "Matt Wringe" <[email protected]>
> >> >> >>>> >>> >> Cc: [email protected]
> >> >> >>>> >>> >> Sent: Monday, June 13, 2016 4:55:55 PM
> >> >> >>>> >>> >> Subject: Re: Metrics deployment
> >> >> >>>> >>> >>
> >> >> >>>> >>> >> Matt
> >> >> >>>> >>> >>
> >> >> >>>> >>> >> Thanks for looking into. I rerun the setup, but had the
> >> >> >>>> >>> >> same
> >> >> >>>> >>> >> issue
> >> >> >>>> >>> >>
> >> >> >>>> >>> >> # oc get pods
> >> >> >>>> >>> >> NAME READY STATUS
> >> >> >>>> >>> >> RESTARTS
> >> >> >>>> >>> >> AGE
> >> >> >>>> >>> >> hawkular-cassandra-1-y2egy 0/1 ContainerCreating
> >> >> >>>> >>> >> 0
> >> >> >>>> >>> >> 5m
> >> >> >>>> >>> >> hawkular-metrics-4b16f 0/1 Running
> >> >> >>>> >>> >> 1
> >> >> >>>> >>> >> 4m
> >> >> >>>> >>> >> heapster-x2gj2 0/1 Running
> >> >> >>>> >>> >> 2
> >> >> >>>> >>> >> 4m
> >> >> >>>> >>> >> metrics-deployer-9v7vc 0/1 Completed
> >> >> >>>> >>> >> 0
> >> >> >>>> >>> >> 6m
> >> >> >>>> >>> >>
> >> >> >>>> >>> >> $ oc logs -f hawkular-cassandra-1-y2egy
> >> >> >>>> >>> >> Error from server: container "hawkular-cassandra-1" in pod
> >> >> >>>> >>> >> "hawkular-cassandra-1-y2egy" is waiting to start:
> >> >> >>>> >>> >> ContainerCreating
> >> >> >>>> >>> >
> >> >> >>>> >>> >Ok, so it looks like something is blocking the Cassandra pod
> >> >> >>>> >>> >from
> >> >> >>>> >>> >starting.
> >> >> >>>> >>> >
> >> >> >>>> >>> >If you are using persistent storage, Cassandra will not start
> >> >> >>>> >>> >until
> >> >> >>>> >>> >the
> >> >> >>>> >>> >PV
> >> >> >>>> >>> >is available. There may be some more information about
> >> >> >>>> >>> >Cassandra
> >> >> >>>> >>> >in
> >> >> >>>> >>> >the
> >> >> >>>> >>> >pod
> >> >> >>>> >>> >section of the console under events.
> >> >> >>>> >>> >
> >> >> >>>> >>> >What command did you use when deploying the deployer?
> >> >> >>>> >>> >
> >> >> >>>> >>> >>
> >> >> >>>> >>> >> $ oc logs -f hawkular-metrics-4b16f
> >> >> >>>> >>> >>
> >> >> >>>> >>> >> 16:54:25,703 DEBUG [org.jboss.as.config] (MSC service
> >> >> >>>> >>> >> thread
> >> >> >>>> >>> >> 1-4)
> >> >> >>>> >>> >> VM
> >> >> >>>> >>> >> Arguments: -Duser.home=/home/jboss -Duser.name=jboss
> >> >> >>>> >>> >> -D[Standalone]
> >> >> >>>> >>> >> -XX:+UseCompressedOops -verbose:gc
> >> >> >>>> >>> >> -Xloggc:/opt/eap/standalone/log/gc.log
> >> >> >>>> >>> >> -XX:+PrintGCDetails -XX:+PrintGCDateStamps
> >> >> >>>> >>> >> -XX:+UseGCLogFileRotation
> >> >> >>>> >>> >> -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=3M
> >> >> >>>> >>> >> -XX:-TraceClassUnloading
> >> >> >>>> >>> >> -Xms1303m -Xmx1303m -XX:MaxPermSize=256m
> >> >> >>>> >>> >> -Djava.net.preferIPv4Stack=true
> >> >> >>>> >>> >> -Djboss.modules.system.pkgs=org.jboss.logmanager
> >> >> >>>> >>> >> -Djava.awt.headless=true
> >> >> >>>> >>> >> -Djboss.modules.policy-permissions=true
> >> >> >>>> >>> >> -Xbootclasspath/p:/opt/eap/jboss-modules.jar:/opt/eap/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-1.5.4.Final-redhat-1.jar:/opt/eap/modules/system/layers/base/org/jboss/logmanager/ext/main/javax.json-1.0.4.jar:/opt/eap/modules/system/layers/base/org/jboss/logmanager/ext/main/jboss-logmanager-ext-1.0.0.Alpha2-redhat-1.jar
> >> >> >>>> >>> >> -Djava.util.logging.manager=org.jboss.logmanager.LogManager
> >> >> >>>> >>> >> -javaagent:/opt/eap/jolokia.jar=port=8778,protocol=https,caCert=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt,clientPrincipal=cn=system:master-proxy,useSslClientAuthentication=true,extraClientCheck=true,host=0.0.0.0,discoveryEnabled=false
> >> >> >>>> >>> >> -Djava.security.egd=file:/dev/./urandom
> >> >> >>>> >>> >> -Dorg.jboss.boot.log.file=/opt/eap/standalone/log/server.log
> >> >> >>>> >>> >> -Dlogging.configuration=file:/opt/eap/standalone/configuration/logging.properties
> >> >> >>>> >>> >> 16:54:27,079 INFO [org.xnio] (MSC service thread 1-3) XNIO
> >> >> >>>> >>> >> Version
> >> >> >>>> >>> >> 3.0.14.GA-redhat-1
> >> >> >>>> >>> >> 16:54:27,083 INFO [org.xnio.nio] (MSC service thread 1-3)
> >> >> >>>> >>> >> XNIO
> >> >> >>>> >>> >> NIO
> >> >> >>>> >>> >> Implementation Version 3.0.14.GA-redhat-1
> >> >> >>>> >>> >> 16:54:27,101 INFO [org.jboss.as.server] (Controller Boot
> >> >> >>>> >>> >> Thread)
> >> >> >>>> >>> >> JBAS015888:
> >> >> >>>> >>> >> Creating http management service using socket-binding
> >> >> >>>> >>> >> (management-http)
> >> >> >>>> >>> >> 16:54:27,104 INFO [org.jboss.remoting] (MSC service thread
> >> >> >>>> >>> >> 1-3)
> >> >> >>>> >>> >> JBoss
> >> >> >>>> >>> >> Remoting version 3.3.5.Final-redhat-1
> >> >> >>>> >>> >>
> >> >> >>>> >>> >> $ oc logs -f heapster-x2gj2
> >> >> >>>> >>> >> Endpoint Check in effect. Checking
> >> >> >>>> >>> >> https://hawkular-metrics:443/hawkular/metrics/status
> >> >> >>>> >>> >> Could not connect to
> >> >> >>>> >>> >> https://hawkular-metrics:443/hawkular/metrics/status.
> >> >> >>>> >>> >> Curl exit code: 6. Status Code 000
> >> >> >>>> >>> >> 'https://hawkular-metrics:443/hawkular/metrics/status' is
> >> >> >>>> >>> >> not
> >> >> >>>> >>> >> accessible
> >> >> >>>> >>> >> [HTTP status code: 000. Curl exit code 6]. Retrying.
> >> >> >>>> >>> >> Could not connect to
> >> >> >>>> >>> >> https://hawkular-metrics:443/hawkular/metrics/status.
> >> >> >>>> >>> >> Curl exit code: 6. Status Code 000
> >> >> >>>> >>> >> 'https://hawkular-metrics:443/hawkular/metrics/status' is
> >> >> >>>> >>> >> not
> >> >> >>>> >>> >> accessible
> >> >> >>>> >>> >> [HTTP status code: 000. Curl exit code 6]. Retrying.
> >> >> >>>> >>> >> Could not connect to
> >> >> >>>> >>> >> https://hawkular-metrics:443/hawkular/metrics/status.
> >> >> >>>> >>> >> Curl exit code: 6. Status Code 000
> >> >> >>>> >>> >>
> >> >> >>>> >>> >>
> >> >> >>>> >>> >> $ oc logs -f metrics-deployer-9v7vc
> >> >> >>>> >>> >>
> >> >> >>>> >>> >> ++ oc create -f -
> >> >> >>>> >>> >> serviceaccount "heapster" created
> >> >> >>>> >>> >> service "heapster" created
> >> >> >>>> >>> >> replicationcontroller "heapster" created
> >> >> >>>> >>> >> + echo 'Success!'
> >> >> >>>> >>> >> Success!
> >> >> >>>> >>> >>
> >> >> >>>> >>> >> --
> >> >> >>>> >>> >> Srinivas Kotaru
> >> >> >>>> >>> >>
> >> >> >>>> >>> >> On 6/13/16, 1:49 PM, "Matt Wringe" <[email protected]>
> >> >> >>>> >>> >> wrote:
> >> >> >>>> >>> >>
> >> >> >>>> >>> >> >
> >> >> >>>> >>> >> >
> >> >> >>>> >>> >> >----- Original Message -----
> >> >> >>>> >>> >> >> From: "Srinivas Naga Kotaru (skotaru)"
> >> >> >>>> >>> >> >> <[email protected]>
> >> >> >>>> >>> >> >> To: [email protected]
> >> >> >>>> >>> >> >> Sent: Monday, June 13, 2016 3:58:12 PM
> >> >> >>>> >>> >> >> Subject: Metrics deployment
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >> Hi
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >> Am trying to configure metrics in our newly installed
> >> >> >>>> >>> >> >> clusters. Am
> >> >> >>>> >>> >> >> seeing
> >> >> >>>> >>> >> >> below errors once metrics-deploy script was successful.
> >> >> >>>> >>> >> >> I
> >> >> >>>> >>> >> >> used
> >> >> >>>> >>> >> >> our
> >> >> >>>> >>> >> >> environment specific HAWKULAR_METRICS_HOSTNAME and
> >> >> >>>> >>> >> >> MASTER_URL
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >> # oc new-app -f metrics-deployer.yaml
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >> Note: customized, CASSANDARA PV, MASTER_URL, and
> >> >> >>>> >>> >> >> HAWKULAR_METRICS_HOSTNAME
> >> >> >>>> >>> >> >> (
> >> >> >>>> >>> >> >> hard coded as values)
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >> template "hawkular-heapster" created
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >> Deploying the Heapster component
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >> ++ echo 'Deploying the Heapster component'
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >> ++ '[' -n '' ']'
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >> ++ oc create -f -
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >> ++ oc process hawkular-heapster -v
> >> >> >>>> >>> >> >> IMAGE_PREFIX=registry.access.redhat.com/openshift3/,IMAGE_VERSION=latest,MASTER_URL=https://lae3-alln-int-idev01.cisco.com:443,NODE_ID=nodename
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >> serviceaccount "heapster" created
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >> service "heapster" created
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >> replicationcontroller "heapster" created
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >> + echo 'Success!'
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >> Success!
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >> # oc get pods
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >> NAME READY STATUS RESTARTS AGE
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >> hawkular-cassandra-1-9nzio 0/1 ContainerCreating 0 4m
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >> hawkular-metrics-hi7mb 0/1 Running 1 4m
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >> heapster-e8gbu 0/1 Running 2 4m
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >> metrics-deployer-64703 0/1 ContainerCreating 0 3s
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >> metrics-deployer-cd1nf 0/1 Completed 0 5m
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >
> >> >> >>>> >>> >> >It looks like none of your containers are fully up and
> >> >> >>>> >>> >> >running
> >> >> >>>> >>> >> >yet.
> >> >> >>>> >>> >> >
> >> >> >>>> >>> >> >Without Cassandra running, Hawkular Metrics will not run,
> >> >> >>>> >>> >> >and
> >> >> >>>> >>> >> >Heapster
> >> >> >>>> >>> >> >will
> >> >> >>>> >>> >> >wait until Hawkular Metrics is fully running.
> >> >> >>>> >>> >> >
> >> >> >>>> >>> >> >Do you see anything in the Cassandra logs? The first step
> >> >> >>>> >>> >> >will
> >> >> >>>> >>> >> >be to
> >> >> >>>> >>> >> >get
> >> >> >>>> >>> >> >Cassandra running properly.
> >> >> >>>> >>> >> >
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >> $ oc logs -f heapster-e8gbu
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >> Endpoint Check in effect. Checking
> >> >> >>>> >>> >> >> https://hawkular-metrics:443/hawkular/metrics/status
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >> Could not connect to
> >> >> >>>> >>> >> >> https://hawkular-metrics:443/hawkular/metrics/status.
> >> >> >>>> >>> >> >> Curl exit code: 6. Status Code 000
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >> 'https://hawkular-metrics:443/hawkular/metrics/status'
> >> >> >>>> >>> >> >> is
> >> >> >>>> >>> >> >> not
> >> >> >>>> >>> >> >> accessible
> >> >> >>>> >>> >> >> [HTTP status code: 000. Curl exit code 6]. Retrying.
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >> Could not connect to
> >> >> >>>> >>> >> >> https://hawkular-metrics:443/hawkular/metrics/status.
> >> >> >>>> >>> >> >> Curl exit code: 6. Status Code 000
> >> >> >>>> >>> >> >
> >> >> >>>> >>> >> >Heapster waits until Hawkular Metrics is started before
> >> >> >>>> >>> >> >trying
> >> >> >>>> >>> >> >to
> >> >> >>>> >>> >> >push
> >> >> >>>> >>> >> >metrics to it. The issue that you are seeing is because
> >> >> >>>> >>> >> >Heapster
> >> >> >>>> >>> >> >could
> >> >> >>>> >>> >> >not
> >> >> >>>> >>> >> >properly connect to Hawkular Metrics. Until the Hawkular
> >> >> >>>> >>> >> >Metrics
> >> >> >>>> >>> >> >service
> >> >> >>>> >>> >> >is
> >> >> >>>> >>> >> >fully up, Heapster will not be able to connect to it.
> >> >> >>>> >>> >> >
> >> >> >>>> >>> >> >
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >> What is the wrong? Why it checking just hawkular-metrics
> >> >> >>>> >>> >> >> rather
> >> >> >>>> >>> >> >> full
> >> >> >>>> >>> >> >> routing
> >> >> >>>> >>> >> >> URL which was provided as HAWKULAR_METRICS_HOSTNAME
> >> >> >>>> >>> >> >
> >> >> >>>> >>> >> >The Hawkular Metrics service has two hostnames: the
> >> >> >>>> >>> >> >internal
> >> >> >>>> >>> >> >hostname
> >> >> >>>> >>> >> >used
> >> >> >>>> >>> >> >by the internal components (eg 'hawkular-metrics') and the
> >> >> >>>> >>> >> >external
> >> >> >>>> >>> >> >hostname (eg what is configured via
> >> >> >>>> >>> >> >HAWKULAR_METRICS_HOSTNAME).
> >> >> >>>> >>> >> >The
> >> >> >>>> >>> >> >OpenShift dns server will resolve hostnames to the name of
> >> >> >>>> >>> >> >services,
> >> >> >>>> >>> >> >which
> >> >> >>>> >>> >> >is where the internal 'hawkular-metrics' comes from.
> >> >> >>>> >>> >> >
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >> --
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >> Srinivas Kotaru
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >> _______________________________________________
> >> >> >>>> >>> >> >> users mailing list
> >> >> >>>> >>> >> >> [email protected]
> >> >> >>>> >>> >> >> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >>
> >> >> >>>> >>> >>
> >> >> >>>> >>>
> >> >> >>>> >>>
> >> >> >>>> >
> >> >> >>>>
> >> >> >>>>
> >> >> >>
> >> >> >
> >> >>
> >> >>
> >>
> >>
>
>
_______________________________________________
users mailing list
[email protected]
http://lists.openshift.redhat.com/openshiftmm/listinfo/users
