Matt
Am rerunning the setup again. Can you give tips how to check if something is
blocking between nodes? I can see DNS is working as I pasted earlier with
examples. Do we need to open any firewall rules? Am assuming this is SDN and
default open with in the cluster. Let me know if I need to explicitely need to
open any ports between the nodes.
Am getting below output from browser if I hit FQDN (This was taken before I
deleted the stack)
{
"errorMsg": "Service unavailable while initializing."
}
--
Srinivas Kotaru
On 6/14/16, 11:27 AM, "Matt Wringe" <[email protected]> wrote:
>----- Original Message -----
>> From: "Srinivas Naga Kotaru (skotaru)" <[email protected]>
>> To: "Matt Wringe" <[email protected]>
>> Cc: [email protected]
>> Sent: Tuesday, June 14, 2016 2:09:49 PM
>> Subject: Re: Metrics deployment
>>
>> Not sure what do you mean by installation. Am just running oc new-app -f
>> metrics-deployer.yaml with default values except HOST_NAME and PV storage.
>
>I would suspect something wrong with your node or cluster installation. Or you
>have firewall rules blocking connections between your nodes so that pod cannot
>connect with each other or access the OpenShift DNS server.
>
>The lifecycle hooks exist to make sure that components only enter the ready
>state when they are fully started and ready.
>
>Can you check the Hawkular Metrics status page and see what that outputs? eg
>https://${HAWKULAR_METRICS_HOSTNAME}/hawkular/metrics/status
>
>>
>> I just deleted entire metrics setup and re running. But not sure this will
>> fix the issue.
>>
>> $ ./delete_metrics-infra.sh
>> replicationcontroller "hawkular-cassandra-1" deleted
>> replicationcontroller "hawkular-metrics" deleted
>> replicationcontroller "heapster" deleted
>> route "hawkular-metrics" deleted
>> service "hawkular-cassandra" deleted
>> service "hawkular-cassandra-nodes" deleted
>> service "hawkular-metrics" deleted
>> service "heapster" deleted
>> pod "heapster-lyf65" deleted
>> serviceaccount "cassandra" deleted
>> serviceaccount "hawkular" deleted
>> serviceaccount "heapster" deleted
>> template "hawkular-cassandra-node-emptydir" deleted
>> template "hawkular-cassandra-node-pv" deleted
>> template "hawkular-cassandra-services" deleted
>> template "hawkular-heapster" deleted
>> template "hawkular-metrics" deleted
>> template "hawkular-support" deleted
>> secret "hawkular-cassandra-certificate" deleted
>> secret "hawkular-cassandra-secrets" deleted
>> secret "hawkular-metrics-account" deleted
>> secret "hawkular-metrics-certificate" deleted
>> secret "hawkular-metrics-secrets" deleted
>> secret "heapster-secrets" deleted
>>
>> --
>> Srinivas Kotaru
>>
>> On 6/14/16, 10:53 AM, "Matt Wringe" <[email protected]> wrote:
>>
>> >----- Original Message -----
>> >> From: "Srinivas Naga Kotaru (skotaru)" <[email protected]>
>> >> To: "Matt Wringe" <[email protected]>
>> >> Cc: [email protected]
>> >> Sent: Tuesday, June 14, 2016 1:37:01 PM
>> >> Subject: Re: Metrics deployment
>> >>
>> >> I removed readiness probes from both hawkular-cassandra-1 &
>> >> hawkular-metrics
>> >> as both status shows probes failed.
>> >
>> >You should not have to remove the probes, this indicates that something is
>> >wrong with your installation.
>> >
>> >>
>> >> It looks good now. Both containers looks and running
>> >> (hawkular-cassandra-1-kr8ka , hawkular-metrics-vhe3u) however
>> >> heapster-7yl34
>> >> logs still shows Could not connect to
>> >> https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 6.
>> >> Status Code 000.
>> >>
>> >> Are we good or still had issues?
>> >>
>> >>
>> >> # oc get pods
>> >> NAME READY STATUS RESTARTS AGE
>> >> hawkular-cassandra-1-kr8ka 1/1 Running 0 6m
>> >> hawkular-metrics-vhe3u 1/1 Running 2 5m
>> >> heapster-7yl34 0/1 Running 2 5m
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> --
>> >> Srinivas Kotaru
>> >>
>> >> On 6/14/16, 10:07 AM, "Srinivas Naga Kotaru (skotaru)" <[email protected]>
>> >> wrote:
>> >>
>> >> >Matt
>> >> >
>> >> >Just want to share more info by running describe pod.
>> >> >
>> >> >It seems to be health probe failing. Do you think it is the issue?
>> >> >
>> >> >
>> >> >
>> >> ># oc describe pod hawkular-cassandra-1-it5uh
>> >> >Name: hawkular-cassandra-1-it5uh
>> >> >Namespace: openshift-infra
>> >> >Node: l3inpn-id2-003.cisco.com/173.36.96.16
>> >> >Start Time: Tue, 14 Jun 2016 16:36:21 +0000
>> >> >Labels:
>> >> >
>> >> > metrics-infra=hawkular-cassandra,name=hawkular-cassandra-1,type=hawkular-cassandra
>> >> >Status: Running
>> >> >IP: 10.1.9.2
>> >> >Controllers: ReplicationController/hawkular-cassandra-1
>> >> >Containers:
>> >> > hawkular-cassandra-1:
>> >> > Container ID:
>> >> >
>> >> > docker://17a9575eb655145859a9207f5c4bde7456f947e27188a056ff2bd08c4ce6ae5d
>> >> > Image:
>> >> >
>> >> > registry.access.redhat.com/openshift3/metrics-cassandra:latest
>> >> > Image ID:
>> >> >
>> >> > docker://ee2117c9848298ca5a0cbbce354fd4adff370435225324ab9d60cd9cd9a95c53
>> >> > Ports: 9042/TCP, 9160/TCP, 7000/TCP, 7001/TCP
>> >> > Command:
>> >> > /opt/apache-cassandra/bin/cassandra-docker.sh
>> >> > --cluster_name=hawkular-metrics
>> >> > --data_volume=/cassandra_data
>> >> > --internode_encryption=all
>> >> > --require_node_auth=true
>> >> > --enable_client_encryption=true
>> >> > --require_client_auth=true
>> >> > --keystore_file=/secret/cassandra.keystore
>> >> > --keystore_password_file=/secret/cassandra.keystore.password
>> >> > --truststore_file=/secret/cassandra.truststore
>> >> > --truststore_password_file=/secret/cassandra.truststore.password
>> >> > --cassandra_pem_file=/secret/cassandra.pem
>> >> > QoS Tier:
>> >> > cpu: BestEffort
>> >> > memory: BestEffort
>> >> > State: Running
>> >> > Started: Tue, 14 Jun 2016 16:37:01 +0000
>> >> > Ready: True
>> >> > Restart Count: 0
>> >> > Readiness: exec
>> >> > [/opt/apache-cassandra/bin/cassandra-docker-ready.sh]
>> >> > delay=0s timeout=1s period=10s #success=1 #failure=3
>> >> > Environment Variables:
>> >> > CASSANDRA_MASTER: true
>> >> > POD_NAMESPACE: openshift-infra (v1:metadata.namespace)
>> >> >Conditions:
>> >> > Type Status
>> >> > Ready True
>> >> >Volumes:
>> >> > cassandra-data:
>> >> > Type: PersistentVolumeClaim (a reference to a
>> >> > PersistentVolumeClaim
>> >> > in
>> >> > the same namespace)
>> >> > ClaimName: metrics-cassandra-1
>> >> > ReadOnly: false
>> >> > hawkular-cassandra-secrets:
>> >> > Type: Secret (a volume populated by a Secret)
>> >> > SecretName: hawkular-cassandra-secrets
>> >> > cassandra-token-4urfd:
>> >> > Type: Secret (a volume populated by a Secret)
>> >> > SecretName: cassandra-token-4urfd
>> >> >Events:
>> >> > FirstSeen LastSeen Count From
>> >> > SubobjectPath Type Reason
>> >> > Message
>> >> > --------- -------- ----- ----
>> >> > ------------- -------- ------
>> >> > -------
>> >> > 27m 27m 1 {default-scheduler }
>> >> > Normal
>> >> > Scheduled Successfully
>> >> > assigned hawkular-cassandra-1-it5uh to l3inpn-id2-003.cisco.com
>> >> > 27m 27m 1 {kubelet
>> >> > l3inpn-id2-003.cisco.com}
>> >> > spec.containers{hawkular-cassandra-1} Normal Pulling
>> >> > pulling image
>> >> > "registry.access.redhat.com/openshift3/metrics-cassandra:latest"
>> >> > 27m 27m 1 {kubelet
>> >> > l3inpn-id2-003.cisco.com}
>> >> > spec.containers{hawkular-cassandra-1} Normal Pulled
>> >> > Successfully
>> >> > pulled
>> >> > image "registry.access.redhat.com/openshift3/metrics-cassandra:latest"
>> >> > 27m 27m 1 {kubelet
>> >> > l3inpn-id2-003.cisco.com}
>> >> > spec.containers{hawkular-cassandra-1} Normal Created
>> >> > Created
>> >> > container
>> >> > with docker id 17a9575eb655
>> >> > 27m 27m 1 {kubelet
>> >> > l3inpn-id2-003.cisco.com}
>> >> > spec.containers{hawkular-cassandra-1} Normal Started
>> >> > Started
>> >> > container
>> >> > with docker id 17a9575eb655
>> >> > 27m 26m 3 {kubelet
>> >> > l3inpn-id2-003.cisco.com}
>> >> > spec.containers{hawkular-cassandra-1} Warning
>> >> > Unhealthy Readiness
>> >> > probe
>> >> > failed: cat: /etc/ld.so.conf.d/*.conf: No such file or directory
>> >> >nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException:
>> >> >'Connection refused'.
>> >> >Cassandra not in the up and normal state. Current state is
>> >> >/opt/apache-cassandra/bin/cassandra-docker-ready.sh: line 28: [: =: unary
>> >> >operator expected
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >--
>> >> >Srinivas Kotaru
>> >> >
>> >> >On 6/14/16, 10:00 AM, "Srinivas Naga Kotaru (skotaru)"
>> >> ><[email protected]>
>> >> >wrote:
>> >> >
>> >> >>Matt
>> >> >>
>> >> >>DNS service is working
>> >> >>
>> >> >>
>> >> >>~ dig +short @master-01 kubernetes.default.svc.cluster.local
>> >> >>172.30.0.1
>> >> >>~ dig +short @master-01 jenkins.alln-test.svc.cluster.local
>> >> >>172.30.85.148
>> >> >>~ dig +short @master-01 cakephp-example.alln-test.svc.cluster.local
>> >> >>172.30.31.6
>> >> >>
>> >> >>I captured hawkular-metrics, it shows the problem. It seems to be unable
>> >> >>to
>> >> >>connect Cassandra cluster
>> >> >>
>> >> >># oc exec hawkular-metrics-qldd8 cat /opt/eap/standalone/log/server.log
>> >> >>
>> >> >>12:40:58,848 INFO [org.jboss.as.clustering.infinispan] (ServerService
>> >> >>Thread Pool -- 31) JBAS010280: Activating Infinispan subsystem.
>> >> >>12:40:58,898 INFO [org.jboss.as.webservices] (ServerService Thread Pool
>> >> >>--
>> >> >>48) JBAS015537: Activating WebServices Extension
>> >> >>12:40:58,901 WARN [org.jboss.as.txn] (ServerService Thread Pool -- 46)
>> >> >>JBAS010153: Node identifier property is set to the default value. Please
>> >> >>make sure it is unique.
>> >> >>12:40:58,902 INFO [org.jboss.as.security] (ServerService Thread Pool --
>> >> >>44) JBAS013371: Activating Security Subsystem
>> >> >>12:40:58,912 INFO [org.jboss.as.naming] (ServerService Thread Pool --
>> >> >>39)
>> >> >>JBAS011800: Activating Naming Subsystem
>> >> >>12:40:58,928 INFO [org.jboss.as.jsf] (ServerService Thread Pool -- 37)
>> >> >>JBAS012615: Activated the following JSF Implementations: [main, 1.2]
>> >> >>12:40:58,939 INFO [org.jboss.as.connector.logging] (MSC service thread
>> >> >>1-2) JBAS010408: Starting JCA Subsystem (IronJacamar
>> >> >>1.0.32.Final-redhat-1)
>> >> >>12:40:58,981 INFO [org.jboss.as.security] (MSC service thread 1-2)
>> >> >>JBAS013370: Current PicketBox version=4.1.1.Final-redhat-1
>> >> >>12:40:58,961 INFO [org.jboss.as.connector.subsystems.datasources]
>> >> >>(ServerService Thread Pool -- 27) JBAS010403: Deploying JDBC-compliant
>> >> >>driver class org.h2.Driver (version 1.3)
>> >> >>12:40:59,039 INFO [org.jboss.as.naming] (MSC service thread 1-2)
>> >> >>JBAS011802: Starting Naming Service
>> >> >>12:40:59,052 INFO [org.jboss.as.mail.extension] (MSC service thread
>> >> >>1-4)
>> >> >>JBAS015400: Bound mail session [java:jboss/mail/Default]
>> >> >>12:40:59,614 INFO [org.apache.coyote.http11.Http11Protocol] (MSC
>> >> >>service
>> >> >>thread 1-1) JBWEB003001: Coyote HTTP/1.1 initializing on :
>> >> >>http-/0.0.0.0:8444
>> >> >>12:40:59,620 INFO [org.apache.coyote.http11.Http11Protocol] (MSC
>> >> >>service
>> >> >>thread 1-2) JBWEB003001: Coyote HTTP/1.1 initializing on :
>> >> >>http-/0.0.0.0:8080
>> >> >>12:40:59,621 INFO [org.apache.coyote.http11.Http11Protocol] (MSC
>> >> >>service
>> >> >>thread 1-1) JBWEB003000: Coyote HTTP/1.1 starting on: http-/0.0.0.0:8444
>> >> >>12:40:59,627 INFO [org.apache.coyote.http11.Http11Protocol] (MSC
>> >> >>service
>> >> >>thread 1-2) JBWEB003000: Coyote HTTP/1.1 starting on: http-/0.0.0.0:8080
>> >> >>12:40:59,654 INFO [org.jboss.as.connector.subsystems.datasources] (MSC
>> >> >>service thread 1-3) JBAS010400: Bound data source
>> >> >>[java:jboss/datasources/ExampleDS]
>> >> >>12:40:59,985 INFO [org.jboss.as.server.deployment] (MSC service thread
>> >> >>1-2) JBAS015876: Starting deployment of "activemq-rar.rar"
>> >> >>(runtime-name:
>> >> >>"activemq-rar.rar")
>> >> >>12:40:59,986 INFO [org.jboss.as.server.deployment] (MSC service thread
>> >> >>1-1) JBAS015876: Starting deployment of "hawkular-metrics-api-jaxrs.war"
>> >> >>(runtime-name: "hawkular-metrics-api-jaxrs.war")
>> >> >>12:40:59,987 INFO [org.jboss.as.server.deployment.scanner] (MSC service
>> >> >>thread 1-3) JBAS015012: Started FileSystemDeploymentService for
>> >> >>directory
>> >> >>/opt/eap/standalone/deployments
>> >> >>12:41:00,011 INFO [org.jboss.as.remoting] (MSC service thread 1-2)
>> >> >>JBAS017100: Listening on 127.0.0.1:9999
>> >> >>12:41:00,030 WARN [org.jboss.as.domain.http.api] (MSC service thread
>> >> >>1-3)
>> >> >>JBAS015102: Unable to load console module for slot eap, disabling
>> >> >>console
>> >> >>12:41:00,042 INFO [org.jboss.ws.common.management] (MSC service thread
>> >> >>1-4) JBWS022052: Starting JBoss Web Services - Stack CXF Server
>> >> >>4.3.5.Final-redhat-3
>> >> >>12:41:00,045 INFO [org.jboss.as.remoting] (MSC service thread 1-4)
>> >> >>JBAS017100: Listening on 0.0.0.0:4447
>> >> >>12:41:01,361 INFO [org.jboss.as.connector.deployers.RADeployer] (MSC
>> >> >>service thread 1-3) IJ020001: Required license terms for
>> >> >>file:/opt/eap/standalone/tmp/vfs/temp/temp912bb9d1372c33d3/activemq-rar.rar-a722089a00658182/contents/
>> >> >>12:41:01,784 WARN [org.jboss.as.dependency.private] (MSC service thread
>> >> >>1-1) JBAS015867: Deployment "deployment.hawkular-metrics-api-jaxrs.war"
>> >> >>is
>> >> >>using a private module ("org.codehaus.jackson.jackson-core-asl:main")
>> >> >>which may be changed or removed in future versions without notice.
>> >> >>12:41:01,785 WARN [org.jboss.as.dependency.private] (MSC service thread
>> >> >>1-1) JBAS015867: Deployment "deployment.hawkular-metrics-api-jaxrs.war"
>> >> >>is
>> >> >>using a private module ("org.codehaus.jackson.jackson-core-asl:main")
>> >> >>which may be changed or removed in future versions without notice.
>> >> >>12:41:01,785 WARN [org.jboss.as.dependency.private] (MSC service thread
>> >> >>1-1) JBAS015867: Deployment "deployment.hawkular-metrics-api-jaxrs.war"
>> >> >>is
>> >> >>using a private module ("org.codehaus.jackson.jackson-mapper-asl:main")
>> >> >>which may be changed or removed in future versions without notice.
>> >> >>12:41:01,785 WARN [org.jboss.as.dependency.private] (MSC service thread
>> >> >>1-1) JBAS015867: Deployment "deployment.hawkular-metrics-api-jaxrs.war"
>> >> >>is
>> >> >>using a private module ("org.codehaus.jackson.jackson-mapper-asl:main")
>> >> >>which may be changed or removed in future versions without notice.
>> >> >>12:41:01,843 INFO [org.jboss.weld.deployer] (MSC service thread 1-1)
>> >> >>JBAS016002: Processing weld deployment hawkular-metrics-api-jaxrs.war
>> >> >>12:41:01,941 INFO [org.jboss.weld.deployer] (MSC service thread 1-1)
>> >> >>JBAS016005: Starting Services for CDI deployment:
>> >> >>hawkular-metrics-api-jaxrs.war
>> >> >>12:41:01,986 INFO [org.jboss.weld.Version] (MSC service thread 1-1)
>> >> >>WELD-000900 1.1.31 (redhat)
>> >> >>12:41:02,013 INFO [org.jboss.weld.deployer] (MSC service thread 1-3)
>> >> >>JBAS016008: Starting weld service for deployment
>> >> >>hawkular-metrics-api-jaxrs.war
>> >> >>12:41:02,820 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>> >> >>metrics
>> >> >>service
>> >> >>12:41:02,835 INFO [org.jboss.web] (ServerService Thread Pool -- 51)
>> >> >>JBAS018210: Register web context: /hawkular/metrics
>> >> >>12:41:02,958 INFO [org.jboss.as.server] (ServerService Thread Pool --
>> >> >>28)
>> >> >>JBAS015859: Deployed "hawkular-metrics-api-jaxrs.war" (runtime-name :
>> >> >>"hawkular-metrics-api-jaxrs.war")
>> >> >>12:41:02,959 INFO [org.jboss.as.server] (ServerService Thread Pool --
>> >> >>28)
>> >> >>JBAS015859: Deployed "activemq-rar.rar" (runtime-name :
>> >> >>"activemq-rar.rar")
>> >> >>12:41:02,994 INFO [org.jboss.as] (Controller Boot Thread) JBAS015961:
>> >> >>Http
>> >> >>management interface listening on http://127.0.0.1:9990/management
>> >> >>12:41:02,994 INFO [org.jboss.as] (Controller Boot Thread) JBAS015951:
>> >> >>Admin console listening on http://127.0.0.1:9990
>> >> >>12:41:02,994 INFO [org.jboss.as] (Controller Boot Thread) JBAS015874:
>> >> >>JBoss EAP 6.4.4.GA (AS 7.5.4.Final-redhat-4) started in 6593ms - Started
>> >> >>267 of 303 services (62 services are lazy, passive or on-demand)
>> >> >>12:41:02,999 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
>> >> >>to
>> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra: unknown
>> >> >>error
>> >> >>12:41:02,999 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [1] Retrying
>> >> >>connecting to Cassandra cluster in [1]s...
>> >> >>12:41:04,000 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>> >> >>metrics
>> >> >>service
>> >> >>12:41:04,001 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
>> >> >>to
>> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>> >> >>12:41:04,001 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [2] Retrying
>> >> >>connecting to Cassandra cluster in [2]s...
>> >> >>12:41:06,002 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>> >> >>metrics
>> >> >>service
>> >> >>12:41:06,002 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
>> >> >>to
>> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>> >> >>12:41:06,003 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [3] Retrying
>> >> >>connecting to Cassandra cluster in [3]s...
>> >> >>12:41:08,203 INFO [org.jboss.resteasy.cdi.i18n] (http-/0.0.0.0:8080-2)
>> >> >>RESTEASY006050: Found BeanManager at java:comp/BeanManager
>> >> >>12:41:08,299 INFO [org.hibernate.validator.internal.util.Version]
>> >> >>(http-/0.0.0.0:8080-2) HV000001: Hibernate Validator
>> >> >>4.3.2.Final-redhat-2
>> >> >>12:41:08,418 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp]
>> >> >>(http-/0.0.0.0:8080-2) HAWKMETRICS200001: Hawkular Metrics starting
>> >> >>12:41:08,421 INFO [org.jboss.resteasy.resteasy_jaxrs.i18n]
>> >> >>(http-/0.0.0.0:8080-2) RESTEASY000320: Deploying
>> >> >>javax.ws.rs.core.Application: class
>> >> >>org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp$Proxy$_$$_WeldClientProxy
>> >> >>12:41:08,421 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp]
>> >> >>(http-/0.0.0.0:8080-2) HAWKMETRICS200001: Hawkular Metrics starting
>> >> >>12:41:08,422 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp]
>> >> >>(http-/0.0.0.0:8080-2) Virtual clock is disabled
>> >> >>12:41:08,422 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp]
>> >> >>(http-/0.0.0.0:8080-2) Virtual clock is disabled
>> >> >>12:41:09,003 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>> >> >>metrics
>> >> >>service
>> >> >>12:41:09,004 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
>> >> >>to
>> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>> >> >>12:41:09,004 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [4] Retrying
>> >> >>connecting to Cassandra cluster in [4]s...
>> >> >>12:41:13,004 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>> >> >>metrics
>> >> >>service
>> >> >>12:41:13,123 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
>> >> >>to
>> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra: unknown
>> >> >>error
>> >> >>12:41:13,124 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [5] Retrying
>> >> >>connecting to Cassandra cluster in [1]s...
>> >> >>12:41:14,124 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>> >> >>metrics
>> >> >>service
>> >> >>12:41:14,124 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
>> >> >>to
>> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>> >> >>12:41:14,125 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [6] Retrying
>> >> >>connecting to Cassandra cluster in [2]s...
>> >> >>12:41:16,125 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>> >> >>metrics
>> >> >>service
>> >> >>12:41:16,126 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
>> >> >>to
>> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>> >> >>12:41:16,126 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [7] Retrying
>> >> >>connecting to Cassandra cluster in [3]s...
>> >> >>12:41:19,126 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>> >> >>metrics
>> >> >>service
>> >> >>12:41:19,127 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
>> >> >>to
>> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>> >> >>12:41:19,127 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [8] Retrying
>> >> >>connecting to Cassandra cluster in [4]s...
>> >> >>12:41:23,128 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>> >> >>metrics
>> >> >>service
>> >> >>12:41:23,249 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
>> >> >>to
>> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra: unknown
>> >> >>error
>> >> >>12:41:23,249 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [9] Retrying
>> >> >>connecting to Cassandra cluster in [1]s...
>> >> >>12:41:24,250 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>> >> >>metrics
>> >> >>service
>> >> >>12:41:24,250 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
>> >> >>to
>> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>> >> >>12:41:24,250 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [10] Retrying
>> >> >>connecting to Cassandra cluster in [2]s...
>> >> >>12:41:26,251 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>> >> >>metrics
>> >> >>service
>> >> >>12:41:26,251 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
>> >> >>to
>> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>> >> >>12:41:26,251 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [11] Retrying
>> >> >>connecting to Cassandra cluster in [3]s...
>> >> >>12:41:29,252 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>> >> >>metrics
>> >> >>service
>> >> >>12:41:29,252 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
>> >> >>to
>> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>> >> >>12:41:29,253 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [12] Retrying
>> >> >>connecting to Cassandra cluster in [4]s...
>> >> >>12:41:33,253 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>> >> >>metrics
>> >> >>service
>> >> >>12:41:33,376 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
>> >> >>to
>> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra: unknown
>> >> >>error
>> >> >>12:41:33,377 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [13] Retrying
>> >> >>connecting to Cassandra cluster in [1]s...
>> >> >>12:41:34,377 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>> >> >>metrics
>> >> >>service
>> >> >>12:41:34,378 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
>> >> >>to
>> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>> >> >>12:41:34,378 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [14] Retrying
>> >> >>connecting to Cassandra cluster in [2]s...
>> >> >>12:41:36,378 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>> >> >>metrics
>> >> >>service
>> >> >>12:41:36,379 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
>> >> >>to
>> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>> >> >>12:41:36,379 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [15] Retrying
>> >> >>connecting to Cassandra cluster in [3]s...
>> >> >>12:41:39,379 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>> >> >>metrics
>> >> >>service
>> >> >>12:41:39,380 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
>> >> >>to
>> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>> >> >>12:41:39,380 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [16] Retrying
>> >> >>connecting to Cassandra cluster in [4]s...
>> >> >>12:41:43,380 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>> >> >>metrics
>> >> >>service
>> >> >>12:41:43,503 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
>> >> >>to
>> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra: unknown
>> >> >>error
>> >> >>12:41:43,504 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [17] Retrying
>> >> >>connecting to Cassandra cluster in [1]s...
>> >> >>12:41:44,504 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>> >> >>metrics
>> >> >>service
>> >> >>12:41:44,505 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
>> >> >>to
>> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>> >> >>12:41:44,505 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [18] Retrying
>> >> >>connecting to Cassandra cluster in [2]s...
>> >> >>
>> >> >>--
>> >> >>Srinivas Kotaru
>> >> >>
>> >> >>On 6/14/16, 6:06 AM, "Matt Wringe" <[email protected]> wrote:
>> >> >>
>> >> >>>----- Original Message -----
>> >> >>>> From: "Srinivas Naga Kotaru (skotaru)" <[email protected]>
>> >> >>>> To: "Matt Wringe" <[email protected]>
>> >> >>>> Cc: [email protected]
>> >> >>>> Sent: Monday, June 13, 2016 7:26:06 PM
>> >> >>>> Subject: Re: Metrics deployment
>> >> >>>>
>> >> >>>> Matt
>> >> >>>>
>> >> >>>> PV issue resolved. Was able to to see PV successfully bounded and
>> >> >>>> Casandra
>> >> >>>> container has been running. However, it seems puzzle not fully yet
>> >> >>>> solved.
>> >> >>>
>> >> >>>Are you sure the OpenShift DNS server is running?
>> >> >>>
>> >> >>>If you are running OSE 3.1, can you please follow this
>> >> >>>https://access.redhat.com/solutions/2329131 and see if you are now
>> >> >>>seeing
>> >> >>>errors in the Hawkular Metrics logs (essentially just run `oc exec
>> >> >>>hawkular-metrics-xxxxx cat /opt/eap/standalone/log/server.log`)
>> >> >>>
>> >> >>>>
>> >> >>>> I could see other container(heapster) not coming up, and seeing below
>> >> >>>> errors
>> >> >>>>
>> >> >>>> [skotaru@l3imas-id2-01 metrics]$ oc logs -f heapster-fnkdc
>> >> >>>> Endpoint Check in effect. Checking
>> >> >>>> https://hawkular-metrics:443/hawkular/metrics/status
>> >> >>>> Could not connect to
>> >> >>>> https://hawkular-metrics:443/hawkular/metrics/status.
>> >> >>>> Curl exit code: 6. Status Code 000
>> >> >>>> 'https://hawkular-metrics:443/hawkular/metrics/status' is not
>> >> >>>> accessible
>> >> >>>> [HTTP status code: 000. Curl exit code 6]. Retrying.
>> >> >>>> Could not connect to
>> >> >>>> https://hawkular-metrics:443/hawkular/metrics/status.
>> >> >>>> Curl exit code: 6. Status Code 000
>> >> >>>> 'https://hawkular-metrics:443/hawkular/metrics/status' is not
>> >> >>>> accessible
>> >> >>>> [HTTP status code: 000. Curl exit code 6]. Retrying.
>> >> >>>>
>> >> >>>>
>> >> >>>> # oc get pv
>> >> >>>> pv-5gb-0011 5Gi RWO Bound
>> >> >>>> openshift-infra/metrics-cassandra-1 22m
>> >> >>>>
>> >> >>>>
>> >> >>>> $ oc get pods
>> >> >>>> NAME READY STATUS RESTARTS AGE
>> >> >>>> hawkular-cassandra-1-2pzd7 1/1 Running 0 20m
>> >> >>>> hawkular-metrics-mf5qf 0/1 Running 7 20m
>> >> >>>> heapster-fnkdc 0/1 Error 6 20m
>> >> >>>> metrics-deployer-cvep0 0/1 Completed 0 21m
>> >> >>>>
>> >> >>>> # oc logs -f hawkular-metrics-mf5qf
>> >> >>>>
>> >> >>>> 19:20:00,819 INFO [org.xnio] (MSC service thread 1-2) XNIO Version
>> >> >>>> 3.0.14.GA-redhat-1
>> >> >>>> 19:20:00,831 INFO [org.jboss.as.server] (Controller Boot Thread)
>> >> >>>> JBAS015888:
>> >> >>>> Creating http management service using socket-binding
>> >> >>>> (management-http)
>> >> >>>> 19:20:00,834 INFO [org.xnio.nio] (MSC service thread 1-2) XNIO NIO
>> >> >>>> Implementation Version 3.0.14.GA-redhat-1
>> >> >>>> 19:20:00,844 INFO [org.jboss.remoting] (MSC service thread 1-2)
>> >> >>>> JBoss
>> >> >>>> Remoting version 3.3.5.Final-redhat-1
>> >> >>>>
>> >> >>>> $ oc logs -f heapster-fnkdc
>> >> >>>> Endpoint Check in effect. Checking
>> >> >>>> https://hawkular-metrics:443/hawkular/metrics/status
>> >> >>>> Could not connect to
>> >> >>>> https://hawkular-metrics:443/hawkular/metrics/status.
>> >> >>>> Curl exit code: 6. Status Code 000
>> >> >>>> 'https://hawkular-metrics:443/hawkular/metrics/status' is not
>> >> >>>> accessible
>> >> >>>> [HTTP status code: 000. Curl exit code 6]. Retrying.
>> >> >>>> Could not connect to
>> >> >>>> https://hawkular-metrics:443/hawkular/metrics/status.
>> >> >>>> Curl exit code: 6. Status Code 000
>> >> >>>> 'https://hawkular-metrics:443/hawkular/metrics/status' is not
>> >> >>>> accessible
>> >> >>>> [HTTP status code: 000. Curl exit code 6]. Retrying.
>> >> >>>> Could not connect to
>> >> >>>> https://hawkular-metrics:443/hawkular/metrics/status.
>> >> >>>> Curl exit code: 6. Status Code 000
>> >> >>>>
>> >> >>>> $ oc logs -f hawkular-cassandra-1-2pzd7
>> >> >>>> INFO 23:00:24 Starting listening for CQL clients on
>> >> >>>> hawkular-cassandra-1-2pzd7/10.1.6.2:9042...
>> >> >>>> INFO 23:00:24 Binding thrift service to
>> >> >>>> hawkular-cassandra-1-2pzd7/10.1.6.2:9160
>> >> >>>> INFO 23:00:24 enabling encrypted thrift connections between client
>> >> >>>> and
>> >> >>>> server
>> >> >>>> INFO 23:00:24 Listening for thrift clients...
>> >> >>>> INFO 23:00:26 Created default superuser role 'cassandra'
>> >> >>>>
>> >> >>>> # oc get svc
>> >> >>>> NAME CLUSTER-IP EXTERNAL-IP PORT(S)
>> >> >>>> AGE
>> >> >>>> hawkular-cassandra 172.30.2.13 <none>
>> >> >>>> 9042/TCP,9160/TCP,7000/TCP,7001/TCP 25m
>> >> >>>> hawkular-cassandra-nodes None <none>
>> >> >>>> 9042/TCP,9160/TCP,7000/TCP,7001/TCP 25m
>> >> >>>> hawkular-metrics 172.30.117.176 <none> 443/TCP
>> >> >>>> 25m
>> >> >>>> heapster 172.30.107.135 <none> 80/TCP
>> >> >>>> 25m
>> >> >>>>
>> >> >>>> #curl -I 172.30.117.176:443//hawkular/metrics/status
>> >> >>>>
>> >> >>>> HTTP/1.1 504 Gateway Timeout
>> >> >>>> Mime-Version: 1.0
>> >> >>>> Date: Mon, 13 Jun 2016 23:25:47 GMT
>> >> >>>> Content-Type: text/html
>> >> >>>> Connection: keep-alive
>> >> >>>> Proxy-Connection: keep-alive
>> >> >>>> Content-Length: 1572
>> >> >>>>
>> >> >>>> --
>> >> >>>> Srinivas Kotaru
>> >> >>>>
>> >> >>>> On 6/13/16, 2:33 PM, "Srinivas Naga Kotaru (skotaru)"
>> >> >>>> <[email protected]>
>> >> >>>> wrote:
>> >> >>>>
>> >> >>>> >Matt
>> >> >>>> >
>> >> >>>> >That is good catch. I ran without USE_PERSISTENT_STORAGE=false and
>> >> >>>> >working
>> >> >>>> >
>> >> >>>> >I adjusted PV to 5Gi and reran. Will update progress.
>> >> >>>> >
>> >> >>>> >Thanks you for your help so far.
>> >> >>>> >
>> >> >>>> >--
>> >> >>>> >Srinivas Kotaru
>> >> >>>> >
>> >> >>>> >On 6/13/16, 2:27 PM, "Matt Wringe" <[email protected]> wrote:
>> >> >>>> >
>> >> >>>> >>
>> >> >>>> >>
>> >> >>>> >>----- Original Message -----
>> >> >>>> >>> From: "Srinivas Naga Kotaru (skotaru)" <[email protected]>
>> >> >>>> >>> To: "Matt Wringe" <[email protected]>
>> >> >>>> >>> Cc: [email protected]
>> >> >>>> >>> Sent: Monday, June 13, 2016 5:21:01 PM
>> >> >>>> >>> Subject: Re: Metrics deployment
>> >> >>>> >>>
>> >> >>>> >>> Oh ok
>> >> >>>> >>>
>> >> >>>> >>> Am using PV for metrics
>> >> >>>> >>>
>> >> >>>> >>> description: "The persistent volume size for each of the
>> >> >>>> >>> Cassandra
>> >> >>>> >>> nodes"
>> >> >>>> >>> name: CASSANDRA_PV_SIZE
>> >> >>>> >>> value: "10Gi"
>> >> >>>> >>>
>> >> >>>> >>> oc get pv
>> >> >>>> >>> NAME CAPACITY ACCESSMODES STATUS CLAIM
>> >> >>>> >>> REASON
>> >> >>>> >>> AGE
>> >> >>>> >>> pv-1gb-001 1Gi RWO Available
>> >> >>>> >>> 4d
>> >> >>>> >>> pv-1gb-002 1Gi RWO Available
>> >> >>>> >>> 4d
>> >> >>>> >>> pv-1gb-003 1Gi RWO Available
>> >> >>>> >>> 4d
>> >> >>>> >>> pv-1gb-004 1Gi RWO Bound thlatt/mongodb
>> >> >>>> >>> 4d
>> >> >>>> >>> pv-1gb-005 1Gi RWO Available
>> >> >>>> >>> 4d
>> >> >>>> >>> pv-2gb-0010 2Gi RWO Available
>> >> >>>> >>> 4d
>> >> >>>> >>> pv-2gb-006 2Gi RWO Available
>> >> >>>> >>> 4d
>> >> >>>> >>> pv-2gb-007 2Gi RWO Available
>> >> >>>> >>> 4d
>> >> >>>> >>> pv-2gb-008 2Gi RWO Available
>> >> >>>> >>> 4d
>> >> >>>> >>> pv-2gb-009 2Gi RWO Available
>> >> >>>> >>> 4d
>> >> >>>> >>> pv-5gb-0011 5Gi RWO Available
>> >> >>>> >>> 4d
>> >> >>>> >>> pv-5gb-0012 5Gi RWO Available
>> >> >>>> >>> 4d
>> >> >>>> >>> pv-5gb-0013 5Gi RWO Available
>> >> >>>> >>> 4d
>> >> >>>> >>> pv-5gb-0014 5Gi RWO Available
>> >> >>>> >>> 4d
>> >> >>>> >>> pv-5gb-0015 5Gi RWO Available
>> >> >>>> >>> 4d
>> >> >>>> >>>
>> >> >>>> >>> am running with below command
>> >> >>>> >>>
>> >> >>>> >>> $ oc new-app -f metrics-deployer.yaml ( hardcoded HOSTNAME,
>> >> >>>> >>> MASTER_API
>> >> >>>> >>> and
>> >> >>>> >>> PV info so not passing any parameters)
>> >> >>>> >>>
>> >> >>>> >>
>> >> >>>> >>I would suspect that Cassandra is blocked because its waiting for
>> >> >>>> >>10Gi
>> >> >>>> >>PV
>> >> >>>> >>to become available, and none of the PV listed above are big
>> >> >>>> >>enough.
>> >> >>>> >>
>> >> >>>> >>>
>> >> >>>> >>> --
>> >> >>>> >>> Srinivas Kotaru
>> >> >>>> >>>
>> >> >>>> >>> On 6/13/16, 2:12 PM, "Matt Wringe" <[email protected]> wrote:
>> >> >>>> >>>
>> >> >>>> >>> >----- Original Message -----
>> >> >>>> >>> >> From: "Srinivas Naga Kotaru (skotaru)" <[email protected]>
>> >> >>>> >>> >> To: "Matt Wringe" <[email protected]>
>> >> >>>> >>> >> Cc: [email protected]
>> >> >>>> >>> >> Sent: Monday, June 13, 2016 4:55:55 PM
>> >> >>>> >>> >> Subject: Re: Metrics deployment
>> >> >>>> >>> >>
>> >> >>>> >>> >> Matt
>> >> >>>> >>> >>
>> >> >>>> >>> >> Thanks for looking into. I rerun the setup, but had the same
>> >> >>>> >>> >> issue
>> >> >>>> >>> >>
>> >> >>>> >>> >> # oc get pods
>> >> >>>> >>> >> NAME READY STATUS
>> >> >>>> >>> >> RESTARTS
>> >> >>>> >>> >> AGE
>> >> >>>> >>> >> hawkular-cassandra-1-y2egy 0/1 ContainerCreating 0
>> >> >>>> >>> >> 5m
>> >> >>>> >>> >> hawkular-metrics-4b16f 0/1 Running 1
>> >> >>>> >>> >> 4m
>> >> >>>> >>> >> heapster-x2gj2 0/1 Running 2
>> >> >>>> >>> >> 4m
>> >> >>>> >>> >> metrics-deployer-9v7vc 0/1 Completed 0
>> >> >>>> >>> >> 6m
>> >> >>>> >>> >>
>> >> >>>> >>> >> $ oc logs -f hawkular-cassandra-1-y2egy
>> >> >>>> >>> >> Error from server: container "hawkular-cassandra-1" in pod
>> >> >>>> >>> >> "hawkular-cassandra-1-y2egy" is waiting to start:
>> >> >>>> >>> >> ContainerCreating
>> >> >>>> >>> >
>> >> >>>> >>> >Ok, so it looks like something is blocking the Cassandra pod
>> >> >>>> >>> >from
>> >> >>>> >>> >starting.
>> >> >>>> >>> >
>> >> >>>> >>> >If you are using persistent storage, Cassandra will not start
>> >> >>>> >>> >until
>> >> >>>> >>> >the
>> >> >>>> >>> >PV
>> >> >>>> >>> >is available. There may be some more information about Cassandra
>> >> >>>> >>> >in
>> >> >>>> >>> >the
>> >> >>>> >>> >pod
>> >> >>>> >>> >section of the console under events.
>> >> >>>> >>> >
>> >> >>>> >>> >What command did you use when deploying the deployer?
>> >> >>>> >>> >
>> >> >>>> >>> >>
>> >> >>>> >>> >> $ oc logs -f hawkular-metrics-4b16f
>> >> >>>> >>> >>
>> >> >>>> >>> >> 16:54:25,703 DEBUG [org.jboss.as.config] (MSC service thread
>> >> >>>> >>> >> 1-4)
>> >> >>>> >>> >> VM
>> >> >>>> >>> >> Arguments: -Duser.home=/home/jboss -Duser.name=jboss
>> >> >>>> >>> >> -D[Standalone]
>> >> >>>> >>> >> -XX:+UseCompressedOops -verbose:gc
>> >> >>>> >>> >> -Xloggc:/opt/eap/standalone/log/gc.log
>> >> >>>> >>> >> -XX:+PrintGCDetails -XX:+PrintGCDateStamps
>> >> >>>> >>> >> -XX:+UseGCLogFileRotation
>> >> >>>> >>> >> -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=3M
>> >> >>>> >>> >> -XX:-TraceClassUnloading
>> >> >>>> >>> >> -Xms1303m -Xmx1303m -XX:MaxPermSize=256m
>> >> >>>> >>> >> -Djava.net.preferIPv4Stack=true
>> >> >>>> >>> >> -Djboss.modules.system.pkgs=org.jboss.logmanager
>> >> >>>> >>> >> -Djava.awt.headless=true
>> >> >>>> >>> >> -Djboss.modules.policy-permissions=true
>> >> >>>> >>> >> -Xbootclasspath/p:/opt/eap/jboss-modules.jar:/opt/eap/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-1.5.4.Final-redhat-1.jar:/opt/eap/modules/system/layers/base/org/jboss/logmanager/ext/main/javax.json-1.0.4.jar:/opt/eap/modules/system/layers/base/org/jboss/logmanager/ext/main/jboss-logmanager-ext-1.0.0.Alpha2-redhat-1.jar
>> >> >>>> >>> >> -Djava.util.logging.manager=org.jboss.logmanager.LogManager
>> >> >>>> >>> >> -javaagent:/opt/eap/jolokia.jar=port=8778,protocol=https,caCert=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt,clientPrincipal=cn=system:master-proxy,useSslClientAuthentication=true,extraClientCheck=true,host=0.0.0.0,discoveryEnabled=false
>> >> >>>> >>> >> -Djava.security.egd=file:/dev/./urandom
>> >> >>>> >>> >> -Dorg.jboss.boot.log.file=/opt/eap/standalone/log/server.log
>> >> >>>> >>> >> -Dlogging.configuration=file:/opt/eap/standalone/configuration/logging.properties
>> >> >>>> >>> >> 16:54:27,079 INFO [org.xnio] (MSC service thread 1-3) XNIO
>> >> >>>> >>> >> Version
>> >> >>>> >>> >> 3.0.14.GA-redhat-1
>> >> >>>> >>> >> 16:54:27,083 INFO [org.xnio.nio] (MSC service thread 1-3)
>> >> >>>> >>> >> XNIO
>> >> >>>> >>> >> NIO
>> >> >>>> >>> >> Implementation Version 3.0.14.GA-redhat-1
>> >> >>>> >>> >> 16:54:27,101 INFO [org.jboss.as.server] (Controller Boot
>> >> >>>> >>> >> Thread)
>> >> >>>> >>> >> JBAS015888:
>> >> >>>> >>> >> Creating http management service using socket-binding
>> >> >>>> >>> >> (management-http)
>> >> >>>> >>> >> 16:54:27,104 INFO [org.jboss.remoting] (MSC service thread
>> >> >>>> >>> >> 1-3)
>> >> >>>> >>> >> JBoss
>> >> >>>> >>> >> Remoting version 3.3.5.Final-redhat-1
>> >> >>>> >>> >>
>> >> >>>> >>> >> $ oc logs -f heapster-x2gj2
>> >> >>>> >>> >> Endpoint Check in effect. Checking
>> >> >>>> >>> >> https://hawkular-metrics:443/hawkular/metrics/status
>> >> >>>> >>> >> Could not connect to
>> >> >>>> >>> >> https://hawkular-metrics:443/hawkular/metrics/status.
>> >> >>>> >>> >> Curl exit code: 6. Status Code 000
>> >> >>>> >>> >> 'https://hawkular-metrics:443/hawkular/metrics/status' is not
>> >> >>>> >>> >> accessible
>> >> >>>> >>> >> [HTTP status code: 000. Curl exit code 6]. Retrying.
>> >> >>>> >>> >> Could not connect to
>> >> >>>> >>> >> https://hawkular-metrics:443/hawkular/metrics/status.
>> >> >>>> >>> >> Curl exit code: 6. Status Code 000
>> >> >>>> >>> >> 'https://hawkular-metrics:443/hawkular/metrics/status' is not
>> >> >>>> >>> >> accessible
>> >> >>>> >>> >> [HTTP status code: 000. Curl exit code 6]. Retrying.
>> >> >>>> >>> >> Could not connect to
>> >> >>>> >>> >> https://hawkular-metrics:443/hawkular/metrics/status.
>> >> >>>> >>> >> Curl exit code: 6. Status Code 000
>> >> >>>> >>> >>
>> >> >>>> >>> >>
>> >> >>>> >>> >> $ oc logs -f metrics-deployer-9v7vc
>> >> >>>> >>> >>
>> >> >>>> >>> >> ++ oc create -f -
>> >> >>>> >>> >> serviceaccount "heapster" created
>> >> >>>> >>> >> service "heapster" created
>> >> >>>> >>> >> replicationcontroller "heapster" created
>> >> >>>> >>> >> + echo 'Success!'
>> >> >>>> >>> >> Success!
>> >> >>>> >>> >>
>> >> >>>> >>> >> --
>> >> >>>> >>> >> Srinivas Kotaru
>> >> >>>> >>> >>
>> >> >>>> >>> >> On 6/13/16, 1:49 PM, "Matt Wringe" <[email protected]> wrote:
>> >> >>>> >>> >>
>> >> >>>> >>> >> >
>> >> >>>> >>> >> >
>> >> >>>> >>> >> >----- Original Message -----
>> >> >>>> >>> >> >> From: "Srinivas Naga Kotaru (skotaru)" <[email protected]>
>> >> >>>> >>> >> >> To: [email protected]
>> >> >>>> >>> >> >> Sent: Monday, June 13, 2016 3:58:12 PM
>> >> >>>> >>> >> >> Subject: Metrics deployment
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> Hi
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> Am trying to configure metrics in our newly installed
>> >> >>>> >>> >> >> clusters. Am
>> >> >>>> >>> >> >> seeing
>> >> >>>> >>> >> >> below errors once metrics-deploy script was successful. I
>> >> >>>> >>> >> >> used
>> >> >>>> >>> >> >> our
>> >> >>>> >>> >> >> environment specific HAWKULAR_METRICS_HOSTNAME and
>> >> >>>> >>> >> >> MASTER_URL
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> # oc new-app -f metrics-deployer.yaml
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> Note: customized, CASSANDARA PV, MASTER_URL, and
>> >> >>>> >>> >> >> HAWKULAR_METRICS_HOSTNAME
>> >> >>>> >>> >> >> (
>> >> >>>> >>> >> >> hard coded as values)
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> template "hawkular-heapster" created
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> Deploying the Heapster component
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> ++ echo 'Deploying the Heapster component'
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> ++ '[' -n '' ']'
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> ++ oc create -f -
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> ++ oc process hawkular-heapster -v
>> >> >>>> >>> >> >> IMAGE_PREFIX=registry.access.redhat.com/openshift3/,IMAGE_VERSION=latest,MASTER_URL=https://lae3-alln-int-idev01.cisco.com:443,NODE_ID=nodename
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> serviceaccount "heapster" created
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> service "heapster" created
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> replicationcontroller "heapster" created
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> + echo 'Success!'
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> Success!
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> # oc get pods
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> NAME READY STATUS RESTARTS AGE
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> hawkular-cassandra-1-9nzio 0/1 ContainerCreating 0 4m
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> hawkular-metrics-hi7mb 0/1 Running 1 4m
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> heapster-e8gbu 0/1 Running 2 4m
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> metrics-deployer-64703 0/1 ContainerCreating 0 3s
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> metrics-deployer-cd1nf 0/1 Completed 0 5m
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >
>> >> >>>> >>> >> >It looks like none of your containers are fully up and
>> >> >>>> >>> >> >running
>> >> >>>> >>> >> >yet.
>> >> >>>> >>> >> >
>> >> >>>> >>> >> >Without Cassandra running, Hawkular Metrics will not run, and
>> >> >>>> >>> >> >Heapster
>> >> >>>> >>> >> >will
>> >> >>>> >>> >> >wait until Hawkular Metrics is fully running.
>> >> >>>> >>> >> >
>> >> >>>> >>> >> >Do you see anything in the Cassandra logs? The first step
>> >> >>>> >>> >> >will
>> >> >>>> >>> >> >be to
>> >> >>>> >>> >> >get
>> >> >>>> >>> >> >Cassandra running properly.
>> >> >>>> >>> >> >
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> $ oc logs -f heapster-e8gbu
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> Endpoint Check in effect. Checking
>> >> >>>> >>> >> >> https://hawkular-metrics:443/hawkular/metrics/status
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> Could not connect to
>> >> >>>> >>> >> >> https://hawkular-metrics:443/hawkular/metrics/status.
>> >> >>>> >>> >> >> Curl exit code: 6. Status Code 000
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> 'https://hawkular-metrics:443/hawkular/metrics/status' is
>> >> >>>> >>> >> >> not
>> >> >>>> >>> >> >> accessible
>> >> >>>> >>> >> >> [HTTP status code: 000. Curl exit code 6]. Retrying.
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> Could not connect to
>> >> >>>> >>> >> >> https://hawkular-metrics:443/hawkular/metrics/status.
>> >> >>>> >>> >> >> Curl exit code: 6. Status Code 000
>> >> >>>> >>> >> >
>> >> >>>> >>> >> >Heapster waits until Hawkular Metrics is started before
>> >> >>>> >>> >> >trying
>> >> >>>> >>> >> >to
>> >> >>>> >>> >> >push
>> >> >>>> >>> >> >metrics to it. The issue that you are seeing is because
>> >> >>>> >>> >> >Heapster
>> >> >>>> >>> >> >could
>> >> >>>> >>> >> >not
>> >> >>>> >>> >> >properly connect to Hawkular Metrics. Until the Hawkular
>> >> >>>> >>> >> >Metrics
>> >> >>>> >>> >> >service
>> >> >>>> >>> >> >is
>> >> >>>> >>> >> >fully up, Heapster will not be able to connect to it.
>> >> >>>> >>> >> >
>> >> >>>> >>> >> >
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> What is the wrong? Why it checking just hawkular-metrics
>> >> >>>> >>> >> >> rather
>> >> >>>> >>> >> >> full
>> >> >>>> >>> >> >> routing
>> >> >>>> >>> >> >> URL which was provided as HAWKULAR_METRICS_HOSTNAME
>> >> >>>> >>> >> >
>> >> >>>> >>> >> >The Hawkular Metrics service has two hostnames: the internal
>> >> >>>> >>> >> >hostname
>> >> >>>> >>> >> >used
>> >> >>>> >>> >> >by the internal components (eg 'hawkular-metrics') and the
>> >> >>>> >>> >> >external
>> >> >>>> >>> >> >hostname (eg what is configured via
>> >> >>>> >>> >> >HAWKULAR_METRICS_HOSTNAME).
>> >> >>>> >>> >> >The
>> >> >>>> >>> >> >OpenShift dns server will resolve hostnames to the name of
>> >> >>>> >>> >> >services,
>> >> >>>> >>> >> >which
>> >> >>>> >>> >> >is where the internal 'hawkular-metrics' comes from.
>> >> >>>> >>> >> >
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> --
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> Srinivas Kotaru
>> >> >>>> >>> >> >>
>> >> >>>> >>> >> >> _______________________________________________
>> >> >>>> >>> >> >> users mailing list
>> >> >>>> >>> >> >> [email protected]
>> >> >>>> >>> >> >> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>> >> >>>> >>> >> >>
>> >> >>>> >>> >>
>> >> >>>> >>> >>
>> >> >>>> >>>
>> >> >>>> >>>
>> >> >>>> >
>> >> >>>>
>> >> >>>>
>> >> >>
>> >> >
>> >>
>> >>
>>
>>
_______________________________________________
users mailing list
[email protected]
http://lists.openshift.redhat.com/openshiftmm/listinfo/users
