Matt PV issue resolved. Was able to to see PV successfully bounded and Casandra container has been running. However, it seems puzzle not fully yet solved.
I could see other container(heapster) not coming up, and seeing below errors [skotaru@l3imas-id2-01 metrics]$ oc logs -f heapster-fnkdc Endpoint Check in effect. Checking https://hawkular-metrics:443/hawkular/metrics/status Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 6. Status Code 000 'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 6]. Retrying. Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 6. Status Code 000 'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 6]. Retrying. # oc get pv pv-5gb-0011 5Gi RWO Bound openshift-infra/metrics-cassandra-1 22m $ oc get pods NAME READY STATUS RESTARTS AGE hawkular-cassandra-1-2pzd7 1/1 Running 0 20m hawkular-metrics-mf5qf 0/1 Running 7 20m heapster-fnkdc 0/1 Error 6 20m metrics-deployer-cvep0 0/1 Completed 0 21m # oc logs -f hawkular-metrics-mf5qf 19:20:00,819 INFO [org.xnio] (MSC service thread 1-2) XNIO Version 3.0.14.GA-redhat-1 19:20:00,831 INFO [org.jboss.as.server] (Controller Boot Thread) JBAS015888: Creating http management service using socket-binding (management-http) 19:20:00,834 INFO [org.xnio.nio] (MSC service thread 1-2) XNIO NIO Implementation Version 3.0.14.GA-redhat-1 19:20:00,844 INFO [org.jboss.remoting] (MSC service thread 1-2) JBoss Remoting version 3.3.5.Final-redhat-1 $ oc logs -f heapster-fnkdc Endpoint Check in effect. Checking https://hawkular-metrics:443/hawkular/metrics/status Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 6. Status Code 000 'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 6]. Retrying. Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 6. Status Code 000 'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 6]. Retrying. Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 6. Status Code 000 $ oc logs -f hawkular-cassandra-1-2pzd7 INFO 23:00:24 Starting listening for CQL clients on hawkular-cassandra-1-2pzd7/10.1.6.2:9042... INFO 23:00:24 Binding thrift service to hawkular-cassandra-1-2pzd7/10.1.6.2:9160 INFO 23:00:24 enabling encrypted thrift connections between client and server INFO 23:00:24 Listening for thrift clients... INFO 23:00:26 Created default superuser role 'cassandra' # oc get svc NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE hawkular-cassandra 172.30.2.13 <none> 9042/TCP,9160/TCP,7000/TCP,7001/TCP 25m hawkular-cassandra-nodes None <none> 9042/TCP,9160/TCP,7000/TCP,7001/TCP 25m hawkular-metrics 172.30.117.176 <none> 443/TCP 25m heapster 172.30.107.135 <none> 80/TCP 25m #curl -I 172.30.117.176:443//hawkular/metrics/status HTTP/1.1 504 Gateway Timeout Mime-Version: 1.0 Date: Mon, 13 Jun 2016 23:25:47 GMT Content-Type: text/html Connection: keep-alive Proxy-Connection: keep-alive Content-Length: 1572 -- Srinivas Kotaru On 6/13/16, 2:33 PM, "Srinivas Naga Kotaru (skotaru)" <[email protected]> wrote: >Matt > >That is good catch. I ran without USE_PERSISTENT_STORAGE=false and working > >I adjusted PV to 5Gi and reran. Will update progress. > >Thanks you for your help so far. > >-- >Srinivas Kotaru > >On 6/13/16, 2:27 PM, "Matt Wringe" <[email protected]> wrote: > >> >> >>----- Original Message ----- >>> From: "Srinivas Naga Kotaru (skotaru)" <[email protected]> >>> To: "Matt Wringe" <[email protected]> >>> Cc: [email protected] >>> Sent: Monday, June 13, 2016 5:21:01 PM >>> Subject: Re: Metrics deployment >>> >>> Oh ok >>> >>> Am using PV for metrics >>> >>> description: "The persistent volume size for each of the Cassandra nodes" >>> name: CASSANDRA_PV_SIZE >>> value: "10Gi" >>> >>> oc get pv >>> NAME CAPACITY ACCESSMODES STATUS CLAIM REASON >>> AGE >>> pv-1gb-001 1Gi RWO Available >>> 4d >>> pv-1gb-002 1Gi RWO Available >>> 4d >>> pv-1gb-003 1Gi RWO Available >>> 4d >>> pv-1gb-004 1Gi RWO Bound thlatt/mongodb >>> 4d >>> pv-1gb-005 1Gi RWO Available >>> 4d >>> pv-2gb-0010 2Gi RWO Available >>> 4d >>> pv-2gb-006 2Gi RWO Available >>> 4d >>> pv-2gb-007 2Gi RWO Available >>> 4d >>> pv-2gb-008 2Gi RWO Available >>> 4d >>> pv-2gb-009 2Gi RWO Available >>> 4d >>> pv-5gb-0011 5Gi RWO Available >>> 4d >>> pv-5gb-0012 5Gi RWO Available >>> 4d >>> pv-5gb-0013 5Gi RWO Available >>> 4d >>> pv-5gb-0014 5Gi RWO Available >>> 4d >>> pv-5gb-0015 5Gi RWO Available >>> 4d >>> >>> am running with below command >>> >>> $ oc new-app -f metrics-deployer.yaml ( hardcoded HOSTNAME, MASTER_API and >>> PV info so not passing any parameters) >>> >> >>I would suspect that Cassandra is blocked because its waiting for 10Gi PV to >>become available, and none of the PV listed above are big enough. >> >>> >>> -- >>> Srinivas Kotaru >>> >>> On 6/13/16, 2:12 PM, "Matt Wringe" <[email protected]> wrote: >>> >>> >----- Original Message ----- >>> >> From: "Srinivas Naga Kotaru (skotaru)" <[email protected]> >>> >> To: "Matt Wringe" <[email protected]> >>> >> Cc: [email protected] >>> >> Sent: Monday, June 13, 2016 4:55:55 PM >>> >> Subject: Re: Metrics deployment >>> >> >>> >> Matt >>> >> >>> >> Thanks for looking into. I rerun the setup, but had the same issue >>> >> >>> >> # oc get pods >>> >> NAME READY STATUS RESTARTS AGE >>> >> hawkular-cassandra-1-y2egy 0/1 ContainerCreating 0 5m >>> >> hawkular-metrics-4b16f 0/1 Running 1 4m >>> >> heapster-x2gj2 0/1 Running 2 4m >>> >> metrics-deployer-9v7vc 0/1 Completed 0 6m >>> >> >>> >> $ oc logs -f hawkular-cassandra-1-y2egy >>> >> Error from server: container "hawkular-cassandra-1" in pod >>> >> "hawkular-cassandra-1-y2egy" is waiting to start: ContainerCreating >>> > >>> >Ok, so it looks like something is blocking the Cassandra pod from starting. >>> > >>> >If you are using persistent storage, Cassandra will not start until the PV >>> >is available. There may be some more information about Cassandra in the pod >>> >section of the console under events. >>> > >>> >What command did you use when deploying the deployer? >>> > >>> >> >>> >> $ oc logs -f hawkular-metrics-4b16f >>> >> >>> >> 16:54:25,703 DEBUG [org.jboss.as.config] (MSC service thread 1-4) VM >>> >> Arguments: -Duser.home=/home/jboss -Duser.name=jboss -D[Standalone] >>> >> -XX:+UseCompressedOops -verbose:gc -Xloggc:/opt/eap/standalone/log/gc.log >>> >> -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation >>> >> -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=3M -XX:-TraceClassUnloading >>> >> -Xms1303m -Xmx1303m -XX:MaxPermSize=256m -Djava.net.preferIPv4Stack=true >>> >> -Djboss.modules.system.pkgs=org.jboss.logmanager -Djava.awt.headless=true >>> >> -Djboss.modules.policy-permissions=true >>> >> -Xbootclasspath/p:/opt/eap/jboss-modules.jar:/opt/eap/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-1.5.4.Final-redhat-1.jar:/opt/eap/modules/system/layers/base/org/jboss/logmanager/ext/main/javax.json-1.0.4.jar:/opt/eap/modules/system/layers/base/org/jboss/logmanager/ext/main/jboss-logmanager-ext-1.0.0.Alpha2-redhat-1.jar >>> >> -Djava.util.logging.manager=org.jboss.logmanager.LogManager >>> >> -javaagent:/opt/eap/jolokia.jar=port=8778,protocol=https,caCert=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt,clientPrincipal=cn=system:master-proxy,useSslClientAuthentication=true,extraClientCheck=true,host=0.0.0.0,discoveryEnabled=false >>> >> -Djava.security.egd=file:/dev/./urandom >>> >> -Dorg.jboss.boot.log.file=/opt/eap/standalone/log/server.log >>> >> -Dlogging.configuration=file:/opt/eap/standalone/configuration/logging.properties >>> >> 16:54:27,079 INFO [org.xnio] (MSC service thread 1-3) XNIO Version >>> >> 3.0.14.GA-redhat-1 >>> >> 16:54:27,083 INFO [org.xnio.nio] (MSC service thread 1-3) XNIO NIO >>> >> Implementation Version 3.0.14.GA-redhat-1 >>> >> 16:54:27,101 INFO [org.jboss.as.server] (Controller Boot Thread) >>> >> JBAS015888: >>> >> Creating http management service using socket-binding (management-http) >>> >> 16:54:27,104 INFO [org.jboss.remoting] (MSC service thread 1-3) JBoss >>> >> Remoting version 3.3.5.Final-redhat-1 >>> >> >>> >> $ oc logs -f heapster-x2gj2 >>> >> Endpoint Check in effect. Checking >>> >> https://hawkular-metrics:443/hawkular/metrics/status >>> >> Could not connect to >>> >> https://hawkular-metrics:443/hawkular/metrics/status. >>> >> Curl exit code: 6. Status Code 000 >>> >> 'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible >>> >> [HTTP status code: 000. Curl exit code 6]. Retrying. >>> >> Could not connect to >>> >> https://hawkular-metrics:443/hawkular/metrics/status. >>> >> Curl exit code: 6. Status Code 000 >>> >> 'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible >>> >> [HTTP status code: 000. Curl exit code 6]. Retrying. >>> >> Could not connect to >>> >> https://hawkular-metrics:443/hawkular/metrics/status. >>> >> Curl exit code: 6. Status Code 000 >>> >> >>> >> >>> >> $ oc logs -f metrics-deployer-9v7vc >>> >> >>> >> ++ oc create -f - >>> >> serviceaccount "heapster" created >>> >> service "heapster" created >>> >> replicationcontroller "heapster" created >>> >> + echo 'Success!' >>> >> Success! >>> >> >>> >> -- >>> >> Srinivas Kotaru >>> >> >>> >> On 6/13/16, 1:49 PM, "Matt Wringe" <[email protected]> wrote: >>> >> >>> >> > >>> >> > >>> >> >----- Original Message ----- >>> >> >> From: "Srinivas Naga Kotaru (skotaru)" <[email protected]> >>> >> >> To: [email protected] >>> >> >> Sent: Monday, June 13, 2016 3:58:12 PM >>> >> >> Subject: Metrics deployment >>> >> >> >>> >> >> >>> >> >> >>> >> >> Hi >>> >> >> >>> >> >> >>> >> >> >>> >> >> Am trying to configure metrics in our newly installed clusters. Am >>> >> >> seeing >>> >> >> below errors once metrics-deploy script was successful. I used our >>> >> >> environment specific HAWKULAR_METRICS_HOSTNAME and MASTER_URL >>> >> >> >>> >> >> >>> >> >> >>> >> >> # oc new-app -f metrics-deployer.yaml >>> >> >> >>> >> >> >>> >> >> >>> >> >> Note: customized, CASSANDARA PV, MASTER_URL, and >>> >> >> HAWKULAR_METRICS_HOSTNAME >>> >> >> ( >>> >> >> hard coded as values) >>> >> >> >>> >> >> >>> >> >> >>> >> >> template "hawkular-heapster" created >>> >> >> >>> >> >> Deploying the Heapster component >>> >> >> >>> >> >> ++ echo 'Deploying the Heapster component' >>> >> >> >>> >> >> ++ '[' -n '' ']' >>> >> >> >>> >> >> ++ oc create -f - >>> >> >> >>> >> >> ++ oc process hawkular-heapster -v >>> >> >> IMAGE_PREFIX=registry.access.redhat.com/openshift3/,IMAGE_VERSION=latest,MASTER_URL=https://lae3-alln-int-idev01.cisco.com:443,NODE_ID=nodename >>> >> >> >>> >> >> serviceaccount "heapster" created >>> >> >> >>> >> >> service "heapster" created >>> >> >> >>> >> >> replicationcontroller "heapster" created >>> >> >> >>> >> >> + echo 'Success!' >>> >> >> >>> >> >> Success! >>> >> >> >>> >> >> >>> >> >> >>> >> >> # oc get pods >>> >> >> >>> >> >> NAME READY STATUS RESTARTS AGE >>> >> >> >>> >> >> hawkular-cassandra-1-9nzio 0/1 ContainerCreating 0 4m >>> >> >> >>> >> >> hawkular-metrics-hi7mb 0/1 Running 1 4m >>> >> >> >>> >> >> heapster-e8gbu 0/1 Running 2 4m >>> >> >> >>> >> >> metrics-deployer-64703 0/1 ContainerCreating 0 3s >>> >> >> >>> >> >> metrics-deployer-cd1nf 0/1 Completed 0 5m >>> >> >> >>> >> > >>> >> >It looks like none of your containers are fully up and running yet. >>> >> > >>> >> >Without Cassandra running, Hawkular Metrics will not run, and Heapster >>> >> >will >>> >> >wait until Hawkular Metrics is fully running. >>> >> > >>> >> >Do you see anything in the Cassandra logs? The first step will be to get >>> >> >Cassandra running properly. >>> >> > >>> >> >> >>> >> >> >>> >> >> >>> >> >> $ oc logs -f heapster-e8gbu >>> >> >> >>> >> >> Endpoint Check in effect. Checking >>> >> >> https://hawkular-metrics:443/hawkular/metrics/status >>> >> >> >>> >> >> Could not connect to >>> >> >> https://hawkular-metrics:443/hawkular/metrics/status. >>> >> >> Curl exit code: 6. Status Code 000 >>> >> >> >>> >> >> 'https://hawkular-metrics:443/hawkular/metrics/status' is not >>> >> >> accessible >>> >> >> [HTTP status code: 000. Curl exit code 6]. Retrying. >>> >> >> >>> >> >> Could not connect to >>> >> >> https://hawkular-metrics:443/hawkular/metrics/status. >>> >> >> Curl exit code: 6. Status Code 000 >>> >> > >>> >> >Heapster waits until Hawkular Metrics is started before trying to push >>> >> >metrics to it. The issue that you are seeing is because Heapster could >>> >> >not >>> >> >properly connect to Hawkular Metrics. Until the Hawkular Metrics service >>> >> >is >>> >> >fully up, Heapster will not be able to connect to it. >>> >> > >>> >> > >>> >> >> >>> >> >> >>> >> >> >>> >> >> >>> >> >> What is the wrong? Why it checking just hawkular-metrics rather full >>> >> >> routing >>> >> >> URL which was provided as HAWKULAR_METRICS_HOSTNAME >>> >> > >>> >> >The Hawkular Metrics service has two hostnames: the internal hostname >>> >> >used >>> >> >by the internal components (eg 'hawkular-metrics') and the external >>> >> >hostname (eg what is configured via HAWKULAR_METRICS_HOSTNAME). The >>> >> >OpenShift dns server will resolve hostnames to the name of services, >>> >> >which >>> >> >is where the internal 'hawkular-metrics' comes from. >>> >> > >>> >> >> >>> >> >> >>> >> >> >>> >> >> >>> >> >> >>> >> >> >>> >> >> >>> >> >> >>> >> >> >>> >> >> >>> >> >> -- >>> >> >> >>> >> >> >>> >> >> Srinivas Kotaru >>> >> >> >>> >> >> _______________________________________________ >>> >> >> users mailing list >>> >> >> [email protected] >>> >> >> http://lists.openshift.redhat.com/openshiftmm/listinfo/users >>> >> >> >>> >> >>> >> >>> >>> > _______________________________________________ users mailing list [email protected] http://lists.openshift.redhat.com/openshiftmm/listinfo/users
