I'm able to connect from each node, but not from the master. But I saw in this issue which is the same: https://github.com/openshift/origin/issues/6293 On each node: * About to connect() to 10.1.0.6 port 8082 (#0) * Trying 10.1.0.6... * Connected to 10.1.0.6 (10.1.0.6) port 8082 (#0) > GET / HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 10.1.0.6:8082 > Accept: */* >
On the master: curl -v 10.1.0.6:8082 * About to connect() to 10.1.0.6 port 8082 (#0) * Trying 10.1.0.6... * Failed connect to 10.1.0.6:8082; Connection timed out * Closing connection 0 curl: (7) Failed connect to 10.1.0.6:8082; Connection timed out I will try to rerun the ansible playbook and describe my master as a node too. But unschedulable. Date: Thu, 25 Feb 2016 09:48:55 -0500 Subject: Re: horizontal autoscaler does not get cpu utilization From: [email protected] To: [email protected] CC: [email protected] Sorry, I mistyped the URL. 8080 is a force of habit. Try curl with -v. And where are you able to connect / not connect from? Masters? Nodes? On Thu, Feb 25, 2016 at 9:30 AM, Den Cowboy <[email protected]> wrote: curl 10.1.0.6:8080 gave me: url: (7) Failed connect to 10.1.0.6:8080; Connection refused curl 10.1.0.6:8082 gave me just an empty line Date: Thu, 25 Feb 2016 09:28:38 -0500 Subject: Re: horizontal autoscaler does not get cpu utilization From: [email protected] To: [email protected] CC: [email protected] That error message is your issue. Are you able to curl 10.1.0.6:8080 from any of your nodes? On Thu, Feb 25, 2016 at 8:35 AM, Den Cowboy <[email protected]> wrote: I keep getting a timout: Error: 'dial tcp 10.1.0.6:8082: i/o timeout' or without the -k curl performs SSL certificate verification by default, using a "bundle" of Certificate Authority (CA) public keys (CA certs). If the default bundle file isn't adequate, you can specify an alternate file ... command: $ curl -H "Authorization: Bearer token" \ > -X GET > https://ip-172-31-xx-xx.xx-xx-1.compute.internal:8443/api/v1/proxy/namespaces/openshift-infra/services/https:heapster:/validate + What was your thought about the errors in the second mail: invalid character 'E' looking for beginning of value Date: Thu, 25 Feb 2016 08:12:19 -0500 Subject: Re: horizontal autoscaler does not get cpu utilization From: [email protected] To: [email protected] CC: [email protected]; [email protected] 2 things: 1) you need to provide authentication when you make the curl request 2) you need a : in between https and heapster (so httpsheapster: becomes https:heapster:) On Thu, Feb 25, 2016 at 3:24 AM, Den Cowboy <[email protected]> wrote: I got this from the curl (as system:admin) curl -k https://ip-xx-xx-xx-xx.xx-xx-1.compute.internal:8443/api/v1/proxy/namespaces/openshift-infra/services/httpsheapster:/validate { "kind": "Status", "apiVersion": "v1", "metadata": {}, "status": "Failure", "message": "User \"system:anonymous\" cannot \"proxy\" \"services\" with name \"httpsheapster:\" in project \"openshift-infra\"", "reason": "Forbidden", "details": { "name": "httpsheapster:", "kind": "services" }, "code": 403 From: [email protected] To: [email protected] Subject: RE: horizontal autoscaler does not get cpu utilization Date: Thu, 25 Feb 2016 07:59:11 +0000 CC: [email protected] Some logs are showing: Failed to reconcile test-scaler: failed to compute desired number of replicas based on CPU utilization for DeploymentConfig/test/test: failed to get cpu utilization: failed to get CPU consumption and request: failed to unmarshall heapster response: invalid character 'E' looking for beginning of value Feb 25 07:48:57 ip-172-31-xx-xx origin-master: E0225 07:48:57.079028 2242 event.go:192] Server rejected event '&api.Event{TypeMeta:unversioned.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:api.ObjectMeta{Name:"test-scaler.14361ecd543d4608", GenerateName:"", Namespace:"test", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:unversioned.Time{Time:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*unversioned.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil)}, InvolvedObject:api.ObjectReference{Kind:"HorizontalPodAutoscaler", Namespace:"test", Name:"test-scaler", UID:"f7bac384-db00-11e5-ac6e-06b94d3c6589", APIVersion:"extensions", ResourceVersion:"13501", FieldPath:""}, Reason:"FailedGetMetrics", Message:"failed to get CPU consumption and request: failed to unmarshall heapster response: invalid character 'E' looking for beginning of value", I used this configuration: $ oc secrets new metrics-deployer nothing=/dev/null > Date: Wed, 24 Feb 2016 13:29:21 -0500 > From: [email protected] > To: [email protected] > CC: [email protected] > Subject: Re: horizontal autoscaler does not get cpu utilization > > > > ----- Original Message ----- > > From: "Den Cowboy" <[email protected]> > > To: [email protected] > > Sent: Wednesday, February 24, 2016 9:35:34 AM > > Subject: RE: horizontal autoscaler does not get cpu utilization > > > > I don't know if this is maybe the issue? > > In my browser https://hawkular-metrics.xx.xx.com/hawkular/metrics/status > > {"MetricsService":"STARTED","Implementation-Version":"0.13.0-SNAPSHOT","Built-From-Git-SHA1":"7dee24acfcfb3beac356e2c4d83b7b1704ebf82x"} > > curl on my master or nodes: > > curl -X GET https://hawkular-metrics.xx.xx.com/hawkular/metrics/status > > curl: (6) Could not resolve host: hawkular-metrics.xx.xx.com; Name or > > service > > not known > > > > I'm just describing the IP of the node where my router is in my local > > /etc/hosts > > like this: xx.xx.xx.xx hawkular-metrics.xx.xx.com > > The router configuration is not used for the HPA and so not being able to > resolve the hostname from within the node or container should not be an issue. > > What the HPA does use is the API proxy. > > You can check if Heapster is accessible via the API proxy through the > following command: > > curl -H "Authorization: Bearer XXXXXXXXXXXXXXXXX" \ > -X GET > https://${KUBERNETES_MASTER}/api/v1/proxy/namespaces/openshift-infra/services/https:heapster:/api/v1/model/ > > Are there any other errors in the OpenShift logs [not the container logs for > Hawkular-Metrics, Cassandra or Heapster, those appear to be working since you > can see metrics in the browser] > > > > > > > From: [email protected] > > To: [email protected] > > Subject: horizontal autoscaler does not get cpu utilization > > Date: Wed, 24 Feb 2016 13:56:24 +0000 > > > > I'm on Origin 1.1.3 > > I've confgured the cluster-metrics (its in the openshift-infra project!). > > I'm > > able to see all the metrics (memory & cpu) on my metrics-tab. > > Now I try to create a simple autoscaler: > > oc autoscale dc/test --min 2 --max 15 --cpu-percent=70 > > > > I've edited the dc of my container so now it's using resources requests and > > limits. > > In my webconsole I see the 2 cirkles and the MiB and millicores used. > > > > But > > oc get hpa > > NAME REFERENCE TARGET CURRENT MINPODS MAXPODS AGE > > test DeploymentConfig/test/scale 70% <waiting> 2 15 21m > > > > Target CPU utilization: 70% > > Current CPU utilization: <not available> > > > > I read it was normal that the current state was waiting in the beginning. > > But > > it's already on <waiting> for 21 minutes. > > How can I check what's going wrong? > > > > The logs of the pod where I've created the autoscaler doesn't show anything > > different than normal: > > AH00558: httpd: Could not reliably determine the server's fully qualified > > domain name, using 10.1.1.14. Set the 'ServerName' directive globally to > > suppress this message > > > > The logs of my metrics (but they seem to work fine): > > casandra: INFO 12:59:51 Initializing hawkular_metrics.leases > > > > heapster: > > I0224 13:28:35.108008 1 manager.go:152] ScrapeMetrics: time: 107.715184ms > > size: 160 > > I0224 13:29:05.000203 1 manager.go:79] Scraping metrics start: 2016-02-24 > > 13:28:30 +0000 UTC, end: 2016-02-24 13:29:00 +0000 UTC > > I0224 13:29:05.118256 1 manager.go:152] ScrapeMetrics: time: 117.97843ms > > size: 160 > > I0224 13:29:35.000209 1 manager.go:79] Scraping metrics start: 2016-02-24 > > 13:29:00 +0000 UTC, end: 2016-02-24 13:29:30 +0000 UTC > > I0224 13:29:35.106305 1 manager.go:152] ScrapeMetrics: time: 106.009207ms > > size: 160 > > I0224 13:30:05.000266 1 manager.go:79] Scraping metrics start: 2016-02-24 > > 13:29:30 +0000 UTC, end: 2016-02-24 13:30:00 +0000 UTC > > I0224 13:30:05.113913 1 manager.go:152] ScrapeMetrics: time: 113.477259ms > > size: 160 > > > > metrics deployer: > > ++ '[' -n '' ']' > > Deploying the Heapster component > > ++ oc process hawkular-heapster -v > > IMAGE_PREFIX=docker.io/openshift/origin-,IMAGE_VERSION=latest,MASTER_URL=https://kubernetes.default.svc:443 > > ++ oc create -f - > > serviceaccount "heapster" created > > service "heapster" created > > replicationcontroller "heapster" created > > + echo 'Success!' > > Success! > > > > Hawkular-metrics: > > [0m[0m12:59:48,619 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0051: > > Admin console listening on http://127.0.0.1:9990 > > [0m[0m12:59:48,620 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: > > WildFly Full 10.0.0.Final (WildFly Core 2.0.10.Final) started in 16058ms - > > Started 374 of 664 services (385 services are lazy, passive or on-demand) > > [0m[0m12:59:52,446 INFO > > [org.hawkular.metrics.core.service.MetricsServiceImpl] > > (metricsservice-lifecycle-thread) HAWKMETRICS100001: Using a key space of > > 'hawkular_metrics' > > [0m[0m12:59:52,561 INFO > > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] > > (metricsservice-lifecycle-thread) HAWKMETRICS200005: Metrics service started > > > > _______________________________________________ users mailing list > > [email protected] > > http://lists.openshift.redhat.com/openshiftmm/listinfo/users > > > > _______________________________________________ > > users mailing list > > [email protected] > > http://lists.openshift.redhat.com/openshiftmm/listinfo/users > > _______________________________________________ users mailing list [email protected] http://lists.openshift.redhat.com/openshiftmm/listinfo/users _______________________________________________ users mailing list [email protected] http://lists.openshift.redhat.com/openshiftmm/listinfo/users
_______________________________________________ users mailing list [email protected] http://lists.openshift.redhat.com/openshiftmm/listinfo/users
