Okay It worked! Thanks:
$ oc get nodes
NAME LABELS
STATUS
AGE
ip-172-xx-xx-xx.xx-xx-1.compute.internal
kubernetes.io/hostname=ip-172-xx-xx-xx.xx-central-1.compute.internal,region=xx,zone=xx
Ready 3d
ip-172-xx-xx-xx.xx-xx-1.compute.internal
kubernetes.io/hostname=ip-172-xx-xx-xx.xx-central-1.compute.internal,region=xx,zone=xx
Ready 3d
ip-172-xx-xx-xx.xx-xx-1.compute.internal
kubernetes.io/hostname=ip-172-xx-xx-xx.xx-central-1.compute.internal,region=xx,zone=xx
Ready,SchedulingDisabled 3m
$ oc get hpa
NAME REFERENCE TARGET CURRENT MINPODS
MAXPODS AGE
test-scaler DeploymentConfig/test/scale 5% 0% 1 10
1d
From: [email protected]
To: [email protected]
Subject: RE: horizontal autoscaler does not get cpu utilization
Date: Thu, 25 Feb 2016 14:52:37 +0000
CC: [email protected]
I'm able to connect from each node, but not from the master.
But I saw in this issue which is the same:
https://github.com/openshift/origin/issues/6293
On each node:
* About to connect() to 10.1.0.6 port 8082 (#0)
* Trying 10.1.0.6...
* Connected to 10.1.0.6 (10.1.0.6) port 8082 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 10.1.0.6:8082
> Accept: */*
>
On the master:
curl -v 10.1.0.6:8082
* About to connect() to 10.1.0.6 port 8082 (#0)
* Trying 10.1.0.6...
* Failed connect to 10.1.0.6:8082; Connection timed out
* Closing connection 0
curl: (7) Failed connect to 10.1.0.6:8082; Connection timed out
I will try to rerun the ansible playbook and describe my master as a node too.
But unschedulable.
Date: Thu, 25 Feb 2016 09:48:55 -0500
Subject: Re: horizontal autoscaler does not get cpu utilization
From: [email protected]
To: [email protected]
CC: [email protected]
Sorry, I mistyped the URL. 8080 is a force of habit.
Try curl with -v. And where are you able to connect / not connect from?
Masters? Nodes?
On Thu, Feb 25, 2016 at 9:30 AM, Den Cowboy <[email protected]> wrote:
curl 10.1.0.6:8080 gave me:
url: (7) Failed connect to 10.1.0.6:8080; Connection refused
curl 10.1.0.6:8082 gave me just an empty line
Date: Thu, 25 Feb 2016 09:28:38 -0500
Subject: Re: horizontal autoscaler does not get cpu utilization
From: [email protected]
To: [email protected]
CC: [email protected]
That error message is your issue. Are you able to curl 10.1.0.6:8080 from any
of your nodes?
On Thu, Feb 25, 2016 at 8:35 AM, Den Cowboy <[email protected]> wrote:
I keep getting a timout:
Error: 'dial tcp 10.1.0.6:8082: i/o timeout'
or without the -k
curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file ...
command:
$ curl -H "Authorization: Bearer token" \
> -X GET
> https://ip-172-31-xx-xx.xx-xx-1.compute.internal:8443/api/v1/proxy/namespaces/openshift-infra/services/https:heapster:/validate
+ What was your thought about the errors in the second mail:
invalid character 'E' looking for beginning of value
Date: Thu, 25 Feb 2016 08:12:19 -0500
Subject: Re: horizontal autoscaler does not get cpu utilization
From: [email protected]
To: [email protected]
CC: [email protected]; [email protected]
2 things:
1) you need to provide authentication when you make the curl request
2) you need a : in between https and heapster (so httpsheapster: becomes
https:heapster:)
On Thu, Feb 25, 2016 at 3:24 AM, Den Cowboy <[email protected]> wrote:
I got this from the curl (as system:admin)
curl -k
https://ip-xx-xx-xx-xx.xx-xx-1.compute.internal:8443/api/v1/proxy/namespaces/openshift-infra/services/httpsheapster:/validate
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {},
"status": "Failure",
"message": "User \"system:anonymous\" cannot \"proxy\" \"services\" with name
\"httpsheapster:\" in project \"openshift-infra\"",
"reason": "Forbidden",
"details": {
"name": "httpsheapster:",
"kind": "services"
},
"code": 403
From: [email protected]
To: [email protected]
Subject: RE: horizontal autoscaler does not get cpu utilization
Date: Thu, 25 Feb 2016 07:59:11 +0000
CC: [email protected]
Some logs are showing:
Failed to reconcile test-scaler: failed to compute desired number of replicas
based on CPU utilization for DeploymentConfig/test/test: failed to get cpu
utilization: failed to get CPU consumption and request: failed to unmarshall
heapster response: invalid character 'E' looking for beginning of value
Feb 25 07:48:57 ip-172-31-xx-xx origin-master: E0225 07:48:57.079028 2242
event.go:192] Server rejected event
'&api.Event{TypeMeta:unversioned.TypeMeta{Kind:"", APIVersion:""},
ObjectMeta:api.ObjectMeta{Name:"test-scaler.14361ecd543d4608", GenerateName:"",
Namespace:"test", SelfLink:"", UID:"", ResourceVersion:"", Generation:0,
CreationTimestamp:unversioned.Time{Time:time.Time{sec:0, nsec:0,
loc:(*time.Location)(nil)}}, DeletionTimestamp:(*unversioned.Time)(nil),
DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil),
Annotations:map[string]string(nil)},
InvolvedObject:api.ObjectReference{Kind:"HorizontalPodAutoscaler",
Namespace:"test", Name:"test-scaler",
UID:"f7bac384-db00-11e5-ac6e-06b94d3c6589", APIVersion:"extensions",
ResourceVersion:"13501", FieldPath:""}, Reason:"FailedGetMetrics",
Message:"failed to get CPU consumption and request: failed to unmarshall
heapster response: invalid character 'E' looking for beginning of value",
I used this configuration:
$ oc secrets new metrics-deployer nothing=/dev/null
> Date: Wed, 24 Feb 2016 13:29:21 -0500
> From: [email protected]
> To: [email protected]
> CC: [email protected]
> Subject: Re: horizontal autoscaler does not get cpu utilization
>
>
>
> ----- Original Message -----
> > From: "Den Cowboy" <[email protected]>
> > To: [email protected]
> > Sent: Wednesday, February 24, 2016 9:35:34 AM
> > Subject: RE: horizontal autoscaler does not get cpu utilization
> >
> > I don't know if this is maybe the issue?
> > In my browser https://hawkular-metrics.xx.xx.com/hawkular/metrics/status
> > {"MetricsService":"STARTED","Implementation-Version":"0.13.0-SNAPSHOT","Built-From-Git-SHA1":"7dee24acfcfb3beac356e2c4d83b7b1704ebf82x"}
> > curl on my master or nodes:
> > curl -X GET https://hawkular-metrics.xx.xx.com/hawkular/metrics/status
> > curl: (6) Could not resolve host: hawkular-metrics.xx.xx.com; Name or
> > service
> > not known
> >
> > I'm just describing the IP of the node where my router is in my local
> > /etc/hosts
> > like this: xx.xx.xx.xx hawkular-metrics.xx.xx.com
>
> The router configuration is not used for the HPA and so not being able to
> resolve the hostname from within the node or container should not be an issue.
>
> What the HPA does use is the API proxy.
>
> You can check if Heapster is accessible via the API proxy through the
> following command:
>
> curl -H "Authorization: Bearer XXXXXXXXXXXXXXXXX" \
> -X GET
> https://${KUBERNETES_MASTER}/api/v1/proxy/namespaces/openshift-infra/services/https:heapster:/api/v1/model/
>
> Are there any other errors in the OpenShift logs [not the container logs for
> Hawkular-Metrics, Cassandra or Heapster, those appear to be working since you
> can see metrics in the browser]
>
> >
> >
> > From: [email protected]
> > To: [email protected]
> > Subject: horizontal autoscaler does not get cpu utilization
> > Date: Wed, 24 Feb 2016 13:56:24 +0000
> >
> > I'm on Origin 1.1.3
> > I've confgured the cluster-metrics (its in the openshift-infra project!).
> > I'm
> > able to see all the metrics (memory & cpu) on my metrics-tab.
> > Now I try to create a simple autoscaler:
> > oc autoscale dc/test --min 2 --max 15 --cpu-percent=70
> >
> > I've edited the dc of my container so now it's using resources requests and
> > limits.
> > In my webconsole I see the 2 cirkles and the MiB and millicores used.
> >
> > But
> > oc get hpa
> > NAME REFERENCE TARGET CURRENT MINPODS MAXPODS AGE
> > test DeploymentConfig/test/scale 70% <waiting> 2 15 21m
> >
> > Target CPU utilization: 70%
> > Current CPU utilization: <not available>
> >
> > I read it was normal that the current state was waiting in the beginning.
> > But
> > it's already on <waiting> for 21 minutes.
> > How can I check what's going wrong?
> >
> > The logs of the pod where I've created the autoscaler doesn't show anything
> > different than normal:
> > AH00558: httpd: Could not reliably determine the server's fully qualified
> > domain name, using 10.1.1.14. Set the 'ServerName' directive globally to
> > suppress this message
> >
> > The logs of my metrics (but they seem to work fine):
> > casandra: INFO 12:59:51 Initializing hawkular_metrics.leases
> >
> > heapster:
> > I0224 13:28:35.108008 1 manager.go:152] ScrapeMetrics: time: 107.715184ms
> > size: 160
> > I0224 13:29:05.000203 1 manager.go:79] Scraping metrics start: 2016-02-24
> > 13:28:30 +0000 UTC, end: 2016-02-24 13:29:00 +0000 UTC
> > I0224 13:29:05.118256 1 manager.go:152] ScrapeMetrics: time: 117.97843ms
> > size: 160
> > I0224 13:29:35.000209 1 manager.go:79] Scraping metrics start: 2016-02-24
> > 13:29:00 +0000 UTC, end: 2016-02-24 13:29:30 +0000 UTC
> > I0224 13:29:35.106305 1 manager.go:152] ScrapeMetrics: time: 106.009207ms
> > size: 160
> > I0224 13:30:05.000266 1 manager.go:79] Scraping metrics start: 2016-02-24
> > 13:29:30 +0000 UTC, end: 2016-02-24 13:30:00 +0000 UTC
> > I0224 13:30:05.113913 1 manager.go:152] ScrapeMetrics: time: 113.477259ms
> > size: 160
> >
> > metrics deployer:
> > ++ '[' -n '' ']'
> > Deploying the Heapster component
> > ++ oc process hawkular-heapster -v
> > IMAGE_PREFIX=docker.io/openshift/origin-,IMAGE_VERSION=latest,MASTER_URL=https://kubernetes.default.svc:443
> > ++ oc create -f -
> > serviceaccount "heapster" created
> > service "heapster" created
> > replicationcontroller "heapster" created
> > + echo 'Success!'
> > Success!
> >
> > Hawkular-metrics:
> > [0m[0m12:59:48,619 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0051:
> > Admin console listening on http://127.0.0.1:9990
> > [0m[0m12:59:48,620 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025:
> > WildFly Full 10.0.0.Final (WildFly Core 2.0.10.Final) started in 16058ms -
> > Started 374 of 664 services (385 services are lazy, passive or on-demand)
> > [0m[0m12:59:52,446 INFO
> > [org.hawkular.metrics.core.service.MetricsServiceImpl]
> > (metricsservice-lifecycle-thread) HAWKMETRICS100001: Using a key space of
> > 'hawkular_metrics'
> > [0m[0m12:59:52,561 INFO
> > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > (metricsservice-lifecycle-thread) HAWKMETRICS200005: Metrics service started
> >
> > _______________________________________________ users mailing list
> > [email protected]
> > http://lists.openshift.redhat.com/openshiftmm/listinfo/users
> >
> > _______________________________________________
> > users mailing list
> > [email protected]
> > http://lists.openshift.redhat.com/openshiftmm/listinfo/users
> >
_______________________________________________
users mailing list
[email protected]
http://lists.openshift.redhat.com/openshiftmm/listinfo/users
_______________________________________________
users mailing list
[email protected]
http://lists.openshift.redhat.com/openshiftmm/listinfo/users
_______________________________________________
users mailing list
[email protected]
http://lists.openshift.redhat.com/openshiftmm/listinfo/users
_______________________________________________
users mailing list
[email protected]
http://lists.openshift.redhat.com/openshiftmm/listinfo/users
