Re: [openstack-dev] Weight normalization in scheduler
Hi again. Thank you for your reply, Sandy. Some more comments inline. On Thu 01 Aug 2013 (10:04), Sandy Walsh wrote: > On 08/01/2013 09:51 AM, Álvaro López García wrote: > > On Thu 01 Aug 2013 (09:07), Sandy Walsh wrote: > >> On 08/01/2013 04:24 AM, Álvaro López García wrote: > >>> Hi all. > >>> > >>> TL;DR: I've created a blueprint [1] regarding weight normalization. > >>> I would be very glad if somebody could examine and comment it. > >> > >> Something must have changed. It's been a while since I've done anything > >> with the scheduler, but normalized weights is the way it was designed > >> and implemented. > > > > It seems reasonable, but it is not there anymore: > > > > class RAMWeigher(weights.BaseHostWeigher): > > (...) > > def _weigh_object(self, host_state, weight_properties): > > """Higher weights win. We want spreading to be the default.""" > > return host_state.free_ram_mb > > Hmm, that's unfortunate. We use our own weighing functions internally, > so perhaps we were unaffected by this change. And that is why we spoted this. We wanted to implement our very own functions apart from the RAMWeigher and we found that RAW values were used. > >> The separate Weighing plug-ins are responsible for taking the specific > >> units (cpu load, disk, ram, etc) and converting them into normalized > >> 0.0-1.0 weights. Internally the plug-ins can work however they like, but > >> their output should be 0-1. > > > > With the current code, this is not true. Anyway, I think this responsability > > should be implemented in the BaseWeightHandler rather than each weigher. > > This way each weigher can return whatever they want, but we will be > > always using a correct value. > > I think the problem with moving it to the base handler is that the base > doesn't know the max range of the value ... of course, this could be > passed down. But yeah, we wouldn't want to duplicate the normalization > code itself in every function. With the code in [1] the weigher can specify the maximum and minimum values where a weight can range if it is needed (it most cases just taking these values from the list of returned values should be enough) and the BaseWeightHandler will normalize the list before adding them up to the objects. I do not see any real advantage in doing it into each weigher. Apart from code duplication it is difficult to maintain in the long term, since any change to the normalization should be propagated to all the weighers (ok, now there's only one ;-) ). [1] https://review.openstack.org/#/c/27160 Cheers, -- Álvaro López García al...@ifca.unican.es Instituto de Física de Cantabria http://alvarolopez.github.io Ed. Juan Jordá, Campus UC tel: (+34) 942 200 969 Avda. de los Castros s/n 39005 Santander (SPAIN) _ http://xkcd.com/571/ ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Weight normalization in scheduler
On 08/01/2013 09:51 AM, Álvaro López García wrote: > On Thu 01 Aug 2013 (09:07), Sandy Walsh wrote: >> >> >> On 08/01/2013 04:24 AM, Álvaro López García wrote: >>> Hi all. >>> >>> TL;DR: I've created a blueprint [1] regarding weight normalization. >>> I would be very glad if somebody could examine and comment it. >> >> Something must have changed. It's been a while since I've done anything >> with the scheduler, but normalized weights is the way it was designed >> and implemented. > > It seems reasonable, but it is not there anymore: > > class RAMWeigher(weights.BaseHostWeigher): > (...) > def _weigh_object(self, host_state, weight_properties): > """Higher weights win. We want spreading to be the default.""" > return host_state.free_ram_mb Hmm, that's unfortunate. We use our own weighing functions internally, so perhaps we were unaffected by this change. > > >> The separate Weighing plug-ins are responsible for taking the specific >> units (cpu load, disk, ram, etc) and converting them into normalized >> 0.0-1.0 weights. Internally the plug-ins can work however they like, but >> their output should be 0-1. > > With the current code, this is not true. Anyway, I think this responsability > should be implemented in the BaseWeightHandler rather than each weigher. > This way each weigher can return whatever they want, but we will be > always using a correct value. I think the problem with moving it to the base handler is that the base doesn't know the max range of the value ... of course, this could be passed down. But yeah, we wouldn't want to duplicate the normalization code itself in every function. >> The multiplier, however, could scale this outside that range (if disk is >> more important than cpu, for example). > > Yes, of course, since the multiplier is applied *after* normalizing the > weights. > >> Actually, I remember it being offset + scale * weight, so you could put >> certain factors in bands: cpu: 1000+, disk: 1+, etc. Hopefully >> offset is still there too? > > No, it is not. poop. You may want to consider adding it back. Offset + Scale * Weight allows for many more weighing constructs. > > Thanks, > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Weight normalization in scheduler
On Thu 01 Aug 2013 (09:07), Sandy Walsh wrote: > > > On 08/01/2013 04:24 AM, Álvaro López García wrote: > > Hi all. > > > > TL;DR: I've created a blueprint [1] regarding weight normalization. > > I would be very glad if somebody could examine and comment it. > > Something must have changed. It's been a while since I've done anything > with the scheduler, but normalized weights is the way it was designed > and implemented. It seems reasonable, but it is not there anymore: class RAMWeigher(weights.BaseHostWeigher): (...) def _weigh_object(self, host_state, weight_properties): """Higher weights win. We want spreading to be the default.""" return host_state.free_ram_mb > The separate Weighing plug-ins are responsible for taking the specific > units (cpu load, disk, ram, etc) and converting them into normalized > 0.0-1.0 weights. Internally the plug-ins can work however they like, but > their output should be 0-1. With the current code, this is not true. Anyway, I think this responsability should be implemented in the BaseWeightHandler rather than each weigher. This way each weigher can return whatever they want, but we will be always using a correct value. > The multiplier, however, could scale this outside that range (if disk is > more important than cpu, for example). Yes, of course, since the multiplier is applied *after* normalizing the weights. > Actually, I remember it being offset + scale * weight, so you could put > certain factors in bands: cpu: 1000+, disk: 1+, etc. Hopefully > offset is still there too? No, it is not. Thanks, -- Álvaro López García al...@ifca.unican.es ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Weight normalization in scheduler
On 08/01/2013 04:24 AM, Álvaro López García wrote: > Hi all. > > TL;DR: I've created a blueprint [1] regarding weight normalization. > I would be very glad if somebody could examine and comment it. Something must have changed. It's been a while since I've done anything with the scheduler, but normalized weights is the way it was designed and implemented. The separate Weighing plug-ins are responsible for taking the specific units (cpu load, disk, ram, etc) and converting them into normalized 0.0-1.0 weights. Internally the plug-ins can work however they like, but their output should be 0-1. The multiplier, however, could scale this outside that range (if disk is more important than cpu, for example). Actually, I remember it being offset + scale * weight, so you could put certain factors in bands: cpu: 1000+, disk: 1+, etc. Hopefully offset is still there too? -S > Recently I've been developing some weighers to be used within nova and I > found that the weight system was using raw values. This makes difficult > for an operator to stablish the importance of a weigher against the rest > of them, since the values can range freely and one big magnitude > returned by a weigher could shade another one. > > One solution is to inflate either the multiplier or the weight that is > returned by the weigher, but this is an ugly hack (for example, if you > increase the RAM on your systems, you will need to adjust the > multipliers again). A much better approach is to use weight > normalization before actually using the weights > > With weight normalization a weigher will still return a list of RAW > values, but the BaseWeightHandler will normalize all of them into a > range of values (0.0 and 1.0) before adding them up. This way, a weight > for a given object will be: > > weight = w1_multiplier * norm(w1) + w2_multiplier * norm(w2) + ... > > This makes easier to stablish the importance of a weigher regarding the > rest, by just adjusting the multiplier. This is explained in [1], and > implemented in [2] (with some suggestions by the reviewers). > > [1] > https://blueprints.launchpad.net/openstack/?searchtext=normalize-scheduler-weights > [2] https://review.openstack.org/#/c/27160/ > > Thanks for your feedback, > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] Weight normalization in scheduler
Hi all. TL;DR: I've created a blueprint [1] regarding weight normalization. I would be very glad if somebody could examine and comment it. Recently I've been developing some weighers to be used within nova and I found that the weight system was using raw values. This makes difficult for an operator to stablish the importance of a weigher against the rest of them, since the values can range freely and one big magnitude returned by a weigher could shade another one. One solution is to inflate either the multiplier or the weight that is returned by the weigher, but this is an ugly hack (for example, if you increase the RAM on your systems, you will need to adjust the multipliers again). A much better approach is to use weight normalization before actually using the weights With weight normalization a weigher will still return a list of RAW values, but the BaseWeightHandler will normalize all of them into a range of values (0.0 and 1.0) before adding them up. This way, a weight for a given object will be: weight = w1_multiplier * norm(w1) + w2_multiplier * norm(w2) + ... This makes easier to stablish the importance of a weigher regarding the rest, by just adjusting the multiplier. This is explained in [1], and implemented in [2] (with some suggestions by the reviewers). [1] https://blueprints.launchpad.net/openstack/?searchtext=normalize-scheduler-weights [2] https://review.openstack.org/#/c/27160/ Thanks for your feedback, -- Álvaro López García al...@ifca.unican.es ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
