Re: [PATCH master 2/6] Add bandwidth tags and bandwidth map fields into Node

[Даниил Лещёв]

It sounds logical, I will make comments more clear.

And thanks a lot for your comments, Viktor!

2016-07-05 15:50 GMT+03:00 Viktor Bachraty :

>
>
> On Thursday, June 23, 2016 at 3:46:12 PM UTC+1, Даниил Лещёв wrote:
>>
>> From: Daniil Leshchev 
>>
>> The bandwidths map store data about network speed
>> between current node and given Node by it's bandwidth tags.
>> ---
>>  src/Ganeti/HTools/Node.hs | 30 ++
>>  1 file changed, 30 insertions(+)
>>
>> diff --git a/src/Ganeti/HTools/Node.hs b/src/Ganeti/HTools/Node.hs
>> index 6749568..4eb1576 100644
>> --- a/src/Ganeti/HTools/Node.hs
>> +++ b/src/Ganeti/HTools/Node.hs
>> @@ -60,6 +60,8 @@ module Ganeti.HTools.Node
>>, setMigrationTags
>>, setRecvMigrationTags
>>, setLocationTags
>> +  , setBandwidthTags
>> +  , setBandwidthToLocation
>>-- * Tag maps
>>, addTags
>>, delTags
>> @@ -98,6 +100,7 @@ module Ganeti.HTools.Node
>>, mkNodeGraph
>>, mkRebootNodeGraph
>>, haveExclStorage
>> +  , calcBandwidthToNode
>>) where
>>
>>  import Prelude ()
>> @@ -111,6 +114,7 @@ import qualified Data.IntMap as IntMap
>>  import Data.List (intercalate, foldl', delete, union, sortBy, groupBy)
>>  import qualified Data.Map as Map
>>  import Data.Ord (comparing)
>> +import Data.Maybe (mapMaybe)
>>  import qualified Data.Set as Set
>>  import Text.Printf (printf)
>>
>> @@ -213,6 +217,10 @@ data Node = Node
>> -- to
>>, locationScore :: Int -- ^ Sum of instance location and desired
>> location
>>   -- scores
>> +  , bandwidthTags :: Set.Set String -- ^ Node's bandwidth tags
>> +  , bandwidthMap :: Map.Map String Int -- ^ Node's network bandwidth
>> between
>> +-- current location and given
>> location
>>
>
> Could you please document here what location does mean in this context
> (peer node with given bandwidth tag, so it won't be confused with location
> tags) ?
>
>
>> +-- in Mbit per second
>>, instanceMap :: Map.Map (String, String) Int -- ^ Number of instances
>> with
>>  -- each
>> exclusion/location tags
>>  -- pair
>> @@ -384,6 +392,8 @@ create name_init mem_t_init mem_n_init mem_f_init
>> , rmigTags = Set.empty
>> , locationTags = Set.empty
>> , locationScore = 0
>> +   , bandwidthTags = Set.empty
>> +   , bandwidthMap = Map.empty
>> , instanceMap = Map.empty
>> }
>>
>> @@ -435,6 +445,15 @@ setRecvMigrationTags t val = t { rmigTags = val }
>>  setLocationTags :: Node -> Set.Set String -> Node
>>  setLocationTags t val = t { locationTags = val }
>>
>> +-- | Set the network bandwidth tags
>> +setBandwidthTags :: Node -> Set.Set String -> Node
>> +setBandwidthTags t val = t { bandwidthTags = val }
>> +
>> +-- | Add network bandwidth to nodes with given bandwidth tag
>> +setBandwidthToLocation :: Node -> String -> Int -> Node
>> +setBandwidthToLocation t tag bandwidth = t { bandwidthMap = new_map }
>> +  where new_map = Map.insert tag bandwidth (bandwidthMap t)
>>
>
> Please make clear here as well that the location doesn't refer to the
> existing location tags. (or maybe use Destination to prevent confusion?)
>
>
>> +
>>  -- | Sets the unnaccounted memory.
>>  setXmem :: Node -> Int -> Node
>>  setXmem t val = t { xMem = val }
>> @@ -556,6 +575,17 @@ calcFmemOfflineOrForthcoming node allInstances =
>>   . filter (not . Instance.usesMemory)
>>   $ nodeInstances
>>
>> +-- | Calculate the network bandwidth between two given nodes
>> +calcBandwidthToNode :: Node -> Node -> Maybe Int
>> +calcBandwidthToNode src dst =
>> +  case bndwths of
>> +[] -> Nothing
>> +_  -> Just $ minimum bndwths
>> +  where dstTags = Set.toList $ bandwidthTags dst
>> +srcMap = bandwidthMap src
>> +mapper = flip Map.lookup srcMap
>> +bndwths = mapMaybe mapper dstTags
>> +
>>  -- | Calculates the desired location score of an instance, given its
>> primary
>>  -- node.
>>  getInstanceDsrdLocScore :: Node -- ^ the primary node of the instance
>> --
>> 1.9.1
>>
>>


-- 
С уважением,
Лещёв Даниил.

Re: [PATCH master 2/6] Add bandwidth tags and bandwidth map fields into Node

['Viktor Bachraty' via ganeti-devel]

Thanks for the explanation (and the patches of course). I like that the 
model can be defined on a nodegroup level (which is the common case) and I 
see the per-node bandwidth tags and maps more like a 'cache' that helps us 
when estimating the cost of a concrete node -> node migration with the 
additional benefit of being able to cover a special corner case when a 
single node has different bandwidth characteristics. It is some data 
duplication, but makes the model more 'regular' and covers the mentioned 
corner case. Any other opinions ?

Also please go ahead with extending the calcBandwithGraph with the concise 
model as you suggested in your last email.



On Friday, July 1, 2016 at 1:09:22 PM UTC+1, Даниил Лещёв wrote:
>
> Thanks a lot for the respond, Viktor!
>
>>  
>> Sorry for the long delay to respond. I think I agree with Iustin here and 
>> it is the same thing I was trying (badly, my fault) to explain during our 
>> meeting last week. Usually clusters are built on a single rack or if they 
>> are multirack then each rack is a separate nodegroup so having a node with 
>> different connectivity than the rest would be a very unusual corner case. I 
>> don't think anyone uses such deployments.
>>
>> As I mentioned earlier, I really like the idea of using tags to specify 
>> the bandwidth. I don't mind supporting node <-> node bandwidth tags, but I 
>> think setting them on a bigger cluster (20+ nodes) for each pair of nodes 
>> is a bad idea. As long as we can set all bandwidths on nodegroup level (or 
>> cover everything with a single cluster tag) that implicitly applies for all 
>> nodes, we can (but don't have to) support a special snowflake when one of 
>> the nodes is different.
>>
>> As I understand from addBandwidthData how 'alltags' are merged, cluster 
>> tags are overridden by group tags and those are overridden by node tags 
>> (with the same prefix) ? Why is then the bandwidth graph calculated form 
>> cluster tags only (are the bandwidths specified always in cluster tags 
>> only, while the group/node tags are only for setting for the location) ? 
>> Could you please explain and document this model in more detail? I think 
>> best would be documenting on a few examples - how would the tags look like 
>> in case e.g.:
>>
>
> Ok, I agree, that the best way is to add an explanation with examples to 
> documentation.
> Btw, I also will give an answer here, in order to continue discussion.
>
>>
>> 1) single nodegroup with 5 nodes, all bandwidths symmetric
>>
>  
> group-01|...|nic:100GBit|
>
> node-01|...|_no_ node bandwidth tags (they are inherrited by group tags)
> node-02|...|_no_ node bandwidth tags (they are inherrited by group tags)
> ...
>
> htools:bandwidth:nic
> htools:bandwidth:nic:100GBit::nic:100GBit::100 So for each node bandwidthTags 
> will be Set("nic:100GBit") and every bandwidthMap will contain only one 
> entry - ("nic:100GBit":100)
>
>
> 2) 3 nodegroups, within each nodegroup symmetric bandwidths, between 
>> nodegroups different bandwidths
>>
>
>
> group-01|...|nic:1000GBit|
> group-02|...|nic:100GBit|
> group-03|...|nic:10GBit|
>
> node-01|...|_no_ node bandwidth tags (they are inherrited by group tags)
> ...
>
> htools:bandwidth:nic
> htools:bandwidth:nic:10GBit::nic:10GBit::10
> htools:bandwidth:nic:100GBit::nic:100GBit::100
> htools:bandwidth:nic:1000GBit::nic:1000GBit::1000 htools:bandwidth:
> nic:10GBit::nic:100GBit::10
> htools:bandwidth:nic:10GBit::nic:1000GBit::10
> htools:bandwidth:nic:100GBit::nic:1000GBit::100
> So for each node bandwidthTags will contain one entry and bandwidthMap 
> will contain three entries.
> for example, for node with ("nic:100GBit") bandwidth tags, the 
> bandwidthMap will be {"nic:100GBit":100, "nic:10GBit"" 10, "nic:1000GBit
> ":100}
>
> The another way I see is to throw out specifing bandwidth between 
> different tags "by hand" (like htools:bandwidth:nic:10GBit::nic:100GBit
> ::10)
> and calculate such bandwidth as minimum of bandwidths form two given tags.
> This would reduce the cluster specification, for example for 2)
>
> group-01|...|nic:1000GBit|
> group-02|...|nic:100GBit|
> group-03|...|nic:10GBit|
>
> node-01|...|_no_ node bandwidth tags (they are inherrited by group tags)
> ...
>
> htools:bandwidth:nic
> htools:bandwidth:nic:10GBit::10
> htools:bandwidth:nic:100GBit::100
> htools:bandwidth:nic:1000GBit::1000
>
>
>
> --
> Sincerely,
> Daniil Leshchev 
>

Re: [PATCH master 2/6] Add bandwidth tags and bandwidth map fields into Node

['Viktor Bachraty' via ganeti-devel]

On Thursday, June 23, 2016 at 3:46:12 PM UTC+1, Даниил Лещёв wrote:
>
> From: Daniil Leshchev  
>
> The bandwidths map store data about network speed 
> between current node and given Node by it's bandwidth tags. 
> --- 
>  src/Ganeti/HTools/Node.hs | 30 ++ 
>  1 file changed, 30 insertions(+) 
>
> diff --git a/src/Ganeti/HTools/Node.hs b/src/Ganeti/HTools/Node.hs 
> index 6749568..4eb1576 100644 
> --- a/src/Ganeti/HTools/Node.hs 
> +++ b/src/Ganeti/HTools/Node.hs 
> @@ -60,6 +60,8 @@ module Ganeti.HTools.Node 
>, setMigrationTags 
>, setRecvMigrationTags 
>, setLocationTags 
> +  , setBandwidthTags 
> +  , setBandwidthToLocation 
>-- * Tag maps 
>, addTags 
>, delTags 
> @@ -98,6 +100,7 @@ module Ganeti.HTools.Node 
>, mkNodeGraph 
>, mkRebootNodeGraph 
>, haveExclStorage 
> +  , calcBandwidthToNode 
>) where 
>   
>  import Prelude () 
> @@ -111,6 +114,7 @@ import qualified Data.IntMap as IntMap 
>  import Data.List (intercalate, foldl', delete, union, sortBy, groupBy) 
>  import qualified Data.Map as Map 
>  import Data.Ord (comparing) 
> +import Data.Maybe (mapMaybe) 
>  import qualified Data.Set as Set 
>  import Text.Printf (printf) 
>   
> @@ -213,6 +217,10 @@ data Node = Node 
> -- to 
>, locationScore :: Int -- ^ Sum of instance location and desired 
> location 
>   -- scores 
> +  , bandwidthTags :: Set.Set String -- ^ Node's bandwidth tags 
> +  , bandwidthMap :: Map.Map String Int -- ^ Node's network bandwidth 
> between 
> +-- current location and given 
> location 
>

Could you please document here what location does mean in this context 
(peer node with given bandwidth tag, so it won't be confused with location 
tags) ?
 

> +-- in Mbit per second 
>, instanceMap :: Map.Map (String, String) Int -- ^ Number of instances 
> with 
>  -- each 
> exclusion/location tags 
>  -- pair 
> @@ -384,6 +392,8 @@ create name_init mem_t_init mem_n_init mem_f_init 
> , rmigTags = Set.empty 
> , locationTags = Set.empty 
> , locationScore = 0 
> +   , bandwidthTags = Set.empty 
> +   , bandwidthMap = Map.empty 
> , instanceMap = Map.empty 
> } 
>   
> @@ -435,6 +445,15 @@ setRecvMigrationTags t val = t { rmigTags = val } 
>  setLocationTags :: Node -> Set.Set String -> Node 
>  setLocationTags t val = t { locationTags = val } 
>   
> +-- | Set the network bandwidth tags 
> +setBandwidthTags :: Node -> Set.Set String -> Node 
> +setBandwidthTags t val = t { bandwidthTags = val } 
> + 
> +-- | Add network bandwidth to nodes with given bandwidth tag 
> +setBandwidthToLocation :: Node -> String -> Int -> Node 
> +setBandwidthToLocation t tag bandwidth = t { bandwidthMap = new_map } 
> +  where new_map = Map.insert tag bandwidth (bandwidthMap t) 
>

Please make clear here as well that the location doesn't refer to the 
existing location tags. (or maybe use Destination to prevent confusion?)
 

> + 
>  -- | Sets the unnaccounted memory. 
>  setXmem :: Node -> Int -> Node 
>  setXmem t val = t { xMem = val } 
> @@ -556,6 +575,17 @@ calcFmemOfflineOrForthcoming node allInstances = 
>   . filter (not . Instance.usesMemory) 
>   $ nodeInstances 
>   
> +-- | Calculate the network bandwidth between two given nodes 
> +calcBandwidthToNode :: Node -> Node -> Maybe Int 
> +calcBandwidthToNode src dst = 
> +  case bndwths of 
> +[] -> Nothing 
> +_  -> Just $ minimum bndwths 
> +  where dstTags = Set.toList $ bandwidthTags dst 
> +srcMap = bandwidthMap src 
> +mapper = flip Map.lookup srcMap 
> +bndwths = mapMaybe mapper dstTags 
> + 
>  -- | Calculates the desired location score of an instance, given its 
> primary 
>  -- node. 
>  getInstanceDsrdLocScore :: Node -- ^ the primary node of the instance 
> -- 
> 1.9.1 
>
>

Re: [PATCH master 2/6] Add bandwidth tags and bandwidth map fields into Node

[Oleg Ponomarev]

Hi Daniil,

>The another way I see is to throw out specifing bandwidth between
different tags "by hand" (like htools:bandwidth:nic:10GBit::nic:100GBit::10)
>and calculate such bandwidth as minimum of bandwidths form two given tags.
>This would reduce the cluster specification, for example for 2)

That makes sense.
My suggestion would be to preserve node-node specification but also use
this idea if no node-node specification is available.

Sincerely,
Oleg

On Fri, Jul 1, 2016 at 3:09 PM Даниил Лещёв  wrote:

> Thanks a lot for the respond, Viktor!
>
>
>> Sorry for the long delay to respond. I think I agree with Iustin here and
>> it is the same thing I was trying (badly, my fault) to explain during our
>> meeting last week. Usually clusters are built on a single rack or if they
>> are multirack then each rack is a separate nodegroup so having a node with
>> different connectivity than the rest would be a very unusual corner case. I
>> don't think anyone uses such deployments.
>>
>> As I mentioned earlier, I really like the idea of using tags to specify
>> the bandwidth. I don't mind supporting node <-> node bandwidth tags, but I
>> think setting them on a bigger cluster (20+ nodes) for each pair of nodes
>> is a bad idea. As long as we can set all bandwidths on nodegroup level (or
>> cover everything with a single cluster tag) that implicitly applies for all
>> nodes, we can (but don't have to) support a special snowflake when one of
>> the nodes is different.
>>
>> As I understand from addBandwidthData how 'alltags' are merged, cluster
>> tags are overridden by group tags and those are overridden by node tags
>> (with the same prefix) ? Why is then the bandwidth graph calculated form
>> cluster tags only (are the bandwidths specified always in cluster tags
>> only, while the group/node tags are only for setting for the location) ?
>> Could you please explain and document this model in more detail? I think
>> best would be documenting on a few examples - how would the tags look like
>> in case e.g.:
>>
>
> Ok, I agree, that the best way is to add an explanation with examples to
> documentation.
> Btw, I also will give an answer here, in order to continue discussion.
>
>>
>> 1) single nodegroup with 5 nodes, all bandwidths symmetric
>>
>
> group-01|...|nic:100GBit|
>
> node-01|...|_no_ node bandwidth tags (they are inherrited by group tags)
> node-02|...|_no_ node bandwidth tags (they are inherrited by group tags)
> ...
>
> htools:bandwidth:nic
> htools:bandwidth:nic:100GBit::nic:100GBit::100 So for each node bandwidthTags
> will be Set("nic:100GBit") and every bandwidthMap will contain only one
> entry - ("nic:100GBit":100)
>
>
> 2) 3 nodegroups, within each nodegroup symmetric bandwidths, between
>> nodegroups different bandwidths
>>
>
>
> group-01|...|nic:1000GBit|
> group-02|...|nic:100GBit|
> group-03|...|nic:10GBit|
>
> node-01|...|_no_ node bandwidth tags (they are inherrited by group tags)
> ...
>
> htools:bandwidth:nic
> htools:bandwidth:nic:10GBit::nic:10GBit::10
> htools:bandwidth:nic:100GBit::nic:100GBit::100
> htools:bandwidth:nic:1000GBit::nic:1000GBit::1000 htools:bandwidth:
> nic:10GBit::nic:100GBit::10
> htools:bandwidth:nic:10GBit::nic:1000GBit::10
> htools:bandwidth:nic:100GBit::nic:1000GBit::100
> So for each node bandwidthTags will contain one entry and bandwidthMap
> will contain three entries.
> for example, for node with ("nic:100GBit") bandwidth tags, the
> bandwidthMap will be {"nic:100GBit":100, "nic:10GBit"" 10, "nic:1000GBit
> ":100}
>
> The another way I see is to throw out specifing bandwidth between
> different tags "by hand" (like htools:bandwidth:nic:10GBit::nic:100GBit
> ::10)
> and calculate such bandwidth as minimum of bandwidths form two given tags.
> This would reduce the cluster specification, for example for 2)
>
> group-01|...|nic:1000GBit|
> group-02|...|nic:100GBit|
> group-03|...|nic:10GBit|
>
> node-01|...|_no_ node bandwidth tags (they are inherrited by group tags)
> ...
>
> htools:bandwidth:nic
> htools:bandwidth:nic:10GBit::10
> htools:bandwidth:nic:100GBit::100
> htools:bandwidth:nic:1000GBit::1000
>
>
>
> --
> Sincerely,
> Daniil Leshchev
>

Re: [PATCH master 2/6] Add bandwidth tags and bandwidth map fields into Node

[Даниил Лещёв]

Thanks a lot for the respond, Viktor!

>
> Sorry for the long delay to respond. I think I agree with Iustin here and
> it is the same thing I was trying (badly, my fault) to explain during our
> meeting last week. Usually clusters are built on a single rack or if they
> are multirack then each rack is a separate nodegroup so having a node with
> different connectivity than the rest would be a very unusual corner case. I
> don't think anyone uses such deployments.
>
> As I mentioned earlier, I really like the idea of using tags to specify
> the bandwidth. I don't mind supporting node <-> node bandwidth tags, but I
> think setting them on a bigger cluster (20+ nodes) for each pair of nodes
> is a bad idea. As long as we can set all bandwidths on nodegroup level (or
> cover everything with a single cluster tag) that implicitly applies for all
> nodes, we can (but don't have to) support a special snowflake when one of
> the nodes is different.
>
> As I understand from addBandwidthData how 'alltags' are merged, cluster
> tags are overridden by group tags and those are overridden by node tags
> (with the same prefix) ? Why is then the bandwidth graph calculated form
> cluster tags only (are the bandwidths specified always in cluster tags
> only, while the group/node tags are only for setting for the location) ?
> Could you please explain and document this model in more detail? I think
> best would be documenting on a few examples - how would the tags look like
> in case e.g.:
>

Ok, I agree, that the best way is to add an explanation with examples to
documentation.
Btw, I also will give an answer here, in order to continue discussion.

>
> 1) single nodegroup with 5 nodes, all bandwidths symmetric
>

group-01|...|nic:100GBit|

node-01|...|_no_ node bandwidth tags (they are inherrited by group tags)
node-02|...|_no_ node bandwidth tags (they are inherrited by group tags)
...

htools:bandwidth:nic
htools:bandwidth:nic:100GBit::nic:100GBit::100 So for each node bandwidthTags
will be Set("nic:100GBit") and every bandwidthMap will contain only one
entry - ("nic:100GBit":100)


2) 3 nodegroups, within each nodegroup symmetric bandwidths, between
> nodegroups different bandwidths
>


group-01|...|nic:1000GBit|
group-02|...|nic:100GBit|
group-03|...|nic:10GBit|

node-01|...|_no_ node bandwidth tags (they are inherrited by group tags)
...

htools:bandwidth:nic
htools:bandwidth:nic:10GBit::nic:10GBit::10
htools:bandwidth:nic:100GBit::nic:100GBit::100
htools:bandwidth:nic:1000GBit::nic:1000GBit::1000 htools:bandwidth:
nic:10GBit::nic:100GBit::10
htools:bandwidth:nic:10GBit::nic:1000GBit::10
htools:bandwidth:nic:100GBit::nic:1000GBit::100
So for each node bandwidthTags will contain one entry and bandwidthMap will
contain three entries.
for example, for node with ("nic:100GBit") bandwidth tags, the bandwidthMap
will be {"nic:100GBit":100, "nic:10GBit"" 10, "nic:1000GBit":100}

The another way I see is to throw out specifing bandwidth between different
tags "by hand" (like htools:bandwidth:nic:10GBit::nic:100GBit::10)
and calculate such bandwidth as minimum of bandwidths form two given tags.
This would reduce the cluster specification, for example for 2)

group-01|...|nic:1000GBit|
group-02|...|nic:100GBit|
group-03|...|nic:10GBit|

node-01|...|_no_ node bandwidth tags (they are inherrited by group tags)
...

htools:bandwidth:nic
htools:bandwidth:nic:10GBit::10
htools:bandwidth:nic:100GBit::100
htools:bandwidth:nic:1000GBit::1000



--
Sincerely,
Daniil Leshchev

Re: [PATCH master 2/6] Add bandwidth tags and bandwidth map fields into Node

['Viktor Bachraty' via ganeti-devel]

On Tuesday, June 28, 2016 at 3:20:12 PM UTC+1, Iustin Pop wrote:
>
> On 28 June 2016 at 16:16, Даниил Лещёв  wrote:
>>
>> I have to main concerns here.
>>>
>>> 1. I still believe it's wrong to model this on a per-node basis, and 
>>> that it should be rather two things: bandwidth available inside a node 
>>> group (between any two arbitrary nodes), and bandwidth available between 
>>> any two node groups. However, your badwidth tag concept is nice. Hmm…
>>>
>>  
>> But with such approach we lose an opportunity to specify network speed 
>> between two specific nodes.
>> Or it is really rare case and we don't need such flexibility?
>>
>  
Sorry for the long delay to respond. I think I agree with Iustin here and 
it is the same thing I was trying (badly, my fault) to explain during our 
meeting last week. Usually clusters are built on a single rack or if they 
are multirack then each rack is a separate nodegroup so having a node with 
different connectivity than the rest would be a very unusual corner case. I 
don't think anyone uses such deployments.

As I mentioned earlier, I really like the idea of using tags to specify the 
bandwidth. I don't mind supporting node <-> node bandwidth tags, but I 
think setting them on a bigger cluster (20+ nodes) for each pair of nodes 
is a bad idea. As long as we can set all bandwidths on nodegroup level (or 
cover everything with a single cluster tag) that implicitly applies for all 
nodes, we can (but don't have to) support a special snowflake when one of 
the nodes is different.

As I understand from addBandwidthData how 'alltags' are merged, cluster 
tags are overridden by group tags and those are overridden by node tags 
(with the same prefix) ? Why is then the bandwidth graph calculated form 
cluster tags only (are the bandwidths specified always in cluster tags 
only, while the group/node tags are only for setting for the location) ? 
Could you please explain and document this model in more detail? I think 
best would be documenting on a few examples - how would the tags look like 
in case e.g.:

1) single nodegroup with 5 nodes, all bandwidths symmetric
2) 3 nodegroups, within each nodegroup symmetric bandwidths, between 
nodegroups different bandwidths

thanks,
viktor


 

>
> That's what I believe, yes. The concept of node groups was introduced do 
> model (approximately) high-bandwidth connectivity, with inter-node-group 
> bandwidth being potentially smaller/more contented, but still symmetrical. 
> I don't know of any deployments that had per-node different bandwidth, but 
> maybe some other people have such cases?
>
> CC-ing a couple of other people to see their input as well.
>
> thanks,
> iustin
>

Re: [PATCH master 2/6] Add bandwidth tags and bandwidth map fields into Node

['Iustin Pop' via ganeti-devel]

On 28 June 2016 at 16:16, Даниил Лещёв  wrote:
>
> I have to main concerns here.
>>
>> 1. I still believe it's wrong to model this on a per-node basis, and that
>> it should be rather two things: bandwidth available inside a node group
>> (between any two arbitrary nodes), and bandwidth available between any two
>> node groups. However, your badwidth tag concept is nice. Hmm…
>>
>
> But with such approach we lose an opportunity to specify network speed
> between two specific nodes.
> Or it is really rare case and we don't need such flexibility?
>

That's what I believe, yes. The concept of node groups was introduced do
model (approximately) high-bandwidth connectivity, with inter-node-group
bandwidth being potentially smaller/more contented, but still symmetrical.
I don't know of any deployments that had per-node different bandwidth, but
maybe some other people have such cases?

CC-ing a couple of other people to see their input as well.

thanks,
iustin

Re: [PATCH master 2/6] Add bandwidth tags and bandwidth map fields into Node

[Даниил Лещёв]

>
>
>
> I have to main concerns here.
>
> 1. I still believe it's wrong to model this on a per-node basis, and that 
> it should be rather two things: bandwidth available inside a node group 
> (between any two arbitrary nodes), and bandwidth available between any two 
> node groups. However, your badwidth tag concept is nice. Hmm…
>
 
But with such approach we lose an opportunity to specify network speed 
between two specific nodes.
Or it is really rare case and we don't need such flexibility?

>
> 2. Storing the information somewhere else then nodes would indeed be 
> useful.
>
> So let's discuss the main point: how do you or anybody else think that 
> bandwidth should be represented? Per node, per node group, etc.?
>
> thanks!
> iustin
>

Re: [PATCH master 2/6] Add bandwidth tags and bandwidth map fields into Node

['Iustin Pop' via ganeti-devel]

On 28 June 2016 at 11:28, Даниил Лещёв  wrote:
>
> > The bandwidths map store data about network speed
>> > between current node and given Node by it's bandwidth tags.
>>
>> Filling this up will take some space in a large cluster. Is it really
>> necessary to store this by nodes (which makes it O(n²) in terms of
>> nodes), or simply in terms of node group wide-bandwidth, plus
>> inter-node group bandwidth?
>>
>
> I think, I definitely said wrong.
> At the moment bandwidth map store information about network speed between
> current node and some bandwidth tag (not another node).
>

I see.


> So at the best case (all nodes in cluster have the same bandwidth tag) you
> store O(n) per cluster and in the worst case O(n^2) (all nodes have
> different bandwidth tags).
> If you only have group-wide bandwidth tags you will store O(n*g) per
> cluster, where n is the number of nodes and g is the number of groups (And
> I think it is the most common case).
>
> Should I use some single place for storing network speed information in
> order to avoid data duplication?
>

I have to main concerns here.

1. I still believe it's wrong to model this on a per-node basis, and that
it should be rather two things: bandwidth available inside a node group
(between any two arbitrary nodes), and bandwidth available between any two
node groups. However, your badwidth tag concept is nice. Hmm…

2. Storing the information somewhere else then nodes would indeed be useful.

So let's discuss the main point: how do you or anybody else think that
bandwidth should be represented? Per node, per node group, etc.?

thanks!
iustin

Re: [PATCH master 2/6] Add bandwidth tags and bandwidth map fields into Node

[Даниил Лещёв]

>
>
> > 
> > The bandwidths map store data about network speed 
> > between current node and given Node by it's bandwidth tags. 
>
> Filling this up will take some space in a large cluster. Is it really 
> necessary to store this by nodes (which makes it O(n²) in terms of 
> nodes), or simply in terms of node group wide-bandwidth, plus 
> inter-node group bandwidth? 
>

I think, I definitely said wrong. 
At the moment bandwidth map store information about network speed between 
current node and some bandwidth tag (not another node). 
So at the best case (all nodes in cluster have the same bandwidth tag) you 
store O(n) per cluster and in the worst case O(n^2) (all nodes have 
different bandwidth tags).
If you only have group-wide bandwidth tags you will store O(n*g) per 
cluster, where n is the number of nodes and g is the number of groups (And 
I think it is the most common case).

Should I use some single place for storing network speed information in 
order to avoid data duplication?

--
Cheers,
Daniil Leshchev

Re: [PATCH master 2/6] Add bandwidth tags and bandwidth map fields into Node

[Iustin Pop]

On 2016-06-23 17:45:34, mele...@gmail.com wrote:
> From: Daniil Leshchev 
> 
> The bandwidths map store data about network speed
> between current node and given Node by it's bandwidth tags.

Filling this up will take some space in a large cluster. Is it really
necessary to store this by nodes (which makes it O(n²) in terms of
nodes), or simply in terms of node group wide-bandwidth, plus 
inter-node group bandwidth?

thanks,
iustin

[PATCH master 2/6] Add bandwidth tags and bandwidth map fields into Node

[meleodr]

From: Daniil Leshchev 

The bandwidths map store data about network speed
between current node and given Node by it's bandwidth tags.
---
 src/Ganeti/HTools/Node.hs | 30 ++
 1 file changed, 30 insertions(+)

diff --git a/src/Ganeti/HTools/Node.hs b/src/Ganeti/HTools/Node.hs
index 6749568..4eb1576 100644
--- a/src/Ganeti/HTools/Node.hs
+++ b/src/Ganeti/HTools/Node.hs
@@ -60,6 +60,8 @@ module Ganeti.HTools.Node
   , setMigrationTags
   , setRecvMigrationTags
   , setLocationTags
+  , setBandwidthTags
+  , setBandwidthToLocation
   -- * Tag maps
   , addTags
   , delTags
@@ -98,6 +100,7 @@ module Ganeti.HTools.Node
   , mkNodeGraph
   , mkRebootNodeGraph
   , haveExclStorage
+  , calcBandwidthToNode
   ) where
 
 import Prelude ()
@@ -111,6 +114,7 @@ import qualified Data.IntMap as IntMap
 import Data.List (intercalate, foldl', delete, union, sortBy, groupBy)
 import qualified Data.Map as Map
 import Data.Ord (comparing)
+import Data.Maybe (mapMaybe)
 import qualified Data.Set as Set
 import Text.Printf (printf)
 
@@ -213,6 +217,10 @@ data Node = Node
-- to
   , locationScore :: Int -- ^ Sum of instance location and desired location
  -- scores
+  , bandwidthTags :: Set.Set String -- ^ Node's bandwidth tags
+  , bandwidthMap :: Map.Map String Int -- ^ Node's network bandwidth between
+-- current location and given location
+-- in Mbit per second
   , instanceMap :: Map.Map (String, String) Int -- ^ Number of instances with
 -- each exclusion/location tags
 -- pair
@@ -384,6 +392,8 @@ create name_init mem_t_init mem_n_init mem_f_init
, rmigTags = Set.empty
, locationTags = Set.empty
, locationScore = 0
+   , bandwidthTags = Set.empty
+   , bandwidthMap = Map.empty
, instanceMap = Map.empty
}
 
@@ -435,6 +445,15 @@ setRecvMigrationTags t val = t { rmigTags = val }
 setLocationTags :: Node -> Set.Set String -> Node
 setLocationTags t val = t { locationTags = val }
 
+-- | Set the network bandwidth tags
+setBandwidthTags :: Node -> Set.Set String -> Node
+setBandwidthTags t val = t { bandwidthTags = val }
+
+-- | Add network bandwidth to nodes with given bandwidth tag
+setBandwidthToLocation :: Node -> String -> Int -> Node
+setBandwidthToLocation t tag bandwidth = t { bandwidthMap = new_map }
+  where new_map = Map.insert tag bandwidth (bandwidthMap t)
+
 -- | Sets the unnaccounted memory.
 setXmem :: Node -> Int -> Node
 setXmem t val = t { xMem = val }
@@ -556,6 +575,17 @@ calcFmemOfflineOrForthcoming node allInstances =
  . filter (not . Instance.usesMemory)
  $ nodeInstances
 
+-- | Calculate the network bandwidth between two given nodes
+calcBandwidthToNode :: Node -> Node -> Maybe Int
+calcBandwidthToNode src dst =
+  case bndwths of
+[] -> Nothing
+_  -> Just $ minimum bndwths
+  where dstTags = Set.toList $ bandwidthTags dst
+srcMap = bandwidthMap src
+mapper = flip Map.lookup srcMap
+bndwths = mapMaybe mapper dstTags
+
 -- | Calculates the desired location score of an instance, given its primary
 -- node.
 getInstanceDsrdLocScore :: Node -- ^ the primary node of the instance
-- 
1.9.1