Re: [vdsm] [Engine-devel] CPU Overcommit Feature
- Original Message - > From: "Doron Fediuck" > To: "Dan Kenigsberg" > Cc: "engine-devel" , vdsm-de...@fedorahosted.org > Sent: Thursday, December 20, 2012 2:14:45 PM > Subject: Re: [Engine-devel] [vdsm] CPU Overcommit Feature > > > > - Original Message - > > From: "Dan Kenigsberg" > > To: "Itamar Heim" > > Cc: "engine-devel" , > > vdsm-de...@fedorahosted.org > > Sent: Thursday, December 20, 2012 11:55:10 AM > > Subject: Re: [Engine-devel] [vdsm] CPU Overcommit Feature > > > > On Thu, Dec 20, 2012 at 10:23:55AM +0200, Itamar Heim wrote: > > > On 12/20/2012 09:43 AM, Dan Kenigsberg wrote: > > > >On Wed, Dec 19, 2012 at 09:53:15AM -0500, Doron Fediuck wrote: > > > >> > > > >>- Original Message - > > > >>>From: "Dan Kenigsberg" > > > >>>To: "Greg Padgett" > > > >>>Cc: "engine-devel" , > > > >>>vdsm-de...@fedorahosted.org > > > >>>Sent: Wednesday, December 19, 2012 3:59:11 PM > > > >>>Subject: Re: [Engine-devel] CPU Overcommit Feature > > > >>> > > > >>>On Mon, Dec 17, 2012 at 09:37:57AM -0500, Greg Padgett wrote: > > > Hi, > > > > > > I've been working on a feature to allow CPU Overcommitment of > > > hosts > > > in a cluster. This first stage allows the engine to consider > > > host > > > cpu threads as cores for the purposes of VM resource > > > allocation. > > > > > > This wiki page has further details, your comments are > > > welcome! > > > http://www.ovirt.org/Features/cpu_overcommit > > > >>> > > > >>>I've commented about the vdsm/engine API on > > > >>>http://gerrit.ovirt.org/#/c/10144/ but it is probably better > > > >>>to > > > >>>reiterate it here. > > > >>> > > > >>>The suggested API is tightly coupled with an ugly hack we > > > >>>pushed > > > >>>to > > > >>>vdsm > > > >>>in order not to solve the issue properly on the first strike. > > > >>> > > > >>>If we had not have report_host_threads_as_cores, I think we'd > > > >>>have a > > > >>>simpler API reporting only cpuThreads and cpuCores; with no > > > >>>funny > > > >>>boolean flags. > > > >>> > > > >>>Let us strive to that position as much as we can. > > > >>> > > > >>>How about asking whoever used report_host_threads_as_cores to > > > >>>unset > > > >>>it > > > >>>once they install Engine 3.2 ? I think that these are very few > > > >>>people, > > > >>>that would not mind this very much. > > > >>> > > > >>>If this is impossible, I'd add a cpuCores2, always reporting > > > >>>the > > > >>>true > > > >>>number, to be used by new Engines. We may even report it only > > > >>>on > > > >>>the > > > >>>very few cases of report_host_threads_as_cores being set. > > > >>> > > > >>>Dan. > > > >> > > > >>Hi Dan, > > > >>Thanks for the review. > > > >> > > > >>I agree simply reporting cores and threads would be the right > > > >>solution. > > > >>However, when you have hyperthreading turned off you get > > > >>cores=threads. > > > >>This is the same situation you have when hyperthreading turned > > > >>on, and > > > >>someone used the vdsm configuration of reporting threads as > > > >>cores. > > > >> > > > >>So the engine won't know the real status of the host. > > > > > > > >This is not surprising, as report_host_threads_as_cores means in > > > >blunt > > > >English "lie to Engine about the number of cores". The newly > > > >suggested > > > >flag says "don't believe what I said in cpuCores, since I'm > > > >lying". Next > > > >thing we'd have is another flag saying that the former flag was > > > >a > > > >lie, > > > >and cpuCores is actually trustworthy. > > > > > > > >Instead of dancing this dance, I suggest to stop lying. > > > > > > > >report_host_threads_as_cores was a hack to assist a older Engine > > > >versions. Engine users that needed it had to set it out-of-band > > > >on > > > >their > > > >hosts. Now if they upgrade their Engine, they can -- as easily > > > >-- > > > >reset > > > >that value. > > > > > > > >If they forget, nothing devastating happens beyond Engine > > > >assuming > > > >that > > > >hyperthreading is off. > > > > > > > >Please consider this suggestion. I find it the simplest for all > > > >involved > > > >parties. > > > > > > the only problem is the new vdsm doesn't know which engine may be > > > using it. > > > if engine would say "getVdsCaps engineVersion=3.2", then vdsm > > > could > > > know engine no longer needs lying to and ignore the flag, > > > re-using > > > same field. > > > > Note that I do not suggest to drop report_host_threads_as_cores > > now. > > I am suggesting to keep on lying even to new Engine. > > If someone thinks that lying is bad, he should reset > > report_host_threads_as_cores. > > > > It seems to me that the suggested API is being coerced by a very > > limited > > use case, that is not going to be really harmed by a > > straight-forward > > API. > > > > Dan. > > Dan, > Did some further checking, and we can go with it; > So basically now we add cpuThreads. Additionally, if the > report_host_threads_as_cores > is
Re: [vdsm] Back to future of vdsm network configuration
- Original Message - > From: "Itamar Heim" > To: "Dan Kenigsberg" > Cc: "Alon Bar-Lev" , "VDSM Project Development" > , "Simon > Grinberg" , "Andrew Cathrow" > Sent: Monday, December 3, 2012 10:56:53 PM > Subject: Re: [vdsm] Back to future of vdsm network configuration > > On 12/03/2012 06:54 PM, Dan Kenigsberg wrote: > > On Mon, Dec 03, 2012 at 04:28:16PM +0200, Itamar Heim wrote: > >> On 12/03/2012 04:25 PM, Dan Kenigsberg wrote: > >>> On Mon, Dec 03, 2012 at 04:35:34AM -0500, Alon Bar-Lev wrote: > >>>> > >>>> > >>>> - Original Message - > >>>>> From: "Mark Wu" > >>>>> To: "VDSM Project Development" > >>>>> > >>>>> Cc: "Alon Bar-Lev" , "Dan Kenigsberg" > >>>>> , "Simon Grinberg" , > >>>>> "Antoni Segura Puimedon" , "Igor Lvovsky" > >>>>> , "Daniel P. Berrange" > >>>>> > >>>>> Sent: Monday, December 3, 2012 7:39:49 AM > >>>>> Subject: Re: [vdsm] Back to future of vdsm network > >>>>> configuration > >>>>> > >>>>> On 11/29/2012 04:24 AM, Alon Bar-Lev wrote: > >>>>>> > >>>>>> - Original Message - > >>>>>>> From: "Dan Kenigsberg" > >>>>>>> To: "Alon Bar-Lev" > >>>>>>> Cc: "Simon Grinberg" , "VDSM Project > >>>>>>> Development" > >>>>>>> Sent: Wednesday, November 28, 2012 10:20:11 PM > >>>>>>> Subject: Re: [vdsm] MTU setting according to ifcfg files. > >>>>>>> > >>>>>>> On Wed, Nov 28, 2012 at 12:49:10PM -0500, Alon Bar-Lev wrote: > >>>>>>>> Itamar though a bomb that we should co-exist on generic > >>>>>>>> host, > >>>>>>>> this > >>>>>>>> is > >>>>>>>> something I do not know to compute. I still waiting for a > >>>>>>>> response > >>>>>>>> of > >>>>>>>> where this requirement came from and if that mandatory. > >>>>>>>> > >>>>>>> This bomb has been ticking since ever. We have ovirt-node > >>>>>>> images > >>>>>>> for > >>>>>>> pure hypervisor nodes, but we support plain Linux nodes, > >>>>>>> where > >>>>>>> local > >>>>>>> admins are free to `yum upgrade` in the least convenient > >>>>>>> moment. > >>>>>>> The > >>>>>>> latter mode can be the stuff that nightmares are made of, but > >>>>>>> it > >>>>>>> also > >>>>>>> allows the flexibility and bleeding-endgeness we all cherish. > >>>>>>> > >>>>>> There is a different between having generic OS and having > >>>>>> generic > >>>>>> setup, running your email server, file server and LDAP on a > >>>>>> node > >>>>>> that running VMs. > >>>>>> > >>>>>> I have no problem in having generic OS (opposed of ovirt-node) > >>>>>> but > >>>>>> have full control over that. > >>>>>> > >>>>>> Alon. > >>>>> Can I say we have got agreement on oVirt should cover two kinds > >>>>> of > >>>>> hypervisors? Stateless slave is good for pure and normal > >>>>> virtualization > >>>>> workload, while generic host can keep the flexibility of > >>>>> customization. > >>>>> In my opinion, it's good for the oVirt community to provide > >>>>> choices > >>>>> for > >>>>> users. They could customize it in production, building and > >>>>> even > >>>>> source > >>>>> code according to their requirements and skills. > >>>> > >>>> I also think it will be good to support both modes! It will also > >>>> good if we can rule the world! :) > >>&g
Re: [vdsm] MTU setting according to ifcfg files.
- Original Message - > From: "Roni Luxenberg" > To: "Alon Bar-Lev" > Cc: "Simon Grinberg" , "VDSM Project Development" > > Sent: Thursday, November 29, 2012 5:13:04 PM > Subject: Re: [vdsm] MTU setting according to ifcfg files. > > > - Original Message - > > > From: "Simon Grinberg" > > > To: "Alon Bar-Lev" > > > Cc: "VDSM Project Development" > > > > > > Sent: Thursday, November 29, 2012 2:35:46 PM > > > Subject: Re: [vdsm] MTU setting according to ifcfg files. > > > > > > > > > > > > - Original Message - > > > > From: "Alon Bar-Lev" > > > > To: "Simon Grinberg" > > > > Cc: "VDSM Project Development" > > > > > > > > Sent: Thursday, November 29, 2012 2:25:03 PM > > > > Subject: Re: [vdsm] MTU setting according to ifcfg files. > > > > > > > > > > > > > > > > - Original Message - > > > > > From: "Simon Grinberg" > > > > > To: "Itamar Heim" > > > > > Cc: "Alon Bar-Lev" , "VDSM Project > > > > > Development" > > > > > , "Andrew > > > > > Cathrow" , "Saggi Mizrahi" > > > > > > > > > > Sent: Thursday, November 29, 2012 2:12:09 PM > > > > > Subject: Re: [vdsm] MTU setting according to ifcfg files. > > > > > > > > > > > > > > > > > > > > - Original Message - > > > > > > From: "Itamar Heim" > > > > > > To: "Saggi Mizrahi" > > > > > > Cc: "Alon Bar-Lev" , "Simon Grinberg" > > > > > > , "VDSM Project Development" > > > > > > , "Andrew Cathrow" > > > > > > > > > > > > Sent: Thursday, November 29, 2012 1:06:29 PM > > > > > > Subject: Re: [vdsm] MTU setting according to ifcfg files. > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > >> Assuming manual changes and distro specific persistence > > > > > > >> make > > > > > > >> the > > > > > > >> problem complex in factor of np complete, as we do not > > > > > > >> know > > > > > > >> what > > > > > > >> was > > > > > > >> changed when and how to revert. > > > > > > >> > > > > > > >> Itamar though a bomb that we should co-exist on generic > > > > > > >> host, > > > > > > >> this > > > > > > >> is > > > > > > >> something I do not know to compute. I still waiting for > > > > > > >> a > > > > > > >> response > > > > > > >> of where this requirement came from and if that > > > > > > >> mandatory. > > > > > > > > > > Just few reasons: > > > > > - One of the key attraction with KVM is that with it, you are > > > > > capable > > > > > to run process/application along side virtual machines. Look > > > > > at > > > > > every KVM presentation out there. > > > > > - Licencing and support, some application (do I hear Oracle?) > > > > > are > > > > > not > > > > > licensed/supported on KVM, but you would still want to use > > > > > free > > > > > cycles for virtual machines (especially on modern servers) > > > > > - 3rd party monitoring and audit tools > > > > > - custom drivers > > > > > - custom SLA policies > > > > > - etc, > > > > > - etc, > > > > > - etc, > > > > > > > > > > You don't want to say, ha if you use VDSM to manage the node > > > > > you > > > > > can't do all of the above. > > > > > > > > Actually, I am. > > > > I claim that we will never be able to stabilize a product if we > > > > go > > > > this way. > > > > There is a very good reason why other virtualization solutions >
Re: [vdsm] MTU setting according to ifcfg files.
- Original Message - > From: "Alon Bar-Lev" > To: "Simon Grinberg" > Cc: "VDSM Project Development" > Sent: Thursday, November 29, 2012 2:25:03 PM > Subject: Re: [vdsm] MTU setting according to ifcfg files. > > > > - Original Message - > > From: "Simon Grinberg" > > To: "Itamar Heim" > > Cc: "Alon Bar-Lev" , "VDSM Project Development" > > , "Andrew > > Cathrow" , "Saggi Mizrahi" > > > > Sent: Thursday, November 29, 2012 2:12:09 PM > > Subject: Re: [vdsm] MTU setting according to ifcfg files. > > > > > > > > - Original Message - > > > From: "Itamar Heim" > > > To: "Saggi Mizrahi" > > > Cc: "Alon Bar-Lev" , "Simon Grinberg" > > > , "VDSM Project Development" > > > , "Andrew Cathrow" > > > > > > Sent: Thursday, November 29, 2012 1:06:29 PM > > > Subject: Re: [vdsm] MTU setting according to ifcfg files. > > > > > > > > > >> > > > >> Assuming manual changes and distro specific persistence make > > > >> the > > > >> problem complex in factor of np complete, as we do not know > > > >> what > > > >> was > > > >> changed when and how to revert. > > > >> > > > >> Itamar though a bomb that we should co-exist on generic host, > > > >> this > > > >> is > > > >> something I do not know to compute. I still waiting for a > > > >> response > > > >> of where this requirement came from and if that mandatory. > > > > Just few reasons: > > - One of the key attraction with KVM is that with it, you are > > capable > > to run process/application along side virtual machines. Look at > > every KVM presentation out there. > > - Licencing and support, some application (do I hear Oracle?) are > > not > > licensed/supported on KVM, but you would still want to use free > > cycles for virtual machines (especially on modern servers) > > - 3rd party monitoring and audit tools > > - custom drivers > > - custom SLA policies > > - etc, > > - etc, > > - etc, > > > > You don't want to say, ha if you use VDSM to manage the node you > > can't do all of the above. > > Actually, I am. > I claim that we will never be able to stabilize a product if we go > this way. > There is a very good reason why other virtualization solutions out > there put similar restriction. > > When and if we finish with rock solid solution using a pure > completely managed slave and have good market share then we can > start thinking about these non deterministic approaches. Actually it's the other way around. Since you are far from there, then many (if not most) users today actually use a full blown host to complement features or required functionality like: Monitoring, Private firewall, central logging, customization for third party devices etc. > Or... maybe this is the marketing advantage we would like, and then > we should FOCUS on this approach, but then we are aiming to low > scale, manual managed solution, and the "other" open source project > will probably consume the higher scale. > > As I wrote there are two solution using CURRENT technology for that: > 1. Move the original host into virtual machine and manage the host as > a whole. > 2. Execute virtual machine with nested virtualization and manage this > VM as if it was our host, in this mode we have no conflict. > > > Stateless by the way, in a sense that after reboot the node goes > > back > > to the original configuration, works very well with the requirement > > above. This means that the admin sets everything required for the > > non virtualized hardware, VDSM configures on top, but after reboot > > all is reverted to the original thus everything else continues to > > work after reboot. > > This is not the way to go in this case, Oracle will not live within > stateless world, nor 1000 other solutions. You missed what I've said: Admin configures state-fully everything required for the 'native' application, VDSM may configure starless on top. After reboot, host goes back to the original configuration that is enough to run the 'native' non managed by VDSM applications. > > Regards, > Alon. > ___ > vdsm-devel mailing list > vdsm-devel@lists.fedorahosted.org > https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel > ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] MTU setting according to ifcfg files.
- Original Message - > From: "Itamar Heim" > To: "Saggi Mizrahi" > Cc: "Alon Bar-Lev" , "Simon Grinberg" , > "VDSM Project Development" > , "Andrew Cathrow" > Sent: Thursday, November 29, 2012 1:06:29 PM > Subject: Re: [vdsm] MTU setting according to ifcfg files. > > On 11/28/2012 01:20 PM, Saggi Mizrahi wrote: > ... > >>> VDSM should not bother with the issue at all, certainly not > >>> playing > >>> a > >>> guessing game. > >>> > >>> Livant, your 0.02$? > >> > >> This exactly the reason why we should either define completely > >> stateless slave host, and apply configuration including what you > >> call 'defaults'. > > Completely stateless is problematic because if the engine is down > > or unavailable and VDSM happens to restart you can't use any of > > your resources. > > that's actually a very good point. going forward we would like to be > able for hosts to continue working when engine is down, even post > reboot. How?, Will you really fire up VMs without central management control? This implies you'll have to go into host based clustering where you'll hit scale limits as any other such a solution. If you do not intend to do the above then why not stateless? Host to remember on wakeup an old configuration may at best not work but at worst may conflict with existing configuration and do unpredictable things to your environment. You also loose the benefit of recovering bad configured host simply by fencing it. > engine passing the policy to the hosts and hosts assuming > that > policy is relevant post boot would allow that. > (though relying on central network services like qunatum will also > cause > an issue for this architecture). > > > > > The way forward is currently to get rid of most of the > > configuration in vdsm.conf. > > Only have things that are necessary for communication with the > > engine (eg. Core dump on\off, management interface\port, SSL > > on\off). > > Other VDSM configuration should have a an API introduced to set > > them and that will be persisted but only configurable by > > management (eg. reserved host mem, guest ram overhead, migration > > timeouts). > > There should be a place where VDSM saves the configuration of owned > > resources (eg. managed storage connections, managed interfaces). > > This will be use by VDSM to make sure that the resources are > > configured properly after restarts\downtimes without the need of > > the engine. > > > > To reiterate, the general logic for system resources should be that > > resources are either owned or used by VDSM, you never share > > ownership. > > Never assume ownership unless expressly given. VDSM has complete > > control over owned resources. VDSM has NO control over unowned > > resources, he can use them but never configure them. > > > > Every other hybrid scheme is just asking for trouble. > > > >> > >> Or, store configuration before we perform any change so we can > >> revert. > >> > >> Assuming manual changes and distro specific persistence make the > >> problem complex in factor of np complete, as we do not know what > >> was > >> changed when and how to revert. > >> > >> Itamar though a bomb that we should co-exist on generic host, this > >> is > >> something I do not know to compute. I still waiting for a response > >> of where this requirement came from and if that mandatory. Just few reasons: - One of the key attraction with KVM is that with it, you are capable to run process/application along side virtual machines. Look at every KVM presentation out there. - Licencing and support, some application (do I hear Oracle?) are not licensed/supported on KVM, but you would still want to use free cycles for virtual machines (especially on modern servers) - 3rd party monitoring and audit tools - custom drivers - custom SLA policies - etc, - etc, - etc, You don't want to say, ha if you use VDSM to manage the node you can't do all of the above. Stateless by the way, in a sense that after reboot the node goes back to the original configuration, works very well with the requirement above. This means that the admin sets everything required for the non virtualized hardware, VDSM configures on top, but after reboot all is reverted to the original thus everything else continues to work after reboot. > > It's all about resource provisioning and ownership delegation. > > hybrid mode is something brought up several times as a use case we > should consider. so far our main concern was that SLA in the host > would > be needed (cgroup for example) between the native and guest > workloads. > as well as making sure hybrid nodes will not contend for critical > resources to reduce the risk of a need to fence them. > ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] MTU setting according to ifcfg files.
- Original Message - > From: "Saggi Mizrahi" > To: "Simon Grinberg" > Cc: "VDSM Project Development" > Sent: Wednesday, November 28, 2012 7:15:35 PM > Subject: Re: [vdsm] MTU setting according to ifcfg files. > > > > - Original Message - > > From: "Simon Grinberg" > > To: "Saggi Mizrahi" > > Cc: "VDSM Project Development" , > > "Barak Azulay" , "Igor > > Lvovsky" > > Sent: Wednesday, November 28, 2012 12:03:03 PM > > Subject: Re: [vdsm] MTU setting according to ifcfg files. > > > > > > > > - Original Message - > > > From: "Saggi Mizrahi" > > > To: "Igor Lvovsky" > > > Cc: "VDSM Project Development" > > > , > > > "Simon Grinberg" , "Barak > > > Azulay" > > > Sent: Wednesday, November 28, 2012 6:49:22 PM > > > Subject: Re: [vdsm] MTU setting according to ifcfg files. > > > > > > OK, I think I need to explain myself better, > > > MTU sizes under 1500 are not interesting as they are only really > > > valid for slow networks which will not be able to support virt > > > workloads anyway. > > > 1500 is "internet MTU" and is the recommended size when > > > communicating > > > with the outside world. > > > > > > MTU is just a size that has to be agreed upon by all participants > > > in > > > the chain. > > > There is no inherent default MTU but default is technically 1500. > > > > > > Reverting to previous value makes no sense unless you are just > > > testing something out. > > > > Yes it does, > > There are networks out there that do use MTU > 1500 as weird as it > > sounds, > It not weird at all, this is why MTU settings exist. > But setting a low MTU will not break the network but will just have > some performance degredation. > > this usually the admin does initial settings on the > > management network and then when you set don't touch all works > > well. > > An example is when you have storage and management on the same > > network. > > > > Now consider the scenario that for some VMs the user wants to limit > > to the 'normal/recommended defaults' so in this case he will have > > to > > set in the logical network property to MTU=1500. when VDSM sets > > this > > chain it supposedly won't touch the interface MTU since it's > > already > > bigger (if it does it's a bug). Now the user has one more logical > > network of VMs with 9000 since he also have VMs using shared > > storage > > on this network. > > > > All works well till now. > > > > But what about when removing the 9000 network? > > Will VDSM 'remember' that it did not touch the interface MTU in the > > first place, or will it try to set it to this recommended MTU?. > It's a question of ownership. Because it's simpler I suggest we > assume ownership and always set the maximum needed (also lowering if > to high). > The engine can query the MTU and make weird decision according. Like > setting the current as default or as a saved value or whatever. > This flow obviously needs user input so VSDM is not the place to put > the decision making. I tend to agree, it's an ownership thing Engine should not allow mixed configuration of 'default vs override' on the same interface. If user wishes to start playing with MTUs he needs to use it carefully and across the board. VDSM should not bother with the issue at all, certainly not playing a guessing game. Livant, your 0.02$? > > > > I have no idea :) > > > > > > > For that case the engine can remember the current MTU and set it > > > back. > > > > > > To sum up, I suggest ignoring any previously set value like we > > > would > > > ignore it if VDSM had set it. > > > It makes no sense to keep it because the semantic of setting the > > > MTU > > > is to override the current configuration. > > > > > > As a side note, having verb to test max MTU for a path might be a > > > good idea to give the engine\user a way to recommend a value to > > > the > > > user. > > > > That is better but not perfect :) > > > > > > > > > > - Original Message - > > > > From: "Saggi Mizrahi" > > > > To: "Igor Lvovsky" > >
Re: [vdsm] MTU setting according to ifcfg files.
- Original Message - > From: "Saggi Mizrahi" > To: "Igor Lvovsky" > Cc: "VDSM Project Development" , "Simon > Grinberg" , "Barak > Azulay" > Sent: Wednesday, November 28, 2012 6:49:22 PM > Subject: Re: [vdsm] MTU setting according to ifcfg files. > > OK, I think I need to explain myself better, > MTU sizes under 1500 are not interesting as they are only really > valid for slow networks which will not be able to support virt > workloads anyway. > 1500 is "internet MTU" and is the recommended size when communicating > with the outside world. > > MTU is just a size that has to be agreed upon by all participants in > the chain. > There is no inherent default MTU but default is technically 1500. > > Reverting to previous value makes no sense unless you are just > testing something out. Yes it does, There are networks out there that do use MTU > 1500 as weird as it sounds, this usually the admin does initial settings on the management network and then when you set don't touch all works well. An example is when you have storage and management on the same network. Now consider the scenario that for some VMs the user wants to limit to the 'normal/recommended defaults' so in this case he will have to set in the logical network property to MTU=1500. when VDSM sets this chain it supposedly won't touch the interface MTU since it's already bigger (if it does it's a bug). Now the user has one more logical network of VMs with 9000 since he also have VMs using shared storage on this network. All works well till now. But what about when removing the 9000 network? Will VDSM 'remember' that it did not touch the interface MTU in the first place, or will it try to set it to this recommended MTU?. I have no idea :) > For that case the engine can remember the current MTU and set it > back. > > To sum up, I suggest ignoring any previously set value like we would > ignore it if VDSM had set it. > It makes no sense to keep it because the semantic of setting the MTU > is to override the current configuration. > > As a side note, having verb to test max MTU for a path might be a > good idea to give the engine\user a way to recommend a value to the > user. That is better but not perfect :) > > - Original Message - > > From: "Saggi Mizrahi" > > To: "Igor Lvovsky" > > Cc: "VDSM Project Development" , > > "Simon Grinberg" > > Sent: Wednesday, November 28, 2012 11:23:52 AM > > Subject: Re: [vdsm] MTU setting according to ifcfg files. > > > > I don't want to keep the last configured MTU. It's problematic. > > Having a stack is even worse. > > VDSM should try not to persist anything if possible. > > > > Also, reverting to the last MTU is raceful and has weird corner > > cases. Best to just assume default it 1500 (Like all major OSs do). > > But since it's not really a default I would call it a recommended > > setting. > > > > - Original Message - > > > From: "Igor Lvovsky" > > > To: "Saggi Mizrahi" > > > Cc: "VDSM Project Development" > > > , > > > "Simon Grinberg" > > > Sent: Wednesday, November 28, 2012 11:10:27 AM > > > Subject: Re: [vdsm] MTU setting according to ifcfg files. > > > > > > > > > > > > - Original Message - > > > > From: "Saggi Mizrahi" > > > > To: "Simon Grinberg" > > > > Cc: "VDSM Project Development" > > > > , > > > > "Igor Lvovsky" > > > > Sent: Wednesday, November 28, 2012 5:30:17 PM > > > > Subject: Re: [vdsm] MTU setting according to ifcfg files. > > > > > > > > I suggest we don't have a default. If you don't specify an MTU > > > > it > > > > will use whatever is already configured. > > > > There is no way to "go back to the defaults" only to set a new > > > > value. > > > > The engine can assume 1500 (in case of ethernet devices) is the > > > > "recommended value". > > > > > > > > > > This is not related to engine. You are right that the actually > > > MTU > > > will the last configured one, > > > but this is exactly a problem. > > > As I already mentioned, if you will add another bridge without > > > custom > > > MTU its users (VMs) > > > can assume that the MTU is 1500 > > > > > > &g
Re: [vdsm] MTU setting according to ifcfg files.
- Original Message - > From: "Igor Lvovsky" > To: "Saggi Mizrahi" > Cc: "Simon Grinberg" , "VDSM Project Development" > > Sent: Wednesday, November 28, 2012 6:10:27 PM > Subject: Re: [vdsm] MTU setting according to ifcfg files. > > > > - Original Message - > > From: "Saggi Mizrahi" > > To: "Simon Grinberg" > > Cc: "VDSM Project Development" , > > "Igor Lvovsky" > > Sent: Wednesday, November 28, 2012 5:30:17 PM > > Subject: Re: [vdsm] MTU setting according to ifcfg files. > > > > I suggest we don't have a default. If you don't specify an MTU it > > will use whatever is already configured. > > There is no way to "go back to the defaults" only to set a new > > value. > > The engine can assume 1500 (in case of ethernet devices) is the > > "recommended value". > > > > This is not related to engine. You are right that the actually MTU > will the last configured one, > but this is exactly a problem. > As I already mentioned, if you will add another bridge without custom > MTU its users (VMs) > can assume that the MTU is 1500 Assumption is the mother of all . What needs to be done is reverting to the old value. Can be done easily by inserting a comment in the ifcfg-file with the MTU prior to the change. When we (hopefully) go into a stateless configuration controlled by the engine/any other manager then it should be determined solely by the manager, and reverted to user defined on reboot. > > > - Original Message - > > > From: "Simon Grinberg" > > > To: "Igor Lvovsky" > > > Cc: "VDSM Project Development" > > > > > > Sent: Wednesday, November 28, 2012 9:53:48 AM > > > Subject: Re: [vdsm] MTU setting according to ifcfg files. > > > > > > > > > > > > - Original Message - > > > > From: "Igor Lvovsky" > > > > To: "VDSM Project Development" > > > > > > > > Cc: "Simon Grinberg" > > > > Sent: Wednesday, November 28, 2012 2:58:52 PM > > > > Subject: [vdsm] MTU setting according to ifcfg files. > > > > > > > > Hi, > > > > > > > > I am working on one of the vdsm bugs that we have and I found > > > > that > > > > initscripts (initscripts-9.03.34-1.el6.x86_64) > > > > behaviour doesn't fits our needs. > > > > So, I would like to raise this issue in the list. > > > > > > > > The issue is MTU setting according to ifcfg files. > > > > I'll try to describe the flow below. > > > > > > > > 1. I started with ifcfg file for the interface without MTU > > > > keyword > > > > at > > > > all > > > > and the proper interface (let say eth0) had the *default* > > > > MTU=1500 > > > > (according to /sys/class/net/eth0/mtu). > > > > 2. I created a bridge with MTU=9000 on top of this interface. > > > > Everything went OK. > > > >After I wrote MTU=9000 on ifcfg-eth0 and ifdown/ifup it, > > > >eth0 > > > >got > > > >the proper MTU. > > > > 3. Now, I removed the bridge and deleted MTU keyword from the > > > > ifcfg-eth0. > > > >But after ifup/ifdown the actual MTU of the eth0 stayed > > > >9000. > > > > > > > > The only way to change it back to 1500 (or something else) is > > > > explicitly set MTU in ifcfg file. > > > > According to Bill Nottingham it is intentional behaviour. > > > > If so, we have a problem in vdsm, because we never set MTU > > > > value > > > > until user ask it explicitly. > > > > > > Actually you are, > > > > > > You where asked for MTU 9000 on the network, > > > As implementation specif you had to do this all the way down the > > > chain > > > Now it's only reasonable that when you cancel the 9000 request > > > then > > > you'll do what is necessary to rollback the changes. > > > It's pity that ifcfg-files don't have the option to set > > > MTU='default', but as you can read this default before you > > > change, > > > then please keep it somewhere and revert to that. > > > > > > > > > > It means that
Re: [vdsm] MTU setting according to ifcfg files.
- Original Message - > From: "Saggi Mizrahi" > To: "Simon Grinberg" > Cc: "VDSM Project Development" , "Igor > Lvovsky" > Sent: Wednesday, November 28, 2012 5:30:17 PM > Subject: Re: [vdsm] MTU setting according to ifcfg files. > > I suggest we don't have a default. If you don't specify an MTU it > will use whatever is already configured. > There is no way to "go back to the defaults" only to set a new value. > The engine can assume 1500 (in case of ethernet devices) is the > "recommended value". I understand, This is why I've suggested to keep the old value and revert to that. Igor, alternately you may always calculate based on the hierarchy leafs, meaning the 'trunk' interface always needs to be set to the maximal MTU required by any of the logical networks, and it needs to be recalculated every time you change something in the hierarchy The problem is what happens if all are removed and then another is configured with MTU set to not override, here you may need to use the saved one. > > - Original Message - > > From: "Simon Grinberg" > > To: "Igor Lvovsky" > > Cc: "VDSM Project Development" > > Sent: Wednesday, November 28, 2012 9:53:48 AM > > Subject: Re: [vdsm] MTU setting according to ifcfg files. > > > > > > > > - Original Message - > > > From: "Igor Lvovsky" > > > To: "VDSM Project Development" > > > > > > Cc: "Simon Grinberg" > > > Sent: Wednesday, November 28, 2012 2:58:52 PM > > > Subject: [vdsm] MTU setting according to ifcfg files. > > > > > > Hi, > > > > > > I am working on one of the vdsm bugs that we have and I found > > > that > > > initscripts (initscripts-9.03.34-1.el6.x86_64) > > > behaviour doesn't fits our needs. > > > So, I would like to raise this issue in the list. > > > > > > The issue is MTU setting according to ifcfg files. > > > I'll try to describe the flow below. > > > > > > 1. I started with ifcfg file for the interface without MTU > > > keyword > > > at > > > all > > > and the proper interface (let say eth0) had the *default* > > > MTU=1500 > > > (according to /sys/class/net/eth0/mtu). > > > 2. I created a bridge with MTU=9000 on top of this interface. > > > Everything went OK. > > >After I wrote MTU=9000 on ifcfg-eth0 and ifdown/ifup it, eth0 > > >got > > >the proper MTU. > > > 3. Now, I removed the bridge and deleted MTU keyword from the > > > ifcfg-eth0. > > >But after ifup/ifdown the actual MTU of the eth0 stayed 9000. > > > > > > The only way to change it back to 1500 (or something else) is > > > explicitly set MTU in ifcfg file. > > > According to Bill Nottingham it is intentional behaviour. > > > If so, we have a problem in vdsm, because we never set MTU value > > > until user ask it explicitly. > > > > Actually you are, > > > > You where asked for MTU 9000 on the network, > > As implementation specif you had to do this all the way down the > > chain > > Now it's only reasonable that when you cancel the 9000 request then > > you'll do what is necessary to rollback the changes. > > It's pity that ifcfg-files don't have the option to set > > MTU='default', but as you can read this default before you change, > > then please keep it somewhere and revert to that. > > > > > > > It means that if we have interface with MTU=9000 on it just > > > because > > > once there was a bridge with such MTU > > > attached to it and now we want to attach regular bridge with > > > *default* MTU=1500 we have a problem. > > > The only thing we can do to avoid this it's set explicitly > > > MTU=1500 > > > in interface's ifcfg file. > > > IMHO it's a bit ugly, but it looks like we have no choice. > > > > > > As usual comments more than welcome... > > > > > > Regards, > > >Igor Lvovsky > > > ___ > > > vdsm-devel mailing list > > > vdsm-devel@lists.fedorahosted.org > > > https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel > > > > > ___ > > vdsm-devel mailing list > > vdsm-devel@lists.fedorahosted.org > > https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel > > > ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] MTU setting according to ifcfg files.
- Original Message - > From: "Igor Lvovsky" > To: "VDSM Project Development" > Cc: "Simon Grinberg" > Sent: Wednesday, November 28, 2012 2:58:52 PM > Subject: [vdsm] MTU setting according to ifcfg files. > > Hi, > > I am working on one of the vdsm bugs that we have and I found that > initscripts (initscripts-9.03.34-1.el6.x86_64) > behaviour doesn't fits our needs. > So, I would like to raise this issue in the list. > > The issue is MTU setting according to ifcfg files. > I'll try to describe the flow below. > > 1. I started with ifcfg file for the interface without MTU keyword at > all > and the proper interface (let say eth0) had the *default* MTU=1500 > (according to /sys/class/net/eth0/mtu). > 2. I created a bridge with MTU=9000 on top of this interface. > Everything went OK. >After I wrote MTU=9000 on ifcfg-eth0 and ifdown/ifup it, eth0 got >the proper MTU. > 3. Now, I removed the bridge and deleted MTU keyword from the > ifcfg-eth0. >But after ifup/ifdown the actual MTU of the eth0 stayed 9000. > > The only way to change it back to 1500 (or something else) is > explicitly set MTU in ifcfg file. > According to Bill Nottingham it is intentional behaviour. > If so, we have a problem in vdsm, because we never set MTU value > until user ask it explicitly. Actually you are, You where asked for MTU 9000 on the network, As implementation specif you had to do this all the way down the chain Now it's only reasonable that when you cancel the 9000 request then you'll do what is necessary to rollback the changes. It's pity that ifcfg-files don't have the option to set MTU='default', but as you can read this default before you change, then please keep it somewhere and revert to that. > It means that if we have interface with MTU=9000 on it just because > once there was a bridge with such MTU > attached to it and now we want to attach regular bridge with > *default* MTU=1500 we have a problem. > The only thing we can do to avoid this it's set explicitly MTU=1500 > in interface's ifcfg file. > IMHO it's a bit ugly, but it looks like we have no choice. > > As usual comments more than welcome... > > Regards, >Igor Lvovsky > ___ > vdsm-devel mailing list > vdsm-devel@lists.fedorahosted.org > https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel > ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] Future of Vdsm network configuration
- Original Message - > From: "Alon Bar-Lev" > To: "Simon Grinberg" > Cc: vdsm-de...@fedorahosted.org, "Mark Wu" > Sent: Tuesday, November 13, 2012 10:13:41 AM > Subject: Re: [vdsm] Future of Vdsm network configuration > > > > - Original Message - > > From: "Simon Grinberg" > > To: "Mark Wu" > > Cc: vdsm-de...@fedorahosted.org, "Alon Bar-Lev" > > Sent: Tuesday, November 13, 2012 10:06:42 AM > > Subject: Re: [vdsm] Future of Vdsm network configuration > > > > > > > > Hypervisor should be a total slave of manager (or cluster), so > > > > I > > > > have > > > > no problem in bypassing/disabling any distribution specific > > > > tool > > > > in > > > > favour of atoms (brctl, iproute), in non persistence mode. > > > > > > > Do you mean just using the utilities (brctl, iproute) on demand > > > and > > > not > > > keeping any network configuration > > > on vdsm host? Then manager needs reconfigure network on every > > > host > > > reboot. Actually, I like this way. > > > It could be more flexible than libvirt's virInterface (netcf or > > > NM) > > > and have fine-grained control to handle some tough cases. > > > Moreover, > > > it's clean than the current mangling network configuration files. > > > > +1, > > I've raised that is the past, I don't think the network > > configuration > > done by the engine should be persisted. This way the admin sets up > > the node in a persistent way such that it always succeeds to boot > > and has a rout to the engine. Engine on node activation updates the > > network, connects to storage etc. > > > > Now that setupNetworks can do it in one atomic operation this is > > the > > way to go, very simple. > > It's also eases the move of a node from a cluster to cluster. With > > current concept, after you move the host you need to modify the > > node > > networks to fit the new cluster topology. With non-persistent, > > placing a host into maintenance should also revert the host to the > > original networking after boot same as it disconnects the storage. > > Now you can easily move the node from cluster to cluster or even a > > differed DC. As soon as you activate - it is configured with the > > new > > DC/Cluster pair requirements. > > > > And it's a valuable step in the direction of -> Go go dynamic host > > allocation :) > > > > So I am not the only insane one where! > Good to know! Hold your horses, all I've said is that I strongly agree that networks should be dynamically set from the engine, I did not comment on how. If there is a cross distribution utility out there that can do this in reliable <\bold> manner and actually offloads <\bold> logic from VDSM, meaning it's simpler then direct usage of mkdev, ip, brctl, etc to configure the hosts networking, it should be considered. What's important is the goal and not the way there. One of my goals, is returning to the stateless node concept the ovirt node had started with so many, many years (5?) ago. It just makes sense. BTU, I've always been insane, they just didn't caught up with me yet. > > Alon > ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] Future of Vdsm network configuration
> > Hypervisor should be a total slave of manager (or cluster), so I have > > no problem in bypassing/disabling any distribution specific tool in > > favour of atoms (brctl, iproute), in non persistence mode. > Do you mean just using the utilities (brctl, iproute) on demand and not > keeping any network configuration > on vdsm host? Then manager needs reconfigure network on every host > reboot. Actually, I like this way. > It could be more flexible than libvirt's virInterface (netcf or NM) > and have fine-grained control to handle some tough cases. Moreover, > it's clean than the current mangling network configuration files. +1, I've raised that is the past, I don't think the network configuration done by the engine should be persisted. This way the admin sets up the node in a persistent way such that it always succeeds to boot and has a rout to the engine. Engine on node activation updates the network, connects to storage etc. Now that setupNetworks can do it in one atomic operation this is the way to go, very simple. It's also eases the move of a node from a cluster to cluster. With current concept, after you move the host you need to modify the node networks to fit the new cluster topology. With non-persistent, placing a host into maintenance should also revert the host to the original networking after boot same as it disconnects the storage. Now you can easily move the node from cluster to cluster or even a differed DC. As soon as you activate - it is configured with the new DC/Cluster pair requirements. And it's a valuable step in the direction of -> Go go dynamic host allocation :) ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] Future of Vdsm network configuration
- Original Message - > From: "Dan Kenigsberg" > To: vdsm-de...@fedorahosted.org > Sent: Sunday, November 11, 2012 4:07:30 PM > Subject: [vdsm] Future of Vdsm network configuration > > Hi, > > Nowadays, when vdsm receives the setupNetowrk verb, it mangles > /etc/sysconfig/network-scripts/ifcfg-* files and restarts the network > service, so they are read by the responsible SysV service. > > This is very much Fedora-oriented, and not up with the new themes > in Linux network configuration. Since we want oVirt and Vdsm to be > distribution agnostic, and support new features, we have to change. > > setupNetwork is responsible for two different things: > (1) configure the host networking interfaces, and > (2) create virtual networks for guests and connect the to the world > over (1). > > Functionality (2) is provided by building Linux software bridges, and > vlan devices. I'd like to explore moving it to Open vSwitch, which > would > enable a host of functionalities that we currently lack (e.g. > tunneling). One thing that worries me is the need to reimplement our > config snapshot/recovery on ovs's database. > > As far as I know, ovs is unable to maintain host level parameters of > interfaces (e.g. eth0's IPv4 address), so we need another > tool for functionality (1): either speak to NetworkManager directly, > or > to use NetCF, via its libvirt virInterface* wrapper. > > I have minor worries about NetCF's breadth of testing and usage; I > know > it is intended to be cross-platform, but unlike ovs, I am not aware > of a > wide Debian usage thereof. On the other hand, its API is ready for > vdsm's > usage for quite a while. > > NetworkManager has become ubiquitous, and we'd better integrate with > it > better than our current setting of NM_CONTROLLED=no. But as DPB tells > us, > https://lists.fedorahosted.org/pipermail/vdsm-devel/2012-November/001677.html > we'd better offload integration with NM to libvirt. For NetworkManager bridge support is in introduction stages https://bugzilla.gnome.org/show_bug.cgi?id=546197 > > We would like to take Network configuration in VDSM to the next level > and make it distribution agnostic in addition for setting the > infrastructure for more advanced features to be used going forward. > The path we think of taking is to integrate with OVS and for feature > completeness use NetCF, via its libvirt virInterface* wrapper. Any > comments or feedback on this proposal is welcomed. > + from netCF site " How can I help ? netcf is in its very early stages, and can be improved in any number of ways. The most pressing needs right now are implementing backends for other distributions (Debian, Ubuntu, etc.) and operating systems (Solaris) testing and using netcf " At a short term I think that it will only add overhead an instability if you'll depend on them. Long term I don't see why not. > Thanks to the oVirt net team members who's input has helped writing > this > email. > > Dan. > ___ > vdsm-devel mailing list > vdsm-devel@lists.fedorahosted.org > https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel > ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] [RFC] Implied UUIDs in API
+1 Long ago it was not enforced and then changed. The motivation for the enforcing as I recall was that for VMs this was also the guest system ID. I don't know if this is the case now days. Regards, Simon Sent from my iPhone, please excuse any typos. ב-31 באוג 2012, בשעה 00:19, Saggi Mizrahi כתב/ה: > Hi, in the API a lot of IDs get passed around are UUIDs. > The point is that as long as you are not the entity generating the UUIDs the > fact that these are UUIDs have no real significance to you. > I suggest removing the validation of UUIDs from the receiving end. There is > no real reason to make sure these are real UUIDs. > It's another restriction we can remove from the interface simplifying the > code and the interface. > > Just to be clear I'm not saying that we should stop using UUIDs. > For example, vdsm will keep generating task IDs as UUIDs. But the > documentation will state that it could be *any* string value. > If for some reason we choose to change the format of task IDs. There will be > no need to change the interface. > > The same goes for VM IDs. Currently the engine uses UUIDs but there is no > reason for VDSM to enforce this and limit the engine from ever changing it in > the future and using other string values. > ___ > Arch mailing list > a...@ovirt.org > http://lists.ovirt.org/mailman/listinfo/arch ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] flowID schema
Hi Ayal, > > actual use case where someone would have actually > > benefited > > from > > this. Sorry joining that late into the discussion. Forget the automatic analyses/past issues recognition that may or may not be possible if we do have flow ID. (Personally I believe it's possible for known issues) Think of large setups, not the few hosts setups that are running in the labs. Dozens of hosts, hundreds of VMs and tasks, running for few months. --->>> Huge logs (yes we even got 50G sized logs) Now try to do what you and Saggy are suggesting said, yes it will work!!! Sometimes it is the only approach that will be possible. But what about efficiency? Some times I found myself spending 30 minutes just getting to the files that I need to view and to the approximate location of the error. If you have a flow ID you can have a tool that will extract all the relevant information from all the hosts participating in that flow and correlate that with the rhevm.log as the anchor. (Valdik actually wrote a tool that follows the SPM and creates an SPM log no matter on which host it resides - this allowed tracking flows and proved itself useful and time saving, especially since when storage issues begin SPM tends to move around :)) In addition to that think about skill levels, For you it's very easy to go and debug, you wrote the code, you know what to expect and what is the normal behavior (most of the time). But what about others? When encountering an issue within a flow for a front line or a customer self debug the easiest way will be to actually compare it to the same flow that succeed (if exist) and understand what was supposed to happen next (or before) and did not. I actually used that technique a lot. Again a flow ID will save time. All you have to do is extract an identical flow that worked using the flow-extractor utility and then compare. I agree that when you get to the really tough cases, they will span across flows and your only alternative may be just to dig in. However for many others, flows tracking will save support calls, support escalations and will considerably help with education. For 2.2 we had Marina going into the code and documenting flows, so support people can learn what to expect and what to look for in the logs. I believe it will also help engineering, lot's of times I had to gather 3 engineers and sat together just to understand a flow in order to debug an issue or understand what to expect if someone complained about a weird behavior. Since each of them only new what his part is doing - all had to go into their code and discuss. With flow ID you can just extract the flow and easily see what is being done (you may find out as happened before that it's not exactly what they meant to do) In short, if we aim to look at RHEV as a system, flows are helpful in many aspects. Instead of having discrete components and requiring a high level of expertise and spending a lot of time (finding the right spot) to debug even minor issues you can look at a system and easily get (most of the time), understand what to expect and get to the most relevant logs fairly fast. Simon. - Original Message - > From: "Dan Yasny" > To: "Ayal Baron" > Cc: "Simon Grinberg" , vdsm-devel@lists.fedorahosted.org > Sent: Friday, February 10, 2012 4:01:01 PM > Subject: Re: [vdsm] flowID schema > > > > - Original Message - > > From: "Ayal Baron" > > To: "Dan Yasny" > > Cc: "Simon Grinberg" , > > vdsm-devel@lists.fedorahosted.org > > Sent: Friday, 10 February, 2012 3:51:01 PM > > Subject: Re: [vdsm] flowID schema > > > > > > > > - Original Message - > > > > > > > > > - Original Message - > > > > From: "Ayal Baron" > > > > To: "Dan Yasny" > > > > Cc: "Simon Grinberg" , > > > > vdsm-devel@lists.fedorahosted.org > > > > Sent: Friday, 10 February, 2012 2:55:46 PM > > > > Subject: Re: [vdsm] flowID schema > > > > > > > > > > > > > > > > - Original Message - > > > > > > > > > > > > > > > - Original Message - > > > > > > From: "Ayal Baron" > > > > > > To: "Dan Yasny" > > > > > > Cc: "Simon Grinberg" , > > > > > > vdsm-devel@lists.fedorahosted.org > > > > > > Sent: Friday, 10 February, 2012 12:50:04 AM > > > > > > Subject: Re: [vdsm] flowID schema > > > > > > > > > > > > > > > > > > > > &
