Please make sure you include a fix for this problem in fuel-library when using RabbitMQ3 with OpenStack: https://bugs.launchpad.net/fuel/+bug/1296922
On Wed, Apr 2, 2014 at 6:53 AM, Vladimir Sharshov <[email protected]> wrote: > First of all: QA says what bug reproduced. Now i try to fix it based on > Lukasz investigations. > > Short how info how to make such iso > (http://172.18.8.252/fuelweb_rabbitmq3-5.0.iso): > > We need this things: > > 1. Patch for fuel-library (https://review.openstack.org/#/c/84707/) > 2. Patch for fuel-main (https://review.openstack.org/#/c/84717/) > 3. Custom RabbitMQ repo: > http://osci-obs.vm.mirantis.net:82/centos-fuel-5.0-stable-14055/centos > > Steps: > > http://docs.mirantis.com/fuel-dev/develop/env.html#building-the-fuel-iso > (steps 1-4) > cd fuel-main > Apply patch for fuel-main: > git fetch https://review.openstack.org/stackforge/fuel-main > refs/changes/17/84717/1 && git cherry-pick FETCH_HEAD > Edit config.mk: > FUELLIB_GERRIT_COMMIT?=refs/changes/07/84707/1 > Build iso (in example i use Moscow mirror, but it is not necessary) and > custom RabbitMQ3 package: > make iso USE_MIRROR=msk > EXTRA_RPM_REPOS="rabbitmq3,http://osci-obs.vm.mirantis.net:82/centos-fuel-5.0-stable-14055/centos"; > Check iso in 'build/iso' folder > > > > > > On Wed, Apr 2, 2014 at 1:40 PM, Vladimir Sharshov <[email protected]> > wrote: >> >> Long awaited iso: http://172.18.8.252/fuelweb_rabbitmq3-5.0.iso (internal >> network only) >> >> Using this iso we can try to repeat problem with progress bar. TTL changes >> included. >> >> >> On Mon, Mar 31, 2014 at 3:38 PM, Vladimir Sharshov >> <[email protected]> wrote: >>> >>> Hi all. >>> >>> Current version of RabbitMQ 3 package have issue which does not allow >>> install it in fresh system. OSCI team now work on it. After it will be done, >>> i will build iso for test. >>> >>> Sorry for delay. >>> >>> >>> On Thu, Mar 27, 2014 at 4:19 PM, Vladimir Sharshov >>> <[email protected]> wrote: >>>> >>>> Thanks! >>>> >>>> > Do you know easy way to do it? >>>> Yes. I do this soon. >>>> >>>> >>>> >>>> >>>> On Thu, Mar 27, 2014 at 3:30 PM, Lukasz Oles <[email protected]> wrote: >>>>> >>>>> hi, comments inline >>>>> >>>>> >>>>> On Thu, Mar 27, 2014 at 10:50 AM, Vladimir Sharshov >>>>> <[email protected]> wrote: >>>>>> >>>>>> I try to reproduce problem with new rabbitmq from this email and all >>>>>> works fine without any delay with 7 astute workers. >>>>>> >>>>>> I think we should build ISO with new RabbitMQ and ttl fixes (today i >>>>>> spend time to reinvestigate already solved problem with ttl). >>>>> >>>>> Do you know easy way to do it? >>>>>> >>>>>> >>>>>> Several questions for you: >>>>>> >>>>>> does this problem repeat only in HA-case >>>>>> https://bugs.launchpad.net/fuel/+bug/1278336 ? >>>>> >>>>> No, I'm testing it by adding two nodes to cluster and pressing "Verify >>>>> netoworks' in Netowork Tab. I'm waiting 10 min to see results on GUI. >>>>>> >>>>>> does problem repeat every time or episodically? >>>>> >>>>> Almost every time. Once it worked, with 2 astute workers, but after >>>>> restart it stopped to work. >>>>>> >>>>>> does this problem affect 'generate diagnostic snapshot' (it take much >>>>>> longer time)? >>>>> >>>>> I don't know. I tested it on 4.1 branch. I will repeat tests on trunk >>>>> version. >>>>> >>>>>> >>>>>> I use simple CentOS cluster (1 controller + 1 compute) and 'generate >>>>>> diagnostic snapshot', because a few month ago in this cases UI have >>>>>> significant delays. As result — all work without delay. Now try with HA. >>>>>> >>>>>> >>>>>> On Tue, Mar 25, 2014 at 3:41 PM, Mike Scherbakov >>>>>> <[email protected]> wrote: >>>>>>> >>>>>>> Great findings, Lukasz! >>>>>>> >>>>>>> Adding larger audience of fuel-dev.. >>>>>>> >>>>>>> >>>>>>> On Tue, Mar 25, 2014 at 3:37 PM, Lukasz Oles <[email protected]> >>>>>>> wrote: >>>>>>>> >>>>>>>> Vladimir, >>>>>>>> >>>>>>>> there is no ISO, just install newest rabbitmq. I attached rpm >>>>>>>> package for centos. >>>>>>>> >>>>>>>> I have done some more investigation and number of workers actually >>>>>>>> doesn't matter. It just gives random results, but I think I found >>>>>>>> solution. >>>>>>>> >>>>>>>> Naily is using asynchronous library to communicate with Rabbitmq. It >>>>>>>> uses amqp library which uses EventMachine. In Naily event loop is >>>>>>>> running in >>>>>>>> main thread but consumer is running in another and publisher in yet >>>>>>>> another >>>>>>>> thread. >>>>>>>> >>>>>>>> To solve the problem with hanging I moved the code for publisher and >>>>>>>> consumer to EM::next_tick block. After this everything is working >>>>>>>> again now. >>>>>>>> EM::next_tick does two things. First, it schedules code to run in >>>>>>>> next event loop iteration. Second it runs this code in event loop >>>>>>>> thread. >>>>>>>> I'm not sure which of this things helps. >>>>>>>> Debugging async code in threads is really hard. Why in the first >>>>>>>> place Naily is using async library? >>>>>>>> >>>>>>>> What do you think about it? Maybe it would be better just move to >>>>>>>> synchronous library like bunny? >>>>>>>> >>>>>>>> Please remember I'm not ruby programmer so I can be missing >>>>>>>> something here. >>>>>>>> >>>>>>>> Regards >>>>>>>> >>>>>>>> >>>>>>>> On Tue, Mar 25, 2014 at 10:42 AM, Vladimir Sharshov >>>>>>>> <[email protected]> wrote: >>>>>>>>> >>>>>>>>> Guys, please share link to ISO. Without it i could not say anything >>>>>>>>> useful about potencial problem with naily. Thanks! >>>>>>>>> >>>>>>>>> >>>>>>>>> On Mon, Mar 24, 2014 at 4:34 PM, Andrey Danin <[email protected]> >>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> A huge ttl value was set in order to allow nodes with >>>>>>>>>> unsynchronized time be able to use mcollective. if a master node has >>>>>>>>>> a local >>>>>>>>>> time more than 6000 seconds in past in comparison with target nodes, >>>>>>>>>> these >>>>>>>>>> target nodes will not be able to answer via mcollective. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Mon, Mar 24, 2014 at 2:07 PM, Vladimir Sharshov >>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>> Hi all! >>>>>>>>>>> >>>>>>>>>>> > When I changed number of workers in naily from 3 to 2 >>>>>>>>>>> > everything started to work >>>>>>>>>>> At now moment we increase this value to 7 as i remember. What >>>>>>>>>>> about shared connection - at now moment it works without any >>>>>>>>>>> issues. Due to >>>>>>>>>>> this limitation only 2 clients in new version looks very strange. >>>>>>>>>>> >>>>>>>>>>> Please share link to iso, i try to reproduce and investigate this >>>>>>>>>>> problem. Thanks! >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Wed, Mar 19, 2014 at 1:36 PM, Mike Scherbakov >>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>> Vladimir - I think you've been working with Naily workers, any >>>>>>>>>>>> thoughts on the issue? >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Wed, Mar 19, 2014 at 1:04 PM, Andrey Korolyov >>>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> On 03/19/2014 12:52 PM, Dmitry Pyzhov wrote: >>>>>>>>>>>>> > + more guys. >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > On Wed, Mar 19, 2014 at 12:10 PM, Lukasz Oles >>>>>>>>>>>>> > <[email protected] >>>>>>>>>>>>> > <mailto:[email protected]>> wrote: >>>>>>>>>>>>> > >>>>>>>>>>>>> > Hello guys, >>>>>>>>>>>>> > >>>>>>>>>>>>> > After a lot of testing and debugging finally I have >>>>>>>>>>>>> > something to share. >>>>>>>>>>>>> > >>>>>>>>>>>>> > First change is in mcollective settings. In >>>>>>>>>>>>> > /etc/mcollective/server.cfg value ttl = 2000000000 is >>>>>>>>>>>>> > too big. >>>>>>>>>>>>> > Rabbitmq returns error. I changed it to 6000, without >>>>>>>>>>>>> > this >>>>>>>>>>>>> > mcollective will not work. >>>>>>>>>>>>> >>>>>>>>>>>>> The problem is not in value itself but in way how rmq drivers >>>>>>>>>>>>> pushes it. >>>>>>>>>>>>> Somehow it turns as a *concatenation* of default value around >>>>>>>>>>>>> 10k and >>>>>>>>>>>>> this one, which is definitely too large for first one` >>>>>>>>>>>>> concatenation >>>>>>>>>>>>> result. Just remove this value entirely as I did before from >>>>>>>>>>>>> the config, >>>>>>>>>>>>> three hours are acceptable enough. >>>>>>>>>>>>> > >>>>>>>>>>>>> > Another problem is with task status update. In my tests >>>>>>>>>>>>> > it hangs for >>>>>>>>>>>>> > about 10 minutes. After that task is updated. >>>>>>>>>>>>> > Unfortunately it's not >>>>>>>>>>>>> > a problem with python but with naily. When I changed >>>>>>>>>>>>> > number of >>>>>>>>>>>>> > workers in naily from 3 to 2 everything started to work. >>>>>>>>>>>>> > I think >>>>>>>>>>>>> > it's because all thread are using the same connection and >>>>>>>>>>>>> > chanel to >>>>>>>>>>>>> > publish results but I'm still investigating it. >>>>>>>>>>>>> > >>>>>>>>>>>>> > Regards >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > On Thu, Mar 13, 2014 at 12:38 PM, Dmitry Pyzhov >>>>>>>>>>>>> > <[email protected] <mailto:[email protected]>> >>>>>>>>>>>>> > wrote: >>>>>>>>>>>>> > >>>>>>>>>>>>> > Lukasz, >>>>>>>>>>>>> > >>>>>>>>>>>>> > Feel free to contact us if you need anything else. >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > On Thu, Mar 13, 2014 at 3:22 PM, Lukasz Oles >>>>>>>>>>>>> > <[email protected] >>>>>>>>>>>>> > <mailto:[email protected]>> wrote: >>>>>>>>>>>>> > >>>>>>>>>>>>> > ok, thx for rpm >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > On Thu, Mar 13, 2014 at 11:40 AM, Dmitry >>>>>>>>>>>>> > Burmistrov >>>>>>>>>>>>> > <[email protected] >>>>>>>>>>>>> > <mailto:[email protected]>> >>>>>>>>>>>>> > wrote: >>>>>>>>>>>>> > >>>>>>>>>>>>> > Package rabbitmq-server has been built from >>>>>>>>>>>>> > changeset: >>>>>>>>>>>>> > http://gerrit.mirantis.com/13455 >>>>>>>>>>>>> > RPM Repository URL: >>>>>>>>>>>>> > >>>>>>>>>>>>> > http:///osci-obs.vm.mirantis.net:82/centos-fuel-5.0-stable-13455/centos >>>>>>>>>>>>> > >>>>>>>>>>>>> > <http://osci-obs.vm.mirantis.net:82/centos-fuel-5.0-stable-13455/centos> >>>>>>>>>>>>> > >>>>>>>>>>>>> > Package rabbitmq-server has been built from >>>>>>>>>>>>> > changeset: >>>>>>>>>>>>> > http://gerrit.mirantis.com/13457 >>>>>>>>>>>>> > DEB Repository URL: >>>>>>>>>>>>> > >>>>>>>>>>>>> > http:///osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.0-stable-13457/ubuntu >>>>>>>>>>>>> > >>>>>>>>>>>>> > <http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.0-stable-13457/ubuntu> >>>>>>>>>>>>> > >>>>>>>>>>>>> > On Thu, Mar 13, 2014 at 2:11 PM, Dmitry >>>>>>>>>>>>> > Pyzhov >>>>>>>>>>>>> > <[email protected] >>>>>>>>>>>>> > <mailto:[email protected]>> wrote: >>>>>>>>>>>>> > > Lukasz, >>>>>>>>>>>>> > > >>>>>>>>>>>>> > > Sorry for long response. Our OSCI team will >>>>>>>>>>>>> > build a >>>>>>>>>>>>> > package. Dmitry B, could >>>>>>>>>>>>> > > you reply with the download link when it is >>>>>>>>>>>>> > available? >>>>>>>>>>>>> > Ticket OSCI-1016. >>>>>>>>>>>>> > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> > > On Wed, Mar 12, 2014 at 2:18 PM, Lukasz >>>>>>>>>>>>> > Oles >>>>>>>>>>>>> > <[email protected] >>>>>>>>>>>>> > <mailto:[email protected]>> wrote: >>>>>>>>>>>>> > >> >>>>>>>>>>>>> > >> Sure, I will look into it. Can you give me >>>>>>>>>>>>> > a link to >>>>>>>>>>>>> > rabbitmq rpm which >>>>>>>>>>>>> > >> you used? >>>>>>>>>>>>> > >> >>>>>>>>>>>>> > >> Regards, >>>>>>>>>>>>> > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>> > >> On Wed, Mar 12, 2014 at 11:10 AM, Dmitry >>>>>>>>>>>>> > Pyzhov >>>>>>>>>>>>> > <[email protected] >>>>>>>>>>>>> > <mailto:[email protected]>> >>>>>>>>>>>>> > >> wrote: >>>>>>>>>>>>> > >>> >>>>>>>>>>>>> > >>> Lukasz, >>>>>>>>>>>>> > >>> >>>>>>>>>>>>> > >>> actually we have no idea what is wrong >>>>>>>>>>>>> > with fresh >>>>>>>>>>>>> > rabbitmq. For some >>>>>>>>>>>>> > >>> reason refresh of task status takes too >>>>>>>>>>>>> > much time. >>>>>>>>>>>>> > Dmitry tried to find the >>>>>>>>>>>>> > >>> root cause, but did not succeed. Could >>>>>>>>>>>>> > you >>>>>>>>>>>>> > investigate the issue? >>>>>>>>>>>>> > >>> >>>>>>>>>>>>> > >>> >>>>>>>>>>>>> > >>> On Tue, Mar 11, 2014 at 9:49 PM, Lukasz >>>>>>>>>>>>> > Oles >>>>>>>>>>>>> > <[email protected] >>>>>>>>>>>>> > <mailto:[email protected]>> wrote: >>>>>>>>>>>>> > >>>> >>>>>>>>>>>>> > >>>> Dmitry, >>>>>>>>>>>>> > >>>> >>>>>>>>>>>>> > >>>> rabbitmq update looks interesting, I can >>>>>>>>>>>>> > look into >>>>>>>>>>>>> > it. Do I need any >>>>>>>>>>>>> > >>>> additional information? >>>>>>>>>>>>> > >>>> >>>>>>>>>>>>> > >>>> regards, >>>>>>>>>>>>> > >>>> >>>>>>>>>>>>> > >>>> >>>>>>>>>>>>> > >>>> On Tue, Mar 11, 2014 at 2:05 PM, Dmitry >>>>>>>>>>>>> > Pyzhov >>>>>>>>>>>>> > <[email protected] >>>>>>>>>>>>> > <mailto:[email protected]>> >>>>>>>>>>>>> > >>>> wrote: >>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>> > >>>>> Great! >>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>> > >>>>> Lukasz, could you help us with rabbitmq >>>>>>>>>>>>> > update? We >>>>>>>>>>>>> > faced an issue with >>>>>>>>>>>>> > >>>>> it: >>>>>>>>>>>>> > https://bugs.launchpad.net/fuel/+bug/1278336 >>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>> > >>>>> Also, could you participate it design >>>>>>>>>>>>> > review: >>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>> > >>>>>>>>>>>>> > https://docs.google.com/document/d/1zqV58LZBLQ-0gllb_i3MyIKIMj-Qx8ELJohjcWs459s/edit?usp=sharing >>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>> > >>>>> On Mon, Mar 10, 2014 at 9:01 PM, Mike >>>>>>>>>>>>> > Scherbakov >>>>>>>>>>>>> > >>>>> <[email protected] >>>>>>>>>>>>> > <mailto:[email protected]>> wrote: >>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>> > >>>>>> Lukasz, >>>>>>>>>>>>> > >>>>>> please take any bugs from >>>>>>>>>>>>> > https://launchpad.net/fuel/+milestone/5.0 >>>>>>>>>>>>> > >>>>>> which are not assigned to particular >>>>>>>>>>>>> > person. You >>>>>>>>>>>>> > are likely to be interested >>>>>>>>>>>>> > >>>>>> in those which are assigned to >>>>>>>>>>>>> > "fuel-python". Of >>>>>>>>>>>>> > course, it's preferred to >>>>>>>>>>>>> > >>>>>> work on Critical and High priority >>>>>>>>>>>>> > bugs in a >>>>>>>>>>>>> > first order. >>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>> > >>>>>> We are in a design phase for 5.0. >>>>>>>>>>>>> > Please take a >>>>>>>>>>>>> > look at >>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>> > >>>>>>>>>>>>> > https://mirantis.jira.com/wiki/display/PRD/5.0+-+Mirantis+OpenStack+release+home+page. >>>>>>>>>>>>> > >>>>>> I'm discussing this still with >>>>>>>>>>>>> > management, and we >>>>>>>>>>>>> > will likely have only part >>>>>>>>>>>>> > >>>>>> of what is on the page. Your comments >>>>>>>>>>>>> > and input >>>>>>>>>>>>> > into design docs (which you >>>>>>>>>>>>> > >>>>>> can find following blueprint link, >>>>>>>>>>>>> > then "Read the >>>>>>>>>>>>> > full spec") is very >>>>>>>>>>>>> > >>>>>> welcome. >>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>> > >>>>>> Start looking over and try to identify >>>>>>>>>>>>> > spot which >>>>>>>>>>>>> > is in most interest >>>>>>>>>>>>> > >>>>>> of you. Dmitry/Evgeny will help to >>>>>>>>>>>>> > identify areas >>>>>>>>>>>>> > where help is mostly >>>>>>>>>>>>> > >>>>>> needed. Sorry for not responding to >>>>>>>>>>>>> > you in time. >>>>>>>>>>>>> > I'll get my team to fix >>>>>>>>>>>>> > >>>>>> this. >>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>> > >>>>>> FYI: Today is holiday in Russia & >>>>>>>>>>>>> > Ukraine >>>>>>>>>>>>> > >>>>>> Thanks, >>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>> > >>>>>> On Thu, Mar 6, 2014 at 12:37 AM, Mike >>>>>>>>>>>>> > Scherbakov >>>>>>>>>>>>> > >>>>>> <[email protected] >>>>>>>>>>>>> > <mailto:[email protected]>> wrote: >>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>> > >>>>>>> It's great. I would be happy to see >>>>>>>>>>>>> > Lukasz >>>>>>>>>>>>> > working on Fuel. >>>>>>>>>>>>> > >>>>>>> Actually, Lukasz already doing great >>>>>>>>>>>>> > job helping >>>>>>>>>>>>> > us with Nailgun >>>>>>>>>>>>> > >>>>>>> scalability issues resolution. >>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>> > >>>>>>> Dmitry, please arrange meeting >>>>>>>>>>>>> > between our >>>>>>>>>>>>> > Python engineers and >>>>>>>>>>>>> > >>>>>>> Lukasz, and identify areas where >>>>>>>>>>>>> > contribution of >>>>>>>>>>>>> > Lukasz will be the most >>>>>>>>>>>>> > >>>>>>> effective. It should be aligned with >>>>>>>>>>>>> > the >>>>>>>>>>>>> > development of our engineers too. >>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>> > >>>>>>> Thanks, >>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>> > >>>>>>> On Tue, Mar 4, 2014 at 3:27 PM, Piotr >>>>>>>>>>>>> > Siwczak >>>>>>>>>>>>> > <[email protected] >>>>>>>>>>>>> > <mailto:[email protected]>> >>>>>>>>>>>>> > >>>>>>> wrote: >>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>> > >>>>>>>> Mike, >>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>> > >>>>>>>> For now I see Lukasz has finished >>>>>>>>>>>>> > his work for >>>>>>>>>>>>> > Softlayer/Express (at >>>>>>>>>>>>> > >>>>>>>> least for the next few weeks)and can >>>>>>>>>>>>> > use his >>>>>>>>>>>>> > time to engage into Fuel >>>>>>>>>>>>> > >>>>>>>> development. Please feel free to >>>>>>>>>>>>> > assign him to >>>>>>>>>>>>> > Fuel tasks. >>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>> > >>>>>>>> -Piotr >>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>> > >>>>>>> -- >>>>>>>>>>>>> > >>>>>>> Mike Scherbakov >>>>>>>>>>>>> > >>>>>>> #mihgen >>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>> > >>>>>> -- >>>>>>>>>>>>> > >>>>>> Mike Scherbakov >>>>>>>>>>>>> > >>>>>> #mihgen >>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>> > >>>> >>>>>>>>>>>>> > >>>> >>>>>>>>>>>>> > >>>> >>>>>>>>>>>>> > >>>> -- >>>>>>>>>>>>> > >>>> Łukasz Oleś >>>>>>>>>>>>> > >>> >>>>>>>>>>>>> > >>> >>>>>>>>>>>>> > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>> > >> -- >>>>>>>>>>>>> > >> Łukasz Oleś >>>>>>>>>>>>> > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > -- >>>>>>>>>>>>> > Łukasz Oleś >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > -- >>>>>>>>>>>>> > Łukasz Oleś >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > -- >>>>>>>>>>>>> > You received this message because you are subscribed to the >>>>>>>>>>>>> > Google >>>>>>>>>>>>> > Groups "fuel-core-team" group. >>>>>>>>>>>>> > To unsubscribe from this group and stop receiving emails from >>>>>>>>>>>>> > it, send >>>>>>>>>>>>> > an email to [email protected] >>>>>>>>>>>>> > <mailto:[email protected]>. >>>>>>>>>>>>> > For more options, visit >>>>>>>>>>>>> > https://groups.google.com/a/mirantis.com/d/optout. >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>>> Google Groups "fuel-core-team" group. >>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>>>> it, send an email to [email protected]. >>>>>>>>>>>>> For more options, visit >>>>>>>>>>>>> https://groups.google.com/a/mirantis.com/d/optout. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Mike Scherbakov >>>>>>>>>>>> #mihgen >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>> Google Groups "fuel-core-team" group. >>>>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>>>> send an email to [email protected]. >>>>>>>>>>> For more options, visit >>>>>>>>>>> https://groups.google.com/a/mirantis.com/d/optout. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Andrey Danin >>>>>>>>>> [email protected] >>>>>>>>>> skype: gcon.monolake >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Łukasz Oleś >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Mike Scherbakov >>>>>>> #mihgen >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Łukasz Oleś >>>> >>>> >>> >> > > -- > You received this message because you are subscribed to the Google Groups > "fuel-core-team" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/a/mirantis.com/d/optout. -- Dmitry Borodaenko -- Mailing list: https://launchpad.net/~fuel-dev Post to : [email protected] Unsubscribe : https://launchpad.net/~fuel-dev More help : https://help.launchpad.net/ListHelp
