Great findings, Lukasz! Adding larger audience of fuel-dev..
On Tue, Mar 25, 2014 at 3:37 PM, Lukasz Oles <[email protected]> wrote: > Vladimir, > > there is no ISO, just install newest rabbitmq. I attached rpm package for > centos. > > I have done some more investigation and number of workers actually doesn't > matter. It just gives random results, but I think I found solution. > > Naily is using asynchronous library to communicate with Rabbitmq. It uses > amqp library which uses EventMachine. In Naily event loop is running in > main thread but consumer is running in another and publisher in yet another > thread. > > To solve the problem with hanging I moved the code for publisher and > consumer to EM::next_tick block. After this everything is working again > now. > EM::next_tick does two things. First, it schedules code to run in next > event loop iteration. Second it runs this code in event loop thread. I'm > not sure which of this things helps. > Debugging async code in threads is really hard. Why in the first place > Naily is using async library? > > What do you think about it? Maybe it would be better just move to > synchronous library like bunny? > > Please remember I'm not ruby programmer so I can be missing something here. > > Regards > > > On Tue, Mar 25, 2014 at 10:42 AM, Vladimir Sharshov < > [email protected]> wrote: > >> Guys, please share link to ISO. Without it i could not say anything >> useful about potencial problem with naily. Thanks! >> >> >> On Mon, Mar 24, 2014 at 4:34 PM, Andrey Danin <[email protected]>wrote: >> >>> A huge ttl value was set in order to allow nodes with unsynchronized >>> time be able to use mcollective. if a master node has a local time more >>> than 6000 seconds in past in comparison with target nodes, these target >>> nodes will not be able to answer via mcollective. >>> >>> >>> On Mon, Mar 24, 2014 at 2:07 PM, Vladimir Sharshov < >>> [email protected]> wrote: >>> >>>> Hi all! >>>> >>>> > When I changed number of workers in naily from 3 to 2 everything >>>> started to work >>>> At now moment we increase this value to 7 as i remember. What about >>>> shared connection - at now moment it works without any issues. Due to this >>>> limitation only 2 clients in new version looks very strange. >>>> >>>> Please share link to iso, i try to reproduce and investigate this >>>> problem. Thanks! >>>> >>>> >>>> On Wed, Mar 19, 2014 at 1:36 PM, Mike Scherbakov < >>>> [email protected]> wrote: >>>> >>>>> Vladimir - I think you've been working with Naily workers, any >>>>> thoughts on the issue? >>>>> >>>>> >>>>> On Wed, Mar 19, 2014 at 1:04 PM, Andrey Korolyov < >>>>> [email protected]> wrote: >>>>> >>>>>> On 03/19/2014 12:52 PM, Dmitry Pyzhov wrote: >>>>>> > + more guys. >>>>>> > >>>>>> > >>>>>> > On Wed, Mar 19, 2014 at 12:10 PM, Lukasz Oles <[email protected] >>>>>> > <mailto:[email protected]>> wrote: >>>>>> > >>>>>> > Hello guys, >>>>>> > >>>>>> > After a lot of testing and debugging finally I have something >>>>>> to share. >>>>>> > >>>>>> > First change is in mcollective settings. In >>>>>> > /etc/mcollective/server.cfg value ttl = 2000000000 is too big. >>>>>> > Rabbitmq returns error. I changed it to 6000, without this >>>>>> > mcollective will not work. >>>>>> >>>>>> The problem is not in value itself but in way how rmq drivers pushes >>>>>> it. >>>>>> Somehow it turns as a *concatenation* of default value around 10k and >>>>>> this one, which is definitely too large for first one` concatenation >>>>>> result. Just remove this value entirely as I did before from the >>>>>> config, >>>>>> three hours are acceptable enough. >>>>>> > >>>>>> > Another problem is with task status update. In my tests it >>>>>> hangs for >>>>>> > about 10 minutes. After that task is updated. Unfortunately >>>>>> it's not >>>>>> > a problem with python but with naily. When I changed number of >>>>>> > workers in naily from 3 to 2 everything started to work. I think >>>>>> > it's because all thread are using the same connection and >>>>>> chanel to >>>>>> > publish results but I'm still investigating it. >>>>>> > >>>>>> > Regards >>>>>> > >>>>>> > >>>>>> > On Thu, Mar 13, 2014 at 12:38 PM, Dmitry Pyzhov >>>>>> > <[email protected] <mailto:[email protected]>> wrote: >>>>>> > >>>>>> > Lukasz, >>>>>> > >>>>>> > Feel free to contact us if you need anything else. >>>>>> > >>>>>> > >>>>>> > On Thu, Mar 13, 2014 at 3:22 PM, Lukasz Oles < >>>>>> [email protected] >>>>>> > <mailto:[email protected]>> wrote: >>>>>> > >>>>>> > ok, thx for rpm >>>>>> > >>>>>> > >>>>>> > On Thu, Mar 13, 2014 at 11:40 AM, Dmitry Burmistrov >>>>>> > <[email protected] <mailto: >>>>>> [email protected]>> >>>>>> > wrote: >>>>>> > >>>>>> > Package rabbitmq-server has been built from >>>>>> changeset: >>>>>> > http://gerrit.mirantis.com/13455 >>>>>> > RPM Repository URL: >>>>>> > http:/// >>>>>> osci-obs.vm.mirantis.net:82/centos-fuel-5.0-stable-13455/centos >>>>>> > < >>>>>> http://osci-obs.vm.mirantis.net:82/centos-fuel-5.0-stable-13455/centos >>>>>> > >>>>>> > >>>>>> > Package rabbitmq-server has been built from >>>>>> changeset: >>>>>> > http://gerrit.mirantis.com/13457 >>>>>> > DEB Repository URL: >>>>>> > http:/// >>>>>> osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.0-stable-13457/ubuntu >>>>>> > < >>>>>> http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.0-stable-13457/ubuntu >>>>>> > >>>>>> > >>>>>> > On Thu, Mar 13, 2014 at 2:11 PM, Dmitry Pyzhov >>>>>> > <[email protected] <mailto:[email protected]>> >>>>>> wrote: >>>>>> > > Lukasz, >>>>>> > > >>>>>> > > Sorry for long response. Our OSCI team will build >>>>>> a >>>>>> > package. Dmitry B, could >>>>>> > > you reply with the download link when it is >>>>>> available? >>>>>> > Ticket OSCI-1016. >>>>>> > > >>>>>> > > >>>>>> > > On Wed, Mar 12, 2014 at 2:18 PM, Lukasz Oles >>>>>> > <[email protected] <mailto:[email protected]>> >>>>>> wrote: >>>>>> > >> >>>>>> > >> Sure, I will look into it. Can you give me a >>>>>> link to >>>>>> > rabbitmq rpm which >>>>>> > >> you used? >>>>>> > >> >>>>>> > >> Regards, >>>>>> > >> >>>>>> > >> >>>>>> > >> On Wed, Mar 12, 2014 at 11:10 AM, Dmitry Pyzhov >>>>>> > <[email protected] <mailto:[email protected] >>>>>> >> >>>>>> > >> wrote: >>>>>> > >>> >>>>>> > >>> Lukasz, >>>>>> > >>> >>>>>> > >>> actually we have no idea what is wrong with >>>>>> fresh >>>>>> > rabbitmq. For some >>>>>> > >>> reason refresh of task status takes too much >>>>>> time. >>>>>> > Dmitry tried to find the >>>>>> > >>> root cause, but did not succeed. Could you >>>>>> > investigate the issue? >>>>>> > >>> >>>>>> > >>> >>>>>> > >>> On Tue, Mar 11, 2014 at 9:49 PM, Lukasz Oles >>>>>> > <[email protected] <mailto:[email protected]>> >>>>>> wrote: >>>>>> > >>>> >>>>>> > >>>> Dmitry, >>>>>> > >>>> >>>>>> > >>>> rabbitmq update looks interesting, I can look >>>>>> into >>>>>> > it. Do I need any >>>>>> > >>>> additional information? >>>>>> > >>>> >>>>>> > >>>> regards, >>>>>> > >>>> >>>>>> > >>>> >>>>>> > >>>> On Tue, Mar 11, 2014 at 2:05 PM, Dmitry Pyzhov >>>>>> > <[email protected] <mailto:[email protected] >>>>>> >> >>>>>> > >>>> wrote: >>>>>> > >>>>> >>>>>> > >>>>> Great! >>>>>> > >>>>> >>>>>> > >>>>> Lukasz, could you help us with rabbitmq >>>>>> update? We >>>>>> > faced an issue with >>>>>> > >>>>> it: >>>>>> https://bugs.launchpad.net/fuel/+bug/1278336 >>>>>> > >>>>> >>>>>> > >>>>> Also, could you participate it design review: >>>>>> > >>>>> >>>>>> > >>>>>> https://docs.google.com/document/d/1zqV58LZBLQ-0gllb_i3MyIKIMj-Qx8ELJohjcWs459s/edit?usp=sharing >>>>>> > >>>>> >>>>>> > >>>>> >>>>>> > >>>>> On Mon, Mar 10, 2014 at 9:01 PM, Mike >>>>>> Scherbakov >>>>>> > >>>>> <[email protected] >>>>>> > <mailto:[email protected]>> wrote: >>>>>> > >>>>>> >>>>>> > >>>>>> Lukasz, >>>>>> > >>>>>> please take any bugs from >>>>>> > https://launchpad.net/fuel/+milestone/5.0 >>>>>> > >>>>>> which are not assigned to particular person. >>>>>> You >>>>>> > are likely to be interested >>>>>> > >>>>>> in those which are assigned to >>>>>> "fuel-python". Of >>>>>> > course, it's preferred to >>>>>> > >>>>>> work on Critical and High priority bugs in a >>>>>> > first order. >>>>>> > >>>>>> >>>>>> > >>>>>> We are in a design phase for 5.0. Please >>>>>> take a >>>>>> > look at >>>>>> > >>>>>> >>>>>> > >>>>>> https://mirantis.jira.com/wiki/display/PRD/5.0+-+Mirantis+OpenStack+release+home+page >>>>>> . >>>>>> > >>>>>> I'm discussing this still with management, >>>>>> and we >>>>>> > will likely have only part >>>>>> > >>>>>> of what is on the page. Your comments and >>>>>> input >>>>>> > into design docs (which you >>>>>> > >>>>>> can find following blueprint link, then >>>>>> "Read the >>>>>> > full spec") is very >>>>>> > >>>>>> welcome. >>>>>> > >>>>>> >>>>>> > >>>>>> Start looking over and try to identify spot >>>>>> which >>>>>> > is in most interest >>>>>> > >>>>>> of you. Dmitry/Evgeny will help to identify >>>>>> areas >>>>>> > where help is mostly >>>>>> > >>>>>> needed. Sorry for not responding to you in >>>>>> time. >>>>>> > I'll get my team to fix >>>>>> > >>>>>> this. >>>>>> > >>>>>> >>>>>> > >>>>>> FYI: Today is holiday in Russia & Ukraine >>>>>> > >>>>>> Thanks, >>>>>> > >>>>>> >>>>>> > >>>>>> >>>>>> > >>>>>> On Thu, Mar 6, 2014 at 12:37 AM, Mike >>>>>> Scherbakov >>>>>> > >>>>>> <[email protected] >>>>>> > <mailto:[email protected]>> wrote: >>>>>> > >>>>>>> >>>>>> > >>>>>>> It's great. I would be happy to see Lukasz >>>>>> > working on Fuel. >>>>>> > >>>>>>> Actually, Lukasz already doing great job >>>>>> helping >>>>>> > us with Nailgun >>>>>> > >>>>>>> scalability issues resolution. >>>>>> > >>>>>>> >>>>>> > >>>>>>> Dmitry, please arrange meeting between our >>>>>> > Python engineers and >>>>>> > >>>>>>> Lukasz, and identify areas where >>>>>> contribution of >>>>>> > Lukasz will be the most >>>>>> > >>>>>>> effective. It should be aligned with the >>>>>> > development of our engineers too. >>>>>> > >>>>>>> >>>>>> > >>>>>>> Thanks, >>>>>> > >>>>>>> >>>>>> > >>>>>>> >>>>>> > >>>>>>> On Tue, Mar 4, 2014 at 3:27 PM, Piotr >>>>>> Siwczak >>>>>> > <[email protected] <mailto: >>>>>> [email protected]>> >>>>>> > >>>>>>> wrote: >>>>>> > >>>>>>>> >>>>>> > >>>>>>>> Mike, >>>>>> > >>>>>>>> >>>>>> > >>>>>>>> For now I see Lukasz has finished his work >>>>>> for >>>>>> > Softlayer/Express (at >>>>>> > >>>>>>>> least for the next few weeks)and can use >>>>>> his >>>>>> > time to engage into Fuel >>>>>> > >>>>>>>> development. Please feel free to assign >>>>>> him to >>>>>> > Fuel tasks. >>>>>> > >>>>>>>> >>>>>> > >>>>>>>> -Piotr >>>>>> > >>>>>>> >>>>>> > >>>>>>> >>>>>> > >>>>>>> >>>>>> > >>>>>>> >>>>>> > >>>>>>> -- >>>>>> > >>>>>>> Mike Scherbakov >>>>>> > >>>>>>> #mihgen >>>>>> > >>>>>> >>>>>> > >>>>>> >>>>>> > >>>>>> >>>>>> > >>>>>> >>>>>> > >>>>>> -- >>>>>> > >>>>>> Mike Scherbakov >>>>>> > >>>>>> #mihgen >>>>>> > >>>>> >>>>>> > >>>>> >>>>>> > >>>> >>>>>> > >>>> >>>>>> > >>>> >>>>>> > >>>> -- >>>>>> > >>>> Łukasz Oleś >>>>>> > >>> >>>>>> > >>> >>>>>> > >> >>>>>> > >> >>>>>> > >> >>>>>> > >> -- >>>>>> > >> Łukasz Oleś >>>>>> > > >>>>>> > > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > -- >>>>>> > Łukasz Oleś >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > -- >>>>>> > Łukasz Oleś >>>>>> > >>>>>> > >>>>>> > -- >>>>>> > You received this message because you are subscribed to the Google >>>>>> > Groups "fuel-core-team" group. >>>>>> > To unsubscribe from this group and stop receiving emails from it, >>>>>> send >>>>>> > an email to [email protected] >>>>>> > <mailto:[email protected]>. >>>>>> > For more options, visit >>>>>> https://groups.google.com/a/mirantis.com/d/optout. >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "fuel-core-team" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> For more options, visit >>>>>> https://groups.google.com/a/mirantis.com/d/optout. >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Mike Scherbakov >>>>> #mihgen >>>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "fuel-core-team" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> For more options, visit >>>> https://groups.google.com/a/mirantis.com/d/optout. >>>> >>> >>> >>> >>> -- >>> Andrey Danin >>> [email protected] >>> skype: gcon.monolake >>> >> >> > > > -- > Łukasz Oleś > -- Mike Scherbakov #mihgen
-- Mailing list: https://launchpad.net/~fuel-dev Post to : [email protected] Unsubscribe : https://launchpad.net/~fuel-dev More help : https://help.launchpad.net/ListHelp
