Lukasz, hi! I try to reproduce problem with new rabbitmq from this email and all works fine without any delay with 7 astute workers.
I think we should build ISO with new RabbitMQ and ttl fixes (today i spend time to reinvestigate already solved problem with ttl). Several questions for you: - does this problem repeat only in HA-case https://bugs.launchpad.net/fuel/+bug/1278336 ? - does problem repeat every time or episodically? - does this problem affect 'generate diagnostic snapshot' (it take much longer time)? I use simple CentOS cluster (1 controller + 1 compute) and 'generate diagnostic snapshot', because a few month ago in this cases UI have significant delays. As result — all work without delay. Now try with HA. On Tue, Mar 25, 2014 at 3:41 PM, Mike Scherbakov <[email protected]>wrote: > Great findings, Lukasz! > > Adding larger audience of fuel-dev.. > > > On Tue, Mar 25, 2014 at 3:37 PM, Lukasz Oles <[email protected]> wrote: > >> Vladimir, >> >> there is no ISO, just install newest rabbitmq. I attached rpm package for >> centos. >> >> I have done some more investigation and number of workers actually >> doesn't matter. It just gives random results, but I think I found solution. >> >> Naily is using asynchronous library to communicate with Rabbitmq. It uses >> amqp library which uses EventMachine. In Naily event loop is running in >> main thread but consumer is running in another and publisher in yet another >> thread. >> >> To solve the problem with hanging I moved the code for publisher and >> consumer to EM::next_tick block. After this everything is working again >> now. >> EM::next_tick does two things. First, it schedules code to run in next >> event loop iteration. Second it runs this code in event loop thread. I'm >> not sure which of this things helps. >> Debugging async code in threads is really hard. Why in the first place >> Naily is using async library? >> >> What do you think about it? Maybe it would be better just move to >> synchronous library like bunny? >> >> Please remember I'm not ruby programmer so I can be missing something >> here. >> >> Regards >> >> >> On Tue, Mar 25, 2014 at 10:42 AM, Vladimir Sharshov < >> [email protected]> wrote: >> >>> Guys, please share link to ISO. Without it i could not say anything >>> useful about potencial problem with naily. Thanks! >>> >>> >>> On Mon, Mar 24, 2014 at 4:34 PM, Andrey Danin <[email protected]>wrote: >>> >>>> A huge ttl value was set in order to allow nodes with unsynchronized >>>> time be able to use mcollective. if a master node has a local time more >>>> than 6000 seconds in past in comparison with target nodes, these target >>>> nodes will not be able to answer via mcollective. >>>> >>>> >>>> On Mon, Mar 24, 2014 at 2:07 PM, Vladimir Sharshov < >>>> [email protected]> wrote: >>>> >>>>> Hi all! >>>>> >>>>> > When I changed number of workers in naily from 3 to 2 everything >>>>> started to work >>>>> At now moment we increase this value to 7 as i remember. What about >>>>> shared connection - at now moment it works without any issues. Due to this >>>>> limitation only 2 clients in new version looks very strange. >>>>> >>>>> Please share link to iso, i try to reproduce and investigate this >>>>> problem. Thanks! >>>>> >>>>> >>>>> On Wed, Mar 19, 2014 at 1:36 PM, Mike Scherbakov < >>>>> [email protected]> wrote: >>>>> >>>>>> Vladimir - I think you've been working with Naily workers, any >>>>>> thoughts on the issue? >>>>>> >>>>>> >>>>>> On Wed, Mar 19, 2014 at 1:04 PM, Andrey Korolyov < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> On 03/19/2014 12:52 PM, Dmitry Pyzhov wrote: >>>>>>> > + more guys. >>>>>>> > >>>>>>> > >>>>>>> > On Wed, Mar 19, 2014 at 12:10 PM, Lukasz Oles <[email protected] >>>>>>> > <mailto:[email protected]>> wrote: >>>>>>> > >>>>>>> > Hello guys, >>>>>>> > >>>>>>> > After a lot of testing and debugging finally I have something >>>>>>> to share. >>>>>>> > >>>>>>> > First change is in mcollective settings. In >>>>>>> > /etc/mcollective/server.cfg value ttl = 2000000000 is too big. >>>>>>> > Rabbitmq returns error. I changed it to 6000, without this >>>>>>> > mcollective will not work. >>>>>>> >>>>>>> The problem is not in value itself but in way how rmq drivers pushes >>>>>>> it. >>>>>>> Somehow it turns as a *concatenation* of default value around 10k and >>>>>>> this one, which is definitely too large for first one` concatenation >>>>>>> result. Just remove this value entirely as I did before from the >>>>>>> config, >>>>>>> three hours are acceptable enough. >>>>>>> > >>>>>>> > Another problem is with task status update. In my tests it >>>>>>> hangs for >>>>>>> > about 10 minutes. After that task is updated. Unfortunately >>>>>>> it's not >>>>>>> > a problem with python but with naily. When I changed number of >>>>>>> > workers in naily from 3 to 2 everything started to work. I >>>>>>> think >>>>>>> > it's because all thread are using the same connection and >>>>>>> chanel to >>>>>>> > publish results but I'm still investigating it. >>>>>>> > >>>>>>> > Regards >>>>>>> > >>>>>>> > >>>>>>> > On Thu, Mar 13, 2014 at 12:38 PM, Dmitry Pyzhov >>>>>>> > <[email protected] <mailto:[email protected]>> wrote: >>>>>>> > >>>>>>> > Lukasz, >>>>>>> > >>>>>>> > Feel free to contact us if you need anything else. >>>>>>> > >>>>>>> > >>>>>>> > On Thu, Mar 13, 2014 at 3:22 PM, Lukasz Oles < >>>>>>> [email protected] >>>>>>> > <mailto:[email protected]>> wrote: >>>>>>> > >>>>>>> > ok, thx for rpm >>>>>>> > >>>>>>> > >>>>>>> > On Thu, Mar 13, 2014 at 11:40 AM, Dmitry Burmistrov >>>>>>> > <[email protected] <mailto: >>>>>>> [email protected]>> >>>>>>> > wrote: >>>>>>> > >>>>>>> > Package rabbitmq-server has been built from >>>>>>> changeset: >>>>>>> > http://gerrit.mirantis.com/13455 >>>>>>> > RPM Repository URL: >>>>>>> > http:/// >>>>>>> osci-obs.vm.mirantis.net:82/centos-fuel-5.0-stable-13455/centos >>>>>>> > < >>>>>>> http://osci-obs.vm.mirantis.net:82/centos-fuel-5.0-stable-13455/centos >>>>>>> > >>>>>>> > >>>>>>> > Package rabbitmq-server has been built from >>>>>>> changeset: >>>>>>> > http://gerrit.mirantis.com/13457 >>>>>>> > DEB Repository URL: >>>>>>> > http:/// >>>>>>> osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.0-stable-13457/ubuntu >>>>>>> > < >>>>>>> http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.0-stable-13457/ubuntu >>>>>>> > >>>>>>> > >>>>>>> > On Thu, Mar 13, 2014 at 2:11 PM, Dmitry Pyzhov >>>>>>> > <[email protected] <mailto:[email protected]>> >>>>>>> wrote: >>>>>>> > > Lukasz, >>>>>>> > > >>>>>>> > > Sorry for long response. Our OSCI team will >>>>>>> build a >>>>>>> > package. Dmitry B, could >>>>>>> > > you reply with the download link when it is >>>>>>> available? >>>>>>> > Ticket OSCI-1016. >>>>>>> > > >>>>>>> > > >>>>>>> > > On Wed, Mar 12, 2014 at 2:18 PM, Lukasz Oles >>>>>>> > <[email protected] <mailto:[email protected]>> >>>>>>> wrote: >>>>>>> > >> >>>>>>> > >> Sure, I will look into it. Can you give me a >>>>>>> link to >>>>>>> > rabbitmq rpm which >>>>>>> > >> you used? >>>>>>> > >> >>>>>>> > >> Regards, >>>>>>> > >> >>>>>>> > >> >>>>>>> > >> On Wed, Mar 12, 2014 at 11:10 AM, Dmitry Pyzhov >>>>>>> > <[email protected] <mailto:[email protected] >>>>>>> >> >>>>>>> > >> wrote: >>>>>>> > >>> >>>>>>> > >>> Lukasz, >>>>>>> > >>> >>>>>>> > >>> actually we have no idea what is wrong with >>>>>>> fresh >>>>>>> > rabbitmq. For some >>>>>>> > >>> reason refresh of task status takes too much >>>>>>> time. >>>>>>> > Dmitry tried to find the >>>>>>> > >>> root cause, but did not succeed. Could you >>>>>>> > investigate the issue? >>>>>>> > >>> >>>>>>> > >>> >>>>>>> > >>> On Tue, Mar 11, 2014 at 9:49 PM, Lukasz Oles >>>>>>> > <[email protected] <mailto:[email protected]>> >>>>>>> wrote: >>>>>>> > >>>> >>>>>>> > >>>> Dmitry, >>>>>>> > >>>> >>>>>>> > >>>> rabbitmq update looks interesting, I can look >>>>>>> into >>>>>>> > it. Do I need any >>>>>>> > >>>> additional information? >>>>>>> > >>>> >>>>>>> > >>>> regards, >>>>>>> > >>>> >>>>>>> > >>>> >>>>>>> > >>>> On Tue, Mar 11, 2014 at 2:05 PM, Dmitry Pyzhov >>>>>>> > <[email protected] <mailto:[email protected] >>>>>>> >> >>>>>>> > >>>> wrote: >>>>>>> > >>>>> >>>>>>> > >>>>> Great! >>>>>>> > >>>>> >>>>>>> > >>>>> Lukasz, could you help us with rabbitmq >>>>>>> update? We >>>>>>> > faced an issue with >>>>>>> > >>>>> it: >>>>>>> https://bugs.launchpad.net/fuel/+bug/1278336 >>>>>>> > >>>>> >>>>>>> > >>>>> Also, could you participate it design review: >>>>>>> > >>>>> >>>>>>> > >>>>>>> https://docs.google.com/document/d/1zqV58LZBLQ-0gllb_i3MyIKIMj-Qx8ELJohjcWs459s/edit?usp=sharing >>>>>>> > >>>>> >>>>>>> > >>>>> >>>>>>> > >>>>> On Mon, Mar 10, 2014 at 9:01 PM, Mike >>>>>>> Scherbakov >>>>>>> > >>>>> <[email protected] >>>>>>> > <mailto:[email protected]>> wrote: >>>>>>> > >>>>>> >>>>>>> > >>>>>> Lukasz, >>>>>>> > >>>>>> please take any bugs from >>>>>>> > https://launchpad.net/fuel/+milestone/5.0 >>>>>>> > >>>>>> which are not assigned to particular >>>>>>> person. You >>>>>>> > are likely to be interested >>>>>>> > >>>>>> in those which are assigned to >>>>>>> "fuel-python". Of >>>>>>> > course, it's preferred to >>>>>>> > >>>>>> work on Critical and High priority bugs in a >>>>>>> > first order. >>>>>>> > >>>>>> >>>>>>> > >>>>>> We are in a design phase for 5.0. Please >>>>>>> take a >>>>>>> > look at >>>>>>> > >>>>>> >>>>>>> > >>>>>>> https://mirantis.jira.com/wiki/display/PRD/5.0+-+Mirantis+OpenStack+release+home+page >>>>>>> . >>>>>>> > >>>>>> I'm discussing this still with management, >>>>>>> and we >>>>>>> > will likely have only part >>>>>>> > >>>>>> of what is on the page. Your comments and >>>>>>> input >>>>>>> > into design docs (which you >>>>>>> > >>>>>> can find following blueprint link, then >>>>>>> "Read the >>>>>>> > full spec") is very >>>>>>> > >>>>>> welcome. >>>>>>> > >>>>>> >>>>>>> > >>>>>> Start looking over and try to identify spot >>>>>>> which >>>>>>> > is in most interest >>>>>>> > >>>>>> of you. Dmitry/Evgeny will help to identify >>>>>>> areas >>>>>>> > where help is mostly >>>>>>> > >>>>>> needed. Sorry for not responding to you in >>>>>>> time. >>>>>>> > I'll get my team to fix >>>>>>> > >>>>>> this. >>>>>>> > >>>>>> >>>>>>> > >>>>>> FYI: Today is holiday in Russia & Ukraine >>>>>>> > >>>>>> Thanks, >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> On Thu, Mar 6, 2014 at 12:37 AM, Mike >>>>>>> Scherbakov >>>>>>> > >>>>>> <[email protected] >>>>>>> > <mailto:[email protected]>> wrote: >>>>>>> > >>>>>>> >>>>>>> > >>>>>>> It's great. I would be happy to see Lukasz >>>>>>> > working on Fuel. >>>>>>> > >>>>>>> Actually, Lukasz already doing great job >>>>>>> helping >>>>>>> > us with Nailgun >>>>>>> > >>>>>>> scalability issues resolution. >>>>>>> > >>>>>>> >>>>>>> > >>>>>>> Dmitry, please arrange meeting between our >>>>>>> > Python engineers and >>>>>>> > >>>>>>> Lukasz, and identify areas where >>>>>>> contribution of >>>>>>> > Lukasz will be the most >>>>>>> > >>>>>>> effective. It should be aligned with the >>>>>>> > development of our engineers too. >>>>>>> > >>>>>>> >>>>>>> > >>>>>>> Thanks, >>>>>>> > >>>>>>> >>>>>>> > >>>>>>> >>>>>>> > >>>>>>> On Tue, Mar 4, 2014 at 3:27 PM, Piotr >>>>>>> Siwczak >>>>>>> > <[email protected] <mailto: >>>>>>> [email protected]>> >>>>>>> > >>>>>>> wrote: >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> Mike, >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> For now I see Lukasz has finished his >>>>>>> work for >>>>>>> > Softlayer/Express (at >>>>>>> > >>>>>>>> least for the next few weeks)and can use >>>>>>> his >>>>>>> > time to engage into Fuel >>>>>>> > >>>>>>>> development. Please feel free to assign >>>>>>> him to >>>>>>> > Fuel tasks. >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> -Piotr >>>>>>> > >>>>>>> >>>>>>> > >>>>>>> >>>>>>> > >>>>>>> >>>>>>> > >>>>>>> >>>>>>> > >>>>>>> -- >>>>>>> > >>>>>>> Mike Scherbakov >>>>>>> > >>>>>>> #mihgen >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> -- >>>>>>> > >>>>>> Mike Scherbakov >>>>>>> > >>>>>> #mihgen >>>>>>> > >>>>> >>>>>>> > >>>>> >>>>>>> > >>>> >>>>>>> > >>>> >>>>>>> > >>>> >>>>>>> > >>>> -- >>>>>>> > >>>> Łukasz Oleś >>>>>>> > >>> >>>>>>> > >>> >>>>>>> > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> > >> -- >>>>>>> > >> Łukasz Oleś >>>>>>> > > >>>>>>> > > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > -- >>>>>>> > Łukasz Oleś >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > -- >>>>>>> > Łukasz Oleś >>>>>>> > >>>>>>> > >>>>>>> > -- >>>>>>> > You received this message because you are subscribed to the Google >>>>>>> > Groups "fuel-core-team" group. >>>>>>> > To unsubscribe from this group and stop receiving emails from it, >>>>>>> send >>>>>>> > an email to [email protected] >>>>>>> > <mailto:[email protected]>. >>>>>>> > For more options, visit >>>>>>> https://groups.google.com/a/mirantis.com/d/optout. >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "fuel-core-team" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to [email protected]. >>>>>>> For more options, visit >>>>>>> https://groups.google.com/a/mirantis.com/d/optout. >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Mike Scherbakov >>>>>> #mihgen >>>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "fuel-core-team" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> For more options, visit >>>>> https://groups.google.com/a/mirantis.com/d/optout. >>>>> >>>> >>>> >>>> >>>> -- >>>> Andrey Danin >>>> [email protected] >>>> skype: gcon.monolake >>>> >>> >>> >> >> >> -- >> Łukasz Oleś >> > > > > -- > Mike Scherbakov > #mihgen >
-- Mailing list: https://launchpad.net/~fuel-dev Post to : [email protected] Unsubscribe : https://launchpad.net/~fuel-dev More help : https://help.launchpad.net/ListHelp
