I also use newrelic and never saw this grey part... On Thu, Mar 5, 2015 at 2:28 PM, Sarkis Varozian <[email protected]> wrote:
> Braulio, > > Are you referring to the vertical grey line? That is the deployment event. > The part that spikes in the first graph is request queue which is a bit > different on newrelic: > http://blog.newrelic.com/2013/01/22/understanding-new-relic-queuing/ > > We are using HAProxy to load balance (round robin) to 4 physical hosts > running unicorn with 6 workers. > > I have not tried to reproduce this on 1 master - I assume this would be > the same. > > I do in fact do the sleep now: > https://gist.github.com/sarkis/1aa296044b1dfd3695ab#file-unicorn-rb-L37 - > the deployment results above had the 1 second sleep in there. > > On Thu, Mar 5, 2015 at 9:13 AM, Bráulio Bhavamitra <[email protected]> > wrote: > >> In the graphs you posted, what is the grey part? It is not described in >> the legend and it seems the problem is entirely there. What reverse proxy >> are you using? >> >> Can you reproduce this with a single master instance? >> >> Could you try this sleep: >> https://gist.github.com/brauliobo/11298486#file-unicorn-conf-rb-L91 >> >> >> On Thu, Mar 5, 2015 at 2:07 PM, Sarkis Varozian <[email protected]> >> wrote: >> >>> Hey All, >>> >>> So I changed up my unicorn.rb a bit from my original post: >>> https://gist.github.com/sarkis/1aa296044b1dfd3695ab >>> >>> I'm also still sending the USR2 signals on deploy staggered with 30 >>> second delay via capistrano: >>> >>> on roles(:web), in: :sequence, wait: 30 >>> >>> As you can see I am now doing a warup via rack MockRequest (I hoped this >>> would warmup the master). However, this is what a deploy looks like on >>> newrelic: >>> >>> >>> https://www.dropbox.com/s/beh7nc8npdfijqp/Screenshot%202015-03-05%2009.05.15.png?dl=0 >>> >>> >>> https://www.dropbox.com/s/w08gpvp7mpik3vs/Screenshot%202015-03-05%2009.06.51.png?dl=0 >>> >>> I'm running out of ideas to get rid of thse latency spikes. Would you >>> guys recommend I try anything else at this point? >>> >>> >>> >>> On Wed, Mar 4, 2015 at 12:40 PM, Sarkis Varozian <[email protected]> >>> wrote: >>> >>>> Eric, >>>> >>>> Thanks for the quick reply. >>>> >>>> We are on Ruby 2.1.5p273 and unicorn 4.8.3. I believe our problem is >>>> the lazy loading - at least thats what all signs point to. I am going to >>>> try and mock request some url endpoints. Currently, I can only think of >>>> '/', as most other parts of the app require a session and auth. I'll report >>>> back with results. >>>> >>>> >>>> >>>> On Wed, Mar 4, 2015 at 12:35 PM, Eric Wong <[email protected]> wrote: >>>> >>>>> Sarkis Varozian <[email protected]> wrote: >>>>> > On Wed, Mar 4, 2015 at 12:17 PM, Michael Fischer < >>>>> [email protected]> >>>>> > wrote: >>>>> > >>>>> > > I'm not exactly sure how preload_app works, but I suspect your app >>>>> is >>>>> > > lazy-loading a number of Ruby libraries while handling the first >>>>> few >>>>> > > requests that weren't automatically loaded during the preload >>>>> process. >>>>> > > >>>>> > > Eric, your thoughts? >>>>> >>>>> (top-posting corrected) >>>>> >>>>> Yeah, preload_app won't help startup speed if much of the app is >>>>> autoloaded. >>>>> >>>>> Sarkis: which Ruby version are you running? IIRC, 1.9.2 had terrible >>>>> startup performance compared to 1.9.3 and later in case you're stuck on >>>>> 1.9.2 >>>>> >>>>> > That does make sense - I was looking at another suggestion from a >>>>> user here >>>>> > (Braulio) of running a "warmup" using rack MockRequest: >>>>> > https://gist.github.com/brauliobo/11298486#file-unicorn-conf-rb-L77 >>>>> > >>>>> > The only issue I am having with the above solution is it is >>>>> happening in >>>>> > the before_fork block - shouldn't I warmup the connection in >>>>> after_fork? >>>>> >>>>> If preload_app is true, you can warmup in before_fork; otherwise it >>>>> needs to be after_fork. >>>>> >>>>> > If >>>>> > I follow the above gist properly it warms up the server with the old >>>>> > activerecord base connection and then its turned off, then turned >>>>> back on >>>>> > in after_fork. I think I am not understanding the sequence of events >>>>> > there... >>>>> >>>>> With preload_app and warmup, you need to ensure any stream connections >>>>> (DB, memcached, redis, etc..) do not get shared between processes, so >>>>> it's standard practice to disconnect in the parent and reconnect in the >>>>> child. >>>>> >>>>> > If this is the case, I should warmup and also check/kill the old >>>>> > master in the after_fork block after the new db, redis, neo4j >>>>> connections >>>>> > are all created. Thoughts? >>>>> >>>>> I've been leaving killing the master outside of the unicorn hooks >>>>> and doing it as a separate step; seemed too fragile to do it in >>>>> hooks from my perspective. >>>>> >>>> >>>> >>>> >>>> -- >>>> *Sarkis Varozian* >>>> [email protected] >>>> >>> >>> >>> >>> -- >>> *Sarkis Varozian* >>> [email protected] >>> >> >> >> >> -- >> "Lute pela sua ideologia. Seja um com sua ideologia. Viva pela sua >> ideologia. Morra por sua ideologia" P.R. Sarkar >> >> EITA - Educação, Informação e Tecnologias para Autogestão >> http://cirandas.net/brauliobo >> http://eita.org.br >> >> "Paramapurusha é meu pai e Parama Prakriti é minha mãe. O universo é meu >> lar e todos nós somos cidadãos deste cosmo. Este universo é a imaginação da >> Mente Macrocósmica, e todas as entidades estão sendo criadas, preservadas e >> destruídas nas fases de extroversão e introversão do fluxo imaginativo >> cósmico. No âmbito pessoal, quando uma pessoa imagina algo em sua mente, >> naquele momento, essa pessoa é a única proprietária daquilo que ela >> imagina, e ninguém mais. Quando um ser humano criado mentalmente caminha >> por um milharal também imaginado, a pessoa imaginada não é a propriedade >> desse milharal, pois ele pertence ao indivíduo que o está imaginando. Este >> universo foi criado na imaginação de Brahma, a Entidade Suprema, por isso >> a propriedade deste universo é de Brahma, e não dos microcosmos que também >> foram criados pela imaginação de Brahma. Nenhuma propriedade deste mundo, >> mutável ou imutável, pertence a um indivíduo em particular; tudo é o >> patrimônio comum de todos." >> Restante do texto em >> http://cirandas.net/brauliobo/blog/a-problematica-de-hoje-em-dia >> > > > > -- > *Sarkis Varozian* > [email protected] > -- "Lute pela sua ideologia. Seja um com sua ideologia. Viva pela sua ideologia. Morra por sua ideologia" P.R. Sarkar EITA - Educação, Informação e Tecnologias para Autogestão http://cirandas.net/brauliobo http://eita.org.br "Paramapurusha é meu pai e Parama Prakriti é minha mãe. O universo é meu lar e todos nós somos cidadãos deste cosmo. Este universo é a imaginação da Mente Macrocósmica, e todas as entidades estão sendo criadas, preservadas e destruídas nas fases de extroversão e introversão do fluxo imaginativo cósmico. No âmbito pessoal, quando uma pessoa imagina algo em sua mente, naquele momento, essa pessoa é a única proprietária daquilo que ela imagina, e ninguém mais. Quando um ser humano criado mentalmente caminha por um milharal também imaginado, a pessoa imaginada não é a propriedade desse milharal, pois ele pertence ao indivíduo que o está imaginando. Este universo foi criado na imaginação de Brahma, a Entidade Suprema, por isso a propriedade deste universo é de Brahma, e não dos microcosmos que também foram criados pela imaginação de Brahma. Nenhuma propriedade deste mundo, mutável ou imutável, pertence a um indivíduo em particular; tudo é o patrimônio comum de todos." Restante do texto em http://cirandas.net/brauliobo/blog/a-problematica-de-hoje-em-dia
