Braulio, Are you referring to the vertical grey line? That is the deployment event. The part that spikes in the first graph is request queue which is a bit different on newrelic: http://blog.newrelic.com/2013/01/22/understanding-new-relic-queuing/
We are using HAProxy to load balance (round robin) to 4 physical hosts running unicorn with 6 workers. I have not tried to reproduce this on 1 master - I assume this would be the same. I do in fact do the sleep now: https://gist.github.com/sarkis/1aa296044b1dfd3695ab#file-unicorn-rb-L37 - the deployment results above had the 1 second sleep in there. On Thu, Mar 5, 2015 at 9:13 AM, Bráulio Bhavamitra <[email protected]> wrote: > In the graphs you posted, what is the grey part? It is not described in > the legend and it seems the problem is entirely there. What reverse proxy > are you using? > > Can you reproduce this with a single master instance? > > Could you try this sleep: > https://gist.github.com/brauliobo/11298486#file-unicorn-conf-rb-L91 > > > On Thu, Mar 5, 2015 at 2:07 PM, Sarkis Varozian <[email protected]> > wrote: > >> Hey All, >> >> So I changed up my unicorn.rb a bit from my original post: >> https://gist.github.com/sarkis/1aa296044b1dfd3695ab >> >> I'm also still sending the USR2 signals on deploy staggered with 30 >> second delay via capistrano: >> >> on roles(:web), in: :sequence, wait: 30 >> >> As you can see I am now doing a warup via rack MockRequest (I hoped this >> would warmup the master). However, this is what a deploy looks like on >> newrelic: >> >> >> https://www.dropbox.com/s/beh7nc8npdfijqp/Screenshot%202015-03-05%2009.05.15.png?dl=0 >> >> >> https://www.dropbox.com/s/w08gpvp7mpik3vs/Screenshot%202015-03-05%2009.06.51.png?dl=0 >> >> I'm running out of ideas to get rid of thse latency spikes. Would you >> guys recommend I try anything else at this point? >> >> >> >> On Wed, Mar 4, 2015 at 12:40 PM, Sarkis Varozian <[email protected]> >> wrote: >> >>> Eric, >>> >>> Thanks for the quick reply. >>> >>> We are on Ruby 2.1.5p273 and unicorn 4.8.3. I believe our problem is the >>> lazy loading - at least thats what all signs point to. I am going to try >>> and mock request some url endpoints. Currently, I can only think of '/', as >>> most other parts of the app require a session and auth. I'll report back >>> with results. >>> >>> >>> >>> On Wed, Mar 4, 2015 at 12:35 PM, Eric Wong <[email protected]> wrote: >>> >>>> Sarkis Varozian <[email protected]> wrote: >>>> > On Wed, Mar 4, 2015 at 12:17 PM, Michael Fischer < >>>> [email protected]> >>>> > wrote: >>>> > >>>> > > I'm not exactly sure how preload_app works, but I suspect your app >>>> is >>>> > > lazy-loading a number of Ruby libraries while handling the first few >>>> > > requests that weren't automatically loaded during the preload >>>> process. >>>> > > >>>> > > Eric, your thoughts? >>>> >>>> (top-posting corrected) >>>> >>>> Yeah, preload_app won't help startup speed if much of the app is >>>> autoloaded. >>>> >>>> Sarkis: which Ruby version are you running? IIRC, 1.9.2 had terrible >>>> startup performance compared to 1.9.3 and later in case you're stuck on >>>> 1.9.2 >>>> >>>> > That does make sense - I was looking at another suggestion from a >>>> user here >>>> > (Braulio) of running a "warmup" using rack MockRequest: >>>> > https://gist.github.com/brauliobo/11298486#file-unicorn-conf-rb-L77 >>>> > >>>> > The only issue I am having with the above solution is it is happening >>>> in >>>> > the before_fork block - shouldn't I warmup the connection in >>>> after_fork? >>>> >>>> If preload_app is true, you can warmup in before_fork; otherwise it >>>> needs to be after_fork. >>>> >>>> > If >>>> > I follow the above gist properly it warms up the server with the old >>>> > activerecord base connection and then its turned off, then turned >>>> back on >>>> > in after_fork. I think I am not understanding the sequence of events >>>> > there... >>>> >>>> With preload_app and warmup, you need to ensure any stream connections >>>> (DB, memcached, redis, etc..) do not get shared between processes, so >>>> it's standard practice to disconnect in the parent and reconnect in the >>>> child. >>>> >>>> > If this is the case, I should warmup and also check/kill the old >>>> > master in the after_fork block after the new db, redis, neo4j >>>> connections >>>> > are all created. Thoughts? >>>> >>>> I've been leaving killing the master outside of the unicorn hooks >>>> and doing it as a separate step; seemed too fragile to do it in >>>> hooks from my perspective. >>>> >>> >>> >>> >>> -- >>> *Sarkis Varozian* >>> [email protected] >>> >> >> >> >> -- >> *Sarkis Varozian* >> [email protected] >> > > > > -- > "Lute pela sua ideologia. Seja um com sua ideologia. Viva pela sua > ideologia. Morra por sua ideologia" P.R. Sarkar > > EITA - Educação, Informação e Tecnologias para Autogestão > http://cirandas.net/brauliobo > http://eita.org.br > > "Paramapurusha é meu pai e Parama Prakriti é minha mãe. O universo é meu > lar e todos nós somos cidadãos deste cosmo. Este universo é a imaginação da > Mente Macrocósmica, e todas as entidades estão sendo criadas, preservadas e > destruídas nas fases de extroversão e introversão do fluxo imaginativo > cósmico. No âmbito pessoal, quando uma pessoa imagina algo em sua mente, > naquele momento, essa pessoa é a única proprietária daquilo que ela > imagina, e ninguém mais. Quando um ser humano criado mentalmente caminha > por um milharal também imaginado, a pessoa imaginada não é a propriedade > desse milharal, pois ele pertence ao indivíduo que o está imaginando. Este > universo foi criado na imaginação de Brahma, a Entidade Suprema, por isso > a propriedade deste universo é de Brahma, e não dos microcosmos que também > foram criados pela imaginação de Brahma. Nenhuma propriedade deste mundo, > mutável ou imutável, pertence a um indivíduo em particular; tudo é o > patrimônio comum de todos." > Restante do texto em > http://cirandas.net/brauliobo/blog/a-problematica-de-hoje-em-dia > -- *Sarkis Varozian* [email protected]
