Re: [galaxy-dev] Restart Galaxy without interruption

[Rémy Dernat]

Hi,

Thanks Laure and Martin.

This is very interesting. However, can it handle the load balancing part as
Gildas asked ? Anyway, thanks for the tip.

Can it be possible to share /etc/supervisor/conf.d/galaxy.conf without
having to use ansible ? I downloaded the tarball, however, I did not find
this file; is it (seems to be) galaxy.j2 in the template directory ?

Best regards,
Remy

2016-11-16 16:36 GMT+01:00 Martin Čech :

> Hi Gildas,
>
> the approach which Main Galaxy takes is as described by Nate in the
> exercise document linked by Laure. Exerpt here:
>
> "In addition, you can gracefully restart the uWSGI Galaxy process with
> sudo supervisorctl signal HUP gx:galaxy. uWSGI is configured to start
> Galaxy in a "master" process and then fork the configured number of worker
> processes. Because of this, if sent a SIGHUP signal, it will kill the
> workers but the master process will hold its socket open, blocking client
> (browser) connections until new workers are forked. This prevents users
> from seeing a proxy error page during restarts."
>
> Using this, you can restart both the uWSGI server and Galaxy handlers with:
>
> $ sudo supervisorctl signal HUP gx:galaxy && sudo supervisorctl restart
> gx:handler0 gx:handler1
>
> without supervisor it would be something like $ pkill -HUP -o -u 
> uwsgi
>
> Thanks for using Galaxy,
>
> Martin
>
> On Wed, Nov 16, 2016 at 5:19 AM Laure QUINTRIC 
> wrote:
>
>> Hi Gildas,
>>
>> Have a look here : https://github.com/martenson/
>> dagobah-training/tree/master/advanced/002a-systemd-supervisor
>>
>> See ya.
>>
>>
>> Laure
>>
>> Le 14/11/2016 à 21:35, Gildas Le Corguillé a écrit :
>>
>> Hi,
>>
>> Since Galaxy need to be restarted sometime (load datatype, add ressources
>> in .loc, apply changes in the galaxy.ini), is anybody of you have solutions
>> to restart it without interrupt the service?
>>
>> We are now using nginx, uwsgi and supervisord.
>>
>> My hope is to split the Galaxy process in two group (1 uwsgi and 2 job
>> handlers per group). That way, we will be able to restart the group1 and
>> then the group2.
>>
>>
>> I was planning to set the 2 uwsgi servers and delegate the load balancing
>> to ngnix (https://www.nginx.com/resources/admin-guide/load-balancer/)
>> But maybe, some of you already implemented that and can provide me some
>> advices or warnings.
>>
>> The job handler part seems easier to deal with
>> --server-name=handler1_%(process_num)s
>> --server-name=handler2_%(process_num)s
>>
>> Thanks by advance
>>
>> Gildas
>>
>> -
>> Gildas Le Corguillé - Bioinformatician/Bioanalyste
>>
>> Plateform ABiMS (Analyses and Bioinformatics for Marine Science)
>> http://abims.sb-roscoff.fr
>>
>> Member of the Workflow4Metabolomics project
>> http://workflow4metabolomics.org
>>
>> Station Biologique de Roscoff - UPMC/CNRS - FR2424
>> Place Georges Teissier 29680 Roscoff FRANCE
>> tel: +33 2 98 29 23 81 <+33%202%2098%2029%2023%2081>
>> --
>>
>>
>>
>>
>>
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>   https://lists.galaxyproject.org/
>>
>> To search Galaxy mailing lists use the unified search at:
>>   http://galaxyproject.org/search/mailinglists/
>>
>>
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>   https://lists.galaxyproject.org/
>>
>> To search Galaxy mailing lists use the unified search at:
>>   http://galaxyproject.org/search/mailinglists/
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Restart Galaxy without interruption

[Martin Čech]

Hi Gildas,

the approach which Main Galaxy takes is as described by Nate in the
exercise document linked by Laure. Exerpt here:

"In addition, you can gracefully restart the uWSGI Galaxy process with sudo
supervisorctl signal HUP gx:galaxy. uWSGI is configured to start Galaxy in
a "master" process and then fork the configured number of worker processes.
Because of this, if sent a SIGHUP signal, it will kill the workers but the
master process will hold its socket open, blocking client (browser)
connections until new workers are forked. This prevents users from seeing a
proxy error page during restarts."

Using this, you can restart both the uWSGI server and Galaxy handlers with:

$ sudo supervisorctl signal HUP gx:galaxy && sudo supervisorctl restart
gx:handler0 gx:handler1

without supervisor it would be something like $ pkill -HUP -o -u 
uwsgi

Thanks for using Galaxy,

Martin

On Wed, Nov 16, 2016 at 5:19 AM Laure QUINTRIC 
wrote:

> Hi Gildas,
>
> Have a look here :
> https://github.com/martenson/dagobah-training/tree/master/advanced/002a-systemd-supervisor
>
> See ya.
>
>
> Laure
>
> Le 14/11/2016 à 21:35, Gildas Le Corguillé a écrit :
>
> Hi,
>
> Since Galaxy need to be restarted sometime (load datatype, add ressources
> in .loc, apply changes in the galaxy.ini), is anybody of you have solutions
> to restart it without interrupt the service?
>
> We are now using nginx, uwsgi and supervisord.
>
> My hope is to split the Galaxy process in two group (1 uwsgi and 2 job
> handlers per group). That way, we will be able to restart the group1 and
> then the group2.
>
>
> I was planning to set the 2 uwsgi servers and delegate the load balancing
> to ngnix (https://www.nginx.com/resources/admin-guide/load-balancer/)
> But maybe, some of you already implemented that and can provide me some
> advices or warnings.
>
> The job handler part seems easier to deal with
> --server-name=handler1_%(process_num)s
> --server-name=handler2_%(process_num)s
>
> Thanks by advance
>
> Gildas
>
> -
> Gildas Le Corguillé - Bioinformatician/Bioanalyste
>
> Plateform ABiMS (Analyses and Bioinformatics for Marine Science)
> http://abims.sb-roscoff.fr
>
> Member of the Workflow4Metabolomics project
> http://workflow4metabolomics.org
>
> Station Biologique de Roscoff - UPMC/CNRS - FR2424
> Place Georges Teissier 29680 Roscoff FRANCE
> tel: +33 2 98 29 23 81 <+33%202%2098%2029%2023%2081>
> --
>
>
>
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Restart Galaxy without interruption

[Laure QUINTRIC]

Hi Gildas,

Have a look here : 
https://github.com/martenson/dagobah-training/tree/master/advanced/002a-systemd-supervisor


See ya.
Laure


Le 14/11/2016 à 21:35, Gildas Le Corguillé a écrit :

Hi,

Since Galaxy need to be restarted sometime (load datatype, add 
ressources in .loc, apply changes in the galaxy.ini), is anybody of 
you have solutions to restart it without interrupt the service?


We are now using nginx, uwsgi and supervisord.

My hope is to split the Galaxy process in two group (1 uwsgi and 2 job 
handlers per group). That way, we will be able to restart the group1 
and then the group2.



I was planning to set the 2 uwsgi servers and delegate the load 
balancing to ngnix 
(https://www.nginx.com/resources/admin-guide/load-balancer/)
But maybe, some of you already implemented that and can provide me 
some advices or warnings.


The job handler part seems easier to deal with
--server-name=handler1_%(process_num)s
--server-name=handler2_%(process_num)s

Thanks by advance

Gildas

-
Gildas Le Corguillé - Bioinformatician/Bioanalyste

Plateform ABiMS (Analyses and Bioinformatics for Marine Science)
http://abims.sb-roscoff.fr

Member of the Workflow4Metabolomics project
http://workflow4metabolomics.org 

Station Biologique de Roscoff - UPMC/CNRS - FR2424
Place Georges Teissier 29680 Roscoff FRANCE
tel: +33 2 98 29 23 81
--





___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/