Re: [galaxy-dev] Keeping Galaxy up to date
Hi Nate, Thank you very much for enlightening me - I tried to get an impression on Ansible playbooks for the last two minutes and I am really exited about it. Principle and syntax are close to what we are doing in plain BASH plus config files, but the respective scripts are for sure more complex, because we have to write explicitly the tasks which are 'ready to use' modules in Ansible (and I really like YAML btw :) ). I am really looking forward to your updated article on the wiki - in the meantime I will forward the verve to my colleagues ;). Maybe we manage to switch on the playbook principle before the planned update. Cheers & thanks again, Sebastian > Hi Sebastian, > > As for Galaxy Main (usegalaxy.org), our process is automated with Ansible, > the playbook for which can be found here: > > https://github.com/galaxyproject/usegalaxy-playbook > > A bit of documentation on what you should look for (new versions of > universe_wsgi.ini.sample, datatypes_conf.xml.sample, etc.) would certainly > be helpful, so I'll expand the "keeping up to date" page on getgalaxy.org > into a new page and add more details. > > On Tue, Sep 9, 2014 at 6:55 AM, Sebastian Schaaf < > sch...@ibe.med.uni-muenchen.de> wrote: > >> Hi Thomas, >> >> We had this topic at the GCC this year, especially within the Galaxy >> Admins BoF. The truth is (please correct me anyone if this is not the >> case >> anymore!) that you are touching an unresolved point. Keeping track of >> tools, settings etc. across the versions is not given. People have their >> 'home-brew' solutions for this, e.g. keeping a (maybe virtual) machine >> in >> spare in order to invest several hours up to some days to test (with or >> without participation of the users) applicability to local >> constraints/histories/workflow/tools. The more testing is automated and >> the less users (or non-default pieces) the faster the update procedure >> can >> be. But there are also instances which did not receive updates for >> months >> or even years. >> >> On our side we will be facing reality within the next two weeks as we >> really need the update. Although preconditions are pretty good (few >> users, >> not that deep modifications, high grade of automation) we expect some >> issues. happy to have a virtualized environment... >> >> To wrap up my answer in a nutshell: no, there is no best practice guide >> and no migration tool, but every single contribution is warmly welcome >> :). >> >> It would be *very* interesting how updates are handled at Galaxy >> Main...? >> >> Cheers, >> >> Sebastian >> (very interested in further feedback) >> >> >> > Hello, >> > >> > I try to keep our Galaxy instance up to date. >> > Very often, some tools disappear and other do not work anymore. This >> > breaks some users workflows. >> > >> > I may miss something. Is there a "best update pratice" to keep a >> Galaxy >> > instance and the tool-sheds fully functionnal ? >> > >> > Regards, >> > >> > Thomas >> > >> > -- >> > Thomas Bellembois, Network and System Administrator, IGFL (France) >> > http://perso.ens-lyon.fr/thomas.bellembois - +33 4 26 73 13 67 >> > (IGFL internal IT doc: http://itdoc.igfl.ens-lyon.fr/itdoc) >> > ___ >> > Please keep all replies on the list by using "reply all" >> > in your mail client. To manage your subscriptions to this >> > and other Galaxy lists, please use the interface at: >> > http://lists.bx.psu.edu/ >> > >> > To search Galaxy mailing lists use the unified search at: >> > http://galaxyproject.org/search/mailinglists/ >> > >> >> >> -- >> Sebastian Schaaf, M.Sc. Bioinformatics >> Faculty Coordinator NGS Infrastructure >> Chair of Biometry and Bioinformatics >> Department of Medical Informatics, >> Biometry and Epidemiology (IBE) >> University of Munich >> Marchioninistr. 15, K U1 (postal) >> Marchioninistr. 17, U 006 (office) >> D-81377 Munich (Germany) >> Tel: +49 89 2180-78178 >> >> ___ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this >> and other Galaxy lists, please use the interface at: >> http://lists.bx.psu.edu/ >> >> To search Galaxy mailing lists use the unified search at: >> http://galaxyproject.org/search/mailinglists/ >> > ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Keeping Galaxy up to date
Hi Sebastian, As for Galaxy Main (usegalaxy.org), our process is automated with Ansible, the playbook for which can be found here: https://github.com/galaxyproject/usegalaxy-playbook A bit of documentation on what you should look for (new versions of universe_wsgi.ini.sample, datatypes_conf.xml.sample, etc.) would certainly be helpful, so I'll expand the "keeping up to date" page on getgalaxy.org into a new page and add more details. On Tue, Sep 9, 2014 at 6:55 AM, Sebastian Schaaf < sch...@ibe.med.uni-muenchen.de> wrote: > Hi Thomas, > > We had this topic at the GCC this year, especially within the Galaxy > Admins BoF. The truth is (please correct me anyone if this is not the case > anymore!) that you are touching an unresolved point. Keeping track of > tools, settings etc. across the versions is not given. People have their > 'home-brew' solutions for this, e.g. keeping a (maybe virtual) machine in > spare in order to invest several hours up to some days to test (with or > without participation of the users) applicability to local > constraints/histories/workflow/tools. The more testing is automated and > the less users (or non-default pieces) the faster the update procedure can > be. But there are also instances which did not receive updates for months > or even years. > > On our side we will be facing reality within the next two weeks as we > really need the update. Although preconditions are pretty good (few users, > not that deep modifications, high grade of automation) we expect some > issues. happy to have a virtualized environment... > > To wrap up my answer in a nutshell: no, there is no best practice guide > and no migration tool, but every single contribution is warmly welcome :). > > It would be *very* interesting how updates are handled at Galaxy Main...? > > Cheers, > > Sebastian > (very interested in further feedback) > > > > Hello, > > > > I try to keep our Galaxy instance up to date. > > Very often, some tools disappear and other do not work anymore. This > > breaks some users workflows. > > > > I may miss something. Is there a "best update pratice" to keep a Galaxy > > instance and the tool-sheds fully functionnal ? > > > > Regards, > > > > Thomas > > > > -- > > Thomas Bellembois, Network and System Administrator, IGFL (France) > > http://perso.ens-lyon.fr/thomas.bellembois - +33 4 26 73 13 67 > > (IGFL internal IT doc: http://itdoc.igfl.ens-lyon.fr/itdoc) > > ___ > > Please keep all replies on the list by using "reply all" > > in your mail client. To manage your subscriptions to this > > and other Galaxy lists, please use the interface at: > > http://lists.bx.psu.edu/ > > > > To search Galaxy mailing lists use the unified search at: > > http://galaxyproject.org/search/mailinglists/ > > > > > -- > Sebastian Schaaf, M.Sc. Bioinformatics > Faculty Coordinator NGS Infrastructure > Chair of Biometry and Bioinformatics > Department of Medical Informatics, > Biometry and Epidemiology (IBE) > University of Munich > Marchioninistr. 15, K U1 (postal) > Marchioninistr. 17, U 006 (office) > D-81377 Munich (Germany) > Tel: +49 89 2180-78178 > > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > http://lists.bx.psu.edu/ > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/mailinglists/ > ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Keeping Galaxy up to date
Hello John, I had forgotten that these migration scripts exist. It should work better now. :) Regards, Thomas -- Thomas Bellembois, Network and System Administrator, IGFL (France) http://perso.ens-lyon.fr/thomas.bellembois - +33 4 26 73 13 67 (IGFL internal IT doc: http://itdoc.igfl.ens-lyon.fr/itdoc) On 09/09/2014 15:05, John Chilton wrote: > Hello Thomas, > > After public debate on Trello, there have been a couple tools that > have indeed disappeared completely but nearly all tools that have gone > away have been migrated to the tool shed and can be recovered with the > migration scripts that are included with each new release > (https://wiki.galaxyproject.org/MigratingToolsFromGalaxyDistribution). > Are you running those scripts and still having those problems or were > you not aware of these scripts? > > My understanding is that if you are running these scripts workflows > and tool re-runs should continue to work. (Am I wrong about this?). > > If particular tools don't disappear but no longer work - this sounds > like a bug and you should let the list or Trello know so that the > devteam can attempt to fix the problem or at least be aware it exists. > > Hopefully at some point soon nearly all of the tools will be > migrated out the core release and the devteam will provide some > mechanism to quickly deploy complete say NGS packages all at once to > new releases. Once everything is in the tool shed then hopefully these > problems will go away. (There will be new problems of course - but > hopefully updates will easier). > > -John > > > On Tue, Sep 9, 2014 at 5:49 AM, Thomas Bellembois > wrote: >> Hello, >> >> I try to keep our Galaxy instance up to date. >> Very often, some tools disappear and other do not work anymore. This >> breaks some users workflows. >> >> I may miss something. Is there a "best update pratice" to keep a Galaxy >> instance and the tool-sheds fully functionnal ? >> >> Regards, >> >> Thomas >> >> -- >> Thomas Bellembois, Network and System Administrator, IGFL (France) >> http://perso.ens-lyon.fr/thomas.bellembois - +33 4 26 73 13 67 >> (IGFL internal IT doc: http://itdoc.igfl.ens-lyon.fr/itdoc) >> ___ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this >> and other Galaxy lists, please use the interface at: >> http://lists.bx.psu.edu/ >> >> To search Galaxy mailing lists use the unified search at: >> http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Keeping Galaxy up to date
Hello Thomas, After public debate on Trello, there have been a couple tools that have indeed disappeared completely but nearly all tools that have gone away have been migrated to the tool shed and can be recovered with the migration scripts that are included with each new release (https://wiki.galaxyproject.org/MigratingToolsFromGalaxyDistribution). Are you running those scripts and still having those problems or were you not aware of these scripts? My understanding is that if you are running these scripts workflows and tool re-runs should continue to work. (Am I wrong about this?). If particular tools don't disappear but no longer work - this sounds like a bug and you should let the list or Trello know so that the devteam can attempt to fix the problem or at least be aware it exists. Hopefully at some point soon nearly all of the tools will be migrated out the core release and the devteam will provide some mechanism to quickly deploy complete say NGS packages all at once to new releases. Once everything is in the tool shed then hopefully these problems will go away. (There will be new problems of course - but hopefully updates will easier). -John On Tue, Sep 9, 2014 at 5:49 AM, Thomas Bellembois wrote: > Hello, > > I try to keep our Galaxy instance up to date. > Very often, some tools disappear and other do not work anymore. This > breaks some users workflows. > > I may miss something. Is there a "best update pratice" to keep a Galaxy > instance and the tool-sheds fully functionnal ? > > Regards, > > Thomas > > -- > Thomas Bellembois, Network and System Administrator, IGFL (France) > http://perso.ens-lyon.fr/thomas.bellembois - +33 4 26 73 13 67 > (IGFL internal IT doc: http://itdoc.igfl.ens-lyon.fr/itdoc) > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > http://lists.bx.psu.edu/ > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Keeping Galaxy up to date
Hi Sebastian, Thanks a lot for this detailed answer. I will think about a custom recipe and try to contribute to this topic. Thomas -- Thomas Bellembois, Network and System Administrator, IGFL (France) http://perso.ens-lyon.fr/thomas.bellembois - +33 4 26 73 13 67 (IGFL internal IT doc: http://itdoc.igfl.ens-lyon.fr/itdoc) On 09/09/2014 12:55, Sebastian Schaaf wrote: > Hi Thomas, > > We had this topic at the GCC this year, especially within the Galaxy > Admins BoF. The truth is (please correct me anyone if this is not the case > anymore!) that you are touching an unresolved point. Keeping track of > tools, settings etc. across the versions is not given. People have their > 'home-brew' solutions for this, e.g. keeping a (maybe virtual) machine in > spare in order to invest several hours up to some days to test (with or > without participation of the users) applicability to local > constraints/histories/workflow/tools. The more testing is automated and > the less users (or non-default pieces) the faster the update procedure can > be. But there are also instances which did not receive updates for months > or even years. > > On our side we will be facing reality within the next two weeks as we > really need the update. Although preconditions are pretty good (few users, > not that deep modifications, high grade of automation) we expect some > issues. happy to have a virtualized environment... > > To wrap up my answer in a nutshell: no, there is no best practice guide > and no migration tool, but every single contribution is warmly welcome :). > > It would be *very* interesting how updates are handled at Galaxy Main...? > > Cheers, > > Sebastian > (very interested in further feedback) > > >> Hello, >> >> I try to keep our Galaxy instance up to date. >> Very often, some tools disappear and other do not work anymore. This >> breaks some users workflows. >> >> I may miss something. Is there a "best update pratice" to keep a Galaxy >> instance and the tool-sheds fully functionnal ? >> >> Regards, >> >> Thomas >> >> -- >> Thomas Bellembois, Network and System Administrator, IGFL (France) >> http://perso.ens-lyon.fr/thomas.bellembois - +33 4 26 73 13 67 >> (IGFL internal IT doc: http://itdoc.igfl.ens-lyon.fr/itdoc) >> ___ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this >> and other Galaxy lists, please use the interface at: >> http://lists.bx.psu.edu/ >> >> To search Galaxy mailing lists use the unified search at: >> http://galaxyproject.org/search/mailinglists/ >> > > ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Keeping Galaxy up to date
Hi Thomas, We had this topic at the GCC this year, especially within the Galaxy Admins BoF. The truth is (please correct me anyone if this is not the case anymore!) that you are touching an unresolved point. Keeping track of tools, settings etc. across the versions is not given. People have their 'home-brew' solutions for this, e.g. keeping a (maybe virtual) machine in spare in order to invest several hours up to some days to test (with or without participation of the users) applicability to local constraints/histories/workflow/tools. The more testing is automated and the less users (or non-default pieces) the faster the update procedure can be. But there are also instances which did not receive updates for months or even years. On our side we will be facing reality within the next two weeks as we really need the update. Although preconditions are pretty good (few users, not that deep modifications, high grade of automation) we expect some issues. happy to have a virtualized environment... To wrap up my answer in a nutshell: no, there is no best practice guide and no migration tool, but every single contribution is warmly welcome :). It would be *very* interesting how updates are handled at Galaxy Main...? Cheers, Sebastian (very interested in further feedback) > Hello, > > I try to keep our Galaxy instance up to date. > Very often, some tools disappear and other do not work anymore. This > breaks some users workflows. > > I may miss something. Is there a "best update pratice" to keep a Galaxy > instance and the tool-sheds fully functionnal ? > > Regards, > > Thomas > > -- > Thomas Bellembois, Network and System Administrator, IGFL (France) > http://perso.ens-lyon.fr/thomas.bellembois - +33 4 26 73 13 67 > (IGFL internal IT doc: http://itdoc.igfl.ens-lyon.fr/itdoc) > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > http://lists.bx.psu.edu/ > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/mailinglists/ > -- Sebastian Schaaf, M.Sc. Bioinformatics Faculty Coordinator NGS Infrastructure Chair of Biometry and Bioinformatics Department of Medical Informatics, Biometry and Epidemiology (IBE) University of Munich Marchioninistr. 15, K U1 (postal) Marchioninistr. 17, U 006 (office) D-81377 Munich (Germany) Tel: +49 89 2180-78178 ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
