Re: [galaxy-dev] Once-run galaxy archives
Problem with automation is I could create dozens of templates over the next several years and consume less time in aggregate than it would take me to automate this. Nonetheless, there is a documentation component here that is important so I did enough to document - if someone wants to automate from there feel free. The template is just a copy of the sqlite database after a fresh Galaxy is launched. I usually just do this against whatever development instance of Galaxy I am working on. For completeness though I have put together a script to automate this task against a fresh install (I think): https://github.com/jmchilton/galaxy-downloads/blob/master/build_sqlite_template.sh Good luck! -John On Tue, Jul 22, 2014 at 4:13 PM, Peter Cock p.j.a.c...@googlemail.com wrote: On Tue, Jul 22, 2014 at 7:41 PM, Eric Rasche rasche.e...@yandex.ru wrote: John, How are those generated? Would you be amenable to scripting that portion and running it once a month? (...say in a cron job, with a passwordless ssh key so you never have to touch it again) Cheers, Eric How to generate it was going to be my next question too ;) I'm impressed with Eric's zeal to automate things. Having a script for making the SQLite template would be good - under git in the same repository? Peter P.S. The schema version 120 template works great, thanks!: https://travis-ci.org/peterjc/pico_galaxy/builds/30592828 https://travis-ci.org/peterjc/galaxy_blast/builds/30592097 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Once-run galaxy archives
On Wed, Jul 23, 2014 at 2:42 PM, John Chilton jmchil...@gmail.com wrote: Problem with automation is I could create dozens of templates over the next several years and consume less time in aggregate than it would take me to automate this. Nonetheless, there is a documentation component here that is important so I did enough to document - if someone wants to automate from there feel free. The template is just a copy of the sqlite database after a fresh Galaxy is launched. I usually just do this against whatever development instance of Galaxy I am working on. For completeness though I have put together a script to automate this task against a fresh install (I think): https://github.com/jmchilton/galaxy-downloads/blob/master/build_sqlite_template.sh Good luck! -John Perfect - documentation target achieved :) In terms of speeding up things like TravisCI using these SQLite database templates, refreshing this every few schema bumps should be enough. Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Once-run galaxy archives
On Mon, Jul 21, 2014 at 6:51 PM, Eric Rasche rasche.e...@yandex.ru wrote: Currently the checkout options consist of hg clones, and archives that mercurial produces. Having pulled or cloned galaxy a few times lately, I'm wondering if anyone would have a use for a once-run galaxy instance in an archive? I.e., I'd clone, run once to grab eggs and do the db migration, then re-tar result and store online. Might cut down on build/test times for those who are using travis or other CIs. Thoughts/opinions? Hi Eric, Given how close you can get now for minimal effort, this seem unnecessary. http://blastedbio.blogspot.co.uk/2013/09/using-travis-ci-for-testing-galaxy-tools.html My TravisCI setup this fetches the latest Galaxy as a tar ball (from a GitHub mirror as it was faster than a git clone which was faster than getting the tar ball from BitBucket, which in turn was faster than using hg clone), and a per-migrated SQLite database (using a bit of Galaxy functionality originally with $GALAXY_TEST_DB_TEMPLATE added to speed up running the functional tests). Note this does not cache the eggs and all the other side effects of the first run like creating config files, so there is room for some speed up. Regards, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Once-run galaxy archives
Hi Aaron and Eric, Am 21.07.2014 22:58, schrieb Aaron Petkau: Hello Eric, That sounds like a pretty good idea. If there was a pre-built image available for whatever release I wanted to test against I could just cache it and (hopefully) get my tests running a bit faster. I'm not sure if anyone else is already doing this? Also, I remember there being mentioned pre-building docker images for each release of Galaxy, which would accomplish something similar, but I'm not really sure how that's being handled. I think Björn's Docker image is kept up to date with Galaxy stable each time it's built https://github.com/bgruening/docker-recipes/blob/master/galaxy/Dockerfile#L51. So, this could be handled by modifying his Dockerfile to build Galaxy at whatever tagged release you want to test against. I will try hard to create with every Galaxy stable release a new Galaxy docker image. You can create a github branch with a specific tag, that will end up as a new tagged version of the main Galaxy Docker image. One downside to docker is that you need to get it installed on your CI server, which may or may not be possible (needs a very recent kernel for example). So true. SL7 for the win! :) Docker, Docker, Docker! Bjoern Aaron On Mon, Jul 21, 2014 at 3:12 PM, Eric Rasche rasche.e...@yandex.ru wrote: Hi Aaron, Yeah, absolutely understandable. I want my tools tested early and often. I abuse my CI server for everything, especially for building and packaging software. In this case I was imagining that I might have it produce an archive on every tagged release, as well as producing a daily archive. All of these would be available on some ftp/http server somewhere with symlinks for latest archives (e.g. galaxy-latest-release.tgz and galaxy-latest-daily.tgz). Would that work for your use case as well? Eric On 07/21/2014 03:02 PM, Aaron Petkau wrote: Hello Eric, Your right about that, downloading the archive, installing all the eggs, and then updating the database takes a bit of time (especially if you're like me and like re-running tests on nearly every change you make :P). I think it would be cool to have a pre-package Galaxy for integration testing which is quick to setup. I once thought of downloading Björn's Docker image from Galaxy Bootstrap and using it that way, but thinking is about as far as I got with that one. One problem I could see is it would have to be re-built on every release of Galaxy you want to test against (whereas mercurial cloning/pulling makes sure you're always up to date with the latest code). Aaron On Mon, Jul 21, 2014 at 2:45 PM, Eric Rasche rasche.e...@yandex.ru mailto:rasche.e...@yandex.ru wrote: Hi Aaron, Good points, I was considering using galaxy bootstrap. This is mostly for the CI folk who want to download an archive, unpack it, and be ready to install/test their tools. The hg clone and egg/db steps seem like unnecessary overhead for this specific use case. Cheers, Eric ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Once-run galaxy archives
Hi Björn, On July 22, 2014 3:17:38 AM CDT, Björn Grüning bjoern.gruen...@gmail.com wrote: Hi Aaron and Eric, Am 21.07.2014 22:58, schrieb Aaron Petkau: Hello Eric, That sounds like a pretty good idea. If there was a pre-built image available for whatever release I wanted to test against I could just cache it and (hopefully) get my tests running a bit faster. I'm not sure if anyone else is already doing this? Also, I remember there being mentioned pre-building docker images for each release of Galaxy, which would accomplish something similar, but I'm not really sure how that's being handled. I think Björn's Docker image is kept up to date with Galaxy stable each time it's built https://github.com/bgruening/docker-recipes/blob/master/galaxy/Dockerfile#L51. So, this could be handled by modifying his Dockerfile to build Galaxy at whatever tagged release you want to test against. I will try hard to create with every Galaxy stable release a new Galaxy docker image. You can create a github branch with a specific tag, that will end up as a new tagged version of the main Galaxy Docker image. Try hard to create? No no, what can we do to automatically create these? I'm not so familiar with how one might build a galaxy release specific docker image, but if you can provide a generalized process, let's stick it in a CI server/cron job somewhere and never worry about it again! One downside to docker is that you need to get it installed on your CI server, which may or may not be possible (needs a very recent kernel for example). So true. SL7 for the win! :) Docker, Docker, Docker! Bjoern Aaron On Mon, Jul 21, 2014 at 3:12 PM, Eric Rasche rasche.e...@yandex.ru wrote: Hi Aaron, Yeah, absolutely understandable. I want my tools tested early and often. I abuse my CI server for everything, especially for building and packaging software. In this case I was imagining that I might have it produce an archive on every tagged release, as well as producing a daily archive. All of these would be available on some ftp/http server somewhere with symlinks for latest archives (e.g. galaxy-latest-release.tgz and galaxy-latest-daily.tgz). Would that work for your use case as well? Eric On 07/21/2014 03:02 PM, Aaron Petkau wrote: Hello Eric, Your right about that, downloading the archive, installing all the eggs, and then updating the database takes a bit of time (especially if you're like me and like re-running tests on nearly every change you make :P). I think it would be cool to have a pre-package Galaxy for integration testing which is quick to setup. I once thought of downloading Björn's Docker image from Galaxy Bootstrap and using it that way, but thinking is about as far as I got with that one. One problem I could see is it would have to be re-built on every release of Galaxy you want to test against (whereas mercurial cloning/pulling makes sure you're always up to date with the latest code). Aaron On Mon, Jul 21, 2014 at 2:45 PM, Eric Rasche rasche.e...@yandex.ru mailto:rasche.e...@yandex.ru wrote: Hi Aaron, Good points, I was considering using galaxy bootstrap. This is mostly for the CI folk who want to download an archive, unpack it, and be ready to install/test their tools. The hg clone and egg/db steps seem like unnecessary overhead for this specific use case. Cheers, Eric ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ -- Sent from my Android device with K-9 Mail. Please excuse my brevity. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Once-run galaxy archives
Hi Peter, On July 22, 2014 3:15:41 AM CDT, Peter Cock p.j.a.c...@googlemail.com wrote: On Mon, Jul 21, 2014 at 6:51 PM, Eric Rasche rasche.e...@yandex.ru wrote: Currently the checkout options consist of hg clones, and archives that mercurial produces. Having pulled or cloned galaxy a few times lately, I'm wondering if anyone would have a use for a once-run galaxy instance in an archive? I.e., I'd clone, run once to grab eggs and do the db migration, then re-tar result and store online. Might cut down on build/test times for those who are using travis or other CIs. Thoughts/opinions? Hi Eric, Given how close you can get now for minimal effort, this seem unnecessary. http://blastedbio.blogspot.co.uk/2013/09/using-travis-ci-for-testing-galaxy-tools.html My TravisCI setup this fetches the latest Galaxy as a tar ball (from a GitHub mirror as it was faster than a git clone which was faster than getting the tar ball from BitBucket, which in turn was faster than using Yes, that post was at least part of the thinking behind this. hg clone), and a per-migrated SQLite database (using a bit of Galaxy functionality originally with $GALAXY_TEST_DB_TEMPLATE added to speed up running the functional tests). I know I've seen that used but was never able to get that working in practice (then again I didn't try that hard). If that's a working/usable feature, then that is already the majority of setup time. Eggs would be nice but not the biggest thing in the world. Note this does not cache the eggs and all the other side effects of the first run like creating config files, so there is room for some speed up. Regards, Peter Cheers, Eric -- Sent from my Android device with K-9 Mail. Please excuse my brevity. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Once-run galaxy archives
On Tue, Jul 22, 2014 at 1:15 PM, Eric Rasche rasche.e...@yandex.ru wrote: Hi Peter, On July 22, 2014 3:15:41 AM CDT, Peter Cock p.j.a.c...@googlemail.com wrote: Given how close you can get now for minimal effort, this seem unnecessary. http://blastedbio.blogspot.co.uk/2013/09/using-travis-ci-for-testing-galaxy-tools.html My TravisCI setup this fetches the latest Galaxy as a tar ball (from a GitHub mirror as it was faster than a git clone which was faster than getting the tar ball from BitBucket, which in turn was faster than using hg clone), Yes, that post was at least part of the thinking behind this. :) .., and a per-migrated SQLite database (using a bit of Galaxy functionality originally with $GALAXY_TEST_DB_TEMPLATE added to speed up running the functional tests). Apologies for grammatical error - I pasted in the environment variable at the wrong point in the sentence. I know I've seen that used but was never able to get that working in practice (then again I didn't try that hard). If that's a working/usable feature, then that is already the majority of setup time. Yes, the creation of the test-database and all the migrations was an obvious low-hanging fruit when we were looking at making running the tool functional tests faster - although originally in the context of running the tests on a local development Galaxy instance. As to using this in practise, currently my TravisCI setup has: export GALAXY_TEST_DB_TEMPLATE=https://github.com/jmchilton/galaxy-downloads/raw/master/db_gx_rev_0117.sqlite I also added that line at the start of my local copy of script run_functional_tests.sh to benefit from this while doing development. That should be all there is to it (but from memory, this is only for use with the SQLite backend). John - could you add a current schema snapshot to https://github.com/jmchilton/galaxy-downloads/ please? Note this does not cache the eggs and all the other side effects of the first run like creating config files, so there is room for some speed up. Eggs would be nice but not the biggest thing in the world. Right. I do like your idea of automatically generated cutting-edge or each stable release Docker images though (even if I have no personal need for them at the moment). Regards, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Once-run galaxy archives
Hi Eric, That sounds like a pretty good idea. If there was a pre-built image available for whatever release I wanted to test against I could just cache it and (hopefully) get my tests running a bit faster. I'm not sure if anyone else is already doing this? Also, I remember there being mentioned pre-building docker images for each release of Galaxy, which would accomplish something similar, but I'm not really sure how that's being handled. I think Björn's Docker image is kept up to date with Galaxy stable each time it's built https://github.com/bgruening/docker-recipes/blob/master/galaxy/Dockerfile#L51. So, this could be handled by modifying his Dockerfile to build Galaxy at whatever tagged release you want to test against. I will try hard to create with every Galaxy stable release a new Galaxy docker image. You can create a github branch with a specific tag, that will end up as a new tagged version of the main Galaxy Docker image. Try hard to create? No no, what can we do to automatically create these? I'm not so familiar with how one might build a galaxy release specific docker image, but if you can provide a generalized process, let's stick it in a CI server/cron job somewhere and never worry about it again! The hardest part is to remind myself ;) The procedure is: 1) create a new branch for the galaxy-docker github account 2) change the version string in the git-clone command in the Dockerfile 3) login into the docker-index site and re-associate a new tag to the new branch ... click the build button I could simplify that a little bit if the galaxy-docker image has its own repository. The docker-index has a build-on-push feature. But currently every image (all branches) are build again. There is no way to only trigger one branch build. Until that is fixed in the docker-index I will do it manually. So you see any improvements in that setup? Let me know! Cheers, Bjoern One downside to docker is that you need to get it installed on your CI server, which may or may not be possible (needs a very recent kernel for example). So true. SL7 for the win! :) Docker, Docker, Docker! Bjoern Aaron On Mon, Jul 21, 2014 at 3:12 PM, Eric Rasche rasche.e...@yandex.ru wrote: Hi Aaron, Yeah, absolutely understandable. I want my tools tested early and often. I abuse my CI server for everything, especially for building and packaging software. In this case I was imagining that I might have it produce an archive on every tagged release, as well as producing a daily archive. All of these would be available on some ftp/http server somewhere with symlinks for latest archives (e.g. galaxy-latest-release.tgz and galaxy-latest-daily.tgz). Would that work for your use case as well? Eric On 07/21/2014 03:02 PM, Aaron Petkau wrote: Hello Eric, Your right about that, downloading the archive, installing all the eggs, and then updating the database takes a bit of time (especially if you're like me and like re-running tests on nearly every change you make :P). I think it would be cool to have a pre-package Galaxy for integration testing which is quick to setup. I once thought of downloading Björn's Docker image from Galaxy Bootstrap and using it that way, but thinking is about as far as I got with that one. One problem I could see is it would have to be re-built on every release of Galaxy you want to test against (whereas mercurial cloning/pulling makes sure you're always up to date with the latest code). Aaron On Mon, Jul 21, 2014 at 2:45 PM, Eric Rasche rasche.e...@yandex.ru mailto:rasche.e...@yandex.ru wrote: Hi Aaron, Good points, I was considering using galaxy bootstrap. This is mostly for the CI folk who want to download an archive, unpack it, and be ready to install/test their tools. The hg clone and egg/db steps seem like unnecessary overhead for this specific use case. Cheers, Eric ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Once-run galaxy archives
Hi Björn, 22.07.2014, 14:26, Björn Grüning bjoern.gruen...@gmail.com: Hi Eric, That sounds like a pretty good idea. If there was a pre-built image available for whatever release I wanted to test against I could just cache it and (hopefully) get my tests running a bit faster. I'm not sure if anyone else is already doing this? Also, I remember there being mentioned pre-building docker images for each release of Galaxy, which would accomplish something similar, but I'm not really sure how that's being handled. I think Björn's Docker image is kept up to date with Galaxy stable each time it's built https://github.com/bgruening/docker-recipes/blob/master/galaxy/Dockerfile#L51. So, this could be handled by modifying his Dockerfile to build Galaxy at whatever tagged release you want to test against. I will try hard to create with every Galaxy stable release a new Galaxy docker image. You can create a github branch with a specific tag, that will end up as a new tagged version of the main Galaxy Docker image. Try hard to create? No no, what can we do to automatically create these? I'm not so familiar with how one might build a galaxy release specific docker image, but if you can provide a generalized process, let's stick it in a CI server/cron job somewhere and never worry about it again! The hardest part is to remind myself ;) The procedure is: 1) create a new branch for the galaxy-docker github account 2) change the version string in the git-clone command in the Dockerfile 3) login into the docker-index site and re-associate a new tag to the new branch ... click the build button I could simplify that a little bit if the galaxy-docker image has its own repository. The docker-index has a build-on-push feature. But currently every image (all branches) are build again. There is no way to only trigger one branch build. Until that is fixed in the docker-index I will do it manually. So you see any improvements in that setup? Let me know! Naturally! 1 and 2 could be automated out with a script. 3 could probably be fixed with a script, but that requires parsing pages/crafting cURL queries and that's less pleasant. Let's have a new repository just for galaxy-docker images. I'll write up a script to check for updates and create+push branches as needed, we can place this in a cron/CI job and have it email you whenever it's run to say hey, associate the branch/trigger a build Cheers, Bjoern One downside to docker is that you need to get it installed on your CI server, which may or may not be possible (needs a very recent kernel for example). So true. SL7 for the win! :) Docker, Docker, Docker! Bjoern Aaron On Mon, Jul 21, 2014 at 3:12 PM, Eric Rasche rasche.e...@yandex.ru wrote: Hi Aaron, Yeah, absolutely understandable. I want my tools tested early and often. I abuse my CI server for everything, especially for building and packaging software. In this case I was imagining that I might have it produce an archive on every tagged release, as well as producing a daily archive. All of these would be available on some ftp/http server somewhere with symlinks for latest archives (e.g. galaxy-latest-release.tgz and galaxy-latest-daily.tgz). Would that work for your use case as well? Eric On 07/21/2014 03:02 PM, Aaron Petkau wrote: Hello Eric, Your right about that, downloading the archive, installing all the eggs, and then updating the database takes a bit of time (especially if you're like me and like re-running tests on nearly every change you make :P). I think it would be cool to have a pre-package Galaxy for integration testing which is quick to setup. I once thought of downloading Björn's Docker image from Galaxy Bootstrap and using it that way, but thinking is about as far as I got with that one. One problem I could see is it would have to be re-built on every release of Galaxy you want to test against (whereas mercurial cloning/pulling makes sure you're always up to date with the latest code). Aaron On Mon, Jul 21, 2014 at 2:45 PM, Eric Rasche rasche.e...@yandex.ru mailto:rasche.e...@yandex.ru wrote: Hi Aaron, Good points, I was considering using galaxy bootstrap. This is mostly for the CI folk who want to download an archive, unpack it, and be ready to install/test their tools. The hg clone and egg/db steps seem like unnecessary overhead for this specific use case. Cheers, Eric ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use
Re: [galaxy-dev] Once-run galaxy archives
:) great I like it! Will do it shortly! Am 22.07.2014 15:36, schrieb Eric Rasche: Hi Björn, 22.07.2014, 14:26, Björn Grüning bjoern.gruen...@gmail.com: Hi Eric, That sounds like a pretty good idea. If there was a pre-built image available for whatever release I wanted to test against I could just cache it and (hopefully) get my tests running a bit faster. I'm not sure if anyone else is already doing this? Also, I remember there being mentioned pre-building docker images for each release of Galaxy, which would accomplish something similar, but I'm not really sure how that's being handled. I think Björn's Docker image is kept up to date with Galaxy stable each time it's built https://github.com/bgruening/docker-recipes/blob/master/galaxy/Dockerfile#L51. So, this could be handled by modifying his Dockerfile to build Galaxy at whatever tagged release you want to test against. I will try hard to create with every Galaxy stable release a new Galaxy docker image. You can create a github branch with a specific tag, that will end up as a new tagged version of the main Galaxy Docker image. Try hard to create? No no, what can we do to automatically create these? I'm not so familiar with how one might build a galaxy release specific docker image, but if you can provide a generalized process, let's stick it in a CI server/cron job somewhere and never worry about it again! The hardest part is to remind myself ;) The procedure is: 1) create a new branch for the galaxy-docker github account 2) change the version string in the git-clone command in the Dockerfile 3) login into the docker-index site and re-associate a new tag to the new branch ... click the build button I could simplify that a little bit if the galaxy-docker image has its own repository. The docker-index has a build-on-push feature. But currently every image (all branches) are build again. There is no way to only trigger one branch build. Until that is fixed in the docker-index I will do it manually. So you see any improvements in that setup? Let me know! Naturally! 1 and 2 could be automated out with a script. 3 could probably be fixed with a script, but that requires parsing pages/crafting cURL queries and that's less pleasant. Let's have a new repository just for galaxy-docker images. I'll write up a script to check for updates and create+push branches as needed, we can place this in a cron/CI job and have it email you whenever it's run to say hey, associate the branch/trigger a build Cheers, Bjoern One downside to docker is that you need to get it installed on your CI server, which may or may not be possible (needs a very recent kernel for example). So true. SL7 for the win! :) Docker, Docker, Docker! Bjoern Aaron On Mon, Jul 21, 2014 at 3:12 PM, Eric Rasche rasche.e...@yandex.ru wrote: Hi Aaron, Yeah, absolutely understandable. I want my tools tested early and often. I abuse my CI server for everything, especially for building and packaging software. In this case I was imagining that I might have it produce an archive on every tagged release, as well as producing a daily archive. All of these would be available on some ftp/http server somewhere with symlinks for latest archives (e.g. galaxy-latest-release.tgz and galaxy-latest-daily.tgz). Would that work for your use case as well? Eric On 07/21/2014 03:02 PM, Aaron Petkau wrote: Hello Eric, Your right about that, downloading the archive, installing all the eggs, and then updating the database takes a bit of time (especially if you're like me and like re-running tests on nearly every change you make :P). I think it would be cool to have a pre-package Galaxy for integration testing which is quick to setup. I once thought of downloading Björn's Docker image from Galaxy Bootstrap and using it that way, but thinking is about as far as I got with that one. One problem I could see is it would have to be re-built on every release of Galaxy you want to test against (whereas mercurial cloning/pulling makes sure you're always up to date with the latest code). Aaron On Mon, Jul 21, 2014 at 2:45 PM, Eric Rasche rasche.e...@yandex.ru mailto:rasche.e...@yandex.ru wrote: Hi Aaron, Good points, I was considering using galaxy bootstrap. This is mostly for the CI folk who want to download an archive, unpack it, and be ready to install/test their tools. The hg clone and egg/db steps seem like unnecessary overhead for this specific use case. Cheers, Eric ___ Please keep all replies on the list by using reply all in your mail client. To manage your
Re: [galaxy-dev] Once-run galaxy archives
That's awesome Björn and Eric. And, I'll also have to go through your Travis CI document Peter. It looks really cool. Aaron On Tue, Jul 22, 2014 at 9:35 AM, Björn Grüning bjoern.gruen...@gmail.com wrote: :) great I like it! Will do it shortly! Am 22.07.2014 15:36, schrieb Eric Rasche: Hi Björn, 22.07.2014, 14:26, Björn Grüning bjoern.gruen...@gmail.com: Hi Eric, That sounds like a pretty good idea. If there was a pre-built image available for whatever release I wanted to test against I could just cache it and (hopefully) get my tests running a bit faster. I'm not sure if anyone else is already doing this? Also, I remember there being mentioned pre-building docker images for each release of Galaxy, which would accomplish something similar, but I'm not really sure how that's being handled. I think Björn's Docker image is kept up to date with Galaxy stable each time it's built https://github.com/bgruening/docker-recipes/blob/master/ galaxy/Dockerfile#L51. So, this could be handled by modifying his Dockerfile to build Galaxy at whatever tagged release you want to test against. I will try hard to create with every Galaxy stable release a new Galaxy docker image. You can create a github branch with a specific tag, that will end up as a new tagged version of the main Galaxy Docker image. Try hard to create? No no, what can we do to automatically create these? I'm not so familiar with how one might build a galaxy release specific docker image, but if you can provide a generalized process, let's stick it in a CI server/cron job somewhere and never worry about it again! The hardest part is to remind myself ;) The procedure is: 1) create a new branch for the galaxy-docker github account 2) change the version string in the git-clone command in the Dockerfile 3) login into the docker-index site and re-associate a new tag to the new branch ... click the build button I could simplify that a little bit if the galaxy-docker image has its own repository. The docker-index has a build-on-push feature. But currently every image (all branches) are build again. There is no way to only trigger one branch build. Until that is fixed in the docker-index I will do it manually. So you see any improvements in that setup? Let me know! Naturally! 1 and 2 could be automated out with a script. 3 could probably be fixed with a script, but that requires parsing pages/crafting cURL queries and that's less pleasant. Let's have a new repository just for galaxy-docker images. I'll write up a script to check for updates and create+push branches as needed, we can place this in a cron/CI job and have it email you whenever it's run to say hey, associate the branch/trigger a build Cheers, Bjoern One downside to docker is that you need to get it installed on your CI server, which may or may not be possible (needs a very recent kernel for example). So true. SL7 for the win! :) Docker, Docker, Docker! Bjoern Aaron On Mon, Jul 21, 2014 at 3:12 PM, Eric Rasche rasche.e...@yandex.ru wrote: Hi Aaron, Yeah, absolutely understandable. I want my tools tested early and often. I abuse my CI server for everything, especially for building and packaging software. In this case I was imagining that I might have it produce an archive on every tagged release, as well as producing a daily archive. All of these would be available on some ftp/http server somewhere with symlinks for latest archives (e.g. galaxy-latest-release.tgz and galaxy-latest-daily.tgz). Would that work for your use case as well? Eric On 07/21/2014 03:02 PM, Aaron Petkau wrote: Hello Eric, Your right about that, downloading the archive, installing all the eggs, and then updating the database takes a bit of time (especially if you're like me and like re-running tests on nearly every change you make :P). I think it would be cool to have a pre-package Galaxy for integration testing which is quick to setup. I once thought of downloading Björn's Docker image from Galaxy Bootstrap and using it that way, but thinking is about as far as I got with that one. One problem I could see is it would have to be re-built on every release of Galaxy you want to test against (whereas mercurial cloning/pulling makes sure you're always up to date with the latest code). Aaron On Mon, Jul 21, 2014 at 2:45 PM, Eric Rasche rasche.e...@yandex.ru mailto:rasche.e...@yandex.ru wrote: Hi Aaron, Good points, I was considering using galaxy bootstrap. This is mostly for the CI folk who want to download an archive, unpack it, and be ready to
Re: [galaxy-dev] Once-run galaxy archives
For my part:Script/infra to generate brancheshttps://cpt.tamu.edu/gitlab/rasche.eric/docker-branch-generator/tree/masterGenerated branches:https://cpt.tamu.edu/gitlab/rasche.eric/generated-docker-branches/branches/recentNeed to patch up a couple issues, but I'm pretty much done on my end. Whenever you're ready, we can move this to github and someone can just run a cron job once a month. :)22.07.2014, 15:35, "Björn Grüning" bjoern.gruen...@gmail.com: :) great I like it! Will do it shortly! Am 22.07.2014 15:36, schrieb Eric Rasche: Hi Björn, 22.07.2014, 14:26, "Björn Grüning" bjoern.gruen...@gmail.com:Hi Eric, That sounds like a pretty good idea. If there was a pre-built image available for whatever release I wanted to test against I could just cache it and (hopefully) get my tests running a bit faster. I'm not sure if anyone else is already doing this? Also, I remember there being mentioned pre-building docker images for each release of Galaxy, which would accomplish something similar, but I'm not really sure how that's being handled. I think Björn's Docker image is kept up to date with Galaxy stable each time it's built https://github.com/bgruening/docker-recipes/blob/master/galaxy/Dockerfile#L51. So, this could be handled by modifying his Dockerfile to build Galaxy at whatever tagged release you want to test against. I will try hard to create with every Galaxy stable release a new Galaxy docker image. You can create a github branch with a specific tag, that will end up as a new tagged version of the main Galaxy Docker image. Try hard to create? No no, what can we do to automatically create these? I'm not so familiar with how one might build a galaxy release specific docker image, but if you can provide a generalized process, let's stick it in a CI server/cron job somewhere and never worry about it again!The hardest part is to remind myself ;)The procedure is:1) create a new branch for the galaxy-docker github account2) change the version string in the git-clone command in the Dockerfile3) login into the docker-index site and re-associate a new tag to thenew branch ... click the build buttonI could simplify that a little bit if the galaxy-docker image has itsown repository. The docker-index has a build-on-push feature.But currently every image (all branches) are build again. There is noway to only trigger one branch build. Until that is fixed in thedocker-index I will do it manually.So you see any improvements in that setup? Let me know! Naturally! 1 and 2 could be automated out with a script. 3 could probably be fixed with a script, but that requires parsing pages/crafting cURL queries and that's less pleasant. Let's have a new repository just for galaxy-docker images. I'll write up a script to check for updates and create+push branches as needed, we can place this in a cron/CI job and have it email you whenever it's run to say "hey, associate the branch/trigger a build"Cheers,Bjoern One downside to docker is that you need to get it installed on your CI server, which may or may not be possible (needs a very recent kernel for example). So true. SL7 for the win! :) Docker, Docker, Docker! Bjoern Aaron On Mon, Jul 21, 2014 at 3:12 PM, Eric Rasche rasche.e...@yandex.ru wrote: Hi Aaron, Yeah, absolutely understandable. I want my tools tested early and often. I abuse my CI server for everything, especially for building and packaging software. In this case I was imagining that I might have it produce an archive on every tagged release, as well as producing a "daily" archive. All of these would be available on some ftp/http server somewhere with symlinks for latest archives (e.g. galaxy-latest-release.tgz and galaxy-latest-daily.tgz). Would that work for your use case as well? Eric On 07/21/2014 03:02 PM, Aaron Petkau wrote: Hello Eric, Your right about that, downloading the archive, installing all the eggs, and then updating the database takes a bit of time (especially if you're like me and like re-running tests on nearly every change you make :P). I think it would be cool to have a pre-package Galaxy for integration testing which is quick to setup. I once thought of downloading Björn's Docker image from Galaxy Bootstrap and using it that way, but thinking is about as far as I got with that one. One problem I could see is it would have to be re-built on every release of Galaxy you want to test against (whereas mercurial cloning/pulling makes sure you're always up to date with the latest code). Aaron On Mon, Jul 21, 2014 at 2:45 PM, Eric Rasche rasche.e...@yandex.ru mailto:rasche.e...@yandex.ru wrote: Hi Aaron, Good points, I was considering using
Re: [galaxy-dev] Once-run galaxy archives
On Tue, Jul 22, 2014 at 7:51 AM, Peter Cock p.j.a.c...@googlemail.com wrote: On Tue, Jul 22, 2014 at 1:15 PM, Eric Rasche rasche.e...@yandex.ru wrote: Hi Peter, On July 22, 2014 3:15:41 AM CDT, Peter Cock p.j.a.c...@googlemail.com wrote: Given how close you can get now for minimal effort, this seem unnecessary. http://blastedbio.blogspot.co.uk/2013/09/using-travis-ci-for-testing-galaxy-tools.html My TravisCI setup this fetches the latest Galaxy as a tar ball (from a GitHub mirror as it was faster than a git clone which was faster than getting the tar ball from BitBucket, which in turn was faster than using hg clone), Yes, that post was at least part of the thinking behind this. :) .., and a per-migrated SQLite database (using a bit of Galaxy functionality originally with $GALAXY_TEST_DB_TEMPLATE added to speed up running the functional tests). Apologies for grammatical error - I pasted in the environment variable at the wrong point in the sentence. I know I've seen that used but was never able to get that working in practice (then again I didn't try that hard). If that's a working/usable feature, then that is already the majority of setup time. Yes, the creation of the test-database and all the migrations was an obvious low-hanging fruit when we were looking at making running the tool functional tests faster - although originally in the context of running the tests on a local development Galaxy instance. As to using this in practise, currently my TravisCI setup has: export GALAXY_TEST_DB_TEMPLATE=https://github.com/jmchilton/galaxy-downloads/raw/master/db_gx_rev_0117.sqlite I also added that line at the start of my local copy of script run_functional_tests.sh to benefit from this while doing development. That should be all there is to it (but from memory, this is only for use with the SQLite backend). John - could you add a current schema snapshot to https://github.com/jmchilton/galaxy-downloads/ please? Hey All, Love this thread and effort! Keep up the good work - would love to replace say blend4j's automatic travisci testing to be backed by dockerized -stable and -central instance. At any rate, I have uploaded a more updated sqlite template: https://github.com/jmchilton/galaxy-downloads/raw/master/db_gx_rev_0120.sqlite. The old template still exists at the same URL so hopefully this doesn't break anything. -John Note this does not cache the eggs and all the other side effects of the first run like creating config files, so there is room for some speed up. Eggs would be nice but not the biggest thing in the world. Right. I do like your idea of automatically generated cutting-edge or each stable release Docker images though (even if I have no personal need for them at the moment). Regards, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Once-run galaxy archives
John, How are those generated? Would you be amenable to scripting that portion and running it once a month? (...say in a cron job, with a passwordless ssh key so you never have to touch it again) Cheers, Eric 22.07.2014, 19:08, John Chilton jmchil...@gmail.com: On Tue, Jul 22, 2014 at 7:51 AM, Peter Cock p.j.a.c...@googlemail.com wrote: On Tue, Jul 22, 2014 at 1:15 PM, Eric Rasche rasche.e...@yandex.ru wrote: Hi Peter, On July 22, 2014 3:15:41 AM CDT, Peter Cock p.j.a.c...@googlemail.com wrote: Given how close you can get now for minimal effort, this seem unnecessary. http://blastedbio.blogspot.co.uk/2013/09/using-travis-ci-for-testing-galaxy-tools.html My TravisCI setup this fetches the latest Galaxy as a tar ball (from a GitHub mirror as it was faster than a git clone which was faster than getting the tar ball from BitBucket, which in turn was faster than using hg clone), Yes, that post was at least part of the thinking behind this. :) .., and a per-migrated SQLite database (using a bit of Galaxy functionality originally with $GALAXY_TEST_DB_TEMPLATE added to speed up running the functional tests). Apologies for grammatical error - I pasted in the environment variable at the wrong point in the sentence. I know I've seen that used but was never able to get that working in practice (then again I didn't try that hard). If that's a working/usable feature, then that is already the majority of setup time. Yes, the creation of the test-database and all the migrations was an obvious low-hanging fruit when we were looking at making running the tool functional tests faster - although originally in the context of running the tests on a local development Galaxy instance. As to using this in practise, currently my TravisCI setup has: export GALAXY_TEST_DB_TEMPLATE=https://github.com/jmchilton/galaxy-downloads/raw/master/db_gx_rev_0117.sqlite I also added that line at the start of my local copy of script run_functional_tests.sh to benefit from this while doing development. That should be all there is to it (but from memory, this is only for use with the SQLite backend). John - could you add a current schema snapshot to https://github.com/jmchilton/galaxy-downloads/ please? Hey All, Love this thread and effort! Keep up the good work - would love to replace say blend4j's automatic travisci testing to be backed by dockerized -stable and -central instance. At any rate, I have uploaded a more updated sqlite template: https://github.com/jmchilton/galaxy-downloads/raw/master/db_gx_rev_0120.sqlite. The old template still exists at the same URL so hopefully this doesn't break anything. -John Note this does not cache the eggs and all the other side effects of the first run like creating config files, so there is room for some speed up. Eggs would be nice but not the biggest thing in the world. Right. I do like your idea of automatically generated cutting-edge or each stable release Docker images though (even if I have no personal need for them at the moment). Regards, Peter -- Eric Rasche Programmer II Center for Phage Technology Texas AM Univesity College Station, TX 77843 Ph: 4046922048 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Once-run galaxy archives
On Tue, Jul 22, 2014 at 7:41 PM, Eric Rasche rasche.e...@yandex.ru wrote: John, How are those generated? Would you be amenable to scripting that portion and running it once a month? (...say in a cron job, with a passwordless ssh key so you never have to touch it again) Cheers, Eric How to generate it was going to be my next question too ;) I'm impressed with Eric's zeal to automate things. Having a script for making the SQLite template would be good - under git in the same repository? Peter P.S. The schema version 120 template works great, thanks!: https://travis-ci.org/peterjc/pico_galaxy/builds/30592828 https://travis-ci.org/peterjc/galaxy_blast/builds/30592097 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Once-run galaxy archives
Hello Eric, I think that's a great idea. You may be interested in checking out Björn's Galaxy Docker image https://github.com/bgruening/docker-recipes/tree/master/galaxy which is very similar to what you're describing. A few other testing frameworks, like Galaxy bootstrap https://github.com/jmchilton/galaxy-bootstrap, use there own caching mechanism too. Aaron On Mon, Jul 21, 2014 at 12:51 PM, Eric Rasche rasche.e...@yandex.ru wrote: Currently the checkout options consist of hg clones, and archives that mercurial produces. Having pulled or cloned galaxy a few times lately, I'm wondering if anyone would have a use for a once-run galaxy instance in an archive? I.e., I'd clone, run once to grab eggs and do the db migration, then re-tar result and store online. Might cut down on build/test times for those who are using travis or other CIs. Thoughts/opinions? Cheers, Eric -- Eric Rasche Programmer II Center for Phage Technology Texas AM Univesity College Station, TX 77843 Ph: 4046922048 http://tel:4046922048 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Once-run galaxy archives
Hi Aaron,Good points, I was considering using galaxy bootstrap. This is mostly for the CI folk who want to download an archive, unpack it, and be ready to install/test their tools. The hg clone and egg/db steps seem like unnecessary overhead for this specific use case.Cheers,Eric ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Once-run galaxy archives
Hi Aaron, Yeah, absolutely understandable. I want my tools tested early and often. I abuse my CI server for everything, especially for building and packaging software. In this case I was imagining that I might have it produce an archive on every tagged release, as well as producing a daily archive. All of these would be available on some ftp/http server somewhere with symlinks for latest archives (e.g. galaxy-latest-release.tgz and galaxy-latest-daily.tgz). Would that work for your use case as well? Eric On 07/21/2014 03:02 PM, Aaron Petkau wrote: Hello Eric, Your right about that, downloading the archive, installing all the eggs, and then updating the database takes a bit of time (especially if you're like me and like re-running tests on nearly every change you make :P). I think it would be cool to have a pre-package Galaxy for integration testing which is quick to setup. I once thought of downloading Björn's Docker image from Galaxy Bootstrap and using it that way, but thinking is about as far as I got with that one. One problem I could see is it would have to be re-built on every release of Galaxy you want to test against (whereas mercurial cloning/pulling makes sure you're always up to date with the latest code). Aaron On Mon, Jul 21, 2014 at 2:45 PM, Eric Rasche rasche.e...@yandex.ru mailto:rasche.e...@yandex.ru wrote: Hi Aaron, Good points, I was considering using galaxy bootstrap. This is mostly for the CI folk who want to download an archive, unpack it, and be ready to install/test their tools. The hg clone and egg/db steps seem like unnecessary overhead for this specific use case. Cheers, Eric ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Once-run galaxy archives
Hello Eric, That sounds like a pretty good idea. If there was a pre-built image available for whatever release I wanted to test against I could just cache it and (hopefully) get my tests running a bit faster. I'm not sure if anyone else is already doing this? Also, I remember there being mentioned pre-building docker images for each release of Galaxy, which would accomplish something similar, but I'm not really sure how that's being handled. I think Björn's Docker image is kept up to date with Galaxy stable each time it's built https://github.com/bgruening/docker-recipes/blob/master/galaxy/Dockerfile#L51. So, this could be handled by modifying his Dockerfile to build Galaxy at whatever tagged release you want to test against. One downside to docker is that you need to get it installed on your CI server, which may or may not be possible (needs a very recent kernel for example). Aaron On Mon, Jul 21, 2014 at 3:12 PM, Eric Rasche rasche.e...@yandex.ru wrote: Hi Aaron, Yeah, absolutely understandable. I want my tools tested early and often. I abuse my CI server for everything, especially for building and packaging software. In this case I was imagining that I might have it produce an archive on every tagged release, as well as producing a daily archive. All of these would be available on some ftp/http server somewhere with symlinks for latest archives (e.g. galaxy-latest-release.tgz and galaxy-latest-daily.tgz). Would that work for your use case as well? Eric On 07/21/2014 03:02 PM, Aaron Petkau wrote: Hello Eric, Your right about that, downloading the archive, installing all the eggs, and then updating the database takes a bit of time (especially if you're like me and like re-running tests on nearly every change you make :P). I think it would be cool to have a pre-package Galaxy for integration testing which is quick to setup. I once thought of downloading Björn's Docker image from Galaxy Bootstrap and using it that way, but thinking is about as far as I got with that one. One problem I could see is it would have to be re-built on every release of Galaxy you want to test against (whereas mercurial cloning/pulling makes sure you're always up to date with the latest code). Aaron On Mon, Jul 21, 2014 at 2:45 PM, Eric Rasche rasche.e...@yandex.ru mailto:rasche.e...@yandex.ru wrote: Hi Aaron, Good points, I was considering using galaxy bootstrap. This is mostly for the CI folk who want to download an archive, unpack it, and be ready to install/test their tools. The hg clone and egg/db steps seem like unnecessary overhead for this specific use case. Cheers, Eric ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
