Re: [galaxy-dev] Nice 'citable' URLs for Galaxy Tool Shed repositories
On Sat, Feb 2, 2013 at 1:56 PM, Peter Cock p.j.a.c...@googlemail.com wrote: On Sat, Feb 2, 2013 at 12:46 AM, Greg Von Kuster g...@bx.psu.edu wrote: Hello Peter, I've added an enhanced version of your implementation for citable URLS in the tool shed to change set revision 8720:e27d0dd12752. This is currently running on the test tool shed, and viewing repositories now displays a link like this: I've used the same routes you set up, so both of the following work: http://testtoolshed.g2.bx.psu.edu/view/peterjc/mira_assembler http://testtoolshed.g2.bx.psu.edu/view/peterjc/ i was under a rush to get this in before the next release cutoff. There is still some work to do on this (e.g., we can add additions routes for citing a specific repository revision, etc) but this should be a good start and it will be in the next Galaxy release currently scheduled for next week. Thanks very much for your contribution on this and please let me know if you encounter any issues or have any additional suggestions. Greg Von Kuster Quick work :) I've noticed one oddity, which is if I go one of the citable URLs like http://testtoolshed.g2.bx.psu.edu/view/peterjc/mira_assembler and then browse away to another section/repository/etc the URL in the browser's address bar does not update. You can be looking at repository B, but the address bar URL still says repository A. (This was one reason I stuck a redirect in my prototype). Do you think this going to be easy to fix, or should we revert to the redirect trick to avoid this 'stale' URL problem? Separately, but perhaps related, it would be nice if via the search or otherwise, the new URLs were automatically used - that would be slightly easier than copying it from the text of the page. However, this is already functional enough to start sharing direct links. Once this goes live, you'll have to brief the whoever writes the new tool alerts for Twitter to use it :) I see the new citable URLs are already in use on the wiki (but not yet working as the live ToolShed doesn't have this update yet): http://wiki.galaxyproject.org/ToolShedToolFeatures#Example_repositories_in_the_main_Galaxy_Tool_Shed_that_define_tool_dependencies Regards, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Nice 'citable' URLs for Galaxy Tool Shed repositories
Hi Peter, On Feb 8, 2013, at 5:26 AM, Peter Cock wrote: On Sat, Feb 2, 2013 at 1:56 PM, Peter Cock p.j.a.c...@googlemail.com wrote: I've noticed one oddity, which is if I go one of the citable URLs like http://testtoolshed.g2.bx.psu.edu/view/peterjc/mira_assembler and then browse away to another section/repository/etc the URL in the browser's address bar does not update. You can be looking at repository B, but the address bar URL still says repository A. (This was one reason I stuck a redirect in my prototype). Do you think this going to be easy to fix, or should we revert to the redirect trick to avoid this 'stale' URL problem? This shouldn't be too difficult to fix. There actually is still a redirect, but it comes after the panel layout. I'll take a look at this today. I see the new citable URLs are already in use on the wiki (but not yet working as the live ToolShed doesn't have this update yet): There is a release scheduled for today, so the main Galaxy tool shed will include this new feature. I always have the wiki updated just before the release. http://wiki.galaxyproject.org/ToolShedToolFeatures#Example_repositories_in_the_main_Galaxy_Tool_Shed_that_define_tool_dependencies Regards, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Fwd: Desire to contribute
On 06/02/13 20:23, Dave Clements wrote: Please let the list know if any of the ideas posted so far grab you, or if you want further explanation. Thanks for your interest and for picking the Galaxy Project. Efforts like these really help the project move forward. Are there any lists that are more developer-oriented than this one? There seems to be a mixture of end-user problem reports and tool-specific integration discussions on this list, but not so much discussion of framework development, or at least it can be hard to tune into the latter. Paul ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Dependencies on other Tool Shed repositories
Hi all, I've been reading http://wiki.galaxyproject.org/ToolShedToolFeatures and focused on the example of EMBOSS where the tool definitions depend on the data type definitions. Here the emboss_5 repository (the tool wrappers) contains depository_dependencies.xml saying: ?xml version=1.0? repositories description=Emboss 5 requires the Galaxy applicable data formats used by Emboss tools. repository toolshed=http://toolshed.g2.bx.psu.edu; name=emboss_datatypes owner=devteam changeset_revision=a89163f31369 / /repositories Is that revision treated as a minimum version, or an absolute one? This concerns me as I have three repositories which will all depend on the blast_datatypes repository for the 'blastxml' format (etc) - so what happens when these three repositories don't all request the same revision of the blast_datatypes repository? What I'm using for now is: ?xml version=1.0? repositories description=This requires the BLAST datatype definitions (e.g. the BLAST XML format). !-- Revision 4:f9a7783ed7b6 on the main tool shed is v0.0.14 which added BLAST databases -- repository toolshed=http://toolshed.g2.bx.psu.edu; name=blast_datatypes owner=devteam changeset_revision=f9a7783ed7b6 / /repositories Regards, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] [galaxy-user] Why SGE needed for galaxy ?
The workers don't need their own copy of galaxy installed, but a shared filesystem is a requirement for galaxy (in any cluster environment -- see the galaxy wiki for more http://wiki.galaxyproject.org/Admin/Config/Performance/Cluster). Cloudman handles managing NFS for you and sharing the galaxy/tools/index/data volumes. In order for workers to communicate with the master instance, they'll need the cloudman installation as well, so you should use the same image. Now that I've answered that, I'm not sure I totally understand your proposed installation yet, but if you're suggesting bypassing cloudman for installation on a private cloud it should be possible. You'd want the master instance up full time running as the galaxy front end, dispatching jobs to a separate cluster managed by SGE/PBS/whatever. Basically the standard cluster configuration outlined in the wiki above, but you'd want your worker nodes automatically configured to mount the shared directories and join the PBS/SGE queue so they could handle jobs. Depending on what type of private cloud you're working with, it might be easier to just see if you can get cloudman to work :) Lastly, I swapped this message to galaxy-dev since it's about installation nuts and bolts. -Dannon On Feb 8, 2013, at 3:02 AM, Zeeshan Ali Shah zas...@pdc.kth.se wrote: Dear Enis, thanks for reply and being you as cloudman developer it is good to see you in the list . Q2: On Workers node we need galaxy installed with its shared directories ? like galaxyindices , galaxydata Q3: For a private cloud setup do you prefare to have a master image with cloudman and galaxy and use the same image for workers as well ? or worker images can be vanilla OS ? BR Zeeshan On W6-Feb 7, 2013, at 11:50 PM, Enis Afgan wrote: Hi Zeeshan, In order to gain from the scalability of the cloud, SGE does need to run. However, CloudMan sets all that up and manages it going forward. Enis On Fri, Feb 8, 2013 at 8:59 AM, Zeeshan Ali Shah zas...@pdc.kth.se wrote: Hi, It seems that cloud man need SGE for scaling . Does SGE need also when run cloud on private cloud ? Zeeshan ___ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using reply all in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list: http://lists.bx.psu.edu/listinfo/galaxy-dev To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using reply all in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list: http://lists.bx.psu.edu/listinfo/galaxy-dev To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Fwd: Desire to contribute
On Feb 8, 2013, at 7:03 AM, Paul Boddie paul.bod...@biotek.uio.no wrote: Are there any lists that are more developer-oriented than this one? There seems to be a mixture of end-user problem reports and tool-specific integration discussions on this list, but not so much discussion of framework development, or at least it can be hard to tune into the latter. There aren't any other development mailing lists specifically for the framework, this is it. You may want to look at our development Trello board to see more of a detailed view (and comment and vote!): https://trello.com/board/galaxy-development/506338ce32ae458f6d15e4b3 -Dannon ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Dependencies on other Tool Shed repositories
Hi Peter, The changset_revision=xxx tag attribute is a minimum version setting. What this means is that the repository being defined will be installed with any updates to the repository within the set of changeset revisions up to and including the next installable changeset revision for the repository. The exception on this rule is if the defined changeset revisions xxx is an installble changeset revision itself, in which case it will be the revision installed. I know this is a bit confusing, so let me know if this is not clear. The following section of the tool shed wiki provides more details about how an installable changeset revision is defined. http://wiki.galaxyproject.org/RepositoryRevisions Greg Von Kuster On Feb 8, 2013, at 7:54 AM, Peter Cock wrote: Hi all, I've been reading http://wiki.galaxyproject.org/ToolShedToolFeatures and focused on the example of EMBOSS where the tool definitions depend on the data type definitions. Here the emboss_5 repository (the tool wrappers) contains depository_dependencies.xml saying: ?xml version=1.0? repositories description=Emboss 5 requires the Galaxy applicable data formats used by Emboss tools. repository toolshed=http://toolshed.g2.bx.psu.edu; name=emboss_datatypes owner=devteam changeset_revision=a89163f31369 / /repositories Is that revision treated as a minimum version, or an absolute one? This concerns me as I have three repositories which will all depend on the blast_datatypes repository for the 'blastxml' format (etc) - so what happens when these three repositories don't all request the same revision of the blast_datatypes repository? What I'm using for now is: ?xml version=1.0? repositories description=This requires the BLAST datatype definitions (e.g. the BLAST XML format). !-- Revision 4:f9a7783ed7b6 on the main tool shed is v0.0.14 which added BLAST databases -- repository toolshed=http://toolshed.g2.bx.psu.edu; name=blast_datatypes owner=devteam changeset_revision=f9a7783ed7b6 / /repositories Regards, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Fwd: Desire to contribute
The highlight enhancement request on Trello: https://trello.com/c/vUtbTQ7l Since it was imported from Bitbucket, the duplicated card I mentioned refers to the wrong Trello card. Please consider! Thanks, Joachim Joachim Jacob Rijvisschestraat 120, 9052 Zwijnaarde Tel: +32 9 244.66.34 Bioinformatics Training and Services (BITS) http://www.bits.vib.be @bitsatvib On 02/06/2013 12:08 PM, Peter Cock wrote: On Wed, Feb 6, 2013 at 11:01 AM, Joachim Jacob |VIB| joachim.ja...@vib.be wrote: In risk of getting a discussion here: a long standing enhancement request is to highlight the current history item one is currently viewing. Situation sketch: I often hide the history items panel to study results (displayed in middle panel) into detail, and when I bring the history item panel back, I often have to search which item I was viewing - no clue at all. It really annoys me, but I don't know whether this can be fixed easily, and how deep you need to dig. Anyway, you will make at least one person happy :-) Cheers, Joachim That sounds like a good usability enhancement - and would likely need some knowledge of the mako template system used in Galaxy, and HTML/CSS for the visual styling too. You said it was a long standing enhancement request - is it filed on Trello? http://galaxyproject.org/trello (I couldn't find it myself). Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Dependencies on other Tool Shed repositories
On Feb 8, 2013, at 7:54 AM, Peter Cock wrote: Hi all, I've been reading http://wiki.galaxyproject.org/ToolShedToolFeatures and focused on the example of EMBOSS where the tool definitions depend on the data type definitions. Here the emboss_5 repository (the tool wrappers) contains depository_dependencies.xml saying: ?xml version=1.0? repositories description=Emboss 5 requires the Galaxy applicable data formats used by Emboss tools. repository toolshed=http://toolshed.g2.bx.psu.edu; name=emboss_datatypes owner=devteam changeset_revision=a89163f31369 / /repositories Is that revision treated as a minimum version, or an absolute one? .. On Fri, Feb 8, 2013 at 1:58 PM, Greg Von Kuster g...@bx.psu.edu wrote: Hi Peter, The changset_revision=xxx tag attribute is a minimum version setting. ... That should be fine then - I've tried to clarify the repository description on http://wiki.galaxyproject.org/ToolShedToolFeatures to say that. Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Data library not properly showing up
Hi all, The data library I have created is only showing up partially. I have made a data libray, via het Admin menu, with one folder in it. I changed permissions, because I wanted it only to be visible to me. That worked fine. Next I created plenty more folders in that data library. Now, the folders created afterwards are not visible in the data library, when accessed via 'Shared data'. I have played a lot with changing permissions of those libraries, but to no avail. Cheers, Joachim -- Joachim Jacob Rijvisschestraat 120, 9052 Zwijnaarde Tel: +32 9 244.66.34 Bioinformatics Training and Services (BITS) http://www.bits.vib.be @bitsatvib ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Deploying LOC files for tool built-in data during a tool installation
Hi list, The tool I am currently wrapping has built-in data, which may be used by the tool users (through a relevant from_data_table + .LOC file configuration). They are .fasta databases which are rather small and are thus bundled in the tool distribution package. Thanks to the tool_dependencies.xml file, said distribution package is downloaded at install time, code is compiled, and since they are here, the data files are copied to $INSTAL L_DIR too , ready to be used. After that, the user still has to edit tool-data/my_fancy_data_files.loc ; but the thing is, during the install I know where these data files are (since I copied those there), so I would like to save the user the trouble and set up this file automagically. I would have two questions: 1/ Is it okay to have tool built-in data files in $INSTAL L_DIR, or would it be considered bad practice? 2/ Is there a way to set up the tool-data/my_fancy_data_files.loc during the install? Here are the options I though of: *shipping a “real” my_fancy_data_files.loc.sample with the good paths already set-up, which is going to be copied as the .loc file (a rather ugly hack) *using more action type=shell_command during install to create my_fancy_data_files.loc (but deploying this file it is not part of the tool dependency install per se) *variant of the previous : shipping my_fancy_data_files.loc as part of the tool distribution package, and copy it through shell_command (same concern than above). Any thoughts? Cheers, -- Jean-Frédéric Bonsai Bioinformatics group___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] DustMasker tool for ncbi_blast_plus
Il giorno mer, 06/02/2013 alle 20.01 +0100, Nicola Soranzo ha scritto: Hi Peter, I added these file formats mostly as placeholders for a future implementation. Now I have changed a bit the tool by removing acclist and seqloc_xml formats since they are not recognized by the last versions of dustmasker (I also sent an email to blast-h...@ncbi.nlm.nih.gov to inform them of this bug). As before, you can find the new version at: https://bitbucket.org/nsoranzo/ncbi_blast_plus I stripped the old commit and did a new one, not a very good practice, sorry about that. Hi Peter, I've added a new commit to this repo which updates the test output files to (recommended) BLAST 2.2.26+, since functional tests were returning errors. Hope you find it useful. Nicola -- Nicola Soranzo, Ph.D. CRS4 Bioinformatics Program Loc. Piscina Manna 09010 Pula (CA), Italy http://www.bioinformatica.crs4.it/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Extract genomic DNA job error
Hello, I am getting this error when using Extract Genomic DNA (version 2.2.2) empty format: fasta, database: hg19 924 warnings, 1st is: Chromosome by name 'chr22' was not found for build 'hg19'. Skipped 924 invalid lines, 1st is #1, chr22 1625633116287937 NM_001136213 0 - 16258185 16287885 0 11 346,119,167,45,71,71,138,107,174,115,684, 0,1853,10597,11805,13541,188 We set up a local instance of Galaxy and a local mirror of the UCSC Table browser in our servers. Thank you for your help, --Ricardo Perez ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Error in database after trying the innocently named, but devastatingly evil test in manage_db.sh
Hi,
I had a small issue with the total usage of my galaxy system showing some
negative random number so I figured my database was a little “off”.
I snooped around for some database management tools and found “manage_db.sh”
which had an option innocently called “test”.
I ran it, without reading the little fine print that states: “Performs the
upgrade and downgrade option on the given database. This is not a real test and
may leave the database in a bad state. You should therefore better run the test
on a copy of your database.”
Well…It surely did leave my database in a bad state since now I get this when I
try to access the toolshed installed tools:
Error Traceback:
View as:http://srv151/admin_toolshed/browse_repositories Interactive |
http://srv151/admin_toolshed/browse_repositories Text |
http://srv151/admin_toolshed/browse_repositories XML
http://srv151/admin_toolshed/browse_repositories (full)
⇝ ProgrammingError: (ProgrammingError) relation
repository_repository_dependency_association does not exist LINE 2: FROM
repository_repository_dependency_association ^ 'SELECT
repository_repository_dependency_association.id AS
repository_repository_dependency_association_id,
repository_repository_dependency_association.create_time AS
repository_repository_dependency_association_create_time,
repository_repository_dependency_association.update_time AS
repository_repository_dependency_association_update_time,
repository_repository_dependency_association.tool_shed_repository_id AS
repository_repository_dependency_association_tool_shed_re_1,
repository_repository_dependency_association.repository_dependency_id AS
repository_repository_dependency_association_repository_d_2 \nFROM
repository_repository_dependency_association \nWHERE %(param_1)s =
repository_repository_dependency_association.tool_shed_repository_id'
{'param_1': 5}
URL: http://srv151/admin_toolshed/browse_repositories
http://srv151/admin_toolshed/browse_repositories
Is there ANY way I can rescue this fiasco? Suffice it to say I did not do it on
a copy and did not do a backup (yes, I learned my lesson), but maybe some SQL
skullduggery can help me?
Thanks
Thon
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
Re: [galaxy-dev] Extract genomic DNA job error
Hi Ricardo, This tool requires external data that you'll need to configure. Here's how: (1) download the 2bit file from UCSC for your genomes into a local directory from here: http://hgdownload.cse.ucsc.edu/goldenPath/hg19/bigZips/ (2) add an entry like this to alignseq.loc (note that you'll need to use tabs to separate the columns: -- seq hg19/galaxy/data/hg19/seq/hg19.2bit -- (3) try running the tool again. Best, J. On Feb 8, 2013, at 2:28 PM, Perez, Ricardo wrote: Hello, I am getting this error when using Extract Genomic DNA (version 2.2.2) empty format: fasta, database: hg19 924 warnings, 1st is: Chromosome by name 'chr22' was not found for build 'hg19'. Skipped 924 invalid lines, 1st is #1, chr22 1625633116287937 NM_001136213 0 - 16258185 16287885 0 11 346,119,167,45,71,71,138,107,174,115,684, 0,1853,10597,11805,13541,188 We set up a local instance of Galaxy and a local mirror of the UCSC Table browser in our servers. Thank you for your help, --Ricardo Perez ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Error in database after trying the innocently named, but devastatingly evil test in manage_db.sh
Hi Thon,
The test option runs the upgrade and downgrade of the latest migration script.
I'm assuming that you had already run the latest migrations available and the
server was up and running OK (other than the dataset size issue) before trying
the test command.
On a COPY of your database, you can try this:
$ sh manage_db.sh version
XYZ
$ sh manage_db.sh downgrade [XYZ -1]
..
done
sh manage_db.sh upgrade XYZ
done
If it works, then you can try it on your real database. Of course, this will
destroy whatever data existed in the table/columns of the latest migration (but
it looks like you already lost them.)
Thanks for using Galaxy,
Dan
On Feb 8, 2013, at 2:32 PM, Thon de Boer wrote:
Hi,
I had a small issue with the total usage of my galaxy system showing some
negative random number so I figured my database was a little “off”.
I snooped around for some database management tools and found “manage_db.sh”
which had an option innocently called “test”.
I ran it, without reading the little fine print that states: “Performs the
upgrade and downgrade option on the given database. This is not a real test
and may leave the database in a bad state. You should therefore better run
the test on a copy of your database.”
Well…It surely did leave my database in a bad state since now I get this when
I try to access the toolshed installed tools:
Error Traceback:
View as: Interactive | Text | XML (full)
⇝ ProgrammingError: (ProgrammingError) relation
repository_repository_dependency_association does not exist LINE 2: FROM
repository_repository_dependency_association ^ 'SELECT
repository_repository_dependency_association.id AS
repository_repository_dependency_association_id,
repository_repository_dependency_association.create_time AS
repository_repository_dependency_association_create_time,
repository_repository_dependency_association.update_time AS
repository_repository_dependency_association_update_time,
repository_repository_dependency_association.tool_shed_repository_id AS
repository_repository_dependency_association_tool_shed_re_1,
repository_repository_dependency_association.repository_dependency_id AS
repository_repository_dependency_association_repository_d_2 \nFROM
repository_repository_dependency_association \nWHERE %(param_1)s =
repository_repository_dependency_association.tool_shed_repository_id'
{'param_1': 5}
URL: http://srv151/admin_toolshed/browse_repositories
Is there ANY way I can rescue this fiasco? Suffice it to say I did not do it
on a copy and did not do a backup (yes, I learned my lesson), but maybe some
SQL skullduggery can help me?
Thanks
Thon
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
Re: [galaxy-dev] [galaxy-user] Why SGE needed for galaxy ?
As far as actually building the image, the recommended method is to use CloudBioLinux build scripts: https://github.com/chapmanb/cloudbiolinux There is a CloudMan flavor of CBL that allows you to build only CloudMan- and Galaxy-required parts: https://github.com/chapmanb/cloudbiolinux/tree/master/contrib/flavor/cloudman On Sat, Feb 9, 2013 at 12:24 AM, Dannon Baker dannonba...@me.com wrote: The workers don't need their own copy of galaxy installed, but a shared filesystem is a requirement for galaxy (in any cluster environment -- see the galaxy wiki for more http://wiki.galaxyproject.org/Admin/Config/Performance/Cluster). Cloudman handles managing NFS for you and sharing the galaxy/tools/index/data volumes. In order for workers to communicate with the master instance, they'll need the cloudman installation as well, so you should use the same image. Now that I've answered that, I'm not sure I totally understand your proposed installation yet, but if you're suggesting bypassing cloudman for installation on a private cloud it should be possible. You'd want the master instance up full time running as the galaxy front end, dispatching jobs to a separate cluster managed by SGE/PBS/whatever. Basically the standard cluster configuration outlined in the wiki above, but you'd want your worker nodes automatically configured to mount the shared directories and join the PBS/SGE queue so they could handle jobs. Depending on what type of private cloud you're working with, it might be easier to just see if you can get cloudman to work :) Lastly, I swapped this message to galaxy-dev since it's about installation nuts and bolts. -Dannon On Feb 8, 2013, at 3:02 AM, Zeeshan Ali Shah zas...@pdc.kth.se wrote: Dear Enis, thanks for reply and being you as cloudman developer it is good to see you in the list . Q2: On Workers node we need galaxy installed with its shared directories ? like galaxyindices , galaxydata Q3: For a private cloud setup do you prefare to have a master image with cloudman and galaxy and use the same image for workers as well ? or worker images can be vanilla OS ? BR Zeeshan On W6-Feb 7, 2013, at 11:50 PM, Enis Afgan wrote: Hi Zeeshan, In order to gain from the scalability of the cloud, SGE does need to run. However, CloudMan sets all that up and manages it going forward. Enis On Fri, Feb 8, 2013 at 8:59 AM, Zeeshan Ali Shah zas...@pdc.kth.se wrote: Hi, It seems that cloud man need SGE for scaling . Does SGE need also when run cloud on private cloud ? Zeeshan ___ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using reply all in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list: http://lists.bx.psu.edu/listinfo/galaxy-dev To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using reply all in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list: http://lists.bx.psu.edu/listinfo/galaxy-dev To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Error in database after trying the innocently named, but devastatingly evil test in manage_db.sh
Hi Dan,
I tried that but it failed ofcourse since the table it tries to drop from 109
- 108 does not exisit.
sh manage_db.sh downgrade 108
109 - 108...
0109_add_repository_dependency_tables DEBUG 2013-02-08 13:04:33,464 Dropping
repository_repository_dependency_association table failed: (ProgrammingError)
table repository_repository_dependency_association does not exist
'\nDROP TABLE repository_repository_dependency_association' {}
0109_add_repository_dependency_tables DEBUG 2013-02-08 13:04:33,464 Dropping
repository_repository_dependency_association table failed: (ProgrammingError)
table repository_repository_dependency_association does not exist
'\nDROP TABLE repository_repository_dependency_association' {}
0109_add_repository_dependency_tables DEBUG 2013-02-08 13:04:33,467 Dropping
repository_dependency table failed: (ProgrammingError) table
repository_dependency does not exist
'\nDROP TABLE repository_dependency' {}
0109_add_repository_dependency_tables DEBUG 2013-02-08 13:04:33,467 Dropping
repository_dependency table failed: (ProgrammingError) table
repository_dependency does not exist
'\nDROP TABLE repository_dependency' {}
done
(galaxy_env)[svcgalaxy@srv151 g]$ sh manage_db.sh version
109
I just went into the database and create the two dummy tables and tried it
again and it seem to work..
I downgraded and upgraded to 109…
Then I restarted Galaxy, reset the metatdata (which failed for 2 of my 13
tools) and then I see this in the manage toolshed:
Error Traceback:
View as:http://srv151/admin_toolshed/browse_repositories Interactive |
http://srv151/admin_toolshed/browse_repositories Text |
http://srv151/admin_toolshed/browse_repositories XML
http://srv151/admin_toolshed/browse_repositories (full)
⇝ ProgrammingError: (ProgrammingError) column
repository_repository_dependency_association.id does not exist LINE 1:
..._7f8dcc0e81d0_de8f CURSOR WITHOUT HOLD FOR SELECT repository... ^ 'SELECT
repository_repository_dependency_association.id AS
repository_repository_dependency_association_id,
repository_repository_dependency_association.create_time AS
repository_repository_dependency_association_create_time,
repository_repository_dependency_association.update_time AS
repository_repository_dependency_association_update_time,
repository_repository_dependency_association.tool_shed_repository_id AS
repository_repository_dependency_association_tool_shed_re_1,
repository_repository_dependency_association.repository_dependency_id AS
repository_repository_dependency_association_repository_d_2 \nFROM
repository_repository_dependency_association \nWHERE %(param_1)s =
repository_repository_dependency_association.tool_shed_repository_id'
{'param_1': 5}
Slightly different error, but an error nonetheless…
Any more ideas?
I am (somewhat) OK with having to reload the toolshed tools but I can’t even
install or manage any new toolsheds…
Thanks
Thon
From: Daniel Blankenberg [mailto:d...@bx.psu.edu]
Sent: Friday, February 08, 2013 12:32 PM
To: Thon de Boer
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] Error in database after trying the innocently named,
but devastatingly evil test in manage_db.sh
Hi Thon,
The test option runs the upgrade and downgrade of the latest migration script.
I'm assuming that you had already run the latest migrations available and the
server was up and running OK (other than the dataset size issue) before trying
the test command.
On a COPY of your database, you can try this:
$ sh manage_db.sh version
XYZ
$ sh manage_db.sh downgrade [XYZ -1]
..
done
sh manage_db.sh upgrade XYZ
done
If it works, then you can try it on your real database. Of course, this will
destroy whatever data existed in the table/columns of the latest migration (but
it looks like you already lost them.)
Thanks for using Galaxy,
Dan
On Feb 8, 2013, at 2:32 PM, Thon de Boer wrote:
Hi,
I had a small issue with the total usage of my galaxy system showing some
negative random number so I figured my database was a little “off”.
I snooped around for some database management tools and found “manage_db.sh”
which had an option innocently called “test”.
I ran it, without reading the little fine print that states: “Performs the
upgrade and downgrade option on the given database. This is not a real test and
may leave the database in a bad state. You should therefore better run the test
on a copy of your database.”
Well…It surely did leave my database in a bad state since now I get this when I
try to access the toolshed installed tools:
Error Traceback:
View as:http://srv151/admin_toolshed/browse_repositories Interactive |
http://srv151/admin_toolshed/browse_repositories Text |
http://srv151/admin_toolshed/browse_repositories XML
http://srv151/admin_toolshed/browse_repositories (full)
⇝ ProgrammingError: (ProgrammingError) relation
Re: [galaxy-dev] Error in database after trying the innocently named, but devastatingly evil test in manage_db.sh
OK..I tried it a few more times and this time with different results! HA! I
knew I wasn’t insane…As long as you try anything often enough in the same
manner, you WILL get a different outcome…Yea uncertainty principle!
I also had to resort to downloading the whole complete galaxy-dist from 11-Jan
since the current development version of galaxy-dist is broken, but I’m back in
business and first order is to create a backup plan
Thon
From: galaxy-dev-boun...@lists.bx.psu.edu
[mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Thon de Boer
Sent: Friday, February 08, 2013 1:23 PM
To: 'Daniel Blankenberg'
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] Error in database after trying the innocently named,
but devastatingly evil test in manage_db.sh
Hi Dan,
I tried that but it failed ofcourse since the table it tries to drop from 109
- 108 does not exisit.
sh manage_db.sh downgrade 108
109 - 108...
0109_add_repository_dependency_tables DEBUG 2013-02-08 13:04:33,464 Dropping
repository_repository_dependency_association table failed: (ProgrammingError)
table repository_repository_dependency_association does not exist
'\nDROP TABLE repository_repository_dependency_association' {}
0109_add_repository_dependency_tables DEBUG 2013-02-08 13:04:33,464 Dropping
repository_repository_dependency_association table failed: (ProgrammingError)
table repository_repository_dependency_association does not exist
'\nDROP TABLE repository_repository_dependency_association' {}
0109_add_repository_dependency_tables DEBUG 2013-02-08 13:04:33,467 Dropping
repository_dependency table failed: (ProgrammingError) table
repository_dependency does not exist
'\nDROP TABLE repository_dependency' {}
0109_add_repository_dependency_tables DEBUG 2013-02-08 13:04:33,467 Dropping
repository_dependency table failed: (ProgrammingError) table
repository_dependency does not exist
'\nDROP TABLE repository_dependency' {}
done
(galaxy_env)[svcgalaxy@srv151 g]$ sh manage_db.sh version
109
I just went into the database and create the two dummy tables and tried it
again and it seem to work..
I downgraded and upgraded to 109…
Then I restarted Galaxy, reset the metatdata (which failed for 2 of my 13
tools) and then I see this in the manage toolshed:
Error Traceback:
View as:http://srv151/admin_toolshed/browse_repositories Interactive |
http://srv151/admin_toolshed/browse_repositories Text |
http://srv151/admin_toolshed/browse_repositories XML
http://srv151/admin_toolshed/browse_repositories (full)
⇝ ProgrammingError: (ProgrammingError) column
repository_repository_dependency_association.id does not exist LINE 1:
..._7f8dcc0e81d0_de8f CURSOR WITHOUT HOLD FOR SELECT repository... ^ 'SELECT
repository_repository_dependency_association.id AS
repository_repository_dependency_association_id,
repository_repository_dependency_association.create_time AS
repository_repository_dependency_association_create_time,
repository_repository_dependency_association.update_time AS
repository_repository_dependency_association_update_time,
repository_repository_dependency_association.tool_shed_repository_id AS
repository_repository_dependency_association_tool_shed_re_1,
repository_repository_dependency_association.repository_dependency_id AS
repository_repository_dependency_association_repository_d_2 \nFROM
repository_repository_dependency_association \nWHERE %(param_1)s =
repository_repository_dependency_association.tool_shed_repository_id'
{'param_1': 5}
Slightly different error, but an error nonetheless…
Any more ideas?
I am (somewhat) OK with having to reload the toolshed tools but I can’t even
install or manage any new toolsheds…
Thanks
Thon
From: Daniel Blankenberg [mailto:d...@bx.psu.edu]
Sent: Friday, February 08, 2013 12:32 PM
To: Thon de Boer
Cc: galaxy-dev@lists.bx.psu.edu mailto:galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] Error in database after trying the innocently named,
but devastatingly evil test in manage_db.sh
Hi Thon,
The test option runs the upgrade and downgrade of the latest migration script.
I'm assuming that you had already run the latest migrations available and the
server was up and running OK (other than the dataset size issue) before trying
the test command.
On a COPY of your database, you can try this:
$ sh manage_db.sh version
XYZ
$ sh manage_db.sh downgrade [XYZ -1]
..
done
sh manage_db.sh upgrade XYZ
done
If it works, then you can try it on your real database. Of course, this will
destroy whatever data existed in the table/columns of the latest migration (but
it looks like you already lost them.)
Thanks for using Galaxy,
Dan
On Feb 8, 2013, at 2:32 PM, Thon de Boer wrote:
Hi,
I had a small issue with the total usage of my galaxy system showing some
negative random number so I figured my database was a little “off”.
I snooped around for
[galaxy-dev] Galaxy February 8, 2013 Distribution News Brief
Galaxy Feb 8, 2013 Distribution News Brief http://wiki.galaxyproject.org/News/2013_02_08_GalaxyNewsBrief *Complete News Brief http://wiki.galaxyproject.org/DevNewsBriefs/2013_02_08 * *Highlights:* * /Improvements/ to our release process http://wiki.galaxyproject.org/DevNewsBriefs/2013_02_08#Improvements_to_Release_Process. *Release tag must be used in the hg update command to upgrade*. More at *usegalaxy.org http://wiki.galaxyproject.org/Admin/Get%20Galaxy*. * Tool Shed /Complex repository dependencies/ http://wiki.galaxyproject.org/DefiningRepositoryDependencies#Complex_repository_dependencies:_tool_dependency_definitions_that_contain_repository_dependency_definitions are introduced, streamlining core dependency use across individual tools. * Also updated in the Tool Shed: multiple repository installation, dependency installation (when defined), and many usability enhancements and fixes. * New /Bedgraph-to-bigwig/ http://wiki.galaxyproject.org/Learn/Datatypes#BedGraph tool plus /Filter/ tool updated. * /Workflows/ now include option to export an image and the core /Framework/ now allows more unified reference genome usage and access. * /Ten Community Pull-Requests/ incorporated, plus another contribution to the tool Shed, addressing general usability, API, tools, error handling, workflows, and more. Thanks!! * Review highlights from the latest monthly */Galaxy Update/* http://wiki.galaxyproject.org/GalaxyUpdates newsletter and Galaxy Project Stats http://wiki.galaxyproject.org/Galaxy%20Project/Statistics! * Plus bug fixes and related enhancements for visualizations, histories, workflows, and tools. *http://getgalaxy.org* *http://bitbucket.org/galaxy/galaxy-dist* *http://galaxy-dist.readthedocs.org * new: $ hg clone http://www.bx.psu.edu/hg/galaxy galaxy-dist upgrade: $ hg pull $ hg update release_2013.02.08 To ensure that you continue to receive updates about *Galaxy Distributions and News Briefs* http://wiki.galaxyproject.org/DevNewsBriefs, be sure to subscribe to the Galaxy-Announce mailing list http://wiki.galaxyproject.org/MailingLists#The_lists, a low-volume list dedicated to News Items http://wiki.galaxyproject.org/News and priority messages from the Galaxy community http://wiki.galaxyproject.org/Community and project core team http://wiki.galaxyproject.org/Galaxy%20Project. /Thanks for using Galaxy,/ The Galaxy Team http://wiki.galaxyproject.org/Galaxy%20Team ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
