Re: [galaxy-dev] Reducing the size of a volume in Galaxy Cloud
Hi Denise, Although the instructions you found can still be used, a couple of steps can be updated now so I'll to that now and hopefully answer your questions in the process. Updated instructions: 1. Start a brand new Galaxy CloudMan cluster/instance (this will be only a temporary cluster needed for the duration of the following steps) 2. ssh into the master instance and delete whatever genomes you don't need/want (these are all located under /mnt/galaxyIndices; using '*rm -rf dir name*' is the command to use) 3. From CloudMan's Admin page, add a new file system of type 'new volume' and desired size (ie, size of data on /mnt/galaxyIndices after you've deleted what you didn't want). Give this file system a name 'indices'. 4. ssh into the master instance and copy over the data from /mnt/galaxyIndices to /mnt/indices using command: rsync -avz /mnt/galaxyIndices/ /mnt/indices/. (do this as 'galaxy' user) 5. Unmount the new 'indices' file system using command: sudo umount /mnt/indices 6. From the AWS console, create a snapshot of the volume for the 'indices' file system. You can retrieve the volume ID from the file system 'details' on the CloudMan Admin page 7. For the cluster you want to keep around (while it is terminated), edit *persistent_data.yaml* in it's bucket on S3 (you can get the bucket name from the CloudMan Admin page - it will look like so *cm-hash*) and replace the existing snap ID for the *galaxyIndices* with the snapshot ID you got in the previous step 8. Start that cluster and you should have a file system from the new snapshot mounted 9. Terminate delete the cluster you created in step 1 Here are my questions to help me to get through step 4: 1) Step 1: Is a “cluster” the same thing as an “instance”? Yes 2) Step 2: I deleted the directories for individual genomes using rm –rf . Is that the correct approach? Yes 3) Step 3: Do I add the newly created EBS volume to the same instance where I deleted the genomes? Or is it added the instance I want to keep? You add the new file system (ie, EBS volume) to the newly created cluster (and you also delete the extra genomes on that same cluster). 4) Step 3: I can see how to attach this newly created volume using the AWS EC2 management console, but how do I mount it? (and unmount it in Step 5?) With the updated instructions above, mounting the file system by hand is no longer necessary. 5) Step 4: what is the syntax for the rsync (or cp) command to copy directories/files from one volume to another volume (within the same instance, or in if they are in different instances)? I included in the command in the instructions now. Hope this helps. Let us know if you have any trouble or if you have any more questions, Enis ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] job_working / tmp folders
I am not aware of any other way new_file_path is accessed. Is it old jobs that were queued up prior to the switch? I think job_working_directory might actually be affected by the settings in your object store configuration. Have you configured an object store? In either case - knowing the specific tools to look at that are still referencing these old files might really help diagnose the problem. Hope this helps. -John On Thu, Oct 30, 2014 at 1:56 PM, Shrum, Donald C dcsh...@admin.fsu.edu wrote: I changed the values for new_file_path and job_working_directory in my config file. After restarting galaxy I’m finding some of the tools still reference the old path which no longer exists. Do these end up hard coded in some place… I haven’t done a find + grep to go looking yet but that may be the next step… Thanks for any help --Don FSU Research Computing Center ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Unable to connect two tools in workflow editor
Can you review this dev thread http://dev.list.galaxyproject.org/Problem-selecting-datasets-with-a-specified-datatype-td4665366.html? In particular can you go to the View data types registry option in the admin menu and see if Galaxy thinks it knows about this datatype? Let me know if Galaxy thinks the datatype exists. If the datatype doesn't exist - can you tell me how you have defined the custom dataytpe - directly in Galaxy, in a tool shed, etc...? The more details about the definition of the datatype you can provide the better. -John On Thu, Oct 30, 2014 at 4:52 PM, David Kelly davidke...@uchicago.edu wrote: Hello, I'm working with two custom tools in my local galaxy instance: The first tool outputs a single aceb file. aceb is a custom datatype I've added that describes a crop experiment. The second tool takes three files as inputs, one of which is an aceb file. I can successfully run these two tools together in Galaxy outside of a workflow. But for some strange reason I cannot connect these two tools in the workflow editor. The connection never turns green and links. Here are the relevant lines in the tool wrappers: Outputs of first tool: outputs data format=aceb name=acebData label=Experiment data with unified format / /outputs Inputs of second tool: inputs param name=acebData type=data format=aceb label=Inut Survey aceb Data / param name=domeData type=data format=dome label=Input Strategy DOME Data / param name=linkData type=data format=txt label=Input Linkage between field overlay and survey / /inputs Any ideas for what could be causing this? Thanks. David ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Configuring r script galaxy tool to install on testtoolshed with dependencies in the same repository
I think a working example that uses this pattern is Bjoern's glimmer tool: https://github.com/bgruening/galaxytools/tree/master/glimmer3 The only obvious difference I see that he sets the environment variable to $REPOSITORY_INSTALL_DIR in tool_dependencies.xml (environment_variable name=GLIMMER_SCRIPT_PATH action=set_to$REPOSITORY_INSTALL_DIR/environment_variable). Does changing that fix anything? He also escapes the environment variable in the command \$R_SCRIPT_PATH instead of $R_SCRIPT_PATH in your case. -John P.S. I find set_environment to be something of an anti-pattern myself - it doesn't map well to other packaging systems such as nix or brew. I think it would be better to resolve things relative to the script from the script itself or to write out a helper script that wraps up R and stuff and resolve paths relative to itself but I guess that is beside the point. On Sat, Oct 25, 2014 at 3:38 PM, Jeremy Liu jeremy@yale.edu wrote: Hi Galaxy Dev, Been having trouble getting my Galaxy tools to work with the testtoolshed. Here is the tools repository: https://jeremyj...@testtoolshed.g2.bx.psu.edu/repos/jeremyjliu/region_motif_enrichment In particular, I've been focusing on getting the region_motif_intersect.r script to work properly when installed from the testtoolshed. intersect.r runs using files located in the same location as where the repository is installed, in the folder region_motif_db. To make sure that the tool knows where these files are, I've been trying to pass the repository install location as an argument in the r script invocation (see region_motif_intersect.xml and tool_dependencies.xml) . To set the repository install location in the environment, I've been basing off of this wiki entry: https://wiki.galaxyproject.org/ToolsWithDependenciesInSameRepository When I install the repository to my local Galaxy, it gives me the following warnings: galaxy.tools.deps DEBUG 2014-10-25 15:12:35,376 Building dependency shell command for dependency 'R_SCRIPT_PATH' galaxy.tools.deps WARNING 2014-10-25 15:12:35,377 Failed to resolve dependency on 'R_SCRIPT_PATH', ignoring Then when I run the tool from Galaxy, this is the run command: PACKAGE_BASE=/Users/jeremyliu1/galaxy-dist/tool-dependency/R/2.15.1/jeremyjliu/region_motif_enrichment/7a3bb055f3d0; export PACKAGE_BASE; . /Users/jeremyliu1/galaxy-dist/tool-dependency/R/2.15.1/jeremyjliu/region_motif_enrichment/7a3bb055f3d0/env.sh; Rscript /Users/jeremyliu1/shed_tools/testtoolshed.g2.bx.psu.edu/repos/jeremyjliu/region_motif_enrichment/7a3bb055f3d0/region_motif_enrichment/region_motif_intersect.r --args $R_SCRIPT_PATH p /Users/jeremyliu1/galaxy-dist/database/files/000/dataset_504.dat /Users/jeremyliu1/galaxy-dist/database/files/000/dataset_515.dat What exactly is going on here? How do I make sure my tool knows where the reference files it needs are, since it won't be in the job working directory? Note: the files that intersect.r needs are not in the toolshed repository, since they are a few GB. I was planning on providing that as a separate download or using the Data Managers to solve the problem. But as long as I can get the tool to look for it in that folder, that would be perfect. Thanks! Jeremy Liu jeremy@yale.edu 352.871.1258 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] bowtie2: no stderr output
It looks like this was a problem for awhile but isn't anymore on main https://trello.com/c/yFeXvhUg. It doesn't look like there has been any changes to the bowtie2 wrapper since it was migrated to the tool shed so I am not sure why it isn't working for you? As an admin can you open a successful bowtie2 dataset and see what the standard error and standard output are saved as - maybe copying them into pastebin or gist. It might help illuminate the problem. -John On Mon, Oct 27, 2014 at 4:31 AM, Elizabeth Chia elc...@gmail.com wrote: Dear all Usually the alignment statistics from bowtie2 will be printed to the stderr file, however, on the local instance, the stderr file is empty after alignment. It still does the alignment, but the statistics are not printed out. This is my hg summary: parent: 13802:f7dd0060c296 latest_2014.06.02 Merged in dannon/galaxy-central-prmaker/stable (pull request #404) branch: stable commit: 288 modified, 357 added, 18 unknown (new branch head) update: 27 new changesets (update) Will updating to the latest version help? Or are there any patches available? Thanks. Regards liz ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] error running cuffdiff
Hello Liz, I wish I had some better news - upgrading might help - we have definitely made modifications to that file since then - but not to address this issue I don't think. What would be really interesting is to see the Galaxy logs around the time of that error - I wonder if there would be some other exceptions logged that explains why that entry doesn't exist in the dictionary. I have seen errors like this caused by changing things around in the tool without modifying the version - I am assuming you haven't modified the cuffdiffs wrapper though? Ideally to track this down there would be a minimal example that can reproduce this problem - have this one dataset with this metadata in your history and run this cufflinks tool - but that would obviously be a lot of work on your part. If you did want to help debug this problem though - you could start a new history try opening the cuffdiff tool - verify it opens - and then slowly copy a few of that histories datasets at a time into the new history and try reopening the cuffdiff wrapper until you get an error. I would bet some combination of history dataset metadata is interacting with the cuffdiff wrapper. -John On Sun, Oct 26, 2014 at 11:26 PM, Elizabeth Chia elc...@gmail.com wrote: Hi Bjorn Thanks for getting back to me. Sorry for the late reply. Was awaiting my user to get back to me. It was the default cuffdiff parameters. If I'm not wrong, we keep experiencing this on/off with other tools too. Appreciate any feedback and help. Thanks so much! Best regards liz On Sat, Oct 18, 2014 at 8:52 PM, Björn Grüning bjoern.gruen...@gmail.com wrote: Hi Elizabeth, can you please let us know which parameters you have used? Thanks, Bjoern Am 18.10.2014 um 14:33 schrieb Elizabeth Chia: Hi there I'm running into this error with cuffdiff: I'm running galaxy locally on a cluster (1 head+3 compute nodes) with MySQL. Here's my hg summary: Any clues? I also keep getting the a similar 149 Internal Server error (with different Key Error) thrown when trying to install some tools. But strangely, after leaving it for awhile, it went back to normal. So I'm not too sure what's causing it. Should I be updating galaxy to the latest stable one? Please let me know if you require any other info. Thanks! liz ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Problems with upload.py and changes to json.py
Dear Galaxy-Dev, Last week I downloaded a fresh version of Galaxy with all stable updates. I've just come across a NameError in the upload.py tool. Traceback (most recent call last): File /home/rob/galaxy-dist/tools/data_source/upload.py, line 390, in module __main__() File /home/rob/galaxy-dist/tools/data_source/upload.py, line 368, in __main__ dataset = from_json_string( line ) NameError: global name 'from_json_string' is not defined A quick investigation led me to: from galaxy.util.json import * and from there to json.py. I compared this file with my previous version and note that there are quite a few changes. The problem seems to be that from_json_string() is not included in the __all__ list of public functions... can someone tell me if this is a bug/oversight or if I'm missing something? Cheers, Rob___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
