Re: [galaxy-dev] EOF error with tool
There are a couple groups working on improved trinity wrappers - I don't think they have reach the tool sheds though :(. As for the details of your specific tool - I think I would really need to see the tool to guess at the issue. Would it be possible to send us the problematic tool XML file or paste it to a service like gist.github.com? As for reloading the tool - if you add yourself as an admin user in galaxy's config/galaxy.ini file you should see a Reload Tool Configuration option in the admin menu that will allow you to manually reload the tool without restarting Galaxy (I generally recommend keeping this open in its own tab). Newer versions of Galaxy have some more experimental features designed to help as well - for instance if you create a virtualenv for Galaxy and install watchdog (pip install watchdog) into it and then enable watch_tools = True in config/galaxy.ini Galaxy should automatically reload tools each time the tool config is updated. I am not sure anyone is really using this feature so it might still have some bugs (in particular I have noticed if the tool panel is really full there are problems so trimming that down might be required to use it). For the rare person like me who really hates using GUIs at all during development - you might also want to write some tests for the tool and run them using planemo (http://planemo.readthedocs.org/). It allows linting and testing tools without a Galaxy interface so you can be pretty confident the tool is working by the time you actually load it into a web browser and view it. -John On Mon, Jan 12, 2015 at 9:36 AM, Francis, Warren w.fran...@lrz.uni-muenchen.de wrote: Hello, I'm getting a strange error when trying to debug a tool. Traceback (most recent call last): File /var/project/galaxy-dist/lib/galaxy/jobs/runners/__init__.py, line 157, in prepare_job job_wrapper.prepare() File /var/project/galaxy-dist/lib/galaxy/jobs/__init__.py, line 811, in prepare self.command_line, self.extra_filenames = tool_evaluator.build() File /var/project/galaxy-dist/lib/galaxy/tools/evaluation.py, line 348, in build self.__build_command_line( ) File /var/project/galaxy-dist/lib/galaxy/tools/evaluation.py, line 364, in __build_command_line command_line = fill_template( command, context=param_dict ) File /var/project/galaxy-dist/lib/galaxy/util/template.py, line 9, in fill_template return str( Template( source=template_text, searchList=[context] ) ) File /var/project/galaxy-dist/eggs/Cheetah-2.2.2-py2.7-linux-x86_64-ucs4.egg/Cheetah/Template.py, line 1004, in __str__ return getattr(self, mainMethName)() File cheetah_DynamicallyCompiledCheetahTemplate_1421072368_91_78155.py, line 139, in respond EOFError: EOF when reading a line The tool is the assembler Trinity. The toolshed version is quite old and the one from the Trinity website does not work, so I tried to fix it (several problems to fix). What is the EOFError? There were a bunch of posts reporting this but no answer as to why this happens. Is there a way to debug a tool without having to restart galaxy each time? Something offline that can check for runtime errors, or at least see if it generates the command correctly. Best, -Warren ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Unable to connect galaxy server after upgrade to latest version
No clue what is wrong - hopefully you figured this out. If you haven't, can you send us your Galaxy log file for the web server loading - you can send it to me personally and galaxy-b...@bx.psu.edu - if you are worried about disclosing private information on the public list? Also it would be interesting to see the output produced when running both hg summary and hg status from your Galaxy working directory. -John On Fri, Jan 16, 2015 at 6:11 PM, Ping Luo luop0...@gmail.com wrote: I installed the latest galaxy server and started it without any change. The server stared on port 8080 and the log file looks fine. But when I typed http://localhost:8080, I got the error Unable to connect. I had worked with version 2014_08_11 and had no problem at all. What could have been wrong? ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] GCC2015 Training Day Topic Voting is open. Please vote!
Hello all,What topics should be offered at the GCC2015 Training Day http://gcc2015.tsl.ac.uk/training-day/? Topics http://gcc2015.tsl.ac.uk/training-day/ for the GCC2015 Training Day http://gcc2015.tsl.ac.uk/training-day/ are selected by you, the Galaxy Community. There are nominated topics http://gcc2015.tsl.ac.uk/training-day/ spanning from basic usage to advanced deployment. No matter what you do with Galaxy, there are topics for you to choose from. Your vote will determine the topics that are offered, which topics should be offered more than once, and which ones should not be scheduled at the same time. Your vote matters. *Topic voting closes 30 January.* The Training Day schedule, including instructors, will be published before early registration opens in February. *Vote Now http://gcc2015.tsl.ac.uk/training-day/* -- *About the GCC2015 Training Day http://gcc2015.tsl.ac.uk/training-day/:* The 2015 Galaxy Community Conference (GCC2015) http://gcc2015.tsl.ac.uk/ will start on 6 July with a Training Day http://gcc2015.tsl.ac.uk/training-day/ featuring parallel tracks, each with several multi-hour workshops. There will be at least one complete track about using Galaxy for biological research, and at least one full track on deploying and managing Galaxy instances. As always, please let us know if you have any questions. Thanks, The GCC2015 Organising Committee http://gcc2015.tsl.ac.uk/organisers/ -- http://galaxyproject.org/ http://getgalaxy.org/ http://usegalaxy.org/ https://wiki.galaxyproject.org/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Manually adding new tools from toolshed
I am skeptical that setting up a local tool shed is worth the effort - it seems like overkill. I think just downloading the tar balls, manually modifying the datatypes_conf.xml is fine. If you set the tool_dependencies_dir in your Galaxy configuration - you can also create a isolated environments for tool requirements (https://wiki.galaxyproject.org/Admin/Config/ToolDependencies) allowing multiple versions of requirements like samtools to be installed simultaneously. I recently added the ability to have multiple versions of tools themselves installed like the tool shed but without the tool shed (https://bitbucket.org/galaxy/galaxy-central/commits/08f8850853d004bf8a456a147436a6694d0a1a11) - it is pretty experimental but it might help if you go down this road. As for how Galaxy knows about the tool shed installs - it is a combination of file system files and database modifications (Galaxy installs the dependencies into tool_dependencies_dir, the tool files into the directory specified in shed_tools_conf.xml, updates the file integrated_tool_panel_conf.xml, and registers tool lineages and installation status in the Galaxy database). It should certainly be possible to install tools from a different server than Galaxy is running on - but you will need to run a Galaxy process on that server that can access all the same files and the Galaxy database. Hope this helps, -John On Fri, Jan 16, 2015 at 10:39 AM, Ryan G ngsbioinformat...@gmail.com wrote: I can try a local toolshed, but the organization I work at frowns upon automatic installation for various reasons. That and it lets me see exactly how a tool is installed and works within my local instance. When a tool is installed from the toolshed, how does Galaxy know about it? From a DB entry, or from the XML file(s) being present somewhere? I sort of like the idea of how /etc/profile.d gets scanned by /etc/bashrc when a user logs into a linux machine. This way, there is no modification of files, just placing the right files in the right directory is all that is needed. Anyway, I need to understand how Galaxy works with the toolshed. On Fri, Jan 16, 2015 at 3:08 AM, Björn Grüning bjoern.gruen...@gmail.com wrote: Hi Ryan, what about a local bootstrapped ToolShed which is in sync with the public one (at least for the users you are interested on). Than you can install from this ToolShed. I guess this is less annoying than maintaining everything on your own. Cheers, Bjoern Am 16.01.2015 um 08:22 schrieb Peter Cock: On Thu, Jan 15, 2015 at 6:57 PM, Ryan G ngsbioinformat...@gmail.com wrote: Because of the way the infrastructure for my Galaxy instance is set up, I need to download and install tools from the toolshed manually. For most of the tools, this is pretty easy, however I'm now trying to add RSEM to my instance and it has some new datatypes. Along with the new data types is python class file. I'm referring to the exist datatypes_conf.xml.sample but don't see a reference to datatype_files as is coded in the RSEM tool. So my question is, how can I manually install a tool such as RSEM from the toolshed? I've cloned it but not sure where to put the files and what to modify in Galaxy's configuration to see the new files. I can modify the datatypes_conf.xml and include the datatypes, but I get the sense this is not correct. Ryan The old pre-ToolShed way to add a new datatype was indeed to add it to datatypes_conf.xml and put any Python file into the galaxy-dist/lib/galaxy/datatypes/ I'm still doing this on some of my locally developed tools, but now they are in the ToolShed perhaps I should switch to using that... Regards, Peter P.S. You also used to need to add an import line to the galaxy-dist/lib/galaxy/datatypes/registry.py but that is not longer required as of this bug fix: https://bitbucket.org/galaxy/galaxy-central/commits/1422966f1ca88472b8685015f5a8cdd3dd0f1db9 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
Re: [galaxy-dev] galaxy and torque - resource allocation
Galaxy generally defers to the DRM (torque in your case) for dealing with these things. In your job_conf.xml you can specify limits for memory or CPUs and Galaxy will pass these along to the DRM at which point it is up to the DRM to enforce these - details depend on if you are using the PBS runner or the DRMAA runner (let me know which and I can try to elobrate if that would be useful). In your particular case - I don't believe torque schedules RAM so things will generally only be... throttled... by CPUs counts. This is what I was told by the admins at MSI when I worked there anyway. If you want to place hard limits on RAM I think you need to upgrade to the MOAB scheduler or switch over to a different DRM entirely like SLURM. Even for DRMs that deal more directly with memory - Galaxy doesn't provide a general mechanism for passing this along to the tool (https://trello.com/c/3RkTDnIn) - so it would be up to the tool to interact with the environment variables your DRM sets . This sounds really terrible in the abstract - but it reality it usually isn't an issue - most of Galaxy's multi-core mappers say have relatively low memory per CPU usage - and for things like assemblers where this is more important - one can usually just assign them to their own CPU or something like that to ensure they get all the memory available. Unlike memory - Galaxy will attempt to pass the number of slots allocated to a job to the tools by setting the GALAXY_SLOTS environment variable. All of the multiple core devteam Galaxy tools at this point use this and so should work - as should probably most multi-core tools from the tool shed - at least the most popular ones. Hope this helps, -John On Tue, Jan 13, 2015 at 11:52 AM, Fernandez Edgar edgar.fernan...@umontreal.ca wrote: Hello gents, Hope everyone had a great holiday break! Wish you guys all the best for 2015! I have a couple of questions about how resources (CPU and memory) is allocated when you have a galaxy and torque installation. So I’ve setup torque with some default and maximum amount of CPU and memory allocations. However, I have some worries when it comes to application (like tophat for example). By default, it takes half the CPU of a server unless specified otherwise. How is the CPU allocation is specified to application like tophat through galaxy? Also, how does galaxy react if a job needs more memory than the limit set by torque? Any information would help me a lot! My sincere salutations to you all!!! Cordialement / Regards, Edgar Fernandez System Administrator (Linux) Direction Générale des Technologies de l'Information et de la Communication ( Bur. : 1-514-343-6111 poste 16568 Université de Montréal PAVILLON ROGER-GAUDRY, bureau X-218 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Display of file content in a combobox
I don't know how to do this exactly - but I think it is possible with a custom datatype. I think you define a custom datatype (you will probably want to extend the Galaxy tabular datatype), then I believe you will need to override the set_meta method on the datatype to collect a new metadata field (perhaps called factors), and finally have the first step produce this datatype. The second step then should define its input data parameter to consume this type, and then define a select param type and within that param name=label type=select label=Factor options filter type=data_meta ref=output_step_1 key=factors / /options /param The example of a tool that works vaguely this way that I know about is JJ's mothur wrappers (but hopefully someone can think of one that actually works with tabular data). Here is the repository: https://toolshed.g2.bx.psu.edu/repos/jjohnson/mothur_toolsuite/file/tip/mothur Here is the datatype definition - he extends Text data - but you would want to extend Tabular instead - but this gives a sense of how to override set_meta to collect metadata and extend a datatype. https://toolshed.g2.bx.psu.edu/repos/jjohnson/mothur_toolsuite/file/tip/mothur/lib/galaxy/datatypes/metagenomics.py Here is a tool than that uses this datatype and the filter on data_meta type. https://toolshed.g2.bx.psu.edu/repos/jjohnson/mothur_toolsuite/file/tip/mothur/tools/mothur/classify.otu.xml It will probably frustrating to get this pattern working the first time - but it is possible. Hope this helps, -John On Mon, Jan 12, 2015 at 10:31 AM, christof.piet...@kws.com christof.piet...@kws.com wrote: Hello, I’d like to display the unique column content of a file in a combobox to give the user the possibility to select factor levels for subsequent analyses, e.g. for combining related factor levels in a meta-analyses. Unfortunately, I failed to connect the tool that reads out the factor levels with the tool that receive the selection via combobox as input argument. I’d be very thankful for any suggestion! Christof ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] FastQC wrapper not seeing files at gzipped
Peter has already voted and if I recall correctly Ryan cannot access Trello - so this might be a waste to bring up - but here is a Trello card for voting on this issue and tracking progress https://trello.com/c/3RkTDnIn. To summarize previous discussion - this would be fantastic to have and Galaxy needs this - but we solve this on usegalaxy.org by using a compressed file system - a more elegant solution when it is a possibility - so it has never been a tier one priority for the devteam. The only update on this is that I don't think we are using a compressed file system anymore so this might become and issue again someday soon. This would be non-trivial to implement - but I have always felt this would be a fairly fun project to work on if anyone really tight on space locally wants to try to tackle it :). -John On Tue, Jan 13, 2015 at 9:54 AM, Ryan G ngsbioinformat...@gmail.com wrote: Agreed. On Mon, Jan 12, 2015 at 10:24 PM, Peter Cock p.j.a.c...@googlemail.com wrote: Hi Ryan, That is the workaround I am using, which means keeping an uncompressed copy of the FASTQ file on our main storage from where Galaxy can see it (for people to use within their histories). From a long term storage perspective this is not ideal - so I am keen for better handling of gzipped files within Galaxy (particularly within libraries which we use for raw data). Peter On Mon, Jan 12, 2015 at 5:20 PM, Ryan G ngsbioinformat...@gmail.com wrote: Yes, I'm doing a link to file on file system when doing a library import. Does this mean I should link to the the uncompressed file? On Mon, Jan 12, 2015 at 12:14 PM, Peter Cock p.j.a.c...@googlemail.com wrote: Ah. Then this is more subtle... are you using the library import option where Galaxy just symlinks to existing files? I thought that was not possible with gzipped files (for the reasons given below). Perhaps this is not being blocked, leading to the confused state you're seeing? Peter On Mon, Jan 12, 2015 at 4:52 PM, Ryan G ngsbioinformat...@gmail.com wrote: Galaxy is not decompressing the file. The file is linked to on the filesystem. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Partial automation for generating those twisty R dependency tool shed installation sequences
Hi Ross, Am 13.01.2015 um 00:13 schrieb Ross: Hi Björn, I'm a bit old fashioned and think I prefer a proper Galaxy tool rather than a notebook :) :) So I've set up a temporary demonstration/test site of a toolfactory generated tool that does what I think I need - can some kind soul please test it and let me know how it goes ? If it's useful, it needs to be adjusted to depend on whatever version of package_R you want to work with - currently just uses the system R for demonstration purposes. I used the toolfactory2 (main toolshed) (which now allows any number of (optionally non editable) parameters!!!) to wrap the script shown at the bottom of https://wiki.galaxyproject.org/SetUpREnvironment. There are currently three parameters - the names of the R/BioC packages from sessionInfo(), the local directory where all the tarballs should be stowed and the XML output prefix to prepend to each row of the generated XML stanza for tool_dependencies.xml The resulting toolshed tarball was uploaded to a local toolshed and then installed to produce a new tool in the tool generators section - r_bioc_depgen Generate dependencies for R/BioC packages If you import the history at http://130.56.252.21/history/list_published you will see the toolfactory job (#1,#2,#3) - rerunning will show how the parameters are defined - fugly but it does work. After generating/uploading/installing the new tool, outputs from a test run are in #4 and #5 for DESeq This is cool! I have to try this on a few packages! This could also be of interest for our GSOC idea: https://wiki.galaxyproject.org/Develop/GSOC/2015Ideas#Fostering_Bioconductor_Collaborations Thanks Ross for working on this! Bjoern Comments and suggestions welcomed! On Sun, Jan 11, 2015 at 10:41 PM, Björn Grüning bjoern.gruen...@gmail.com wrote: Hi Ross, you are absolutely right. My download_store repository is exactly for this purpose. https://github.com/bgruening/download_store If you are interested we could integrate your additional magic into the notebook. Thanks, Bjoern Am 11.01.2015 um 01:33 schrieb Ross: Hi, Björn, Looks pretty similar! Aren't the links your notebook generates transient? I think if you put them into a tool_dependencies.xml, they will fail permanently immediately after any of the package authors updates one of the relevant svn repositories? AFAIK, it looks like the whole BioC/CRAN infrastructure is automated so a link that works today like http://cran.fhcrc.org/src/contrib/Rcpp_0.11.3.tar.gz will fail when Rcpp next gets updated and Rcpp_0.11.3.tar.gz is migrated to http://cran.fhcrc.org/src/contrib/00Archive/Rcpp/ with a replacement (eg) http://cran.fhcrc.org/src/contrib/Rcpp_0.11.4.tar.gz appearing in the contrib directory? That's why my more complex script downloads all the latest archives into my local github archive repo and generates a permanent link to suit that github repo. We definitely need an automated solution as this is a really infuriating aspect of trying to make code relying on R/BioC packages reproducible. On Thu, Jan 8, 2015 at 11:28 PM, Björn Grüning bjoern.gruen...@gmail.com wrote: Hi Ross, this is great! Have you seen this notebook? http://nbviewer.ipython.org/github/bgruening/notebooks/blob/master/R/extract_all_dependencies_from_an_r_package.ipynb It tries to do the same thing. Maybe it's also worth to mention? Maybe we can enhance it? Thanks, Bjoern Am 08.01.2015 um 08:09 schrieb Ross: This may be helpful for anyone else struggling to get complex nested R package dependency installation from the tool shed sorted out. That whole can of worms. While we have setup_r_packages, the developer still has to figure out the right magical incantation and make sure the tarballs are available. https://wiki.galaxyproject.org/SetUpREnvironment has some notes I've started - contribitions welcome. It has a more or less reusable R script to generate tool_dependencies.xml boilerplate, assuming you set the constant libdir to your local git repository path where those tarballs will be downloaded from. I hope this helps someone! Could make a tool to do this if enough developers want access to it without the pain of managing yet another R script? ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
