[galaxy-dev] bowtie2 indexes & errors
Hi, I observe next error: [2017-03-21 10:30:56] Beginning TopHat run (v2.1.0) --- [2017-03-21 10:30:56] Checking for Bowtie Bowtie version:2.2.5.0 [2017-03-21 10:30:56] Checking for Bowtie index files (genome).. Error: Could not find Bowtie 2 index files (/home/galaxy/tool-data/GRCm38_84/bowtie2_index/GRCm38_84/GRCm38_84.*.bt2) And this error is clear: no index files with *.bt2. But we have for GRCm38_84: ../GRCm38_84/GRCm38_84.3.bt2l ../GRCm38_84/GRCm38_84.rev.1.bt2l ../GRCm38_84/GRCm38_84.1.bt2l ../GRCm38_84/GRCm38_84.rev.2.bt2l ../GRCm38_84/GRCm38_84.2.bt2l ../GRCm38_84/GRCm38_84.4.bt2l ../GRCm38_84/GRCm38_84.fa as I understand from the manual for bowtie2 "l" at the end of files means that indexes were created for large file. If it is correct way then why Galaxy tools do not understand this extension and how it should be solve? Okay I have simply created symbolic links to them with *.bt2 extensions (I don't know if *.tb2 and *.tb2l have the same format) and run job again: [2017-03-21 11:40:41] Beginning TopHat run (v2.1.0) --- [2017-03-21 11:40:41] Checking for Bowtie Bowtie version:2.2.5.0 [2017-03-21 11:40:41] Checking for Bowtie index files (genome).. [2017-03-21 11:40:41] Checking for reference FASTA file [2017-03-21 11:40:41] Generating SAM header for /home/galaxy/tool-data/GRCm38_84/bowtie2_index/GRCm38_84/GRCm38_84 [2017-03-21 11:40:50] Preparing reads left reads: min. length=101, max. length=101, 37099122 kept reads (11326 discarded) right reads: min. length=101, max. length=101, 36957824 kept reads (152624 discarded) [2017-03-21 12:11:25] Mapping left_kept_reads to genome GRCm38_84 with Bowtie2 [FAILED] Error running bowtie: Error reading _plen[] array: 895748346, 4294967292 Error: Encountered internal Bowtie 2 exception (#1) Command: /home/galaxy/tools/dependencies/bowtie2/2.2.5/iuc/package_bowtie_2_2_5/30bd7eaeddbf/bin/bowtie2-align-s --wrapper basic-0 -k 20 -D 15 -R 2 -N 0 -L 20 -i S,1,1.25 --gbar 4 --mp 6,2 --np 1 --rdg 5,3 --rfg 5,3 --score-min C,-14,0 -p 8 --sam-no-hd -x /home/galaxy/tooldata/GRCm38_84/bowtie2_index/GRCm38_84/GRCm38_84 - (ERR): bowtie2-align exited with value 1 If this is not an extension problem then what is an origin of this error and how to solve it? Thanx, Nikolaos ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
Re: [galaxy-dev] Request more details from Nikolaos Tur about pyBamTools & pyBamParser dependency packages message
Hi, sorry for long delay answer. Solution which Peter suggested is absolutely right - owner of packages has to add changes. Because I did not have any reaction to my message but needed quick solution and wanted to stay in consistent with standard Galaxy installation procedure and neither do any fork of Daniel's packages and setup then local toolsched repository for tools or install these packages locally, I did some trick. First I installed these packages. Installation of course failed. Then I open them in web interface, uninstalled them, modified tool_dependencies.xml files (see below) and installed them again from the web interface. Sorry if I miss something that was several months ago. shell_command was modified. ... hg clone -r ba0341816b15fff5e0a554658d6579a60e1b12b1 https://bitbucket.org/dan/pybamtools pyBamTools $INSTALL_DIR/lib/python export PYTHONPATH=$PYTHONPATH:$INSTALL_DIR/lib/python && sed -i '/use_setuptools()/ s/^/#/' setup.py && sed -i '/from distribute_setup import use_set uptools/ s/^/#/' setup.py && python setup.py install --home $INSTALL_DIR $INSTALL_DIR/lib/python ... instead of export PYTHONPATH=$PYTHONPATH:$INSTALL_DIR/lib/python && python setup.py install --home $INSTALL_DIR Idea I think is clear, after clone of tools, modify first setup.py files and then compile and install. Hope it can be useful, Nikolaos. On Tue, Jan 10, 2017 at 12:54 PM, Jennifer Hsieh <humbleambit...@gmail.com> wrote: > Hi Peter, > Thank you so much. > I'll see what I can figure out with these pull requests. I'm still quite > inexperienced with these types of files. This is probably a good > opportunity to learn some new code/structures. > If I can't manage to figure it out, I hope there's another variant caller > with different dependencies I can use for my project. > > Best, > Jen > > > On 9 Jan 2017 4:06 p.m., "Peter Briggs" <peter.bri...@manchester.ac.uk> > wrote: > > Hello Jennifer > > I came across the same problem with the naive variant caller at the end of > last year, when trying to upgrade our local Galaxy instance. > > As you describe, the issue is actually with the two dependency packages: > > -- package_pybamparser_0_0_1 > -- package_pybamtools_0_0_2 > > I think I found the same solution as Nikolaos to the underlying problem > (i.e. the setup.py files in these two packages are no longer compatible > with the newer versions of the Python 'setuptools'), but I didn't have time > to follow it up then. > > I've now submitted pull requests with my attempted fixes on the Bitbucket > repositories which hold the source code for the dependencies (URLs below), > so at least you can see how I've tried to address the problem. However I > don't know of an easy way to apply these changes inside the Galaxy context > - my hope is that the tool maintainer (Daniel Blankenberg, cc'ed) will take > a look and update the dependencies on the toolshed. > > Not sure if this helps - maybe someone else on this list has a way to > apply the patches directly? > > Best wishes > > Peter > > Pull request URLs: > > -- https://bitbucket.org/dan/pybamtools/pull-requests/1/ > -- https://bitbucket.org/dan/pybamparser/pull-requests/3/ > > > > On 06/01/17 14:50, Jennifer Hsieh wrote: > >> Hi, >> I've installed a local instance of Galaxy (on Ubuntu 16.04, python >> 2.7.12) and have run in to this exact problem >> (posted by Nikolaos >> Tur https://www.mail-archive.com/galaxy-dev@lists.galaxyproject. >> org/msg03955.html) >> while trying to add naive_variant_caller tool. >> Naive_variant_caller can't install because these two dependency packages >> can't be installed. >> I basically get exactly the same error message that ends with "IOError: >> Could not build the egg." >> >> Unfortunately I'm very new to both Galaxy and Python (Taking Coursera >> Galaxy course at the moment) and I don't really understand the solution >> Nikolaos described about modifying the setup.py to use "setuptools" >> instead of "distribute". I don't even know where to find setup.py... >> >> Can anyone help me with step-by-step details as to how to solve this >> dependency problem? >> >> Much Thanks! >> -- >> Jennifer Hsieh >> PhD Neuroscience (student) >> Department of Psychiatry and Mental Health >> University of Cape Town >> Tell: +27 21 447 3157 >> Cell: +27 74 841 4761 >> >> >> >> >> >> ___ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this >> and o
[galaxy-dev] building indexes
Hi, I try to build indexes for ch38 (GRCh38.84 Toplevel) using data managers for BWA-MEM and TwoBit. Both failed. Error building index: Error in faToTwoBit, index overflow at CHR_HSCHR4_5_CTG12. The 2bit format does not support indexes larger than 4Gb, please split up into smaller files. [bwa_index] Update BWT... [bwt_bwtupdate_core] Failed to allocate 46629609168 bytes at bwtindex.c line 135: Cannot allocate memory Error building index. So, how to split genome for TwoBit and how much RAM is needed for BWA indexes? These errors are from runs on machine with 45GB RAM. Thanx, Nikolaos ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] several versions of compiler
Hi, On Fri, Sep 23, 2016 at 2:13 PM, Björn Grüning <bjoern.gruen...@gmail.com> wrote: > Hi, > > Am 23.09.2016 um 10:42 schrieb Nikolaos Tur: > > Hi, > > > > maybe this was discussed or documented already but I'm lost. How to tell > > installed tools by tool_shed that I have newer version of gcc and pls > > use that one instead of default? > > Do you want to recompile all tools installed from the TS with a new GCC > version? > just only one which requires more recent version of gcc Nikolaos ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] several versions of compiler
Hi, maybe this was discussed or documented already but I'm lost. How to tell installed tools by tool_shed that I have newer version of gcc and pls use that one instead of default? Thank you very much, Nikolaos ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] why modified workflow doesn't save?
Hi, after modification of workflow it does not save in workflow editor. Just shows saving bar for ever. In logs there are not so much info, just mentions that using version of tools are different from specified in this workflow. I work with "CloudMap Hawaiian ..." workflow. Galaxy is 16.04 with version of tools at June/July time. Workflow file is a text file and potentially can be modified by hands. In this case on what I have to pay attention to change it in text editor. Is there any tutorial for this? Thanx, Nikolaos. ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] CloudMap_Hawaiian_Variant & missing tools on new Galaxy instance
Hi, imported from usegalaxy.org workflow CloudMap_Hawaiian_Variant_Mapping_with_WGS to local Galaxy instance (16.04) shows missing tools or their different versions. For instance: *Tool name* Add or Replace Groups, *id* toolshed.g2.bx.psu.edu/repos/devteam/picard/picard_ARRG/1.56.0, *version* 1.56.0 *Tool name* Mark Duplicate reads, *id* toolshed.g2.bx.psu.edu/repos/devteam/picard/rgPicardMarkDups/1.56.0, *version* 1.56.0 Also it requires SNP 1.56 version. Our Galaxy instance has the latest tools version for 16.04. Does it mean I have to install older version of tools or modify workflow to use latest one? (As I see workflow asks for programs with different ids what was in older versions: picard_ARRG in 1.56 but it probably is already picard_AddOrReplaceReadGroups in 1.136. Or bedtools from aaronquinlan repository with id="bedtools_genomecoveragebed_bedgraph" and which is not available in 16.04 repositories but probably this program is in bedtools from iuc with id="bedtools_genomecoveragebed"). Also I see that tools on usegalaxy.org are even newer than our but this workflow is imported there without any errors. Does it mean that on usegalaxy.org there are many versions of the same tool but they are not visible in tools panel? Does some one have modified version of this workflow (or similar to try just cloudmap tests) to new Galaxy instance? Thanx, Nikolaos ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] flexbar v2.7 & required format extension
thanx Mina, I thought about this. If no one still did this, question is where to add soft link creation. Probably in flexbar.xml. Now flexbar supports fasta and fastaq format, also their gziped/bziped versions. How to determine data format in Galaxy? I'm not an expert neither in Galaxy nor biology, just try to administrate Galaxy instance. Thanx, Nikolaos On Sat, Jun 18, 2016 at 2:54 PM, Mina Bashir <mina.bas...@medunigraz.at> wrote: > Hi, > You can solve the extension issue using soft links in your tool wrapper. > E.g.: > ln -s $input_fasta $input_fasta.fna > > Best, > Mina > > Sent from my iPhone > > > On Jun 18, 2016, at 12:04, Nikolaos Tur <nikolaos@gmail.com> wrote: > > > > Hi, > > > > does someone use latest version of flexbar 2.7 with Galaxy? It seems > newest flexbar changes way how it works with data. Now it requires format > extension for data. But Galaxy operates data with .dat extension in > database/files/. > > Also there is flexbar wrapper for version 2.7, so it should be somehow > integrated to the Galaxy. Our Galaxy is 16.04. > > > > Thanx, > > Nikolaos > > ___ > > Please keep all replies on the list by using "reply all" > > in your mail client. To manage your subscriptions to this > > and other Galaxy lists, please use the interface at: > > https://lists.galaxyproject.org/ > > > > To search Galaxy mailing lists use the unified search at: > > http://galaxyproject.org/search/mailinglists/ > > ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
