[galaxy-dev] Re: King Skrupellos hack

[Daniel Blankenberg]

Hi Pat,

I wanted to make sure that you were able to figure this out. As suggested
previously, this is most likely just a user uploaded file, but if you are
an admin of the Galaxy instance then you should be able to figure it out.
If you want to learn a bit more about this specific dataset in your Galaxy
instance, log in as a Galaxy admin_user, and go to the show_params page,
i.e.: http://mbac.gmu.edu:8080/datasets/6d3270914ac86d96/show_params

As an admin user, you will be able to see the file location on disk, the
tool and command line used to generate the dataset, etc. Please let us know
how it goes.



Thanks for using Galaxy,

Dan


On Tue, Jul 30, 2019 at 8:23 PM Pat Gillevet  wrote:

> Dannon
>
> Here is the link and the corresponding page.  I agree that it is probably
> innocuous but I cannot find the data file in the galaxy database.
> Any hints so I can remove it would be welcome…   the GMU IT department is
> bugging me about it.
>
> thanks
> Pat
>
> http://mbac.gmu.edu:8080/datasets/6d3270914ac86d96/display
>
>
>
>
>
> On Jul 30, 2019, at 8:16 AM, Dannon Baker  dannon.ba...@gmail.com>> wrote:
>
> Sorry for the delayed response here, I think this fell through the cracks
> while folks were engaged with GCC this year.  The attachment didn't make it
> through, can you provide more information?  For what it's worth, we've seen
> 'hacks' like this in the past simply be that someone uploads data with spam
> text/links in the contents, so it may not be much to worry about.
>
> On Thu, Jul 11, 2019 at 7:44 AM Pat Gillevet  pgill...@gmu.edu>> wrote:
>
> Has anyone been hacked through the galaxy port 8080?  We have an aberrant
> link on our install and I cannot find where it is hidden.
>
> The message that comes up is attached.
>
> Pat
>
>
> 
> Patrick M. Gillevet, Ph.D
>Director, Microbiome Analysis Center
>   Professor, Department of Biology
> George Mason University, Science & Technology Campus
> 10900 University Boulevard, MSN 4D4
>  Manassas, Virginia  20110
> Office 703-993-1057 Room Colgan Hall-426 FAX 703-993-8430
>   http://mbac.gmu.edu<
> http://mbac.gmu.edu/>
>
> **
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   %(web_page_url)s
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/<
> http://secure-web.cisco.com/1WyIBPyi6RnnURKuDPdOUjWkO6rKvRhOFFsGHDo0sTYBVMPVsu1OZl92sB9OO94U0wIkC9EXXyEWONR8XQ91KFx-1Nd-5zCEsohAJBeiNBg5OA2i3Rklfy-jcXE0M_7L9y-BWSTSJ1krePydY_ATaXs5G9xRwy5RqEUmH9JQOeI3ZZYo1AfMZsyx6NpfHFSwasZp2ZcyycPIMVUoYeuvf_rgInFXh0Cyti1UCWlirO28oTOAhFECCkpGGXIfw9AID5wkUHakwRioe77BXz0Jr29WZ8WY1vS4Hl80mtVrArHvd22fGCPrRBvNqv5slknz_1tACzdrEzjYt-kic5dC1QByG__dVbIMFsffsBFnXbPqwY1E4K7kmLZ8MxjPUzdUO0JHJH0rdgue9YevRxPkhWk-MkQ_Vjq-kDYF-FzMtaMBMgU4NiEMCrMcCfzFNtQEb/http%3A%2F%2Fgalaxyproject.org%2Fsearch%2F
> >
>
>
> 
> Patrick M. Gillevet, Ph.D
>Director, Microbiome Analysis Center
>   Professor, Department of Biology
> George Mason University, Science & Technology Campus
> 10900 University Boulevard, MSN 4D4
>  Manassas, Virginia  20110
> Office 703-993-1057 Room Colgan Hall-426 FAX 703-993-8430
>   http://mbac.gmu.edu
>
> **
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   %(web_page_url)s
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  %(web_page_url)s

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] galaxy behind firewall

[Daniel Blankenberg]

HI Matthias,

At the firewall level, one could allow pass through to only the specific
IPs/sites that you want to enable for external displays, instead of just
opening up the headnode to the entire world. Additional access restrictions
could be enforced at the apache/nginx proxy level as well.

If opening the firewall is definitely a non-starter, you could implement a
Galaxy tool do do as you suggest, with the output of the tool being an HTML
dataset containing the link to the external site with the freshly uploaded
data/URL included as parameter. Doing this in a generic (all external
displays) fashion, would require a bit of development (which would probably
be better-spent adding this sort of behavior as a configurable backend
function of the external display application framework), but for a single
specific use-case, such as Phinch, is rather trivial.


Dan

On Tue, Sep 19, 2017 at 5:32 AM, Matthias Bernt  wrote:

> Hi,
>
> our galaxy is behind a firewall which seems to imply that many of the nice
>>> possibilities to use online services which download data from galaxy do not
>>> work, e.g. phinch or the UCSC browser.
>>>
>>> Has anybody ideas for (or experience with) workarounds?
>>>
>>
>> Maybe you can elaborate about these firewall issues first ? What do you
>> mean by "do not work" ?
>> Did you get in touch with your security team ?
>>
>
> For instance, biom files can be visualised at phinch. If a user clicks the
> link "view biom at Phinch" a new website is opened:
>
> http://www.bx.psu.edu/~dan/Phinch/index.html?biomURL=http%
> 3A%2F%2Fbioinf2-dev%3A8080%2Fdisplay_application%2F7aee12
> da589cb4d4%2Fbiom_simple%2Fphinch_dan%2F772601c34b88e564%2Fdata%2Fgalaxy_
> 7aee12da589cb4d4.biom
>
> so the phinch website tries to download the biom file from our local
> Galaxy bioinf2-dev which is behind the firewall. Similar mechanism is at
> work to visualize annotations at the UCSC genome browser.
>
> There is no way that our security team will open the firewall for galaxy
> which runs on a head node of our compute cluster.
>
> For instance, we have a local own cloud which we might exploit, i.e.,
>>> upload the data to OC and modify the link.
>>>
>>
>> What kind of integration are you intending with ownCloud exactly ?
>>
>
> I was just thinking that one could create a tool that uploads the data set
> to OC, makes it publicly accessible via a link, and redirects to a website
> (essentially replacing the link to the galaxy by the one to OC).
>
>
>
> Cheers,
> Matthias
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>  https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>  http://galaxyproject.org/search/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] Create a new datatype svg

[Daniel Blankenberg]

Hi Jochen,

I’m not sure I follow exactly. The SVG format is basically an XML file that can 
be visualized as a graphic. Whether it will be displayed as a graphic within 
your browser or as text generally depends upon the mime type that is set. If 
the mime type is set as text/plain, it will render as plain text, if it is set 
as image/svg+xml it will be rendered as an image; the previously mentioned flag 
will control which is used for this datatype. However, the actual file contents 
are the same.


Dan


> On Feb 8, 2017, at 2:46 AM, Jochen Bick <jochen.b...@usys.ethz.ch> wrote:
> 
> Thanks Dan,
> 
> I would like to have it as plain text but no as a separate file in the
> history but more like the solution of bam and bai files. Do you know how
> to do that?
> 
> Cheers Jochen
> 
> On 07.02.2017 21:26, Daniel Blankenberg wrote:
>> Just a quick note, if you want svg to be rendered as a graphics correctly 
>> via the eye preview, you’ll need to set “serve_xss_vulnerable_mimetypes = 
>> True" in galaxy.ini. Otherwise it will appear as plaintext xml. But do be 
>> aware of the security concerns if you change this setting.  
>> Downloading/saving the svg and then manually opening will work regardless.
>> 
>> 
>> Thanks for using Galaxy,
>> 
>> Dan
>> 
> 

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Create a new datatype svg

[Daniel Blankenberg]

Just a quick note, if you want svg to be rendered as a graphics correctly via 
the eye preview, you’ll need to set “serve_xss_vulnerable_mimetypes = True" in 
galaxy.ini. Otherwise it will appear as plaintext xml. But do be aware of the 
security concerns if you change this setting.  Downloading/saving the svg and 
then manually opening will work regardless.


Thanks for using Galaxy,

Dan


> On Feb 7, 2017, at 3:17 PM, Björn Grüning  wrote:
> 
> Hi,
> 
> just change it to svg. If the output is empty this could be a tool problem.
> 
> Ciao,
> Bjoern
> 
> Am 07.02.2017 um 14:19 schrieb Jochen Bick:
>> Hi Björn,
>> 
>> thanks, so I just change the format to svg? or is there more to do?
>> The output appears to be empty?
>> 
>> Cheers Jochen
>> 
>> 
>> On 07.02.2017 13:34, Björn Grüning wrote:
>>> Hi Jochen,
>>> 
>>> svg is supported, see here:
>>> 
>>> https://github.com/galaxyproject/galaxy/blob/dev/config/datatypes_conf.xml.sample#L282
>>> 
>>> Cheers,
>>> Bjoern
>>> 
>>> Am 07.02.2017 um 12:45 schrieb Jochen Bick:
 Hi all,
 
 I'm working on a tool that will have svg as an output file. Is this data
 type already established? If not how do I start?
 
 Cheers Jochen
 ___
 Please keep all replies on the list by using "reply all"
 in your mail client.  To manage your subscriptions to this
 and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/
 
 To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
 
>> 
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>  https://lists.galaxyproject.org/
> 
> To search Galaxy mailing lists use the unified search at:
>  http://galaxyproject.org/search/mailinglists/

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Request more details from Nikolaos Tur about pyBamTools & pyBamParser dependency packages message

[Daniel Blankenberg]

Hi all,

These packages should now install correctly from the toolshed with Peter’s PR. 
Please let me know if you encounter additional issues.



Thanks for using Galaxy,

Dan

> On Jan 31, 2017, at 10:47 PM, Jennifer Hsieh <humbleambit...@gmail.com> wrote:
> 
> Thank so much!
> This is exactly what I needed.
> What a gem :)
> 
> Best,
> Jen
> 
> On Tue, Jan 31, 2017 at 3:48 PM, Nikolaos Tur <nikolaos@gmail.com 
> <mailto:nikolaos@gmail.com>> wrote:
> Hi,
> 
> sorry for long delay answer. Solution which Peter suggested is absolutely 
> right - owner of packages has to add changes.
> Because I did not have any reaction to my message but needed quick solution 
> and wanted to stay in consistent with
> standard Galaxy installation procedure and neither do any fork of Daniel's 
> packages and setup then local toolsched 
> repository for tools or install these packages locally, I did some trick.
> 
> First I installed these packages. Installation of course failed. Then I open 
> them in web interface, uninstalled them, 
> modified  tool_dependencies.xml files (see below) and installed them again 
> from the web interface.
> Sorry if I miss something that was several months ago.
> 
> shell_command was modified.
> 
> ...
> 
> hg clone -r 
> ba0341816b15fff5e0a554658d6579a60e1b12b1 https://bitbucket.org/dan/pybamtools 
> <https://bitbucket.org/dan/pybamtools> pyBamTools
> $INSTALL_DIR/lib/python
> export 
> PYTHONPATH=$PYTHONPATH:$INSTALL_DIR/lib/python && sed -i '/use_setuptools()/ 
> s/^/#/' setup.py && sed -i '/from distribute_setup import use_set
> uptools/ s/^/#/' setup.py && python setup.py install --home 
> $INSTALL_DIR
> 
>  action="prepend_to">$INSTALL_DIR/lib/python
> 
> 
> ...
> 
> instead of 
> 
> export 
> PYTHONPATH=$PYTHONPATH:$INSTALL_DIR/lib/python && python setup.py install 
> --home $INSTALL_DIR
> 
> Idea I think is clear, after clone of tools, modify first setup.py files and 
> then compile and install.
> 
> Hope it can be useful,
> Nikolaos.
> 
> On Tue, Jan 10, 2017 at 12:54 PM, Jennifer Hsieh <humbleambit...@gmail.com 
> <mailto:humbleambit...@gmail.com>> wrote:
> Hi Peter, 
> Thank you so much. 
> I'll see what I can figure out with these pull requests. I'm still quite 
> inexperienced with these types of files. This is probably a good opportunity 
> to learn some new code/structures. 
> If I can't manage to figure it out, I hope there's another variant caller 
> with different dependencies I can use for my project. 
> 
> Best, 
> Jen 
> 
> 
> On 9 Jan 2017 4:06 p.m., "Peter Briggs" <peter.bri...@manchester.ac.uk 
> <mailto:peter.bri...@manchester.ac.uk>> wrote:
> Hello Jennifer
> 
> I came across the same problem with the naive variant caller at the end of 
> last year, when trying to upgrade our local Galaxy instance.
> 
> As you describe, the issue is actually with the two dependency packages:
> 
> -- package_pybamparser_0_0_1
> -- package_pybamtools_0_0_2
> 
> I think I found the same solution as Nikolaos to the underlying problem (i.e. 
> the setup.py files in these two packages are no longer compatible with the 
> newer versions of the Python 'setuptools'), but I didn't have time to follow 
> it up then.
> 
> I've now submitted pull requests with my attempted fixes on the Bitbucket 
> repositories which hold the source code for the dependencies (URLs below), so 
> at least you can see how I've tried to address the problem. However I don't 
> know of an easy way to apply these changes inside the Galaxy context - my 
> hope is that the tool maintainer (Daniel Blankenberg, cc'ed) will take a look 
> and update the dependencies on the toolshed.
> 
> Not sure if this helps - maybe someone else on this list has a way to apply 
> the patches directly?
> 
> Best wishes
> 
> Peter
> 
> Pull request URLs:
> 
> -- https://bitbucket.org/dan/pybamtools/pull-requests/1/ 
> <https://bitbucket.org/dan/pybamtools/pull-requests/1/>
> -- https://bitbucket.org/dan/pybamparser/pull-requests/3/ 
> <https://bitbucket.org/dan/pybamparser/pull-requests/3/>
> 
> 
> 
> On 06/01/17 14:50, Jennifer Hsieh wrote:
> Hi,
> I've installed a local instance of Galaxy (on Ubuntu 16.04, python
> 2.7.12) and have run in to this exact problem
> (posted by Nikolaos
> Tur 
> https://www.mail-archive.com/galaxy-dev@lists.galaxyproject.org/msg03955.html 
> <https://www.mail-archive.com/galaxy-dev@lists.galaxyproject.org/msg03955.html>)
> while trying to add naive_variant_caller tool.
> Naive_variant_caller can't install because these 

Re: [galaxy-dev] Question about Data Managers

[Daniel Blankenberg]

Hi Carlos,

You’ll want to set 'galaxy_data_manager_data_path' in your galaxy.ini file.


Thanks for using Galaxy,

Dan


> On Dec 21, 2016, at 3:17 PM, Carlos Lijeron  wrote:
> 
> Everyone,
>  
> I’m trying to setup various data managers to automatically download and index 
> reference indices.  However, I can’t figure out where to change the default 
> location where these are saved.   As you can see in the attached image,  the 
> default location for the indexes is the … /tool-data/dbkey/index_name 
> However, I would like to save them in /projects/reference_datasets which is a 
> shared location that has a lot of space available.
>  
> Please advise.
>  
> Thank you  all !
>  
>  
> Carlos.
>  
>  
> 
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>  https://lists.galaxyproject.org/ 
> 
> To search Galaxy mailing lists use the unified search at:
>  http://galaxyproject.org/search/mailinglists/ 
> 
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Galaxy Security Vulnerability - Arbitrary code execution

[Daniel Blankenberg]

The server is back up. Thanks for reporting this issue.

Dan

> On Dec 19, 2016, at 10:02 AM, Nicola Soranzo  wrote:
> 
> Thanks David,
> the issue has been already reported, see 
> https://github.com/galaxyproject/galaxy/issues/3332 
>  , hopefully someone 
> from the Galaxy Team will look into this soon.
> 
> You can get the patch from GitHub in the mean time, it has been applied to 
> all branches starting from release_14.10 .
> 
> Cheers,
> Nicola
> 
> On 19/12/16 14:39, Lapointe, David wrote:
>> This site does not seem to be working for getting patches. 
>> depot.galaxyproject.org
>> -- 
>> David Lapointe Ph.D.
>> Sr. Bioinformatics Specialist
>> Research Technology (RT)
>> Tufts Technology Services (TTS)
>> 16 Dearborn Road
>> Somerville MA 02144
>> 
>> Phone:  617-627-5319
>> Fax: 617-627-3667
>> http://it.tufts.edu 
>> 
>> From: galaxy-dev < 
>> galaxy-dev-boun...@lists.galaxyproject.org
>>  > on behalf of Nate 
>> Coraor >
>> Date: Thursday, December 15, 2016 at 3:13 PM
>> To: " 
>> galaxy-annou...@lists.galaxyproject.org
>>  " 
>> > >
>> Cc: Galaxy Committers > >, Galaxy Dev List 
>> > >
>> Subject: [galaxy-dev] Galaxy Security Vulnerability - Arbitrary code 
>> execution
>> 
>> *DESCRIPTION*
>> 
>> A security vulnerability was recently discovered by David Wyde in two Galaxy 
>> tools that are provided with and enabled by default in the Galaxy server. 
>> These vulnerabilities allow malicious actors to execute arbitrary code as 
>> the user running the Galaxy server.
>> 
>> The vulnerable tools are "Filter GFF data by attribute" and "Filter GFF data 
>> by feature count", two tools which share code with each other and the more 
>> general "Filter data on any column using simple expressions" tool. The 
>> latter was fixed in a previous security disclosure but these GFF variants of 
>> the tool were missed when updating the Filter tool.
>> 
>> These tools use the Python `eval` and `exec` functions and do not properly 
>> sanitize input to these functions.
>> 
>> Additionally, an XSS vulnerability with the upload tool: An uploaded file's 
>> name was not properly sanitized, and so a specially crafted filename 
>> uploaded to the Galaxy server could be used as an XSS attack vector. This 
>> vulnerability has already been fixed in the 16.07 and 16.10 Galaxy releases.
>> 
>> The Galaxy Committers would like to thank David for disclosing these 
>> vulnerabilities. If you discover security vulnerabilities in Galaxy, please 
>> report them to the Galaxy Committers at 
>> galaxy-committ...@lists.galaxyproject.org 
>> .
>> 
>> *AFFECTED VERSIONS*
>> 
>> These arbitrary code execution vulnerability is present in all known 
>> releases of Galaxy using the default tool config or a tool config in which 
>> these tools are loaded. To determine if your Galaxy server is affected, 
>> check your tool_conf.xml for the following lines:
>> 
>> 
>> 
>> 
>> If you are unsure, you can also check to see if the tools have loaded by 
>> searching for them in your Galaxy server logs, e.g.:
>> 
>>   % grep gff_filter paster.log
>>   galaxy.tools.toolbox.base DEBUG 2016-12-08 13:25:16,930 Loaded tool id: 
>> gff_filter_by_attribute, version: 0.1 into tool panel..
>>   galaxy.tools.toolbox.base DEBUG 2016-12-08 13:25:16,930 Loaded tool id: 
>> gff_filter_by_feature_count, version: 0.1 into tool panel..
>> 
>> *IMPACT*
>> 
>> This vulnerability can be exploited to execute arbitrary code as the user 
>> that runs Galaxy jobs on any system where Galaxy runs those jobs 
>> (potentially a cluster). Such code can be used to provide a remote shell to 
>> an attacker, or read/write/delete files that the Galaxy user has appropriate 
>> permissions on.
>> 
>> *SOLUTION*
>> 
>> Multiple solutions exist for the arbitrary code execution vulnerability:
>> 
>> 1. Update your Galaxy code. A fix for the issue has been applied to stable 
>> releases of Galaxy back to version 14.10.
>> 
>> 2. Disable the affected tools.
>> 
>> 3. Patch the affected tools. Patches to fix the vulnerabilities (depending 
>> on the version of Galaxy that you are running) can be found below.
>> 
>> *INSTRUCTIONS*
>> 
>> 1. To apply the fix, first identify your current Galaxy release version 
>> using the `git branch` command. If you are on a 'release_YY.MM' branch, you 
>> can update with:
>> 
>>   % git pull
>> 
>> The 

Re: [galaxy-dev] Galaxy Blast

[Daniel Blankenberg]

Hi Peter and Harry,

When a tool is installed from the toolshed, the tool data table loc files are 
installed to the configured location and are then referenced in 
shed_tool_data_table_conf.xml. You should be able to modify the .loc files as 
referenced in this file, or, as you did, add a new entry to the 
tool_data_table_conf.xml file. Each instance of the tool data table (one will 
exist per tool shed repository revision that has been installed) will be merged 
together into a single entity that can then be accessed by the tools. You can 
also find the paths to the .loc files for each tool data table under the admin 
interface. Data Managers will write to the specific .loc file that was created 
during their installation.

Please let me know if I can provide additional clarification. 



Thanks for using Galaxy,

Dan


> On Jul 26, 2016, at 6:08 AM, Peter Cock  wrote:
> 
> Hello all,
> 
> Harry (CC'd) emailed me about some teething trouble getting
> nucleotide BLAST databases to appear in the drop down lists
> (see below).
> 
> I suspect this is partly due to the partial data table sample that
> ships with Galaxy:
> 
> https://github.com/galaxyproject/galaxy/blob/dev/config/tool_data_table_conf.xml.sample
> 
> Can any of the Galaxy Data Manager expects advise please?
> 
> Thanks,
> 
> Peter
> 
> 
> -- Forwarded message --
> From: Harry Wright 
> Date: Tue, Jul 26, 2016 at 9:25 AM
> Subject: RE: Galaxy Blast
> To: Peter Cock 
> 
> 
> Hi Peter,
> 
> Sorry for the delay in response. If you think it would be useful please post 
> it.
> 
> I have downloaded what I thought was the newest versions of NCIB
> blast+ 2.2.31 from the toolshed.
> 
> The tool_data_table_conf.xml.sample I altered was in galaxy_dir/config/.
> 
> If you need anything else please ask.
> 
> Thanks
> Harry
> 
> -Original Message-
> From: Peter Cock [mailto:p.j.a.c...@googlemail.com]
> Sent: 22 July 2016 16:28
> To: Harry Wright
> Subject: Re: Galaxy Blast
> 
> Hi Harry,
> 
> Hmm. I'm a little puzzled. That file looks fine on our GitHub and on
> the Tool Shed:
> 
> https://github.com/peterjc/galaxy_blast/blob/master/tool-data/tool_data_table_conf.xml.sample
> 
> http://toolshed.g2.bx.psu.edu/view/devteam/ncbi_blast_plus (browse tip files)
> 
> Can you tell me exactly which version of things you got from the Tool
> Shed, and exactly which tool_data_table_conf.xml and
> tool_data_table_conf.xml.sample you had to edit?
> 
> My guess is the problem is this partial example that ships with Galaxy:
> 
> https://github.com/galaxyproject/galaxy/blob/dev/config/tool_data_table_conf.xml.sample
> 
> (There is talk of moving datatypes like this back into the Galaxy
> core, which would simply this)
> 
> May I forward this email to the public Galaxy developers mailing list please?
> 
> Thanks,
> 
> Peter
> 
> On Fri, Jul 22, 2016 at 3:59 PM, Harry Wright
>  wrote:
>> 
>> 
>> Dear Peter,
>> 
>> 
>> 
>> I have just installed a new instance of Galaxy here at Moredun and
>> have been busy setting up tools for us to use. One of the tools that I
>> have set up is blast with the NCBI nr and nt databases sitting in the
>> backend as a Locally installed database. I downloaded both files and
>> formatted them into databases. I then add lines to the blastdb_p.loc
>> and blastdb.loc. When I restarted the server I found that I could
>> access nr through the interface, for blastx, but not nt, using blastn.
>> For nt I was getting the message “No option available” in the
>> Nucleotide BLAST database drop down when I selected “Locally installed BLAST 
>> database”.
>> 
>> 
>> 
>> After some searching I found reference to
>> config/tool_data_table_conf.xml.sample. When I look there I could only
>> find reference to blastdb_p and not blastdb.
>> 
>> 
>> 
>> I added the lines below to the file and changed the file name to
>> tool_data_table_conf.xml. I  restarted galaxy to find that I could now
>> use nt in blastn.
>> 
>> 
>> 
>> 
>> 
>>
>> 
>>value, name, path
>> 
>>
>> 
>>
>> 
>> 
>> 
>> I was wondering if there was something that I have missed or has this
>> section of the code been missed out of the tool_data_table_conf.xml
>> file in the installation download?
>> 
>> 
>> 
>> Regards
>> 
>> Harry
>> 
>> 
>> 
>> Dr. Harry Wright
>> 
>> Moredun Research Institute
>> 
>> Pentlands Science Park
>> 
>> Bush Loan
>> 
>> Penicuik
>> 
>> Midlothian EH26 0PZ
>> 
>> Tel - 0131 4455111
>> 
>> Fax - 0131 4456235
>> 
>> 
>> 
>> The Moredun Research Institute is a limited company registered in Scotland.
>> Registered number: SC149440
>> 
>> The Moredun Research Institute is a charity registered in Scotland.
>> Registered number: SC022353
>> 
>> Registered office: Moredun Research Institute, Pentlands Science Park,
>> Bush Loan, Penicuik, Midlothian, EH26 0PZ, UK
>> 
>> 
> 
> 
> 
> The Moredun Research Institute is a limited 

Re: [galaxy-dev] Troubleshooting BAM file upload issues

[Daniel Blankenberg]

Hi Amelia,

I’ve provided a patch below that will disable the html check, so you might be 
able to see the uploaded file content to help with debugging.


Thanks,

Dan


$ git diff
diff --git a/lib/galaxy/datatypes/sniff.py b/lib/galaxy/datatypes/sniff.py
index ff9584d..5b608ee 100644
--- a/lib/galaxy/datatypes/sniff.py
+++ b/lib/galaxy/datatypes/sniff.py
@@ -440,8 +440,6 @@ def handle_uploaded_dataset_file( filename, 
datatypes_registry, ext='auto', is_m
 if check_binary( filename ):
 if not Binary.is_ext_unsniffable(ext) and not 
datatypes_registry.get_datatype_by_extension( ext ).sniff( filename ):
 raise InappropriateDatasetContentError( 'The binary uploaded file 
contains inappropriate content.' )
-elif check_html( filename ):
-raise InappropriateDatasetContentError( 'The uploaded file contains 
inappropriate HTML content.' )
 return ext
 
 AUTO_DETECT_EXTENSIONS = [ 'auto' ]  # should 'data' also cause auto detect?
diff --git a/tools/data_source/upload.py b/tools/data_source/upload.py
index 59fa187..0d659c8 100644
--- a/tools/data_source/upload.py
+++ b/tools/data_source/upload.py
@@ -280,11 +280,6 @@ def add_file( dataset, registry, json_file, output_path ):
 err_msg = "You must manually set the 'File Format' 
to '%s' when uploading %s files." % ( ext.capitalize(), ext )
 file_err( err_msg, dataset, json_file )
 return
-if not data_type:
-# We must have a text file
-if check_html( dataset.path ):
-file_err( 'The uploaded file contains inappropriate HTML 
content', dataset, json_file )
-return
 if data_type != 'binary':
 if link_data_only == 'copy_files':
 if dataset.type in ( 'server_dir', 'path_paste' ) and 
data_type not in [ 'gzip', 'bz2', 'zip' ]:




On Mar 25, 2016, at 2:32 PM, Amelia Ireland <airel...@lbl.gov> wrote:

> Hi Christian,
> 
> Thanks for the tip. I tried this, but to no avail. I think that if the 
> problem was with the file ordering, I wouldn't be able to upload it 
> successfully to other galaxy installations (e.g. the main Galaxy server, 
> Galaxy install on my laptop).
> 
> It seems as though the issue is either the file upload process somehow doing 
> weird things to the file, or something going wrong in launching samtools and 
> running the script to process the uploaded file. 
> 
> Thanks,
> Amelia.
> 
> On 24 March 2016 at 02:03, Christian Brenninkmeijer 
> <christian.brenninkmei...@manchester.ac.uk> wrote:
> Try using samtools to sort the BAM before uploading.
> 
> samtools sort original.bam  sorted.bam
> 
> This will help with two things.
> 
> 1. Check the original file is not corrupt
> 
> 2. Put the BAM in the only order galaxy will accept.
> 
> Christian
> University of Manchester
> From: galaxy-dev [galaxy-dev-boun...@lists.galaxyproject.org] on behalf of 
> Amelia Ireland [airel...@lbl.gov]
> Sent: Wednesday, March 23, 2016 8:24 PM
> To: Daniel Blankenberg
> Cc: galaxy-dev@lists.galaxyproject.org
> Subject: Re: [galaxy-dev] Troubleshooting BAM file upload issues
> 
> I'm using the standard 'Upload file' tool, and choosing some local files. I 
> can successfully upload these files to the galaxy installation on my laptop 
> and to the usegalaxy.org server, but uploading them to the galaxy 
> installation on the server results in their appearing to be empty and the 
> error message about uploaded file containing inappropriate content.
> 
> I'm not sure where to proceed from here. Some things I'd like to explore, but 
> am not sure how to do so in Galaxy world, are:
> 
> - check that the file has uploaded properly
> - manually run the tool script on the input file to ensure there's not some 
> error that I'm missing
> 
> 
> 
> 
> On 23 March 2016 at 12:53, Daniel Blankenberg <d...@bx.psu.edu> wrote:
> Hi Amelia,
> 
> How are you uploading the files? 
> 
> If you are using a copy and pasted URL, can you make sure that the example 
> file exists and is accessible to the machine where the Galaxy job is running. 
> 
> A 404 error, for example, can often masquerade as a “The uploaded file 
> contains inappropriate HTML content” error (although this shouldn’t be the 
> outcome of this tool, it should be a red dataset stating a 404 error, in this 
> case).
> 
> 
> 
> Thanks for using Galaxy,
> 
> Dan
> 
> 
> 
> 
> On Mar 23, 2016, at 3:30 PM, Amelia Ireland <airel...@lbl.gov> wrote:
> 
>> On 23 March 2016 at 01:00, Björn Grüning <bjoern.gruen...@gmail.com> wrote:
>> Hi Amelia,
>> 
>> can you pleae check if you have samtools installed in your 

Re: [galaxy-dev] Troubleshooting BAM file upload issues

[Daniel Blankenberg]

Hi Amelia,

How are you uploading the files? 

If you are using a copy and pasted URL, can you make sure that the example file 
exists and is accessible to the machine where the Galaxy job is running. 

A 404 error, for example, can often masquerade as a “The uploaded file contains 
inappropriate HTML content” error (although this shouldn’t be the outcome of 
this tool, it should be a red dataset stating a 404 error, in this case).



Thanks for using Galaxy,

Dan





On Mar 23, 2016, at 3:30 PM, Amelia Ireland  wrote:

> On 23 March 2016 at 01:00, Björn Grüning  wrote:
> Hi Amelia,
> 
> can you pleae check if you have samtools installed in your PATH?
> 
> I've got it in my tool dependencies dir and it's also available as an 
> Environment module.
> 
> The tools dependencies file looks like this:
> 
> 
> 
>   
>   
>   
>   
>/>
> 
> 
> Debug messages when the tool runs:
> 
> galaxy.tools.deps DEBUG 2016-03-23 12:23:05,556 Building dependency shell 
> command for dependency 'samtools'
> galaxy.tools.deps DEBUG 2016-03-23 12:23:05,556 Find dependency samtools 
> version None
> galaxy.tools.deps DEBUG 2016-03-23 12:23:05,556 Resolver tool_shed_packages 
> returned  
> (isnull? True)
> galaxy.tools.deps DEBUG 2016-03-23 12:23:05,556 Resolver galaxy_packages 
> returned  
> (isnull? True)
> galaxy.tools.deps DEBUG 2016-03-23 12:23:05,556 Resolver galaxy_packages 
> returned  
> (isnull? True)
> galaxy.tools.deps DEBUG 2016-03-23 12:23:05,556 Resolver modules returned 
>  
> (isnull? False)
> galaxy.jobs.runners DEBUG 2016-03-23 12:23:05,567 (217) command is: 
> /global/homes/a/aireland/galaxy/database/job_working_directory/000/217/tool_script.sh;
>  return_code=$?; 
> MODULEPATH=/usr/share/Modules:/usr/share/Modules/modulefiles:/etc/modulefiles:/usr/common/usg/Modules/modulefiles:/usr/syscom/nsg/modulefiles:/usr/common/jgi/Modules/modulefiles;
>  export MODULEPATH; eval `/usr/bin/modulecmd sh load samtools`; python 
> "/global/homes/a/aireland/galaxy/database/job_working_directory/000/217/set_metadata_NFVQiC.py"
>  "/global/homes/a/aireland/galaxy/database/tmp/tmp5vpBro" 
> "/global/homes/a/aireland/galaxy/database/job_working_directory/000/217/galaxy.json"
>  
> "/global/homes/a/aireland/galaxy/database/job_working_directory/000/217/metadata_in_HistoryDatasetAssociation_349_ZqiKgT,/global/homes/a/aireland/galaxy/database/job_working_directory/000/217/metadata_kwds_HistoryDatasetAssociation_349_CH5HZd,/global/homes/a/aireland/galaxy/database/job_working_directory/000/217/metadata_out_HistoryDatasetAssociation_349_Hb4hhP,/global/homes/a/aireland/galaxy/database/job_working_directory/000/217/metadata_results_HistoryDatasetAssociation_349_2e3dj6,/global/homes/a/aireland/galaxy/database/files/000/dataset_320.dat,/global/homes/a/aireland/galaxy/database/job_working_directory/000/217/metadata_override_HistoryDatasetAssociation_349_pEc2Gh"
>  5242880; sh -c "exit $return_code"
> galaxy.jobs.runners.local DEBUG 2016-03-23 12:23:05,583 (217) executing job 
> script: 
> /global/homes/a/aireland/galaxy/database/job_working_directory/000/217/galaxy_217.sh
> galaxy.jobs DEBUG 2016-03-23 12:23:05,599 (217) Persisting job destination 
> (destination id: local:///)
> galaxy.jobs.runners.local DEBUG 2016-03-23 12:23:05,799 execution finished: 
> /global/homes/a/aireland/galaxy/database/job_working_directory/000/216/galaxy_216.sh
>  
> 
> Am 22.03.2016 um 19:28 schrieb Amelia Ireland:
> > I'm setting up a Galaxy instance on a new server and am having some
> > issues with BAM file uploads. I'm using the BAM files supplied with
> > JBrowse for testing purposes. I have Galaxy installed on my laptop and
> > file uploads are working fine, so it's evidently an issue with the
> > server and/or OS configuration.
> >
> > Attempting to upload a BAM file returns the message
> >
> > "The uploaded binary file contains inappropriate content"
> >
> > and the upload is empty.
> >
> > The server is running Scientific Linux and uses the Environment Modules
> > system for dependency management. I can successfully upload bed and
> > bigwig files, which suggests that samtools is being loaded and
> > functioning correctly.
> >
> > If someone can suggest some troubleshooting steps, I would be most grateful.
> >
> > Thanks!
> > Amelia
> >
> > Integrated Microbial Genomics (IMG)
> > JGI / LBL
> >
> >
> >
> > ___
> > Please keep all replies on the list by using "reply all"
> > in your mail client.  To manage your subscriptions to this
> > and other Galaxy lists, please use the interface at:
> >   https://lists.galaxyproject.org/
> >
> > To search Galaxy mailing lists use the unified search at:
> >   http://galaxyproject.org/search/mailinglists/
> >
> 
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy 

Re: [galaxy-dev] Query

[Daniel Blankenberg]

Hi Chang-Hyuk,


If you click on the bug icon for your error'd dataset you should be able to get 
the full traceback.


Thanks for using Galaxy,


Dan



> Dear Sir,
> 
> that is all of error message.
> I attached the file to be tried.
> 
> best regards,
> Chang-Hyuk Yoo.
> 
> 2016-02-25 오전 4:13에 Vipin TS 이(가) 쓴 글:
>> Hi Chang-Hyuk, 
>> 
>> Looks like the traceback truncated, Could you please pass me the complete 
>> error message from the job run, then I will have a look.   
>> 
>> Vipin | Rätsch Lab
>> 
>> On Tue, Feb 23, 2016 at 11:04 PM, 유창혁  wrote:
>> Dear Sir,
>> 
>> I used your "Convert Format" from gff file of NCBI.
>> but I encounter the following message.
>> What can I do?
>> 
>> Best regards,
>> Chang-Hyuk Yoo.
>> Traceback (most recent call last):
>>   File 
>> "/home/galaxy/shed_tools/toolshed.g2.bx.psu.edu/repos/vipints/fml_gff3togtf/5c6f33e20fcc/fml_gff3togtf/gff_to_gtf.py",
>>  line 76, in 
>> printGTF(Transcriptdb) 
>>   File "/home/galaxy/shed_tools/toolshed.g2.
>> 
>> 
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>   https://lists.galaxyproject.org/
>> 
>> To search Galaxy mailing lists use the unified search at:
>>   http://galaxyproject.org/search/mailinglists/
>> 
> 
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>  https://lists.galaxyproject.org/
> 
> To search Galaxy mailing lists use the unified search at:
>  http://galaxyproject.org/search/mailinglists/

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] BED12 Data Format

[Daniel Blankenberg]

Hi Lance,

Ah, yes, using bed12 would be problematic, as all bedstrict types are currently 
set to not be uploadable or datatype assignable. This stems from a long history 
of abuse of the BED format in Galaxy (where datasets should have been generic 
‘interval’ matching bed metadata of chrome,start,end of 1,2,3). I added these 
datatypes to force conversions of bed/interval files to Real bed/6/12 files, 
especially as needed by external visualization tools — that way you can click 
on a ‘bed’ (fake) file and Galaxy will convert it to Real bed and load the 
external service, or run the Galaxy tool on the properly formatted data.

Basically, there are restrictions in place to try to ensure that a bedstrict 
datatype is actually BED conforming. I am open to loosening these restrictions 
up, however, and allowing them to function as ’normal' datatypes (users would 
be free to shoot themselves in the foot by mis-assigning the datatype). In the 
meantime, you should be able to test this by flipping ‘allow_datatype_change’ 
in interval.py to True.

Thoughts?


Thanks,

Dan



On Jan 27, 2016, at 3:38 PM, Lance Parsons <lpars...@princeton.edu> wrote:

> Thanks for the info Dan. I've explored using bed12, but I have a few 
> questions.
> 
> 1. When attempting to use 'bed12' or 'Bed12' as a file type in a test, I get 
> the following error: 
> 
> Exception: {u'message': {u'type': u'error', u'data': {u'file_type': u"An 
> invalid option was selected for file_type, u'Bed12', please verify.", 
> u'files_metadata': [u"An invalid option was selected for file_type, u'Bed12', 
> please verify."]}}}
> 
> 2. I see that Bed12 is in the datatypes_conf.sample file. Is there a way to 
> add a converter for that datatype? Perhaps something 
> like:https://wiki.galaxyproject.org/ToolShedDatatypesFeatures#Including_datatype_converters_and_display_applications?
>  My concern is that since it already exists, I wouldn't be able to add a 
> converter. Also, the sniffer doesn't seem to work (it just finds the files as 
> "bed", thus my desire to specify ftype in tests).
> 
> Thanks,
> Lance
> 
> Daniel Blankenberg wrote:
>> 
>> Hi Lance,
>> 
>> FWIW, there is an existing bedstrict and bed12 (and bed6) datatypes in 
>> Galaxy. The strict datatypes are currently usually created by implicit 
>> datatype converters and are most often used by some external display 
>> applications that need standards conforming files.  bed6/12 are subclasses 
>> of bedstrict. They can of course be consumed or created by any sort of tool. 
>> Please let us know if we can provide additional information.
>> 
>> 
>> Thanks for using Galaxy,
>> 
>> Dan
>> 
>> 
>> On Jan 14, 2016, at 4:26 PM, Lance Parsons<lpars...@princeton.edu>  wrote:
>> 
>>> 
>>> Does anyone know of any efforts to create a BED12 datatype for Galaxy? 
>>> Since some tools require BED12 and the automatic convertion from GFF-to-BED 
>>> does not seem to generate a BED12, it seems it might be a worthwhile 
>>> addition.
>>> 
>>> If not, what would be the best way to go about doing this? Making it part 
>>> of the core galaxy (which would allow multiple tools to share the same data 
>>> type definitions) or making it part of a toolshed tool (which I'm not sure 
>>> how to do)? BTW, I'm thinking about RSeQC at the moment, but I know other 
>>> tools use/require this format.
>>> 
>>> -- 
>>> Lance Parsons - Scientific Programmer
>>> Carl C. Icahn Laboratory - Room 141 (Temporary)
>>> Lewis-Sigler Institute for Integrative Genomics
>>> Princeton University
>>> 
>>> _______
>>> Please keep all replies on the list by using "reply all"
>>> in your mail client.  To manage your subscriptions to this
>>> and other Galaxy lists, please use the interface at:
>>> https://lists.galaxyproject.org/
>>> 
>>> To search Galaxy mailing lists use the unified search at:
>>> http://galaxyproject.org/search/mailinglists/
>> 
> 
> 
>> Daniel Blankenberg January 25, 2016 at 9:47 AM
>> Hi Lance,
>> 
>> FWIW, there is an existing bedstrict and bed12 (and bed6) datatypes in 
>> Galaxy. The strict datatypes are currently usually created by implicit 
>> datatype converters and are most often used by some external display 
>> applications that need standards conforming files. bed6/12 are subclasses of 
>> bedstrict. They can of course be consumed or created by any sort of tool. 
>> Please let us know if we can provide additional information.
>>

Re: [galaxy-dev] BED12 Data Format

[Daniel Blankenberg]

Hi Lance,

FWIW, there is an existing bedstrict and bed12 (and bed6) datatypes in Galaxy. 
The strict datatypes are currently usually created by implicit datatype 
converters and are most often used by some external display applications that 
need standards conforming files.  bed6/12 are subclasses of bedstrict. They can 
of course be consumed or created by any sort of tool. Please let us know if we 
can provide additional information.


Thanks for using Galaxy,

Dan


On Jan 14, 2016, at 4:26 PM, Lance Parsons  wrote:

> Does anyone know of any efforts to create a BED12 datatype for Galaxy? Since 
> some tools require BED12 and the automatic convertion from GFF-to-BED does 
> not seem to generate a BED12, it seems it might be a worthwhile addition.
> 
> If not, what would be the best way to go about doing this? Making it part of 
> the core galaxy (which would allow multiple tools to share the same data type 
> definitions) or making it part of a toolshed tool (which I'm not sure how to 
> do)? BTW, I'm thinking about RSeQC at the moment, but I know other tools 
> use/require this format.
> 
> -- 
> Lance Parsons - Scientific Programmer
> Carl C. Icahn Laboratory - Room 141 (Temporary)
> Lewis-Sigler Institute for Integrative Genomics
> Princeton University
> 
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
> https://lists.galaxyproject.org/
> 
> To search Galaxy mailing lists use the unified search at:
> http://galaxyproject.org/search/mailinglists/

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Pass Working Directory from xml

[Daniel Blankenberg]

Hi Todd,

Can you just pass in “.”?


   the_tool.py -o ‘.'


Thanks for using Galaxy,

Dan


On Dec 11, 2015, at 4:00 PM, Todd Oakley  wrote:

> Greetings,
>I feel this must be simple, but I cannot find it.
> 
> I'm making a wrapper for a tool that requires the output directory to be 
> specified on the command line of a python script (otherwise it uses the path 
> of the input file for the multiple output files). How can I pass the working 
> directory to the tool from the xml file?
> 
> 
>the_tool.py -o $DIRECTORY_WHERE_GALAXY_RUNS_THE_TOOL
> 
> This is similar to from_work_dir in the outputs -- but I need the path of 
> the directory that does not include any file names.
> 
> 
> Thanks!
> 
> Todd Oakley, UCSB
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
> https://lists.galaxyproject.org/
> 
> To search Galaxy mailing lists use the unified search at:
> http://galaxyproject.org/search/mailinglists/

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] QIIME tools for Galaxy (WIP, call for collaborators)

[Daniel Blankenberg]

Hi Lance,

I looked at this a bit ago and had similar concerns, particularly with the 
outputs and inputs not being well-defined. In addition to the output tar ball 
—> local, extract —> upload not being great, as you mention, the input 
datatypes, etc, could use some work — in the very least, we should definitely 
create a nice biom datatype and have some converters available (import and 
export).  

Definitely worth spending some extra time to make sure that we have the data 
flowing well between each of the different parts/tools, and even better to make 
sure that its done in a way that allows mixing and matching with other 
non-qiime tools.


One thing that we want to avoid is large amounts of manual massaging of the 
automatically generated xml; fixing things up once might not be too bad, but 
having to do it with each new tool version can be “frustrating". Although 
perhaps having a good starting point and only needing to manually modify for 
any updates could be good enough (I’m not very familiar with the extent of 
typical changes between qiime versions to make a call on how much changes).




Dan

(resending since I received a message bounce from list)


On Oct 6, 2015, at 9:59 AM, Daniel Blankenberg <d...@bx.psu.edu> wrote:

> Hi Lance,
> 
> I looked at this a bit ago and had similar concerns, particularly with the 
> outputs and inputs not being well-defined. In addition to the output tar ball 
> —> local, extract —> upload not being great, as you mention, the input 
> datatypes, etc, could use some work — in the very least, we should definitely 
> create a nice biom datatype and have some converters available (import and 
> export).  
> 
> Definitely worth spending some extra time to make sure that we have the data 
> flowing well between each of the different parts/tools, and even better to 
> make sure that its done in a way that allows mixing and matching with other 
> non-qiime tools.
> 
> 
> One thing that we want to avoid is large amounts of manual massaging of the 
> automatically generated xml; fixing things up once might not be too bad, but 
> having to do it with each new tool version can be “frustrating". Although 
> perhaps having a good starting point and only needing to manually modify for 
> any updates could be good enough (I’m not very familiar with the extent of 
> typical changes between qiime versions to make a call on how much changes).
> 
> 
> 
> 
> Dan
> 
> 
> On Oct 5, 2015, at 5:26 PM, Lance Parsons <lpars...@princeton.edu> wrote:
> 
>> I was recently asked if I could provide a QIIME analysis pipeline for 16S 
>> data in Galaxy using tools in the QIIME pipeline (http://qiime.org/).
>> 
>> I did a bit of looking around for existing Galaxy wrappers and found an 
>> application that generates the wrappers for QIIME scripts for Galaxy 
>> (https://github.com/qiime/qiime-galaxy). This is a very well written 
>> application that does a great job of wrapping the QIIME scripts for Galaxy. 
>> However, there are a few things about it that don't quite fit my needs.
>> 
>> 1. The tools output tgz files of all of the output files. This means that to 
>> execute a pipeline, the user would have to download the tgz files, untar, 
>> and then upload whichever file(s) are needed for the next step.
>> 2. There is no toolshed repository to install the dependencies for these 
>> tools, making it a tricky for administrators to automate and also maintain 
>> various versions of QIIME going forward.
>> 3. There are no toolshed versions of the tools themselves, which also makes 
>> installation and integration a bit tricky and makes it hard to me to create 
>> and manage updates, fixes, tweaks, etc. There are also no tests, etc.
>> 
>> For these reasons I decided to investigate the feasibility of using the 
>> generated wrappers as a basis for a "toolshed" version of QIIME. If anyone 
>> is interested in helping, or has suggestions, or is working on something 
>> related, I'd be very happy to collaborate.
>> 
>> The repository for the WIP is at 
>> https://github.com/lparsons/galaxy_tools/tree/qiime/tools/qiime1.9.0. There 
>> is also a package on the testtoolshed as well as a first pass at 
>> package_qiime_1_9_1 
>> (https://github.com/lparsons/galaxy_tools/tree/qiime/packages/package_qiime_1_9_1).
>> 
>> -- 
>> Lance Parsons - Scientific Programmer
>> 134 Carl C. Icahn Laboratory
>> Lewis-Sigler Institute for Integrative Genomics
>> Princeton University
>> 
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To ma

Re: [galaxy-dev] Workflow editor stuck on loading workflow

[Daniel Blankenberg]

Just a quick general suggestion: have you tried clearing your browser cache?


Thanks for using Galaxy,

Dan



On Jul 15, 2015, at 11:18 AM, Ryan G ngsbioinformat...@gmail.com wrote:

 The error I see is TypeError: c.effectiveMapOver is not a function in 
 static/scripts/galaxy.workflow_editor.canvas.js?v=1436897004
 
 On Tue, Jul 14, 2015 at 4:50 PM, Björn Grüning bjoern.gruen...@gmail.com 
 wrote:
 Hi Ryan,
 
 try to look at the JS console. I suspect some problem with one of the
 tools in your workflow.
 
 Cheers,
 Bjoern
 
 Am 14.07.2015 um 22:02 schrieb Ryan G:
  Hmmm this only happens on a specific workflow.  How can I debug this?
 
  On Tue, Jul 14, 2015 at 3:37 PM, Ryan G ngsbioinformat...@gmail.com wrote:
 
  It looks like a recent commit since 6/26 to 15.05 is break the workflow
  editor.  I have a pipeline I'm trying to edit.  I updated my instance of
  Galaxy and now when I try to edit a workflow, the editor is stuck at
  Loading workflow editor.
 
 
 
 
 
 
  ___
  Please keep all replies on the list by using reply all
  in your mail client.  To manage your subscriptions to this
  and other Galaxy lists, please use the interface at:
https://lists.galaxyproject.org/
 
  To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
 
 
 ___
 Please keep all replies on the list by using reply all
 in your mail client.  To manage your subscriptions to this
 and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/
 
 To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] galaxy.ini

[Daniel Blankenberg]

It is under config/. You’ll want to copy it from  /config/galaxy.ini.sample

Thanks for using Galaxy,

Dan


On Jul 14, 2015, at 6:35 AM, Bryan Hepworth bryan.hepwo...@newcastle.ac.uk 
wrote:

 Hi
 
 For the woefully inept, where in the heck is galaxy.ini stored?
 
 I saw the documentation that said change 127.0.0.1 to 0.0.0.0 and the file 
 name, can't find it to do so.
 
 I know it all works as I've installed a module and that all shows up and 
 works on localhost.
 
 Thanks
 
 Bryan
 ___
 Please keep all replies on the list by using reply all
 in your mail client.  To manage your subscriptions to this
 and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/
 
 To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
 

___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] [SPAM] RE: GalaxyAdmins Online Meetup this Thursday: Peter Briggs on Galactic Engineering at Manchester

[Daniel Blankenberg]

We’re going to try this one now: https://connect.johnshopkins.edu/r283zwzv0yi


On Jun 18, 2015, at 11:09 AM, Hans-Rudolf Hotz hansrudolf.h...@fmi.ch wrote:

 sorry my (copy paste) mistake
 
 https://connect.johnshopkins.edu/galaxyadmins201506/
 
 
 On Jun 18, 2015, at 5:08 PM, Christian Brenninkmeijer 
 christian.brenninkmei...@manchester.ac.uk wrote:
 
 Does anyone have a link to the meeting?
 Page below only has a general link not to this meeting.
 From: galaxy-dev [galaxy-dev-boun...@lists.galaxyproject.org] on behalf of 
 Hans-Rudolf Hotz [hansrudolf.h...@fmi.ch]
 Sent: Thursday, June 18, 2015 4:03 PM
 To: Galaxy Dev List
 Subject: Re: [galaxy-dev] GalaxyAdmins Online Meetup this Thursday: Peter 
 Briggs on Galactic Engineering at Manchester
 
 Hi everyone, the online meetup will (hopefully) start in a few minutes at:
 
 https://wiki.galaxyproject.org/Community/GalaxyAdmins/Meetups/2015_06_18
 
 
 
 Regards, Hans-Rudolf
 
 
 On Jun 15, 2015, at 3:47 PM, Dave Clements cleme...@galaxyproject.org 
 wrote:
 
 Hello all,
 
 The June 2015 online GalaxyAdmins meetup will be this Thursday, 18 June, at 
 5pm Central European Time (see your local time).
 
 Peter Briggs of the University of Manchester and author of the Galactic 
 Engineer blog will talk about his Galaxy work deploying Galaxy and 
 developing tools within the Bioinformatics Core Facility at the University 
 of Manchester.
 
 John Chilton will also give an update on recent and upcoming work on 
 Planemo, a set of command-line utilities to assist in building and 
 publishing Galaxy tools.
 
 We'll also discuss the upcoming GalaxyAdmins meetup, and the hackathons at 
 GCC2015.
 
 We'll post a link on the meetup page on Thursday.  
 
 Hope to see/hear you on Thursday,
 
 Dave C 
 
 -- 
 http://galaxyproject.org/
 http://getgalaxy.org/
 http://usegalaxy.org/
 https://wiki.galaxyproject.org/
 ___
 Please keep all replies on the list by using reply all
 in your mail client.  To manage your subscriptions to this
 and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/
 
 To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
 
 
 
 ___
 Please keep all replies on the list by using reply all
 in your mail client.  To manage your subscriptions to this
 and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/
 
 To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] code tagset in tools

[Daniel Blankenberg]

The code tag won’t be going away any time soon, but in many cases there are 
other ways to do what this tag does, using the built-in dynamic options (in 
particular, currently for tab-delimited files), that are much preferred. So it 
is deprecated, and it should be used sparingly, but there are times when there 
isn’t a better way for tools to dynamically build their options without using a 
code file.

In the case of the get microbial data tool, if it was working prior to 15.03, 
it should still be working, afaik we haven’t changed the handling of these 
recently (with the exception of not loading them inside of a ToolShed 
application instance).  Its been a while since I’ve played with that specific 
tool (and we deprecated it ourselves and removed it from main), so I can’t 
confirm off-the-cuff whether it should still be working or not. If something 
really did break with respect to the code files, we should take a look though. 


I think using a Data Library is the perfect replacement for for the microbial 
import tool, and would cause less file duplication when used. But what exactly 
is no longer working?


Thanks for using Galaxy,

Dan

On Apr 29, 2015, at 10:38 AM, Peter Cock p.j.a.c...@googlemail.com wrote:

 On Wed, Apr 29, 2015 at 6:33 AM, Simon Gladman
 simon.glad...@unimelb.edu.au wrote:
 Hi dev,
 
 Now that the code tagset is deprecated, how do we replace that
 functionality. For example the get microbial data tool runs the script
 listed in the code tagset to load the list of available microbe data when
 the tool is selected. This doesn't seem to work in the latest stable 15.03.1
 or 15.03.2.
 
 How do we configure tools like this now? Or is there an updated version of
 the microbial_import tool?
 
 Cheers,
 
 Simon.
 
 Good question Simon.
 
 I am still using code for advanced parameter validation, e.g.
 https://github.com/peterjc/pico_galaxy/blob/64aecfa6041b3b7fe2c70a30236a7aa15e650523/tools/mira4_assembler/mira4_de_novo.xml
 
 i.e. Validation before the job gets submitted to the cluster or run.
 If the code tag suddenly goes away, my tool will still work but
 would be a little less user friendly (some errors would only be
 spotted after the tool starts running).
 
 Peter
 ___
 Please keep all replies on the list by using reply all
 in your mail client.  To manage your subscriptions to this
 and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/
 
 To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
 

___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Multiple outputs of unknown types

[Daniel Blankenberg]

Hi Davids,

For some a somewhat more complex example, have a look at the GenomeSpace 
importer tools under tools/genomespace in the distribution. They always create 
at least 1 dataset, but can create any number of files, of any type of 
datatype. In this tool, it will use the filetype declared by GenomeSpace, or it 
will fall back to using Galaxy’s sniffing functions of it doesn’t know the 
datatypes.

This makes use of the ‘galaxy.json’ file — 
json_params['job_config']['TOOL_PROVIDED_JOB_METADATA_FILE']


Thanks,

Dan


On Apr 8, 2015, at 9:24 AM, John Chilton jmchil...@gmail.com wrote:

 I had my own use case for this a few weeks ago and I couldn't figure
 out anyway to configure this right now - I was hoping Dan might
 respond to your question with some crazy trick though. For now I have
 created a Trello card for the implementation of this feature - please
 vote for it:
 
 https://trello.com/c/FZ67pAAE
 
 -John
 
 
 On Wed, Apr 1, 2015 at 7:29 PM, David Kelly davidke...@uchicago.edu wrote:
 Hello,
 
 I've looked through the documentation about tools that produce multiple
 output files, but it seems like you're required to know the data types ahead
 of time (or make assumptions based on input data). But what about a case
 where you don't know what the output datatypes will be? Is it just possible
 to have N output files imported and sniffed?
 
 Thanks,
 Davids
 
 ___
 Please keep all replies on the list by using reply all
 in your mail client.  To manage your subscriptions to this
 and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/
 
 To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
 ___
 Please keep all replies on the list by using reply all
 in your mail client.  To manage your subscriptions to this
 and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/
 
 To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
 

___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] a question on data manager and data table

[Daniel Blankenberg]

Hi Rui,

The Extract Genomic DNA Tool has not yet been updated to work with tool Data 
Tables. 

Additionally, this tool requires TwoBit formatted files when selecting from 
built-in data — when selecting a FASTA from your history, the fasta file is 
converted to TwoBit internally in the tool before extracting sequence chunks.

There is a TwoBit builder Data Manager under development 
(https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_twobit_builder),
 that hasn’t made it out to the ToolShed just yet, but it won’t be helpful 
until the  the Extract Genomic DNA Tool is updated: 
https://trello.com/c/8unnSp7H


Thanks for using Galaxy,

Dan


On Apr 7, 2015, at 12:16 AM, Beginner TI ruiwang...@gmail.com wrote:

 Hi All,
 
 I just made a new local instance, installed the fetch_all_fasta data manager, 
 downloaded mm9 fasta. Then I noticed the following:
 
 in tool_data_table_conf.xml, it has:
 
 table name=all_fasta comment_char=#
 columnsvalue, dbkey, name, path/columns
 file path=tool-data/all_fasta.loc /
 /table
 
 however in shed_tool_data_table_conf.xml, it has:
 
 table comment_char=# name=all_fasta
 columnsvalue, dbkey, name, path/columns
 file 
 path=/auto/rcf-proj/yc1/galaxy-suite/galaxy-dist/tool-data/toolshed.g2.bx.psu.edu/repos/devteam/data_manager_fetch_genome_all_fasta/cca219f2b212/all_fasta.loc
  /
 
 tool_shed_repositorytool_shedtoolshed.g2.bx.psu.edu/tool_shedrepository_namedata_manager_fetch_genome_all_fasta/repository_namerepository_ownerdevteam/repository_ownerinstalled_changeset_revisioncca219f2b212/installed_changeset_revision/tool_shed_repository/table
 
 in tool-data, all_fasta.loc is empty, but in the all_fasta.loc of the 
 shed_tool entry, it shows:
 
 mm9 mm9 Mouse July 2007 (NCBI37/mm9) (mm9)  
 /auto/rcf-proj/yc1/galaxy-suite/galaxy-dist/tool-data/mm9/seq/mm9.fa
 
 So if I try Extract Genomic DNA function, I could see the parameter passed 
 to the command line is 
 
 -g /auto/rcf-proj/yc1/galaxy-suite/galaxy-dist/tool-data 
 
 which does not host the data. It should be at least
 
 -g /auto/rcf-proj/yc1/galaxy-suite/galaxy-dist/tool-data/mm9/seq
 
 I thought that data manager will automatically populate these loc files? Am I 
 missing something obvious?  I could manually modify tool_data_table_conf.xml 
 to let it point to
 
 /auto/rcf-proj/yc1/galaxy-suite/galaxy-dist/tool-data/toolshed.g2.bx.psu.edu/repos/devteam/data_manager_fetch_genome_all_fasta/cca219f2b212/all_fasta.loc
 
 This would work, but it is ugly. Could someone please give me a hand to fix 
 this?
 
 Thanks,
 Rui
 ___
 Please keep all replies on the list by using reply all
 in your mail client.  To manage your subscriptions to this
 and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/
 
 To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] using DataManager to add custom genomes

[Daniel Blankenberg]

Hi Nikhil,

Have a look at 
https://toolshed.g2.bx.psu.edu/view/devteam/data_manager_fetch_genome_dbkeys_all_fasta
 to add a new genome/fasta. Once you have the genome entry, then you can use 
additional Data Manager to make tool specific (bwa, samtools, etc) indexes.


Thanks for using Galaxy,

Dan


On Apr 2, 2015, at 5:41 AM, Nikhil Joshi najo...@ucdavis.edu wrote:

 Of course this would be for a local install of galaxy.
 
 On Thu, Apr 2, 2015 at 2:29 AM, Nikhil Joshi najo...@ucdavis.edu wrote:
 Hi all,
 
 Is it possible to use a DataManager (or anything) to allow a user (or admin) 
 to upload their genome of choice and have it be indexed and be used as a 
 built-in/cached genome *without* having to go on the command-line? I.e., only 
 using the Galaxy interface?
 
 - Nik.
 
 -- 
 Nikhil Joshi
 Bioinformatics Analyst/Programmer
 UC Davis Bioinformatics Core
 http://bioinformatics.ucdavis.edu/
 najoshi -at- ucdavis -dot- edu
 530.752.2698 (w)
 
 
 
 -- 
 Nikhil Joshi
 Bioinformatics Analyst/Programmer
 UC Davis Bioinformatics Core
 http://bioinformatics.ucdavis.edu/
 najoshi -at- ucdavis -dot- edu
 530.752.2698 (w)
 ___
 Please keep all replies on the list by using reply all
 in your mail client.  To manage your subscriptions to this
 and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/
 
 To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/