Re: [galaxy-dev] Tools that make datasets

[Peter van Heusden]

Just a quick check - did you refresh your history to confirm that the
dataset *is* empty? We had the same thing at SANBI but it turns out that
Galaxy creates an empty output collection and then only populates it
sometime after job completion (this is a know UI bug).

See:
http://pvh.wp.sanbi.ac.za/2015/09/18/adventures-in-galaxy-output-collections/

On 20 October 2015 at 08:48, Steve Cassidy  wrote:

> Hi all,
>   I'm trying to understand how to write a tool that generates a dataset
> rather than a single output file.  I've tried following all of the examples
> but I'm stuck, so I thought I would distil down the simplest example I
> could write and ask for help here.
>
> So here's my example:
>
> https://gist.github.com/stevecassidy/0fa45ad5853faacb5f55
>
> it's a simple python script that writes three files to a directory named
> for the single input parameter.
>
> I think one of the problems I'm having is knowing where to write the
> output to. I've run this under planemo serve and the job runs, creating the
> output directory within the 'job_working_directory/000/1/SampleDataset'
> directory, however my dataset doesn't contain anything so clearly my
> outputs directive isn't working:
>
> 
> 
>  directory="$job_name" />
> 
> 
>
> ($job_name is the name of the directory that is being written to,
> SampleDataset in this case)
>
> Any help in getting this example working would be appreciated.
>
> Thanks,
>
> Steve
>
>
>
>
>
> --
> Department of Computing, Macquarie University
> http://web.science.mq.edu.au/~cassidy/
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Tools that make datasets

[Steve Cassidy]

Thanks Peter,
  I did see that proviso somewhere but no, refreshing doesn't help.

That page was one of those that I referred to getting to this point.

Steve

On 20 October 2015 at 18:33, Peter van Heusden  wrote:

> Just a quick check - did you refresh your history to confirm that the
> dataset *is* empty? We had the same thing at SANBI but it turns out that
> Galaxy creates an empty output collection and then only populates it
> sometime after job completion (this is a know UI bug).
>
> See:
> http://pvh.wp.sanbi.ac.za/2015/09/18/adventures-in-galaxy-output-collections/
>
> On 20 October 2015 at 08:48, Steve Cassidy 
> wrote:
>
>> Hi all,
>>   I'm trying to understand how to write a tool that generates a dataset
>> rather than a single output file.  I've tried following all of the examples
>> but I'm stuck, so I thought I would distil down the simplest example I
>> could write and ask for help here.
>>
>> So here's my example:
>>
>> https://gist.github.com/stevecassidy/0fa45ad5853faacb5f55
>>
>> it's a simple python script that writes three files to a directory named
>> for the single input parameter.
>>
>> I think one of the problems I'm having is knowing where to write the
>> output to. I've run this under planemo serve and the job runs, creating the
>> output directory within the 'job_working_directory/000/1/SampleDataset'
>> directory, however my dataset doesn't contain anything so clearly my
>> outputs directive isn't working:
>>
>> 
>> 
>> > directory="$job_name" />
>> 
>> 
>>
>> ($job_name is the name of the directory that is being written to,
>> SampleDataset in this case)
>>
>> Any help in getting this example working would be appreciated.
>>
>> Thanks,
>>
>> Steve
>>
>>
>>
>>
>>
>> --
>> Department of Computing, Macquarie University
>> http://web.science.mq.edu.au/~cassidy/
>>
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>   https://lists.galaxyproject.org/
>>
>> To search Galaxy mailing lists use the unified search at:
>>   http://galaxyproject.org/search/mailinglists/
>>
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
>



-- 
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Developing package wrappers

[Tiago Rodrigues Antao]

Hi,

I am afraid that I might be overlooking something obvious but...

I am trying to add a few tools to the RAD-Seq portfolio of Galaxy. For
that effect I am designing a few package wrappers. My problem is to
understand the best practice to develop wrappers for dependencies of
external tools.

For normal tools, that is quite easy: one uses the galaxy/tool/XXX
to do all development and testing.

But for dependency wrappers, the only solution that I have found is to
upload the wrapper to a toolshed and installing it via the admin
interface... This is cumbersome. Is there any way to test/develop a
package wrapper so that I can test tool_dependencies.xml without
uploading it to a toolshed and installing it from there?

Thanks,
Tiago
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Tools that make datasets

[Peter van Heusden]

I suspect that the problem might be in the  then. I'm
not an export on this, but "__name_and_ext__" turns into the
regexp r"(?P.*)\.(?P[^\.]+)?" in
lib/galaxy/tools/parameters/output_collect.py, and is used by the
DatasetCollector (line 358). This looks like it should match the filenames
you're creating, but I'm not 100% sure how that code works. One thing I
notice is the "directory" argument. If you write jobs to the current
directory instead of "output_path" can you get it to work?

Peter

On 20 October 2015 at 09:52, Steve Cassidy  wrote:

> Thanks Peter,
>   I did see that proviso somewhere but no, refreshing doesn't help.
>
> That page was one of those that I referred to getting to this point.
>
> Steve
>
> On 20 October 2015 at 18:33, Peter van Heusden  wrote:
>
>> Just a quick check - did you refresh your history to confirm that the
>> dataset *is* empty? We had the same thing at SANBI but it turns out that
>> Galaxy creates an empty output collection and then only populates it
>> sometime after job completion (this is a know UI bug).
>>
>> See:
>> http://pvh.wp.sanbi.ac.za/2015/09/18/adventures-in-galaxy-output-collections/
>>
>> On 20 October 2015 at 08:48, Steve Cassidy 
>> wrote:
>>
>>> Hi all,
>>>   I'm trying to understand how to write a tool that generates a dataset
>>> rather than a single output file.  I've tried following all of the examples
>>> but I'm stuck, so I thought I would distil down the simplest example I
>>> could write and ask for help here.
>>>
>>> So here's my example:
>>>
>>> https://gist.github.com/stevecassidy/0fa45ad5853faacb5f55
>>>
>>> it's a simple python script that writes three files to a directory named
>>> for the single input parameter.
>>>
>>> I think one of the problems I'm having is knowing where to write the
>>> output to. I've run this under planemo serve and the job runs, creating the
>>> output directory within the 'job_working_directory/000/1/SampleDataset'
>>> directory, however my dataset doesn't contain anything so clearly my
>>> outputs directive isn't working:
>>>
>>> 
>>> 
>>> >> directory="$job_name" />
>>> 
>>> 
>>>
>>> ($job_name is the name of the directory that is being written to,
>>> SampleDataset in this case)
>>>
>>> Any help in getting this example working would be appreciated.
>>>
>>> Thanks,
>>>
>>> Steve
>>>
>>>
>>>
>>>
>>>
>>> --
>>> Department of Computing, Macquarie University
>>> http://web.science.mq.edu.au/~cassidy/
>>>
>>> ___
>>> Please keep all replies on the list by using "reply all"
>>> in your mail client.  To manage your subscriptions to this
>>> and other Galaxy lists, please use the interface at:
>>>   https://lists.galaxyproject.org/
>>>
>>> To search Galaxy mailing lists use the unified search at:
>>>   http://galaxyproject.org/search/mailinglists/
>>>
>>
>>
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>   https://lists.galaxyproject.org/
>>
>> To search Galaxy mailing lists use the unified search at:
>>   http://galaxyproject.org/search/mailinglists/
>>
>
>
>
> --
> Department of Computing, Macquarie University
> http://web.science.mq.edu.au/~cassidy/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Tools that make datasets

[Steve Cassidy]

Yes, I'm sure that's where the problem lies. Writing out to the current
directory doesn't work.  The files get written to
'job_working_directory/000/1/' but if I run the Upload File tool the result
is placed in 'files/000/'.  I think I need to work out where to write the
files, I found some references to $__new_file_path__ but that doesn't seem
to help.

Steve



On 20 October 2015 at 19:57, Peter van Heusden  wrote:

> I suspect that the problem might be in the  then. I'm
> not an export on this, but "__name_and_ext__" turns into the
> regexp r"(?P.*)\.(?P[^\.]+)?" in
> lib/galaxy/tools/parameters/output_collect.py, and is used by the
> DatasetCollector (line 358). This looks like it should match the filenames
> you're creating, but I'm not 100% sure how that code works. One thing I
> notice is the "directory" argument. If you write jobs to the current
> directory instead of "output_path" can you get it to work?
>
> Peter
>
> On 20 October 2015 at 09:52, Steve Cassidy 
> wrote:
>
>> Thanks Peter,
>>   I did see that proviso somewhere but no, refreshing doesn't help.
>>
>> That page was one of those that I referred to getting to this point.
>>
>> Steve
>>
>> On 20 October 2015 at 18:33, Peter van Heusden  wrote:
>>
>>> Just a quick check - did you refresh your history to confirm that the
>>> dataset *is* empty? We had the same thing at SANBI but it turns out that
>>> Galaxy creates an empty output collection and then only populates it
>>> sometime after job completion (this is a know UI bug).
>>>
>>> See:
>>> http://pvh.wp.sanbi.ac.za/2015/09/18/adventures-in-galaxy-output-collections/
>>>
>>> On 20 October 2015 at 08:48, Steve Cassidy 
>>> wrote:
>>>
 Hi all,
   I'm trying to understand how to write a tool that generates a dataset
 rather than a single output file.  I've tried following all of the examples
 but I'm stuck, so I thought I would distil down the simplest example I
 could write and ask for help here.

 So here's my example:

 https://gist.github.com/stevecassidy/0fa45ad5853faacb5f55

 it's a simple python script that writes three files to a directory
 named for the single input parameter.

 I think one of the problems I'm having is knowing where to write the
 output to. I've run this under planemo serve and the job runs, creating the
 output directory within the 'job_working_directory/000/1/SampleDataset'
 directory, however my dataset doesn't contain anything so clearly my
 outputs directive isn't working:

 
 
 >>> directory="$job_name" />
 
 

 ($job_name is the name of the directory that is being written to,
 SampleDataset in this case)

 Any help in getting this example working would be appreciated.

 Thanks,

 Steve





 --
 Department of Computing, Macquarie University
 http://web.science.mq.edu.au/~cassidy/

 ___
 Please keep all replies on the list by using "reply all"
 in your mail client.  To manage your subscriptions to this
 and other Galaxy lists, please use the interface at:
   https://lists.galaxyproject.org/

 To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/

>>>
>>>
>>> ___
>>> Please keep all replies on the list by using "reply all"
>>> in your mail client.  To manage your subscriptions to this
>>> and other Galaxy lists, please use the interface at:
>>>   https://lists.galaxyproject.org/
>>>
>>> To search Galaxy mailing lists use the unified search at:
>>>   http://galaxyproject.org/search/mailinglists/
>>>
>>
>>
>>
>> --
>> Department of Computing, Macquarie University
>> http://web.science.mq.edu.au/~cassidy/
>>
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
>



-- 
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] FW: Galaxy on Centos via Apache - connection refused

[Makis Ladoukakis]

Hello,

I am sorry but I have really no experience with setting up the Apache web 
server so I am not really sure how to do that. Can you please help me out with 
it? My apache configuration file is in /etc/httpd/conf/ directory and there are 
no directories such as /sites-available/ or /sites-enabled/ (as I would find in 
an ubuntu installation).

What I did already (after some advice from the server admin) is open up the 
8081 port like that:



firewall-cmd
 --permanent  --add-port=8081/tcp

firewall-cmd
 --reload


and then I got another error:



 [cgi:error] [pid
29603] [client 115.230.124.164:4559] script not found or unable to stat:
/var/www/cgi-bin/common

[autoindex:error]
[pid 29716] [client 218.76.28.36:4468] AH01276: Cannot serve directory
/var/www/html/: No matching DirectoryIndex (index.html,index.php) found, and
server-generated directory index forbidden by Options directive


which I tried to solve by adding welcome.html as a recognizable filename in the 
apache configuration:





DirectoryIndex index.html welcome.html




but nothing worked and now the error_log shows the following:

[Tue Oct 20 13:15:23.719295 2015] [mpm_prefork:notice] [pid 29598] AH00170: 
caught SIGWINCH, shutting down gracefully
[Tue Oct 20 13:15:24.810684 2015] [core:notice] [pid 46896] SELinux policy 
enabled; httpd running as context system_u:system_r:httpd_t:s0
[Tue Oct 20 13:15:24.811647 2015] [suexec:notice] [pid 46896] AH01232: suEXEC 
mechanism enabled (wrapper: /usr/sbin/suexec)
[Tue Oct 20 13:15:24.846399 2015] [so:warn] [pid 46896] AH01574: module 
wsgi_module is already loaded, skipping
[Tue Oct 20 13:15:24.847316 2015] [auth_digest:notice] [pid 46896] AH01757: 
generating secret for digest authentication ...
[Tue Oct 20 13:15:24.848294 2015] [lbmethod_heartbeat:notice] [pid 46896] 
AH02282: No slotmem from mod_heartmonitor
[Tue Oct 20 13:15:24.870033 2015] [mpm_prefork:notice] [pid 46896] AH00163: 
Apache/2.4.6 (CentOS) PHP/5.4.16 mod_wsgi/3.4 Python/2.7.5 configured -- 
resuming normal operations
[Tue Oct 20 13:15:24.870075 2015] [core:notice] [pid 46896] AH00094: Command 
line: '/usr/sbin/httpd -D FOREGROUND'

And the webpage that galaxy is supposed to appear is still blank.

Any ideas?

Thank you,
Makis

 

Date: Tue, 20 Oct 2015 11:01:44 +1030
Subject: Re: [galaxy-dev] FW: Galaxy on Centos via Apache - connection refused
From: jcsanch...@gmail.com
To: makis4e...@hotmail.com
CC: galaxy-...@lists.bx.psu.edu

hi,
Maybe sounds silly, but have you tried to put the apache configuration in a 
virtual host within the sites-enable site?

cheersjc
On Tue, Oct 20, 2015 at 12:36 AM, Makis Ladoukakis  
wrote:



Forwading to this list too. I am not sure if they are two separate lists.

Makis

From: makis4e...@hotmail.com
To: galaxy-dev@lists.galaxyproject.org
Date: Mon, 19 Oct 2015 17:04:13 +0300
Subject: [galaxy-dev] Galaxy on Centos via Apache -  connection refused




Dear all,

I've been trying to set up a Galaxy instance on my CentOS server but even when 
I manage to run the simplest configuration correctly I can't seem to get it 
right on my browser as the connection is refused.

I have changed the host to 0.0.0.0, the port to 8081 and made the appropriate 
changes in Apache configuration file:

RewriteEngine on
RewriteRule ^/galaxy_test(.*) http://localhost:8081$1 [P]
RewriteRule ^/galaxy_test$ /galaxy_test/ [R]
RewriteRule ^/galaxy_test/static/style/(.*) 
/home/galaxy_test/galaxy/static/june_2007_style/blue/$1 [L]
RewriteRule ^/galaxy_test/static/scripts/(.*) 
/home/galaxy_test/galaxy/static/scripts/packed/$1 [L]
RewriteRule ^/galaxy_test/static/(.*) /home/galaxy_test/galaxy/static/$1 [L]
RewriteRule ^/galaxy_test/favicon.ico 
/home/galaxy_test/galaxy/static/favicon.ico [L]
RewriteRule ^/galaxy_test/robots.txt /home/galaxy_test/galaxy/static/robots.txt 
[L]

So when I tried to run it the first time I got the "temporarily out of service" 
error on my browser and the following error in apache log:

[Mon Oct 19 05:56:45.308714 2015] [proxy:error] [pid 18505] (13)Permission 
denied: AH00957: HTTP: attempt to connect to 127.0.0.1:8081 (*) failed
[Mon Oct 19 05:56:45.308774 2015] [proxy_http:error] [pid 18505] [client 
185.25.151.159:39629] AH01114: HTTP: failed to make connection to backend: 
localhost

Searching the internet I found the following solution:



/usr/sbin/setsebool
httpd_can_network_connect 1


/usr/sbin/setsebool -P
httpd_can_network_connect 1




but when I restarted my galaxy my browser would now show just a blank page and 
the error log of apache was:

[Mon Oct 19 15:29:09.425120 2015] [proxy:error] [pid 29699] (111)Connection 
refused: AH00957: HTTP: attempt to connect to 127.0.0.1:8081 (*) failed
[Mon Oct 19 15:29:09.425167 2015] [proxy_http:error] [pid 29699] [client 
147.102.86.121:51865] AH01114: HTTP: failed to make connection to backend: 
localhost


Has anyone encountered that?

Kind regards,
Makis






P.S. In case that helps the output of iptables -L is 

Re: [galaxy-dev] recurring issues with revisions of packages from toolshed

[Sarah DIEHL]

Hi,

I'm afraid it won't be easy to reproduce my problem with a clean galaxy 
install. I think it has to do with the history of the instance, a lot of back 
and forth on the tool installations and also lot's of failed installations (due 
to my environment or network issues).

Investigating the issue Marius mentioned also seems worthwhile to me, it sounds 
like it could be the underlying bug. What might also help is a way to really 
purge all traces of a tool or tool dependency (and all it's versions) from the 
disk and the database. Currently the only way to solve my numpy revision issue 
would be to delete the whole tool install database (which I learned to put in a 
different db and I already resorted to that once in the beginning). I had a 
look at the database and it looks messed up, but I don't know enough about it 
to feel comfortable changing entries. I'm happy to send a DB dump along, if 
this is helpful for anyone.

Best regards,
Sarah



Sarah Diehl
HPC System Administrator

UNIVERSITÉ DU LUXEMBOURG

LUXEMBOURG CENTRE FOR SYSTEMS BIOMEDICINE
Campus Belval | Biotech II
6, avenue du Swing
L-4371 Belvaux
T +352 46 66 44 5360
sarah.di...@uni.lu 
http://lcsb.uni.lu
-
This message is confidential and may contain privileged information. It is 
intended for the named recipient only. If you receive it in error please notify 
me and permanently delete the original message and any copies.
-


From: Marius van den Beek 
>
Date: Tuesday 20 October 2015 09:53
To: Björn Grüning >
Cc: Sarah DIEHL >, 
"galaxy-dev@lists.galaxyproject.org" 
>
Subject: Re: [galaxy-dev] recurring issues with revisions of packages from 
toolshed

Sorry to hijack the thread, but I think the issue that Sarah has described 
could be related to
https://github.com/galaxyproject/galaxy/issues/667.
(I know it's not exactly the same, but chances are we fix this, we 
automatically fix Sarah's issue.)
There is also an example to reproduce 
https://github.com/galaxyproject/galaxy/issues/667#issuecomment-138878174
And I made a bit of work on this in Pull Request: 
https://github.com/galaxyproject/galaxy/pull/707
But unfortunately at the time this was a bit over my head.
Bjoern, if you want to work on this, let me know, maybe I can give you a hand.

Hope that's helpful,
Marius

On 19 October 2015 at 23:17, Björn Grüning 
> wrote:
Hi Sarah,

can you create a reproducible example in a Docker container for me?
Together with a friend I will have a look at this. But we need to short
reproducible example :)

Thanks,
Bjoern


> Dear all,
>
> I continuously have several issues with tool dependency packages from
> the toolshed and their revisions. It looks like the tools I install
> have requirements for different revisions of the same package_xy
> repository. What I end up with quite often is having the same package
> installed in multiple revisions (see attached
> "galaxytools_multiple_revisions.jpg" screenshot). When I then update
> the older revision, it's still listet twice and additionally the tool
> that had the older one as a dependency now says it's missing a
> repository dependency.
>
> Although this is already quite annoying and creates a mess within the
> tool list, I could live with it. However, sometimes it completely
> messes up revisions. In the second attached screenshot
> (numpy_revision_issue.jpg) you can see that this numpy repository
> thinks it is revision "300877695495", but it's actual location has
> revision "8b3a5c7061d8". This causes tools that depend on revision
> "300877695495" to think it's there, but when the dependency is
> resolved at runtime of the tool, numpy is not found (since it's
> location is in a different revision's folder) and the run fails. As
> you can also see from the screenshot, resetting the metadata doesn't
> do anything about this. Also reinstalling doesn't work, since the
> database keeps the faulty record and uses it upon reinstallation.
>
> I think it's a problem that when I install tools that have outdated
> revisions of a package repository as a dependency, they don't
> recognize that a newer version is available (and already installed),
> but instead insist on installing the older revision, too. I don't
> understand why they even have this strict requirement on a specific
> version of a package repo, and not just the version of the tool
> inside.
>
> Btw. I also noticed that the installation of R packages (e.g. in
> package_dexseq_1_14) silently fails. Some of the requirements and
> also the dexseq R package itself failed the installation, but Galaxy
> didn't recognize any error and it still showed up 

Re: [galaxy-dev] FW: Galaxy on Centos via Apache - connection refused

[Juan Carlos]

Hi,
If you have a line in your Apache conf like 
"Listen 80"
change to 
"Listen 8081"



> On 20 Oct 2015, at 21:00, Makis Ladoukakis  wrote:
> 
> Hello,
> 
> I am sorry but I have really no experience with setting up the Apache web 
> server so I am not really sure how to do that. Can you please help me out 
> with it? My apache configuration file is in /etc/httpd/conf/ directory and 
> there are no directories such as /sites-available/ or /sites-enabled/ (as I 
> would find in an ubuntu installation).
> 
> What I did already (after some advice from the server admin) is open up the 
> 8081 port like that:
> 
> firewall-cmd  --permanent  --add-port=8081/tcp
> 
> firewall-cmd  --reload
> 
> 
> and then I got another error:
> 
>  [cgi:error] [pid 29603] [client 115.230.124.164:4559] script not found or 
> unable to stat: /var/www/cgi-bin/common
> [autoindex:error] [pid 29716] [client 218.76.28.36:4468] AH01276: Cannot 
> serve directory /var/www/html/: No matching DirectoryIndex 
> (index.html,index.php) found, and server-generated directory index forbidden 
> by Options directive
> 
> which I tried to solve by adding welcome.html as a recognizable filename in 
> the apache configuration:
> 
> 
> DirectoryIndex index.html welcome.html
> 
> 
> 
> but nothing worked and now the error_log shows the following:
> 
> [Tue Oct 20 13:15:23.719295 2015] [mpm_prefork:notice] [pid 29598] AH00170: 
> caught SIGWINCH, shutting down gracefully
> [Tue Oct 20 13:15:24.810684 2015] [core:notice] [pid 46896] SELinux policy 
> enabled; httpd running as context system_u:system_r:httpd_t:s0
> [Tue Oct 20 13:15:24.811647 2015] [suexec:notice] [pid 46896] AH01232: suEXEC 
> mechanism enabled (wrapper: /usr/sbin/suexec)
> [Tue Oct 20 13:15:24.846399 2015] [so:warn] [pid 46896] AH01574: module 
> wsgi_module is already loaded, skipping
> [Tue Oct 20 13:15:24.847316 2015] [auth_digest:notice] [pid 46896] AH01757: 
> generating secret for digest authentication ...
> [Tue Oct 20 13:15:24.848294 2015] [lbmethod_heartbeat:notice] [pid 46896] 
> AH02282: No slotmem from mod_heartmonitor
> [Tue Oct 20 13:15:24.870033 2015] [mpm_prefork:notice] [pid 46896] AH00163: 
> Apache/2.4.6 (CentOS) PHP/5.4.16 mod_wsgi/3.4 Python/2.7.5 configured -- 
> resuming normal operations
> [Tue Oct 20 13:15:24.870075 2015] [core:notice] [pid 46896] AH00094: Command 
> line: '/usr/sbin/httpd -D FOREGROUND'
> 
> And the webpage that galaxy is supposed to appear is still blank.
> 
> Any ideas?
> 
> Thank you,
> Makis
> 
>  
> 
> Date: Tue, 20 Oct 2015 11:01:44 +1030
> Subject: Re: [galaxy-dev] FW: Galaxy on Centos via Apache - connection refused
> From: jcsanch...@gmail.com
> To: makis4e...@hotmail.com
> CC: galaxy-...@lists.bx.psu.edu
> 
> hi,
> 
> Maybe sounds silly, but have you tried to put the apache configuration in a 
> virtual host within the sites-enable site?
> 
> 
> cheers
> jc
> 
> On Tue, Oct 20, 2015 at 12:36 AM, Makis Ladoukakis  
> wrote:
> Forwading to this list too. I am not sure if they are two separate lists.
> 
> Makis
> 
> From: makis4e...@hotmail.com
> To: galaxy-dev@lists.galaxyproject.org
> Date: Mon, 19 Oct 2015 17:04:13 +0300
> Subject: [galaxy-dev] Galaxy on Centos via Apache - connection refused
> 
> 
> Dear all,
> 
> I've been trying to set up a Galaxy instance on my CentOS server but even 
> when I manage to run the simplest configuration correctly I can't seem to get 
> it right on my browser as the connection is refused.
> 
> I have changed the host to 0.0.0.0, the port to 8081 and made the appropriate 
> changes in Apache configuration file:
> 
> RewriteEngine on
> RewriteRule ^/galaxy_test(.*) http://localhost:8081$1 [P]
> RewriteRule ^/galaxy_test$ /galaxy_test/ [R]
> RewriteRule ^/galaxy_test/static/style/(.*) 
> /home/galaxy_test/galaxy/static/june_2007_style/blue/$1 [L]
> RewriteRule ^/galaxy_test/static/scripts/(.*) 
> /home/galaxy_test/galaxy/static/scripts/packed/$1 [L]
> RewriteRule ^/galaxy_test/static/(.*) /home/galaxy_test/galaxy/static/$1 [L]
> RewriteRule ^/galaxy_test/favicon.ico 
> /home/galaxy_test/galaxy/static/favicon.ico [L]
> RewriteRule ^/galaxy_test/robots.txt 
> /home/galaxy_test/galaxy/static/robots.txt [L]
> 
> So when I tried to run it the first time I got the "temporarily out of 
> service" error on my browser and the following error in apache log:
> 
> [Mon Oct 19 05:56:45.308714 2015] [proxy:error] [pid 18505] (13)Permission 
> denied: AH00957: HTTP: attempt to connect to 127.0.0.1:8081 (*) failed
> [Mon Oct 19 05:56:45.308774 2015] [proxy_http:error] [pid 18505] [client 
> 185.25.151.159:39629] AH01114: HTTP: failed to make connection to backend: 
> localhost
> 
> Searching the internet I found the following solution:
> 
> /usr/sbin/setsebool httpd_can_network_connect 1
> /usr/sbin/setsebool -P httpd_can_network_connect 1
> 
> 
> but when I restarted my galaxy my browser would now show just a blank page 
> and the error log of apache was:
> 
> [Mon 

Re: [galaxy-dev] Developing package wrappers

[Bjoern Gruening]

Hi Tiago,

for simple tool_dependencies you can use the new planemo depbash command 
from Peter:

https://github.com/galaxyproject/planemo/pull/310

For more complicated once I recommend to use the TTS for the time being.
You can create the initial repository install from it and change your 
tool_dependency file on your harddrive (after installation)

and simply reinstall the tool after every change.

What kind of dependencies do you have? Python/Perl?
Consider contributing to tools-iuc 
(https://github.com/galaxyproject/tools-iuc). We can probably help you 
and guide you your way.


Thanks for working on RAD-seq!
Bjoern


On 20.10.2015 11:01, Tiago Rodrigues Antao wrote:

Hi,

I am afraid that I might be overlooking something obvious but...

I am trying to add a few tools to the RAD-Seq portfolio of Galaxy. For
that effect I am designing a few package wrappers. My problem is to
understand the best practice to develop wrappers for dependencies of
external tools.

For normal tools, that is quite easy: one uses the galaxy/tool/XXX
to do all development and testing.

But for dependency wrappers, the only solution that I have found is to
upload the wrapper to a toolshed and installing it via the admin
interface... This is cumbersome. Is there any way to test/develop a
package wrapper so that I can test tool_dependencies.xml without
uploading it to a toolshed and installing it from there?

Thanks,
Tiago
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Tools that make datasets

[Steve Cassidy]

Hi all,
  I'm trying to understand how to write a tool that generates a dataset
rather than a single output file.  I've tried following all of the examples
but I'm stuck, so I thought I would distil down the simplest example I
could write and ask for help here.

So here's my example:

https://gist.github.com/stevecassidy/0fa45ad5853faacb5f55

it's a simple python script that writes three files to a directory named
for the single input parameter.

I think one of the problems I'm having is knowing where to write the output
to. I've run this under planemo serve and the job runs, creating the output
directory within the 'job_working_directory/000/1/SampleDataset' directory,
however my dataset doesn't contain anything so clearly my outputs directive
isn't working:







($job_name is the name of the directory that is being written to,
SampleDataset in this case)

Any help in getting this example working would be appreciated.

Thanks,

Steve





-- 
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] recurring issues with revisions of packages from toolshed

[Marius van den Beek]

Sorry to hijack the thread, but I think the issue that Sarah has described
could be related to
https://github.com/galaxyproject/galaxy/issues/667.
(I know it's not exactly the same, but chances are we fix this, we
automatically fix Sarah's issue.)
There is also an example to reproduce
https://github.com/galaxyproject/galaxy/issues/667#issuecomment-138878174
And I made a bit of work on this in Pull Request:
https://github.com/galaxyproject/galaxy/pull/707
But unfortunately at the time this was a bit over my head.
Bjoern, if you want to work on this, let me know, maybe I can give you a
hand.

Hope that's helpful,
Marius

On 19 October 2015 at 23:17, Björn Grüning 
wrote:

> Hi Sarah,
>
> can you create a reproducible example in a Docker container for me?
> Together with a friend I will have a look at this. But we need to short
> reproducible example :)
>
> Thanks,
> Bjoern
>
>
> > Dear all,
> >
> > I continuously have several issues with tool dependency packages from
> > the toolshed and their revisions. It looks like the tools I install
> > have requirements for different revisions of the same package_xy
> > repository. What I end up with quite often is having the same package
> > installed in multiple revisions (see attached
> > "galaxytools_multiple_revisions.jpg" screenshot). When I then update
> > the older revision, it's still listet twice and additionally the tool
> > that had the older one as a dependency now says it's missing a
> > repository dependency.
> >
> > Although this is already quite annoying and creates a mess within the
> > tool list, I could live with it. However, sometimes it completely
> > messes up revisions. In the second attached screenshot
> > (numpy_revision_issue.jpg) you can see that this numpy repository
> > thinks it is revision "300877695495", but it's actual location has
> > revision "8b3a5c7061d8". This causes tools that depend on revision
> > "300877695495" to think it's there, but when the dependency is
> > resolved at runtime of the tool, numpy is not found (since it's
> > location is in a different revision's folder) and the run fails. As
> > you can also see from the screenshot, resetting the metadata doesn't
> > do anything about this. Also reinstalling doesn't work, since the
> > database keeps the faulty record and uses it upon reinstallation.
> >
> > I think it's a problem that when I install tools that have outdated
> > revisions of a package repository as a dependency, they don't
> > recognize that a newer version is available (and already installed),
> > but instead insist on installing the older revision, too. I don't
> > understand why they even have this strict requirement on a specific
> > version of a package repo, and not just the version of the tool
> > inside.
> >
> > Btw. I also noticed that the installation of R packages (e.g. in
> > package_dexseq_1_14) silently fails. Some of the requirements and
> > also the dexseq R package itself failed the installation, but Galaxy
> > didn't recognize any error and it still showed up as "Installed".
> >
> > Best regards, Sarah
> >
> >
> >  Sarah Diehl HPC System Administrator
> >
> > UNIVERSITÉ DU LUXEMBOURG
> >
> > LUXEMBOURG CENTRE FOR SYSTEMS BIOMEDICINE Campus Belval | Biotech II
> > 6, avenue du Swing L-4371 Belvaux T +352 46 66 44 5360
> > sarah.di...@uni.lu
> > http://lcsb.uni.lu - This message is
> > confidential and may contain privileged information. It is intended
> > for the named recipient only. If you receive it in error please
> > notify me and permanently delete the original message and any
> > copies. -
> >
> >
> >
> >
> > ___ Please
> > keep all replies on the list by using "reply all" in your mail
> > client.  To manage your subscriptions to this and other Galaxy lists,
> > please use the interface at: https://lists.galaxyproject.org/
> >
> > To search Galaxy mailing lists use the unified search at:
> > http://galaxyproject.org/search/mailinglists/
> >
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Developing package wrappers

[Tiago Rodrigues Antao]

On Tue, 20 Oct 2015 14:48:08 +0200
Bjoern Gruening  wrote:

> Hi Tiago,
> 
> for simple tool_dependencies you can use the new planemo depbash
> command from Peter:
> https://github.com/galaxyproject/planemo/pull/310

Gosh, this is exactly what the doctor ordered. Thanks (to Peter also).

 
> What kind of dependencies do you have? Python/Perl?
> Consider contributing to tools-iuc 
> (https://github.com/galaxyproject/tools-iuc). We can probably help
> you and guide you your way.
> 
> Thanks for working on RAD-seq!


I am actually updating the raxml tool for now.
But in my new position I am heavily involved with RAD-seq data and I
am working on some new galaxy tools to help with more recent analytical
methods (e.g. integrating cap3 with stacks). If there is any reported
need on the RAD-seq front, email me and I might be able to help...

Tiago
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Tools that make datasets

[Steve Cassidy]

Ah, thankyou, yes, I can now get results by using patterns to match the
output.  I used your example but prepend 'simple' to the filename and then
searches for that with:


this solves the problem for the sample script but not generally since in
general I can't predict the filenames that will be generated - this is a
tool for downloading data from a repository which could be text, audio or
video data.

If I don't use the 'simple' prefix and omit the file extension I still get
my data but I also get three other files which are temporary scripts
generated by galaxy and placed in the working directory.  So, back to
trying to put things in a subdirectory. It turns out that the issue I was
having was as you pointed out earlier, the directory attribute to
discover_datasets doesn't allow variables, so I need to write to a fixed
directory name:


This now works!

I had thought that I'd need to use a unique directory name but since galaxy
runs each job in a separate directory, this isn't required.  My real tool
now works too after following the same pattern.

Thanks for your help.  I'll see if I can write this up in a blog post.

Steve



On 21 October 2015 at 00:06, Peter van Heusden  wrote:

> I poked around at your tool XML and the code a bit and the problem is
> directory="$job_name". Galaxy expects to collect files from the job's
> working directory - basically the current working directory the job runs
> in. The directory= argument doesn't have variables expanded as far as I can
> tell. In any event it is used in walk_over_extra_files() that is in
> lib/galaxy/tools/parameters/output_collect.py - if you look there you see
> that it is simply appended to the job's working directory.
>
> So if you use:
>
> 
>
> (note the  and  - this is effectively the regexp
> r"(?P.*)\.txt" with the < and > escaped out)
>
> And alter the code so that it just writes files to the current directory,
> then you'll pick up the files one.txt, two,txt and three.txt.
>
> Peter
>
> On 20 October 2015 at 12:28, Steve Cassidy 
> wrote:
>
>> Sorry, it was just an example of a tool that works - the file that it
>> writes out is put into that directory, so I assume that's where my files
>> should end up too.
>>
>> Steve
>>
>> On 20 October 2015 at 21:12, Peter van Heusden  wrote:
>>
>>> Sorry, I don't understand - what does the Upload File tool have to do
>>> with this?
>>>
>>> On 20 October 2015 at 11:49, Steve Cassidy 
>>> wrote:
>>>
 Yes, I'm sure that's where the problem lies. Writing out to the current
 directory doesn't work.  The files get written to
 'job_working_directory/000/1/' but if I run the Upload File tool the result
 is placed in 'files/000/'.  I think I need to work out where to write the
 files, I found some references to $__new_file_path__ but that doesn't seem
 to help.

 Steve



 On 20 October 2015 at 19:57, Peter van Heusden  wrote:

> I suspect that the problem might be in the  then.
> I'm not an export on this, but "__name_and_ext__" turns into the
> regexp r"(?P.*)\.(?P[^\.]+)?" in
> lib/galaxy/tools/parameters/output_collect.py, and is used by the
> DatasetCollector (line 358). This looks like it should match the filenames
> you're creating, but I'm not 100% sure how that code works. One thing I
> notice is the "directory" argument. If you write jobs to the current
> directory instead of "output_path" can you get it to work?
>
> Peter
>
> On 20 October 2015 at 09:52, Steve Cassidy 
> wrote:
>
>> Thanks Peter,
>>   I did see that proviso somewhere but no, refreshing doesn't help.
>>
>> That page was one of those that I referred to getting to this point.
>>
>> Steve
>>
>> On 20 October 2015 at 18:33, Peter van Heusden 
>> wrote:
>>
>>> Just a quick check - did you refresh your history to confirm that
>>> the dataset *is* empty? We had the same thing at SANBI but it turns out
>>> that Galaxy creates an empty output collection and then only populates 
>>> it
>>> sometime after job completion (this is a know UI bug).
>>>
>>> See:
>>> http://pvh.wp.sanbi.ac.za/2015/09/18/adventures-in-galaxy-output-collections/
>>>
>>> On 20 October 2015 at 08:48, Steve Cassidy 
>>> wrote:
>>>
 Hi all,
   I'm trying to understand how to write a tool that generates a
 dataset rather than a single output file.  I've tried following all of 
 the
 examples but I'm stuck, so I thought I would distil down the simplest
 example I could write and ask for help here.

 So here's my example:

 https://gist.github.com/stevecassidy/0fa45ad5853faacb5f55

 it's a simple python 

Re: [galaxy-dev] Developing package wrappers

[Peter Cock]

On Tue, Oct 20, 2015 at 10:01 AM, Tiago Rodrigues Antao  wrote:
> Hi,
>
> I am afraid that I might be overlooking something obvious but...
>
> I am trying to add a few tools to the RAD-Seq portfolio of Galaxy. For
> that effect I am designing a few package wrappers. My problem is to
> understand the best practice to develop wrappers for dependencies of
> external tools.
>
> For normal tools, that is quite easy: one uses the galaxy/tool/XXX
> to do all development and testing.
>
> But for dependency wrappers, the only solution that I have found is to
> upload the wrapper to a toolshed and installing it via the admin
> interface... This is cumbersome. Is there any way to test/develop a
> package wrapper so that I can test tool_dependencies.xml without
> uploading it to a toolshed and installing it from there?
>
> Thanks,
> Tiago

Hi Tiago,

If you've not seen planemo, I suggest you have a look at
https://github.com/galaxyproject/planemo and
https://github.com/galaxyproject/planemo or the slides John
gave at GCC 2015:

http://gcc2015.tsl.ac.uk/abstracts/#Planemo_8211_A_Galaxy_Tool_SDK

For tool testing during development this ought to work:

planemo test path/to/your/tool

For tool dependency testing, I used to rely on the nightly tests
run on the main and Test Tool Shed, currently unavailable. One
option here is to run your own local tool shed just for development,
but the Test Tool Shed is a reasonable substitute. See also:

planemo shed_test --help

I recently started work on the new "planemo dependency_script"
which attempts to parse tool_dependencies.xml into a bash
script - that is still a work in progress but might help here too:

https://github.com/galaxyproject/planemo/issues/303
https://github.com/galaxyproject/planemo/commit/f798c7e29b2276ce68b828e72fc6a6460c73792b

See also https://github.com/galaxyproject/planemo/issues/19
for the wider issue of testing tool_dependencies.xml with
planemo without a tool shed.

Regards,

Peter
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Tools that make datasets

[Peter van Heusden]

I poked around at your tool XML and the code a bit and the problem is
directory="$job_name". Galaxy expects to collect files from the job's
working directory - basically the current working directory the job runs
in. The directory= argument doesn't have variables expanded as far as I can
tell. In any event it is used in walk_over_extra_files() that is in
lib/galaxy/tools/parameters/output_collect.py - if you look there you see
that it is simply appended to the job's working directory.

So if you use:



(note the  and  - this is effectively the regexp
r"(?P.*)\.txt" with the < and > escaped out)

And alter the code so that it just writes files to the current directory,
then you'll pick up the files one.txt, two,txt and three.txt.

Peter

On 20 October 2015 at 12:28, Steve Cassidy  wrote:

> Sorry, it was just an example of a tool that works - the file that it
> writes out is put into that directory, so I assume that's where my files
> should end up too.
>
> Steve
>
> On 20 October 2015 at 21:12, Peter van Heusden  wrote:
>
>> Sorry, I don't understand - what does the Upload File tool have to do
>> with this?
>>
>> On 20 October 2015 at 11:49, Steve Cassidy 
>> wrote:
>>
>>> Yes, I'm sure that's where the problem lies. Writing out to the current
>>> directory doesn't work.  The files get written to
>>> 'job_working_directory/000/1/' but if I run the Upload File tool the result
>>> is placed in 'files/000/'.  I think I need to work out where to write the
>>> files, I found some references to $__new_file_path__ but that doesn't seem
>>> to help.
>>>
>>> Steve
>>>
>>>
>>>
>>> On 20 October 2015 at 19:57, Peter van Heusden  wrote:
>>>
 I suspect that the problem might be in the  then.
 I'm not an export on this, but "__name_and_ext__" turns into the
 regexp r"(?P.*)\.(?P[^\.]+)?" in
 lib/galaxy/tools/parameters/output_collect.py, and is used by the
 DatasetCollector (line 358). This looks like it should match the filenames
 you're creating, but I'm not 100% sure how that code works. One thing I
 notice is the "directory" argument. If you write jobs to the current
 directory instead of "output_path" can you get it to work?

 Peter

 On 20 October 2015 at 09:52, Steve Cassidy 
 wrote:

> Thanks Peter,
>   I did see that proviso somewhere but no, refreshing doesn't help.
>
> That page was one of those that I referred to getting to this point.
>
> Steve
>
> On 20 October 2015 at 18:33, Peter van Heusden 
> wrote:
>
>> Just a quick check - did you refresh your history to confirm that the
>> dataset *is* empty? We had the same thing at SANBI but it turns out that
>> Galaxy creates an empty output collection and then only populates it
>> sometime after job completion (this is a know UI bug).
>>
>> See:
>> http://pvh.wp.sanbi.ac.za/2015/09/18/adventures-in-galaxy-output-collections/
>>
>> On 20 October 2015 at 08:48, Steve Cassidy 
>> wrote:
>>
>>> Hi all,
>>>   I'm trying to understand how to write a tool that generates a
>>> dataset rather than a single output file.  I've tried following all of 
>>> the
>>> examples but I'm stuck, so I thought I would distil down the simplest
>>> example I could write and ask for help here.
>>>
>>> So here's my example:
>>>
>>> https://gist.github.com/stevecassidy/0fa45ad5853faacb5f55
>>>
>>> it's a simple python script that writes three files to a directory
>>> named for the single input parameter.
>>>
>>> I think one of the problems I'm having is knowing where to write the
>>> output to. I've run this under planemo serve and the job runs, creating 
>>> the
>>> output directory within the 'job_working_directory/000/1/SampleDataset'
>>> directory, however my dataset doesn't contain anything so clearly my
>>> outputs directive isn't working:
>>>
>>> 
>>> 
>>> >> directory="$job_name" />
>>> 
>>> 
>>>
>>> ($job_name is the name of the directory that is being written to,
>>> SampleDataset in this case)
>>>
>>> Any help in getting this example working would be appreciated.
>>>
>>> Thanks,
>>>
>>> Steve
>>>
>>>
>>>
>>>
>>>
>>> --
>>> Department of Computing, Macquarie University
>>> http://web.science.mq.edu.au/~cassidy/
>>>
>>> ___
>>> Please keep all replies on the list by using "reply all"
>>> in your mail client.  To manage your subscriptions to this
>>> and other Galaxy lists, please use the interface at:
>>>   https://lists.galaxyproject.org/
>>>
>>> To search Galaxy mailing lists use