Re: [galaxy-dev] Dynamic Tool UI based on script

[Steve Cassidy]

 I don’t like ‘hacks’!! I solved this by pulling the relevant bits of the 
module into my script so that it runs with no dependencies. It’s basically just 
making an HTTP GET request with an authentication header, so not too hard to 
achieve.  Should be safe to put in the toolshed.

Thanks, this is a big improvement in our UI (and I vote for not deprecating 
this feature, maybe even making it more robust).

Steve

—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

On 7 Dec 2017, at 8:50 pm, Hans-Rudolf Hotz <h...@fmi.ch<mailto:h...@fmi.ch>> 
wrote:


Sure

However, tools relying on "dynamic_options" and extra "code" scripts are 
usually 'hacks' for accessing external data/services in a local galaxy 
installation. And such tools are not intended for the toolshed anywayIMHO.


Hans-Rudolf


On 12/07/2017 09:37 AM, Steve Cassidy wrote:
I could do that but it would mean that no-one could install the tool
from the toolshed without also installing this package.

Maybe I just need to hack this together.

Steve


—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

On 7 Dec 2017, at 6:59 pm, Hans-Rudolf Hotz <h...@fmi.ch
<mailto:h...@fmi.ch>> wrote:

Hi Steve

what happened, when you add this module to the environment the galaxy
server is running in? (e.g. by listing the path in the 'run.sh' file)

Regards, Hans-Rudolf


On 12/07/2017 03:05 AM, Steve Cassidy wrote:
Hi all,
 this would help in a tool I’m trying to write too but the problem I
have implementing this approach is that my script will need an external
python module - is there any way that I can list dependencies for the
script?  As it happens these are the same dependancies that the tool
itself will need once executed.

Thanks,

Steve

—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

On 24 Nov 2017, at 6:52 pm, Hans-Rudolf Hotz <h...@fmi.ch
<mailto:h...@fmi.ch>> wrote:

Hi Evan

Yes, this is possible with "dynamic_options"

e.g.:




   2017
   2016
   2015







...


...





The python function "ds_readFiles" is defined in a extra code file
"more_code.py", which is stored in the same directory as the tool
definition file. The function returns all files with a matching file
name (i.e.year), and could look like:


rDir = "/data/"

def ds_readFiles(year):

 l = os.listdir(rDir)
 p = re.compile('^%s_[0­9]{4}_M[0­9]{5}_[0­9]'%year)
 l.sort()
 path = rDir
 filesoptions = [(s) for s in l if os.path.exists(path + s)]
 files = [(s,s,False) for s in filesoptions if p.match(s)]
 return files



a similar case is also described here:

http://dev.list.galaxyproject.org/Dynamic-Tool-Parameter-Lists-td4175828.html


Hope this helps, Hans-Rudolf


On 11/23/2017 06:02 PM, Evan Clark wrote:
Is it possible to execute a script within a tool xml confit to retrieve
some data and append to the ui depending on the requested data. I.e
if I
wanted to get a list of files and provide them as select options before
hitting the execute button.


---
Sent from VMware Boxer


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] Dynamic Tool UI based on script

[Steve Cassidy]

I could do that but it would mean that no-one could install the tool from the 
toolshed without also installing this package.

Maybe I just need to hack this together.

Steve


—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

On 7 Dec 2017, at 6:59 pm, Hans-Rudolf Hotz <h...@fmi.ch<mailto:h...@fmi.ch>> 
wrote:

Hi Steve

what happened, when you add this module to the environment the galaxy server is 
running in? (e.g. by listing the path in the 'run.sh' file)

Regards, Hans-Rudolf


On 12/07/2017 03:05 AM, Steve Cassidy wrote:
Hi all,
  this would help in a tool I’m trying to write too but the problem I
have implementing this approach is that my script will need an external
python module - is there any way that I can list dependencies for the
script?  As it happens these are the same dependancies that the tool
itself will need once executed.

Thanks,

Steve

—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

On 24 Nov 2017, at 6:52 pm, Hans-Rudolf Hotz <h...@fmi.ch
<mailto:h...@fmi.ch>> wrote:

Hi Evan

Yes, this is possible with "dynamic_options"

e.g.:




2017
2016
2015







...


...





The python function "ds_readFiles" is defined in a extra code file
"more_code.py", which is stored in the same directory as the tool
definition file. The function returns all files with a matching file
name (i.e.year), and could look like:


rDir = "/data/"

def ds_readFiles(year):

  l = os.listdir(rDir)
  p = re.compile('^%s_[0­9]{4}_M[0­9]{5}_[0­9]'%year)
  l.sort()
  path = rDir
  filesoptions = [(s) for s in l if os.path.exists(path + s)]
  files = [(s,s,False) for s in filesoptions if p.match(s)]
  return files



a similar case is also described here:

http://dev.list.galaxyproject.org/Dynamic-Tool-Parameter-Lists-td4175828.html


Hope this helps, Hans-Rudolf


On 11/23/2017 06:02 PM, Evan Clark wrote:
Is it possible to execute a script within a tool xml confit to retrieve
some data and append to the ui depending on the requested data. I.e if I
wanted to get a list of files and provide them as select options before
hitting the execute button.


---
Sent from VMware Boxer


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] Dynamic Tool UI based on script

[Steve Cassidy]

Hi all,
  this would help in a tool I’m trying to write too but the problem I have 
implementing this approach is that my script will need an external python 
module - is there any way that I can list dependencies for the script?  As it 
happens these are the same dependancies that the tool itself will need once 
executed.

Thanks,

Steve

—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

On 24 Nov 2017, at 6:52 pm, Hans-Rudolf Hotz > 
wrote:

Hi Evan

Yes, this is possible with "dynamic_options"

e.g.:




 2017
 2016
 2015







...


...





The python function "ds_readFiles" is defined in a extra code file
"more_code.py", which is stored in the same directory as the tool definition 
file. The function returns all files with a matching file name (i.e.year), and 
could look like:


rDir = "/data/"

def ds_readFiles(year):

   l = os.listdir(rDir)
   p = re.compile('^%s_[0­9]{4}_M[0­9]{5}_[0­9]'%year)
   l.sort()
   path = rDir
   filesoptions = [(s) for s in l if os.path.exists(path + s)]
   files = [(s,s,False) for s in filesoptions if p.match(s)]
   return files



a similar case is also described here:

http://dev.list.galaxyproject.org/Dynamic-Tool-Parameter-Lists-td4175828.html


Hope this helps, Hans-Rudolf


On 11/23/2017 06:02 PM, Evan Clark wrote:
Is it possible to execute a script within a tool xml confit to retrieve
some data and append to the ui depending on the requested data. I.e if I
wanted to get a list of files and provide them as select options before
hitting the execute button.


---
Sent from VMware Boxer


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

[galaxy-dev] planemo with --docker and --conda_dependency_resolution

[Steve Cassidy]

Hi,
 I’m trying to test a few tools, one of which uses a docker container, the 
others use dependencies that can be found in conda.

I can test each one by using either —docker or —conda_dependency_resolution 
with planemo, but when I try to give both of these, it seems to want to only 
use docker and fails because it can’t find a usable container.

I’ve hunted around the planemo sources but i can’t see what magic is creating 
this problem.  Am I just asking too much?

Any pointers appreciated.

Thanks,

Steve

—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Tool needs a particular file extension -- now Datatype help

[Steve Cassidy]

Thanks all,
  it seems that my real problem is that the audio file (.wav) is not being 
identified as a valid datatype and ending up as a zero length text file. So, I 
need to start to explore the world of datatypes.

Following the docs 
(https://wiki.galaxyproject.org/Admin/Datatypes/Adding%20Datatypes) I can 
modify datatypes_conf.xml in my Galaxy sources and add a new datatype for wav 
files:

http://web.science.mq.edu.au/~cassidy

On 21 Oct. 2016, at 12:58 pm, Peter Cock 
<p.j.a.c...@googlemail.com<mailto:p.j.a.c...@googlemail.com>> wrote:

Using a soft link for this is a common pattern, and should be followed with &&
(ideally using XML CDATA to avoid escaping everything like  etc),
and quote the filenames just in case there are any spaces. e.g.

https://github.com/galaxyproject/tools-iuc/blob/master/tools/trinity/run_de_analysis.xml#L16


For reference, in tools-iuc there are over 400 soft link examples:

$ grep "ln -s" tools/*/*.xml | wc -l
446

Peter

On Fri, Oct 21, 2016 at 5:48 PM, Steve Cassidy <steve.cass...@mq.edu.au> wrote:
Hi,
I’m wrapping a tool that needs it’s input to have a known file extension
(an audio file, eg. .wav).  Since Galaxy stores all data as .dat files the
tool is falling over since it doesn’t know what .dat is.

I thought I’d be able to get around this by hard linking the .dat file to
the same name with a .wav extension (dataset_1.dat.wav), this works when I
try it with the tool on the command line but within Galaxy it fails, here’s
my :

   ln $signal ${signal}.wav 
   /home/maus/maus OUTFORMAT=TextGrid LANGUAGE=$language
   BPF=$bpf INSKANTEXTGRID=$inskantextgrid
INSORTTEXTGRID=$insorttextgrid
   MODUS=$modus MAUSSHIFT=$mausshift MINPAUSLEN=$minpauslen
WEIGHT=$weight
   INSPROB=$insprob NOINITIALFINALSILENCE=$noinitialfinalsilence
OUTSYMBOL=$outsymbol
   OUT=$output SIGNAL=${signal}.wav

resulting in the job command line:

ln /tmp/tmp7AZvx7/files/000/dataset_2.dat
/tmp/tmp7AZvx7/files/000/dataset_2.dat.wav & /home/maus/maus
OUTFORMAT=TextGrid LANGUAGE=aus BPF=/tmp/tmp7AZvx7/files/000/dataset_1.dat
INSKANTEXTGRID=false INSORTTEXTGRID=false MODUS=standard MAUSSHIFT=10
MINPAUSLEN=5 WEIGHT=7.0 INSPROB=0.0 NOINITIALFINALSILENCE=no OUTSYMBOL=sampa
OUT=/tmp/tmp7AZvx7/files/000/dataset_3.dat
SIGNAL=/tmp/tmp7AZvx7/files/000/dataset_2.dat.wav

I’m getting an error message from the tool:

sox FAIL formats: can't open input file
`/tmp/tmp7AZvx7/files/000/dataset_2.dat.wav': WAVE: RIFF header not found

this suggests that the hard link didn’t get made.  I tried copying the file
instead but got the same result.

I could go in and patch the tool script to be more forgiving but it would be
good to find a solution that didn’t require that if possible.

Any pointers appreciated.

Steve
—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Tool needs a particular file extension

[Steve Cassidy]

Hi,
 I’m wrapping a tool that needs it’s input to have a known file extension (an 
audio file, eg. .wav).  Since Galaxy stores all data as .dat files the tool is 
falling over since it doesn’t know what .dat is.

I thought I’d be able to get around this by hard linking the .dat file to the 
same name with a .wav extension (dataset_1.dat.wav), this works when I try it 
with the tool on the command line but within Galaxy it fails, here’s my 
:

ln $signal ${signal}.wav 
/home/maus/maus OUTFORMAT=TextGrid LANGUAGE=$language
BPF=$bpf INSKANTEXTGRID=$inskantextgrid INSORTTEXTGRID=$insorttextgrid
MODUS=$modus MAUSSHIFT=$mausshift MINPAUSLEN=$minpauslen WEIGHT=$weight
INSPROB=$insprob NOINITIALFINALSILENCE=$noinitialfinalsilence 
OUTSYMBOL=$outsymbol
OUT=$output SIGNAL=${signal}.wav

resulting in the job command line:

ln /tmp/tmp7AZvx7/files/000/dataset_2.dat 
/tmp/tmp7AZvx7/files/000/dataset_2.dat.wav & /home/maus/maus OUTFORMAT=TextGrid 
LANGUAGE=aus BPF=/tmp/tmp7AZvx7/files/000/dataset_1.dat INSKANTEXTGRID=false 
INSORTTEXTGRID=false MODUS=standard MAUSSHIFT=10 MINPAUSLEN=5 WEIGHT=7.0 
INSPROB=0.0 NOINITIALFINALSILENCE=no OUTSYMBOL=sampa 
OUT=/tmp/tmp7AZvx7/files/000/dataset_3.dat 
SIGNAL=/tmp/tmp7AZvx7/files/000/dataset_2.dat.wav

I’m getting an error message from the tool:

sox FAIL formats: can't open input file 
`/tmp/tmp7AZvx7/files/000/dataset_2.dat.wav': WAVE: RIFF header not found

this suggests that the hard link didn’t get made.  I tried copying the file 
instead but got the same result.

I could go in and patch the tool script to be more forgiving but it would be 
good to find a solution that didn’t require that if possible.

Any pointers appreciated.

Steve
—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Storing data in the user session

[Steve Cassidy]

Thanks Katherine, in my case I want to store the API key for an external 
service (not Galaxy) and get access to that from a tool so that the tool can 
act on behalf of the user.

Steve

—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

On 17 Oct. 2016, at 5:26 pm, Katherine Beaulieu 
<katherine.beaulieu...@gmail.com<mailto:katherine.beaulieu...@gmail.com>> wrote:

Hi Steve,
In terms of accessing the user's api key within a tool definition file, have a 
look at this thread I had started. John had suggested something but I haven't 
gotten it to work but maybe you will have more luck than me.
http://dev.list.galaxyproject.org/Workflow-ID-Galaxy-Instance-URL-API-Key-in-Galaxy-Tool-Config-td4669906.html#a4670104
Cheers,
Katherine

On Mon, Oct 17, 2016 at 5:17 PM, Steve Cassidy 
<steve.cass...@mq.edu.au<mailto:steve.cass...@mq.edu.au>> wrote:
Hi Bjoern,
 just wanting to revisit this question and perhaps get an update on what you 
were doing for extending the user profile.

I just noticed in the galaxy sources the option to add ‘forms’ to be filled out 
at registration time. The admin interface seems to allow me to create  a form 
but it doesn’t appear when I try to register.  The code for this seems to be 
2-5 years old so perhaps it’s a forgotten corner?

https://github.com/galaxyproject/galaxy/tree/dev/templates/admin/forms

Anyway, if you have made any progress on this I’d be happy to hear about it.  
I’d like to be able to store the user’s API key in the profile and access it 
from a tool definition file.

Cheers,

Steve
—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

On 28 Jul. 2016, at 2:59 pm, Björn Grüning 
<bjoern.gruen...@gmail.com<mailto:bjoern.gruen...@gmail.com>> wrote:

Hi Steve & Mohammed,

I'm working on a user-profile extension to get credentials for a user,
but this will take a few more weeks sorry.

In the meantime can you look at the following idea:

my_programm.py --credentials $USER_CREDENTIALS --arguments

The variable $USER_CREDENTIALS is set during job execution with dynamic
destination mappings
https://wiki.galaxyproject.org/Admin/Config/Jobs#Dynamic_Destination_Mapping

What do you think, would this work for you?
This assumes you as admin can pass along the user credentials if you
know the username.

If not please get in touch and I will tell you about my plans to extend
the user-preferences maybe you can help me with this to speed things up.

Ciao,
Bjoern


Hi Bjorn,
sure, I don’t want to have the credentials stored in the tool but the tool
needs to act on behalf of the user to retrieve data from the repository, so
it needs access to credentials.  This could for example be a token generated
via an OAuth2 exchange - basically something that is sent along with every
http request to the remote resource to authenticate the user.

My thought is that this could be stored in the user profile if this can be made
available to the tool in the same way that __user_email__ is now.

When you say ‘on the framework level’ do you mean by Galaxy?

Steve
—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

On 28 Jul 2016, at 5:23 PM, Björn Grüning 
<bjoern.gruen...@gmail.com<mailto:bjoern.gruen...@gmail.com>> wrote:

Hi Steve,

can you explain the larger use-case behind it. Imho you should not
populate a tool with credentials, this should be handled on the
framework level.

Hope you are fine,
Bjoern

Am 28.07.2016 um 03:15 schrieb Steve Cassidy:
Hi again,
 I’m looking for the right way to store some user credentials in the
galaxy session so that tools can work on behalf of the user with our
repository.

Currently users have an API key and they need to upload it as a data
item to that is then passed to each tool that needs it as input.  This
doesn’t seem like the right solution since the API key becomes part of
the history and so would be shared if the history were shared.

What would be better would be a way of storing the API key in the user
session and then being able to pass that into a tool.

I note that there are a few user session variables available in the tool
xml file: __user__, __user_email__, __user_name__ and __user_id__.
There is also a user preferences page where they can fill out a few
details.  However, I can’t see a mechanism to extend this in any way and
have extra properties in the preferences pane that would then be
available to tools via the template file.

I can see that the standard practice for the data sources on
usegalaxy.org<http://usegalaxy.org/> 
<http://usegalaxy.org<http://usegalaxy.org/>> is to send the user off to a
repository website to find data that is then posted back to the galaxy
server.  This isn’t appropriate for us since the interaction with the
repository is not just for downloading a single dataset - tools will
query the repository, download various kinds of data and poss

Re: [galaxy-dev] Storing data in the user session

[Steve Cassidy]

Hi Bjoern,
 just wanting to revisit this question and perhaps get an update on what you 
were doing for extending the user profile.

I just noticed in the galaxy sources the option to add ‘forms’ to be filled out 
at registration time. The admin interface seems to allow me to create  a form 
but it doesn’t appear when I try to register.  The code for this seems to be 
2-5 years old so perhaps it’s a forgotten corner?

https://github.com/galaxyproject/galaxy/tree/dev/templates/admin/forms

Anyway, if you have made any progress on this I’d be happy to hear about it.  
I’d like to be able to store the user’s API key in the profile and access it 
from a tool definition file.

Cheers,

Steve
—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

On 28 Jul. 2016, at 2:59 pm, Björn Grüning 
<bjoern.gruen...@gmail.com<mailto:bjoern.gruen...@gmail.com>> wrote:

Hi Steve & Mohammed,

I'm working on a user-profile extension to get credentials for a user,
but this will take a few more weeks sorry.

In the meantime can you look at the following idea:

my_programm.py --credentials $USER_CREDENTIALS --arguments

The variable $USER_CREDENTIALS is set during job execution with dynamic
destination mappings
https://wiki.galaxyproject.org/Admin/Config/Jobs#Dynamic_Destination_Mapping

What do you think, would this work for you?
This assumes you as admin can pass along the user credentials if you
know the username.

If not please get in touch and I will tell you about my plans to extend
the user-preferences maybe you can help me with this to speed things up.

Ciao,
Bjoern


Hi Bjorn,
sure, I don’t want to have the credentials stored in the tool but the tool
needs to act on behalf of the user to retrieve data from the repository, so
it needs access to credentials.  This could for example be a token generated
via an OAuth2 exchange - basically something that is sent along with every
http request to the remote resource to authenticate the user.

My thought is that this could be stored in the user profile if this can be made
available to the tool in the same way that __user_email__ is now.

When you say ‘on the framework level’ do you mean by Galaxy?

Steve
—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

On 28 Jul 2016, at 5:23 PM, Björn Grüning <bjoern.gruen...@gmail.com> wrote:

Hi Steve,

can you explain the larger use-case behind it. Imho you should not
populate a tool with credentials, this should be handled on the
framework level.

Hope you are fine,
Bjoern

Am 28.07.2016 um 03:15 schrieb Steve Cassidy:
Hi again,
 I’m looking for the right way to store some user credentials in the
galaxy session so that tools can work on behalf of the user with our
repository.

Currently users have an API key and they need to upload it as a data
item to that is then passed to each tool that needs it as input.  This
doesn’t seem like the right solution since the API key becomes part of
the history and so would be shared if the history were shared.

What would be better would be a way of storing the API key in the user
session and then being able to pass that into a tool.

I note that there are a few user session variables available in the tool
xml file: __user__, __user_email__, __user_name__ and __user_id__.
There is also a user preferences page where they can fill out a few
details.  However, I can’t see a mechanism to extend this in any way and
have extra properties in the preferences pane that would then be
available to tools via the template file.

I can see that the standard practice for the data sources on
usegalaxy.org <http://usegalaxy.org> is to send the user off to a
repository website to find data that is then posted back to the galaxy
server.  This isn’t appropriate for us since the interaction with the
repository is not just for downloading a single dataset - tools will
query the repository, download various kinds of data and possibly upload
new data, so we need to store user credentials in some way.

Is there already a good way to achieve this or is this an enhancement to
Galaxy?

Thanks,

Steve


—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy
<http://web.science.mq.edu.au/%7Ecassidy>



___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Help wanted: setting up a production server

[Steve Cassidy]

Hi all,
  I’m working on setting up a Galaxy server for use with speech and language 
data.  While I can probably get there myself eventually, I’m thinking it would 
be good to get some (paid) help if it were available.

If you have some experience setting up a public facing Galaxy instance, and you 
have some time available in the next couple of weeks, please get in touch.

Any location is probably ok as long as we can work out how to reimburse you.  
We (and the servers) are based in Australia but I’m currently in New York.

Thanks,

Steve

—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Testtoolshed status

[Steve Cassidy]

I’m getting a 500 response from the testtoolshed when I try to upload a tool 
via planemo:

➜  alveo-galaxy-tools git:(master) ✗ planemo shed_update --shed_target 
testtoolshed apitools
cd '/Users/steve/projects/alveo-galaxy-tools/apitools' && git rev-parse HEAD
cd '/Users/steve/projects/alveo-galaxy-tools/apitools' && git diff --quiet
Could not update alveoimport
Unexpected HTTP status code: 500: 
  Internal Server Error
  
Internal Server Error
The server has either erred or is incapable of performing
the requested operation.




WSGI Server
  


Repository metadata updated successfully for repository alveoimport.
Failed to update a repository.

In addition, I just got a “Tool Shed could not be reached” message via the web 
interface, so i’m wondering if something is up.

Thanks,

Steve
—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Conda: confused dependencies with multiple R versions and rpy

[Steve Cassidy]

Hi,
  trying to get my tools going, part 29.  I have a Docker image with the latest 
Galaxy release, patched to fix the Conda install bug I found earlier, my tools 
are installed ok and I’ve added a few from the toolshed for good measure.

One of my tools requires R 3.0.2 which is installed ok.   I also installed 
scatterplot from the toolshed which requires R 2.11; that is also installed ok 
by conda. However, I get this message when trying to run scatterplot:


Traceback (most recent call last):
  File 
"/shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/scatterplot/d243056b22ed/scatterplot/scatterplot.py",
 line 5, in 
from rpy import *
  File 
"/tool_deps/rpy/1.0.3/devteam/package_rpy_1_0_3/82170c94ca7c/lib/python/rpy.py",
 line 134, in 
""" % RVERSION)
RuntimeError: No module named _rpy3002

  RPy module can not be imported. Please check if your rpy
  installation supports R 3.0.2. If you have multiple R versions
  installed, you may need to set RHOME before importing rpy. For
  example:

  >>> from rpy_options import set_options
  >>> set_options(RHOME='c:/progra~1/r/rw2011/')
  >>> from rpy import *


So it seems that rpy is installed and is aware of one version of R but not the 
other, and the other is being found (perhaps because it’s the default install 
in the Docker image?)

FYI my docker image recipe is here:

https://github.com/Alveo/docker-galaxy-alveo

and the image itself is here:

https://hub.docker.com/r/stevecassidy/alveo-galaxy/

Any pointers to the way forward appreciated.

Thanks,
Steve
—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Putting it all together - toolshed tool + conda dependency

[Steve Cassidy]

Looks like I’m hitting this issue:

https://github.com/galaxyproject/galaxy/issues/2797

Steve

—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

> On 29 Aug 2016, at 11:09 AM, Steve Cassidy <steve.cass...@mq.edu.au> wrote:
> 
> Thanks Marius,
>   I tried a build based on the image you mentioned and get the following 
> error:
> 
> /export/galaxy-central/database/job_working_directory/000/1/tool_script.sh: 
> line 9: /tool_deps/_conda/bin/activate: No such file or directory
> /export/galaxy-central/database/job_working_directory/000/1/galaxy_1.sh: line 
> 66: /tool_deps/_conda/bin/activate: No such file or directory
> 
> I tried again with the 16.07 release image but got the same result.  
> 
> Checking the /tool_deps/_conda/bin/ directory, indeed there is no activate 
> script, in fact it contains:
> 
> root@db343fbf7372:/galaxy-central# ls /tool_deps/_conda/bin/
> 2to3  conda  c_rehash  easy_install-3.5  idle3.5  pippydoc3   
>  python   python3.5
> python3.5m python3-config  pyvenv-3.5  tclsh8.5  wheelxz
> 2to3-3.5  conda-env  easy_install  idle3 openssl  pydoc  pydoc3.5 
>  
> python3  python3.5-config  python3.5m-config  pyvenv  sqlite3 
> unxz  wish8.5
> 
> 
> Any pointers appreciated.
> 
> Thanks,
> 
> Steve
> 
> —
> Department of Computing, Macquarie University
> http://web.science.mq.edu.au/~cassidy
> 
>> On 26 Aug 2016, at 6:16 PM, Marius van den Beek <m.vandenb...@gmail.com> 
>> wrote:
>> 
>> Hi Steve,
>> 
>> Looks like you're running an older version of galaxy in the docker 
>> container, since newer galaxy will build the metadata environment in a 
>> separate environment called 'conda-metadata-env',
>> 
>> and we have also changed the way that the handlers receive their Python 
>> environment (that's why sqlalachemy couldn't be found).
>> 
>> Can you try updating the container or building from the dev branch? I think 
>> the necessary changes for conda should be in the dev branch, which you can 
>> pull with docker pull quay.io/bgruening/galaxy:dev.
>> 
>> Best,
>> 
>> Marius
>> 
>> 
>> On Aug 26, 2016 6:49 AM, "Steve Cassidy" <steve.cass...@mq.edu.au> wrote:
>> Ok, probing further trying to understand this. It looks like the job runner 
>> is trying to call set_meta() from galaxy_ext/metadata/set_metadata.py.  In 
>> there there’s a line that tries to set up the PATH:
>> 
>> # insert *this* galaxy before all others on sys.path
>> sys.path.insert( 1, os.path.abspath( os.path.join( os.path.dirname( __file__ 
>> ), os.pardir, os.pardir ) ) )
>> 
>> In my case the result is:
>> 
>> ['/export/galaxy-central/database/job_working_directory/000/3’,
>> '/galaxy-central/lib’,
>> '/galaxy-central/lib’,
>> '/export/galaxy-central/database/job_working_directory/000/3/conda-env/lib/python27.zip’,
>> '/export/galaxy-central/database/job_working_directory/000/3/conda-env/lib/python2.7’,
>> '/export/galaxy-central/database/job_working_directory/000/3/conda-env/lib/python2.7/plat-linux2’,
>> '/export/galaxy-central/database/job_working_directory/000/3/conda-env/lib/python2.7/lib-tk’,
>> '/export/galaxy-central/database/job_working_directory/000/3/conda-env/lib/python2.7/lib-old’,
>> '/export/galaxy-central/database/job_working_directory/000/3/conda-env/lib/python2.7/lib-dynload’,
>> '/export/galaxy-central/database/job_working_directory/000/3/conda-env/lib/python2.7/site-packages’,
>> '/export/galaxy-central/database/job_working_directory/000/3/conda-env/lib/python2.7/site-packages/setuptools-25.1.6-py2.7.egg’
>> ]
>> 
>> Looking for sqlalchemy, I find it installed at:
>> 
>> /galaxy-central/.venv/lib/python2.7/site-packages/sqlalchemy
>> 
>> so clearly that line of code is not doing what it should. I’m guessing that 
>> the Conda resolver is running the script within a conda env and the BETA 
>> magic hasn’t quite made it to the right place yet…
>> 
>> BTW I was trying to understand the flags you mentioned but I can’t find 
>> GALAXY_CONFIG_ENABLE_BETA_TOOL_COMMAND_ISOLATION anywhere other than in 
>> sample Dockerfiles - is this some kind of magic incantation that creates a 
>> rift in space-time…enquiring minds want to know!!!
>> 
>> Steve
>> —
>> Department of Computing, Macquarie University
>> http://web.science.mq.edu.au/~cassidy
>> 
>>> On 25 Aug 2016, at 9:54 PM, Steve Cassidy <steve.cass...@mq.edu.au> wrote:
>>> 
>>> Nah, definitely baby st

Re: [galaxy-dev] Putting it all together - toolshed tool + conda dependency

[Steve Cassidy]

Nah, definitely baby steps…

so, in the repo you point to there seems to be an error in the Dockerfile, the 
ENV line should use the var=value syntax to have more than one setting on one 
line (maybe that’s changed recently in docker).

with this I built a new docker image, when I run my first tool it takes an age 
while it’s installing the deps, then it crashes with:

Traceback (most recent call last):
  File 
"/export/galaxy-central/database/job_working_directory/000/1/set_metadata_QUJQLD.py",
 line 1, in 
from galaxy_ext.metadata.set_metadata import set_metadata; set_metadata()
  File "/galaxy-central/lib/galaxy_ext/metadata/set_metadata.py", line 23, in 

from sqlalchemy.orm import clear_mappers
ImportError: No module named sqlalchemy.orm

and the output:

discarding /galaxy-central/tool_deps/_conda/bin from PATH
prepending 
/export/galaxy-central/database/job_working_directory/000/1/conda-env/bin to 
PATH
discarding /galaxy-central/tool_deps/_conda/bin from PATH
prepending 
/export/galaxy-central/database/job_working_directory/000/1/conda-env/bin to 
PATH

I’m guessing this is some kind of conflict between python versions in and out 
of conda environments? Surely sqlalchemy would be installed for Galaxy to work? 
 

I’ll try to dig around this in the morning but if it rings a bell…

Steve
—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

> On 25 Aug 2016, at 8:43 PM, Björn Grüning <bjoern.gruen...@gmail.com> wrote:
> 
> Hi Steve,
> 
> you call this baby-steps? I think this is huge! :)
> 
> All what you are missing is to enable conda in Galaxy.
> Look at Gregs new flavour which is entirely Conda/Galaxy based.
> 
> You need to enable these env vars to make Galaxy conda enabled:
> 
> https://github.com/gregvonkuster/docker-galaxy-csg/blob/master/Dockerfile#L9
> 
> Hope this helps,
> Bjoern
> 
> Am 25.08.2016 um 12:32 schrieb Steve Cassidy:
>> Hi all,
>>  I’m making baby steps towards having a repeatable installation for my
>> tools.  But I’m now stuck, so help would be appreciated.
>> 
>> I have a tool that works and is in the test toolshed (alveoimport in
>> Data Sources). 
>> 
>> It depends on my python package which is now part of bioconda (pyalveo,
>> version 0.6). 
>> 
>> I can run my tool via planemo, which works I think because I have
>> pyalveo installed in a local venv.  
>> 
>> If I try to run the docker image (derived from  bgruening/galaxy-stable
>> but with the testtoolshed added) I am able to install my tool, but it
>> doesn’t pick up the dependency, so it doesn’t work. 
>> 
>> I tried running with planemo turning on conda dependency resolution
>> (following https://pypi.python.org/pypi/planemo/): 
>> 
>> planemo serve --galaxy_branch release_16.01 --conda_dependency_resolution .
>> 
>> It seems to have a go, but fails:
>> 
>> galaxy.tools.deps DEBUG 2016-08-25 17:32:22,449 Building dependency
>> shell command for dependency 'pyalveo'
>> galaxy.tools.deps DEBUG 2016-08-25 17:32:22,449 Find dependency pyalveo
>> version 0.6
>> galaxy.tools.deps DEBUG 2016-08-25 17:32:22,450 Resolver conda returned
>> 
>> (isnull? True)
>> galaxy.tools.deps DEBUG 2016-08-25 17:32:22,450 Resolver conda returned
>> 
>> (isnull? True)
>> galaxy.tools.deps WARNING 2016-08-25 17:32:22,451 Failed to resolve
>> dependency on 'pyalveo', ignoring
>> 
>> So, what’s the easiest route to a Galaxy deployment with my tool
>> installed. The Docker route would be best I think, but what do I have to
>> add to bgruening/galaxy-stable to have conda find my dependencies.
>> 
>> Thanks in advance,
>> 
>> Steve
>> 
>> —
>> Department of Computing, Macquarie University
>> http://web.science.mq.edu.au/~cassidy
>> <http://web.science.mq.edu.au/%7Ecassidy>
>> 
>> 
>> 
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>  https://lists.galaxyproject.org/
>> 
>> To search Galaxy mailing lists use the unified search at:
>>  http://galaxyproject.org/search/mailinglists/
>> 

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Putting it all together - toolshed tool + conda dependency

[Steve Cassidy]

Hi all,
  I’m making baby steps towards having a repeatable installation for my tools.  
But I’m now stuck, so help would be appreciated.

I have a tool that works and is in the test toolshed (alveoimport in Data 
Sources).

It depends on my python package which is now part of bioconda (pyalveo, version 
0.6).

I can run my tool via planemo, which works I think because I have pyalveo 
installed in a local venv.

If I try to run the docker image (derived from  bgruening/galaxy-stable but 
with the testtoolshed added) I am able to install my tool, but it doesn’t pick 
up the dependency, so it doesn’t work.

I tried running with planemo turning on conda dependency resolution (following 
https://pypi.python.org/pypi/planemo/):

planemo serve --galaxy_branch release_16.01 --conda_dependency_resolution .

It seems to have a go, but fails:

galaxy.tools.deps DEBUG 2016-08-25 17:32:22,449 Building dependency shell 
command for dependency 'pyalveo'
galaxy.tools.deps DEBUG 2016-08-25 17:32:22,449 Find dependency pyalveo version 
0.6
galaxy.tools.deps DEBUG 2016-08-25 17:32:22,450 Resolver conda returned 
 (isnull? 
True)
galaxy.tools.deps DEBUG 2016-08-25 17:32:22,450 Resolver conda returned 
 (isnull? 
True)
galaxy.tools.deps WARNING 2016-08-25 17:32:22,451 Failed to resolve dependency 
on 'pyalveo', ignoring

So, what’s the easiest route to a Galaxy deployment with my tool installed. The 
Docker route would be best I think, but what do I have to add to 
bgruening/galaxy-stable to have conda find my dependencies.

Thanks in advance,

Steve

—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Storing data in the user session

[Steve Cassidy]

Hi Mohamed,

The problem with this solution is that the password again becomes part of the 
history since it is one of the inputs to the tool.  If I want to publish this 
workflow then the password can’t be visible.  For this reason password storage 
needs to be outside of workflows and handled somewhere else in Galaxy.

Steve



—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

On 28 Jul 2016, at 6:04 PM, Mohamed Kassam 
<k.mam...@gmail.com<mailto:k.mam...@gmail.com>> wrote:

Hi Steve,
there is another solution is that the user is giving his credential via the 
tool. The only thing is missing the tag password on the xml then you can 
encrypt the password.
I tried to add the new paramter on the build.py but I have some errors, don't 
know if someone arrived to fixed it.

Mohamed

2016-07-28 10:00 GMT+02:00 Steve Cassidy 
<steve.cass...@mq.edu.au<mailto:steve.cass...@mq.edu.au>>:
Hi Mohamed,
   In my case, the credentials on Galaxy are not generally the same as on the 
remote system so we need some way to store a second set of credentials in the 
galaxy session.

So, for example, user Fred has an account on Galaxy 
(f...@here.com<mailto:f...@here.com>) and wants to get data from Alveo as input 
to a workflow. For the Alveo tools to get data, they need Fred’s access token 
from Alveo which Fred can retrieve by logging in to the Alveo website.  Fred 
enters the access key into his preferences in Galaxy.  This key is then passed 
to one or more tools that access Alveo on his behalf:

 - the query tool finds words matching a given pattern from a spoken corpus 
(needs the token)
 - the get-data tool retrieves the audio files for each word (needs the token)
 - the automatic alignment tool finds the locations of the vowel sounds in each 
word
 - the annotation storer tool sends these locations back to Alveo as a new 
annotation set (needs the token)

and so on, at various times in the workflow, the tools need to talk to the 
repository to query, get data or add new data.

At the moment, I need the token to be a data item in the history, but it would 
be better if it were part of the user session in Galaxy.

Steve


—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

On 28 Jul 2016, at 5:46 PM, Mohamed Kassam 
<k.mam...@gmail.com<mailto:k.mam...@gmail.com>> wrote:

Hello Cassidy,
I have the same problem like you because I need the user credential to access 
to an external tools.
I tried the user session variables available in the tool
 xml file: __user__, __user_email__, __user_name__ and __user_id__. I t works 
perfectly I don't know if you need the password, because this is the only 
information is missing.

Mohamed

2016-07-28 9:42 GMT+02:00 Steve Cassidy 
<steve.cass...@mq.edu.au<mailto:steve.cass...@mq.edu.au>>:
Hi Bjorn,
 sure, I don’t want to have the credentials stored in the tool but the tool
needs to act on behalf of the user to retrieve data from the repository, so
it needs access to credentials.  This could for example be a token generated
via an OAuth2 exchange - basically something that is sent along with every
http request to the remote resource to authenticate the user.

My thought is that this could be stored in the user profile if this can be made
available to the tool in the same way that __user_email__ is now.

When you say ‘on the framework level’ do you mean by Galaxy?

Steve
—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

> On 28 Jul 2016, at 5:23 PM, Björn Grüning 
> <bjoern.gruen...@gmail.com<mailto:bjoern.gruen...@gmail.com>> wrote:
>
> Hi Steve,
>
> can you explain the larger use-case behind it. Imho you should not
> populate a tool with credentials, this should be handled on the
> framework level.
>
> Hope you are fine,
> Bjoern
>
> Am 28.07.2016 um 03:15 schrieb Steve Cassidy:
>> Hi again,
>>   I’m looking for the right way to store some user credentials in the
>> galaxy session so that tools can work on behalf of the user with our
>> repository.
>>
>> Currently users have an API key and they need to upload it as a data
>> item to that is then passed to each tool that needs it as input.  This
>> doesn’t seem like the right solution since the API key becomes part of
>> the history and so would be shared if the history were shared.
>>
>> What would be better would be a way of storing the API key in the user
>> session and then being able to pass that into a tool.
>>
>> I note that there are a few user session variables available in the tool
>> xml file: __user__, __user_email__, __user_name__ and __user_id__.
>> There is also a user preferences page where they can fill out a few
>> details.  However, I can’t see a mechanism to extend this in any way and
>> have extra pr

Re: [galaxy-dev] Storing data in the user session

[Steve Cassidy]

Hi Mohamed,
   In my case, the credentials on Galaxy are not generally the same as on the 
remote system so we need some way to store a second set of credentials in the 
galaxy session.

So, for example, user Fred has an account on Galaxy 
(f...@here.com<mailto:f...@here.com>) and wants to get data from Alveo as input 
to a workflow. For the Alveo tools to get data, they need Fred’s access token 
from Alveo which Fred can retrieve by logging in to the Alveo website.  Fred 
enters the access key into his preferences in Galaxy.  This key is then passed 
to one or more tools that access Alveo on his behalf:

 - the query tool finds words matching a given pattern from a spoken corpus 
(needs the token)
 - the get-data tool retrieves the audio files for each word (needs the token)
 - the automatic alignment tool finds the locations of the vowel sounds in each 
word
 - the annotation storer tool sends these locations back to Alveo as a new 
annotation set (needs the token)

and so on, at various times in the workflow, the tools need to talk to the 
repository to query, get data or add new data.

At the moment, I need the token to be a data item in the history, but it would 
be better if it were part of the user session in Galaxy.

Steve


—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

On 28 Jul 2016, at 5:46 PM, Mohamed Kassam 
<k.mam...@gmail.com<mailto:k.mam...@gmail.com>> wrote:

Hello Cassidy,
I have the same problem like you because I need the user credential to access 
to an external tools.
I tried the user session variables available in the tool
 xml file: __user__, __user_email__, __user_name__ and __user_id__. I t works 
perfectly I don't know if you need the password, because this is the only 
information is missing.

Mohamed

2016-07-28 9:42 GMT+02:00 Steve Cassidy 
<steve.cass...@mq.edu.au<mailto:steve.cass...@mq.edu.au>>:
Hi Bjorn,
 sure, I don’t want to have the credentials stored in the tool but the tool
needs to act on behalf of the user to retrieve data from the repository, so
it needs access to credentials.  This could for example be a token generated
via an OAuth2 exchange - basically something that is sent along with every
http request to the remote resource to authenticate the user.

My thought is that this could be stored in the user profile if this can be made
available to the tool in the same way that __user_email__ is now.

When you say ‘on the framework level’ do you mean by Galaxy?

Steve
—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

> On 28 Jul 2016, at 5:23 PM, Björn Grüning 
> <bjoern.gruen...@gmail.com<mailto:bjoern.gruen...@gmail.com>> wrote:
>
> Hi Steve,
>
> can you explain the larger use-case behind it. Imho you should not
> populate a tool with credentials, this should be handled on the
> framework level.
>
> Hope you are fine,
> Bjoern
>
> Am 28.07.2016 um 03:15 schrieb Steve Cassidy:
>> Hi again,
>>   I’m looking for the right way to store some user credentials in the
>> galaxy session so that tools can work on behalf of the user with our
>> repository.
>>
>> Currently users have an API key and they need to upload it as a data
>> item to that is then passed to each tool that needs it as input.  This
>> doesn’t seem like the right solution since the API key becomes part of
>> the history and so would be shared if the history were shared.
>>
>> What would be better would be a way of storing the API key in the user
>> session and then being able to pass that into a tool.
>>
>> I note that there are a few user session variables available in the tool
>> xml file: __user__, __user_email__, __user_name__ and __user_id__.
>> There is also a user preferences page where they can fill out a few
>> details.  However, I can’t see a mechanism to extend this in any way and
>> have extra properties in the preferences pane that would then be
>> available to tools via the template file.
>>
>> I can see that the standard practice for the data sources on
>> usegalaxy.org<http://usegalaxy.org/> 
>> <http://usegalaxy.org<http://usegalaxy.org/>> is to send the user off to a
>> repository website to find data that is then posted back to the galaxy
>> server.  This isn’t appropriate for us since the interaction with the
>> repository is not just for downloading a single dataset - tools will
>> query the repository, download various kinds of data and possibly upload
>> new data, so we need to store user credentials in some way.
>>
>> Is there already a good way to achieve this or is this an enhancement to
>> Galaxy?
>>
>> Thanks,
>>
>> Steve
>>
>>
>> —
>> Department of Computing, Macqu

Re: [galaxy-dev] Storing data in the user session

[Steve Cassidy]

Hi Bjorn,
 sure, I don’t want to have the credentials stored in the tool but the tool 
needs to act on behalf of the user to retrieve data from the repository, so
it needs access to credentials.  This could for example be a token generated
via an OAuth2 exchange - basically something that is sent along with every 
http request to the remote resource to authenticate the user.  

My thought is that this could be stored in the user profile if this can be made
available to the tool in the same way that __user_email__ is now. 

When you say ‘on the framework level’ do you mean by Galaxy? 

Steve
—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

> On 28 Jul 2016, at 5:23 PM, Björn Grüning <bjoern.gruen...@gmail.com> wrote:
> 
> Hi Steve,
> 
> can you explain the larger use-case behind it. Imho you should not
> populate a tool with credentials, this should be handled on the
> framework level.
> 
> Hope you are fine,
> Bjoern
> 
> Am 28.07.2016 um 03:15 schrieb Steve Cassidy:
>> Hi again,
>>   I’m looking for the right way to store some user credentials in the
>> galaxy session so that tools can work on behalf of the user with our
>> repository. 
>> 
>> Currently users have an API key and they need to upload it as a data
>> item to that is then passed to each tool that needs it as input.  This
>> doesn’t seem like the right solution since the API key becomes part of
>> the history and so would be shared if the history were shared.  
>> 
>> What would be better would be a way of storing the API key in the user
>> session and then being able to pass that into a tool.
>> 
>> I note that there are a few user session variables available in the tool
>> xml file: __user__, __user_email__, __user_name__ and __user_id__.
>> There is also a user preferences page where they can fill out a few
>> details.  However, I can’t see a mechanism to extend this in any way and
>> have extra properties in the preferences pane that would then be
>> available to tools via the template file.  
>> 
>> I can see that the standard practice for the data sources on
>> usegalaxy.org <http://usegalaxy.org> is to send the user off to a
>> repository website to find data that is then posted back to the galaxy
>> server.  This isn’t appropriate for us since the interaction with the
>> repository is not just for downloading a single dataset - tools will
>> query the repository, download various kinds of data and possibly upload
>> new data, so we need to store user credentials in some way. 
>> 
>> Is there already a good way to achieve this or is this an enhancement to
>> Galaxy? 
>> 
>> Thanks,
>> 
>> Steve
>> 
>> 
>> —
>> Department of Computing, Macquarie University
>> http://web.science.mq.edu.au/~cassidy
>> <http://web.science.mq.edu.au/%7Ecassidy>
>> 
>> 
>> 
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>  https://lists.galaxyproject.org/
>> 
>> To search Galaxy mailing lists use the unified search at:
>>  http://galaxyproject.org/search/mailinglists/

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Storing data in the user session

[Steve Cassidy]

Hi again,
   I’m looking for the right way to store some user credentials in the galaxy 
session so that tools can work on behalf of the user with our repository.

Currently users have an API key and they need to upload it as a data item to 
that is then passed to each tool that needs it as input.  This doesn’t seem 
like the right solution since the API key becomes part of the history and so 
would be shared if the history were shared.

What would be better would be a way of storing the API key in the user session 
and then being able to pass that into a tool.

I note that there are a few user session variables available in the tool xml 
file: __user__, __user_email__, __user_name__ and __user_id__.  There is also a 
user preferences page where they can fill out a few details.  However, I can’t 
see a mechanism to extend this in any way and have extra properties in the 
preferences pane that would then be available to tools via the template file.

I can see that the standard practice for the data sources on 
usegalaxy.org is to send the user off to a repository 
website to find data that is then posted back to the galaxy server.  This isn’t 
appropriate for us since the interaction with the repository is not just for 
downloading a single dataset - tools will query the repository, download 
various kinds of data and possibly upload new data, so we need to store user 
credentials in some way.

Is there already a good way to achieve this or is this an enhancement to Galaxy?

Thanks,

Steve


—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Papers that cite Galaxy workflows

[Steve Cassidy]

Thanks Dave, that’s just what I’m looking for, I found a few good examples of 
workflow citations.

Steve

—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

On 27 Jul 2016, at 6:29 AM, Dave Clements 
<cleme...@galaxyproject.org<mailto:cleme...@galaxyproject.org>> wrote:

Hi Steve,

I don't have a precise answer to your question, but the Galaxy CiteULike 
archive might be helpful here.  Any paper that uses, extends, or references 
Galaxy gets put in that library.  Papers are tagged with one or more of 17 
tags<https://wiki.galaxyproject.org/CiteULike>.  There aren't specific tags for 
"defines and publishes a workflow" or "reuses a previously published workflow" 
but I'm pretty sure I've come across many papers that do this.

A good place to start though would be papers with the shared 
tag<http://www.citeulike.org/group/16008/tag/shared>: Papers that have 
published/shared workflows, histories, datasets, pages, visualizations in a 
Galaxy instance.

Also, not sure if 
CloudMap<http://www.citeulike.org/group/16008/article/11450604> qualifies here, 
but if it does, many papers that reference it are reusing the pipeline defined 
in it.

Hope this helps,

Dave C

On Thu, Jul 21, 2016 at 7:21 PM, Steve Cassidy 
<steve.cass...@mq.edu.au<mailto:steve.cass...@mq.edu.au>> wrote:
Hi all,
 I’m writing a paper that will include some evangelism for Galaxy in language 
sciences and I’d like to be able to cite examples of papers in the Bio field 
that cite Galaxy workflows.  Even better would be the re-use of a Galaxy 
workflow cited in another paper.  Does anyone have any good example papers they 
could refer me to.

Thanks,

Steve
—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/



--
http://galaxyproject.org/
http://getgalaxy.org/
http://usegalaxy.org/
https://wiki.galaxyproject.org/

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Papers that cite Galaxy workflows

[Steve Cassidy]

Hi all,
 I’m writing a paper that will include some evangelism for Galaxy in language 
sciences and I’d like to be able to cite examples of papers in the Bio field 
that cite Galaxy workflows.  Even better would be the re-use of a Galaxy 
workflow cited in another paper.  Does anyone have any good example papers they 
could refer me to.

Thanks,

Steve
—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] planemo shed_create bug? on planemo/server docker image

[Steve Cassidy]

Hi Bjorn,
  thanks for the response, very happy to go with the future way of doing 
things.  I did see the references to conda but couldn’t quite decipher what was 
needed. 

Are you saying that I should create a new recipe in bioconda for my dependency? 
Does it matter that this is not in the least ‘bio’ - these are for our language 
analysis tools, in the first instance this is a tool to pull data from our 
language repository.  

How does Galaxy know about the bioconda channel - is there a configuration 
setting pointing to the channels to find dependencies in? Could we create a 
language-conda channel to house language dependencies? Does that question make 
sense (not totally up on conda yet).

Steve 

—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

> On 14 Jul 2016, at 5:00 PM, Björn Grüning <bjoern.gruen...@gmail.com> wrote:
> 
> Hi Steve,
> 
> are you fine restricting your tools to a recent Galaxy version, let's
> say 6 month old?
> 
> If you forget about tool_depenencies.xml files, completely forget these!
> Conda packaging is the way to go and way more easy for you.
> 
> Let me know and I will give a short introduction or follow the
> description on https://github.com/bioconda/bioconda-recipes to create a
> PR and a package for your tool.
> 
> This can be used by Galaxy automatically.
> Thanks,
> Bjoern
> 
> Am 14.07.2016 um 03:49 schrieb Steve Cassidy:
>> I’m just getting my head around tool dependencies and in doing so trying to 
>> push my tools to a local toolshed which I can handily run with the 
>> planemo/server docker image via Kitematic.
>> 
>> That toolshed is set up with an admin user 
>> plan...@test.com<mailto:plan...@test.com> so I have a stanza in my 
>> .planemo.yml:
>> 
>> shed_username: "stevecassidy"
>> sheds:
>>  http://localhost:32768/:
>>key: ""
>>email: "plan...@test.com<mailto:plan...@test.com>"
>>password: "planemo"
>> 
>> Now I try to create a repository in the toolshed with planemo:
>> 
>> $ planemo shed_create --shed_target http://localhost:32768/
>> Repository created
>> cd '/Users/steve/projects/hcsvlab-galaxy/tools/alveo' && git rev-parse HEAD
>> cd '/Users/steve/projects/hcsvlab-galaxy/tools/alveo' && git diff --quiet
>> Repository [alveoimport] does not exist in the targeted Tool Shed.
>> 
>> 
>> Note the conflicting messages here. The repository is created (and I can 
>> confirm that it is there) but then I get the message that the repo does not 
>> exist in the Toolshed.  After a bit of digging it turns out that this is 
>> because the repo is owned by the user ‘planemo’ rather than my username as 
>> declared in .planemo.yml.  If I add —owner plane to the command then it 
>> works fine.
>> 
>> So I’m not sure if this is a bug in the shed_create code or a weird 
>> interaction with the setup on planemo/server.
>> 
>> Steve
>> 
>> —
>> Department of Computing, Macquarie University
>> http://web.science.mq.edu.au/~cassidy
>> 
>> 
>> 
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>  https://lists.galaxyproject.org/
>> 
>> To search Galaxy mailing lists use the unified search at:
>>  http://galaxyproject.org/search/mailinglists/

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] planemo shed_create bug? on planemo/server docker image

[Steve Cassidy]

I’m just getting my head around tool dependencies and in doing so trying to 
push my tools to a local toolshed which I can handily run with the 
planemo/server docker image via Kitematic.

That toolshed is set up with an admin user 
plan...@test.com so I have a stanza in my .planemo.yml:

shed_username: "stevecassidy"
sheds:
  http://localhost:32768/:
key: ""
email: "plan...@test.com"
password: "planemo"

Now I try to create a repository in the toolshed with planemo:

$ planemo shed_create --shed_target http://localhost:32768/
Repository created
cd '/Users/steve/projects/hcsvlab-galaxy/tools/alveo' && git rev-parse HEAD
cd '/Users/steve/projects/hcsvlab-galaxy/tools/alveo' && git diff --quiet
Repository [alveoimport] does not exist in the targeted Tool Shed.


Note the conflicting messages here. The repository is created (and I can 
confirm that it is there) but then I get the message that the repo does not 
exist in the Toolshed.  After a bit of digging it turns out that this is 
because the repo is owned by the user ‘planemo’ rather than my username as 
declared in .planemo.yml.  If I add —owner plane to the command then it works 
fine.

So I’m not sure if this is a bug in the shed_create code or a weird interaction 
with the setup on planemo/server.

Steve

—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Position Vacant - Galaxy and Language Research in Sydney

[Steve Cassidy]

Hi all,
 I’d like to draw your attention to the following vacancy on our project in 
Sydney, Australia:

https://goo.gl/6NcY3g

This is on the Alveo project and I’m posting here because part of the platform 
is a Galaxy server with tools developed to work on language data (text and 
speech).   The role would suit someone with knowledge of Galaxy and some 
exposure to general eResearch problems; knowledge of language research methods 
would obviously be a bonus.

The ad mentions a part-time role for 6 months but in practice we have funding 
for longer than this and for a full time role for the right candidate. Please 
get in touch if you are interested in finding out about the post. International 
candidates are welcome to apply.

Steve
—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Tools that make datasets

[Steve Cassidy]

A brief writeup of my experiences:

http://web.science.mq.edu.au/~cassidy/wordpress/2015/10/21/galaxy-tool-generating-datasets/

Steve

On 21 October 2015 at 18:27, Peter van Heusden <p...@sanbi.ac.za> wrote:

> These names have meaning in CollectedDatasetMatch:
>
> designation means designation if that exists else name
> name means name
> dbkey means dbkey
> ext means ext
> visible means visible
>
>
> On 21 October 2015 at 08:44, Steve Cassidy <steve.cass...@mq.edu.au>
> wrote:
>
>> Thanks, yes that works, though I'm not really sure what the difference is
>> between __name__ and __designation__. They both seem to have the same
>> effect in my example.
>>
>> Thanks for your help.
>>
>> Steve
>>
>>
>> On 21 October 2015 at 16:49, Peter van Heusden <p...@sanbi.ac.za> wrote:
>>
>>> Since you're using a directory, you can use one of the built in patterns:
>>>
>>> DEFAULT_EXTRA_FILENAME_PATTERN = 
>>> r"primary_DATASET_ID_(?P[^_]+)_(?P[^_]+)_(?P[^_]+)(_(?P[^_]+))?"
>>>
>>> NAMED_PATTERNS = {
>>> "__default__": DEFAULT_EXTRA_FILENAME_PATTERN,
>>> "__name__": r"(?P.*)",
>>> "__designation__": r"(?P.*)",
>>> "__name_and_ext__": r"(?P.*)\.(?P[^\.]+)?",
>>> "__designation_and_ext__": r"(?P.*)\.(?P[^\._]+)?",
>>> }
>>>
>>> In terms of docs, I don't know what the future is - the Galaxy wiki or 
>>> http://galaxy.readthedocs.org/en/master/
>>>
>>>
>>> On 21 October 2015 at 04:26, Steve Cassidy <steve.cass...@mq.edu.au>
>>> wrote:
>>>
>>>> Ah, thankyou, yes, I can now get results by using patterns to match the
>>>> output.  I used your example but prepend 'simple' to the filename and then
>>>> searches for that with:
>>>>
>>>> 
>>>> this solves the problem for the sample script but not generally since
>>>> in general I can't predict the filenames that will be generated - this is a
>>>> tool for downloading data from a repository which could be text, audio or
>>>> video data.
>>>>
>>>> If I don't use the 'simple' prefix and omit the file extension I still
>>>> get my data but I also get three other files which are temporary scripts
>>>> generated by galaxy and placed in the working directory.  So, back to
>>>> trying to put things in a subdirectory. It turns out that the issue I was
>>>> having was as you pointed out earlier, the directory attribute to
>>>> discover_datasets doesn't allow variables, so I need to write to a fixed
>>>> directory name:
>>>>
>>>> 
>>>> This now works!
>>>>
>>>> I had thought that I'd need to use a unique directory name but since
>>>> galaxy runs each job in a separate directory, this isn't required.  My real
>>>> tool now works too after following the same pattern.
>>>>
>>>> Thanks for your help.  I'll see if I can write this up in a blog post.
>>>>
>>>> Steve
>>>>
>>>>
>>>>
>>>> On 21 October 2015 at 00:06, Peter van Heusden <p...@sanbi.ac.za> wrote:
>>>>
>>>>> I poked around at your tool XML and the code a bit and the problem is
>>>>> directory="$job_name". Galaxy expects to collect files from the job's
>>>>> working directory - basically the current working directory the job runs
>>>>> in. The directory= argument doesn't have variables expanded as far as I 
>>>>> can
>>>>> tell. In any event it is used in walk_over_extra_files() that is in
>>>>> lib/galaxy/tools/parameters/output_collect.py - if you look there you see
>>>>> that it is simply appended to the job's working directory.
>>>>>
>>>>> So if you use:
>>>>>
>>>>> 
>>>>>
>>>>> (note the  and  - this is effectively the regexp
>>>>> r"(?P.*)\.txt" with the < and > escaped out)
>>>>>
>>>>> And alter the code so that it just writes files to the current
>>>>> directory, then you'll pick up the files one.txt, two,txt and three.txt.
>>>>>
>>>>> Peter
>>>>>
>>>>> On 20 October 2015 at 12:28, Steve Cassidy <steve.cass...@mq.edu.au>
>>>>> w

Re: [galaxy-dev] Tools that make datasets

[Steve Cassidy]

Thanks Peter,
  I did see that proviso somewhere but no, refreshing doesn't help.

That page was one of those that I referred to getting to this point.

Steve

On 20 October 2015 at 18:33, Peter van Heusden <p...@sanbi.ac.za> wrote:

> Just a quick check - did you refresh your history to confirm that the
> dataset *is* empty? We had the same thing at SANBI but it turns out that
> Galaxy creates an empty output collection and then only populates it
> sometime after job completion (this is a know UI bug).
>
> See:
> http://pvh.wp.sanbi.ac.za/2015/09/18/adventures-in-galaxy-output-collections/
>
> On 20 October 2015 at 08:48, Steve Cassidy <steve.cass...@mq.edu.au>
> wrote:
>
>> Hi all,
>>   I'm trying to understand how to write a tool that generates a dataset
>> rather than a single output file.  I've tried following all of the examples
>> but I'm stuck, so I thought I would distil down the simplest example I
>> could write and ask for help here.
>>
>> So here's my example:
>>
>> https://gist.github.com/stevecassidy/0fa45ad5853faacb5f55
>>
>> it's a simple python script that writes three files to a directory named
>> for the single input parameter.
>>
>> I think one of the problems I'm having is knowing where to write the
>> output to. I've run this under planemo serve and the job runs, creating the
>> output directory within the 'job_working_directory/000/1/SampleDataset'
>> directory, however my dataset doesn't contain anything so clearly my
>> outputs directive isn't working:
>>
>> 
>> 
>> > directory="$job_name" />
>> 
>> 
>>
>> ($job_name is the name of the directory that is being written to,
>> SampleDataset in this case)
>>
>> Any help in getting this example working would be appreciated.
>>
>> Thanks,
>>
>> Steve
>>
>>
>>
>>
>>
>> --
>> Department of Computing, Macquarie University
>> http://web.science.mq.edu.au/~cassidy/
>>
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>   https://lists.galaxyproject.org/
>>
>> To search Galaxy mailing lists use the unified search at:
>>   http://galaxyproject.org/search/mailinglists/
>>
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
>



-- 
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Tools that make datasets

[Steve Cassidy]

Yes, I'm sure that's where the problem lies. Writing out to the current
directory doesn't work.  The files get written to
'job_working_directory/000/1/' but if I run the Upload File tool the result
is placed in 'files/000/'.  I think I need to work out where to write the
files, I found some references to $__new_file_path__ but that doesn't seem
to help.

Steve



On 20 October 2015 at 19:57, Peter van Heusden <p...@sanbi.ac.za> wrote:

> I suspect that the problem might be in the  then. I'm
> not an export on this, but "__name_and_ext__" turns into the
> regexp r"(?P.*)\.(?P[^\.]+)?" in
> lib/galaxy/tools/parameters/output_collect.py, and is used by the
> DatasetCollector (line 358). This looks like it should match the filenames
> you're creating, but I'm not 100% sure how that code works. One thing I
> notice is the "directory" argument. If you write jobs to the current
> directory instead of "output_path" can you get it to work?
>
> Peter
>
> On 20 October 2015 at 09:52, Steve Cassidy <steve.cass...@mq.edu.au>
> wrote:
>
>> Thanks Peter,
>>   I did see that proviso somewhere but no, refreshing doesn't help.
>>
>> That page was one of those that I referred to getting to this point.
>>
>> Steve
>>
>> On 20 October 2015 at 18:33, Peter van Heusden <p...@sanbi.ac.za> wrote:
>>
>>> Just a quick check - did you refresh your history to confirm that the
>>> dataset *is* empty? We had the same thing at SANBI but it turns out that
>>> Galaxy creates an empty output collection and then only populates it
>>> sometime after job completion (this is a know UI bug).
>>>
>>> See:
>>> http://pvh.wp.sanbi.ac.za/2015/09/18/adventures-in-galaxy-output-collections/
>>>
>>> On 20 October 2015 at 08:48, Steve Cassidy <steve.cass...@mq.edu.au>
>>> wrote:
>>>
>>>> Hi all,
>>>>   I'm trying to understand how to write a tool that generates a dataset
>>>> rather than a single output file.  I've tried following all of the examples
>>>> but I'm stuck, so I thought I would distil down the simplest example I
>>>> could write and ask for help here.
>>>>
>>>> So here's my example:
>>>>
>>>> https://gist.github.com/stevecassidy/0fa45ad5853faacb5f55
>>>>
>>>> it's a simple python script that writes three files to a directory
>>>> named for the single input parameter.
>>>>
>>>> I think one of the problems I'm having is knowing where to write the
>>>> output to. I've run this under planemo serve and the job runs, creating the
>>>> output directory within the 'job_working_directory/000/1/SampleDataset'
>>>> directory, however my dataset doesn't contain anything so clearly my
>>>> outputs directive isn't working:
>>>>
>>>> 
>>>> 
>>>> >>> directory="$job_name" />
>>>> 
>>>> 
>>>>
>>>> ($job_name is the name of the directory that is being written to,
>>>> SampleDataset in this case)
>>>>
>>>> Any help in getting this example working would be appreciated.
>>>>
>>>> Thanks,
>>>>
>>>> Steve
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Department of Computing, Macquarie University
>>>> http://web.science.mq.edu.au/~cassidy/
>>>>
>>>> ___
>>>> Please keep all replies on the list by using "reply all"
>>>> in your mail client.  To manage your subscriptions to this
>>>> and other Galaxy lists, please use the interface at:
>>>>   https://lists.galaxyproject.org/
>>>>
>>>> To search Galaxy mailing lists use the unified search at:
>>>>   http://galaxyproject.org/search/mailinglists/
>>>>
>>>
>>>
>>> ___
>>> Please keep all replies on the list by using "reply all"
>>> in your mail client.  To manage your subscriptions to this
>>> and other Galaxy lists, please use the interface at:
>>>   https://lists.galaxyproject.org/
>>>
>>> To search Galaxy mailing lists use the unified search at:
>>>   http://galaxyproject.org/search/mailinglists/
>>>
>>
>>
>>
>> --
>> Department of Computing, Macquarie University
>> http://web.science.mq.edu.au/~cassidy/
>>
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
>



-- 
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Tools that make datasets

[Steve Cassidy]

Hi all,
  I'm trying to understand how to write a tool that generates a dataset
rather than a single output file.  I've tried following all of the examples
but I'm stuck, so I thought I would distil down the simplest example I
could write and ask for help here.

So here's my example:

https://gist.github.com/stevecassidy/0fa45ad5853faacb5f55

it's a simple python script that writes three files to a directory named
for the single input parameter.

I think one of the problems I'm having is knowing where to write the output
to. I've run this under planemo serve and the job runs, creating the output
directory within the 'job_working_directory/000/1/SampleDataset' directory,
however my dataset doesn't contain anything so clearly my outputs directive
isn't working:







($job_name is the name of the directory that is being written to,
SampleDataset in this case)

Any help in getting this example working would be appreciated.

Thanks,

Steve





-- 
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Tools that make datasets

[Steve Cassidy]

Ah, thankyou, yes, I can now get results by using patterns to match the
output.  I used your example but prepend 'simple' to the filename and then
searches for that with:


this solves the problem for the sample script but not generally since in
general I can't predict the filenames that will be generated - this is a
tool for downloading data from a repository which could be text, audio or
video data.

If I don't use the 'simple' prefix and omit the file extension I still get
my data but I also get three other files which are temporary scripts
generated by galaxy and placed in the working directory.  So, back to
trying to put things in a subdirectory. It turns out that the issue I was
having was as you pointed out earlier, the directory attribute to
discover_datasets doesn't allow variables, so I need to write to a fixed
directory name:


This now works!

I had thought that I'd need to use a unique directory name but since galaxy
runs each job in a separate directory, this isn't required.  My real tool
now works too after following the same pattern.

Thanks for your help.  I'll see if I can write this up in a blog post.

Steve



On 21 October 2015 at 00:06, Peter van Heusden <p...@sanbi.ac.za> wrote:

> I poked around at your tool XML and the code a bit and the problem is
> directory="$job_name". Galaxy expects to collect files from the job's
> working directory - basically the current working directory the job runs
> in. The directory= argument doesn't have variables expanded as far as I can
> tell. In any event it is used in walk_over_extra_files() that is in
> lib/galaxy/tools/parameters/output_collect.py - if you look there you see
> that it is simply appended to the job's working directory.
>
> So if you use:
>
> 
>
> (note the  and  - this is effectively the regexp
> r"(?P.*)\.txt" with the < and > escaped out)
>
> And alter the code so that it just writes files to the current directory,
> then you'll pick up the files one.txt, two,txt and three.txt.
>
> Peter
>
> On 20 October 2015 at 12:28, Steve Cassidy <steve.cass...@mq.edu.au>
> wrote:
>
>> Sorry, it was just an example of a tool that works - the file that it
>> writes out is put into that directory, so I assume that's where my files
>> should end up too.
>>
>> Steve
>>
>> On 20 October 2015 at 21:12, Peter van Heusden <p...@sanbi.ac.za> wrote:
>>
>>> Sorry, I don't understand - what does the Upload File tool have to do
>>> with this?
>>>
>>> On 20 October 2015 at 11:49, Steve Cassidy <steve.cass...@mq.edu.au>
>>> wrote:
>>>
>>>> Yes, I'm sure that's where the problem lies. Writing out to the current
>>>> directory doesn't work.  The files get written to
>>>> 'job_working_directory/000/1/' but if I run the Upload File tool the result
>>>> is placed in 'files/000/'.  I think I need to work out where to write the
>>>> files, I found some references to $__new_file_path__ but that doesn't seem
>>>> to help.
>>>>
>>>> Steve
>>>>
>>>>
>>>>
>>>> On 20 October 2015 at 19:57, Peter van Heusden <p...@sanbi.ac.za> wrote:
>>>>
>>>>> I suspect that the problem might be in the  then.
>>>>> I'm not an export on this, but "__name_and_ext__" turns into the
>>>>> regexp r"(?P.*)\.(?P[^\.]+)?" in
>>>>> lib/galaxy/tools/parameters/output_collect.py, and is used by the
>>>>> DatasetCollector (line 358). This looks like it should match the filenames
>>>>> you're creating, but I'm not 100% sure how that code works. One thing I
>>>>> notice is the "directory" argument. If you write jobs to the current
>>>>> directory instead of "output_path" can you get it to work?
>>>>>
>>>>> Peter
>>>>>
>>>>> On 20 October 2015 at 09:52, Steve Cassidy <steve.cass...@mq.edu.au>
>>>>> wrote:
>>>>>
>>>>>> Thanks Peter,
>>>>>>   I did see that proviso somewhere but no, refreshing doesn't help.
>>>>>>
>>>>>> That page was one of those that I referred to getting to this point.
>>>>>>
>>>>>> Steve
>>>>>>
>>>>>> On 20 October 2015 at 18:33, Peter van Heusden <p...@sanbi.ac.za>
>>>>>> wrote:
>>>>>>
>>>>>>> Just a quick check - did you refresh your history to confirm that
>>>>>>> the dataset *is* empty? We had the same thing

Re: [galaxy-dev] planemo and tool dependencies

[Steve Cassidy]

Thanks Christian,
  I'm not sure I understand how to make use of this.  My tool dependency is
described in an XML file which is in the same source directory as the tool
xml and python files.  The tool xml file contains:

python_nltk_tools

and the tool_dependencies.xml file contains the definition:



...

how do I tell planemo to tell galaxy that it should refer to this tool
dependency file to resolve the requirement? At the moment it's loading all
of the tools but doesn't seem to be concerned that the requirement isn't
met.

Thanks for your help.

Steve



On 3 September 2015 at 18:06, Christian Brenninkmeijer <
christian.brenninkmei...@manchester.ac.uk> wrote:

> Hi,
>
> I wrote an extension that will help planemo pull in tool dependencies.
>
> See
> https://github.com/galaxyproject/galaxy/blob/master/lib/galaxy/tools/deps/resolvers/unlinked_tool_shed_packages.py
>
> It is for when the dependency was installed into galaxy but where planemo
> is not using the database to find that tool.
>
> It works my helping planemo guess where the tool dependency may be.
>
> Hope that helps.
> Christian
> University of Manchester
> --
> *From:* galaxy-dev [galaxy-dev-boun...@lists.galaxyproject.org] on behalf
> of Steve Cassidy [steve.cass...@mq.edu.au]
> *Sent:* Thursday, September 03, 2015 9:00 AM
> *To:* galaxy-dev@lists.galaxyproject.org
> *Subject:* [galaxy-dev] planemo and tool dependencies
>
> Hi all,
>   I'm trying to test some tools under planemo and can't find out how to
> have it install tool dependencies while setting up the environment.
>
> The dependency is on a particular python module and we have a
> tool_dependencies.xml file to describe it. I can't see a way to tell
> planemo serve etc to refer to that file to get the dependencies.
>
> Any help appreciated.
>
> Steve
>
> --
> Department of Computing, Macquarie University
> http://web.science.mq.edu.au/~cassidy/
>



-- 
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] planemo and tool dependencies

[Steve Cassidy]

Hi all,
  I'm trying to test some tools under planemo and can't find out how to
have it install tool dependencies while setting up the environment.

The dependency is on a particular python module and we have a
tool_dependencies.xml file to describe it. I can't see a way to tell
planemo serve etc to refer to that file to get the dependencies.

Any help appreciated.

Steve

-- 
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/