Hi John,

     Thanks for your reply.
Question 1:
      I used the relative path (e.g. ../../../applications/executable) which 
still does not work. I have managed to put the script in the same folder which 
works though against the initial design.
Question 2:
      I pass the $output to the perl script within which a file is generated 
with the filename contained in $output. The Galaxy server can recognize the 
output. However when I generate another file with the name 
"primary_$job.id_additional_visible_txt" in the same folder as the $output. In 
theory, it should also be recognized by Galaxy as being in the 
job_working_directory. But it is not. Could you reckon the reason? 
Question 3:
      Your explanation is very clear. Thank you again for this. The initial 
purpose of asking this question is related to question 2: I was wondering which 
"correct" folder I should use to put the additional optional outputs. Now as 
you suggested, I should not worry about it at all.
Question 4: 
     I have managed to convert a RAW file (900M) to mzML (2.3G). But if there 
is a limit on the file size, the spectral file maybe need to be split into 
multiple mgf files. 

Best regards!
Jun

-----Original Message-----
From: John Chilton [mailto:jmchil...@gmail.com] 
Sent: Monday, March 3, 2014 2:27 PM
To: Jun Fan
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] LWR questions

Hey Jun,

Thanks for your continued interest in the LWR. Responses inline.

On Sun, Mar 2, 2014 at 7:00 PM, Jun Fan <j....@qmul.ac.uk> wrote:
> Hi all,
>
>
>
>       LWR sounds very attractive and promising. However how to program 
> on galaxy side to utilize LWR server is not thoroughly documented.

Critique noted, I have tried to make it seamless but when you move a job from 
Linux to Windows there are bound to be severe limitations and I should document 
these more clearly and provide a guide for writing Windows compatible tools 
(https://bitbucket.org/jmchilton/lwr/issue/9/provide-documentation-for-writing-windows).

> I have
> several questions to ask:
>
> 1.       I separate the tool wrappers (.xml) and the applications (.pl, .py
> etc) in two different folders. In the traditional way it works fine. 
> When I use the legacy way of using LWR (adding galaxy:tool_runners 
> section in the universal_wsgi.ini file), the error message says that 
> the application cannot be found. Does the executable have to be in the 
> same folder with the wrapper?

Do you hard code absolute paths to your scripts in the tool XML files then? The 
LWR does not support that. If you place a single script in the same folder as 
the XML (a simple light-weight wrapper as is fairly
common) - that is supported - otherwise you will need to place all the required 
dependencies on the LWR's path on the remote Windows server.
This could get awkward if used with interpreter tag for instance - if you send 
me examples of your tools I can provide some guidance.

I guess in general my advice for Windows tools would be if you can use a simple 
one-file wrapper script placed next the tool you are fine - just make sure the 
underlying application is loaded from the path instead of via absolute paths - 
and then make sure all dependencies for the wrapper script are installed on the 
remote server and the underlying application is on the LWR's path.

(If your remote server is a *nix server there is new support for using 
tool_dependency_dir style Galaxy dependencies that resolve on the remote server 
- but this will not work with Windows (enhancements to provide this would be 
welcomed though) ).

>
> 2.       In the thread
> http://dev.list.galaxyproject.org/Managing-Data-Locality-td4662438.htm
> l#a4662570 John only mentioned $GALAXY_ROOT/databases/files, does this 
> mean that the dynamic multiple output is not supported as it uses 
> $GALAXY_ROOT/databases/tmp folder?

https://bitbucket.org/galaxy/galaxy-central/src/eafb4208db665dfeed7d8f3e6f0d2b61ce2f4fc8/universe_wsgi.ini.sample?at=default#cl-213

My understanding is that writing those files to the databases/tmp is somewhat 
deprecated - ideally these should be written to the working directory of the 
job. The newest version of the LWR *should* attempt to transfer various kinds 
of working directory files back to the Galaxy server after the job is complete 
- these files *should* include such variable output files. This hasn't been 
tested on Windows though
- if there are problems please let me know and I can try to address them -

https://bitbucket.org/jmchilton/lwr/src/51f097d3d517064159bd959c61cee71f7894ea92/lwr/lwr_client/staging/down.py?at=default#cl-17

>
> 3.       What types of files are supposed to be stored under tools directory
> (tool_files), working directory (working), inputs (inputs), configs 
> directory (configs), all of which are subfolders of 
> ${staging_directory}/${job_identifier}?

In theory, you shouldn't need to be worried about these implementations details 
as an application programmer, it should feel just like Galaxy (again with 
limitations).  But for what it is worth - the LWR client will transfer tool 
wrappers it infers are necessary to run the job to `tool_files`, the contents 
of any `configfile` templates defined for the job to `configs` 
(https://wiki.galaxyproject.org/Admin/Tools/ToolConfigSyntax#A.3Cconfigfile.3E_tag_set),
and any inputs it infers are nessecary to `inputs`.

`working` is will be the working directory for the job - so any files your 
script or application writes to relative directories at runtime will be 
relative to this directory and finally all output paths (say you have $output 
defined in your Cheetah) will be rewritten so they target the outputs directory.

>
> 4.       Is there file size limit on the LWR server?

It is quite possible there is. Before I had left MSI we had transferred 
multi-gigabyte files to and from the LWR - but I don't know if we had ever 
actually transferred a file over 4Gb (an obvious potential limit). I get the 
sense that the underlying paste server is kind of taxed by these. Like Galaxy 
itself, it could presumably be hosted in a more robust web server. I have done 
some looking at chaussette for instance which sees to have support for Windows.

Hope this helps somewhat,
-John

>
>
>
> Best regards!
>
> Jun
>
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this and other 
> Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Reply via email to