On Thu, Mar 6, 2014 at 8:04 PM, Jun Fan <j....@qmul.ac.uk> wrote:
> Hi John,
>      Thanks for your reply.
> Question 1:
>       I used the relative path (e.g. ../../../applications/executable) which 
> still does not work. I have managed to put the script in the same folder 
> which works though against the initial design.

Yeah - the LWR would not really handle relative paths like this and I
would be hesitant to add support. It is not a pattern for tools I have
seen others use and I would like to move the LWR and Galaxy toward
more isolation and structure for both jobs and tools for security
reasons - and this would be moving in the other direction.

I think ideally you would make the perl scripts executable and put
them on both Galaxy and the LWR's path - this way you structure things
the way you want - I think it would even give you more flexibility on
the Galaxy side - no need to impose constraints on how the tools are
laid out relative to the executable.

> Question 2:
>       I pass the $output to the perl script within which a file is generated 
> with the filename contained in $output. The Galaxy server can recognize the 
> output. However when I generate another file with the name 
> "primary_$job.id_additional_visible_txt" in the same folder as the $output. 
> In theory, it should also be recognized by Galaxy as being in the 
> job_working_directory. But it is not. Could you reckon the reason?

In Galaxy or the LWR? Either way I suspect the problem is the same -
$output is going to point a path not in the job working directory. In
Galaxy this will be a file like say
database/files/000/dataset_00101.dat whereas the job working directory
will be something like database/job_working_directory/007. (Unless you
have set outputs_to_working_directory in your Galaxy universe_wsgi.ini
file - let me know if this is the case - this option is possibly not
compatible with the LWR).

Likewise in the LWR $output will point to
<staging_directory>/<job_id>/outputs/dataset_00101.dat and the working
directory will be something like <staging_directory>/job_id/working.
These files need to be placed in the working directory (in Perl do
your open command with no path - just a simple filename).

> Question 3:
>       Your explanation is very clear. Thank you again for this. The initial 
> purpose of asking this question is related to question 2: I was wondering 
> which "correct" folder I should use to put the additional optional outputs. 
> Now as you suggested, I should not worry about it at all.
> Question 4:
>      I have managed to convert a RAW file (900M) to mzML (2.3G). But if there 
> is a limit on the file size, the spectral file maybe need to be split into 
> multiple mgf files.

The LWR was recently enhanced to support LWR initiated transfers to
and from Galaxy instead of Galaxy initiated transfer to and from the
LWR - the upshot of this is there should be not file size limit (if in
fact there is with the LWR - I am still not sure there are). I haven't
tested that configuration of the LWR on Windows - so if you do
encounter limits and want to do some testing for me let me know :).

Thanks for your interest in Galaxy,

> Best regards!
> Jun
> -----Original Message-----
> From: John Chilton [mailto:jmchil...@gmail.com]
> Sent: Monday, March 3, 2014 2:27 PM
> To: Jun Fan
> Cc: galaxy-dev@lists.bx.psu.edu
> Subject: Re: [galaxy-dev] LWR questions
> Hey Jun,
> Thanks for your continued interest in the LWR. Responses inline.
> On Sun, Mar 2, 2014 at 7:00 PM, Jun Fan <j....@qmul.ac.uk> wrote:
>> Hi all,
>>       LWR sounds very attractive and promising. However how to program
>> on galaxy side to utilize LWR server is not thoroughly documented.
> Critique noted, I have tried to make it seamless but when you move a job from 
> Linux to Windows there are bound to be severe limitations and I should 
> document these more clearly and provide a guide for writing Windows 
> compatible tools 
> (https://bitbucket.org/jmchilton/lwr/issue/9/provide-documentation-for-writing-windows).
>> I have
>> several questions to ask:
>> 1.       I separate the tool wrappers (.xml) and the applications (.pl, .py
>> etc) in two different folders. In the traditional way it works fine.
>> When I use the legacy way of using LWR (adding galaxy:tool_runners
>> section in the universal_wsgi.ini file), the error message says that
>> the application cannot be found. Does the executable have to be in the
>> same folder with the wrapper?
> Do you hard code absolute paths to your scripts in the tool XML files then? 
> The LWR does not support that. If you place a single script in the same 
> folder as the XML (a simple light-weight wrapper as is fairly
> common) - that is supported - otherwise you will need to place all the 
> required dependencies on the LWR's path on the remote Windows server.
> This could get awkward if used with interpreter tag for instance - if you 
> send me examples of your tools I can provide some guidance.
> I guess in general my advice for Windows tools would be if you can use a 
> simple one-file wrapper script placed next the tool you are fine - just make 
> sure the underlying application is loaded from the path instead of via 
> absolute paths - and then make sure all dependencies for the wrapper script 
> are installed on the remote server and the underlying application is on the 
> LWR's path.
> (If your remote server is a *nix server there is new support for using 
> tool_dependency_dir style Galaxy dependencies that resolve on the remote 
> server - but this will not work with Windows (enhancements to provide this 
> would be welcomed though) ).
>> 2.       In the thread
>> http://dev.list.galaxyproject.org/Managing-Data-Locality-td4662438.htm
>> l#a4662570 John only mentioned $GALAXY_ROOT/databases/files, does this
>> mean that the dynamic multiple output is not supported as it uses
>> $GALAXY_ROOT/databases/tmp folder?
> https://bitbucket.org/galaxy/galaxy-central/src/eafb4208db665dfeed7d8f3e6f0d2b61ce2f4fc8/universe_wsgi.ini.sample?at=default#cl-213
> My understanding is that writing those files to the databases/tmp is somewhat 
> deprecated - ideally these should be written to the working directory of the 
> job. The newest version of the LWR *should* attempt to transfer various kinds 
> of working directory files back to the Galaxy server after the job is 
> complete - these files *should* include such variable output files. This 
> hasn't been tested on Windows though
> - if there are problems please let me know and I can try to address them -
> https://bitbucket.org/jmchilton/lwr/src/51f097d3d517064159bd959c61cee71f7894ea92/lwr/lwr_client/staging/down.py?at=default#cl-17
>> 3.       What types of files are supposed to be stored under tools directory
>> (tool_files), working directory (working), inputs (inputs), configs
>> directory (configs), all of which are subfolders of
>> ${staging_directory}/${job_identifier}?
> In theory, you shouldn't need to be worried about these implementations 
> details as an application programmer, it should feel just like Galaxy (again 
> with limitations).  But for what it is worth - the LWR client will transfer 
> tool wrappers it infers are necessary to run the job to `tool_files`, the 
> contents of any `configfile` templates defined for the job to `configs` 
> (https://wiki.galaxyproject.org/Admin/Tools/ToolConfigSyntax#A.3Cconfigfile.3E_tag_set),
> and any inputs it infers are nessecary to `inputs`.
> `working` is will be the working directory for the job - so any files your 
> script or application writes to relative directories at runtime will be 
> relative to this directory and finally all output paths (say you have $output 
> defined in your Cheetah) will be rewritten so they target the outputs 
> directory.
>> 4.       Is there file size limit on the LWR server?
> It is quite possible there is. Before I had left MSI we had transferred 
> multi-gigabyte files to and from the LWR - but I don't know if we had ever 
> actually transferred a file over 4Gb (an obvious potential limit). I get the 
> sense that the underlying paste server is kind of taxed by these. Like Galaxy 
> itself, it could presumably be hosted in a more robust web server. I have 
> done some looking at chaussette for instance which sees to have support for 
> Windows.
> Hope this helps somewhat,
> -John
>> Best regards!
>> Jun
>> ___________________________________________________________
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this and other
>> Galaxy lists, please use the interface at:
>>   http://lists.bx.psu.edu/
>> To search Galaxy mailing lists use the unified search at:
>>   http://galaxyproject.org/search/mailinglists/

Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

To search Galaxy mailing lists use the unified search at:

Reply via email to