On Thu, Mar 6, 2014 at 8:04 PM, Jun Fan <j....@qmul.ac.uk> wrote:
> Hi John,
>
>      Thanks for your reply.
> Question 1:
>       I used the relative path (e.g. ../../../applications/executable) which 
> still does not work. I have managed to put the script in the same folder 
> which works though against the initial design.

Yeah - the LWR would not really handle relative paths like this and I
would be hesitant to add support. It is not a pattern for tools I have
seen others use and I would like to move the LWR and Galaxy toward
more isolation and structure for both jobs and tools for security
reasons - and this would be moving in the other direction.

I think ideally you would make the perl scripts executable and put
them on both Galaxy and the LWR's path - this way you structure things
the way you want - I think it would even give you more flexibility on
the Galaxy side - no need to impose constraints on how the tools are
laid out relative to the executable.

> Question 2:
>       I pass the $output to the perl script within which a file is generated 
> with the filename contained in $output. The Galaxy server can recognize the 
> output. However when I generate another file with the name 
> "primary_$job.id_additional_visible_txt" in the same folder as the $output. 
> In theory, it should also be recognized by Galaxy as being in the 
> job_working_directory. But it is not. Could you reckon the reason?

In Galaxy or the LWR? Either way I suspect the problem is the same -
$output is going to point a path not in the job working directory. In
Galaxy this will be a file like say
database/files/000/dataset_00101.dat whereas the job working directory
will be something like database/job_working_directory/007. (Unless you
have set outputs_to_working_directory in your Galaxy universe_wsgi.ini
file - let me know if this is the case - this option is possibly not
compatible with the LWR).

Likewise in the LWR $output will point to
<staging_directory>/<job_id>/outputs/dataset_00101.dat and the working
directory will be something like <staging_directory>/job_id/working.
These files need to be placed in the working directory (in Perl do
your open command with no path - just a simple filename).

> Question 3:
>       Your explanation is very clear. Thank you again for this. The initial 
> purpose of asking this question is related to question 2: I was wondering 
> which "correct" folder I should use to put the additional optional outputs. 
> Now as you suggested, I should not worry about it at all.
> Question 4:
>      I have managed to convert a RAW file (900M) to mzML (2.3G). But if there 
> is a limit on the file size, the spectral file maybe need to be split into 
> multiple mgf files.

The LWR was recently enhanced to support LWR initiated transfers to
and from Galaxy instead of Galaxy initiated transfer to and from the
LWR - the upshot of this is there should be not file size limit (if in
fact there is with the LWR - I am still not sure there are). I haven't
tested that configuration of the LWR on Windows - so if you do
encounter limits and want to do some testing for me let me know :).

Thanks for your interest in Galaxy,
-John

>
> Best regards!
> Jun
>
> -----Original Message-----
> From: John Chilton [mailto:jmchil...@gmail.com]
> Sent: Monday, March 3, 2014 2:27 PM
> To: Jun Fan
> Cc: galaxy-dev@lists.bx.psu.edu
> Subject: Re: [galaxy-dev] LWR questions
>
> Hey Jun,
>
> Thanks for your continued interest in the LWR. Responses inline.
>
> On Sun, Mar 2, 2014 at 7:00 PM, Jun Fan <j....@qmul.ac.uk> wrote:
>> Hi all,
>>
>>
>>
>>       LWR sounds very attractive and promising. However how to program
>> on galaxy side to utilize LWR server is not thoroughly documented.
>
> Critique noted, I have tried to make it seamless but when you move a job from 
> Linux to Windows there are bound to be severe limitations and I should 
> document these more clearly and provide a guide for writing Windows 
> compatible tools 
> (https://bitbucket.org/jmchilton/lwr/issue/9/provide-documentation-for-writing-windows).
>
>> I have
>> several questions to ask:
>>
>> 1.       I separate the tool wrappers (.xml) and the applications (.pl, .py
>> etc) in two different folders. In the traditional way it works fine.
>> When I use the legacy way of using LWR (adding galaxy:tool_runners
>> section in the universal_wsgi.ini file), the error message says that
>> the application cannot be found. Does the executable have to be in the
>> same folder with the wrapper?
>
> Do you hard code absolute paths to your scripts in the tool XML files then? 
> The LWR does not support that. If you place a single script in the same 
> folder as the XML (a simple light-weight wrapper as is fairly
> common) - that is supported - otherwise you will need to place all the 
> required dependencies on the LWR's path on the remote Windows server.
> This could get awkward if used with interpreter tag for instance - if you 
> send me examples of your tools I can provide some guidance.
>
> I guess in general my advice for Windows tools would be if you can use a 
> simple one-file wrapper script placed next the tool you are fine - just make 
> sure the underlying application is loaded from the path instead of via 
> absolute paths - and then make sure all dependencies for the wrapper script 
> are installed on the remote server and the underlying application is on the 
> LWR's path.
>
> (If your remote server is a *nix server there is new support for using 
> tool_dependency_dir style Galaxy dependencies that resolve on the remote 
> server - but this will not work with Windows (enhancements to provide this 
> would be welcomed though) ).
>
>>
>> 2.       In the thread
>> http://dev.list.galaxyproject.org/Managing-Data-Locality-td4662438.htm
>> l#a4662570 John only mentioned $GALAXY_ROOT/databases/files, does this
>> mean that the dynamic multiple output is not supported as it uses
>> $GALAXY_ROOT/databases/tmp folder?
>
> https://bitbucket.org/galaxy/galaxy-central/src/eafb4208db665dfeed7d8f3e6f0d2b61ce2f4fc8/universe_wsgi.ini.sample?at=default#cl-213
>
> My understanding is that writing those files to the databases/tmp is somewhat 
> deprecated - ideally these should be written to the working directory of the 
> job. The newest version of the LWR *should* attempt to transfer various kinds 
> of working directory files back to the Galaxy server after the job is 
> complete - these files *should* include such variable output files. This 
> hasn't been tested on Windows though
> - if there are problems please let me know and I can try to address them -
>
> https://bitbucket.org/jmchilton/lwr/src/51f097d3d517064159bd959c61cee71f7894ea92/lwr/lwr_client/staging/down.py?at=default#cl-17
>
>>
>> 3.       What types of files are supposed to be stored under tools directory
>> (tool_files), working directory (working), inputs (inputs), configs
>> directory (configs), all of which are subfolders of
>> ${staging_directory}/${job_identifier}?
>
> In theory, you shouldn't need to be worried about these implementations 
> details as an application programmer, it should feel just like Galaxy (again 
> with limitations).  But for what it is worth - the LWR client will transfer 
> tool wrappers it infers are necessary to run the job to `tool_files`, the 
> contents of any `configfile` templates defined for the job to `configs` 
> (https://wiki.galaxyproject.org/Admin/Tools/ToolConfigSyntax#A.3Cconfigfile.3E_tag_set),
> and any inputs it infers are nessecary to `inputs`.
>
> `working` is will be the working directory for the job - so any files your 
> script or application writes to relative directories at runtime will be 
> relative to this directory and finally all output paths (say you have $output 
> defined in your Cheetah) will be rewritten so they target the outputs 
> directory.
>
>>
>> 4.       Is there file size limit on the LWR server?
>
> It is quite possible there is. Before I had left MSI we had transferred 
> multi-gigabyte files to and from the LWR - but I don't know if we had ever 
> actually transferred a file over 4Gb (an obvious potential limit). I get the 
> sense that the underlying paste server is kind of taxed by these. Like Galaxy 
> itself, it could presumably be hosted in a more robust web server. I have 
> done some looking at chaussette for instance which sees to have support for 
> Windows.
>
> Hope this helps somewhat,
> -John
>
>>
>>
>>
>> Best regards!
>>
>> Jun
>>
>>
>> ___________________________________________________________
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this and other
>> Galaxy lists, please use the interface at:
>>   http://lists.bx.psu.edu/
>>
>> To search Galaxy mailing lists use the unified search at:
>>   http://galaxyproject.org/search/mailinglists/

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Reply via email to