Hey Jun,

Thanks for your continued interest in the LWR. Responses inline.

On Sun, Mar 2, 2014 at 7:00 PM, Jun Fan <j....@qmul.ac.uk> wrote:
> Hi all,
>
>
>
>       LWR sounds very attractive and promising. However how to program on
> galaxy side to utilize LWR server is not thoroughly documented.

Critique noted, I have tried to make it seamless but when you move a
job from Linux to Windows there are bound to be severe limitations and
I should document these more clearly and provide a guide for writing
Windows compatible tools
(https://bitbucket.org/jmchilton/lwr/issue/9/provide-documentation-for-writing-windows).

> I have
> several questions to ask:
>
> 1.       I separate the tool wrappers (.xml) and the applications (.pl, .py
> etc) in two different folders. In the traditional way it works fine. When I
> use the legacy way of using LWR (adding galaxy:tool_runners section in the
> universal_wsgi.ini file), the error message says that the application cannot
> be found. Does the executable have to be in the same folder with the
> wrapper?

Do you hard code absolute paths to your scripts in the tool XML files
then? The LWR does not support that. If you place a single script in
the same folder as the XML (a simple light-weight wrapper as is fairly
common) - that is supported - otherwise you will need to place all the
required dependencies on the LWR's path on the remote Windows server.
This could get awkward if used with interpreter tag for instance - if
you send me examples of your tools I can provide some guidance.

I guess in general my advice for Windows tools would be if you can use
a simple one-file wrapper script placed next the tool you are fine -
just make sure the underlying application is loaded from the path
instead of via absolute paths - and then make sure all dependencies
for the wrapper script are installed on the remote server and the
underlying application is on the LWR's path.

(If your remote server is a *nix server there is new support for using
tool_dependency_dir style Galaxy dependencies that resolve on the
remote server - but this will not work with Windows (enhancements to
provide this would be welcomed though) ).

>
> 2.       In the thread
> http://dev.list.galaxyproject.org/Managing-Data-Locality-td4662438.html#a4662570
> John only mentioned $GALAXY_ROOT/databases/files, does this mean that the
> dynamic multiple output is not supported as it uses
> $GALAXY_ROOT/databases/tmp folder?

https://bitbucket.org/galaxy/galaxy-central/src/eafb4208db665dfeed7d8f3e6f0d2b61ce2f4fc8/universe_wsgi.ini.sample?at=default#cl-213

My understanding is that writing those files to the databases/tmp is
somewhat deprecated - ideally these should be written to the working
directory of the job. The newest version of the LWR *should* attempt
to transfer various kinds of working directory files back to the
Galaxy server after the job is complete - these files *should* include
such variable output files. This hasn't been tested on Windows though
- if there are problems please let me know and I can try to address
them -

https://bitbucket.org/jmchilton/lwr/src/51f097d3d517064159bd959c61cee71f7894ea92/lwr/lwr_client/staging/down.py?at=default#cl-17

>
> 3.       What types of files are supposed to be stored under tools directory
> (tool_files), working directory (working), inputs (inputs), configs
> directory (configs), all of which are subfolders of
> ${staging_directory}/${job_identifier}?

In theory, you shouldn't need to be worried about these
implementations details as an application programmer, it should feel
just like Galaxy (again with limitations).  But for what it is worth -
the LWR client will transfer tool wrappers it infers are necessary to
run the job to `tool_files`, the contents of any `configfile`
templates defined for the job to `configs`
(https://wiki.galaxyproject.org/Admin/Tools/ToolConfigSyntax#A.3Cconfigfile.3E_tag_set),
and any inputs it infers are nessecary to `inputs`.

`working` is will be the working directory for the job - so any files
your script or application writes to relative directories at runtime
will be relative to this directory and finally all output paths (say
you have $output defined in your Cheetah) will be rewritten so they
target the outputs directory.

>
> 4.       Is there file size limit on the LWR server?

It is quite possible there is. Before I had left MSI we had
transferred multi-gigabyte files to and from the LWR - but I don't
know if we had ever actually transferred a file over 4Gb (an obvious
potential limit). I get the sense that the underlying paste server is
kind of taxed by these. Like Galaxy itself, it could presumably be
hosted in a more robust web server. I have done some looking at
chaussette for instance which sees to have support for Windows.

Hope this helps somewhat,
-John

>
>
>
> Best regards!
>
> Jun
>
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Reply via email to