Hello Nate

Thanks for the follow up, and the tip off about manipulating the env variables 
in job_conf.xml for specific job destinations - I didn't know about that but I 
could imagine other situations where it might be useful!

Best wishes

Peter

--
Peter Briggs peter.bri...@manchester.ac.uk
Bioinformatics Core Facility University of Manchester
B.1083 Michael Smith Bldg Tel: (0161) 2751482

________________________________
From: Nate Coraor [n...@bx.psu.edu]
Sent: Monday, December 19, 2016 5:50 PM
To: Peter Briggs
Cc: Galaxy Dev List
Subject: Re: [galaxy-dev] Tools pick up Python modules from Galaxy core rather 
than installed tool dependencies

Hi Peter,

Thanks for posting your findings. You can also directly manipulate the 
PYTHONPATH in <env> tags in job_conf.xml <destination>s, which may be enough, 
although this would be tedious for every dependency.

--nate

On Fri, Dec 16, 2016 at 5:35 AM, Peter Briggs 
<peter.bri...@manchester.ac.uk<mailto:peter.bri...@manchester.ac.uk>> wrote:
I think I've figured this out in the end, posting my solution in case anyone 
else comes across it in future.

The issue was due to a change in the way that Python setuptools adds its 
installed modules to the search path (sys.path) via easy-install.pth files. 
Later versions of setuptools (>=25.0.0) append to rather than rewrite sys.path, 
so modules installed as eggs don't take precedence over older versions in the 
Galaxy core (for example "galaxy-ops" and "galaxy_sequence_utils").

One fix is to set the SETUPTOOLS_SYS_PATH_TECHNIQUE environment variable to 
"rewrite" and reinstall the modules (to update the module's "easy-install.pth" 
file). This should work for all recent versions of setuptools (and fixed the 
problems below for me), but might not work for future versions of setuptools - 
see http://setuptools.readthedocs.io/en/latest/history.html#v25-0-0.

Alternatively: the "galaxy_sequence_utils" package (used by 
"fastq_paired_end_joiner", maybe others?) is available in bioconda, so using 
that is probably a better long-term fix (not tested yet though). But I don't 
think "galaxy-ops" (used by "join" and other tools) is there yet.

Best wishes

Peter

--
Peter Briggs peter.bri...@manchester.ac.uk<mailto:peter.bri...@manchester.ac.uk>
Bioinformatics Core Facility University of Manchester
B.1083 Michael Smith Bldg Tel: (0161) 2751482


________________________________________
From: galaxy-dev 
[galaxy-dev-boun...@lists.galaxyproject.org<mailto:galaxy-dev-boun...@lists.galaxyproject.org>]
 on behalf of Peter Briggs 
[peter.bri...@manchester.ac.uk<mailto:peter.bri...@manchester.ac.uk>]
Sent: Wednesday, December 14, 2016 4:13 PM
To: Galaxy Dev List
Subject: [galaxy-dev] Tools pick up Python modules from Galaxy core rather than 
installed tool dependencies

Dear devs

I was wondering if anyone had any suggestions as to the cause of a
peculiar error I'm seeing when running the latest versions of the
devteam "join" and "fastq_paired_end_joiner".

The tools appear to be picking up Python modules from the Galaxy
library, rather than from the installed dependencies. The PYTHONPATH
appears to be correct (i.e. tool dependency appears before Galaxy
library path) but sys.path shows the dependency's Python egg appearing
after the Galaxy library path.

E.g. for the paired-end joiner:

PYTHONPATH=/mnt/rvmi/centaurus/galaxy/devel/tool_dependencies/galaxy_sequence_utils/1.0.1/iuc/package_galaxy_sequence_utils_1_0_1/c1ab450748ba/lib/python:/mnt/rvmi/centaurus/galaxy/devel/galaxy-dist/lib

sys.path = ['',
'/mnt/rvmi/centaurus/galaxy/devel/tool_dependencies/galaxy_sequence_utils/1.0.1/iuc/package_galaxy_sequence_utils_1_0_1/c1ab450748ba/lib/python',
'/mnt/rvmi/centaurus/galaxy/devel/galaxy-dist/lib',
'/mnt/rvmi/centaurus/galaxy/devel/tool_dependencies/galaxy_sequence_utils/1.0.1/iuc/package_galaxy_sequence_utils_1_0_1/c1ab450748ba/lib/python/galaxy_sequence_utils-1.0.1-py2.7.egg',
'/home/centaurus-galaxy/fastq_paired_end_joiner_problem/venv/lib/python27.zip',
'/home/centaurus-galaxy/fastq_paired_end_joiner_problem/venv/lib/python2.7',
'/home/centaurus-galaxy/fastq_paired_end_joiner_problem/venv/lib/python2.7/plat-linux2',
'/home/centaurus-galaxy/fastq_paired_end_joiner_problem/venv/lib/python2.7/lib-tk',
'/home/centaurus-galaxy/fastq_paired_end_joiner_problem/venv/lib/python2.7/lib-old',
'/home/centaurus-galaxy/fastq_paired_end_joiner_problem/venv/lib/python2.7/lib-dynload',
'/mnt/rvmi/centaurus/galaxy/python/2.7.10/lib/python2.7',
'/mnt/rvmi/centaurus/galaxy/python/2.7.10/lib/python2.7/plat-linux2',
'/mnt/rvmi/centaurus/galaxy/python/2.7.10/lib/python2.7/lib-tk',
'/home/centaurus-galaxy/fastq_paired_end_joiner_problem/venv/lib/python2.7/site-packages']

Error:

Traceback (most recent call last):
   File
"/mnt/rvmi/centaurus/galaxy/devel/shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/fastq_paired_end_joiner/6a7f5da7c76d/fastq_paired_end_joiner/fastq_paired_end_joiner.py<http://toolshed.g2.bx.psu.edu/repos/devteam/fastq_paired_end_joiner/6a7f5da7c76d/fastq_paired_end_joiner/fastq_paired_end_joiner.py>",
line 158, in <module>
     main()
   File
"/mnt/rvmi/centaurus/galaxy/devel/shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/fastq_paired_end_joiner/6a7f5da7c76d/fastq_paired_end_joiner/fastq_paired_end_joiner.py<http://toolshed.g2.bx.psu.edu/repos/devteam/fastq_paired_end_joiner/6a7f5da7c76d/fastq_paired_end_joiner/fastq_paired_end_joiner.py>",
line 136, in main
     joiner = fq.fastqJoiner( input1_type, paste=paste )
TypeError: __init__() got an unexpected keyword argument 'paste'

I know it's a long shot but can anyone suggest where to start looking at
where this is going wrong?

Thanks,

Peter

Ps I'm using Galaxy v16.07 and the problem occurs on our production
platform but not in a local Galaxy install on my desktop.

--
Peter Briggs peter.bri...@manchester.ac.uk<mailto:peter.bri...@manchester.ac.uk>
Bioinformatics Core Facility University of Manchester
B.1083 Michael Smith Bldg Tel: (0161) 2751482
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Reply via email to