Re: [galaxy-dev] ImportError and Project life cycle

[Matthias Bernt]

Hi Martin,

First of all: Since I have upgraded from 16.10 to 17.10 the problem 
disappeared :)



I did uncheck the checkbox for TS dependencies during the tool
installation. I was wondering that galaxy_sequence_utils is not shown as
installed in "Manage installed tools". It seems to make no difference if
I install it manually or not.



I recommend leaving the box checked so Galaxy will use Conda to install
the dependency.


OK.


Is the tool still missing imports after you install it this way?


No, its shown in the list of installed tools in version 1 of the tool. 
Version 2 seems to solely rely on conda.



Currently 'Managed installed repositories' does _not_ show
Conda-installed dependencies - the UI is lacking behind the system's
capabilities - our apologies.


Thanks a lot for the explanations.

Cheers,
Matthias



smime.p7s
Description: S/MIME Cryptographic Signature
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] ImportError and Project life cycle

[Martin Čech]

Hi Matthias,

> I did uncheck the checkbox for TS dependencies during the tool
> installation. I was wondering that galaxy_sequence_utils is not shown as
> installed in "Manage installed tools". It seems to make no difference if
> I install it manually or not.

I recommend leaving the box checked so Galaxy will use Conda to install the
dependency.
Is the tool still missing imports after you install it this way?

Currently 'Managed installed repositories' does _not_ show Conda-installed
dependencies - the UI is lacking behind the system's capabilities - our
apologies.


Martin




On Mon, Feb 13, 2017 at 4:17 AM Matthias Bernt  wrote:

> Dear Martin,
>
> Thanks for your input. Before answering the raised questions I would
> like to add some more information that I just found:
>
> I've noticed something odd (to me) which is maybe the source of the
> problem. My galaxy installation has galaxy_utils installed twice:
>
> - lib/galaxy_utils/sequence/fastq.py
>
> -
>
> database/dependencies/galaxy_sequence_utils/1.0.0/devteam/package_galaxy_utils_1_0/8882f14715b5/lib/python/galaxy_sequence_utils-1.0.0-py2.7.egg/galaxy_utils/sequence/fastq.py
>
> The two files have different content. Only the former imports the six
> module and the former is used by fastq_trimmer_by_quality (as you see in
> the python stack trace).
>
> The former is already installed in a fresh clone of galaxy (I have
> checked for 16.07 and 16.10).
>
> The Dependency Resolver Details show that fastq_trimmer_by_quality
> requires galaxy_sequence_utils 1.0.1, but I can not find this version in
> my galaxy installation after installing fastq_trimmer_by_quality (find .
> -iname "galaxy_sequence_utils").
>
> If I install galaxy_sequence_utils 1.0.1 manually (only available from
> TS) the 1.0.0 version in the database/dependencies/galaxy_sequence_utils
> is replaced by 1.0.1 (which seems also odd to me - shouldn't both
> versions be available?).
>
> Anyway, it makes no difference to the python import error if this
> version is installed or not, since still the version in lib/ is used.
>
> > what version of Galaxy are you running?
>
> I am using galaxy 16.10.
>
> > The tool you have installed seems to support only new Galaxies that have
> > Conda dependency resolution enabled. We are currently in the middle of
> > the transition of the Galaxy's ecosystem to this new dependency manager
> > so there are rough edges here and there - sorry about that. You can read
> > details about Conda and dependency resolution
> > here: https://docs.galaxyproject.org/en/master/admin/conda_faq.html
> >
> > Since Galaxy 16.07 you can switch the `conda_auto_init` in Galaxy's
> > config to True and after restart Galaxy will install Conda itself.
>
> I'm using the default dependency_resolvers_conf.xml (ie I don't have the
> file). Furthermore I have set:
>
> conda_auto_install = True
> conda_auto_init = True
> conda_copy_dependencies = True
>
> > Therefore I recommend you (with Galaxy newer than 16.07) to uninstall
> > the tool, install Conda (with conda_auto_init) and install the tool
> > again. That will install the needed dependency for the tool. If you have
> > older Galaxy please update it.
>
> I did uncheck the checkbox for TS dependencies during the tool
> installation. I was wondering that galaxy_sequence_utils is not shown as
> installed in "Manage installed tools". It seems to make no difference if
> I install it manually or not.
>
> Cheers,
> Matthias
>
> > Please let us know about any questions you have and thanks for using
> Galaxy.
> >
> > Martin
> >
> > On Fri, Feb 10, 2017 at 10:38 AM Peter Cock  > > wrote:
> >
> > On Fri, Feb 10, 2017 at 3:26 PM, Matthias Bernt  > > wrote:
> > > Dear galaxy-dev list,
> > >
> > > I'm currently trying to introduce myself to the galaxy server,
> i.e.,
> > > learning how to use and administer a galaxy server.
> > >
> > > I have installed galaxy following this excellent tutorial:
> > >
> > > https://github.com/martenson/dagobah-training
> > >
> > > Then I tried the tool "FASTQ Quality Trimmer by sliding window"
> > which is
> > > mentioned in NGS101-6 tutorial. I installed the tool from tool
> > shed, but I
> > > get a python error (the Admin panel shows that all dependencies are
> > > installed):
> > >
> > > Traceback (most recent call last):
> > >   File
> > >
> > "/home/berntm/shed_tools/
> toolshed.g2.bx.psu.edu/repos/devteam/fastq_trimmer_by_quality/25c24379693a/fastq_trimmer_by_quality/fastq_trimmer_by_quality.py
> > <
> http://toolshed.g2.bx.psu.edu/repos/devteam/fastq_trimmer_by_quality/25c24379693a/fastq_trimmer_by_quality/fastq_trimmer_by_quality.py
> >",
> > > line 3, in 
> > > from galaxy_utils.sequence.fastq import fastqReader,
> fastqWriter
> > >   File "/home/berntm/galaxy/lib/galaxy_utils/sequence/fastq.py",
> > line 7, in
> > > 
> > > f

Re: [galaxy-dev] ImportError and Project life cycle

[Peter Cock]

On Mon, Feb 13, 2017 at 9:17 AM, Matthias Bernt  wrote:
> Dear Martin,
>
> Thanks for your input. Before answering the raised questions I would like to
> add some more information that I just found:
>
> I've noticed something odd (to me) which is maybe the source of the problem.
> My galaxy installation has galaxy_utils installed twice:
>
> - lib/galaxy_utils/sequence/fastq.py
>
> -
> database/dependencies/galaxy_sequence_utils/1.0.0/devteam/package_galaxy_utils_1_0/8882f14715b5/lib/python/galaxy_sequence_utils-1.0.0-py2.7.egg/galaxy_utils/sequence/fastq.py
>

Something like that is to be expected right now as this is in
transition, see also:

http://dev.list.galaxyproject.org/Tools-using-Galaxy-s-Python-library-td4670472.html

In the next release of Galaxy, there will not be a copy of the code at
lib/galaxy_utils/sequence/fastq.py - see this pull request for the change
https://github.com/galaxyproject/galaxy/pull/3551

Originally tools were allowed to access Galaxy's internal Python code,
here lib/galaxy_utils/sequence/fastq.py - but that is changing. In the
next Galaxy release tools should explicitly depend on the new
packaged version of Galaxy's sequence_utils - that's where the
second copy you've seen comes from:

https://github.com/galaxyproject/sequence_utils

This does now import six as part of work to make sure it can be
used on Python 2 and Python 3:

https://github.com/galaxyproject/sequence_utils/blob/master/galaxy_utils/sequence/sequence.py

Galaxy itself will depend on the now separate sequence_utils
package.

--

As to what is going wrong in your situation, I'm hoping one of the
Galaxy developers working on this can help.

Peter
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] ImportError and Project life cycle

[Matthias Bernt]

Dear Martin,

Thanks for your input. Before answering the raised questions I would 
like to add some more information that I just found:


I've noticed something odd (to me) which is maybe the source of the 
problem. My galaxy installation has galaxy_utils installed twice:


- lib/galaxy_utils/sequence/fastq.py

- 
database/dependencies/galaxy_sequence_utils/1.0.0/devteam/package_galaxy_utils_1_0/8882f14715b5/lib/python/galaxy_sequence_utils-1.0.0-py2.7.egg/galaxy_utils/sequence/fastq.py


The two files have different content. Only the former imports the six 
module and the former is used by fastq_trimmer_by_quality (as you see in 
the python stack trace).


The former is already installed in a fresh clone of galaxy (I have 
checked for 16.07 and 16.10).


The Dependency Resolver Details show that fastq_trimmer_by_quality 
requires galaxy_sequence_utils 1.0.1, but I can not find this version in 
my galaxy installation after installing fastq_trimmer_by_quality (find . 
-iname "galaxy_sequence_utils").


If I install galaxy_sequence_utils 1.0.1 manually (only available from 
TS) the 1.0.0 version in the database/dependencies/galaxy_sequence_utils 
is replaced by 1.0.1 (which seems also odd to me - shouldn't both 
versions be available?).


Anyway, it makes no difference to the python import error if this 
version is installed or not, since still the version in lib/ is used.



what version of Galaxy are you running?


I am using galaxy 16.10.


The tool you have installed seems to support only new Galaxies that have
Conda dependency resolution enabled. We are currently in the middle of
the transition of the Galaxy's ecosystem to this new dependency manager
so there are rough edges here and there - sorry about that. You can read
details about Conda and dependency resolution
here: https://docs.galaxyproject.org/en/master/admin/conda_faq.html

Since Galaxy 16.07 you can switch the `conda_auto_init` in Galaxy's
config to True and after restart Galaxy will install Conda itself.


I'm using the default dependency_resolvers_conf.xml (ie I don't have the 
file). Furthermore I have set:


conda_auto_install = True
conda_auto_init = True
conda_copy_dependencies = True


Therefore I recommend you (with Galaxy newer than 16.07) to uninstall
the tool, install Conda (with conda_auto_init) and install the tool
again. That will install the needed dependency for the tool. If you have
older Galaxy please update it.


I did uncheck the checkbox for TS dependencies during the tool 
installation. I was wondering that galaxy_sequence_utils is not shown as 
installed in "Manage installed tools". It seems to make no difference if 
I install it manually or not.


Cheers,
Matthias


Please let us know about any questions you have and thanks for using Galaxy.

Martin

On Fri, Feb 10, 2017 at 10:38 AM Peter Cock mailto:p.j.a.c...@googlemail.com>> wrote:

On Fri, Feb 10, 2017 at 3:26 PM, Matthias Bernt mailto:m.be...@ufz.de>> wrote:
> Dear galaxy-dev list,
>
> I'm currently trying to introduce myself to the galaxy server, i.e.,
> learning how to use and administer a galaxy server.
>
> I have installed galaxy following this excellent tutorial:
>
> https://github.com/martenson/dagobah-training
>
> Then I tried the tool "FASTQ Quality Trimmer by sliding window"
which is
> mentioned in NGS101-6 tutorial. I installed the tool from tool
shed, but I
> get a python error (the Admin panel shows that all dependencies are
> installed):
>
> Traceback (most recent call last):
>   File
>

"/home/berntm/shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/fastq_trimmer_by_quality/25c24379693a/fastq_trimmer_by_quality/fastq_trimmer_by_quality.py

",
> line 3, in 
> from galaxy_utils.sequence.fastq import fastqReader, fastqWriter
>   File "/home/berntm/galaxy/lib/galaxy_utils/sequence/fastq.py",
line 7, in
> 
> from six import Iterator, string_types
> ImportError: No module named six
>
> Do you have an idea what could be wrong? I guess that this is to
little
> information, but I did not know which kind of additional
information would
> be helpful.

This appears to be a missing dependency on the Python module "six"
which is used to help write code which will work under both Python 2
and Python 3.

This is likely a recent regression - hopefully one of the Galaxy team
can comment on this, likely the Tool Shed definition for this tool needs
a minor tweak (and then you can apply the update via the Galaxy
Admin controls within your browser).

> Another question: At my institution quite a lot of NGS projects
are carried
> out. After a project is finished all methods and data are stored
to a long
> term read only tape archive. For the futu

Re: [galaxy-dev] ImportError and Project life cycle

[Björn Grüning]

Hi Matthias,

to your life cycle question, do you have more information about your
filesystem(s) and what you have specifically in mind? Everything is
possible I guess, from a hack to a advanced solution like iRODS.

My feeling is that this should be implemented on the FS level and not on
the Galaxy level.

Ciao,
Bjoern


> Dear galaxy-dev list,
> 
> I'm currently trying to introduce myself to the galaxy server, i.e.,
> learning how to use and administer a galaxy server.
> 
> I have installed galaxy following this excellent tutorial:
> 
> https://github.com/martenson/dagobah-training
> 
> Then I tried the tool "FASTQ Quality Trimmer by sliding window" which is
> mentioned in NGS101-6 tutorial. I installed the tool from tool shed, but
> I get a python error (the Admin panel shows that all dependencies are
> installed):
> 
> Traceback (most recent call last):
>   File
> "/home/berntm/shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/fastq_trimmer_by_quality/25c24379693a/fastq_trimmer_by_quality/fastq_trimmer_by_quality.py",
> line 3, in 
> from galaxy_utils.sequence.fastq import fastqReader, fastqWriter
>   File "/home/berntm/galaxy/lib/galaxy_utils/sequence/fastq.py", line 7,
> in 
> from six import Iterator, string_types
> ImportError: No module named six
> 
> Do you have an idea what could be wrong? I guess that this is to little
> information, but I did not know which kind of additional information
> would be helpful.
> 
> Another question: At my institution quite a lot of NGS projects are
> carried out. After a project is finished all methods and data are stored
> to a long term read only tape archive. For the future the idea is to use
> galaxy. Which options are there to implement such a project life cycle
> with galaxy?
> 
> A last general question: would it be better to send a separate mail to
> the list when I have multiple questions?
> 
> Best regards,
> Matthias Bernt
> 
> 
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>  https://lists.galaxyproject.org/
> 
> To search Galaxy mailing lists use the unified search at:
>  http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] ImportError and Project life cycle

[Martin Čech]

Hi Matthias,

what version of Galaxy are you running?

The tool you have installed seems to support only new Galaxies that have
Conda dependency resolution enabled. We are currently in the middle of the
transition of the Galaxy's ecosystem to this new dependency manager so
there are rough edges here and there - sorry about that. You can read
details about Conda and dependency resolution here:
https://docs.galaxyproject.org/en/master/admin/conda_faq.html

Since Galaxy 16.07 you can switch the `conda_auto_init` in Galaxy's config
to True and after restart Galaxy will install Conda itself.

Therefore I recommend you (with Galaxy newer than 16.07) to uninstall the
tool, install Conda (with conda_auto_init) and install the tool again. That
will install the needed dependency for the tool. If you have older Galaxy
please update it.

Please let us know about any questions you have and thanks for using Galaxy.

Martin

On Fri, Feb 10, 2017 at 10:38 AM Peter Cock 
wrote:

> On Fri, Feb 10, 2017 at 3:26 PM, Matthias Bernt  wrote:
> > Dear galaxy-dev list,
> >
> > I'm currently trying to introduce myself to the galaxy server, i.e.,
> > learning how to use and administer a galaxy server.
> >
> > I have installed galaxy following this excellent tutorial:
> >
> > https://github.com/martenson/dagobah-training
> >
> > Then I tried the tool "FASTQ Quality Trimmer by sliding window" which is
> > mentioned in NGS101-6 tutorial. I installed the tool from tool shed, but
> I
> > get a python error (the Admin panel shows that all dependencies are
> > installed):
> >
> > Traceback (most recent call last):
> >   File
> > "/home/berntm/shed_tools/
> toolshed.g2.bx.psu.edu/repos/devteam/fastq_trimmer_by_quality/25c24379693a/fastq_trimmer_by_quality/fastq_trimmer_by_quality.py
> ",
> > line 3, in 
> > from galaxy_utils.sequence.fastq import fastqReader, fastqWriter
> >   File "/home/berntm/galaxy/lib/galaxy_utils/sequence/fastq.py", line 7,
> in
> > 
> > from six import Iterator, string_types
> > ImportError: No module named six
> >
> > Do you have an idea what could be wrong? I guess that this is to little
> > information, but I did not know which kind of additional information
> would
> > be helpful.
>
> This appears to be a missing dependency on the Python module "six"
> which is used to help write code which will work under both Python 2
> and Python 3.
>
> This is likely a recent regression - hopefully one of the Galaxy team
> can comment on this, likely the Tool Shed definition for this tool needs
> a minor tweak (and then you can apply the update via the Galaxy
> Admin controls within your browser).
>
> > Another question: At my institution quite a lot of NGS projects are
> carried
> > out. After a project is finished all methods and data are stored to a
> long
> > term read only tape archive. For the future the idea is to use galaxy.
> Which
> > options are there to implement such a project life cycle with galaxy?
>
> Not that I personally am aware of, no. There are optional disk quotas to
> encourage users to remove (delete) old data.
>
> > A last general question: would it be better to send a separate mail to
> the
> > list when I have multiple questions?
>
> Yes please, with useful email subjects would be best.
>
> Peter
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] ImportError and Project life cycle

[Peter Cock]

On Fri, Feb 10, 2017 at 3:26 PM, Matthias Bernt  wrote:
> Dear galaxy-dev list,
>
> I'm currently trying to introduce myself to the galaxy server, i.e.,
> learning how to use and administer a galaxy server.
>
> I have installed galaxy following this excellent tutorial:
>
> https://github.com/martenson/dagobah-training
>
> Then I tried the tool "FASTQ Quality Trimmer by sliding window" which is
> mentioned in NGS101-6 tutorial. I installed the tool from tool shed, but I
> get a python error (the Admin panel shows that all dependencies are
> installed):
>
> Traceback (most recent call last):
>   File
> "/home/berntm/shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/fastq_trimmer_by_quality/25c24379693a/fastq_trimmer_by_quality/fastq_trimmer_by_quality.py",
> line 3, in 
> from galaxy_utils.sequence.fastq import fastqReader, fastqWriter
>   File "/home/berntm/galaxy/lib/galaxy_utils/sequence/fastq.py", line 7, in
> 
> from six import Iterator, string_types
> ImportError: No module named six
>
> Do you have an idea what could be wrong? I guess that this is to little
> information, but I did not know which kind of additional information would
> be helpful.

This appears to be a missing dependency on the Python module "six"
which is used to help write code which will work under both Python 2
and Python 3.

This is likely a recent regression - hopefully one of the Galaxy team
can comment on this, likely the Tool Shed definition for this tool needs
a minor tweak (and then you can apply the update via the Galaxy
Admin controls within your browser).

> Another question: At my institution quite a lot of NGS projects are carried
> out. After a project is finished all methods and data are stored to a long
> term read only tape archive. For the future the idea is to use galaxy. Which
> options are there to implement such a project life cycle with galaxy?

Not that I personally am aware of, no. There are optional disk quotas to
encourage users to remove (delete) old data.

> A last general question: would it be better to send a separate mail to the
> list when I have multiple questions?

Yes please, with useful email subjects would be best.

Peter
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] ImportError and Project life cycle

[Matthias Bernt]

Dear galaxy-dev list,

I'm currently trying to introduce myself to the galaxy server, i.e., 
learning how to use and administer a galaxy server.


I have installed galaxy following this excellent tutorial:

https://github.com/martenson/dagobah-training

Then I tried the tool "FASTQ Quality Trimmer by sliding window" which is 
mentioned in NGS101-6 tutorial. I installed the tool from tool shed, but 
I get a python error (the Admin panel shows that all dependencies are 
installed):


Traceback (most recent call last):
  File 
"/home/berntm/shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/fastq_trimmer_by_quality/25c24379693a/fastq_trimmer_by_quality/fastq_trimmer_by_quality.py", 
line 3, in 

from galaxy_utils.sequence.fastq import fastqReader, fastqWriter
  File "/home/berntm/galaxy/lib/galaxy_utils/sequence/fastq.py", line 
7, in 

from six import Iterator, string_types
ImportError: No module named six

Do you have an idea what could be wrong? I guess that this is to little 
information, but I did not know which kind of additional information 
would be helpful.


Another question: At my institution quite a lot of NGS projects are 
carried out. After a project is finished all methods and data are stored 
to a long term read only tape archive. For the future the idea is to use 
galaxy. Which options are there to implement such a project life cycle 
with galaxy?


A last general question: would it be better to send a separate mail to 
the list when I have multiple questions?


Best regards,
Matthias Bernt


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/