Re: [galaxy-dev] recurring issues with revisions of packages from toolshed

2015-10-20 Thread Sarah DIEHL 
Hi,

I'm afraid it won't be easy to reproduce my problem with a clean galaxy 
install. I think it has to do with the history of the instance, a lot of back 
and forth on the tool installations and also lot's of failed installations (due 
to my environment or network issues).

Investigating the issue Marius mentioned also seems worthwhile to me, it sounds 
like it could be the underlying bug. What might also help is a way to really 
purge all traces of a tool or tool dependency (and all it's versions) from the 
disk and the database. Currently the only way to solve my numpy revision issue 
would be to delete the whole tool install database (which I learned to put in a 
different db and I already resorted to that once in the beginning). I had a 
look at the database and it looks messed up, but I don't know enough about it 
to feel comfortable changing entries. I'm happy to send a DB dump along, if 
this is helpful for anyone.

Best regards,
Sarah



Sarah Diehl
HPC System Administrator

UNIVERSITÉ DU LUXEMBOURG

LUXEMBOURG CENTRE FOR SYSTEMS BIOMEDICINE
Campus Belval | Biotech II
6, avenue du Swing
L-4371 Belvaux
T +352 46 66 44 5360
sarah.di...@uni.lu<mailto:sarah.di...@uni.lu> 
http://lcsb.uni.lu<http://lcsb.uni.lu/>
-
This message is confidential and may contain privileged information. It is 
intended for the named recipient only. If you receive it in error please notify 
me and permanently delete the original message and any copies.
-


From: Marius van den Beek 
<m.vandenb...@gmail.com<mailto:m.vandenb...@gmail.com>>
Date: Tuesday 20 October 2015 09:53
To: Björn Grüning <bjoern.gruen...@gmail.com<mailto:bjoern.gruen...@gmail.com>>
Cc: Sarah DIEHL <sarah.di...@uni.lu<mailto:sarah.di...@uni.lu>>, 
"galaxy-dev@lists.galaxyproject.org<mailto:galaxy-dev@lists.galaxyproject.org>" 
<galaxy-dev@lists.galaxyproject.org<mailto:galaxy-dev@lists.galaxyproject.org>>
Subject: Re: [galaxy-dev] recurring issues with revisions of packages from 
toolshed

Sorry to hijack the thread, but I think the issue that Sarah has described 
could be related to
https://github.com/galaxyproject/galaxy/issues/667.
(I know it's not exactly the same, but chances are we fix this, we 
automatically fix Sarah's issue.)
There is also an example to reproduce 
https://github.com/galaxyproject/galaxy/issues/667#issuecomment-138878174
And I made a bit of work on this in Pull Request: 
https://github.com/galaxyproject/galaxy/pull/707
But unfortunately at the time this was a bit over my head.
Bjoern, if you want to work on this, let me know, maybe I can give you a hand.

Hope that's helpful,
Marius

On 19 October 2015 at 23:17, Björn Grüning 
<bjoern.gruen...@gmail.com<mailto:bjoern.gruen...@gmail.com>> wrote:
Hi Sarah,

can you create a reproducible example in a Docker container for me?
Together with a friend I will have a look at this. But we need to short
reproducible example :)

Thanks,
Bjoern


> Dear all,
>
> I continuously have several issues with tool dependency packages from
> the toolshed and their revisions. It looks like the tools I install
> have requirements for different revisions of the same package_xy
> repository. What I end up with quite often is having the same package
> installed in multiple revisions (see attached
> "galaxytools_multiple_revisions.jpg" screenshot). When I then update
> the older revision, it's still listet twice and additionally the tool
> that had the older one as a dependency now says it's missing a
> repository dependency.
>
> Although this is already quite annoying and creates a mess within the
> tool list, I could live with it. However, sometimes it completely
> messes up revisions. In the second attached screenshot
> (numpy_revision_issue.jpg) you can see that this numpy repository
> thinks it is revision "300877695495", but it's actual location has
> revision "8b3a5c7061d8". This causes tools that depend on revision
> "300877695495" to think it's there, but when the dependency is
> resolved at runtime of the tool, numpy is not found (since it's
> location is in a different revision's folder) and the run fails. As
> you can also see from the screenshot, resetting the metadata doesn't
> do anything about this. Also reinstalling doesn't work, since the
> database keeps the faulty record and uses it upon reinstallation.
>
> I think it's a problem that when I install tools that have outdated
> revisions of a package repository as a dependency, they don't
> recognize that a newer version is available (and already installed),
> but instead insist on installing the older revision, too. I don't
> understand why they even have this strict requirement on a specific
> version of a package repo, and not just the version of the tool
> inside.
>
> Btw. I also noticed that

Re: [galaxy-dev] recurring issues with revisions of packages from toolshed

2015-10-20 Thread Marius van den Beek 
Sorry to hijack the thread, but I think the issue that Sarah has described
could be related to
https://github.com/galaxyproject/galaxy/issues/667.
(I know it's not exactly the same, but chances are we fix this, we
automatically fix Sarah's issue.)
There is also an example to reproduce
https://github.com/galaxyproject/galaxy/issues/667#issuecomment-138878174
And I made a bit of work on this in Pull Request:
https://github.com/galaxyproject/galaxy/pull/707
But unfortunately at the time this was a bit over my head.
Bjoern, if you want to work on this, let me know, maybe I can give you a
hand.

Hope that's helpful,
Marius

On 19 October 2015 at 23:17, Björn Grüning 
wrote:

> Hi Sarah,
>
> can you create a reproducible example in a Docker container for me?
> Together with a friend I will have a look at this. But we need to short
> reproducible example :)
>
> Thanks,
> Bjoern
>
>
> > Dear all,
> >
> > I continuously have several issues with tool dependency packages from
> > the toolshed and their revisions. It looks like the tools I install
> > have requirements for different revisions of the same package_xy
> > repository. What I end up with quite often is having the same package
> > installed in multiple revisions (see attached
> > "galaxytools_multiple_revisions.jpg" screenshot). When I then update
> > the older revision, it's still listet twice and additionally the tool
> > that had the older one as a dependency now says it's missing a
> > repository dependency.
> >
> > Although this is already quite annoying and creates a mess within the
> > tool list, I could live with it. However, sometimes it completely
> > messes up revisions. In the second attached screenshot
> > (numpy_revision_issue.jpg) you can see that this numpy repository
> > thinks it is revision "300877695495", but it's actual location has
> > revision "8b3a5c7061d8". This causes tools that depend on revision
> > "300877695495" to think it's there, but when the dependency is
> > resolved at runtime of the tool, numpy is not found (since it's
> > location is in a different revision's folder) and the run fails. As
> > you can also see from the screenshot, resetting the metadata doesn't
> > do anything about this. Also reinstalling doesn't work, since the
> > database keeps the faulty record and uses it upon reinstallation.
> >
> > I think it's a problem that when I install tools that have outdated
> > revisions of a package repository as a dependency, they don't
> > recognize that a newer version is available (and already installed),
> > but instead insist on installing the older revision, too. I don't
> > understand why they even have this strict requirement on a specific
> > version of a package repo, and not just the version of the tool
> > inside.
> >
> > Btw. I also noticed that the installation of R packages (e.g. in
> > package_dexseq_1_14) silently fails. Some of the requirements and
> > also the dexseq R package itself failed the installation, but Galaxy
> > didn't recognize any error and it still showed up as "Installed".
> >
> > Best regards, Sarah
> >
> >
> >  Sarah Diehl HPC System Administrator
> >
> > UNIVERSITÉ DU LUXEMBOURG
> >
> > LUXEMBOURG CENTRE FOR SYSTEMS BIOMEDICINE Campus Belval | Biotech II
> > 6, avenue du Swing L-4371 Belvaux T +352 46 66 44 5360
> > sarah.di...@uni.lu
> > http://lcsb.uni.lu - This message is
> > confidential and may contain privileged information. It is intended
> > for the named recipient only. If you receive it in error please
> > notify me and permanently delete the original message and any
> > copies. -
> >
> >
> >
> >
> > ___ Please
> > keep all replies on the list by using "reply all" in your mail
> > client.  To manage your subscriptions to this and other Galaxy lists,
> > please use the interface at: https://lists.galaxyproject.org/
> >
> > To search Galaxy mailing lists use the unified search at:
> > http://galaxyproject.org/search/mailinglists/
> >
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] recurring issues with revisions of packages from toolshed

2015-10-19 Thread Björn Grüning 
Hi Sarah,

can you create a reproducible example in a Docker container for me?
Together with a friend I will have a look at this. But we need to short
reproducible example :)

Thanks,
Bjoern


> Dear all,
> 
> I continuously have several issues with tool dependency packages from
> the toolshed and their revisions. It looks like the tools I install
> have requirements for different revisions of the same package_xy
> repository. What I end up with quite often is having the same package
> installed in multiple revisions (see attached
> "galaxytools_multiple_revisions.jpg" screenshot). When I then update
> the older revision, it's still listet twice and additionally the tool
> that had the older one as a dependency now says it's missing a
> repository dependency.
> 
> Although this is already quite annoying and creates a mess within the
> tool list, I could live with it. However, sometimes it completely
> messes up revisions. In the second attached screenshot
> (numpy_revision_issue.jpg) you can see that this numpy repository
> thinks it is revision "300877695495", but it's actual location has
> revision "8b3a5c7061d8". This causes tools that depend on revision
> "300877695495" to think it's there, but when the dependency is
> resolved at runtime of the tool, numpy is not found (since it's
> location is in a different revision's folder) and the run fails. As
> you can also see from the screenshot, resetting the metadata doesn't
> do anything about this. Also reinstalling doesn't work, since the
> database keeps the faulty record and uses it upon reinstallation.
> 
> I think it's a problem that when I install tools that have outdated
> revisions of a package repository as a dependency, they don't
> recognize that a newer version is available (and already installed),
> but instead insist on installing the older revision, too. I don't
> understand why they even have this strict requirement on a specific
> version of a package repo, and not just the version of the tool
> inside.
> 
> Btw. I also noticed that the installation of R packages (e.g. in
> package_dexseq_1_14) silently fails. Some of the requirements and
> also the dexseq R package itself failed the installation, but Galaxy
> didn't recognize any error and it still showed up as "Installed".
> 
> Best regards, Sarah
> 
> 
>  Sarah Diehl HPC System Administrator
> 
> UNIVERSITÉ DU LUXEMBOURG
> 
> LUXEMBOURG CENTRE FOR SYSTEMS BIOMEDICINE Campus Belval | Biotech II 
> 6, avenue du Swing L-4371 Belvaux T +352 46 66 44 5360 
> sarah.di...@uni.lu
> http://lcsb.uni.lu - This message is
> confidential and may contain privileged information. It is intended
> for the named recipient only. If you receive it in error please
> notify me and permanently delete the original message and any
> copies. -
> 
> 
> 
> 
> ___ Please
> keep all replies on the list by using "reply all" in your mail
> client.  To manage your subscriptions to this and other Galaxy lists,
> please use the interface at: https://lists.galaxyproject.org/
> 
> To search Galaxy mailing lists use the unified search at: 
> http://galaxyproject.org/search/mailinglists/
> 
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] recurring issues with revisions of packages from toolshed

2015-10-17 Thread Björn Grüning 

Hi Sarah,

as far as I can tell this is exactly how it should work.
Thanks for clarifying!
Bjoern

> Just to avoid confusion, I would like to explain how I understand the
> meaning of "revisions" and "versions".
> 
> Revisions are what shows up the in tool list as the orange triangle
> with a "!". You can simply update the tool and it's replaced by the
> newer revision. So revisions should be mostly bug fixes and such,
> which don't change the behaviour of the tool. Version changes show up
> with the red up arrow and if you install a new version of a tool it
> will not replace the older one, but both will be active. So a version
> change usually means an update of the underlying program and other
> changes that influence behaviour and/or output of the tool.
> 
> What I was talking about in my mail were different revisions. Since
> those are not supposed to change the behaviour or the version of the
> program/package behind, I don't see why always using the latest
> revision should be an issue.
> 
> 
>  Sarah Diehl HPC System Administrator
> 
> UNIVERSITÉ DU LUXEMBOURG
> 
> LUXEMBOURG CENTRE FOR SYSTEMS BIOMEDICINE Campus Belval | Biotech II 
> 6, avenue du Swing L-4371 Belvaux T +352 46 66 44 5360 
> sarah.di...@uni.lu<mailto:sarah.di...@uni.lu>
> http://lcsb.uni.lu<http://lcsb.uni.lu/> - This message is
> confidential and may contain privileged information. It is intended
> for the named recipient only. If you receive it in error please
> notify me and permanently delete the original message and any
> copies. -
> 
> 
> From: Christian Brenninkmeijer
> <christian.brenninkmei...@manchester.ac.uk<mailto:christian.brenninkmei...@manchester.ac.uk>>
>
> 
Date: Thursday 15 October 2015 11:43
> To: Sarah DIEHL <sarah.di...@uni.lu<mailto:sarah.di...@uni.lu>>,
> "galaxy-dev@lists.galaxyproject.org<mailto:galaxy-dev@lists.galaxyproject.org>"
> <galaxy-dev@lists.galaxyproject.org<mailto:galaxy-dev@lists.galaxyproject.org>>
>
> 
Subject: RE: [galaxy-dev] recurring issues with revisions of packages
from toolshed
> 
> Note these are personal views of another admin not official Galaxy
> views.
> 
> The version issues is a hard one to solve, as automatically updating
> upstream project is also dangerous as that leaves them in a state
> that has not be verified by the developer.
> 
> Best solution could be for the toolshed to warn if a tool upload
> would pull in multiple versions.
> 
> As for the R install issue. 1. I too find it extremely frustrating
> how often Galaxy just silently fails.
> 
> 2. It could be due to a missing R package. The galaxy tool to
> determine required R packages appears to miss ones only listed as
> linked in. For example I had one which depends on lme4. This failed
> to build because RcppEigen was missing Yet it RccpEigen is only
> listed as "LinkingTo:"
> 
> Christian  From: galaxy-dev
> [galaxy-dev-boun...@lists.galaxyproject.org<mailto:galaxy-dev-boun...@lists.galaxyproject.org>]
> on behalf of Sarah DIEHL
> [sarah.di...@uni.lu<mailto:sarah.di...@uni.lu>] Sent: Wednesday,
> October 14, 2015 2:05 PM To:
> galaxy-dev@lists.galaxyproject.org<mailto:galaxy-dev@lists.galaxyproject.org>
>
> 
Subject: [galaxy-dev] recurring issues with revisions of packages from
toolshed
> 
> Dear all,
> 
> I continuously have several issues with tool dependency packages from
> the toolshed and their revisions. It looks like the tools I install
> have requirements for different revisions of the same package_xy
> repository. What I end up with quite often is having the same package
> installed in multiple revisions (see attached
> "galaxytools_multiple_revisions.jpg" screenshot). When I then update
> the older revision, it's still listet twice and additionally the tool
> that had the older one as a dependency now says it's missing a
> repository dependency.
> 
> Although this is already quite annoying and creates a mess within the
> tool list, I could live with it. However, sometimes it completely
> messes up revisions. In the second attached screenshot
> (numpy_revision_issue.jpg) you can see that this numpy repository
> thinks it is revision "300877695495", but it's actual location has
> revision "8b3a5c7061d8". This causes tools that depend on revision
> "300877695495" to think it's there, but when the dependency is
> resolved at runtime of the tool, numpy is not found (since it's
> location is in a different revision's folder) and the run fails. As
> you can also see from the screenshot, resetting the metadata doesn't
> do anything

Re: [galaxy-dev] recurring issues with revisions of packages from toolshed

2015-10-17 Thread Björn Grüning 


Am 15.10.2015 um 13:20 schrieb Christian Brenninkmeijer:
> Regarding package dependency.
> Assume there packages A, B and C
> B depends on A and C on A and B
> 
> At the Start there is
> A0, B0 and C0
> B0 depends on A0 and C0 on A0 and B0
> 
> Now if A is updated there is a A1
> But B0 still depends on A0 and C0 still on A0 and B0
> 
> Now if C is updated you have C1
> C1 detects the latest versions of A and B so depends on A1 and B0
> But B0 still depends on A0
> 
> The result is that C1 directly depends on A1 and indirectly (via B0) on A0
> 
> Could the tool shed detect this double dependency and refuse to load C1 
> unless it specifically references A0 and B0


As far as I know (@Dave, please correct me if I'm wrong) A0 does not
exists for Galaxy as soon as you have A1 installed.
Galaxy should see during runtime that A is a "package", a special
repository only containing tool dependencies, and should only take the
most recent version. Imho a different behaviour is a bug. And I actually
don't know why A0 is not deleted as soon as A1 is installed successfully.

Revisions as Sarah has described in her mail do not exist for
tool-dependency packages, afaik.


> ===
> Regard R 
> I agree that as a wrapper it Galaxy is highly dependent on what the 
> underlying tools report.
> 
> And if they change what they report that is a disaster.
> 
> What I do in my testing is after all the R is loaded try to run an R script 
> which loads the packages as a test.
> 
> Something like:
> library(Cairo)
> args<-commandArgs(TRUE)
> writeLines(capture.output(sessionInfo()), args[1])
> sessionInfo()
> 
> But even then the tool goes green with an R error,
> 
> Christian
> 
> 
> From: Björn Grüning [bjoern.gruen...@gmail.com]
> Sent: Thursday, October 15, 2015 10:48 AM
> To: Christian Brenninkmeijer; Sarah DIEHL; galaxy-dev@lists.galaxyproject.org
> Subject: Re: [galaxy-dev] recurring issues with revisions of packages from 
> toolshed
> 
> Am 15.10.2015 um 11:43 schrieb Christian Brenninkmeijer:
>> Note these are personal views of another admin not official Galaxy views.
>>
>> The version issues is a hard one to solve, as automatically updating 
>> upstream project is also dangerous as that leaves them in a state that has 
>> not be verified by the developer.
>>
>> Best solution could be for the toolshed to warn if a tool upload would pull 
>> in multiple versions.
> 
> Can you elaborate there a little bit more?
> 
>> As for the R install issue.
>> 1. I too find it extremely frustrating how often Galaxy just silently fails.
> 
> We are working on this.
> 
> https://github.com/bgruening/galaxy/pull/8
> 
> But it is really hard to convince R to provide us with a real error
> code. I spend several days, but different R versions, fail differently
> or not at all ..
> 
> I will keep you posted on this and hopefully the latest idea from us
> works out.
> 
> Sorry for any inconvenience,
> Bjoern
> 
>> 2. It could be due to a missing R package.
>> The galaxy tool to determine required R packages appears to miss ones only 
>> listed as linked in.
>> For example I had one which depends on lme4.
>> This failed to build because RcppEigen was missing
>> Yet it RccpEigen is only listed as "LinkingTo:"
>>
>> Christian
>> 
>> From: galaxy-dev [galaxy-dev-boun...@lists.galaxyproject.org] on behalf of 
>> Sarah DIEHL [sarah.di...@uni.lu]
>> Sent: Wednesday, October 14, 2015 2:05 PM
>> To: galaxy-dev@lists.galaxyproject.org
>> Subject: [galaxy-dev] recurring issues with revisions of packages from 
>> toolshed
>>
>> Dear all,
>>
>> I continuously have several issues with tool dependency packages from the 
>> toolshed and their revisions. It looks like the tools I install have 
>> requirements for different revisions of the same package_xy repository. What 
>> I end up with quite often is having the same package installed in multiple 
>> revisions (see attached "galaxytools_multiple_revisions.jpg" screenshot). 
>> When I then update the older revision, it's still listet twice and 
>> additionally the tool that had the older one as a dependency now says it's 
>> missing a repository dependency.
>>
>> Although this is already quite annoying and creates a mess within the tool 
>> list, I could live with it. However, sometimes it completely messes up 
>> revisions. In the second attached screenshot (numpy_revision_issue.jpg) you 
>> can see that this numpy repository thinks it is revision "300877695495&qu

Re: [galaxy-dev] recurring issues with revisions of packages from toolshed

2015-10-15 Thread Sarah DIEHL 
Just to avoid confusion, I would like to explain how I understand the meaning 
of "revisions" and "versions".

Revisions are what shows up the in tool list as the orange triangle with a "!". 
You can simply update the tool and it's replaced by the newer revision. So 
revisions should be mostly bug fixes and such, which don't change the behaviour 
of the tool.
Version changes show up with the red up arrow and if you install a new version 
of a tool it will not replace the older one, but both will be active. So a 
version change usually means an update of the underlying program and other 
changes that influence behaviour and/or output of the tool.

What I was talking about in my mail were different revisions. Since those are 
not supposed to change the behaviour or the version of the program/package 
behind, I don't see why always using the latest revision should be an issue.



Sarah Diehl
HPC System Administrator

UNIVERSITÉ DU LUXEMBOURG

LUXEMBOURG CENTRE FOR SYSTEMS BIOMEDICINE
Campus Belval | Biotech II
6, avenue du Swing
L-4371 Belvaux
T +352 46 66 44 5360
sarah.di...@uni.lu<mailto:sarah.di...@uni.lu> 
http://lcsb.uni.lu<http://lcsb.uni.lu/>
-
This message is confidential and may contain privileged information. It is 
intended for the named recipient only. If you receive it in error please notify 
me and permanently delete the original message and any copies.
-


From: Christian Brenninkmeijer 
<christian.brenninkmei...@manchester.ac.uk<mailto:christian.brenninkmei...@manchester.ac.uk>>
Date: Thursday 15 October 2015 11:43
To: Sarah DIEHL <sarah.di...@uni.lu<mailto:sarah.di...@uni.lu>>, 
"galaxy-dev@lists.galaxyproject.org<mailto:galaxy-dev@lists.galaxyproject.org>" 
<galaxy-dev@lists.galaxyproject.org<mailto:galaxy-dev@lists.galaxyproject.org>>
Subject: RE: [galaxy-dev] recurring issues with revisions of packages from 
toolshed

Note these are personal views of another admin not official Galaxy views.

The version issues is a hard one to solve, as automatically updating upstream 
project is also dangerous as that leaves them in a state that has not be 
verified by the developer.

Best solution could be for the toolshed to warn if a tool upload would pull in 
multiple versions.

As for the R install issue.
1. I too find it extremely frustrating how often Galaxy just silently fails.

2. It could be due to a missing R package.
The galaxy tool to determine required R packages appears to miss ones only 
listed as linked in.
For example I had one which depends on lme4.
This failed to build because RcppEigen was missing
Yet it RccpEigen is only listed as "LinkingTo:"

Christian

From: galaxy-dev 
[galaxy-dev-boun...@lists.galaxyproject.org<mailto:galaxy-dev-boun...@lists.galaxyproject.org>]
 on behalf of Sarah DIEHL [sarah.di...@uni.lu<mailto:sarah.di...@uni.lu>]
Sent: Wednesday, October 14, 2015 2:05 PM
To: 
galaxy-dev@lists.galaxyproject.org<mailto:galaxy-dev@lists.galaxyproject.org>
Subject: [galaxy-dev] recurring issues with revisions of packages from toolshed

Dear all,

I continuously have several issues with tool dependency packages from the 
toolshed and their revisions. It looks like the tools I install have 
requirements for different revisions of the same package_xy repository. What I 
end up with quite often is having the same package installed in multiple 
revisions (see attached "galaxytools_multiple_revisions.jpg" screenshot). When 
I then update the older revision, it's still listet twice and additionally the 
tool that had the older one as a dependency now says it's missing a repository 
dependency.

Although this is already quite annoying and creates a mess within the tool 
list, I could live with it. However, sometimes it completely messes up 
revisions. In the second attached screenshot (numpy_revision_issue.jpg) you can 
see that this numpy repository thinks it is revision "300877695495", but it's 
actual location has revision "8b3a5c7061d8". This causes tools that depend on 
revision "300877695495" to think it's there, but when the dependency is 
resolved at runtime of the tool, numpy is not found (since it's location is in 
a different revision's folder) and the run fails. As you can also see from the 
screenshot, resetting the metadata doesn't do anything about this. Also 
reinstalling doesn't work, since the database keeps the faulty record and uses 
it upon reinstallation.

I think it's a problem that when I install tools that have outdated revisions 
of a package repository as a dependency, they don't recognize that a newer 
version is available (and already installed), but instead insist on installing 
the older revision, too. I don't understand why they even have this strict 
requirement on a specific version of a package repo, and not just the version 
of the tool insid

Re: [galaxy-dev] recurring issues with revisions of packages from toolshed

2015-10-15 Thread Christian Brenninkmeijer 
Regarding package dependency.
Assume there packages A, B and C
B depends on A and C on A and B

At the Start there is
A0, B0 and C0
B0 depends on A0 and C0 on A0 and B0

Now if A is updated there is a A1
But B0 still depends on A0 and C0 still on A0 and B0

Now if C is updated you have C1
C1 detects the latest versions of A and B so depends on A1 and B0
But B0 still depends on A0

The result is that C1 directly depends on A1 and indirectly (via B0) on A0

Could the tool shed detect this double dependency and refuse to load C1 unless 
it specifically references A0 and B0

===
Regard R 
I agree that as a wrapper it Galaxy is highly dependent on what the underlying 
tools report.

And if they change what they report that is a disaster.

What I do in my testing is after all the R is loaded try to run an R script 
which loads the packages as a test.

Something like:
library(Cairo)
args<-commandArgs(TRUE)
writeLines(capture.output(sessionInfo()), args[1])
sessionInfo()

But even then the tool goes green with an R error,

Christian


From: Björn Grüning [bjoern.gruen...@gmail.com]
Sent: Thursday, October 15, 2015 10:48 AM
To: Christian Brenninkmeijer; Sarah DIEHL; galaxy-dev@lists.galaxyproject.org
Subject: Re: [galaxy-dev] recurring issues with revisions of packages from 
toolshed

Am 15.10.2015 um 11:43 schrieb Christian Brenninkmeijer:
> Note these are personal views of another admin not official Galaxy views.
>
> The version issues is a hard one to solve, as automatically updating upstream 
> project is also dangerous as that leaves them in a state that has not be 
> verified by the developer.
>
> Best solution could be for the toolshed to warn if a tool upload would pull 
> in multiple versions.

Can you elaborate there a little bit more?

> As for the R install issue.
> 1. I too find it extremely frustrating how often Galaxy just silently fails.

We are working on this.

https://github.com/bgruening/galaxy/pull/8

But it is really hard to convince R to provide us with a real error
code. I spend several days, but different R versions, fail differently
or not at all ..

I will keep you posted on this and hopefully the latest idea from us
works out.

Sorry for any inconvenience,
Bjoern

> 2. It could be due to a missing R package.
> The galaxy tool to determine required R packages appears to miss ones only 
> listed as linked in.
> For example I had one which depends on lme4.
> This failed to build because RcppEigen was missing
> Yet it RccpEigen is only listed as "LinkingTo:"
>
> Christian
> 
> From: galaxy-dev [galaxy-dev-boun...@lists.galaxyproject.org] on behalf of 
> Sarah DIEHL [sarah.di...@uni.lu]
> Sent: Wednesday, October 14, 2015 2:05 PM
> To: galaxy-dev@lists.galaxyproject.org
> Subject: [galaxy-dev] recurring issues with revisions of packages from 
> toolshed
>
> Dear all,
>
> I continuously have several issues with tool dependency packages from the 
> toolshed and their revisions. It looks like the tools I install have 
> requirements for different revisions of the same package_xy repository. What 
> I end up with quite often is having the same package installed in multiple 
> revisions (see attached "galaxytools_multiple_revisions.jpg" screenshot). 
> When I then update the older revision, it's still listet twice and 
> additionally the tool that had the older one as a dependency now says it's 
> missing a repository dependency.
>
> Although this is already quite annoying and creates a mess within the tool 
> list, I could live with it. However, sometimes it completely messes up 
> revisions. In the second attached screenshot (numpy_revision_issue.jpg) you 
> can see that this numpy repository thinks it is revision "300877695495", but 
> it's actual location has revision "8b3a5c7061d8". This causes tools that 
> depend on revision "300877695495" to think it's there, but when the 
> dependency is resolved at runtime of the tool, numpy is not found (since it's 
> location is in a different revision's folder) and the run fails. As you can 
> also see from the screenshot, resetting the metadata doesn't do anything 
> about this. Also reinstalling doesn't work, since the database keeps the 
> faulty record and uses it upon reinstallation.
>
> I think it's a problem that when I install tools that have outdated revisions 
> of a package repository as a dependency, they don't recognize that a newer 
> version is available (and already installed), but instead insist on 
> installing the older revision, too. I don't understand why they even have 
> this strict requirement on a specific version of a package repo, and not just 
> the version of the tool inside.
>
> Btw. I also noticed that the installation of R p

Re: [galaxy-dev] recurring issues with revisions of packages from toolshed

2015-10-15 Thread Björn Grüning 


Am 15.10.2015 um 11:43 schrieb Christian Brenninkmeijer:
> Note these are personal views of another admin not official Galaxy views.
> 
> The version issues is a hard one to solve, as automatically updating upstream 
> project is also dangerous as that leaves them in a state that has not be 
> verified by the developer.
> 
> Best solution could be for the toolshed to warn if a tool upload would pull 
> in multiple versions.

Can you elaborate there a little bit more?

> As for the R install issue.
> 1. I too find it extremely frustrating how often Galaxy just silently fails.

We are working on this.

https://github.com/bgruening/galaxy/pull/8

But it is really hard to convince R to provide us with a real error
code. I spend several days, but different R versions, fail differently
or not at all ..

I will keep you posted on this and hopefully the latest idea from us
works out.

Sorry for any inconvenience,
Bjoern

> 2. It could be due to a missing R package.
> The galaxy tool to determine required R packages appears to miss ones only 
> listed as linked in.
> For example I had one which depends on lme4.
> This failed to build because RcppEigen was missing
> Yet it RccpEigen is only listed as "LinkingTo:"
> 
> Christian
> 
> From: galaxy-dev [galaxy-dev-boun...@lists.galaxyproject.org] on behalf of 
> Sarah DIEHL [sarah.di...@uni.lu]
> Sent: Wednesday, October 14, 2015 2:05 PM
> To: galaxy-dev@lists.galaxyproject.org
> Subject: [galaxy-dev] recurring issues with revisions of packages from 
> toolshed
> 
> Dear all,
> 
> I continuously have several issues with tool dependency packages from the 
> toolshed and their revisions. It looks like the tools I install have 
> requirements for different revisions of the same package_xy repository. What 
> I end up with quite often is having the same package installed in multiple 
> revisions (see attached "galaxytools_multiple_revisions.jpg" screenshot). 
> When I then update the older revision, it's still listet twice and 
> additionally the tool that had the older one as a dependency now says it's 
> missing a repository dependency.
> 
> Although this is already quite annoying and creates a mess within the tool 
> list, I could live with it. However, sometimes it completely messes up 
> revisions. In the second attached screenshot (numpy_revision_issue.jpg) you 
> can see that this numpy repository thinks it is revision "300877695495", but 
> it's actual location has revision "8b3a5c7061d8". This causes tools that 
> depend on revision "300877695495" to think it's there, but when the 
> dependency is resolved at runtime of the tool, numpy is not found (since it's 
> location is in a different revision's folder) and the run fails. As you can 
> also see from the screenshot, resetting the metadata doesn't do anything 
> about this. Also reinstalling doesn't work, since the database keeps the 
> faulty record and uses it upon reinstallation.
> 
> I think it's a problem that when I install tools that have outdated revisions 
> of a package repository as a dependency, they don't recognize that a newer 
> version is available (and already installed), but instead insist on 
> installing the older revision, too. I don't understand why they even have 
> this strict requirement on a specific version of a package repo, and not just 
> the version of the tool inside.
> 
> Btw. I also noticed that the installation of R packages (e.g. in 
> package_dexseq_1_14) silently fails. Some of the requirements and also the 
> dexseq R package itself failed the installation, but Galaxy didn't recognize 
> any error and it still showed up as "Installed".
> 
> Best regards,
> Sarah
> 
> 
> 
> Sarah Diehl
> HPC System Administrator
> 
> UNIVERSITÉ DU LUXEMBOURG
> 
> LUXEMBOURG CENTRE FOR SYSTEMS BIOMEDICINE
> Campus Belval | Biotech II
> 6, avenue du Swing
> L-4371 Belvaux
> T +352 46 66 44 5360
> sarah.di...@uni.lu<mailto:sarah.di...@uni.lu> 
> http://lcsb.uni.lu<http://lcsb.uni.lu/>
> -
> This message is confidential and may contain privileged information. It is 
> intended for the named recipient only. If you receive it in error please 
> notify me and permanently delete the original message and any copies.
> -
> 
> 
> 
> 
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
> 
> To search Galaxy mailing lists use the unified search at:
>   http://gala

Re: [galaxy-dev] recurring issues with revisions of packages from toolshed

2015-10-15 Thread Christian Brenninkmeijer 
Note these are personal views of another admin not official Galaxy views.

The version issues is a hard one to solve, as automatically updating upstream 
project is also dangerous as that leaves them in a state that has not be 
verified by the developer.

Best solution could be for the toolshed to warn if a tool upload would pull in 
multiple versions.

As for the R install issue.
1. I too find it extremely frustrating how often Galaxy just silently fails.

2. It could be due to a missing R package.
The galaxy tool to determine required R packages appears to miss ones only 
listed as linked in.
For example I had one which depends on lme4.
This failed to build because RcppEigen was missing
Yet it RccpEigen is only listed as "LinkingTo:"

Christian

From: galaxy-dev [galaxy-dev-boun...@lists.galaxyproject.org] on behalf of 
Sarah DIEHL [sarah.di...@uni.lu]
Sent: Wednesday, October 14, 2015 2:05 PM
To: galaxy-dev@lists.galaxyproject.org
Subject: [galaxy-dev] recurring issues with revisions of packages from toolshed

Dear all,

I continuously have several issues with tool dependency packages from the 
toolshed and their revisions. It looks like the tools I install have 
requirements for different revisions of the same package_xy repository. What I 
end up with quite often is having the same package installed in multiple 
revisions (see attached "galaxytools_multiple_revisions.jpg" screenshot). When 
I then update the older revision, it's still listet twice and additionally the 
tool that had the older one as a dependency now says it's missing a repository 
dependency.

Although this is already quite annoying and creates a mess within the tool 
list, I could live with it. However, sometimes it completely messes up 
revisions. In the second attached screenshot (numpy_revision_issue.jpg) you can 
see that this numpy repository thinks it is revision "300877695495", but it's 
actual location has revision "8b3a5c7061d8". This causes tools that depend on 
revision "300877695495" to think it's there, but when the dependency is 
resolved at runtime of the tool, numpy is not found (since it's location is in 
a different revision's folder) and the run fails. As you can also see from the 
screenshot, resetting the metadata doesn't do anything about this. Also 
reinstalling doesn't work, since the database keeps the faulty record and uses 
it upon reinstallation.

I think it's a problem that when I install tools that have outdated revisions 
of a package repository as a dependency, they don't recognize that a newer 
version is available (and already installed), but instead insist on installing 
the older revision, too. I don't understand why they even have this strict 
requirement on a specific version of a package repo, and not just the version 
of the tool inside.

Btw. I also noticed that the installation of R packages (e.g. in 
package_dexseq_1_14) silently fails. Some of the requirements and also the 
dexseq R package itself failed the installation, but Galaxy didn't recognize 
any error and it still showed up as "Installed".

Best regards,
Sarah



Sarah Diehl
HPC System Administrator

UNIVERSITÉ DU LUXEMBOURG

LUXEMBOURG CENTRE FOR SYSTEMS BIOMEDICINE
Campus Belval | Biotech II
6, avenue du Swing
L-4371 Belvaux
T +352 46 66 44 5360
sarah.di...@uni.lu<mailto:sarah.di...@uni.lu> 
http://lcsb.uni.lu<http://lcsb.uni.lu/>
-
This message is confidential and may contain privileged information. It is 
intended for the named recipient only. If you receive it in error please notify 
me and permanently delete the original message and any copies.
-

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/