Re: [galaxy-dev] Test Toolshed Biopython package dependency NumPy

2013-10-22 Thread Peter Cock 
Hi Dave,

The nightly tests for my tools using Biopython are mostly working, e.g.

http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_filter_by_id
http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_rename

However, I had an odd failure last night:

http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_select_by_id/9e9e3b860aa0

Automated tool test results

Tool test results
Automated test environment
Time tested: ~ 11 hours ago
System: Linux 3.0.0-1-amd64
Architecture: x86_64
Python version: 2.7.2+
Galaxy revision: 11038:315018626d87
Galaxy database version: 117
Tool shed revision: 11020:95815478f355
Tool shed database version: 21
Tool shed mercurial version: 2.2.3
Tests that failed
Tool id: seq_select_by_id
Tool version: seq_select_by_id
Test: test_tool_00
(functional.test_toolbox.TestForTool_testtoolshed.g2.bx.psu.edu/repos/peterjc/seq_select_by_id/seq_select_by_id/0.0.6)
Stderr:
Fatal error: Exit code 1 ()
Biopython 1.54 or later is required
Traceback:
Traceback (most recent call last):
  File 
/var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py,
line 171, in test_tool
self.do_it( td, shed_tool_id=shed_tool_id )
  File 
/var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py,
line 102, in do_it
self.verify_dataset_correctness( outfile, hid=elem_hid,
maxseconds=testdef.maxseconds, attributes=attributes,
shed_tool_id=shed_tool_id )
  File 
/var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/test/base/twilltestcase.py,
line 782, in verify_dataset_correctness
self._assert_dataset_state( elem, 'ok' )
  File 
/var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/test/base/twilltestcase.py,
line 606, in _assert_dataset_state
raise AssertionError( errmsg )
AssertionError: Expecting dataset state 'ok', but state is 'error'.
Dataset blurb: error


There are no installation problems shown, yet the Biopython
import line inside seq_select_by_id.py is failing.

Could you send me the logs from this failure? Perhaps we
have a silent installation failure?

Thanks,

Peter
___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Test Toolshed Biopython package dependency NumPy

2013-10-15 Thread Dave Bouvier 

Björn, Peter,

In the course of my investigation, I've discovered a possible 
improvement to the lapack build recipe. If you change the cmake command 
from:


cmake .. -DCMAKE_INSTALL_PREFIX=$INSTALL_DIR/lapack

to:

cmake .. -DCMAKE_INSTALL_PREFIX=$INSTALL_DIR/lapack 
-DCMAKE_Fortran_FLAGS='-O2 -fPIC'


the need for the earlier sed command is eliminated, which may reduce the 
likelihood of environment-specific failures. Specifically, I attempted 
to install get_orfs_or_cdss on a local (linux) system, and the numpy 
installation's error message informed me that lapack was built without 
the -fPIC flag.



   --Dave B.

On 10/11/2013 04:51 AM, Peter Cock wrote:

On Thu, Oct 10, 2013 at 10:00 AM, Peter Cock p.j.a.c...@googlemail.com wrote:

On Wed, Oct 9, 2013 at 10:01 PM, Björn Grüning
bjoern.gruen...@pharmazie.uni-freiburg.de wrote:

Am Mittwoch, den 09.10.2013, 19:15 +0100 schrieb Peter Cock:

On Tue, Oct 8, 2013 at 9:57 PM, Dave Bouvier d...@bx.psu.edu wrote:

Peter, Björn,

I have restored gfortran to the repository test host. I believe it was
inadvertently removed, possibly as a side effect of updating packages or
removing packages that were causing conflicts with the Galaxy eggs. Numpy
and biopython should now install successfully, as they do in my local
environment.

--Dave B.


Thanks Dave, fingers crossed for tonight's test run :)

Bjoern - it looks like we can leave the NumPy definition as is...


Ah sorry, totally forgot about it :(



Progress, Fortran is working and lapack seems to have installed, but:
e.g. http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_rename

Installation errors - no functional tests were run for any tools in
this changeset revision
Tool dependencies
TypeNameVersion
biopython package 1.62
Error
running install Numerical Python (NumPy) is not installed. This
package is required for many Biopython features. Please install it
before you install Biopython. You can install Biopython anyway, but
anything dependent on NumPy will not work. If you do this, and later
install NumPy, you should then re-install Biopython. You can find
NumPy at http://numpy.scipy.org
TypeNameVersion
numpy package 1.7.1
Error
Running from numpy source directory.
/var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/tmp/tmp3kNwuR/tmp-toolshed-mtdGJ0d5H/numpy-1.7.1/numpy/distutils/system_info.py:1494:
UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not
found. Directories to search for the libraries can be specified in the
numpy/distutils/site.cfg file (section [atlas]) or by setting the
ATLAS environment variable. warnings.warn(AtlasNotFoundError.__doc__)
/var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/tmp/tmp3kNwuR/tmp-toolshed-mtdGJ0d5H/numpy-1.7.1/numpy/distutils/system_info.py:1408:
UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not
found. Directories to search for the libraries can be specified in the
numpy/distutils/site.cfg file (section [atlas]) or by setting the
ATLAS environment variable. warnings.warn(AtlasNotFoundError.__doc__)
error: Command /usr/bin/gfortran -Wall -Wall -shared
build/temp.linux-x86_64-2.7/numpy/linalg/lapack_litemodule.o
build/temp.linux-x86_64-2.7/numpy/linalg/python_xerbla.o
-L/ToolDependencies/lapack/3.4.2/iuc/package_lapack_3_4/60957bd68fe2/lapack/lib
-Lbuild/temp.linux-x86_64-2.7 -llapack -lblas -lgfortran -o
build/lib.linux-x86_64-2.7/numpy/linalg/lapack_lite.so failed with
exit status 1


Most of that is warnings about not using ATLAS (which we
expect) but I'm not quite sure why it fails...


Hi Guys,

Bjoern - good news for you, the installation of NumPy can
work on the current Test Tool Shed.

Dave - bad news for you, last night I had 3 NumPy install failures
(all different) and 2 successes - which to me suggests a possible
race condition in the nightly testing?

(1) Last night on the Test Tool Shed this failed as in email above:

http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_primer_clip/834cf6618a14

(2) This failed with the simpler message:

http://testtoolshed.g2.bx.psu.edu/view/peterjc/get_orfs_or_cdss/308ec20faefb

Installation errors - no functional tests were run for any tools in
this changeset revision
Tool dependencies
TypeNameVersion
biopython package 1.62
Error
running install Numerical Python (NumPy) is not installed. This
package is required for many Biopython features. Please install it
before you install Biopython. You can install Biopython anyway, but
anything dependent on NumPy will not work. If you do this, and later
install NumPy, you should then re-install Biopython. You can find
NumPy at http://numpy.scipy.org

(3) Here the installation seems to have failed silently, leading to a
test failure at the import stage:
http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_select_by_id/9e9e3b860aa0

Tests that failed
Tool id: seq_select_by_id
Tool

Re: [galaxy-dev] Test Toolshed Biopython package dependency NumPy

2013-10-11 Thread Peter Cock 
On Thu, Oct 10, 2013 at 10:00 AM, Peter Cock p.j.a.c...@googlemail.com wrote:
 On Wed, Oct 9, 2013 at 10:01 PM, Björn Grüning
 bjoern.gruen...@pharmazie.uni-freiburg.de wrote:
 Am Mittwoch, den 09.10.2013, 19:15 +0100 schrieb Peter Cock:
 On Tue, Oct 8, 2013 at 9:57 PM, Dave Bouvier d...@bx.psu.edu wrote:
  Peter, Björn,
 
  I have restored gfortran to the repository test host. I believe it was
  inadvertently removed, possibly as a side effect of updating packages or
  removing packages that were causing conflicts with the Galaxy eggs. Numpy
  and biopython should now install successfully, as they do in my local
  environment.
 
 --Dave B.

 Thanks Dave, fingers crossed for tonight's test run :)

 Bjoern - it looks like we can leave the NumPy definition as is...

 Ah sorry, totally forgot about it :(


 Progress, Fortran is working and lapack seems to have installed, but:
 e.g. http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_rename

 Installation errors - no functional tests were run for any tools in
 this changeset revision
 Tool dependencies
 TypeNameVersion
 biopython package 1.62
 Error
 running install Numerical Python (NumPy) is not installed. This
 package is required for many Biopython features. Please install it
 before you install Biopython. You can install Biopython anyway, but
 anything dependent on NumPy will not work. If you do this, and later
 install NumPy, you should then re-install Biopython. You can find
 NumPy at http://numpy.scipy.org
 TypeNameVersion
 numpy package 1.7.1
 Error
 Running from numpy source directory.
 /var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/tmp/tmp3kNwuR/tmp-toolshed-mtdGJ0d5H/numpy-1.7.1/numpy/distutils/system_info.py:1494:
 UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not
 found. Directories to search for the libraries can be specified in the
 numpy/distutils/site.cfg file (section [atlas]) or by setting the
 ATLAS environment variable. warnings.warn(AtlasNotFoundError.__doc__)
 /var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/tmp/tmp3kNwuR/tmp-toolshed-mtdGJ0d5H/numpy-1.7.1/numpy/distutils/system_info.py:1408:
 UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not
 found. Directories to search for the libraries can be specified in the
 numpy/distutils/site.cfg file (section [atlas]) or by setting the
 ATLAS environment variable. warnings.warn(AtlasNotFoundError.__doc__)
 error: Command /usr/bin/gfortran -Wall -Wall -shared
 build/temp.linux-x86_64-2.7/numpy/linalg/lapack_litemodule.o
 build/temp.linux-x86_64-2.7/numpy/linalg/python_xerbla.o
 -L/ToolDependencies/lapack/3.4.2/iuc/package_lapack_3_4/60957bd68fe2/lapack/lib
 -Lbuild/temp.linux-x86_64-2.7 -llapack -lblas -lgfortran -o
 build/lib.linux-x86_64-2.7/numpy/linalg/lapack_lite.so failed with
 exit status 1


 Most of that is warnings about not using ATLAS (which we
 expect) but I'm not quite sure why it fails...

Hi Guys,

Bjoern - good news for you, the installation of NumPy can
work on the current Test Tool Shed.

Dave - bad news for you, last night I had 3 NumPy install failures
(all different) and 2 successes - which to me suggests a possible
race condition in the nightly testing?

(1) Last night on the Test Tool Shed this failed as in email above:

http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_primer_clip/834cf6618a14

(2) This failed with the simpler message:

http://testtoolshed.g2.bx.psu.edu/view/peterjc/get_orfs_or_cdss/308ec20faefb

Installation errors - no functional tests were run for any tools in
this changeset revision
Tool dependencies
TypeNameVersion
biopython package 1.62
Error
running install Numerical Python (NumPy) is not installed. This
package is required for many Biopython features. Please install it
before you install Biopython. You can install Biopython anyway, but
anything dependent on NumPy will not work. If you do this, and later
install NumPy, you should then re-install Biopython. You can find
NumPy at http://numpy.scipy.org

(3) Here the installation seems to have failed silently, leading to a
test failure at the import stage:
http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_select_by_id/9e9e3b860aa0

Tests that failed
Tool id: seq_select_by_id
Tool version: seq_select_by_id
Test: test_tool_00
(functional.test_toolbox.TestForTool_testtoolshed.g2.bx.psu.edu/repos/peterjc/seq_select_by_id/seq_select_by_id/0.0.6)
Stderr:
Fatal error: Exit code 1 ()
Biopython 1.54 or later is required

(4, 5) However these also depend on Biopython 1.62 and worked:

http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_rename/d08767ae73ce
(failed last night)

http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_filter_by_id/8a34c565a473

My guess is a race condition where multiple attempts are being made in
parallel to install NumPy (in the same place), and this causes some of
these

Re: [galaxy-dev] Test Toolshed Biopython package dependency NumPy

2013-10-10 Thread Peter Cock 
On Wed, Oct 9, 2013 at 10:01 PM, Björn Grüning
bjoern.gruen...@pharmazie.uni-freiburg.de wrote:
 Am Mittwoch, den 09.10.2013, 19:15 +0100 schrieb Peter Cock:
 On Tue, Oct 8, 2013 at 9:57 PM, Dave Bouvier d...@bx.psu.edu wrote:
  Peter, Björn,
 
  I have restored gfortran to the repository test host. I believe it was
  inadvertently removed, possibly as a side effect of updating packages or
  removing packages that were causing conflicts with the Galaxy eggs. Numpy
  and biopython should now install successfully, as they do in my local
  environment.
 
 --Dave B.

 Thanks Dave, fingers crossed for tonight's test run :)

 Bjoern - it looks like we can leave the NumPy definition as is...

 Ah sorry, totally forgot about it :(


Progress, Fortran is working and lapack seems to have installed, but:
e.g. http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_rename

Installation errors - no functional tests were run for any tools in
this changeset revision
Tool dependencies
TypeNameVersion
biopython package 1.62
Error
running install Numerical Python (NumPy) is not installed. This
package is required for many Biopython features. Please install it
before you install Biopython. You can install Biopython anyway, but
anything dependent on NumPy will not work. If you do this, and later
install NumPy, you should then re-install Biopython. You can find
NumPy at http://numpy.scipy.org
TypeNameVersion
numpy package 1.7.1
Error
Running from numpy source directory.
/var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/tmp/tmp3kNwuR/tmp-toolshed-mtdGJ0d5H/numpy-1.7.1/numpy/distutils/system_info.py:1494:
UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not
found. Directories to search for the libraries can be specified in the
numpy/distutils/site.cfg file (section [atlas]) or by setting the
ATLAS environment variable. warnings.warn(AtlasNotFoundError.__doc__)
/var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/tmp/tmp3kNwuR/tmp-toolshed-mtdGJ0d5H/numpy-1.7.1/numpy/distutils/system_info.py:1408:
UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not
found. Directories to search for the libraries can be specified in the
numpy/distutils/site.cfg file (section [atlas]) or by setting the
ATLAS environment variable. warnings.warn(AtlasNotFoundError.__doc__)
error: Command /usr/bin/gfortran -Wall -Wall -shared
build/temp.linux-x86_64-2.7/numpy/linalg/lapack_litemodule.o
build/temp.linux-x86_64-2.7/numpy/linalg/python_xerbla.o
-L/ToolDependencies/lapack/3.4.2/iuc/package_lapack_3_4/60957bd68fe2/lapack/lib
-Lbuild/temp.linux-x86_64-2.7 -llapack -lblas -lgfortran -o
build/lib.linux-x86_64-2.7/numpy/linalg/lapack_lite.so failed with
exit status 1


Most of that is warnings about not using ATLAS (which we
expect) but I'm not quite sure why it fails...

Peter

___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Test Toolshed Biopython package dependency NumPy

2013-10-09 Thread Peter Cock 
On Tue, Oct 8, 2013 at 9:57 PM, Dave Bouvier d...@bx.psu.edu wrote:
 Peter, Björn,

 I have restored gfortran to the repository test host. I believe it was
 inadvertently removed, possibly as a side effect of updating packages or
 removing packages that were causing conflicts with the Galaxy eggs. Numpy
 and biopython should now install successfully, as they do in my local
 environment.

--Dave B.

Thanks Dave, fingers crossed for tonight's test run :)

Bjoern - it looks like we can leave the NumPy definition as is...

Regards,

Peter

___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Test Toolshed Biopython package dependency NumPy

2013-10-09 Thread Björn Grüning 
Am Mittwoch, den 09.10.2013, 19:15 +0100 schrieb Peter Cock:
 On Tue, Oct 8, 2013 at 9:57 PM, Dave Bouvier d...@bx.psu.edu wrote:
  Peter, Björn,
 
  I have restored gfortran to the repository test host. I believe it was
  inadvertently removed, possibly as a side effect of updating packages or
  removing packages that were causing conflicts with the Galaxy eggs. Numpy
  and biopython should now install successfully, as they do in my local
  environment.
 
 --Dave B.
 
 Thanks Dave, fingers crossed for tonight's test run :)
 
 Bjoern - it looks like we can leave the NumPy definition as is...

Ah sorry, totally forgot about it :(

 Regards,
 
 Peter



___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Test Toolshed Biopython package dependency NumPy

2013-10-08 Thread Peter Cock 
On Mon, Oct 7, 2013 at 8:52 PM, Björn Grüning
bjoern.gruen...@pharmazie.uni-freiburg.de wrote:

 Hi Dave, Bjeorn,

 Right now NumPy isn't installing at all on the Test Tool Shed.

 Dave - is is possible that the system was updated and the
 Fortran compiler removed? e.g.

 ...

 Since this seems to be breaking lapack, perhaps we should
 further simplify package_numpy_1_7 not to use lapack or
 ATLAS? What do you think Bjoern?

 I'm a little bit sceptic to remove features only because the Test
 Utility can not handle it.
 On the other hand I understand that numpy should just work and that your
 test failing since month because of my dependencies. I will get rid of
 it tomorrow. As a result we can setup_virtualenv and install numpy from
 pip directly I think.

 Is that ok for you?
 Cheers,
 Bjoern

Hi Bjoern,

I think the real problem here is we don't yet have a clear
base installation:
https://trello.com/c/gYlRdqwr/774-toolshed-define-a-base-set-of-utilities-that-can-be-expected-to-be-on-systems-into-which-tool-dependencies-are-being-installed
https://trello.com/c/7VTlX9rD/844-toolshed-tool-dependency-installation-requirements

What is worse is the system used by the ToolShed for
running the nightly tests appears to change without
notice, e.g. here it looks like something with the Fortran
compiler has changed, for the BLAST+ wrappers the
default shell seems to have changed between bash
and sh).

I think the ToolShed system should be a role model of
the agreed standard set of system level dependencies.

Given that, it seems your lapack package has some
dependencies which are not necessarily present, and
this failure is cascading up to break NumPy and then
Biopython, and thus multiple tools on the Tool Shed :(

I do think that in the short term the best option is to
 further simplify package_numpy_1_7 not to use
lapack or ATLAS - although this is not ideal :(

On the positive side, your hard work setting up packages
for NumPy, lapack and Altas is drawing attention to the
need for a clearer set of base packages. :)

Thanks,

Peter

___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Test Toolshed Biopython package dependency NumPy

2013-10-08 Thread Dave Bouvier 

Peter, Björn,

I have restored gfortran to the repository test host. I believe it was 
inadvertently removed, possibly as a side effect of updating packages or 
removing packages that were causing conflicts with the Galaxy eggs. 
Numpy and biopython should now install successfully, as they do in my 
local environment.


   --Dave B.

On 10/08/2013 05:25 AM, Peter Cock wrote:

On Mon, Oct 7, 2013 at 8:52 PM, Björn Grüning
bjoern.gruen...@pharmazie.uni-freiburg.de wrote:


Hi Dave, Bjeorn,

Right now NumPy isn't installing at all on the Test Tool Shed.

Dave - is is possible that the system was updated and the
Fortran compiler removed? e.g.

...

Since this seems to be breaking lapack, perhaps we should
further simplify package_numpy_1_7 not to use lapack or
ATLAS? What do you think Bjoern?


I'm a little bit sceptic to remove features only because the Test
Utility can not handle it.
On the other hand I understand that numpy should just work and that your
test failing since month because of my dependencies. I will get rid of
it tomorrow. As a result we can setup_virtualenv and install numpy from
pip directly I think.

Is that ok for you?
Cheers,
Bjoern


Hi Bjoern,

I think the real problem here is we don't yet have a clear
base installation:
https://trello.com/c/gYlRdqwr/774-toolshed-define-a-base-set-of-utilities-that-can-be-expected-to-be-on-systems-into-which-tool-dependencies-are-being-installed
https://trello.com/c/7VTlX9rD/844-toolshed-tool-dependency-installation-requirements

What is worse is the system used by the ToolShed for
running the nightly tests appears to change without
notice, e.g. here it looks like something with the Fortran
compiler has changed, for the BLAST+ wrappers the
default shell seems to have changed between bash
and sh).

I think the ToolShed system should be a role model of
the agreed standard set of system level dependencies.

Given that, it seems your lapack package has some
dependencies which are not necessarily present, and
this failure is cascading up to break NumPy and then
Biopython, and thus multiple tools on the Tool Shed :(

I do think that in the short term the best option is to
  further simplify package_numpy_1_7 not to use
lapack or ATLAS - although this is not ideal :(

On the positive side, your hard work setting up packages
for NumPy, lapack and Altas is drawing attention to the
need for a clearer set of base packages. :)

Thanks,

Peter


___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Test Toolshed Biopython package dependency NumPy

2013-10-07 Thread Björn Grüning 
Am Montag, den 07.10.2013, 14:07 +0100 schrieb Peter Cock:
 On Tue, Oct 1, 2013 at 9:18 PM, Dave Bouvier d...@bx.psu.edu wrote:
  I've done some investigating, and it looks like there is an irregularity
  with the installation of package_numpy_1_7, and somehow
  package_biopython_1_62 is not finding the (successfully) installed numpy
  dependency. I duplicated the dependency chain for seq_select_by_id on my
  local tool shed, and got the following error for biopython's tool dependency
  record when installing seq_select_by_id into my local Galaxy instance:
 
  export
  PYTHONPATH=$PYTHONPATH:/var/galaxy/tool_dependencies/biopython/1.62/devteam/package_biopython_1_62/c3c8665cb0af/lib/python
  
  export PATH=$PATH:$PATH_NUMPY 
  export PYTHONPATH=$PYTHONPATH:$PYTHONPATH_NUMPY 
  python setup.py install --install-lib
  /var/galaxy/tool_dependencies/biopython/1.62/devteam/package_biopython_1_62/c3c8665cb0af/lib/python
 
  STDOUT
  running install
 
  Numerical Python (NumPy) is not installed.
 
  This package is required for many Biopython features.  Please install
  it before you install Biopython. You can install Biopython anyway, but
  anything dependent on NumPy will not work. If you do this, and later
  install NumPy, you should then re-install Biopython.
 
  You can find NumPy at http://numpy.scipy.org
 
  Note that when I installed seq_select_by_id, numpy installed correctly, and
  PYTHONPATH was populated with the right values.
 
 --Dave B.
 
 Hi Dave, Bjeorn,
 
 Right now NumPy isn't installing at all on the Test Tool Shed.
 
 Dave - is is possible that the system was updated and the
 Fortran compiler removed? e.g.
 
 http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_primer_clip/834cf6618a14
 reports:
 
 Installation errors - no functional tests were run for any tools in
 this changeset revision
 Tool dependencies
 TypeNameVersion
 lapack package 3.4.2
 Error
 CMake Error: your Fortran compiler: CMAKE_Fortran_COMPILER-NOTFOUND
 was not found. Please set CMAKE_Fortran_COMPILER to a valid compiler
 path or name. CMake Error at
 /usr/share/cmake-2.8/Modules/CMakeFortranInformation.cmake:27
 (get_filename_component): get_filename_component called with incorrect
 number of arguments Call Stack (most recent call first):
 CMakeLists.txt:2 (project) CMake Error: CMAKE_Fortran_COMPILER not
 set, after EnableLanguage CMake Error: Internal CMake error,
 TryCompile configure of cmake failed CMake Error at
 /usr/share/cmake-2.8/Modules/CMakeFortranInformation.cmake:27
 (get_filename_component): get_filename_component called with incorrect
 number of arguments Call Stack (most recent call first):
 CMakeLists.txt:2 (project) CMake Error: CMAKE_Fortran_COMPILER not
 set, after EnableLanguage CMake Error: Internal CMake error,
 TryCompile configure of cmake failed CMake Error at
 /usr/share/cmake-2.8/Modules/CMakeFortranInformation.cmake:27
 (get_filename_component): get_filename_component called with incorrect
 number of arguments Call Stack (most recent call first):
 CMakeLists.txt:2 (project) CMake Error: CMAKE_Fortran_COMPILER not
 set, after EnableLanguage CMake Error: Internal CMake error,
 TryCompile configure of cmake failed CMake Error at
 /usr/share/cmake-2.8/Modules/CMakeFortranInformation.cmake:27
 (get_filename_component): get_filename_component called with incorrect
 number of arguments Call Stack (most recent call first):
 CMakeLists.txt:2 (project) CMake Error: CMAKE_Fortran_COMPILER not
 set, after EnableLanguage CMake Error: Internal CMake error,
 TryCompile configure of cmake failed CMake Error at
 /usr/share/cmake-2.8/Modules/CMakeFortranInformation.cmake:27
 (get_filename_component): get_filename_component called with incorrect
 number of arguments Call Stack (most recent call first):
 CMakeLists.txt:2 (project) CMake Error: CMAKE_Fortran_COMPILER not
 set, after EnableLanguage CMake Error: Internal CMake error,
 TryCompile configure of cmake failed
 
 Since this seems to be breaking lapack, perhaps we should
 further simplify package_numpy_1_7 not to use lapack or
 ATLAS? What do you think Bjoern?

I'm a little bit sceptic to remove features only because the Test
Utility can not handle it.
On the other hand I understand that numpy should just work and that your
test failing since month because of my dependencies. I will get rid of
it tomorrow. As a result we can setup_virtualenv and install numpy from
pip directly I think.

Is that ok for you?
Cheers,
Bjoern

 http://toolshed.g2.bx.psu.edu/view/iuc/package_numpy_1_7
 http://testtoolshed.g2.bx.psu.edu/view/iuc/package_numpy_1_7
 
 Regards,
 
 Peter



___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:

Re: [galaxy-dev] Test Toolshed Biopython package dependency NumPy

2013-10-01 Thread Dave Bouvier 
I've done some investigating, and it looks like there is an irregularity 
with the installation of package_numpy_1_7, and somehow 
package_biopython_1_62 is not finding the (successfully) installed numpy 
dependency. I duplicated the dependency chain for seq_select_by_id on my 
local tool shed, and got the following error for biopython's tool 
dependency record when installing seq_select_by_id into my local Galaxy 
instance:


export 
PYTHONPATH=$PYTHONPATH:/var/galaxy/tool_dependencies/biopython/1.62/devteam/package_biopython_1_62/c3c8665cb0af/lib/python 


export PATH=$PATH:$PATH_NUMPY 
export PYTHONPATH=$PYTHONPATH:$PYTHONPATH_NUMPY 
python setup.py install --install-lib 
/var/galaxy/tool_dependencies/biopython/1.62/devteam/package_biopython_1_62/c3c8665cb0af/lib/python


STDOUT
running install

Numerical Python (NumPy) is not installed.

This package is required for many Biopython features.  Please install
it before you install Biopython. You can install Biopython anyway, but
anything dependent on NumPy will not work. If you do this, and later
install NumPy, you should then re-install Biopython.

You can find NumPy at http://numpy.scipy.org

Note that when I installed seq_select_by_id, numpy installed correctly, 
and PYTHONPATH was populated with the right values.


   --Dave B.

On 09/30/2013 09:10 AM, Bjoern Gruening wrote:

Am Montag, den 30.09.2013, 14:02 +0100 schrieb Peter Cock:

On Mon, Sep 30, 2013 at 1:58 PM, Dave Bouvier d...@bx.psu.edu wrote:

Peter,

It looks like there may be a version conflict between package_numpy and
either a system numpy or the Galaxy numpy egg. I'll investigate and keep you
updated on my progress.

--Dave B.


Thanks Dave,

If Tool authors can assume NumPy is already present via the Galaxy
numpy egg and/or the system Python's numpy, that might be simpler...


The question is how to import the eggs into the tools? Is that possible
at all? For example pysam is shipped with Galaxy right? But is there a
way to use it inside of a tool?
But I think even if that is possible we need separate pysam/numpy
repositories for reproducibility.


I agree with Björn, it's probably not a good idea to use Galaxy eggs in 
tools and their dependencies, since the egg versions can change at any 
time, which might impact reproducibilty.




Cheers,
Bjoern


Peter
___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/





___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Test Toolshed Biopython package dependency NumPy

2013-09-30 Thread Peter Cock 
On Sun, Sep 29, 2013 at 1:17 PM, Björn Grüning wrote:
 I updated R, numpy, scipy, scikit and removed the atlas dependency. It
 seems to work fine for the ChemicalToolBox. I do not remove the lapack
 dependency, because I did not get any complains until now.

I'm seeing problems on the Test Tool Shed with NumPy :(

The updated no-ATLAS numpy is revision f0490401cfff
http://testtoolshed.g2.bx.psu.edu/view/iuc/package_numpy_1_7

My tools using NumPy via Biopython 1.62 do seem to be
picking up this new NumPy package, which is good.

However, the install of NumPy seems to be failing in strange
and *different* ways - which I suspect points to a Tool Shed
test framework bug, perhaps in sharing the dependencies
between tools?

--

http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_select_by_id /9e9e3b860aa0
Looks like partial NumPy install?

Installation errors - no functional tests were run for any tools in
this changeset revision
Tool dependencies
TypeNameVersion
biopython package 1.62
Error
Traceback (most recent call last): File setup.py, line 463, in
module setup(**setup_args) File
/usr/lib/python2.7/distutils/core.py, line 152, in setup
dist.run_commands() File /usr/lib/python2.7/distutils/dist.py, line
953, in run_commands self.run_command(cmd) File
/usr/lib/python2.7/distutils/dist.py, line 972, in run_command
cmd_obj.run() File setup.py, line 216, in run if
check_dependencies_once(): File setup.py, line 123, in
check_dependencies_once _CHECKED = check_dependencies() File
setup.py, line 163, in check_dependencies if is_Numpy_installed():
File setup.py, line 282, in is_Numpy_installed return
bool(can_import(numpy)) File setup.py, line 274, in can_import
return __import__(module_name) File
/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/__init__.py,
line 137, in module import add_newdocs File
/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/add_newdocs.py,
line 9, in module from numpy.lib import add_newdoc File
/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/lib/__init__.py,
line 13, in module from polynomial import * File
/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/lib/polynomial.py,
line 11, in module import numpy.core.numeric as NX AttributeError:
'module' object has no attribute 'core'

--

http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_select_by_id/9e9e3b860aa0
Apparently the same error:

Installation errors - no functional tests were run for any tools in
this changeset revision
Tool dependencies
TypeNameVersion
biopython package 1.62
Error
Traceback (most recent call last): File setup.py, line 463, in
module setup(**setup_args) File
/usr/lib/python2.7/distutils/core.py, line 152, in setup
dist.run_commands() File /usr/lib/python2.7/distutils/dist.py, line
953, in run_commands self.run_command(cmd) File
/usr/lib/python2.7/distutils/dist.py, line 972, in run_command
cmd_obj.run() File setup.py, line 216, in run if
check_dependencies_once(): File setup.py, line 123, in
check_dependencies_once _CHECKED = check_dependencies() File
setup.py, line 163, in check_dependencies if is_Numpy_installed():
File setup.py, line 282, in is_Numpy_installed return
bool(can_import(numpy)) File setup.py, line 274, in can_import
return __import__(module_name) File
/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/__init__.py,
line 137, in module import add_newdocs File
/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/add_newdocs.py,
line 9, in module from numpy.lib import add_newdoc File
/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/lib/__init__.py,
line 13, in module from polynomial import * File
/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/lib/polynomial.py,
line 11, in module import numpy.core.numeric as NX AttributeError:
'module' object has no attribute 'core'

--

http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_rename/d08767ae73ce
Looks like a different partial NumPy install?

Installation errors - no functional tests were run for any tools in
this changeset revision
Tool dependencies
TypeNameVersion
biopython package 1.62
Error
Traceback (most recent call last): File setup.py, line 393, in
module if is_Numpy_installed(): File setup.py, line 282, in
is_Numpy_installed return bool(can_import(numpy)) File setup.py,
line 274, in can_import return __import__(module_name) File
/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/__init__.py,
line 137, in module import add_newdocs File
/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/add_newdocs.py,
line 9, in module from numpy.lib import add_newdoc File
/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/lib/__init__.py,
line 4, in module from

Re: [galaxy-dev] Test Toolshed Biopython package dependency NumPy

2013-09-30 Thread Bjoern Gruening 
Hi,

it does not mean anything, but for me its working locally on a Fedora
(stable) machine.

If I can anything do to fix that, please let me know.
Bjoern

 On Sun, Sep 29, 2013 at 1:17 PM, Björn Grüning wrote:
  I updated R, numpy, scipy, scikit and removed the atlas dependency. It
  seems to work fine for the ChemicalToolBox. I do not remove the lapack
  dependency, because I did not get any complains until now.
 
 I'm seeing problems on the Test Tool Shed with NumPy :(
 
 The updated no-ATLAS numpy is revision f0490401cfff
 http://testtoolshed.g2.bx.psu.edu/view/iuc/package_numpy_1_7
 
 My tools using NumPy via Biopython 1.62 do seem to be
 picking up this new NumPy package, which is good.
 
 However, the install of NumPy seems to be failing in strange
 and *different* ways - which I suspect points to a Tool Shed
 test framework bug, perhaps in sharing the dependencies
 between tools?
 
 --
 
 http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_select_by_id /9e9e3b860aa0
 Looks like partial NumPy install?
 
 Installation errors - no functional tests were run for any tools in
 this changeset revision
 Tool dependencies
 TypeNameVersion
 biopython package 1.62
 Error
 Traceback (most recent call last): File setup.py, line 463, in
 module setup(**setup_args) File
 /usr/lib/python2.7/distutils/core.py, line 152, in setup
 dist.run_commands() File /usr/lib/python2.7/distutils/dist.py, line
 953, in run_commands self.run_command(cmd) File
 /usr/lib/python2.7/distutils/dist.py, line 972, in run_command
 cmd_obj.run() File setup.py, line 216, in run if
 check_dependencies_once(): File setup.py, line 123, in
 check_dependencies_once _CHECKED = check_dependencies() File
 setup.py, line 163, in check_dependencies if is_Numpy_installed():
 File setup.py, line 282, in is_Numpy_installed return
 bool(can_import(numpy)) File setup.py, line 274, in can_import
 return __import__(module_name) File
 /ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/__init__.py,
 line 137, in module import add_newdocs File
 /ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/add_newdocs.py,
 line 9, in module from numpy.lib import add_newdoc File
 /ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/lib/__init__.py,
 line 13, in module from polynomial import * File
 /ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/lib/polynomial.py,
 line 11, in module import numpy.core.numeric as NX AttributeError:
 'module' object has no attribute 'core'
 
 --
 
 http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_select_by_id/9e9e3b860aa0
 Apparently the same error:
 
 Installation errors - no functional tests were run for any tools in
 this changeset revision
 Tool dependencies
 TypeNameVersion
 biopython package 1.62
 Error
 Traceback (most recent call last): File setup.py, line 463, in
 module setup(**setup_args) File
 /usr/lib/python2.7/distutils/core.py, line 152, in setup
 dist.run_commands() File /usr/lib/python2.7/distutils/dist.py, line
 953, in run_commands self.run_command(cmd) File
 /usr/lib/python2.7/distutils/dist.py, line 972, in run_command
 cmd_obj.run() File setup.py, line 216, in run if
 check_dependencies_once(): File setup.py, line 123, in
 check_dependencies_once _CHECKED = check_dependencies() File
 setup.py, line 163, in check_dependencies if is_Numpy_installed():
 File setup.py, line 282, in is_Numpy_installed return
 bool(can_import(numpy)) File setup.py, line 274, in can_import
 return __import__(module_name) File
 /ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/__init__.py,
 line 137, in module import add_newdocs File
 /ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/add_newdocs.py,
 line 9, in module from numpy.lib import add_newdoc File
 /ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/lib/__init__.py,
 line 13, in module from polynomial import * File
 /ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/lib/polynomial.py,
 line 11, in module import numpy.core.numeric as NX AttributeError:
 'module' object has no attribute 'core'
 
 --
 
 http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_rename/d08767ae73ce
 Looks like a different partial NumPy install?
 
 Installation errors - no functional tests were run for any tools in
 this changeset revision
 Tool dependencies
 TypeNameVersion
 biopython package 1.62
 Error
 Traceback (most recent call last): File setup.py, line 393, in
 module if is_Numpy_installed(): File setup.py, line 282, in
 is_Numpy_installed return bool(can_import(numpy)) File setup.py,
 line 274, in can_import return __import__(module_name) File
 /ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/__init__.py,
 line 137, in module import add_newdocs File

Re: [galaxy-dev] Test Toolshed Biopython package dependency NumPy

2013-09-30 Thread Peter Cock 
On Mon, Sep 30, 2013 at 12:23 PM, Bjoern Gruening
bjoern.gruen...@gmail.com wrote:
 Hi,

 it does not mean anything, but for me its working locally on a Fedora
 (stable) machine.

 If I can anything do to fix that, please let me know.
 Bjoern

Thanks Bjoern - I'm pretty sure this is a Tool Shed issue,
rather than the Biopython or NumPy package definitions
themselves.

Peter
___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Test Toolshed Biopython package dependency NumPy

2013-09-30 Thread Bjoern Gruening 
Am Montag, den 30.09.2013, 14:02 +0100 schrieb Peter Cock:
 On Mon, Sep 30, 2013 at 1:58 PM, Dave Bouvier d...@bx.psu.edu wrote:
  Peter,
 
  It looks like there may be a version conflict between package_numpy and
  either a system numpy or the Galaxy numpy egg. I'll investigate and keep you
  updated on my progress.
 
 --Dave B.
 
 Thanks Dave,
 
 If Tool authors can assume NumPy is already present via the Galaxy
 numpy egg and/or the system Python's numpy, that might be simpler...

The question is how to import the eggs into the tools? Is that possible
at all? For example pysam is shipped with Galaxy right? But is there a
way to use it inside of a tool?
But I think even if that is possible we need separate pysam/numpy
repositories for reproducibility.

Cheers,
Bjoern

 Peter
 ___
 Please keep all replies on the list by using reply all
 in your mail client.  To manage your subscriptions to this
 and other Galaxy lists, please use the interface at:
   http://lists.bx.psu.edu/
 
 To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/



___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/