[easybuild] error when using requirements file for Pythonpackage

[Arnau]

Hi again,

Trying to install a python package from a requirements.txt using
use_pip_requirement . But pip fails with the error:

Collecting pyzmq>=17
  Downloading pyzmq-24.0.1.tar.gz (1.2 MB)
  1.2/1.2 MB 100.5 MB/s eta 0:00:00
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'error'
  error: subprocess-exited-with-error
[...]
File
"/apps/prod/easybuild/sl7.x86_64.foss-2021a/software/Python/3.11.1-GCCcore-10.3.0-bare/lib/python3.11/site-packages/setuptools/build_meta.py",
line 335, in run_setup
  exec(code, locals())
File "", line 34, in 
  ModuleNotFoundError: No module named 'packaging'
  [end of output]


the pip install command that EB builds is:


/apps/prod/easybuild/sl7.x86_64.foss-2021a/software/Python/3.11.1-GCCcore-10.3.0-bare/bin/python
-m pip install
--prefix=/apps/prod/easybuild/sl7.x86_64.foss-2021a/software/Python/3.11.1-GCCcore-10.3.0-jupyter
--ignore-installed  --no-build-isolation  --requirement
/home/x2bioc/git/easyconfigs/Python/requirements.txt


which I ran outside easybuild and works perfectly (I loaded Python 3.11 and
ran tha bove command).

my question is: what is EB doing differently in its environment to the pip
command? Am I missing any option

My easyconfig looks like this:

easyblock = 'PythonPackage'

name = 'Python'

versionsuffix = '-jupyter'
version = '3.11.1'

homepage = 'https://python.org/'
description = """Python is a programming language that lets you work more
quickly and integrate your systems
 more effectively."""

toolchain = {'name': 'GCCcore', 'version': '10.3.0'}
toolchainopts = {'pic': True}

builddependencies = [
('UnZip', '6.0'),
]

dependencies = [
('Python', '3.11.1', '-bare'),
]

use_pip= True
sanity_pip_check= False
runtest= False
sources=['requirements.txt']
use_pip_requirement= True
unpack_sources = False
use_pip_for_deps = True


TIA,
Arnau

[easybuild] [ANN] EasyBuild v4.7.0

[Kenneth Hoste]

Dear EasyBuilders,

I'm happy to announce the release of EasyBuild v4.7.0 [1].

Since the start of 2023, we have managed to hold back on releasing 
EasyBuild v4.7.0, but we can't keep it from you any longer, since this 
beyond any doubt is the best EasyBuild release to date.


Hopefully you did not fall for the silly attempts to distract you from 
this release, like some open source software projects making a lot of 
fuss of something trivial like a new logo, or people celebrating just 
another lap around the sun for some reason...



The executive summary for EasyBuild v4.7.0:

  - running EasyBuild on top of Python 2 is deprecated;

  - various minor bug fixes and enhancements to EasyBuild framework, 
including support for generating documentation in MarkDown format;


  - enhancements and changes to the (still experimental) support for 
easystack files;


  - two new generic easyblocks, for installing (bundles of) Julia 
packages: JuliaPackage and JuliaBundle;


  - various minor bug fixes and enhancements to existing easyconfigs 
and easyblocks;


  - easyconfigs for the 2022b update of the common toolchains: 
foss/2022b and intel/2022b;


  - support for installing 107 (!) new software applications and 
libraries, and updated versions of already supported software;




EasyBuild v4.7.0 is primarily a feature release, but it also includes 
various minor bug fixes and enhancements.


Highlights for this release are listed below. More details are available 
in the release notes [2] which includes links to the respective pull 
requests for more detailed information.


(this information is also available at 
https://github.com/easybuilders/easybuild/releases/tag/easybuild-v4.7.0)



## Highlighted enhancements

[enhancements that (may) warrant updating existing installations are 
marked with (***)]


- also run unit tests with Python 3.11;

- add support for checksums specified in external checksums.json file;

- take into account custom configuration options specified in easystack 
files - see also https://docs.easybuild.io/en/latest/Easystack-files.html;


- add support for using --output-format=md (MarkDown);

- add support for postinstallmsgs;

- add gfbf as subtoolchain of foss;

- make iimkl toolchain aware of intel-compilers;

- add generic easyblocks for installing (bundle of) Julia packages: 
JuliaPackage + JuliaBundle;


- updates and enhancements for various generic easyblocks, incl. 
ConfigureMake, PythonPackage;


- (***) updates and enhancements for various software-specific 
easyblocks: BerkeleyGW, Clang, Clang-AOMP, ESMF, libxml2, LLVM, OpenCV, 
OpenMPI, PETSc, QScintill, TensorFlow, Xmipp;



## Prominent bug fixes & changes

[bug fixes or changes that (may) warrant reinstalling easyconfigs are 
marked with (***)]


- print deprecation warning with running EasyBuild with Python 2;

- various tweaks to docstrings and help messages to fix problems with 
auto-generated documentation in MarkDown format;


- vendor distutils.version.LooseVersion as easybuild.tools.LooseVersion 
(since distutils is deprecated in Python 3.10, to be removed in Python 
3.12);


- auto-enable use of oneAPI C/C++ compilers for intel-compilers >= 2022.2.0;

- use -march=x86-64 -mtune=generic instead of -xSSE2 when using Intel 
oneAPI compilers;


- drop support for easystack files using 'software' top-level key;

- make PythonPackage easyblock compatible with --sanity-check-only by 
loading module early during sanity check step;


- (***) update HDF5 easyblock to use --enable-threadsafe configure 
option to make C API thread safe;


- (***) bug fixes for various software-specific easyblocks: ANSYS, 
Clang, MRtrix, PyTorch;



## Supported software


New version of common toolchains: foss/2022b, intel/2022b

Support for installing 107 new software packages has been added, 
including (but not limited to):


  Alfred
  bamFilters
  CAT-BAT
  CloudCompare
  COBRApy
  CUDA-Samples
  DeepLabCut
  DRAGMAP
  ESM-2
  HighFive
  IJulia
  KaHIP
  LHAPDF
  Magics
  MEMOTE
  MONAI
  MOOSE
  MultilevelEstimators
  napari
  nf-core-mag
  OmegaFold
  pyccel
  PyDamage
  pyWannier90
  scvi-tools
  SELFIES
  SISSO++
  spaCy
  Squidpy
  TBA
  trimesh
  UCX-ROCm

  => This brings the total number of supported software packages to 
2,904 (excluding extensions)!

 An up-to-date list of supported software is available at [3].


Various software updates have been added, including (but not limited to):

  AOCC 4.0.0
  CDO 2.1.1
  CubeGUI/CubeLib/CubeWriter 4.8
  CUDA 11.8.0 + 12.0.0
  deepdiff 5.8.1
  FLANN 1.9.1
  GATK 4.3.0.0
  GDCM 3.0.20
  Ghostscript 10.0.0
  GRASS 8.2.0
  JasPer 4.0.0
  libspatialite 5.0.1
  LLVM 15.0.5
  Mesa 22.2.4
  mold 1.7.1
  MRChem 1.1.1
  ncbi-vdb 3.0.0
  nglview 3.0.3
  nodejs 18.12.1
  Octave 7.1.0
  OpenFOAM 10
  OpenFold 1.0.1
  OpenMolcas 22.10
  OpenStackClient 6.0.0
  PAPI 7.0.0,
  PETSc 3.17.4
  pyproj 3.4.0
  Python 3.10.8
  PyTorch 1.12.1
  PyTorch-Geometric 2.1.0
  QGIS 3.28.1
  Qt5 5.15.7
  

Re: [easybuild] PythonBundle and use_pip_requirement

[Kenneth Hoste]

Hi Arnau,

When using PythonBundle, you should specify 'sources' for specific 
extensions in exts_list (not "top-level").



regards,

Kenneth

On 09/01/2023 14:53, Arnau wrote:

Dear all,

I'd like to install a bunch of python packages from a requirements.txt 
instead of listing each of them as extension. So I'd like to use the 
use_pip_requirement option.


According to the doc :

install using ‘python -m pip install –requirement’. The sources is 
expected to be the requirements file.


but as soon as I add the sources to point to the requirements file EB 
complains with the error:


" List of sources for bundle itself must be empty, found 
['requirements.txt']"


If I do not add any sources at all, it does not install anythings at all.

so, what is the correct way of using this option in a PythonBundle?

(eb 4.6.2)

TIA,
Arna

Re: [easybuild] PythonBundle and use_pip_requirement

[Maxime Boissonneault]

Hi Arna,

I think this option is only supported for PythonPackage. This is where 
we use it, for example:


https://github.com/ComputeCanada/easybuild-easyconfigs/blob/computecanada-main/easybuild/easyconfigs/s/SciPy-Stack/SciPy-Stack-2022a-GCCcore-9.3.0.eb

Best,

Maxime Boissonneault


Le 2023-01-09 à 08:53, Arnau a écrit :

Dear all,

I'd like to install a bunch of python packages from a requirements.txt 
instead of listing each of them as extension. So I'd like to use the 
use_pip_requirement option.


According to the doc :

install using ‘python -m pip install –requirement’. The sources is 
expected to be the requirements file.


but as soon as I add the sources to point to the requirements file EB 
complains with the error:


" List of sources for bundle itself must be empty, found 
['requirements.txt']"


If I do not add any sources at all, it does not install anythings at all.

so, what is the correct way of using this option in a PythonBundle?

(eb 4.6.2)

TIA,
Arna

[easybuild] PythonBundle and use_pip_requirement

[Arnau]

Dear all,

I'd like to install a bunch of python packages from a requirements.txt
instead of listing each of them as extension. So I'd like to use the
use_pip_requirement option.

According to the doc :

install using ‘python -m pip install –requirement’. The sources is expected
to be the requirements file.

but as soon as I add the sources to point to the requirements file EB
complains with the error:

" List of sources for bundle itself must be empty, found
['requirements.txt']"

If I do not add any sources at all, it does not install anythings at all.

so, what is the correct way of using this option in a PythonBundle?

(eb 4.6.2)

TIA,
Arna