[easybuild] easy_update

[Dey, John F]

Easy_update has been updated for Python usage. Easy_update updates the 
exts_list for Python EasyBuild configs. The program is in the scripts directory 
of our easybuild repository:   
https://github.com/FredHutch/easybuild-life-sciences


-  Updated to use Pypi.org

-  Updates checksums

-  Easy_update has improved meta data debugging for Python modules.

-  search feature to locate single modules and output is in format for 
exts_list

-  R update features are broken for now, but Python was a necessity.


Search for current version of iPython:
easy_update.py --search iPython --pyver 3.7 --meta
('iPython', '6.4.0', {
'source_urls': ['PYPI_SOURCE'],
'checksums': 
['eca537aa61592aca2fef4adea12af8e42f5c335004dfa80c78caf80e8b525e5c'],
}),
'filename': 'ipython-6.4.0.tar.gz'
'packagetype': 'sdist'
'url': 
'https://files.pythonhosted.org/packages/ee/01/2a85cd07f5a43fa2e86d60001c213647252662d44a0c2e3d69471a058f1b/ipython-6.4.0.tar.gz'
'python_version': 'source'
'requires_dist': '[u'numpy; python_version >= "3.4" and extra == \'test\'', 
u"ipykernel; extra == 'test'", u"nbformat; extra == 'test'", u"pygments; extra 
== 'test'", u"testpath; extra == 'test'", u"requests; extra == 'test'", u"nose 
(>=0.10.1); extra== 'test'", u"qtconsole; extra == 'qtconsole'", u"ipyparallel; 
extra == 'parallel'", u"ipywidgets; extra == 'notebook'", u"notebook; extra == 
'notebook'", u"nbformat; extra == 'nbformat'", u"nbconvert; extra == 
'nbconvert'", u"ipykernel; extra == 'kernel'", u"Sphinx (>=1.3); extra == 
'doc'", u"pygments; extra == 'all'", u"nbconvert; extra == 'all'", 
u"ipywidgets; extra == 'all'", u"qtconsole; extra == 'all'", u"testpath; extra 
== 'all'", u"Sphinx (>=1.3); extra == 'all'", u"requests; extra == 'all'", 
u"ipyparallel; extra == 'all'", u"nose (>=0.10.1); extra == 'all'", u"notebook; 
extra == 'all'", u"ipykernel; extra == 'all'", u"nbformat; extra == 'all'", 
u'win-unicode-console (>=0.5); sys_platform == "win32" and python_version < 
"3.6"', u'colorama; sys_platform == "win32"', u'appnope; sys_platform == 
"darwin"', u'pexpect; sys_platform != "win32"', u'pathlib2; python_version == 
"3.3"', u'typing; python_version <= "3.4"', u'backcall', u'pygments', 
u'prompt-toolkit (<2.0.0,>=1.0.15)', u'traitlets (>=4.2)', u'simplegeneric 
(>0.8)', u'pickleshare', u'decorator', u'jedi (>=0.10)', u'setuptools 
(>=18.5)']'
'requires_python': '>=3.3'


Example run with the current Python 3.7.0 Easyconfig:
./easy_update.py --verbose Python-3.7.0-foss-2018b.eb
versionsuffix not defined
  setuptools : 39.2.0 -> 40.0.0 (update) [42, 1]
 pip : 10.0.1 (keep) [42, 2]
nose : 1.3.7  (keep) [42, 3]
   numpy : 1.14.5 (keep) [42, 4]
   scipy : 1.1.0  (keep) [42, 5]
   blist : 1.3.6  (keep) [42, 6]
  mpi4py : 3.0.0  (keep) [42, 7]
paycheck : 1.0.2  (keep) [42, 8]
 pbr : 4.0.4 -> 4.1.0   (update) [42, 9]
  Cython : 0.28.3 -> 0.28.4 (update) [42, 10]
 six : 1.11.0 (keep) [42, 11]
 python-dateutil : 2.7.3  (keep) [42, 12]
deap : 1.2.2  (keep) [42, 13]
   decorator : 4.3.0  (keep) [42, 14]
   liac-arff : 2.2.2  (keep) [42, 15]
pycrypto : 2.6.1  (keep) [42, 16]
   ecdsa : 0.13   (keep) [42, 17]
   pycparser : 2.18   (keep) [42, 18]
cffi : 1.11.5 (keep) [42, 19]
  asn1crypto : 0.24.0 (keep) [42, 20]
idna : 2.7(keep) [42, 21]
cryptography : 2.2.2  (keep) [42, 22]
  pyasn1 : 0.4.3  (keep) [42, 23]
  PyNaCl : 1.2.1  (keep) [42, 24]
  bcrypt : 3.1.4  (keep) [42, 25]
paramiko : 2.4.1  (keep) [42, 26]
   pyparsing : 2.2.0  (keep) [42, 27]
   netifaces : 0.10.7 (keep) [42, 28]
 netaddr : 0.7.19 (keep) [42, 29]
mock : 2.0.0  (keep) [42, 30]
pytz : 2018.4 -> 2018.5 (update) [42, 31]
  pandas : 0.23.3 (keep) [42, 32]
   bitstring : 3.1.5  (keep) [42, 33]
  virtualenv : 16.0.0 (keep) [42, 34]
  docopt : 0.6.2  (keep) [42, 35]
  joblib : 0.11 -> 0.12.0   (update) [42, 36]
 chardet : 

Re: [easybuild] Python-3.7.0 No module named '_ssl'

[Dey, John F]

Thanks, Openss version does seem to be the issue. My openssl version is bellow 
the minimum.

John Dey

From:  on behalf of Mikael Öhman 

Reply-To: "easybuild@lists.ugent.be" 
Date: Friday, July 13, 2018 at 7:24 AM
To: "easybuild@lists.ugent.be" 
Subject: Re: [easybuild] Python-3.7.0 No module named '_ssl'

Hej John,
Isn't this what happens to do old OpenSSL on CentOS6(or similar)? I don't 
recall the exact error I got myself, but I think this was it.
You have to modify the script to also build an EB-supplied OpenSSL when the 
OS-depenency is to old (there are some comments in the config).

Best regards, Mikael

On Fri, Jul 13, 2018 at 2:53 PM Dey, John F 
mailto:jf...@fredhutch.org>> wrote:
Using EasyBuild 3.6.0 and Python-3.7.0 easyconfig from github. All modules 
build without issue but sanity check fails.

== FAILED: Installation ended unsuccessfully (build directory: 
/app/easybuild/build/Python/3.7.0/foss-2016b):build failed (first 300 chars): 
Sanity check failed: sanity check command python -c 'import _ssl' exited 
withcode 1 (output: Traceback (most recent call last):
  File "", line 1, in 
ModuleNotFoundError: No module named '_ssl'
)


--
John Dey
HPC Operations
Scientific Computing
O 206.667.4308
M 360.649.2731
E jf...@fredhutch.org

[Users/john/Library/Containers/com.microsoft.Out]
Fred Hutchinson Cancer Research Center
1100 Fairview Ave. 
N.,
 Mail Stop J3-516
Seattle, WA 98109
fredhutch.org

Re: [easybuild] Python-3.7.0 No module named '_ssl'

[Mikael Öhman]

Hej John,
Isn't this what happens to do old OpenSSL on CentOS6(or similar)? I don't
recall the exact error I got myself, but I think this was it.
You have to modify the script to also build an EB-supplied OpenSSL when the
OS-depenency is to old (there are some comments in the config).

Best regards, Mikael

On Fri, Jul 13, 2018 at 2:53 PM Dey, John F  wrote:

> Using EasyBuild 3.6.0 and Python-3.7.0 easyconfig from github. All modules
> build without issue but sanity check fails.
>
>
>
> == FAILED: Installation ended unsuccessfully (build directory:
> /app/easybuild/build/Python/3.7.0/foss-2016b):build failed (first 300
> chars): Sanity check failed: sanity check command python -c 'import _ssl'
> exited withcode 1 (output: Traceback (most recent call last):
>
>   File "", line 1, in 
>
> ModuleNotFoundError: No module named '_ssl'
>
> )
>
>
>
>
>
> --
>
> *John Dey*
>
> *HPC Operations*
>
> Scientific Computing
> *O* *206.667.4308 <(206)%20667-4308>*
> *M* *360.649.2731 <(360)%20649-2731>*
>
> *E* jf...@fredhutch.org
>
>
>
> [image:
> /Users/john/Library/Containers/com.microsoft.Outlook/Data/Library/Caches/Signatures/signature_1878591012]
> *Fred Hutchinson Cancer Research Center*
> 1100 Fairview Ave. N
> .,
> Mail Stop J3-516
> Seattle, WA 98109
> *fredhutch.org *
>
>
>
>
>

[easybuild] RE: Python-3.7.0 No module named '_ssl'

[Vanzo, Davide]

John,

Could you please attach the full log?

--
Davide Vanzo, PhD
Application Developer
Adjunct Assistant Professor of Chemical and Biomolecular Engineering
Advanced Computing Center for Research and Education (ACCRE)
Vanderbilt University - Hill Center 201
(615)-875-9137
www.vanderbilt.edu/accre


On 2018-07-13 07:53:56-05:00 easybuild-requ...@lists.ugent.be wrote:
Using EasyBuild 3.6.0 and Python-3.7.0 easyconfig from github. All modules 
build without issue but sanity check fails.

== FAILED: Installation ended unsuccessfully (build directory: 
/app/easybuild/build/Python/3.7.0/foss-2016b):build failed (first 300 chars): 
Sanity check failed: sanity check command python -c 'import _ssl' exited 
withcode 1 (output: Traceback (most recent call last):
  File "", line 1, in 
ModuleNotFoundError: No module named '_ssl'
)


--
John Dey
HPC Operations
Scientific Computing
O 206.667.4308
M 360.649.2731
E jf...@fredhutch.org

[/Users/john/Library/Containers/com.microsoft.Outlook/Data/Library/Caches/Signatures/signature_1878591012]
Fred Hutchinson Cancer Research Center
1100 Fairview Ave. 
N.,
 Mail Stop J3-516
Seattle, WA 98109
fredhutch.org

[easybuild] Python-3.7.0 No module named '_ssl'

[Dey, John F]

Using EasyBuild 3.6.0 and Python-3.7.0 easyconfig from github. All modules 
build without issue but sanity check fails.

== FAILED: Installation ended unsuccessfully (build directory: 
/app/easybuild/build/Python/3.7.0/foss-2016b):build failed (first 300 chars): 
Sanity check failed: sanity check command python -c 'import _ssl' exited 
withcode 1 (output: Traceback (most recent call last):
  File "", line 1, in 
ModuleNotFoundError: No module named '_ssl'
)


--
John Dey
HPC Operations
Scientific Computing
O 206.667.4308
M 360.649.2731
E jf...@fredhutch.org

[/Users/john/Library/Containers/com.microsoft.Outlook/Data/Library/Caches/Signatures/signature_1878591012]
Fred Hutchinson Cancer Research Center
1100 Fairview Ave. 
N., Mail 
Stop J3-516
Seattle, WA 98109
fredhutch.org

Re: [easybuild] PyPi legacy is gone.... and ideas on source_urls for Python extensions

[Jack Perdue]

Jack Perdue
Lead Systems Administrator
High Performance Research Computing
TAMU Division of Research
j-per...@tamu.eduhttp://hprc.tamu.edu
HPRC Helpdesk: h...@hprc.tamu.edu

On 07/12/2018 02:13 PM, Bart Oldeman wrote:

Hi Jack,

I think what Kenneth was trying to say is that you need to use:
'source_urls': ['https://pypi.python.org/packages/source/i/ipython/' 
],


I didn't get that impression, given that the URL no longer exist.

== 2018-07-13 06:59:18,068 filetools.py:476 WARNING URL 
https://pypi.python.org/packages/source/i/ipython/ipython-6.4.tar.gz was 
not found (HTTP response code 404), not trying again
== 2018-07-13 06:59:18,069 filetools.py:494 WARNING Download of 
https://pypi.python.org/packages/source/i/ipython/ipython-6.4.tar.gz to 
/sw/eb/sources/i/IPython/extensions/ipython-6.4.tar.gz failed, done trying


instead since PYPI_LOWER_SOURCE does not work for extensions.


Is that indeed still the case?  I see some old issues in the EasyBuild 
framework

e.g.

https://github.com/easybuilders/easybuild-framework/issues/1597

(there are more)

In any case:

1) its broken

2) someone with more than a clue than I should fix it

Otherwise, I would think just about all Python extensions
from PyPi are now broken given non-existent URLs
(due to the demise of PyPi legacy).   I'm surprised I'm
the only one that has this issue.

jack



--
Dr. Bart E. Oldeman | bart.olde...@mcgill.ca 
 | bart.olde...@calculquebec.ca 


Scientific Computing Analyst / Analyste en calcul scientifique
McGill HPC Centre / Centre de Calcul Haute Performance de McGill | 
http://www.hpc.mcgill.ca 

Calcul Québec | http://www.calculquebec.ca 

Compute/Calcul Canada | http://www.computecanada.ca 


Tel/Tél: 514-396-8926 | Fax/Télécopieur: 514-396-8934