Re: [easybuild] PyPi legacy is gone.... and ideas on source_urls for Python extensions

[Jack Perdue]

Jack Perdue
Lead Systems Administrator
High Performance Research Computing
TAMU Division of Research
j-per...@tamu.eduhttp://hprc.tamu.edu
HPRC Helpdesk: h...@hprc.tamu.edu

On 07/12/2018 02:13 PM, Bart Oldeman wrote:

Hi Jack,

I think what Kenneth was trying to say is that you need to use:
'source_urls': ['https://pypi.python.org/packages/source/i/ipython/' 
],


I didn't get that impression, given that the URL no longer exist.

== 2018-07-13 06:59:18,068 filetools.py:476 WARNING URL 
https://pypi.python.org/packages/source/i/ipython/ipython-6.4.tar.gz was 
not found (HTTP response code 404), not trying again
== 2018-07-13 06:59:18,069 filetools.py:494 WARNING Download of 
https://pypi.python.org/packages/source/i/ipython/ipython-6.4.tar.gz to 
/sw/eb/sources/i/IPython/extensions/ipython-6.4.tar.gz failed, done trying


instead since PYPI_LOWER_SOURCE does not work for extensions.


Is that indeed still the case?  I see some old issues in the EasyBuild 
framework

e.g.

https://github.com/easybuilders/easybuild-framework/issues/1597

(there are more)

In any case:

1) its broken

2) someone with more than a clue than I should fix it

Otherwise, I would think just about all Python extensions
from PyPi are now broken given non-existent URLs
(due to the demise of PyPi legacy).   I'm surprised I'm
the only one that has this issue.

jack



--
Dr. Bart E. Oldeman | bart.olde...@mcgill.ca 
 | bart.olde...@calculquebec.ca 


Scientific Computing Analyst / Analyste en calcul scientifique
McGill HPC Centre / Centre de Calcul Haute Performance de McGill | 
http://www.hpc.mcgill.ca 

Calcul Québec | http://www.calculquebec.ca 

Compute/Calcul Canada | http://www.computecanada.ca 


Tel/Tél: 514-396-8926 | Fax/Télécopieur: 514-396-8934

Re: [easybuild] PyPi legacy is gone.... and ideas on source_urls for Python extensions

[Bart Oldeman]

Hi Jack,

I think what Kenneth was trying to say is that you need to use:
'source_urls': ['https://pypi.python.org/packages/source/i/ipython/'],

instead since PYPI_LOWER_SOURCE does not work for extensions.
-- 
Dr. Bart E. Oldeman | bart.olde...@mcgill.ca | bart.olde...@calculquebec.ca
Scientific Computing Analyst / Analyste en calcul scientifique
McGill HPC Centre / Centre de Calcul Haute Performance de McGill |
http://www.hpc.mcgill.ca
Calcul Québec | http://www.calculquebec.ca
Compute/Calcul Canada | http://www.computecanada.ca
Tel/Tél: 514-396-8926 | Fax/Télécopieur: 514-396-8934

Re: [easybuild] PyPi legacy is gone.... and ideas on source_urls for Python extensions

[Jack Perdue]

Howdy Kenneth,

On 07/11/2018 03:59 PM, Kenneth Hoste wrote:

Hi Jack,

On 11/07/2018 20:04, Jack Perdue wrote:

Howdy Kenneth,

For example, I'm trying to build IPython and as
a Bundle with extensions trying to fetch files from:

  https://files.pythonhosted.org/packages/source/i/ipython/

which doesn't exist.


That's not what I'm seeing with the just released EasyBuild v3.6.2 
(this behavior hasn't changed compared to EasyBuild v3.6.1).


Here's a relevant bit from the EasyBuild log that shows it's 
downloading directly from https://pypi.python.org/packages (not via 
https://files.pythonhosted.org/packages).


== 2018-07-11 22:53:46,457 filetools.py:471 INFO Downloaded file 
ipython_urls.html from url https://pypi.python.org/simple/ipython to 
/Users/kehoste/work/TMP/eb-_E1vS6/tmppmy5pk/ipython_urls.html
== 2018-07-11 22:53:46,457 filetools.py:491 INFO Successful download 
of file ipython_urls.html from url 
https://pypi.python.org/simple/ipython to path 
/Users/kehoste/work/TMP/eb-_E1vS6/tmppmy5pk/ipython_urls.html
== 2018-07-11 22:53:51,753 filetools.py:471 INFO Downloaded file 
ipython-6.2.1.tar.gz from url 
https://pypi.python.org/packages/fa/50/974211502bd72873728d44c3013fe79875c819c8fb69f778bcfd67bc7d38/ipython-6.2.1.tar.gz 
to /tmp/i/IPython/extensions/ipython-6.2.1.tar.gz
== 2018-07-11 22:53:51,753 filetools.py:491 INFO Successful download 
of file ipython-6.2.1.tar.gz from url 
https://pypi.python.org/packages/fa/50/974211502bd72873728d44c3013fe79875c819c8fb69f778bcfd67bc7d38/ipython-6.2.1.tar.gz 
to path /tmp/i/IPython/extensions/ipython-6.2.1.tar.gz
== 2018-07-11 22:53:51,753 easyblock.py:730 INFO Successfully 
downloaded source file ipython-6.2.1.tar.gz from 
https://pypi.python.org/packages/fa/50/974211502bd72873728d44c3013fe79875c819c8fb69f778bcfd67bc7d38/ipython-6.2.1.tar.gz


That's not what I am seeing here.  I have:

    ('ipython', version, {
    'source_urls': [PYPI_LOWER_SOURCE],
    'modulename': 'IPython',
    'checksums': [
'eca537aa61592aca2fef4adea12af8e42f5c335004dfa80c78caf80e8b525e5c', # 
ipython-6.4.tar.gz

    ],
    }),

and this is in the log:

== 2018-07-12 13:22:45,782 filetools.py:444 DEBUG Trying to download 
%(namelower)s_urls.html from 
https://pypi.python.org/simple/%(namelower)s to 
/tmp/j-perdue/tmp/eb-3gbeni/tmpHM8L22/%(namelower)s_urls.html
== 2018-07-12 13:22:45,782 filetools.py:451 DEBUG Using timeout of 10 
seconds for initiating download
== 2018-07-12 13:22:45,782 filetools.py:1136 DEBUG Not creating existing 
path /tmp/j-perdue/tmp/eb-3gbeni/tmpHM8L22
== 2018-07-12 13:22:45,917 filetools.py:476 WARNING URL 
https://pypi.python.org/simple/%(namelower)s was not found (HTTP 
response code 400), not trying again
== 2018-07-12 13:22:45,917 filetools.py:494 WARNING Download of 
https://pypi.python.org/simple/%(namelower)s to 
/tmp/j-perdue/tmp/eb-3gbeni/tmpHM8L22/%(namelower)s_urls.html failed, 
done trying
== 2018-07-12 13:22:45,918 filetools.py:400 DEBUG Failed to download 
https://pypi.python.org/simple/%(namelower)s to determine available PyPI 
URLs for %(namelower)s
== 2018-07-12 13:22:45,918 filetools.py:436 DEBUG Failed to extract hash 
using pattern 
'.*/ipython-6\.4\.tar\.gz(?:#md5=[a-f0-9]{32}|#sha256=[a-f0-9]{64})$' 
from list of URLs: []
== 2018-07-12 13:22:45,918 easyblock.py:707 DEBUG Failed to derive 
alternate PyPI URL for 
https://pypi.python.org/packages/source/%(nameletterlower)s/%(namelower)s/ipython-6.4.tar.gz, 
so retaining the original
== 2018-07-12 13:22:45,918 easyblock.py:717 DEBUG Trying to download 
file ipython-6.4.tar.gz from 
https://pypi.python.org/packages/source/%(nameletterlower)s/%(namelower)s/ipython-6.4.tar.gz 
to /sw/eb/sources/i/IPython/extensions/ipython-6.4.tar.gz ...
== 2018-07-12 13:22:45,918 filetools.py:444 DEBUG Trying to download 
ipython-6.4.tar.gz from 
https://pypi.python.org/packages/source/%(nameletterlower)s/%(namelower)s/ipython-6.4.tar.gz 
to /sw/eb/sources/i/IPython/extensions/ipython-6.4.tar.gz
== 2018-07-12 13:22:45,918 filetools.py:451 DEBUG Using timeout of 10 
seconds for initiating download
== 2018-07-12 13:22:45,919 filetools.py:1136 DEBUG Not creating existing 
path /sw/eb/sources/i/IPython/extensions
== 2018-07-12 13:22:46,050 filetools.py:476 WARNING URL 
https://pypi.python.org/packages/source/%(nameletterlower)s/%(namelower)s/ipython-6.4.tar.gz 
was not found (HTTP response code 400), not trying again
== 2018-07-12 13:22:46,051 filetools.py:494 WARNING Download of 
https://pypi.python.org/packages/source/%(nameletterlower)s/%(namelower)s/ipython-6.4.tar.gz 
to /sw/eb/sources/i/IPython/extensions/ipython-6.4.tar.gz failed, done 
trying


Any ideas why that isn't getting expanded  properly?  Your's seems
to have worked.

jack







See:

IPython-6.2.1-foss-2017a-Python-3.6.4.eb

(which is  IPython-6.4-foss-2017b-Python-3.6.3.eb here).

I saw some issues in framework covering problems
with the templates in extensions.  Should
I change all the hard-coded dirs for extensions
as 

Re: [easybuild] PyPi legacy is gone.... and ideas on source_urls for Python extensions

[Kenneth Hoste]

Hi Jack,

On 11/07/2018 20:04, Jack Perdue wrote:

Howdy Kenneth,

For example, I'm trying to build IPython and as
a Bundle with extensions trying to fetch files from:

  https://files.pythonhosted.org/packages/source/i/ipython/

which doesn't exist.


That's not what I'm seeing with the just released EasyBuild v3.6.2 (this 
behavior hasn't changed compared to EasyBuild v3.6.1).


Here's a relevant bit from the EasyBuild log that shows it's downloading 
directly from https://pypi.python.org/packages (not via 
https://files.pythonhosted.org/packages).


== 2018-07-11 22:53:46,457 filetools.py:471 INFO Downloaded file 
ipython_urls.html from url https://pypi.python.org/simple/ipython to 
/Users/kehoste/work/TMP/eb-_E1vS6/tmppmy5pk/ipython_urls.html
== 2018-07-11 22:53:46,457 filetools.py:491 INFO Successful download of 
file ipython_urls.html from url https://pypi.python.org/simple/ipython 
to path /Users/kehoste/work/TMP/eb-_E1vS6/tmppmy5pk/ipython_urls.html
== 2018-07-11 22:53:51,753 filetools.py:471 INFO Downloaded file 
ipython-6.2.1.tar.gz from url 
https://pypi.python.org/packages/fa/50/974211502bd72873728d44c3013fe79875c819c8fb69f778bcfd67bc7d38/ipython-6.2.1.tar.gz 
to /tmp/i/IPython/extensions/ipython-6.2.1.tar.gz
== 2018-07-11 22:53:51,753 filetools.py:491 INFO Successful download of 
file ipython-6.2.1.tar.gz from url 
https://pypi.python.org/packages/fa/50/974211502bd72873728d44c3013fe79875c819c8fb69f778bcfd67bc7d38/ipython-6.2.1.tar.gz 
to path /tmp/i/IPython/extensions/ipython-6.2.1.tar.gz
== 2018-07-11 22:53:51,753 easyblock.py:730 INFO Successfully downloaded 
source file ipython-6.2.1.tar.gz from 
https://pypi.python.org/packages/fa/50/974211502bd72873728d44c3013fe79875c819c8fb69f778bcfd67bc7d38/ipython-6.2.1.tar.gz




See:

IPython-6.2.1-foss-2017a-Python-3.6.4.eb

(which is  IPython-6.4-foss-2017b-Python-3.6.3.eb here).

I saw some issues in framework covering problems
with the templates in extensions.  Should
I change all the hard-coded dirs for extensions
as just PYPI_SOURCE and let EB handle the rest?


PYPI_SOURCE probably isn't going to work with extensions...

But if you use source URLs like 
https://pypi.python.org/packages/source/i/ipython/, EasyBuild will 
automagically translate them to the correct hash URL via 
https://pypi.python.org/simple ...



regards,

Kenneth




Jack Perdue
Lead Systems Administrator
High Performance Research Computing
TAMU Division of Research
j-per...@tamu.edu    http://hprc.tamu.edu
HPRC Helpdesk: h...@hprc.tamu.edu

On 07/11/2018 12:50 PM, Kenneth Hoste wrote:

Hi Jack,

On 11/07/2018 01:38, Jack Perdue wrote:

As recently as a week or two ago, the "legacy"
PyPi site was still serving files from old source_urls.

That no longer seems to be the case.

And the new URLs seem to be a random mess.

Has anyone pondered how to deail with this?


EasyBuild has dealt with this for a while (since v2.8.0), by 
auto-translating PyPI URLs, see 
https://github.com/easybuilders/easybuild-framework/pull/1749 .


Are you saying that functionality is broken?

Seems to be working fine (we also check this in the framework test 
suite, of course):


$ python -c "from test.framework.utilities import init_config; from 
easybuild.tools.filetools import derive_alt_pypi_url; init_config(); 
print 
derive_alt_pypi_url('https://pypi.python.org/packages/source/e/easybuild/easybuild-2.7.0.tar.gz')" 



INFO: This is (based on) vsc.install.shared_setup 0.11.2
https://pypi.python.org/packages/5b/03/e135b19fadeb9b1ccb45eac9f60ca2dc3afe72d099f6bd84e03cb131f9bf/easybuild-2.7.0.tar.gz 




The source tarball can be downloaded using "curl -OL" just fine...


regards,

Kenneth

Re: [easybuild] PyPi legacy is gone.... and ideas on source_urls for Python extensions

[Jack Perdue]

Howdy Kenneth,

For example, I'm trying to build IPython and as
a Bundle with extensions trying to fetch files from:

 https://files.pythonhosted.org/packages/source/i/ipython/

which doesn't exist.

See:

IPython-6.2.1-foss-2017a-Python-3.6.4.eb

(which is  IPython-6.4-foss-2017b-Python-3.6.3.eb here).

I saw some issues in framework covering problems
with the templates in extensions.  Should
I change all the hard-coded dirs for extensions
as just PYPI_SOURCE and let EB handle the rest?


Jack Perdue
Lead Systems Administrator
High Performance Research Computing
TAMU Division of Research
j-per...@tamu.eduhttp://hprc.tamu.edu
HPRC Helpdesk: h...@hprc.tamu.edu

On 07/11/2018 12:50 PM, Kenneth Hoste wrote:

Hi Jack,

On 11/07/2018 01:38, Jack Perdue wrote:

As recently as a week or two ago, the "legacy"
PyPi site was still serving files from old source_urls.

That no longer seems to be the case.

And the new URLs seem to be a random mess.

Has anyone pondered how to deail with this?


EasyBuild has dealt with this for a while (since v2.8.0), by 
auto-translating PyPI URLs, see 
https://github.com/easybuilders/easybuild-framework/pull/1749 .


Are you saying that functionality is broken?

Seems to be working fine (we also check this in the framework test 
suite, of course):


$ python -c "from test.framework.utilities import init_config; from 
easybuild.tools.filetools import derive_alt_pypi_url; init_config(); 
print 
derive_alt_pypi_url('https://pypi.python.org/packages/source/e/easybuild/easybuild-2.7.0.tar.gz')"


INFO: This is (based on) vsc.install.shared_setup 0.11.2
https://pypi.python.org/packages/5b/03/e135b19fadeb9b1ccb45eac9f60ca2dc3afe72d099f6bd84e03cb131f9bf/easybuild-2.7.0.tar.gz 




The source tarball can be downloaded using "curl -OL" just fine...


regards,

Kenneth

Re: [easybuild] PyPi legacy is gone.... and ideas on source_urls for Python extensions

[Kenneth Hoste]

Hi Jack,

On 11/07/2018 01:38, Jack Perdue wrote:

As recently as a week or two ago, the "legacy"
PyPi site was still serving files from old source_urls.

That no longer seems to be the case.

And the new URLs seem to be a random mess.

Has anyone pondered how to deail with this?


EasyBuild has dealt with this for a while (since v2.8.0), by 
auto-translating PyPI URLs, see 
https://github.com/easybuilders/easybuild-framework/pull/1749 .


Are you saying that functionality is broken?

Seems to be working fine (we also check this in the framework test 
suite, of course):


$ python -c "from test.framework.utilities import init_config; from 
easybuild.tools.filetools import derive_alt_pypi_url; init_config(); 
print 
derive_alt_pypi_url('https://pypi.python.org/packages/source/e/easybuild/easybuild-2.7.0.tar.gz')"


INFO: This is (based on) vsc.install.shared_setup 0.11.2
https://pypi.python.org/packages/5b/03/e135b19fadeb9b1ccb45eac9f60ca2dc3afe72d099f6bd84e03cb131f9bf/easybuild-2.7.0.tar.gz


The source tarball can be downloaded using "curl -OL" just fine...


regards,

Kenneth