Hi Jack,
On 11/07/2018 20:04, Jack Perdue wrote:
Howdy Kenneth,
For example, I'm trying to build IPython and as
a Bundle with extensions trying to fetch files from:
https://files.pythonhosted.org/packages/source/i/ipython/
which doesn't exist.
That's not what I'm seeing with the just released EasyBuild v3.6.2 (this
behavior hasn't changed compared to EasyBuild v3.6.1).
Here's a relevant bit from the EasyBuild log that shows it's downloading
directly from https://pypi.python.org/packages (not via
https://files.pythonhosted.org/packages).
== 2018-07-11 22:53:46,457 filetools.py:471 INFO Downloaded file
ipython_urls.html from url https://pypi.python.org/simple/ipython to
/Users/kehoste/work/TMP/eb-_E1vS6/tmppmy5pk/ipython_urls.html
== 2018-07-11 22:53:46,457 filetools.py:491 INFO Successful download of
file ipython_urls.html from url https://pypi.python.org/simple/ipython
to path /Users/kehoste/work/TMP/eb-_E1vS6/tmppmy5pk/ipython_urls.html
== 2018-07-11 22:53:51,753 filetools.py:471 INFO Downloaded file
ipython-6.2.1.tar.gz from url
https://pypi.python.org/packages/fa/50/974211502bd72873728d44c3013fe79875c819c8fb69f778bcfd67bc7d38/ipython-6.2.1.tar.gz
to /tmp/i/IPython/extensions/ipython-6.2.1.tar.gz
== 2018-07-11 22:53:51,753 filetools.py:491 INFO Successful download of
file ipython-6.2.1.tar.gz from url
https://pypi.python.org/packages/fa/50/974211502bd72873728d44c3013fe79875c819c8fb69f778bcfd67bc7d38/ipython-6.2.1.tar.gz
to path /tmp/i/IPython/extensions/ipython-6.2.1.tar.gz
== 2018-07-11 22:53:51,753 easyblock.py:730 INFO Successfully downloaded
source file ipython-6.2.1.tar.gz from
https://pypi.python.org/packages/fa/50/974211502bd72873728d44c3013fe79875c819c8fb69f778bcfd67bc7d38/ipython-6.2.1.tar.gz
See:
IPython-6.2.1-foss-2017a-Python-3.6.4.eb
(which is IPython-6.4-foss-2017b-Python-3.6.3.eb here).
I saw some issues in framework covering problems
with the templates in extensions. Should
I change all the hard-coded dirs for extensions
as just PYPI_SOURCE and let EB handle the rest?
PYPI_SOURCE probably isn't going to work with extensions...
But if you use source URLs like
https://pypi.python.org/packages/source/i/ipython/, EasyBuild will
automagically translate them to the correct hash URL via
https://pypi.python.org/simple ...
regards,
Kenneth
Jack Perdue
Lead Systems Administrator
High Performance Research Computing
TAMU Division of Research
j-per...@tamu.edu http://hprc.tamu.edu
HPRC Helpdesk: h...@hprc.tamu.edu
On 07/11/2018 12:50 PM, Kenneth Hoste wrote:
Hi Jack,
On 11/07/2018 01:38, Jack Perdue wrote:
As recently as a week or two ago, the "legacy"
PyPi site was still serving files from old source_urls.
That no longer seems to be the case.
And the new URLs seem to be a random mess.
Has anyone pondered how to deail with this?
EasyBuild has dealt with this for a while (since v2.8.0), by
auto-translating PyPI URLs, see
https://github.com/easybuilders/easybuild-framework/pull/1749 .
Are you saying that functionality is broken?
Seems to be working fine (we also check this in the framework test
suite, of course):
$ python -c "from test.framework.utilities import init_config; from
easybuild.tools.filetools import derive_alt_pypi_url; init_config();
print
derive_alt_pypi_url('https://pypi.python.org/packages/source/e/easybuild/easybuild-2.7.0.tar.gz')"
INFO: This is (based on) vsc.install.shared_setup 0.11.2
https://pypi.python.org/packages/5b/03/e135b19fadeb9b1ccb45eac9f60ca2dc3afe72d099f6bd84e03cb131f9bf/easybuild-2.7.0.tar.gz
The source tarball can be downloaded using "curl -OL" just fine...
regards,
Kenneth
