Thanks very much! I added it as a dependency to the easyconfig, since we use GCC here. Now on to other modules (and R).
Do you guys have an idea of how big/small the easybuild user base is? Double figures, triple? Just curious. Todd From: Kenneth Hoste <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Thursday, February 13, 2014 at 6:45 PM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: [easybuild] trouble with python modules and R packages built as extensions Hi Todd, OK, after some digging around, we figured out that the problem is not with your system Python, but with the one EasyBuild is building. The reason we've missed this so far is because we usually have openssl-dev installed system-wide, so our EB Python builds do have SSL support. We can fix things such that EasyBuild will always built an SSL-enabled Python, but it gets tricky for a couple of reasons. First of all: telling the Python build system to use an OpenSSL lib in a non-standard location is tricky, but not impossible. Petar opened a PR with a rather hackish solution, I figured out that there's a (slightly) better approach, see https://github.com/hpcugent/easybuild-easyblocks/pull/359 . This works. Next up is specifying OpenSSL as a dependency in the Python easyconfig. No biggy, except that building OpenSSL with the Intel compilers (the ictce toolchain) is a problem (the OpenSSL tests fail, which is not a good sign). One way around that problem is to simply link against an OpenSSL library that was built with the system compilers (i.e. using a dummy-dummy toolchain). It's dirty, but that's what also happens when you rely on the OpenSSL lib provided by the OS. See https://github.com/hpcugent/easybuild-easyconfigs/pull/703 for the PR that adds the OpenSSL deps in the Python easyconfigs. This probably needs a little bit more love (i.e. make sure that the SSL support effectively works when OpenSSL is listed as a dependency), but this should resolve the problem you've been seeing. We're doing what we can to get these fixes included in EasyBuild v1.11, but that's already running late, so no promises there. regards, Kenneth On 12/02/14 15:46, Heywood, Todd wrote: First, in reply to Jens, the _ssl module is installed; shows up under “help(‘modules’)”, and: heywood@bnbmgmt2:~$ python Python 2.7.2 (default, Jun 14 2012, 09:57:05) [GCC 4.4.6 20110731 (Red Hat 4.4.6-3)] on linux2 Type "help", "copyright", "credits" or "license" for more information. import _ssl And that show the older Python that is part of our “base” install. I am using easy build for extra, optional, or more up to date installs of specific things with specific dependencies differing from our base (toolchains are a good concept in this scenario). Like Python-2.7.5 and 3, and R-3. I ran this: "eb -fd -r $EBCONFIGS ./Python-2.7.5-goalf-1.5.12-no-OFED.eb “, where the last 2 extensions are: ('distribute', '0.6.28', { 'source_urls': ['https://pypi.python.org/packages/source/d/distribute/'], }), ('matplotlib', '1.3.1', { 'source_urls': ['https://downloads.sourceforge.net/project/matplotlib/matplotlib/matplotlib-1.3.1/'], }), Got this, running with the debug option: heywood@bnbmgmt2:/sonas-hs/it/hpc/home/eb/ebfiles/configs/cshl$ eb -fd -r $EBCONFIGS ./Python-2.7.5-goalf-1.5.12-no-OFED.eb == temporary log file in case of crash /tmp/easybuild-SMsBQ5/easybuild-tDl804.log == resolving dependencies ... == processing EasyBuild easyconfig /sonas-hs/it/hpc/home/eb/ebfiles/configs/cshl/Python-2.7.5-goalf-1.5.12-no-OFED.eb == building and installing Python/2.7.5-goalf-1.5.12-no-OFED... == fetching files... == creating build dir, resetting environment... == unpacking... == patching... == preparing... == configuring... == building... == testing... == installing... == taking care of extensions... ERROR: EasyBuild encountered an exception (at easybuild/software/EasyBuild/1.10.0/lib/python2.7/site-packages/easybuild_framework-1.10.0-py2.7.egg/easybuild/main.py:809 in build_and_install_software): autoBuild Failed (last 300 chars): D/lib/python2.7/urllib2.py", line 382, in _call_chain result = func(*args) File "/opt/eb/software/Python/2.7.5-goalf-1.5.12-no-OFED/lib/python2.7/urllib2.py", line 1247, in unknown_open raise URLError('unknown url type: %s' % type) urllib2.URLError: <urlopen error unknown url type: https> heywood@bnbmgmt2:/sonas-hs/it/hpc/home/eb/ebfiles/configs/cshl$ Log file attached. Thanks, Todd From: Stijn De Weirdt <[email protected]<mailto:[email protected]><mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]><mailto:[email protected]>" <[email protected]<mailto:[email protected]><mailto:[email protected]>> Date: Wednesday, February 12, 2014 at 3:12 AM To: "[email protected]<mailto:[email protected]><mailto:[email protected]>" <[email protected]<mailto:[email protected]><mailto:[email protected]>> Subject: Re: [easybuild] trouble with python modules and R packages built as extensions hi todd, this: urllib2.URLError: <urlopen error unknown url type: https> what python version is EB using to run? (if it is an OS provided one, can you add the full package version/) and can you produce a debug run and send us the output? stijn On 02/11/2014 06:37 PM, Heywood, Todd wrote: First, thanks for help on the earlier issue of module unloading, so that I can give you guys another issue :-). I am trying to build new versions of Python and R, and have long lists of required modules and packages. I¹m adding lines under the exts_list section of .eb files. I often get crashes due to "unknown url type² errors, which I think come from modules/packages trying to go find moudles/packages they depend on, and download them. I have been able to get by some by finding exact URL¹s and versions for dependencies and putting lines in the exts_list for these before the module/package in question. However, this is not working for one Python module I¹m trying to install now, and I figure I should ask about the real issue here instead of just trying to work around it. So, trying to install the matplotlib Python module with (eb ‹robot) with this: ('matplotlib', '1.3.1', { 'source_urls': ['https://downloads.sourceforge.net/project/matplotlib/matplotlib/matplotli b-1.3.1/'], }), The output, and log file excerpt are below. Matplotlib is apparently trying to download the ³distribute² dependency module. So I put this line above the matplotlib line: ('distribute', '0.6.28', { 'source_urls': ['https://pypi.python.org/packages/source/d/distribute/'], }), But the eaasybuild install still crashes with the same error. Output from build: == taking care of extensions... ERROR: EasyBuild encountered an exception (at easybuild/software/EasyBuild/1.10.0/lib/python2.7/site-packages/easybuild_f ramework-1.10.0-py2.7.egg/easybuild/main.py:809 in build_and_install_software): autoBuild Failed (last 300 chars): D/lib/python2.7/urllib2.py", line 382, in _call_chain result = func(*args) File "/opt/eb/software/Python/2.7.5-goalf-1.5.12-no-OFED/lib/python2.7/urllib2.p y", line 1247, in unknown_open raise URLError('unknown url type: %s' % type) urllib2.URLError: <urlopen error unknown url type: https> From the log file: == 2014-02-11 11:35:44,089 main.filetools ERROR EasyBuild crashed with an error (at easybuild/software/EasyBuild/1.10.0/lib/python2.7/site-packages/easybuild_f ramework-1.10.0-py2.7.egg/easybuild/tools/fil\ etools.py:799 in parse_cmd_output): cmd "python setup.py build" exited with exitcode 1 and output: Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.28.tar.g z Traceback (most recent call last): File "setup.py", line 10, in <module> use_setuptools() File "/sonas-hs/it/hpc/home/eb/build/Python/2.7.5/goalf-1.5.12-no-OFED.10/matplo tlib/matplotlib-1.3.1/distribute_setup.py", line 163, in use_setuptools return _do_download(version, download_base, to_dir, download_delay) File "/sonas-hs/it/hpc/home/eb/build/Python/2.7.5/goalf-1.5.12-no-OFED.10/matplo tlib/matplotlib-1.3.1/distribute_setup.py", line 132, in _do_download to_dir, download_delay) File "/sonas-hs/it/hpc/home/eb/build/Python/2.7.5/goalf-1.5.12-no-OFED.10/matplo tlib/matplotlib-1.3.1/distribute_setup.py", line 212, in download_setuptools src = urlopen(url) File "/opt/eb/software/Python/2.7.5-goalf-1.5.12-no-OFED/lib/python2.7/urllib2.p y", line 127, in urlopen return _opener.open(url, data, timeout) File "/opt/eb/software/Python/2.7.5-goalf-1.5.12-no-OFED/lib/python2.7/urllib2.p y", line 410, in open response = meth(req, response) File "/opt/eb/software/Python/2.7.5-goalf-1.5.12-no-OFED/lib/python2.7/urllib2.p y", line 523, in http_response 'http', request, response, code, msg, hdrs) File "/opt/eb/software/Python/2.7.5-goalf-1.5.12-no-OFED/lib/python2.7/urllib2.p y", line 442, in error result = self._call_chain(*args) File "/opt/eb/software/Python/2.7.5-goalf-1.5.12-no-OFED/lib/python2.7/urllib2.p y", line 382, in _call_chain result = func(*args) File "/opt/eb/software/Python/2.7.5-goalf-1.5.12-no-OFED/lib/python2.7/urllib2.p y", line 629, in http_error_302 return self.parent.open(new, timeout=req.timeout) File "/opt/eb/software/Python/2.7.5-goalf-1.5.12-no-OFED/lib/python2.7/urllib2.p y", line 404, in open response = self._open(req, data) File "/opt/eb/software/Python/2.7.5-goalf-1.5.12-no-OFED/lib/python2.7/urllib2.p y", line 427, in _open 'unknown_open', req) File "/opt/eb/software/Python/2.7.5-goalf-1.5.12-no-OFED/lib/python2.7/urllib2.p y", line 382, in _call_chain result = func(*args) File "/opt/eb/software/Python/2.7.5-goalf-1.5.12-no-OFED/lib/python2.7/urllib2.p y", line 1247, in unknown_open raise URLError('unknown url type: %s' % type) urllib2.URLError: <urlopen error unknown url type: https> == 2014-02-11 11:35:44,090 main ERROR EasyBuild crashed with an error (at easybuild/software/EasyBuild/1.10.0/lib/python2.7/site-packages/easybuild_f ramework-1.10.0-py2.7.egg/easybuild/tools/build_log.py:\ 96 in exception): EasyBuild encountered an exception (at easybuild/software/EasyBuild/1.10.0/lib/python2.7/site-packages/easybuild_f ramework-1.10.0-py2.7.egg/easybuild/main.py:809 in build_and_install_sof\ tware): autoBuild Failed (last 300 chars): D/lib/python2.7/urllib2.py", line 382, in _call_chain result = func(*args) File "/opt/eb/software/Python/2.7.5-goalf-1.5.12-no-OFED/lib/python2.7/urllib2.p y", line 1247, in unknown_open raise URLError('unknown url type: %s' % type) urllib2.URLError: <urlopen error unknown url type: https> Any ideas? Both for this specific module, and the fact that ³unknown url type² is very common? Thanks, Todd
