Hi Olaf,
On 28/04/16 15:28, Olaf Walter wrote:
Hi Kenneth,
when we install Python packages using pip from one Python binary, and
then try importing them with a different Python binary, we see the
following error message:
ImportError:
…Python/2.7.8-foss-2014b/lib/python2.7/lib-dynload/operator.so:
undefined symbol: _PyUnicodeUCS2_AsDefaultEncodedString
You mean, using a Python binary (the former one) that was installed
using an EasyBuild version which doesn't use --enable-unicode=* yet, right?
Yes, that won't work, the unicode setting for both Python builds has to
match...
You are right, this only happens when we re-run eb and build a new
Python binary, either using -f, or when updating to a new minor
version, such as 2.7.11 next to 2.7.8.
I assume I can nail the unicode setting somehow in the easyconfig.
What would be the best way to do this? Just adding
configopts = '--enable-unicode=ucs2 '
does not work, as it results in the following configure step:
./configure
--prefix=/gpfs01/sw/eb20160420/software/Python/2.7.8-foss-2014b
--enable-unicode=ucs2 --with-threads --enable-shared
--enable-unicode=ucs4
This way, your setting gets overridden, indeed.
You could do the more sneaky thing using '#' to comment out what the
Python easyblock appends:
# ensure that ucs2 setting for unicode is used, so this build
matches with the other Python builds
configopts = "--enable-unicode=ucs2 --with-threads --enable-shared #"
which should result in something like:
./configure --prefix=... --enable-unicode=ucs2 --with-threads
--enable-shared #--with-threads --enable-shared --enable-unicode=ucs4
That's sort of a hack though, we should add support in the Python
easyblock to override the guessing it does now, i.e. we should support
something like this:
with_unicode = 'ucs2'
Alan: thoughts?
Olaf: do note that you will be 'stuck' to ucs2 this way, while your
system Python was apparently built with usc4...
So, system Python packages are/will not be compatible with the ones you
install using this Python build (which is the whole point of adding this
Unicode detection to the Python easyblock).
regards,
Kenneth
Best regards,
Olaf
*Von:*[email protected]
[mailto:[email protected]] *Im Auftrag von *Kenneth Hoste
*Gesendet:* Donnerstag, 28. April 2016 15:06
*An:* [email protected]
*Betreff:* Re: [easybuild] UCS-2 and UCS-4 in Python easyblock
Hi Olaf,
On 28/04/16 14:59, Olaf Walter wrote:
Dear easybuilders,
we jumped to eb 2.7, and our Python builds became incompatible. We
think the cause is:
https://github.com/hpcugent/easybuild-easyblocks/pull/817
https://github.com/hpcugent/easybuild-easyblocks/issues/651
From the Python docs at https://docs.python.org/2/c-api/unicode.html:
Note that UCS2 and UCS4 Python builds are not binary compatible.
Please keep this in mind when writing extensions or interfaces.
From the python.py easyblock:
# Need to be careful to match the unicode settings to the
underlying python
The reasoning seems to be that the eb Python should be compatible
with Python packages provided by the OS.
IMHO, this is less important than ensuring binary compatibility
between Python releases that are built by different easybuild
releases.
Actually I think, one of the benefits of easybuild is that the
resulting binaries are not so much dependent on the underlying OS
version. The introduced change achieves the opposite.
Can you elaborate on what you mean by "our Python builds became
incompatible"?
What is incompatible with what, exactly?
The change you're pointing out only affects things if you reinstall
the Python module itself; it does not affect installing individual
Python packages.
One of the reasons why we use easybuild is to create reproducible
software builds. For that scenario, would you recommend sticking with
a frozen easybuild release? Please let me know what you think.
EasyBuild releases are backwards compatible with the releases before
that (as long as the major version number (i.e. 2, now) is the same),
with the exception of bug fixes.
For more strict reproducibility, you should probably stick to the same
EasyBuild version, but then you'll miss out on new features,
enhancements (like the significant speedups we've been working on for
EasyBuild itself), and various bug fixes...
regards,
Kenneth
