Hi Kenneth, when we install Python packages using pip from one Python binary, and then try importing them with a different Python binary, we see the following error message:
ImportError: ...Python/2.7.8-foss-2014b/lib/python2.7/lib-dynload/operator.so: undefined symbol: _PyUnicodeUCS2_AsDefaultEncodedString You are right, this only happens when we re-run eb and build a new Python binary, either using -f, or when updating to a new minor version, such as 2.7.11 next to 2.7.8. I assume I can nail the unicode setting somehow in the easyconfig. What would be the best way to do this? Just adding configopts = '--enable-unicode=ucs2 ' does not work, as it results in the following configure step: ./configure --prefix=/gpfs01/sw/eb20160420/software/Python/2.7.8-foss-2014b --enable-unicode=ucs2 --with-threads --enable-shared --enable-unicode=ucs4 Best regards, Olaf Von: [email protected] [mailto:[email protected]] Im Auftrag von Kenneth Hoste Gesendet: Donnerstag, 28. April 2016 15:06 An: [email protected] Betreff: Re: [easybuild] UCS-2 and UCS-4 in Python easyblock Hi Olaf, On 28/04/16 14:59, Olaf Walter wrote: Dear easybuilders, we jumped to eb 2.7, and our Python builds became incompatible. We think the cause is: https://github.com/hpcugent/easybuild-easyblocks/pull/817 https://github.com/hpcugent/easybuild-easyblocks/issues/651 From the Python docs at https://docs.python.org/2/c-api/unicode.html: Note that UCS2 and UCS4 Python builds are not binary compatible. Please keep this in mind when writing extensions or interfaces. From the python.py easyblock: # Need to be careful to match the unicode settings to the underlying python The reasoning seems to be that the eb Python should be compatible with Python packages provided by the OS. IMHO, this is less important than ensuring binary compatibility between Python releases that are built by different easybuild releases. Actually I think, one of the benefits of easybuild is that the resulting binaries are not so much dependent on the underlying OS version. The introduced change achieves the opposite. Can you elaborate on what you mean by "our Python builds became incompatible"? What is incompatible with what, exactly? The change you're pointing out only affects things if you reinstall the Python module itself; it does not affect installing individual Python packages. One of the reasons why we use easybuild is to create reproducible software builds. For that scenario, would you recommend sticking with a frozen easybuild release? Please let me know what you think. EasyBuild releases are backwards compatible with the releases before that (as long as the major version number (i.e. 2, now) is the same), with the exception of bug fixes. For more strict reproducibility, you should probably stick to the same EasyBuild version, but then you'll miss out on new features, enhancements (like the significant speedups we've been working on for EasyBuild itself), and various bug fixes... regards, Kenneth
