Sniffing out the ELF loader is definitely more complicated than ideal – e.g. it adds a "find the python binary" step that could go wrong – but, ok, if that were the only barrier maybe we could manage.
The bigger problem is: how do we figure out whether a wheel built against *that* musl on *that* machine will work with *this* musl on *this* machine? For glibc, this involves three pieces, each of which is non-trivial: - the glibc maintainers provide some careful, documented guarantees about when a library built against one glibc version will run with another glibc version, and they encode this in machine-readable form in their symbol versions - auditwheel checks the symbol versions to derive a summary of what the wheel needs, and stores it the wheel metadata - pip checks the local system's glibc version against this metadata A simple "is this musl or not?" check isn't useful on its own. We also need some musl equivalent for this other machinery. (It doesn't have to work the same way, but it has to accomplish the same result.) If you want to keep moving this forward you're going to have to talk to the musl maintainers. -n On Mon, Feb 25, 2019, 09:49 Alexander Revin <lyss...@gmail.com> wrote: > I've put combined code here: > https://gist.github.com/lyssdod/f51579ae8d93c8657a5564aefc2ffbca > > Just download it, make executable and run. > > amd64 Alpine: > # ./guess_pyruntime.py > Interpreter extracted: /lib/ld-musl-x86_64.so.1 > Running on musl > > amd64 Gentoo glibc: > # ./guess_pyruntime.py > Interpreter extracted: /lib64/ld-linux-x86-64.so.2 > Running on glibc version 2.27 > > On Thu, Feb 21, 2019 at 3:57 AM Alexander Revin <lyss...@gmail.com> wrote: > > > > Hi Nathaniel, > > > > Thanks for your answer. > > > > Basing on your example of RHEL and Ubuntu, let's take RHEL 6 which > > uses glibc 2.12. If you cross-compile for it (using the same gcc RHEL > > uses), wheel surely will work on Ubuntu 18.10 :) > > I think it's not of an issue, since wheels are built with these > > minimal runtime requirements anyway, unless they're built on a local > > machine – but in this case they will just work™; anyway, having a > > working toolchain is not the scope of Python tooling. Gentoo has an > > awesome crossdev tool for creating cross-toolchains, and there's > > crosstool-ng of course. It looks like there are workarounds for > > putting code built on newer systems to older ones though, but they > > seem to be pretty tedious ([1]) > > > > Speaking of runtime detection, I see it this way for example – since > > one of the most reliable ways to check for program's dependencies is > > invoking something like ldd or objdump, it can essentially be done the > > same way: > > > > 1. Pick the minimal required code to extract ELF's ".interp" field [2] > > (I used code from [3]); > > 2. Process sys.executable with it; > > > > Here what it returns (grepping by "/lib" because it starts from a new > line): > > > > Gentoo amd64 glibc > > # python3 readelf.py $(which python3) | grep "/lib" > > b'/lib64/ld-linux-x86-64.so.2\x00' > > > > Alpine amd64 docker (official python3 alpine image): > > # python3 readelf.py $(which python3) | grep "/lib" > > b'/lib/ld-musl-x86_64.so.1\x00' > > > > 3. Essentially that's enough in my opinion, but we can go further and > > do what ldd does: > > > > # /lib/ld-musl-x86_64.so.1 --list $(which python3) > > /lib/ld-musl-x86_64.so.1 (0x7fbc36567000) > > libpython3.7m.so.1.0 => /usr/local/lib/libpython3.7m.so.1.0 > (0x7fbc3622a000) > > libc.musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x7fbc36567000) > > > > Basically it's just string matching here, and the only question now if > > the name of dynamic linker is enough or all libs should be iterated > > until perfect "musl" or "libc" match. Parsing "Dynamic section" turns > > out to be pretty useless – it's empty on Alpine (or parsing code is > > buggy). If ".interp" field is not available, then interpreter is > > statically linked :) > > > > 4. If any glibc-specific functionality is needed at this point, code > > from PEP 513 is really good. Maybe it's also better to put it first > > and use ELF parsing if it failed to open glibc in the first place. > > > > > > Thanks, > > Alex > > > > > > > > [1] > https://snorfalorpagus.net/blog/2016/07/17/compiling-python-extensions-for-old-glibc-versions/ > > [2] https://www.linuxjournal.com/article/1060 > > [3] > https://github.com/detailyang/readelf/blob/master/readelf/readelf.py#L545 > > > > On Wed, Feb 20, 2019 at 1:50 AM Nathaniel Smith <n...@pobox.com> wrote: > > > > > > On Tue, Feb 19, 2019 at 3:28 PM Alexander Revin <lyss...@gmail.com> > wrote: > > > > > > > > Hi all, > > > > > > > > I have an idea regarding Python binary wheels on non-glibc platforms, > > > > and it seems that initially I've posted it to the wrong list ([1]) > > > > > > > > Long story short, the proposal is to use platform tuples (like > > > > compiler ones) for wheel names, which will allow much broader > platform > > > > support, for example: > > > > > > > > package-1.0-cp36-cp36m-amd64_linux_gnu.whl > > > > package-1.0-cp36-cp36m-amd64_linux_musl.whl > > > > > > > > So eventually only {platform tag} part will be modified. Glibc/musl > > > > detection is quite trivial and eventually will be based on existing > > > > one in PEP 513 [2]. > > > > > > The challenge here is: the purpose of a target triple is to tell a > > > compiler/linker toolchain which kind of code they should generate, > > > e.g. when cross-compiling. The purpose of a wheel tag is to tell you > > > whether a given wheel will work on a given system. It turns out these > > > are different things :-). > > > > > > For example, Ubuntu 18.10 and RHEL 6 are both 'amd64-linux-gnu', > > > because they use the same instruction set, the same binary format > > > (ELF), etc. But if you build a wheel on Ubuntu 18.10, it definitely > > > will not work on RHEL 6. (The other way around might work, if you do > > > other things right.) > > > > > > In practice Windows and macOS are already fine; the place where this > > > would be useful is Linux wheels for platforms that use non-Intel-based > > > architectures or non-glibc-libcs. We do have an idea for making it > > > easier to support newer glibcs and also extending to all > > > architectures: > https://mail.python.org/archives/list/distutils-sig@python.org/thread/6AFS4HKX6PVAS76EQNI7JNTGZZRHQ6SQ/ > > > > > > Adding musl is a bit trickier since I'm not sure what the details of > > > their ABI compatibility are, and they intentionally make it difficult > > > to figure out whether you're running on musl. But if someone could > > > convince them to publish more information then we could fix that too. > > > > > > -n > > > > > > -- > > > Nathaniel J. Smith -- https://vorpus.org >
-- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/SUT2LEK6PDVNRLLYM7EJTB3SE24PY53J/
