Re: [Python-Dev] AIX to stable, what does that take?
On 05/10/2018 22:01, Rob Boehne wrote: > On 10/5/18, 10:33 AM, "Python-Dev on behalf of Michael Haubenwallner" > michael.haubenwall...@ssi-schaefer.com> wrote: > > > > >... I build everything myself, using xlc > >(gcc introduces the need for a GNU RTE, e.g., glibc). > > Using gcc does *not* require to use glibc or even GNU binutils at all. > Except for gcc's own runtime libraries, there's no need for a GNU RTE. > In fact, in Gentoo Prefix I do use gcc as the compiler, configured to > use AIX provided binutils (as, ld, nm, ...), with AIX libc as RTE. > > I think the author was referring to the dependency on libgcc_s when using gcc. > It's typical for native UNIX package builders to use gcc only when necessary > because the correct runtime is always installed (if the os running it is > newer) and therefore won't clash when something else in the process space is > using a different version of libgcc_s (I'm not sure what the ABI guarantees > are with libgcc_s specifically, and neither are UNIX packagers - not > necessarily anyway) Thank you Rob. My core mistake is calling it glibc (that is the gnome libc not that I think back), and libgcc* are something else entirely. In any case, I need to get my facts more accurate. > It also eliminates the need to ship a version of libgcc_s as a shared library. That would make life easier. Would probably have to package gcc on my own to get it work that way though. > > > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] AIX to stable, what does that take?
On 05/10/2018 16:15, Michael Haubenwallner wrote:
> Hi Michael,
>
> being on a similar road with Gentoo Prefix, I really do appreciate
> your AIX related work!
>
> However, for two (not so minor) topics I've got a little different
> experience, which I think should be mentioned here for completion:
Always.
>
> On 10/04/2018 11:13 AM, Michael Felt wrote:
>> On 10/4/2018 10:30 AM, INADA Naoki wrote:
>>> Hello,
>>>
>>> First of all, congratulations on passing all test on AIX.
>>> As a one of core developer, I don't know anything about AIX.
>>> If my change breaks AIX build, I can't investigate what's happened.
>>>
>>> So I think we need following in devguide:
>>>
>>> * Brief description about AIX, from developer's point of view.
>> This I might be able to do. Bullet form:
>> ... I build everything myself, using xlc
>> (gcc introduces the need for a GNU RTE, e.g., glibc).
> Using gcc does *not* require to use glibc or even GNU binutils at all.
> Except for gcc's own runtime libraries, there's no need for a GNU RTE.
> In fact, in Gentoo Prefix I do use gcc as the compiler, configured to
> use AIX provided binutils (as, ld, nm, ...), with AIX libc as RTE.
Well, this is something I learned - second hand - from someone who
worked hard to make much more OSS available than I. Probably wrong then
- in how I came to my conclusion - but the few things I tried to compile
"asis" to shared libraries would not work without also a lot of the gcc
compiler libraries. While I could have bitten the bullet and just found
a way to add those I was warned that different versions of gcc need
different level of supporting files.
>> * finally, a bit deeper: while the AIX linker loader supports svr4
>> shared libraries (it is the data, not the file name) it also supports
>> having multiple shared libraries in a classic archive. So, rather that
>> .../lib/libxxx.so and .../lib64/libxxx.so AIX prefers .../lib/libxxx.a
>> with two so-called members, with same or different names. The one found
>> is not it's name, but the symbol name and size of the ABI (32-bit or 64-bit)
> While this all is true, having multiple *versions* of one shared library in
> one single file is a PITA for package managers - both human or software.
Yes, it is a necessary pain. My secret is a) do not touch /usr/lib -
leave what is as it is, and in the few situations where it must be in
/usr/lib I add/replace named archive members with my new ones - and all
the other ones get extracted, modify a flag in their respective header -
so that the linker knows they are only to be used for applications that
expect them - not for new applications.
> But fortunately, the AIX linker does support so called "Import Files",
> allowing for *filename based* shared library versioning like on Linux,
> while still allowing for both ABIs in a single library archive file.
>
> For example, libtool provides the --with-aix-soname={aix|svr4|both}
> configure flag since libtool-2.4.4. Although the default will stay
> at 'aix' here, in Gentoo Prefix I do use 'svr4' only. This actually
> is a package manager's decision, ideally for all depending packages.
> As gcc does use libtool, for more information please refer to
> https://gcc.gnu.org/install/configure.html#WithAixSoname
> But note that "Import Files" should work with xlc as well.
Actually, more detail than I really want to know. I recall the day when
a library was a collection of "static" .o files, and ranlib was nearly
always needed - Or you told the linker to link against that library
multiple times. And I recall going to my first conference where "RPC"
was the next greatest thing, and shared libraries were going to be such
a space savior - both on disk and in memory. And I was always more of a
bsd fan having schooled myself on UNIX v7, then bsd (2.9 iirc) and bsd
4.1, 4.2 (in comes tcpip) and 4.3. But I diverge :p
To return briefly to the question of what is AIX for the developer -
very flexible. You can choose your architecture and it generally, just
works. I wrote some scripts as a front-end for packaging and most
packages are a one-liner - that runs configure, make, makeinstall (using
DESTDIR) and then packaging the DESTDIR.
As far as python development and AIX goes I am open to helping others.
Been doing that for more years than I care to count. :)
>
> Thanks!
> /haubi/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] AIX to stable, what does that take?
On 10/5/18, 10:33 AM, "Python-Dev on behalf of Michael Haubenwallner" wrote: > >... I build everything myself, using xlc >(gcc introduces the need for a GNU RTE, e.g., glibc). Using gcc does *not* require to use glibc or even GNU binutils at all. Except for gcc's own runtime libraries, there's no need for a GNU RTE. In fact, in Gentoo Prefix I do use gcc as the compiler, configured to use AIX provided binutils (as, ld, nm, ...), with AIX libc as RTE. I think the author was referring to the dependency on libgcc_s when using gcc. It's typical for native UNIX package builders to use gcc only when necessary because the correct runtime is always installed (if the os running it is newer) and therefore won't clash when something else in the process space is using a different version of libgcc_s (I'm not sure what the ABI guarantees are with libgcc_s specifically, and neither are UNIX packagers - not necessarily anyway) It also eliminates the need to ship a version of libgcc_s as a shared library. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] AIX to stable, what does that take?
Hi Michael,
being on a similar road with Gentoo Prefix, I really do appreciate
your AIX related work!
However, for two (not so minor) topics I've got a little different
experience, which I think should be mentioned here for completion:
On 10/04/2018 11:13 AM, Michael Felt wrote:
> On 10/4/2018 10:30 AM, INADA Naoki wrote:
>> Hello,
>>
>> First of all, congratulations on passing all test on AIX.
>> As a one of core developer, I don't know anything about AIX.
>> If my change breaks AIX build, I can't investigate what's happened.
>>
>> So I think we need following in devguide:
>>
>> * Brief description about AIX, from developer's point of view.
> This I might be able to do. Bullet form:
> ... I build everything myself, using xlc
> (gcc introduces the need for a GNU RTE, e.g., glibc).
Using gcc does *not* require to use glibc or even GNU binutils at all.
Except for gcc's own runtime libraries, there's no need for a GNU RTE.
In fact, in Gentoo Prefix I do use gcc as the compiler, configured to
use AIX provided binutils (as, ld, nm, ...), with AIX libc as RTE.
> * finally, a bit deeper: while the AIX linker loader supports svr4
> shared libraries (it is the data, not the file name) it also supports
> having multiple shared libraries in a classic archive. So, rather that
> .../lib/libxxx.so and .../lib64/libxxx.so AIX prefers .../lib/libxxx.a
> with two so-called members, with same or different names. The one found
> is not it's name, but the symbol name and size of the ABI (32-bit or 64-bit)
While this all is true, having multiple *versions* of one shared library in
one single file is a PITA for package managers - both human or software.
But fortunately, the AIX linker does support so called "Import Files",
allowing for *filename based* shared library versioning like on Linux,
while still allowing for both ABIs in a single library archive file.
For example, libtool provides the --with-aix-soname={aix|svr4|both}
configure flag since libtool-2.4.4. Although the default will stay
at 'aix' here, in Gentoo Prefix I do use 'svr4' only. This actually
is a package manager's decision, ideally for all depending packages.
As gcc does use libtool, for more information please refer to
https://gcc.gnu.org/install/configure.html#WithAixSoname
But note that "Import Files" should work with xlc as well.
Thanks!
/haubi/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] AIX to stable, what does that take?
On 10/4/2018 10:30 AM, INADA Naoki wrote: > Hello, > > First of all, congratulations on passing all test on AIX. > >> My assumption is that it needs to (at least) pass all tests - and that >> is why I keep asking for attention. All the PRs to fix individual tests >> mean less if they are not merged, for whatever reason. >> >> However, maybe there is another way, or even something additional >> needed. Maybe something I cannot provide and then I can adjust my >> expectations and goals. > As a one of core developer, I don't know anything about AIX. > If my change breaks AIX build, I can't investigate what's happened. > > So I think we need following in devguide: > > * Brief description about AIX, from developer's point of view. This I might be able to do. Bullet form: * Committed to POSIX standard (valid when release came out, so AIX 5.3 confirms to a different standard than AIX 7.2) * While Linux affinity is recognized - GNU (or GNP - GNU not POSIX) integration is not guaranteed. - GNU rte is not provided under support. There is a so-called Toolbox, GNU an other OSS utilities supplied by many packaged as RPMs. Unfortunately, different RPM providers (Michael Perlz, BULL Freeware, IBM, and others) have different numbering (the part after the package version, e.g., python-2.7.10-XXX so they do not mix well). Headache for both admins and developers trying to develop in a GNU-like environment. * As a consultant, fedup with what is called the "RPM hell" by many AIX admins - I do not use any RPMs. I build everything myself, using xlc (gcc introduces the need for a GNU RTE, e.g., glibc). I package using installp (the "native") AIX package manager, and strive to make the packages independent (these days). When there are dependencies I try to build them as static libraries so that they do not become an additional install dependency. * finally, a bit deeper: while the AIX linker loader supports svr4 shared libraries (it is the data, not the file name) it also supports having multiple shared libraries in a classic archive. So, rather that .../lib/libxxx.so and .../lib64/libxxx.so AIX prefers .../lib/libxxx.a with two so-called members, with same or different names. The one found is not it's name, but the symbol name and size of the ABI (32-bit or 64-bit) * Hope that is enough of the key issues for now. ** In general, GNU autotools (autoreconf -f -v) works well, as does configure.ac et al. for createing OSS Makefiles. > * How to run AIX on (VirtualBox, AWS EC2, Azure, GCP) easily. Easily! ? :) - well, on a POWER server it was easy enough for me to follow the step by step instructions for creating a buildbot. If I had a POWER server with more resources I would examine using the AIX internal WPAR - however, a POWER server configured to use PowerVC uses "EC2" aka cloud-init for creating a virtual machine. With that environment it should be "easy" to provide additional instructions to cloud-init-ec2. Or, I provide you a login on my personal server that I run the buildbot on. etc. etc. - Where there is a will, there is a way. > * How to set up a development environment for Python. Again, follow the instructions for setting up a buildbot. > * How to build Python. git clone ... autoreconf -v -f (as needed) ./configure --with-pydebug #gcc compiler ./configure --with-pydebug --without-computed-gotos # xlc compiler make make test > * How to debug C code. I learned, 40 years ago, using adb (a debugger) - I do a lot of single-stepping. gdb is not the default debugger. If I were a developer I would probably dig into the AIX debuggers (there are at least two, kdb (kernel debugger, which I do use occaisionally for performance issues) and another I try to avoid. I add fprintf statements and am looking at learning how to use probevue. In short, you probably have many much better ideas on how to debug C than I do :) > > And even though there is a developer guide, it will take more long time > than fixing issues on AIX, compared Linux, macOS, and Windows. > > But without this guide, it feels almost impossible to maintain AIX build to > me. IMHO: The AIX build is stable, but this is unrecognized because it does have differences that cause tests to fail. I can think of one test that PASSes, but should fail. And another test that passes, but should have failed (in test_uuid) I have submitted a PR. I tried to fix "all" in one PR, which confused people - so redid it as two (got _uuid working in Python 3.7 ! yes!!) but the "original" to fix uuid.py and test_uuid.py is still "awaiting change review". My gut feeling to maintaining AIX is: a) all test pass so a potential regression is flagged; b) someone such as myself who knows the platform and can establish a "root cause" on why it is failing with AIX so that c) a developer becomes aware and can decide to ignore or adjust; d) alternatives - such as work around an implementation limitation (as I try to do, e.g., for test_time and the related _datetime.c) is yet another path. In other wor
Re: [Python-Dev] AIX to stable, what does that take?
Hello, First of all, congratulations on passing all test on AIX. > My assumption is that it needs to (at least) pass all tests - and that > is why I keep asking for attention. All the PRs to fix individual tests > mean less if they are not merged, for whatever reason. > > However, maybe there is another way, or even something additional > needed. Maybe something I cannot provide and then I can adjust my > expectations and goals. As a one of core developer, I don't know anything about AIX. If my change breaks AIX build, I can't investigate what's happened. So I think we need following in devguide: * Brief description about AIX, from developer's point of view. * How to run AIX on (VirtualBox, AWS EC2, Azure, GCP) easily. * How to set up a development environment for Python. * How to build Python. * How to debug C code. And even though there is a developer guide, it will take more long time than fixing issues on AIX, compared Linux, macOS, and Windows. But without this guide, it feels almost impossible to maintain AIX build to me. Regards, -- INADA Naoki ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] AIX to stable, what does that take?
In the buildbots AIX is marked as "unstable"? What is needed to get it marked as a "stable" platform - that being one of my short-term goals. My assumption is that it needs to (at least) pass all tests - and that is why I keep asking for attention. All the PRs to fix individual tests mean less if they are not merged, for whatever reason. However, maybe there is another way, or even something additional needed. Maybe something I cannot provide and then I can adjust my expectations and goals. Regards, Michael ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
