Re: On time64 and Large File Support

[Sam James]

> On 12 Nov 2022, at 02:20, Zack Weinberg via Libc-alpha 
>  wrote:
> 
> Sam James  writes:
>>> On 11 Nov 2022, at 09:19, Florian Weimer  wrote:
>>> We need to support legacy binaries on i386.  Few libraries are
>>> explicitly dual-ABI.  Whether it's safe to switch libraries above glibc
>>> to LFS or time64 needs to be evaluated on a per-library basis.  For most
>>> distributions, no one is going to do that work, and we have to stick to
>>> whathever we are building today.
>> 
>> While I agree, I don't think it's as well-known that it should be that
>> these are ABI breaking and require inspection. It's being done ad-hoc
>> or in many cases, not at all.
> …
 We're now (possibly) on the eve of an autoconf 2.72 release which
 contains two changes of note [2][3]
 1. addition of a new AC_SYS_YEAR2038 macro;
 2. making AC_SYS_LARGEFILE change behaviour to imply AC_SYS_YEAR2038.
 
 Indeed, the gnulib version of change #2 is exactly how we ended up with
 wget/gnutls breaking [1]. I feel this shows that the only approach
 "supported" by glibc right now is untenable.
>>> 
>>> AC_SYS_LARGEFILE defaulting to AC_SYS_YEAR2038 is extremely destructive
>>> for Fedora unfortunately.
>>> 
>>> I thought the gnulib change has been reverted?
>>> 
>>> I really wish the rest of GNU would talk to glibc maintainers before
>>> overriding glibc maintainer decisions.
> 
> There seems to be a disconnect between Paul Eggert’s position on these
> changes and everyone else on this thread’s position.
> 
> I don’t think Paul considered the new behavior of AC_SYS_LARGEFILE to be
> overriding the glibc maintainers. Rather, I think he was only thinking
> about applications, not libraries, and only about source distribution.
> If an application is being built on the machine where it’ll be used, and
> both it and the system support building it with 64-bit time_t and off_t,
> *why wouldn’t you*?  It has no impact on anything else to do that, and
> it future-proofs your installataion.
> 
> But everyone else is worrying about cases where, either, an application
> is built with 64-bit time_t and/or off_t and then copied to a different
> system where the underlying libraries *don’t* support that, or a *shared
> library* is rebuilt with 64-bit time_t and/or off_t and that silently
> changes its ABI out from under programs that use it.
> 

Precisely.

> (It’s unfortunate, IMNSHO, that glibc chose not to provide a time_t
> equivalent of _LARGEFILE64_SOURCE, as this means it’s much more
> difficult for any shared library other than glibc to offer *both* time_t
> and time64_t binary interfaces.)
> 
> (It’s also unfortunate that, 20+ years after the invention of ELF symbol
> versioning, it’s still ridiculously difficult.  So many macros and
> assembly hacks and fighting with libtool.  But I digress.)
> 
> I am honestly not sure what to do about this in the long term, but for
> the proposed “this weekend, just bugfixes” Autoconf 2.72, I do think it
> makes sense to back out change #2, only — that is, AC_SYS_YEAR2038 will
> exist, but AC_SYS_LARGEFILE will *not* imply AC_SYS_YEAR2038.  That will
> limit the impact of AC_SYS_YEAR2038 to packages that have explicitly
> added it, and should make it safe for Fedora and Gentoo to drop in 2.72
> in order to unblock C23 testing — am I correct?  It doesn’t resolve the
> larger issue, but it gives us more time to think about what the
> resolution ought to be.
> 
> What do you think?

This is really I think the best option while allowing us time & space
to complete the larger discussion.

We keep AC_SYS_YEAR2038 which lets upstreams opt in,
but it avoids the risk of sudden LFS-into-time64 rollover.

Year 2038 work is really important (which is why I brought
up the topic) but I don't want confusion around it to make
people avoid adopting 2.72 or leave us without a 2.72
at all.

I do sympathise with Paul's position, but I'm not
a believer in piecemeal anyway now and it's fair to say there's no
consensus on this yet.

Paul's point Is fair that distros who wish can set the cache var to
ff, so *if* you and Paul want to keep it in.

I could live with a bigger "please check this!" in NEWS. I don't think it's a
total disaster as long as we brief distros first, as we're already
doing it for gnuilb & upstreams may well end up doing themselves,
so the train is on its way out of the station anyway.

(I think we need to keep the discussion going on that front,
but I think it'd be wrong to let it block this release, as it's
not a simple problem and I'm still unsure how I fully feel
about the issues at its core.)

I don't think it's conservative to make a macro suddenly
do a new thing which might require testing on the part
of projects using autoconf. Even if I get the aim.

And if we want to change it later, we still can. If we all
magically reach consensus in a month, there's nothing
stopping that being pursued.

So let's go for keep AC_SYS_YEAR2038 only?

Best,
sam


signature.asc
Description: 

Re: On time64 and Large File Support

[Zack Weinberg]

Sam James  writes:
>> On 11 Nov 2022, at 09:19, Florian Weimer  wrote:
>> We need to support legacy binaries on i386.  Few libraries are
>> explicitly dual-ABI.  Whether it's safe to switch libraries above glibc
>> to LFS or time64 needs to be evaluated on a per-library basis.  For most
>> distributions, no one is going to do that work, and we have to stick to
>> whathever we are building today.
>
> While I agree, I don't think it's as well-known that it should be that
> these are ABI breaking and require inspection. It's being done ad-hoc
> or in many cases, not at all.
…
>>> We're now (possibly) on the eve of an autoconf 2.72 release which
>>> contains two changes of note [2][3]
>>> 1. addition of a new AC_SYS_YEAR2038 macro;
>>> 2. making AC_SYS_LARGEFILE change behaviour to imply AC_SYS_YEAR2038.
>>> 
>>> Indeed, the gnulib version of change #2 is exactly how we ended up with
>>> wget/gnutls breaking [1]. I feel this shows that the only approach
>>> "supported" by glibc right now is untenable.
>> 
>> AC_SYS_LARGEFILE defaulting to AC_SYS_YEAR2038 is extremely destructive
>> for Fedora unfortunately.
>> 
>> I thought the gnulib change has been reverted?
>> 
>> I really wish the rest of GNU would talk to glibc maintainers before
>> overriding glibc maintainer decisions.

There seems to be a disconnect between Paul Eggert’s position on these
changes and everyone else on this thread’s position.

I don’t think Paul considered the new behavior of AC_SYS_LARGEFILE to be
overriding the glibc maintainers. Rather, I think he was only thinking
about applications, not libraries, and only about source distribution.
If an application is being built on the machine where it’ll be used, and
both it and the system support building it with 64-bit time_t and off_t,
*why wouldn’t you*?  It has no impact on anything else to do that, and
it future-proofs your installataion.

But everyone else is worrying about cases where, either, an application
is built with 64-bit time_t and/or off_t and then copied to a different
system where the underlying libraries *don’t* support that, or a *shared
library* is rebuilt with 64-bit time_t and/or off_t and that silently
changes its ABI out from under programs that use it.

(It’s unfortunate, IMNSHO, that glibc chose not to provide a time_t
equivalent of _LARGEFILE64_SOURCE, as this means it’s much more
difficult for any shared library other than glibc to offer *both* time_t
and time64_t binary interfaces.)

(It’s also unfortunate that, 20+ years after the invention of ELF symbol
versioning, it’s still ridiculously difficult.  So many macros and
assembly hacks and fighting with libtool.  But I digress.)

I am honestly not sure what to do about this in the long term, but for
the proposed “this weekend, just bugfixes” Autoconf 2.72, I do think it
makes sense to back out change #2, only — that is, AC_SYS_YEAR2038 will
exist, but AC_SYS_LARGEFILE will *not* imply AC_SYS_YEAR2038.  That will
limit the impact of AC_SYS_YEAR2038 to packages that have explicitly
added it, and should make it safe for Fedora and Gentoo to drop in 2.72
in order to unblock C23 testing — am I correct?  It doesn’t resolve the
larger issue, but it gives us more time to think about what the
resolution ought to be.

What do you think?
zw

Re: On time64 and Large File Support

[Paul Eggert]

On 2022-11-11 03:38, Florian Weimer wrote:

But that said, these binaries are broken anyway in 2038?

No, I expect users to run them in time-shifted VMs or containers.


That's reasonable for systems where accurate timestamps are not 
important and where time_t width mismatches would just get in the way.


However, if this is the expected approach, I suggest having a standard 
and well-documented way to timeshift VMs and containers, and unless you 
want to cede the field entirely to other platforms this documentation 
effort and testing should be done now rather than in the year 2037. 
(Another thing to add to your bulging to-do list)

Re: How can Autoconf help with the transition to stricter compilation defaults?

[Aaron Ballman]

On Thu, Nov 10, 2022 at 4:05 PM Paul Eggert  wrote:
>
> On 2022-11-10 10:19, Aaron Ballman wrote:
> > In terms of the Clang side of things, I don't think we've formed any
> > sort of official stance on how to handle that yet. It's UB (you can
> > declare the C standard library interface without UB but calling any
> > function with a mismatched signature is UB)
>
> The Autoconf-generated code is never executed, so this is not a runtime
> issue; it's merely an issue of undefined behavior during translation.

FWIW, the only thing the (Clang) compiler is aware of is translation.
So from the frontend perspective, we can't tell the difference between
"trust me this is safe because it never gets executed" and "this is a
CVE". We believe the runtime behavior is sufficiently dangerous to
warrant a conservative view that any call to a function will be a call
that gets executed at runtime, hence a definitive signature mismatch
is something we feel comfortable diagnosing (in some form) by default.

> A
> problem could occur with a picky compiler or linker that rejects modules
> with mismatched function type declarations. Does Clang do that, or
> require or use such a linker? If not, there's no practical problem here.
> If so, it'd be helpful if Clang continued to support its traditional
> behavior that doesn't reject Autoconf's test cases, and for this to be
> the default.

Clang doesn't require such a linker (we work with various system linkers).

> Autoconf arose because one cannot ask something like "Can I call the
> renameat2 function with its usual signature?" in standard C code and one
> must escape into something like the shell to handle such questions. C23
> and GCC and Clang have added a few features to answer such questions,
> such as __has_include. But these features don't go nearly far enough.
> For example, __has_include tells me only whether the include file
>  exists; it won't tell me whether I can successfully include
> , or whether  will declare the function 'bar', or whether
> 'bar' will have a signature compatible with my code's calls to 'bar'.
>
> If I could request a single thing from the C23/GCC/Clang side, I'd ask
> for better facilities to be able to ask such questions within C code,
> without using the shell. Then a good chunk of Autoconf could dry up and
> blow away.
>
> I realize that I'm asking for a lot. For example, a traditional
> implementation cannot answer the renameat2 question without consulting
> the linker. That being said, this ability is essential for modular
> programming at the low level, and if compilers don't provide this
> ability Autoconf will simply have to do the best it can, regardless of
> whether it generates source code that relies on undefined behavior.

This would be challenging for an implementation like Clang where we
work with an arbitrary C runtime library (which may be dynamically
loaded on the target machine) and an arbitrary linker.

~Aaron

Re: On time64 and Large File Support

[Florian Weimer]

* Sam James:

>> On 11 Nov 2022, at 09:19, Florian Weimer  wrote:
>> 
>> * Sam James:
>> 
>>> In Gentoo, we've been planning out what we should do for time64 on
>>> glibc [0] and concluded that we need some support in glibc for a newer
>>> option. I'll outline why below.
>>> 
>>> Proposal: glibc gains two new build-time configure options:
>>> * --enable-hard-time64
>>> * --enable-hard-lfs
>> 
>> We should define new target triplets for this if it's really required.
>
> I hadn't considered that option. I'll reflect on it. Please let me know
> if you have further thoughts on this.
>
> But that said, these binaries are broken anyway in 2038?

No, I expect users to run them in time-shifted VMs or containers.
Wrong dates are better than no ability to run anything at all.

And whoever can recompile to switch to time64 can just recompile for a
64-bit target.  There are some embedded users that stick to 32-bit for
cost savings, but I think the cost allocation is quite wrong: They save
a bit on per-unit costs, but they do not really contribute back to GNU
(and most don't even use glibc, although there is some use out there).

>> We need to support legacy binaries on i386.  Few libraries are
>> explicitly dual-ABI.  Whether it's safe to switch libraries above glibc
>> to LFS or time64 needs to be evaluated on a per-library basis.  For most
>> distributions, no one is going to do that work, and we have to stick to
>> whathever we are building today.
>
> While I agree, I don't think it's as well-known that it should be that
> these are ABI breaking and require inspection. It's being done ad-hoc
> or in many cases, not at all.
>
> Part of the use of this thread is, if nothing else, we can show upstreams
> and other distros It if they're confused.
>
> It's very easy to miss that a package has started enabling LFS
> and then your opportunity to catch the ABI breakage is gone.
>
> It doesn't help that I (and I suspect most distribution maintainers)
> do all development on x86_64 and hence even ABI diffing isn't
> going to notice. You have to specifically diff the build system, which I
> do, but it's not easy if it's buried deep within gnulib or something.

I really assumed that setting the default in glibc would solve this for
everyone: binary distributions keep using time32, and source-based
embedded distributions can switch to time64 if they want to. *sigh*

I mean, we have things like more compact stack usage through certain
ABI-breaking GCC options.  The kernel can use those safely, but few
people attempt to apply them to userspace.  There, having the right
default in the toolchain is sufficient.  I didn't expect time64 to be
different.

Thanks,
Florian

Re: On time64 and Large File Support

[Sam James]

> On 11 Nov 2022, at 09:19, Florian Weimer  wrote:
> 
> * Sam James:
> 
>> In Gentoo, we've been planning out what we should do for time64 on
>> glibc [0] and concluded that we need some support in glibc for a newer
>> option. I'll outline why below.
>> 
>> Proposal: glibc gains two new build-time configure options:
>> * --enable-hard-time64
>> * --enable-hard-lfs
> 
> We should define new target triplets for this if it's really required.

I hadn't considered that option. I'll reflect on it. Please let me know
if you have further thoughts on this.

But that said, these binaries are broken anyway in 2038?

> We need to support legacy binaries on i386.  Few libraries are
> explicitly dual-ABI.  Whether it's safe to switch libraries above glibc
> to LFS or time64 needs to be evaluated on a per-library basis.  For most
> distributions, no one is going to do that work, and we have to stick to
> whathever we are building today.

While I agree, I don't think it's as well-known that it should be that
these are ABI breaking and require inspection. It's being done ad-hoc
or in many cases, not at all.

Part of the use of this thread is, if nothing else, we can show upstreams
and other distros It if they're confused.

It's very easy to miss that a package has started enabling LFS
and then your opportunity to catch the ABI breakage is gone.

It doesn't help that I (and I suspect most distribution maintainers)
do all development on x86_64 and hence even ABI diffing isn't
going to notice. You have to specifically diff the build system, which I
do, but it's not easy if it's buried deep within gnulib or something.

> 
>> These would hard-enable the relevant #defines within glibc's headers
>> and ensure that any binaries built with such a glibc have both Large
>> File Support (LFS) and time64 support.
>> 
>> I've come to the conclusion it's infeasible to try handle the
>> migration piecemeal. Various mismatches can and will occur (and while
>> it's more likely with time64, it's possible with LFS too) [1].
>> 
>> We're now (possibly) on the eve of an autoconf 2.72 release which contains 
>> two changes
>> of note [2][3]
>> 1. addition of a new AC_SYS_YEAR2038 macro;
>> 2. making AC_SYS_LARGEFILE change behaviour to imply AC_SYS_YEAR2038.
>> 
>> Indeed, the gnulib version of change #2 is exactly how we ended up with
>> wget/gnutls breaking [1]. I feel this shows that the only approach
>> "supported" by glibc right now is untenable.
> 
> AC_SYS_LARGEFILE defaulting to AC_SYS_YEAR2038 is extremely destructive
> for Fedora unfortunately.
> 
> I thought the gnulib change has been reverted?
> 
> I really wish the rest of GNU would talk to glibc maintainers before
> overriding glibc maintainer decisions.  If we cannot revert this in
> autoconf (and gnulib), this will very much endanger the Fedora i386
> port.  Debian will probably be impacted in the same way.
> 

Right, we need to be unified on this, because it's confusing enough
without unilateralism.



signature.asc
Description: Message signed with OpenPGP

Re: How can Autoconf help with the transition to stricter compilation defaults?

[Sam James]

> On 10 Nov 2022, at 17:16, Zack Weinberg  wrote:
> 
> I’m the closest thing Autoconf has to a lead maintainer at present.
> 
> It’s come to my attention (via https://lwn.net/Articles/913505/ and
> https://fedoraproject.org/wiki/Changes/PortingToModernC) that GCC and
> Clang both plan to disable several “legacy” C language features by
> default in a near-future release (GCC 14, Clang 16) (see the Fedora
> wiki link for a list).  I understand that this change potentially
> breaks a lot of old dusty code, and in particular that
> Autoconf-generated configure scripts use constructs that may *silently
> give the wrong answer to a probe* when a stricter compiler is in use.

Thank you for asking. The fixes in git get us there, I think, as far
as you can, with the exception of the stuff you (and I) mention below.

> 
> [...]
> 
> The biggest remaining (potential) problem, that I’m aware of, is that
> AC_CHECK_FUNC unconditionally declares the function we’re probing for
> as ‘char NAME (void)’, and asks the compiler to call it with no
> arguments, regardless of what its prototype actually is.  It is not
> clear to me whether this will still work with the planned changes to
> the compilers.  Both GCC 12 and Clang 14 have on-by-default warnings
> triggered by ‘extern char memcpy(void);’ (or any other standard
> library function whose prototype is coded into the compiler) and this
> already causes problems for people who run configure scripts with
> CC='cc -Werror'.  Unfortunately this is very hard to fix — we would
> have to build a comprehensive list of library functions into Autoconf,
> mapping each to either its documented prototype or to a header where
> it ought to be declared; in the latter case we would also have to make
> e.g. AC_CHECK_FUNCS([getaddrinfo]) imply AC_CHECK_HEADERS([sys/types.h
> sys/socket.h netdb.h]) which might mess up configure scripts that
> aren’t expecting headers to be probed at that point.
> 
> How important do you think it is for this to be fixed?
> 
> Are there any other changes you would like to see in a near-future
> Autoconf 2.72 in order to make this transition easier?

This might be a WONTFIX but let me mention it just for
the record:
1. AC_CHECK_FUNCS is, as you've noticed, very noisy.

I would support having a hardcoded list for certain CHOSTs
as Rich suggests to reduce noise, but I don't think we can
do this accurately very quickly.

I think for Gentoo's efforts, I might just build up a set
of cache variables for glibc/musl on each arch to
reduce the noise in logs. But that's time consuming
and brittle still, so I'm not sure.

(Note that Gentoo and Fedora are taking two complementary
but different approaches here:
we're diffing config.logs and other compiler
output, in addition to build logs, while Florian for Fedora
Is building a list of functions which *we know* are available
In a specific environment and patching gcc to spit out
logs when something in said list is missing. This mitigates
noise for things like functions in libbsd, which I'm finding
a bit of a pain.)

I'll see what others say.

2. I often have to set the following cache variables to
reduce noise in logs:
* ac_cv_c_undeclared_builtin_options="none needed"
* ac_cv_header_sys_types_h_makedev=no
* gl_cv_compiler_check_decl_option="-Werror=implicit-function-declaration" 
(obviously this is gnulib)
* gl_cv_minmax_in_limits_h=no

I don't know if we can do anything to make these tests smarter
or just leave it as-is. It's fine if we can't, as exporting the cache
vars is not a bad workaround for us when doing testing.

> 
> zw
> 
> p.s. GCC and Clang folks: As long as you’re changing the defaults out
> from under people, can you please also remove the last few predefined
> user-namespace macros (-Dlinux, -Dunix, -Darm, etc) from all the
> -std=gnuXX modes?

I support this as well. This kind of thing has led to endless
bugs in userland, see https://reviews.llvm.org/D137511.



signature.asc
Description: Message signed with OpenPGP

RE: [PATCH] Basic support for checking NFSv4 ACLs in Linux

[Ondrej Valousek]

Hi Andreas,

> Historically, I've suggested taking care of these kinds of things in the 
> richacl project, but that effort has been shot down upstream, and that 
> project has been dead for a long time.

I think I have complete picture now. I also think that eventually I can 
eventually fix gnulib (hopefully the gnulib folks stay supportive) as there are 
already some pieces of code for AIX that I could possibly reuse.

Once I have something I'll kindly ask you for your opinion again.

Thanks,
Ondrej