Re: [musl] Re: [PATCH resent] uapi libc compat: allow non-glibc to opt out of uapi definitions

[Felix Janda]

Carlos O'Donell wrote:
> On 04/25/2017 02:37 AM, Hauke Mehrtens wrote:
> > 
> > 
> > On 03/08/2017 01:46 PM, David Woodhouse wrote:
> >> On Fri, 2016-11-11 at 07:08 -0500, Felix Janda wrote:
> >>> Currently, libc-compat.h detects inclusion of specific glibc headers,
> >>> and defines corresponding _UAPI_DEF_* macros, which in turn are used in
> >>> uapi headers to prevent definition of conflicting structures/constants.
> >>> There is no such detection for other c libraries, for them the
> >>> _UAPI_DEF_* macros are always defined as 1, and so none of the possibly
> >>> conflicting definitions are suppressed.
> >>>
> >>> This patch enables non-glibc c libraries to request the suppression of
> >>> any specific interface by defining the corresponding _UAPI_DEF_* macro
> >>> as 0.
> >>
> >> Ick. It's fairly horrid for kernel headers to be reacting to __GLIBC__
> >> in any way. That's just wrong.
> >>
> >> It makes more sense for C libraries to define the __UAPI_DEF_xxx for
> >> themselves as and when they add their own support for certain things,
> >> and for the kernel not to have incestuous knowledge of them.
> >>
> >> The part you add here in the #else /* !__GLIBC__ */ part is what we
> >> should do at *all* times.
> >>
> >> I understand that we'll want to grandfather in the glibc horridness,
> >> but let's make it clear that that's what it is, by letting it set the
> >> appropriate __UAPI_DEF_xxx macros to zero, and then continue through to
> >> your new part. Something like this (incremental to yours):
> > 
> > Felix's and this change and are looking better than my patches here:
> > https://lkml.org/lkml/2017/3/12/235
> > 
> > Is someone working on brining this into the mainline kernel?
> > 
> > Is it correct that only the comments should be improved?
> > musl only supports including the musl header files before the kernel
> > anyway, I assume that it is not needed to make the kernel uapi code
> > support also the other order.
> 
> Please repost to linux-api so other relevant C library authors can review
> the changes and comment on how they might be harmonized.

>From the beginning, this patch was CCed to linux-api. Let me repost
anyway with a new (hopefully clearer) commit message.

> Whether or not you  support both inclusion orders, kernel first, or libc 
> first,
> is a property of your libc implementation.
> 
> Today glibc aims to support both inclusion orders since we see applications
> with either order, and the ordering should not matter in this case. You either
> get one or the other without needing to know any special rules about header
> ordering.

This patch does not change anything for glibc (or uclibc), it just, in
a not very intrusive way, improves the situation for any other libc.

Felix

Re: [musl] Re: [PATCH resent] uapi libc compat: allow non-glibc to opt out of uapi definitions

[Felix Janda]

Carlos O'Donell wrote:
> On 04/25/2017 02:37 AM, Hauke Mehrtens wrote:
> > 
> > 
> > On 03/08/2017 01:46 PM, David Woodhouse wrote:
> >> On Fri, 2016-11-11 at 07:08 -0500, Felix Janda wrote:
> >>> Currently, libc-compat.h detects inclusion of specific glibc headers,
> >>> and defines corresponding _UAPI_DEF_* macros, which in turn are used in
> >>> uapi headers to prevent definition of conflicting structures/constants.
> >>> There is no such detection for other c libraries, for them the
> >>> _UAPI_DEF_* macros are always defined as 1, and so none of the possibly
> >>> conflicting definitions are suppressed.
> >>>
> >>> This patch enables non-glibc c libraries to request the suppression of
> >>> any specific interface by defining the corresponding _UAPI_DEF_* macro
> >>> as 0.
> >>
> >> Ick. It's fairly horrid for kernel headers to be reacting to __GLIBC__
> >> in any way. That's just wrong.
> >>
> >> It makes more sense for C libraries to define the __UAPI_DEF_xxx for
> >> themselves as and when they add their own support for certain things,
> >> and for the kernel not to have incestuous knowledge of them.
> >>
> >> The part you add here in the #else /* !__GLIBC__ */ part is what we
> >> should do at *all* times.
> >>
> >> I understand that we'll want to grandfather in the glibc horridness,
> >> but let's make it clear that that's what it is, by letting it set the
> >> appropriate __UAPI_DEF_xxx macros to zero, and then continue through to
> >> your new part. Something like this (incremental to yours):
> > 
> > Felix's and this change and are looking better than my patches here:
> > https://lkml.org/lkml/2017/3/12/235
> > 
> > Is someone working on brining this into the mainline kernel?
> > 
> > Is it correct that only the comments should be improved?
> > musl only supports including the musl header files before the kernel
> > anyway, I assume that it is not needed to make the kernel uapi code
> > support also the other order.
> 
> Please repost to linux-api so other relevant C library authors can review
> the changes and comment on how they might be harmonized.

>From the beginning, this patch was CCed to linux-api. Let me repost
anyway with a new (hopefully clearer) commit message.

> Whether or not you  support both inclusion orders, kernel first, or libc 
> first,
> is a property of your libc implementation.
> 
> Today glibc aims to support both inclusion orders since we see applications
> with either order, and the ordering should not matter in this case. You either
> get one or the other without needing to know any special rules about header
> ordering.

This patch does not change anything for glibc (or uclibc), it just, in
a not very intrusive way, improves the situation for any other libc.

Felix

Re: [musl] Re: [PATCH resent] uapi libc compat: allow non-glibc to opt out of uapi definitions

[Florian Weimer]

On 04/25/2017 02:29 PM, Carlos O'Donell wrote:
> In glibc right now we support including linux or glibc header files first,
> and this has always been a requirement from the start. This requirement 
> dictates
> that the kernel know which libc it's being used with so it can tailor 
> coordination.

“right now we support” is not a support commitment, but more of an
aspiration, right?

A lot of combinations are broken, especially when kernel headers are
included first.  Maybe it's time for us to admit defeat and come up with
a simpler scheme which actually works.

Thanks,
Florian

Re: [musl] Re: [PATCH resent] uapi libc compat: allow non-glibc to opt out of uapi definitions

[Florian Weimer]

On 04/25/2017 02:29 PM, Carlos O'Donell wrote:
> In glibc right now we support including linux or glibc header files first,
> and this has always been a requirement from the start. This requirement 
> dictates
> that the kernel know which libc it's being used with so it can tailor 
> coordination.

“right now we support” is not a support commitment, but more of an
aspiration, right?

A lot of combinations are broken, especially when kernel headers are
included first.  Maybe it's time for us to admit defeat and come up with
a simpler scheme which actually works.

Thanks,
Florian

Re: [musl] Re: [PATCH resent] uapi libc compat: allow non-glibc to opt out of uapi definitions

[Rich Felker]

On Tue, Apr 25, 2017 at 08:29:00AM -0400, Carlos O'Donell wrote:
> On 04/25/2017 02:45 AM, Hauke Mehrtens wrote:
> > On 03/08/2017 05:39 PM, Carlos O'Donell wrote:
> >> Any header needing compat with a libc includes libc-compat.h (per the 
> >> documented way the model works). With this patch any included linux kernel
> >> header that also includes libc-compat.h would immediately define all 
> >> the __UAPI_DEF_* constants to 1 as-if it had defined those structures, 
> >> but it has not.
> >>
> >> For example, with these two patches applied, the inclusion of linux/if.h
> >> would define __UAPI_DEF_XATTR to 1, but linux/if.h has not defined
> >> XATTR_CREATE or other constants, so a subsequent inclusion sys/xattrs.h
> >> from userspace would _not_ define XATTR_CREATE because __UAPI_DEF_XATTR set
> >> to 1 indicates the kernel has.
> >>
> >> I don't want to read into the model you are proposing and would rather you
> >> document the semantics clearly so we can all see what you mean.
> > 
> > What about moving the code from libc-comapt.h into the specific header
> > files? This way including linux/if.h would not have an impact on
> > __UAPI_DEF_XATTR, because this is defined in linux/xattr.h. We would
> > still have a problem when the specific Linux header file gets included
> > before the libc header file, but at least musl does not support this anyway.
> 
> The point of libc-compat.h is to contain the libc-related logic to a single 
> header
> where it can be reviewed easily as a whole for each libc.
> 
> Headers that include libc-compat.h need not have any libc-related logic, they 
> need
> only guard their structures with the appropriate __UAPI_DEF* macros per the 
> rules
> described in libc-compat.h.
> 
> This way we minimize any changes to the header files and keep the complex
> logic in one place where the libc authors can discuss it.
> 
> In glibc right now we support including linux or glibc header files first,
> and this has always been a requirement from the start. This requirement 
> dictates
> that the kernel know which libc it's being used with so it can tailor 
> coordination.
> 
> If musl only needs header coordination in one direction, then support only 
> that
> direction, but please do not presume to simplify the code by deleting a bunch 
> of
> things that were worked into the kernel to ensure header inclusion ordering 
> works
> in both ways.

Agreed.

On the musl side, we really don't want to be playing cat-and-mouse
having to follow every kernel change and rework things when subtle
differences from kernel-provided definitions might conflict. Saying
(with a macro) "we've got this, please don't try to redefine it" is
easy and maitenance-free; trying to make do with a definition that may
or may not be entirely compatible with libc types or namespace
constraints is nontrivial, and I'd rather it (including kernel defs
first) just not work from the outset than break somewhere down the
line and turn into an argument over whether it needs to be fixed and
if so how.

But none of this justifies breaking stuff that's working for glibc or
preventing them from cleanly supporting both orders.

Rich

Re: [musl] Re: [PATCH resent] uapi libc compat: allow non-glibc to opt out of uapi definitions

[Rich Felker]

On Tue, Apr 25, 2017 at 08:29:00AM -0400, Carlos O'Donell wrote:
> On 04/25/2017 02:45 AM, Hauke Mehrtens wrote:
> > On 03/08/2017 05:39 PM, Carlos O'Donell wrote:
> >> Any header needing compat with a libc includes libc-compat.h (per the 
> >> documented way the model works). With this patch any included linux kernel
> >> header that also includes libc-compat.h would immediately define all 
> >> the __UAPI_DEF_* constants to 1 as-if it had defined those structures, 
> >> but it has not.
> >>
> >> For example, with these two patches applied, the inclusion of linux/if.h
> >> would define __UAPI_DEF_XATTR to 1, but linux/if.h has not defined
> >> XATTR_CREATE or other constants, so a subsequent inclusion sys/xattrs.h
> >> from userspace would _not_ define XATTR_CREATE because __UAPI_DEF_XATTR set
> >> to 1 indicates the kernel has.
> >>
> >> I don't want to read into the model you are proposing and would rather you
> >> document the semantics clearly so we can all see what you mean.
> > 
> > What about moving the code from libc-comapt.h into the specific header
> > files? This way including linux/if.h would not have an impact on
> > __UAPI_DEF_XATTR, because this is defined in linux/xattr.h. We would
> > still have a problem when the specific Linux header file gets included
> > before the libc header file, but at least musl does not support this anyway.
> 
> The point of libc-compat.h is to contain the libc-related logic to a single 
> header
> where it can be reviewed easily as a whole for each libc.
> 
> Headers that include libc-compat.h need not have any libc-related logic, they 
> need
> only guard their structures with the appropriate __UAPI_DEF* macros per the 
> rules
> described in libc-compat.h.
> 
> This way we minimize any changes to the header files and keep the complex
> logic in one place where the libc authors can discuss it.
> 
> In glibc right now we support including linux or glibc header files first,
> and this has always been a requirement from the start. This requirement 
> dictates
> that the kernel know which libc it's being used with so it can tailor 
> coordination.
> 
> If musl only needs header coordination in one direction, then support only 
> that
> direction, but please do not presume to simplify the code by deleting a bunch 
> of
> things that were worked into the kernel to ensure header inclusion ordering 
> works
> in both ways.

Agreed.

On the musl side, we really don't want to be playing cat-and-mouse
having to follow every kernel change and rework things when subtle
differences from kernel-provided definitions might conflict. Saying
(with a macro) "we've got this, please don't try to redefine it" is
easy and maitenance-free; trying to make do with a definition that may
or may not be entirely compatible with libc types or namespace
constraints is nontrivial, and I'd rather it (including kernel defs
first) just not work from the outset than break somewhere down the
line and turn into an argument over whether it needs to be fixed and
if so how.

But none of this justifies breaking stuff that's working for glibc or
preventing them from cleanly supporting both orders.

Rich

Re: [musl] Re: [PATCH resent] uapi libc compat: allow non-glibc to opt out of uapi definitions

[Carlos O'Donell]

On 04/25/2017 02:45 AM, Hauke Mehrtens wrote:
> On 03/08/2017 05:39 PM, Carlos O'Donell wrote:
>> Any header needing compat with a libc includes libc-compat.h (per the 
>> documented way the model works). With this patch any included linux kernel
>> header that also includes libc-compat.h would immediately define all 
>> the __UAPI_DEF_* constants to 1 as-if it had defined those structures, 
>> but it has not.
>>
>> For example, with these two patches applied, the inclusion of linux/if.h
>> would define __UAPI_DEF_XATTR to 1, but linux/if.h has not defined
>> XATTR_CREATE or other constants, so a subsequent inclusion sys/xattrs.h
>> from userspace would _not_ define XATTR_CREATE because __UAPI_DEF_XATTR set
>> to 1 indicates the kernel has.
>>
>> I don't want to read into the model you are proposing and would rather you
>> document the semantics clearly so we can all see what you mean.
> 
> What about moving the code from libc-comapt.h into the specific header
> files? This way including linux/if.h would not have an impact on
> __UAPI_DEF_XATTR, because this is defined in linux/xattr.h. We would
> still have a problem when the specific Linux header file gets included
> before the libc header file, but at least musl does not support this anyway.

The point of libc-compat.h is to contain the libc-related logic to a single 
header
where it can be reviewed easily as a whole for each libc.

Headers that include libc-compat.h need not have any libc-related logic, they 
need
only guard their structures with the appropriate __UAPI_DEF* macros per the 
rules
described in libc-compat.h.

This way we minimize any changes to the header files and keep the complex
logic in one place where the libc authors can discuss it.

In glibc right now we support including linux or glibc header files first,
and this has always been a requirement from the start. This requirement dictates
that the kernel know which libc it's being used with so it can tailor 
coordination.

If musl only needs header coordination in one direction, then support only that
direction, but please do not presume to simplify the code by deleting a bunch of
things that were worked into the kernel to ensure header inclusion ordering 
works
in both ways.

I have lots of customers writing code that includes kernel headers and letting 
them
include headers safely in any order where both headers provide the same define 
is
the simplest thing.

If you have a suggestion, please describe your proposed model in detail, and 
provide
a patch that explains and show how it works.

-- 
Cheers,
Carlos.

Re: [musl] Re: [PATCH resent] uapi libc compat: allow non-glibc to opt out of uapi definitions

[Carlos O'Donell]

On 04/25/2017 02:45 AM, Hauke Mehrtens wrote:
> On 03/08/2017 05:39 PM, Carlos O'Donell wrote:
>> Any header needing compat with a libc includes libc-compat.h (per the 
>> documented way the model works). With this patch any included linux kernel
>> header that also includes libc-compat.h would immediately define all 
>> the __UAPI_DEF_* constants to 1 as-if it had defined those structures, 
>> but it has not.
>>
>> For example, with these two patches applied, the inclusion of linux/if.h
>> would define __UAPI_DEF_XATTR to 1, but linux/if.h has not defined
>> XATTR_CREATE or other constants, so a subsequent inclusion sys/xattrs.h
>> from userspace would _not_ define XATTR_CREATE because __UAPI_DEF_XATTR set
>> to 1 indicates the kernel has.
>>
>> I don't want to read into the model you are proposing and would rather you
>> document the semantics clearly so we can all see what you mean.
> 
> What about moving the code from libc-comapt.h into the specific header
> files? This way including linux/if.h would not have an impact on
> __UAPI_DEF_XATTR, because this is defined in linux/xattr.h. We would
> still have a problem when the specific Linux header file gets included
> before the libc header file, but at least musl does not support this anyway.

The point of libc-compat.h is to contain the libc-related logic to a single 
header
where it can be reviewed easily as a whole for each libc.

Headers that include libc-compat.h need not have any libc-related logic, they 
need
only guard their structures with the appropriate __UAPI_DEF* macros per the 
rules
described in libc-compat.h.

This way we minimize any changes to the header files and keep the complex
logic in one place where the libc authors can discuss it.

In glibc right now we support including linux or glibc header files first,
and this has always been a requirement from the start. This requirement dictates
that the kernel know which libc it's being used with so it can tailor 
coordination.

If musl only needs header coordination in one direction, then support only that
direction, but please do not presume to simplify the code by deleting a bunch of
things that were worked into the kernel to ensure header inclusion ordering 
works
in both ways.

I have lots of customers writing code that includes kernel headers and letting 
them
include headers safely in any order where both headers provide the same define 
is
the simplest thing.

If you have a suggestion, please describe your proposed model in detail, and 
provide
a patch that explains and show how it works.

-- 
Cheers,
Carlos.

Re: [musl] Re: [PATCH resent] uapi libc compat: allow non-glibc to opt out of uapi definitions

[Carlos O'Donell]

On 04/25/2017 02:37 AM, Hauke Mehrtens wrote:
> 
> 
> On 03/08/2017 01:46 PM, David Woodhouse wrote:
>> On Fri, 2016-11-11 at 07:08 -0500, Felix Janda wrote:
>>> Currently, libc-compat.h detects inclusion of specific glibc headers,
>>> and defines corresponding _UAPI_DEF_* macros, which in turn are used in
>>> uapi headers to prevent definition of conflicting structures/constants.
>>> There is no such detection for other c libraries, for them the
>>> _UAPI_DEF_* macros are always defined as 1, and so none of the possibly
>>> conflicting definitions are suppressed.
>>>
>>> This patch enables non-glibc c libraries to request the suppression of
>>> any specific interface by defining the corresponding _UAPI_DEF_* macro
>>> as 0.
>>
>> Ick. It's fairly horrid for kernel headers to be reacting to __GLIBC__
>> in any way. That's just wrong.
>>
>> It makes more sense for C libraries to define the __UAPI_DEF_xxx for
>> themselves as and when they add their own support for certain things,
>> and for the kernel not to have incestuous knowledge of them.
>>
>> The part you add here in the #else /* !__GLIBC__ */ part is what we
>> should do at *all* times.
>>
>> I understand that we'll want to grandfather in the glibc horridness,
>> but let's make it clear that that's what it is, by letting it set the
>> appropriate __UAPI_DEF_xxx macros to zero, and then continue through to
>> your new part. Something like this (incremental to yours):
> 
> Felix's and this change and are looking better than my patches here:
> https://lkml.org/lkml/2017/3/12/235
> 
> Is someone working on brining this into the mainline kernel?
> 
> Is it correct that only the comments should be improved?
> musl only supports including the musl header files before the kernel
> anyway, I assume that it is not needed to make the kernel uapi code
> support also the other order.

Please repost to linux-api so other relevant C library authors can review
the changes and comment on how they might be harmonized.

Whether or not you  support both inclusion orders, kernel first, or libc first,
is a property of your libc implementation.

Today glibc aims to support both inclusion orders since we see applications
with either order, and the ordering should not matter in this case. You either
get one or the other without needing to know any special rules about header
ordering.

-- 
Cheers,
Carlos.

Re: [musl] Re: [PATCH resent] uapi libc compat: allow non-glibc to opt out of uapi definitions

[Carlos O'Donell]

On 04/25/2017 02:37 AM, Hauke Mehrtens wrote:
> 
> 
> On 03/08/2017 01:46 PM, David Woodhouse wrote:
>> On Fri, 2016-11-11 at 07:08 -0500, Felix Janda wrote:
>>> Currently, libc-compat.h detects inclusion of specific glibc headers,
>>> and defines corresponding _UAPI_DEF_* macros, which in turn are used in
>>> uapi headers to prevent definition of conflicting structures/constants.
>>> There is no such detection for other c libraries, for them the
>>> _UAPI_DEF_* macros are always defined as 1, and so none of the possibly
>>> conflicting definitions are suppressed.
>>>
>>> This patch enables non-glibc c libraries to request the suppression of
>>> any specific interface by defining the corresponding _UAPI_DEF_* macro
>>> as 0.
>>
>> Ick. It's fairly horrid for kernel headers to be reacting to __GLIBC__
>> in any way. That's just wrong.
>>
>> It makes more sense for C libraries to define the __UAPI_DEF_xxx for
>> themselves as and when they add their own support for certain things,
>> and for the kernel not to have incestuous knowledge of them.
>>
>> The part you add here in the #else /* !__GLIBC__ */ part is what we
>> should do at *all* times.
>>
>> I understand that we'll want to grandfather in the glibc horridness,
>> but let's make it clear that that's what it is, by letting it set the
>> appropriate __UAPI_DEF_xxx macros to zero, and then continue through to
>> your new part. Something like this (incremental to yours):
> 
> Felix's and this change and are looking better than my patches here:
> https://lkml.org/lkml/2017/3/12/235
> 
> Is someone working on brining this into the mainline kernel?
> 
> Is it correct that only the comments should be improved?
> musl only supports including the musl header files before the kernel
> anyway, I assume that it is not needed to make the kernel uapi code
> support also the other order.

Please repost to linux-api so other relevant C library authors can review
the changes and comment on how they might be harmonized.

Whether or not you  support both inclusion orders, kernel first, or libc first,
is a property of your libc implementation.

Today glibc aims to support both inclusion orders since we see applications
with either order, and the ordering should not matter in this case. You either
get one or the other without needing to know any special rules about header
ordering.

-- 
Cheers,
Carlos.

Re: [musl] Re: [PATCH resent] uapi libc compat: allow non-glibc to opt out of uapi definitions

[Hauke Mehrtens]

On 03/08/2017 05:39 PM, Carlos O'Donell wrote:
> Any header needing compat with a libc includes libc-compat.h (per the 
> documented way the model works). With this patch any included linux kernel
> header that also includes libc-compat.h would immediately define all 
> the __UAPI_DEF_* constants to 1 as-if it had defined those structures, 
> but it has not.
> 
> For example, with these two patches applied, the inclusion of linux/if.h
> would define __UAPI_DEF_XATTR to 1, but linux/if.h has not defined
> XATTR_CREATE or other constants, so a subsequent inclusion sys/xattrs.h
> from userspace would _not_ define XATTR_CREATE because __UAPI_DEF_XATTR set
> to 1 indicates the kernel has.
> 
> I don't want to read into the model you are proposing and would rather you
> document the semantics clearly so we can all see what you mean.

What about moving the code from libc-comapt.h into the specific header
files? This way including linux/if.h would not have an impact on
__UAPI_DEF_XATTR, because this is defined in linux/xattr.h. We would
still have a problem when the specific Linux header file gets included
before the libc header file, but at least musl does not support this anyway.

Hauke

Re: [musl] Re: [PATCH resent] uapi libc compat: allow non-glibc to opt out of uapi definitions

[Hauke Mehrtens]

On 03/08/2017 05:39 PM, Carlos O'Donell wrote:
> Any header needing compat with a libc includes libc-compat.h (per the 
> documented way the model works). With this patch any included linux kernel
> header that also includes libc-compat.h would immediately define all 
> the __UAPI_DEF_* constants to 1 as-if it had defined those structures, 
> but it has not.
> 
> For example, with these two patches applied, the inclusion of linux/if.h
> would define __UAPI_DEF_XATTR to 1, but linux/if.h has not defined
> XATTR_CREATE or other constants, so a subsequent inclusion sys/xattrs.h
> from userspace would _not_ define XATTR_CREATE because __UAPI_DEF_XATTR set
> to 1 indicates the kernel has.
> 
> I don't want to read into the model you are proposing and would rather you
> document the semantics clearly so we can all see what you mean.

What about moving the code from libc-comapt.h into the specific header
files? This way including linux/if.h would not have an impact on
__UAPI_DEF_XATTR, because this is defined in linux/xattr.h. We would
still have a problem when the specific Linux header file gets included
before the libc header file, but at least musl does not support this anyway.

Hauke

Re: [musl] Re: [PATCH resent] uapi libc compat: allow non-glibc to opt out of uapi definitions

[Hauke Mehrtens]

On 03/08/2017 01:46 PM, David Woodhouse wrote:
> On Fri, 2016-11-11 at 07:08 -0500, Felix Janda wrote:
>> Currently, libc-compat.h detects inclusion of specific glibc headers,
>> and defines corresponding _UAPI_DEF_* macros, which in turn are used in
>> uapi headers to prevent definition of conflicting structures/constants.
>> There is no such detection for other c libraries, for them the
>> _UAPI_DEF_* macros are always defined as 1, and so none of the possibly
>> conflicting definitions are suppressed.
>>
>> This patch enables non-glibc c libraries to request the suppression of
>> any specific interface by defining the corresponding _UAPI_DEF_* macro
>> as 0.
> 
> Ick. It's fairly horrid for kernel headers to be reacting to __GLIBC__
> in any way. That's just wrong.
> 
> It makes more sense for C libraries to define the __UAPI_DEF_xxx for
> themselves as and when they add their own support for certain things,
> and for the kernel not to have incestuous knowledge of them.
> 
> The part you add here in the #else /* !__GLIBC__ */ part is what we
> should do at *all* times.
> 
> I understand that we'll want to grandfather in the glibc horridness,
> but let's make it clear that that's what it is, by letting it set the
> appropriate __UAPI_DEF_xxx macros to zero, and then continue through to
> your new part. Something like this (incremental to yours):

Felix's and this change and are looking better than my patches here:
https://lkml.org/lkml/2017/3/12/235

Is someone working on brining this into the mainline kernel?

Is it correct that only the comments should be improved?
musl only supports including the musl header files before the kernel
anyway, I assume that it is not needed to make the kernel uapi code
support also the other order.

Hauke

Re: [musl] Re: [PATCH resent] uapi libc compat: allow non-glibc to opt out of uapi definitions

[Hauke Mehrtens]

On 03/08/2017 01:46 PM, David Woodhouse wrote:
> On Fri, 2016-11-11 at 07:08 -0500, Felix Janda wrote:
>> Currently, libc-compat.h detects inclusion of specific glibc headers,
>> and defines corresponding _UAPI_DEF_* macros, which in turn are used in
>> uapi headers to prevent definition of conflicting structures/constants.
>> There is no such detection for other c libraries, for them the
>> _UAPI_DEF_* macros are always defined as 1, and so none of the possibly
>> conflicting definitions are suppressed.
>>
>> This patch enables non-glibc c libraries to request the suppression of
>> any specific interface by defining the corresponding _UAPI_DEF_* macro
>> as 0.
> 
> Ick. It's fairly horrid for kernel headers to be reacting to __GLIBC__
> in any way. That's just wrong.
> 
> It makes more sense for C libraries to define the __UAPI_DEF_xxx for
> themselves as and when they add their own support for certain things,
> and for the kernel not to have incestuous knowledge of them.
> 
> The part you add here in the #else /* !__GLIBC__ */ part is what we
> should do at *all* times.
> 
> I understand that we'll want to grandfather in the glibc horridness,
> but let's make it clear that that's what it is, by letting it set the
> appropriate __UAPI_DEF_xxx macros to zero, and then continue through to
> your new part. Something like this (incremental to yours):

Felix's and this change and are looking better than my patches here:
https://lkml.org/lkml/2017/3/12/235

Is someone working on brining this into the mainline kernel?

Is it correct that only the comments should be improved?
musl only supports including the musl header files before the kernel
anyway, I assume that it is not needed to make the kernel uapi code
support also the other order.

Hauke

Re: [musl] Re: [PATCH resent] uapi libc compat: allow non-glibc to opt out of uapi definitions

[Rich Felker]

On Wed, Mar 08, 2017 at 07:51:29PM -0500, Carlos O'Donell wrote:
> On 03/08/2017 07:14 PM, Szabolcs Nagy wrote:
> > * Carlos O'Donell  [2017-03-08 10:53:00 -0500]:
> >> On 11/11/2016 07:08 AM, Felix Janda wrote:
> >>> fixes the following compiler errors when  is included
> >>> after musl :
> >>>
> >>> ./linux/in6.h:32:8: error: redefinition of 'struct in6_addr'
> >>> ./linux/in6.h:49:8: error: redefinition of 'struct sockaddr_in6'
> >>> ./linux/in6.h:59:8: error: redefinition of 'struct ipv6_mreq'
> >>
> >> Do you have plans for fixing the error when the inclusion order is the 
> >> other way?
> > 
> > the other way (linux header included first) is
> > problematic because linux headers don't follow
> > all the standards the libc follows, they violate
> > namespace rules in their struct definitions, so
> > the libc definitions are necessarily incompatible
> > with them and thus different translation units can
> > end up refering to the same object through
> > incompatible types which is undefined.
> > (even if the abi matches and thus works across
> > the syscall interface, a sufficiently smart
> > toolchain can break such code at link time,
> > and since the libc itself uses its own definitons
> > that's what user code should use too).
> > 
> > there should be a way to include standard conform
> > libc headers and linux headers into the same tu,
> > at least the case when all conflicting definitions
> > come from the libc should work and i think that
> > should be the scope of these libc-compat.h changes.
> > (of course if glibc tries to support arbitrary
> > interleavings then the changes should not break that)
> 
> You can get non-standard defines even when including the
> linux headers _after_ libc headers because linux headers
> should rightly continue to define things that are required
> for linux-specific applications.
> 
> IMO the fact that the UAPI headers may cause problems with
> standards conformance is orthogonal to the discussion of 
> _how_ we fix inclusion order issues.
> 
> Some of the network headers can be used in relative safety
> and need to be used for some applications. It is those cases
> where I'd like to see an inclusion guard design that works
> for both inclusion orders.

The issue has been discussed on our side (musl) and our position so
far is that we don't want to try to support the case of including the
kernel headers before the libc headers, at least not at this time.
It's a big rabbit hole of stuff that could go wrong. This doesn't
preclude the kernel folks trying to make things so that it _can_ be
supported more smoothly.

Rich

Re: [musl] Re: [PATCH resent] uapi libc compat: allow non-glibc to opt out of uapi definitions

[Rich Felker]

On Wed, Mar 08, 2017 at 07:51:29PM -0500, Carlos O'Donell wrote:
> On 03/08/2017 07:14 PM, Szabolcs Nagy wrote:
> > * Carlos O'Donell  [2017-03-08 10:53:00 -0500]:
> >> On 11/11/2016 07:08 AM, Felix Janda wrote:
> >>> fixes the following compiler errors when  is included
> >>> after musl :
> >>>
> >>> ./linux/in6.h:32:8: error: redefinition of 'struct in6_addr'
> >>> ./linux/in6.h:49:8: error: redefinition of 'struct sockaddr_in6'
> >>> ./linux/in6.h:59:8: error: redefinition of 'struct ipv6_mreq'
> >>
> >> Do you have plans for fixing the error when the inclusion order is the 
> >> other way?
> > 
> > the other way (linux header included first) is
> > problematic because linux headers don't follow
> > all the standards the libc follows, they violate
> > namespace rules in their struct definitions, so
> > the libc definitions are necessarily incompatible
> > with them and thus different translation units can
> > end up refering to the same object through
> > incompatible types which is undefined.
> > (even if the abi matches and thus works across
> > the syscall interface, a sufficiently smart
> > toolchain can break such code at link time,
> > and since the libc itself uses its own definitons
> > that's what user code should use too).
> > 
> > there should be a way to include standard conform
> > libc headers and linux headers into the same tu,
> > at least the case when all conflicting definitions
> > come from the libc should work and i think that
> > should be the scope of these libc-compat.h changes.
> > (of course if glibc tries to support arbitrary
> > interleavings then the changes should not break that)
> 
> You can get non-standard defines even when including the
> linux headers _after_ libc headers because linux headers
> should rightly continue to define things that are required
> for linux-specific applications.
> 
> IMO the fact that the UAPI headers may cause problems with
> standards conformance is orthogonal to the discussion of 
> _how_ we fix inclusion order issues.
> 
> Some of the network headers can be used in relative safety
> and need to be used for some applications. It is those cases
> where I'd like to see an inclusion guard design that works
> for both inclusion orders.

The issue has been discussed on our side (musl) and our position so
far is that we don't want to try to support the case of including the
kernel headers before the libc headers, at least not at this time.
It's a big rabbit hole of stuff that could go wrong. This doesn't
preclude the kernel folks trying to make things so that it _can_ be
supported more smoothly.

Rich

Re: [musl] Re: [PATCH resent] uapi libc compat: allow non-glibc to opt out of uapi definitions

[Carlos O'Donell]

On 03/08/2017 11:25 AM, Rich Felker wrote:
> On Wed, Mar 08, 2017 at 10:53:00AM -0500, Carlos O'Donell wrote:
>> On 11/11/2016 07:08 AM, Felix Janda wrote:
>>> Currently, libc-compat.h detects inclusion of specific glibc headers,
>>> and defines corresponding _UAPI_DEF_* macros, which in turn are used in
>>> uapi headers to prevent definition of conflicting structures/constants.
>>> There is no such detection for other c libraries, for them the
>>> _UAPI_DEF_* macros are always defined as 1, and so none of the possibly
>>> conflicting definitions are suppressed.
>>>
>>> This patch enables non-glibc c libraries to request the suppression of
>>> any specific interface by defining the corresponding _UAPI_DEF_* macro
>>> as 0.
>>>
>>> This patch together with the recent musl libc commit
>>>
>>> http://git.musl-libc.org/cgit/musl/commit/?id=04983f2272382af92eb8f8838964ff944fbb8258
>>
>> Would it be possible to amend the musl patch to define the macros to 1.
> 
> I don't follow. They're defined to 0 explicitly to tell the kernel
> headers not to define their own versions of these structs, etc. since
> they would clash. Defining to 1 would have the opposite meaning.

My apologies, I must have misread the original musl patch.

Defining them to a known value is exactly what I was looking for.

The other outstanding questions remain.

-- 
Cheers,
Carlos.

Re: [musl] Re: [PATCH resent] uapi libc compat: allow non-glibc to opt out of uapi definitions

[Carlos O'Donell]

On 03/08/2017 11:25 AM, Rich Felker wrote:
> On Wed, Mar 08, 2017 at 10:53:00AM -0500, Carlos O'Donell wrote:
>> On 11/11/2016 07:08 AM, Felix Janda wrote:
>>> Currently, libc-compat.h detects inclusion of specific glibc headers,
>>> and defines corresponding _UAPI_DEF_* macros, which in turn are used in
>>> uapi headers to prevent definition of conflicting structures/constants.
>>> There is no such detection for other c libraries, for them the
>>> _UAPI_DEF_* macros are always defined as 1, and so none of the possibly
>>> conflicting definitions are suppressed.
>>>
>>> This patch enables non-glibc c libraries to request the suppression of
>>> any specific interface by defining the corresponding _UAPI_DEF_* macro
>>> as 0.
>>>
>>> This patch together with the recent musl libc commit
>>>
>>> http://git.musl-libc.org/cgit/musl/commit/?id=04983f2272382af92eb8f8838964ff944fbb8258
>>
>> Would it be possible to amend the musl patch to define the macros to 1.
> 
> I don't follow. They're defined to 0 explicitly to tell the kernel
> headers not to define their own versions of these structs, etc. since
> they would clash. Defining to 1 would have the opposite meaning.

My apologies, I must have misread the original musl patch.

Defining them to a known value is exactly what I was looking for.

The other outstanding questions remain.

-- 
Cheers,
Carlos.

Re: [musl] Re: [PATCH resent] uapi libc compat: allow non-glibc to opt out of uapi definitions

[Rich Felker]

On Wed, Mar 08, 2017 at 10:53:00AM -0500, Carlos O'Donell wrote:
> On 11/11/2016 07:08 AM, Felix Janda wrote:
> > Currently, libc-compat.h detects inclusion of specific glibc headers,
> > and defines corresponding _UAPI_DEF_* macros, which in turn are used in
> > uapi headers to prevent definition of conflicting structures/constants.
> > There is no such detection for other c libraries, for them the
> > _UAPI_DEF_* macros are always defined as 1, and so none of the possibly
> > conflicting definitions are suppressed.
> > 
> > This patch enables non-glibc c libraries to request the suppression of
> > any specific interface by defining the corresponding _UAPI_DEF_* macro
> > as 0.
> > 
> > This patch together with the recent musl libc commit
> > 
> > http://git.musl-libc.org/cgit/musl/commit/?id=04983f2272382af92eb8f8838964ff944fbb8258
> 
> Would it be possible to amend the musl patch to define the macros to 1.

I don't follow. They're defined to 0 explicitly to tell the kernel
headers not to define their own versions of these structs, etc. since
they would clash. Defining to 1 would have the opposite meaning.

Rich

Re: [musl] Re: [PATCH resent] uapi libc compat: allow non-glibc to opt out of uapi definitions

[Rich Felker]

On Wed, Mar 08, 2017 at 10:53:00AM -0500, Carlos O'Donell wrote:
> On 11/11/2016 07:08 AM, Felix Janda wrote:
> > Currently, libc-compat.h detects inclusion of specific glibc headers,
> > and defines corresponding _UAPI_DEF_* macros, which in turn are used in
> > uapi headers to prevent definition of conflicting structures/constants.
> > There is no such detection for other c libraries, for them the
> > _UAPI_DEF_* macros are always defined as 1, and so none of the possibly
> > conflicting definitions are suppressed.
> > 
> > This patch enables non-glibc c libraries to request the suppression of
> > any specific interface by defining the corresponding _UAPI_DEF_* macro
> > as 0.
> > 
> > This patch together with the recent musl libc commit
> > 
> > http://git.musl-libc.org/cgit/musl/commit/?id=04983f2272382af92eb8f8838964ff944fbb8258
> 
> Would it be possible to amend the musl patch to define the macros to 1.

I don't follow. They're defined to 0 explicitly to tell the kernel
headers not to define their own versions of these structs, etc. since
they would clash. Defining to 1 would have the opposite meaning.

Rich