Re: NULL pointer arithmetic issues
At Mon, 9 Mar 2020 17:36:24 +0100, Joerg Sonnenberger wrote: Subject: Re: NULL pointer arithmetic issues > > I consider it as something even worse. Just like the case of passing > NULL pointers to memcpy and friends with zero as size, this > interpretation / restriction in the standard is actively harmful to some > code for the sake of potential optimisation opportunities in other code. > It seems to be a poor choice at that. I.e. it requires adding > conditional branches for something that behaves sanely everywhere but > may the DS9k. Indeed. I way the very existence of anything called "Undefined Behaviour" and its exploitation by optimizers is evil. (by definition, especially if we accept as valid the claim that "Premature optimization is the root of all evil in programming" -- this is of course a little bit of a stretch since my claim could be twisted to say that any and all automatic optimzation by a compiler or toolchain is evil, but of course that's not exactly my intent -- normal optimization which does not change the behaviour and intent of the code is, IMO, OK, but defining "intent" is obviously the problem) So in Standard C all "Undefined Behaviour" should be changed to "Implementation Defined" and it should be required that the implementation is not allowed to abuse any such things for the evil purpose of premature optimzation. For this kind of thing adding an integer to a pointer (or the equivalent, e.g. taking the address of a field in a struct pointed to by a nil pointer) should always do just that, even if the pointer can be proven to be a nil pointer at compile time. It is wrong to do anything else, and absolute insanity to remove any other code just because the compiler assumes SIGSEGV would/should/could happen before the other code gets a chance to run. -- Greg A. Woods Kelowna, BC +1 250 762-7675 RoboHack Planix, Inc. Avoncote Farms pgpGIvFrvwd81.pgp Description: OpenPGP Digital Signature
Re: NULL pointer arithmetic issues
On Mon, Mar 9, 2020 at 2:53 PM Martin Husemann wrote: > > On Mon, Mar 09, 2020 at 12:41:37PM -0400, Aaron Ballman wrote: > > > You could view NULL as a special pointer pointing to an inaccessible > > > zero sized object. Adding 0 to it still points to the same special > > > non-object. I guess that is how the C++ folks see it. > > > > This wording has always been UB in C. It stopped being UB in C++20 but > > was UB prior to that. I did not see a core issue pertaining to this > > (though it's possible I've missed it -- the C++ core issues list is a > > bit hard to track these days), so I am not certain why this change was > > made so recently. > > AFAICT it has always been well defined in C++, though the wording changed > a bit. In C++11 (n3337) it is 5.7 note 7: "If the value of 0 is added or > subtracted from a pointer value, the result compares equal to the original > pointer value." Same text but moved one note down to 8 in n3797 (C++14). > N4659 (C++17) makes it explicit for NULL pointers in 8.7 note 7 (as > Taylor cited earlier in this thread): "If the value 0 is added or subtracted > from a null pointer value, the result is a null pointer value." Yes, that's my mistake, sorry about that misinformation! (Taylor also brought it up off-list.) ~~Aaron > > Martin
Re: NULL pointer arithmetic issues
On Mon, Mar 09, 2020 at 12:41:37PM -0400, Aaron Ballman wrote: > > You could view NULL as a special pointer pointing to an inaccessible > > zero sized object. Adding 0 to it still points to the same special > > non-object. I guess that is how the C++ folks see it. > > This wording has always been UB in C. It stopped being UB in C++20 but > was UB prior to that. I did not see a core issue pertaining to this > (though it's possible I've missed it -- the C++ core issues list is a > bit hard to track these days), so I am not certain why this change was > made so recently. AFAICT it has always been well defined in C++, though the wording changed a bit. In C++11 (n3337) it is 5.7 note 7: "If the value of 0 is added or subtracted from a pointer value, the result compares equal to the original pointer value." Same text but moved one note down to 8 in n3797 (C++14). N4659 (C++17) makes it explicit for NULL pointers in 8.7 note 7 (as Taylor cited earlier in this thread): "If the value 0 is added or subtracted from a null pointer value, the result is a null pointer value." Martin
Re: NULL pointer arithmetic issues
On Mon, Mar 9, 2020 at 12:36 PM Joerg Sonnenberger wrote: > > On Mon, Mar 09, 2020 at 09:50:50AM -0400, Aaron Ballman wrote: > > On Sun, Mar 8, 2020 at 2:30 PM Taylor R Campbell > > > I ask because in principle a conformant implementation could compile > > > the NetBSD kernel into a useless blob that does nothing -- we rely on > > > all sorts of behaviour relative to a real physical machine that is not > > > defined by the letter of the standard, like inline asm, or converting > > > integers from the VM system's virtual address allocator into pointers > > > to objects. But such an implementation would not be useful. > > > > Whether an optimizer elects to use this form of UB to make > > optimization decisions is a matter of QoI. My personal feeling is that > > I don't trust this sort of optimization -- it takes code the > > programmer wrote and makes it behave in a fundamentally different > > manner. I'm in support of UBSan diagnosing these constructs because it > > is UB and an optimizer is definitely allowed to optimize based on it > > but I wouldn't be in support of an optimizer that aggressively > > optimizes on this. > > I consider it as something even worse. Just like the case of passing > NULL pointers to memcpy and friends with zero as size, this > interpretation / restriction in the standard is actively harmful to some > code for the sake of potential optimisation opportunities in other code. > It seems to be a poor choice at that. I.e. it requires adding > conditional branches for something that behaves sanely everywhere but > may the DS9k. +1 -- I'm personally not a fan of allowing these kinds of optimizations. I refer to the entire class of similar operations as "exploiting undefined behavior" for this reason. ~~Aaron > > Joerg
Re: NULL pointer arithmetic issues
On Mon, Mar 9, 2020 at 11:51 AM Martin Husemann wrote: > > On Mon, Mar 09, 2020 at 09:38:31AM -0400, Aaron Ballman wrote: > > The way I read this is: > > > > "If the pointer operand points to an element of an array object" -- it > > does not (null is not a valid array object). > > "Moreover, if the expression P points to the last element of an array > > object" -- it does not (null is not a valid array object). > > "If both the pointer operand and the result point to elements of the > > same array object, or one past the last element of the array > > object..." -- it does not (there is no valid array so they cannot > > point to elements of the same array). > > All fine. > > > "...otherwise, the behavior is undefined." -- this is where we hit the UB. > > But it started with "If the pointer operand points to an element of an > array object" and my reading is that the whole section does not apply. I don't think that's the correct predicate to read. The full sentence I reach is: "If both the pointer operand and the result point to elements of the same array object, or one past the last element of the array object, the evaluation shall not produce an overflow; otherwise, the behavior is undefined." The pointer operand and the result do not point to elements of the same array object or one past the last element, so I hit the "otherwise" case. But even under your interpretation, I see the result as being UB -- anything the standard doesn't explicitly specify is also undefined behavior, so if the whole section doesn't apply, nothing else specifies the behavior. > NULL is special, and we did not change it's value, so it stays a NULL > pointer. The section you cited just does not tell anything about arithmetic > on NULL pointers - and the wording makes it sound like this is more an > oversight than a concious descision. That was why I asked on the reflectors; there was sufficient confusion from people reading the standard to warrant asking "are we reading this properly" and the answer was: yes, we are interpreting it properly and that NULL + 0 is UB. > Eespecially adding or subtracting zero. > > You could view NULL as a special pointer pointing to an inaccessible > zero sized object. Adding 0 to it still points to the same special > non-object. I guess that is how the C++ folks see it. This wording has always been UB in C. It stopped being UB in C++20 but was UB prior to that. I did not see a core issue pertaining to this (though it's possible I've missed it -- the C++ core issues list is a bit hard to track these days), so I am not certain why this change was made so recently. ~~Aaron > > > Martin
Re: NULL pointer arithmetic issues
On Mon, Mar 09, 2020 at 09:50:50AM -0400, Aaron Ballman wrote: > On Sun, Mar 8, 2020 at 2:30 PM Taylor R Campbell > > I ask because in principle a conformant implementation could compile > > the NetBSD kernel into a useless blob that does nothing -- we rely on > > all sorts of behaviour relative to a real physical machine that is not > > defined by the letter of the standard, like inline asm, or converting > > integers from the VM system's virtual address allocator into pointers > > to objects. But such an implementation would not be useful. > > Whether an optimizer elects to use this form of UB to make > optimization decisions is a matter of QoI. My personal feeling is that > I don't trust this sort of optimization -- it takes code the > programmer wrote and makes it behave in a fundamentally different > manner. I'm in support of UBSan diagnosing these constructs because it > is UB and an optimizer is definitely allowed to optimize based on it > but I wouldn't be in support of an optimizer that aggressively > optimizes on this. I consider it as something even worse. Just like the case of passing NULL pointers to memcpy and friends with zero as size, this interpretation / restriction in the standard is actively harmful to some code for the sake of potential optimisation opportunities in other code. It seems to be a poor choice at that. I.e. it requires adding conditional branches for something that behaves sanely everywhere but may the DS9k. Joerg
Re: NULL pointer arithmetic issues
On Mon, Mar 09, 2020 at 09:38:31AM -0400, Aaron Ballman wrote: > The way I read this is: > > "If the pointer operand points to an element of an array object" -- it > does not (null is not a valid array object). > "Moreover, if the expression P points to the last element of an array > object" -- it does not (null is not a valid array object). > "If both the pointer operand and the result point to elements of the > same array object, or one past the last element of the array > object..." -- it does not (there is no valid array so they cannot > point to elements of the same array). All fine. > "...otherwise, the behavior is undefined." -- this is where we hit the UB. But it started with "If the pointer operand points to an element of an array object" and my reading is that the whole section does not apply. NULL is special, and we did not change it's value, so it stays a NULL pointer. The section you cited just does not tell anything about arithmetic on NULL pointers - and the wording makes it sound like this is more an oversight than a concious descision. Eespecially adding or subtracting zero. You could view NULL as a special pointer pointing to an inaccessible zero sized object. Adding 0 to it still points to the same special non-object. I guess that is how the C++ folks see it. Martin
Re: NULL pointer arithmetic issues
On Sun, Mar 8, 2020 at 2:30 PM Taylor R Campbell
wrote:
>
> > Date: Sun, 8 Mar 2020 20:52:29 +0300
> > From: Roman Lebedev
> >
> > so we are allowed to lower that in clang front-end as:
> >
> > int
> > statu(char *a)
> > {
> > __builtin_assume(a != NULL);
> > a += getuid() - geteuid();
> > __builtin_assume(a != NULL);
>
> Allowed, yes.
>
> What I'm wondering is whether this is something Clang will actually do
> -- and whether it is for any serious reason other than to capriciously
> screw people who write software for real machines instead of for the
> pure C abstract machine to the letter of the standard (and if so,
> whether -fno-delete-null-pointer-checks is enough to disable it).
>
> Evidently making that assumption _not_ allowed in C++, so presumably
> it's not important for performance purposes. It's also not important
> for expressive purposes, because I could just as well have written
> assert(a != NULL).
>
> > > I was told by Roman that it was checked during a C committee meeting and
> > > confirmed to be an intentional UB.
> > Correction: Aaron Ballman asked about this in non-public WG14 reflector
> > mailing list, it wasn't a committee meeting, but the point still stands.
>
> What does `intentional' UB mean, exactly? What is the intent behind
> having p + 0 for null p be undefined in C, while the C++ committee saw
> fit to define it?
Intentional as in: the question was considered and it was decided to
make that scenario explicitly be UB (as opposed to it being UB because
we didn't say anything about it in the standard or never considered
the question in the first place).
> Is it because there technically once existed C implementations on
> bizarre architectures with wacky arithmetic on pointers like Zeta-C,
> or is it because there are actual useful consequences to inferring
> that p must be nonnull if we evaluate p + 0?
As for intent, I can only speculate (I wasn't there for the original
specification work on this. I might not have even been alive for the
original specification work on this.): performance gains. For
instance, a segmented memory architecture may have multiple value
representations of a null pointer. Even a flat memory architecture can
see potential optimization gains from making an assumption that
arithmetic on a pointer means the pointer is valid.
> I ask because in principle a conformant implementation could compile
> the NetBSD kernel into a useless blob that does nothing -- we rely on
> all sorts of behaviour relative to a real physical machine that is not
> defined by the letter of the standard, like inline asm, or converting
> integers from the VM system's virtual address allocator into pointers
> to objects. But such an implementation would not be useful.
Whether an optimizer elects to use this form of UB to make
optimization decisions is a matter of QoI. My personal feeling is that
I don't trust this sort of optimization -- it takes code the
programmer wrote and makes it behave in a fundamentally different
manner. I'm in support of UBSan diagnosing these constructs because it
is UB and an optimizer is definitely allowed to optimize based on it
but I wouldn't be in support of an optimizer that aggressively
optimizes on this.
~~Aaron
Re: NULL pointer arithmetic issues
On Mon, Mar 9, 2020 at 9:21 AM Martin Husemann wrote: > > On Mon, Mar 09, 2020 at 01:34:23PM +0100, Kamil Rytarowski wrote: > > > We instruct a C compiler that pointer used in the pserialize macros is > > never NULL, as the side effect of adding to it 0. > > I question that side effect. > > The C++ standard disagrees with your interpration, I have not seen clear > statements in any official C standard, and we have been discussing here > tons of totally unrelated things. Untill proven otherwise I consider that > ubsan behaviour a bug. C17 6.5.6p8 says: When an expression that has integer type is added to or subtracted from a pointer, the result has the type of the pointer operand. If the pointer operand points to an element of an array object, and the array is large enough, the result points to an element offset from the original element such that the difference of the subscripts of the resulting and original array elements equals the integer expression. In other words, if the expression P points to the i-th element of an array object, the expressions (P)+N (equivalently, N+(P)) and (P)-N (where N has the value n) point to, respectively, the i + n-th and i - n-th elements of the array object, provided they exist. Moreover, if the expression P points to the last element of an array object, the expression (P)+1 points one past the last element of the array object, and if the expression Q points one past the last element of an array object, the expression (Q)-1 points to the last element of the array object. If both the pointer operand and the result point to elements of the same array object, or one past the last element of the array object, the evaluation shall not produce an overflow; otherwise, the behavior is undefined. If the result points one past the last element of the array object, it shall not be used as the operand of a unary * operator that is evaluated. The way I read this is: "If the pointer operand points to an element of an array object" -- it does not (null is not a valid array object). "Moreover, if the expression P points to the last element of an array object" -- it does not (null is not a valid array object). "If both the pointer operand and the result point to elements of the same array object, or one past the last element of the array object..." -- it does not (there is no valid array so they cannot point to elements of the same array). "...otherwise, the behavior is undefined." -- this is where we hit the UB. When discussed on the committee reflector there were no objections to that interpretation, so this is my understanding of the C committee's current position. If you still think this is unclear, you can file a clarification request with the committee to get an official response considered by the full committee. ~~Aaron
Re: NULL pointer arithmetic issues
On Mon, Mar 09, 2020 at 01:34:23PM +0100, Kamil Rytarowski wrote: > We instruct a C compiler that pointer used in the pserialize macros is > never NULL, as the side effect of adding to it 0. I question that side effect. The C++ standard disagrees with your interpration, I have not seen clear statements in any official C standard, and we have been discussing here tons of totally unrelated things. Untill proven otherwise I consider that ubsan behaviour a bug. Martin
Re: NULL pointer arithmetic issues
On 09.03.2020 07:05, Martin Husemann wrote: > Also note that the getuid()/geteuid() example here is IMHO unrelated to the > original issue that caused this discussion, so I am not even convinced this > is NOT a ubsan bug. We instruct a C compiler that pointer used in the pserialize macros is never NULL, as the side effect of adding to it 0. As the pointer can be NULL, this at least confuses the compiler and can result in a miscompilation. We workaround it today with -fno-delete-null-pointer-checks in RUMP. In regular userland we shall avoid NULL pointer arithmetic. signature.asc Description: OpenPGP digital signature
Re: NULL pointer arithmetic issues
On Sun, Mar 08, 2020 at 09:58:05PM +0100, Kamil Rytarowski wrote: > Aaron (as he was mentioned by name), is a voting member in the C++ > committee and actively working on harmonizing C and C++ standards. There > is a good chance that C and C++ will be synced here (they definitely > should). Yes, they should. In this regard one of them is sane and one is totaly absurd and actively harmfull to existing working code. Unbreak C, please. Also note that the getuid()/geteuid() example here is IMHO unrelated to the original issue that caused this discussion, so I am not even convinced this is NOT a ubsan bug. Martin
Re: NULL pointer arithmetic issues
On 08.03.2020 19:42, Mouse wrote: >>> The correct fix is not to disable the null-pointer-check option but >>> to remove the broken automatic non-null arguments in GCC. > >> The C standard and current usage (GNU) disagrees here. > > GNU is not some kind of arbiter of "current usage" (whatever _that_ > means). > >> memcpy (along with a number of other functions) must not accept NULL >> arguments and compiler can optimize the code based on these >> assumptions. > > Then such functions - or the language in which they are embedded - is > not suitable for writing kernels. > Theoretical C is not suitable for writing kernels and we need extensions for the freestanding environment. We require at least assembly stubs. > But, is it "must not accept null arguments" or is it "may do anything > they like when presented with null arguments"? Actually both. > But the answer is to fix the problem, not to > twist the code into a pretzel to work around the compiler's refusal to > be suitable for the job. > We use -fno-delete-null-pointer-checks to disable these optimizations. signature.asc Description: OpenPGP digital signature
Re: NULL pointer arithmetic issues
On 08.03.2020 19:30, Taylor R Campbell wrote:
>> Date: Sun, 8 Mar 2020 20:52:29 +0300
>> From: Roman Lebedev
>>
>> so we are allowed to lower that in clang front-end as:
>>
>> int
>> statu(char *a)
>> {
>> __builtin_assume(a != NULL);
>> a += getuid() - geteuid();
>> __builtin_assume(a != NULL);
>
> Allowed, yes.
>
> What I'm wondering is whether this is something Clang will actually do
> -- and whether it is for any serious reason other than to capriciously
> screw people who write software for real machines instead of for the
> pure C abstract machine to the letter of the standard (and if so,
> whether -fno-delete-null-pointer-checks is enough to disable it).
>
> Evidently making that assumption _not_ allowed in C++, so presumably
> it's not important for performance purposes. It's also not important
> for expressive purposes, because I could just as well have written
> assert(a != NULL).
>
>>> I was told by Roman that it was checked during a C committee meeting and
>>> confirmed to be an intentional UB.
>> Correction: Aaron Ballman asked about this in non-public WG14 reflector
>> mailing list, it wasn't a committee meeting, but the point still stands.
>
> What does `intentional' UB mean, exactly? What is the intent behind
> having p + 0 for null p be undefined in C, while the C++ committee saw
> fit to define it?
>
> Is it because there technically once existed C implementations on
> bizarre architectures with wacky arithmetic on pointers like Zeta-C,
> or is it because there are actual useful consequences to inferring
> that p must be nonnull if we evaluate p + 0?
>
> I ask because in principle a conformant implementation could compile
> the NetBSD kernel into a useless blob that does nothing -- we rely on
> all sorts of behaviour relative to a real physical machine that is not
> defined by the letter of the standard, like inline asm, or converting
> integers from the VM system's virtual address allocator into pointers
> to objects. But such an implementation would not be useful.
>
Aaron (as he was mentioned by name), is a voting member in the C++
committee and actively working on harmonizing C and C++ standards. There
is a good chance that C and C++ will be synced here (they definitely
should).
signature.asc
Description: OpenPGP digital signature
Re: NULL pointer arithmetic issues
>> The correct fix is not to disable the null-pointer-check option but >> to remove the broken automatic non-null arguments in GCC. > The C standard and current usage (GNU) disagrees here. GNU is not some kind of arbiter of "current usage" (whatever _that_ means). > memcpy (along with a number of other functions) must not accept NULL > arguments and compiler can optimize the code based on these > assumptions. Then such functions - or the language in which they are embedded - is not suitable for writing kernels. But, is it "must not accept null arguments" or is it "may do anything they like when presented with null arguments"? Given the lack of any way for it to "not accept" arguments, I suspect it is the latter, in which case it is not the language but rather the compiler, the compiler that mis-"optimizes" those cases into broken code, that is not suitable for kernel use. The "automatic non-null arguments" are not apporpriate in a language used for writing kernels. Whether that means changing the basic language, changing compilers, changing compiler flags, changing a file somewhere, I don't know. But the answer is to fix the problem, not to twist the code into a pretzel to work around the compiler's refusal to be suitable for the job. /~\ The ASCII Mouse \ / Ribbon Campaign X Against HTMLmo...@rodents-montreal.org / \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
Re: NULL pointer arithmetic issues
> Date: Sun, 8 Mar 2020 20:52:29 +0300
> From: Roman Lebedev
>
> so we are allowed to lower that in clang front-end as:
>
> int
> statu(char *a)
> {
> __builtin_assume(a != NULL);
> a += getuid() - geteuid();
> __builtin_assume(a != NULL);
Allowed, yes.
What I'm wondering is whether this is something Clang will actually do
-- and whether it is for any serious reason other than to capriciously
screw people who write software for real machines instead of for the
pure C abstract machine to the letter of the standard (and if so,
whether -fno-delete-null-pointer-checks is enough to disable it).
Evidently making that assumption _not_ allowed in C++, so presumably
it's not important for performance purposes. It's also not important
for expressive purposes, because I could just as well have written
assert(a != NULL).
> > I was told by Roman that it was checked during a C committee meeting and
> > confirmed to be an intentional UB.
> Correction: Aaron Ballman asked about this in non-public WG14 reflector
> mailing list, it wasn't a committee meeting, but the point still stands.
What does `intentional' UB mean, exactly? What is the intent behind
having p + 0 for null p be undefined in C, while the C++ committee saw
fit to define it?
Is it because there technically once existed C implementations on
bizarre architectures with wacky arithmetic on pointers like Zeta-C,
or is it because there are actual useful consequences to inferring
that p must be nonnull if we evaluate p + 0?
I ask because in principle a conformant implementation could compile
the NetBSD kernel into a useless blob that does nothing -- we rely on
all sorts of behaviour relative to a real physical machine that is not
defined by the letter of the standard, like inline asm, or converting
integers from the VM system's virtual address allocator into pointers
to objects. But such an implementation would not be useful.
Re: NULL pointer arithmetic issues
Warning: i'm not subscribed to the lists CC'd here.
On Sun, Mar 8, 2020 at 8:26 PM Kamil Rytarowski wrote:
>
> On 08.03.2020 18:00, Taylor R Campbell wrote:
> > Thanks for doing the research.
> >
> >> Date: Sun, 8 Mar 2020 15:30:02 +0100
> >> From: Kamil Rytarowski
> >>
> >> NULL+0 was added to UBSan proactively as it is as of today not
> >> miscompiled, but it is planned to so in not so distant time. It is a
> >> chicken-egg problem.
> >
> > If you say it is planned, can you please cite the plan?
> >
>
> Adding Roman Lebedev, we discussed about enabling NULL+0 optimization.
> Teaching LLVM about this will allow more aggressive optimization, so
> code like this:
>
> http://netbsd.org/~kamil/null_plus_0-ub.c
>
> will report different results depending on optimization levels.
I believe this point is correct. Given
int
statu(char *a)
{
a += getuid() - geteuid();
return a == NULL;
}
in C, as per all the wording cited below \/, we are allowed to assume
that given `ptr + offset`, regardless of `offset`,
both `ptr` and `ptr + offset` are non-NULL,
so we are allowed to lower that in clang front-end as:
int
statu(char *a)
{
__builtin_assume(a != NULL);
a += getuid() - geteuid();
__builtin_assume(a != NULL);
return a == NULL;
}
And that will naturally fold to
int
statu(char *a)
{
(void)getuid();
(void)geteuid();
return false;
}
.. which isn't what the code initially intended to do.
I.e. it's a dormant miscompile.
> > In C++, adding zero to a null pointer is explicitly allowed and
> > guaranteed to return a null pointer. See, for example, C++11 5.7
> > `Additive operators', clause 7, p. 117: `If the value 0 is added to or
> > subtracted from a pointer value, the result compares equal to the
> > original pointer value.' C++17 clarifies in 8.7 `Additive operators',
> > clause 7, p. 132: `If the value 0 is added to or subtracted from a
> > null pointer value, the result is a null pointer.'
> >
> > So it would be a little surprising to me if compilers -- which tend to
> > focus their efforts on C++ more than C these days -- went out of their
> > way to act on the inference that evaluating p + 0 implies p is nonnull
> > in C.
> >
>
> I underlined the C language in my message. More elaborated answer here:
>
> https://reviews.llvm.org/D67122#inline-612172
>
> I was told by Roman that it was checked during a C committee meeting and
> confirmed to be an intentional UB.
Correction: Aaron Ballman asked about this in non-public WG14 reflector
mailing list, it wasn't a committee meeting, but the point still stands.
> >> There is however a fallback. If we want to use NULL+0, we must use
> >> -fno-delete-null-pointer-checks that avoids miscompilation and raising
> >> UBSan reports. If we want to allow NULL+x we must enable -fwrapv.
> >
> > Adding -fno-delete-null-pointer-checks for clang too sounds sensible
> > to me in general, but please check with joerg first. It remains
> > unclear to me that it's necessary here.
> >
>
> Today it will calm down LLVM UBSan. In future potentially avoid
> miscompilation.
>
> There are also memcpy(NULL,NULL,0)-like cases that need research why
> they happen in the first place.
Roman
Re: NULL pointer arithmetic issues
On 08.03.2020 18:12, Joerg Sonnenberger wrote: > On Sun, Mar 08, 2020 at 03:33:57PM +0100, Kamil Rytarowski wrote: >> There was also a request to make a proof that memcpy(NULL,NULL,0) is UB >> and can be miscompiled. >> >> Here is a reproducer: >> >> http://netbsd.org/~kamil/memcpy-ub.c >> >> 131 kamil@rugged /tmp $ gcc -O0 memcpy.c >> >> 132 kamil@rugged /tmp $ ./a.out >> >> 1 >> >> 133 kamil@rugged /tmp $ gcc -O2 memcpy.c >> 134 kamil@rugged /tmp $ ./a.out >> 0 >> >> A fallback for freestanding environment is to use >> -fno-delete-null-pointer-check. > > The correct fix is not to disable the null-pointer-check option but to > remove the broken automatic non-null arguments in GCC. > > Joerg > The C standard and current usage (GNU) disagrees here. memcpy (along with a number of other functions) must not accept NULL arguments and compiler can optimize the code based on these assumptions. -fno-delete-null-pointer-check is a fallback disabling this optimizations. signature.asc Description: OpenPGP digital signature
Re: NULL pointer arithmetic issues
On 08.03.2020 18:11, Joerg Sonnenberger wrote: > On Sun, Mar 08, 2020 at 03:30:02PM +0100, Kamil Rytarowski wrote: >> NULL+x is now miscompiled by Clang/LLVM after this commit: >> >> https://reviews.llvm.org/rL369789 >> >> This broke various programs like: >> >> "Performing base + offset pointer arithmetic is only allowed when base >> itself is not nullptr. In other words, the compiler is assumed to allow >> that base + offset is always non-null, which an upcoming compiler >> release will do in this case. The result is that CommandStream.cpp, >> which calls this in a loop until the result is nullptr, will never >> terminate (until it runs junk data and crashes)." > > As you said, using a non-zero offset. Noone here argued that using > non-zero offsets is or should be valid since that would obviously create > a pointer outside the zero-sized object. > > Joerg > We catch NULL + x at least here: Undefined Behavior in t_subr_prf.c:179:9, pointer expression with base 0 overflowed to 0x14 Undefined Behavior in t_subr_prf.c:179:9, pointer expression with base 0 overflowed to 0xa signature.asc Description: OpenPGP digital signature
Re: NULL pointer arithmetic issues
On 08.03.2020 18:00, Taylor R Campbell wrote: > Thanks for doing the research. > >> Date: Sun, 8 Mar 2020 15:30:02 +0100 >> From: Kamil Rytarowski >> >> NULL+0 was added to UBSan proactively as it is as of today not >> miscompiled, but it is planned to so in not so distant time. It is a >> chicken-egg problem. > > If you say it is planned, can you please cite the plan? > Adding Roman Levedev, we discussed about enabling NULL+0 optimization. Teaching LLVM about this will allow more aggressive optimization, so code like this: http://netbsd.org/~kamil/null_plus_0-ub.c will report different results depending on optimization levels. > In C++, adding zero to a null pointer is explicitly allowed and > guaranteed to return a null pointer. See, for example, C++11 5.7 > `Additive operators', clause 7, p. 117: `If the value 0 is added to or > subtracted from a pointer value, the result compares equal to the > original pointer value.' C++17 clarifies in 8.7 `Additive operators', > clause 7, p. 132: `If the value 0 is added to or subtracted from a > null pointer value, the result is a null pointer.' > > So it would be a little surprising to me if compilers -- which tend to > focus their efforts on C++ more than C these days -- went out of their > way to act on the inference that evaluating p + 0 implies p is nonnull > in C. > I underlined the C language in my message. More elaborated answer here: https://reviews.llvm.org/D67122#inline-612172 I was told by Roman that it was checked during a C committee meeting and confirmed to be an intentional UB. >> There is however a fallback. If we want to use NULL+0, we must use >> -fno-delete-null-pointer-checks that avoids miscompilation and raising >> UBSan reports. If we want to allow NULL+x we must enable -fwrapv. > > Adding -fno-delete-null-pointer-checks for clang too sounds sensible > to me in general, but please check with joerg first. It remains > unclear to me that it's necessary here. > Today it will calm down LLVM UBSan. In future potentially avoid miscompilation. There are also memcpy(NULL,NULL,0)-like cases that need research why they happen in the first place. signature.asc Description: OpenPGP digital signature
Re: NULL pointer arithmetic issues
On Sun, Mar 08, 2020 at 03:33:57PM +0100, Kamil Rytarowski wrote: > There was also a request to make a proof that memcpy(NULL,NULL,0) is UB > and can be miscompiled. > > Here is a reproducer: > > http://netbsd.org/~kamil/memcpy-ub.c > > 131 kamil@rugged /tmp $ gcc -O0 memcpy.c > > 132 kamil@rugged /tmp $ ./a.out > > 1 > > 133 kamil@rugged /tmp $ gcc -O2 memcpy.c > 134 kamil@rugged /tmp $ ./a.out > 0 > > A fallback for freestanding environment is to use > -fno-delete-null-pointer-check. The correct fix is not to disable the null-pointer-check option but to remove the broken automatic non-null arguments in GCC. Joerg
Re: NULL pointer arithmetic issues
On Sun, Mar 08, 2020 at 03:30:02PM +0100, Kamil Rytarowski wrote: > NULL+x is now miscompiled by Clang/LLVM after this commit: > > https://reviews.llvm.org/rL369789 > > This broke various programs like: > > "Performing base + offset pointer arithmetic is only allowed when base > itself is not nullptr. In other words, the compiler is assumed to allow > that base + offset is always non-null, which an upcoming compiler > release will do in this case. The result is that CommandStream.cpp, > which calls this in a loop until the result is nullptr, will never > terminate (until it runs junk data and crashes)." As you said, using a non-zero offset. Noone here argued that using non-zero offsets is or should be valid since that would obviously create a pointer outside the zero-sized object. Joerg
Re: NULL pointer arithmetic issues
Thanks for doing the research. > Date: Sun, 8 Mar 2020 15:30:02 +0100 > From: Kamil Rytarowski > > NULL+0 was added to UBSan proactively as it is as of today not > miscompiled, but it is planned to so in not so distant time. It is a > chicken-egg problem. If you say it is planned, can you please cite the plan? In C++, adding zero to a null pointer is explicitly allowed and guaranteed to return a null pointer. See, for example, C++11 5.7 `Additive operators', clause 7, p. 117: `If the value 0 is added to or subtracted from a pointer value, the result compares equal to the original pointer value.' C++17 clarifies in 8.7 `Additive operators', clause 7, p. 132: `If the value 0 is added to or subtracted from a null pointer value, the result is a null pointer.' So it would be a little surprising to me if compilers -- which tend to focus their efforts on C++ more than C these days -- went out of their way to act on the inference that evaluating p + 0 implies p is nonnull in C. > There is however a fallback. If we want to use NULL+0, we must use > -fno-delete-null-pointer-checks that avoids miscompilation and raising > UBSan reports. If we want to allow NULL+x we must enable -fwrapv. Adding -fno-delete-null-pointer-checks for clang too sounds sensible to me in general, but please check with joerg first. It remains unclear to me that it's necessary here.
Re: NULL pointer arithmetic issues
On 22.02.2020 17:25, Kamil Rytarowski wrote:
> When running the ATF tests under MKLIBCSANITIZER [1], there are many
> NULL pointer arithmetic issues .
>
> http://netbsd.org/~kamil/mksanitizer-reports/ubsan-2020-02-22-null-pointer.txt
>
> These issues are in macros like:
> - IN_ADDRHASH_READER_FOREACH()
> - IN_ADDRLIST_WRITER_INSERT_TAIL()
> - IFADDR_READER_FOREACH()
> - etc
>
> These macros wrap internally pserialize-safe linked lists.
>
> What's the proper approach to address this issue?
>
> These reports are responsible for around half of all kinds of the
> remaining Undefined Behavior unique issues when executing ATF tests.
>
>
> [1] ./build.sh -N0 -U -V MAKECONF=/dev/null -V HAVE_LLVM=yes -V MKGCC=no
> -V MKLLVM=yes -V MKLIBCSANITIZER=yes -j8 -u -O /public/netbsd-llvm
> distribution
>
>
>
NULL + 0 and NULL + x are both Undefined Behavior (as confirmed by a C
committee member, it was discussed and confirmed to be intentional).
NULL+x is now miscompiled by Clang/LLVM after this commit:
https://reviews.llvm.org/rL369789
This broke various programs like:
"Performing base + offset pointer arithmetic is only allowed when base
itself is not nullptr. In other words, the compiler is assumed to allow
that base + offset is always non-null, which an upcoming compiler
release will do in this case. The result is that CommandStream.cpp,
which calls this in a loop until the result is nullptr, will never
terminate (until it runs junk data and crashes)."
https://github.com/google/filament/pull/1566
LLVM middle-end uses those guarantees for transformations.
This pushed the LLVM devs to implement this NULL+x and NULL+0 check in
UBSan:
https://reviews.llvm.org/D67122
NULL+0 was added to UBSan proactively as it is as of today not
miscompiled, but it is planned to so in not so distant time. It is a
chicken-egg problem.
There is however a fallback. If we want to use NULL+0, we must use
-fno-delete-null-pointer-checks that avoids miscompilation and raising
UBSan reports. If we want to allow NULL+x we must enable -fwrapv.
This means that we can avoid pserialize reports by enabling these CFLAGS
for clang as well.
22 # We are compiling the kernel code with
no-delete-null-pointer-checks,
23 # and compiling without it, causes issues at least on sh3 by adding
24 # aborts after kern_assert on NULL pointer checks.
25 CFLAGS+=${${ACTIVE_CC} == "gcc":?
-fno-delete-null-pointer-checks :}
26
https://nxr.netbsd.org/xref/src/sys/rump/Makefile.rump#25
I'm going to test it and switch -fno-delete-null-pointer-check on
Clang/LLVM.
For the historical note, a reproducer miscompiling NULL + x:
http://netbsd.org/~kamil/null_plus_x-ub.c
136 kamil@chieftec /tmp $ /usr/local/bin/clang -O0 null_plus_x-ub.c
137 kamil@chieftec /tmp $ ./a.out
1
138 kamil@chieftec /tmp $ /usr/local/bin/clang -O2 null_plus_x-ub.c
139 kamil@chieftec /tmp $ ./a.out
0
151 kamil@chieftec /tmp $ /usr/local/bin/clang -O3 -fwrapv null_plus_x-ub.c
152 kamil@chieftec /tmp $ ./a.out
1
NULL+0 is planned to follow up.
signature.asc
Description: OpenPGP digital signature
Re: NULL pointer arithmetic issues
At Wed, 26 Feb 2020 00:12:49 -0500 (EST), Mouse wrote: Subject: Re: NULL pointer arithmetic issues > > > This is the main point of my original rant. "Undefined Behaviour" as > > it has been interpreted by Optimization Warriors has given us an > > unusable language. > > I'd say that it's given you unusuable implementations of the language. > The problem is not the language; it's the compiler(s). (Well, unless > you consider the language to be the problem because it's possible to > implement it badly. I don't.) I don't think the C language (in all lower-case, un-quoted, plainly) is the problem -- I think the problem is the wording of the modern standard, and the unfortunate choice to use the phrase "undefined behaviour" for certain things. This has given "license" to optimization warriors -- and their over-optimization is the root of the evil I see in current compilers. It is this unfortunate choice of describing things as "undefined" within the language that has made modern "Standard C" unusable (especially for any and all legacy code, which is most of it, right?). If we outlawed the use of the phrase "undefined behaviour" and made all instances of it into "implementation defined behviour", with a very specific caveat that such instances did not, would not, and could not, ever allow optimizers to even think of violating any possible conceivable principle of least astonishment. E.g. in the example I gave, the only thing allowed would be for the implementation to do as it please IFF and when the pointer passed was actually a nil pointer at runtime (and perhaps in this case with a strong hint that the best and ideal behaviour would be something akin to calling abort()). -- Greg A. Woods Kelowna, BC +1 250 762-7675 RoboHack Planix, Inc. Avoncote Farms pgpEf9I3wNR2o.pgp Description: OpenPGP Digital Signature
Re: NULL pointer arithmetic issues
Johnny Billquist writes: > Well, the d-dpace don't start at 1, and also, the PDP-11 isn't that > fond of odd addresses. :-) True - I should have said that they started actual allocation of data objects at a non-zero address. :) > Here is the current "state": Thanks! That's really interesting, particularly the rationale given. I also wasn't aware that the mechanism used was so straightforward. -tih -- Most people who graduate with CS degrees don't understand the significance of Lisp. Lisp is the most important idea in computer science. --Alan Kay
Re: NULL pointer arithmetic issues
>>> char *lp = p->s;
>>> if (p == NULL || lp == NULL) {
>> This code is, and always has been, broken; it is accessing p->s
>> before it knows that p isn't nil.
> How do you know for sure?
> What if all calls to foo() are written as such:
> if (p) foo(p);
The code is still broken: it implies, with its test of p, that it's
supposed to be tolerant of p being nil - which it isn't. Even if none
of the extant calls point up that problem.
Of course, this was supposed to be security/safety-sensitive code,
right? So naturally it has an interface contract. What does that
specify for the case where p is nil? If it specifies that such calls
are erroneous, then the code is just sloppy and misleading (which I
suppose could be considered brokenness at a different level), but I
_still_ don't want whoever wrote it anywhere near anything
security/safety-sensitive I'm using.
/~\ The ASCII Mouse
\ / Ribbon Campaign
X Against HTMLmo...@rodents-montreal.org
/ \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
Re: NULL pointer arithmetic issues
> This is the main point of my original rant. "Undefined Behaviour" as > it has been interpreted by Optimization Warriors has given us an > unusable language. I'd say that it's given you unusuable implementations of the language. The problem is not the language; it's the compiler(s). (Well, unless you consider the language to be the problem because it's possible to implement it badly. I don't.) /~\ The ASCII Mouse \ / Ribbon Campaign X Against HTMLmo...@rodents-montreal.org / \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
Re: NULL pointer arithmetic issues
At Mon, 24 Feb 2020 22:15:22 -0500 (EST), Mouse
wrote:
Subject: Re: NULL pointer arithmetic issues
>
> > Greg A. Woods wrote:
> >
> > NO MORE "undefined behaviour"!!! Pick something sane and stick to it!
> >
> > The problem with modern "Standard" C is that instead of refining
> > the definition of the abstract machine to match the most common
> > and/or logical behaviour of existing implementations, the standards
> > committee chose to throw the baby out with the bath water and make
> > whole swaths of conditions into so-called "undefined behaviour"
> > conditions.
>
> Unfortunately for your argument, they did this because there are
> "existing implementations" that disagree severely over the points in
> question.
I don't believe that's quite right.
True "Undefined Behaviour" is not usually the explanation for
differences between implementations. That's normally what the Language
Lawyers call "Implementation Defined" behaviour.
"Undefined behaviour" is used for things like dereferencing a nil
pointer. There's little disagreement about that being "undefined by
definition" -- even ignoring the Language Lawyers. We can hopefully
agree upon that even using the original K edition's language:
"C guarantees that no pointer that validly points at data will
contain zero"
The problem though is that C gives more rope than you might ever think
possible in some situations, such as for example, the chances of
dereferencing a nil pointer with poorly written code.
The worse problem though is when compiler writers, what I'll call
"Optimization Warrior Lawyers", start abusing any and every possible
instance of "Undefined Behaviour" to their advantage.
This is worse than ignoring Hoare's advice -- this is the very epitome
of premature optimization -- this is pure evil.
This is breaking otherwise readable and usable code.
I give you again my example:
> > An excellent example are the data-flow optimizations that are now
> > commonly abused to elide security/safety-sensitive code:
>
> > int
> > foo(struct bar *p)
> > {
> > char *lp = p->s;
> >
> > if (p == NULL || lp == NULL) {
> > return -1;
> > }
>
> This code is, and always has been, broken; it is accessing p->s before
> it knows that p isn't nil.
How do you know for sure? How does the compiler know? Serious questions.
What if all calls to foo() are written as such:
if (p) foo(p);
I agree this might not be "fail-safe" code, or in any other way
advisable, but it was perfectly fine in the world before UB Optimization
Warriors, however today's "Standard C" gives compilers license to
replace "foo()" with a trap or call to "abort()", etc.
I.e. it takes a real "C Language Lawyer(tm)" to know that past certain
optimization levels the sequence points prevent this from happening.
In the past I could equally assume the optimizer would rewrite the first
bit of foo() as:
if (! p || ! p->s) return -1;
In 35 years of C programming I've never before had to pay such close
attention to such minute details. I need tools now to audit old code
for such things, and my current experience to date suggests UBSan is not
up to this task -- i.e. runtime reports are useless (perhaps even with
high-code-coverage unit tests).
This is the main point of my original rant. "Undefined Behaviour" as it
has been interpreted by Optimization Warriors has given us an unusable
language.
--
Greg A. Woods
Kelowna, BC +1 250 762-7675 RoboHack
Planix, Inc. Avoncote Farms
pgpytwmS13Epg.pgp
Description: OpenPGP Digital Signature
Re: NULL pointer arithmetic issues
On 2020-02-25 12:33, Tom Ivar Helbekkmo wrote: Johnny Billquist writes: But yes, on the PDP11 [having nothing mapped at address 0] was/is not the case. Memory space is too precious to allow some of it to be wasted for this... Yup - and I assume the "hack" Kamil alludes to is the practice of actually starting the data segment for split I/D programs at address 1 instead of 0, to make sure that no actual pointer is 0, thus allowing the straightforward comparison of a pointer with 0 to see if it's set. Well, the d-dpace don't start at 1, and also, the PDP-11 isn't that fond of odd addresses. :-) Actually, you could not even start a page at address 1 if you wanted. (I believe they also initialized address 0 to 0, to stop indirect references through it from reaching random data. I guess Franz may have depended on this in some way, e.g. expecting to be able to test *p directly, instead of first p and then *p. Do enough of this, and you've soon bummed a significant amount of valuable code space...) It used to, but not for some time now. Here is the current "state": /* * Paragraph below retained for historical purposes. * * The following zero has a number of purposes - it serves as a null terminated * string for uninitialized string pointers on separate I machines for * instance. But we never would have put it here for that reason; programs * which use uninitialized pointer *should* die. The real reason it's here is * so you can declare "char blah[] = "foobar" at the start of a C program * and not have printf generate "(null)" when you try to print it because * blah is at address zero on separate I machines ... sick, sick, sick ... * * In porting bits and pieces of the 4.4-Lite C library the global program * name location '___progname' was needed. Rather than take up another two * bytes of D space the 0th location was used. The '(null)' string was * removed from doprnt.s so now when programs use uninitialized pointers * they will be rewarded with argv[0]. This is no sicker than before and * may cause bad programs to die sooner. */ .data .globl ___progname, _strrchr ___progname: 0 Johnny -- Johnny Billquist || "I'm on a bus || on a psychedelic trip email: b...@softjar.se || Reading murder books pdp is alive! || tryin' to stay hip" - B. Idol
Re: NULL pointer arithmetic issues
Johnny Billquist writes: > But yes, on the PDP11 [having nothing mapped at address 0] was/is not > the case. Memory space is too precious to allow some of it to be > wasted for this... Yup - and I assume the "hack" Kamil alludes to is the practice of actually starting the data segment for split I/D programs at address 1 instead of 0, to make sure that no actual pointer is 0, thus allowing the straightforward comparison of a pointer with 0 to see if it's set. (I believe they also initialized address 0 to 0, to stop indirect references through it from reaching random data. I guess Franz may have depended on this in some way, e.g. expecting to be able to test *p directly, instead of first p and then *p. Do enough of this, and you've soon bummed a significant amount of valuable code space...) -tih -- Most people who graduate with CS degrees don't understand the significance of Lisp. Lisp is the most important idea in computer science. --Alan Kay
Re: NULL pointer arithmetic issues
> I wrote the following rant some time ago and posted it somewhere
> I'll throw it in here for some more fuel
> NO MORE "undefined behaviour"!!! Pick something sane and stick to it!
> The problem with modern "Standard" C is that instead of refining
> the definition of the abstract machine to match the most common
> and/or logical behaviour of existing implementations, the standards
> committee chose to throw the baby out with the bath water and make
> whole swaths of conditions into so-called "undefined behaviour"
> conditions.
Unfortunately for your argument, they did this because there are
"existing implementations" that disagree severely over the points in
question. A spec that mandated such things as the "pointers are really
just memory addresses" model you sketch below would, at best, simply
get ignored by implementors on machines that don't match it. Perhaps
that's what you'd want. Personally, I prefer the actual choice.
> An excellent example are the data-flow optimizations that are now
> commonly abused to elide security/safety-sensitive code:
> int
> foo(struct bar *p)
> {
> char *lp = p->s;
>
> if (p == NULL || lp == NULL) {
> return -1;
> }
This code is, and always has been, broken; it is accessing p->s before
it knows that p isn't nil. If you're really unlucky you'll be on a
machine where there are device registers at address 0 and you'll poke a
device register with that read. If you're less lucky you'll be on
MS-DOS or a PDP-11 or some such and silently and harmlessly get a
meaningless value for lp. If you're lucky you'll get a segfault or
moral equivalent. Anyone who thinks this sort of sloppiness is
appropriate in security/safety-sensitive code please stay far, far away
from anything that might run on my machines. Yes, an optimizer _might_
defer the fetch of lp, but it also might not, for any of many reasons;
relying on its doing so is extremely brittle, most definitely not
appropriate for anything security/safety-sensitive.
That said, I do agree that simply dropping the p==NULL check but
preserving the fetch of lp is, if anything, even more broken; it is
gross abuse of the latitude permitted by the undefined-behaviour rules.
But that is a quality-of-implementation issue.
> Worse yet this example stems from actual Linux kernel code [...]
Good gods. I'm gladder than ever I don't run Linux.
> [...], yet again any programmer worth their salt knows that the
> address of an field in a struct is simply the sum of the struct's
> base address and the offset of the field, [...]
That's what a mediocre C programmer thinks. A good one knows there is
a difference between the abstract machine and the implementation and
realizes that, while that is a common implementation, it is far from
the only possible one, and it is inappropriate to rely on it being an
accurate description (except in code not intended to be portable, like
a kernel's pmap layer).
> Worst of all consider this example:
> size_t o = offsetof(p, s);
> And then consider an extremely common example of "offsetof()" [...]
Such an implementation of offsetof() is nonportable, exactly because it
assumes things like your sketch based on the "pointers are just memory
addresses" model. Providing it in application code constitutes
nonportable code, just as much as assuming shorts are 18 bits does.
(What, you mean you're not on a 36-bit machine? What sort of weird
hardware are you using?)
An implementation may provide it, yes, if - IF! - it knows the
associated compiler handles that code such that offsetof() returns the
correct result. But what would you expect it to do in, say, Zeta-C?
Or do you think Zeta-C should not exist?
> or possibly (for those who know that pointers are not always "just"
> integers):
> #define offsetof(type, member) ((size_t)(unsigned long)((&((type
> *)0)->member) - (type *)0))
That has never worked in C since, oh, I dunno, V7? and probably never
will; it tries to subtract pointers that point to different types.
What I think of as the usual implementation along those lines would be
something like
((size_t)((char *)&((type *)0)->member - (char *)0))
(note the lack of an intermediate cast to unsigned long; size_t may be
wider than unsigned long, though admittedly it's unlikely offsetof()
will need to return a value greater than the largest unsigned long).
/~\ The ASCII Mouse
\ / Ribbon Campaign
X Against HTMLmo...@rodents-montreal.org
/ \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
Re: NULL pointer arithmetic issues
I very much agree that pointer arithmetic MUST NOT be "undefined", even
if it includes "NULL" and/or "0". The warning that begat this thread is
insane!
Note I say this as someone who is very empathetic to implementers who
might try to make C work in any strange hardware systems where "null"
pointers are not actually all zeros in the hardware. I hope to be one.
At Mon, 24 Feb 2020 14:41:26 +0100, Kamil Rytarowski wrote:
Subject: Re: NULL pointer arithmetic issues
>
> Please join the C committee as a voting member or at least submit papers
> with language changes. Complaining here won't change anything.
>
> (Out of people in the discussion, I am involved in wg14 discussions and
> submit papers.)
If you are active on the wg14 committee, perhaps you can be convinced to
argue on "our" behalf? [0.5 :-)]
I wrote the following rant some time ago and posted it somewhere
(probably on G+ because I don't find it now with a quick search).
I'll throw it in here for some more fuel
NO MORE "undefined behaviour"!!! Pick something sane and stick to it!
The problem with modern "Standard" C is that instead of refining the
definition of the abstract machine to match the most common and/or
logical behaviour of existing implementations, the standards committee
chose to throw the baby out with the bath water and make whole swaths
of conditions into so-called "undefined behaviour" conditions.
An excellent example are the data-flow optimizations that are now
commonly abused to elide security/safety-sensitive code:
int
foo(struct bar *p)
{
char *lp = p->s;
if (p == NULL || lp == NULL) {
return -1;
}
lp[0] = '\0';
return 0;
}
Any programmer worth their salt will assume the compiler can calculate
the offset of 's' at compile time and thus anyone ignorant of C's new
"undefined behaviour" rules will guess that at worst some location on
the stack will be assigned a value pulled from low memory (if that
doesn't cause a SIGSEGV), but more likely the de-reference of 'p'
won't happen right away because we all know that any optimizer worth
it's salt SHOULD defer it until the first use of 'lp', perhaps not
even allocating any stack space for 'lp' at all!
Worse yet this example stems from actual Linux kernel code like this:
static int
podhd_try_init(struct usb_interface *interface,
struct usb_line6_podhd *podhd)
{
struct usb_line6 *line6 = >line6;
if ((interface == NULL) || (podhd == NULL))
return ENODEV;
}
Here some language-lawyer-wannabees [[LLWs]] might try in vain to argue over
the interpretation of "dereferencing", yet again any programmer worth
their salt knows that the address of an field in a struct is simply
the sum of the struct's base address and the offset of the field, the
latter of which the compiler obviously knows at compile time, and
adding a value to a NULL pointer should never be considered invalid or
undefined!
[[ You have to start from somewhere, after all Why not zero? ]]
(I suspect the LLWs are being misled by the congruence between "a->b"
and "(*a).b".)
Worst of all consider this example:
void *
foo(struct bar *p)
{
size_t o = offsetof(p, s);
if (s == NULL)
return NULL;
}
And then consider an extremely common example of "offsetof()" which
might very well appear in a legacy application's own code because it
pre-dated , though indeed this very definition has been used
in by several standard compiler implementations, and indeed
it was specifically allowed in general by ISO C90 (and only more
recently denied by C11, sort of):
#define offsetof(type, member) ((size_t)(unsigned long)(&((type
*)0)->member))
or possibly (for those who know that pointers are not always "just"
integers):
#define offsetof(type, member) ((size_t)(unsigned long)((&((type
*)0)->member) - (type *)0))
Here we have very effectively and entirely hidden the fact that the
'->' operator is used with 's'.
Any sane person with some understanding of programming languages
should agree that it is wrong to assume that calculating the address
of an lvalue "evaluates" that lvalue. In C the '->' and '[]'
operators are arithmetic operators, not (immediately and on their own)
memory access operators.
Sadly C's new undefined behaviour rules as interpreted by some
compiler maintainers now allow the compiler to STUPIDLY assume that
since the programme
Re: NULL pointer arithmetic issues
> Date: Mon, 24 Feb 2020 11:42:01 +0100 > From: Kamil Rytarowski > > Forbidding NULL pointer arithmetic is not just for C purists trolls. It > is now in C++ mainstream and already in C2x draft. > > The newer C standard will most likely (already accepted by the > committee) adopt nullptr on par with nullptr from C++. In C++ we can > "#define NULL nullptr" and possibly the same will be possible in C. > > http://www.open-std.org/jtc1/sc22/wg14/www/docs/n2394.pdf > > This will change all arithmetic code operating on NULL into syntax error. Arithmetic on bare NULL is already an error, flagged by the options -Wpointer-arith -Werror which we already use, and arithmetic on the proposed nullptr will remain so. This question is not about that, or about syntax. The question is whether it is realistic to imagine that a compiler we will ever use to build the kernel -- particularly with the option -fno-delete-null-pointer-checks as we already use to build the kernel with gcc -- will actually meaningfully distinguish the fragments char *x = NULL; return x; and char *x = NULL; return x + 0; Will two programs that differ only by this fragment actually behave differently on any serious C implementation we use in NetBSD, ignoring the pedantry of ubsan? (The question is the same if you substitute the proposed nullptr for NULL; it's about the meaning of + on a null pointer, not whether the program is syntactically written with the letters `NULL' or `nullptr'.) The second program technically has undefined behaviour because in, e.g., C99 6.5.6 `Additive operators', the meaning of + is defined on pointer/integer operands only when the pointer is to an object in an array and the sum stays within the array or points one past the end -- in other words, there's nothing in C99 formally defining what x + 0 means when x is a null pointer. Why is the standard written this way? I surmise that it's because technically there exist implementations such as Zeta-C where a `pointer' is not simply a virtual address in a machine register but actually a pair of a Lisp array and an index into it. NetBSD does not run on such implementations. Corners of the standard that serve _only_ to accommodate such implementations are not relevant to NetBSD on their own. The standard is also technically written so that a null pointer is not necessarily stored as all bits zero in memory, so char *x; memset(, 0, sizeof x); return x; is not guaranteed to return a null pointer. However, NetBSD only runs on C implementations where it actually is guaranteed to return a null pointer, and we rely on this pervasively. If we make _only_ the assumptions that the standard formally guarantees, then ubsan would be right to object that char *x; memset(, 0, sizeof x); return x == NULL ? 0 : *(char *)x; has undefined behaviour. But in NetBSD this is guaranteed to return 0 and so if ubsan flagged it we would treat that as a useless false alarm that detracts from the value of ubsan as a tool. If you can present a compelling argument that C implementations which are _relevant to NetBSD_ -- not merely technically allowed by the letter of the standard like Zeta-C -- will actually behave differently from how I described, please present that. Otherwise please find a way to suppress the false alarm in the tool so it doesn't waste any more time. (And please do the same for memcpy(x,NULL,0)/memcpy(NULL,y,0)!)
Re: NULL pointer arithmetic issues
>> int one(void) { return(1); }
>> then (one()-one()) is not [a null pointer constant]
> As you say, it's an integer expression. And I read that "or" part as
> just an expression, which this is. So I believe it is a valid way to
> creating something that can be converted to a NULL pointer.
C99 words it as
[#3] An integer constant expression with the value 0, or
such an expression cast to type void *, is called a null
pointer constant.
That little word "such" is important. As I read it, that means a null
pointer constant is "[a]n integer constant expression with the value 0,
or an integer constant expression with the value 0 cast to type void *"
(not just "an integer expression with the value 0 cast to...").
> if (expression) statement; shall execute statement if expression not
> equals 0, according to the standard.
Yes:
[#2] In both forms, the first substatement is executed if
the expression compares unequal to 0. In the else form, the
second substatement is executed if the expression compares
equal to 0. If the first substatement is reached via a
label, the second substatement is not executed.
> So, where does that leave this code:
> char *p;
> if (p) foo();
> p is not an integer. How do you compare it to 0?
The same way you do in
if (p != 0) foo();
How else?
I would admittedly prefer slightly more verbose wording, saying
explicitly that the 0 to which the control expression is compared is,
when applicable, a null pointer constant. (Comparison of a pointer
with a null pointer constant is specifically permitted for == and != -
see 6.5.9.)
/~\ The ASCII Mouse
\ / Ribbon Campaign
X Against HTMLmo...@rodents-montreal.org
/ \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
Re: NULL pointer arithmetic issues
On 2020-02-25 02:12, Mouse wrote:
Oh. And I actually do not believe it has to be a constant.
You are correct; it does not need to be a simple constant.
The text says "integer constant expression with the value 0, or such
an expression..."
Yes. (void *)(1-1) is a valid null pointer constant. So, on an
all-ASCII system, is (0x1f+(3*5)-'.'). But, in the presence of
int one(void) { return(1); }
then (one()-one()) is not - it is an integer expression with value
zero, but it is not an integer _constant_ expression. It's entirely
possible that (int *)(one()-one()) will produce a different pointer
from (int *)(1-1) - the latter is a null pointer; the former might or
might not be, depending on the implementation.
As you say, it's an integer expression. And I read that "or" part as
just an expression, which this is. So I believe it is a valid way to
creating something that can be converted to a NULL pointer.
Also:
if (expression) statement; shall execute statement if expression not
equals 0, according to the standard.
So, where does that leave this code:
char *p;
.
.
if (p) foo();
p is not an integer. How do you compare it to 0?
Johnny
--
Johnny Billquist || "I'm on a bus
|| on a psychedelic trip
email: b...@softjar.se || Reading murder books
pdp is alive! || tryin' to stay hip" - B. Idol
Re: NULL pointer arithmetic issues
> Oh. And I actually do not believe it has to be a constant.
You are correct; it does not need to be a simple constant.
> The text says "integer constant expression with the value 0, or such
> an expression..."
Yes. (void *)(1-1) is a valid null pointer constant. So, on an
all-ASCII system, is (0x1f+(3*5)-'.'). But, in the presence of
int one(void) { return(1); }
then (one()-one()) is not - it is an integer expression with value
zero, but it is not an integer _constant_ expression. It's entirely
possible that (int *)(one()-one()) will produce a different pointer
from (int *)(1-1) - the latter is a null pointer; the former might or
might not be, depending on the implementation. Similarly,
int i;
i = 0;
if ((int *)i == (int *)0) ... else ...
may test unequal. (I have a very fuzzy memory that says POSIX may
impose additional restrictions that might affect this; I'm talking
strictly about C99 here.)
/~\ The ASCII Mouse
\ / Ribbon Campaign
X Against HTMLmo...@rodents-montreal.org
/ \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
Re: NULL pointer arithmetic issues
On Mon, Feb 24, 2020 at 05:35:22PM -0500, Mouse wrote: > > Unless I remember wrong, older C standards explicitly say that the > > integer 0 can be converted to a pointer, and that will be the NULL > > pointer, and a NULL pointer cast as an integer shall give the value > > 0. > > The only one I have anything close to a copy of is C99, for which I > have a very late draft. > > Based on that: > > You are not quite correct. Any integer may be converted to a pointer, > and any pointer may be converted to an integer - but the mapping is > entirely implementation-dependent, except in the integer->pointer > direction when the integer is a "null pointer constant", defined as > "[a]n integer constant expression with the value 0" (or such an > expression cast to void *, though not if we're talking specifically > about integers), in which case "the resulting pointer, called a null > pointer, is guaranteed to compare unequal to a pointer to any object or > function". You could have meant that, but what you wrote could also be > taken as applying to the _run-time_ integer value 0, which C99's > promise does not apply to. (Quotes are from 6.3.2.3.) > > I don't think there is any promise that converting a null pointer of > any type back to an integer will necessarily produce a zero integer. > The wording was the same for C89 and there is this paragraph in K (second edition, p 102): "Pointers and integers are not interchangeable. Zero is the sole exception: the constant zero may be assigned to a pointer, and a pointer may be compared with the constant zero. The symbolic constant NULL is often used in place of zero, as a mnemonic to indicate more clearly that this is a special value for a pointer. [...]" I interpret this (the paragraph above and the standard) as: in comparing a pointer to the constant zero, the constant zero is converted to a pointer of NULL value, thus comparing pointer to pointer and not comparing an integer value (the integer value of the pointer) to an integer value (0). So defining NULL as the casting of 0 is (was?) in the C standard, the actual value of the expression i.e. of an incorrect (NULL) pointer being implementation defined. FWIW, -- Thierry Laronde http://www.kergis.com/ http://www.sbfa.fr/ Key fingerprint = 0FF7 E906 FBAF FE95 FD89 250D 52B1 AE95 6006 F40C
Re: NULL pointer arithmetic issues
> On Feb 24, 2020, at 4:08 PM, Kamil Rytarowski wrote: > > NULL in C is expected to be harmonized with nullptr from C++. This is insanity. C++ is a cesspit where dreams of elegant code go to die. -- thorpej ...proud owner of a "C++ Barf Bag" that hangs outside his office...
Re: NULL pointer arithmetic issues
On 24.02.2020 23:35, Mouse wrote:
>> Unless I remember wrong, older C standards explicitly say that the
>> integer 0 can be converted to a pointer, and that will be the NULL
>> pointer, and a NULL pointer cast as an integer shall give the value
>> 0.
>
> The only one I have anything close to a copy of is C99, for which I
> have a very late draft.
>
> Based on that:
>
> You are not quite correct. Any integer may be converted to a pointer,
> and any pointer may be converted to an integer - but the mapping is
> entirely implementation-dependent, except in the integer->pointer
> direction when the integer is a "null pointer constant", defined as
> "[a]n integer constant expression with the value 0" (or such an
> expression cast to void *, though not if we're talking specifically
> about integers), in which case "the resulting pointer, called a null
> pointer, is guaranteed to compare unequal to a pointer to any object or
> function". You could have meant that, but what you wrote could also be
> taken as applying to the _run-time_ integer value 0, which C99's
> promise does not apply to. (Quotes are from 6.3.2.3.)
>
> I don't think there is any promise that converting a null pointer of
> any type back to an integer will necessarily produce a zero integer.
>
> /~\ The ASCII Mouse
> \ / Ribbon Campaign
> X Against HTML mo...@rodents-montreal.org
> / \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
>
$ cat test.cpp
#include
int
main(int argc, char **argv)
{
if (((char *)0)[argc])
return 1;
else
return 0;
}
$ g++ test.cpp
$ ./a.out
Memory fault (core dumped)
And some variations:
$ g++ test.cpp
test.cpp: In function ‘int main(int, char**)’:
test.cpp:6:15: warning: converting NULL to non-pointer type
[-Wconversion-null]
if (NULL[argc])
^
test.cpp:6:15: error: invalid types ‘long int[int]’ for array subscript
$ g++ test.cpp
test.cpp: In function ‘int main(int, char**)’:
test.cpp:6:18: error: invalid types ‘std::nullptr_t[int]’ for array
subscript
if (nullptr[argc])
^
NULL in C is expected to be harmonized with nullptr from C++.
We still can store NULL/nullptr in variables as before and there is no
change in the produced code. The only change is on the syntax level as
we can catch more bugs earlier. Whenever a compiler will be smart enough
to deduce that the code is nullptr[0] it will raise an error.
signature.asc
Description: OpenPGP digital signature
Re: NULL pointer arithmetic issues
On 2020-02-24 21:24, Kamil Rytarowski wrote: On 24.02.2020 21:18, Mouse wrote: If we use 0x0, it can be a valid pointer. If we use NULL, it's not expected to work and will eventually generate a syntax erro. Then someone has severely broken compatability with older versions of C. 0x0 and (when one of the suitable #includes has been done) NULL have both historically been perfectly good null pointer constants. Also...syntax error? Really? _Syntax_ error?? I'd really like to see what they've done to the grammar to lead to that; I'm having trouble imagining how that would be done. The process of evaluation of the NULL semantics is not a recent thing. Not so long time, still in the NetBSD times, it was a general practice to allow dereferencing the NULL pointer and expect zeroed bytes over there. We still maintain compatibility with this behavior (originated as a hack in PDP11) in older NetBSD releases (NetBSD-0.9 Franz Lisp binaries depend on this). Really? I thought we usually do not have anything mapped at address 0 to explicitly catch any dereferencing of NULL pointers. But yes, on the PDP11 this was/is not the case. Memory space is too precious to allow some of it to be wasted for this... (Even if there are a comment about it in 2.11BSD, bemoaning this fact...) Johnny -- Johnny Billquist || "I'm on a bus || on a psychedelic trip email: b...@softjar.se || Reading murder books pdp is alive! || tryin' to stay hip" - B. Idol
Re: NULL pointer arithmetic issues
On 2020-02-25 00:24, Johnny Billquist wrote: On 2020-02-24 23:35, Mouse wrote: Unless I remember wrong, older C standards explicitly say that the integer 0 can be converted to a pointer, and that will be the NULL pointer, and a NULL pointer cast as an integer shall give the value 0. The only one I have anything close to a copy of is C99, for which I have a very late draft. Based on that: You are not quite correct. Any integer may be converted to a pointer, and any pointer may be converted to an integer - but the mapping is entirely implementation-dependent, except in the integer->pointer direction when the integer is a "null pointer constant", defined as "[a]n integer constant expression with the value 0" (or such an expression cast to void *, though not if we're talking specifically about integers), in which case "the resulting pointer, called a null pointer, is guaranteed to compare unequal to a pointer to any object or function". You could have meant that, but what you wrote could also be taken as applying to the _run-time_ integer value 0, which C99's promise does not apply to. (Quotes are from 6.3.2.3.) I don't think there is any promise that converting a null pointer of any type back to an integer will necessarily produce a zero integer. Maybe we are reading things differently...? Looking at 6.3.2.3... As far as I read, paragraph 3 says: "An integer constant expression with the value 0, or such an expression cast to type void *, is called a null pointer constant.55) If a null pointer constant is converted to a pointer type, the resulting pointer, called a null pointer, is guaranteed to compare unequal to a pointer to any object or function." Essentially, the integer constant 0 can be casted to a pointer, and that pointer is then a null pointer constand, also called a null pointer. And footnote 55 says: Oh. And I actually do not believe it has to be a constant. The text says "integer constant expression with the value 0, or such an expression..." So either a constant expression, or just an expression, which gives a 0, can be cast to a pointer, that that will be the NULL pointer. (I realized when reading, that I might have implied that it only applied to constanst, which I think it does not.) But I might have misunderstood everything, of course... Johnny -- Johnny Billquist || "I'm on a bus || on a psychedelic trip email: b...@softjar.se || Reading murder books pdp is alive! || tryin' to stay hip" - B. Idol
Re: NULL pointer arithmetic issues
On 2020-02-24 23:35, Mouse wrote: Unless I remember wrong, older C standards explicitly say that the integer 0 can be converted to a pointer, and that will be the NULL pointer, and a NULL pointer cast as an integer shall give the value 0. The only one I have anything close to a copy of is C99, for which I have a very late draft. Based on that: You are not quite correct. Any integer may be converted to a pointer, and any pointer may be converted to an integer - but the mapping is entirely implementation-dependent, except in the integer->pointer direction when the integer is a "null pointer constant", defined as "[a]n integer constant expression with the value 0" (or such an expression cast to void *, though not if we're talking specifically about integers), in which case "the resulting pointer, called a null pointer, is guaranteed to compare unequal to a pointer to any object or function". You could have meant that, but what you wrote could also be taken as applying to the _run-time_ integer value 0, which C99's promise does not apply to. (Quotes are from 6.3.2.3.) I don't think there is any promise that converting a null pointer of any type back to an integer will necessarily produce a zero integer. Maybe we are reading things differently...? Looking at 6.3.2.3... As far as I read, paragraph 3 says: "An integer constant expression with the value 0, or such an expression cast to type void *, is called a null pointer constant.55) If a null pointer constant is converted to a pointer type, the resulting pointer, called a null pointer, is guaranteed to compare unequal to a pointer to any object or function." Essentially, the integer constant 0 can be casted to a pointer, and that pointer is then a null pointer constand, also called a null pointer. And footnote 55 says: "The macro NULL is defined in (and other headers) as a null pointer constant; see 7.17." So, 0 casted as a pointer gives a NULL pointer. And paragraph 6 says: "Any pointer type may be converted to an integer type. Except as previously specified, the result is implementation-defined. If the result cannot be represented in the integer type, the behavior is undefined. The result need not be in the range of values of any integer type." And I can only read the "previously specified" to refer to the equivalence between a NULL pointer and integer 0, because nothing before paragraph 6 talks about pointer to integer, so I can't see how it can be read as something more specific than all the things mentioned in the prebious 6 paragraphs. Johnny -- Johnny Billquist || "I'm on a bus || on a psychedelic trip email: b...@softjar.se || Reading murder books pdp is alive! || tryin' to stay hip" - B. Idol
Re: NULL pointer arithmetic issues
On 2020-02-24 21:18, Mouse wrote: If we use 0x0, it can be a valid pointer. If we use NULL, it's not expected to work and will eventually generate a syntax erro. Then someone has severely broken compatability with older versions of C. 0x0 and (when one of the suitable #includes has been done) NULL have both historically been perfectly good null pointer constants. Also...syntax error? Really? _Syntax_ error?? I'd really like to see what they've done to the grammar to lead to that; I'm having trouble imagining how that would be done. Unless I remember wrong, older C standards explicitly say that the integer 0 can be converted to a pointer, and that will be the NULL pointer, and a NULL pointer cast as an integer shall give the value 0. This is also used in such things as code traversing linked lists to check for the end of the list... And the C standard also explicitly allows the NULL pointer to not be represented by something with all bits cleared. Only that casting to/from integers have a very defined behavior. Johnny -- Johnny Billquist || "I'm on a bus || on a psychedelic trip email: b...@softjar.se || Reading murder books pdp is alive! || tryin' to stay hip" - B. Idol
Re: NULL pointer arithmetic issues
> Unless I remember wrong, older C standards explicitly say that the > integer 0 can be converted to a pointer, and that will be the NULL > pointer, and a NULL pointer cast as an integer shall give the value > 0. The only one I have anything close to a copy of is C99, for which I have a very late draft. Based on that: You are not quite correct. Any integer may be converted to a pointer, and any pointer may be converted to an integer - but the mapping is entirely implementation-dependent, except in the integer->pointer direction when the integer is a "null pointer constant", defined as "[a]n integer constant expression with the value 0" (or such an expression cast to void *, though not if we're talking specifically about integers), in which case "the resulting pointer, called a null pointer, is guaranteed to compare unequal to a pointer to any object or function". You could have meant that, but what you wrote could also be taken as applying to the _run-time_ integer value 0, which C99's promise does not apply to. (Quotes are from 6.3.2.3.) I don't think there is any promise that converting a null pointer of any type back to an integer will necessarily produce a zero integer. /~\ The ASCII Mouse \ / Ribbon Campaign X Against HTMLmo...@rodents-montreal.org / \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
Re: NULL pointer arithmetic issues
If we use 0x0, it can be a valid pointer. If we use NULL, it's not expected to work and [...] >> Then someone has severely broken compatability with older versions >> of C. [...] > The process of evaluation of the NULL semantics is not a recent > thing. No, it's not. But I was talking about the equivalence of 0x0 and NULL (in pointer contexts), not about what happens when you indirect through the result of converting either to a specific object pointer type. If 0x0 and NULL do different things in a pointer context, someone has severely broken backward compatability, regardless of what either of the "different things" is. There's a _lot_ of code that depends on the (historically promised by the spec) assurance that any of the various historically specified ways of spelling a null pointer constant, including 0x0 and, with a suitable #include, NULL, _is_ a null pointer constant. /~\ The ASCII Mouse \ / Ribbon Campaign X Against HTMLmo...@rodents-montreal.org / \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
Re: NULL pointer arithmetic issues
Kamil Rytarowski wrote: >We still maintain compatibility with this behavior (originated as a hack >in PDP11) in older NetBSD releases (NetBSD-0.9 Franz Lisp binaries >depend on this). I presume this was built for NetBSD/vax. A 68k build of Franz Lisp wouldn't try to dereference a NULL pointer.
Re: NULL pointer arithmetic issues
On 24.02.2020 21:18, Mouse wrote: >>> If we use 0x0, it can be a valid pointer. > >>> If we use NULL, it's not expected to work and will eventually >>> generate a syntax erro. > > Then someone has severely broken compatability with older versions of > C. 0x0 and (when one of the suitable #includes has been done) NULL > have both historically been perfectly good null pointer constants. > > Also...syntax error? Really? _Syntax_ error?? I'd really like to see > what they've done to the grammar to lead to that; I'm having trouble > imagining how that would be done. > The process of evaluation of the NULL semantics is not a recent thing. Not so long time, still in the NetBSD times, it was a general practice to allow dereferencing the NULL pointer and expect zeroed bytes over there. We still maintain compatibility with this behavior (originated as a hack in PDP11) in older NetBSD releases (NetBSD-0.9 Franz Lisp binaries depend on this). > /~\ The ASCII Mouse > \ / Ribbon Campaign > X Against HTML mo...@rodents-montreal.org > / \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B > signature.asc Description: OpenPGP digital signature
Re: NULL pointer arithmetic issues
>> If we use 0x0, it can be a valid pointer. >> If we use NULL, it's not expected to work and will eventually >> generate a syntax erro. Then someone has severely broken compatability with older versions of C. 0x0 and (when one of the suitable #includes has been done) NULL have both historically been perfectly good null pointer constants. Also...syntax error? Really? _Syntax_ error?? I'd really like to see what they've done to the grammar to lead to that; I'm having trouble imagining how that would be done. /~\ The ASCII Mouse \ / Ribbon Campaign X Against HTMLmo...@rodents-montreal.org / \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
re: NULL pointer arithmetic issues
> > Nonsense, I think it's fair to classify that as a bug. That sort of
> > stuff is *not* supposed to happen if -ffreestanding is passed to the
> > compiler.
>
> If we use 0x0, it can be a valid pointer.
>
> If we use NULL, it's not expected to work and will eventually generate a
> syntax erro.
this is not true in GCC, since a while now -- 2 years ago i
had to write this code to stop GCC from emitting "trap" when
accessing the 0x0 pointer, from powerpc/oea/oea_machdep.c:
/*
* Load pointer with 0 behind GCC's back, otherwise it will
* emit a "trap" instead.
*/
static __inline__ uintptr_t
zero_value(void)
{
uintptr_t dont_tell_gcc;
__asm volatile ("li %0, 0" : "=r"(dont_tell_gcc) :);
return dont_tell_gcc;
}
.mrg.
Re: NULL pointer arithmetic issues
In article , Kamil Rytarowski wrote: >-=-=-=-=-=- >-=-=-=-=-=- > >On 24.02.2020 13:41, Joerg Sonnenberger wrote: >> On Mon, Feb 24, 2020 at 11:42:01AM +0100, Kamil Rytarowski wrote: >>> Forbidding NULL pointer arithmetic is not just for C purists trolls. It >>> is now in C++ mainstream and already in C2x draft. >> >> This is not true. NULL pointer arithmetic and nullptr arithmetic are >> *very* different things. Do not conflate them. >> >> Joerg >> > >As noted, they are allowed to be practically the same in C++. The C >proposal (n2394) NULL is marked as deprecated and NULL should be set to >nullptr. This is just a proposal; once it becomes part of the standard we can worry about it. I agree with the rest of the people that we should (for now) change these cases in the sanitizer to not produce errors instead of making the code more complicated, to make the sanitizer happy. christos
Re: NULL pointer arithmetic issues
On 24.02.2020 15:35, Don Lee wrote: > >> On Feb 24, 2020, at 8:05 AM, Mouse wrote: >> > RUST is better defined that C and is indeed used in OS development > these days ...so? I don't see how this is related to the rest of the discussion. >>> As C is considered as not suitable for OS development, >> >> Once again, there is no such language as C. There is a family of >> closely related languages collectively called C. >> >> But it's actually the compiler, not the language. >> >>> there is an escape plan, already with a successful story in this >>> domain. >> >> There's another one, and one that doesn't require the complete rewrite >> a switch as drastic as C->rust would: various compilers (including >> older versions of the gcc family) that don't think it reasonable to >> take clear code and language-lawyer it into broken executables. >> > We need to be mindful of the gargantuan body of code written in “C”, > expecting the “old” behavior, much of it no longer having any sort of support. > > Software lives almost as long as government programs. > > -dgl- > While there, CHERI CPU can catch invalid intermediates (invalid pointer, before dereferencing). This is something that breaks a lot of old C code. tcpdump (that still preserves ifdefs for MSDOS) received rewrite to remove these types of bugs. https://www.cl.cam.ac.uk/~dc552/papers/asplos15-memory-safe-c.pdf signature.asc Description: OpenPGP digital signature
Re: NULL pointer arithmetic issues
> On Feb 24, 2020, at 8:05 AM, Mouse wrote: > RUST is better defined that C and is indeed used in OS development these days >>> ...so? I don't see how this is related to the rest of the >>> discussion. >> As C is considered as not suitable for OS development, > > Once again, there is no such language as C. There is a family of > closely related languages collectively called C. > > But it's actually the compiler, not the language. > >> there is an escape plan, already with a successful story in this >> domain. > > There's another one, and one that doesn't require the complete rewrite > a switch as drastic as C->rust would: various compilers (including > older versions of the gcc family) that don't think it reasonable to > take clear code and language-lawyer it into broken executables. > We need to be mindful of the gargantuan body of code written in “C”, expecting the “old” behavior, much of it no longer having any sort of support. Software lives almost as long as government programs. -dgl-
Re: NULL pointer arithmetic issues
On 24.02.2020 15:04, Jason Thorpe wrote: > >> On Feb 24, 2020, at 4:22 AM, Kamil Rytarowski wrote: >> >> A compiler once being smart enough can introduce ILL/SEGV traps into >> code that performs operations on NULL pointers. This already bitten us >> when we were registering a handler at address 0x0 for the kernel code, >> GCC changed the operation into a cpu trap. (IIRC it was in the sparc code.) > > Nonsense, I think it's fair to classify that as a bug. That sort of stuff is > *not* supposed to happen if -ffreestanding is passed to the compiler. > > -- thorpej > If we use 0x0, it can be a valid pointer. If we use NULL, it's not expected to work and will eventually generate a syntax erro. UBSan as a runtime tool tries to indirectly catch the latter with the former and is prone to some rare false positives (so far not reported). If a compiler is too smart for 0x0 pointers, transforming them to abort traps, it is a compiler bug. I noted that this already happens. On 24.02.2020 15:05, Mouse wrote: > (3) If you have reason to think the C committee would be interested in > having me as a member, let me know whom to talk to. I might or might > not actually end up interested in joining, but I'd like more info. http://www.open-std.org/jtc1/sc22/wg14/ signature.asc Description: OpenPGP digital signature
Re: NULL pointer arithmetic issues
> On Feb 24, 2020, at 4:22 AM, Kamil Rytarowski wrote: > > A compiler once being smart enough can introduce ILL/SEGV traps into > code that performs operations on NULL pointers. This already bitten us > when we were registering a handler at address 0x0 for the kernel code, > GCC changed the operation into a cpu trap. (IIRC it was in the sparc code.) Nonsense, I think it's fair to classify that as a bug. That sort of stuff is *not* supposed to happen if -ffreestanding is passed to the compiler. -- thorpej
Re: NULL pointer arithmetic issues
>> C is not a language. C is a family of closely related languages. >> Some of them are suitable for OS implementation. It appears some of >> the more recent ones are not, but this does not mean the older ones >> also aren't. > From my perception the trend is inversed. Things that were undefined > or unspecified in older revisions of C, are more clearly defined now. You seem to be confusing "clearly defined" with "useful". Modern members of the C family may indeed be more clearly defined. That is not the problem. The problem is...hmm, actually, I misspoke upthread. It is not the language that is the problem; it is the compiler. Unless - and I find this highly unlikely - there is something in the latest versions of C _requiring_ the compiler to perform these unexpected transformations, the language itself is fine; it is the compiler that is at fault, in that it chooses to take advantage of the not-forbidden-by-the-spec latitude to "optimize" code in unexpected ways. >> [...]; it is not a compiler's place to take the position of "ha ha, >> the code you wrote is clear but I can find a way to lawyer it into >> formally undefined behaviour, so I'm going to transform it into >> something I know damn well you didn't expect". > Please join the C committee as a voting member or at least submit > papers with language changes. Complaining here won't change > anything. (1) It might get NetBSD to stop trying to insist on using a compiler that is not suitable for the purpose. (2) As I realized above, it's not the language that's the problem. (3) If you have reason to think the C committee would be interested in having me as a member, let me know whom to talk to. I might or might not actually end up interested in joining, but I'd like more info. >>> RUST is better defined that C and is indeed used in OS development >>> these days >> ...so? I don't see how this is related to the rest of the >> discussion. > As C is considered as not suitable for OS development, Once again, there is no such language as C. There is a family of closely related languages collectively called C. But it's actually the compiler, not the language. > there is an escape plan, already with a successful story in this > domain. There's another one, and one that doesn't require the complete rewrite a switch as drastic as C->rust would: various compilers (including older versions of the gcc family) that don't think it reasonable to take clear code and language-lawyer it into broken executables. /~\ The ASCII Mouse \ / Ribbon Campaign X Against HTMLmo...@rodents-montreal.org / \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
Re: NULL pointer arithmetic issues
On 24.02.2020 14:03, Mouse wrote: It is now in C++ mainstream and already in C2x draft. >>> Then those are not suitable languages for OS implementations. >> This battle is lost for C > > C is not a language. C is a family of closely related languages. > If we tread C as gnu89 gnu99 gnu11 k etc this is true. > Some of them are suitable for OS implementation. It appears some of > the more recent ones are not, but this does not mean the older ones > also aren't. > From my perception the trend is inversed. Things that were undefined or unspecified in older revisions of C, are more clearly defined now. > Undefined behaviour as a way of describing differences between > implementations, things that it limits portability to depend on, is > useful. Undefined behaviour as a license-by-fiat for compilers to > unnecessarily transform code in unexpected ways is not. Software > languages and their compilers exist to serve their users, not the other > way around; it is not a compiler's place to take the position of "ha > ha, the code you wrote is clear but I can find a way to lawyer it into > formally undefined behaviour, so I'm going to transform it into > something I know damn well you didn't expect". > Please join the C committee as a voting member or at least submit papers with language changes. Complaining here won't change anything. (Out of people in the discussion, I am involved in wg14 discussions and submit papers.) >> RUST is better defined that C and is indeed used in OS development >> these days > > ...so? I don't see how this is related to the rest of the discussion. > As C is considered as not suitable for OS development, there is an escape plan, already with a successful story in this domain. > /~\ The ASCII Mouse > \ / Ribbon Campaign > X Against HTML mo...@rodents-montreal.org > / \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B > signature.asc Description: OpenPGP digital signature
Re: NULL pointer arithmetic issues
On 24.02.2020 13:41, Joerg Sonnenberger wrote: > On Mon, Feb 24, 2020 at 11:42:01AM +0100, Kamil Rytarowski wrote: >> Forbidding NULL pointer arithmetic is not just for C purists trolls. It >> is now in C++ mainstream and already in C2x draft. > > This is not true. NULL pointer arithmetic and nullptr arithmetic are > *very* different things. Do not conflate them. > > Joerg > As noted, they are allowed to be practically the same in C++. The C proposal (n2394) NULL is marked as deprecated and NULL should be set to nullptr. signature.asc Description: OpenPGP digital signature
Re: NULL pointer arithmetic issues
>>> It is now in C++ mainstream and already in C2x draft. >> Then those are not suitable languages for OS implementations. > This battle is lost for C C is not a language. C is a family of closely related languages. Some of them are suitable for OS implementation. It appears some of the more recent ones are not, but this does not mean the older ones also aren't. Undefined behaviour as a way of describing differences between implementations, things that it limits portability to depend on, is useful. Undefined behaviour as a license-by-fiat for compilers to unnecessarily transform code in unexpected ways is not. Software languages and their compilers exist to serve their users, not the other way around; it is not a compiler's place to take the position of "ha ha, the code you wrote is clear but I can find a way to lawyer it into formally undefined behaviour, so I'm going to transform it into something I know damn well you didn't expect". > RUST is better defined that C and is indeed used in OS development > these days ...so? I don't see how this is related to the rest of the discussion. /~\ The ASCII Mouse \ / Ribbon Campaign X Against HTMLmo...@rodents-montreal.org / \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
Re: NULL pointer arithmetic issues
On Mon, Feb 24, 2020 at 11:42:01AM +0100, Kamil Rytarowski wrote: > Forbidding NULL pointer arithmetic is not just for C purists trolls. It > is now in C++ mainstream and already in C2x draft. This is not true. NULL pointer arithmetic and nullptr arithmetic are *very* different things. Do not conflate them. Joerg
Re: NULL pointer arithmetic issues
On 24.02.2020 12:14, Mouse wrote: >> Forbidding NULL pointer arithmetic is not just for C purists trolls. >> It is now in C++ mainstream and already in C2x draft. > > Then those are not suitable languages for OS implementations. > > I'm with campbell and mrg on this one. It is not appropriate to twist > NetBSD's code into a pretzel to work around "bugs" created by language > committees deciding to give compilers new latitutde to "optimize" > meaningful code into trash. > This battle is lost for C and not be fought on a downstream user of a C compiler (Matt Thomas insisted at some point to get the kernel buildable with C++ and patched it for this..). A compiler once being smart enough can introduce ILL/SEGV traps into code that performs operations on NULL pointers. This already bitten us when we were registering a handler at address 0x0 for the kernel code, GCC changed the operation into a cpu trap. (IIRC it was in the sparc code.) Looking at it from the proper perspective, the only rumpkernel reported NULL->0 arithmetic is performed by the pserialize macros. Once we will patch them, the problem can go away. So claim about twisting the kernel code or churn is exaggeration. RUST is better defined that C and is indeed used in OS development these days (there are startups doing OS development in RUST, e.g. https://github.com/oxidecomputer). signature.asc Description: OpenPGP digital signature
Re: NULL pointer arithmetic issues
> Forbidding NULL pointer arithmetic is not just for C purists trolls. > It is now in C++ mainstream and already in C2x draft. Then those are not suitable languages for OS implementations. I'm with campbell and mrg on this one. It is not appropriate to twist NetBSD's code into a pretzel to work around "bugs" created by language committees deciding to give compilers new latitutde to "optimize" meaningful code into trash. /~\ The ASCII Mouse \ / Ribbon Campaign X Against HTMLmo...@rodents-montreal.org / \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
Re: NULL pointer arithmetic issues
On 24.02.2020 05:03, Taylor R Campbell wrote: >> Date: Sun, 23 Feb 2020 22:51:08 +0100 >> From: Kamil Rytarowski >> >> On 23.02.2020 20:08, Taylor R Campbell wrote: >> Date: Sun, 23 Feb 2020 22:51:08 +0100 >> From: Kamil Rytarowski >> >> On 23.02.2020 20:08, Taylor R Campbell wrote: Date: Sat, 22 Feb 2020 17:25:42 +0100 From: Kamil Rytarowski What's the proper approach to address this issue? >>> >>> What do these reports mean? >>> >>> UBSan: Undefined Behavior in >>> /usr/src/sys/rump/net/lib/libnet/../../../../netinet6/in6.c:2351:2, pointer >>> expression with base 0 overflowed to 0 >> >> We added 0 to a NULL pointer. >> >> They can be triggered by code like: >> >> char *p = NULL; >> p += 0; > > It seems to me the proper approach is to teach the tool to accept > this, and to avoid cluttering the tree with churn to work around the > tool's deficiency, unless there's actually a serious compelling > argument -- beyond a language-lawyering troll -- that (char *)NULL + 0 > is meaningfully undefined. > > We already assume, for example, that memset(...,0,...) is the same as > initialization to null pointers where the object in question is a > pointer or has pointers as subobjects. > Forbidding NULL pointer arithmetic is not just for C purists trolls. It is now in C++ mainstream and already in C2x draft. The newer C standard will most likely (already accepted by the committee) adopt nullptr on par with nullptr from C++. In C++ we can "#define NULL nullptr" and possibly the same will be possible in C. http://www.open-std.org/jtc1/sc22/wg14/www/docs/n2394.pdf This will change all arithmetic code operating on NULL into syntax error. > I think we should treat memcpy(NULL,NULL,0) similarly and tell the > tool `no, on NetBSD that really is defined and we're not interested in > hearing about theoretical nasal demons from armchair language > lawyers'. > memcpy(3) and other string functions are different. It is undefined if we just run it with memcpy(rand(), rand(), 0) and the first two arguments point to invalid memory. memcpy(0, 0, x) have another issue with overlapping memory that makes it undefined. In theory memcpy(x,y,z) where x or y are 0 is valid, whenever we map 0x0 in the address space, but that is so rare that GCC defines these arguments as nonnull. signature.asc Description: OpenPGP digital signature
re: NULL pointer arithmetic issues
> It seems to me the proper approach is to teach the tool to accept > this, and to avoid cluttering the tree with churn to work around the > tool's deficiency, unless there's actually a serious compelling > argument -- beyond a language-lawyering troll -- that (char *)NULL + 0 > is meaningfully undefined. > > We already assume, for example, that memset(...,0,...) is the same as > initialization to null pointers where the object in question is a > pointer or has pointers as subobjects. > > I think we should treat memcpy(NULL,NULL,0) similarly and tell the > tool `no, on NetBSD that really is defined and we're not interested in > hearing about theoretical nasal demons from armchair language > lawyers'. well said. i 100% agree. these extreme edge-cases of UB that have a very clear definition don't seem to he helpful in finding any real class of bugs and only seem to be good at cluttering up working code. .mrg.
Re: NULL pointer arithmetic issues
> Date: Sun, 23 Feb 2020 22:51:08 +0100 > From: Kamil Rytarowski > > On 23.02.2020 20:08, Taylor R Campbell wrote: > Date: Sun, 23 Feb 2020 22:51:08 +0100 > From: Kamil Rytarowski > > On 23.02.2020 20:08, Taylor R Campbell wrote: > >> Date: Sat, 22 Feb 2020 17:25:42 +0100 > >> From: Kamil Rytarowski > >> > >> What's the proper approach to address this issue? > > > > What do these reports mean? > > > > UBSan: Undefined Behavior in > > /usr/src/sys/rump/net/lib/libnet/../../../../netinet6/in6.c:2351:2, pointer > > expression with base 0 overflowed to 0 > > We added 0 to a NULL pointer. > > They can be triggered by code like: > > char *p = NULL; > p += 0; It seems to me the proper approach is to teach the tool to accept this, and to avoid cluttering the tree with churn to work around the tool's deficiency, unless there's actually a serious compelling argument -- beyond a language-lawyering troll -- that (char *)NULL + 0 is meaningfully undefined. We already assume, for example, that memset(...,0,...) is the same as initialization to null pointers where the object in question is a pointer or has pointers as subobjects. I think we should treat memcpy(NULL,NULL,0) similarly and tell the tool `no, on NetBSD that really is defined and we're not interested in hearing about theoretical nasal demons from armchair language lawyers'.
Re: NULL pointer arithmetic issues
On 23.02.2020 20:08, Taylor R Campbell wrote:
>> Date: Sat, 22 Feb 2020 17:25:42 +0100
>> From: Kamil Rytarowski
>>
>> When running the ATF tests under MKLIBCSANITIZER [1], there are many
>> NULL pointer arithmetic issues .
>>
>> http://netbsd.org/~kamil/mksanitizer-reports/ubsan-2020-02-22-null-pointer.txt
>>
>> These issues are in macros like:
>> - IN_ADDRHASH_READER_FOREACH()
>> - IN_ADDRLIST_WRITER_INSERT_TAIL()
>> - IFADDR_READER_FOREACH()
>> - etc
>>
>> These macros wrap internally pserialize-safe linked lists.
>>
>> What's the proper approach to address this issue?
>
> What do these reports mean?
>
> UBSan: Undefined Behavior in
> /usr/src/sys/rump/net/lib/libnet/../../../../netinet6/in6.c:2351:2, pointer
> expression with base 0 overflowed to 0
>
We added 0 to a NULL pointer.
They can be triggered by code like:
char *p = NULL;
p += 0;
or
char *p = NULL;
if (p[0] == NULL) {}
signature.asc
Description: OpenPGP digital signature
Re: NULL pointer arithmetic issues
> Date: Sat, 22 Feb 2020 17:25:42 +0100 > From: Kamil Rytarowski > > When running the ATF tests under MKLIBCSANITIZER [1], there are many > NULL pointer arithmetic issues . > > http://netbsd.org/~kamil/mksanitizer-reports/ubsan-2020-02-22-null-pointer.txt > > These issues are in macros like: > - IN_ADDRHASH_READER_FOREACH() > - IN_ADDRLIST_WRITER_INSERT_TAIL() > - IFADDR_READER_FOREACH() > - etc > > These macros wrap internally pserialize-safe linked lists. > > What's the proper approach to address this issue? What do these reports mean? UBSan: Undefined Behavior in /usr/src/sys/rump/net/lib/libnet/../../../../netinet6/in6.c:2351:2, pointer expression with base 0 overflowed to 0
Re: NULL pointer arithmetic issues
On 22.02.2020 19:39, Joerg Sonnenberger wrote: > On Sat, Feb 22, 2020 at 05:25:42PM +0100, Kamil Rytarowski wrote: >> When running the ATF tests under MKLIBCSANITIZER [1], there are many >> NULL pointer arithmetic issues . > > Which flags are the sanitizers using? Because I wouldn't be surprised if > they just hit _PSLIST_VALIDATE_PTRS and friends. > > Joerg > This patch did not help. I double checked that this branch is really taken. Index: sys/sys/pslist.h === RCS file: /cvsroot/src/sys/sys/pslist.h,v retrieving revision 1.7 diff -u -r1.7 pslist.h --- sys/sys/pslist.h1 Dec 2019 15:28:19 - 1.7 +++ sys/sys/pslist.h22 Feb 2020 20:51:42 - @@ -32,6 +32,7 @@ #ifndef_SYS_PSLIST_H #define_SYS_PSLIST_H +#include #include #include @@ -288,7 +289,9 @@ * Type-safe macros for convenience. */ -#if defined(__COVERITY__) || defined(__LGTM_BOT__) +#if defined(__COVERITY__) || defined(__LGTM_BOT__) || \ + __has_feature(undefined_behavior_sanitizer) || \ + defined(__SANITIZE_UNDEFINED__) #define_PSLIST_VALIDATE_PTRS(P, Q) 0 #define_PSLIST_VALIDATE_CONTAINER(P, T, F) 0 #else signature.asc Description: OpenPGP digital signature
Re: NULL pointer arithmetic issues
On Sat, Feb 22, 2020 at 05:25:42PM +0100, Kamil Rytarowski wrote: > When running the ATF tests under MKLIBCSANITIZER [1], there are many > NULL pointer arithmetic issues . Which flags are the sanitizers using? Because I wouldn't be surprised if they just hit _PSLIST_VALIDATE_PTRS and friends. Joerg
