Re: (ITS#8988) Undefined Behavior in slapadd

2020-02-03 Thread openldap-technical 
  This message is in MIME format.  The first part should be readable text,
  while the remaining parts are likely unreadable without MIME-aware tools.

---1463811718-1330329680-1580755187=:215607
Content-Type: text/plain; charset=ISO-8859-15
Content-Transfer-Encoding: 8BIT


Hello,

On Mon, 3 Feb 2020, h...@symas.com wrote:

> No. The POSIX spec is *not* a part of the C spec - yet it is still valid.

POSIX spec *is* a C standard that exists in addition to the ISO C 
standard. ISO C defines the so-called "basic language and features" of C 
if you can call it that.

> Which simply proves the point that just because something is not defined
> in the C spec does not make it invalid. The compiler guys are idiots for
> taking this position.

Like Jeff already said, wrongly aligned object access *is* defined by the 
ISO C specification. It is wrong and explicitly defined as Undefined 
Behaviour. Invoking UB is playing with fire, and calling compiler writers 
with offending names does not change that fact at all.

Spending long time on comp.lang.c should be mandatory for all C 
programmers out there. It is shocking to invoke UB and not bother to fix 
it, instead blaming compiler writers and C standard writers.

Best regards,
Jokke Hämäläinen
---1463811718-1330329680-1580755187=:215607--

Re: (ITS#8988) Undefined Behavior in slapadd

2020-02-03 Thread hyc 
openldap-techni...@kolttonen.fi wrote:
>   This message is in MIME format.  The first part should be readable text,
>   while the remaining parts are likely unreadable without MIME-aware tools.
> 
> ---1463811718-806940296-1580670508=:196090
> Content-Type: text/plain; charset=ISO-8859-15
> Content-Transfer-Encoding: 8BIT
> 
> 
> Hello,
> 
> On Sat, 8 Jun 2019, h...@symas.com wrote:
> 
>> The gcc/clang folks have their heads up their asses. They've 
>> deliberately misinterpreted the spec claiming undefined behaviors are 
>> forbidden, supposedly to enable essential optimizations. Most of which 
>> only apply to obscure corner cases in compiler benchmark suites, that 
>> nobody in the real world ever benefits from.
> 
> I realize this thread is very old, but Jeff and the C compiler folks are 
> right. All C programs that invoke undefined behavior are illegal C 
> programs and should be fixed.
> 
> Posix threads are well-defined by Posix standards, so calling them 
> "undefined behaviour" is not a valid argument.

No. The POSIX spec is *not* a part of the C spec - yet it is still valid.
Which simply proves the point that just because something is not defined
in the C spec does not make it invalid. The compiler guys are idiots for
taking this position.

-- 
  -- Howard Chu
  CTO, Symas Corp.   http://www.symas.com
  Director, Highland Sun http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/

Re: (ITS#8988) Undefined Behavior in slapadd

2020-02-02 Thread openldap-technical 
  This message is in MIME format.  The first part should be readable text,
  while the remaining parts are likely unreadable without MIME-aware tools.

---1463811718-326687762-1580674674=:196994
Content-Type: text/plain; charset=ISO-8859-15
Content-Transfer-Encoding: 8BIT



On Sun, 2 Feb 2020, openldap-techni...@kolttonen.fi wrote:

> I realize this thread is very old, but Jeff and the C compiler folks are 
> right. All C programs that invoke undefined behavior are illegal C 
> programs and should be fixed.
> 
> Posix threads are well-defined by Posix standards, so calling them 
> "undefined behaviour" is not a valid argument.

To make it clear that UB in C programs is horrible, let me provide a real 
world example of UB: Cyrus IMAPD had an unnoticed dormant strcpy() related 
UB bug for several years. You see, the manual page of strcpy() says that 
the src and dst strings must not overlap, or else the C program invokes 
UB.

For many years, Cyrus operated correctly even though the src and dst 
strings *did* overlap in one part of their database code. UB of course 
allows this behaviour too. The reason was because GNU libc folks had 
written their strcpy() implementation in such a way that breaking the 
contract of having non-overlapping strings did not cause any problems.

Then, years later, GNU libc folks deciced to optimize (or otherwise 
change) their strcpy() implementation. Now the new implementation punished 
all UB invokers, and we saw our Cyrus mailbox database slowly but surely 
getting corrupted.

It is indeed shocking to hear that LMDB implementation invokes UB. GCC and 
Clang could change their behaviour tomorrow, and LMDB could get corrupted, 
crash or whatever. And certainly the blame would be on LMDB code, not the 
compiler writers.

Best Regards,
Jokke Hämäläinen
---1463811718-326687762-1580674674=:196994--

Re: (ITS#8988) Undefined Behavior in slapadd

2020-02-02 Thread openldap-technical 
  This message is in MIME format.  The first part should be readable text,
  while the remaining parts are likely unreadable without MIME-aware tools.

---1463811718-806940296-1580670508=:196090
Content-Type: text/plain; charset=ISO-8859-15
Content-Transfer-Encoding: 8BIT


Hello,

On Sat, 8 Jun 2019, h...@symas.com wrote:

> The gcc/clang folks have their heads up their asses. They've 
> deliberately misinterpreted the spec claiming undefined behaviors are 
> forbidden, supposedly to enable essential optimizations. Most of which 
> only apply to obscure corner cases in compiler benchmark suites, that 
> nobody in the real world ever benefits from.

I realize this thread is very old, but Jeff and the C compiler folks are 
right. All C programs that invoke undefined behavior are illegal C 
programs and should be fixed.

Posix threads are well-defined by Posix standards, so calling them 
"undefined behaviour" is not a valid argument.

Best regards,
Jokke Hämäläinen
---1463811718-806940296-1580670508=:196090--

Re: (ITS#8988) Undefined Behavior in slapadd

2019-06-08 Thread hyc 
Jeffrey Walton wrote:
> On Sat, Jun 8, 2019 at 3:47 AM Howard Chu  wrote:
>>
>> Jeffrey Walton wrote:
>>> On Fri, Jun 7, 2019 at 2:37 PM Howard Chu  wrote:

 Jeffrey Walton wrote:
> On Fri, Jun 7, 2019 at 10:08 AM Howard Chu  wrote:
>>
>> Jeffrey Walton wrote:
>>> On Fri, Jun 7, 2019 at 9:59 AM Howard Chu  wrote:

 noloa...@gmail.com wrote:
> On Fri, Jun 7, 2019 at 9:32 AM Howard Chu  wrote:
>>
>> noloa...@gmail.com wrote:
>> ...
>>> I encourage OpenLDAP to fix the undefined behavior. OpenLDAP is an
>>> important project, and the undefined behavior is causing too many
>>> tangential problems.
>>
>> Undefined behavior is not a bug, nor is it prohibited by the C spec. 
>> It is a necessary
>> part of the language for its intended use as a system programming 
>> language, writing
>> machine-specific programs. Anyone who says it is prohibited by the 
>> spec is wrong.
>
> I don't believe this is correct.
>
> Maybe you are thinking of implementation defined behavior?

 That would apply to how a particular compiler implementation treats 
 some piece of code.
 Whether a CPU supports unaligned access is machine-defined. Since our 
 code is already
 properly ifdef'd for the machines which do or don't support it, the 
 fact that this is
 "non-portable" is not an issue.
>>>
>>> The undefined behavior is causing OpenLDAP to fail testing.
>>>
>>> Worse, it is causing test failures in programs and libraries which use
>>> OpenLDAP. The OpenLDAP bugs have cross-pollinated into other programs
>>> and libraries. It is not simply contained or limited to OpenLDAP.
>>>
>>> That's a big problem for testing a QA folks.
>>
>> Then the tests are broken, because these are not bugs.
>
> They are absolutely OpenLDAP bugs. The unaligned accesses are
> Undefined Behavior.

 Simply because the C standard doesn't specify what the behavior is doesn't 
 make it a bug.
>>>
>>> The C standard does specify the behavior. It falls clearly in
>>> Undefined Behavior.
>>
>> The fact a behavior is undefined does not make it illegal.
> 
> A program with undefined behavior is, by definition, and illegal program.

This is pure nonsense. You realize, don't you, that the C spec has no definition
of threads? All multithreaded programs are using undefined behavior. And yet
here we are, using threads for over 2 decades.

The gcc/clang folks have their heads up their asses. They've deliberately
misinterpreted the spec claiming undefined behaviors are forbidden, supposedly
to enable essential optimizations. Most of which only apply to obscure corner
cases in compiler benchmark suites, that nobody in the real world ever benefits 
from.

-- 
  -- Howard Chu
  CTO, Symas Corp.   http://www.symas.com
  Director, Highland Sun http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/

Re: (ITS#8988) Undefined Behavior in slapadd

2019-06-08 Thread noloader 
On Sat, Jun 8, 2019 at 3:47 AM Howard Chu  wrote:
>
> Jeffrey Walton wrote:
> > On Fri, Jun 7, 2019 at 2:37 PM Howard Chu  wrote:
> >>
> >> Jeffrey Walton wrote:
> >>> On Fri, Jun 7, 2019 at 10:08 AM Howard Chu  wrote:
> 
>  Jeffrey Walton wrote:
> > On Fri, Jun 7, 2019 at 9:59 AM Howard Chu  wrote:
> >>
> >> noloa...@gmail.com wrote:
> >>> On Fri, Jun 7, 2019 at 9:32 AM Howard Chu  wrote:
> 
>  noloa...@gmail.com wrote:
>  ...
> > I encourage OpenLDAP to fix the undefined behavior. OpenLDAP is an
> > important project, and the undefined behavior is causing too many
> > tangential problems.
> 
>  Undefined behavior is not a bug, nor is it prohibited by the C spec. 
>  It is a necessary
>  part of the language for its intended use as a system programming 
>  language, writing
>  machine-specific programs. Anyone who says it is prohibited by the 
>  spec is wrong.
> >>>
> >>> I don't believe this is correct.
> >>>
> >>> Maybe you are thinking of implementation defined behavior?
> >>
> >> That would apply to how a particular compiler implementation treats 
> >> some piece of code.
> >> Whether a CPU supports unaligned access is machine-defined. Since our 
> >> code is already
> >> properly ifdef'd for the machines which do or don't support it, the 
> >> fact that this is
> >> "non-portable" is not an issue.
> >
> > The undefined behavior is causing OpenLDAP to fail testing.
> >
> > Worse, it is causing test failures in programs and libraries which use
> > OpenLDAP. The OpenLDAP bugs have cross-pollinated into other programs
> > and libraries. It is not simply contained or limited to OpenLDAP.
> >
> > That's a big problem for testing a QA folks.
> 
>  Then the tests are broken, because these are not bugs.
> >>>
> >>> They are absolutely OpenLDAP bugs. The unaligned accesses are
> >>> Undefined Behavior.
> >>
> >> Simply because the C standard doesn't specify what the behavior is doesn't 
> >> make it a bug.
> >
> > The C standard does specify the behavior. It falls clearly in
> > Undefined Behavior.
>
> The fact a behavior is undefined does not make it illegal.

A program with undefined behavior is, by definition, and illegal program.

> > And Appendix J.2, Undefined Behavior, says:
> >
> > Conversion between two pointer types produces a result
> > that is incorrectly aligned (6.3.2.3).
> >
> > You are violating the C standard in macros like COPY_PGNO. Effectively
> > you are casting a uint8_t array to an uint16_t*. You cannot do that
> > because uint16_t* increases the alignment requirements.
>
> All internal fields of LMDB are (at least) 2-byte aligned, so these copies 
> always
> meet alignment requirements.

No, they are not. Some of them are 1-byte aligned. UBsan tells you that.

Others require 4-byte alignment. UBsan tells you that, too.

Jeff

Re: (ITS#8988) Undefined Behavior in slapadd

2019-06-08 Thread hyc 
Jeffrey Walton wrote:
> On Fri, Jun 7, 2019 at 2:37 PM Howard Chu  wrote:
>>
>> Jeffrey Walton wrote:
>>> On Fri, Jun 7, 2019 at 10:08 AM Howard Chu  wrote:

 Jeffrey Walton wrote:
> On Fri, Jun 7, 2019 at 9:59 AM Howard Chu  wrote:
>>
>> noloa...@gmail.com wrote:
>>> On Fri, Jun 7, 2019 at 9:32 AM Howard Chu  wrote:

 noloa...@gmail.com wrote:
 ...
> I encourage OpenLDAP to fix the undefined behavior. OpenLDAP is an
> important project, and the undefined behavior is causing too many
> tangential problems.

 Undefined behavior is not a bug, nor is it prohibited by the C spec. 
 It is a necessary
 part of the language for its intended use as a system programming 
 language, writing
 machine-specific programs. Anyone who says it is prohibited by the 
 spec is wrong.
>>>
>>> I don't believe this is correct.
>>>
>>> Maybe you are thinking of implementation defined behavior?
>>
>> That would apply to how a particular compiler implementation treats some 
>> piece of code.
>> Whether a CPU supports unaligned access is machine-defined. Since our 
>> code is already
>> properly ifdef'd for the machines which do or don't support it, the fact 
>> that this is
>> "non-portable" is not an issue.
>
> The undefined behavior is causing OpenLDAP to fail testing.
>
> Worse, it is causing test failures in programs and libraries which use
> OpenLDAP. The OpenLDAP bugs have cross-pollinated into other programs
> and libraries. It is not simply contained or limited to OpenLDAP.
>
> That's a big problem for testing a QA folks.

 Then the tests are broken, because these are not bugs.
>>>
>>> They are absolutely OpenLDAP bugs. The unaligned accesses are
>>> Undefined Behavior.
>>
>> Simply because the C standard doesn't specify what the behavior is doesn't 
>> make it a bug.
> 
> The C standard does specify the behavior. It falls clearly in
> Undefined Behavior.

The fact a behavior is undefined does not make it illegal.

> And Appendix J.2, Undefined Behavior, says:
> 
> Conversion between two pointer types produces a result
> that is incorrectly aligned (6.3.2.3).
> 
> You are violating the C standard in macros like COPY_PGNO. Effectively
> you are casting a uint8_t array to an uint16_t*. You cannot do that
> because uint16_t* increases the alignment requirements.

All internal fields of LMDB are (at least) 2-byte aligned, so these copies 
always
meet alignment requirements.

-- 
  -- Howard Chu
  CTO, Symas Corp.   http://www.symas.com
  Director, Highland Sun http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/

Re: (ITS#8988) Undefined Behavior in slapadd

2019-06-08 Thread noloader 
On Fri, Jun 7, 2019 at 2:37 PM Howard Chu  wrote:
>
> Jeffrey Walton wrote:
> > On Fri, Jun 7, 2019 at 10:08 AM Howard Chu  wrote:
> >>
> >> Jeffrey Walton wrote:
> >>> On Fri, Jun 7, 2019 at 9:59 AM Howard Chu  wrote:
> 
>  noloa...@gmail.com wrote:
> > On Fri, Jun 7, 2019 at 9:32 AM Howard Chu  wrote:
> >>
> >> noloa...@gmail.com wrote:
> >> ...
> >>> I encourage OpenLDAP to fix the undefined behavior. OpenLDAP is an
> >>> important project, and the undefined behavior is causing too many
> >>> tangential problems.
> >>
> >> Undefined behavior is not a bug, nor is it prohibited by the C spec. 
> >> It is a necessary
> >> part of the language for its intended use as a system programming 
> >> language, writing
> >> machine-specific programs. Anyone who says it is prohibited by the 
> >> spec is wrong.
> >
> > I don't believe this is correct.
> >
> > Maybe you are thinking of implementation defined behavior?
> 
>  That would apply to how a particular compiler implementation treats some 
>  piece of code.
>  Whether a CPU supports unaligned access is machine-defined. Since our 
>  code is already
>  properly ifdef'd for the machines which do or don't support it, the fact 
>  that this is
>  "non-portable" is not an issue.
> >>>
> >>> The undefined behavior is causing OpenLDAP to fail testing.
> >>>
> >>> Worse, it is causing test failures in programs and libraries which use
> >>> OpenLDAP. The OpenLDAP bugs have cross-pollinated into other programs
> >>> and libraries. It is not simply contained or limited to OpenLDAP.
> >>>
> >>> That's a big problem for testing a QA folks.
> >>
> >> Then the tests are broken, because these are not bugs.
> >
> > They are absolutely OpenLDAP bugs. The unaligned accesses are
> > Undefined Behavior.
>
> Simply because the C standard doesn't specify what the behavior is doesn't 
> make it a bug.

The C standard does specify the behavior. It falls clearly in
Undefined Behavior.

> It would be unreasonable to expect the standards body to exhaustively list 
> all of the
> possible behaviors on all of the possible hardware architectures that the 
> language is
> implemented for. Leaving it undefined in the spec only means they chose not 
> to specify
> a behavior. That means the actual behavior depends on the underlying machine. 
> On the
> x86 family the behavior is well defined.

You seem to have a misunderstanding of the standard and unaligned
accesses. You also seem to be conflating two different concepts.

First, unaligned accesses, are undefined behavior. They are forbidden
by the standard in 6.3.2.3 Pointers:

7. A pointer to an object type may be converted to
a pointer to a different object type. If the resulting pointer
is not correctly aligned for the referenced type, the
behavior is undefined.

And Appendix J.2, Undefined Behavior, says:

Conversion between two pointer types produces a result
that is incorrectly aligned (6.3.2.3).

You are violating the C standard in macros like COPY_PGNO. Effectively
you are casting a uint8_t array to an uint16_t*. You cannot do that
because uint16_t* increases the alignment requirements.

Tools like UBsan are telling you when you violate the rules. They
specifically flagged COPY_PGNO as incurring undefined behavior. (Tools
like UBsan will miss some UB, but that's a different error and not
applicable to this discussion).

You insistence on claiming the code is correct is befuddling. You are
effectively saying the world is wrong, including the C standard, the
compiler writers, other projects, the QA personnel testing the
software and users like me.

There's nothing special about me being wrong. I've had to acknowledged
that often. However, I would take pause in claiming the C standard is
wrong or the compiler writers, like GCC and LLVM devs, are wrong.
That's pretty bold.

Second, conflating concepts, is the difference between the C standard
and processor capabilities. The C code has undefined behavior, and it
does not matter what the processor is capable of.

And it does not matter if the compiler decides to generate code that
performs unaligned accesses. The compiler is free to do what it wants,
even when it frustrates the user or makes the user envious. However,
you cannot wrote code that uses unaligned objects even though the
compiler may generate equivalent code.

Jeff

Re: (ITS#8988) Undefined Behavior in slapadd

2019-06-07 Thread hyc 
Howard Chu wrote:
> h...@symas.com wrote:
>> Jeffrey Walton wrote:
>>> On Fri, Jun 7, 2019 at 10:08 AM Howard Chu  wrote:

 Jeffrey Walton wrote:
> On Fri, Jun 7, 2019 at 9:59 AM Howard Chu  wrote:
>>
>> noloa...@gmail.com wrote:
>>> On Fri, Jun 7, 2019 at 9:32 AM Howard Chu  wrote:

 noloa...@gmail.com wrote:
 ...
> I encourage OpenLDAP to fix the undefined behavior. OpenLDAP is an
> important project, and the undefined behavior is causing too many
> tangential problems.

 Undefined behavior is not a bug, nor is it prohibited by the C spec. 
 It is a necessary
 part of the language for its intended use as a system programming 
 language, writing
 machine-specific programs. Anyone who says it is prohibited by the 
 spec is wrong.
>>>
>>> I don't believe this is correct.
>>>
>>> Maybe you are thinking of implementation defined behavior?
>>
>> That would apply to how a particular compiler implementation treats some 
>> piece of code.
>> Whether a CPU supports unaligned access is machine-defined. Since our 
>> code is already
>> properly ifdef'd for the machines which do or don't support it, the fact 
>> that this is
>> "non-portable" is not an issue.
>
> The undefined behavior is causing OpenLDAP to fail testing.
>
> Worse, it is causing test failures in programs and libraries which use
> OpenLDAP. The OpenLDAP bugs have cross-pollinated into other programs
> and libraries. It is not simply contained or limited to OpenLDAP.
>
> That's a big problem for testing a QA folks.

 Then the tests are broken, because these are not bugs.
>>>
>>> They are absolutely OpenLDAP bugs. The unaligned accesses are
>>> Undefined Behavior.
>>
>> Simply because the C standard doesn't specify what the behavior is doesn't 
>> make it a bug.
>> It would be unreasonable to expect the standards body to exhaustively list 
>> all of the
>> possible behaviors on all of the possible hardware architectures that the 
>> language is
>> implemented for. Leaving it undefined in the spec only means they chose not 
>> to specify
>> a behavior. That means the actual behavior depends on the underlying 
>> machine. On the
>> x86 family the behavior is well defined.
> 
> Also note: Behaviors are left undefined when they are non-portable. But being 
> non-portable
> isn't illegal in C. The entire purpose of conditional compilation is to 
> support non-portable
> code.
> 
Worth reading 
https://www.yodaiken.com/2018/12/31/undefined-behavior-and-the-purpose-of-c/

-- 
  -- Howard Chu
  CTO, Symas Corp.   http://www.symas.com
  Director, Highland Sun http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/

Re: (ITS#8988) Undefined Behavior in slapadd

2019-06-07 Thread hyc 
h...@symas.com wrote:
> Jeffrey Walton wrote:
>> On Fri, Jun 7, 2019 at 10:08 AM Howard Chu  wrote:
>>>
>>> Jeffrey Walton wrote:
 On Fri, Jun 7, 2019 at 9:59 AM Howard Chu  wrote:
>
> noloa...@gmail.com wrote:
>> On Fri, Jun 7, 2019 at 9:32 AM Howard Chu  wrote:
>>>
>>> noloa...@gmail.com wrote:
>>> ...
 I encourage OpenLDAP to fix the undefined behavior. OpenLDAP is an
 important project, and the undefined behavior is causing too many
 tangential problems.
>>>
>>> Undefined behavior is not a bug, nor is it prohibited by the C spec. It 
>>> is a necessary
>>> part of the language for its intended use as a system programming 
>>> language, writing
>>> machine-specific programs. Anyone who says it is prohibited by the spec 
>>> is wrong.
>>
>> I don't believe this is correct.
>>
>> Maybe you are thinking of implementation defined behavior?
>
> That would apply to how a particular compiler implementation treats some 
> piece of code.
> Whether a CPU supports unaligned access is machine-defined. Since our 
> code is already
> properly ifdef'd for the machines which do or don't support it, the fact 
> that this is
> "non-portable" is not an issue.

 The undefined behavior is causing OpenLDAP to fail testing.

 Worse, it is causing test failures in programs and libraries which use
 OpenLDAP. The OpenLDAP bugs have cross-pollinated into other programs
 and libraries. It is not simply contained or limited to OpenLDAP.

 That's a big problem for testing a QA folks.
>>>
>>> Then the tests are broken, because these are not bugs.
>>
>> They are absolutely OpenLDAP bugs. The unaligned accesses are
>> Undefined Behavior.
> 
> Simply because the C standard doesn't specify what the behavior is doesn't 
> make it a bug.
> It would be unreasonable to expect the standards body to exhaustively list 
> all of the
> possible behaviors on all of the possible hardware architectures that the 
> language is
> implemented for. Leaving it undefined in the spec only means they chose not 
> to specify
> a behavior. That means the actual behavior depends on the underlying machine. 
> On the
> x86 family the behavior is well defined.

Also note: Behaviors are left undefined when they are non-portable. But being 
non-portable
isn't illegal in C. The entire purpose of conditional compilation is to support 
non-portable
code.

-- 
  -- Howard Chu
  CTO, Symas Corp.   http://www.symas.com
  Director, Highland Sun http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/

Re: (ITS#8988) Undefined Behavior in slapadd

2019-06-07 Thread hyc 
Jeffrey Walton wrote:
> On Fri, Jun 7, 2019 at 10:08 AM Howard Chu  wrote:
>>
>> Jeffrey Walton wrote:
>>> On Fri, Jun 7, 2019 at 9:59 AM Howard Chu  wrote:

 noloa...@gmail.com wrote:
> On Fri, Jun 7, 2019 at 9:32 AM Howard Chu  wrote:
>>
>> noloa...@gmail.com wrote:
>> ...
>>> I encourage OpenLDAP to fix the undefined behavior. OpenLDAP is an
>>> important project, and the undefined behavior is causing too many
>>> tangential problems.
>>
>> Undefined behavior is not a bug, nor is it prohibited by the C spec. It 
>> is a necessary
>> part of the language for its intended use as a system programming 
>> language, writing
>> machine-specific programs. Anyone who says it is prohibited by the spec 
>> is wrong.
>
> I don't believe this is correct.
>
> Maybe you are thinking of implementation defined behavior?

 That would apply to how a particular compiler implementation treats some 
 piece of code.
 Whether a CPU supports unaligned access is machine-defined. Since our code 
 is already
 properly ifdef'd for the machines which do or don't support it, the fact 
 that this is
 "non-portable" is not an issue.
>>>
>>> The undefined behavior is causing OpenLDAP to fail testing.
>>>
>>> Worse, it is causing test failures in programs and libraries which use
>>> OpenLDAP. The OpenLDAP bugs have cross-pollinated into other programs
>>> and libraries. It is not simply contained or limited to OpenLDAP.
>>>
>>> That's a big problem for testing a QA folks.
>>
>> Then the tests are broken, because these are not bugs.
> 
> They are absolutely OpenLDAP bugs. The unaligned accesses are
> Undefined Behavior.

Simply because the C standard doesn't specify what the behavior is doesn't make 
it a bug.
It would be unreasonable to expect the standards body to exhaustively list all 
of the
possible behaviors on all of the possible hardware architectures that the 
language is
implemented for. Leaving it undefined in the spec only means they chose not to 
specify
a behavior. That means the actual behavior depends on the underlying machine. 
On the
x86 family the behavior is well defined.

-- 
  -- Howard Chu
  CTO, Symas Corp.   http://www.symas.com
  Director, Highland Sun http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/

Re: (ITS#8988) Undefined Behavior in slapadd

2019-06-07 Thread noloader 
On Fri, Jun 7, 2019 at 10:08 AM Howard Chu  wrote:
>
> Jeffrey Walton wrote:
> > On Fri, Jun 7, 2019 at 9:59 AM Howard Chu  wrote:
> >>
> >> noloa...@gmail.com wrote:
> >>> On Fri, Jun 7, 2019 at 9:32 AM Howard Chu  wrote:
> 
>  noloa...@gmail.com wrote:
>  ...
> > I encourage OpenLDAP to fix the undefined behavior. OpenLDAP is an
> > important project, and the undefined behavior is causing too many
> > tangential problems.
> 
>  Undefined behavior is not a bug, nor is it prohibited by the C spec. It 
>  is a necessary
>  part of the language for its intended use as a system programming 
>  language, writing
>  machine-specific programs. Anyone who says it is prohibited by the spec 
>  is wrong.
> >>>
> >>> I don't believe this is correct.
> >>>
> >>> Maybe you are thinking of implementation defined behavior?
> >>
> >> That would apply to how a particular compiler implementation treats some 
> >> piece of code.
> >> Whether a CPU supports unaligned access is machine-defined. Since our code 
> >> is already
> >> properly ifdef'd for the machines which do or don't support it, the fact 
> >> that this is
> >> "non-portable" is not an issue.
> >
> > The undefined behavior is causing OpenLDAP to fail testing.
> >
> > Worse, it is causing test failures in programs and libraries which use
> > OpenLDAP. The OpenLDAP bugs have cross-pollinated into other programs
> > and libraries. It is not simply contained or limited to OpenLDAP.
> >
> > That's a big problem for testing a QA folks.
>
> Then the tests are broken, because these are not bugs.

They are absolutely OpenLDAP bugs. The unaligned accesses are
Undefined Behavior.

* https://stackoverflow.com/a/46790815/608639
* https://stackoverflow.com/a/46790813/608639
* https://stackoverflow.com/a/28895321/608639
* https://stackoverflow.com/a/51126453/608639
* https://stackoverflow.com/q/39908946/608639
...

I get that you think your project is fine. Many folks take that myopic
view of the world. I call it the greedy model.

Greedy models show up all over the place. For example, every website
that requires a different account and different 16 character password.
That's the greedy password model where a site think it is the only one
someone visits.

What baffles me is your insistence on polluting other program and
library results. That's not your place to screw up testing for other
programs and libraries.

Jeff

Re: (ITS#8988) Undefined Behavior in slapadd

2019-06-07 Thread hyc 
Jeffrey Walton wrote:
> On Fri, Jun 7, 2019 at 9:59 AM Howard Chu  wrote:
>>
>> noloa...@gmail.com wrote:
>>> On Fri, Jun 7, 2019 at 9:32 AM Howard Chu  wrote:

 noloa...@gmail.com wrote:
 ...
> I encourage OpenLDAP to fix the undefined behavior. OpenLDAP is an
> important project, and the undefined behavior is causing too many
> tangential problems.

 Undefined behavior is not a bug, nor is it prohibited by the C spec. It is 
 a necessary
 part of the language for its intended use as a system programming 
 language, writing
 machine-specific programs. Anyone who says it is prohibited by the spec is 
 wrong.
>>>
>>> I don't believe this is correct.
>>>
>>> Maybe you are thinking of implementation defined behavior?
>>
>> That would apply to how a particular compiler implementation treats some 
>> piece of code.
>> Whether a CPU supports unaligned access is machine-defined. Since our code 
>> is already
>> properly ifdef'd for the machines which do or don't support it, the fact 
>> that this is
>> "non-portable" is not an issue.
> 
> The undefined behavior is causing OpenLDAP to fail testing.
> 
> Worse, it is causing test failures in programs and libraries which use
> OpenLDAP. The OpenLDAP bugs have cross-pollinated into other programs
> and libraries. It is not simply contained or limited to OpenLDAP.
> 
> That's a big problem for testing a QA folks.

Then the tests are broken, because these are not bugs.

-- 
  -- Howard Chu
  CTO, Symas Corp.   http://www.symas.com
  Director, Highland Sun http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/

Re: (ITS#8988) Undefined Behavior in slapadd

2019-06-07 Thread noloader 
On Fri, Jun 7, 2019 at 9:59 AM Howard Chu  wrote:
>
> noloa...@gmail.com wrote:
> > On Fri, Jun 7, 2019 at 9:32 AM Howard Chu  wrote:
> >>
> >> noloa...@gmail.com wrote:
> >> ...
> >>> I encourage OpenLDAP to fix the undefined behavior. OpenLDAP is an
> >>> important project, and the undefined behavior is causing too many
> >>> tangential problems.
> >>
> >> Undefined behavior is not a bug, nor is it prohibited by the C spec. It is 
> >> a necessary
> >> part of the language for its intended use as a system programming 
> >> language, writing
> >> machine-specific programs. Anyone who says it is prohibited by the spec is 
> >> wrong.
> >
> > I don't believe this is correct.
> >
> > Maybe you are thinking of implementation defined behavior?
>
> That would apply to how a particular compiler implementation treats some 
> piece of code.
> Whether a CPU supports unaligned access is machine-defined. Since our code is 
> already
> properly ifdef'd for the machines which do or don't support it, the fact that 
> this is
> "non-portable" is not an issue.

The undefined behavior is causing OpenLDAP to fail testing.

Worse, it is causing test failures in programs and libraries which use
OpenLDAP. The OpenLDAP bugs have cross-pollinated into other programs
and libraries. It is not simply contained or limited to OpenLDAP.

That's a big problem for testing a QA folks.

Jeff

Re: (ITS#8988) Undefined Behavior in slapadd

2019-06-07 Thread hyc 
noloa...@gmail.com wrote:
> On Fri, Jun 7, 2019 at 9:32 AM Howard Chu  wrote:
>>
>> noloa...@gmail.com wrote:
>> ...
>>> I encourage OpenLDAP to fix the undefined behavior. OpenLDAP is an
>>> important project, and the undefined behavior is causing too many
>>> tangential problems.
>>
>> Undefined behavior is not a bug, nor is it prohibited by the C spec. It is a 
>> necessary
>> part of the language for its intended use as a system programming language, 
>> writing
>> machine-specific programs. Anyone who says it is prohibited by the spec is 
>> wrong.
> 
> I don't believe this is correct.
> 
> Maybe you are thinking of implementation defined behavior?

That would apply to how a particular compiler implementation treats some piece 
of code.
Whether a CPU supports unaligned access is machine-defined. Since our code is 
already
properly ifdef'd for the machines which do or don't support it, the fact that 
this is
"non-portable" is not an issue.

-- 
  -- Howard Chu
  CTO, Symas Corp.   http://www.symas.com
  Director, Highland Sun http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/

Re: (ITS#8988) Undefined Behavior in slapadd

2019-06-07 Thread noloader 
On Fri, Jun 7, 2019 at 9:32 AM Howard Chu  wrote:
>
> noloa...@gmail.com wrote:
> ...
> > I encourage OpenLDAP to fix the undefined behavior. OpenLDAP is an
> > important project, and the undefined behavior is causing too many
> > tangential problems.
>
> Undefined behavior is not a bug, nor is it prohibited by the C spec. It is a 
> necessary
> part of the language for its intended use as a system programming language, 
> writing
> machine-specific programs. Anyone who says it is prohibited by the spec is 
> wrong.

I don't believe this is correct.

Maybe you are thinking of implementation defined behavior?

Jeff

Re: (ITS#8988) Undefined Behavior in slapadd

2019-06-07 Thread hyc 
noloa...@gmail.com wrote:
> On Thu, Jun 6, 2019 at 7:39 PM Quanah Gibson-Mount  wrote:
>>
>> --On Thursday, March 07, 2019 10:10 AM + noloa...@gmail.com wrote:
>>
>>> Full_Name: JW
>>> Version: 2.4.47
>>> OS: Linux (Fedora 29, x86_64, fully patched)
>>> URL: ftp://ftp.openldap.org/incoming/
>>> Submission from: (NULL) (151.196.22.177)
>>>
>>> I added -fsanitize=undefined to CFLAGS.
>>
>> In this case, the compiler is the problem.
> 
> I'm pretty sure the compiler is correct.
> 
> The instrumentation does not produce false positives. It operates on
> real data and simply does not produce false positives.
> 
>> LMDB is 100% correct regarding
>> alignment on all supported CPUs (which is most of them).  The x86 family
>> fully supports unaligned access, so LMDB makes use of this.
> 
> It is not the processor; it is the C language. The C language does not allow 
> it.

That is incorrect. The C language allows it, but does not fully specify the 
behavior.
On some machines unaligned accesses are allowed, on others it is not. On X86 
family
it is allowed, and we use it.
> 
>> On other CPUs,
>> which do not support unaligned access (e.g., SPARC), LMDB doesn't use them.
> 
> The undefined behavior exists even after disabling the unaligned
> accesses with custom patches. For example, there is still undefined
> behavior even after changes like this:
> 
> -#if defined(__i386) || defined(__x86_64)
> -#define MISALIGNED_OK 1
> -#else
> #define ALIGNER (sizeof(size_t)-1)
> -#endif
> 
> We also fixed COPY_PGNO but the undefined behavior remained.
> 
> The GCC Compile Farm (https://cfarm.tetaneutral.net/machines/list/)
> has three SPARC machines. SPARC does not tolerate unaligned accesses
> well, especially due to the optimized 64-bit move instruction.

We have already tested on SPARC and as we already noted above, we do not use
unaligned accesses on SPARC.

> OpenLDAP might consider testing on the SPARC machines.
> 
> In the bigger picture, OpenLDAP is causing other program and library
> testing to fail testing. When we test other programs and libraries
> that depend upon OpenLDAP in a similar fashion, the undefined behavior
> in OpenLDAP surfaces in the other program and libraries. It is making
> a lot of extra work for folks who perform extra testing.
> 
> I encourage OpenLDAP to fix the undefined behavior. OpenLDAP is an
> important project, and the undefined behavior is causing too many
> tangential problems.

Undefined behavior is not a bug, nor is it prohibited by the C spec. It is a 
necessary
part of the language for its intended use as a system programming language, 
writing
machine-specific programs. Anyone who says it is prohibited by the spec is 
wrong.

> Jeff
> 
> 
> 
> 


-- 
  -- Howard Chu
  CTO, Symas Corp.   http://www.symas.com
  Director, Highland Sun http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/

Re: (ITS#8988) Undefined Behavior in slapadd

2019-06-06 Thread noloader 
On Thu, Jun 6, 2019 at 7:39 PM Quanah Gibson-Mount  wrote:
>
> --On Thursday, March 07, 2019 10:10 AM + noloa...@gmail.com wrote:
>
> > Full_Name: JW
> > Version: 2.4.47
> > OS: Linux (Fedora 29, x86_64, fully patched)
> > URL: ftp://ftp.openldap.org/incoming/
> > Submission from: (NULL) (151.196.22.177)
> >
> > I added -fsanitize=undefined to CFLAGS.
>
> In this case, the compiler is the problem.

I'm pretty sure the compiler is correct.

The instrumentation does not produce false positives. It operates on
real data and simply does not produce false positives.

> LMDB is 100% correct regarding
> alignment on all supported CPUs (which is most of them).  The x86 family
> fully supports unaligned access, so LMDB makes use of this.

It is not the processor; it is the C language. The C language does not allow it.

> On other CPUs,
> which do not support unaligned access (e.g., SPARC), LMDB doesn't use them.

The undefined behavior exists even after disabling the unaligned
accesses with custom patches. For example, there is still undefined
behavior even after changes like this:

-#if defined(__i386) || defined(__x86_64)
-#define MISALIGNED_OK 1
-#else
#define ALIGNER (sizeof(size_t)-1)
-#endif

We also fixed COPY_PGNO but the undefined behavior remained.

The GCC Compile Farm (https://cfarm.tetaneutral.net/machines/list/)
has three SPARC machines. SPARC does not tolerate unaligned accesses
well, especially due to the optimized 64-bit move instruction.
OpenLDAP might consider testing on the SPARC machines.

In the bigger picture, OpenLDAP is causing other program and library
testing to fail testing. When we test other programs and libraries
that depend upon OpenLDAP in a similar fashion, the undefined behavior
in OpenLDAP surfaces in the other program and libraries. It is making
a lot of extra work for folks who perform extra testing.

I encourage OpenLDAP to fix the undefined behavior. OpenLDAP is an
important project, and the undefined behavior is causing too many
tangential problems.

Jeff

Re: (ITS#8988) Undefined Behavior in slapadd

2019-06-06 Thread quanah 
--On Thursday, March 07, 2019 10:10 AM + noloa...@gmail.com wrote:

> Full_Name: JW
> Version: 2.4.47
> OS: Linux (Fedora 29, x86_64, fully patched)
> URL: ftp://ftp.openldap.org/incoming/
> Submission from: (NULL) (151.196.22.177)
>
>
> I added -fsanitize=undefined to CFLAGS.

Hello,

In this case, the compiler is the problem.  LMDB is 100% correct regarding 
alignment on all supported CPUs (which is most of them).  The x86 family 
fully supports unaligned access, so LMDB makes use of this.  On other CPUs, 
which do not support unaligned access (e.g., SPARC), LMDB doesn't use them.

This ITS will be closed.

--Quanah



--

Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:

(ITS#8988) Undefined Behavior in slapadd

2019-03-07 Thread noloader 
Full_Name: JW
Version: 2.4.47
OS: Linux (Fedora 29, x86_64, fully patched)
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (151.196.22.177)


I added -fsanitize=undefined to CFLAGS.

> Starting test001-slapadd for mdb...
running defines.sh
Running slapadd to build slapd database...
../../../libraries/liblmdb/mdb.c:7544:26: runtime error: member access within
misaligned address 0x023fe67a for type 'struct MDB_page', which requires 8
byte alignment
0x023fe67a: note: pointer points here
 00 00  00 00 03 00 00 00 00 00  00 00 00 00 52 00 10 00  66 00 00 00 00 00 00
00  00 00 00 00 00 00
  ^
../../../libraries/liblmdb/mdb.c:7545:3: runtime error: member access within
misaligned address 0x023fe67a for type 'struct MDB_page', which requires 8
byte alignment
0x023fe67a: note: pointer points here
 00 00  00 00 03 00 00 00 00 00  00 00 00 00 52 00 10 00  66 00 00 00 00 00 00
00  00 00 00 00 00 00
  ^
../../../libraries/liblmdb/mdb.c:6046:8: runtime error: member access within
misaligned address 0x023fe67a for type 'struct MDB_page', which requires 8
byte alignment
0x023fe67a: note: pointer points here
 00 00  00 00 03 00 00 00 00 00  00 00 00 00 52 00 10 00  66 00 00 00 00 00 00
00  00 00 00 00 00 00
  ^
../../../libraries/liblmdb/mdb.c:2418:7: runtime error: member access within
misaligned address 0x023fe67a for type 'struct MDB_page', which requires 8
byte alignment
0x023fe67a: note: pointer points here
 00 00  00 00 03 00 00 00 00 00  00 00 00 00 52 00 10 00  66 00 00 00 00 00 00
00  00 00 00 00 00 00
  ^
../../../libraries/liblmdb/mdb.c:6938:10: runtime error: member access within
misaligned address 0x023fe67a for type 'struct MDB_page', which requires 8
byte alignment
0x023fe67a: note: pointer points here
 00 00  00 00 03 00 00 00 00 00  00 00 00 00 52 00 10 00  66 00 00 00 00 00 00
00  00 00 00 00 00 00
  ^
../../../libraries/liblmdb/mdb.c:6939:6: runtime error: member access within
misaligned address 0x023fe67a for type 'struct MDB_page', which requires 8
byte alignment
0x023fe67a: note: pointer points here
 00 00  00 00 03 00 00 00 00 00  00 00 00 00 52 00 10 00  66 00 00 00 00 00 00
00  00 00 00 00 00 00
  ^
../../../libraries/liblmdb/mdb.c:6939:6: runtime error: member access within
misaligned address 0x023fe67a for type 'struct MDB_page', which requires 8
byte alignment
0x023fe67a: note: pointer points here
 00 00  00 00 03 00 00 00 00 00  00 00 00 00 52 00 10 00  66 00 00 00 00 00 00
00  00 00 00 00 00 00
  ^
../../../libraries/liblmdb/mdb.c:7287:6: runtime error: member access within
misaligned address 0x023fe67a for type 'struct MDB_page', which requires 8
byte alignment
0x023fe67a: note: pointer points here
 00 00  00 00 03 00 00 00 00 00  00 00 00 00 52 00 10 00  66 00 00 00 00 00 00
00  00 00 00 00 00 00
  ^
../../../libraries/liblmdb/mdb.c:7303:18: runtime error: member access within
misaligned address 0x023fe67a for type 'struct MDB_page', which requires 8
byte alignment
0x023fe67a: note: pointer points here
 00 00  00 00 03 00 00 00 00 00  00 00 00 00 52 00 10 00  66 00 00 00 00 00 00
00  00 00 00 00 00 00
  ^
../../../libraries/liblmdb/mdb.c:7303:18: runtime error: member access within
misaligned address 0x023fe67a for type 'struct MDB_page', which requires 8
byte alignment
0x023fe67a: note: pointer points here
 00 00  00 00 03 00 00 00 00 00  00 00 00 00 52 00 10 00  66 00 00 00 00 00 00
00  00 00 00 00 00 00
  ^
../../../libraries/liblmdb/mdb.c:7306:6: runtime error: member access within
misaligned address 0x023fe67a for type 'struct MDB_page', which requires 8
byte alignment
0x023fe67a: note: pointer points here
 00 00  00 00 03 00 00 00 00 00  00 00 00 00 52 00 10 00  66 00 00 00 00 00 00
00  00 00 00 00 00 00
  ^
../../../libraries/liblmdb/mdb.c:7335:11: runtime error: member access within
misaligned address 0x023fe67a for type 'struct MDB_page', which requires 8
byte alignment
0x023fe67a: note: pointer points here
 00 00  00 00 03 00 00 00 00 00  00 00 00 00 52 00 10 00  66 00 00 00 00 00 00
00  00 00 00 00 00 00
  ^
../../../libraries/liblmdb/mdb.c:7339:12: runtime error: member access within
misaligned address 0x023fe67a for type 'struct MDB_page', which requires 8
byte alignment
0x023fe67a: note: pointer points here
 00 00  00 00 03 00 00 00 00 00  00 00 00 00 52 00 10 00  66 00 00 00 00 00 00
00  00 00 00 00 00 00
  ^
../../../libraries/liblmdb/mdb.c:7341:20: runtime error: member access within
misaligned address 0x023fe67a for type 'struct MDB_page', which requires 8
byte alignment
0x023fe67a: note: pointer points here
 00 00  00 00 03 00 00 00 00 00  00 00 00 00 52 00 10 00  66 00 00 00 00 00 00
00  00 00 00 00 00 00
  ^
../../../libraries/liblmdb/mdb.c:7342:15: runtime error: member access