[sqlite] Article about pointer abuse in SQLite

[John McKown]

On Thu, Mar 24, 2016 at 5:59 PM, Keith Medcalf  wrote:

> On Thursday, 24 March, 2016 08:27, Tim Streater 
> said,
>
> > On 24 Mar 2016 at 13:41, Jim Callahan 
> > wrote:
>
> > > Assuming one did not get a cryptic compiler message (and in those days
> > all compiler messages were cryptic, "Probable user error:") ...
>
> > Not always: If you used JAIL as a variable in a computed GOTO, the XDS
> > FORTRAN compiler for the Sigma 7 would, in response to the statement:
>
> >GOTO JAIL
>
> > give you the message:
>
> > *** Warning: Go directly; do not pass GO; do not collect $200
>
> Ah the memories ...
>

?Me too. Along with one that I got that really upset me:

ABOVE BUL

That was the _entire_ message! Turns out BUL is "Background Upper Limit".
I.e. I had too big of an array, so the program wouldn't fit into? memory.
And, the ever popular:

END OF FILE ON UNIT 05 AT RUN TIME


-- 
How many surrealists does it take to screw in a lightbulb? One to hold the
giraffe and one to fill the bathtub with brightly colored power tools.

Maranatha! <><
John McKown

[sqlite] Article about pointer abuse in SQLite

[Keith Medcalf]

On Thursday, 24 March, 2016 08:27, Tim Streater  said,

> On 24 Mar 2016 at 13:41, Jim Callahan 
> wrote:

> > Assuming one did not get a cryptic compiler message (and in those days
> all compiler messages were cryptic, "Probable user error:") ...

> Not always: If you used JAIL as a variable in a computed GOTO, the XDS
> FORTRAN compiler for the Sigma 7 would, in response to the statement:

>GOTO JAIL

> give you the message:

> *** Warning: Go directly; do not pass GO; do not collect $200

Ah the memories ...

[sqlite] Article about pointer abuse in SQLite

[Simon Slavin]

On 24 Mar 2016, at 2:49pm, Richard Hipp  wrote:

> On 3/24/16, Simon Slavin  wrote:
>> 
>> I want the next generation of compilers to require
>> that the programmer specify an initial value (a constant, not a calculated
>> value) for every variable they define, including every array element when
>> they define an array.
> 
> At https://www.sqlite.org/src/info/03b2a622?ln=3553-3560 is just one
> example where unnecessary local variable initialization has a large
> performance impact.  There are *many* other such examples in SQLite.
> This one just has the best comment.

Yup.  The compiler has to decide whether to compile the statement (presumably 
it uses path analysis).  But it should be in the source code to force the 
programmer to worry about the problem.  Purely my opinion.

Simon.

[sqlite] Article about pointer abuse in SQLite

[Simon Slavin]

I've been leaving this thread alone because I've been very interested in 
various people's contributions.  But I'm a security professional so I know 
stuff most programmers don't know, and I've seen enough wrong information here 
that I'm concerned someone will use it to design something badly.  So please 
excuse me.  I'm not pouncing on any one person's post, since no one post has 
been worth pouncing on.  I'm reacting to the thread in general.



Useful and plausible initial variable values lead to situations where a 
programmer/debugger may think things are working when they're not.  Clutter is 
good.  And intentional clutter is better than random clutter.

Zero is the worst value to have as a default because it is useful and plausible 
and will trigger no errors.  Zero is a useful initial value in many situations. 
 It can be used as a pointer without errors (it's small and always on a word 
boundary no matter what word size you're using).  It can be used as a memory 
offset without errors.  It is a low number and thus can be included in 
calculations without being obvious.

Common values (for over 30 years) for initialisation are numbers like 
0xDEADBEEF.  This value has these advantages:

a) It is memorable and easy to recognise so if any programmer/debugger sees it 
they know they have a bug somewhere.
b) It is odd, meaning that any attempt to use it as a pointer for CPUs which 
demand that pointers be aligned to words will immediately trigger 'segment 
fault' (or equivalent) error.
c) Not just the whole value but every byte has the top bit set, which means 
that however you use it as a pointer you will get an 'illegal memory access' 
(or equivalent) error because you're using more memory than was assigned in 
that block.
d) It has a very high value, meaning that if it's used as a memory offset 
you'll get n 'illegal memory access' (or equivalent) error because you're using 
more memory than was assigned in that block.
e) It has a very high value, meaning that if it is included in any calculation 
the result will be obviously wrong.

It is bad for initialisation to be done by the compiler when it allocates an 
App new memory.  You could write your own compiler which did not do this, and 
in that way spy on other app's memory.  In operating systems designed for 
security it is done by the operating system /when the previous application 
releases the memory, before that memory is returned to the free pool/.  This 
means that even if someone finds a way to trick the OS into reading memory 
which hadn't been allocated to an App yet, its contents would already have been 
overwritten.

Of course, many of us are still using operating systems written under the 
assumption that one App must trust another.  This got better for a while as 
people who worried about desktop computers started worrying about rogue Apps.  
But for the last decade it has got worse because so many of our computing 
devices are tiny and need to operated on battery power (phones, watches, GPS 
devices) and don't have enough free CPU to wipe memory every time it's 
released.  So, annoyingly, the device which knows the most personal information 
about you, your mobile phone, is probably the one with the worst security.

In closing, I hope that the construction used in C and other languages

int i, loopCounter;
float interestRate, currentDebt;

is going to go away.  I want the next generation of compilers to require that 
the programmer specify an initial value (a constant, not a calculated value) 
for every variable they define, including every array element when they define 
an array.

Simon.

[sqlite] Article about pointer abuse in SQLite

[Tim Streater]

On 24 Mar 2016 at 13:41, Jim Callahan  wrote:

> Assuming one did not get a cryptic compiler message (and in those days all
> compiler messages were cryptic, "Probable user error:") ...

Not always: If you used JAIL as a variable in a computed GOTO, the XDS FORTRAN 
compiler for the Sigma 7 would, in response to the statement:

   GOTO JAIL

give you the message:

*** Warning: Go directly; do not pass GO; do not collect $200


--
Cheers  --  Tim

[sqlite] Article about pointer abuse in SQLite

[Richard Hipp]

On 3/24/16, Marc L. Allen  wrote:
> With everything that routine does, the extra initialization negatively
> impacts operation?
>

It did when I put in that comment in 2009.  At that point in time, the
routine was very hot.  Subsequent optimizations have enabled us to
completely bypass that routine.  The code in question now only runs as
a verification step when compiled with -DSQLITE_DEBUG.  So these days
the initialization does not matter.

But the point is the same.  Sometimes an unnecessary initialization in
a hot routine can make a big difference in performance.

-- 
D. Richard Hipp
drh at sqlite.org

[sqlite] Article about pointer abuse in SQLite

[Richard Hipp]

On 3/24/16, Simon Slavin  wrote:
>
> I want the next generation of compilers to require
> that the programmer specify an initial value (a constant, not a calculated
> value) for every variable they define, including every array element when
> they define an array.

At https://www.sqlite.org/src/info/03b2a622?ln=3553-3560 is just one
example where unnecessary local variable initialization has a large
performance impact.  There are *many* other such examples in SQLite.
This one just has the best comment.
-- 
D. Richard Hipp
drh at sqlite.org

[sqlite] Article about pointer abuse in SQLite

[Igor Tandetnik]

On 3/23/2016 1:23 PM, Doug Nebeker wrote:
>> For obvious security reasons all allocations from the Operating System are 
>> pre-initialized to 0x00.
>
> Time to bash Windows, but according to the docs for HeapAlloc, memory is not 
> automatically initialized to 0
>
> https://msdn.microsoft.com/en-us/library/windows/desktop/aa366597(v=vs.85).aspx
>
> This fits with my experience as well.

HeapAlloc doesn't, but VirtualAlloc does. VirtualAlloc is how you obtain 
pages of memory from the kernel (and those come zero-initialized); 
HeapAlloc is a user-mode heap manager further sub-allocating those 
pages. The garbage you see in uninitialized heap allocations came from 
your own process; you don't get to observe random data from other 
processes this way.
-- 
Igor Tandetnik

[sqlite] Article about pointer abuse in SQLite

[Marc L. Allen]

With everything that routine does, the extra initialization negatively impacts 
operation?

-Original Message-
From: sqlite-users-bounces at mailinglists.sqlite.org 
[mailto:sqlite-users-boun...@mailinglists.sqlite.org] On Behalf Of Richard Hipp
Sent: Thursday, March 24, 2016 10:50 AM
To: SQLite mailing list 
Subject: Re: [sqlite] Article about pointer abuse in SQLite

On 3/24/16, Simon Slavin  wrote:
>
> I want the next generation of compilers to require that the programmer 
> specify an initial value (a constant, not a calculated
> value) for every variable they define, including every array element 
> when they define an array.

At https://www.sqlite.org/src/info/03b2a622?ln=3553-3560 is just one example 
where unnecessary local variable initialization has a large performance impact. 
 There are *many* other such examples in SQLite.
This one just has the best comment.
--
D. Richard Hipp
drh at sqlite.org
___
sqlite-users mailing list
sqlite-users at mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users


This email and any attachments are only for use by the intended recipient(s) 
and may contain legally privileged, confidential, proprietary or otherwise 
private information. Any unauthorized use, reproduction, dissemination, 
distribution or other disclosure of the contents of this e-mail or its 
attachments is strictly prohibited. If you have received this email in error, 
please notify the sender immediately and delete the original.

[sqlite] Article about pointer abuse in SQLite

[Scott Robison]

Forgive top posting, on my phone. But two points.

1. There is a flag if you want zero initialized memory from HeapAlloc.

2. These types of heaps are private to the process and must be created
first. The heap creation process reserves virtual address space and
allocates memory on demand from a lower OS layer which does zero initialize
on first allocation.

So while it is true that HeapAlloc can return non zero initialized memory,
it is not a cross process security violation because HeapCreate's
implementation does not allow that.

Freeing a heap allocation does not return that memory to the OS. It's still
reserved to your process. If later reallocated, it will only contain
whatever your app wrote to it. Unless you destroy the heap, which should
return the memory and address space to the OS.

There are a lot of different memory APIs with different characteristics.
On Mar 24, 2016 6:01 AM, "Doug Nebeker"  wrote:

> > For obvious security reasons all allocations from the Operating System
> are pre-initialized to 0x00.
>
> Time to bash Windows, but according to the docs for HeapAlloc, memory is
> not automatically initialized to 0
>
>
> https://msdn.microsoft.com/en-us/library/windows/desktop/aa366597(v=vs.85).aspx
>
> This fits with my experience as well.
>
> Doug
>
>
>
> -Original Message-
> From: sqlite-users-bounces at mailinglists.sqlite.org [mailto:
> sqlite-users-bounces at mailinglists.sqlite.org] On Behalf Of Keith Medcalf
> Sent: Tuesday, March 22, 2016 8:41 PM
> To: SQLite mailing list
> Subject: Re: [sqlite] Article about pointer abuse in SQLite
>
>
> > This discussion on the nature of undefined behaviour code is
> > interesting.  I don't know the reasoning, but it seems that VS6 often
> > initialized things to 0xcd in debug mode and (usually) had memory
> > uninitialized to 0x00 when complied in Release (perhaps 0x00 just
> > happens to be what was on the stack or heap).  I presume this wasn't
> > just to make people suffer  when things don't work the same in debug
> > vs release mode.
>
> The initialization of memory to non-0x00 is a compiler function.
>
> For obvious security reasons all allocations from the Operating System are
> pre-initialized to 0x00.  This is so that your program cannot request a big
> hunk of virtual memory which is full of a predecessor process data and then
> proceed to search it for nifty things like previously used private keys,
> userids, passwords, and so forth.  Such behaviour is required for any
> Operating Systems to obtain any security certification level whatsoever.
>
>
>
>
> ___
> sqlite-users mailing list
> sqlite-users at mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
> ___
> sqlite-users mailing list
> sqlite-users at mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>

[sqlite] Article about pointer abuse in SQLite

[Jim Callahan]

I have painful memories from programming 101 in the twilight of the punch
card era
that initializing variables was a big deal in FORTRAN and COBOL. After
declaring a long list of variables for a convoluted assignment one would
have to go back and set equal to zero in FORTRAN and "MOVE SPACES TO" in
COBOL. There was no "cut and paste" this was the punch card era (we also
had to walk seven miles uphill barefoot in the snow to school -- joke).
Assuming one did not get a cryptic compiler message (and in those days all
compiler messages were cryptic, "Probable user error:") one would get a
bizarre run time error at a time when one did not clearly understand the
difference between "compile time" and "run time" (the batch job was
initialized with cryptic JCL cards and returned one long printout on green
bar paper so it took careful reading to understand what the computer was
doing).  The UNIX shells (or even the MS-DOS command line) were such an
improvement over IBM mainframe JCL hell.

In a shared mainframe or minicomputer environment there was constant
clutter in memory. This was long before Internet era security concerns.

Wikipedia has an article on "Uninitialized Variable" with a C example and
references
a C standard:
"ISO/IEC 9899:TC3 (Current C standard)"
<http://www.open-std.org/JTC1/SC22/WG14/www/docs/n1256.pdf> (PDF).
2007-09-07. p. 126. Retrieved 2008-09-26. Section 6.7.8, paragraph 10.

If uninitialized variables are allowed and memory is not automatically set
to zero the program will have non-deterministic run-time behavior because
of the random clutter in memory.

Jim Callahan







<https://www.avast.com/sig-email?utm_medium=email_source=link_campaign=sig-email_content=webmail_term=oa-2115-v2-a>
This
email has been sent from a virus-free computer protected by Avast.
www.avast.com
<https://www.avast.com/sig-email?utm_medium=email_source=link_campaign=sig-email_content=webmail_term=oa-2115-v2-a>
<#DDB4FAA8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

On Wed, Mar 23, 2016 at 1:23 PM, Doug Nebeker  wrote:

> > For obvious security reasons all allocations from the Operating System
> are pre-initialized to 0x00.
>
> Time to bash Windows, but according to the docs for HeapAlloc, memory is
> not automatically initialized to 0
>
>
> https://msdn.microsoft.com/en-us/library/windows/desktop/aa366597(v=vs.85).aspx
>
> This fits with my experience as well.
>
> Doug
>
>
>
> -Original Message-
> From: sqlite-users-bounces at mailinglists.sqlite.org [mailto:
> sqlite-users-bounces at mailinglists.sqlite.org] On Behalf Of Keith Medcalf
> Sent: Tuesday, March 22, 2016 8:41 PM
> To: SQLite mailing list
> Subject: Re: [sqlite] Article about pointer abuse in SQLite
>
>
> > This discussion on the nature of undefined behaviour code is
> > interesting.  I don't know the reasoning, but it seems that VS6 often
> > initialized things to 0xcd in debug mode and (usually) had memory
> > uninitialized to 0x00 when complied in Release (perhaps 0x00 just
> > happens to be what was on the stack or heap).  I presume this wasn't
> > just to make people suffer  when things don't work the same in debug
> > vs release mode.
>
> The initialization of memory to non-0x00 is a compiler function.
>
> For obvious security reasons all allocations from the Operating System are
> pre-initialized to 0x00.  This is so that your program cannot request a big
> hunk of virtual memory which is full of a predecessor process data and then
> proceed to search it for nifty things like previously used private keys,
> userids, passwords, and so forth.  Such behaviour is required for any
> Operating Systems to obtain any security certification level whatsoever.
>
>
>
>
> ___
> sqlite-users mailing list
> sqlite-users at mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
> ___
> sqlite-users mailing list
> sqlite-users at mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>

[sqlite] Article about pointer abuse in SQLite

[Doug Nebeker]

> For obvious security reasons all allocations from the Operating System are 
> pre-initialized to 0x00.  

Time to bash Windows, but according to the docs for HeapAlloc, memory is not 
automatically initialized to 0

https://msdn.microsoft.com/en-us/library/windows/desktop/aa366597(v=vs.85).aspx

This fits with my experience as well.

Doug



-Original Message-
From: sqlite-users-bounces at mailinglists.sqlite.org 
[mailto:sqlite-users-boun...@mailinglists.sqlite.org] On Behalf Of Keith Medcalf
Sent: Tuesday, March 22, 2016 8:41 PM
To: SQLite mailing list
Subject: Re: [sqlite] Article about pointer abuse in SQLite


> This discussion on the nature of undefined behaviour code is 
> interesting.  I don't know the reasoning, but it seems that VS6 often 
> initialized things to 0xcd in debug mode and (usually) had memory 
> uninitialized to 0x00 when complied in Release (perhaps 0x00 just 
> happens to be what was on the stack or heap).  I presume this wasn't 
> just to make people suffer  when things don't work the same in debug 
> vs release mode.

The initialization of memory to non-0x00 is a compiler function.

For obvious security reasons all allocations from the Operating System are 
pre-initialized to 0x00.  This is so that your program cannot request a big 
hunk of virtual memory which is full of a predecessor process data and then 
proceed to search it for nifty things like previously used private keys, 
userids, passwords, and so forth.  Such behaviour is required for any Operating 
Systems to obtain any security certification level whatsoever. 




___
sqlite-users mailing list
sqlite-users at mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] Article about pointer abuse in SQLite

[Donald Shepherd]

On Wed, 23 Mar 2016 12:59 am Adam Devita  wrote:

>
> This discussion on the nature of undefined behaviour code is
> interesting.  I don't know the reasoning, but it seems that VS6 often
> initialized things to 0xcd in debug mode and (usually) had memory
> uninitialized to 0x00 when complied in Release (perhaps 0x00 just
> happens to be what was on the stack or heap).  I presume this wasn't
> just to make people suffer  when things don't work the same in debug
> vs release mode.
>

It's not uncommon for compilers to initialise variables to definitely bad
values in debug mode to help find these kinds of bugs.

However if you were getting 00s in VC++ you were getting lucky and would
probably continue to get lucky until an unexpected stack allocation changed
the usual location, after which all bets are off (speaking from VC++
experience and VC++6 specifically).

>

[sqlite] Article about pointer abuse in SQLite

[James K. Lowden]

On Tue, 22 Mar 2016 11:00:24 -0500
"Marc L. Allen"  wrote:

> I don't think compilers "run" your code.  

Provided we're talking about a C compiler, you're right.  Optimizers
don't run the code, they reason about it.  

> The fact that the code never actually allows that path to occur is
> beyond the scope of most compilers, isn't it?

Yes and no.  If the compiler can prove a particular branch can never be
taken, it can remove it because the logic of the program will not be
affected.  If it cannot prove that, the code will remain.  For example,
given

int foo = 0;
if (foo)
exit(0);

the compiler can delete lines 2 & 3.  If there's no other reference to
foo, it can delete line 1, too.  However, 

extern int foo;
if (foo)
exit(0);
and
int foo = 0;
extern int *pfoo;
pfoo = 
if (foo)
exit(0);

both leave most optimzers flat-footed.  The potential for another
module to affect the value of foo means the code could run, and thus
must remain.  

--jkl

[sqlite] Article about pointer abuse in SQLite

[James K. Lowden]

On Tue, 22 Mar 2016 09:58:52 -0400
Adam Devita  wrote:

> I don't know the reasoning, but it seems that VS6 often
> initialized things to 0xcd in debug mode and (usually) had memory
> uninitialized to 0x00 when complied in Release (perhaps 0x00 just
> happens to be what was on the stack or heap).  

I would be talking out of school here if you're talking about C#.  For
C and C++, the 0xcd initialization helps make (mis)use of uninitalized
objects more obvious.  If the allocated buffer happens to be
zero-initialized, things like printf will make them appear empty when
they're actually invalid.  

This link has a nice discussion: 


http://stackoverflow.com/questions/2769247/controling-crt-memory-initialization

and includes a link to the documented behavior: 

https://msdn.microsoft.com/en-us/library/Aa270812

--jkl

[sqlite] Article about pointer abuse in SQLite

[James K. Lowden]

On Tue, 22 Mar 2016 09:56:57 +0100
"Cezary H. Noweta"  wrote:

> On 2016-03-22 00:35, James K. Lowden wrote:
> >[...]  An example from Clang's discussion is
> >
> > int i = 10 << 31;
> 
> Could you provide a link for that discussion? (Or google's phrase to 
> retrieve such link?)

I'm sorry, no.  Not for the first time I wish my browser had a feature
like "find links in history with documents matching regex".  

I didn't read it on the Clang mailing list.  I think I saw it by
reference in Regehr's discussion of "friendly C".  It specifically
mentioned  10 << 31  as an example of an "integer" requiring 35 bits,
something gcc assigns silently and clang diagnoses with a warning.  

If you haven't seen it, 

http://blog.regehr.org/archives/1180

is a good starting point.  It mentions "Towards Optimization-Safe
Systems: Analyzing the Impact of Undefined Behavior" 
(http://pdos.csail.mit.edu/papers/stack:sosp13.pdf), which is where I
learned that sharp-edged optimization is not a brand-new phenomenon.  

DJB provides a properly justified grumpy, frustrated view, 


https://groups.google.com/forum/m/#!msg/boring-crypto/48qa1kWignU/o8GGp2K1DAAJ

wherein he mentions one of the defenses for the status quo, 

"that a boring C compiler can't possibly support the desired 
 system _performance_. Even if this were true (which I very much 
 doubt), why would it be more important than system _correctness_?"

That should be the only argument needed.  DJB is concerned about
security.  DRH is concerned about correctness.  The serious C
programmer doesn't breath who prizes performance over correctness, yet
that is the license the compiler writers have granted themselves.  

--jkl

[sqlite] Article about pointer abuse in SQLite

[Keith Medcalf]

> This discussion on the nature of undefined behaviour code is
> interesting.  I don't know the reasoning, but it seems that VS6 often
> initialized things to 0xcd in debug mode and (usually) had memory
> uninitialized to 0x00 when complied in Release (perhaps 0x00 just
> happens to be what was on the stack or heap).  I presume this wasn't
> just to make people suffer  when things don't work the same in debug
> vs release mode.

The initialization of memory to non-0x00 is a compiler function.

For obvious security reasons all allocations from the Operating System are 
pre-initialized to 0x00.  This is so that your program cannot request a big 
hunk of virtual memory which is full of a predecessor process data and then 
proceed to search it for nifty things like previously used private keys, 
userids, passwords, and so forth.  Such behaviour is required for any Operating 
Systems to obtain any security certification level whatsoever. 

[sqlite] Article about pointer abuse in SQLite

[Marc L. Allen]

I don't think compilers "run" your code.  When looking for uninitialized 
variables, it simply looks for a potential path through the code that uses a 
variable without it being initialized.

The fact that the code never actually allows that path to occur is beyond the 
scope of most compilers, isn't it?

-Original Message-
From: sqlite-users-bounces at mailinglists.sqlite.org 
[mailto:sqlite-users-boun...@mailinglists.sqlite.org] On Behalf Of J Decker
Sent: Tuesday, March 22, 2016 11:43 AM
To: SQLite mailing list 
Subject: Re: [sqlite] Article about pointer abuse in SQLite

On Tue, Mar 22, 2016 at 6:58 AM, Adam Devita  wrote:
> It may be pedantic, but VS2016 will stop complaining if you edit your 
> definition of s to large_struct s=new large_struct();  //set s to an 
> actual instance of large_struct. c people can think of s as a pointer, 
> and in c# the members are set to their default values.
>
The point was, the structure had some 20 members, and 90% of the time the 
conditions don't exist for it to be initialized.  So rather than initialize it 
90% of the time for no use, I added checks to optimize the object's creation.

> J Decker's point could also have been made by using int x in place of 
> large_struct s . and sub x for s.x  , since it is a contrived example 
> anyway.  The only way to use x is if another conditional on another 
> variable that follows it in code and it is initialized.
>
> if one writes
> const bool arbitrary_true_false = true;   //note the const as Scott
> Doctor implied, makes the error go away.
>

It's not a const though, it's a variable the changes during runtime and allows 
for the creation of such an object. It's not 'contrived' it was an example that 
I ran into while developing (several times in fact).

similarly soemthing like (I haven't run it though compilers, so don't know if 
this is nested enough to cause the same issue... but it's easy to see how a 
compiler/error checker would similarly be confused.

void f() { int a, b;
   for( a = 0; a < 2; a++ ) {
   if(  a == 0 ) b = 1234;
   }
   printf( "b is never uniniialized here : %d", b ); }

> -
> This discussion on the nature of undefined behaviour code is 
> interesting.  I don't know the reasoning, but it seems that VS6 often 
> initialized things to 0xcd in debug mode and (usually) had memory 
> uninitialized to 0x00 when complied in Release (perhaps 0x00 just 
> happens to be what was on the stack or heap).  I presume this wasn't 
> just to make people suffer  when things don't work the same in debug 
> vs release mode.
>
> Does the tool help (in the sqlite in practice) point out things that 
> could be problematic?  Is it a compiler's variant of  "hay,  you are 
> depending on implemented, not documented behaviour" ?
>
> regards,
> Adam DeVita
>
>
> On Tue, Mar 22, 2016 at 7:27 AM, Scott Doctor  
> wrote:
>>
>> It is uninitialized. you are setting an initial value within an if 
>> statement. For the compiler, the code has NOT actually executed. so 
>> it does not use the value of the variable arbitrary_true_false. If it 
>> was a #define then it would use the value but still give an error 
>> because it is not a compiler directive #if but a code if.
>>
>> The logic is that the first instance of assignment is within a conditional.
>> That is a particularly nasty kind of bug and should be reported as an error.
>> because if later you decide to change arbitrary_true_false to false, 
>> then s.x would not be initialized before use. the compiler is correct 
>> to issue the warning. Give s.x a value after/at initialization, but 
>> before the if statement to give it a desired initial value then 
>> recompile, that should fix the error.
>>
>> Compilers only set the code to initialize the variable at 
>> declaration, not actually use the values during compile. If it was 
>> declared as a constant using a compiler directive such as #define, 
>> then the compiler would use the value in the logic and still give an 
>> error, but a different one because the conditional would always 
>> evaluate true (or false depending on what it was set to)
>>
>>
>> On 03/21/2016 21:31, J Decker wrote:
>>>
>>> On Mon, Mar 21, 2016 at 8:40 PM, Scott Doctor 
>>> 
>>> wrote:
>>>>
>>>> you are missing
>>>>
>>>> using System;
>>>
>>> whatever.  It still fails because it says the variable is 
>>> uninitilalized.  THe only thing that doesn't is actually running it.
>>>
>>> That same pattern not matter what the language triggers 
>>> warning/error checkers
>>>>
>>>> 

[sqlite] Article about pointer abuse in SQLite

[Adam Devita]

It may be pedantic, but VS2016 will stop complaining if you edit your
definition of s to
large_struct s=new large_struct();  //set s to an actual instance of
large_struct. c people can think of s as a pointer, and in c# the
members are set to their default values.

J Decker's point could also have been made by using int x in place of
large_struct s . and sub x for s.x  , since it is a contrived example
anyway.  The only way to use x is if another conditional on another
variable that follows it in code and it is initialized.

if one writes
const bool arbitrary_true_false = true;   //note the const as Scott
Doctor implied, makes the error go away.

-
This discussion on the nature of undefined behaviour code is
interesting.  I don't know the reasoning, but it seems that VS6 often
initialized things to 0xcd in debug mode and (usually) had memory
uninitialized to 0x00 when complied in Release (perhaps 0x00 just
happens to be what was on the stack or heap).  I presume this wasn't
just to make people suffer  when things don't work the same in debug
vs release mode.

Does the tool help (in the sqlite in practice) point out things that
could be problematic?  Is it a compiler's variant of  "hay,  you are
depending on implemented, not documented behaviour" ?

regards,
Adam DeVita


On Tue, Mar 22, 2016 at 7:27 AM, Scott Doctor  wrote:
>
> It is uninitialized. you are setting an initial value within an if
> statement. For the compiler, the code has NOT actually executed. so it does
> not use the value of the variable arbitrary_true_false. If it was a #define
> then it would use the value but still give an error because it is not a
> compiler directive #if but a code if.
>
> The logic is that the first instance of assignment is within a conditional.
> That is a particularly nasty kind of bug and should be reported as an error.
> because if later you decide to change arbitrary_true_false to false, then
> s.x would not be initialized before use. the compiler is correct to issue
> the warning. Give s.x a value after/at initialization, but before the if
> statement to give it a desired initial value then recompile, that should fix
> the error.
>
> Compilers only set the code to initialize the variable at declaration, not
> actually use the values during compile. If it was declared as a constant
> using a compiler directive such as #define, then the compiler would use the
> value in the logic and still give an error, but a different one because the
> conditional would always evaluate true (or false depending on what it was
> set to)
>
>
> On 03/21/2016 21:31, J Decker wrote:
>>
>> On Mon, Mar 21, 2016 at 8:40 PM, Scott Doctor 
>> wrote:
>>>
>>> you are missing
>>>
>>> using System;
>>
>> whatever.  It still fails because it says the variable is
>> uninitilalized.  THe only thing that doesn't is actually running it.
>>
>> That same pattern not matter what the language triggers warning/error
>> checkers
>>>
>>> 
>>> Scott Doctor
>>> scott at scottdoctor.com
>>> --
>>>
>>>
>>> On 3/21/2016 5:21 PM, J Decker wrote:

 So far I just see analysis tools fail for the same sorts of valid
 code...

 this is a bit of C# but the same idea causes the same warnings and
 there's nothign tecniclally wrong with this.



 class test
 {
  struct large_struct { public int x; }
  bool arbitrary_true_false = true;
  void method()
  {
 bool initialized = false;
 large_struct s;
 if( arbitrary_true_false )
 {
initialized = true;
s.x = 1;
 }
 if( initialized )
 {
Console.WriteLine( "this fails(during compile) as
 uninitialized: {0}", s.x );
 }
  }
 }

 On Mon, Mar 21, 2016 at 4:35 PM, James K. Lowden
  wrote:
>
> On Mon, 21 Mar 2016 13:48:06 -0700
> Scott Perry  wrote:
>
>> Compilers allow you to choose your standard; --std=c11 means
>> something very specific (and unchanging)
>
> They do.  And that covers what the standard covers.  The standard also
> has limits.  It includes constructs that are syntactically permitted
> but whose behavior is left undefined, known by the scarred as "UB" for
> "undefined behavior". An example from Clang's discussion is
>
>   int i = 10 << 31;
>
> The standard says << is a shift operator.  It places no limit on the
> number of bits to be shifted.  If that number is so large that the
> product cannot be represented by the assigned variable, that is *not*
> an error.  The standard allows the compiler to do anything or nothing
> with it.  As you may imagine, the varieties of anything and nothing are
> many.
>
> Compiler writers are well aware that "nothing" is faster done than
> "something".  Over time, they have gotten more aggressive in 

[sqlite] Article about pointer abuse in SQLite

[Cezary H. Noweta]

Hello,

On 2016-03-22 00:35, James K. Lowden wrote:
>[...]  An example from Clang's discussion is
>
>   int i = 10 << 31;

Could you provide a link for that discussion? (Or google's phrase to 
retrieve such link?)

-- best regards

Cezary H. Noweta

[sqlite] Article about pointer abuse in SQLite

[J Decker]

On Tue, Mar 22, 2016 at 6:58 AM, Adam Devita  wrote:
> It may be pedantic, but VS2016 will stop complaining if you edit your
> definition of s to
> large_struct s=new large_struct();  //set s to an actual instance of
> large_struct. c people can think of s as a pointer, and in c# the
> members are set to their default values.
>
The point was, the structure had some 20 members, and 90% of the time
the conditions don't exist for it to be initialized.  So rather than
initialize it 90% of the time for no use, I added checks to optimize
the object's creation.

> J Decker's point could also have been made by using int x in place of
> large_struct s . and sub x for s.x  , since it is a contrived example
> anyway.  The only way to use x is if another conditional on another
> variable that follows it in code and it is initialized.
>
> if one writes
> const bool arbitrary_true_false = true;   //note the const as Scott
> Doctor implied, makes the error go away.
>

It's not a const though, it's a variable the changes during runtime
and allows for the creation of such an object. It's not 'contrived' it
was an example that I ran into while developing (several times in
fact).

similarly soemthing like (I haven't run it though compilers, so don't
know if this is nested enough to cause the same issue... but it's easy
to see how a compiler/error checker would similarly be confused.

void f() { int a, b;
   for( a = 0; a < 2; a++ ) {
   if(  a == 0 ) b = 1234;
   }
   printf( "b is never uniniialized here : %d", b );
}

> -
> This discussion on the nature of undefined behaviour code is
> interesting.  I don't know the reasoning, but it seems that VS6 often
> initialized things to 0xcd in debug mode and (usually) had memory
> uninitialized to 0x00 when complied in Release (perhaps 0x00 just
> happens to be what was on the stack or heap).  I presume this wasn't
> just to make people suffer  when things don't work the same in debug
> vs release mode.
>
> Does the tool help (in the sqlite in practice) point out things that
> could be problematic?  Is it a compiler's variant of  "hay,  you are
> depending on implemented, not documented behaviour" ?
>
> regards,
> Adam DeVita
>
>
> On Tue, Mar 22, 2016 at 7:27 AM, Scott Doctor  
> wrote:
>>
>> It is uninitialized. you are setting an initial value within an if
>> statement. For the compiler, the code has NOT actually executed. so it does
>> not use the value of the variable arbitrary_true_false. If it was a #define
>> then it would use the value but still give an error because it is not a
>> compiler directive #if but a code if.
>>
>> The logic is that the first instance of assignment is within a conditional.
>> That is a particularly nasty kind of bug and should be reported as an error.
>> because if later you decide to change arbitrary_true_false to false, then
>> s.x would not be initialized before use. the compiler is correct to issue
>> the warning. Give s.x a value after/at initialization, but before the if
>> statement to give it a desired initial value then recompile, that should fix
>> the error.
>>
>> Compilers only set the code to initialize the variable at declaration, not
>> actually use the values during compile. If it was declared as a constant
>> using a compiler directive such as #define, then the compiler would use the
>> value in the logic and still give an error, but a different one because the
>> conditional would always evaluate true (or false depending on what it was
>> set to)
>>
>>
>> On 03/21/2016 21:31, J Decker wrote:
>>>
>>> On Mon, Mar 21, 2016 at 8:40 PM, Scott Doctor 
>>> wrote:

 you are missing

 using System;
>>>
>>> whatever.  It still fails because it says the variable is
>>> uninitilalized.  THe only thing that doesn't is actually running it.
>>>
>>> That same pattern not matter what the language triggers warning/error
>>> checkers

 
 Scott Doctor
 scott at scottdoctor.com
 --


 On 3/21/2016 5:21 PM, J Decker wrote:
>
> So far I just see analysis tools fail for the same sorts of valid
> code...
>
> this is a bit of C# but the same idea causes the same warnings and
> there's nothign tecniclally wrong with this.
>
>
>
> class test
> {
>  struct large_struct { public int x; }
>  bool arbitrary_true_false = true;
>  void method()
>  {
> bool initialized = false;
> large_struct s;
> if( arbitrary_true_false )
> {
>initialized = true;
>s.x = 1;
> }
> if( initialized )
> {
>Console.WriteLine( "this fails(during compile) as
> uninitialized: {0}", s.x );
> }
>  }
> }
>
> On Mon, Mar 21, 2016 at 4:35 PM, James K. Lowden
>  wrote:
>>
>> On Mon, 21 Mar 2016 13:48:06 -0700
>> Scott Perry  wrote:
>>
>>> 

[sqlite] Article about pointer abuse in SQLite

[J Decker]

On Tue, Mar 22, 2016 at 4:27 AM, Scott Doctor  wrote:
>
> It is uninitialized. you are setting an initial value within an if
> statement. For the compiler, the code has NOT actually executed. so it does
> not use the value of the variable arbitrary_true_false. If it was a #define
> then it would use the value but still give an error because it is not a
> compiler directive #if but a code if.
>
> The logic is that the first instance of assignment is within a conditional.
> That is a particularly nasty kind of bug and should be reported as an error.
> because if later you decide to change arbitrary_true_false to false, then
> s.x would not be initialized before use. the compiler is correct to issue
> the warning. Give s.x a value after/at initialization, but before the if
> statement to give it a desired initial value then recompile, that should fix
> the error.
>
> Compilers only set the code to initialize the variable at declaration, not
> actually use the values during compile. If it was declared as a constant
> using a compiler directive such as #define, then the compiler would use the
> value in the logic and still give an error, but a different one because the
> conditional would always evaluate true (or false depending on what it was
> set to)
>

The usage is never uninitinalized.  You can pass false all you want,
once you make it ot passing it true, the initialized gets set, and the
thing IS initialized.
There is no cause for warning or error in either of these cases.
There is NEVER a time when it will be used and be unassigned.
See - it's too complex for even human minds to reason through how
could a dumb tool have any hope?
>
> On 03/21/2016 21:31, J Decker wrote:
>>
>> On Mon, Mar 21, 2016 at 8:40 PM, Scott Doctor 
>> wrote:
>>>
>>> you are missing
>>>
>>> using System;
>>
>> whatever.  It still fails because it says the variable is
>> uninitilalized.  THe only thing that doesn't is actually running it.
>>
>> That same pattern not matter what the language triggers warning/error
>> checkers
>>>
>>> 
>>> Scott Doctor
>>> scott at scottdoctor.com
>>> --
>>>
>>>
>>> On 3/21/2016 5:21 PM, J Decker wrote:

 So far I just see analysis tools fail for the same sorts of valid
 code...

 this is a bit of C# but the same idea causes the same warnings and
 there's nothign tecniclally wrong with this.



 class test
 {
  struct large_struct { public int x; }
  bool arbitrary_true_false = true;
  void method()
  {
 bool initialized = false;
 large_struct s;
 if( arbitrary_true_false )
 {
initialized = true;
s.x = 1;
 }
 if( initialized )
 {
Console.WriteLine( "this fails(during compile) as
 uninitialized: {0}", s.x );
 }
  }
 }

 On Mon, Mar 21, 2016 at 4:35 PM, James K. Lowden
  wrote:
>
> On Mon, 21 Mar 2016 13:48:06 -0700
> Scott Perry  wrote:
>
>> Compilers allow you to choose your standard; --std=c11 means
>> something very specific (and unchanging)
>
> They do.  And that covers what the standard covers.  The standard also
> has limits.  It includes constructs that are syntactically permitted
> but whose behavior is left undefined, known by the scarred as "UB" for
> "undefined behavior". An example from Clang's discussion is
>
>   int i = 10 << 31;
>
> The standard says << is a shift operator.  It places no limit on the
> number of bits to be shifted.  If that number is so large that the
> product cannot be represented by the assigned variable, that is *not*
> an error.  The standard allows the compiler to do anything or nothing
> with it.  As you may imagine, the varieties of anything and nothing are
> many.
>
> Compiler writers are well aware that "nothing" is faster done than
> "something".  Over time, they have gotten more aggressive in simply
> deleting UB code.  As a consequence, programmers who thought they wrote
> standards-conforming code get burned when they upgrade/change
> compilers.  Mysterious and sometimes subtle errors are introduced by
> the compiler for the user's benefit.
>
> Your googlefu will turn up lots of discussion.  One I liked that wasn't
> on Page 1:
>
>
>
> http://blog.frama-c.com/index.php?post/2013/10/09/Overflow-float-integer
>
> --jkl
> ___
> sqlite-users mailing list
> sqlite-users at mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

 ___
 sqlite-users mailing list
 sqlite-users at mailinglists.sqlite.org
 http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] Article about pointer abuse in SQLite

[Scott Doctor]

It is uninitialized. you are setting an initial value within an if 
statement. For the compiler, the code has NOT actually executed. so it 
does not use the value of the variable arbitrary_true_false. If it was a 
#define then it would use the value but still give an error because it 
is not a compiler directive #if but a code if.

The logic is that the first instance of assignment is within a 
conditional. That is a particularly nasty kind of bug and should be 
reported as an error. because if later you decide to change 
arbitrary_true_false to false, then s.x would not be initialized before 
use. the compiler is correct to issue the warning. Give s.x a value 
after/at initialization, but before the if statement to give it a 
desired initial value then recompile, that should fix the error.

Compilers only set the code to initialize the variable at declaration, 
not actually use the values during compile. If it was declared as a 
constant using a compiler directive such as #define, then the compiler 
would use the value in the logic and still give an error, but a 
different one because the conditional would always evaluate true (or 
false depending on what it was set to)

On 03/21/2016 21:31, J Decker wrote:
> On Mon, Mar 21, 2016 at 8:40 PM, Scott Doctor  
> wrote:
>> you are missing
>>
>> using System;
> whatever.  It still fails because it says the variable is
> uninitilalized.  THe only thing that doesn't is actually running it.
>
> That same pattern not matter what the language triggers warning/error checkers
>> 
>> Scott Doctor
>> scott at scottdoctor.com
>> --
>>
>>
>> On 3/21/2016 5:21 PM, J Decker wrote:
>>> So far I just see analysis tools fail for the same sorts of valid code...
>>>
>>> this is a bit of C# but the same idea causes the same warnings and
>>> there's nothign tecniclally wrong with this.
>>>
>>>
>>>
>>> class test
>>> {
>>>  struct large_struct { public int x; }
>>>  bool arbitrary_true_false = true;
>>>  void method()
>>>  {
>>> bool initialized = false;
>>> large_struct s;
>>> if( arbitrary_true_false )
>>> {
>>>initialized = true;
>>>s.x = 1;
>>> }
>>> if( initialized )
>>> {
>>>Console.WriteLine( "this fails(during compile) as
>>> uninitialized: {0}", s.x );
>>> }
>>>  }
>>> }
>>>
>>> On Mon, Mar 21, 2016 at 4:35 PM, James K. Lowden
>>>  wrote:
 On Mon, 21 Mar 2016 13:48:06 -0700
 Scott Perry  wrote:

> Compilers allow you to choose your standard; --std=c11 means
> something very specific (and unchanging)
 They do.  And that covers what the standard covers.  The standard also
 has limits.  It includes constructs that are syntactically permitted
 but whose behavior is left undefined, known by the scarred as "UB" for
 "undefined behavior". An example from Clang's discussion is

   int i = 10 << 31;

 The standard says << is a shift operator.  It places no limit on the
 number of bits to be shifted.  If that number is so large that the
 product cannot be represented by the assigned variable, that is *not*
 an error.  The standard allows the compiler to do anything or nothing
 with it.  As you may imagine, the varieties of anything and nothing are
 many.

 Compiler writers are well aware that "nothing" is faster done than
 "something".  Over time, they have gotten more aggressive in simply
 deleting UB code.  As a consequence, programmers who thought they wrote
 standards-conforming code get burned when they upgrade/change
 compilers.  Mysterious and sometimes subtle errors are introduced by
 the compiler for the user's benefit.

 Your googlefu will turn up lots of discussion.  One I liked that wasn't
 on Page 1:


 http://blog.frama-c.com/index.php?post/2013/10/09/Overflow-float-integer

 --jkl
 ___
 sqlite-users mailing list
 sqlite-users at mailinglists.sqlite.org
 http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>>> ___
>>> sqlite-users mailing list
>>> sqlite-users at mailinglists.sqlite.org
>>> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>>>
>>>
>> ___
>> sqlite-users mailing list
>> sqlite-users at mailinglists.sqlite.org
>> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
> ___
> sqlite-users mailing list
> sqlite-users at mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>
>

-- 
-
Scott Doctor
scott at scottdoctor.com

[sqlite] Article about pointer abuse in SQLite

[J Decker]

On Mon, Mar 21, 2016 at 8:40 PM, Scott Doctor  wrote:
> you are missing
>
> using System;

whatever.  It still fails because it says the variable is
uninitilalized.  THe only thing that doesn't is actually running it.

That same pattern not matter what the language triggers warning/error checkers
>
> 
> Scott Doctor
> scott at scottdoctor.com
> --
>
>
> On 3/21/2016 5:21 PM, J Decker wrote:
>>
>> So far I just see analysis tools fail for the same sorts of valid code...
>>
>> this is a bit of C# but the same idea causes the same warnings and
>> there's nothign tecniclally wrong with this.
>>
>>
>>
>> class test
>> {
>> struct large_struct { public int x; }
>> bool arbitrary_true_false = true;
>> void method()
>> {
>>bool initialized = false;
>>large_struct s;
>>if( arbitrary_true_false )
>>{
>>   initialized = true;
>>   s.x = 1;
>>}
>>if( initialized )
>>{
>>   Console.WriteLine( "this fails(during compile) as
>> uninitialized: {0}", s.x );
>>}
>> }
>> }
>>
>> On Mon, Mar 21, 2016 at 4:35 PM, James K. Lowden
>>  wrote:
>>>
>>> On Mon, 21 Mar 2016 13:48:06 -0700
>>> Scott Perry  wrote:
>>>
 Compilers allow you to choose your standard; --std=c11 means
 something very specific (and unchanging)
>>>
>>> They do.  And that covers what the standard covers.  The standard also
>>> has limits.  It includes constructs that are syntactically permitted
>>> but whose behavior is left undefined, known by the scarred as "UB" for
>>> "undefined behavior". An example from Clang's discussion is
>>>
>>>  int i = 10 << 31;
>>>
>>> The standard says << is a shift operator.  It places no limit on the
>>> number of bits to be shifted.  If that number is so large that the
>>> product cannot be represented by the assigned variable, that is *not*
>>> an error.  The standard allows the compiler to do anything or nothing
>>> with it.  As you may imagine, the varieties of anything and nothing are
>>> many.
>>>
>>> Compiler writers are well aware that "nothing" is faster done than
>>> "something".  Over time, they have gotten more aggressive in simply
>>> deleting UB code.  As a consequence, programmers who thought they wrote
>>> standards-conforming code get burned when they upgrade/change
>>> compilers.  Mysterious and sometimes subtle errors are introduced by
>>> the compiler for the user's benefit.
>>>
>>> Your googlefu will turn up lots of discussion.  One I liked that wasn't
>>> on Page 1:
>>>
>>>
>>> http://blog.frama-c.com/index.php?post/2013/10/09/Overflow-float-integer
>>>
>>> --jkl
>>> ___
>>> sqlite-users mailing list
>>> sqlite-users at mailinglists.sqlite.org
>>> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>>
>> ___
>> sqlite-users mailing list
>> sqlite-users at mailinglists.sqlite.org
>> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>>
>>
>
> ___
> sqlite-users mailing list
> sqlite-users at mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] Article about pointer abuse in SQLite

[Scott Doctor]

you are missing

using System;


Scott Doctor
scott at scottdoctor.com
--

On 3/21/2016 5:21 PM, J Decker wrote:
> So far I just see analysis tools fail for the same sorts of valid code...
>
> this is a bit of C# but the same idea causes the same warnings and
> there's nothign tecniclally wrong with this.
>
>
>
> class test
> {
> struct large_struct { public int x; }
> bool arbitrary_true_false = true;
> void method()
> {
>bool initialized = false;
>large_struct s;
>if( arbitrary_true_false )
>{
>   initialized = true;
>   s.x = 1;
>}
>if( initialized )
>{
>   Console.WriteLine( "this fails(during compile) as
> uninitialized: {0}", s.x );
>}
> }
> }
>
> On Mon, Mar 21, 2016 at 4:35 PM, James K. Lowden
>  wrote:
>> On Mon, 21 Mar 2016 13:48:06 -0700
>> Scott Perry  wrote:
>>
>>> Compilers allow you to choose your standard; --std=c11 means
>>> something very specific (and unchanging)
>> They do.  And that covers what the standard covers.  The standard also
>> has limits.  It includes constructs that are syntactically permitted
>> but whose behavior is left undefined, known by the scarred as "UB" for
>> "undefined behavior". An example from Clang's discussion is
>>
>>  int i = 10 << 31;
>>
>> The standard says << is a shift operator.  It places no limit on the
>> number of bits to be shifted.  If that number is so large that the
>> product cannot be represented by the assigned variable, that is *not*
>> an error.  The standard allows the compiler to do anything or nothing
>> with it.  As you may imagine, the varieties of anything and nothing are
>> many.
>>
>> Compiler writers are well aware that "nothing" is faster done than
>> "something".  Over time, they have gotten more aggressive in simply
>> deleting UB code.  As a consequence, programmers who thought they wrote
>> standards-conforming code get burned when they upgrade/change
>> compilers.  Mysterious and sometimes subtle errors are introduced by
>> the compiler for the user's benefit.
>>
>> Your googlefu will turn up lots of discussion.  One I liked that wasn't
>> on Page 1:
>>
>>  
>> http://blog.frama-c.com/index.php?post/2013/10/09/Overflow-float-integer
>>
>> --jkl
>> ___
>> sqlite-users mailing list
>> sqlite-users at mailinglists.sqlite.org
>> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
> ___
> sqlite-users mailing list
> sqlite-users at mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>
>

[sqlite] Article about pointer abuse in SQLite

[James K. Lowden]

On Mon, 21 Mar 2016 13:48:06 -0700
Scott Perry  wrote:

> Compilers allow you to choose your standard; --std=c11 means
> something very specific (and unchanging) 

They do.  And that covers what the standard covers.  The standard also
has limits.  It includes constructs that are syntactically permitted
but whose behavior is left undefined, known by the scarred as "UB" for
"undefined behavior". An example from Clang's discussion is

int i = 10 << 31;

The standard says << is a shift operator.  It places no limit on the
number of bits to be shifted.  If that number is so large that the
product cannot be represented by the assigned variable, that is *not*
an error.  The standard allows the compiler to do anything or nothing
with it.  As you may imagine, the varieties of anything and nothing are
many.  

Compiler writers are well aware that "nothing" is faster done than
"something".  Over time, they have gotten more aggressive in simply
deleting UB code.  As a consequence, programmers who thought they wrote
standards-conforming code get burned when they upgrade/change
compilers.  Mysterious and sometimes subtle errors are introduced by
the compiler for the user's benefit.  

Your googlefu will turn up lots of discussion.  One I liked that wasn't
on Page 1:

http://blog.frama-c.com/index.php?post/2013/10/09/Overflow-float-integer

--jkl

[sqlite] Article about pointer abuse in SQLite

[J Decker]

So far I just see analysis tools fail for the same sorts of valid code...

this is a bit of C# but the same idea causes the same warnings and
there's nothign tecniclally wrong with this.



class test
{
   struct large_struct { public int x; }
   bool arbitrary_true_false = true;
   void method()
   {
  bool initialized = false;
  large_struct s;
  if( arbitrary_true_false )
  {
 initialized = true;
 s.x = 1;
  }
  if( initialized )
  {
 Console.WriteLine( "this fails(during compile) as
uninitialized: {0}", s.x );
  }
   }
}

On Mon, Mar 21, 2016 at 4:35 PM, James K. Lowden
 wrote:
> On Mon, 21 Mar 2016 13:48:06 -0700
> Scott Perry  wrote:
>
>> Compilers allow you to choose your standard; --std=c11 means
>> something very specific (and unchanging)
>
> They do.  And that covers what the standard covers.  The standard also
> has limits.  It includes constructs that are syntactically permitted
> but whose behavior is left undefined, known by the scarred as "UB" for
> "undefined behavior". An example from Clang's discussion is
>
> int i = 10 << 31;
>
> The standard says << is a shift operator.  It places no limit on the
> number of bits to be shifted.  If that number is so large that the
> product cannot be represented by the assigned variable, that is *not*
> an error.  The standard allows the compiler to do anything or nothing
> with it.  As you may imagine, the varieties of anything and nothing are
> many.
>
> Compiler writers are well aware that "nothing" is faster done than
> "something".  Over time, they have gotten more aggressive in simply
> deleting UB code.  As a consequence, programmers who thought they wrote
> standards-conforming code get burned when they upgrade/change
> compilers.  Mysterious and sometimes subtle errors are introduced by
> the compiler for the user's benefit.
>
> Your googlefu will turn up lots of discussion.  One I liked that wasn't
> on Page 1:
>
> 
> http://blog.frama-c.com/index.php?post/2013/10/09/Overflow-float-integer
>
> --jkl
> ___
> sqlite-users mailing list
> sqlite-users at mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] Article about pointer abuse in SQLite

[Scott Robison]

On Mar 21, 2016 2:48 PM, "Scott Perry"  wrote:
>
> On Mar 21, 2016, at 3:17 AM, Klaas Van B.  wrote:
> >
> >>> On 3/19/16, James K. Lowden  wrote:
> >
> >>> ... If the correctness of the code is
> >>> subject to change by the compiler's interpretation of the language,
how
> >>> is the programmer to prevent it?
> >
> >> On Sat, 19 Mar 2016 15:50:43 -0400 Richard Hipp  wrote:
> >
> >> ... But subsequent revisions of the
> >> C-language standards changed that.  How does one write code that will
> >> comply with language standards that keep changing out from under you?
> >
> > It's like trying to live according to the law while they're changing
the constitution.
>
> This is a false dichotomy. Compilers allow you to choose your standard;
--std=c11 means something very specific (and unchanging) about the
behaviour you can expect to be defined.

Unfortunately, gcc library header files warn about perfectly valid code in
C89 mode due to changes in C99. It's not a big deal until people take
compiler warnings as prima facie evidence that code is broken.

[sqlite] Article about pointer abuse in SQLite

[Scott Perry]

On Mar 21, 2016, at 1:48 PM, Scott Perry  wrote:
> 
> On Mar 21, 2016, at 3:17 AM, Klaas Van B.  wrote:
>> 
 On 3/19/16, James K. Lowden  wrote:
>> 
 ... If the correctness of the code is
 subject to change by the compiler's interpretation of the language, how
 is the programmer to prevent it?
>> 
>>> On Sat, 19 Mar 2016 15:50:43 -0400 Richard Hipp  wrote:
>> 
>>> ... But subsequent revisions of the
>>> C-language standards changed that.  How does one write code that will
>>> comply with language standards that keep changing out from under you?
>> 
>> It's like trying to live according to the law while they're changing the 
>> constitution.
> 
> This is a false dichotomy. Compilers allow you to choose your standard; 
> --std=c11 means something very specific (and unchanging) about the behaviour 
> you can expect to be defined.

(and yes, I did mean to say false equivalence in my response)

[sqlite] Article about pointer abuse in SQLite

[Scott Perry]

On Mar 21, 2016, at 3:17 AM, Klaas Van B.  wrote:
> 
>>> On 3/19/16, James K. Lowden  wrote:
> 
>>> ... If the correctness of the code is
>>> subject to change by the compiler's interpretation of the language, how
>>> is the programmer to prevent it?
> 
>> On Sat, 19 Mar 2016 15:50:43 -0400 Richard Hipp  wrote:
> 
>> ... But subsequent revisions of the
>> C-language standards changed that.  How does one write code that will
>> comply with language standards that keep changing out from under you?
> 
> It's like trying to live according to the law while they're changing the 
> constitution.

This is a false dichotomy. Compilers allow you to choose your standard; 
--std=c11 means something very specific (and unchanging) about the behaviour 
you can expect to be defined.

[sqlite] Article about pointer abuse in SQLite

[Klaas Van B.]

>> On 3/19/16, James K. Lowden  wrote:

>> ... If the correctness of the code is
>> subject to change by the compiler's interpretation of the language, how
>> is the programmer to prevent it?

> On Sat, 19 Mar 2016 15:50:43 -0400 Richard Hipp  wrote:

> ... But subsequent revisions of the
> C-language standards changed that.  How does one write code that will
> comply with language standards that keep changing out from under you?

It's like trying to live according to the law while they're changing the 
constitution.


Kind regards/Vriendelijke groeten.
Klaas `Z4us` Van B., CEO/CIO LI#437429414

[sqlite] Article about pointer abuse in SQLite

[Scott Robison]

On Mar 19, 2016 1:19 PM, "James K. Lowden"  wrote:
>
> On Sat, 19 Mar 2016 02:04:35 -0600
> Scott Robison  wrote:
>
> > As he says, there's not real choice between fast and
> > > correct
> >
> > Except that testing can verify something is correct for a given
> > environment.
>
> That's actually not true, on a couple of levels.

True, and that was a sloppy phrasing on my part. Please replace "correct"
with "sufficiently correct for the use case". If you are writing software
for one environment you are in a very different situation than if you are
trying to write truly portable software intended to be built and used in
any arbitrary environment.

> Second, you can't test the future.  If the correctness of the code is
> subject to change by the compiler's interpretation of the language, how
> is the programmer to prevent it?

In addition to DRH's comment about the moving target set by evolving
standards: even if there was nothing undefined by the standard you can't
trust a new release of a compiler or library to have not introduced some
bug that impacts you. Hence the need to test the heck out of any intended
release. It's not proof of correctness but it is evidence of a likely
sufficiently correct program.

>
> > > finally drive gcc & friends in the direction of working
> > > with their users for a change.  Or make them irrelevant.
> >
> > I think they'd continue to be popular with people looking to eek out
> > as much performance as possible.
>
> You may be right.  As a consultant I've often felt I was hired to
> sprinkle magic pixie performance dust on the system.  People want to
> believe that performance is found in tools.  How come there's no -O5?
>
> In truth, every performance problem I've encountered was a design
> problem, often obvious, always unnecessary.  "Use a better compiler"
> has never been the solution.  Unloading mounds of unnecessary
> processing with a pitchfork is.

I guess it depends on what you're doing. Some applications (I'm thinking
games at the moment) need both quality design and quality code generation.
When you know your application is being targeted for a platform you'll
utilize undefined behavior if it eeks out a little bit more performance. If
necessary you'll hide UB in conditional blocks so that you get the best out
of different platforms.

I agree that most applications do not need anything like this, and I think
it is a good idea to steer clear of UB, but it is not universally true.

[sqlite] Article about pointer abuse in SQLite

[Richard Hipp]

On 3/19/16, James K. Lowden  wrote:
>
> Second, you can't test the future.  If the correctness of the code is
> subject to change by the compiler's interpretation of the language, how
> is the programmer to prevent it?
>

Indeed.  Every bit of the code examined by Prof. Regehr was
well-defined according to K  But subsequent revisions of the
C-language standards changed that.  How does one write code that will
comply with language standards that keep changing out from under you?
-- 
D. Richard Hipp
drh at sqlite.org

[sqlite] Article about pointer abuse in SQLite

[James K. Lowden]

On Sat, 19 Mar 2016 02:04:35 -0600
Scott Robison  wrote:

> As he says, there's not real choice between fast and
> > correct
> 
> Except that testing can verify something is correct for a given
> environment.  

That's actually not true, on a couple of levels.  

"[T]esting can be used very effectively to show the presence of
bugs but never to show their absence."
-- EWD303

I think that should be called Dijkstra's Dictum.  It's not just quip;
it's a concise insight into limits of testing versus proving
correctness.  

Second, you can't test the future.  If the correctness of the code is
subject to change by the compiler's interpretation of the language, how
is the programmer to prevent it?  

> > finally drive gcc & friends in the direction of working
> > with their users for a change.  Or make them irrelevant.
> 
> I think they'd continue to be popular with people looking to eek out
> as much performance as possible.

You may be right.  As a consultant I've often felt I was hired to
sprinkle magic pixie performance dust on the system.  People want to
believe that performance is found in tools.  How come there's no -O5?  

In truth, every performance problem I've encountered was a design
problem, often obvious, always unnecessary.  "Use a better compiler"
has never been the solution.  Unloading mounds of unnecessary
processing with a pitchfork is.  

Doubtless there are some well tested, highly stable applications run at
scale, for which 5% is a measurable and meaningful gain.  IMO they're
actually the ones driving UB treatment by compiler writers.  The
other 99% stand to gain from a compiler that emphasizes correctness and
predictable behavior.  

--jkl

[sqlite] Article about pointer abuse in SQLite

[Scott Robison]

On Mar 18, 2016 11:12 PM, "James K. Lowden" 
wrote:
>
> On Fri, 18 Mar 2016 16:33:56 -0600
> Scott Robison  wrote:
>
> > I'd rather have code that might use some "undefined behavior" and
> > generates the right answer than code that always conformed to defined
> > behavior yet was logically flawed.
>
> Code that falls under undefined behavior *is* logically flawed, by
> definition.  Whether or not it works, it's not specified to.  The
> compiler may have generated perfectly correct machine code, but another
> compiler or some future version of your present compiler may not.

Perhaps I should have said "undefined behavior as per the standard". Code
that does what is intended for the intended target environment utilizing a
specific tool chain is not logically flawed just because the standard calls
a construct "undefined behavior".

> You might share my beef with the compiler writers, though: lots things
> that are left undefined shouldn't be.

Not just compiler writers but standards organizations. Lots of overlap to
be sure, but not 100%.

Because hardware architecture
> varies, some practices that do work and have worked and are expected to
> work on a wide variety of machines are UB.  A recent thread on using
> void* for a function pointer is an example: dlsym(2) returns a function
> pointer defined as void*, but the C standard says void* can only refer
> to data, not functions!

So I guess casting between function pointers might be arguably safer if the
void* form of a function was called void(*)(void) though even then there
can be multiple forms of function pointers (like near vs far pointers in
x86 real mode).

>
> Machines exist for which the size of a function pointer is not
> sizeof(void*).  Source code that assumes they are the same size is not
> portable to those architectures.  Fine.  But a particular compiler
> generates code for a particular architecture.  On x86 hardware, all
> pointers have always been and will always be the same size.  All
> Linux/Posix code relies on that, too, along with a host of other
> assumptions. If that ever changed, a boat load of code would have to be
> changed.  Why does the compiler writer feel it's in his interest or
> mine to warn me about that not-happening eventuality?  For the machine
> I'm compilng for, the code is *not* in error.  For some future machine,
> maybe it will be; let's leave that until then.
>
> I was looking at John Regehr's blog the other day.  I think it was
> there that I learned that the practice of dropping UB code on the floor
> has been going on longer than I'd realized; it's just that gcc has been
> more aggressive in recent years.  I think it was there I saw this
> construction:
>
> if( p < p + n)
> error
>
> where p is a pointer.  On lots of architectures, for large n, p + n can
> be negative.  The test works.  Or did.  The C standard says that's
> UB, though. It doesn't promise the pointer will go negative.  It doesn't
> promise it won't.  It doesn't promise not to tell your mother about
> it.  And, in one recent version, it doesn't compile it.  Warning?  No.
> Error? No.  Machine code?  No!  It's UB, so no code is generated (ergo,
> no error handling)!  Even though the hardware instructions that would
> be -- that used to be -- generated work as implied by the code.

Yes, these are the sorts of things that are frustrating.

> Postel's Law is to be liberal in what you accept and conservative in
> what you emit.  The compilers have been practicing the opposite,
> thwarting common longstanding practice just because they "can".
>
> Dan Bernstein is calling for a new C compiler that is 100%
> deterministic: no UB.  All UB per the standard would be defined by the
> compiler.  And maybe a few goodies, like zero-initialized automatic
> (stack) variables.

Neat idea, though undefined behavior isn't always bad. Interesting article
at http://blog.llvm.org/2011/05/what-every-c-programmer-should-know.html
that goes into some considerations.

>
> Such a compiler would enjoy great popularity, even if it imposed, say,
> a 5% performance penalty, because C programmers would have greater
> confidence in their code working as expected. They'd have some
> assurance that the compiler wouldn't cut them off at the knees in its
> next release.  As he says, there's not real choice between fast and
> correct

Except that testing can verify something is correct for a given environment.

  If the "always defined befavior" compiler got off the ground,
> may it would finally drive gcc & friends in the direction of working
> with their users for a change.  Or make them irrelevant.

I think they'd continue to be popular with people looking to eek out as
much performance as possible.

[sqlite] Article about pointer abuse in SQLite

[James K. Lowden]

On Fri, 18 Mar 2016 16:33:56 -0600
Scott Robison  wrote:

> I'd rather have code that might use some "undefined behavior" and
> generates the right answer than code that always conformed to defined
> behavior yet was logically flawed. 

Code that falls under undefined behavior *is* logically flawed, by
definition.  Whether or not it works, it's not specified to.  The
compiler may have generated perfectly correct machine code, but another
compiler or some future version of your present compiler may not.  

You might share my beef with the compiler writers, though: lots things
that are left undefined shouldn't be.  Because hardware architecture
varies, some practices that do work and have worked and are expected to
work on a wide variety of machines are UB.  A recent thread on using
void* for a function pointer is an example: dlsym(2) returns a function
pointer defined as void*, but the C standard says void* can only refer
to data, not functions!  

Machines exist for which the size of a function pointer is not 
sizeof(void*).  Source code that assumes they are the same size is not
portable to those architectures.  Fine.  But a particular compiler
generates code for a particular architecture.  On x86 hardware, all
pointers have always been and will always be the same size.  All
Linux/Posix code relies on that, too, along with a host of other
assumptions. If that ever changed, a boat load of code would have to be
changed.  Why does the compiler writer feel it's in his interest or
mine to warn me about that not-happening eventuality?  For the machine
I'm compilng for, the code is *not* in error.  For some future machine,
maybe it will be; let's leave that until then.  

I was looking at John Regehr's blog the other day.  I think it was
there that I learned that the practice of dropping UB code on the floor
has been going on longer than I'd realized; it's just that gcc has been
more aggressive in recent years.  I think it was there I saw this
construction:

if( p < p + n)
error

where p is a pointer.  On lots of architectures, for large n, p + n can
be negative.  The test works.  Or did.  The C standard says that's
UB, though. It doesn't promise the pointer will go negative.  It doesn't
promise it won't.  It doesn't promise not to tell your mother about
it.  And, in one recent version, it doesn't compile it.  Warning?  No.
Error? No.  Machine code?  No!  It's UB, so no code is generated (ergo,
no error handling)!  Even though the hardware instructions that would
be -- that used to be -- generated work as implied by the code.

Postel's Law is to be liberal in what you accept and conservative in
what you emit.  The compilers have been practicing the opposite,
thwarting common longstanding practice just because they "can".  

Dan Bernstein is calling for a new C compiler that is 100%
deterministic: no UB.  All UB per the standard would be defined by the
compiler.  And maybe a few goodies, like zero-initialized automatic
(stack) variables.  

Such a compiler would enjoy great popularity, even if it imposed, say,
a 5% performance penalty, because C programmers would have greater
confidence in their code working as expected. They'd have some
assurance that the compiler wouldn't cut them off at the knees in its
next release.  As he says, there's not real choice between fast and
correct  If the "always defined befavior" compiler got off the ground,
may it would finally drive gcc & friends in the direction of working
with their users for a change.  Or make them irrelevant.  

--jkl

[sqlite] Article about pointer abuse in SQLite

[Simon Slavin]

On 18 Mar 2016, at 10:33pm, Scott Robison  wrote:

> I'd rather have code that might use some "undefined behavior" and generates
> the right answer than code that always conformed to defined behavior yet
> was logically flawed. Mind you, I don't often have to worry about my code
> being compiled in a huge number of environments.

That's the problem.

The development team don't know what compilers their code is going to be 
compiled on.  They can argue that a compiler which violates /documented/ 
behaviour is broken.  But they can't argue that a compiler which correctly 
compiles C code is broken.  There is, by definition, nothing wrong with such a 
compiler and if it causes your code to do The Wrong Thing then your code is 
broken.

However nobody has come up with such a compiler.  Someone could write one, I 
suppose[1].  Then they'd compile SQLite with it.  Then they'd find a test case 
which caused SQLite to do The Wrong Thing.  Then they'd submit a bug report.  
Until that has happened, nobody has proved there's anything wrong with SQLite.

Frankly the fact that I can take a cross-compiler for an embedded processor in 
... say, a smart thermostat ... compile the amalgamation version on it and be 
sure it's going to work is very pleasing.

Simon.

[1] Or find an open source compiler and make some simple modifications to it.

[sqlite] Article about pointer abuse in SQLite

[jose isaias cabrera]

Scott Robison wrote...

> Also, it seems a lot of people today are critical of programs that utilize
> defined behavior in the context of the standard to which they were 
> written,
> but not to a later standard (such as ANSI C / C89 / C90 vs C99 or C11 or
> whatever). It's great to conform to both when reasonable, but it may not
> always be reasonable (as I can't necessarily conceive of every 
> possibility).

Good thought, Scott. I have come to the conclusion that 
"political-correctness" has even invaded computer science.  Ever since 
classes started to be pushed by teachers, all of these "ideal methods" have 
taken more power than they should have.  It's not that they are bad, but, 
they are just other methods for some folks to be able to choose. I have so 
much to say, but so little time.  Thank Simon for bringing this theme.

jos? 

[sqlite] Article about pointer abuse in SQLite

[Simon Slavin]

"SQLite is a carefully engineered and thoroughly tested piece of software. Even 
so, it contains undefined behaviors because, until recently, no good checker 
for these behaviors existed. If anything is going to save us from UB hell, it?s 
tools combined with developers who care to listen to them. "

Simon.

[sqlite] Article about pointer abuse in SQLite

[Richard Hipp]

On 3/18/16, Igor Tandetnik  wrote:
> On 3/18/2016 4:40 PM, Keith Medcalf wrote:
>> There is no such thing as "undefined behaviour".  The machine code does
>> exactly what it is told to do
>
> But SQLite is not written in machine code. It is (largely) written in C.

SQLite is written in C, but the focus of testing is the resulting
machine code.  So, there is a point of view that says that it doesn't
really matter if the C code is "undefined" or not.  What matters is
whether or not the resulting machine code computes the correct answer.
That's what we test: does the machine code compute the correct answer.

Our testing standard is to cause every machine-code branch instruction
to take both decisions (jump and fall through) at least once, in an
as-deployed configuration.  That means every instruction in the
machine code is tested and testing occurs with no special compile-time
options or flags.  The code that gets tested is exactly byte-for-byte
the same machine code that gets deployed in the field.  This is what
they mean by "fly what you test and test what you fly".

That said, we also do source-code validation on SQLite.  We work to
make SQLite compile without warnings, we run it through static
analyzers, and I address any "undefined behaviors" in the source code
that John Regehr or others report.  SQLite contains thousands of
assert() statements which are really another form of source-code
validation (as the asserts do not appear in deployment builds.)  Just
the other day, I wrote a custom static code analyzer for SQLite that
runs on every "make" looking for a particular kind of programming
error (see https://www.sqlite.org/src/artifact/4f65e1a6748e42f2).  We
also compile and test SQLite on as many different compilers as we can,
with all kinds of different compiler settings, and verify that they
all get the same answer, which is yet another kind of source-code
validation.

All of this source-code validation is well and good, and I recommend
it.  But the real focus of testing is SQLite is the machine code.  And
so to a first approximation, Keith is correct: "undefined behavior" at
the source code level does not matter.

You cannot say that a program is correct by only looking at source
code.  You have to validate the machine code.  Compilers make
mistakes.  SQLite has, at various times, hit bugs in each of GCC,
Clang, and MSVC, all of which have now been fixed.  Do not trust your
compiler.

One point of view is that compilers that attempt to take advantage of
obscure "undefined behavior" to boost performance, but instead break
programs, are in fact buggy.  (Compiler writers tend to disagree with
this viewpoint.  I'm not saying it is the correct point of view, just
a point of view that is widely held.)  So watching out for obscure
"undefined behavior" that breaks your program is really the same thing
as watching out for bugs in your compiler.  Both of these happen, and
in my experience, with about the same frequency.

-- 
D. Richard Hipp
drh at sqlite.org

[sqlite] Article about pointer abuse in SQLite

[Igor Tandetnik]

On 3/18/2016 4:40 PM, Keith Medcalf wrote:
> There is no such thing as "undefined behaviour".  The machine code does 
> exactly what it is told to do

But SQLite is not written in machine code. It is (largely) written in C. 
And C language most certainly has the concept of undefined behavior - 
roughly, a language construct or situation for which a C compiler is not 
obliged to produce any particular machine code; or in other words, can 
produce machine code that does anything at all. See also:

https://en.wikipedia.org/wiki/Undefined_behavior
http://catb.org/jargon/html/N/nasal-demons.html

> Things will only be non-deterministic and perhaps undefined when run on 
> Quantum Computers using Heisenberg registers for intermediate results.

There's plenty of non-deterministic behavior on regular von Neumann 
machines with more than one core. See e.g. 
https://en.wikipedia.org/wiki/Race_condition . In any case, "undefined 
behavior" is not at all the same thing as "non-deterministic behavior". 
A conforming C (or C++) program does not (by definition of "conforming") 
exhibit undefined behavior, but may very well be non-deterministic.
-- 
Igor Tandetnik

[sqlite] Article about pointer abuse in SQLite

[Scott Robison]

On Fri, Mar 18, 2016 at 4:03 PM, Richard Hipp  wrote:

> On 3/18/16, Igor Tandetnik  wrote:
> > On 3/18/2016 4:40 PM, Keith Medcalf wrote:
> >> There is no such thing as "undefined behaviour".  The machine code does
> >> exactly what it is told to do
> >
> > But SQLite is not written in machine code. It is (largely) written in C.
>
> SQLite is written in C, but the focus of testing is the resulting
> machine code.  So, there is a point of view that says that it doesn't
> really matter if the C code is "undefined" or not.  What matters is
> whether or not the resulting machine code computes the correct answer.
> That's what we test: does the machine code compute the correct answer.
>
> That said, we also do source-code validation on SQLite.  We work to
> make SQLite compile without warnings, we run it through static
> analyzers, and I address any "undefined behaviors" in the source code
> that John Regehr or others report.  SQLite contains thousands of
>

I'd rather have code that might use some "undefined behavior" and generates
the right answer than code that always conformed to defined behavior yet
was logically flawed. Mind you, I don't often have to worry about my code
being compiled in a huge number of environments.

Also, it seems a lot of people today are critical of programs that utilize
defined behavior in the context of the standard to which they were written,
but not to a later standard (such as ANSI C / C89 / C90 vs C99 or C11 or
whatever). It's great to conform to both when reasonable, but it may not
always be reasonable (as I can't necessarily conceive of every possibility).

-- 
Scott Robison

[sqlite] Article about pointer abuse in SQLite

[Marc L. Allen]

I had a long response to this, but it occurs to me that you're just being 
pedantic for fun.  Am I wrong?

-Original Message-
From: sqlite-users-bounces at mailinglists.sqlite.org 
[mailto:sqlite-users-boun...@mailinglists.sqlite.org] On Behalf Of Keith Medcalf
Sent: Friday, March 18, 2016 4:41 PM
To: SQLite mailing list 
Subject: Re: [sqlite] Article about pointer abuse in SQLite


There is no such thing as "undefined behaviour".  The machine code does exactly 
what it is told to do in exactly the manner in which it has been told to do it 
and obtains exactly the correct answer every time.

That the computation is "advanced beyond the realm of understanding of the 
observer" does not make the behaviour undefined.  It is perfectly defined, 
however, it is occasionally necessary to describe things as "undefined", 
oftentimes because it is too complicated to explain.  Just because someone says 
something as "undefined" does not mean that is so.  It is simply a euphemism 
for "I don't understand how it did that/what it is supposed to be doing (or, 
more often an appeal to self-proclaimed authority which said that such 
behaviour was undefined" without  having to admit fault, much in the same way 
that "supported" is a euphemism for "make money from".

Things will only be non-deterministic and perhaps undefined when run on Quantum 
Computers using Heisenberg registers for intermediate results.

> <http://blog.regehr.org/archives/1292>

> "SQLite is a carefully engineered and thoroughly tested piece of software.
> Even so, it contains undefined behaviors because, until recently, no 
> good checker for these behaviors existed. If anything is going to save 
> us from UB hell, it?s tools combined with developers who care to listen to 
> them. "

> Simon.




___
sqlite-users mailing list
sqlite-users at mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users



This email and any attachments are only for use by the intended recipient(s) 
and may contain legally privileged, confidential, proprietary or otherwise 
private information. Any unauthorized use, reproduction, dissemination, 
distribution or other disclosure of the contents of this e-mail or its 
attachments is strictly prohibited. If you have received this email in error, 
please notify the sender immediately and delete the original.

[sqlite] Article about pointer abuse in SQLite

[Keith Medcalf]

There is no such thing as "undefined behaviour".  The machine code does exactly 
what it is told to do in exactly the manner in which it has been told to do it 
and obtains exactly the correct answer every time.

That the computation is "advanced beyond the realm of understanding of the 
observer" does not make the behaviour undefined.  It is perfectly defined, 
however, it is occasionally necessary to describe things as "undefined", 
oftentimes because it is too complicated to explain.  Just because someone says 
something as "undefined" does not mean that is so.  It is simply a euphemism 
for "I don't understand how it did that/what it is supposed to be doing (or, 
more often an appeal to self-proclaimed authority which said that such 
behaviour was undefined" without  having to admit fault, much in the same way 
that "supported" is a euphemism for "make money from".

Things will only be non-deterministic and perhaps undefined when run on Quantum 
Computers using Heisenberg registers for intermediate results.

> 

> "SQLite is a carefully engineered and thoroughly tested piece of software.
> Even so, it contains undefined behaviors because, until recently, no good
> checker for these behaviors existed. If anything is going to save us from
> UB hell, it?s tools combined with developers who care to listen to them. "

> Simon.

[sqlite] Article about pointer abuse in SQLite

[Scott Hess]

Not sure where you're going with this.  "Undefined behavior" in this case
is obviously referring to things defined by the C standard.  Things not
defined by the standard can (and do) change over time as compilers advance,
and also often differ between compilers from different vendors.

-scott


On Fri, Mar 18, 2016 at 1:40 PM, Keith Medcalf  wrote:

>
> There is no such thing as "undefined behaviour".  The machine code does
> exactly what it is told to do in exactly the manner in which it has been
> told to do it and obtains exactly the correct answer every time.
>
> That the computation is "advanced beyond the realm of understanding of the
> observer" does not make the behaviour undefined.  It is perfectly defined,
> however, it is occasionally necessary to describe things as "undefined",
> oftentimes because it is too complicated to explain.  Just because someone
> says something as "undefined" does not mean that is so.  It is simply a
> euphemism for "I don't understand how it did that/what it is supposed to be
> doing (or, more often an appeal to self-proclaimed authority which said
> that such behaviour was undefined" without  having to admit fault, much in
> the same way that "supported" is a euphemism for "make money from".
>
> Things will only be non-deterministic and perhaps undefined when run on
> Quantum Computers using Heisenberg registers for intermediate results.
>
> > 
>
> > "SQLite is a carefully engineered and thoroughly tested piece of
> software.
> > Even so, it contains undefined behaviors because, until recently, no good
> > checker for these behaviors existed. If anything is going to save us from
> > UB hell, it?s tools combined with developers who care to listen to them.
> "
>
> > Simon.
>
>
>
>
> ___
> sqlite-users mailing list
> sqlite-users at mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>