Re: Nested functions [was Re: valgrind]

[Chris Hanson]

On Mar 24, 2022, at 8:16 PM, Mouse  wrote:

> Nested functions are not closures, or at least not what I know as
> closures.  A nested function pointer (conceptually) goes invalid as
> soon as anything it refers to goes out of scope, or at the latest as
> soon as its smallest enclosing block exits (or possibly when its
> smallest enclosing function returns).  The thing I know as a closure
> would preserve the referred-to objects as long as the closure is
> potentially callable.  This requires more reference tracking than C
> typically makes possible.

Closures can easily be used as nested functions, and the blocks feature that 
was created by Apple explicitly addresses this by including runtime functions.

1. You can just declare and use a block if it's not going to go out of scope, 
as if it were a nested function, with just slightly different syntax than 
nested functions. (You do this by declaring a block-typed variable and 
immediately assigning the block to it, then calling through the variable. It 
behaves like a function pointer.)

2. If you have a variable in the outer scope that you want to modify in the 
block, you have to annotate the variable with __block, so the block context 
(activation record) created for the bock can find it. Otherwise everything in 
the enclosing scope is effectively immutable (though one must of course be 
aware that an immutable pointer is not the same as a pointer to immutable data).

3. If you're passing the block to another function or block, whatever you pass 
it to should *either* annotate its block argument as non-escaping via the right 
__attribute__(()) to indicate it doesn't need any memory management, or it 
should Block_copy() the block it's handed and work only with the copy. That 
ensures the block context is hoisted to the heap. Following Apple's general 
patterns, a Block_copy() of a heap block is identical to a BlocK_retain() so 
the latter should really never be called.

4. If something calls Block_copy() because it wants to hold onto a block, it 
needs to call Block_release() to eventually dispose of the block.

Here are the language and runtime specs. They're a great language addition and 
used pervasively on Apple platforms as a result.

Language: https://clang.llvm.org/docs/BlockLanguageSpec.html 

Runtime/ABI: https://clang.llvm.org/docs/Block-ABI-Apple.html 


  -- Chris

Re: pcc [was Re: valgrind]

[Anders Magnusson]

Den 2022-03-22 kl. 18:47, skrev Koning, Paul:



On Mar 22, 2022, at 1:21 PM, Greg A. Woods  wrote:

At Mon, 21 Mar 2022 08:54:43 -0400 (EDT), Mouse  
wrote:
Subject: pcc [was Re: valgrind]

I've been making very-spare-time progress on building my own
compiler on and off for some years now; perhaps I'll eventually get
somewhere.  [...]

Have you looked at pcc?  http://pcc.ludd.ltu.se/ and in our source
tree in src/external/bsd/pcc .

No, I haven't.  I should - it may well end up being quicker to move an
existing compiler in the directions I want to go than to write my own.

...
I also really like PCC.  (I remember teething pains getting used to it
back when it first replaced Ritchie C on my university's PDP-11/60, but
once I actually used it for real code (i.e. assignments in those days),
and soon on the Vax too, I really liked it.)

I'm really sad that I still cannot build NetBSD entirely with PCC as the
native and only default compiler.

Out of curiosity: how does PCC code quality compare with that of GCC and (for 
targets that it supports) Clang?
Last time I tested (maybe 8 years ago?) against gcc with bytebench 
(IIRC? some of the benchmarks anyway) on i386 it
generated code that was slightly smaller (gcc -O6) and on average 6% 
slower than gcc.


The slowliness was mostly due to the lack of strength reduction in loops.

-- R

Re: pcc [was Re: valgrind]

[Anders Magnusson]

Den 2022-03-22 kl. 18:49, skrev bch:



On Tue, Mar 22, 2022 at 10:21 Greg A. Woods  wrote:

At Mon, 21 Mar 2022 08:54:43 -0400 (EDT), Mouse
 wrote:
Subject: pcc [was Re: valgrind]
>
> >> I've been making very-spare-time progress on building my own
> >> compiler on and off for some years now; perhaps I'll
eventually get
> >> somewhere.  [...]
> > Have you looked at pcc? http://pcc.ludd.ltu.se/ and in our source
> > tree in src/external/bsd/pcc .
>
> No, I haven't.  I should - it may well end up being quicker to
move an
> existing compiler in the directions I want to go than to write
my own.

I would like to add my voice too.

I _really_ like valgrind.  It is immensely valuable and infinitely
better than any of the compiler so-called "sanitizers" (except
maybe the
Undefined Behaviour sanitizer in Clang, which, sadly, is a necessary
evil if one is to use such a modern language bastardizer like Clang).

It's a little ugly to use, and it's a very tough task-master, but I'm
really sad that I cannot use it easily and regularly on NetBSD.

(I'm just about as sad that it no longer works on modern macOS
either.)


I also really like PCC.  (I remember teething pains getting used to it
back when it first replaced Ritchie C on my university's
PDP-11/60, but
once I actually used it for real code (i.e. assignments in those
days),
and soon on the Vax too, I really liked it.)

I'm really sad that I still cannot build NetBSD entirely with PCC
as the
native and only default compiler.



Is there a case for another concerted  campaign on pcc like ragge@ did 
those years ago?

If I get help with finding out what do not work then I can easily fix it.
Just fetch the latest pcc and try out :-)

-- R

Re: Nested functions [was Re: valgrind]

[David Holland]

On Thu, Mar 24, 2022 at 11:16:58PM -0400, Mouse wrote:
 > > The conclusion over the past ~25 years has been that there isn't;
 > > qsort and things like it work "well enough" and real uses for
 > > closures that really motivate the feature come up rarely enough that
 > > it doesn't happen.
 > 
 > Nested functions are not closures, or at least not what I know as
 > closures.  A nested function pointer (conceptually) goes invalid as
 > soon as anything it refers to goes out of scope, or at the latest as
 > soon as its smallest enclosing block exits (or possibly when its
 > smallest enclosing function returns).  The thing I know as a closure
 > would preserve the referred-to objects as long as the closure is
 > potentially callable.  This requires more reference tracking than C
 > typically makes possible.

There's a term for scope-restricted closures (that become invalid this
way) that I'm currently forgetting. They're still closures in most of
the important ways, and it's a useful concept for a language where
copying memory isn't free.

 > > There is no solution based on trampolines that'll pass security (or
 > > at least security-theatre) muster.  Unless maybe by doing something
 > > that's horrifying in other ways.
 > 
 > The safest alternative that comes to my mind is to have two stacks, one
 > for trampolines and one for everything else.  But that requires
 > something much like two stack pointers, including assist from the
 > setjmp/longjmp implementation and, if applicable, threads.

That's not s3kure! You can still get arbitrary code execution by
scribbling in it.

 > > (For example: you could declare a static limit on how many instances
 > > of the closure you'll ever produce, make a global array to stuff the
 > > data pointer in, and statically generate N trampoline entry points
 > > that read from that array and call the primary function.  But there
 > > are many other ways in which this is horrible.)
 > 
 > But there are use cases for which it is not a stupid implementation;
 > [...]

I think to keep it safe you need more code analysis than you're likely
to get with C code. But IDK.

-- 
David A. Holland
dholl...@netbsd.org

Re: valgrind

[Andrew Cagney]

On Fri, 25 Mar 2022 at 10:03, Reinoud Zandijk  wrote:

> > I guess they can think so because nested functions are a gcc extension.
> > Quite a useful one, I think, having been taught programming with Algol
> > 68 which of course does have nested functions; but an extension
> > nevertheless.
>
> AFAIR, it comes for GCC's support for pascal which has nested functions
> defined. Its just that for 'C' it was an easy addition since the compiler
> already suported it.

I suspect a bigger contributor may have been that all the cool
languages (and Ada) supported this feature; so why not GCC.   At the
time (pre-EGCS?) any language trying to use GCC as a front end really
struggled (except perhaps Ada), and Pascal more so than most.

However this is all hearsay.  My attempt at archology failed.  All I
found was a bug fix to the code in 1998.

Re: valgrind

[Reinoud Zandijk]

On Tue, Mar 22, 2022 at 06:17:50PM +0100, Rhialto wrote:
> On Sun 20 Mar 2022 at 23:17:57 -0400, Mouse wrote:
> > clang is - or at least was last I checked - under the impression that
> > nested functions are little-used and thus are not worth supporting.
> 
> I guess they can think so because nested functions are a gcc extension.
> Quite a useful one, I think, having been taught programming with Algol
> 68 which of course does have nested functions; but an extension
> nevertheless.

AFAIR, it comes for GCC's support for pascal which has nested functions
defined. Its just that for 'C' it was an easy addition since the compiler
already suported it.

Reinoud

Re: Nested functions [was Re: valgrind]

[Mouse]

> [...] said, moving to fat function pointers on machines that don't
> already use them is a real ABI change and therefore a big deal; but
> it could be done if there were a compelling argument to justify going
> through all the associated dark rituals.

Or as a private experiment, in which compatibility with other ABIs is
irrelevant.  That's what I would, putatively, be doing.

> The conclusion over the past ~25 years has been that there isn't;
> qsort and things like it work "well enough" and real uses for
> closures that really motivate the feature come up rarely enough that
> it doesn't happen.

Nested functions are not closures, or at least not what I know as
closures.  A nested function pointer (conceptually) goes invalid as
soon as anything it refers to goes out of scope, or at the latest as
soon as its smallest enclosing block exits (or possibly when its
smallest enclosing function returns).  The thing I know as a closure
would preserve the referred-to objects as long as the closure is
potentially callable.  This requires more reference tracking than C
typically makes possible.

> There is no solution based on trampolines that'll pass security (or
> at least security-theatre) muster.  Unless maybe by doing something
> that's horrifying in other ways.

The safest alternative that comes to my mind is to have two stacks, one
for trampolines and one for everything else.  But that requires
something much like two stack pointers, including assist from the
setjmp/longjmp implementation and, if applicable, threads.

> (For example: you could declare a static limit on how many instances
> of the closure you'll ever produce, make a global array to stuff the
> data pointer in, and statically generate N trampoline entry points
> that read from that array and call the primary function.  But there
> are many other ways in which this is horrible.)

But there are use cases for which it is not a stupid implementation;
for example, if no containing function is ever called recursively or
reentrantly, and of course if the limit is high enough, it is obviously
safe and quite possibly one of the fastest techniques available (since
"creating" a trampoline can be made very fast, not needing even D$
pushes and I$ flushes).  But, under those conditions, the autos in the
enclosing function can be promoted to static storage duration and the
nested function turned, essentially, into an ordinary function.

Are the cases where the compiler can prove that's true a big enough
fraction of the use cases to be useful?  I don't know, not even for my
own code.  I _think_ one of the heavier uses of the technique - my ssh
implementation - could work just fine with this approach, but I haven't
looked at every nested function closely enough to be sure.  (It needs
more than nested functions, though; one of its uses of nested functions
is to combine them with what gcc calls nonlocal gotos as a throw-out
mechanism better in many respects than setjmp/longjmp.)

/~\ The ASCII Mouse
\ / Ribbon Campaign
 X  Against HTMLmo...@rodents-montreal.org
/ \ Email!   7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

Re: Nested functions [was Re: valgrind]

[David Holland]

On Tue, Mar 22, 2022 at 09:16:36PM -0400, Mouse wrote:
 > Indeed, you can have different sizes for pointers to different object
 > types, too.  I _think_ pointers to different function types can have
 > different sizes, but I'm less certain of that.  (There would be little
 > point, since all function pointer types have to have the same
 > information content; see below.)

I don't think they can, but I wouldn't swear to it and, as you note,
it's a moot point.

 > > However there are, I fear, too many programs that somewhere convert a
 > > function pointer to (void *) and at that point things break.
 > 
 > That is a bug.  It always has been.  Such code is broken and it
 > deserves to be rednered _obviously_ broken so it can get fixed.

...and it already doesn't work on some platforms. I don't think
there's actually that much of this and finding it is a fairly
straightforward static analysis problem.

That said, moving to fat function pointers on machines that don't
already use them is a real ABI change and therefore a big deal; but it
could be done if there were a compelling argument to justify going
through all the associated dark rituals.

The conclusion over the past ~25 years has been that there isn't;
qsort and things like it work "well enough" and real uses for closures
that really motivate the feature come up rarely enough that it doesn't
happen. And that's why we are where we are.

(More than ~25 years ago people probably cared, or purported to care,
about an extra 4 bytes per function pointer enough to consider it not
worthwhile on those grounds.)

There is no solution based on trampolines that'll pass security (or at
least security-theatre) muster. Unless maybe by doing something that's
horrifying in other ways.

(For example: you could declare a static limit on how many instances
of the closure you'll ever produce, make a global array to stuff the
data pointer in, and statically generate N trampoline entry points
that read from that array and call the primary function. But there are
many other ways in which this is horrible.)

-- 
David A. Holland
dholl...@netbsd.org

Re: pcc [was Re: valgrind]

[Anders Magnusson]

Den 2022-03-23 kl. 21:55, skrev Greg A. Woods:

At Wed, 23 Mar 2022 20:56:27 +0100, Anders Magnusson  
wrote:
Subject: Re: pcc [was Re: valgrind]

Den 2022-03-23 kl. 19:37, skrev Greg A. Woods:

Heh.  I would say PCC's generated code doesn't compare to either modern
GCC or LLVM/Clang's output.

I would say the main reason is PCC doesn't (as far as I know) employ
any "Undefined Behaviour" caveat to optimize code, for example.

I'll let the reader decide which might have the "higher" quality.

I would really want to know what you base these three statements on?

Well I've read a great deal of PDP-11 assembler as produced by PCC, and
I've fought with LLVM/Clang (and to a lesser extent with GCC) and their
undefined behaviour sanitizers (and valgrind) when trying to port old
code to these new compilers and to understand what they have done to it.

I also have way more experience than I ever really wanted in finding
bugs in a wide variety of compilers that are effectively from the same
era as PCC (e.g. especially Lattice C and early Microsoft C, which as I
recall started life as Lattice C).

Modern optimizers that take advantage of UB to do their thing can cause
very strange bugs (hidden bugs, when the UB sanitizer isn't used),
especially with legacy code, or indeed with modern code written by naive
programmers.

Note of course that I'm explicitly _not_ talking about the quality of
the _input_ code, but of the generated assembler code, and I'm assuming
that's what Paul was asking about.

One thing I don't have a good feel for though is how the code produced
by modern GCC and LLVM/Clang looks when they are told to "leave it
alone" after the first step, i.e. with "-O0", and especially as compared
to PCC with -O0.  I _think_ they should be about the same, but I dunno.
Although older compilers like PCC are very naive and simplistic in how
they generate code, my feeling is that modern compilers are even more
naive in their first step of code generation as they have come to rely
even more on their own optimizers to clean things up.  That's pure
speculation though -- I haven't worked directly with assembler code very
much at all since I left the likes of the 6502 and 8086 behind.

I get the feeling that you are talking about pcc as it were in 1979 or so?
You seem to be unaware that things have happened the last 40+ years... :-)

-- R

Re: pcc [was Re: valgrind]

[Greg A. Woods]

At Wed, 23 Mar 2022 20:56:27 +0100, Anders Magnusson  
wrote:
Subject: Re: pcc [was Re: valgrind]
>
> Den 2022-03-23 kl. 19:37, skrev Greg A. Woods:
> >
> > Heh.  I would say PCC's generated code doesn't compare to either modern
> > GCC or LLVM/Clang's output.
> >
> > I would say the main reason is PCC doesn't (as far as I know) employ
> > any "Undefined Behaviour" caveat to optimize code, for example.
> >
> > I'll let the reader decide which might have the "higher" quality.
>
> I would really want to know what you base these three statements on?

Well I've read a great deal of PDP-11 assembler as produced by PCC, and
I've fought with LLVM/Clang (and to a lesser extent with GCC) and their
undefined behaviour sanitizers (and valgrind) when trying to port old
code to these new compilers and to understand what they have done to it.

I also have way more experience than I ever really wanted in finding
bugs in a wide variety of compilers that are effectively from the same
era as PCC (e.g. especially Lattice C and early Microsoft C, which as I
recall started life as Lattice C).

Modern optimizers that take advantage of UB to do their thing can cause
very strange bugs (hidden bugs, when the UB sanitizer isn't used),
especially with legacy code, or indeed with modern code written by naive
programmers.

Note of course that I'm explicitly _not_ talking about the quality of
the _input_ code, but of the generated assembler code, and I'm assuming
that's what Paul was asking about.

One thing I don't have a good feel for though is how the code produced
by modern GCC and LLVM/Clang looks when they are told to "leave it
alone" after the first step, i.e. with "-O0", and especially as compared
to PCC with -O0.  I _think_ they should be about the same, but I dunno.
Although older compilers like PCC are very naive and simplistic in how
they generate code, my feeling is that modern compilers are even more
naive in their first step of code generation as they have come to rely
even more on their own optimizers to clean things up.  That's pure
speculation though -- I haven't worked directly with assembler code very
much at all since I left the likes of the 6502 and 8086 behind.

--
Greg A. Woods 

Kelowna, BC +1 250 762-7675   RoboHack 
Planix, Inc.  Avoncote Farms 


pgphG58pf3Aww.pgp
Description: OpenPGP Digital Signature

Re: pcc [was Re: valgrind]

[Anders Magnusson]

Den 2022-03-23 kl. 19:37, skrev Greg A. Woods:

At Tue, 22 Mar 2022 17:47:55 +, "Koning, Paul"  wrote:
Subject: Re: pcc [was Re: valgrind]


Out of curiosity: how does PCC code quality compare with that of
GCC and (for targets that it supports) Clang?

Heh.  I would say PCC's generated code doesn't compare to either modern
GCC or LLVM/Clang's output.

I would say the main reason is PCC doesn't (as far as I know) employ
any "Undefined Behaviour" caveat to optimize code, for example.

I'll let the reader decide which might have the "higher" quality.

I would really want to know what you base these three statements on?

-- R

Re: pcc [was Re: valgrind]

[Greg A. Woods]

At Tue, 22 Mar 2022 17:47:55 +, "Koning, Paul"  wrote:
Subject: Re: pcc [was Re: valgrind]
>
>
> Out of curiosity: how does PCC code quality compare with that of
> GCC and (for targets that it supports) Clang?

Heh.  I would say PCC's generated code doesn't compare to either modern
GCC or LLVM/Clang's output.

I would say the main reason is PCC doesn't (as far as I know) employ
any "Undefined Behaviour" caveat to optimize code, for example.

I'll let the reader decide which might have the "higher" quality.

--
Greg A. Woods 

Kelowna, BC +1 250 762-7675   RoboHack 
Planix, Inc.  Avoncote Farms 


pgp_oLwFqoKCk.pgp
Description: OpenPGP Digital Signature

Re: pcc [was Re: valgrind]

[Koning, Paul]

> On Mar 22, 2022, at 1:21 PM, Greg A. Woods  wrote:
> 
> At Mon, 21 Mar 2022 08:54:43 -0400 (EDT), Mouse  
> wrote:
> Subject: pcc [was Re: valgrind]
>> 
>>>> I've been making very-spare-time progress on building my own
>>>> compiler on and off for some years now; perhaps I'll eventually get
>>>> somewhere.  [...]
>>> Have you looked at pcc?  http://pcc.ludd.ltu.se/ and in our source
>>> tree in src/external/bsd/pcc .
>> 
>> No, I haven't.  I should - it may well end up being quicker to move an
>> existing compiler in the directions I want to go than to write my own.
> 
> ...
> I also really like PCC.  (I remember teething pains getting used to it
> back when it first replaced Ritchie C on my university's PDP-11/60, but
> once I actually used it for real code (i.e. assignments in those days),
> and soon on the Vax too, I really liked it.)
> 
> I'm really sad that I still cannot build NetBSD entirely with PCC as the
> native and only default compiler.

Out of curiosity: how does PCC code quality compare with that of GCC and (for 
targets that it supports) Clang?

paul

Nested functions [was Re: valgrind]

[Mouse]

>> Can't you?  Does C require function pointers to have the same type,
>> or compatible structure, as data pointers?
> No, I don't think that it does.

Correct.

> You could have different sizes for those.

Indeed, you can have different sizes for pointers to different object
types, too.  I _think_ pointers to different function types can have
different sizes, but I'm less certain of that.  (There would be little
point, since all function pointer types have to have the same
information content; see below.)

> However there are, I fear, too many programs that somewhere convert a
> function pointer to (void *) and at that point things break.

That is a bug.  It always has been.  Such code is broken and it
deserves to be rednered _obviously_ broken so it can get fixed.  I can
understand commercial compiler vendors' reluctance to "break customer
code" even if the reality is more "pointing out where customer code was
already broken and just historically getting away with it".  But I have
more trouble understanding it for things like gcc.

But then, there is a lot of confusion around code portability.  There's
at least one relatively popular open-source project - SQLite, fuzzy
memory says - that has a FAQ list entry that goes something like
"$PROJECT provokes these warnings!" with a response that used to read
like "we test it heavily, the code is fine", not understanding that the
warnings are not so much about the code generated today on today's
machines, which they are correct that testing can address; it is about
tomorrow's architecture and/or tomorrow's new compiler release, or new
compiler, getting the code correct so it will work there too.

> And there is not really a "generic function pointer type" that you
> could sensibly use instead, I think.

There is, actually.  Any function pointer type will do.  Any function
pointer can be cast to any other function pointer type and back without
change - at least as of C99; "A pointer to a function of one type may
be converted to a pointer to a function of another type and back again;
the result shall compare equal to the original pointer.".  What you
can't do is call through the pointer when it points to the wrong
function type: "If a converted pointer is used to call a function whose
type is not compatible with the pointed-to type, the behavior is
undefined.".

> Possibly, a trampoline could be created on the heap, and then made
> executable and un-writable. Maybe that's considered too complicated /
> system dependent / expensive by gcc?

(1) it is ugly for the runtime to be mallocing behind the scenes, (2)
this breaks in the presence of longjmp (it is hard to stop it from
leaking trampolines when longjmping through a stack frame that created
a trampoline), and (3) it is difficult to use safely in the presence of
signal pointers or threads.  Oh, and (4) yes, changing memory
protection is system-dependent and expensive, and there are even some
Harvard(ish) architectures on which generating new executable code at
runtime is architecturally impossible.  (Few to none of them are
targeted by gcc, I suspect.)

(2) could be addressed if someone were to design an unwind-protect for
longjmp (which arguably should have existed all along), but that brings
it back to the "gcc wants to be compatible with existing systems"
issue; remember, back when gcc arose it usually wasn't the system
compiler, so it had to be compatible.  Arguably, now that gcc at least
sometimes _is_ the system compiler, it would make sense for it to grow
a way to handle nested functions better.  But I suspect there is
comparatively little will to do that in the gcc crowd, or it would have
happened long since.  And the heap techniques still run into (3); I
think fat function pointers is the rightest way to do it when
compatibility with a preexisting ABI is not a concern.

To bring this back to NetBSD, I ran into (4) myself back sometime in
2000-2005ish, I think it was.  There were two issues I ran into; I
can't recall which one was first.  One was that some new NetBSD release
brought in totally non-executable stack on at least one of the
architectures I ran, making it impossible to use gcc nested functions
at all.  That I had to fix in the kernel.  The other was that gcc's
configuration made at least one syscall per trampoline generated.  The
results were correct, but it was intolerably slow for at least one
program I cared about (I think it was a search program with a nested
function pointer being generated in a relatively deeply nested
routine).  I forget what I did there - just made the whole stack
executable, maybe?

/~\ The ASCII Mouse
\ / Ribbon Campaign
 X  Against HTMLmo...@rodents-montreal.org
/ \ Email!   7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

Re: valgrind

[Joerg Sonnenberger]

Am Tue, Mar 22, 2022 at 09:01:19PM + schrieb RVP:
> On Tue, 22 Mar 2022, Joerg Sonnenberger wrote:
> 
> > Am Mon, Mar 21, 2022 at 11:09:41PM + schrieb RVP:
> > > Sanitizers are OK, but, they don't seem to work in some cases:
> > 
> > Neither case is a memory leak. They are both reachable memory
> > allocations.
> > 
> 
> Yah: Reachable is what valgrind reports too, but, it still flags
> it as "lost". My point being that I'm glad we have sanitizers, but,
> you still need valgrind for some common use-cases.

Yeah and it regulary results in "bug reports" from the majority of
people that don't understand the difference. That said, I think
draconian mode of gperftools's heap cheaper can provide the check you
want here. Just be aware that it is pure noise most of the time.

Joerg

Re: valgrind

[Koning, Paul]

> On Mar 22, 2022, at 5:11 PM, Rhialto  wrote:
> 
> On Tue 22 Mar 2022 at 20:59:05 +, Koning, Paul wrote:
> ...
>> If it does, that would mean you can't get a pointer to a nested
>> function, which is no different from the C++ rule that you can't get a
>> (plain function) pointer to a member function.
> 
> True. But the trampoline function on the stack are only needed to create
> a pointer to a nested function. I don't think there is any issue if you
> never create such pointers. But that would make nested functions a lot
> less useful. You couldn't use one with qsort(3) for instance (as the
> comparison function), which is exactly the sort of scenario that would
> be perfect for a nested function.

Yes, though in that case the pointer is of a specific type, so you don't
need what you pointed out as missing, the generic function pointer.

> Possibly, a trampoline could be created on the heap, and then made
> executable and un-writable. Maybe that's considered too complicated /
> system dependent / expensive by gcc?

Perhaps.  But the bigger issue is that the heap isn't normally 
executable either, and all the same arguments for why the stack
should not be executable apply to the heap also.

paul

Re: valgrind

[Rhialto]

On Tue 22 Mar 2022 at 20:59:05 +, Koning, Paul wrote:
> > But these sorts of strategies require bigger function pointers, which
> > you basically can't do in C.
> 
> Can't you?  Does C require function pointers to have the same type, or
> compatible structure, as data pointers?

No, I don't think that it does. You could have different sizes for
those. However there are, I fear, too many programs that somewhere
convert a function pointer to (void *) and at that point things break.
And there is not really a "generic function pointer type" that you could
sensibly use instead, I think.

> If it does, that would mean you can't get a pointer to a nested
> function, which is no different from the C++ rule that you can't get a
> (plain function) pointer to a member function.

True. But the trampoline function on the stack are only needed to create
a pointer to a nested function. I don't think there is any issue if you
never create such pointers. But that would make nested functions a lot
less useful. You couldn't use one with qsort(3) for instance (as the
comparison function), which is exactly the sort of scenario that would
be perfect for a nested function.

Possibly, a trampoline could be created on the heap, and then made
executable and un-writable. Maybe that's considered too complicated /
system dependent / expensive by gcc?

>   paul
-Olaf.
-- 
___ "Buying carbon credits is a bit like a serial killer paying someone else to
\X/  have kids to make his activity cost neutral." -The BOFHfalu.nl@rhialto


signature.asc
Description: PGP signature

Re: valgrind

[RVP]

On Tue, 22 Mar 2022, Joerg Sonnenberger wrote:


Am Mon, Mar 21, 2022 at 11:09:41PM + schrieb RVP:

Sanitizers are OK, but, they don't seem to work in some cases:


Neither case is a memory leak. They are both reachable memory
allocations.



Yah: Reachable is what valgrind reports too, but, it still flags
it as "lost". My point being that I'm glad we have sanitizers, but,
you still need valgrind for some common use-cases.

-RVP

Re: valgrind

[Koning, Paul]

> On Mar 22, 2022, at 4:53 PM, Rhialto  wrote:
> 
> On Tue 22 Mar 2022 at 17:46:49 +, Koning, Paul wrote:
>> I don't believe ALGOL implementations needed executable stacks to implement 
>> nested functions, for example.
> 
> No, a common way to do it (as in the Dragon Book), if I recall
> correctly, the address of the function combined with a stack frame
> pointer. And nested functions keep a static link to the lexically
> enclosing scope, or alternatively a "display". I'm not sure if A68 would
> automatically move locals from surrounding scopes into the heap (like in
> a closure); it could well be that there was run-time checking instead if
> you tried to use a function pointer outside of the scope of any of the
> locals it used.
> 
> But these sorts of strategies require bigger function pointers, which
> you basically can't do in C.

Can't you?  Does C require function pointers to have the same type, or 
compatible structure, as data pointers?

If it does, that would mean you can't get a pointer to a nested function, which 
is no different from the C++ rule that you can't get a (plain function) pointer 
to a member function.

paul

Re: valgrind

[Rhialto]

On Tue 22 Mar 2022 at 21:53:37 +0100, Rhialto wrote:
> No, a common way to do it (as in the Dragon Book), if I recall
> correctly, the address of the function combined with a stack frame
> pointer. And nested functions keep a static link to the lexically
> enclosing scope, or alternatively a "display".

https://pages.cs.wisc.edu/~fischer/cs536.s06/course.hold/html/NOTES/8.RUNTIME-VAR-ACCESS.html
explains both methods briefly, for those interested.

-Olaf.
-- 
___ "Buying carbon credits is a bit like a serial killer paying someone else to
\X/  have kids to make his activity cost neutral." -The BOFHfalu.nl@rhialto


signature.asc
Description: PGP signature

Re: valgrind

[Rhialto]

On Tue 22 Mar 2022 at 17:46:49 +, Koning, Paul wrote:
> I don't believe ALGOL implementations needed executable stacks to implement 
> nested functions, for example.

No, a common way to do it (as in the Dragon Book), if I recall
correctly, the address of the function combined with a stack frame
pointer. And nested functions keep a static link to the lexically
enclosing scope, or alternatively a "display". I'm not sure if A68 would
automatically move locals from surrounding scopes into the heap (like in
a closure); it could well be that there was run-time checking instead if
you tried to use a function pointer outside of the scope of any of the
locals it used.

But these sorts of strategies require bigger function pointers, which
you basically can't do in C.

>   paul
-Olaf.
-- 
___ "Buying carbon credits is a bit like a serial killer paying someone else to
\X/  have kids to make his activity cost neutral." -The BOFHfalu.nl@rhialto


signature.asc
Description: PGP signature

Re: nested functions [was Re: valgrind]

[Koning, Paul]

> On Mar 22, 2022, at 2:23 PM, Mouse  wrote:
> 
> 
> [EXTERNAL EMAIL] 
> 
>>> I found an interesting article about why they're bad...
>>> https://urldefense.com/v3/__https://thephd.dev/lambdas-nested-functions-block-expressions-oh-my__;!!LpKI!1zB1gatUTEiM-j9CQ_6N-NWd4jS8UvW5iTSMRgW1tEyW_mK0mG2pU48LnUwJJBvA$
>>>  [thephd[.]dev]
>> That's a good argument for why GCC's implementation of nested functions is b$
> 
> What security blunder is that?  Based on your next line, I'm going to
> assume it's "implementing them via stack trampolines".  (I would have
> to go to a work machine to tell, because thephd.dev has apparently
> drunk the "it's good to ram HTTPS down everyone's throat" koolaid.
> Even the stack trampoline mechanism, I would say, is not a security
> blunder per se; I see it as a security issue only in that it
> exacerbates the effects of certain other security issues.  Also don't
> forget that early gcc arose in a very different environment from
> today's.)

Yes, I did mean executable stacks.  True, the world was different
once, though executable stacks are also problematic on some older
architectures for entirely different reasons.  For example, you can't
necessarily use them on PDP-11s.

>> I don't believe ALGOL implementations needed executable stacks to implement $
> 
> Neither would gcc...IF it can set the ABI.  There really was very
> little choice for gcc when it started.  It had to be ABI-compatible
> with existing procedure calling sequences.  It also had to be
> compatible with existing longjmps. That eliminates pretty close to
> everything _but_ stack trampolines.

True.  Do general ABIs like the VAX one have this issue or is it 
specific to just some of the ABIs?

> Other ways of doing nested functions is one of the things I want to
> experiment with.

I know the classic answer in ALGOL is "displays", which go all the
way back to the first ALGOL compilers, but I don't have the details
in my head.

paul

Re: pcc [was Re: valgrind]

[bch]

On Tue, Mar 22, 2022 at 10:21 Greg A. Woods  wrote:

> At Mon, 21 Mar 2022 08:54:43 -0400 (EDT), Mouse 
> wrote:
> Subject: pcc [was Re: valgrind]
> >
> > >> I've been making very-spare-time progress on building my own
> > >> compiler on and off for some years now; perhaps I'll eventually get
> > >> somewhere.  [...]
> > > Have you looked at pcc?  http://pcc.ludd.ltu.se/ and in our source
> > > tree in src/external/bsd/pcc .
> >
> > No, I haven't.  I should - it may well end up being quicker to move an
> > existing compiler in the directions I want to go than to write my own.
>
> I would like to add my voice too.
>
> I _really_ like valgrind.  It is immensely valuable and infinitely
> better than any of the compiler so-called "sanitizers" (except maybe the
> Undefined Behaviour sanitizer in Clang, which, sadly, is a necessary
> evil if one is to use such a modern language bastardizer like Clang).
>
> It's a little ugly to use, and it's a very tough task-master, but I'm
> really sad that I cannot use it easily and regularly on NetBSD.
>
> (I'm just about as sad that it no longer works on modern macOS either.)
>
>
> I also really like PCC.  (I remember teething pains getting used to it
> back when it first replaced Ritchie C on my university's PDP-11/60, but
> once I actually used it for real code (i.e. assignments in those days),
> and soon on the Vax too, I really liked it.)
>
> I'm really sad that I still cannot build NetBSD entirely with PCC as the
> native and only default compiler.
>


Is there a case for another concerted  campaign on pcc like ragge@ did
those years ago?

-bch


>
> I would like to do whatever I can to help fix these two problems, but
> I'm not anywhere near able to even begin to fix them on my own (well I
> could hack on NetBSD to make it work with PCC, but I don't want to be a
> one-man-band on a project of that scale).
>
> --
> Greg A. Woods 
>
> Kelowna, BC +1 250 762-7675   RoboHack 
> Planix, Inc.  Avoncote Farms 
>

nested functions [was Re: valgrind]

[Mouse]

>> I found an interesting article about why they're bad...
>> https://thephd.dev/lambdas-nested-functions-block-expressions-oh-my
> That's a good argument for why GCC's implementation of nested functions is b$

What security blunder is that?  Based on your next line, I'm going to
assume it's "implementing them via stack trampolines".  (I would have
to go to a work machine to tell, because thephd.dev has apparently
drunk the "it's good to ram HTTPS down everyone's throat" koolaid.
Even the stack trampoline mechanism, I would say, is not a security
blunder per se; I see it as a security issue only in that it
exacerbates the effects of certain other security issues.  Also don't
forget that early gcc arose in a very different environment from
today's.)

> I don't believe ALGOL implementations needed executable stacks to implement $

Neither would gcc...IF it can set the ABI.  There really was very
little choice for gcc when it started.  It had to be ABI-compatible
with existing procedure calling sequences.  It also had to be
compatible with existing longjmps. That eliminates pretty close to
everything _but_ stack trampolines.

Other ways of doing nested functions is one of the things I want to
experiment with.

/~\ The ASCII Mouse
\ / Ribbon Campaign
 X  Against HTMLmo...@rodents-montreal.org
/ \ Email!   7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

Re: valgrind

[Koning, Paul]

> On Mar 22, 2022, at 1:17 PM, Rhialto  wrote:
> 
> On Sun 20 Mar 2022 at 23:17:57 -0400, Mouse wrote:
>> clang is - or at least was last I checked - under the impression that
>> nested functions are little-used and thus are not worth supporting.
> 
> I guess they can think so because nested functions are a gcc extension.
> Quite a useful one, I think, having been taught programming with Algol
> 68 which of course does have nested functions; but an extension
> nevertheless.
> 
> I found an interesting article about why they're bad...
> https://thephd.dev/lambdas-nested-functions-block-expressions-oh-my

That's a good argument for why GCC's implementation of nested functions is bad. 
 It doesn't mean other implementations that avoid the security blunder GCC made 
are bad.

I don't believe ALGOL implementations needed executable stacks to implement 
nested functions, for example.

paul

Re: pcc [was Re: valgrind]

[Greg A. Woods]

At Mon, 21 Mar 2022 08:54:43 -0400 (EDT), Mouse  
wrote:
Subject: pcc [was Re: valgrind]
>
> >> I've been making very-spare-time progress on building my own
> >> compiler on and off for some years now; perhaps I'll eventually get
> >> somewhere.  [...]
> > Have you looked at pcc?  http://pcc.ludd.ltu.se/ and in our source
> > tree in src/external/bsd/pcc .
>
> No, I haven't.  I should - it may well end up being quicker to move an
> existing compiler in the directions I want to go than to write my own.

I would like to add my voice too.

I _really_ like valgrind.  It is immensely valuable and infinitely
better than any of the compiler so-called "sanitizers" (except maybe the
Undefined Behaviour sanitizer in Clang, which, sadly, is a necessary
evil if one is to use such a modern language bastardizer like Clang).

It's a little ugly to use, and it's a very tough task-master, but I'm
really sad that I cannot use it easily and regularly on NetBSD.

(I'm just about as sad that it no longer works on modern macOS either.)


I also really like PCC.  (I remember teething pains getting used to it
back when it first replaced Ritchie C on my university's PDP-11/60, but
once I actually used it for real code (i.e. assignments in those days),
and soon on the Vax too, I really liked it.)

I'm really sad that I still cannot build NetBSD entirely with PCC as the
native and only default compiler.


I would like to do whatever I can to help fix these two problems, but
I'm not anywhere near able to even begin to fix them on my own (well I
could hack on NetBSD to make it work with PCC, but I don't want to be a
one-man-band on a project of that scale).

--
Greg A. Woods 

Kelowna, BC +1 250 762-7675   RoboHack 
Planix, Inc.  Avoncote Farms 


pgpIqUBNiiYDq.pgp
Description: OpenPGP Digital Signature

Re: valgrind

[Rhialto]

On Sun 20 Mar 2022 at 23:17:57 -0400, Mouse wrote:
> clang is - or at least was last I checked - under the impression that
> nested functions are little-used and thus are not worth supporting.

I guess they can think so because nested functions are a gcc extension.
Quite a useful one, I think, having been taught programming with Algol
68 which of course does have nested functions; but an extension
nevertheless.

I found an interesting article about why they're bad...
https://thephd.dev/lambdas-nested-functions-block-expressions-oh-my

-Olaf.
-- 
___ "Buying carbon credits is a bit like a serial killer paying someone else to
\X/  have kids to make his activity cost neutral." -The BOFHfalu.nl@rhialto


signature.asc
Description: PGP signature

Re: valgrind

[Mouse]

> PS. Mouse's email server will, as usual, reject mails sent from SDF I
> think... ;)

I had a look at my logs.

This is true at present, but it's mx.sdf.org's fault.  It's dropping
the connection 50 seconds into my banner delay, a point at which the
SMTP specs say any client-side timeout should be at least five minutes.

/~\ The ASCII Mouse
\ / Ribbon Campaign
 X  Against HTMLmo...@rodents-montreal.org
/ \ Email!   7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

Re: valgrind

[Joerg Sonnenberger]

Am Mon, Mar 21, 2022 at 11:09:41PM + schrieb RVP:
> Sanitizers are OK, but, they don't seem to work in some cases:

Neither case is a memory leak. They are both reachable memory
allocations.

Joerg

re: valgrind

[RVP]

On Mon, 21 Mar 2022, matthew green wrote:


try clang, which usually has newer/better sanitizers.



Sanitizers are OK, but, they don't seem to work in some cases:

Case 1: Global allocation
---
$ cat global_alloc.c
#include 
#include 

void* p;

int
main(void)
{
p = malloc(1000);
memset(p, '\0', 1000);
return 0;
}
$ clang --version
clang version 13.0.1
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/llvm/bin
$ clang -Wall -g -fsanitize=address -o 1 global_alloc.c
$ env ASAN_OPTIONS=detect_leaks=1 ./1
$ 
---




Case 2: Static allocation
---
$ cat static_alloc.c
#include 
#include 

static char*
mkbuf(size_t len)
{
static void* sbuf = NULL;
static size_t slen = 0;

if (len > slen) {
sbuf = realloc(sbuf, len);
slen = len;
}
return sbuf;
}

int
main(void)
{
char* p = mkbuf(1000);
memset(p, '\0', 1000);
return 0;
}
$ clang -Wall -g -fsanitize=address -o 2 static_alloc.c
$ env ASAN_OPTIONS=detect_leaks=1 ./2
$ 
---


Valgrind, of course, detects both those leaks.

-RVP

PS. Mouse's email server will, as usual, reject mails sent from SDF I
think... ;)

Re: pcc [was Re: valgrind]

[tlaronde]

Le Mon, Mar 21, 2022 at 08:54:43AM -0400, Mouse a écrit :
> >> I've been making very-spare-time progress on building my own
> >> compiler on and off for some years now; perhaps I'll eventually get
> >> somewhere.  [...]
> > Have you looked at pcc?  http://pcc.ludd.ltu.se/ and in our source
> > tree in src/external/bsd/pcc .
> 
> No, I haven't.  I should - it may well end up being quicker to move an
> existing compiler in the directions I want to go than to write my own.
> 

And FWIW, there is also the collection of compilers in Plan9, that has
now been released under the MIT license: https://p9f.org/ (Plan9
foundation).
-- 
Thierry Laronde 
 http://www.kergis.com/
http://kertex.kergis.com/
   http://www.sbfa.fr/
Key fingerprint = 0FF7 E906 FBAF FE95 FD89  250D 52B1 AE95 6006 F40C

pcc [was Re: valgrind]

[Mouse]

>> I've been making very-spare-time progress on building my own
>> compiler on and off for some years now; perhaps I'll eventually get
>> somewhere.  [...]
> Have you looked at pcc?  http://pcc.ludd.ltu.se/ and in our source
> tree in src/external/bsd/pcc .

No, I haven't.  I should - it may well end up being quicker to move an
existing compiler in the directions I want to go than to write my own.

Thank you!

/~\ The ASCII Mouse
\ / Ribbon Campaign
 X  Against HTMLmo...@rodents-montreal.org
/ \ Email!   7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

GPLv3 [was Re: valgrind]

[Mouse]

>> But it's less a lack of interest and more an unwillingness to ignore
>> the licensing issues.  `Modern' GCC is licensed under the GPLv3.
> So?  That doesn't affect the licensing of code compiled by gcc.

Doesn't it?  I can't tell - that's, basically, the problem.  Your
opinion that it doesn't is noted, but, unless you are currently a
member of the Ontario bar, that opinion is of minimal relevance to me.

Furthermore, licensing of code compiled by gcc is only part of the
question.  The rest of the question is whether I'm allowed to use gcc,
to have it produce that compiled code, at all.  That too I can't tell.

/~\ The ASCII Mouse
\ / Ribbon Campaign
 X  Against HTMLmo...@rodents-montreal.org
/ \ Email!   7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

Re: valgrind

[Simon Burge]

Mouse wrote:

> I've been making very-spare-time progress on building my own compiler
> on and off for some years now; perhaps I'll eventually get somewhere.
> (It's something I want to do regardless; there are some directions I'd
> like to take the language to see what results)

Have you looked at pcc?  http://pcc.ludd.ltu.se/ and in our source tree
in src/external/bsd/pcc .

Cheers,
Simon.

Re: valgrind

[Koning, Paul]

> On Mar 20, 2022, at 7:58 PM, Mouse  wrote:
> 
> 
> [EXTERNAL EMAIL] 
> 
>>> Perhaps I just need a better approach
> 
>> I know you've previously expressed a lack of interest in such things
>> when we've talked previously, but I've found the built-in sanitizers
>> in modern GCC useful for approximating the functionality of valgrind
>> on NetBSD.
> 
> To some extent, perhaps.
> 
> But it's less a lack of interest and more an unwillingness to ignore
> the licensing issues.  `Modern' GCC is licensed under the GPLv3.

So?  That doesn't affect the licensing of code compiled by gcc.

paul

Re: valgrind

[Mouse]

>> But it's less a lack of interest and more an unwillingness to ignore
>> the licensing issues.  `Modern' GCC is licensed under the GPLv3.
> try clang, which usually has newer/better sanitizers.

clang is - or at least was last I checked - under the impression that
nested functions are little-used and thus are not worth supporting.

That may be - presumably is; the clang people are not stupid - correct
as applied to the C ecosystem at large.  It definitely is not correct
as applied to my code.  Unless/until they change their collective mind
and implement nested functions, clang is a nonstarter for me.

Also, at work, I had occasion to do a clang build-from-source.  The
resulting executables were multiple _giga_bytes:

[Pavilion] 14> ls -l bin/clang-10
-rwxr-xr-x  1 root 2535346040 Aug 10  2020 bin/clang-10

That too is a showstopper for me.  They're turned off at the moment,
but I have two machines, one in live use when I'm working on what I set
it up for and the other a live use desktop until COVID-19 put paid to
my working outside my home, for which that one file would take up more
than half the machine's total available disk space.

I've been making very-spare-time progress on building my own compiler
on and off for some years now; perhaps I'll eventually get somewhere.
(It's something I want to do regardless; there are some directions I'd
like to take the language to see what results)

/~\ The ASCII Mouse
\ / Ribbon Campaign
 X  Against HTMLmo...@rodents-montreal.org
/ \ Email!   7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

re: valgrind

[matthew green]

> But it's less a lack of interest and more an unwillingness to ignore
> the licensing issues.  `Modern' GCC is licensed under the GPLv3.  I

try clang, which usually has newer/better sanitizers.

Re: valgrind

[Mouse]

>> Perhaps I just need a better approach

> I know you've previously expressed a lack of interest in such things
> when we've talked previously, but I've found the built-in sanitizers
> in modern GCC useful for approximating the functionality of valgrind
> on NetBSD.

To some extent, perhaps.

But it's less a lack of interest and more an unwillingness to ignore
the licensing issues.  `Modern' GCC is licensed under the GPLv3.  I
have tried twice to understand that license and people tell me I've
still missed major pieces.  If I have to hire a lawyer to figure out
the license, it's not suitable for my purposes - in my opinion not
suitable for any open-source work, actually - no matter what it turns
out to permit.

(As this implies, my experience with valgrind has, of course, been
entirely at work.)

/~\ The ASCII Mouse
\ / Ribbon Campaign
 X  Against HTMLmo...@rodents-montreal.org
/ \ Email!   7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

Re: valgrind

[nia]

On Wed, Mar 09, 2022 at 08:39:06PM -0500, Mouse wrote:
> Perhaps I just need a better approach

I know you've previously expressed a lack of interest in such
things when we've talked previously, but I've found the built-in
sanitizers in modern GCC useful for approximating the functionality
of valgrind on NetBSD.

For example, -fsanitize=leak can log leaks at runtime.