Re: Creative ideas on how to debug heap corruption

[Moritz Angermann]

Thanks everyone. I have indeed been trying to get somewhere with sanity
checking. That used to help quite a bit for the deadstripping stuff that
happened on iOS a long time ago, but that was also much more deterministic.
Maybe I'll try to see if running it through qemu will give me some more
determinism. That at least gives somewhat predictable allocations. It could
still end up being some annoying memory ordering issues, the llvm backend
just managed to happen to not run into by luck, or optimisation passes.

On Mon, Aug 31, 2020 at 10:29 PM Csaba Hruska 
wrote:

> Fuzzing:
>
>1. generate simple random stg programs
>2. compile and run with RTS sanity checking enabled
>3. compare the program result between different backends
>
> The fuzzer should cover all codegen cases and all code in RTS. Maybe this
> could be checked by the existing tools.
>
> On Mon, Aug 31, 2020 at 4:19 PM George Colpitts 
> wrote:
>
>> +Moritz
>>
>> On Mon, Aug 31, 2020 at 11:17 AM George Colpitts <
>> george.colpi...@gmail.com> wrote:
>>
>>> I assume you're familiar with the following from
>>> https://www.aosabook.org/en/ghc.html and that this facility is still
>>> there. Just in case you are not:
>>>
>>> So, the debug RTS has an optional mode that we call *sanity checking*.
>>> Sanity checking enables all kinds of expensive assertions, and can make the
>>> program run many times more slowly. In particular, sanity checking runs a
>>> full scan of the heap to check for dangling pointers (amongst other
>>> things), before *and* after every GC. The first job when investigating
>>> a runtime crash is to run the program with sanity checking turned on;
>>> sometimes this will catch the invariant violation well before the program
>>> actually crashes.
>>>
>>>
>>> On Mon, Aug 31, 2020 at 11:08 AM Csaba Hruska 
>>> wrote:
>>>
 Dump the whole heap into file during GC traversal or taking the whole
 allocated area. hmm, maybe this is the same as core dump.

 On Mon, Aug 31, 2020 at 11:00 AM Ben Lippmeier 
 wrote:

>
>
> > On 31 Aug 2020, at 5:54 pm, Moritz Angermann <
> moritz.angerm...@gmail.com> wrote:
> >
> > If anyone has some create ideas, I'd love to hear them.  I've been
> wondering
> > if just logging allocations (offset, range, type) would help
> figuring out what we
> > expected to be there; and then maybe try to break on the allocation,
> (and
> > subsequent writes).
> >
> > I'm sure some have been down this road before.
>
> Force a GC before every allocation, and make the GC check the validity
> of the objects before it moves anything. I think this used to be possible
> by compiling the runtime system in debug mode.
>
> The usual pain of heap corruption is that once the heap is corrupted
> it may be several GC cycles before you get the actual crash, and in the
> meantime the objects have all been moved around. The GC walks over all the
> objects by nature, so get it to validate the heap every time it does, then
> force it to run as often as you possibly can.
>
> A user space approach is to use a library like vacuum or packman that
> also walks over the heap objects directly.
>
> http://hackage.haskell.org/package/vacuum-2.2.0.0/docs/GHC-Vacuum.html
> https://hackage.haskell.org/package/packman
>
> Ben.
>
>
>
> ___
> ghc-devs mailing list
> ghc-devs@haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
 ___
 ghc-devs mailing list
 ghc-devs@haskell.org
 http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

>>>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: COMPLETE pragmas

[Edward Kmett]

I'd be over the moon with happiness if I could hang COMPLETE pragmas on
polymorphic types.

I have 3 major issues with COMPLETE as it exists.

1.) Is what is mentioned here:

Examples for me come up when trying to build a completely unboxed 'linear'
library using backpack. In the end I want/need to supply a pattern synonym
that works over, say, all the 2d vector types, extracting their elements,
but right now I just get spammed by incomplete coverage warnings.

type family Elem t :: Type
class D2 where
  _V2 :: Iso' t (Elem t, Elem t)

pattern V2 :: D2 t => Elem t -> Elem t -> t
pattern V2 a b <- (view _V2 -> (a,b)) where
  V2 a b = review _V2 (a,b)

There is no way to hang a COMPLETE pragma on that.

2.) Another scenario that I'd really love to see supported with COMPLETE
pragmas is a way to use | notation with them like you can with MINIMAL
pragmas.

If you make smart constructors for a dozen constructors in your term type
(don't judge me!), you wind up needing 2^12 COMPLETE pragmas to describe
all the ways you might mix regular and start constructors today.

{# COMPLETE (Lam | LAM), (Var | VAR), ... #-}

would let you get away with a single such definition. This comes up when
you have some kind of monoid that acts on terms and you want to push it
down through
the syntax tree invisibly to the user. Explicit substitutions, shifts in
position in response to source code edits, etc.

3.) I had one other major usecase where I failed to be able to use a
COMPLETE pragma:

type Option a = (# a | (##) #)

pattern Some :: a -> Option a
pattern Some a = (# a | #)

pattern None :: Option a
pattern None = (# | (##) #)

{-# COMPLETE Some, None #-}

These worked _within_ a module, but was forgotten across module boundaries,
which forced me to rather drastically change the module structure of a
package, but it sounds a lot like the issue being discussed. No types to
hang it on in the interface file. With the ability to define unlifted
newtypes I guess this last one is less of a concern now?

-Edward

On Mon, Aug 31, 2020 at 2:29 PM Richard Eisenberg  wrote:

> Hooray Sebastian!
>
> Somehow, I knew cluing you into this conundrum would help find a solution.
> The approach you describe sounds quite plausible.
>
> Yet: types *do* matter, of course. So, I suppose the trick is this: have
> the COMPLETE sets operate independent of types, but then use types in the
> PM-checker when determining impossible cases? And, about your idea for
> having pattern synonyms store pointers to their COMPLETE sets: I think data
> constructors can also participate. But maybe there is always at least one
> pattern synonym (which would be a reasonable restriction), so I guess you
> can look at the pattern-match as a whole and use the pattern synonym to
> find the relevant COMPLETE set(s).
>
> Thanks for taking a look!
> Richard
>
> On Aug 31, 2020, at 4:23 PM, Sebastian Graf  wrote:
>
> Hi Richard,
>
> Am Mo., 31. Aug. 2020 um 21:30 Uhr schrieb Richard Eisenberg <
> r...@richarde.dev>:
>
>> Hi Sebastian,
>>
>> I enjoyed your presentation last week at ICFP!
>>
>
> Thank you :) I'm glad you liked it!
>
> This thread (
>> https://ghc-devs.haskell.narkive.com/NXBBDXg1/suppressing-false-incomplete-pattern-matching-warnings-for-polymorphic-pattern-synonyms)
>> played out before you became so interested in pattern-match coverage. I'd
>> be curious for your thoughts there -- do you agree with the conclusions in
>> the thread?
>>
>
> I vaguely remember reading this thread. As you write there
> 
>
> And, while I know it doesn't work today, what's wrong (in theory) with
>>
>> {-# COMPLETE LL #-}
>>
>> No types! (That's a rare thing for me to extol...)
>>
>> I feel I must be missing something here.
>>
>
> Without reading the whole thread, I think that solution is very possible.
> The thread goes on to state that we currently attach COMPLETE sets to type
> constructors, but that is only an implementational thing. I asked Matt (who
> implemented it) somewhere and he said the only reason to attach it to type
> constructors was because it was the easiest way to implement serialisation
> to interface files.
>
> The thread also mentions that type-directed works better for the
> pattern-match checker. In fact I disagree; we have to thin out COMPLETE
> sets all the time anyway when new type evidence comes up, for example. It's
> quite a hassle to find all the COMPLETE sets of the type constructors a
> given type can be "represented" (I mean equality modulo type family
> reductions here) as. I'm pretty sure it's broken in multiple ways, as
> #18276  points out.
>
> Disregarding a bit of busy work for implementing serialisation to
> interface files, it's probably far simpler to give each COMPLETE set a
> Name/Unique and refer to them from the pattern synonyms that mention them
> 

Re: COMPLETE pragmas

[Richard Eisenberg]

Hooray Sebastian!

Somehow, I knew cluing you into this conundrum would help find a solution. The 
approach you describe sounds quite plausible.

Yet: types *do* matter, of course. So, I suppose the trick is this: have the 
COMPLETE sets operate independent of types, but then use types in the 
PM-checker when determining impossible cases? And, about your idea for having 
pattern synonyms store pointers to their COMPLETE sets: I think data 
constructors can also participate. But maybe there is always at least one 
pattern synonym (which would be a reasonable restriction), so I guess you can 
look at the pattern-match as a whole and use the pattern synonym to find the 
relevant COMPLETE set(s).

Thanks for taking a look!
Richard

> On Aug 31, 2020, at 4:23 PM, Sebastian Graf  wrote:
> 
> Hi Richard,
> 
> Am Mo., 31. Aug. 2020 um 21:30 Uhr schrieb Richard Eisenberg 
> mailto:r...@richarde.dev>>:
> Hi Sebastian,
> 
> I enjoyed your presentation last week at ICFP!
> 
> Thank you :) I'm glad you liked it!
> 
> This thread 
> (https://ghc-devs.haskell.narkive.com/NXBBDXg1/suppressing-false-incomplete-pattern-matching-warnings-for-polymorphic-pattern-synonyms
>  
> )
>  played out before you became so interested in pattern-match coverage. I'd be 
> curious for your thoughts there -- do you agree with the conclusions in the 
> thread?
> 
> I vaguely remember reading this thread. As you write there 
> 
> 
> And, while I know it doesn't work today, what's wrong (in theory) with
> 
> {-# COMPLETE LL #-}
> 
> No types! (That's a rare thing for me to extol...)
> 
> I feel I must be missing something here.
> 
> Without reading the whole thread, I think that solution is very possible. The 
> thread goes on to state that we currently attach COMPLETE sets to type 
> constructors, but that is only an implementational thing. I asked Matt (who 
> implemented it) somewhere and he said the only reason to attach it to type 
> constructors was because it was the easiest way to implement serialisation to 
> interface files.
> 
> The thread also mentions that type-directed works better for the 
> pattern-match checker. In fact I disagree; we have to thin out COMPLETE sets 
> all the time anyway when new type evidence comes up, for example. It's quite 
> a hassle to find all the COMPLETE sets of the type constructors a given type 
> can be "represented" (I mean equality modulo type family reductions here) as. 
> I'm pretty sure it's broken in multiple ways, as #18276 
>  points out.
> 
> Disregarding a bit of busy work for implementing serialisation to interface 
> files, it's probably far simpler to give each COMPLETE set a Name/Unique and 
> refer to them from the pattern synonyms that mention them (we'd have to get 
> creative for orphans, though). The relation is quite like between a type 
> class instance and the type in its head. A more worked example is here: 
> https://gitlab.haskell.org/ghc/ghc/-/issues/18277#note_287827 
> 
> 
> So, it's on my longer term TODO list to fix this.
> 
> 
> My motivation for asking is https://github.com/conal/linalg/pull/54 
>  (you don't need to read the whole 
> thing), which can be boiled down to a request for a COMPLETE pragma that 
> works at a polymorphic result type. (Or a COMPLETE pragma written in a module 
> that is not the defining module for a pattern synonym.) 
> https://gitlab.haskell.org/ghc/ghc/-/issues/14422 
>  describes a similar, but 
> even more challenging scenario.
> 
> I'll answer in the thread. (Oh, you also found #14422.) I think the approach 
> above will also fix #14422.
> 
> Do you see any ways forward here?
> . 
> 
> Thanks!
> Richard
> 
> Maybe I'll give it a try tomorrow. 

___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: COMPLETE pragmas

[Sebastian Graf]

Hi Richard,

Am Mo., 31. Aug. 2020 um 21:30 Uhr schrieb Richard Eisenberg <
r...@richarde.dev>:

> Hi Sebastian,
>
> I enjoyed your presentation last week at ICFP!
>

Thank you :) I'm glad you liked it!

This thread (
> https://ghc-devs.haskell.narkive.com/NXBBDXg1/suppressing-false-incomplete-pattern-matching-warnings-for-polymorphic-pattern-synonyms)
> played out before you became so interested in pattern-match coverage. I'd
> be curious for your thoughts there -- do you agree with the conclusions in
> the thread?
>

I vaguely remember reading this thread. As you write there


And, while I know it doesn't work today, what's wrong (in theory) with
>
> {-# COMPLETE LL #-}
>
> No types! (That's a rare thing for me to extol...)
>
> I feel I must be missing something here.
>

Without reading the whole thread, I think that solution is very possible.
The thread goes on to state that we currently attach COMPLETE sets to type
constructors, but that is only an implementational thing. I asked Matt (who
implemented it) somewhere and he said the only reason to attach it to type
constructors was because it was the easiest way to implement serialisation
to interface files.

The thread also mentions that type-directed works better for the
pattern-match checker. In fact I disagree; we have to thin out COMPLETE
sets all the time anyway when new type evidence comes up, for example. It's
quite a hassle to find all the COMPLETE sets of the type constructors a
given type can be "represented" (I mean equality modulo type family
reductions here) as. I'm pretty sure it's broken in multiple ways, as #18276
 points out.

Disregarding a bit of busy work for implementing serialisation to interface
files, it's probably far simpler to give each COMPLETE set a Name/Unique
and refer to them from the pattern synonyms that mention them (we'd have to
get creative for orphans, though). The relation is quite like between a
type class instance and the type in its head. A more worked example is
here: https://gitlab.haskell.org/ghc/ghc/-/issues/18277#note_287827

So, it's on my longer term TODO list to fix this.


> My motivation for asking is https://github.com/conal/linalg/pull/54 (you
> don't need to read the whole thing), which can be boiled down to a request
> for a COMPLETE pragma that works at a polymorphic result type. (Or a
> COMPLETE pragma written in a module that is not the defining module for a
> pattern synonym.) https://gitlab.haskell.org/ghc/ghc/-/issues/14422
> describes a similar, but even more challenging scenario.
>

I'll answer in the thread. (Oh, you also found #14422.) I think the
approach above will also fix #14422.

>
> Do you see any ways forward here?
>
.

>
> Thanks!
> Richard


Maybe I'll give it a try tomorrow.
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: HsPragTick

[Richard Eisenberg]

I'm content with ripping it out and waiting to see if anyone complains. Without 
documentation or any obvious users, it seems hard to do otherwise.

That said, if you reading this have a better idea than we do about who uses {-# 
GENERATED ... #-} pragmas, do speak up!

Thanks,
Richard

> On Aug 31, 2020, at 10:07 AM, Vladislav Zavialov  
> wrote:
> 
> I was under impression it was somehow related to HPC. Since I'm not 
> sufficiently familiar with HPC's inner workings, I kept it around just to be 
> safe.
> 
> - Vlad
> 
> On Mon, Aug 31, 2020, 17:03 Ryan Scott  > wrote:
> I think that HsPragTick is unused as of [1]. In fact, I was under the 
> impression that [1] removed HsPragTick entirely (as the commit message) would 
> suggest, but upon further inspection, that doesn't appear to be the case. 
> Vlad, do you recall why HsPragTick was kept around?
> 
> Ryan S.
> -
> [1] https://gitlab.haskell.org/ghc/ghc/-/merge_requests/2154 
> ___
> ghc-devs mailing list
> ghc-devs@haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

COMPLETE pragmas

[Richard Eisenberg]

Hi Sebastian,

I enjoyed your presentation last week at ICFP!

This thread 
(https://ghc-devs.haskell.narkive.com/NXBBDXg1/suppressing-false-incomplete-pattern-matching-warnings-for-polymorphic-pattern-synonyms)
 played out before you became so interested in pattern-match coverage. I'd be 
curious for your thoughts there -- do you agree with the conclusions in the 
thread?

My motivation for asking is https://github.com/conal/linalg/pull/54 (you don't 
need to read the whole thing), which can be boiled down to a request for a 
COMPLETE pragma that works at a polymorphic result type. (Or a COMPLETE pragma 
written in a module that is not the defining module for a pattern synonym.) 
https://gitlab.haskell.org/ghc/ghc/-/issues/14422 describes a similar, but even 
more challenging scenario.

Do you see any ways forward here?

Thanks!
Richard
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Implicit reboxing of unboxed tuple in let-patterns

[Richard Eisenberg]

> On Aug 31, 2020, at 10:34 AM, Spiwack, Arnaud  wrote:
> 
> That being said, Richard seemed to feel rather strongly about this one. 
> Richard, do you still agree with your then position that let (#x, y#) = … 
> being a lazy pattern (hence implicitly boxes the pair) is the right semantics?

I admit I don't feel as strongly any more. My argument in that thread was from 
the standpoint of a language designer: there is really no reason, a priori, for 
an unboxed-tuple binding to be strict. What controls strictness is whether the 
bound variables are of unlifted type. However, I'm currently in more sympathy 
with language users, who (for whatever reason) seem to think that bindings with 
#s in them should be strict. (I have this intuition myself, even though it's 
not quite warranted on technical grounds.)

What do we think of

> pattern Unl x y = (# x, y #)
> 
> ex1, ex2 :: ()
> ex1 = let Unl x y = Unl undefined undefined in ()
> ex2 = let Unl x y = undefined in ()


? Today, both ex1 and ex2 evaluate to (). If we were to change the 
specification here, would we consider any unlifted-type pattern (where the type 
of the pattern itself is unlifted, independent of the type of any of its bound 
variables) to be banged? Or would it be a super-special case for unboxed tuples?

Richard___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

WebUI for GHC/Haskell tooling (eventlog)

[Csaba Hruska]

Hello,

I've written a blog post  about my
WebUI based eventlog related tool.
It is also related to eventlog2html and ghc-debug.
I'm interested in your opinion and ideas related to ghc debug/profiling
tooling.
If you have time please read the post and it would be great to hear some
positive or negative feedback from you. It would be great to discuss this
topic with you.

Thanks,
Csaba
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Implicit reboxing of unboxed tuple in let-patterns

[John Cotton Ericson]

I haven't used unboxed tuples enough to perhaps feel the pain, but on 
paper the current design makes sense to me. The laziness of the binding 
is suppose to have to do with the runtime rep of the binding itself, not 
any enclosing pattern.


For example take

    {-# LANGUAGE ScopedTypeVariables #-}
    {-# LANGUAGE MagicHash #-}

    import GHC.Base

    data Foo = Foo Int# Int#

    main = pure ()
    where Foo x y = Foo undefined undefined

This program will fail even though x and y are unused.

While this principle may not match how this stuff is used in practice, 
the alternative of making the strictness of the bindings depend on more 
than their runtime reps seems less-local / more ad-hoc to me.


John

On 8/31/20 10:34 AM, Spiwack, Arnaud wrote:


I’ve been pointed to 
https://github.com/ghc-proposals/ghc-proposals/pull/35 where this was 
debated a few years ago. With much of the same arguments as today.


Simon Marlow said

making an unboxed tuple binding lazy by default seems to be
intuitively the wrong choice. I guarantee I would get tripped up
by this! Giving unboxed tuples an implicit bang seems reasonable
to me.

I can share that I got tripped by it. And so were other members of my 
team.


That being said, Richard seemed to feel rather strongly about this 
one. Richard, do you still agree with your then position that |let 
(#x, y#) = …| being a lazy pattern (hence implicitly boxes the pair) 
is the right semantics?



On Fri, Aug 28, 2020 at 8:26 PM chessai > wrote:


Arnaud,

I have dealt with this in the past and find the laziness extremely
counterintuitive and never wanted. Every time I have let-bound an
unboxed tuple, I have never wanted that boxing to occur. Perhaps
there is a good reason this is the case but I wish it would change.

On Fri, Aug 28, 2020, 08:26 Spiwack, Arnaud
mailto:arnaud.spiw...@tweag.io>> wrote:

Hi Carter,

We are using |let !(#x,y#) = …| actually. Having the strict
behaviour is not particularly difficult. You can even use
|case … of (#x, y#) ->…| directly, it’s not too bad. My
complaint, as it were, is solely about the potential for mistakes.


On Fri, Aug 28, 2020 at 3:20 PM Carter Schonwald
mailto:carter.schonw...@gmail.com>> wrote:

Have you tried using do notation for bindings you want to
keep strict, with Eg the identity monad?  That doesn’t
address  the design critique but gives you a path forward ?

I do agree that the semantics / default recursivity Of let
bindings  can be inappropriate for non recursive code ,
but would any other non uniform semantics or optimization
be safe?

On Fri, Aug 28, 2020 at 9:05 AM Spiwack, Arnaud
mailto:arnaud.spiw...@tweag.io>>
wrote:

Dear all,



I discovered the hard way, yesterday, that lazy let
pattern
matching is allowed on unboxed tuples. And that it
implicitly reboxes
the pattern.



Here is how the manual describes it, from the relevant
section

:





You can have an unboxed tuple in a pattern
binding, thus



|f x = let (# p,q #) = h x in ..body.. |



If the types of |p| and |q| are not unboxed, the
resulting binding is lazy like any other Haskell
pattern binding. The above example desugars like this:



|f x = let t = case h x of { (# p,q #) -> (p,q) }
p = fst t q = snd t in ..body.. |



Indeed, the bindings can even be recursive.





Notice how |h x| is lazily bound, hence won’t
necessarily be run when
|body| is forced. as opposed to if I had written, for
instance,



|let u = hx in ..body.. |



My question is: are we happy with this? I did find
this extremely
surprising. If I’m using unboxed tuples, it’s because
I want to
guarantee to myself a strict, unboxed behaviour. But a
very subtle
syntactic detail seems to break this expectation for
me. My
expectation would be that I would need to explicitly
rebox things
before they get lazy again.



I find that this behaviour invites trouble. But you
may disagree. Let
me know!





___

ghc-devs mailing list

  

Re: Creative ideas on how to debug heap corruption

[Csaba Hruska]

Fuzzing:

   1. generate simple random stg programs
   2. compile and run with RTS sanity checking enabled
   3. compare the program result between different backends

The fuzzer should cover all codegen cases and all code in RTS. Maybe this
could be checked by the existing tools.

On Mon, Aug 31, 2020 at 4:19 PM George Colpitts 
wrote:

> +Moritz
>
> On Mon, Aug 31, 2020 at 11:17 AM George Colpitts <
> george.colpi...@gmail.com> wrote:
>
>> I assume you're familiar with the following from
>> https://www.aosabook.org/en/ghc.html and that this facility is still
>> there. Just in case you are not:
>>
>> So, the debug RTS has an optional mode that we call *sanity checking*.
>> Sanity checking enables all kinds of expensive assertions, and can make the
>> program run many times more slowly. In particular, sanity checking runs a
>> full scan of the heap to check for dangling pointers (amongst other
>> things), before *and* after every GC. The first job when investigating a
>> runtime crash is to run the program with sanity checking turned on;
>> sometimes this will catch the invariant violation well before the program
>> actually crashes.
>>
>>
>> On Mon, Aug 31, 2020 at 11:08 AM Csaba Hruska 
>> wrote:
>>
>>> Dump the whole heap into file during GC traversal or taking the whole
>>> allocated area. hmm, maybe this is the same as core dump.
>>>
>>> On Mon, Aug 31, 2020 at 11:00 AM Ben Lippmeier 
>>> wrote:
>>>


 > On 31 Aug 2020, at 5:54 pm, Moritz Angermann <
 moritz.angerm...@gmail.com> wrote:
 >
 > If anyone has some create ideas, I'd love to hear them.  I've been
 wondering
 > if just logging allocations (offset, range, type) would help figuring
 out what we
 > expected to be there; and then maybe try to break on the allocation,
 (and
 > subsequent writes).
 >
 > I'm sure some have been down this road before.

 Force a GC before every allocation, and make the GC check the validity
 of the objects before it moves anything. I think this used to be possible
 by compiling the runtime system in debug mode.

 The usual pain of heap corruption is that once the heap is corrupted it
 may be several GC cycles before you get the actual crash, and in the
 meantime the objects have all been moved around. The GC walks over all the
 objects by nature, so get it to validate the heap every time it does, then
 force it to run as often as you possibly can.

 A user space approach is to use a library like vacuum or packman that
 also walks over the heap objects directly.

 http://hackage.haskell.org/package/vacuum-2.2.0.0/docs/GHC-Vacuum.html
 https://hackage.haskell.org/package/packman

 Ben.



 ___
 ghc-devs mailing list
 ghc-devs@haskell.org
 http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

>>> ___
>>> ghc-devs mailing list
>>> ghc-devs@haskell.org
>>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>>>
>>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Creative ideas on how to debug heap corruption

[George Colpitts]

+Moritz

On Mon, Aug 31, 2020 at 11:17 AM George Colpitts 
wrote:

> I assume you're familiar with the following from
> https://www.aosabook.org/en/ghc.html and that this facility is still
> there. Just in case you are not:
>
> So, the debug RTS has an optional mode that we call *sanity checking*.
> Sanity checking enables all kinds of expensive assertions, and can make the
> program run many times more slowly. In particular, sanity checking runs a
> full scan of the heap to check for dangling pointers (amongst other
> things), before *and* after every GC. The first job when investigating a
> runtime crash is to run the program with sanity checking turned on;
> sometimes this will catch the invariant violation well before the program
> actually crashes.
>
>
> On Mon, Aug 31, 2020 at 11:08 AM Csaba Hruska 
> wrote:
>
>> Dump the whole heap into file during GC traversal or taking the whole
>> allocated area. hmm, maybe this is the same as core dump.
>>
>> On Mon, Aug 31, 2020 at 11:00 AM Ben Lippmeier 
>> wrote:
>>
>>>
>>>
>>> > On 31 Aug 2020, at 5:54 pm, Moritz Angermann <
>>> moritz.angerm...@gmail.com> wrote:
>>> >
>>> > If anyone has some create ideas, I'd love to hear them.  I've been
>>> wondering
>>> > if just logging allocations (offset, range, type) would help figuring
>>> out what we
>>> > expected to be there; and then maybe try to break on the allocation,
>>> (and
>>> > subsequent writes).
>>> >
>>> > I'm sure some have been down this road before.
>>>
>>> Force a GC before every allocation, and make the GC check the validity
>>> of the objects before it moves anything. I think this used to be possible
>>> by compiling the runtime system in debug mode.
>>>
>>> The usual pain of heap corruption is that once the heap is corrupted it
>>> may be several GC cycles before you get the actual crash, and in the
>>> meantime the objects have all been moved around. The GC walks over all the
>>> objects by nature, so get it to validate the heap every time it does, then
>>> force it to run as often as you possibly can.
>>>
>>> A user space approach is to use a library like vacuum or packman that
>>> also walks over the heap objects directly.
>>>
>>> http://hackage.haskell.org/package/vacuum-2.2.0.0/docs/GHC-Vacuum.html
>>> https://hackage.haskell.org/package/packman
>>>
>>> Ben.
>>>
>>>
>>>
>>> ___
>>> ghc-devs mailing list
>>> ghc-devs@haskell.org
>>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>>>
>> ___
>> ghc-devs mailing list
>> ghc-devs@haskell.org
>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>>
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Creative ideas on how to debug heap corruption

[George Colpitts]

I assume you're familiar with the following from
https://www.aosabook.org/en/ghc.html and that this facility is still there.
Just in case you are not:

So, the debug RTS has an optional mode that we call *sanity checking*.
Sanity checking enables all kinds of expensive assertions, and can make the
program run many times more slowly. In particular, sanity checking runs a
full scan of the heap to check for dangling pointers (amongst other
things), before *and* after every GC. The first job when investigating a
runtime crash is to run the program with sanity checking turned on;
sometimes this will catch the invariant violation well before the program
actually crashes.


On Mon, Aug 31, 2020 at 11:08 AM Csaba Hruska 
wrote:

> Dump the whole heap into file during GC traversal or taking the whole
> allocated area. hmm, maybe this is the same as core dump.
>
> On Mon, Aug 31, 2020 at 11:00 AM Ben Lippmeier  wrote:
>
>>
>>
>> > On 31 Aug 2020, at 5:54 pm, Moritz Angermann <
>> moritz.angerm...@gmail.com> wrote:
>> >
>> > If anyone has some create ideas, I'd love to hear them.  I've been
>> wondering
>> > if just logging allocations (offset, range, type) would help figuring
>> out what we
>> > expected to be there; and then maybe try to break on the allocation,
>> (and
>> > subsequent writes).
>> >
>> > I'm sure some have been down this road before.
>>
>> Force a GC before every allocation, and make the GC check the validity of
>> the objects before it moves anything. I think this used to be possible by
>> compiling the runtime system in debug mode.
>>
>> The usual pain of heap corruption is that once the heap is corrupted it
>> may be several GC cycles before you get the actual crash, and in the
>> meantime the objects have all been moved around. The GC walks over all the
>> objects by nature, so get it to validate the heap every time it does, then
>> force it to run as often as you possibly can.
>>
>> A user space approach is to use a library like vacuum or packman that
>> also walks over the heap objects directly.
>>
>> http://hackage.haskell.org/package/vacuum-2.2.0.0/docs/GHC-Vacuum.html
>> https://hackage.haskell.org/package/packman
>>
>> Ben.
>>
>>
>>
>> ___
>> ghc-devs mailing list
>> ghc-devs@haskell.org
>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>>
> ___
> ghc-devs mailing list
> ghc-devs@haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: HsPragTick

[Vladislav Zavialov]

I was under impression it was somehow related to HPC. Since I'm not
sufficiently familiar with HPC's inner workings, I kept it around just to
be safe.

- Vlad

On Mon, Aug 31, 2020, 17:03 Ryan Scott  wrote:

> I think that HsPragTick is unused as of [1]. In fact, I was under the
> impression that [1] removed HsPragTick entirely (as the commit message)
> would suggest, but upon further inspection, that doesn't appear to be the
> case. Vlad, do you recall why HsPragTick was kept around?
>
> Ryan S.
> -
> [1] https://gitlab.haskell.org/ghc/ghc/-/merge_requests/2154
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Creative ideas on how to debug heap corruption

[Csaba Hruska]

Dump the whole heap into file during GC traversal or taking the whole
allocated area. hmm, maybe this is the same as core dump.

On Mon, Aug 31, 2020 at 11:00 AM Ben Lippmeier  wrote:

>
>
> > On 31 Aug 2020, at 5:54 pm, Moritz Angermann 
> wrote:
> >
> > If anyone has some create ideas, I'd love to hear them.  I've been
> wondering
> > if just logging allocations (offset, range, type) would help figuring
> out what we
> > expected to be there; and then maybe try to break on the allocation, (and
> > subsequent writes).
> >
> > I'm sure some have been down this road before.
>
> Force a GC before every allocation, and make the GC check the validity of
> the objects before it moves anything. I think this used to be possible by
> compiling the runtime system in debug mode.
>
> The usual pain of heap corruption is that once the heap is corrupted it
> may be several GC cycles before you get the actual crash, and in the
> meantime the objects have all been moved around. The GC walks over all the
> objects by nature, so get it to validate the heap every time it does, then
> force it to run as often as you possibly can.
>
> A user space approach is to use a library like vacuum or packman that also
> walks over the heap objects directly.
>
> http://hackage.haskell.org/package/vacuum-2.2.0.0/docs/GHC-Vacuum.html
> https://hackage.haskell.org/package/packman
>
> Ben.
>
>
>
> ___
> ghc-devs mailing list
> ghc-devs@haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: HsPragTick

[Ryan Scott]

I think that HsPragTick is unused as of [1]. In fact, I was under the
impression that [1] removed HsPragTick entirely (as the commit message)
would suggest, but upon further inspection, that doesn't appear to be the
case. Vlad, do you recall why HsPragTick was kept around?

Ryan S.
-
[1] https://gitlab.haskell.org/ghc/ghc/-/merge_requests/2154
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Creative ideas on how to debug heap corruption

[Ben Lippmeier]

> On 31 Aug 2020, at 5:54 pm, Moritz Angermann  
> wrote:
> 
> If anyone has some create ideas, I'd love to hear them.  I've been wondering
> if just logging allocations (offset, range, type) would help figuring out 
> what we
> expected to be there; and then maybe try to break on the allocation, (and
> subsequent writes).
> 
> I'm sure some have been down this road before. 

Force a GC before every allocation, and make the GC check the validity of the 
objects before it moves anything. I think this used to be possible by compiling 
the runtime system in debug mode.

The usual pain of heap corruption is that once the heap is corrupted it may be 
several GC cycles before you get the actual crash, and in the meantime the 
objects have all been moved around. The GC walks over all the objects by 
nature, so get it to validate the heap every time it does, then force it to run 
as often as you possibly can.

A user space approach is to use a library like vacuum or packman that also 
walks over the heap objects directly.

http://hackage.haskell.org/package/vacuum-2.2.0.0/docs/GHC-Vacuum.html
https://hackage.haskell.org/package/packman

Ben.



___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Creative ideas on how to debug heap corruption

[Moritz Angermann]

Hi there!

as some of you may know, I've been working on an aarch64 native code
generator.  Now I've hit a situation where my stage2 compiler somehow
corrupts my heap.  Initially I thought this would likely be missing memory
barriers, however they are emitted.  This doesn't mean it can't be, but at
least it's not as simple as "they are just missing".

The crashes I see are non deterministic, in fact I sometimes even manage
to compile a Hello World module, without crashes.  Other times it crashes
with unknown closure errors or it just crashes.  But it always crashes
during GC.  Changing the nursery size make it crasha bit more frequent,
but nothing obvious sticks out yet.

If anyone has some create ideas, I'd love to hear them.  I've been wondering
if just logging allocations (offset, range, type) would help figuring out
what we
expected to be there; and then maybe try to break on the allocation, (and
subsequent writes).

I'm sure some have been down this road before.

Cheers,
 Moritz
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs