Re: Meaning of -i and -hidir

[Ryan Yates]

The behavior of `-hidir` and `-i` changed in response to this issue:

https://gitlab.haskell.org/ghc/ghc/-/issues/16500

We just ran into this and I think the change is unfortunate.  I don't think
`-hidir` (a flag about output) should override the input search flag.  I
would like it to at least merge the two so it searches both the `-hidir`
and the `-i`s rather than overriding.  Reverting the change would be fine
too (I think) because you can always specify the same path as `-hidir` and
`-i`.  As it is now, you can control *both* the input and the output.

On Sat, Oct 23, 2021 at 4:26 PM Carter Schonwald 
wrote:

> Could be!  I’m far from a computer much of this week
>
> On Sat, Oct 23, 2021 at 2:36 PM Brandon Allbery 
> wrote:
>
>> I would expect that to be -I and for -i to specify module paths (which
>> might well mean .hi).
>>
>> On Sat, Oct 23, 2021 at 2:32 PM Carter Schonwald <
>> carter.schonw...@gmail.com> wrote:
>>
>>> I would assume the -i is for include c header search paths but I
>>> could be wrong
>>>
>>> On Sat, Oct 23, 2021 at 6:00 AM Oleg Grenrus 
>>> wrote:
>>>
 This looks like a bug.

 -hidir documentation says

 > Redirects all generated interface files into ⟨dir⟩, instead of the
 default.
 > Please also note that when doing incremental compilation (by ghc
 --make or ghc -c), this directory is where GHC looks into to find interface
 files.

 And documentation for -i doesn't mention looking for interface files.

 So by doing

 % ghc -dynamic-too -c lib/Lib.hs -odir odir -hidir hidir_Lib
 % ghc -dynamic-too -c Main.hs -odir odir -hidir hidir_Main -ihidir_Lib
 1
 % ghc odir/Lib.o odir/Main.o -o Demo
 % ./Demo
 True

 everything compiles, TH is run, and demo works. The result files are

 % find .
 .
 ./Demo
 ./Main.hs
 ./hidir_Main
 ./hidir_Main/Main.dyn_hi
 ./hidir_Main/Main.hi
 ./odir
 ./odir/Main.dyn_o
 ./odir/Main.o
 ./odir/Lib.dyn_o
 ./odir/Lib.o
 ./hidir_Lib
 ./hidir_Lib/Lib.dyn_hi
 ./hidir_Lib/Lib.hi
 ./lib
 ./lib/Lib.hs

 ---

 The confusing error is caused by Lib module in libiserv:
 https://hackage.haskell.org/package/libiserv
 which GHC picks!

 If we rename your Lib module to Library, the error is way better: Could
 not find module ‘Library’

 I also tried using same -hidir when compiling both modules, then GHC
 still cannot find the Library interface, even the documentation says it
 should.

 Please open a GHC issue at https://gitlab.haskell.org/ghc/ghc/-/issues



 - Oleg
 On 22.10.2021 19.16, Domínguez, Facundo wrote:

 Dear devs,

 I'm confused about the meaning of -hidir and -i. Here's my experiment
 with both ghc-9.2.0 and ghc-8.10.4.

 > $ find
 > ./Main.hs
 > ./lib/Lib.hs
 >
 > $ ghc -dynamic-too -c lib/Lib.hs -odir odir -hidir hidir_Lib
 >
 > $ ghc -dynamic-too -c Main.hs -odir odir -ihidir_Lib -hidir hidir_Main
 >
 > Main.hs:3:1: error:
 > Bad interface file: hidir_Main/Lib.hi
 > hidir_Main/Lib.hi: openBinaryFile: does not exist (No such
 file or directory)
 >   |
 > 3 | import Lib
 >   | ^^

 If I only use -hidir, it still fails with another error

 > $ ghc -dynamic-too -c Main.hs -odir odir -hidir hidir_Lib
 >
 > Main.hs:7:29: error: Variable not in scope: f
 >   |
 > 7 | main = print $(runIO (print f) >> [| True |])
 >   | ^

 If I use both -i and -hidir pointing to the same folder, then it works!

 > $ ghc -dynamic-too -c Main.hs -odir odir -ihidir_Lib -hidir hidir_Lib

 Now, is this behavior a bug or a feature? And if it is a bug, what is
 the expected behavior?

 Programs copied below.

 Thank you!
 Facundo

 > $ cat lib/Lib.hs
 > module Lib where
 >
 > f :: Int
 > f = 1
 >
 > $ cat Main.hs
 > {-# LANGUAGE TemplateHaskell #-}
 >
 > import Lib
 > import Language.Haskell.TH 
 >
 > main :: IO ()
 > main = print $(runIO (print f) >> [| True |])

 ___
 ghc-devs mailing 
 listghc-devs@haskell.orghttp://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

 ___
 ghc-devs mailing list
 ghc-devs@haskell.org
 http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

>>> ___
>>> ghc-devs mailing list
>>> ghc-devs@haskell.org
>>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>>>
>>
>>
>> --
>> brandon s allbery kf8nh
>> allber...@gmail.com
>>
> ___
> ghc-devs mailing list
> ghc-devs@haskell.org
> 

Re: Fwd: Restricted sums in BoxedRep

[Ryan Yates]

I'm not sure what would lead you to have abundant enough unboxed MVar's to
make a difference with GC.  I'm happy to be wrong, but I think most
situations where you want an MVar you want the indirection.  If you don't
want the indirection you probably don't want some feature of MVar and
should use something else.

On Thu, Oct 15, 2020 at 4:27 PM Carter Schonwald 
wrote:

> Ryan: would an unboxed value Mvar, Eg to write StrictMvar (), avoid
> creating extra gc pressure / traversal a?
>
> On Thu, Oct 15, 2020 at 4:23 PM Ryan Yates  wrote:
>
>> Ah yes.  In my research work I extended Mutable Constructor fields to
>> allow a mutable array field to help with that problem.  This allowed me to
>> have a Nil constructor in a sum type and a Node constructor with normal
>> fields as well as an array of mutable fields.  Pointer tagging worked as
>> expected so a case match on a dereference of a field would not need to
>> follow the pointer and no intermediate objects were between Node objects.
>>
>>
>>
>> On Thu, Oct 15, 2020 at 4:08 PM David Feuer 
>> wrote:
>>
>>> This might be lost in the noise for MVar and TVar, but for arrays, there
>>> are tremendous advantages to cutting out the extra heap objects.
>>>
>>> On Thu, Oct 15, 2020, 1:10 PM Ryan Yates  wrote:
>>>
>>>> I haven't been following this discussion too closely, but in my
>>>> research work I found that some of the benefits that I wanted in this
>>>> direction were already there with pointer tagging.
>>>>
>>>> On Thu, Oct 15, 2020 at 12:59 PM David Feuer 
>>>> wrote:
>>>>
>>>>> Yes, that's something quite different. We'd need a whole different
>>>>> heap object type for such MVars and TVars. My approach covers the case
>>>>> where the unboxed thing can only take on a few values, for some value of 
>>>>> "a
>>>>> few" which, depending on implementation, may or may not be very small. If
>>>>> the nulls point to actual heap objects in pre-allocated pinned memory
>>>>> (say), then up to 64 or so might be reasonable. If they point to "invalid"
>>>>> address space, then the numbers could go up a good bit.
>>>>>
>>>>> On Thu, Oct 15, 2020, 12:50 PM Carter Schonwald <
>>>>> carter.schonw...@gmail.com> wrote:
>>>>>
>>>>>> Indeed, I mean things that aren’t pointery, and could be represented
>>>>>> by a tvar paired with a mutable byte array or mvar with mutable byte 
>>>>>> array,
>>>>>> but which we’d want considered as a single heap object from the rts/gc
>>>>>> perspective.
>>>>>>
>>>>>> On Thu, Oct 15, 2020 at 11:58 AM David Feuer 
>>>>>> wrote:
>>>>>>
>>>>>>> Sorry, unlifted, not unboxed...
>>>>>>>
>>>>>>> On Thu, Oct 15, 2020, 11:57 AM David Feuer 
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Putting unboxed things in TVar, MVar, etc., is part of Andrew
>>>>>>>> Martin's accepted BoxedRep proposal.
>>>>>>>>
>>>>>>>> On Thu, Oct 15, 2020, 11:44 AM Carter Schonwald <
>>>>>>>> carter.schonw...@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> A related idea that came up recently and is perhaps simpler ties
>>>>>>>>> into this via the lens of having unboxed Mvars/tvars (even if it’s
>>>>>>>>> restricted to just things we can embed in a word64#)
>>>>>>>>>
>>>>>>>>> This came up in
>>>>>>>>> https://gitlab.haskell.org/ghc/ghc/-/issues/18798#note_307410,
>>>>>>>>> where viktor had millions of independent mvars holding what’s 
>>>>>>>>> essentially a
>>>>>>>>> strict unit ()!
>>>>>>>>>
>>>>>>>>> The motivation in this later scenario is that in high concurrency
>>>>>>>>> settings, the less trivial stuff the gc needs to trace under updates, 
>>>>>>>>> the
>>>>>>>>> better ghc scales.
>>>>>>>>>
>>>>>>>>> This may not be a use case david has in mind, but certainly seems
>>>>>>>>> related.
>>>>>>>

Re: Fwd: Restricted sums in BoxedRep

[Ryan Yates]

Ah yes.  In my research work I extended Mutable Constructor fields to allow
a mutable array field to help with that problem.  This allowed me to have a
Nil constructor in a sum type and a Node constructor with normal fields as
well as an array of mutable fields.  Pointer tagging worked as expected so
a case match on a dereference of a field would not need to follow the
pointer and no intermediate objects were between Node objects.



On Thu, Oct 15, 2020 at 4:08 PM David Feuer  wrote:

> This might be lost in the noise for MVar and TVar, but for arrays, there
> are tremendous advantages to cutting out the extra heap objects.
>
> On Thu, Oct 15, 2020, 1:10 PM Ryan Yates  wrote:
>
>> I haven't been following this discussion too closely, but in my research
>> work I found that some of the benefits that I wanted in this direction were
>> already there with pointer tagging.
>>
>> On Thu, Oct 15, 2020 at 12:59 PM David Feuer 
>> wrote:
>>
>>> Yes, that's something quite different. We'd need a whole different heap
>>> object type for such MVars and TVars. My approach covers the case where the
>>> unboxed thing can only take on a few values, for some value of "a few"
>>> which, depending on implementation, may or may not be very small. If the
>>> nulls point to actual heap objects in pre-allocated pinned memory (say),
>>> then up to 64 or so might be reasonable. If they point to "invalid" address
>>> space, then the numbers could go up a good bit.
>>>
>>> On Thu, Oct 15, 2020, 12:50 PM Carter Schonwald <
>>> carter.schonw...@gmail.com> wrote:
>>>
>>>> Indeed, I mean things that aren’t pointery, and could be represented by
>>>> a tvar paired with a mutable byte array or mvar with mutable byte array,
>>>> but which we’d want considered as a single heap object from the rts/gc
>>>> perspective.
>>>>
>>>> On Thu, Oct 15, 2020 at 11:58 AM David Feuer 
>>>> wrote:
>>>>
>>>>> Sorry, unlifted, not unboxed...
>>>>>
>>>>> On Thu, Oct 15, 2020, 11:57 AM David Feuer 
>>>>> wrote:
>>>>>
>>>>>> Putting unboxed things in TVar, MVar, etc., is part of Andrew
>>>>>> Martin's accepted BoxedRep proposal.
>>>>>>
>>>>>> On Thu, Oct 15, 2020, 11:44 AM Carter Schonwald <
>>>>>> carter.schonw...@gmail.com> wrote:
>>>>>>
>>>>>>> A related idea that came up recently and is perhaps simpler ties
>>>>>>> into this via the lens of having unboxed Mvars/tvars (even if it’s
>>>>>>> restricted to just things we can embed in a word64#)
>>>>>>>
>>>>>>> This came up in
>>>>>>> https://gitlab.haskell.org/ghc/ghc/-/issues/18798#note_307410,
>>>>>>> where viktor had millions of independent mvars holding what’s 
>>>>>>> essentially a
>>>>>>> strict unit ()!
>>>>>>>
>>>>>>> The motivation in this later scenario is that in high concurrency
>>>>>>> settings, the less trivial stuff the gc needs to trace under updates, 
>>>>>>> the
>>>>>>> better ghc scales.
>>>>>>>
>>>>>>> This may not be a use case david has in mind, but certainly seems
>>>>>>> related.
>>>>>>>
>>>>>>> Phrased more succinctly: gc perf dominates large heap / many core
>>>>>>> computation in Haskell via sensitivity to allocation volume / mutation
>>>>>>> volume (to ensure generational hypothesis stays valid), and providing 
>>>>>>> tools
>>>>>>> to incrementally reduce the pressure with local changes would be good.
>>>>>>>
>>>>>>> So I’d propose / suggest that a baby step towards what david asks
>>>>>>> would be for us to work out some manner of unboxed tvar/mvar ref 
>>>>>>> machinery
>>>>>>> that supports unboxed values.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Oct 15, 2020 at 5:32 AM Andreas Klebinger <
>>>>>>> klebinger.andr...@gmx.at> wrote:
>>>>>>>
>>>>>>>> From a implementors perspective my main questions would be:
>>>>>>>>
>>>>>>>> * How big i

Re: Fwd: Restricted sums in BoxedRep

[Ryan Yates]

I haven't been following this discussion too closely, but in my research
work I found that some of the benefits that I wanted in this direction were
already there with pointer tagging.

On Thu, Oct 15, 2020 at 12:59 PM David Feuer  wrote:

> Yes, that's something quite different. We'd need a whole different heap
> object type for such MVars and TVars. My approach covers the case where the
> unboxed thing can only take on a few values, for some value of "a few"
> which, depending on implementation, may or may not be very small. If the
> nulls point to actual heap objects in pre-allocated pinned memory (say),
> then up to 64 or so might be reasonable. If they point to "invalid" address
> space, then the numbers could go up a good bit.
>
> On Thu, Oct 15, 2020, 12:50 PM Carter Schonwald <
> carter.schonw...@gmail.com> wrote:
>
>> Indeed, I mean things that aren’t pointery, and could be represented by a
>> tvar paired with a mutable byte array or mvar with mutable byte array, but
>> which we’d want considered as a single heap object from the rts/gc
>> perspective.
>>
>> On Thu, Oct 15, 2020 at 11:58 AM David Feuer 
>> wrote:
>>
>>> Sorry, unlifted, not unboxed...
>>>
>>> On Thu, Oct 15, 2020, 11:57 AM David Feuer 
>>> wrote:
>>>
 Putting unboxed things in TVar, MVar, etc., is part of Andrew Martin's
 accepted BoxedRep proposal.

 On Thu, Oct 15, 2020, 11:44 AM Carter Schonwald <
 carter.schonw...@gmail.com> wrote:

> A related idea that came up recently and is perhaps simpler ties into
> this via the lens of having unboxed Mvars/tvars (even if it’s restricted 
> to
> just things we can embed in a word64#)
>
> This came up in
> https://gitlab.haskell.org/ghc/ghc/-/issues/18798#note_307410, where
> viktor had millions of independent mvars holding what’s essentially a
> strict unit ()!
>
> The motivation in this later scenario is that in high concurrency
> settings, the less trivial stuff the gc needs to trace under updates, the
> better ghc scales.
>
> This may not be a use case david has in mind, but certainly seems
> related.
>
> Phrased more succinctly: gc perf dominates large heap / many core
> computation in Haskell via sensitivity to allocation volume / mutation
> volume (to ensure generational hypothesis stays valid), and providing 
> tools
> to incrementally reduce the pressure with local changes would be good.
>
> So I’d propose / suggest that a baby step towards what david asks
> would be for us to work out some manner of unboxed tvar/mvar ref machinery
> that supports unboxed values.
>
>
>
> On Thu, Oct 15, 2020 at 5:32 AM Andreas Klebinger <
> klebinger.andr...@gmx.at> wrote:
>
>> From a implementors perspective my main questions would be:
>>
>> * How big is the benefit in practice? How many use cases are there?
>> * How bad are the costs? (Runtime overhead, rts complexity, ...)
>>
>> The details of how this would be exposed to a user would are
>> important.
>> But if the costs are too high for the drawbacks then it becomes a
>> moot point.
>>
>>
>> David Feuer schrieb am 14.10.2020 um 22:21:
>>
>> Forwarded from Andrew Martin below. I think we want more than just
>> Maybe (more than one null), but the nesting I described is certainly more
>> convenience than necessity.
>>
>> -- Forwarded message -
>> From: Andrew Martin 
>> Date: Wed, Oct 14, 2020, 4:14 PM
>> Subject: Re: Restricted sums in BoxedRep
>> To: David Feuer 
>>
>>
>> You'll have to forward this to the ghc-devs list to share it with
>> others since I'm not currently subscribed to it, but I've had this same
>> thought before. It is discussed at
>> https://github.com/andrewthad/impure-containers/issues/12. Here's
>> the relevant excerpt:
>>
>>> Relatedly, I was thinking the other day that after finishing
>>> implementing
>>> https://github.com/ghc-proposals/ghc-proposals/blob/master/proposals/0203-pointer-rep.rst,
>>> I should really look at seeing if it's possible to add this
>>> maybe-of-a-lifted value trick straight to GHC. I think that with:
>>>
>>> data RuntimpRep
>>>   = BoxedRep Levity
>>>   | MaybeBoxedRep Levity
>>>   | IntRep
>>>   | ...
>>>
>>> data BuiltinMaybe :: forall (v :: Levity). TYPE v -> TYPE 
>>> ('MaybeBoxedRep v)
>>>
>>> This doesn't have the nesting issues because the kind system
>>> prevents nesting. But anyway, back to the original question. I would
>>> recommend not using Maybe.Unsafe and using unpacked-maybe instead.
>>> The latter is definitely safe, and it only costs an extra machine word 
>>> of
>>> space in each data constructor it gets used in, and it doesn't introduce
>>> more indirections.
>>>
>>
>> On Tue, Oct 13, 2020 at 

Re: GHC Logo

[Ryan Yates]

Cats are warm and fuzzy.

On Wed, Sep 2, 2020 at 10:38 AM Richard Eisenberg  wrote:

> On Sep 2, 2020, someone wrote to me privately saying:
>
> > I was thinking Cats for some reason.
>
> Ooh. I'm picturing a cat with its tail wrapped around a lambda, or
> something like that. And Simon PJ does have a cat named Haskell who could
> perhaps be the model. :)
>
> Richard
>
> On Sep 2, 2020, at 10:16 AM, Richard Eisenberg  wrote:
>
> I'm oddly drawn to the idea of a turtle -- except that turtles are slow.
> But animals are cute. Maybe something involving a fox, given that foxes can
> be clever? Octopuses are also known to be very clever, but maybe GitHub has
> octopuses covered.
>
> On Sep 1, 2020, at 8:42 PM, Carter Schonwald 
> wrote:
>
> Ben, what if we have someone draw a cartoony version of your box turtle? i
> feel like that would be a pretty cute logo! totally ahistorical, but would
> certainly be cute!
>
> On Tue, Sep 1, 2020 at 7:51 PM Daneel Yaitskov 
> wrote:
>
>> Hi,
>>
>> Is it a contest for picking up a new logo?
>> As for me logo "λ GHC" is redundant, because H stands for Haskell and λ here
>> means Haskell.
>>
>> So logo should be GλC.
>>
>> Best Regards,
>> Daniil.
>>
>> On Sat, Aug 15, 2020, 8:50 AM Ben Gamari  wrote:
>>
>>> Hi everyone,
>>>
>>> Recently a sponsor asked for a logo for our project. As far as I know,
>>> GHC doesn't really have a consistent logo; the closest that we have had
>>> is the stylized "GHC" on the top of ghc.haskell.org.
>>>
>>> To accomodate the request, I took a few minutes and reworked the
>>> typography of the Thompson-Wheeler Haskell logo for use by GHC. I
>>> couldn't positively identify the typeface used for the "Haskell" text,
>>> but I believe that the extra-bold Cantarell face that I chose in the GHC
>>> variant has a similar feel to the Haskell logo and is free to use.
>>>
>>> I've posted the logo on the Wiki for future reference [1]. Feedback is
>>> very much welcome.
>>>
>>> Cheers,
>>>
>>> - Ben
>>>
>>>
>>>
>>> [1] https://gitlab.haskell.org/ghc/ghc/-/wikis/logo
>>> ___
>>> ghc-devs mailing list
>>> ghc-devs@haskell.org
>>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>>>
>> ___
>> ghc-devs mailing list
>> ghc-devs@haskell.org
>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>>
> ___
> ghc-devs mailing list
> ghc-devs@haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
>
> ___
> ghc-devs mailing list
> ghc-devs@haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
>
> ___
> ghc-devs mailing list
> ghc-devs@haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Wiki markup

[Ryan Yates]

Hi Simon,

I think you can just indent the code block to achieve this.

Ryan

On Mon, Jun 3, 2019 at 12:46 PM Simon Peyton Jones via ghc-devs <
ghc-devs@haskell.org> wrote:

> This is a GitLab wiki markup question.
>
> Suppose I want to have a bulleted list, in which a single item has some
> text, then some laid-out code, then more text.  I want the second blob of
> text to be indented as part of the bullet.
>
> Absent the laid-out code, indenting the second para 2 spaces seems to do
> it:
>
> * first bit of text
>
>
>
>   Second bit of text
>
> But if there is code in the middle, like this
>
> * first bit of text
>
> ```
>
> code
>
> ```
>
>   Second bit of text
>
> the second bit of text is not indented.
>
> Does anyone know how to achieve the indentation?
>
> Simon
> ___
> ghc-devs mailing list
> ghc-devs@haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: MutVar# and GC

[Ryan Yates]

Hi,

Because GHC's GC is generational it needs a way to handle heap objects from
older generations that point into younger generations.  This only happens
when an older object is mutated to point to a younger object.  GHC solves
this by invoking the GC write barrier (not to be confused with write
barriers for memory synchronization) `dirty_MUT_VAR`.  This will add that
mutable object to a mutable list that will be traversed in minor GCs along
with young generation roots.  Additionally the write barrier will mark the
heap object as "dirty" to avoid adding it to the list more than once.

Ryan

On Thu, Jan 17, 2019 at 4:29 PM chessai .  wrote:

> Devs,
>
> I've heard from a few friends that MutVars, TVars, etc. are more
> challenging for the garbage collector. I'm writing to ask if someone can
> answer: 1. Is this true, and 2: Why? I can't seem to find anything like a
> writeup or documentation that mentions this. The HeapObjects trac page also
> mentions nothing about these supposed difficulties that GC faces with
> mutable heap objects.
>
> Thanks
> ___
> ghc-devs mailing list
> ghc-devs@haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Use NULL instead of END_X_QUEUE closures?

[Ryan Yates]

Hi Ömer,

These are pointed to by objects traversed by GC.  They have info tables
like any other heap object that GC can understand.  I think this is a much
simpler invariant to hold then to have some heap objects point to NULL.

Ryan

On Mon, May 7, 2018 at 3:34 PM, Ömer Sinan Ağacan 
wrote:

> Currently we sometimes use special closures to mark end of lists of
> different
> objects. Some examples:
>
> - END_TSO_QUEUE
> - END_STM_WATCH_QUEUE
> - END_STM_CHUNK_LIST
>
> But we also use NULL for the same thing, e.g. in weak pointer lists
> (old_weak_ptr_list, weak_ptr_list).
>
> I'm wondering why we need special marker objects (which are actual closures
> with info tables) instead of using NULL consistently. Current approach
> causes a
> minor problem when working on the RTS because every time I traverse a list
> I
> need to remember how the list is terminated (e.g. NULL when traversing weak
> pointer lists, END_TSO_QUEUE when traversing TSO lists).
>
> Ömer
> ___
> ghc-devs mailing list
> ghc-devs@haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Where to raise an issue for the stm library

[Ryan Yates]

Hi Harendra,

I think STM issues are fine to report to GHC's trac (please someone else
chime in if this isn't the case):

https://ghc.haskell.org/trac/ghc/wiki/ReportABug

If that isn't the appropriate place it can get sorted out from there.

Ryan


On Fri, Aug 25, 2017 at 3:06 PM, Harendra Kumar 
wrote:

> Can someone tell me where to raise an issue for the stm library? The
> hackage link sends me here http://git.haskell.org/packages/stm.git. But
> this page provides no clue on raising issues. I wondered if GHC trac itself
> is used for this library. I saw some other libraries in the drop down list
> but not this one in the GHC trac ticket creation page. It is frustrating
> that figuring out how to raise an issue is not easy for such an important
> library.
>
> -harendra
>
> ___
> ghc-devs mailing list
> ghc-devs@haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: NCG lowering of sqrt

[Ryan Yates]

Hi Kavon,

I looked a bit and it does not appear that there is an SSE sqrt in the
native code gen.  It should be easy to add (see a similar addition here:
https://phabricator.haskell.org/D3265).  The x87 version was available for
32-bit.  I think if you use the LLVM backend it will give you the SSE sqrt.

Ryan

On Fri, Apr 28, 2017 at 9:27 AM, Kavon Farvardin  wrote:

> Given a Cmm expression such as
>
> (_c8Gq::F64) = call MO_F64_Sqrt(_s8oX::F64);   // CmmUnsafeForeignCall
>
> the native code generator produces an actual call to the sqrt C function,
> which has the side-effect of causing all floating-point registers to be
> dumped as they are caller-saved. In the nbody benchmark, this is
> particularly bad for a rather hot piece of code (see below).
>
> Ideally the NCG would recognize this foreign call and instead use the
> `sqrtsd` SSE instruction when targeting x86-64.
>
> Does anyone know if the NCG can produce this instruction? I think it would
> be beneficial, as the below would turn into one or two instructions.
>
> Other math functions such as sin/cos require x87 FPU instructions, which
> as far as I know we're not using.
>
>
> ;;;
> ; NCG generates this in parts of the nbody benchmark
> ; to compute the sqrt
> ;
> subq $8,%rsp
> movsd %xmm9,176(%rsp)  ; all floating-point registers
> movsd %xmm1,184(%rsp)  ; are caller-saved in SysV ABI
> movsd %xmm2,192(%rsp)
> movsd %xmm3,200(%rsp)
> movq %rdi,208(%rsp)
> movq %rcx,216(%rsp)
> movq %rsi,224(%rsp)
> movsd %xmm4,232(%rsp)
> movsd %xmm5,240(%rsp)
> movsd %xmm6,248(%rsp)
> movsd %xmm7,256(%rsp)
> movsd %xmm8,264(%rsp)
> movsd %xmm11,272(%rsp)
> call _sqrt
> ;; the loads
> ;; below are interleaved
> ;; with computations
> addq $8,%rsp
> movsd 264(%rsp),%xmm1
> movsd 240(%rsp),%xmm2
> movsd 224(%rsp),%xmm2
> movsd 232(%rsp),%xmm4
> movq 200(%rsp),%rax
> movsd 248(%rsp),%xmm4
> movsd 256(%rsp),%xmm4
> movq 216(%rsp),%rcx
> movsd 192(%rsp),%xmm2
>
>
> ~kavon
>
> ___
> ghc-devs mailing list
> ghc-devs@haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: DeriveFoldable treatment of tuples is surprising

[Ryan Yates]

Thanks for the clarification!

Ryan

On Wed, Mar 22, 2017 at 9:47 AM, Ryan Scott  wrote:

> I believe what Sven was saying is not that the Foldable instance for
> tuples are given "special treatment" (which is arguably an orthogonal
> discussion), but rather that -XDeriveFoldable special-cases tuples, which
> is certainly true.
>
> As Edward noted, there is one possible justification for this behavior
> w.r.t. things like newtype V3 a = V3 (a, a, a) deriving Foldable. But to be
> honest, I find this justification tenuous at best, given the confusion it
> causes when explaining how DeriveFunctor/DeriveFoldable/DeriveTraversable
> work to newcomers. Removing this special case would not only be simple, but
> it would also lead to a more consistent story overall.
>
> I would be curious to know how much code in the wild is actually taking
> advantage of a trick like newtype V3 a = V3 (a, a, a) deriving Foldable. If
> the breakage isn't terrible, then I propose we just rip off this wart.
>
> (This is basically a rehash of the thoughts I left at
> https://ghc.haskell.org/trac/ghc/ticket/13465#comment:3)
>
> Ryan S.
>
> ___
> ghc-devs mailing list
> ghc-devs@haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: DeriveFoldable treatment of tuples is surprising

[Ryan Yates]

On Wed, Mar 22, 2017 at 4:12 AM, Sven Panne  wrote:

> 2017-03-21 22:29 GMT+01:00 Edward Kmett :
>
>> [... In general I think the current behavior is the least surprising as
>> it "walks all the a's it can" and is the only definition compatible with
>> further extension with Traversable. [...]
>>
>
> OTOH, the current behavior contradicts my intuition that wrapping a type
> into data/newtype plus using the deriving machinery is basically a no-op
> (modulo bottoms etc.). When I e.g. wrap a type t, I would be very surprised
> if the Eq/Ord instances of the wrapped type would behave differently than
> the one on t. I know that this is very handwavy argument, but I think the
> current behavior is *very* surprising.
>
> Somehow the current behavior seems to be incompatible with the FTP, where
> pairs are given a special treatment (if that't the right/intuitive choice
> is a completely different topic, though).
>

I'm not sure what you mean by "pairs are given a special treatment".
Tuples are given the only possible treatment:

data (,) a b = (a,b)

The b is the only place to fold over with a Foldable or change with a
Functor instance.  When things are monomorphic there are more options and
that leads to the least surprising, fold over all the options for:

data Pair a = Pair a a

or

data X a = X (a,a)

The (a,a) here is most certainly not the same thing as (a,b).  There is
something that is a bit surprising to me in that DerivingFoldable will not
a user declared data type for pair with two arguments:

> data Pair a = Pair a a deriving (Functor, Foldable, Show)
> data X a = X (Pair a) deriving (Functor, Foldable, Show)
> length (X (Pair 1 2))
2
> data Tup a b  = Tup a b deriving (Functor, Foldable, Show)
> data Y a = Y (Tup a a) deriving (Functor, Show)

:10:34: error:
• Can't make a derived instance of ‘Functor Y’:
Constructor ‘Y’ must use the type variable only as the last
argument of a data type
• In the data declaration for ‘Y’
> data Y a = Y (Tup a a) deriving (Foldable, Show)

:11:34: error:
• Can't make a derived instance of ‘Foldable Y’:
Constructor ‘Y’ must use the type variable only as the last
argument of a data type
• In the data declaration for ‘Y’
> data Y a = Y (Tup a a) deriving (Foldable, Show)

But it is happy to do just that with (a,a).


Ryan
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: setnumcapabilities001 failure

[Ryan Yates]

Right, it is compiler effects at this boundary that I'm worried about,
values that are not read from memory after the changes have been made, not
memory effects or data races.

On Fri, Oct 28, 2016 at 3:02 AM, Simon Marlow <marlo...@gmail.com> wrote:

> Hi Ryan, I don't think that's the issue.  Those variables can only be
> modified in setNumCapabilities, which acquires *all* the capabilities
> before it makes any changes.  There should be no other threads running RTS
> code(*) while we change the number of capabilities.  In particular we
> shouldn't be in releaseGCThreads while enabled_capabilities is being
> changed.
>
> (*) well except for the parts at the boundary with the external world
> which run without a capability, such as rts_lock() which acquires a
> capability.
>
> Cheers
> Simon
>
> On 27 Oct 2016 17:10, "Ryan Yates" <fryguy...@gmail.com> wrote:
>
>> Briefly looking at the code it seems like several global variables
>> involved should be volatile: n_capabilities, enabled_capabilities, and
>> capabilities.  Perhaps in a loop like in scheduleDoGC the compiler moves
>> the reads of n_capabilites or capabilites outside the loop.  A failed
>> requestSync in that loop would not get updated values for those global
>> pointers.  That particular loop isn't doing that optimization for me, but I
>> think it could happen without volatile.
>>
>> Ryan
>>
>> On Thu, Oct 27, 2016 at 9:18 AM, Ben Gamari <b...@smart-cactus.org> wrote:
>>
>>> Simon Marlow <marlo...@gmail.com> writes:
>>>
>>> > I haven't been able to reproduce the failure yet. :(
>>> >
>>> Indeed I've also not seen it in my own local builds. It's quite an
>>> fragile failure.
>>>
>>> Cheers,
>>>
>>> - Ben
>>>
>>>
>>> ___
>>> ghc-devs mailing list
>>> ghc-devs@haskell.org
>>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>>>
>>>
>>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: setnumcapabilities001 failure

[Ryan Yates]

Briefly looking at the code it seems like several global variables involved
should be volatile: n_capabilities, enabled_capabilities, and
capabilities.  Perhaps in a loop like in scheduleDoGC the compiler moves
the reads of n_capabilites or capabilites outside the loop.  A failed
requestSync in that loop would not get updated values for those global
pointers.  That particular loop isn't doing that optimization for me, but I
think it could happen without volatile.

Ryan

On Thu, Oct 27, 2016 at 9:18 AM, Ben Gamari  wrote:

> Simon Marlow  writes:
>
> > I haven't been able to reproduce the failure yet. :(
> >
> Indeed I've also not seen it in my own local builds. It's quite an
> fragile failure.
>
> Cheers,
>
> - Ben
>
>
> ___
> ghc-devs mailing list
> ghc-devs@haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Semantics of MVars and IORefs?

[Ryan Yates]

Hi Iavor,

You might be interested in what Edward has written about this:

http://blog.ezyang.com/2014/01/so-you-want-to-add-a-new-concurrency-primitive-to-ghc/

I would say when we do have a memory model for GHC the program you gave
will almost certainly be correct.  MVar operations should be full
synchronization operations.  There are some bugs on relaxed systems with
initialization that I think are being addressed.  I can't find the tickets
at the moment.

Ryan

On Fri, Oct 21, 2016 at 1:19 PM, Iavor Diatchki 
wrote:

> Hello,
>
> recently, I and a few colleagues have been wondering about the interaction
> between IORefs and MVars, and we can't seem to find any explicit
> documentation stating the behavior, so I was wondering if anyone might know
> the answer.
>
> The question is:  is it safe to use IORefs in a multi-threaded program,
> provided that the uses are within a "critical section" implemented with
> MVars.  Here is a simple program to illustrate the situation: we have two
> threads, each thread takes a lock, increments a counter, then releases the
> lock:
>
> > import Control.Concurrent
> > import Data.IORef
> >
> > main :: IO ()
> > main =
> >   do lock<- newMVar ()
> >  counter <- newIORef 0
> >  forkIO (thread lock counter)
> >  thread lock counter
> >
> > thread :: MVar () -> IORef Integer -> IO a
> > thread lock counter =
> >   do takeMVar lock
> >  value <- readIORef counter
> >  print value
> >  writeIORef counter (value + 1)
> >  putMVar lock ()
> >  thread lock counter
>
> The question is if this program has a race condition or not, due to the
> use of IORefs?  More explicitly, the concern is if a write to an IORef in
> one thread is guaranteed to be seen by a read from the same IORef in
> another thread, provided that there is proper synchronization between the
> two.
>
> -Iavor
>
>
>
>
>
> ___
> ghc-devs mailing list
> ghc-devs@haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Is Safe Haskell intended to allow segfaults?

[Ryan Yates]

Hi Ryan,

I have similar concerns with safety and STM.  In particular, lazy
validation allows for execution after inconsistent reads from TVars.  The
obvious problem to avoid is falling into an infinite loop.  As long as
-fno-omit-yields is used (on every module?) and maybe some other conditions
like eventually GC happens, the transaction will be validated and killed
off. But other problems can happen.  Consider a transactional hash table
with an array and some hashes already computed to be inside the array.  If
an execution sees updated hashes, but not the updated array, then using
unsafeRead could lead to a segfault.  I don't think this is completely
obvious, especially when people will reach for STM precisely to avoid this
sort of problem.  I worry about code that abstracts over mutable variables
that work given sequential execution, but could fail with STM.  ByteString
can lead to similar issues.  Data.ByteString.replicate can be asked to
allocate very large pinned data leading to immediate heap overflow.  But if
the request is from an inconsistent view of data it seams the programmer
has already done their due diligence in preventing this from happening!

Anyway, I would like to work toward reasoning about these things more
precisely.

On Wed, Aug 10, 2016 at 10:23 AM, Ryan Newton  wrote:

> Hi Edward,
>
> On Tue, Aug 9, 2016 at 11:58 PM, Edward Kmett  wrote:
>>
>> 1.) If you remove IO from being able to be compiled inside Safe code _at
>> all_ most packages I have that bother to expose Safe information will have
>> to stop bothering.
>>
>
> I definitely wouldn't argue for removing it entirely.  But it's good to
> know that there are instances where IO functions get mixed up in safe
> modules.  I'll try to systematically find all of these on hackage, but in
> the meantime do you have a sample list of modules?
>
> My modest starting proposal is marking certain Foreign.* modules as Unsafe
> rather than Trustworthy.  We'll find all the modules affected.  But, again,
> are there any modules you know of offhand that are affected?  They should
> fall into two categories:
>
>1. Safe modules that must become Trustworthy (if they import Foreign
>bits, but don't expose the ability to corrupt memory to the clients of
>their APIs).
>2. Safe modules that must become Unsafe or be split further into
>smaller modules.
>
> Obviously (2) is the biggest source of potential disruption.
>
> I wouldn't ask anyone to accept a patch on GHC until we'd explored these
> impacts pretty thoroughly.
>
> I'd have to cut up too many APIs into too many fine-grained pieces.
>>
>
> Yeah, the module-level business is pretty annoying.  "vector' removed
> ".Safe" modules and no one has gotten around to adding the ".Unsafe".
>
>
>> 2.) Assuming instead that you're talking about a stronger-than-Safe
>> additional language extension, say ReallySafe or SafeIO, it all comes down
>> to what the user is allowed to do in IO, doesn't it? What effects are users
>> granted access to? We don't have a very fine-grained system for IO-effect
>> management, and it seems pretty much any choice you pick for what to put in
>> the sandbox will be wrong for some users, so you'd need some sort of pragma
>> for each IO operation saying what bins it falls into and to track that
>> while type checking, etc.
>>
>
> Well, *maybe* it is a slippery slope that leads to a full effect system.
> But I'd like to see these issues enumerated.  Does memory safety as a goal
> really involve so many different effects?  Do you think there will be 1, 3,
> 10, or 100 things beyond Foreign.Ptr to worry about?
>
> 3.) On the other hand, someone could _build_ an effect system in Haskell
>> that happens to sit on top of IO, holding effects in an HList, undischarged
>> nullary class constraint, etc.
>>
>
> Well, sure, I hope we will continue to aim for this as well.  This is
> effectively what we do with our "LVish" Par monad, where we use Safe
> Haskell to ensure users cannot break the effect system in -XSafe code.
>
> Best,
>  -Ryan
>
> ___
> ghc-devs mailing list
> ghc-devs@haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Temporarily pinning a thread to a capability

[Ryan Yates]

A thread with TSO_LOCKED can be migrated if the number of capabilities
decreases.

Ryan

On Tue, Oct 27, 2015 at 11:35 PM, Edward Kmett  wrote:

> Would anything go wrong with a thread id if I pinned it to a capability
> after the fact?
>
> I could in theory do so just by setting
>
> tso->flags |= TSO_LOCKED
>
> and then disabling this later by restoring the TSO flags.
>
> I can't think of anything but I figured folks here might be able to think
> of invariants I don't know about.
>
> Usage scenario:
>
> I have a number of things where I can't afford a map from a ThreadId# or
> even its internal id to a per-thread value for bounded wait-free
> structures.
>
> On the other hand, I can afford one entry per capability and to make a
> handful of primitives that can't be preempted, letting me use normal
> writes, not even a CAS, to update the capability-local variable in a
> primitive (indexing into an array of size based on the number of
> capabilities). This lets me bound the amount of "helpers" to consider by
> the capability count rather than the potentially much larger and much more
> variable number of live threads.
>
> However, I may need to access this stuff in "pure" code that wasn't
> written with my needs in mind, so I need to at least temporarily pin the
> current thread to a fixed capability for the duration when that happens.
>
> This isn't perfect, it won't react to a growing number of capabilities
> nicely in the future, but it does handle a lot of things I can't do now at
> all without downgrading to lock-free and starving a lot of computations, so
> I'm hoping the answer is "it all works". =)
>
> -Edward
>
> ___
> ghc-devs mailing list
> ghc-devs@haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Temporarily pinning a thread to a capability

[Ryan Yates]

I figured as much :D

On Wed, Oct 28, 2015 at 4:43 PM, Edward Kmett <ekm...@gmail.com> wrote:

> If the number of capabilities is increased or decreased while everything I
> have here is running I'm going to have to blow up the world anyways.
>
> Basically I'll need to rely on an invariant that setNumCapabilities is
> called before you spin up these Par-like computations.
>
> -Edward
>
> On Wed, Oct 28, 2015 at 4:28 PM, Ryan Yates <fryguy...@gmail.com> wrote:
>
>> A thread with TSO_LOCKED can be migrated if the number of capabilities
>> decreases.
>>
>> Ryan
>>
>> On Tue, Oct 27, 2015 at 11:35 PM, Edward Kmett <ekm...@gmail.com> wrote:
>>
>>> Would anything go wrong with a thread id if I pinned it to a capability
>>> after the fact?
>>>
>>> I could in theory do so just by setting
>>>
>>> tso->flags |= TSO_LOCKED
>>>
>>> and then disabling this later by restoring the TSO flags.
>>>
>>> I can't think of anything but I figured folks here might be able to
>>> think of invariants I don't know about.
>>>
>>> Usage scenario:
>>>
>>> I have a number of things where I can't afford a map from a ThreadId# or
>>> even its internal id to a per-thread value for bounded wait-free
>>> structures.
>>>
>>> On the other hand, I can afford one entry per capability and to make a
>>> handful of primitives that can't be preempted, letting me use normal
>>> writes, not even a CAS, to update the capability-local variable in a
>>> primitive (indexing into an array of size based on the number of
>>> capabilities). This lets me bound the amount of "helpers" to consider by
>>> the capability count rather than the potentially much larger and much more
>>> variable number of live threads.
>>>
>>> However, I may need to access this stuff in "pure" code that wasn't
>>> written with my needs in mind, so I need to at least temporarily pin the
>>> current thread to a fixed capability for the duration when that happens.
>>>
>>> This isn't perfect, it won't react to a growing number of capabilities
>>> nicely in the future, but it does handle a lot of things I can't do now at
>>> all without downgrading to lock-free and starving a lot of computations, so
>>> I'm hoping the answer is "it all works". =)
>>>
>>> -Edward
>>>
>>> ___
>>> ghc-devs mailing list
>>> ghc-devs@haskell.org
>>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>>>
>>>
>>
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Unlifted data types

[Ryan Yates]

https://ghc.haskell.org/trac/ghc/wiki/UnliftedDataTypes

On Thu, Oct 8, 2015 at 6:02 AM, Simon Peyton Jones 
wrote:

> | I've added a section on parametric levity polymorphism to the wiki.
> | Sorry it took so long.
>
> What's the wiki page?
>
> Simon
> |
> | I might add some thoughts about first-class `!a` being the only
> | semantic hole in our current strict data type situation later if I
> | remember to do so.
> ___
> ghc-devs mailing list
> ghc-devs@haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: ArrayArrays

[Ryan Yates]

I completely agree.  I would love to spend some time during ICFP and
friends talking about what it could look like.  My small array for STM
changes for the RTS can be seen here [1].  It is on a branch somewhere
between 7.8 and 7.10 and includes irrelevant STM bits and some
confusing naming choices (sorry), but should cover all the details
needed to implement it for a non-STM context.  The biggest surprise
for me was following small array too closely and having a word/byte
offset miss-match [2].

[1]: 
https://github.com/fryguybob/ghc/compare/ghc-htm-bloom...fryguybob:ghc-htm-mut
[2]: https://ghc.haskell.org/trac/ghc/ticket/10413

Ryan

On Fri, Aug 28, 2015 at 10:09 PM, Edward Kmett ekm...@gmail.com wrote:
 I'd love to have that last 10%, but its a lot of work to get there and more
 importantly I don't know quite what it should look like.

 On the other hand, I do have a pretty good idea of how the primitives above
 could be banged out and tested in a long evening, well in time for 7.12. And
 as noted earlier, those remain useful even if a nicer typed version with an
 extra level of indirection to the sizes is built up after.

 The rest sounds like a good graduate student project for someone who has
 graduate students lying around. Maybe somebody at Indiana University who has
 an interest in type theory and parallelism can find us one. =)

 -Edward

 On Fri, Aug 28, 2015 at 8:48 PM, Ryan Yates fryguy...@gmail.com wrote:

 I think from my perspective, the motivation for getting the type
 checker involved is primarily bringing this to the level where users
 could be expected to build these structures.  it is reasonable to
 think that there are people who want to use STM (a context with
 mutation already) to implement a straight forward data structure that
 avoids extra indirection penalty.  There should be some places where
 knowing that things are field accesses rather then array indexing
 could be helpful, but I think GHC is good right now about handling
 constant offsets.  In my code I don't do any bounds checking as I know
 I will only be accessing my arrays with constant indexes.  I make
 wrappers for each field access and leave all the unsafe stuff in
 there.  When things go wrong though, the compiler is no help.  Maybe
 template Haskell that generates the appropriate wrappers is the right
 direction to go.
 There is another benefit for me when working with these as arrays in
 that it is quite simple and direct (given the hoops already jumped
 through) to play with alignment.  I can ensure two pointers are never
 on the same cache-line by just spacing things out in the array.

 On Fri, Aug 28, 2015 at 7:33 PM, Edward Kmett ekm...@gmail.com wrote:
  They just segfault at this level. ;)
 
  Sent from my iPhone
 
  On Aug 28, 2015, at 7:25 PM, Ryan Newton rrnew...@gmail.com wrote:
 
  You presumably also save a bounds check on reads by hard-coding the
  sizes?
 
  On Fri, Aug 28, 2015 at 3:39 PM, Edward Kmett ekm...@gmail.com wrote:
 
  Also there are 4 different things here, basically depending on two
  independent questions:
 
  a.) if you want to shove the sizes into the info table, and
  b.) if you want cardmarking.
 
  Versions with/without cardmarking for different sizes can be done
  pretty
  easily, but as noted, the infotable variants are pretty invasive.
 
  -Edward
 
  On Fri, Aug 28, 2015 at 6:36 PM, Edward Kmett ekm...@gmail.com wrote:
 
  Well, on the plus side you'd save 16 bytes per object, which adds up
  if
  they were small enough and there are enough of them. You get a bit
  better
  locality of reference in terms of what fits in the first cache line of
  them.
 
  -Edward
 
  On Fri, Aug 28, 2015 at 6:14 PM, Ryan Newton rrnew...@gmail.com
  wrote:
 
  Yes. And for the short term I can imagine places we will settle with
  arrays even if it means tracking lengths unnecessarily and
  unsafeCoercing
  pointers whose types don't actually match their siblings.
 
  Is there anything to recommend the hacks mentioned for fixed sized
  array
  objects *other* than using them to fake structs? (Much to
  derecommend, as
  you mentioned!)
 
  On Fri, Aug 28, 2015 at 3:07 PM Edward Kmett ekm...@gmail.com
  wrote:
 
  I think both are useful, but the one you suggest requires a lot more
  plumbing and doesn't subsume all of the usecases of the other.
 
  -Edward
 
  On Fri, Aug 28, 2015 at 5:51 PM, Ryan Newton rrnew...@gmail.com
  wrote:
 
  So that primitive is an array like thing (Same pointed type,
  unbounded
  length) with extra payload.
 
  I can see how we can do without structs if we have arrays,
  especially
  with the extra payload at front. But wouldn't the general solution
  for
  structs be one that that allows new user data type defs for #
  types?
 
 
 
  On Fri, Aug 28, 2015 at 4:43 PM Edward Kmett ekm...@gmail.com
  wrote:
 
  Some form of MutableStruct# with a known number of words and a
  known
  number of pointers is basically what Ryan Yates was suggesting
  above, but
  where

Re: ArrayArrays

[Ryan Yates]

I think from my perspective, the motivation for getting the type
checker involved is primarily bringing this to the level where users
could be expected to build these structures.  it is reasonable to
think that there are people who want to use STM (a context with
mutation already) to implement a straight forward data structure that
avoids extra indirection penalty.  There should be some places where
knowing that things are field accesses rather then array indexing
could be helpful, but I think GHC is good right now about handling
constant offsets.  In my code I don't do any bounds checking as I know
I will only be accessing my arrays with constant indexes.  I make
wrappers for each field access and leave all the unsafe stuff in
there.  When things go wrong though, the compiler is no help.  Maybe
template Haskell that generates the appropriate wrappers is the right
direction to go.
There is another benefit for me when working with these as arrays in
that it is quite simple and direct (given the hoops already jumped
through) to play with alignment.  I can ensure two pointers are never
on the same cache-line by just spacing things out in the array.

On Fri, Aug 28, 2015 at 7:33 PM, Edward Kmett ekm...@gmail.com wrote:
 They just segfault at this level. ;)

 Sent from my iPhone

 On Aug 28, 2015, at 7:25 PM, Ryan Newton rrnew...@gmail.com wrote:

 You presumably also save a bounds check on reads by hard-coding the sizes?

 On Fri, Aug 28, 2015 at 3:39 PM, Edward Kmett ekm...@gmail.com wrote:

 Also there are 4 different things here, basically depending on two
 independent questions:

 a.) if you want to shove the sizes into the info table, and
 b.) if you want cardmarking.

 Versions with/without cardmarking for different sizes can be done pretty
 easily, but as noted, the infotable variants are pretty invasive.

 -Edward

 On Fri, Aug 28, 2015 at 6:36 PM, Edward Kmett ekm...@gmail.com wrote:

 Well, on the plus side you'd save 16 bytes per object, which adds up if
 they were small enough and there are enough of them. You get a bit better
 locality of reference in terms of what fits in the first cache line of them.

 -Edward

 On Fri, Aug 28, 2015 at 6:14 PM, Ryan Newton rrnew...@gmail.com wrote:

 Yes. And for the short term I can imagine places we will settle with
 arrays even if it means tracking lengths unnecessarily and unsafeCoercing
 pointers whose types don't actually match their siblings.

 Is there anything to recommend the hacks mentioned for fixed sized array
 objects *other* than using them to fake structs? (Much to derecommend, as
 you mentioned!)

 On Fri, Aug 28, 2015 at 3:07 PM Edward Kmett ekm...@gmail.com wrote:

 I think both are useful, but the one you suggest requires a lot more
 plumbing and doesn't subsume all of the usecases of the other.

 -Edward

 On Fri, Aug 28, 2015 at 5:51 PM, Ryan Newton rrnew...@gmail.com
 wrote:

 So that primitive is an array like thing (Same pointed type, unbounded
 length) with extra payload.

 I can see how we can do without structs if we have arrays, especially
 with the extra payload at front. But wouldn't the general solution for
 structs be one that that allows new user data type defs for # types?



 On Fri, Aug 28, 2015 at 4:43 PM Edward Kmett ekm...@gmail.com wrote:

 Some form of MutableStruct# with a known number of words and a known
 number of pointers is basically what Ryan Yates was suggesting above, 
 but
 where the word counts were stored in the objects themselves.

 Given that it'd have a couple of words for those counts it'd likely
 want to be something we build in addition to MutVar# rather than a
 replacement.

 On the other hand, if we had to fix those numbers and build info
 tables that knew them, and typechecker support, for instance, it'd get
 rather invasive.

 Also, a number of things that we can do with the 'sized' versions
 above, like working with evil unsized c-style arrays directly inline at 
 the
 end of the structure cease to be possible, so it isn't even a pure win 
 if we
 did the engineering effort.

 I think 90% of the needs I have are covered just by adding the one
 primitive. The last 10% gets pretty invasive.

 -Edward

 On Fri, Aug 28, 2015 at 5:30 PM, Ryan Newton rrnew...@gmail.com
 wrote:

 I like the possibility of a general solution for mutable structs
 (like Ed said), and I'm trying to fully understand why it's hard.

 So, we can't unpack MutVar into constructors because of object
 identity problems. But what about directly supporting an extensible 
 set of
 unlifted MutStruct# objects, generalizing (and even replacing) 
 MutVar#? That
 may be too much work, but is it problematic otherwise?

 Needless to say, this is also critical if we ever want best in class
 lockfree mutable structures, just like their Stm and sequential
 counterparts.

 On Fri, Aug 28, 2015 at 4:43 AM Simon Peyton Jones
 simo...@microsoft.com wrote:

 At the very least I'll take this email and turn it into a short
 article.

 Yes, please do

Re: ArrayArrays

[Ryan Yates]

Hi Edward,

I've been working on removing indirection in STM and I added a heap
object like SmallArray, but with a mix of words and pointers (as well
as a header with metadata for STM).  It appears to work well now, but
it is missing the type information.  All the pointers have the same
type which works fine for your Upper.  In my case I use it to
represent a red-black tree node [1].

Also all the structures I make are fixed size and it would be nice if
the compiler could treat that fix size like a constant in code
generation.  I don't know what the right design is or what would be
needed, but it seems simple enough to give the right typing
information to something like this and basically get a mutable struct.
I'm talking about this work at HIW and really hope to find someone
interested in extending this expressiveness to let us write something
that looks clear in Haskell, but gives the heap representation that we
really need for performance.  From the RTS perspective I think there
are any obstacles.

[1]: 
https://github.com/fryguybob/ghc-stm-benchmarks/blob/master/benchmarks/RBTree-Throughput/RBTreeNode.hs

Ryan

On Fri, Aug 21, 2015 at 12:25 AM, Edward Kmett ekm...@gmail.com wrote:
 When (ab)using them for this purpose, SmallArrayArray's would be very handy
 as well.

 Consider right now if I have something like an order-maintenance structure I
 have:

 data Upper s = Upper {-# UNPACK #-} !(MutableByteArray s) {-# UNPACK #-}
 !(MutVar s (Upper s)) {-# UNPACK #-} !(MutVar s (Upper s))

 data Lower s = Lower {-# UNPACK #-} !(MutVar s (Upper s)) {-# UNPACK #-}
 !(MutableByteArray s) {-# UNPACK #-} !(MutVar s (Lower s)) {-# UNPACK #-}
 !(MutVar s (Lower s))

 The former contains, logically, a mutable integer and two pointers, one for
 forward and one for backwards. The latter is basically the same thing with a
 mutable reference up pointing at the structure above.

 On the heap this is an object that points to a structure for the bytearray,
 and points to another structure for each mutvar which each point to the
 other 'Upper' structure. So there is a level of indirection smeared over
 everything.

 So this is a pair of doubly linked lists with an upward link from the
 structure below to the structure above.

 Converted into ArrayArray#s I'd get

 data Upper s = Upper (MutableArrayArray# s)

 w/ the first slot being a pointer to a MutableByteArray#, and the next 2
 slots pointing to the previous and next previous objects, represented just
 as their MutableArrayArray#s. I can use sameMutableArrayArray# on these for
 object identity, which lets me check for the ends of the lists by tying
 things back on themselves.

 and below that

 data Lower s = Lower (MutableArrayArray# s)

 is similar, with an extra MutableArrayArray slot pointing up to an upper
 structure.

 I can then write a handful of combinators for getting out the slots in
 question, while it has gained a level of indirection between the wrapper to
 put it in * and the MutableArrayArray# s in #, that one can be basically
 erased by ghc.

 Unlike before I don't have several separate objects on the heap for each
 thing. I only have 2 now. The MutableArrayArray# for the object itself, and
 the MutableByteArray# that it references to carry around the mutable int.

 The only pain points are

 1.) the aforementioned limitation that currently prevents me from stuffing
 normal boxed data through a SmallArray or Array into an ArrayArray leaving
 me in a little ghetto disconnected from the rest of Haskell,

 and

 2.) the lack of SmallArrayArray's, which could let us avoid the card marking
 overhead. These objects are all small, 3-4 pointers wide. Card marking
 doesn't help.

 Alternately I could just try to do really evil things and convert the whole
 mess to SmallArrays and then figure out how to unsafeCoerce my way to glory,
 stuffing the #'d references to the other arrays directly into the SmallArray
 as slots, removing the limitation  we see here by aping the
 MutableArrayArray# s API, but that gets really really dangerous!

 I'm pretty much willing to sacrifice almost anything on the altar of speed
 here, but I'd like to be able to let the GC move them and collect them which
 rules out simpler Ptr and Addr based solutions.

 -Edward

 On Thu, Aug 20, 2015 at 9:01 PM, Manuel M T Chakravarty
 c...@cse.unsw.edu.au wrote:

 That’s an interesting idea.

 Manuel

  Edward Kmett ekm...@gmail.com:
 
  Would it be possible to add unsafe primops to add Array# and SmallArray#
  entries to an ArrayArray#? The fact that the ArrayArray# entries are all
  directly unlifted avoiding a level of indirection for the containing
  structure is amazing, but I can only currently use it if my leaf level data
  can be 100% unboxed and distributed among ByteArray#s. It'd be nice to be
  able to have the ability to put SmallArray# a stuff down at the leaves to
  hold lifted contents.
 
  I accept fully that if I name the wrong type when I go to access one of
  the fields it'll 

Re: Can we fix our Trac that doesn't lose new ticket text

[Ryan Yates]

My very pessimistic rule of thumb is don't trust a form field unless the
site actively saves a draft as you type.

On Fri, Apr 24, 2015 at 10:13 AM, Eric Crockett ecrocke...@gmail.com
wrote:

 My Firefox also has this issue. So it seems the problem is the browser
 itself. How can this be fixed on the server side?

 On Fri, Apr 24, 2015 at 10:07 AM, Edward Z. Yang ezy...@mit.edu wrote:

 Whatever the case, it affects me, it affects SPJ, so we ought to fix it!
 (Change your browser is not a good answer :)

 Edward

 Excerpts from Eric Crockett's message of 2015-04-24 15:02:35 +0100:
  On the contrary, I did precisely that just a few days ago (
  https://ghc.haskell.org/trac/ghc/ticket/10338), and to my surprise,
 trac
  *did* save my new ticket text. Perhaps this is a browser-specific issue?
  I'm using 64-bit Chrome 42.0.2311.90.
 
  On Fri, Apr 24, 2015 at 9:54 AM, Edward Z. Yang ezy...@mit.edu wrote:
 
   Steps to reproduce:
  
   1. Click New Ticket
   2. Type some text into the box
   3. Press Back in your browser
   4. Press Forward in your browser
  
   If you try this with an official Trac this doesn't happen, so either
   this was fixed in a new version, or we have a plugin installed which
   is causing this to happen.
  
   Current version of Trac is 1.0.5, we're currently running 1.0.1
  
   Thanks,
   Edward
   ___
   ghc-devs mailing list
   ghc-devs@haskell.org
   http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
  



 ___
 ghc-devs mailing list
 ghc-devs@haskell.org
 http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs


___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: STM and GC write barrier

[Ryan Yates]

Ok, that explains it.  Thanks!

On Thu, Nov 6, 2014 at 7:34 AM, Simon Marlow marlo...@gmail.com wrote:

 On 15/09/2014 17:50, Ryan Yates wrote:

 I'm starting to get to the bottom of something that has been puzzling me
 for a while.  Recently commit 6d784c43592290ec16db8b7f0f2a012dff3ed497
 [1] introduced the GC write barrier for TVars.  Not fully understanding
 things and reading the commit message to be saying that this was an
 optimization, I implemented my hardware transactional memory support
 without the write barrier (avoiding the extra work inside a
 transaction).  This resulted in occasional crashes where a TVar which
 was point to a valid heap object when it was committed, pointed to
 garbage later.  My understanding was that the TVar ended up in a later
 generation then the value that it came to point to and without getting
 added to the mut list, the value was collected.  Adding the write
 barrier to all TVars written in my hardware transaction made the problem
 go away.

 While this all makes sense to me, what doesn't make as much sense is how
 STM prior to the commit above was not susceptible to the same problem.
   Is there some machinery to avoid this issue that I'm still missing?


 Sorry for the delay, I'm just catching up with this mailing list.

 Prior to this commit, the garbage collector would treat *all* TVars in the
 old generation as potentially modified, and traverse them all during every
 GC.  This is expensive when there are a lot of TVars, which meant that
 TVar-based data structures such as TChan would perform very badly with a
 large number of elements.  The fix is to track writes to TVars in the old
 generation, which is what that commit did.

 Cheers,
 Simon

___
ghc-devs mailing list
ghc-devs@haskell.org
http://www.haskell.org/mailman/listinfo/ghc-devs

STM and GC write barrier

[Ryan Yates]

Hi All,

I'm starting to get to the bottom of something that has been puzzling me
for a while.  Recently commit 6d784c43592290ec16db8b7f0f2a012dff3ed497
[1] introduced
the GC write barrier for TVars.  Not fully understanding things and reading
the commit message to be saying that this was an optimization, I
implemented my hardware transactional memory support without the write
barrier (avoiding the extra work inside a transaction).  This resulted in
occasional crashes where a TVar which was point to a valid heap object when
it was committed, pointed to garbage later.  My understanding was that the
TVar ended up in a later generation then the value that it came to point to
and without getting added to the mut list, the value was collected.  Adding
the write barrier to all TVars written in my hardware transaction made the
problem go away.

While this all makes sense to me, what doesn't make as much sense is how
STM prior to the commit above was not susceptible to the same problem.  Is
there some machinery to avoid this issue that I'm still missing?

Ryan


[1]:
https://github.com/ghc/ghc/commit/6d784c43592290ec16db8b7f0f2a012dff3ed497
___
ghc-devs mailing list
ghc-devs@haskell.org
http://www.haskell.org/mailman/listinfo/ghc-devs

Re: Possible bug related to stm and exceptions

[Ryan Yates]

The bug that Luite and I uncovered is
http://ghc.haskell.org/trac/ghc/ticket/7930.  It would not be related.
 There was a bug relating to `catchSTM` that was fixed recently:
http://ghc.haskell.org/trac/ghc/ticket/8035.  And another related to
profiling: http://ghc.haskell.org/trac/ghc/ticket/8298.  I doubt either of
these is related.  I'm happy to help narrow things down.

Ryan


On Thu, Oct 17, 2013 at 4:39 AM, Simon Marlow marlo...@gmail.com wrote:

 On 17/10/2013 03:01, Andreas Voellmy wrote:

 Hi all,

 I have a program that uses STM heavily and also performs lots of foreign
 calls. I've noticed that sometimes the program uses 100% CPU
 indefinitely and uses lots of memory - I see it go up to about 5GB
 before I kill it. I've grabbed some preliminary samples of stack traces
 and see lots stm related stuff (e.g. lots of stg_atomically_frame_info
 and stmCommitTransaction entries).  I can pretty reliably get the
 behavior to happen now by closing a socket that my Haskell program is
 trying to recv from. When this causes an exception to be raised
 (something like recv: resource vanished (Connection reset by peer)) ,
 then this behavior gets triggered.  I haven't pinned down the bug yet,
 but I'm suspecting it is STM related - somehow the exception causes some
 STM transaction to go wrong.

 Are there any known bugs that sound similar to this?

 BTW, this is with GHC 7.6.3 from a recent HP release on OS X.


 Please create a ticket and dump all the information you have in it. There
 might be something we can tell from the stack trace, but if not we'll need
 a way to reproduce it.

 Cheers,
 Simon


 __**_
 ghc-devs mailing list
 ghc-devs@haskell.org
 http://www.haskell.org/**mailman/listinfo/ghc-devshttp://www.haskell.org/mailman/listinfo/ghc-devs

___
ghc-devs mailing list
ghc-devs@haskell.org
http://www.haskell.org/mailman/listinfo/ghc-devs

Re: STM Commentary

[Ryan Yates]

Thanks!  I'll work on getting it onto the wiki soon.  My next step is to
apply hardware transactions to improve STM performance.  For this I will
need to collect some meaningful STM benchmarks.  If anyone has some
particular use cases or code where they would like to see improved STM
performance on I would love to know.  There are several potential
improvements to the existing STM performance that I will also investigate
along the way.

Ryan

On Mon, May 20, 2013 at 10:04 AM, Simon Peyton-Jones
simo...@microsoft.comwrote:

  Ryan

 ** **

 Great stuff.  What you have is almost certainly more accurate, complete,
 and up to date than anything else we have, so thank you.  As others have
 said, putting it on the GHC wiki would be much the best place.

 ** **

 Perhaps here: http://hackage.haskell.org/trac/ghc/wiki/Commentary/Rts/STM*
 ***

 ** **

 Simon

 * *

 *Microsoft Research Limited (company number 03369488) is registered in
 England and Wales*

 *Registered office 21 Station Road, Cambridge, CB1 2FB ***

 ** **

 *From:* ghc-devs-boun...@haskell.org [mailto:ghc-devs-boun...@haskell.org]
 *On Behalf Of *Ryan Yates
 *Sent:* 17 May 2013 21:06
 *To:* ghc-devs@haskell.org
 *Cc:* Michael L. Scott
 *Subject:* STM Commentary

 ** **

 Hi all,

 ** **

 For a while now I've been working on understanding GHC's STM
 implementation and I finally have something to share.  I've written a
 commentary on the implementation which can be found here:

 ** **

 http://fryguybob.github.io/STM-Commentary/

 ** **

 The source is here:

 ** **

 https://github.com/fryguybob/STM-Commentary/

 ** **

 It still could use a bit of polish and I would love to hear about any
 missing details, confusion, or misunderstandings.  My goal in writing this
 was to fill in the gap found here
 http://hackage.haskell.org/trac/ghc/wiki/Commentary/Rts/STM.  I happy to
 have it live where it is most appropriate.  If anyone has opinions on that,
 let me know.

 ** **

 Any comments are welcome, it may be easiest to just comment on the source
 directly on Github, but I will, of course, take feedback in any form.

 ** **

 Thanks,

 ** **

 Ryan Yates

 ** **

 ** **

___
ghc-devs mailing list
ghc-devs@haskell.org
http://www.haskell.org/mailman/listinfo/ghc-devs

STM Commentary

[Ryan Yates]

Hi all,

For a while now I've been working on understanding GHC's STM implementation
and I finally have something to share.  I've written a commentary on the
implementation which can be found here:

http://fryguybob.github.io/STM-Commentary/

The source is here:

https://github.com/fryguybob/STM-Commentary/

It still could use a bit of polish and I would love to hear about any
missing details, confusion, or misunderstandings.  My goal in writing this
was to fill in the gap found here
http://hackage.haskell.org/trac/ghc/wiki/Commentary/Rts/STM.  I happy to
have it live where it is most appropriate.  If anyone has opinions on that,
let me know.

Any comments are welcome, it may be easiest to just comment on the source
directly on Github, but I will, of course, take feedback in any form.

Thanks,

Ryan Yates
___
ghc-devs mailing list
ghc-devs@haskell.org
http://www.haskell.org/mailman/listinfo/ghc-devs

Re: STM Commentary

[Ryan Yates]

On Fri, May 17, 2013 at 4:19 PM, Ian Lynagh i...@well-typed.com wrote:

 Why not put it on
 http://hackage.haskell.org/trac/ghc/wiki/Commentary/Rts/STM
 ? If you sign up for an account on the trac then you should be able to
 edit that page.


Sure, I wanted to make sure that anyone who wanted had a chance to review
it before it ends up somewhere official :D.

Ryan
___
ghc-devs mailing list
ghc-devs@haskell.org
http://www.haskell.org/mailman/listinfo/ghc-devs