Re: Is Safe Haskell intended to allow segfaults?

[Ryan Newton]

Yes, it is peek and poke that are dangerous.  It was Foreign.Storable that
I wanted to mark as Unsafe.

But we do sometimes run into examples where there's an A and a B, and if
you import both, you can make A+B which blows up.  So preventing access to
A+B may mean arbitrarily marking one or the other (or both) as Unsafe.

What I was hoping for examples of are modules you have that are Safe and
import Foreign.Storable.




On Fri, Aug 12, 2016 at 9:49 AM, Edward Kmett <ekm...@gmail.com> wrote:

> As for a sample list of modules, let's just start with your very first
> example, Foreign.Ptr:
>
> In and of itself nothing in Foreign.Ptr is unsafe! It allows a bit of
> arithmetic on a type you can't actually use with anything, and provides an
> IO action mixed into an otherwise pure module that happens to create a
> FunPtr slot from a haskell function. In fact this module is a textbook
> example of an otherwise perfectly cromulent Trustworthy module today that
> happens to have a single IO action in it.
>
> I can grab Ptr from it, use its Storable instance to make a default
> signature for other safe code and still be perfectly safe.
>
> It gives no tools for manipulating the contents of the Ptr. It is no more
> dangerous than an Int with a phantom type argument.
>
> You could randomly declare that this module is Unsafe because it combines
> badly with APIs that would be safe if you could rely on any Ptr T actually
> pointing to a T, and that users could then be granted the power to ferry
> them around, but we don't trust a user to be able to do that today.
>
> It's the combinators that read/write to a Ptr are the dangerous bits, not
> pure math.
>
> -Edward
>
>
> On Wed, Aug 10, 2016 at 10:23 AM, Ryan Newton <rrnew...@gmail.com> wrote:
>
>> Hi Edward,
>>
>> On Tue, Aug 9, 2016 at 11:58 PM, Edward Kmett <ekm...@gmail.com> wrote:
>>>
>>> 1.) If you remove IO from being able to be compiled inside Safe code _at
>>> all_ most packages I have that bother to expose Safe information will have
>>> to stop bothering.
>>>
>>
>> I definitely wouldn't argue for removing it entirely.  But it's good to
>> know that there are instances where IO functions get mixed up in safe
>> modules.  I'll try to systematically find all of these on hackage, but in
>> the meantime do you have a sample list of modules?
>>
>> My modest starting proposal is marking certain Foreign.* modules as
>> Unsafe rather than Trustworthy.  We'll find all the modules affected.  But,
>> again, are there any modules you know of offhand that are affected?  They
>> should fall into two categories:
>>
>>1. Safe modules that must become Trustworthy (if they import Foreign
>>bits, but don't expose the ability to corrupt memory to the clients of
>>their APIs).
>>2. Safe modules that must become Unsafe or be split further into
>>smaller modules.
>>
>> Obviously (2) is the biggest source of potential disruption.
>>
>> I wouldn't ask anyone to accept a patch on GHC until we'd explored these
>> impacts pretty thoroughly.
>>
>> I'd have to cut up too many APIs into too many fine-grained pieces.
>>>
>>
>> Yeah, the module-level business is pretty annoying.  "vector' removed
>> ".Safe" modules and no one has gotten around to adding the ".Unsafe".
>>
>>
>>> 2.) Assuming instead that you're talking about a stronger-than-Safe
>>> additional language extension, say ReallySafe or SafeIO, it all comes down
>>> to what the user is allowed to do in IO, doesn't it? What effects are users
>>> granted access to? We don't have a very fine-grained system for IO-effect
>>> management, and it seems pretty much any choice you pick for what to put in
>>> the sandbox will be wrong for some users, so you'd need some sort of pragma
>>> for each IO operation saying what bins it falls into and to track that
>>> while type checking, etc.
>>>
>>
>> Well, *maybe* it is a slippery slope that leads to a full effect
>> system.  But I'd like to see these issues enumerated.  Does memory safety
>> as a goal really involve so many different effects?  Do you think there
>> will be 1, 3, 10, or 100 things beyond Foreign.Ptr to worry about?
>>
>> 3.) On the other hand, someone could _build_ an effect system in Haskell
>>> that happens to sit on top of IO, holding effects in an HList, undischarged
>>> nullary class constraint, etc.
>>>
>>
>> Well, sure, I hope we will continue to aim for this as well.  This is
>> effectively what we do with our "LVish" Par monad, where we use Safe
>> Haskell to ensure users cannot break the effect system in -XSafe code.
>>
>> Best,
>>  -Ryan
>>
>
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Is Safe Haskell intended to allow segfaults?

[Ryan Newton]

Hi Simon,

On Wed, Aug 10, 2016 at 4:24 AM, Simon Marlow >
wrote:

> Definining a safe subset of IO is usually  an application-specific
> decision, e.g. do you want to allow access to the filesystem but without
> allowing openFile "/dev/mem"?  It's a minefield.
>

I respect your intuition that this is a "minefield" (and Edward's "big
mess"), but I still want to unpack this further.

I would propose that a MemSafe app is a two-part contract that is half an
OS responsibility and half language/runtime responsibility.

   1. OS: guarantee that all system calls (including file access) from the
   process do not violate the processes memory, outside of certain opt-in DMA
   regions.
   2. Lang/runtime: create a binary "ensuring" that all instructions
   executed in the process maintain the type safety of memory and follow the
   memory model.  (Scare quotes due to large TCB.)

With this division, "/dev/mem" and subprocess/ptrace are definitely the job
of the OS or containerization.  Blame for a bug would fall to category 1.

A MemSafe app needs a launcher/harness to check that category 1 is enforced
properly.  I'm not proposing that GHC needs to generate that.  It should be
separate.

Ryan, if you want to give an operational semantics for memory in IO, why
> not start from the subset of IO that you can accurately model - the basic
> IO structure together with a set of primitives - and define the semantics
> for that?  That's typically what we do when we're talking about something
> in the IO monad.
>

Yes, indeed, that has been the

approach  for
decades (and the same thing you and I do with other monads like Par
).
There's something unsatisfying here though -- we love to build model
languages that include the effects we want to talk about *at that time* --
perhaps just MVars, just putChar, or just STM.  But of course there's a big
leap from these small treatments to GHC, with its multitude of moving
parts.  That's partly why I think SafeHaskell is so great; it's a Full
Language which aims for serious, statically enforced guarantees.

Well, almost a full language ;-).  You can't run programs in it, unless you
*extend* the TCB.  It's kind of like Rust if you were required to use an
"unsafe" block to write main().

Anyway, here's a draft paper that does propose a small model language.  It
adds a memory model to Haskell plus IORefs/STRefs/TVars, and explains how,
when executed on a machine with relaxed TSO memory (store buffers), the IO
writes need to be fenced, but the ST and STM ones don't:

   http://www.cs.indiana.edu/~rrnewton/papers/sc-haskell_draft.pdf

Do we need a memory model?  I concur with David Terei's arguments from 5
years ago
.
Note however, that the implementation in this paper is TSO only.
Implementation on ARM would be a bit different, but the relationship
between IO/ST and IO/STM would stay the same.

Ryan Yates & others, I'm curious if you think the treatment of STM is
satisfactory.  Like ppopp05, we do not model retrying explicitly, and our
goal here is to show the interaction of the different effects.

Cheers,
  -Ryan
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Is Safe Haskell intended to allow segfaults?

[Ryan Newton]

Heh, ok, segfaults themselves are a red herring.  More precisely:

The operational semantics for a SafeIO language should always accurately
model its memory state.  The application should not compute (take a step in
the semantics) in a way that exposes corrupt memory or arbitrary undefined
behavior.  Nor should it violate the memory model.

Moving immediately to the "Terminated" state is fine, whether it be due to
out-of-memory, kill -SEGV, cosmic rays, or hardware failure.  An FFI call
that corrupts memory is bad (may result in arbitrary behavior, not just
termination) as is ptrace'ing.

Naturally, all Unsafe code is part of the TCB, as is the OS and GHC, and
low-level data structure libs and bindings.  It's a big TCB.  Still, it's
something to be able to write an app that doesn't automatically get added
to this TCB just by virtue of being an *app* (main::IO).




On Tue, Aug 9, 2016 at 10:52 PM, Brandon Allbery 
wrote:

> On Tue, Aug 9, 2016 at 4:19 PM, Edward Z. Yang  wrote:
>
>> If you can execute subprocesses, you could always spawn gdb to
>> attach via ptrace() to the parent process and then poke around
>> memory.
>>
>
> Don't even need that if you're just talking segfaults, you can always
> spawn a subprocess "kill -SEGV $PPID" :)
>
> Unless you have full control over all the code that could be run in
> subprocesses, it's not going to be safe much less Safe.
>
> --
> brandon s allbery kf8nh   sine nomine
> associates
> allber...@gmail.com
> ballb...@sinenomine.net
> unix, openafs, kerberos, infrastructure, xmonad
> http://sinenomine.net
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Is Safe Haskell intended to allow segfaults?

[Ryan Newton]

I'm hearing that Safe Haskell is great for pure use cases (lambda bot).
But that doesn't depend on being able to write arbitrary IO code inside the
Safe bubble, does it?  In fact *all* of IO could be outside the safe
boundary for this use case, could it not?  Are there any existing cases
where it is important to be able to build up unsafe IO values inside -XSafe
code?

Edward, why does it seem like a losing proposition?  Are there further
problems that come to mind?  ezyang mentioned the subprocess problem.  I
don't have a strong opinion on that one.  But I tend to think the safe IO
language *should* allow subprocess calls, and its a matter of configuring
your OS to not allow ptrace in that situation.  This would be part of a set
of requirements for how to compile and launch a complete "Safe Haskell"
*program* in order to get a guarantee.

My primary interest is actually not segfault-freedom, per-se, but being
able to define a memory model for Safe Haskell (for which I'd suggest
sequential consistency).  FFI undermines that, and peek/poke seems like it
should cluster with FFI as an unsafe feature.  I'm not inclined to give a
memory model to peek or FFI -- at that level you get what the architecture
gives you -- but I do want a memory model for IORefs, IOVectors, etc.

We're poking at the Stackage package set now to figure out what pressure
point to push on to increase the percentage of Stackage that is Safe.  I'll
be able to say more when we have more data on dependencies and problem
points.  Across all of hackage, Safe Haskell has modest use: of the ~100K
modules on Hackage, ~636 are marked Safe, ~874 trustworthy, and ~118
Unsafe.  It should be easy to check if any of this Safe code is currently
importing "Foreign.*" or using FFI.

My general plea is that we not give the imperative partition of Haskell too
much the short end of the stick [1]. There is oodles of code in IO (or
MonadIO), and probably relatively little in "RIO".  To my knowledge, we
don't have great ways to coin "RIO" newtypes without having to wrap and
reexport rather a lot of IO functions.  Maybe if APIs like MVars or files
were overloaded in a class then GND could do some of the work...

  -Ryan

[1] In safety guarantees, in optimizations, primops, whatever...  For
instance, I find in microbenchmarks that IO code still runs 2X slower than
pure code, even if no IO effects are performed.



On Tue, Aug 9, 2016 at 5:13 PM, Edward Kmett <ekm...@gmail.com> wrote:

> I've always treated Safe Haskell as "Safe until you allow IO" -- in that
> all 'evil' things get tainted by an IO type that you can't get rid of by
> the usual means. So if you go to run pure Safe Haskell code in say,
> lambdabot, which doesn't give the user a means to execute IO, it can't
> segfault if all of the Trustworthy modules you depend upon actually are
> trustworthy.
>
> Trying to shore up segfault safety under Safe in IO seems like a losing
> proposition.
>
> -Edward
>
> On Mon, Aug 8, 2016 at 1:27 PM, Ryan Newton <rrnew...@gmail.com> wrote:
>
>> We're trying to spend some cycles pushing on Safe Haskell within the
>> stackage packages.  (It's looking like a slog.)
>>
>> But we're running up against some basic questions regarding the core
>> packages and Safe Haskell guarantees.  The manual currently says:
>> <https://downloads.haskell.org/~ghc/latest/docs/html/users_guide/safe_haskell.html#safe-language>
>>
>>
>> *Functions in the IO monad are still allowed and behave as usual. *
>> As usual?  So it is ok to segfault GHC?  Elsewhere it says "in the safe
>> language you can trust the types", and I'd always assumed that meant Safe
>> Haskell is a type safe language, even in the IO fragment.
>>
>> Was there an explicit decision to allow segfaults and memory corruption?
>> This can happen not just with FFI calls but with uses of Ptrs within
>> Haskell, for example the following:
>>
>>
>> ```
>>
>> {-# LANGUAGE Safe #-}
>>
>> module Main where
>>
>> import Foreign.Marshal.Alloc
>>
>> import Foreign.Storable
>>
>> import Foreign.Ptr
>>
>> import System.Random
>>
>>
>> fn :: Ptr Int -> IO ()
>>
>> fn p = do
>>
>>   -- This is kosher:
>>
>>   poke p 3
>>
>>   print =<< peek p
>>
>>   -- This should crash the system:
>>
>>   ix <- randomIO
>>
>>   pokeElemOff p ix 0xcc
>>
>>
>>
>> main = alloca fn
>>
>> ```
>>
>>
>>   -Ryan
>>
>> ___
>> ghc-devs mailing list
>> ghc-devs@haskell.org
>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>>
>>
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Is Safe Haskell intended to allow segfaults?

[Ryan Newton]

On Mon, Aug 8, 2016 at 8:46 PM, Mikhail Glushenkov  wrote:
>
> Yes, this can be done with JNI, see e.g. [1]. Additionally, by using
> sun.misc.Unsafe [2], one can cause segfaults even from pure Java.
> [1] https://www.cs.princeton.edu/~appel/papers/safejni.pdf
> [2] http://www.inf.usi.ch/faculty/lanza/Downloads/Mast2015a.pdf


Ah, I see. I thought that, ruling out FFI, that you couldn't segfault with
pure Java code.  Good to know about the unsafe interface.

On Mon, Aug 8, 2016 at 7:32 PM, David Terei  wrote:
>
> If you have the energy, it'd be great to put some of this thinking
> into a wiki page (https://ghc.haskell.org/trac/ghc/wiki/SafeHaskell)
> and flesh out a first approximation of what IO API's cause issues. Is
> it just Ptr not carrying bounds around with it? Maybe, but then we
> need to secure how Ptr's can be created, which excludes FFI returning
> Ptr's.
>

Yes, we can add a Wiki page.

Btw I was thinking more of kicking FFI/peek/poke outside of the Safe
bubble, not changing their operational behavior.  First of all, it's nigh
impossible to lock down FFI calls.

When someone, e.g. Bob Harper
,
points out a problem in Haskell, we sometimes respond "hey, *Safe Haskell*
is the real objet d'art!  It's a safe language."  Yet it isn't really a
full *language* if people cannot write and run programs in it!  (Because
every program must be ultimately be `main::IO()`.)  Kicking out segfaulty
features would still leave a safe language that people can write complete
programs in.

Best,
  -Ryan
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Is Safe Haskell intended to allow segfaults?

[Ryan Newton]

Hi Ed,

Thanks, I went back to the paper and read those sections.

It sounds like this design heavily reflects the untrusted-code use case.  I
believe ceding the entire IO monad is too pessimistic, and it's also
against the *spirit* of the type safety guarantee mentioned in the paper:

"Type safety. In Milner’s famous phrase, well-typed programs do not go
wrong. "
There's not really very much discussion of this "punting on IO" decision in
the paper.  The paper mentions not deleting users files as an example of a
higher level security policy -- and a reason not  to try to lock down IO --
but that's not really the issue here.

Sure, there are many use cases that require an RIO newtype, but the memory
model and memory protection are really internal to the language.  My
interest is in a type safe language with a memory model *in the IO monad*.
I think it's quite *reasonable to expect Safe Haskell to be as safe as
Java! * There are hundreds of thousands or millions of lines of code
written in the IO monad [1], and a lot of that code could probably be
memory safe by construction.

One question I have for people is which of the following they would favor:

   1. Redefine -XSafe to guarantee something about IO and its memory safety
   (and even its memory model)
   2. Add a fourth safety level, "SafeExceptIO", that corresponds to the
   current guarantees, while extending "Safe" to say something about IO.
   3. Leave safe Haskell as it is.

(2) sounds a bit clunky to me, and I favor (1) most of all.

Best,
  -Ryan

[1] 17M lines on hackage total, hard to count how much is in an IO monad or
related monad.


On Mon, Aug 8, 2016 at 2:05 PM, Edward Z. Yang  wrote:

> Hello Ryan,
>
> The guarantee that Safe Haskell gives with regards to IO is a little
> subtle and is mentioned in Section 3.1 of the paper, and detailed
> in Section 5.1. Essentially, to use Safe Haskell, you are responsible
> for defining the type at which untrusted code is to be called.
> Using an untrusted value at type IO a in main imposes no safety
> restrictions by design--it's up to the user of Safe Haskell to
> decide what kind of security properties it needs out of user code.
>
> Edward
>
> Excerpts from Ryan Newton's message of 2016-08-08 13:27:16 -0400:
> > We're trying to spend some cycles pushing on Safe Haskell within the
> > stackage packages.  (It's looking like a slog.)
> >
> > But we're running up against some basic questions regarding the core
> > packages and Safe Haskell guarantees.  The manual currently says:
> >  users_guide/safe_haskell.html#safe-language>
> >
> >
> > *Functions in the IO monad are still allowed and behave as usual. *
> > As usual?  So it is ok to segfault GHC?  Elsewhere it says "in the safe
> > language you can trust the types", and I'd always assumed that meant Safe
> > Haskell is a type safe language, even in the IO fragment.
> >
> > Was there an explicit decision to allow segfaults and memory corruption?
> > This can happen not just with FFI calls but with uses of Ptrs within
> > Haskell, for example the following:
> >
> >
> > ```
> >
> > {-# LANGUAGE Safe #-}
> >
> > module Main where
> >
> > import Foreign.Marshal.Alloc
> >
> > import Foreign.Storable
> >
> > import Foreign.Ptr
> >
> > import System.Random
> >
> >
> > fn :: Ptr Int -> IO ()
> >
> > fn p = do
> >
> >   -- This is kosher:
> >
> >   poke p 3
> >
> >   print =<< peek p
> >
> >   -- This should crash the system:
> >
> >   ix <- randomIO
> >
> >   pokeElemOff p ix 0xcc
> >
> >
> >
> > main = alloca fn
> >
> > ```
> >
> >
> >   -Ryan
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Is Safe Haskell intended to allow segfaults?

[Ryan Newton]

>
> Pretty sure it's impossible to allow IO without enabling all of it one way
> or another. And as soon as you allow *any* IO, you're open to various kinds
> of failures including segfaults. The only way you will get your type system
> to prevent that is fully specified dependent typing both in your program
> *and in the OS level interfaces*. I wish you luck on the latter.
>

I don't agree.  This would seem to imply that type safe / memory safe
imperative languages are impossible, and that OCaml/Java/C#/etc are a false
promise!

Which APIs and dangers are you thinking of?  I'm worried about bounds
checks on memory access, and the capability of foreign code to poke at
arbitrary memory.

But other APIs like these should be fine:

   - IORef API
   - MVar API
   - STM API
   - File IO API
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Is Safe Haskell intended to allow segfaults?

[Ryan Newton]

We're trying to spend some cycles pushing on Safe Haskell within the
stackage packages.  (It's looking like a slog.)

But we're running up against some basic questions regarding the core
packages and Safe Haskell guarantees.  The manual currently says:



*Functions in the IO monad are still allowed and behave as usual. *
As usual?  So it is ok to segfault GHC?  Elsewhere it says "in the safe
language you can trust the types", and I'd always assumed that meant Safe
Haskell is a type safe language, even in the IO fragment.

Was there an explicit decision to allow segfaults and memory corruption?
This can happen not just with FFI calls but with uses of Ptrs within
Haskell, for example the following:


```

{-# LANGUAGE Safe #-}

module Main where

import Foreign.Marshal.Alloc

import Foreign.Storable

import Foreign.Ptr

import System.Random


fn :: Ptr Int -> IO ()

fn p = do

  -- This is kosher:

  poke p 3

  print =<< peek p

  -- This should crash the system:

  ix <- randomIO

  pokeElemOff p ix 0xcc



main = alloca fn

```


  -Ryan
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Benchmarking harnesses for a more modern nofib?

[Ryan Newton]

FYI, moving discussion off the ghc-devs list to avoid spamming it.  Check
out this mailing list if interested:

https://groups.google.com/forum/#!forum/haskell-benchmark-infrastructure

So far, we're discussing benchmark harnesses and perf dashboards, but I'm
sure we'll get to the benchmarks themselves at some point.






On Tue, Apr 5, 2016 at 11:59 AM, Carter Schonwald <
carter.schonw...@gmail.com> wrote:

> The cause of the initial blowup was adding the vector test suite to the
> normal cabal file, in both 0O and O2 forms.  This was done so that fusion
> rules based bugs wouldn't lie in hiding for years.
>
> It sounds like the issues that resulted in built blowup are fixed in 8.0.
> My guess is it's a combination of some type class coercion / equality
> manipulation getting better plus other improvements in 8.0, both.
>
>
> On Tuesday, April 5, 2016, Dominic Steinitz <domi...@steinitz.org> wrote:
>
>> Ryan Newton  gmail.com> writes:
>>
>> >  Hi all, Is anyone currently working in, or interested in helping
>> > with, a new benchmark suite for Haskell?  Perhaps, packaging up
>> > existing apps and app benchmarks into a new benchmark suite that
>> > gives a broad picture of Haskell application performance today?
>>
>> I am very interested. I recently encountered a serious performance
>> regression from 7.8 to 7.10 which seems to be fixed in 8.0. Now it's
>> not clear whether this was a library change or ghc itself. I suspect
>> the latter but given the performance is better in 8.0, I was not
>> motivated to confirm this.
>>
>> I am happy to wrap up my example in whatever format but note that it
>> does pull in quite a few libraries.
>>
>> Dominic.
>> ___
>> ghc-devs mailing list
>> ghc-devs@haskell.org
>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>>
>
> ___
> ghc-devs mailing list
> ghc-devs@haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Benchmarking harnesses for a more modern nofib?

[Ryan Newton]

Sorry, bit too late: excuse typos:

Also, these days companies are building substantial apps in Hackage.
>

In *Haskell*, of course.
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Benchmarking harnesses for a more modern nofib?

[Ryan Newton]

Hi all,

Is anyone currently working in, or interested in helping with, a new
benchmark suite for Haskell?  Perhaps, packaging up existing apps and app
benchmarks into a new benchmark suite that gives a broad picture of Haskell
application performance today?

Background: We run nofib, and we run the shootout benchmarks.  But when we
want to evaluate basic changes to GHC optimizations or data representation,
these really don't give us a full picture of whether a change is beneficial.

A few years ago, fibon  tried to
gather some Hackage benchmarks.  This may work even better with Stackage,
where there are 180 benchmark suites among the 1770 packages currently.

Also, these days companies are building substantial apps in Hackage.  Which
substantial apps could or should go into a benchmark suite?  I see Warp and
other web server benchmarks

all
over the web.  But is there a harness that can time some of this code while
running inside a single-machine, easy-setup benchmark suite?

Best,
  -Ryan
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Semantics of IORefs in GHC

[Ryan Newton]

Hi Madan,

Yes, GHC, like Java, is an "unsafe language" currently ;-).  It does not
protect its abstractions against buggy programs.  Namely both Haskell and
Java do not protect semi-colon/(>>).

Really, we should probably have some additional monad to distinguish
data-race-free (DRF) IO programs from other IO programs that may have data
races.

Or we could just make sequential consistency the law of the land for IO, as
you propose.  One thing I'd really like to work on -- if someone were
interested in collaborating -- is testing the overhead of making sequential
composition the norm in this way.

I had a nice conversation with your SNAPL co-author Satish about this
recently.  It seems like we could survey a broad swath of Haskell code to
see if fine-grained use of "writeIORef" and "readIORef" are at all common.
My hunch is that IORefs used in concurrent apps (web servers, etc), all use
atomicModifyIORef anyway.

Thus like you I think we could fence read/writeIORef with acceptable perf
for most real apps.  You can still have "reallyUnsafeReadIORef" for
specific purposes like implementing concurrent data structures.

Best,
 -Ryan



On Mon, Mar 14, 2016 at 10:06 AM, Simon Peyton Jones <simo...@microsoft.com>
wrote:

> Maclan
>
>
>
> I’m glad you enjoyed the awkward squad paper.
>
>
>
> I urge you to write to the Haskell Café mailing list and/or ghc-devs.
> Lots of smart people there.  Ryan Newton is working on this kind of stuff;
> I’ve cc’d him.
>
>
>
> But my rough answer would be: IORefs are really only meant for
> single-threaded work.  Use STM for concurrent communication.
>
>
>
> That’s not to say that we are done!  The Haskell community doesn’t have
> many people like you, who care about the detail of the memory model.  So
> please do help us J.   For example, perhaps we could guarantee a simple
> sequential memory model without much additional cost?
>
>
>
> Simon
>
>
>
> *From:* Madan Musuvathi
> *Sent:* 11 March 2016 19:35
> *To:* Simon Peyton Jones <simo...@microsoft.com>
> *Subject:* Semantics of IORefs in GHC
>
>
>
> Dear Simon,
>
> I really enjoyed reading your awkward squad paper
> <http://research.microsoft.com/en-us/um/people/simonpj/papers/marktoberdorf/mark.pdf>.
> Thank you for writing such an accessible paper.
>
>
>
> My current understanding is that the implementation of IORefs in GHC
> breaks the simple semantics you develop in this paper. In particular, by
> not inserting sufficient fences around reads and writes of IORefs, a
> Haskell program is exposed to the weak-memory-consistency effects of the
> underlying hardware and possibly the backend C compiler. As a result, the
> monadic bind operator no longer has the simple semantics of sequential
> composition. Is my understanding correct?
>
>
>
> This is very troublesome as this weaker semantics can lead to unforeseen
> consequences even in pure functional parts of a program. For example, when
> a reference to an object is passed through an IORef to another thread, the
> latter thread is not guaranteed to see the updates of the first thread. So,
> it is quite possible for some (pure functional) code to be processing
> objects with broken invariants or partially-constructed objects. In the
> extreme, this could lead to type-unsafety unless the GHC compiler is taking
> careful precautions to avoid this. (Many of these problems are unlikely to
> show up on x86 machines, but will be common on ARM.)
>
>
>
> I am sure the GHC community is addressing these problems one way or the
> other. But, my question is WHY?  Why can’t GHC tighten the semantics of
> IORefs so that the bind operation simply means sequential composition?
> Given that Haskell has a clean separation between pure functional parts and
> “awkward” parts of the program, the overheads of these fenced IORefs should
> be acceptable.
>
>
>
> My coauthors and I wrote a recent SNAPL article
> <http://research.microsoft.com/apps/pubs/default.aspx?id=252150> about
> this problem for other (“less-beautiful” J) imperative languages like C#
> and Java. I really believe we should support sequential composition in our
> programming languages.
>
>
>
> madan
>
>
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Better calling conventions for strict functions (bang patterns)?

[Ryan Newton]

Thanks Simon for linking that issue!  Does the patch linked there
<https://ghc.haskell.org/trac/ghc/attachment/ticket/8905/0001-EXPERIMENTAL-save-the-continuation-after-the-tag-che.patch>
already
cover what you suggested in your last mail?  I think no, it's a more
limited change, but I'm having trouble understanding exactly what.

I've also got one really basic question -- was it decided long ago that all
these stack limit checks are cheaper than having a guard page at the end of
each stack and faulting to grow the stack?  (I couldn't find a place that
this rationale was described on wiki.)

Best,
  -Ryan


On Sun, Nov 1, 2015 at 10:05 AM, Simon Marlow <marlo...@gmail.com> wrote:

> Yes, I think we can probably do a better job of compiling case
> expressions.  The current compilation strategy optimises for code size, but
> at the expense of performance in the fast path.  We can tweak this
> tradeoff, perhaps under the control of a flag.
>
> Ideally the sequence should start by assuming that the closure is already
> evaluated, e.g.
>
>  loop:
>tag = R2 & 7;
>if (tag == 1) then // code for []
>else if (tag == 2) then // code for (:)
>else evaluate; jump back to loop
>
> The nice thing is that now that we don't need proc points, "loop" is just
> a label that we can directly jump to.  Unfortunately this only works when
> using the NCG, not with LLVM, because LLVM requires proc points, so we
> might need to fall back to a different strategy for LLVM.
>
> Similar topics came up here: https://ghc.haskell.org/trac/ghc/ticket/8905
> and I think there was another ticket but I can't find it now.
>
> Cheers
> Simon
>
> On 23/10/2015 19:00, Ryan Newton wrote:
>
>>  1. Small tweaks: The CMM code above seems to be /betting/ than the
>> thunk is unevaluated, because it does the stack check and stack
>> write /before/ the predicate test that checks if the thunk is
>> evaluated (if(R1 & 7!= 0) gotoc3aO; elsegotoc3aP;).  With a
>> bang-pattern function, couldn't it make the opposite bet?  That
>> is, branch on whether the thunk is evaluated first, and then the
>> wasted computation is only a single correctly predicted branch
>> (and a read of a tag that we need to read anyway).
>>
>> Oh, a small further addition would be needed for this tweak.  In the
>> generated code above "Sp = Sp + 8;" happens /late/, but I think it could
>> happen right after the call to the thunk.  In general, does it seem
>> feasible to separate the slowpath from fastpath as in the following
>> tweak of the example CMM?
>>
>>
>> *  // Skip to the chase if it's already evaluated:*
>> *  start:*
>> *  if (R2 & 7 != 0) goto fastpath; else goto slowpath;*
>> *
>> *
>> *  slowpath:   // Formerly c3aY*
>> *  if ((Sp + -8) < SpLim) goto c3aZ; else goto c3b0;*
>> *  c3aZ:*
>> *  // nop*
>> *  R1 = PicBaseReg + foo_closure;*
>> *  call (I64[BaseReg - 8])(R2, R1) args: 8, res: 0, upd: 8;*
>> *  c3b0:*
>> *  I64[Sp - 8] = PicBaseReg + block_c3aO_info;*
>> *  R1 = R2;*
>> *  Sp = Sp - 8;*
>>
>> *  call (I64[R1])(R1) returns to fastpath, args: 8, res: 8, upd: 8;*
>> *  // Sp bump moved to here so it's separate from "fastpath"*
>> *  Sp = Sp + 8;*
>> *
>> *
>> *  fastpath: // Formerly c3aO*
>> *  if (R1 & 7 >= 2) goto c3aW; else goto c3aX;*
>> *  c3aW:*
>> *  R1 = P64[R1 + 6] & (-8);*
>> *  call (I64[R1])(R1) args: 8, res: 0, upd: 8;*
>> *  c3aX:*
>> *  R1 = PicBaseReg + lvl_r39S_closure;*
>> *  call (I64[R1])(R1) args: 8, res: 0, upd: 8;*
>>
>>
>>
>>
>>
>> ___
>> ghc-devs mailing list
>> ghc-devs@haskell.org
>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>>
>>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Better calling conventions for strict functions (bang patterns)?

[Ryan Newton]

Hi all,

With module-level Strict and StrictData pragmas coming soon, one obvious
question is what kind of the code quality GHC can achieve for strict
programs.

When it came up in discussion in our research group we realized we didn't
actually know whether the bang patterns, `f !x`, on function arguments were
enforced by caller or callee.

Here's a Gist that shows the compilation of a trivial function:

foo :: Maybe Int -> Intfoo !x = case x of Just y -> y

   https://gist.github.com/rrnewton/1ac722189c65f26fe9ac

If that function is compiled to *assume* its input is in WHNF, it should be
just as efficient as the isomorphic MLton/OCaml code, right?  It only needs
to branch on the tag, do a field dereference, and return.

But as you can see from the STG and CMM generated, foo *does indeed* enter
the thunk, adding an extra indirect jump.  Here's the body:

c3aY: if ((Sp + -8) < SpLim) goto c3aZ; else goto c3b0; c3aZ: // nop R1 =
PicBaseReg + foo_closure; call (I64[BaseReg - 8])(R2, R1) args: 8, res: 0,
upd: 8; c3b0: I64[Sp - 8] = PicBaseReg + block_c3aO_info; R1 = R2; Sp = Sp
- 8; if (R1 & 7 != 0) goto c3aO; else goto c3aP; c3aP: call (I64[R1])(R1)
returns to c3aO, args: 8, res: 8, upd: 8; c3aO: if (R1 & 7 >= 2) goto c3aW;
else goto c3aX; c3aW: R1 = P64[R1 + 6] & (-8); Sp = Sp + 8; call
(I64[R1])(R1) args: 8, res: 0, upd: 8; c3aX: R1 = PicBaseReg +
lvl_r39S_closure; Sp = Sp + 8; call (I64[R1])(R1) args: 8, res: 0, upd: 8;


The call inside c3aP is entering "x" as a thunk, which also incurs all of
the stack limit check code.  I believe that IF the input could be assumed
to be in WHNF, everything above the label "c3aO" could be omitted.

So... if GHC is going to be a fabulous pure *and* imperative language, and
a fabulous lazy *and* strict compiler/runtime.. is there some work we can
do here to improve this situation? Would the following make sense:

   - Put together a benchmark suite of all-strict programs with
   Strict/StrictData (compare a few benchmark's generated code to MLton, if
   time allows)
   - Modify GHC to change calling conventions for bang patterns -- caller
   enforces WHNF rather than callee.  Existing strictness/demand/cardinality
   analysis would stay the same.

Unless there's something I'm really missing here, the result should be that
you can have a whole chain of strict function calls, each of which knows
its arguments and the arguments it passes to its callees are all in WHNF,
without ever generating thunk-entry sequences.

Thanks for your time,
  -Ryan
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Better calling conventions for strict functions (bang patterns)?

[Ryan Newton]

>
>
>1. Small tweaks: The CMM code above seems to be *betting* than the
>thunk is unevaluated, because it does the stack check and stack write
>*before* the predicate test that checks if the thunk is evaluated (if
>(R1 & 7 != 0) goto c3aO; else goto c3aP;).  With a bang-pattern
>function, couldn't it make the opposite bet?  That is, branch on whether
>the thunk is evaluated first, and then the wasted computation is only a
>single correctly predicted branch (and a read of a tag that we need to read
>anyway).
>
> Oh, a small further addition would be needed for this tweak.  In the
generated code above "Sp = Sp + 8;" happens *late*, but I think it could
happen right after the call to the thunk.  In general, does it seem
feasible to separate the slowpath from fastpath as in the following tweak
of the example CMM?


*  // Skip to the chase if it's already evaluated:*
*  start:*
*  if (R2 & 7 != 0) goto fastpath; else goto slowpath;*

*  slowpath:   // Formerly c3aY*
*  if ((Sp + -8) < SpLim) goto c3aZ; else goto c3b0;*
*  c3aZ:*
*  // nop*
*  R1 = PicBaseReg + foo_closure;*
*  call (I64[BaseReg - 8])(R2, R1) args: 8, res: 0, upd: 8;*
*  c3b0:*
*  I64[Sp - 8] = PicBaseReg + block_c3aO_info;*
*  R1 = R2;*
*  Sp = Sp - 8;*

*  call (I64[R1])(R1) returns to fastpath, args: 8, res: 8, upd: 8;*
*  // Sp bump moved to here so it's separate from "fastpath"*
*  Sp = Sp + 8;*

*  fastpath: // Formerly c3aO*
*  if (R1 & 7 >= 2) goto c3aW; else goto c3aX;*
*  c3aW:*
*  R1 = P64[R1 + 6] & (-8);*
*  call (I64[R1])(R1) args: 8, res: 0, upd: 8;*
*  c3aX:*
*  R1 = PicBaseReg + lvl_r39S_closure;*
*  call (I64[R1])(R1) args: 8, res: 0, upd: 8;*
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Better calling conventions for strict functions (bang patterns)?

[Ryan Newton]

Ah, yes, so just to give a concrete example in this thread, if we take the
`foo` function above and say `map foo ls`, we may well get unevaluated
arguments to foo.  (And this is almost precisely the same as the first
example that Strict-Core paper!)

Thanks for the paper reference.  I read it and it's great -- just what I
was looking for.  An approach that eliminates any jealousy of ML/Scheme
compiler techniques vis a vis calling conventions ;-).  I'm also wondering
if there are some incremental steps that can be taken, short of what is
proposed in the paper.

   1. Small tweaks: The CMM code above seems to be *betting* than the thunk
   is unevaluated, because it does the stack check and stack write *before* the
   predicate test that checks if the thunk is evaluated (if (R1 & 7 != 0)
   goto c3aO; else goto c3aP;).  With a bang-pattern function, couldn't it
   make the opposite bet?  That is, branch on whether the thunk is evaluated
   first, and then the wasted computation is only a single correctly predicted
   branch (and a read of a tag that we need to read anyway).

   2. The option of multiple entrypoints which is considered and discarded
   as fragile in the beginning of the paper (for direct call vs indirect / 1st
   order vs higher order).  That fragile option is along the lines of what I
   wanted to discuss on this thread.  It does seem like a tricky phase
   ordering concern, but how bad is it exactly?  The conflict with the a
   case-expr rewrite is illustrated clearly in the paper, but that just means
   that such optimizations must happen *before *the final choice of which
   function entrypoint to call, doesn't it?  I'm not 100% sure where it could
   go in the current compiler pipeline, but couldn't the adjustment of call
   target from "foo" to "foo_with_known_whnf_args" happen quite late?

Cheers,
  -Ryan

P.S. One of the students CC'd, Ryan Scott, is currently on internship at
Intel labs and is working to (hopefully) liberate the Intell Haskell
Research Compiler as open source.  Like the 2009 paper, it also uses a
strict IR, and I think it will be interesting to see exactly how it handles
the conversion from Core to its IR.  (Probably the same as Fig 10 in the
paper.)


On Fri, Oct 23, 2015 at 10:11 AM, Simon Peyton Jones <simo...@microsoft.com>
wrote:

> It’s absolutely the case that bang patterns etc tell the caller what to
> do, but the function CANNOT ASSUME that its argument is evaluated.  Reason:
> higher order functions.
>
>
>
> I think that the way to allow functions that can assume their arg is
> evaluated is through types: see Type are calling conventions
> <http://research.microsoft.com/~simonpj/papers/strict-core/tacc-hs09.pdf>.
> But it’d be a fairly big deal to implement.
>
>
>
> Simon
>
>
>
>
>
> *From:* ghc-devs [mailto:ghc-devs-boun...@haskell.org] *On Behalf Of *Ryan
> Newton
> *Sent:* 23 October 2015 14:54
> *To:* ghc-devs@haskell.org; Ömer Sinan Ağacan; Ryan Scott; Chao-Hong
> Chen; Johan Tibell
> *Subject:* Better calling conventions for strict functions (bang
> patterns)?
>
>
>
> Hi all,
>
>
>
> With module-level Strict and StrictData pragmas coming soon, one obvious
> question is what kind of the code quality GHC can achieve for strict
> programs.
>
>
>
> When it came up in discussion in our research group we realized we didn't
> actually know whether the bang patterns, `f !x`, on function arguments were
> enforced by caller or callee.
>
>
>
> Here's a Gist that shows the compilation of a trivial function:
>
> foo :: Maybe Int -> Int
>
> foo !x =
>
>   case x of
>
>Just y -> y
>
>
>
>https://gist.github.com/rrnewton/1ac722189c65f26fe9ac
> <https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fgist.github.com%2frrnewton%2f1ac722189c65f26fe9ac=01%7c01%7csimonpj%40064d.mgd.microsoft.com%7cb006dcdbfe834ebb6c1e08d2dbb16c03%7c72f988bf86f141af91ab2d7cd011db47%7c1=qxrT8r1VSP97xQUF2qqkLlxEtSGi9VOzfmORl25W%2fWY%3d>
>
>
>
> If that function is compiled to *assume* its input is in WHNF, it should
> be just as efficient as the isomorphic MLton/OCaml code, right?  It only
> needs to branch on the tag, do a field dereference, and return.
>
>
>
> But as you can see from the STG and CMM generated, foo *does indeed*
> enter the thunk, adding an extra indirect jump.  Here's the body:
>
>   c3aY:
>
>   if ((Sp + -8) < SpLim) goto c3aZ; else goto c3b0;
>
>   c3aZ:
>
>   // nop
>
>   R1 = PicBaseReg + foo_closure;
>
>   call (I64[BaseReg - 8])(R2, R1) args: 8, res: 0, upd: 8;
>
>   c3b0:
>
>   I64[Sp - 8] = PicBaseReg + block_c3aO_info;
>
>   R1 = R2;
>
>   Sp = Sp - 8;
>
> 

Re: Converting unboxed sum types in StgCmm

[Ryan Newton]

>
>
>-
>data RepType = UbxTupleRep [UnaryType]
>| UbxSumRep [UnaryType]
>| UnaryRep UnaryType
>
> Not, fully following, but ... this reptype business is orthogonal to
whether you add a normal type to the STG level that models anonymous,
untagged unions, right?

That is, when using Any for pointer types, they could use indicative
phantom types, like "Any (Union Bool Char)", even if there's not full
support for doing anything useful with (Union Bool Char) by itself.  Maybe
the casting machinery could greenlight a cast from Any (Union Bool Char) to
Bool at least?

There's already the unboxed union itself, (|# #|) , but that's different
than a pointer to a union of types...
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Converting unboxed sum types in StgCmm

[Ryan Newton]

>
> It seems to me that we just want to rewrite the case altogether into
> something that looks at the tag field of the data constructor. Also, in stg
> we use the same DataCon as in core, but after stg the unboxed sum case
> really only has one constructor (one with the union of all the fields),
> which makes it awkward to reuse the original DataCon.
>

Is there a problem with introducing a totally new datatype at this point in
the compile to represent the product (tag, wordish1, ..., wordishN, ptr1
... ptrM)?  Or, if it is an anonymous product, why can't it use existing
unboxed sum machinery?

Also, as an architecture thing, is there a reason this shouldn't be its own
stg->stg pass?

(P.S. "wordish" above has a weaselly suffix because as Dan pointed out,
some unboxed things are > 64 bits.)
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: ArrayArrays

[Ryan Newton]

Ah, incidentally that introduces an interesting difference between
atomicModify and CAS.  CAS should be able to work on mutable locations in
that subset of # that are represented by a gcptr, whereas Edward pointed
out that atomicModify cannot.

(Indeed, to use lock-free algorithms with these new unboxed mutable
structures we'll need CAS on the slots.)

On Mon, Sep 7, 2015 at 4:16 PM, Edward Kmett <ekm...@gmail.com> wrote:

> I had a brief discussion with Richard during the Haskell Symposium about
> how we might be able to let parametricity help a bit in reducing the space
> of necessarily primops to a slightly more manageable level.
>
> Notably, it'd be interesting to explore the ability to allow parametricity
> over the portion of # that is just a gcptr.
>
> We could do this if the levity polymorphism machinery was tweaked a bit.
> You could envision the ability to abstract over things in both * and the
> subset of # that are represented by a gcptr, then modifying the existing
> array primitives to be parametric in that choice of levity for their
> argument so long as it was of a "heap object" levity.
>
> This could make the menagerie of ways to pack
> {Small}{Mutable}Array{Array}# references into a
> {Small}{Mutable}Array{Array}#' actually typecheck soundly, reducing the
> need for folks to descend into the use of the more evil structure
> primitives we're talking about, and letting us keep a few more principles
> around us.
>
> Then in the cases like `atomicModifyMutVar#` where it needs to actually be
> in * rather than just a gcptr, due to the constructed field selectors it
> introduces on the heap then we could keep the existing less polymorphic
> type.
>
> -Edward
>
> On Mon, Sep 7, 2015 at 9:59 AM, Simon Peyton Jones <simo...@microsoft.com>
> wrote:
>
>> It was fun to meet and discuss this.
>>
>>
>>
>> Did someone volunteer to write a wiki page that describes the proposed
>> design?  And, I earnestly hope, also describes the menagerie of currently
>> available array types and primops so that users can have some chance of
>> picking the right one?!
>>
>>
>>
>> Thanks
>>
>>
>>
>> Simon
>>
>>
>>
>> *From:* ghc-devs [mailto:ghc-devs-boun...@haskell.org] *On Behalf Of *Ryan
>> Newton
>> *Sent:* 31 August 2015 23:11
>> *To:* Edward Kmett; Johan Tibell
>> *Cc:* Simon Marlow; Manuel M T Chakravarty; Chao-Hong Chen; ghc-devs;
>> Ryan Scott; Ryan Yates
>> *Subject:* Re: ArrayArrays
>>
>>
>>
>> Dear Edward, Ryan Yates, and other interested parties --
>>
>>
>>
>> So when should we meet up about this?
>>
>>
>>
>> May I propose the Tues afternoon break for everyone at ICFP who is
>> interested in this topic?  We can meet out in the coffee area and
>> congregate around Edward Kmett, who is tall and should be easy to find ;-).
>>
>>
>>
>> I think Ryan is going to show us how to use his new primops for combined
>> array + other fields in one heap object?
>>
>>
>>
>> On Sat, Aug 29, 2015 at 9:24 PM Edward Kmett <ekm...@gmail.com> wrote:
>>
>> Without a custom primitive it doesn't help much there, you have to store
>> the indirection to the mask.
>>
>>
>>
>> With a custom primitive it should cut the on heap root-to-leaf path of
>> everything in the HAMT in half. A shorter HashMap was actually one of the
>> motivating factors for me doing this. It is rather astoundingly difficult
>> to beat the performance of HashMap, so I had to start cheating pretty
>> badly. ;)
>>
>>
>>
>> -Edward
>>
>>
>>
>> On Sat, Aug 29, 2015 at 5:45 PM, Johan Tibell <johan.tib...@gmail.com>
>> wrote:
>>
>> I'd also be interested to chat at ICFP to see if I can use this for my
>> HAMT implementation.
>>
>>
>>
>> On Sat, Aug 29, 2015 at 3:07 PM, Edward Kmett <ekm...@gmail.com> wrote:
>>
>> Sounds good to me. Right now I'm just hacking up composable accessors for
>> "typed slots" in a fairly lens-like fashion, and treating the set of slots
>> I define and the 'new' function I build for the data type as its API, and
>> build atop that. This could eventually graduate to template-haskell, but
>> I'm not entirely satisfied with the solution I have. I currently
>> distinguish between what I'm calling "slots" (things that point directly to
>> another SmallMutableArrayArray# sans wrapper) and "fields" which point
>> directly to the usual Haskell data types because unifying the two notion

Re: RFC: Unpacking sum types

[Ryan Newton]

Just a small comment about syntax.

Why is there an "_n" suffix on the type constructor?  Isn't it
syntactically evident how many things are in the |# .. | ..  #| block?

More generally, are the parser changes and the wild new syntax strictly
necessary?

Could we instead just have a new keyword, but have at look like a normal
type constructor?  For example, the type:

   (Sum# T1 T2 T3)

Where "UnboxedSum" can't be partially applied, and is variable arity.
Likewise, "MkSum#" could be a keyword/syntactic-form:

   (MkSum# 1 3 expr)
  case x of MkSum# 1 3 v -> e

Here "1" and "3" are part of the syntactic form, not expressions.  But it
can probably be handled after parsing and doesn't require the "_n_m"
business.

  -Ryan


On Tue, Sep 1, 2015 at 6:10 PM Johan Tibell  wrote:

> After some discussions with SPJ I've now rewritten the proposal in terms
> of unboxed sums (which should suffer from the extra seq problem you mention
> above).
>
> On Tue, Sep 1, 2015 at 11:31 AM, Dan Doel  wrote:
>
>> I wonder: are there issues with strict/unpacked fields in the sum
>> type, with regard to the 'fill in stuff' behavior?
>>
>> For example:
>>
>> data C = C1 !Int | C2 ![Int]
>>
>> data D = D1 !Double {-# UNPACK #-} !C
>>
>> Naively we might think:
>>
>> data D' = D1 !Double !Tag !Int ![Int]
>>
>> But this is obviously not going to work at the
>> Haskell-implemented-level. Since we're at a lower level, we could just
>> not seq the things from the opposite constructor, but are there
>> problems that arise from that? Also of course the !Int will probably
>> also be unpacked, so such prim types need different handling (fill
>> with 0, I guess).
>>
>> --
>>
>> Also, I guess this is orthogonal, but having primitive, unboxed sums
>> (analogous to unboxed tuples) would be nice as well. Conceivably they
>> could be used as part of the specification of unpacked sums, since we
>> can apparently put unboxed tuples in data types now. I'm not certain
>> if they would cover all cases, though (like the strictness concerns
>> above).
>>
>> -- Dan
>>
>>
>> On Tue, Sep 1, 2015 at 1:23 PM, Johan Tibell 
>> wrote:
>> > I have a draft design for unpacking sum types that I'd like some
>> feedback
>> > on. In particular feedback both on:
>> >
>> >  * the writing and clarity of the proposal and
>> >  * the proposal itself.
>> >
>> > https://ghc.haskell.org/trac/ghc/wiki/UnpackedSumTypes
>> >
>> > -- Johan
>> >
>> >
>> > ___
>> > ghc-devs mailing list
>> > ghc-devs@haskell.org
>> > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>> >
>>
>
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: RFC: Unpacking sum types

[Ryan Newton]

>
> If we expose it on the Haskell level, I find MkSum_1_2# the right thing
> to do: It makes it clear that (conceptually) there really is a
> constructor of that name, and it is distinct from MkSum_2_2#, and the
> user cannot do computation with these indices.
>

I don't mind MkSum_1_2#, it avoids the awkwardness of attaching it to a
closing delimiter.  But...  it does still introduce the idea of cutting up
tokens to get numbers out of them, which is kind of hacky.  (There seems to
be a conserved particle of hackiness here that can't be eliminate, but it
doesn't bother me too much.)
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: ArrayArrays

[Ryan Newton]

Dear Edward, Ryan Yates, and other interested parties --

So when should we meet up about this?

May I propose the Tues afternoon break for everyone at ICFP who is
interested in this topic?  We can meet out in the coffee area and
congregate around Edward Kmett, who is tall and should be easy to find ;-).

I think Ryan is going to show us how to use his new primops for combined
array + other fields in one heap object?

On Sat, Aug 29, 2015 at 9:24 PM Edward Kmett <ekm...@gmail.com> wrote:

> Without a custom primitive it doesn't help much there, you have to store
> the indirection to the mask.
>
> With a custom primitive it should cut the on heap root-to-leaf path of
> everything in the HAMT in half. A shorter HashMap was actually one of the
> motivating factors for me doing this. It is rather astoundingly difficult
> to beat the performance of HashMap, so I had to start cheating pretty
> badly. ;)
>
> -Edward
>
> On Sat, Aug 29, 2015 at 5:45 PM, Johan Tibell <johan.tib...@gmail.com>
> wrote:
>
>> I'd also be interested to chat at ICFP to see if I can use this for my
>> HAMT implementation.
>>
>> On Sat, Aug 29, 2015 at 3:07 PM, Edward Kmett <ekm...@gmail.com> wrote:
>>
>>> Sounds good to me. Right now I'm just hacking up composable accessors
>>> for "typed slots" in a fairly lens-like fashion, and treating the set of
>>> slots I define and the 'new' function I build for the data type as its API,
>>> and build atop that. This could eventually graduate to template-haskell,
>>> but I'm not entirely satisfied with the solution I have. I currently
>>> distinguish between what I'm calling "slots" (things that point directly to
>>> another SmallMutableArrayArray# sans wrapper) and "fields" which point
>>> directly to the usual Haskell data types because unifying the two notions
>>> meant that I couldn't lift some coercions out "far enough" to make them
>>> vanish.
>>>
>>> I'll be happy to run through my current working set of issues in person
>>> and -- as things get nailed down further -- in a longer lived medium than
>>> in personal conversations. ;)
>>>
>>> -Edward
>>>
>>> On Sat, Aug 29, 2015 at 7:59 AM, Ryan Newton <rrnew...@gmail.com> wrote:
>>>
>>>> I'd also love to meet up at ICFP and discuss this.  I think the array
>>>> primops plus a TH layer that lets (ab)use them many times without too much
>>>> marginal cost sounds great.  And I'd like to learn how we could be either
>>>> early users of, or help with, this infrastructure.
>>>>
>>>> CC'ing in Ryan Scot and Omer Agacan who may also be interested in
>>>> dropping in on such discussions @ICFP, and Chao-Hong Chen, a Ph.D. student
>>>> who is currently working on concurrent data structures in Haskell, but will
>>>> not be at ICFP.
>>>>
>>>>
>>>> On Fri, Aug 28, 2015 at 7:47 PM, Ryan Yates <fryguy...@gmail.com>
>>>> wrote:
>>>>
>>>>> I completely agree.  I would love to spend some time during ICFP and
>>>>> friends talking about what it could look like.  My small array for STM
>>>>> changes for the RTS can be seen here [1].  It is on a branch somewhere
>>>>> between 7.8 and 7.10 and includes irrelevant STM bits and some
>>>>> confusing naming choices (sorry), but should cover all the details
>>>>> needed to implement it for a non-STM context.  The biggest surprise
>>>>> for me was following small array too closely and having a word/byte
>>>>> offset miss-match [2].
>>>>>
>>>>> [1]:
>>>>> https://github.com/fryguybob/ghc/compare/ghc-htm-bloom...fryguybob:ghc-htm-mut
>>>>> [2]: https://ghc.haskell.org/trac/ghc/ticket/10413
>>>>>
>>>>> Ryan
>>>>>
>>>>> On Fri, Aug 28, 2015 at 10:09 PM, Edward Kmett <ekm...@gmail.com>
>>>>> wrote:
>>>>> > I'd love to have that last 10%, but its a lot of work to get there
>>>>> and more
>>>>> > importantly I don't know quite what it should look like.
>>>>> >
>>>>> > On the other hand, I do have a pretty good idea of how the
>>>>> primitives above
>>>>> > could be banged out and tested in a long evening, well in time for
>>>>> 7.12. And
>>>>> > as noted earlier, those remain useful even if a nicer typed version
>>>>> with an
>

Re: ArrayArrays

[Ryan Newton]

You presumably also save a bounds check on reads by hard-coding the sizes?

On Fri, Aug 28, 2015 at 3:39 PM, Edward Kmett ekm...@gmail.com wrote:

 Also there are 4 different things here, basically depending on two
 independent questions:

 a.) if you want to shove the sizes into the info table, and
 b.) if you want cardmarking.

 Versions with/without cardmarking for different sizes can be done pretty
 easily, but as noted, the infotable variants are pretty invasive.

 -Edward

 On Fri, Aug 28, 2015 at 6:36 PM, Edward Kmett ekm...@gmail.com wrote:

 Well, on the plus side you'd save 16 bytes per object, which adds up if
 they were small enough and there are enough of them. You get a bit better
 locality of reference in terms of what fits in the first cache line of them.

 -Edward

 On Fri, Aug 28, 2015 at 6:14 PM, Ryan Newton rrnew...@gmail.com wrote:

 Yes. And for the short term I can imagine places we will settle with
 arrays even if it means tracking lengths unnecessarily and unsafeCoercing
 pointers whose types don't actually match their siblings.

 Is there anything to recommend the hacks mentioned for fixed sized array
 objects *other* than using them to fake structs? (Much to derecommend, as
 you mentioned!)

 On Fri, Aug 28, 2015 at 3:07 PM Edward Kmett ekm...@gmail.com wrote:

 I think both are useful, but the one you suggest requires a lot more
 plumbing and doesn't subsume all of the usecases of the other.

 -Edward

 On Fri, Aug 28, 2015 at 5:51 PM, Ryan Newton rrnew...@gmail.com
 wrote:

 So that primitive is an array like thing (Same pointed type, unbounded
 length) with extra payload.

 I can see how we can do without structs if we have arrays, especially
 with the extra payload at front. But wouldn't the general solution for
 structs be one that that allows new user data type defs for # types?



 On Fri, Aug 28, 2015 at 4:43 PM Edward Kmett ekm...@gmail.com wrote:

 Some form of MutableStruct# with a known number of words and a known
 number of pointers is basically what Ryan Yates was suggesting above, but
 where the word counts were stored in the objects themselves.

 Given that it'd have a couple of words for those counts it'd likely
 want to be something we build in addition to MutVar# rather than a
 replacement.

 On the other hand, if we had to fix those numbers and build info
 tables that knew them, and typechecker support, for instance, it'd get
 rather invasive.

 Also, a number of things that we can do with the 'sized' versions
 above, like working with evil unsized c-style arrays directly inline at 
 the
 end of the structure cease to be possible, so it isn't even a pure win if
 we did the engineering effort.

 I think 90% of the needs I have are covered just by adding the one
 primitive. The last 10% gets pretty invasive.

 -Edward

 On Fri, Aug 28, 2015 at 5:30 PM, Ryan Newton rrnew...@gmail.com
 wrote:

 I like the possibility of a general solution for mutable structs
 (like Ed said), and I'm trying to fully understand why it's hard.

 So, we can't unpack MutVar into constructors because of object
 identity problems. But what about directly supporting an extensible set 
 of
 unlifted MutStruct# objects, generalizing (and even replacing) MutVar#?
 That may be too much work, but is it problematic otherwise?

 Needless to say, this is also critical if we ever want best in class
 lockfree mutable structures, just like their Stm and sequential
 counterparts.

 On Fri, Aug 28, 2015 at 4:43 AM Simon Peyton Jones 
 simo...@microsoft.com wrote:

 At the very least I'll take this email and turn it into a short
 article.

 Yes, please do make it into a wiki page on the GHC Trac, and maybe
 make a ticket for it.


 Thanks



 Simon



 *From:* Edward Kmett [mailto:ekm...@gmail.com]
 *Sent:* 27 August 2015 16:54
 *To:* Simon Peyton Jones
 *Cc:* Manuel M T Chakravarty; Simon Marlow; ghc-devs
 *Subject:* Re: ArrayArrays



 An ArrayArray# is just an Array# with a modified invariant. It
 points directly to other unlifted ArrayArray#'s or ByteArray#'s.



 While those live in #, they are garbage collected objects, so this
 all lives on the heap.



 They were added to make some of the DPH stuff fast when it has to
 deal with nested arrays.



 I'm currently abusing them as a placeholder for a better thing.



 The Problem

 -



 Consider the scenario where you write a classic doubly-linked list
 in Haskell.



 data DLL = DLL (IORef (Maybe DLL) (IORef (Maybe DLL)



 Chasing from one DLL to the next requires following 3 pointers on
 the heap.



 DLL ~ IORef (Maybe DLL) ~ MutVar# RealWorld (Maybe DLL) ~ Maybe
 DLL ~ DLL



 That is 3 levels of indirection.



 We can trim one by simply unpacking the IORef with
 -funbox-strict-fields or UNPACK



 We can trim another by adding a 'Nil' constructor for DLL and
 worsening our representation.



 data DLL = DLL !(IORef DLL) !(IORef DLL) | Nil



 but now we're still stuck with a level of indirection



 DLL ~ MutVar# RealWorld

Re: ArrayArrays

[Ryan Newton]

Yes. And for the short term I can imagine places we will settle with arrays
even if it means tracking lengths unnecessarily and unsafeCoercing pointers
whose types don't actually match their siblings.

Is there anything to recommend the hacks mentioned for fixed sized array
objects *other* than using them to fake structs? (Much to derecommend, as
you mentioned!)
On Fri, Aug 28, 2015 at 3:07 PM Edward Kmett ekm...@gmail.com wrote:

 I think both are useful, but the one you suggest requires a lot more
 plumbing and doesn't subsume all of the usecases of the other.

 -Edward

 On Fri, Aug 28, 2015 at 5:51 PM, Ryan Newton rrnew...@gmail.com wrote:

 So that primitive is an array like thing (Same pointed type, unbounded
 length) with extra payload.

 I can see how we can do without structs if we have arrays, especially
 with the extra payload at front. But wouldn't the general solution for
 structs be one that that allows new user data type defs for # types?



 On Fri, Aug 28, 2015 at 4:43 PM Edward Kmett ekm...@gmail.com wrote:

 Some form of MutableStruct# with a known number of words and a known
 number of pointers is basically what Ryan Yates was suggesting above, but
 where the word counts were stored in the objects themselves.

 Given that it'd have a couple of words for those counts it'd likely want
 to be something we build in addition to MutVar# rather than a replacement.

 On the other hand, if we had to fix those numbers and build info tables
 that knew them, and typechecker support, for instance, it'd get rather
 invasive.

 Also, a number of things that we can do with the 'sized' versions above,
 like working with evil unsized c-style arrays directly inline at the end of
 the structure cease to be possible, so it isn't even a pure win if we did
 the engineering effort.

 I think 90% of the needs I have are covered just by adding the one
 primitive. The last 10% gets pretty invasive.

 -Edward

 On Fri, Aug 28, 2015 at 5:30 PM, Ryan Newton rrnew...@gmail.com wrote:

 I like the possibility of a general solution for mutable structs (like
 Ed said), and I'm trying to fully understand why it's hard.

 So, we can't unpack MutVar into constructors because of object identity
 problems. But what about directly supporting an extensible set of unlifted
 MutStruct# objects, generalizing (and even replacing) MutVar#? That may be
 too much work, but is it problematic otherwise?

 Needless to say, this is also critical if we ever want best in class
 lockfree mutable structures, just like their Stm and sequential
 counterparts.

 On Fri, Aug 28, 2015 at 4:43 AM Simon Peyton Jones 
 simo...@microsoft.com wrote:

 At the very least I'll take this email and turn it into a short
 article.

 Yes, please do make it into a wiki page on the GHC Trac, and maybe
 make a ticket for it.


 Thanks



 Simon



 *From:* Edward Kmett [mailto:ekm...@gmail.com]
 *Sent:* 27 August 2015 16:54
 *To:* Simon Peyton Jones
 *Cc:* Manuel M T Chakravarty; Simon Marlow; ghc-devs
 *Subject:* Re: ArrayArrays



 An ArrayArray# is just an Array# with a modified invariant. It points
 directly to other unlifted ArrayArray#'s or ByteArray#'s.



 While those live in #, they are garbage collected objects, so this all
 lives on the heap.



 They were added to make some of the DPH stuff fast when it has to deal
 with nested arrays.



 I'm currently abusing them as a placeholder for a better thing.



 The Problem

 -



 Consider the scenario where you write a classic doubly-linked list in
 Haskell.



 data DLL = DLL (IORef (Maybe DLL) (IORef (Maybe DLL)



 Chasing from one DLL to the next requires following 3 pointers on the
 heap.



 DLL ~ IORef (Maybe DLL) ~ MutVar# RealWorld (Maybe DLL) ~ Maybe DLL
 ~ DLL



 That is 3 levels of indirection.



 We can trim one by simply unpacking the IORef with
 -funbox-strict-fields or UNPACK



 We can trim another by adding a 'Nil' constructor for DLL and
 worsening our representation.



 data DLL = DLL !(IORef DLL) !(IORef DLL) | Nil



 but now we're still stuck with a level of indirection



 DLL ~ MutVar# RealWorld DLL ~ DLL



 This means that every operation we perform on this structure will be
 about half of the speed of an implementation in most other languages
 assuming we're memory bound on loading things into cache!



 Making Progress

 --



 I have been working on a number of data structures where the
 indirection of going from something in * out to an object in # which
 contains the real pointer to my target and coming back effectively doubles
 my runtime.



 We go out to the MutVar# because we are allowed to put the MutVar#
 onto the mutable list when we dirty it. There is a well defined
 write-barrier.



 I could change out the representation to use



 data DLL = DLL (MutableArray# RealWorld DLL) | Nil



 I can just store two pointers in the MutableArray# every time, but
 this doesn't help _much_ directly. It has reduced the amount of distinct

Re: ArrayArrays

[Ryan Newton]

I like the possibility of a general solution for mutable structs (like Ed
said), and I'm trying to fully understand why it's hard.

So, we can't unpack MutVar into constructors because of object identity
problems. But what about directly supporting an extensible set of unlifted
MutStruct# objects, generalizing (and even replacing) MutVar#? That may be
too much work, but is it problematic otherwise?

Needless to say, this is also critical if we ever want best in class
lockfree mutable structures, just like their Stm and sequential
counterparts.

On Fri, Aug 28, 2015 at 4:43 AM Simon Peyton Jones simo...@microsoft.com
wrote:

 At the very least I'll take this email and turn it into a short article.

 Yes, please do make it into a wiki page on the GHC Trac, and maybe make a
 ticket for it.


 Thanks



 Simon



 *From:* Edward Kmett [mailto:ekm...@gmail.com]
 *Sent:* 27 August 2015 16:54
 *To:* Simon Peyton Jones
 *Cc:* Manuel M T Chakravarty; Simon Marlow; ghc-devs
 *Subject:* Re: ArrayArrays



 An ArrayArray# is just an Array# with a modified invariant. It points
 directly to other unlifted ArrayArray#'s or ByteArray#'s.



 While those live in #, they are garbage collected objects, so this all
 lives on the heap.



 They were added to make some of the DPH stuff fast when it has to deal
 with nested arrays.



 I'm currently abusing them as a placeholder for a better thing.



 The Problem

 -



 Consider the scenario where you write a classic doubly-linked list in
 Haskell.



 data DLL = DLL (IORef (Maybe DLL) (IORef (Maybe DLL)



 Chasing from one DLL to the next requires following 3 pointers on the heap.



 DLL ~ IORef (Maybe DLL) ~ MutVar# RealWorld (Maybe DLL) ~ Maybe DLL ~
 DLL



 That is 3 levels of indirection.



 We can trim one by simply unpacking the IORef with -funbox-strict-fields
 or UNPACK



 We can trim another by adding a 'Nil' constructor for DLL and worsening
 our representation.



 data DLL = DLL !(IORef DLL) !(IORef DLL) | Nil



 but now we're still stuck with a level of indirection



 DLL ~ MutVar# RealWorld DLL ~ DLL



 This means that every operation we perform on this structure will be about
 half of the speed of an implementation in most other languages assuming
 we're memory bound on loading things into cache!



 Making Progress

 --



 I have been working on a number of data structures where the indirection
 of going from something in * out to an object in # which contains the real
 pointer to my target and coming back effectively doubles my runtime.



 We go out to the MutVar# because we are allowed to put the MutVar# onto
 the mutable list when we dirty it. There is a well defined write-barrier.



 I could change out the representation to use



 data DLL = DLL (MutableArray# RealWorld DLL) | Nil



 I can just store two pointers in the MutableArray# every time, but this
 doesn't help _much_ directly. It has reduced the amount of distinct
 addresses in memory I touch on a walk of the DLL from 3 per object to 2.



 I still have to go out to the heap from my DLL and get to the array object
 and then chase it to the next DLL and chase that to the next array. I do
 get my two pointers together in memory though. I'm paying for a card
 marking table as well, which I don't particularly need with just two
 pointers, but we can shed that with the SmallMutableArray# machinery
 added back in 7.10, which is just the old array code a a new data type,
 which can speed things up a bit when you don't have very big arrays:



 data DLL = DLL (SmallMutableArray# RealWorld DLL) | Nil



 But what if I wanted my object itself to live in # and have two mutable
 fields and be able to share the sme write barrier?



 An ArrayArray# points directly to other unlifted array types. What if we
 have one # - * wrapper on the outside to deal with the impedence mismatch
 between the imperative world and Haskell, and then just let the
 ArrayArray#'s hold other arrayarrays.



 data DLL = DLL (MutableArrayArray# RealWorld)



 now I need to make up a new Nil, which I can just make be a special
 MutableArrayArray# I allocate on program startup. I can even abuse pattern
 synonyms. Alternately I can exploit the internals further to make this
 cheaper.



 Then I can use the readMutableArrayArray# and writeMutableArrayArray#
 calls to directly access the preceding and next entry in the linked list.



 So now we have one DLL wrapper which just 'bootstraps me' into a strict
 world, and everything there lives in #.



 next :: DLL - IO DLL

 next (DLL m) = IO $ \s - case readMutableArrayArray# s of

(# s', n #) - (# s', DLL n #)



 It turns out GHC is quite happy to optimize all of that code to keep
 things unboxed. The 'DLL' wrappers get removed pretty easily when they are
 known strict and you chain operations of this sort!



 Cleaning it Up

 --



 Now I have one outermost indirection pointing to an array that points
 directly to other 

Re: ArrayArrays

[Ryan Newton]

So that primitive is an array like thing (Same pointed type, unbounded
length) with extra payload.

I can see how we can do without structs if we have arrays, especially with
the extra payload at front. But wouldn't the general solution for structs
be one that that allows new user data type defs for # types?


On Fri, Aug 28, 2015 at 4:43 PM Edward Kmett ekm...@gmail.com wrote:

 Some form of MutableStruct# with a known number of words and a known
 number of pointers is basically what Ryan Yates was suggesting above, but
 where the word counts were stored in the objects themselves.

 Given that it'd have a couple of words for those counts it'd likely want
 to be something we build in addition to MutVar# rather than a replacement.

 On the other hand, if we had to fix those numbers and build info tables
 that knew them, and typechecker support, for instance, it'd get rather
 invasive.

 Also, a number of things that we can do with the 'sized' versions above,
 like working with evil unsized c-style arrays directly inline at the end of
 the structure cease to be possible, so it isn't even a pure win if we did
 the engineering effort.

 I think 90% of the needs I have are covered just by adding the one
 primitive. The last 10% gets pretty invasive.

 -Edward

 On Fri, Aug 28, 2015 at 5:30 PM, Ryan Newton rrnew...@gmail.com wrote:

 I like the possibility of a general solution for mutable structs (like Ed
 said), and I'm trying to fully understand why it's hard.

 So, we can't unpack MutVar into constructors because of object identity
 problems. But what about directly supporting an extensible set of unlifted
 MutStruct# objects, generalizing (and even replacing) MutVar#? That may be
 too much work, but is it problematic otherwise?

 Needless to say, this is also critical if we ever want best in class
 lockfree mutable structures, just like their Stm and sequential
 counterparts.

 On Fri, Aug 28, 2015 at 4:43 AM Simon Peyton Jones simo...@microsoft.com
 wrote:

 At the very least I'll take this email and turn it into a short article.

 Yes, please do make it into a wiki page on the GHC Trac, and maybe make
 a ticket for it.


 Thanks



 Simon



 *From:* Edward Kmett [mailto:ekm...@gmail.com]
 *Sent:* 27 August 2015 16:54
 *To:* Simon Peyton Jones
 *Cc:* Manuel M T Chakravarty; Simon Marlow; ghc-devs
 *Subject:* Re: ArrayArrays



 An ArrayArray# is just an Array# with a modified invariant. It points
 directly to other unlifted ArrayArray#'s or ByteArray#'s.



 While those live in #, they are garbage collected objects, so this all
 lives on the heap.



 They were added to make some of the DPH stuff fast when it has to deal
 with nested arrays.



 I'm currently abusing them as a placeholder for a better thing.



 The Problem

 -



 Consider the scenario where you write a classic doubly-linked list in
 Haskell.



 data DLL = DLL (IORef (Maybe DLL) (IORef (Maybe DLL)



 Chasing from one DLL to the next requires following 3 pointers on the
 heap.



 DLL ~ IORef (Maybe DLL) ~ MutVar# RealWorld (Maybe DLL) ~ Maybe DLL
 ~ DLL



 That is 3 levels of indirection.



 We can trim one by simply unpacking the IORef with -funbox-strict-fields
 or UNPACK



 We can trim another by adding a 'Nil' constructor for DLL and worsening
 our representation.



 data DLL = DLL !(IORef DLL) !(IORef DLL) | Nil



 but now we're still stuck with a level of indirection



 DLL ~ MutVar# RealWorld DLL ~ DLL



 This means that every operation we perform on this structure will be
 about half of the speed of an implementation in most other languages
 assuming we're memory bound on loading things into cache!



 Making Progress

 --



 I have been working on a number of data structures where the indirection
 of going from something in * out to an object in # which contains the real
 pointer to my target and coming back effectively doubles my runtime.



 We go out to the MutVar# because we are allowed to put the MutVar# onto
 the mutable list when we dirty it. There is a well defined write-barrier.



 I could change out the representation to use



 data DLL = DLL (MutableArray# RealWorld DLL) | Nil



 I can just store two pointers in the MutableArray# every time, but this
 doesn't help _much_ directly. It has reduced the amount of distinct
 addresses in memory I touch on a walk of the DLL from 3 per object to 2.



 I still have to go out to the heap from my DLL and get to the array
 object and then chase it to the next DLL and chase that to the next array.
 I do get my two pointers together in memory though. I'm paying for a card
 marking table as well, which I don't particularly need with just two
 pointers, but we can shed that with the SmallMutableArray# machinery
 added back in 7.10, which is just the old array code a a new data type,
 which can speed things up a bit when you don't have very big arrays:



 data DLL = DLL (SmallMutableArray# RealWorld DLL) | Nil



 But what if I

Re: Updating submodules

[Ryan Newton]

Has everyone seen the git man page generator ;-)?  Hilarious.

http://git-man-page-generator.lokaltog.net/


On Tue Jan 06 2015 at 7:49:30 AM Simon Peyton Jones simo...@microsoft.com
wrote:

 Following a chat with Herbert, I've updated
 https://ghc.haskell.org/trac/ghc/wiki/Repositories

 Please check/proof-read

 Simon

 |  -Original Message-
 |  From: ghc-devs [mailto:ghc-devs-boun...@haskell.org] On Behalf Of
 |  Simon Peyton Jones
 |  Sent: 06 January 2015 10:49
 |  To: Herbert Valerio Riedel
 |  Cc: ghc-devs@haskell.org
 |  Subject: RE: Updating submodules
 |
 |  |   * There is no .git/config in libraries/parallel.
 |  (Whereas
 |  |   there is for another submodule, libraries/hoopl.)
 |  |
 |  |   * There is, however, a .git file which points to
 |  |  .git/modules/libraries/parallel
 |  |
 |  |  That's most likely because libraries/hoopl wasn't created via `git
 |  | submodule` but rather inherited from a Git checkout where
 |  | libraries/hoopl was an decoupled (not yet submodule) sub-repo...
 |
 |  Yes, that's plausible.  So the hoopl one is wrong, and the parallel
 |  one is right. But how do I fix hoopl?  (Short of blowing away the
 |  whole repository, which I can't do because it has lots of commits in
 |  it.)
 |
 |  |  In any case, if you manage Git remotes (while in libraries/hoopl)
 |  via
 |  | the `git remote` command, Git takes care of following the
 |  symlinked
 |  |  .git folder...
 |
 |  OK.  But in this case what do I do?
 |
 |  |   * In .git/modules/libraries/parallel/config, I see a url of  
 |  | https://git.haskell.org/packages/parallel.git.  But I can't push to
 |  
 |  | this URL.
 |  |
 |  |  yes, that's our mirrored copy of github.com/haskell/parallel/
 |  |
 |  |   * That matches the url in
 |  |   https://ghc.haskell.org/trac/ghc/wiki/Repositories, but
 |  contradicts
 |  |  the url in 'packages', which says
 |  |
 |  |   ssh://g...@github.com/haskell/parallel.git
 |  |
 |  |  yes, that's exactly the upstream URL you're supposed to push to...
 |  |  (and since it's a ssh:// protocl url, it means you should have
 |  push-
 |  | rights there)
 |
 |  So
 |
 |  * I *push* to ssh://g...@github.com/haskell/parallel.git
 |  * I *pull* from https://git.haskell.org/packages/parallel.git
 |
 |  Is that right?  Then again, how can I get the right URLs in the right
 |  place?
 |
 |
 |  |  The comment there is probably a bit misleading;
 |  |
 |  |  - in the upstreamurl field just means that the official
 |  upstream
 |  | repo is at git.haskell.org, and you should use the usual
 |  | ssh://git.haskell.org/... URL for pushing...
 |
 |  OK, so they are *ALL* sub-modules, and - is just shorthand for a
 |  particular URL.  Would it be possible to fix the comment?
 |
 |  Simon
 |  ___
 |  ghc-devs mailing list
 |  ghc-devs@haskell.org
 |  http://www.haskell.org/mailman/listinfo/ghc-devs
 ___
 ghc-devs mailing list
 ghc-devs@haskell.org
 http://www.haskell.org/mailman/listinfo/ghc-devs

___
ghc-devs mailing list
ghc-devs@haskell.org
http://www.haskell.org/mailman/listinfo/ghc-devs

Random maintainership -- Was: [core libraries] RE: Core libraries bug tracker

[Ryan Newton]

Dear core library folks  others,

 On Tue, Aug 19, 2014 at 10:31 AM, Simon Peyton Jones 
simo...@microsoft.com wrote:
 Some core libraries (e.g. random) have a maintainer that isn’t the
committee.

Ah, since it came up, maybe this is a good time to discuss that particular
maintainership.  I'm afraid that since it isn't close to my current work
(and I'm pre-tenure!) I haven't been able to really push the random library
forward the way it deserves to be pushed these last three years.  Shall we
move maintainership of it to the core libraries committee?

Also/alternatively Thomas Miedema thomasmied...@gmail.com has stepped
forward as a volunteer for taking over maintainership.

The library was in limbo in part because it was clear that some API changes
needed to be made and but there wasn't a major consensus building design
effort around that topic.  One thing that was already agreed upon on via
the libraries list decision process was to separate out SplittableGen.
 Duncan Coutts was in favor of this and also (I think) had some other ideas
about API changes that should be made.

On the implementation front, my hope was that tf-random could replace
random as the default/standard library. Koen and Michal support this, but I
think they didn't want to become the maintainers themselves yet.  (I think
that was to maintain some separation, and get buy-in from someone other
than them, the implementors, before/during the transition).

Best,
 -Ryan



On Tue, Aug 19, 2014 at 5:55 PM, Simon Peyton Jones simo...@microsoft.com
wrote:

 If you don't mind the extra traffic in the ghc trac, I'm open to the plan
to work there.



 OK great.



 Let’s agree that:

 ·The “owner” of a Core Libraries ticket is the person responsible
for progressing it – or “Core Libraries Committee” as one possibility.

 ·The “component” should identify the ticket as belonging to the
core libraries committee, not GHC.  We have a bunch of components like
“libraries/base”, “libraries/directory”, etc, but I’m sure that doesn’t
cover all the core libraries, and even if it did, it’s probably too fine
grain.  I suggest having just “Core Libraries”.



 Actions:

 ·Edward: update the Core Libraries home page (where is that?) to
point people to the Trac, tell them how to correctly submit a ticket, etc?

 ·Edward: send email to tell everyone about the new plan.

 ·Austin: add the same guidance to the GHC bug tracker.

 ·Austin: add “core libraries committee” as something that can be
an owner.

 ·Austin: change the “components” list to replace all the
“libraires/*” stuff with “Core Libraries”.



 Thanks



 Simon





 From: haskell-core-librar...@googlegroups.com [mailto:
haskell-core-librar...@googlegroups.com] On Behalf Of Edward Kmett
 Sent: 19 August 2014 16:23
 To: Simon Peyton Jones
 Cc: core-libraries-commit...@haskell.org; ghc-devs@haskell.org
 Subject: Re: [core libraries] RE: Core libraries bug tracker



 Hi Simon,



 If you don't mind the extra traffic in the ghc trac, I'm open to the plan
to work there.



 I was talking to Eric Mertens a few days ago about this and he agreed to
take lead on getting us set up to actually build tickets for items that go
into the libraries@ proposal process, so we have something helping to force
us to come to a definitive conclusion rather than letting things trail off.



 -Edward



 On Tue, Aug 19, 2014 at 10:31 AM, Simon Peyton Jones 
simo...@microsoft.com wrote:

 Edward, and core library colleagues,

 Any views on this?  It would be good to make progress.

 Thanks

 Simon



 From: ghc-devs [mailto:ghc-devs-boun...@haskell.org] On Behalf Of Simon
Peyton Jones
 Sent: 04 August 2014 16:01
 To: core-libraries-commit...@haskell.org
 Cc: ghc-devs@haskell.org
 Subject: Core libraries bug tracker



 Edward, and core library colleagues,

 This came up in our weekly GHC discussion

 · Does the Core Libraries Committee have a Trac?  Surely, surely
you should, else you’ll lose track of issues.

 · Would you like to use GHC’s Trac for the purpose?   Advantages:

 o   People often report core library issues on GHC’s Trac anyway, so
telling them to move it somewhere else just creates busy-work --- and maybe
they won’t bother, which leaves it in our pile.

 o   Several of these libraries are closely coupled to GHC, and you might
want to milestone some library tickets with an upcoming GHC release

 · If so we’d need a canonical way to identify tickets as CLC
issues.  Perhaps by making “core-libraries” the owner?  Or perhaps the
“Component” field?

 · Some core libraries (e.g. random) have a maintainer that isn’t
the committee.  So that maintainer should be the owner of the ticket. Or
the CLC might like a particular member to own a ticket.  Either way, that
suggest using the “Component” field to identify CLC tickets

 · Or maybe you want a Trac of your own?

 The underlying issue from our end is that we’d like a way to

 ·   

Re: Adding atomic primops

[Ryan Newton]

Yes, that's exactly what I'd like to see as well.  Also, we must confess
that most worldwide GHC cycles are currently spent on x86.  Arm will surely
become more important over time.  But what's right for x86 is probably
right for us for the near/medium term.




On Tue, May 20, 2014 at 11:04 AM, Johan Tibell johan.tib...@gmail.comwrote:

 Sequentially consistent also corresponds to, according to the LLVM docs:
 the C++0x/C1x memory_order_seq_cst, Java volatile, and the gcc-compatible
 __sync_* builtins, so we'll be in good company.


 On Tue, May 20, 2014 at 5:01 PM, Johan Tibell johan.tib...@gmail.comwrote:

 After some discussion with Simon, I will go ahead and try to implement
 fully sequentially consistent versions of these primops. We can add other
 versions that take the consistency model as an argument later, if needed.


 On Thu, May 15, 2014 at 9:03 PM, Carter Schonwald 
 carter.schonw...@gmail.com wrote:

 Hey Johan,
 on https://ghc.haskell.org/trac/ghc/ticket/7883 Ryan helped articulate
 what he'd want wrt memory ordering semantics.

 One point is that It might be totally valid and reasonable to provide
 *both* variants, though if we only were to do one, the strong ordering
 guarantees might be a better default, albeit your use case and others does
 benefit from using the weakest / simplest primops possible,


 On Thu, May 15, 2014 at 6:01 AM, Johan Tibell johan.tib...@gmail.comwrote:

 I will update the wiki page (and later CmmSink) with the guarantees we
 expect CallishMachOps to provide. We do need to pick what kind of guarantee
 we want to provide. Options are here:
 http://en.cppreference.com/w/cpp/atomic/memory_order

 Do we want to have these CallishMachOps to imply a full memory fence
 CPU instruction or some more relaxed ordering (e.g. only atomicity)?

 ___
 ghc-devs mailing list
 ghc-devs@haskell.org
 http://www.haskell.org/mailman/listinfo/ghc-devs





___
ghc-devs mailing list
ghc-devs@haskell.org
http://www.haskell.org/mailman/listinfo/ghc-devs

Re: Adding atomic primops

[Ryan Newton]

Hi John,

My understanding is that GHC does do reordering of memory operations, in
 the CmmSink pass (
 http://www.haskell.org/ghc/docs/latest/html/libraries/ghc/src/CmmSink.html,

 http://blog.ezyang.com/2014/01/so-you-want-to-add-a-new-concurrency-primitive-to-ghc/).
 In particular ghc may currently reorder loads, although it doesn't seem to
 reorder stores (at this time).  And ghc *currently* only performs these
 reorderings in limited cases.


Thanks!  That's a great summary by ezyang and I hadn't yet read it.

I have to confess that I don't understand what you mean by suggesting we
 should implement these primops without having an underlying memory model.


Oh, no, I think we should work on getting as close as possible to said
memory model.  But given how much work it's been for other languages I hope
we can pipeline some of the implementation tasks, at the same time as that
is going on.  For
example, we've got one GSOC working on a concurrent data structure, and it
would be great if they could have a GHC branch to test inlined primops,
even if there is much more work to do to verify the safety of that
prototype before it makes it to the master branch.
 (There's also the social question of getting someone with enough time to
do this memory model -- or port it from other languages.  Suggestions?)


 What does atomicReadIntArray# mean if there's no memory model in which
 its defined?  How can you be sure that other ghc hackers won't break some
 assumptions that the implementations rely upon, if they don't have a memory
 model to describe how operations should behave?  This is strongly related
 to the TODO: in Johan's proposal: just because that may be true now (I
 don't know if it is!) doesn't mean it will remain so in the future.


I totally agree.  I'm just not sure where we should go with it.
I think as a practical matter we need some kind of memory model fuzz
tester for GHC, since we certainly won't have formal verification
throughout the whole compiler.  There's been some neat work on this in the
memory models community, it seems.  It sounds like Jan-Willem also suggests
that there should be invasive changes made to the compiler to reify the
memory model as a DAG of dependencies on the individual operations.

I guess the concrete choice for Johan's proposed changes is to figure out
if its possible to quickly hack the relevant CMM passes that might reorder
(to be primop aware), rather than make the more general improvement that
Jan-Willem suggested in the comments on ezyang's post.

Of course, given the limited number of GHC hackers, everything may work out
 fine in the end.  It just seems to me that it would be more sensible to
 define the semantics first and build the primops around them, rather than
 primops first and semantics later.


I think it's a bit of a chicken and egg, because we need to get interesting
enough data structures and libraries to make someone care enough to do this
work ;-).

I think we need a collaboration with someone who does memory model stuff
for a living if we could get them interested or make this into research
somehow.
___
ghc-devs mailing list
ghc-devs@haskell.org
http://www.haskell.org/mailman/listinfo/ghc-devs

Re: Adding atomic primops

[Ryan Newton]

This would be fantastic and I'd be happy to work with you to expose the
extra ops through the atomic-primops pkg.

This would continue our trend of supporting gcc intrinsic style atomics
(full fence, no configurabilit of the memory model a la lllvm).

These word sized ops are most important, but ultimately there are the other
sizes to support as well -- even 128 bit.

On Sunday, May 4, 2014, Johan Tibell johan.tib...@gmail.com wrote:

 Hi,

 I found myself needing atomic operations on basic, mutable numeric values.
 I wrote a short design doc that I'd like feedback on:

 https://ghc.haskell.org/trac/ghc/wiki/AtomicPrimops

 I will try to implement these for 7.10.

 -- Johan



-- 
Sent from Gmail Mobile
___
ghc-devs mailing list
ghc-devs@haskell.org
http://www.haskell.org/mailman/listinfo/ghc-devs

Re: Adding atomic primops

[Ryan Newton]

Hi John  Carter,

Deferring for a moment issues with CAS on lazy/pointer values, which
pertain to Carter's question, I want to understand the ordering concern WRT
only the IntArray primops that Johan listed.  First, to my knowledge GHC
doesn't do any reordering of effectful memory operations (e.g.
readMutVar/writeMutVar).  That is, I thought that threaded State# param
was ironclad from the compiler's point of view.  (In fact, if GHC does any
optimization of these imperative bits, could someone point me to it?)

For Johan's primops to work, each primop must represent a full memory fence
that is respected both by the architecture, and by *both* compilers (GHC 
LLVM).  Since I don't think GHC is a problem, let's talk about LLVM.  We
need to verify that LLVM understands not to float regular loads and stores
past one of its own atomic instructions.  If that is the case (even without
anything being marked volatile), then I think we are in ok shape, right?

In the grand scheme of things isn't this a place where purity has done us
good?  All the memory traffic related to pure computation is and should be
separate from these primops on mutable vars and arrays, and there shouldn't
need to be any special ordering constraints *between* those two classes,
should there?

(Part 2)* CAS on lazy/pointer values* -- this bit *was *an ill-specified
mess to begin with, for sure ;-).  In part, the problem was that it was
somewhat unusual and sussed out other
cabal/GHChttps://github.com/rrnewton/haskell-lockfree/issues/10
issues
(including a regression in
7.8https://github.com/rrnewton/haskell-lockfree/issues/28that I
think is a GHC bug,  but haven't filed yet [where a spurious ! on
a function arg of Any type changes behavior]).  Even once the commitment
was made to use the ticketed approach.  (Carter, it's described in this
trac ticket https://ghc.haskell.org/trac/ghc/ticket/8157.)

Johan, FYI, I think we should actually backpedal on casMutVar# and
casArray# and not expose a type like the one we have in 7.8:

   MutableArray# s a - Int# - a - a - State# s - (# State# s, Int#, a #)


I can't think of any good reason to use it, since we should never trust
that a return value -- it's poison, and covering it up successfully
requires careful attention to
NOINLINEshttps://github.com/rrnewton/haskell-lockfree/issues/5.
 Rather, for 7.10, better to commit directly IMHO to the Any type to beat
back the compiler:

  casMutVarTicketed# :: MutVar# RealWorld a - Any a - Any a -
   State# RealWorld - (# State# RealWorld, Int#, Any a #)

Cheers,
  -Ryan




 IMHO I think the desire to include these features is leading to a slightly
 cavalier attitude towards reordering concerns.  Even assuming that the Cmm
 code generator doesn't reorder reads/writes around `CallishMachOp`, I don't
 see why this behavior should always be true, leading to possible future
 pain.  Also, it's possible that LLVM may decide to reorder memory accesses
 AIUI because the underlying LLVM variables won't be synchronized.

 In a nutshell, even though everything may work now I suspect it'll become
 an ill-specified mess in a few years time.  I don't have a ton of
 experience implementing these sorts of features in compilers though, so
 probably only worth a half cent.

 John L.
 On May 4, 2014 3:10 AM, Johan Tibell johan.tib...@gmail.com wrote:

 Hi,

 I found myself needing atomic operations on basic, mutable numeric
 values. I wrote a short design doc that I'd like feedback on:

 https://ghc.haskell.org/trac/ghc/wiki/AtomicPrimops

 I will try to implement these for 7.10.

 -- Johan


 ___
 ghc-devs mailing list
 ghc-devs@haskell.org
 http://www.haskell.org/mailman/listinfo/ghc-devs


 ___
 ghc-devs mailing list
 ghc-devs@haskell.org
 http://www.haskell.org/mailman/listinfo/ghc-devs


___
ghc-devs mailing list
ghc-devs@haskell.org
http://www.haskell.org/mailman/listinfo/ghc-devs

Re: Adding atomic primops

[Ryan Newton]

 For Johan's primops to work, each primop must represent a full memory
 fence that is respected both by the architecture, and by *both* compilers
 (GHC  LLVM).  Since I don't think GHC is a problem, let's talk about LLVM.
  We need to verify that LLVM understands not to float regular loads and
 stores past one of its own atomic instructions.  If that is the case (even
 without anything being marked volatile), then I think we are in ok shape,
 right?


Clarification -- this is assuming we're using the SequentiallyConsistent
setting in the LLVM backend to get full fences on each op, which correspond
to the gcc-compatible __sync_* builtins:

   http://llvm.org/docs/Atomics.html#sequentiallyconsistent
___
ghc-devs mailing list
ghc-devs@haskell.org
http://www.haskell.org/mailman/listinfo/ghc-devs

Re: Adding atomic primops

[Ryan Newton]

One last comment -- none of the above is to suggest that I don't think we
should eventually have a memory model (a la Java or C++11).  But I (and
Johan) don't think the addition of the primops Johan listed should wait on
it.  Further, I don't think these primops make the state of affairs any
worse, given that we've *already* had the combination of IORef operations 
parallel IO Threads for a long time, without a memory model.

I think the informal agreement we've been muddling along with is something
like this:

   - IORef operations have the same behavior as the analogous C operations
   -- no implied synchronization
   - all IORef ops are volatile wrt GHC (GHC won't reordered)
   - atomicModifyIORef does what its name implies

Though I confess, I'm personally unclear on what the agreement is in at
least two places:

   - What Haskell operations constitute grabbing a lock to protect IORef
   reads and writes?  (We often use MVar based strategies for locking, but do
   they give a *guarantee* that they provide the necessary memory fences
   for the previous/subsequent IORef operations?)
   - Is the de-facto volatile status I implied before extended to the
   backends (C / LLVM)?  I don't know but assume not.  Note that even if not,
   this doesn't cause a problem for the proposed atomic primops, all of which
   are themselves

Perhaps I and others get away with this level of murkiness because we
depend on IORefs so little, with so much happening in the pure code ;-).

Ah, and last of all -- while we do need to sort out all this stuff -- I
want to point out that adding Johan's proposed primops isn't the key
decision point.  That ship sailed with 7.2 ;-).  This is just about
fleshing out what's already there (e.g. fetch and Xor in addition to fetch
and Add) and improving the implementations by going to in-line primops.

Best,
  -Ryan


On Mon, May 5, 2014 at 12:25 AM, Ryan Newton rrnew...@gmail.com wrote:

 For Johan's primops to work, each primop must represent a full memory
 fence that is respected both by the architecture, and by *both*compilers 
 (GHC  LLVM).  Since I don't think GHC is a problem, let's talk
 about LLVM.  We need to verify that LLVM understands not to float regular
 loads and stores past one of its own atomic instructions.  If that is the
 case (even without anything being marked volatile), then I think we are
 in ok shape, right?


 Clarification -- this is assuming we're using the SequentiallyConsistent
 setting in the LLVM backend to get full fences on each op, which correspond
 to the gcc-compatible __sync_* builtins:

http://llvm.org/docs/Atomics.html#sequentiallyconsistent



___
ghc-devs mailing list
ghc-devs@haskell.org
http://www.haskell.org/mailman/listinfo/ghc-devs

Re: Adding atomic primops

[Ryan Newton]

Oh, just the first CAS primop -- the initial decision to include these
kinds of ops.

Sent from my phone.
On May 5, 2014 12:59 AM, Carter Schonwald carter.schonw...@gmail.com
wrote:

 what sailed in ghc 7.2?


 On Mon, May 5, 2014 at 12:46 AM, Ryan Newton rrnew...@gmail.com wrote:

 One last comment -- none of the above is to suggest that I don't think we
 should eventually have a memory model (a la Java or C++11).  But I (and
 Johan) don't think the addition of the primops Johan listed should wait on
 it.  Further, I don't think these primops make the state of affairs any
 worse, given that we've *already* had the combination of IORef
 operations  parallel IO Threads for a long time, without a memory model.

 I think the informal agreement we've been muddling along with is
 something like this:

- IORef operations have the same behavior as the analogous C
operations -- no implied synchronization
- all IORef ops are volatile wrt GHC (GHC won't reordered)
- atomicModifyIORef does what its name implies

 Though I confess, I'm personally unclear on what the agreement is in at
 least two places:

- What Haskell operations constitute grabbing a lock to protect
IORef reads and writes?  (We often use MVar based strategies for locking,
but do they give a *guarantee* that they provide the necessary memory
fences for the previous/subsequent IORef operations?)
- Is the de-facto volatile status I implied before extended to the
backends (C / LLVM)?  I don't know but assume not.  Note that even if not,
this doesn't cause a problem for the proposed atomic primops, all of which
are themselves

 Perhaps I and others get away with this level of murkiness because we
 depend on IORefs so little, with so much happening in the pure code ;-).

 Ah, and last of all -- while we do need to sort out all this stuff -- I
 want to point out that adding Johan's proposed primops isn't the key
 decision point.  That ship sailed with 7.2 ;-).  This is just about
 fleshing out what's already there (e.g. fetch and Xor in addition to fetch
 and Add) and improving the implementations by going to in-line primops.

 Best,
   -Ryan


 On Mon, May 5, 2014 at 12:25 AM, Ryan Newton rrnew...@gmail.com wrote:

 For Johan's primops to work, each primop must represent a full memory
 fence that is respected both by the architecture, and by *both*compilers 
 (GHC  LLVM).  Since I don't think GHC is a problem, let's talk
 about LLVM.  We need to verify that LLVM understands not to float regular
 loads and stores past one of its own atomic instructions.  If that is the
 case (even without anything being marked volatile), then I think we are
 in ok shape, right?


 Clarification -- this is assuming we're using the
 SequentiallyConsistent setting in the LLVM backend to get full fences on
 each op, which correspond to the gcc-compatible __sync_* builtins:

http://llvm.org/docs/Atomics.html#sequentiallyconsistent




 ___
 ghc-devs mailing list
 ghc-devs@haskell.org
 http://www.haskell.org/mailman/listinfo/ghc-devs



___
ghc-devs mailing list
ghc-devs@haskell.org
http://www.haskell.org/mailman/listinfo/ghc-devs

Re: Support for glibc 2.12 in 7.8 RC2

[Ryan Newton]

Yes, this is a really annoying issue on RHEL, which includes many
supercomputers.

Sent from my phone.
On Feb 7, 2014 1:35 PM, Rob Stewart robstewar...@gmail.com wrote:

 Hi,

 The 7.8 RC1 status page [1] states RC1 requires glibc 2.15, and that
 RC2 will support glibc 2.13 to target Debian 7. CentOS 6 packages
 glibc 2.12 and this will be the glibc version in the CentOS 6
 lifetime.

 Is there a possibility of stretching lower bounded support in RC2 down
 to glibc 2.12? This would allow all CentOS users using any of the 6.*
 releases to use 7.8, I think.

 [1] https://ghc.haskell.org/trac/ghc/wiki/Status/GHC-7.8/RC1

 --
 Rob
 ___
 ghc-devs mailing list
 ghc-devs@haskell.org
 http://www.haskell.org/mailman/listinfo/ghc-devs

___
ghc-devs mailing list
ghc-devs@haskell.org
http://www.haskell.org/mailman/listinfo/ghc-devs

Re: Bad news: apparent bug in casMutVar going back to 7.2

[Ryan Newton]

Hi Carter  others,

Carter, yes, this is CAS on pointers and in my next mail I'll try to come
up with some hypotheses as to why we may have (remaining) problems there.

But first, I have been assured that on x86 there is no failure mode in
which doing a comparison on the value read by CAS should not correctly
diagnose success or failure (same as directly reading the Zero Flag) [1].

And yet, there's this discrepancy, where the modified casMutVar that I
linked to does not have the failure.  As for reproducing the failure,
either of the two following tests will currently show problems:

   - Two threads try to casIORef False-True, both succeed
   - 120 threads try to read, increment, CAS until they succeed.  The total
   is often not 120 because multiple threads think the successfully
   incremented, say, 33-34.

Here's a specific recipe for the latter test on GHC 7.6.3 Mac or Linux:


*git clone g...@github.com:rrnewton/haskell-lockfree-queue.git *
*cd haskell-lockfree-queue/AtomicPrimops/*

*git checkout 1a1e7e55f6706f9e5754*

*cabal sandbox init*

*cabal install -f-withTH -fforeign ./ ./testing --enable-tests*

*./testing/dist/dist-sandbox-*/build/test-atomic-primops/test-atomic-primops
-t n_threads*

You may have to run the last line several times to see the failure.

Best,
  -Ryan

[1] I guess the __sync_bool_compare_and_swap intrinsic which reads ZF is
there just to avoid the extra comparison.

[2] P.S. I'd like to try this on GHC head, but the RHEL 6 machine I usually
use to build it is currently not validating (below error, commit
65d05d7334).  After I debug this gmp problem I'll confirm that the bug
under discussion applies on the 7.8 branch.

./sync-all checkout ghc-7.8
sh validate
...
/usr/bin/ld: libraries/integer-gmp/gmp/objs/aors.o: relocation R_X86_64_32
against `__gmpz_sub' can not be used when making a shared object; recompile
with -fPIC
libraries/integer-gmp/gmp/objs/aors.o: could not read symbols: Bad value




On Sat, Feb 1, 2014 at 2:55 AM, Carter Schonwald carter.schonw...@gmail.com
 wrote:

 Ryan, is your benchmark using CAS on pointers, or immediate words? trying
 to get atomic primops to build on my 7.8 build on my mac


 On Sat, Feb 1, 2014 at 2:44 AM, Carter Schonwald 
 carter.schonw...@gmail.com wrote:

 https://ghc.haskell.org/trac/ghc/ticket/8724#ticket is the ticket

 when i'm more awake i'll experiment some more


 On Sat, Feb 1, 2014 at 2:33 AM, Carter Schonwald 
 carter.schonw...@gmail.com wrote:

 i have a ticket for tracking this, though i'm thinking my initial
 attempt at a patch generates the same object code as it did before.

 @ryan, what CPU variant are you testing this on? is this on a NUMA
 machine or something?


 On Sat, Feb 1, 2014 at 1:58 AM, Carter Schonwald 
 carter.schonw...@gmail.com wrote:

 woops, i mean cmpxchgq


 On Sat, Feb 1, 2014 at 1:36 AM, Carter Schonwald 
 carter.schonw...@gmail.com wrote:

 ok, i can confirm that on my 64bit mac, both clang and gcc
 use cmpxchgl rather than cmpxchg
 i'll whip up a strawman patch on head that can be cherrypicked /
 tested out by ryan et al


 On Sat, Feb 1, 2014 at 1:12 AM, Carter Schonwald 
 carter.schonw...@gmail.com wrote:

 Hey Ryan,
 looking at this closely
 Why isn't CAS using CMPXCHG8B on 64bit architectures?  Could that be
 the culprit?

 Could the issue be that we've not had a good stress test that would
 create values that are equal on the 32bit range, but differ on the 64bit
 range, and you're hitting that?

 Could you try seeing if doing that change fixes things up?
 (I may be completely wrong, but just throwing this out as a naive
 obvious guess)


 On Sat, Feb 1, 2014 at 12:58 AM, Ryan Newton rrnew...@gmail.comwrote:

 Then again... I'm having trouble seeing how the spec on page 3-149
 of the Intel manual would allow the behavior I'm seeing:


 http://www.intel.com/content/dam/www/public/us/en/documents/manuals/64-ia-32-architectures-software-developer-manual-325462.pdf

 Nevertheless, this is exactly the behavior we're seeing with the
 current Haskell primops.  Two threads simultaneously performing the same
 CAS(p,a,b) can both think that they succeeded.





 On Sat, Feb 1, 2014 at 12:33 AM, Ryan Newton rrnew...@gmail.comwrote:

 I commented on the commit here:


 https://github.com/ghc/ghc/commit/521b792553bacbdb0eec138b150ab0626ea6f36b

 The problem is that our cas routine in SMP.h is similar to the C
 compiler intrinsic __sync_val_compare_and_swap, in that it returns the 
 old
 value.  But it seems we cannot use a comparison against that old value 
 to
 determine whether or not the CAS succeeded.  (I believe the CAS may 
 fail
 due to contention, but the old value may happen to look like our old 
 value.)

 Unfortunately, this didn't occur to me until it started causing
 bugs [1] [2].  Fixing casMutVar# fixes these bugs.  However, the way 
 I'm
 currently fixing CAS in the atomic-primops package is by using
 __sync_bool_compare_and_swap:


 https://github.com/rrnewton/haskell-lockfree

Re: Bad news: apparent bug in casMutVar going back to 7.2

[Ryan Newton]

Ok, here's another experiment, on this commit:


https://github.com/rrnewton/haskell-lockfree/commit/399bb19fa02eaf2f2eab5d02c4b608535362f9bc

Here, if I use GCC's __sync_val_compare_and_swap instead of GHC's version
of cas(), the problem also goes away.  I think these two implementations
should behave identically, and that they don't perhaps indicates that there
is something off about the inline asm, as Carter was suggesting:

*#if i386_HOST_ARCH || x86_64_HOST_ARCH*
*__asm__ __volatile__ (*
*   lock\ncmpxchg %3,%1*
*  :=a(o), =m (*(volatile unsigned int *)p) *
*  :0 (o), r (n));*
*return o;*

The x86 CAS instruction must put the return value in the accumulator
register, and indeed this constrains o to be allocated to the accumulator
register, while the new value n can be in any register.

So if there's a problem, I don't know what it is.  Except I'm not sure what
the ramifications of o being a function parameter AND having an =a
constraint on it are...

   -Ryan



On Sat, Feb 1, 2014 at 11:27 AM, Carter Schonwald 
carter.schonw...@gmail.com wrote:

 Hey Ryan, i've made the leap to using 7.8 on my machine, so i'll first
 have to get some pull requests in on atomic-primops before I can test it
 locally :), expect those patches later today!

 looks like gcc's inline ASM logic is pretty correct, after testing it a
 bit locally, pardon my speculative jumping the gun.


 On Sat, Feb 1, 2014 at 9:10 AM, Ryan Newton rrnew...@gmail.com wrote:

 Hi Carter  others,

 Carter, yes, this is CAS on pointers and in my next mail I'll try to come
 up with some hypotheses as to why we may have (remaining) problems there.

 But first, I have been assured that on x86 there is no failure mode in
 which doing a comparison on the value read by CAS should not correctly
 diagnose success or failure (same as directly reading the Zero Flag) [1].

 And yet, there's this discrepancy, where the modified casMutVar that I
 linked to does not have the failure.  As for reproducing the failure,
 either of the two following tests will currently show problems:

- Two threads try to casIORef False-True, both succeed
- 120 threads try to read, increment, CAS until they succeed.  The
total is often not 120 because multiple threads think the successfully
incremented, say, 33-34.

 Here's a specific recipe for the latter test on GHC 7.6.3 Mac or Linux:


 *git clone g...@github.com:rrnewton/haskell-lockfree-queue.git *
 *cd haskell-lockfree-queue/AtomicPrimops/*

 *git checkout 1a1e7e55f6706f9e5754*

 *cabal sandbox init*

 *cabal install -f-withTH -fforeign ./ ./testing --enable-tests*

 *./testing/dist/dist-sandbox-*/build/test-atomic-primops/test-atomic-primops
 -t n_threads*

 You may have to run the last line several times to see the failure.

 Best,
   -Ryan

 [1] I guess the __sync_bool_compare_and_swap intrinsic which reads ZF is
 there just to avoid the extra comparison.

 [2] P.S. I'd like to try this on GHC head, but the RHEL 6 machine I
 usually use to build it is currently not validating (below error, commit
 65d05d7334).  After I debug this gmp problem I'll confirm that the bug
 under discussion applies on the 7.8 branch.

 ./sync-all checkout ghc-7.8
 sh validate
 ...
 /usr/bin/ld: libraries/integer-gmp/gmp/objs/aors.o: relocation
 R_X86_64_32 against `__gmpz_sub' can not be used when making a shared
 object; recompile with -fPIC
 libraries/integer-gmp/gmp/objs/aors.o: could not read symbols: Bad value




 On Sat, Feb 1, 2014 at 2:55 AM, Carter Schonwald 
 carter.schonw...@gmail.com wrote:

 Ryan, is your benchmark using CAS on pointers, or immediate words?
 trying to get atomic primops to build on my 7.8 build on my mac


 On Sat, Feb 1, 2014 at 2:44 AM, Carter Schonwald 
 carter.schonw...@gmail.com wrote:

 https://ghc.haskell.org/trac/ghc/ticket/8724#ticket is the ticket

 when i'm more awake i'll experiment some more


 On Sat, Feb 1, 2014 at 2:33 AM, Carter Schonwald 
 carter.schonw...@gmail.com wrote:

 i have a ticket for tracking this, though i'm thinking my initial
 attempt at a patch generates the same object code as it did before.

 @ryan, what CPU variant are you testing this on? is this on a NUMA
 machine or something?


 On Sat, Feb 1, 2014 at 1:58 AM, Carter Schonwald 
 carter.schonw...@gmail.com wrote:

 woops, i mean cmpxchgq


 On Sat, Feb 1, 2014 at 1:36 AM, Carter Schonwald 
 carter.schonw...@gmail.com wrote:

 ok, i can confirm that on my 64bit mac, both clang and gcc
 use cmpxchgl rather than cmpxchg
 i'll whip up a strawman patch on head that can be cherrypicked /
 tested out by ryan et al


 On Sat, Feb 1, 2014 at 1:12 AM, Carter Schonwald 
 carter.schonw...@gmail.com wrote:

 Hey Ryan,
 looking at this closely
 Why isn't CAS using CMPXCHG8B on 64bit architectures?  Could that
 be the culprit?

 Could the issue be that we've not had a good stress test that would
 create values that are equal on the 32bit range, but differ on the 
 64bit
 range, and you're

Re: Bad news: apparent bug in casMutVar going back to 7.2

[Ryan Newton]

Ok, my bad, sorry all.

This is NOT a problem that will crop up in 7.8.   Rather, it's just a
problem with the duplicated bits of GHC RTS functionality that were stuck
into the atomic-primops library.  It was a C preprocessor problem that was
causing the inline asm we were discussing in this thread to not actually be
called.

Still, I'd like to be reminded of the rational for all this conditional
inline asm rather than using the C compiler intrinsics!  Anyone?

Best,
  -Ryan



On Sat, Feb 1, 2014 at 2:17 PM, Carter Schonwald carter.schonw...@gmail.com
 wrote:

 I got the test suite running on my (2 core) machine mac book air, with 7.8
 i've run it several times, not seeing any failures


 On Sat, Feb 1, 2014 at 1:03 PM, Carter Schonwald 
 carter.schonw...@gmail.com wrote:

 hr I have a crazy idea

 Compare RAX with r/m64. If equal, ZF is set and r64 is loaded into r/m64.
 Else, clear ZF
 and load r/m64 into RAX. is what the docs say for the cmpxchng
 instruction

 so RAX is the old values,  (EAX in the  32bit case). And it looks like we
 dont' set that explicitly when calling the asm .. CMPXCHG r/m64, r64

  hrmmm



 On Sat, Feb 1, 2014 at 12:52 PM, Ryan Newton rrnew...@gmail.com wrote:

 Ok, here's another experiment, on this commit:


 https://github.com/rrnewton/haskell-lockfree/commit/399bb19fa02eaf2f2eab5d02c4b608535362f9bc

 Here, if I use GCC's __sync_val_compare_and_swap instead of GHC's
 version of cas(), the problem also goes away.  I think these two
 implementations should behave identically, and that they don't perhaps
 indicates that there is something off about the inline asm, as Carter was
 suggesting:

 *#if i386_HOST_ARCH || x86_64_HOST_ARCH*
 *__asm__ __volatile__ (*
 *   lock\ncmpxchg %3,%1*
 *  :=a(o), =m (*(volatile unsigned int *)p) *
 *  :0 (o), r (n));*
 *return o;*

 The x86 CAS instruction must put the return value in the accumulator
 register, and indeed this constrains o to be allocated to the accumulator
 register, while the new value n can be in any register.

 So if there's a problem, I don't know what it is.  Except I'm not sure
 what the ramifications of o being a function parameter AND having an =a
 constraint on it are...

-Ryan



 On Sat, Feb 1, 2014 at 11:27 AM, Carter Schonwald 
 carter.schonw...@gmail.com wrote:

 Hey Ryan, i've made the leap to using 7.8 on my machine, so i'll first
 have to get some pull requests in on atomic-primops before I can test it
 locally :), expect those patches later today!

 looks like gcc's inline ASM logic is pretty correct, after testing it a
 bit locally, pardon my speculative jumping the gun.


 On Sat, Feb 1, 2014 at 9:10 AM, Ryan Newton rrnew...@gmail.com wrote:

 Hi Carter  others,

 Carter, yes, this is CAS on pointers and in my next mail I'll try to
 come up with some hypotheses as to why we may have (remaining) problems
 there.

 But first, I have been assured that on x86 there is no failure mode in
 which doing a comparison on the value read by CAS should not correctly
 diagnose success or failure (same as directly reading the Zero Flag) [1].

 And yet, there's this discrepancy, where the modified casMutVar that I
 linked to does not have the failure.  As for reproducing the failure,
 either of the two following tests will currently show problems:

- Two threads try to casIORef False-True, both succeed
- 120 threads try to read, increment, CAS until they succeed.  The
total is often not 120 because multiple threads think the successfully
incremented, say, 33-34.

 Here's a specific recipe for the latter test on GHC 7.6.3 Mac or Linux:


 *git clone g...@github.com:rrnewton/haskell-lockfree-queue.git *
 *cd haskell-lockfree-queue/AtomicPrimops/*

 *git checkout 1a1e7e55f6706f9e5754*

 *cabal sandbox init*

 *cabal install -f-withTH -fforeign ./ ./testing --enable-tests*

 *./testing/dist/dist-sandbox-*/build/test-atomic-primops/test-atomic-primops
 -t n_threads*

 You may have to run the last line several times to see the failure.

 Best,
   -Ryan

 [1] I guess the __sync_bool_compare_and_swap intrinsic which reads ZF
 is there just to avoid the extra comparison.

 [2] P.S. I'd like to try this on GHC head, but the RHEL 6 machine I
 usually use to build it is currently not validating (below error, commit
 65d05d7334).  After I debug this gmp problem I'll confirm that the bug
 under discussion applies on the 7.8 branch.

 ./sync-all checkout ghc-7.8
 sh validate
 ...
 /usr/bin/ld: libraries/integer-gmp/gmp/objs/aors.o: relocation
 R_X86_64_32 against `__gmpz_sub' can not be used when making a shared
 object; recompile with -fPIC
 libraries/integer-gmp/gmp/objs/aors.o: could not read symbols: Bad
 value




 On Sat, Feb 1, 2014 at 2:55 AM, Carter Schonwald 
 carter.schonw...@gmail.com wrote:

 Ryan, is your benchmark using CAS on pointers, or immediate words?
 trying to get atomic primops to build on my 7.8 build on my mac


 On Sat, Feb 1, 2014 at 2:44 AM, Carter

Re: Bad news: apparent bug in casMutVar going back to 7.2

[Ryan Newton]

Then again... I'm having trouble seeing how the spec on page 3-149 of the
Intel manual would allow the behavior I'm seeing:

http://www.intel.com/content/dam/www/public/us/en/documents/manuals/64-ia-32-architectures-software-developer-manual-325462.pdf

Nevertheless, this is exactly the behavior we're seeing with the current
Haskell primops.  Two threads simultaneously performing the same CAS(p,a,b)
can both think that they succeeded.





On Sat, Feb 1, 2014 at 12:33 AM, Ryan Newton rrnew...@gmail.com wrote:

 I commented on the commit here:


 https://github.com/ghc/ghc/commit/521b792553bacbdb0eec138b150ab0626ea6f36b

 The problem is that our cas routine in SMP.h is similar to the C
 compiler intrinsic __sync_val_compare_and_swap, in that it returns the old
 value.  But it seems we cannot use a comparison against that old value to
 determine whether or not the CAS succeeded.  (I believe the CAS may fail
 due to contention, but the old value may happen to look like our old value.)

 Unfortunately, this didn't occur to me until it started causing bugs [1]
 [2].  Fixing casMutVar# fixes these bugs.  However, the way I'm currently
 fixing CAS in the atomic-primops package is by using
 __sync_bool_compare_and_swap:


 https://github.com/rrnewton/haskell-lockfree/commit/f9716ddd94d5eff7420256de22cbf38c02322d7a#diff-be3304b3ecdd8e1f9ed316cd844d711aR200

 What is the best fix for GHC itself?   Would it be ok for GHC to include a
 C compiler intrinsic like __sync_val_compare_and_swap?  Otherwise we need
 another big ifdbef'd function like cas in SMP.h that has the
 architecture-specific inline asm across all architectures.  I can write the
 x86 one, but I'm not eager to try the others.

 Best,
-Ryan

 [1] https://github.com/iu-parfunc/lvars/issues/70
 [2] https://github.com/rrnewton/haskell-lockfree/issues/15


___
ghc-devs mailing list
ghc-devs@haskell.org
http://www.haskell.org/mailman/listinfo/ghc-devs

Re: GHC 7.8 release status

[Ryan Newton]

Thanks for the reminder.  Wiki is updated; atomics branch is merged.  The
only further work I plan to do in the near term is add additional tests.


On Wed, Sep 4, 2013 at 9:52 AM, Simon Peyton-Jones simo...@microsoft.comwrote:

  Friends

 The 7.8 release is imminent. This email is to ask abou the status of your
 contributions*.  In each case could you update the wiki with the current
 state of play, and your intentions, including dates. *  That is, don’t
 put your reply in email: it on the status page below; though by all means
 send email too!

 Summary here: http://ghc.haskell.org/trac/ghc/wiki/Status/GHC-7.8

 *Also : What is missing from the list that should be done?*

 **· ***Patrick Palka*: status of ghc –make –j?

 **· ***Nick*: status of your three items?

 **· ***Pedro/Richard*: is all the Typeable stuff, and gcast and
 friends, finished?

 **· ***Geoff*: what about the new Template Haskell story?

 **· ***Iavor*: when do you think you can merge?

 **· ***Austin*: what about ARMv7?

 **· ***Edsko/Thomas/Luite*: if you want anything for 7.8 it’ll
 have to be jolly soon.  At the moment I don’t even know the motivation or
 design, let alone implementation.  Could you make a wiki page explaining
 the proposed design?  Is it really important to do this for 7.8?

 **· ***Dynamic GHCi*.  I have no idea who is driving this, or how
 important it is.

 **· ***Ryan*: atomic stuff.  All merged?

 **· ***AMP warnings*: David Luposchainsky is driving this.

 ** **

 Thanks!

 Simon

 *Microsoft Research Limited (company number 03369488) is registered in
 England and Wales *

 *Registered office is at 21 Station Road, Cambridge, CB1 2FB*

 ** **

___
ghc-devs mailing list
ghc-devs@haskell.org
http://www.haskell.org/mailman/listinfo/ghc-devs

Re: GHC 7.8 release status

[Ryan Newton]

By the way, the parallel IO manager is also new in 7.8 right?  I'm not sure
but I think it may have something to do with the excessive system time bug
I just filed:

http://ghc.haskell.org/trac/ghc/ticket/8224
___
ghc-devs mailing list
ghc-devs@haskell.org
http://www.haskell.org/mailman/listinfo/ghc-devs

Re: Any GHC plans to adopt tf-random?

[Ryan Newton]

Hi all,

It looks like since the paper has come out the code has been released to
Hackage.  I personally would love to replace System.Random with something
like this.

I think the question of whether it's time comes down to some due-diligence
issues outside the scope of the paper.  This library has C code, have
people tested it extensively on Mac/Linux/Windows?  Does it have any
performance divots we should know about?  (Though the performance of the
legacy implementation was bad in many respects -- it
is a low bar.)

  -Ryan




On Wed, Nov 6, 2013 at 3:12 PM, Nicolas Frisby nicolas.fri...@gmail.comwrote:

 Adding Ryan Newton to the TO field, as the maintainer of random.


 On Wed, Nov 6, 2013 at 2:02 PM, Nicolas Frisby 
 nicolas.fri...@gmail.comwrote:

 Bugs #3575 #3620 have to do with splitting StdGens yielding dependent
 generators.

 The Haskell 2013 paper *Splittable Pseudorandom Number Generators Using
 Cryptographic Hashing* by Claessen and Pałka yielded the tf-random
 package on Hackage.

 Are there any plans to integrate these corrections into System.Random?

 Thanks.



___
ghc-devs mailing list
ghc-devs@haskell.org
http://www.haskell.org/mailman/listinfo/ghc-devs

Re: Proposal: GHC.Generics marked UNSAFE for SafeHaskell

[Ryan Newton]

But what stops the user from defining their own instances if they in fact
did not derive it?

The explicit False in Pedro's formulation seems to serve this purpose.



On Mon, Oct 14, 2013 at 11:20 PM, Nicolas Frisby
nicolas.fri...@gmail.comwrote:

 The formulation as a type family seems to conflict with the open-world
 principle. Would a class-based encoding instead be sufficient?

 -- the proposed, special wired-in class
 class Derives (t :: k1) (c :: k2)

 -- GHC would infer these from Pedro's example's declarations
 instance Derives MyData Eq
 instance Derives MyData Generic
 instance Derives MyData Show

 NB that there is no instance Derives MyData Ord, but standalone deriving
 could yield one later

 On Mon, Oct 14, 2013 at 10:02 PM, Ryan Newton rrnew...@gmail.com wrote:

 Hey, that's an awesome formulation!  Thanks Pedro.

 Any idea how much work this would be to implement in GHC, if it did
 garner approval?


 On Tue, Oct 8, 2013 at 3:48 AM, José Pedro Magalhães 
 drei...@gmail.comwrote:

 Hi,

 On Mon, Oct 7, 2013 at 10:32 AM, Dag Odenhall dag.odenh...@gmail.comwrote:

 Here‘s a thought: doesn’t Generic already have an unused phantom type
 that's only there “just in case we need to add something in the future”?

 No, it doesn't.

 Just a thought: what if we had a type family

 type family Derives (t :: k1) (c :: k2) :: Bool

 which would automatically be instantiated by GHC appropriately? E.g., if
 the user had the following code:

 data MyData = MyData deriving (Eq, Generic)
 deriving instance Show MyData
 instance Ord MyData

 GHC would automatically instantiate:

 type instance Derives MyData Eq  = True
 type instance Derives MyData Generic = True
 type instance Derives MyData Show= True
 type instance Derives MyData Ord = False

 Would this be something Ryan could use for detecting safe instances for
 LVish?


 Cheers,
 Pedro

 ___
 ghc-devs mailing list
 ghc-devs@haskell.org
 http://www.haskell.org/mailman/listinfo/ghc-devs



 ___
 ghc-devs mailing list
 ghc-devs@haskell.org
 http://www.haskell.org/mailman/listinfo/ghc-devs



___
ghc-devs mailing list
ghc-devs@haskell.org
http://www.haskell.org/mailman/listinfo/ghc-devs

Re: Proposal: GHC.Generics marked UNSAFE for SafeHaskell

[Ryan Newton]

 Here‘s a thought: doesn’t Generic already have an unused phantom type
 that's only there “just in case we need to add something in the future”?
 How about using that to track whether an instance was derived or not, and
 enforce *that* with SafeHaskell? Your SafeEq can then use a Generic
 Derived constraint or whatever.

Ah this sounds promising.  You mean that 'x' param in from/to?  So to use
that, whereever I call to/from I would have to constrain that x to be
Derived.

But do I understand correctly that GHC would have to modified so that its
derived instances of Generic force that x param to be Derived?
___
ghc-devs mailing list
ghc-devs@haskell.org
http://www.haskell.org/mailman/listinfo/ghc-devs

Re: Proposal: GHC.Generics marked UNSAFE for SafeHaskell

[Ryan Newton]

On Mon, Oct 7, 2013 at 10:22 AM, Edward Kmett ekm...@gmail.com wrote:

 Am I correct in understanding your issue arises from manually rolled
 instances of Generic, not from Generic itself?


Exactly.


 Wouldn't then perhaps the better fix be to resurrect the old rule for
 derived Typeable instances and apply it to Generic and Generic1 instead?

The new rule would be that if you hand-implemented Generic or Generic1 in
 your module it isn't Safe.


Ah... so you're proposing adding an extra rule to the GHC compiler itself?

Isn't it simpler to do it just with a library-level fix?  I.e. my proposal
is that GHC.Generics is marked as Unsafe.  And then if you really want you
can mark your module as TrustWorthy even with manual instances.  (And then,
yes, we'd need a GHC.Generics.Safe to pull the Generic symbol itself from
when we just want to derive it.)

That would make it so that derived Generic, Generic1 would be considered
 Safe, and you wouldn't break literally every user of the library who care
 about Safe Haskell.


Isn't that also accomplished just by import GHC.Generic.Safe (Generic)?
 And the handful of hypothetical people that are using this right now can
make that one-line fix.  I will *personally* make that fix for all those
people if they would come forward and share the link to their code ;-).

As it stands the things are damn-near impossible to get right instantiating
 them by hand anyways, so I expect this would affect only 1 or 2 users
 rather than all of them!


Exactly!
___
ghc-devs mailing list
ghc-devs@haskell.org
http://www.haskell.org/mailman/listinfo/ghc-devs

Re: Proposal: GHC.Generics marked UNSAFE for SafeHaskell

[Ryan Newton]

 My understanding is that the

 proposed changes to ghc/Generic would mean that all data used with
 LVish would have to expose it's implementation to everybody through
 the derived Generic instance.


Either that or they would have use TrustWorthy to mark a module containing
custom equality/ordering operations as safe according to our definition,
and take the proof obligations that come with that.

In fact, this is what we would plan to do to opt in ADTs like bytestring,
vector, containers.

We can trust as many people/libraries as we like, but we want to retain the
ability to run untrusted code with the right set of flags to ensure that it
can't break our library.  (And that's what we can't do without locking down
the Generic backdoor.)
___
ghc-devs mailing list
ghc-devs@haskell.org
http://www.haskell.org/mailman/listinfo/ghc-devs

Re: Proposal: GHC.Generics marked UNSAFE for SafeHaskell

[Ryan Newton]

On Sun, Oct 6, 2013 at 6:28 PM, Ganesh Sittampalam gan...@earth.li wrote:

  - Referential transparency: e.g. no unsafePerformIO

 - Module boundary control: no abstraction violation like Template
 Haskell and GeneralizedNewtypeDeriving
  - Semantic consistency: importing a safe module can't change existing
 code, so no OverlappingInstances and the like

Is this change necessary to preserve the existing properties, or are you
 hoping to add a new one?


I'm not currently aware of ways to break these invariants *just* with
GHC.Generics.  Hmm, but I would like to know why it is marked trustworthy
and not inferred-safe...

My argument is more that it is a violation in spirit to write bad Generic
instances that in turn lead to bad fromRep and toRep conversions.  (In a
similar way to bad Typeable insteances leading to bad type conversions).

Second, the usage intent of GHC.Generics as far as I can see, is that users
99.9% of the time will be using -XDeriveGeneric.

Third, without making this change I know of no way to provide safe/correct
Eq and Ord instances in a way that extends to user datatypes.  And in LVish
we have one example of what I think could be a valuable Safe-Haskell
compliant parallel programming library (see the POPL'14 paper for more
detailshttp://www.cs.indiana.edu/~rrnewton/papers/2013_07_LVish_quasiDet_working_draft.pdf),
but which will never be fully water-tight without vetted Eq and Ord
instances.


 I also understand that you want to require 'standard' Generic instances
 on types - will that mean that module authors are forced to expose
 internals to use your library?


Great point!  My intent was to short-circuit that by providing
TrustWorthy SafeEq and SafeOrd instances for standard types.

If you know any better ways to go forward, let me know!

Also, if anyone uses SafeHaskell with non-derived Generic instances, please
speak up!

Cheers,
  -Ryan
___
ghc-devs mailing list
ghc-devs@haskell.org
http://www.haskell.org/mailman/listinfo/ghc-devs

Re: Proposal: GHC.Generics marked UNSAFE for SafeHaskell

[Ryan Newton]

I can't know for certain but I think I would bet money on nothing.

Edward  David may know more about what actual use SafeHaskell is getting?





On Sun, Oct 6, 2013 at 5:46 PM, Johan Tibell johan.tib...@gmail.com wrote:

 What would break if we changed it to Unsafe?

 On Sun, Oct 6, 2013 at 2:32 PM, Ryan Newton rrnew...@gmail.com wrote:
  Hi all,
 
  We had a discussion on haskell-cafe, which only confirmed the
 unreliability
  of Eq and Ord.  Fine.  But if I instead want to define a SafeEq class
  and instances based on GHC.Generics, then I can't have users writing
 Generic
  instances that lie about the structure of their types.
 
  But as you can see this module is currently marked Trustworthy:
 
 
 
 http://www.haskell.org/ghc/docs/7.4.1/html/libraries/ghc-prim-0.2.0.0/GHC-Generics.html
 
  I simply propose that this is changed to Unsafe! [1]
 
  Context:
  The bottom line is that I believe in the Safe-Haskell mission here, and I
  want to be able to deliver practical libraries that give a strong
 guarantee
  that types can't be broken.   Currently, the LVish library has one big
 hole
  in it because of this Eq/Ord limitation; the problem is documented here.
 If we can provide incontrovertible Safe-Haskel guarantees, this is a
  huge, qualitative difference compared to what's possible in the C++,
 Java's
  and MLs of the world.  There are plenty of libraries that provide
 guarantees
  like deterministic parallelism IFF the user does everything right and
  breaks no rules (CnC, Pochoir, etc).  But we can do better!
 
  Best,
-Ryan
 
  [1]  Small detail... some of these bindings, like the class name
 Generic
  itself, will still need to be accessible from a Trustworthy library.  I
  propose GHC.Generics.Safe for that, following existing precedent.
 
  On Sun, Oct 6, 2013 at 5:18 PM, Ryan Newton rrnew...@gmail.com wrote:
 
  Thanks for the responses all.
 
  I'm afraid the point about GHC.Generics got lost here.  I'll respond and
  then rename this as a specific library proposal.
 
  I don't want to fix the world's Eq instances, but I am ok with requiring
  that people derive Generic for any data they want to put in an LVar
  container.  (From which I can give them a SafeEq instance.)
 
  It's not just LVish that is in this boat any library that tries to
  provide deterministic parallelism outside of the IO monad has some very
 fine
  lines to walk.  Take a look at Accelerate.  It is deterministic (as
 long as
  you run only with the CUDA backend and only on one specific GPU...
 otherwise
  fold topologies may look different and non-associative folds may leak).
  Besides, runAcc does a huge amount of implicit IO (writing to disk,
  calling nvcc, etc)!  At the very least this could fail if the disk if
 full.
  But then again, regular pure computations fail when memory runs
 out... so
  I'm ok grouping these in the same bucket for now.  Determinism modulo
  available resources.
 
  A possible problem with marking instance Eq as an unsafe feature is
  that many modules would be only Trustworthy instead of Safe.
 
 
  My proposal is actually more narrow than that.  My proposal is to mark
  GHC.Generics as Unsafe.
 
  That way I can define my own SafeEq, and know that someone can't break
 it
  by making a Generic instance that lies.  It is very hard for me to see
 why
  people should be able to make their own Generic instances (that might
 lie
  about the structure of the type), in Safe-Haskell.
 
 
  That would go against my every purely functional module is
 automatically
  safe because the compiler checks that it cannot launch the missiles
  understanding of Safe Haskell.
 
 
  Heh, that may already be violated by the fact that you can't use other
  extensions like OverlappingInstances, or provide your own Typeable
  instances.
 
 
 
  Actually, Eq instances are not unsafe per se, but only if I also use
 some
  other module that assumes certain properties about all Eq instances in
  scope. So in order to check safety, two independent modules (the
 provider
  and the consumer of the Eq instance) would have to cooperate.
 
 
  I've found, that this is a very common problem that we have when trying
 to
  make our libraries Safe-Haskell compliant -- often we want to permit and
  deny combinations of modules.  I don't have a solution I'm afraid.
 
 
 
 
  ___
  Libraries mailing list
  librar...@haskell.org
  http://www.haskell.org/mailman/listinfo/libraries
 

___
ghc-devs mailing list
ghc-devs@haskell.org
http://www.haskell.org/mailman/listinfo/ghc-devs

Re: Proposal: GHC.Generics marked UNSAFE for SafeHaskell

[Ryan Newton]

Let me tweak that -- I think there will be places where import
GHC.Generics must be changed to import GHC.Generics.Safe to get the
Generic symbol.

Alternatively, some essential piece, like the class Generic's members could
be hidden from GHC.Generics and moved into GHC.Generics.Internal (and that
would be the Unsafe one).

That would probably be a good idea for minimizing impact on current
SafeHaskell code but it might have more impact on non-safehaskell code,
which would never have detected this status switch in the first place.
 Also, .Safe is more in line with the existing conventions.







On Sun, Oct 6, 2013 at 11:18 PM, Ryan Newton rrnew...@gmail.com wrote:

 I can't know for certain but I think I would bet money on nothing.

 Edward  David may know more about what actual use SafeHaskell is getting?





 On Sun, Oct 6, 2013 at 5:46 PM, Johan Tibell johan.tib...@gmail.comwrote:

 What would break if we changed it to Unsafe?

 On Sun, Oct 6, 2013 at 2:32 PM, Ryan Newton rrnew...@gmail.com wrote:
  Hi all,
 
  We had a discussion on haskell-cafe, which only confirmed the
 unreliability
  of Eq and Ord.  Fine.  But if I instead want to define a SafeEq
 class
  and instances based on GHC.Generics, then I can't have users writing
 Generic
  instances that lie about the structure of their types.
 
  But as you can see this module is currently marked Trustworthy:
 
 
 
 http://www.haskell.org/ghc/docs/7.4.1/html/libraries/ghc-prim-0.2.0.0/GHC-Generics.html
 
  I simply propose that this is changed to Unsafe! [1]
 
  Context:
  The bottom line is that I believe in the Safe-Haskell mission here, and
 I
  want to be able to deliver practical libraries that give a strong
 guarantee
  that types can't be broken.   Currently, the LVish library has one big
 hole
  in it because of this Eq/Ord limitation; the problem is documented here.
 If we can provide incontrovertible Safe-Haskel guarantees, this is a
  huge, qualitative difference compared to what's possible in the C++,
 Java's
  and MLs of the world.  There are plenty of libraries that provide
 guarantees
  like deterministic parallelism IFF the user does everything right and
  breaks no rules (CnC, Pochoir, etc).  But we can do better!
 
  Best,
-Ryan
 
  [1]  Small detail... some of these bindings, like the class name
 Generic
  itself, will still need to be accessible from a Trustworthy library.  I
  propose GHC.Generics.Safe for that, following existing precedent.
 
  On Sun, Oct 6, 2013 at 5:18 PM, Ryan Newton rrnew...@gmail.com wrote:
 
  Thanks for the responses all.
 
  I'm afraid the point about GHC.Generics got lost here.  I'll respond
 and
  then rename this as a specific library proposal.
 
  I don't want to fix the world's Eq instances, but I am ok with
 requiring
  that people derive Generic for any data they want to put in an LVar
  container.  (From which I can give them a SafeEq instance.)
 
  It's not just LVish that is in this boat any library that tries to
  provide deterministic parallelism outside of the IO monad has some
 very fine
  lines to walk.  Take a look at Accelerate.  It is deterministic (as
 long as
  you run only with the CUDA backend and only on one specific GPU...
 otherwise
  fold topologies may look different and non-associative folds may leak).
  Besides, runAcc does a huge amount of implicit IO (writing to disk,
  calling nvcc, etc)!  At the very least this could fail if the disk if
 full.
  But then again, regular pure computations fail when memory runs
 out... so
  I'm ok grouping these in the same bucket for now.  Determinism modulo
  available resources.
 
  A possible problem with marking instance Eq as an unsafe feature is
  that many modules would be only Trustworthy instead of Safe.
 
 
  My proposal is actually more narrow than that.  My proposal is to mark
  GHC.Generics as Unsafe.
 
  That way I can define my own SafeEq, and know that someone can't break
 it
  by making a Generic instance that lies.  It is very hard for me to see
 why
  people should be able to make their own Generic instances (that might
 lie
  about the structure of the type), in Safe-Haskell.
 
 
  That would go against my every purely functional module is
 automatically
  safe because the compiler checks that it cannot launch the missiles
  understanding of Safe Haskell.
 
 
  Heh, that may already be violated by the fact that you can't use other
  extensions like OverlappingInstances, or provide your own Typeable
  instances.
 
 
 
  Actually, Eq instances are not unsafe per se, but only if I also use
 some
  other module that assumes certain properties about all Eq instances in
  scope. So in order to check safety, two independent modules (the
 provider
  and the consumer of the Eq instance) would have to cooperate.
 
 
  I've found, that this is a very common problem that we have when
 trying to
  make our libraries Safe-Haskell compliant -- often we want to permit
 and
  deny combinations of modules.  I don't have

Re: Proposal: GHC.Generics marked UNSAFE for SafeHaskell

[Ryan Newton]

Indeed!  That seems to be a direct violation of this language in the manual:

*An important part of this is that safe compiled code is not able to
examine or create data values using data constructors that it cannot import
*

And the funny part is that that is unrelated to my particular problem of
the user making their own Generic instances and is instead related to
simply exposing to...

Thanks for putting that together.

On Mon, Oct 7, 2013 at 1:13 AM, John Lato jwl...@gmail.com wrote:

 On Sun, Oct 6, 2013 at 10:14 PM, Ryan Newton rrnew...@gmail.com wrote:


 On Sun, Oct 6, 2013 at 6:28 PM, Ganesh Sittampalam gan...@earth.liwrote:

  - Referential transparency: e.g. no unsafePerformIO

  - Module boundary control: no abstraction violation like Template
 Haskell and GeneralizedNewtypeDeriving
  - Semantic consistency: importing a safe module can't change existing
 code, so no OverlappingInstances and the like

 Is this change necessary to preserve the existing properties, or are you
 hoping to add a new one?


 I'm not currently aware of ways to break these invariants *just* with
 GHC.Generics.  Hmm, but I would like to know why it is marked trustworthy
 and not inferred-safe...


 How about this demo repo? https://github.com/JohnLato/safe-bugtest

 I'm really not a safe haskell expert, but I believe this is a
 demonstration of using GHC.Generics to violate a module's abstraction
 boundaries with SafeHaskell enabled.

 If I'm incorrect, I would appreciate if somebody could explain my error.
 If, however, I'm correct, then I think that Ryan's proposal of marking
 GHC.Generics Unsafe is the best way to remedy the problem.

 A possible stumbling block may involve base and package-trust, but I'm not
 certain of the current status.

 John L.

___
ghc-devs mailing list
ghc-devs@haskell.org
http://www.haskell.org/mailman/listinfo/ghc-devs

Re: Anyone else failing to validate on 'linker_unload'?

[Ryan Newton]

 Ryan, can you do one final thing? When you run that program, be sure
 to specify `+RTS -Dl` (must be linked with -debug.) This will enable
 all the debug output where the linker is concerned. There will be a
 few hundred lines just for initialization (based on my machine.) If my
 theory is correct, you'll probably see stuff like 'Unloading object
 file ...' right as the invalid read/segfault occurs.


Hi Austin,

I did this, and it produced a 97MB text file of debug output, the tail end
of which was:

*initLinker: idempotent return*
*lookupSymbol: value of stg_gc_unpt_r1 is 0x485570*
*`stg_gc_unpt_r1' resolves to 0x485570Reloc: P = 0x40b510f3   S = 0x485570
  A = 0xfffc*
*relocations for section 3 using symtab 8*
*Rel entry   0 is raw( (nil) 0x80001  (nil))   lookupSymbol: looking up
base_ControlziApplicative_zdfApplicativeIO3_info*
*initLinker: start*
*initLinker: idempotent return*
*lookupSymbol: value of base_ControlziApplicative_zdfApplicativeIO3_info is
0x40b51058*
*`base_ControlziApplicative_zdfApplicativeIO3_info' resolves to
0x40b51058Reloc: P = 0x40b51100   S = 0x40b51058   A = (nil)*
*resolveObjs: done*
*lookupSymbol: looking up f*
*initLinker: start*
*initLinker: idempotent return*
*lookupSymbol: value of f is 0x440330c0*
*initLinker: start*
*initLinker: idempotent return*
*unloadObj: Test.o*
*Checking whether to unload Test.o*
*Unloading object file Test.o*

And that's when it segfaulted (notusing valgrind).  If it is of any use,
here is the full output, which fortunately compresses down to 4.4MB:


http://www.cs.indiana.edu/~rrnewton/temp/linker_unload_debug_output.txt.bz2

Best,
  -Ryan

P.S. Here is the equivalent output from the same thing being run under
valgrind:

initLinker: idempotent return
lookupSymbol: value of base_ControlziApplicative_zdfApplicativeIO3_info is
0x4c15058
`base_ControlziApplicative_zdfApplicativeIO3_info' resolves to
0x4c15058Reloc: P = 0x4c15100   S = 0x4c15058   A = (nil)
resolveObjs: done
lookupSymbol: looking up f
initLinker: start
initLinker: idempotent return
lookupSymbol: value of f is 0x4c0f0c0
initLinker: start
initLinker: idempotent return
unloadObj: Test.o
Checking whether to unload Test.o
Unloading object file Test.o
==9030== Invalid read of size 8
==9030==at 0x492502: checkUnload (CheckUnload.c:286)
==9030==by 0x476580: GarbageCollect (GC.c:666)
==9030==by 0x46ADCD: scheduleDoGC (Schedule.c:1652)
==9030==by 0x46B976: performGC_ (Schedule.c:2551)
==9030==by 0x46B9AE: performMajorGC (Schedule.c:2565)
==9030==by 0x4043E1: main (in
/home/beehive/ryan_scratch/ghc-working/testsuite/tests/rts/linker_unload2)
==9030==  Address 0x95c4580 is 80 bytes inside a block of size 120 free'd
==9030==at 0x4A063F0: free (vg_replace_malloc.c:446)
==9030==by 0x4656D5: stgFree (RtsUtils.c:107)
==9030==by 0x45DDF4: freeObjectCode (Linker.c:2087)
==9030==by 0x4924CF: checkUnload (CheckUnload.c:295)
==9030==by 0x476580: GarbageCollect (GC.c:666)
==9030==by 0x46ADCD: scheduleDoGC (Schedule.c:1652)
==9030==by 0x46B976: performGC_ (Schedule.c:2551)
==9030==by 0x46B9AE: performMajorGC (Schedule.c:2565)
==9030==by 0x4043E1: main (in
/home/beehive/ryan_scratch/ghc-working/testsuite/tests/rts/linker_unload2)
==9030==
___
ghc-devs mailing list
ghc-devs@haskell.org
http://www.haskell.org/mailman/listinfo/ghc-devs

Anyone having trouble with make install in HEAD?

[Ryan Newton]

Earlier this summer I had no problem with configure; make; make install
 But recently I'm getting errors like this on GHC HEAD:

/u/rrnewton/opt/ghc-7.7.20130831/lib/ghc-7.7.20130831/bin/ghc-pkg:
error while loading shared libraries: libHSbin-package-db-0.0.0
.0-ghc7.7.20130831.so: cannot open shared object file: No such file or
directory

That is, the inplace build is fully functional but it just won't install.

Any ideas?

Thanks,
  -Ryan
___
ghc-devs mailing list
ghc-devs@haskell.org
http://www.haskell.org/mailman/listinfo/ghc-devs

Design discussion for atomic primops to land in 7.8

[Ryan Newton]

There's a ticket that describes the design here:
http://ghc.haskell.org/trac/ghc/ticket/8157#comment:1
It is a fairly simple extension of the casMutVar# that has been in since
7.2.  The implementation is on the `atomics` branch currently.

Feel free to add your views either here or on that task's comments.

One example of an alternative design would be Carter's proposal to expose
something closer to the full LLVM concurrency
opshttp://llvm.org/docs/Atomics.html
:

Schonwald carter.schonw...@gmail.com wrote:

 i'm kinda thinking that we should do the analogue of exposing all the
 different memory model level choices (because its not that hard to add
 that), and when the person building it has an old version of GCC, it falls
 back to the legacy atomic operations?

 This also gives a nice path to how to upgrade to the inline asm approach.


These LLVM ops include many parameterized configurations of loads, stores,
cmpxchg, atomicrmw and barriers.  In fact, it implements much more than is
natively supported in most hardware, but it provides a uniform abstraction.

My original thought was that any kind of abstraction like that would be
built and maintained as a Haskell library, and only the most rudimentary
operations (required to get access to process features) would be exposed as
primops.  Let's call this the small set of concurrent ops.

If we want the big set I think we're doomed to *reproduce* the logic that
maps LLVM concurrency abstractions onto machine ops irrespective of whether
those abstractions are implemented as Haskell functions or as primops:

   - If the former, then the Haskell library must map the full set of ops
   to the reduced small set (just like LLVM does internally)
   - If we instead have a large set of LLVM-isomorphic primops then to
   support the same primops *in the native code backend *will, again,
   require reimplementing all configurations of all operations.

Unless... we want to make concurrency ops something that require the LLVM
backend?

Right now there is not a *performance* disadvantage to supporting a smaller
rather than a larger set of concurrency ops (LLVM has to emulate these
things anyway, or round up to more expensive ops).  The scenario where it
would be good to target ALL of LLVMs interface would be if processors and
LLVM improved in the future, and we automatically got the benefit of better
HW support for some op on on some arch.

I'm a bit skeptical of that proposition itself, however.  I personally
don't really like a world where we program with virtual operations that
don't really exist (and thus can't be *tested* against properly).  Absent
formal verification, it seems hard to get this code right anyway.  Errors
will be undetectable on existing architectures.

  -Ryan
___
ghc-devs mailing list
ghc-devs@haskell.org
http://www.haskell.org/mailman/listinfo/ghc-devs

Re: Design discussion for atomic primops to land in 7.8

[Ryan Newton]

Well, what's the long term plan?  Is the LLVM backend going to become the
only backend at some point?



On Thu, Aug 22, 2013 at 1:43 PM, Carter Schonwald 
carter.schonw...@gmail.com wrote:

 Hey Ryan,
 you raise some very good points.

 The most important point you raise (I think) is this:
 it would be very very nice to (where feasible) to add analogous machinery
 to the native code gen, so that its not falling behind the llvm one quite
 as much.

 at least for these atomic operations (unlike the SIMD ones),
 it may be worth investigating whats needed to add those to the native code
 gen as well.

 (adding simd support on the native codegen would be nice too, but probably
 *substantially *more work)



 On Thu, Aug 22, 2013 at 11:40 AM, Ryan Newton rrnew...@gmail.com wrote:

 There's a ticket that describes the design here:
 http://ghc.haskell.org/trac/ghc/ticket/8157#comment:1
 It is a fairly simple extension of the casMutVar# that has been in since
 7.2.  The implementation is on the `atomics` branch currently.

 Feel free to add your views either here or on that task's comments.

 One example of an alternative design would be Carter's proposal to expose
 something closer to the full LLVM concurrency 
 opshttp://llvm.org/docs/Atomics.html
 :

 Schonwald carter.schonw...@gmail.com wrote:

 i'm kinda thinking that we should do the analogue of exposing all the
 different memory model level choices (because its not that hard to add
 that), and when the person building it has an old version of GCC, it falls
 back to the legacy atomic operations?

 This also gives a nice path to how to upgrade to the inline asm approach.


 These LLVM ops include many parameterized configurations of loads,
 stores, cmpxchg, atomicrmw and barriers.  In fact, it implements much more
 than is natively supported in most hardware, but it provides a uniform
 abstraction.

 My original thought was that any kind of abstraction like that would be
 built and maintained as a Haskell library, and only the most rudimentary
 operations (required to get access to process features) would be exposed as
 primops.  Let's call this the small set of concurrent ops.

 If we want the big set I think we're doomed to *reproduce* the logic
 that maps LLVM concurrency abstractions onto machine ops irrespective of
 whether those abstractions are implemented as Haskell functions or as
 primops:

- If the former, then the Haskell library must map the full set of
ops to the reduced small set (just like LLVM does internally)
- If we instead have a large set of LLVM-isomorphic primops then
to support the same primops *in the native code backend *will, again,
require reimplementing all configurations of all operations.

 Unless... we want to make concurrency ops something that require the LLVM
 backend?

 Right now there is not a *performance* disadvantage to supporting a
 smaller rather than a larger set of concurrency ops (LLVM has to emulate
 these things anyway, or round up to more expensive ops).  The scenario
 where it would be good to target ALL of LLVMs interface would be if
 processors and LLVM improved in the future, and we automatically got the
 benefit of better HW support for some op on on some arch.

 I'm a bit skeptical of that proposition itself, however.  I personally
 don't really like a world where we program with virtual operations that
 don't really exist (and thus can't be *tested* against properly).
  Absent formal verification, it seems hard to get this code right anyway.
  Errors will be undetectable on existing architectures.

   -Ryan




 ___
 ghc-devs mailing list
 ghc-devs@haskell.org
 http://www.haskell.org/mailman/listinfo/ghc-devs



___
ghc-devs mailing list
ghc-devs@haskell.org
http://www.haskell.org/mailman/listinfo/ghc-devs

Re: how to checkout proper submodules

[Ryan Newton]

Hi all,

I just reread this thread again.  Is this one of these situations where *almost
everyone agrees, but the fix just didn't happen*?

In particular, there is still no formal relationship between versions of
the compiler and versions of the testsuite that tests it -- that seems odd!
 Can we please make *testsuite at least *a sub-module?  If we count this
long email thread as rough consensus, is it just waiting on someone of
sufficient authority typing a git submodule add command (and tweaking
sync-all accordingly)?

Also, Jan's suggestion sounded good -- that once all child repos are git
submodules then sync-all can be replaced with something that helps out with
git submodule branching, as it helps out with multi-repo branching now (a
little bit).

Best,
  -Ryan





On Wed, Jun 5, 2013 at 2:02 PM, Jan Stolarek jan.stola...@p.lodz.pl wrote:

 I think that testsuite should be included in the main GHC repo. I don't
 recall any other project
 that has its tests placed in a separate repository. The nhc argument
 doesn't convince me - after
 all, most test that are added nowadays are GHC specific.

 Janek

 ___
 ghc-devs mailing list
 ghc-devs@haskell.org
 http://www.haskell.org/mailman/listinfo/ghc-devs

___
ghc-devs mailing list
ghc-devs@haskell.org
http://www.haskell.org/mailman/listinfo/ghc-devs

Re: how to checkout proper submodules

[Ryan Newton]

Ok, resuming after release makes sense.

Regarding whether it reached a conclusion:

What struck me about this particular discussion was the *lack* of
disagreement (relative to say, the records debate).  It seemed like no one
was arguing for the status quo and just about everyone agreed that moving
to all-submodules is better than the current mix.

Still, one could argue that making an improvement is premature if (1) there
is significant transition cost to make the change, or (2) it puts you on
some kind of local optima that makes it harder to get to a higher peak.
 Yet in the case of all-submodules vs. ugly-mix, the transition cost is
very low, and it doesn't preclude any future improvements.  (For example,
it is completely reasonable to later decide to copy certain modules into
the tree rather than using submodules.)

But maybe I'm under-estimating the severity of the anti-submodule
grumbling... that is, I may not not be accurately distinguishing the
submodules have their annoyances but they are the lesser evil opinion
from I will adamantly oppose adding any more submodules.

Best,
  -Ryan



On Thu, Aug 22, 2013 at 4:14 PM, Simon Peyton-Jones
simo...@microsoft.comwrote:

  There was a long discussion about this a couple of months ago.  It did
 not reach a conclusion, but it is merely parked, not abandoned. I hope that
 you can all pick it up again after the release.

 ** **

 Simon

 ** **

 *From:* ghc-devs [mailto:ghc-devs-boun...@haskell.org] *On Behalf Of *Austin
 Seipp
 *Sent:* 22 August 2013 20:31
 *To:* Ryan Newton
 *Cc:* ghc-devs@haskell.org; Edward Kmett
 *Subject:* Re: how to checkout proper submodules

 ** **

 Simon and I discussed this a little today. I think there are several
 legitimate points made throughout the threads here, but the problem is
 clear: consistent builds are difficult, if not legitimately impossible.
 That's a very big problem.

 ** **

 Right now, it is far too late into release cycle to do anything drastic
 I'm afraid. Once we branch, we can feasibly start making good changes in
 this direction. One problem however is that we don't even have a clear
 writeup over what all the relevant points are (aside from this + all the
 ranting I did elsewhere, which is loosely in my head still.) Earlier today,
 I preemptively created this page, but have not jotted down any of my notes:
 http://ghc.haskell.org/trac/ghc/wiki/GitSubmoduleProblem

 ** **

 For a short recap, here is what I think:

 ** **

  1) Several repositories should really just become part of GHC's
 repository. I'd argue that includes testsuite, nofib, and several others
 (integer-gmp/integer-simple, hpc, etc.) They don't need to be submodules
 and making them so is unnecessary complexity, when they can realistically
 never be used with anything else. This cuts down on something like 10
 repositories, IIRC.

 ** **

  2) Several more should become submodules, where 'more' = the libraries
 under the new Core Libraries Committee. They will be taking over several of
 the other free floating repositories that are not currently submodules. We
 no longer will 'own' them, as it is.

 ** **

  3) 'base' and 'ghc-prim' are up for more debate it seems. Roman wants
 them in particular for haskell-suite, but really he only wants a repository
 to work with from what I remember. I'm not sure what to do here. Making
 them a submodule is realistic, but I'm honestly a little afraid of
 submodules for a package which is so highly traffic'd by developers
 (another reason I don't want e.g. testsuite as a submodule, either.)

 ** **

 The first two points alone should help a lot in making builds more
 reliable and reproducible, but it will require changes in the development
 workflow. In particular, it's much easier to lose work with submodules -
 especially for those among us who are not Git masters. So we should take
 the time to clearly explain all of this. But 1  2 should cover a large
 part the current setup, and most repos are very low traffic. Also, I'd like
 to take the time to have a discussion with Edward Kmett (who I have CC'd)
 about point 2 to make sure we're on the same page here. But I haven't done
 this yet.

 ** **

 Point 3 seems to really be the most contentious, since a few other things
 come with it. Should we give up on 'base' being usable by other compilers?
 Historically that's why it's separate. But really it's easy to write code
 against 'base' that will never work with another compiler anyway. But maybe
 that can be fixed. And will the base split - also slated for post 7.8 -
 also change the ownership of significant parts of the library, based on how
 it is implemented? There were several things floating around this.

 ** **

 Regardless of point 3 and all that, something should and will be done
 soon. I'll put this up on the wiki later when I have time. We just need a
 directly spelled out plan of attack.

 ** **

 ** **

 On Thu, Aug 22, 2013 at 2:04 PM, Ryan

Re: Reproducible build recipes for GHC development VMs?

[Ryan Newton]

Oh, good question.  It was Ubuntu 12.04 LTS so I just assumed it would be
too old without checking.



On Thu, Aug 22, 2013 at 3:38 AM, Simon Marlow marlo...@gmail.com wrote:

 On 21/08/13 15:44, Ryan Newton wrote:

 Hi all,

 Returning to the topic discussed by Simon M. and others here:

 http://projects.haskell.org/**pipermail/haskell-platform/**
 2009-July/000572.htmlhttp://projects.haskell.org/pipermail/haskell-platform/2009-July/000572.html

 This is my attempt at a script for bootstrapping a GHC-validating VM:

 http://parfunk.blogspot.com/**2013/08/zero-to-ghc-**
 development-in-ubuntu-vm-in.**htmlhttp://parfunk.blogspot.com/2013/08/zero-to-ghc-development-in-ubuntu-vm-in.html

 Let me know if there's a better way, or if you'd like to help get this
 kind of thing into an even more accessible form (Amazon AMI, Chef
 recipe, etc).


 I do this occasionally with an Amazon VM.  Incidentally, why did you
 download the GHC 7.6.3 tarball rather than apt-get install ghc?  The
 current GHC on Ubuntu is 7.4 which is enough to bootstrap GHC HEAD.

 Cheers,
 Simon



___
ghc-devs mailing list
ghc-devs@haskell.org
http://www.haskell.org/mailman/listinfo/ghc-devs

GHC branch atomics pushed

[Ryan Newton]

Hi all,

There should be very little that is controversial about this branch because
it adds primops upon which nothing else depends and which users will not
see directly.  But for now I've added it to a branch called atomics
(likewise use the atomics testsuite branch of course).  As soon as Carter
and I work out a bit more of the future roadmap I'll merge it into master.

Carter, would you like to use this branch to develop the improved LLVM
support for the primops that are now added there?

Cheers,
  -Ryan

P.S. Carter, as you requested I'll add an extra trac ticket for discussion.
___
ghc-devs mailing list
ghc-devs@haskell.org
http://www.haskell.org/mailman/listinfo/ghc-devs

Reproducible build recipes for GHC development VMs?

[Ryan Newton]

Hi all,

Returning to the topic discussed by Simon M. and others here:


http://projects.haskell.org/pipermail/haskell-platform/2009-July/000572.html

This is my attempt at a script for bootstrapping a GHC-validating VM:


http://parfunk.blogspot.com/2013/08/zero-to-ghc-development-in-ubuntu-vm-in.html

Let me know if there's a better way, or if you'd like to help get this kind
of thing into an even more accessible form (Amazon AMI, Chef recipe, etc).

Cheers,
  -Ryan
___
ghc-devs mailing list
ghc-devs@haskell.org
http://www.haskell.org/mailman/listinfo/ghc-devs

Re: Reproducible build recipes for GHC development VMs?

[Ryan Newton]

Luite,

Neat!  That sounds perfect.  If it can build/install the compiler, then
it's also ready to go for validation of patches.

I've never used vagrant myself but I'll give it a try.  Is this the
absolute easiest thing for people to do?  Or should I just put a (sadly
multi GB) virtual box image on my website?

  -Ryan



On Wed, Aug 21, 2013 at 10:51 AM, Luite Stegeman stege...@gmail.com wrote:

 We've been using Vagrant and puppet for building GHC HEAD with some
 patches and GHCJS on 32 and 64 bit ubuntu. This way, rebuilding the whole
 VM from scratch is just one command (vagrant up), the VM can either copy
 files to the host, through a shared filesystem, or just use the network to
 report results.

 I'd be happy to help setting this up for GHC

 https://github.com/ghcjs/ghcjs-build


 On Wed, Aug 21, 2013 at 4:44 PM, Ryan Newton rrnew...@gmail.com wrote:

 Hi all,

 Returning to the topic discussed by Simon M. and others here:


 http://projects.haskell.org/pipermail/haskell-platform/2009-July/000572.html

 This is my attempt at a script for bootstrapping a GHC-validating VM:


 http://parfunk.blogspot.com/2013/08/zero-to-ghc-development-in-ubuntu-vm-in.html

 Let me know if there's a better way, or if you'd like to help get this
 kind of thing into an even more accessible form (Amazon AMI, Chef recipe,
 etc).

 Cheers,
   -Ryan



 ___
 ghc-devs mailing list
 ghc-devs@haskell.org
 http://www.haskell.org/mailman/listinfo/ghc-devs



___
ghc-devs mailing list
ghc-devs@haskell.org
http://www.haskell.org/mailman/listinfo/ghc-devs

Re: 7.8 Feature window

[Ryan Newton]

Hi all,

It is not merged into master presently but I would like to propose the
three new primops that are on the atomics branch for inclusion in 7.8.
 These are pretty much completely apart from everything else and don't
break any existing code.

For the public library that exposes these things (atomic-primops) it will
be a great boon to be able to depend on them in  7.8 and not have to wait
yet another release cycle [1].

Best,
  -Ryan

[1] P.S. 7.8 will already be a breaking change to atomic-primops, because
of the change in CMM syntax.  So if it has to be #ifdef'd anyway, we might
as well go straight to the Right Thing rather than having a proliferation
of intermediate hacks.




On Tue, Aug 20, 2013 at 1:01 PM, Austin Seipp ase...@pobox.com wrote:

 All,

 GHC 7.8's release is drawing near. We would like to make a release
 candidate sometime around ICFP, which will be in late September.
 Unfortunately that's just over a month a way, so the clock is ticking!

 The tree will need a few weeks of stabilization. After that, we will
 release an RC, and likely branch. Then things will roughly return to normal.

 The exact date for feature cutoff is not set yet (but I will follow up
 soon on this.) So, I'd like a show of hands and a quick 'check in' for
 outstanding work for 7.8. There are a few things we know for sure are - or
 were - tentatively scheduled for this release:

  * SIMD improvements
  * New Template Haskell
  * Constraint solver for type naturals

 These are - as far as I'm aware - the largest outstanding features which
 not quite yet in HEAD.

 For the release, we would like to minimize 'disruptive' features, because
 7.8 already has many large changes. In particular, Dynamic GHCi and dynamic
 builds will likely prove the biggest challenge 'in the field', so we would
 like plenty of time to stress this as best we can for the RC, and the
 release itself.

 There are some things which we are fairly certain will not make it:

  * Joachim's new newtype coercion implementation
  * Adam's new record implementation

  There are some things I'm not very privvy to perhaps, but could still go
 in:

  * Nicolas possibly had some optimisation improvements according to Simon.

  * Edsko had a small patch for extended plugin functionality in HEAD, but
 Luite and Thomas also have input here. Status is uncertain.

  * ERDI was working on pattern synonyms. I believe you were having some
 trouble with the implementation. Can someone help him if necessary?

 Finally, there are loose ends to tie off:

  * I believe Simon and Jose were having discussions about the new Typeable
 implementation, regarding hand-written instances. This should be fine for
 7.8 and is mostly some behavioral tweaking I think.

 I've undoubtedly missed things here. Please fill me in. :)

 Note that before the freeze, you should interpret 'disruptive' with your
 own good judgement. Smaller patches and improvements are certainly welcome
 as always, and you shouldn't wait on me to push something if you feel good
 about it. If you're ever unsure, just ask. Worst case is something gets
 backed out, but it's nothing we cannot come back to.

 --
 Regards,
 Austin - PGP: 4096R/0x91384671

 ___
 ghc-devs mailing list
 ghc-devs@haskell.org
 http://www.haskell.org/mailman/listinfo/ghc-devs


___
ghc-devs mailing list
ghc-devs@haskell.org
http://www.haskell.org/mailman/listinfo/ghc-devs

Re: GHC branch atomics pushed

[Ryan Newton]

Done!


On Wed, Aug 21, 2013 at 12:21 PM, Simon Peyton-Jones
simo...@microsoft.comwrote:

  Great.  Can you add a bullet to
 http://ghc.haskell.org/trac/ghc/wiki/Status/GHC-7.8?


 Simon

 ** **

 *From:* ghc-devs [mailto:ghc-devs-boun...@haskell.org] *On Behalf Of *Ryan
 Newton
 *Sent:* 21 August 2013 13:12
 *To:* Carter Schonwald; ghc-devs@haskell.org
 *Subject:* GHC branch atomics pushed

 ** **

 Hi all,

 ** **

 There should be very little that is controversial about this branch
 because it adds primops upon which nothing else depends and which users
 will not see directly.  But for now I've added it to a branch called
 atomics (likewise use the atomics testsuite branch of course).  As soon
 as Carter and I work out a bit more of the future roadmap I'll merge it
 into master.

 ** **

 Carter, would you like to use this branch to develop the improved LLVM
 support for the primops that are now added there?

 ** **

 Cheers,

   -Ryan

 ** **

 P.S. Carter, as you requested I'll add an extra trac ticket for discussion.
 

 ** **

___
ghc-devs mailing list
ghc-devs@haskell.org
http://www.haskell.org/mailman/listinfo/ghc-devs

What are the current rules about object duplication during GC?

[Ryan Newton]

Is the status quo still the same as it was in this paper?

http://research.microsoft.com/en-us/um/people/simonpj/papers/parallel-gc/par-gc-ismm08.pdf

That is, there is a low-probability that a race will result in immutable
objects being duplicated, but not mutable objects.  But that leaves me
wondering exactly which types are currently included in the duplicatable
vs. take the lock category [2].

The reason I'm asking is because *defeating GHC compile-time and runtime
optimizations* is precisely the goal of the recent work to make a
safe/reliable infrastructure for CAS-based lockfree data structures in GHC.
 For example, the atomic-primops package uses the Any kind and type
coercions to store objects (pointers) in a form that makes them opaque to
the compiler, preventing them from being unboxed and reboxed (changing
pointer identity).

Actually, the current policy of this library that false negatives (failing
CAS) must be tolerated, which is *usually* not an onerous obligation.  So
if the heap objects in question CAN be duplicated, that is still ok, but I
would like to know if it is the case, that is, if Any is treated any
differently than, say, Int.

My guess is *no, *but I'd like to confirm this.  That is, it seems
that Anyeffects the compiler (which promises
not to enter the
pointerhttp://www.haskell.org/ghc/docs/7.6.3/html/libraries/ghc-prim-0.3.0.0/GHC-Prim.html#g:25),
but the physical representation of a thunk or function doesn't change, and
thus the GC must treat those objects exactly as it would if they were
*not*coerced to
Any.  Sound right?

Cheers,
  -Ryan

[1] P.S. The places I checked before asking on this list were here:
  http://ghc.haskell.org/trac/ghc/wiki/Commentary/Rts/Storage/GC

And here:  http://ghc.haskell.org/trac/ghc/wiki/GarbageCollectorNotes

If there is newer documentation for the GC, please let me know.

[2] One hint is on line 531 of
StgMiscClosures.cmmhttps://github.com/ghc/ghc/blob/d61c623ed6b2d352474a7497a65015dbf6a72e12/rts/StgMiscClosures.cmm#L531
:

// PRIM rather than CONSTR, because PRIM objects cannot be duplicated by
the GC.

Yet there are 62 different types of objects in ClosureTypes.hs, so I'm not
sure what all the rules are.
___
ghc-devs mailing list
ghc-devs@haskell.org
http://www.haskell.org/mailman/listinfo/ghc-devs

Re: Proposal: provide cas and barriers symbols even without -threaded

[Ryan Newton]

Do you have a branch already lined up for your LLVM-atomics work?



On Sat, Aug 10, 2013 at 7:02 PM, Carter Schonwald 
carter.schonw...@gmail.com wrote:

 huh, did I suggest viewing it as a bug fix? my mistake! (a branch would
 make sense)


 On Sat, Aug 10, 2013 at 12:40 PM, Ryan Newton rrnew...@gmail.com wrote:

 Well for new features like this (rather than bug fix), I'd prefer if I
 could get commit access and at least push it to a branch.  I can create a
 new trac ticket too.


 On Saturday, August 3, 2013, Carter Schonwald wrote:

 took a quick look,  awesome! this will make it MUCH MUCH easier for me
 to do my work. Thank you very much.

 off hand, to prevent patch confusion,
  it naively seems like the nicest way to post the patches to trac is to
 post a *new ticket to trac* that links to the main one,
  plus add a comment on the main ticket a link to the new ticket for the
 c/cmm based versions of the primops.

  At least, given that theres likely going to be a bit of discussion on
 just your ticket perhaps, better to factor that into a related ticket to
 make it easier to keep track of that?

 (i'm also possibly over thinking this enormously, so i could be way off
 base)



 On Sat, Aug 3, 2013 at 9:31 PM, Carter Schonwald 
 carter.schonw...@gmail.com wrote:

 nvm, githubs backup, i'll have a look! :)


 On Sat, Aug 3, 2013 at 9:05 PM, Carter Schonwald 
 carter.schonw...@gmail.com wrote:

 awesome! (this will also make my work easier)

 ryan: github is down, could you put the branch on bitbucket or some such
 so I can have a lookseee/clone locally?

 thanks!
 -Carter


 On Sat, Aug 3, 2013 at 4:01 AM, Ryan Newton rrnew...@gmail.com wrote:

 Just to keep you all up to date...  I'm adding the primops in question
 and validating the individual commits before putting them here:

 https://github.com/rrnewton/ghc/commits/atomicPrimOps

 The basic idea for using these extensions is:

- the atomic-primops library will work in 7.6 or 7.7+.  It will use
ifdefs to decide whether to use its own primops or GHC-builtin
- future versions will simply get faster, as Carter replaces
out-of-line primops that *also* use C calls, with inline primops / LLVM
equivalents

 Shall I stick a patch on a ticket, or will someone volunteer to pull?
  What's the protocol for requesting commit access anyway?  (By the way, can
 someone share the reason that pull-requests to the github ghc mirror are
 such a no-no?  They seem no worse than a patch in an email which the big 
 warning
 sign https://github.com/ghc/ghc recommends.)

 Best,
   -Ryan

 P.S. FYI, I'm periodically getting these:

 0 caused framework failures
 0 unexpected passes
 1 unexpected failures

  Unexpected failures:
  perf/compiler  T1969 [stat not good enough] (normal)

 Can that just be because of running on a loaded machine?  How narrow are
 these windows?


 On Thu, Aug 1, 2013 at 12:32 PM, Ryan Newton rrnew...@gmail.com wrote:

 On Sun, Jul 21, 2013 at 3:32 AM, Carter Schonwald 
 carter.schonw...@gmail.com wrote:

 ok, could you add those comments (about additional operations to
 consider) to the ticket?


 Sure.  Just did that.


 relatedly: if we want these atomic ops to use the sequential analogues
 when we're not using the threaded run time system, does that mean
 we need to have a symbol / constant variable exposed in the RTS we link
 in, so that the inline code branches on a linktime constant value / symbol
 (something like isThreadedRTS:: Bool, )  or some sort of analogue
 thereof?


 



 --
 Sent from Gmail Mobile



___
ghc-devs mailing list
ghc-devs@haskell.org
http://www.haskell.org/mailman/listinfo/ghc-devs

Re: Proposal: provide cas and barriers symbols even without -threaded

[Ryan Newton]

Well for new features like this (rather than bug fix), I'd prefer if I
could get commit access and at least push it to a branch.  I can create a
new trac ticket too.


On Saturday, August 3, 2013, Carter Schonwald wrote:

 took a quick look,  awesome! this will make it MUCH MUCH easier for me to
 do my work. Thank you very much.

 off hand, to prevent patch confusion,
  it naively seems like the nicest way to post the patches to trac is to
 post a *new ticket to trac* that links to the main one,
  plus add a comment on the main ticket a link to the new ticket for the
 c/cmm based versions of the primops.

  At least, given that theres likely going to be a bit of discussion on
 just your ticket perhaps, better to factor that into a related ticket to
 make it easier to keep track of that?

 (i'm also possibly over thinking this enormously, so i could be way off
 base)



 On Sat, Aug 3, 2013 at 9:31 PM, Carter Schonwald 
 carter.schonw...@gmail.com wrote:

 nvm, githubs backup, i'll have a look! :)


 On Sat, Aug 3, 2013 at 9:05 PM, Carter Schonwald 
 carter.schonw...@gmail.com wrote:

 awesome! (this will also make my work easier)

 ryan: github is down, could you put the branch on bitbucket or some such
 so I can have a lookseee/clone locally?

 thanks!
 -Carter


 On Sat, Aug 3, 2013 at 4:01 AM, Ryan Newton rrnew...@gmail.com wrote:

 Just to keep you all up to date...  I'm adding the primops in question and
 validating the individual commits before putting them here:

 https://github.com/rrnewton/ghc/commits/atomicPrimOps

 The basic idea for using these extensions is:

- the atomic-primops library will work in 7.6 or 7.7+.  It will use
ifdefs to decide whether to use its own primops or GHC-builtin
- future versions will simply get faster, as Carter replaces
out-of-line primops that *also* use C calls, with inline primops / LLVM
equivalents

 Shall I stick a patch on a ticket, or will someone volunteer to pull?
  What's the protocol for requesting commit access anyway?  (By the way, can
 someone share the reason that pull-requests to the github ghc mirror are
 such a no-no?  They seem no worse than a patch in an email which the big 
 warning
 sign https://github.com/ghc/ghc recommends.)

 Best,
   -Ryan

 P.S. FYI, I'm periodically getting these:

 0 caused framework failures
 0 unexpected passes
 1 unexpected failures

  Unexpected failures:
  perf/compiler  T1969 [stat not good enough] (normal)

 Can that just be because of running on a loaded machine?  How narrow are
 these windows?


 On Thu, Aug 1, 2013 at 12:32 PM, Ryan Newton rrnew...@gmail.com wrote:

 On Sun, Jul 21, 2013 at 3:32 AM, Carter Schonwald 
 carter.schonw...@gmail.com wrote:

 ok, could you add those comments (about additional operations to consider)
 to the ticket?


 Sure.  Just did that.


 relatedly: if we want these atomic ops to use the sequential analogues
 when we're not using the threaded run time system, does that mean
 we need to have a symbol / constant variable exposed in the RTS we link
 in, so that the inline code branches on a linktime constant value / symbol
 (something like isThreadedRTS:: Bool, )  or some sort of analogue
 thereof?


 



-- 
Sent from Gmail Mobile
___
ghc-devs mailing list
ghc-devs@haskell.org
http://www.haskell.org/mailman/listinfo/ghc-devs

Re: Proposal: provide cas and barriers symbols even without -threaded

[Ryan Newton]

  That would fix the -threaded/unthreaded disparity.  But I still don't
 see how to access this stuff properly from foreign-primops in a library
 such that GHCI doesn't barf when trying to load the library


 If you're referring to the problem with the missing stg_MUT_VAR_CLEAN_info
 symbol, I'll push a fix for that soon.  Or is there something else?


Ah, yes, I think that will be addressed by your fix.  It's good to hear
that it is considered an ok thing to depend on RTS symbols under all ways.
___
ghc-devs mailing list
ghc-devs@haskell.org
http://www.haskell.org/mailman/listinfo/ghc-devs

Call for talks: Haskell Implementors Workshop 2013, Sept 22, Boston

[Ryan Newton]

Please pass on this announcement!  The deadline is in two weeks.*
*
*
*
* Call for Talks*
   ACM SIGPLAN Haskell Implementors' Workshop

http://haskell.org/haskellwiki/HaskellImplementorsWorkshop/2013
Boston, USA, September 22th, 2013
The workshop will be held in conjunction with ICFP 2013
http://www.icfpconference.org/icfp2013/

*Important dates*

Proposal Deadline:  *13th August2013 (by midnight, any timezone)*
Notification:   27th August2013
Workshop:   22th September 2013

The Haskell Implementors' Workshop is to be held alongside ICFP 2013
this year in Boston. There will be no proceedings; it is an informal
gathering of people involved in the design and development of Haskell
implementations, tools, libraries, and supporting infrastructure.

This relatively new workshop reflects the growth of the user community:
there is a clear need for a well-supported tool chain for the
development, distribution, deployment, and configuration of Haskell
software. The aim is for this workshop to give the people involved with
building the infrastructure behind this ecosystem an opportunity to bat
around ideas, share experiences, and ask for feedback from fellow
experts.

We intend the workshop to have an informal and interactive feel, with a
flexible timetable and plenty of room for ad-hoc discussion, demos, and
impromptu short talks.


Scope and target audience
-

It is important to distinguish the Haskell Implementors' Workshop from
the Haskell Symposium which is also co-located with ICFP 2013. The
Haskell Symposium is for the publication of Haskell-related research. In
contrast, the Haskell Implementors' Workshop will have no proceedings --
although we will aim to make talk videos, slides and presented data
available with the consent of the speakers.

In the Haskell Implementors' Workshop, we hope to study the underlying
technology. We want to bring together anyone interested in the
nitty-gritty details behind turning plain-text source code into a
deployed product. Having said that, members of the wider Haskell
community are more than welcome to attend the workshop -- we need your
feedback to keep the Haskell ecosystem thriving.

The scope covers any of the following topics. There may be some topics
that people feel we've missed, so by all means submit a proposal even if
it doesn't fit exactly into one of these buckets:

  * Compilation techniques
  * Language features and extensions
  * Type system implementation
  * Concurrency and parallelism: language design and implementation
  * Performance, optimisation and benchmarking
  * Virtual machines and run-time systems
  * Libraries and tools for development or deployment


Talks
-

At this stage we would like to invite proposals from potential speakers
for a relatively short talk. We are aiming for 20 minute talks with 10
minutes for questions and changeovers. We want to hear from people
writing compilers, tools, or libraries, people with cool ideas for
directions in which we should take the platform, proposals for new
features to be implemented, and half-baked crazy ideas. Please submit a
talk title and abstract of no more than 200 words.

Submissions should be made via EasyChair.  The website is:
  https://www.easychair.org/conferences/?conf=hiw2013

If you don't have an account you can create one here:
  https://www.easychair.org/account/signup.cgi

Because the submission is an abstract only, please click the abstract
only button when you make your submission.  There is no need to
attach a separate file.

We will also have a lightning talks session which will be organised on
the day. These talks will be 2-10 minutes, depending on available time.
Suggested topics for lightning talks are to present a single idea, a
work-in-progress project, a problem to intrigue and perplex Haskell
implementors, or simply to ask for feedback and collaborators.


Organisers
--

  * Ryan Newton(Indiana University)
  * Neal Glew  (Intel Labs)
  * Edward Yang(Stanford University)
  * Thomas Schilling   (University of Kent)
  * Geoffrey Mainland  (Drexel University)
___
ghc-devs mailing list
ghc-devs@haskell.org
http://www.haskell.org/mailman/listinfo/ghc-devs

Re: Proposal: provide cas and barriers symbols even without -threaded

[Ryan Newton]

Edward,

This makes sense to me.  Especially because eliding-synchronization is
already the convention followed in SMP.hs, where, for example,
write_barrier becomes noops if !THREADED_RTS.

All I would need would be linkable symbols for those noops (a la
Inlines.chttps://github.com/ghc/ghc/blob/master/rts/Inlines.c),
not just the #defines that are currently in SMP.h

I think providing these symbols *reliably* would be complementary to
Carter's proposal to handle them better in the LLVM backend.  In fact,
Carter's proposal is more motivation, for me to be using the official
versions in my .cmm ccalls.
   Right now I've literally copy-pasted the relevant code from SMP.h, into
C code called DUP_cas, DUP_write_barrier etc (yuck).  And these
duplicated versions would be missed by the CMM-LLVM conversion Carter has
proposed.

  -Ryan




On Thu, Jul 18, 2013 at 12:11 PM, Edward Z. Yang ezy...@mit.edu wrote:

 I want to note something, which is that if we did link in
 cas/store_load_barrier, then your lockfree queue would always be
 synchronized, even if you didn't compile with -threaded.  Perhaps this
 is not a big deal, but it is generally nice to not pay the cost of
 synchronization when it is unnecessary.  So it would be better if there
 were threaded/nonthreaded variants which you could use instead.  How
 does that sound? (Fortunately, you are not inlining the functions, so
 it's totally possible for this to happen.  We'd have a tougher row
 to hoe if you needed to inline these functions.)

 Edward

 Excerpts from Ryan Newton's message of Thu Jul 18 06:17:44 -0700 2013:
  The atomic-primops library depends on symbols such as
 store_load_barrier
  and cas, which are defined in SMP.h.  Thus the result is that if the
  program is linked WITHOUT -threaded, the user gets a linker error about
  undefined symbols.
 
  The specific place it's used is in the 'foreign C' bits of this .cmm
 code:
 
 
 
 https://github.com/rrnewton/haskell-lockfree-queue/blob/87e63b21b2a6c375e93c30b98c28c1d04f88781c/AtomicPrimops/cbits/primops.cmm
 
  I'm trying to explore hacks that will enable me to pull in those
 functions
  during compile time, without duplicating a whole bunch of code from the
  RTS.  But it's a fragile business.
 
  It seems to me that some of these routines have general utility.  In
 future
  versions of GHC, could we consider linking in those routines irrespective
  of -threaded?
 
-Ryan

___
ghc-devs mailing list
ghc-devs@haskell.org
http://www.haskell.org/mailman/listinfo/ghc-devs

Re: Proposal: provide cas and barriers symbols even without -threaded

[Ryan Newton]

Hi Carter,

Yes, SMP.h is where I've copy pasted the duplicate functionality from
(since I can't presently rely on linking the symbols).

Your proposal for the LLVM backend sounds **great**.  But it also is going
to provide additional constraints for getting atomic-primops right.
   The goal of atomic-primops is to be a stable Haskell-level interface
into the relevant CAS and fetch-and-add stuff.  The reason this is
important is that one has to be very careful to defeat the GHC optimizer in
all the relevant places and make pointer equality a reliable property.  I
would like to get atomic-primops to work reliably in 7.4, 7.6 [and 7.8] and
have more native support in future GHC releases, where maybe the foreign
primops would become unecessary.  (They are a pain and have already exposed
one blocking cabal bug, fixed in upcoming 1.17.)

A couple additional suggestions for the proposal in ticket #7883:

   - we should use more unique symbols than cas, especially for this
   rewriting trick.  How about ghc_cas or something?
   - it would be great to get at least fetch-and-add in addition to CAS and
   barriers
   - if we reliably provide this set of special symbols, libraries like
   atomic-primops may use them in the .cmm and benefit from the CMM-LLVM
   substitutions
   - if we include all the primops I need in GHC proper the previous bullet
   will stop applying ;-)

Cheers,
  -Ryan

P.S. Just as a bit of motivation, here are some recent performance numbers.
 We often wonder about how close our pure values in a box approach comes
to efficient lock-free structures.  Well here are some numbers about using
a proper unboxed counter in the Haskell heap, vs using an IORef Int and
atomicModifyIORef':  Up to 100X performance difference on some platforms
for microbenchmarks that hammer a counter:


https://github.com/rrnewton/haskell-lockfree-queue/blob/fb12d1121690553e4f737af258848f279147ea24/AtomicPrimops/DEVLOG.md#20130718-timing-atomic-counter-ops

And here are the performance and scaling advantages of using ChaseLev
(based on atomic-primops), over a traditional pure-in-a-box structure
(IORef Data.Seq). The following are timings of ChaseLev/traditional
respectively on a 32 core westmere:

fib(42) 1 threads:  21s
fib(42) 2 threads:  10.1s
fib(42) 4 threads:  5.2s (100%prod)
fib(42) 8 threads:  2.7s - 3.2s (100%prod)
fib(42) 16 threads: 1.28s
fib(42) 24 threads: 1.85s
fib(42) 32 threads: 4.8s (high variance)

(hive) fib(42) 1 threads:  41.8s  (95% prod)
(hive) fib(42) 2 threads:  25.2s  (66% prod)
(hive) fib(42) 4 threads:  14.6s  (27% prod, 135GB alloc)
(hive) fib(42) 8 threads:  17.1s  (26% prod)
(hive) fib(42) 16 threads: 16.3s  (13% prod)
(hive) fib(42) 24 threads: 21.2s  (30% prod)
(hive) fib(42) 32 threads: 29.3s  (33% prod)

And that is WITH the inefficiency of doing a ccall on every single atomic
operation.

Notes on parfib performance are here:

https://github.com/rrnewton/haskell-lockfree-queue/blob/d6d3e9eda2a487a5f055b1f51423954bb6b6bdfa/ChaseLev/Test.hs#L158







On Fri, Jul 19, 2013 at 5:05 PM, Carter Schonwald 
carter.schonw...@gmail.com wrote:

 ryan, the relevant machinery on the C side is here, see
 ./includes/stg/SMP.h :
 https://github.com/ghc/ghc/blob/7cc8a3cc5c2970009b83844ff9cc4e27913b8559/includes/stg/SMP.h

 (unless i'm missing something)


 On Fri, Jul 19, 2013 at 4:53 PM, Carter Schonwald 
 carter.schonw...@gmail.com wrote:

 Ryan,
 if you look at line 270, you'll see the CAS is a C call
 https://github.com/ghc/ghc/blob/95e6865ecf06b2bd80fa737e4fa4a24beaae25c5/rts/PrimOps.cmm#L270


 What Simon is alluding to is some work I started (but need to finish)
 http://ghc.haskell.org/trac/ghc/ticket/7883 is the relevant ticket, and
 I'll need to sort out doing the same on the native code gen too

 there ARE no write barrier primops, they're baked into the CAS machinery
 in ghc's rts


 On Fri, Jul 19, 2013 at 1:02 PM, Ryan Newton rrnew...@gmail.com wrote:

 Yes, I'd absolutely rather not suffer C call overhead for these
 functions (or the CAS functions).  But isn't that how it's done currently
 for the casMutVar# primop?


 https://github.com/ghc/ghc/blob/95e6865ecf06b2bd80fa737e4fa4a24beaae25c5/rts/PrimOps.cmm#L265

 To avoid the overhead, is it necessary to make each primop in-line
 rather than out-of-line, or just to get rid of the ccall?

 Another reason it would be good to package these with GHC is that I'm
 having trouble building robust libraries of foreign primops that work under
 all ways (e.g. GHCI).  For example, this bug:

 https://github.com/rrnewton/haskell-lockfree-queue/issues/10

 If I write .cmm code that depends on RTS functionality like
 stg_MUT_VAR_CLEAN_info, then it seems to work fine when in compiled mode
 (with/without threading, profiling), but I get link errors from GHCI where
 these symbols aren't defined.

 I've got a draft of the relevant primops here:


 https://github.com/rrnewton/haskell-lockfree-queue

Re: Proposal: provide cas and barriers symbols even without -threaded

[Ryan Newton]

Hi Simon,

That sounds like a good solution and I'll attempt a patch.  I think the fix
is only three lines.  That is, replace these three lines with EXTERN_INLINE
C functions:

#define write_barrier()  /* nothing */
#define store_load_barrier() /* nothing */
#define load_load_barrier()  /* nothing */

That would fix the -threaded/unthreaded disparity.  But I still don't see
how to access this stuff properly from foreign-primops in a library such
that GHCI doesn't barf when trying to load the library

  -Ryan
___
ghc-devs mailing list
ghc-devs@haskell.org
http://www.haskell.org/mailman/listinfo/ghc-devs

Re: Proposal: provide cas and barriers symbols even without -threaded

[Ryan Newton]

Sorry, rewrite was too overloaded a term to use here.  I was just
referring to the proposal to substitute the cas funcall with the right
llvm operation.

That is, the approach would pattern match for the CMM code ccall cas or
foreign C cas (I'm afraid I don't know the difference between those)
and replace it with the equivalent LLVM op, right?

I think the assumption there is that the native codegen would still have to
suffer the funcall overhead and use the C versions.  I don't know exactly
what the changes would look like to make barriers/CAS all proper inline
primops, because it would have to reproduce in the code generator all the
platform-specific #ifdef'd C code that is currently in SMP.h.  Which I
guess is doable, but probably only for someone who knows the native GHC
codegen properly...



On Sat, Jul 20, 2013 at 2:30 AM, Carter Schonwald 
carter.schonw...@gmail.com wrote:

 Ryan, could you explain what you want more precisely? Specifically what
 you want in terms of exposed primops using the terminology / vocabulary in
 http://llvm.org/docs/LangRef.html#ordering and
 http://llvm.org/docs/Atomics.html ?

  I'll first do the work for just the LLVM backend, and Ill likely need
 some active guidance / monitoring for the native codegen analogues

 (also asked this on ticket for documentation purposes)


 On Sat, Jul 20, 2013 at 2:18 AM, Ryan Newton rrnew...@gmail.com wrote:

 Hi Carter,

 Yes, SMP.h is where I've copy pasted the duplicate functionality from
 (since I can't presently rely on linking the symbols).

 Your proposal for the LLVM backend sounds **great**.  But it also is
 going to provide additional constraints for getting atomic-primops right.

The goal of atomic-primops is to be a stable Haskell-level interface
 into the relevant CAS and fetch-and-add stuff.  The reason this is
 important is that one has to be very careful to defeat the GHC optimizer in
 all the relevant places and make pointer equality a reliable property.  I
 would like to get atomic-primops to work reliably in 7.4, 7.6 [and 7.8] and
 have more native support in future GHC releases, where maybe the foreign
 primops would become unecessary.  (They are a pain and have already exposed
 one blocking cabal bug, fixed in upcoming 1.17.)

 A couple additional suggestions for the proposal in ticket #7883:

- we should use more unique symbols than cas, especially for this
rewriting trick.  How about ghc_cas or something?
- it would be great to get at least fetch-and-add in addition to CAS
and barriers
- if we reliably provide this set of special symbols, libraries like
atomic-primops may use them in the .cmm and benefit from the CMM-LLVM
substitutions
- if we include all the primops I need in GHC proper the previous
bullet will stop applying ;-)

 Cheers,
   -Ryan

 P.S. Just as a bit of motivation, here are some recent performance
 numbers.  We often wonder about how close our pure values in a box
 approach comes to efficient lock-free structures.  Well here are some
 numbers about using a proper unboxed counter in the Haskell heap, vs using
 an IORef Int and atomicModifyIORef':  Up to 100X performance difference
 on some platforms for microbenchmarks that hammer a counter:


 https://github.com/rrnewton/haskell-lockfree-queue/blob/fb12d1121690553e4f737af258848f279147ea24/AtomicPrimops/DEVLOG.md#20130718-timing-atomic-counter-ops

 And here are the performance and scaling advantages of using ChaseLev
 (based on atomic-primops), over a traditional pure-in-a-box structure
 (IORef Data.Seq). The following are timings of ChaseLev/traditional
 respectively on a 32 core westmere:

 fib(42) 1 threads:  21s
 fib(42) 2 threads:  10.1s
 fib(42) 4 threads:  5.2s (100%prod)
 fib(42) 8 threads:  2.7s - 3.2s (100%prod)
 fib(42) 16 threads: 1.28s
 fib(42) 24 threads: 1.85s
 fib(42) 32 threads: 4.8s (high variance)

 (hive) fib(42) 1 threads:  41.8s  (95% prod)
 (hive) fib(42) 2 threads:  25.2s  (66% prod)
 (hive) fib(42) 4 threads:  14.6s  (27% prod, 135GB alloc)
 (hive) fib(42) 8 threads:  17.1s  (26% prod)
 (hive) fib(42) 16 threads: 16.3s  (13% prod)
 (hive) fib(42) 24 threads: 21.2s  (30% prod)
 (hive) fib(42) 32 threads: 29.3s  (33% prod)

 And that is WITH the inefficiency of doing a ccall on every single
 atomic operation.

 Notes on parfib performance are here:


 https://github.com/rrnewton/haskell-lockfree-queue/blob/d6d3e9eda2a487a5f055b1f51423954bb6b6bdfa/ChaseLev/Test.hs#L158







 On Fri, Jul 19, 2013 at 5:05 PM, Carter Schonwald 
 carter.schonw...@gmail.com wrote:

 ryan, the relevant machinery on the C side is here, see
 ./includes/stg/SMP.h :
 https://github.com/ghc/ghc/blob/7cc8a3cc5c2970009b83844ff9cc4e27913b8559/includes/stg/SMP.h

 (unless i'm missing something)


 On Fri, Jul 19, 2013 at 4:53 PM, Carter Schonwald 
 carter.schonw...@gmail.com wrote:

 Ryan,
 if you look at line 270, you'll see the CAS is a C call
 https

Re: Proposal: provide cas and barriers symbols even without -threaded

[Ryan Newton]

Yes, I'd absolutely rather not suffer C call overhead for these functions
(or the CAS functions).  But isn't that how it's done currently for the
casMutVar# primop?

https://github.com/ghc/ghc/blob/95e6865ecf06b2bd80fa737e4fa4a24beaae25c5/rts/PrimOps.cmm#L265

To avoid the overhead, is it necessary to make each primop in-line rather
than out-of-line, or just to get rid of the ccall?

Another reason it would be good to package these with GHC is that I'm
having trouble building robust libraries of foreign primops that work under
all ways (e.g. GHCI).  For example, this bug:

https://github.com/rrnewton/haskell-lockfree-queue/issues/10

If I write .cmm code that depends on RTS functionality like
stg_MUT_VAR_CLEAN_info, then it seems to work fine when in compiled mode
(with/without threading, profiling), but I get link errors from GHCI where
these symbols aren't defined.

I've got a draft of the relevant primops here:

https://github.com/rrnewton/haskell-lockfree-queue/blob/master/AtomicPrimops/cbits/primops.cmm

Which includes:

   - variants of CAS for MutableArray# and MutableByteArray#
   - fetch-and-add for MutableByteArray#

Also, there are some tweaks to support the new ticketed interface for
safer CAS:


http://hackage.haskell.org/packages/archive/atomic-primops/0.3/doc/html/Data-Atomics.html#g:3

I started adding some of these primops to GHC proper (still as
out-of-line), but not all of them.  I had gone with the foreign primop
route instead...

   https://github.com/rrnewton/ghc/commits/master

  -Ryan

P.S. Where is the write barrier primop?  I don't see it listed in
prelude/primops.txt...





On Fri, Jul 19, 2013 at 11:41 AM, Carter Schonwald 
carter.schonw...@gmail.com wrote:

 I guess I should find the time to finish the CAS primop work I volunteered
 to do then. Ill look into in a few days.


 On Friday, July 19, 2013, Simon Marlow wrote:

 On 18/07/13 14:17, Ryan Newton wrote:

 The atomic-primops library depends on symbols such as
 store_load_barrier and cas, which are defined in SMP.h.  Thus the
 result is that if the program is linked WITHOUT -threaded, the user
 gets a linker error about undefined symbols.

 The specific place it's used is in the 'foreign C' bits of this .cmm
 code:

 https://github.com/rrnewton/**haskell-lockfree-queue/blob/**
 87e63b21b2a6c375e93c30b98c28c1**d04f88781c/AtomicPrimops/**
 cbits/primops.cmmhttps://github.com/rrnewton/haskell-lockfree-queue/blob/87e63b21b2a6c375e93c30b98c28c1d04f88781c/AtomicPrimops/cbits/primops.cmm

 I'm trying to explore hacks that will enable me to pull in those
 functions during compile time, without duplicating a whole bunch of code
 from the RTS.  But it's a fragile business.

 It seems to me that some of these routines have general utility.  In
 future versions of GHC, could we consider linking in those routines
 irrespective of -threaded?


 We should make the non-THREADED versions EXTERN_INLINE too, so that there
 will be (empty) functions to call in rts/Inlines.c.  Want to submit a patch?

 A better solution would be to make them into primops.  You don't really
 want to be calling out to a C function to implement a memory barrier. We
 have this for write_barrier(), but none of the others so far.  Of couse
 that's a larger change.

 Cheers,
 Simon



 __**_
 ghc-devs mailing list
 ghc-devs@haskell.org
 http://www.haskell.org/**mailman/listinfo/ghc-devshttp://www.haskell.org/mailman/listinfo/ghc-devs


___
ghc-devs mailing list
ghc-devs@haskell.org
http://www.haskell.org/mailman/listinfo/ghc-devs

Proposal: provide cas and barriers symbols even without -threaded

[Ryan Newton]

The atomic-primops library depends on symbols such as store_load_barrier
and cas, which are defined in SMP.h.  Thus the result is that if the
program is linked WITHOUT -threaded, the user gets a linker error about
undefined symbols.

The specific place it's used is in the 'foreign C' bits of this .cmm code:


https://github.com/rrnewton/haskell-lockfree-queue/blob/87e63b21b2a6c375e93c30b98c28c1d04f88781c/AtomicPrimops/cbits/primops.cmm

I'm trying to explore hacks that will enable me to pull in those functions
during compile time, without duplicating a whole bunch of code from the
RTS.  But it's a fragile business.

It seems to me that some of these routines have general utility.  In future
versions of GHC, could we consider linking in those routines irrespective
of -threaded?

  -Ryan
___
ghc-devs mailing list
ghc-devs@haskell.org
http://www.haskell.org/mailman/listinfo/ghc-devs

Re: [GHC] #7820: Installing profiling library BREAKS non-profiling executable

[Ryan Newton]

By the way, does anyone have a positive example of a package on hackage
that (1) uses foreign primops and (2) is robust across a wide variety of
platforms and build methods?  It would be great to work from such an
example.

Thanks,
  -Ryan



On Mon, Apr 8, 2013 at 10:57 AM, GHC cvs-...@haskell.org wrote:

 #7820: Installing profiling library BREAKS non-profiling executable

 --+-
 Reporter:  rrnewton   |  Owner:
 Type:  bug| Status:  new
 Priority:  normal |  Component:  Compiler
  Version:  7.6.2  |   Keywords:  profiling
   Os:  Linux  |   Architecture:  x86_64 (amd64)
  Failure:  Runtime crash  |  Blockedby:
 Blocking: |Related:

 --+-
  I am trying to work through problems with different GHC ways so that I
  can broadly deploy a library of foreign primops for lockfree data
  structures.  Here's the package I am testing at the moment:

  https://github.com/rrnewton/haskell-lockfree-
  queue/tree/5d990fe43a983fc5ed32c3f58cb2e59df71b00b6/AtomicPrimops

  Here is a recipe for reproducing the bug from within that directory.  Both
  of these build modes work FINE:

  (cabal install --enable-library-profiling; cd testing; make prof;
  ./Test_prof.exe)
  (cabal install --disable-library-profiling; cd testing; make;
  ./Test_threaded.exe)

  That is, building both library and executable with profiling, or without
  profiling, works fine.  But then this creates an executable that
  segfaults:

  (cabal install --enable-library-profiling; cd testing; make;
  ./Test_threaded.exe)

  At runtime the call to the .cmm-defined foreign primop just returns zero,
  and then the process segfaults when it tries to use that as a Haskell
  value.

  But my understanding is that --enable-library-profiling should just add
  extra binaries to .cabal and NOT change the behavior of the non-profiling
  binaries.

  Yet I can confirm that I see slightly different binary sizes with the non-
  profiling .o and .a files installed in ~/.cabal/lib depending on whether
  --enable-library-profiling is activated.  Perhaps this in itself is a
  cabal bug, but I haven't tracked down exactly what the difference is yet.

  I have confirmed the above behavior under:
* Mac OS (ghc 7.4.2 and 7.6.2)
* Linux, RHEL 6 (ghc 7.4.2 and 7.6.2)

 --
 Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/7820
 GHC http://www.haskell.org/ghc/
 The Glasgow Haskell Compiler

___
ghc-devs mailing list
ghc-devs@haskell.org
http://www.haskell.org/mailman/listinfo/ghc-devs