Re: Array heap objects

[David Feuer]

No, I don't think that gets me all the way to what I want, although it
might be a reasonable approach to one *part* of it. The main idea I'm after
is efficient representations of things like HashMap, where nodes in a tree
have variable numbers of children. First, let's look at how HashMap is
actually defined (with inconsequential simplifications & clarifications):

data HashMap k v
= Empty
| BitmapIndexed !Word !(SmallArray (HashMap k v))
| Leaf !Word !(Leaf k v)
| Collision !Word !(SmallArray (Leaf k v))

The internal nodes are represented by BitmapIndexed constructors. The
biggest goal is to flatten those BitmapIndexed values out, so let's start
with just that. Conceptually, imagine we have

data HashMap# :: Type -> Type -> TYPE 'UnliftedRep -- hopefully soon
('BoxedRep 'Unlifted)
pattern BitmapIndexed# :: Word# -> SmallArray# (HashMap k v) -> HashMap# k v

I want a value produced by `BitmapIndexed#` to be a single heap object with:

1. Enough information to tell that it was produced by `BitmapIndexed#`.
2. The included `Word#` (representing the hash).
3. Pointers to all the children.

One way to think of this is as unpacking arrays into (unlifted)
user-defined types. Something like

unlifted data HashMap#
  = Empty#
  | BitmapIndexed# !Word {-# UnpackArray #-} {-# Unpack #-} !(SmallArray
(HashMap k v))
  | ...

Where these fake pragmas indicate a sort of double unpacking, where the
array size and elements
are unpacked into the `BitmapIndexed#` constructor.

Now let's get into the leaves. We may well want to unpack the leaves into
their parents. This gets
a little messy. If we have a totally wild bitmap approach, this is pretty
straightforward, because we
can indicate which elements are pointers (to child nodes, keys, or values)
and which elements are
unboxed (hashes), and do some fancy arithmetic to figure out where each
thing is. Is there a
cleaner, more general approach? I think that's going to be very hard. From
a "pure" standpoint,
we could imagine some sort of succinct data structure that lets us locate
the nth element and figure
out which alternative of the sum it is. But then we'd need some *very*
principled operations for
constructing arrays; the usual unrestricted mutation approach is not going
to work at all.

On Thu, Oct 8, 2020, 3:12 PM Ben Gamari  wrote:

> David Feuer  writes:
>
> > Ah, too bad about reuse. What do you mean about walking over both
> pointers
> > and non-pointers? The extra word (for pointers-first layout) or few words
> > (for bitmapped) will be more than made up for in most cases by not
> needing
> > extra heap objects between a node and its children.
> >
> Let's consider the case that you want an array of (String, Int#) pairs.
> Today you have a choice of two representations:
>
>  * structure-of-arrays:
>
>type Array = (Array# String, ByteArray#)
>
>where the ByteArray# contains packed `Int#`s.
>
>  * array-of-structures:
>
>data Entry = Entry String !Int#
>
>type Array = Array# Entry
>
> In the latter case you indeed incur an extra indirection, but not in the
> former. It is true that locality may be in the former (especially
> if your entry is a wide product rather than the pair used here) but on
> the whole it's often a good choice.
>
> I think you are asking for a very general MixedArray#, which would allow
> you to specify dynamically whether each word of the array is a pointer
> or not. This seems like a lot of power and may complicate garbage
> collection. Apologies if I have misunderstood.
>
> I think you can get most of the power of this idea with a much simpler
> idea: rather than allow configuration of each word dynamically, specify
> the array as a bunch of packed unboxed tuples. For instance, in Haskell
> this might look like:
>
>  -- This class would need to be magic since the compiler would need
>  -- to generate a info table describing the entry layout for each
>  -- instance.
>  class HasArrayRep (a :: k)
>
>  data PackedArray# (a :: k)
>
>  newPackedArray# :: HasArrayRep a
>  => Int#
>  -> State# s -> (# State# s, PackedArray# a)
>  readPackedArray# :: HasArrayRep a
>   => Int -> PackedArray# a ->
>   -> State# s -> (# State# s, a)
>  -- ... etc.
>
> Implementation would involve the introduction of a new array closure
> type and suitable scavenging logic. Each HasArrayRep instance would emit
> an info table; the layout section of this info table would contain a
> bitmap (of the same sort used for stack frames) which would describe the
> layout of a single array element. To avoid unaligned accesses you would
> want to round up the element size to the nearest word size, potentially
> incurring some slop (e.g. each element of a `PackedArray# (# Word#,
> Word8# #)` would require two words). However, sub-word-size fields could
> in principle be packed into a single word (finally allowing us to get
> some 

Re: msys woes

[Ben Gamari]

Phyx  writes:

>> I'm not sure what you mean by "the tag won't move". Could you clarify?
>
> I mean that the tag called 8.8.4-release will still point to the a broken
> hash no? Since it represents the hash of the commit the release was made
> from.
>
Ahh, yes. 8.8.4-release will not change. In general we do not change
tags after their corresponding release has been announced.

Cheers,

- Ben


signature.asc
Description: PGP signature
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: msys woes

[Shayne Fletcher]

On Thu, Oct 8, 2020 at 4:28 PM Ben Gamari  wrote:

> Phyx  writes:
>
> > Right,
> >
> > I think seems to be a bug in the script that wasn't noticed before.
> >
> > On line 40 there are () missing around the condition. So it's checking
> both
> > URLs i think.
> >
> > The download to Haskell.org succeeds but the msys2 one fails since that
> > package is gone.
> >
> > The whole thing after the || needs to be in ().
> >
> > This would need to be fixed in that branch but the tag won't move.
> >
> > Any ideas here Ben?
> >
> I'm not sure what you mean by "the tag won't move". Could you clarify?
>
> Regardless, I do see that some parens are needed here. I have opened
> #18821 to track this.
>
> For what it's worth, the download script was revamped in GHC 8.10 so I
> don't think there is anything needed on master or ghc-8.10.
>
>
I confirm that the specific script 'mk/get-win32-tarballs.sh' is not
present in 8.10. There is a get-win32-tarballs.py there. Looks like the
last version with the .sh was 8.8.4


> Cheers,
>
> - Ben
>


-- 
Shayne Fletcher
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: msys woes

[Phyx]

> I'm not sure what you mean by "the tag won't move". Could you clarify?

I mean that the tag called 8.8.4-release will still point to the a broken
hash no? Since it represents the hash of the commit the release was made
from.

Sent from my Mobile

On Thu, Oct 8, 2020, 21:28 Ben Gamari  wrote:

> Phyx  writes:
>
> > Right,
> >
> > I think seems to be a bug in the script that wasn't noticed before.
> >
> > On line 40 there are () missing around the condition. So it's checking
> both
> > URLs i think.
> >
> > The download to Haskell.org succeeds but the msys2 one fails since that
> > package is gone.
> >
> > The whole thing after the || needs to be in ().
> >
> > This would need to be fixed in that branch but the tag won't move.
> >
> > Any ideas here Ben?
> >
> I'm not sure what you mean by "the tag won't move". Could you clarify?
>
> Regardless, I do see that some parens are needed here. I have opened
> #18821 to track this.
>
> For what it's worth, the download script was revamped in GHC 8.10 so I
> don't think there is anything needed on master or ghc-8.10.
>
> Cheers,
>
> - Ben
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: msys woes

[Ben Gamari]

Phyx  writes:

> Right,
>
> I think seems to be a bug in the script that wasn't noticed before.
>
> On line 40 there are () missing around the condition. So it's checking both
> URLs i think.
>
> The download to Haskell.org succeeds but the msys2 one fails since that
> package is gone.
>
> The whole thing after the || needs to be in ().
>
> This would need to be fixed in that branch but the tag won't move.
>
> Any ideas here Ben?
>
I'm not sure what you mean by "the tag won't move". Could you clarify?

Regardless, I do see that some parens are needed here. I have opened
#18821 to track this.

For what it's worth, the download script was revamped in GHC 8.10 so I
don't think there is anything needed on master or ghc-8.10.

Cheers,

- Ben


signature.asc
Description: PGP signature
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Array heap objects

[Ben Gamari]

David Feuer  writes:

> Ah, too bad about reuse. What do you mean about walking over both pointers
> and non-pointers? The extra word (for pointers-first layout) or few words
> (for bitmapped) will be more than made up for in most cases by not needing
> extra heap objects between a node and its children.
>
Let's consider the case that you want an array of (String, Int#) pairs.
Today you have a choice of two representations:

 * structure-of-arrays:

   type Array = (Array# String, ByteArray#)

   where the ByteArray# contains packed `Int#`s.

 * array-of-structures:

   data Entry = Entry String !Int#

   type Array = Array# Entry

In the latter case you indeed incur an extra indirection, but not in the
former. It is true that locality may be in the former (especially
if your entry is a wide product rather than the pair used here) but on
the whole it's often a good choice.

I think you are asking for a very general MixedArray#, which would allow
you to specify dynamically whether each word of the array is a pointer
or not. This seems like a lot of power and may complicate garbage
collection. Apologies if I have misunderstood.

I think you can get most of the power of this idea with a much simpler
idea: rather than allow configuration of each word dynamically, specify
the array as a bunch of packed unboxed tuples. For instance, in Haskell
this might look like:

 -- This class would need to be magic since the compiler would need
 -- to generate a info table describing the entry layout for each
 -- instance.
 class HasArrayRep (a :: k)

 data PackedArray# (a :: k)

 newPackedArray# :: HasArrayRep a
 => Int#
 -> State# s -> (# State# s, PackedArray# a)
 readPackedArray# :: HasArrayRep a
  => Int -> PackedArray# a ->
  -> State# s -> (# State# s, a)
 -- ... etc.

Implementation would involve the introduction of a new array closure
type and suitable scavenging logic. Each HasArrayRep instance would emit
an info table; the layout section of this info table would contain a
bitmap (of the same sort used for stack frames) which would describe the
layout of a single array element. To avoid unaligned accesses you would
want to round up the element size to the nearest word size, potentially
incurring some slop (e.g. each element of a `PackedArray# (# Word#,
Word8# #)` would require two words). However, sub-word-size fields could
in principle be packed into a single word (finally allowing us to get
some mileage out of the sub-word-size primitive types we now have).

Perhaps this helps in your use-case?

Cheers,

- Ben


signature.asc
Description: PGP signature
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: msys woes

[Shayne Fletcher]

On Thu, Oct 8, 2020 at 1:18 PM Phyx  wrote:

> From the looks of it the repo.msys2.org system doesn't have enough space
> to restore all these old packages.
>
> So we'll have to handle it on our end. The easiest way to get your builds
> working again is by applying a patch with the fix to the source tree.
>
>
Nice one Phyx. I confirm that applying that patch at my end gets me back in
the game.

-- 
Shayne Fletcher
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: msys woes

[Phyx]

>From the looks of it the repo.msys2.org system doesn't have enough space to
restore all these old packages.

So we'll have to handle it on our end. The easiest way to get your builds
working again is by applying a patch with the fix to the source tree.

Sent from my Mobile

On Thu, Oct 8, 2020, 17:55 Phyx  wrote:

> Right,
>
> I think seems to be a bug in the script that wasn't noticed before.
>
> On line 40 there are () missing around the condition. So it's checking
> both URLs i think.
>
> The download to Haskell.org succeeds but the msys2 one fails since that
> package is gone.
>
> The whole thing after the || needs to be in ().
>
> This would need to be fixed in that branch but the tag won't move.
>
> Any ideas here Ben?
>
> Sent from my Mobile
>
> On Thu, Oct 8, 2020, 17:42 Shayne Fletcher 
> wrote:
>
>>
>>
>> On Thu, Oct 8, 2020 at 12:19 PM Shayne Fletcher <
>> shayne.fletcher...@gmail.com> wrote:
>>
>>>
>>>
>>> On Thu, Oct 8, 2020 at 12:13 PM Phyx  wrote:
>>>
 Hi Shayne,

 8.8.1 had a different mechanism but the primary is still haskell.org.


>>> I think we are looking at ghc-8.4.3 here (being used on 8.8.1 sources).
>>>
>>>
 You don't need to run the entire pipeline to test this. Just run

 ./mk/get-win32-tarballs.sh download x86_64


>>> I've thrown that into the build script. Stand-by. Let's see what we get.
>>>
>>>
>> Here we go!
>>
>> ```
>>
>> 2020-10-08T16:35:14.8128041Z Downloading 
>> mingw-w64-x86_64-crt-git-5.0.0.4795.e3d96cb1-1-any.pkg.tar.xz to 
>> ghc-tarballs/mingw-w64/x86_64...
>> 2020-10-08T16:35:15.3839177Z #=#=#
>> 2020-10-08T16:35:15.3841316Z
>> 2020-10-08T16:35:15.4836014Z 
>>   22.6%
>> 2020-10-08T16:35:15.4918420Z 
>> ##
>> 92.9%
>> 2020-10-08T16:35:15.4921362Z 
>>  
>> 100.0%
>> 2020-10-08T16:35:15.7741545Z #=#=#
>> 2020-10-08T16:35:15.8742272Z ##O#- #
>> 2020-10-08T16:35:15.9760569Z ##O=#  #
>> 2020-10-08T16:35:16.9768720Z #=#=-#  #
>> 2020-10-08T16:35:17.4160288Z -#O#- #   #
>> 2020-10-08T16:35:17.4162806Z -=#=#   #   #
>> 2020-10-08T16:35:17.4164112Z curl: (22) The requested URL returned error: 
>> 404 Not Found
>> 2020-10-08T16:35:17.4407190Z
>> 2020-10-08T16:35:17.4425090Z ERROR: Download failed.
>> 2020-10-08T16:35:17.4564686Z ghc-lib-gen.EXE: Failed when running system 
>> command: stack --stack-yaml hadrian/stack.yaml exec -- bash -c 
>> "./mk/get-win32-tarballs.sh download x86_64"
>> ```
>>
>> Hope this helps!
>>
>> --
>> Shayne Fletcher
>>
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: msys woes

[Phyx]

Right,

I think seems to be a bug in the script that wasn't noticed before.

On line 40 there are () missing around the condition. So it's checking both
URLs i think.

The download to Haskell.org succeeds but the msys2 one fails since that
package is gone.

The whole thing after the || needs to be in ().

This would need to be fixed in that branch but the tag won't move.

Any ideas here Ben?

Sent from my Mobile

On Thu, Oct 8, 2020, 17:42 Shayne Fletcher 
wrote:

>
>
> On Thu, Oct 8, 2020 at 12:19 PM Shayne Fletcher <
> shayne.fletcher...@gmail.com> wrote:
>
>>
>>
>> On Thu, Oct 8, 2020 at 12:13 PM Phyx  wrote:
>>
>>> Hi Shayne,
>>>
>>> 8.8.1 had a different mechanism but the primary is still haskell.org.
>>>
>>>
>> I think we are looking at ghc-8.4.3 here (being used on 8.8.1 sources).
>>
>>
>>> You don't need to run the entire pipeline to test this. Just run
>>>
>>> ./mk/get-win32-tarballs.sh download x86_64
>>>
>>>
>> I've thrown that into the build script. Stand-by. Let's see what we get.
>>
>>
> Here we go!
>
> ```
>
> 2020-10-08T16:35:14.8128041Z Downloading 
> mingw-w64-x86_64-crt-git-5.0.0.4795.e3d96cb1-1-any.pkg.tar.xz to 
> ghc-tarballs/mingw-w64/x86_64...
> 2020-10-08T16:35:15.3839177Z #=#=#
> 2020-10-08T16:35:15.3841316Z
> 2020-10-08T16:35:15.4836014Z  
>  22.6%
> 2020-10-08T16:35:15.4918420Z 
> ##
> 92.9%
> 2020-10-08T16:35:15.4921362Z 
>  
> 100.0%
> 2020-10-08T16:35:15.7741545Z #=#=#
> 2020-10-08T16:35:15.8742272Z ##O#- #
> 2020-10-08T16:35:15.9760569Z ##O=#  #
> 2020-10-08T16:35:16.9768720Z #=#=-#  #
> 2020-10-08T16:35:17.4160288Z -#O#- #   #
> 2020-10-08T16:35:17.4162806Z -=#=#   #   #
> 2020-10-08T16:35:17.4164112Z curl: (22) The requested URL returned error: 404 
> Not Found
> 2020-10-08T16:35:17.4407190Z
> 2020-10-08T16:35:17.4425090Z ERROR: Download failed.
> 2020-10-08T16:35:17.4564686Z ghc-lib-gen.EXE: Failed when running system 
> command: stack --stack-yaml hadrian/stack.yaml exec -- bash -c 
> "./mk/get-win32-tarballs.sh download x86_64"
> ```
>
> Hope this helps!
>
> --
> Shayne Fletcher
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: msys woes

[Shayne Fletcher]

On Thu, Oct 8, 2020 at 12:19 PM Shayne Fletcher <
shayne.fletcher...@gmail.com> wrote:

>
>
> On Thu, Oct 8, 2020 at 12:13 PM Phyx  wrote:
>
>> Hi Shayne,
>>
>> 8.8.1 had a different mechanism but the primary is still haskell.org.
>>
>>
> I think we are looking at ghc-8.4.3 here (being used on 8.8.1 sources).
>
>
>> You don't need to run the entire pipeline to test this. Just run
>>
>> ./mk/get-win32-tarballs.sh download x86_64
>>
>>
> I've thrown that into the build script. Stand-by. Let's see what we get.
>
>
Here we go!

```

2020-10-08T16:35:14.8128041Z Downloading
mingw-w64-x86_64-crt-git-5.0.0.4795.e3d96cb1-1-any.pkg.tar.xz to
ghc-tarballs/mingw-w64/x86_64...
2020-10-08T16:35:15.3839177Z #=#=#
2020-10-08T16:35:15.3841316Z
2020-10-08T16:35:15.4836014Z 
22.6%
2020-10-08T16:35:15.4918420Z
##
   92.9%
2020-10-08T16:35:15.4921362Z

100.0%
2020-10-08T16:35:15.7741545Z #=#=#
2020-10-08T16:35:15.8742272Z ##O#- #
2020-10-08T16:35:15.9760569Z ##O=#  #
2020-10-08T16:35:16.9768720Z #=#=-#  #
2020-10-08T16:35:17.4160288Z -#O#- #   #
2020-10-08T16:35:17.4162806Z -=#=#   #   #
2020-10-08T16:35:17.4164112Z curl: (22) The requested URL returned
error: 404 Not Found
2020-10-08T16:35:17.4407190Z
2020-10-08T16:35:17.4425090Z ERROR: Download failed.
2020-10-08T16:35:17.4564686Z ghc-lib-gen.EXE: Failed when running
system command: stack --stack-yaml hadrian/stack.yaml exec -- bash -c
"./mk/get-win32-tarballs.sh download x86_64"
```

Hope this helps!

-- 
Shayne Fletcher
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: msys woes

[Shayne Fletcher]

On Thu, Oct 8, 2020 at 12:13 PM Phyx  wrote:

> Hi Shayne,
>
> 8.8.1 had a different mechanism but the primary is still haskell.org.
>
>
I think we are looking at ghc-8.4.3 here (being used on 8.8.1 sources).


> You don't need to run the entire pipeline to test this. Just run
>
> ./mk/get-win32-tarballs.sh download x86_64
>
>
I've thrown that into the build script. Stand-by. Let's see what we get.

-- 
Shayne Fletcher
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: msys woes

[Phyx]

Hi Shayne,

8.8.1 had a different mechanism but the primary is still haskell.org.

You don't need to run the entire pipeline to test this. Just run

./mk/get-win32-tarballs.sh download x86_64

This should echo to stdout more information.

Kind regards,
Tamar

Sent from my Mobile

On Thu, Oct 8, 2020, 15:54 Shayne Fletcher 
wrote:

>
>
> On Thu, Oct 8, 2020 at 10:51 AM Shayne Fletcher <
> shayne.fletcher...@gmail.com> wrote:
>
>>
>> Hi Phyx,
>>
>> On Thu, Oct 8, 2020 at 9:10 AM Phyx  wrote:
>>
>>> > `./configure --enable-tarballs-autodownload` GHC build step on Windows
>>> has been failing because repo.msys2.org
>>>
>>> Afaik GHC doesn't rely ok repo.msys2.org for builds, only for
>>> mirroring. The primary url is haskell.org
>>> https://downloads.haskell.org/ghc/mingw/
>>>
>>> So it's down time shouldn't have affected you (and works for me).
>>>
>>>
>> [...]
>>
>> A procedure to reproduce it would be,
>> ```
>> cd ghc
>> git fetch --tags && git checkout ghc-8.8.1-release
>> git submodule update --init --recursive
>> stack --stack-yaml hadrian/stack.yaml exec -- bash -c "./configure
>> --enable-tarballs-autodownload"
>> ```
>>
>
> Oops, obviously, there should be a boot in there first.
> ```
> cd ghc
> git fetch --tags && git checkout ghc-8.8.1-release
> git submodule update --init --recursive
> stack --stack-yaml hadrian/stack.yaml exec -- bash -c ./boot
> stack --stack-yaml hadrian/stack.yaml exec -- bash -c "./configure
> --enable-tarballs-autodownload"
> ```
>
> --
> Shayne Fletcher
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: msys woes

[Shayne Fletcher]

On Thu, Oct 8, 2020 at 11:18 AM Sylvain Henry  wrote:

> Could you share the contents of "missing-win32-tarballs" log file?
>

Where exactly does that file reside? In any case I think the answer is
likely no Sylvain as it will be on an inaccessible Azure host.

-- 
Shayne Fletcher
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Array heap objects

[David Feuer]

Ah, too bad about reuse. What do you mean about walking over both pointers
and non-pointers? The extra word (for pointers-first layout) or few words
(for bitmapped) will be more than made up for in most cases by not needing
extra heap objects between a node and its children.

Simon has explained that these unsafe coercions between lifted and unlifted
are really *too* unsafe (so we'd probably have to NOINLINE like mad). Once
we have BoxedRep, we can fix that part by offering unsafe primops that
allow lifted things to be read from/written to arrays of unlifted things
and vice versa, but I'm not sure that's so useful if we can't get a little
bit of unboxed (non-pointer) stuff in there too.

On Thu, Oct 8, 2020, 8:32 AM Ben Gamari  wrote:

> On October 7, 2020 2:02:53 PM EDT, David Feuer 
> wrote:
> >Yes, the bitmap structures used to encode the structure of stack
> >frames. These *look* like they might be reusable for mix-and-match
> >arrays in Haskell-land. If so, that could be a pretty cheap new
> >feature.
>
> Ahh, yes. In principle we could expose a new array type allowing the user
> to specify the pointer-ness of each entry. This wouldn't reuse any of the
> large bitmap machinery since these must be specified in the info table
> rather than the closure but the idea is similar. Implementation would
> require a new set of primops along with GC support but I suspect that none
> of this is too hard.
>
> All these being said, this may not be terribly efficient for garbage
> collection since the GC will need to walk over both pointers and
> non-pointers, trashing a good amount of cache in the process. Moreover, the
> array itself would be larger, due to the two bitmaps.
>
> I suspect most applications would be on the whole better served by a
> structure of arrays approach to achieve the same idea.  If you have
> multiple pointers and locality is a concern then you can even pack all of
> your pointers into a single array with some indexing magic and an
> unsafeCoerce.
>
> Cheers,
>
> - Ben
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: msys woes

[Sylvain Henry]

Could you share the contents of "missing-win32-tarballs" log file?

Thanks,
Sylvain

On 08/10/2020 16:51, Shayne Fletcher wrote:


Hi Phyx,

On Thu, Oct 8, 2020 at 9:10 AM Phyx > wrote:


> `./configure --enable-tarballs-autodownload` GHC build step on
Windows has been failing because repo.msys2.org


Afaik GHC doesn't rely ok repo.msys2.org 
for builds, only for mirroring. The primary url is haskell.org
 https://downloads.haskell.org/ghc/mingw/

So it's down time shouldn't have affected you (and works for me).


I should have mentioned... as always the situation is more complicated 😉
  - This is in the context of ghc-lib CI;
  - I don't have direct access to a windows box;

A procedure to reproduce it would be,
```
cd ghc
git fetch --tags && git checkout ghc-8.8.1-release
git submodule update --init --recursive
stack --stack-yaml hadrian/stack.yaml exec -- bash -c "./configure 
--enable-tarballs-autodownload"

```
Looking into the hadrian/stack/yaml on that tag, that's an 8.4.3 
resolverin case that's relevant.


Which url does it say is inaccessible?


Sadly, doesn't say
```
2020-10-08T14:41:46.4566084Z configure: loading site script 
/usr/local/etc/config.site
2020-10-08T14:41:46.5682736Z checking for gfind... no
2020-10-08T14:41:46.5700226Z checking for find... /usr/bin/find
2020-10-08T14:41:46.6978096Z checking for sort... /usr/bin/sort
2020-10-08T14:41:46.9667887Z checking for GHC Git commit id... inferred 
9c787d4d24f2b515934c8503ee2bbd7cfac4da20
2020-10-08T14:41:47.566Z checking for ghc... 
/c/Users/VssAdministrator/AppData/Local/Programs/stack/x86_64-windows/ghc-8.4.3/bin/ghc
2020-10-08T14:41:48.0265433Z checking version of ghc... 8.4.3
2020-10-08T14:41:49.2214006Z GHC path canonicalised to: 
c:/Users/VssAdministrator/AppData/Local/Programs/stack/x86_64-windows/ghc-8.4.3/bin/ghc
2020-10-08T14:41:49.8138282Z checking build system type... x86_64-w64-mingw32
2020-10-08T14:41:49.8152277Z checking host system type... x86_64-w64-mingw32
2020-10-08T14:41:49.8172851Z checking target system type... x86_64-w64-mingw32
2020-10-08T14:41:49.8197624Z Host platform inferred as: x86_64-unknown-mingw32
2020-10-08T14:41:50.1249990Z Target platform inferred as: x86_64-unknown-mingw32
2020-10-08T14:41:51.4787854Z GHC build  : x86_64-unknown-mingw32
2020-10-08T14:41:51.4788960Z GHC host   : x86_64-unknown-mingw32
2020-10-08T14:41:51.4790058Z GHC target : x86_64-unknown-mingw32
2020-10-08T14:41:51.4791914Z LLVM target: x86_64-unknown-windows
2020-10-08T14:41:51.6809080Z checking for path to top of build tree... 
D:/a/1/s/ghc
2020-10-08T14:41:51.7094005Z configure: Checking for Windows toolchain 
tarballs...
2020-10-08T14:41:53.0985704Z #=#=#
2020-10-08T14:41:53.0986841Z
2020-10-08T14:41:53.1745226Z ###
   26.4%
2020-10-08T14:41:53.1762650Z 
 100.0%
2020-10-08T14:41:53.4899178Z #=#=#
2020-10-08T14:41:53.5914142Z ##O#- #
2020-10-08T14:41:53.6973216Z ##O=#  #
2020-10-08T14:41:53.7049675Z #=#=-#  #
2020-10-08T14:41:53.7051106Z curl: (22) The requested URL returned error: 404 
Not Found
2020-10-08T14:41:53.7596464Z
2020-10-08T14:41:53.7600456Z ERROR: Download failed.
2020-10-08T14:41:53.7614446Z
2020-10-08T14:41:53.7615639Z Error fetching msys2 tarballs; see errors 
above. ```


--
Shayne Fletcher

___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: msys woes

[Shayne Fletcher]

On Thu, Oct 8, 2020 at 10:51 AM Shayne Fletcher <
shayne.fletcher...@gmail.com> wrote:

>
> Hi Phyx,
>
> On Thu, Oct 8, 2020 at 9:10 AM Phyx  wrote:
>
>> > `./configure --enable-tarballs-autodownload` GHC build step on Windows
>> has been failing because repo.msys2.org
>>
>> Afaik GHC doesn't rely ok repo.msys2.org for builds, only for mirroring.
>> The primary url is haskell.org https://downloads.haskell.org/ghc/mingw/
>>
>> So it's down time shouldn't have affected you (and works for me).
>>
>>
> [...]
>
> A procedure to reproduce it would be,
> ```
> cd ghc
> git fetch --tags && git checkout ghc-8.8.1-release
> git submodule update --init --recursive
> stack --stack-yaml hadrian/stack.yaml exec -- bash -c "./configure
> --enable-tarballs-autodownload"
> ```
>

Oops, obviously, there should be a boot in there first.
```
cd ghc
git fetch --tags && git checkout ghc-8.8.1-release
git submodule update --init --recursive
stack --stack-yaml hadrian/stack.yaml exec -- bash -c ./boot
stack --stack-yaml hadrian/stack.yaml exec -- bash -c "./configure
--enable-tarballs-autodownload"
```

-- 
Shayne Fletcher
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: msys woes

[Shayne Fletcher]

Hi Phyx,

On Thu, Oct 8, 2020 at 9:10 AM Phyx  wrote:

> > `./configure --enable-tarballs-autodownload` GHC build step on Windows
> has been failing because repo.msys2.org
>
> Afaik GHC doesn't rely ok repo.msys2.org for builds, only for mirroring.
> The primary url is haskell.org https://downloads.haskell.org/ghc/mingw/
>
> So it's down time shouldn't have affected you (and works for me).
>
>
I should have mentioned... as always the situation is more complicated 😉
  - This is in the context of ghc-lib CI;
  - I don't have direct access to a windows box;

A procedure to reproduce it would be,
```
cd ghc
git fetch --tags && git checkout ghc-8.8.1-release
git submodule update --init --recursive
stack --stack-yaml hadrian/stack.yaml exec -- bash -c "./configure
--enable-tarballs-autodownload"
```
Looking into the hadrian/stack/yaml on that tag, that's an 8.4.3 resolver
in case that's relevant.


> Which url does it say is inaccessible?
>

Sadly, doesn't say
```

2020-10-08T14:41:46.4566084Z configure: loading site script
/usr/local/etc/config.site
2020-10-08T14:41:46.5682736Z checking for gfind... no
2020-10-08T14:41:46.5700226Z checking for find... /usr/bin/find
2020-10-08T14:41:46.6978096Z checking for sort... /usr/bin/sort
2020-10-08T14:41:46.9667887Z checking for GHC Git commit id...
inferred 9c787d4d24f2b515934c8503ee2bbd7cfac4da20
2020-10-08T14:41:47.566Z checking for ghc...
/c/Users/VssAdministrator/AppData/Local/Programs/stack/x86_64-windows/ghc-8.4.3/bin/ghc
2020-10-08T14:41:48.0265433Z checking version of ghc... 8.4.3
2020-10-08T14:41:49.2214006Z GHC path canonicalised to:
c:/Users/VssAdministrator/AppData/Local/Programs/stack/x86_64-windows/ghc-8.4.3/bin/ghc
2020-10-08T14:41:49.8138282Z checking build system type... x86_64-w64-mingw32
2020-10-08T14:41:49.8152277Z checking host system type... x86_64-w64-mingw32
2020-10-08T14:41:49.8172851Z checking target system type... x86_64-w64-mingw32
2020-10-08T14:41:49.8197624Z Host platform inferred as: x86_64-unknown-mingw32
2020-10-08T14:41:50.1249990Z Target platform inferred as: x86_64-unknown-mingw32
2020-10-08T14:41:51.4787854Z GHC build  : x86_64-unknown-mingw32
2020-10-08T14:41:51.4788960Z GHC host   : x86_64-unknown-mingw32
2020-10-08T14:41:51.4790058Z GHC target : x86_64-unknown-mingw32
2020-10-08T14:41:51.4791914Z LLVM target: x86_64-unknown-windows
2020-10-08T14:41:51.6809080Z checking for path to top of build tree...
D:/a/1/s/ghc
2020-10-08T14:41:51.7094005Z configure: Checking for Windows toolchain
tarballs...
2020-10-08T14:41:53.0985704Z #=#=#
2020-10-08T14:41:53.0986841Z
2020-10-08T14:41:53.1745226Z ###
26.4%
2020-10-08T14:41:53.1762650Z

100.0%
2020-10-08T14:41:53.4899178Z #=#=#
2020-10-08T14:41:53.5914142Z ##O#- #
2020-10-08T14:41:53.6973216Z ##O=#  #
2020-10-08T14:41:53.7049675Z #=#=-#  #
2020-10-08T14:41:53.7051106Z curl: (22) The requested URL returned
error: 404 Not Found
2020-10-08T14:41:53.7596464Z
2020-10-08T14:41:53.7600456Z ERROR: Download failed.
2020-10-08T14:41:53.7614446Z

2020-10-08T14:41:53.7615639Z Error fetching msys2 tarballs; see errors
above. ```

-- 
Shayne Fletcher
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

"boilerplate" tool released

[Tseen She]

Hello all,

I have just released "boilerplate" to Hackage

  http://hackage.haskell.org/package/boilerplate

boilerplate is a command line tool that generates explicit boilerplate and
inserts it into your source code, enclosed in comments that text editors
can easily hide from view. Like the way IDEs do it for languages such as
Java, C++ and Go.

This is possible by using the ghc api (via my library and tool "hsinspect")
to parse type information from the current module and then use that to
drive template expansion using source code marker and a simple template
language for rules.

As a demo I have created rules for the most common stock classes (Eq, Ord,
Show), some language extensions (Functor, Foldable, Traversable) and some
popular classes (FFunctor, Aeson's FromJSON / ToJSON), and QuickCheck's Gen
(which is not a typeclass!).

My main hope for this little tool is that it will allow entry level teams
who are adopting Haskell to stick to the Haskell2010 standards without
having to learn advanced language extensions or get into type level
programming.

I also hope that more libraries can use this code to produce more portable
code! e.g. targeting whole program optimising compilers that only support
Haskell2010.

(thanks to SPJ for helping me get over the final hurdle by pointing me to
freeKiTyVarsTypeVars . extractHsTyRdrTyVars)

Best regards,
Tseen She
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: msys woes

[Phyx]

> `./configure --enable-tarballs-autodownload` GHC build step on Windows
has been failing because repo.msys2.org

Afaik GHC doesn't rely ok repo.msys2.org for builds, only for mirroring.
The primary url is haskell.org https://downloads.haskell.org/ghc/mingw/

So it's down time shouldn't have affected you (and works for me).

Which url does it say is inaccessible?

Sent from my Mobile

On Thu, Oct 8, 2020, 13:50 Shayne Fletcher 
wrote:

> For the last 5 days, the `./configure --enable-tarballs-autodownload` GHC
> build step on Windows has been failing because repo.msys2.org has been
> unavailable (https://github.com/msys2/MSYS2-packages/issues/2171).
> Apparently that's fixed today and the server can now be reached but still
> the configure fails now with 'curl: (22) The requested URL returned
> error: 404 Not Found' 😢 It would be great if someone with an understanding
> of what precisely `--enable-tarballs-autodownload` translates to was
> to file a bug-report with the right folks!
>
> --
> Shayne Fletcher
> ___
> ghc-devs mailing list
> ghc-devs@haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

msys woes

[Shayne Fletcher]

For the last 5 days, the `./configure --enable-tarballs-autodownload` GHC
build step on Windows has been failing because repo.msys2.org has been
unavailable (https://github.com/msys2/MSYS2-packages/issues/2171).
Apparently that's fixed today and the server can now be reached but still
the configure fails now with 'curl: (22) The requested URL returned error:
404 Not Found' 😢 It would be great if someone with an understanding of
what precisely `--enable-tarballs-autodownload` translates to was to file a
bug-report with the right folks!

-- 
Shayne Fletcher
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Array heap objects

[Ben Gamari]

On October 7, 2020 2:02:53 PM EDT, David Feuer  wrote:
>Yes, the bitmap structures used to encode the structure of stack
>frames. These *look* like they might be reusable for mix-and-match
>arrays in Haskell-land. If so, that could be a pretty cheap new
>feature.

Ahh, yes. In principle we could expose a new array type allowing the user to 
specify the pointer-ness of each entry. This wouldn't reuse any of the large 
bitmap machinery since these must be specified in the info table rather than 
the closure but the idea is similar. Implementation would require a new set of 
primops along with GC support but I suspect that none of this is too hard.

All these being said, this may not be terribly efficient for garbage collection 
since the GC will need to walk over both pointers and non-pointers, trashing a 
good amount of cache in the process. Moreover, the array itself would be 
larger, due to the two bitmaps.

I suspect most applications would be on the whole better served by a structure 
of arrays approach to achieve the same idea.  If you have multiple pointers and 
locality is a concern then you can even pack all of your pointers into a single 
array with some indexing magic and an unsafeCoerce.

Cheers,

- Ben
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs