Re: Fwd: Release policies

[Michael Snoyman]

On Thu, Dec 14, 2017 at 12:27 PM, Boespflug, Mathieu  wrote:

[snip]

* Or a middle ground: make feature freeze a thing. Meaning that for a
> couple of months before a major GHC release, the major new Cabal isn't
> technically released yet, but like GHC itself within this period, it's
> pretty staid, so not so much a moving target, and something downstream
> tooling authors can possibly adapt to even without any grace period on
> new metadata features. This assumes that the 2 months of feature
> freeze are enough time for downstream tooling. Thoughts from any of
> those maintainers?
>
>
Short answer: if there's a clear idea in advance of when this feature
freeze is going to happen, I think we can coordinate releases of downstream
tooling (Stack being the most important, but stackage-curator playing in as
well) so that 2 months is sufficient. I'll talk with the rest of the Stack
team to see if there are any concerns.

Longer answer: Stack intentionally avoids depending on the internals of
Cabal wherever possible. Instead of calling library functions directly from
within Haskell code to perform builds, for example, it interacts with the
Setup.hs files over their command line interface.[1] This has two results:

* Stack can usually start using new GHC/Cabal versions without a new Stack
release, since it's just shelling out for the actual build
* There's not usually very much code churn needed in Stack to upgrade to a
newer Cabal release

This past release was an exception because of all of the changes that
landed, both the new cabal grammar to support the ^>= operator (making the
old parser incapable of lossily parsing new files) and API changes (I think
mostly around Backpack, though there was some code cleanup as well). In
particular, the main interface we need from Cabal—the package description
data types and parser—changed significantly enough that it took significant
effort to upgrade. There were also new features added (like sub libraries
and foreign libraries) that weren't immediately supported by the old Stack
version, and had to be manually added in.

Tying this up: generally upgrading to a new Cabal release should be fine,
and the only concern I'd have is fitting it into a release schedule with
Stack. The complications that could slow that down are:

* Changes to the command line interface that Stack uses (hopefully those
are exceedingly rare)
* Major overhauls to the Stack-facing API

Michael

[1] This allows for more reproducible builds of older snapshots, insuring
that the exact same Cabal library is performing the builds
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: GHC-API unloading from external interpreter

[Moritz Angermann]

Hi,

I believe you are looking for the `Message` GADT in `GHCi.Message`

```
-- -
-- The RPC protocol between GHC and the interactive server

-- | A @Message a@ is a message that returns a value of type @a@.
-- These are requests sent from GHC to the server.
data Message a where
```
maybe specifically the `UnloadObj` message.

Cheers,
 Moritz


> On Dec 15, 2017, at 1:46 PM, Németh Boldizsár  wrote:
> 
> Dear GHC Developers,
> 
> I'm using the GHC API to load Haskell modules and extract their syntax tree. 
> I need to generate code where TH is used (by default I do in-memory linking). 
> I've run into some trouble with external interpreter. When 
> -fexternal-interpreter is NOT set, I use Linker.unload to remove bytecode 
> from the global linker before reloading modules. However it is not applicable 
> when external interpreter is used and I could not find corresponding 
> functionality in the GHCi module (where communication with external 
> interpreter is implemented).
> 
> I've recently created 2 tickets related to this question: 
> https://ghc.haskell.org/trac/ghc/ticket/14576, 
> https://ghc.haskell.org/trac/ghc/ticket/14577
> 
> Can someone help me with this problem?
> 
> Best Regards,
> Boldizsár Németh
> 
> ___
> ghc-devs mailing list
> ghc-devs@haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

GHC-API unloading from external interpreter

[Németh Boldizsár]

Dear GHC Developers,

I'm using the GHC API to load Haskell modules and extract their syntax 
tree. I need to generate code where TH is used (by default I do 
in-memory linking). I've run into some trouble with external 
interpreter. When -fexternal-interpreter is NOT set, I use Linker.unload 
to remove bytecode from the global linker before reloading modules. 
However it is not applicable when external interpreter is used and I 
could not find corresponding functionality in the GHCi module (where 
communication with external interpreter is implemented).


I've recently created 2 tickets related to this question: 
https://ghc.haskell.org/trac/ghc/ticket/14576, 
https://ghc.haskell.org/trac/ghc/ticket/14577


Can someone help me with this problem?

Best Regards,
Boldizsár Németh

___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Fwd: Release policies

[Boespflug, Mathieu]

>> * But actually if we look at their respective release notes, GHC 8.2.1 was
>> relased in July 2017, even though the Cabal website claims that
>> Cabal-2.0.0.2 was released in August 2017 (see
>> https://www.haskell.org/cabal/download.html). So it looks like GHC didn't
>> just not give enough lead time about an upstream dependency it shipped
>> with, it shipped with an unreleased version of Cabal!
>
> Perhaps this is true and I admit I wasn't happy about releasing the compiler
> without a Cabal release. However, there was no small amount of pressure to
> push forward nevertheless as the release was already quite late and the
> expectation was a Cabal release would be coming shortly after the GHC
> release. Coordination issues like this are a major reason why I think it
> would be better if GHC were more decoupled from its dependencies'
> upstreams.

I have the same sentiment. Do you think this is feasible in the case
of Cabal? Even if say something like Backpack shows up all over again?
If so, are there concrete changes that could be made to support the
following workflow:

* upstreams develop their respective libraries independently of GHC
using their own testing.
* If they want GHC to ship a newer version, they create a Diff. As
Manuel proposed in a separate thread, this must be before feature
freeze, unless...
* ... a critical issue is found in the upstream release, in which case
upstream cuts a new release, and submits a Diff again.
* GHC always has the option to back out an offending upgrade, and
revert to a known good version. In fact it should preemptively do so
while waiting for a new release of upstream.
* In general, GHC does not track git commits of upstream dependencies
in an unknown state of quality, but tracks vetted and tested releases
instead.

>> * GHC should never under any circumstance ship with an unreleased version
>> of any independently maintained dependency. Cabal is one such dependency.
>> This should hold true for anything else. We could just add that policy to
>> the Release Policy.
>>
> We can adopt this as a policy, but doing so very well may mean that GHC
> will be subject to schedule slips beyond its control. We can hope that
> upstream maintainers will be responsive, but there is little we can do
> when they are not.

Why not? If GHC only ever tracks upstream releases (as I think it
should), not git commits in unknown state, then we don't need upstream
maintainer responsiveness. Because at any point in time, all GHC
dependencies are already released. If GHC should ship with a newer
version of a dependency, the onus is on the upstream maintainer to
submit a Diff asking GHC to move to the latest version. Are there good
reasons for GHC to track patches not upstreamed and released?

>> * Stronger still, GHC should not switch to a new major release of a
>> dependency at any time during feature freeze ahead of a release. E.g. if
>> Cabal-3.0.0 ships before feature freeze for GHC-9.6, then maybe it's fair
>> game to include in GHC. But not if Cabal-3.0.0 hasn't shipped yet.
>>
> Yes, this I agree with. I think we can be more accomodating of minor
> bumps to fix bugs which may come to light during the freeze, but major
> releases should be avoided.

Agreed.

>> * The 3-release backwards compat rule should apply in all circumstances.
>> That means major version bumps of any library GHC ships with, including
>> base, should not imply any breaking change in the API's of any such library.
>>
> I'm not sure I follow what you are suggesting here.

Nothing new: just that the 3-release policy doesn't just apply to
base, but also anything else that happens to ship with GHC (including
Cabal). Perhaps that already the policy?

>> * GHC does have control over reinstallable packages (like text and
>> bytestring): GHC need not ship with the latest versions of these, if indeed
>> they introduce breaking changes that would contravene the 3-release policy.
>>
>> * Note: today, users are effectively tied to whatever version of the
>> packages ships with GHC (i.e. the "reinstallable" bit is problematic today
>> for various technical reasons). That's why a breaking change in bytestring
>> is technically a breaking change in GHC.
>>
> I don't follow: Only a small fraction of packages, namely those that
> explicitly link against the `ghc` library, are tied. Can you clarify
> what technical reasons you are referring to here?

Builds often fail for strange reasons when both bytestring-0.10.2 and
bytestring-0.10.1 are in scope. Some libraries in a build plan pick up
one version where some pick up another. The situation here might well
be better than it used to be, but at this point in time Stackage works
hard to ensure that in any given package set, there is *exactly one*
version of any package. That's why Stackage aligns versions of core
packages to whatever ships with the GHC version the package set is
based on.

So in this sense, AFAIK a bug in bytestring can't be worked around by
reinstalling 

Re: Fwd: Release policies

[Boespflug, Mathieu]

>> On 14 December 2017 at 00:19, Gershom B  wrote:
>>>
>>> Mathieu:
>>>
>>> I think the points about better tooling for documenting the correct
>>> claims in the release process are well taken. Updating the release
>>> notes manually leaves way too much room for error.
>>>
> Indeed, the release notes have historically been a massive headache.
> Happily, I wrote a bit of the necessary tooling to fix this a few weeks
> ago [1].
>
> [1] e4dc2cd51902a8cd83476f861cf52996e5adf157

Very cool!
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Fwd: Release policies

[Ben Gamari]

"Boespflug, Mathieu"  writes:

> -- Forwarded message --
> From: Boespflug, Mathieu 
> Date: 13 December 2017 at 23:03
> Subject: Re: Release policies
> To: Simon Peyton Jones 
> Cc: ghc-devops-gr...@haskell.org
>
>
> [replying to ghc-devops-group@, which I assume based on your email's
> content is the mailing list you intended.]
>
> Hi Simon,
>
> feedback from downstream consumers of Cabal metadata (e.g. build tool
> authors) will be particularly useful for the discussion here. Here are my
> thoughts as a bystander.
>
> It's worth trying to identify what problems came up during the integer-gmp
> incident in Trac #14558:
>
> * GHC 8.2.1 shipped with integer-gmp-1.0.1.0 but the release notes said
> otherwise.
>
> * GHC 8.2.1 shipped with Cabal-2.0.0.2, but specifically claimed in the
> release notes that cabal-install-1.24 (and by implication any other build
> tool based on Cabal-the-library version 1.24) was supported: "GHC 8.2 only
> works with cabal-install version 1.24 or later. Please upgrade if you have
> an older version of cabal-install."
> 
> * GHC 8.2.2 also claimed Cabal-1.24 support.
>
> * GHC 8.2.1 was released in July 2017 with Cabal-2.0.0.2, a brand new major
> release with breaking changes to the metadata format, without much lead
> time for downstream tooling authors (like Stack) to adapt.
> 
> * But actually if we look at their respective release notes, GHC 8.2.1 was
> relased in July 2017, even though the Cabal website claims that
> Cabal-2.0.0.2 was released in August 2017 (see
> https://www.haskell.org/cabal/download.html). So it looks like GHC didn't
> just not give enough lead time about an upstream dependency it shipped
> with, it shipped with an unreleased version of Cabal!

Perhaps this is true and I admit I wasn't happy about releasing the compiler
without a Cabal release. However, there was no small amount of pressure to
push forward nevertheless as the release was already quite late and the
expectation was a Cabal release would be coming shortly after the GHC
release. Coordination issues like this are a major reason why I think it
would be better if GHC were more decoupled from its dependencies'
upstreams.

I think the approach that we discussed at ICFP, where library
authors must upstream their version bumps before the freeze, just like
any library, is perhaps one way forward although I suspect it exceptions
will need to be made.

> * Libraries that ship with GHC are usually also uploaded to Hackage, to
> make the documentation easily accessible, but integer-gmp-1.0.1.0 was not
> uploaded to Hackage until 4 months after the release.
> 
> * The metadata for integer-gmp-1.0.1.0 as uploaded to Hackage differed from
> the metadata that was actually in the source tarball of GHC-8.2.1 and
> GHC-8.2.2.
>
> * The metadata for integer-gmp-1.0.1.0 as uploaded to Hackage included
> Cabal-2.0 specific syntactic sugar, making the metadata unreadable using
> any tooling that did not link against the Cabal-2.0.0.2 library (or any
> later version).
>
> * It so happened that one particular version of one particular downstream
> build tool, Stack, had a bug, compounding the bad effects of the previous
> point. But a new release has now been made, and in any case that's not a
> problem for GHC to solve. So let's keep that out of the discussion here.
>
> So I suggest we discuss ways to eliminate or reduce the likelihood of any
> of the above problems from occurring again. Here are some ideas:
>
> * GHC should never under any circumstance ship with an unreleased version
> of any independently maintained dependency. Cabal is one such dependency.
> This should hold true for anything else. We could just add that policy to
> the Release Policy.
>
We can adopt this as a policy, but doing so very well may mean that GHC
will be subject to schedule slips beyond its control. We can hope that
upstream maintainers will be responsive, but there is little we can do
when they are not. Of course, if we adopt the policy of disallowing all
but essentially core library bumps during the freeze then we may be able
to mitigate this.

> * Stronger still, GHC should not switch to a new major release of a
> dependency at any time during feature freeze ahead of a release. E.g. if
> Cabal-3.0.0 ships before feature freeze for GHC-9.6, then maybe it's fair
> game to include in GHC. But not if Cabal-3.0.0 hasn't shipped yet.
>
Yes, this I agree with. I think we can be more accomodating of minor
bumps to fix bugs which may come to light during the freeze, but major
releases should be avoided.

> * The 3-release backwards compat rule should apply in all circumstances.
> That means major version bumps of any library GHC ships with, including
> base, should not imply any breaking change in the API's of any such library.
>
I'm not sure I follow what you are suggesting here.

> * GHC does have control over reinstallable packages (like text and
> bytestring): GHC 

Re: Fwd: Release policies

[Ben Gamari]

"Boespflug, Mathieu"  writes:

> Hi Gerhom,
>
> On 14 December 2017 at 00:19, Gershom B  wrote:
>>
>> Mathieu:
>>
>> I think the points about better tooling for documenting the correct
>> claims in the release process are well taken. Updating the release
>> notes manually leaves way too much room for error.
>>
Indeed, the release notes have historically been a massive headache.
Happily, I wrote a bit of the necessary tooling to fix this a few weeks
ago [1].


[1] e4dc2cd51902a8cd83476f861cf52996e5adf157

>> However, I think you are incorrect that GHC 8.2.1 and 8.2.2 did not
>> have cabal-install 1.24 support. They did. it works with them.
>
> They did, and indeed Stack too worked just fine with them, but that
> was assuming that integer-gmp-1.0.1.0 really was what was shipped in
> the tarballs, not what it was on Hackage (until it got recently
> revised). I don't know which version of integer-gmp-1.0.1.0 was the
> intended one. They both have the same version number and neither seems
> more authoritative than the other to me. Had the Hackage one been the
> one that shipped, then I'm not sure that cabal-install-1.24 would have
> worked. Stack broke the moment what was on Hackage and what was in GHC
> bindists did not line up anymore. And with release notes mentioning
> incorrect version numbers, harder still to tell.
>
I agree that the cabal files uploaded to Hackage should match what is
released or, if not, there should be a very good reason for divergence.

[snip]


>
>> The general motivation of making a "feature freeze" more of a "freeze
>> all the moving parts, really" I do agree with. Having a real freeze is
>> part of a better release process, and it should allow all the
>> downstream consumers of everything more time to really catch up. This
>> is just one instance of that need.
>
> Agreed.
>
Also agreed.

Cheers,

- Ben



signature.asc
Description: PGP signature
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Fwd: Release policies

[Boespflug, Mathieu]

Hi Gershom,

thanks for the extra input. So we've confirmed two facts:

* GHC (intended to) ship with only Cabal-2.0 support, but there was a
mistake in the release notes so this was unclear to downstream tooling
authors.
* Cabal-2.0 was released anywhere between slightly *after* and
*exactly at the same as* GHC, despite GHC itself shipping with
Cabal-2.0.

I'm not too concerned by the first point: so long as Cabal-X does not
introduce breaking changes, the fact that GHC-Y ultimately shipped
with Cabal-X shouldn't be a problem. And this kind of bug in the
release notes should go away provided more automation.

The second one is more interesting. It is, as you point out, a product
of GHC and Cabal being intimately linked and co-developed to a large
extent. This leads to a simultaneous release that poses a concrete
problem:

* if new Cabal versions are used immediately in GHC, then that gives
no time at all ahead of a GHC release for downstream tooling authors
to adapt, because Cabal is, up until the point of the GHC release, a
moving target.

Three possible solutions:

* Provided no API breaking changes in Cabal, if no metadata that ships
with GHC uses new Cabal features for some period of time before
release, then the problem goes away.
* Or something close to what Manuel proposed in another thread: ship
in GHC-X+1 the Cabal version that was co-developed during the
development cycle of GHC-X.
* Or a middle ground: make feature freeze a thing. Meaning that for a
couple of months before a major GHC release, the major new Cabal isn't
technically released yet, but like GHC itself within this period, it's
pretty staid, so not so much a moving target, and something downstream
tooling authors can possibly adapt to even without any grace period on
new metadata features. This assumes that the 2 months of feature
freeze are enough time for downstream tooling. Thoughts from any of
those maintainers?


On 14 December 2017 at 01:27, Gershom B  wrote:
> On Wed, Dec 13, 2017 at 7:06 PM, Boespflug, Mathieu  wrote:
>>
>> But crucially, what *is* the policy around Cabal versions? This
>> comment, https://ghc.haskell.org/trac/ghc/ticket/14558#comment:23
>> claims "if Stack doesn't support the version of Cabal that ships with
>> a certain version of GHC, it shouldn't claim that it supports that
>> version of GHC. The same applies to cabal-install". Is any build tool
>> linked against Cabal-X by definition "not a supported configuration"
>> by GHC-Z if it ships with Cabal-Y such that X < Y?
>
> My understanding is that this is the general thought, yes. In fact,
> I've been told that even though cabal-install 1.24 did end up working
> with the GHC 8.2.x series, the release notes, which were not updated
> properly, actually _were supposed_ to say cabal-install 2.0.0.0 was
> what was supported there. I believe future cabal-installs will warn
> when used with a ghc with a newer Cabal-lib than they were built
> against...
>
>> Right. But switching from Cabal-2 to Cabal-3 (a hypothetical at this
>> point) sounds like a whole new set of features transitively just made
>> it into the compiler. Is that something we're happy to happen during
>> feature freeze?
>
> Right. After freeze, the compiler itself shouldn't switch from Cabal-2
> to Cabal-3. But I would imagine rather that the Cabal-3 tree and the
> compiler tree would be updated in tandem, and then the "freeze" would
> sort of apply to both in tandem as well. So there wouldn't be big
> changes after the freeze, but nor would the compiler be coupled to a
> _released_ lib. Rather, they would develop together, freeze together,
> and release together.
>
>> I don't disagree. But then we'd need to abandon any notion that
>> versions of packages on Hackage and versions of packages in the GHC
>> release tarball always match up. Might even be worth calling that out
>> explicitly in the policy.
>
> Not exactly. The tarball of the package on hackage should match the
> release tarball. Revisions don't change the tarball. They just add
> additional metadata to the index as well that cabal-install knows how
> to use in conjunction with the tarball:
> https://github.com/haskell-infra/hackage-trustees/blob/master/revisions-information.md#what-are-revisions
>
> --Gershom
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs