Re: Status of Harbormaster
Erik de Castro Lopo writes: > Ben Gamari wrote: > >> It's a bit unclear how far we should extend test coverage. In the future >> I think I will at very least add an i386 Ubuntu environment, but we >> could go farther still. For instance these platforms immediately come to >> mind, >> >> * x86_64 FreeBSD >> * x86_64 Solaris >> * ARM Linux (although this could be quite tricky given the speed of >>these machines) >> * AArch64 Linux > > That would be awesome if we could get access to a decent (by which I mean > server grade, with at least 4 cores and 8 Gig of RAM). > I've wondered how one of the Scaleway ARMs would fare. Their specs are seem to be comparable to the Odroid XU4 which I use for my own builds, which barely passes. Perhaps dedicating some of the 50GB SSD to swap would help. > The other option for ARM/Linux and AArch64/Linux is cross-compile builds. > Just building GHC as a cross-compiler for these targets would shave out > a lot of bugs. Let me know if you're interested. Its really pretty easy > to set up on a Debian or Ubuntu system. > I am absolutely interested but I'd first like to stabilize what we have. Let's chat about cross-compilation once that happens. Don't hesitate to remind me in a few weeks. Cheers, - Ben signature.asc Description: PGP signature ___ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Re: Status of Harbormaster
On 10/ 4/16 09:18 AM, Erik de Castro Lopo wrote: * AArch64 Linux That would be awesome if we could get access to a decent (by which I mean server grade, with at least 4 cores and 8 Gig of RAM). Just ask for account on GNU GCC Compile Farm. They do have X-gene V1 machine in the farm, pretty powerful box especially in comparison with my pandaboard. I for example keep running POWER7 buildbot for GHC there. Cheers, Karel ___ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Re: Status of Harbormaster
Ben Gamari wrote: > It's a bit unclear how far we should extend test coverage. In the future > I think I will at very least add an i386 Ubuntu environment, but we > could go farther still. For instance these platforms immediately come to > mind, > > * x86_64 FreeBSD > * x86_64 Solaris > * ARM Linux (although this could be quite tricky given the speed of >these machines) > * AArch64 Linux That would be awesome if we could get access to a decent (by which I mean server grade, with at least 4 cores and 8 Gig of RAM). The other option for ARM/Linux and AArch64/Linux is cross-compile builds. Just building GHC as a cross-compiler for these targets would shave out a lot of bugs. Let me know if you're interested. Its really pretty easy to set up on a Debian or Ubuntu system. Erik -- -- Erik de Castro Lopo http://www.mega-nerd.com/ ___ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Status of Harbormaster
Hello everyone, Over the last few weeks I have been gradually pushing away at increasing our Harbormaster coverage. I'm happy to report that Harbormaster should now test commits on, * x86_64 Ubuntu Linux * x86_64 Mac OS X Sierra * x86_64 Windows (although the bugs are still being worked out here) Differentials are tested on, * x86_64 Ubuntu Linux * x86_64 Windows For those of you following along at home I've roughly documented the configuration on the Phabricator Wiki [1]. One open question is whether we want to enable Differential building on the OS X box. The security implications of allowing essentially anonymous users to build and run untrusted code in our own CI environment are already quite sticky; to enable Differential building on the OS X box would mean that we would be running untrusted code on someone else's hardware, which seems like it may be a step too far. It would be nice to find a way to extend Differential builds to other platforms in the future, however, so we can ensure that we catch bad patches before they even make it to the tree. It's a bit unclear how far we should extend test coverage. In the future I think I will at very least add an i386 Ubuntu environment, but we could go farther still. For instance these platforms immediately come to mind, * x86_64 FreeBSD * x86_64 Solaris * ARM Linux (although this could be quite tricky given the speed of these machines) * AArch64 Linux There is certainly a maintenance and complexity tradeoff that comes with extending our coverage like this, however, so it's quite unclear where the right compromise lies. I'd love to hear what you think. Happy hacking, - Ben [1] https://phabricator.haskell.org/w/ghc_harbormaster/ signature.asc Description: PGP signature ___ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Re: Action Required: Status of Harbormaster for Differentials
Richard Eisenberg writes: > Hi Ben, > > Thanks for tackling this difficult issue -- I know you've been looking > for a better solution. > > As I understand it, anyone will now need to add a public key to Phab > in order to upload patches. This (as you admit) will increase the > barriers to contributions. Are the instructions for how a new > contributor should submit patches (without a public key and therefore > without CI) posted very visibly somewhere? > I have some edits of the Phabricator page [1] in the Wiki on-going as we speak (although mpickering deserves credit for doing the original edits working in mentions of the SSH key requirement). Cheers, - Ben [1] https://ghc.haskell.org/trac/ghc/wiki/Phabricator signature.asc Description: PGP signature ___ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Re: Action Required: Status of Harbormaster for Differentials
Hi Ben, Thanks for tackling this difficult issue -- I know you've been looking for a better solution. As I understand it, anyone will now need to add a public key to Phab in order to upload patches. This (as you admit) will increase the barriers to contributions. Are the instructions for how a new contributor should submit patches (without a public key and therefore without CI) posted very visibly somewhere? Thanks, Richard > On Aug 26, 2016, at 9:38 AM, Ben Gamari wrote: > > tl;dr. I am working on restoring Harbormaster support for Differentials. > Towards this end, GHC's Phabricator instance will soon require > that you add an SSH public key to your account in order to upload > patches with `arc diff`. > > > Hello everyone, > > As you are no doubt aware Harbormaster has not been building > Differentials for a very long time now. I am currently looking at > options for restoring this service but it appears that doing so will > require a bit of action on the part of our contributors. > > Specically, you will need to add an SSH key to Phabricator. You can do > so by navigating to the Phabricator Settings [1] page, selecting the > "Personal Account Settings" option, and selecting "SSH Public Keys" from > the list on the left. There you can click on the "SSH Key Actions" > menu on the top-right corner and click on the "Upload Public Key" > item. Here you can specify a key name and paste the key contents > (e.g. the contents of $HOME/.ssh/id_rsa.pub). > > The reason for this new requirement is that `arc diff` will soon push > submitted diffs to a "staging area", a Git repository managed by Phabricator > containing a branch for each submitted Differential. This will allow more > reliable Harbormaster builds and merges as we will no longer need to > rely on Phabricator correctly applying Differential patches (which has > historically been problematic, especially for Differentials where the > base commit is unavailable). > > If there are no objections, I would like to enable the staging area for > the rGHC repository in three days, on Monday, 29 August 2016. After > this date `arc diff` will try to push submitted differentials to the > staging area, which will require a public key. Note, however, that > pushing to the staging area can always be disabled with `arc diff`'s > `--skip-staging` flag (although we won't be able to run CI builds on > Differentials submitted in this way). > > To be clear: I would have liked to provide an option that did not > require key-based authentication, but sadly neither Phabricator nor > Gitolite provide any such option. I apologize for the added friction > that this new requirement imposes. If enough people feel strongly that > this is too onerous then I'm happy to entertain alternative solutions. > > Cheers, > > - Ben > > > [1] https://phabricator.haskell.org/settings/ > ___ > ghc-devs mailing list > ghc-devs@haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs ___ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Re: Action Required: Status of Harbormaster for Differentials
Ben Gamari writes: > tl;dr. I am working on restoring Harbormaster support for Differentials. >Towards this end, GHC's Phabricator instance will soon require >that you add an SSH public key to your account in order to upload >patches with `arc diff`. > For the record, I just flipped the switch on this and enabled building of Differentials with Harbormaster. So far things look good but please let me know if you encounter any problems. Thanks! Cheers, - Ben signature.asc Description: PGP signature ___ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Re: Action Required: Status of Harbormaster for Differentials
Ben Gamari writes: > Joachim Breitner writes: > >> Hi, >> ... > Ahh, sure. Right, it's expected that ssh requires a key. Indeed http is > more fitting for your bot. > >> I’ll wait for the CDN to get working. >> > I'll let you know when things have been sorted. > Well this is unfortunate: it looks like GHC might be too big [1] for Phabricator's HTTP git hosting logic. Unfortunately it's hard to claim this with much confidence since the endpoint provides no error message beyond a 500 status code. I'm not really sure what to advise here; I'll try to raise a flag with the Phacility people but even if I could confirm that T4369 is the culprit, it's a bug that still unfixed after more than two years and I don't hold out much hope that there will be a fix in the near-term. They suggest that affected users instead use SSH. Sadly this would mean that you would need to provide a key to your bot. It is certainly not a very satisfying answer, but I'm afraid it is likely the best we'll be able to do for a while. Cheers, - Ben [1] https://secure.phabricator.com/T4369 signature.asc Description: PGP signature ___ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Re: Action Required: Status of Harbormaster for Differentials
Joachim Breitner writes: > Hi, > > Am Montag, den 29.08.2016, 10:03 -0400 schrieb Ben Gamari: >> >> Regardless, you should be able to clone the repo via ssh. > > Permission denied (publickey,keyboard-interactive). > fatal: Could not read from remote repository. > > Please make sure you have the correct access rights > and the repository exists. > > (This is without using an ssh key added to Phab – after all, I need a > build service to poll that, and don’t want to hand out keys to that.) > Ahh, sure. Right, it's expected that ssh requires a key. Indeed http is more fitting for your bot. > I’ll wait for the CDN to get working. > I'll let you know when things have been sorted. >> > Ok, I think I will have to add custom code to take care of DRs in >> > gipeda, but it is probably worth it. >> > >> This code won't strictly speaking be in gipeda, right? Rather, it would >> merely be a bit more logic in the build script, no? > > Both. > > The build script needs to know that there are two repositories to poll from > > Gipeda needs to produce a section, like the one for branches, for DRs, > which have slightly different semantics (in particular, they have an > explicit base commit that comparisons should be made against, which is > not the case for git branches). > Ahhh, I see. A nice point. Cheers, - Ben signature.asc Description: PGP signature ___ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Re: Action Required: Status of Harbormaster for Differentials
Hi, Am Montag, den 29.08.2016, 10:03 -0400 schrieb Ben Gamari: > > > it does not seem to be world-readable: > > > > $ git fetch phab > > fatal: unable to access 'http://phabricator.haskell.org/diffusion/G > > HCDIFF/GHC-Differentials.git/': The requested URL returned error: > > 500 > > > Actually it looks like this was a configuration issue which broke > Phabricator's HTTP git hosting. I believe I have fixed this but now > seem to be fighting our CDN which I'll need help from the > infrastructure > guys for. > > Regardless, you should be able to clone the repo via ssh. Permission denied (publickey,keyboard-interactive). fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. (This is without using an ssh key added to Phab – after all, I need a build service to poll that, and don’t want to hand out keys to that.) I’ll wait for the CDN to get working. > > > The only thing you'll find in that repository currently is tags (not > > > branches as I stated earlier; sorry!) with names of the form > > > phabricator/base/1234 and phabricator/diff/1234. These contain the base > > > and head commits of revision 1234, respectively. Note that 1234 does not > > > identify a differential; rather it is a revision of a differential (e.g. > > > the number that you see in the "ID" column of the history table fn a > > > > > > > > ifferential in the web interface). > > > > Ok, I think I will have to add custom code to take care of DRs in > > gipeda, but it is probably worth it. > > > This code won't strictly speaking be in gipeda, right? Rather, it would > merely be a bit more logic in the build script, no? Both. The build script needs to know that there are two repositories to poll from Gipeda needs to produce a section, like the one for branches, for DRs, which have slightly different semantics (in particular, they have an explicit base commit that comparisons should be made against, which is not the case for git branches). Hmm. Links to git diffs would also have to go to a different place. Annoying complexity there. Maybe a separate gipeda instance is simpler (but would get in the way of comparing commits on Phab with commits on the main repo, and duplicate building work). I’ll have to think more about how this can be done in a viable way. Greetings, Joachim -- -- Joachim “nomeata” Breitner m...@joachim-breitner.de • https://www.joachim-breitner.de/ XMPP: nome...@joachim-breitner.de • OpenPGP-Key: 0xF0FBF51F Debian Developer: nome...@debian.org signature.asc Description: This is a digitally signed message part ___ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Re: Action Required: Status of Harbormaster for Differentials
Joachim Breitner writes: > [ Unknown signature status ] > Hi, > > Am Freitag, den 26.08.2016, 17:39 -0400 schrieb Ben Gamari: >> Indeed, it's available at [1]. > > it does not seem to be world-readable: > > $ git fetch phab > fatal: unable to access > 'http://phabricator.haskell.org/diffusion/GHCDIFF/GHC-Differentials.git/': > The requested URL returned error: 500 > Actually it looks like this was a configuration issue which broke Phabricator's HTTP git hosting. I believe I have fixed this but now seem to be fighting our CDN which I'll need help from the infrastructure guys for. Regardless, you should be able to clone the repo via ssh. >> The only thing you'll find in that repository currently is tags (not >> branches as I stated earlier; sorry!) with names of the form >> phabricator/base/1234 and phabricator/diff/1234. These contain the base >> and head commits of revision 1234, respectively. Note that 1234 does not >> identify a differential; rather it is a revision of a differential (e.g. >> the number that you see in the "ID" column of the history table fn a >> > ifferential in the web interface). > > Ok, I think I will have to add custom code to take care of DRs in > gipeda, but it is probably worth it. > This code won't strictly speaking be in gipeda, right? Rather, it would merely be a bit more logic in the build script, no? > Can I assume that phabricator/base/1234 is always (or at least usually) > a commit on master, or some other branch in the ghc repository? > Often it will be. However, I have in the past often opened "chains" of Differentials where the HEAD of one is the base commit of the next. Naturally in this case the base commit will not be in master. Cheers, - Ben signature.asc Description: PGP signature ___ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Re: Action Required: Status of Harbormaster for Differentials
Phyx writes: > Hi Ben, > > Out of curiosity which platforms will be getting harbormaster support? Is > this only Linux x86_64? > > Any plans to expand this in the future to all Tier 1 platforms? > Indeed I would like to get to this point. At the moment I am focusing on Linux x86_64 and Darwin x86_64. I may try to bring up an ARM as well. The other platforms would be subject to machine availability. Cheers, - Ben signature.asc Description: PGP signature ___ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Re: Action Required: Status of Harbormaster for Differentials
Hi, Am Freitag, den 26.08.2016, 17:39 -0400 schrieb Ben Gamari: > Indeed, it's available at [1]. it does not seem to be world-readable: $ git fetch phab fatal: unable to access 'http://phabricator.haskell.org/diffusion/GHCDIFF/GHC-Differentials.git/': The requested URL returned error: 500 > The only thing you'll find in that repository currently is tags (not > branches as I stated earlier; sorry!) with names of the form > phabricator/base/1234 and phabricator/diff/1234. These contain the base > and head commits of revision 1234, respectively. Note that 1234 does not > identify a differential; rather it is a revision of a differential (e.g. > the number that you see in the "ID" column of the history table fn a > > ifferential in the web interface). Ok, I think I will have to add custom code to take care of DRs in gipeda, but it is probably worth it. Can I assume that phabricator/base/1234 is always (or at least usually) a commit on master, or some other branch in the ghc repository? Greetings, Joachim -- Joachim “nomeata” Breitner m...@joachim-breitner.de • https://www.joachim-breitner.de/ XMPP: nome...@joachim-breitner.de • OpenPGP-Key: 0xF0FBF51F Debian Developer: nome...@debian.org signature.asc Description: This is a digitally signed message part ___ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Re: Action Required: Status of Harbormaster for Differentials
Hi Ben, Out of curiosity which platforms will be getting harbormaster support? Is this only Linux x86_64? Any plans to expand this in the future to all Tier 1 platforms? Cheers, Tamar On Fri, Aug 26, 2016, 22:39 Ben Gamari wrote: > Joachim Breitner writes: > > > [ Unknown signature status ] > > Hi, > > > > Am Freitag, den 26.08.2016, 09:38 -0400 schrieb Ben Gamari: > >> The reason for this new requirement is that `arc diff` will soon push > >> submitted diffs to a "staging area", a Git repository managed by > >> Phabricator containing a branch for each submitted Differential. > > > > this is great, because it means that we can run performance builds via > > perf.haskell.org on all DRs automatically! > > > > Will the staging repository be publicly available? > > > Indeed, it's available at [1]. There are no > > > And will it also contain the usual branches (master, ghc-8.0, wip/), so > > that I can simply take that as the upstream repo for gipeda’s sake? > > > Not as things are currently configured and I'm not sure how easy it > would be to reliably accomplish this. > > The only thing you'll find in that repository currently is tags (not > branches as I stated earlier; sorry!) with names of the form > phabricator/base/1234 and phabricator/diff/1234. These contain the base > and head commits of revision 1234, respectively. Note that 1234 does not > identify a differential; rather it is a revision of a differential (e.g. > the number that you see in the "ID" column of the history table fn a > differential in the web interface). > > Cheers, > > - Ben > > > [1] https://phabricator.haskell.org/diffusion/GHCDIFF/ > ___ > ghc-devs mailing list > ghc-devs@haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > ___ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Re: Action Required: Status of Harbormaster for Differentials
Joachim Breitner writes: > [ Unknown signature status ] > Hi, > > Am Freitag, den 26.08.2016, 09:38 -0400 schrieb Ben Gamari: >> The reason for this new requirement is that `arc diff` will soon push >> submitted diffs to a "staging area", a Git repository managed by >> Phabricator containing a branch for each submitted Differential. > > this is great, because it means that we can run performance builds via > perf.haskell.org on all DRs automatically! > > Will the staging repository be publicly available? > Indeed, it's available at [1]. There are no > And will it also contain the usual branches (master, ghc-8.0, wip/), so > that I can simply take that as the upstream repo for gipeda’s sake? > Not as things are currently configured and I'm not sure how easy it would be to reliably accomplish this. The only thing you'll find in that repository currently is tags (not branches as I stated earlier; sorry!) with names of the form phabricator/base/1234 and phabricator/diff/1234. These contain the base and head commits of revision 1234, respectively. Note that 1234 does not identify a differential; rather it is a revision of a differential (e.g. the number that you see in the "ID" column of the history table fn a differential in the web interface). Cheers, - Ben [1] https://phabricator.haskell.org/diffusion/GHCDIFF/ signature.asc Description: PGP signature ___ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Re: Action Required: Status of Harbormaster for Differentials
Hi, Am Freitag, den 26.08.2016, 09:38 -0400 schrieb Ben Gamari: > The reason for this new requirement is that `arc diff` will soon push > submitted diffs to a "staging area", a Git repository managed by > Phabricator containing a branch for each submitted Differential. this is great, because it means that we can run performance builds via perf.haskell.org on all DRs automatically! Will the staging repository be publicly available? And will it also contain the usual branches (master, ghc-8.0, wip/), so that I can simply take that as the upstream repo for gipeda’s sake? Greetings, Joachim -- Joachim “nomeata” Breitner m...@joachim-breitner.de • https://www.joachim-breitner.de/ XMPP: nome...@joachim-breitner.de • OpenPGP-Key: 0xF0FBF51F Debian Developer: nome...@debian.org signature.asc Description: This is a digitally signed message part ___ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Action Required: Status of Harbormaster for Differentials
tl;dr. I am working on restoring Harbormaster support for Differentials. Towards this end, GHC's Phabricator instance will soon require that you add an SSH public key to your account in order to upload patches with `arc diff`. Hello everyone, As you are no doubt aware Harbormaster has not been building Differentials for a very long time now. I am currently looking at options for restoring this service but it appears that doing so will require a bit of action on the part of our contributors. Specically, you will need to add an SSH key to Phabricator. You can do so by navigating to the Phabricator Settings [1] page, selecting the "Personal Account Settings" option, and selecting "SSH Public Keys" from the list on the left. There you can click on the "SSH Key Actions" menu on the top-right corner and click on the "Upload Public Key" item. Here you can specify a key name and paste the key contents (e.g. the contents of $HOME/.ssh/id_rsa.pub). The reason for this new requirement is that `arc diff` will soon push submitted diffs to a "staging area", a Git repository managed by Phabricator containing a branch for each submitted Differential. This will allow more reliable Harbormaster builds and merges as we will no longer need to rely on Phabricator correctly applying Differential patches (which has historically been problematic, especially for Differentials where the base commit is unavailable). If there are no objections, I would like to enable the staging area for the rGHC repository in three days, on Monday, 29 August 2016. After this date `arc diff` will try to push submitted differentials to the staging area, which will require a public key. Note, however, that pushing to the staging area can always be disabled with `arc diff`'s `--skip-staging` flag (although we won't be able to run CI builds on Differentials submitted in this way). To be clear: I would have liked to provide an option that did not require key-based authentication, but sadly neither Phabricator nor Gitolite provide any such option. I apologize for the added friction that this new requirement imposes. If enough people feel strongly that this is too onerous then I'm happy to entertain alternative solutions. Cheers, - Ben [1] https://phabricator.haskell.org/settings/ signature.asc Description: PGP signature ___ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs