Hi all,
Most everything I could get done without blocking on external forces
has been done now, and it's time for the status report.
Completed actions
-----------------
* The buildstream repository has been migrated successfully to:
https://github.com/apache/buildstream/
This includes merge requests and issues, barring some minor
loss as outlined in the migration wiki page[0]
* The releases have been setup in the new github repository:
https://github.com/apache/buildstream/releases
All documentation packages from prior releases have been carried
over from historical builds on gitlab, and attached to the releases
on github.
The release announcement emails for every historical release have
also been linked to every release on github.
* The base pre/post merge CI has been setup on github - tests are run
in pre merge CI and bleeding edge master documentation is published
whenever merging something to master.
This excludes:
- Overnight tests
- Remote execution tests
- Windows platform (partial) tests
* References to gitlab have been replaced with references to github.
This includes:
- The buildstream repository
- The docs-website repository
- The website repository
* The docs-website documentation aggregation has been updated.
This repository now obtains the documentation packages for the
releases it decides to publish documentation for from github releases
instead of gitlab pipeline artifacts.
Todo list
---------
* Configuration of protected branches.
This is a little complicated without access to the repository
configuration UI (I managed to temporarily lock myself out during
the migration).
The most important part will be to ensure we dont allow any
non-fastforward merges on master or any release branch, this is
rather high priority.
* Recover lost CI
Some work and thought needs to be put into how we can revive some
of our lost CI:
- Overnight tests
These take too long for standard github runners, can we get support
from Apache infra to run the heavy duty overnights ?
- Remote execution tests
This is just complicated to achieve in github, requires work.
- Windows platform (partial) tests
This needs further investigation.
* Status checks for CI which is running on pull requests don't seem
to appear in the pull request UI.
This may be only an issue for pull requests submitted from "forked"
repositories.
Since we expect most contributions to come from "forked" repos on
github, it would be good to fix this (and from prior experience, I've
seen this work properly before on other projects).
E.g.: At the bottom of the page here:
https://github.com/apache/buildstream/pull/1436
We can see some status about the PR, but it is missing the status
from "workflow actions".
* The github "pages" needs to be properly enabled.
We're facing some minor technical difficulty here, this was tested
to work in the test migration but needs assistance from Apache infra
team to setup properly on github.
The documentation is being published to our `gh-pages` branch
automatically but it is not appearing yet at:
https://buildstream.apache.github.io/buildstream/
* Badges and links for latest releases and snapshots need to be fixed.
These will be fixed once github pages is fixed, as the "latest build"
of master documentation is the single source of truth for these
badges.
* The docs-website currently excludes documentation for the master
branch.
Again, this will be fixed once github pages is sorted.
* The code coverage support in the github UI is not enabled yet.
This requires some support from Apache infra, the "codecov"
github application installed on apache github
(https://codecov.io/gh/apache) needs to be given the proper
permission to the new buildstream repository.
Once that is done, I can merge Tom Pollards work:
https://github.com/buildstream-migration/buildstream/pull/8
And it should provide us with a beautiful code coverage UX
(somewhat better than what we had with gitlab, even).
* Documentation updates
While updating the websites and docs I noticed there are some
parts which were not easy to update.
Mostly:
- Instructions for contributors on how to contribute patches
- Instructions for maintainers concerning the release process
These bits of documentation are mostly in CONTRIBUTING.rst and
need to be updated once we have figured out our workflow a bit
more clearly.
* Repositories still need to be migrated
The following repos:
- https://gitlab.com/BuildStream/website/
- https://gitlab.com/BuildStream/docs-website/
Are to be migrated to github too.
This entails:
- Obtaining new github repositories under the apache namespace
- Implementing the CI for building and publishing these docs in
github (this is already done in the corresponding tested
migrated repositories https://github.com/buildstream-migration/)
- Updating the CNAME records for the dns to resolve to the new
locations.
Looking forward
---------------
Not strictly as a part of this migration, but related:
* Cache server used for overnight tests
We are currently hosting a testing CAS service independently of the
tests, mostly for the purpose of the overnight tests.
What should we do about this ? Shut it down ?
Move this service to apache infra in some place that we can easily
update the service and redeploy it without much hassle ?
* Move to freenode IRC
We're currently looking into how we can get proper ownership of our
own #buildstream channel on freenode, once that is done we will
announce and move our IRC to freenode.
At this point, we will try to update all documentation pointing to
our IRC channel.
As agreed upon late last year, will will update this to equally
promote the use of "slack".
Cheers,
-Tristan
[0]: https://gitlab.com/BuildStream/buildstream/-/wikis/GitHubMigration
On Sat, 2021-01-02 at 19:33 +0900, Tristan Van Berkom wrote:
> Hi all, and happy new year !
>
> The migration script has completed and I have now locked down the
> BuildStream project on gitlab, setting it to archive mode, and
> replacing the main README.rst with a notice and link to the new
> repository on GitHub:
>
> https://github.com/apache/buildstream/
>
> There is still some follow up work to do, such as migrating the
> historical release data and docs builds, and setting up the CI and
> review process on GitHub properly.
>
> I will report back to the list when things are good to go.
>
> Cheers,
> -Tristan
>
> On Wed, 2020-12-30 at 20:40 +0900, Tristan Van Berkom wrote:
> > Hi again,
> >
> > On Wed, 2020-12-30 at 19:34 +0900, Tristan Van Berkom wrote:
> > > Hi all,
> > >
> > > As expected, something always goes wrong.
> > >
> > > You will have received a lot of spam mail from the dev mailing
> > > list now
> > > for the issue migration of only the first 20 issues and their
> > > comments,
> > > so I have temporarily halted the migration until I can sort this.
> > >
> >
> > Crisis averted; we were able to disable the notifications and the
> > migration is chugging along smoothly and silently.
> >
> > Cheers,
> > -Tristan
> >
> > > Cheers,
> > > -Tristan
> > >
> > >
> > > On Wed, 2020-12-30 at 18:40 +0900, Tristan Van Berkom wrote:
> > > > Hi all,
> > > >
> > > > I am about to run the migration script which will migrate issue
> > > > and
> > > > merge request and such records from gitlab to github.
> > > >
> > > > This is expected to take over 12 hours to complete, and is not
> > > > expected
> > > > to generate immense spam after having tested this before and
> > > > making
> > > > adjustments to the migration script.
> > > >
> > > > That said, things can always go wrong, as this is a one-off it
> > > > will
> > > > be
> > > > best to let the script complete.
> > > >
> > > > I will not be setting the gitlab repository into a read-only
> > > > archived
> > > > state until after the migration completes, as this is untested,
> > > > so:
> > > >
> > > > !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
> > > > !!! !!!
> > > > !!! Any new issues, merge requests, or comments !!!
> > > > !!! made to gitlab after this point may very well !!!
> > > > !!! be lost. !!!
> > > > !!! !!!
> > > > !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
> > > >
> > > > Please refrain from updating the buildstream repo in gitlab
> > > > during
> > > > this
> > > > time.
> > > >
> > > > This has been tested a few times already, but of course many
> > > > things
> > > > can
> > > > go wrong, fingers crossed !
> > > >
> > > > Cheers,
> > > > -Tristan
> > > >
> > > > On Mon, 2020-12-28 at 19:21 +0900, Tristan Van Berkom wrote:
> > > > > Hi all,
> > > > >
> > > > > As we have known for some time now, the BuildStream project
> > > > > will
> > > > > be
> > > > > moving to GitHub.
> > > > >
> > > > > Since we are in the holiday season and we are confident that
> > > > > it
> > > > > will
> > > > > not be disruptive, we are going to do our best to push this
> > > > > over
> > > > > the
> > > > > line during this lull, and kick off 2021 development using a
> > > > > github
> > > > > workflow.
> > > > >
> > > > > Migration plan
> > > > > --------------
> > > > > Chandan made a lot of progress some months ago to get CI
> > > > > mostly
> > > > > working
> > > > > in github[0].
> > > > >
> > > > > Tom Pollard, Douglas Winship, Chris Polin and I took another
> > > > > crack
> > > > > at
> > > > > it last month and we've ironed out most of the creases to our
> > > > > satisfaction.
> > > > >
> > > > > This mostly includes solving the release process and the
> > > > > problem
> > > > > of automatically publishing documentation of prior releases.
> > > > >
> > > > > We created a writeup of our plan to perform the migration
> > > > > here[1].
> > > > >
> > > > >
> > > > > What are we migrating ?
> > > > > -----------------------
> > > > > We plan to migrate the following repositories to the apache
> > > > > namespace
> > > > > in github, at https://github.com/apache/.
> > > > >
> > > > > * The main BuildStream core repository
> > > > > https://gitlab.com/BuildStream/buildstream/
> > > > >
> > > > > * The repository which generates the website
> > > > > https://gitlab.com/BuildStream/website/
> > > > >
> > > > > * The repository which aggregates and publishes docs for
> > > > > released
> > > > > versions of BuildStream
> > > > > https://gitlab.com/BuildStream/docs-website/
> > > > >
> > > > > There are no plans to migrate the plugin repositories at this
> > > > > time.
> > > > >
> > > > > While we may end up creating some new plugin specific
> > > > > repositories
> > > > > on
> > > > > github in the future, we expect mainly to continue to use
> > > > > gitlab
> > > > > for
> > > > > plugin repositories in the long term, this will be easier for
> > > > > plugin
> > > > > maintainers to release asynchronously from core BuildStream
> > > > > and
> > > > > with a
> > > > > bit less bureaucracy.
> > > > >
> > > > >
> > > > > BuildStream first
> > > > > -----------------
> > > > > We have recently acquired
> > > > > https://github.com/apache/buildstream/
> > > > > for
> > > > > our purposes to migrate the core BuildStream repository.
> > > > >
> > > > > All three repositories will not be migrated in one go,
> > > > > BuildStream
> > > > > will
> > > > > be migrated first.
> > > > >
> > > > >
> > > > > Loss
> > > > > ----
> > > > > Things may be a bit shaky for a time, there will be some
> > > > > minor
> > > > > loss
> > > > > in
> > > > > the migration, but we've tested the migration and we're
> > > > > confident
> > > > > that
> > > > > loss is minimal.
> > > > >
> > > > > For example, some things like already merged/closed merge
> > > > > requests
> > > > > may
> > > > > not be migrated as pull requests, and comments and issues
> > > > > will
> > > > > not
> > > > > automatically be assigned to your corresponding username on
> > > > > github.
> > > > >
> > > > > Thanks to Douglas's scripting we have managed to preserve
> > > > > references to
> > > > > gitlab usernames referred to in comments: links to users in
> > > > > historical
> > > > > issues and merge requests will refer to users on gitlab, and
> > > > > not
> > > > > users
> > > > > which might not exist on github, or might exist but not be
> > > > > the
> > > > > same
> > > > > person.
> > > > >
> > > > > Some of the CI will have to be revived post migration, mostly
> > > > > this
> > > > > is
> > > > > limited to remote execution testing - we have agreed to
> > > > > address
> > > > > this
> > > > > after the migration.
> > > > >
> > > > >
> > > > > Summary
> > > > > -------
> > > > > With any luck, we will get this done swiftly by the time
> > > > > people
> > > > > start
> > > > > coming back to work in 2021, and I hope to see you all next
> > > > > year
> > > > > on
> > > > > github :)
> > > > >
> > > > > I will be periodically replying to this thread to announce
> > > > > timeframes
> > > > > as I coordinate with Apache folks and sort out permissions
> > > > > and
> > > > > such.
> > > > >
> > > > > And of course, Merry Christmas and Happy New Year !!
> > > > >
> > > > > Cheers,
> > > > > -Tristan
> > > > >
> > > > >
> > > > > [0]:
> > > > > https://lists.apache.org/thread.html/r229ff54d163389b3e8e42688174336117a139fad9fd63d034db1cc23%40%3Cdev.buildstream.apache.org%3E
> > > > > [1]:
> > > > > https://gitlab.com/BuildStream/buildstream/-/wikis/GitHubMigration
> > > > >
> > > > >
> > > > >
