On the sponsors, we could have a try. The problem here is the process of the donation? IIRC there is a thread on the infra mailing list about how to donate machines to a specific project and the discussion did not go well...
Sean Busbey <[email protected]> 于2020年7月21日周二 上午11:13写道： > We could check with ASF infra for the current state of things wrt GitHub > actions. I believe there is a queue set up across ASF projects. > > It has the same resource issue Travis had; things are fine until some > critical mass of projects seeking better perf realize some new option is > available and then quickly all available resources are consumed. > > AFAICT the only option that gets us the same or better as the H* nodes will > be finding sponsors and running our own. > > On Mon, Jul 20, 2020, 21:55 张铎(Duo Zhang) <[email protected]> wrote: > > > I think our nightly, flakey, and pre commit jobs should be transferred > as a > > whole? They depend on each other. > > > > I offer my help on the transition. > > > > And on github CI, does ASF have a special deal with github? If not, I do > > not think the default resource can fit our requirements... > > > > > > > > Sean Busbey <[email protected]> 于2020年7月21日周二 上午1:49写道： > > > > > Hi folks! > > > > > > Back in April there was a brief discussion[1] about ASF Infra's > > > notification that builds.a.o is going away and we are currently slated > > > to migrate to a set of CI servers for "Hadoop and related projects". > > > This is the ci farm that will contain the bulk of the H* worker nodes > > > that are donated by Yahoo!, which are the nodes we've been running on > > > for ages[2]. > > > > > > Migration discussion still happens on the [email protected] > > > list[3] and recently ASF Infra set a target date of August 15th for > > > turning off the existing builds.a.o server[4]. > > > > > > That gives us a little under 4 weeks to have things up and working on > > > the new ci-hadoop.a.o jenkins coordinator[5]. it’s not clear to me > > > that the level of effort we’ll need to spend is worth what we get out > > > of a continuation of the status quo on builds.a.o. I did a quick test > > > by updating the nightly job on ci-hadoop.a.o to run just branch-2, > > > since that has been stable on builds.a.o. It failed with a Jenkins > > > pipeline DSL syntax error[6] so I'm assuming migrating will be a slog. > > > > > > As far as I can see our options are: > > > > > > * Do nothing. Have no testing or automated website publication in mid > > > August. > > > * Transition website publication and nothing else (probably can be > > > done in a day) > > > * Transition just precommit testing for various repos (probably can be > > > done in a few days) > > > * Transition everything (no idea how long it takes due to nightly, > > > flaky stuff, etc) > > > > > > The alternatives if we do not transition any given job to ci-hadoop: > > > > > > * Try to move to GitHub Actions > > > * Try to move to Travis CI > > > * Try to move to Jenkins infra we maintain ourselves (presumably by > > > soliciting project specific donations for worker nodes on cloud > > > vendors) > > > > > > It's important to remember that as a project we have a heavy footprint > > > wherever our nightly tests run. For context, a given branch's nightly > > > can keep 3-4 executors busy for 6+ hours on the current builds.a.o > > > setup. There's been a bunch of great work lately on bringing down what > > > it takes to run the full test suite, but applying that work to nightly > > > is itself a significant undertaking. > > > > > > What are folks thinking? Most importantly who is ready to work towards > > > any given approach? > > > > > > [1] [DISCUSS] Migrating HBase to new CI Master > > > https://s.apache.org/fux1o > > > > > > [2] https://builds.apache.org/view/H-L/view/HBase/ > > > > > > [3] > > https://lists.apache.org/[email protected] > > > > > > [4] [IMPORTANT] - 2 more HADOOP nodes migrated over to ci-hadoop > > > https://s.apache.org/7e1nq > > > > > > [5] https://ci-hadoop.apache.org/job/HBase/ > > > > > > [6] > > > > > > https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/2/console > > > > > >
