Hi Duo, I’m trying to create a similar job for Apache ZooKeeper, but unfortunately haven’t got too much help on the Apache builds@ list so far, so I’m rather asking you if you don’t mind.
First, how have you set up the Hbase Github account that you use in this job to access the repo? Thanks, Andor > On 2020. Jul 27., at 2:22, 张铎(Duo Zhang) <[email protected]> wrote: > > The pre commit job has been migrated to c-hadoop.a.o. > > I have disabled periodical scan for the old job on builds.a.o, as we still > need to view the pre commit result on it do not delete for now. Will delete > it later, maybe after several weeks. > > The new job is here > > https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/ > > Thanks. > > 张铎(Duo Zhang) <[email protected]> 于2020年7月25日周六 下午9:44写道： > >> >> https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/5/console >> >> >> We successfully finished a nightly build. >> >> But seems the jiraComment did not work. I haven't seen the comment >> on HBASE-24757... >> >> 张铎(Duo Zhang) <[email protected]> 于2020年7月25日周六 下午4:51写道： >> >>> After installing two new jenkins plugins, the pre commit job seems fine >>> now. >>> >>> The last failure is because of a timeout, I assume the problem is that we >>> do not have enough executors so all the jobs are executed sequentially. >>> >>> Maybe we could move the pre commit job to the new env first? The nightly >>> job and flaky job require more resources, and we need the output of these >>> jenkins jobs(the flaky test list). >>> >>> Thanks. >>> >>> >>> >>> 张铎(Duo Zhang) <[email protected]> 于2020年7月24日周五 下午4:36写道： >>> >>>> The problem seems because of this: >>>> >>>> https://issues.jenkins-ci.org/browse/JENKINS-48556 >>>> >>>> I triggered the job again, it passed the timestamps call, and will keep >>>> an eye on it. >>>> >>>> 张铎(Duo Zhang) <[email protected]> 于2020年7月21日周二 上午11:18写道： >>>> >>>>> On the sponsors, we could have a try. >>>>> >>>>> The problem here is the process of the donation? IIRC there is a thread >>>>> on the infra mailing list about how to donate machines to a specific >>>>> project and the discussion did not go well... >>>>> >>>>> Sean Busbey <[email protected]> 于2020年7月21日周二 上午11:13写道： >>>>> >>>>>> We could check with ASF infra for the current state of things wrt >>>>>> GitHub >>>>>> actions. I believe there is a queue set up across ASF projects. >>>>>> >>>>>> It has the same resource issue Travis had; things are fine until some >>>>>> critical mass of projects seeking better perf realize some new option >>>>>> is >>>>>> available and then quickly all available resources are consumed. >>>>>> >>>>>> AFAICT the only option that gets us the same or better as the H* nodes >>>>>> will >>>>>> be finding sponsors and running our own. >>>>>> >>>>>> On Mon, Jul 20, 2020, 21:55 张铎(Duo Zhang) <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> I think our nightly, flakey, and pre commit jobs should be >>>>>> transferred as a >>>>>>> whole? They depend on each other. >>>>>>> >>>>>>> I offer my help on the transition. >>>>>>> >>>>>>> And on github CI, does ASF have a special deal with github? If not, >>>>>> I do >>>>>>> not think the default resource can fit our requirements... >>>>>>> >>>>>>> >>>>>>> >>>>>>> Sean Busbey <[email protected]> 于2020年7月21日周二 上午1:49写道： >>>>>>> >>>>>>>> Hi folks! >>>>>>>> >>>>>>>> Back in April there was a brief discussion[1] about ASF Infra's >>>>>>>> notification that builds.a.o is going away and we are currently >>>>>> slated >>>>>>>> to migrate to a set of CI servers for "Hadoop and related >>>>>> projects". >>>>>>>> This is the ci farm that will contain the bulk of the H* worker >>>>>> nodes >>>>>>>> that are donated by Yahoo!, which are the nodes we've been running >>>>>> on >>>>>>>> for ages[2]. >>>>>>>> >>>>>>>> Migration discussion still happens on the [email protected] >>>>>>>> list[3] and recently ASF Infra set a target date of August 15th for >>>>>>>> turning off the existing builds.a.o server[4]. >>>>>>>> >>>>>>>> That gives us a little under 4 weeks to have things up and working >>>>>> on >>>>>>>> the new ci-hadoop.a.o jenkins coordinator[5]. it’s not clear to me >>>>>>>> that the level of effort we’ll need to spend is worth what we get >>>>>> out >>>>>>>> of a continuation of the status quo on builds.a.o. I did a quick >>>>>> test >>>>>>>> by updating the nightly job on ci-hadoop.a.o to run just branch-2, >>>>>>>> since that has been stable on builds.a.o. It failed with a Jenkins >>>>>>>> pipeline DSL syntax error[6] so I'm assuming migrating will be a >>>>>> slog. >>>>>>>> >>>>>>>> As far as I can see our options are: >>>>>>>> >>>>>>>> * Do nothing. Have no testing or automated website publication in >>>>>> mid >>>>>>>> August. >>>>>>>> * Transition website publication and nothing else (probably can be >>>>>>>> done in a day) >>>>>>>> * Transition just precommit testing for various repos (probably >>>>>> can be >>>>>>>> done in a few days) >>>>>>>> * Transition everything (no idea how long it takes due to nightly, >>>>>>>> flaky stuff, etc) >>>>>>>> >>>>>>>> The alternatives if we do not transition any given job to >>>>>> ci-hadoop: >>>>>>>> >>>>>>>> * Try to move to GitHub Actions >>>>>>>> * Try to move to Travis CI >>>>>>>> * Try to move to Jenkins infra we maintain ourselves (presumably by >>>>>>>> soliciting project specific donations for worker nodes on cloud >>>>>>>> vendors) >>>>>>>> >>>>>>>> It's important to remember that as a project we have a heavy >>>>>> footprint >>>>>>>> wherever our nightly tests run. For context, a given branch's >>>>>> nightly >>>>>>>> can keep 3-4 executors busy for 6+ hours on the current builds.a.o >>>>>>>> setup. There's been a bunch of great work lately on bringing down >>>>>> what >>>>>>>> it takes to run the full test suite, but applying that work to >>>>>> nightly >>>>>>>> is itself a significant undertaking. >>>>>>>> >>>>>>>> What are folks thinking? Most importantly who is ready to work >>>>>> towards >>>>>>>> any given approach? >>>>>>>> >>>>>>>> [1] [DISCUSS] Migrating HBase to new CI Master >>>>>>>> https://s.apache.org/fux1o >>>>>>>> >>>>>>>> [2] https://builds.apache.org/view/H-L/view/HBase/ >>>>>>>> >>>>>>>> [3] >>>>>>> >>>>>> https://lists.apache.org/[email protected] >>>>>>>> >>>>>>>> [4] [IMPORTANT] - 2 more HADOOP nodes migrated over to ci-hadoop >>>>>>>> https://s.apache.org/7e1nq >>>>>>>> >>>>>>>> [5] https://ci-hadoop.apache.org/job/HBase/ >>>>>>>> >>>>>>>> [6] >>>>>>>> >>>>>>> >>>>>> https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/2/console >>>>>>>> >>>>>>> >>>>>> >>>>>
