So any updates here Gavin? Is the ssh transfer plugin available now?
Thanks. Sean Busbey <[email protected]> 于2022年1月23日周日 09:04写道： > It looks like I currently do have access to manage the ci-hbase controller. > > > Could you check that the git-websites label handling where a common > machine gets leased to the ci-hbase controller is set up properly? or give > me a pointer on how I would check this myself? > > I started trying to migrate our website generation job again and it's been > waiting for a node with the git-websites label to show up for ~8 hours at > this point. > > https://ci-hbase.apache.org/job/hbase_generate_website/3/ > > > > On Sat, Jan 22, 2022 at 2:42 AM Gavin McDonald <[email protected]> > wrote: > >> Hi, >> Timestamper plugin installed. >> >> By the way I thought I had given the PMC members permissions to manage >> Jenkins >> on ci-hbase including installing plugins etc, can someone check? >> >> >> On Sat, Jan 22, 2022 at 7:34 AM 张铎(Duo Zhang) <[email protected]> >> wrote: >> >> > We faced a strange issue.. >> > >> > rg.codehaus.groovy.control.MultipleCompilationErrorsException: startup >> failed: >> > WorkflowScript: 29: Invalid option type "timestamps". Valid option >> types: [buildDiscarder, catchError, checkoutToSubdirectory, >> disableConcurrentBuilds, disableResume, durabilityHint, >> githubProjectProperty, newContainerPerStage, overrideIndexTriggers, >> parallelsAlwaysFailFast, preserveStashes, quietPeriod, rateLimitBuilds, >> retry, script, skipDefaultCheckout, skipStagesAfterUnstable, timeout, >> waitUntil, warnError, withChecks, withContext, withCredentials, withEnv, >> wrap, ws] @ line 29, column 5. >> > timestamps() >> > ^ >> > >> > 1 error >> > >> > at >> org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:310) >> > at >> org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1085) >> > at >> org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:603) >> > at >> org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581) >> > at >> org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558) >> > at >> groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298) >> > at >> groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268) >> > at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688) >> > at groovy.lang.GroovyShell.parse(GroovyShell.java:700) >> > at >> org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:142) >> > at >> org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:127) >> > at >> org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:571) >> > at >> org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:523) >> > at >> org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:334) >> > at >> hudson.model.ResourceController.execute(ResourceController.java:97) >> > at hudson.model.Executor.run(Executor.java:429) >> > >> > >> > After googling, it seems a restart of jenkins can fix it, or we do not >> > have timestamper plugin installed? Mind taking a look Gavin? >> > >> > Thanks. >> > >> > 张铎(Duo Zhang) <[email protected]> 于2022年1月22日周六 11:40写道： >> > >> >> Oh, just saw that we could do this by ourselves. >> >> >> >> Let me have a try. >> >> >> >> 张铎(Duo Zhang) <[email protected]> 于2022年1月22日周六 11:18写道： >> >> >> >>> OK, so while copying you could keep the old jenkins builds? >> >>> That's awesome. >> >>> >> >>> Then please help 'copying' HBase-Find-Flaky-Tests and >> HBase-Flaky-Tests >> >>> to ci-hbase too? We need the history builds to calucate the flaky >> test list >> >>> so if all the old build data can be kept then it will help us a lot. >> >>> >> >>> Thanks~ >> >>> >> >>> 张铎(Duo Zhang) <[email protected]> 于2022年1月22日周六 11:03写道： >> >>> >> >>>> Thank you Gavin. >> >>>> >> >>>> Let me first move some light weighted job first to see if everything >> is >> >>>> OK. >> >>>> >> >>>> HBase Nightly, some flaky related jobs, and also the main pre commit >> >>>> job for HBase are tied together. We need to move them at once. >> >>>> >> >>>> Gavin McDonald <[email protected]> 于2022年1月22日周六 02:44写道： >> >>>> >> >>>>> Thanks for your patience >> >>>>> >> >>>>> I have 'copied' over a job - >> >>>>> https://ci-hbase.apache.org/job/HBase%20Nightly/ >> >>>>> >> >>>>> which of course made me aware of what plugins I needed to install. >> >>>>> >> >>>>> Feel free to delete that, or use it and delete the old, and let me >> >>>>> know if have any issues >> >>>>> with any other missing plugins. >> >>>>> >> >>>>> >> >>>>> >> >>>>> On Thu, Jan 20, 2022 at 1:11 PM 张铎(Duo Zhang) < >> [email protected]> >> >>>>> wrote: >> >>>>> >> >>>>>> I’ve also replied on the jira issue. >> >>>>>> >> >>>>>> You can try to copy a job to the new Jenkins. >> >>>>>> >> >>>>>> Thanks. >> >>>>>> >> >>>>>> Gavin McDonald <[email protected]>于2022年1月20日 周四18:45写道： >> >>>>>> >> >>>>>>> Im taking a look ... >> >>>>>>> >> >>>>>>> On Thu, Jan 20, 2022 at 6:54 AM Sean Busbey <[email protected]> >> >>>>>>> wrote: >> >>>>>>> >> >>>>>>> > Hi Gavin >> >>>>>>> > >> >>>>>>> > I went to start migrating jobs to the ci-hbase controller, but >> the >> >>>>>>> new job >> >>>>>>> > workflow doesn't include pipelines or multibranch pipelines. We >> >>>>>>> use both of >> >>>>>>> > these job types for nearly all of our work. Can they be enabled? >> >>>>>>> > >> >>>>>>> > On Tue, Jan 18, 2022 at 5:04 AM Gavin McDonald < >> >>>>>>> [email protected]> >> >>>>>>> > wrote: >> >>>>>>> > >> >>>>>>> >> >> >>>>>>> >> >> >>>>>>> >> On 2022/01/18 09:20:24 "张铎(Duo Zhang)" wrote: >> >>>>>>> >> > Thank you Gavin. >> >>>>>>> >> > >> >>>>>>> >> > We have some urls in our jenkins files which references the >> >>>>>>> artifacts of >> >>>>>>> >> > the jenkins jobs, so a simple copy or move can not work. For >> >>>>>>> example, >> >>>>>>> >> here: >> >>>>>>> >> > >> >>>>>>> >> > >> >>>>>>> >> >> >>>>>>> >> https://github.com/apache/hbase/blob/c9bcd87b34a15d200a55ec7fdc2b1d86e3367a8c/dev-support/Jenkinsfile_GitHub#L47 >> >>>>>>> >> > >> >>>>>>> >> > So I prefer we migrate the jobs by ourselves. Will file a >> HBase >> >>>>>>> issue >> >>>>>>> >> to do >> >>>>>>> >> > this. >> >>>>>>> >> > >> >>>>>>> >> > And will report any problems on INFRA_22760 while migrating. >> >>>>>>> When the >> >>>>>>> >> > migration is half done, you can move for example, 5 hbase >> nodes >> >>>>>>> from >> >>>>>>> >> > ci-hadoop to ci-hbase, and once the migration is finally >> done, >> >>>>>>> you can >> >>>>>>> >> move >> >>>>>>> >> > all the remaining hbase nodes from ci-hadoop to ci-hbase. >> >>>>>>> >> > >> >>>>>>> >> > WDYT? >> >>>>>>> >> >> >>>>>>> >> However you want to do it is fine with me :) >> >>>>>>> >> >> >>>>>>> >> You have 10 nodes on there right now to get you started. >> >>>>>>> >> >> >>>>>>> >> Let me know how you get on >> >>>>>>> >> >> >>>>>>> >> > >> >>>>>>> >> > Thanks. >> >>>>>>> >> > >> >>>>>>> >> > Gavin McDonald <[email protected]> 于2022年1月18日周二 16:16写道： >> >>>>>>> >> > >> >>>>>>> >> > > Hi All, >> >>>>>>> >> > > >> >>>>>>> >> > > I created an Infra ticket for tracking. >> >>>>>>> >> > > >> >>>>>>> >> > > https://issues.apache.org/jira/browse/INFRA-22760 >> >>>>>>> >> > > >> >>>>>>> >> > > For now, I left your current nodes hbase1 to hbase10 in >> >>>>>>> ci-hadoop.a.o >> >>>>>>> >> > > >> >>>>>>> >> > > I have created for you 10 new nodes so far out of our >> Hetzner >> >>>>>>> DC >> >>>>>>> >> > > of a similar specifications to the hbase1-10. >> >>>>>>> >> > > >> >>>>>>> >> > > Those are hbase11-20 and are available right now for >> testing >> >>>>>>> on >> >>>>>>> >> > > ci-hbase.apache.org >> >>>>>>> >> > > >> >>>>>>> >> > > I do not know your workflow well enough to decide the next >> >>>>>>> steps for >> >>>>>>> >> you >> >>>>>>> >> > > for testing. >> >>>>>>> >> > > >> >>>>>>> >> > > Options include: >> >>>>>>> >> > > >> >>>>>>> >> > > 1. I can 'move' all jobs out of ci-hadoop onto ci-hbase >> >>>>>>> >> > > 2. I can 'move' one or a number of jobs out of ci-hadoop >> onto >> >>>>>>> ci-hbase >> >>>>>>> >> > > 3. Similar to 1 or 2, but 'copy' any number of jobs over >> for >> >>>>>>> testing. >> >>>>>>> >> > > - I like copy, however quite a few projects jobs would >> >>>>>>> not be >> >>>>>>> >> > > appropriate >> >>>>>>> >> > > to have the same job running twice, including those >> that >> >>>>>>> >> deploy, or >> >>>>>>> >> > > those that >> >>>>>>> >> > > build from PRs and continue a workflow to report >> back, >> >>>>>>> or >> >>>>>>> >> approve >> >>>>>>> >> > > for merge. >> >>>>>>> >> > > So for a copy, would need to be a job that just >> 'builds >> >>>>>>> and >> >>>>>>> >> tests' >> >>>>>>> >> > > with no outflow. >> >>>>>>> >> > > 4. You guys create new jobs on ci-hbase.apache.org to test >> >>>>>>> the nodes >> >>>>>>> >> > > setup and the >> >>>>>>> >> > > main controller setup, without affecting current jobs. >> >>>>>>> >> > > >> >>>>>>> >> > > Obviously, you may to tweak version controlled script files >> >>>>>>> such as a >> >>>>>>> >> > > JenkinsFile, which makes the copy options harder unless >> >>>>>>> testing from a >> >>>>>>> >> > > branch created for the purpose. >> >>>>>>> >> > > >> >>>>>>> >> > > I may be overthinking things, and option 1 , the simplest >> and >> >>>>>>> most >> >>>>>>> >> > > straight forward from my end would also suit you guys and >> we >> >>>>>>> deal >> >>>>>>> >> with any >> >>>>>>> >> > > breakage as it happens (I'm thinking of integrations with >> >>>>>>> other >> >>>>>>> >> services, >> >>>>>>> >> > > Credentials/Secrets, Plugins I haven't installed that are >> >>>>>>> needed etc.) >> >>>>>>> >> > > >> >>>>>>> >> > > Let me know please ASAP how you want to proceed so we can >> get >> >>>>>>> using >> >>>>>>> >> these >> >>>>>>> >> > > nodes, after which I can then proceed to move hbase1-10 >> nodes >> >>>>>>> over >> >>>>>>> >> also. >> >>>>>>> >> > > And then start adding more into the mix. >> >>>>>>> >> > > >> >>>>>>> >> > > Oh. The 'git-websites' label should be a no-op. It is >> >>>>>>> available on >> >>>>>>> >> > > ci-hbase.a.o as the nodes with that label are shared and >> >>>>>>> leased out to >> >>>>>>> >> > > controllers as needed upon request of a job. >> >>>>>>> >> > > >> >>>>>>> >> > > Thanks >> >>>>>>> >> > > >> >>>>>>> >> > > Gav... >> >>>>>>> >> > > >> >>>>>>> >> > > >> >>>>>>> >> > > On 2021/12/09 15:07:26 "张铎(Duo Zhang)" wrote: >> >>>>>>> >> > > > Thanks you Gavin. >> >>>>>>> >> > > > >> >>>>>>> >> > > > You can do it in your convenience, just let me know so I >> >>>>>>> could >> >>>>>>> >> change the >> >>>>>>> >> > > > label in the jenkinsfile of our jenkins jobs, at least >> our >> >>>>>>> pre >> >>>>>>> >> commit job >> >>>>>>> >> > > > still uses the Hadoop label. >> >>>>>>> >> > > > >> >>>>>>> >> > > > And another thing is about the hbase_generate_website >> job, >> >>>>>>> it needs >> >>>>>>> >> to be >> >>>>>>> >> > > > executed on the nodes under the git-websites label. I'm >> not >> >>>>>>> sure if >> >>>>>>> >> the >> >>>>>>> >> > > > label is still available after the migration. >> >>>>>>> >> > > > >> >>>>>>> >> > > > Gavin McDonald <[email protected]> 于2021年12月8日周三 >> >>>>>>> 17:28写道： >> >>>>>>> >> > > > >> >>>>>>> >> > > > > Hi HBase devs. >> >>>>>>> >> > > > > >> >>>>>>> >> > > > > This is to let you know that Infra has created a >> >>>>>>> dedicated Jenkins >> >>>>>>> >> > > > > Controller at https://ci-hbase.apache.org . >> >>>>>>> >> > > > > >> >>>>>>> >> > > > > You have new incoming donated agents which will get >> >>>>>>> connected to >> >>>>>>> >> this >> >>>>>>> >> > > new >> >>>>>>> >> > > > > controller by Infra. >> >>>>>>> >> > > > > >> >>>>>>> >> > > > > You also have 10 or so agents connected to >> >>>>>>> >> > > https://ci-hadoop.apache.org - >> >>>>>>> >> > > > > I >> >>>>>>> >> > > > > would like to move these agents - and all of your jobs >> - >> >>>>>>> over to >> >>>>>>> >> the >> >>>>>>> >> > > new >> >>>>>>> >> > > > > controller as soon as possible. >> >>>>>>> >> > > > > >> >>>>>>> >> > > > > You may or may not need to reconfigure your jobs and or >> >>>>>>> >> JenkinsFile or >> >>>>>>> >> > > > > other. >> >>>>>>> >> > > > > >> >>>>>>> >> > > > > Can I get the go ahead to move these at my convenience >> or >> >>>>>>> suggest >> >>>>>>> >> > > please a >> >>>>>>> >> > > > > timeline for Infra to move these agents. >> >>>>>>> >> > > > > >> >>>>>>> >> > > > > Thanks! >> >>>>>>> >> > > > > >> >>>>>>> >> > > > > -- >> >>>>>>> >> > > > > >> >>>>>>> >> > > > > *Gavin McDonald* >> >>>>>>> >> > > > > Systems Administrator >> >>>>>>> >> > > > > ASF Infrastructure Team >> >>>>>>> >> > > > > >> >>>>>>> >> > > > >> >>>>>>> >> > > >> >>>>>>> >> > >> >>>>>>> >> >> >>>>>>> > >> >>>>>>> >> >>>>>>> -- >> >>>>>>> >> >>>>>>> *Gavin McDonald* >> >>>>>>> Systems Administrator >> >>>>>>> ASF Infrastructure Team >> >>>>>>> >> >>>>>> >> >>>>> >> >>>>> -- >> >>>>> >> >>>>> *Gavin McDonald* >> >>>>> Systems Administrator >> >>>>> ASF Infrastructure Team >> >>>>> >> >>>> >> >> -- >> >> *Gavin McDonald* >> Systems Administrator >> ASF Infrastructure Team >> >
