Hi Gavin, the ssh transfer plugin is fine, but now we miss python on hbase11-20.
I've filed INFRA-22855, PTAL. Thanks. 张铎(Duo Zhang) <[email protected]> 于2022年2月8日周二 20:19写道： > Thank you Gavin. WIll take a look soon! > > Gavin McDonald <[email protected]> 于2022年2月8日周二 18:31写道： > >> Give it a try now please >> >> On Sat, Feb 5, 2022 at 8:55 AM 张铎(Duo Zhang) <[email protected]> >> wrote: >> >> > Hi, Gavin, any luck on the ssh transfer plugin? >> > >> > Thanks. >> > >> > 张铎(Duo Zhang) <[email protected]> 于2022年1月29日周六 14:53写道： >> > >> >> So any updates here Gavin? >> >> >> >> Is the ssh transfer plugin available now? >> >> >> >> Thanks. >> >> >> >> Sean Busbey <[email protected]> 于2022年1月23日周日 09:04写道： >> >> >> >>> It looks like I currently do have access to manage the ci-hbase >> >>> controller. >> >>> >> >>> >> >>> Could you check that the git-websites label handling where a common >> >>> machine gets leased to the ci-hbase controller is set up properly? or >> give >> >>> me a pointer on how I would check this myself? >> >>> >> >>> I started trying to migrate our website generation job again and it's >> >>> been waiting for a node with the git-websites label to show up for ~8 >> hours >> >>> at this point. >> >>> >> >>> https://ci-hbase.apache.org/job/hbase_generate_website/3/ >> >>> >> >>> >> >>> >> >>> On Sat, Jan 22, 2022 at 2:42 AM Gavin McDonald <[email protected]> >> >>> wrote: >> >>> >> >>>> Hi, >> >>>> Timestamper plugin installed. >> >>>> >> >>>> By the way I thought I had given the PMC members permissions to >> manage >> >>>> Jenkins >> >>>> on ci-hbase including installing plugins etc, can someone check? >> >>>> >> >>>> >> >>>> On Sat, Jan 22, 2022 at 7:34 AM 张铎(Duo Zhang) <[email protected] >> > >> >>>> wrote: >> >>>> >> >>>> > We faced a strange issue.. >> >>>> > >> >>>> > rg.codehaus.groovy.control.MultipleCompilationErrorsException: >> >>>> startup failed: >> >>>> > WorkflowScript: 29: Invalid option type "timestamps". Valid option >> >>>> types: [buildDiscarder, catchError, checkoutToSubdirectory, >> >>>> disableConcurrentBuilds, disableResume, durabilityHint, >> >>>> githubProjectProperty, newContainerPerStage, overrideIndexTriggers, >> >>>> parallelsAlwaysFailFast, preserveStashes, quietPeriod, >> rateLimitBuilds, >> >>>> retry, script, skipDefaultCheckout, skipStagesAfterUnstable, timeout, >> >>>> waitUntil, warnError, withChecks, withContext, withCredentials, >> withEnv, >> >>>> wrap, ws] @ line 29, column 5. >> >>>> > timestamps() >> >>>> > ^ >> >>>> > >> >>>> > 1 error >> >>>> > >> >>>> > at >> >>>> >> org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:310) >> >>>> > at >> >>>> >> org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1085) >> >>>> > at >> >>>> >> org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:603) >> >>>> > at >> >>>> >> org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581) >> >>>> > at >> >>>> >> org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558) >> >>>> > at >> >>>> >> groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298) >> >>>> > at >> >>>> groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268) >> >>>> > at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688) >> >>>> > at groovy.lang.GroovyShell.parse(GroovyShell.java:700) >> >>>> > at >> >>>> >> org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:142) >> >>>> > at >> >>>> >> org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:127) >> >>>> > at >> >>>> >> org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:571) >> >>>> > at >> >>>> >> org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:523) >> >>>> > at >> >>>> >> org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:334) >> >>>> > at >> >>>> hudson.model.ResourceController.execute(ResourceController.java:97) >> >>>> > at hudson.model.Executor.run(Executor.java:429) >> >>>> > >> >>>> > >> >>>> > After googling, it seems a restart of jenkins can fix it, or we do >> not >> >>>> > have timestamper plugin installed? Mind taking a look Gavin? >> >>>> > >> >>>> > Thanks. >> >>>> > >> >>>> > 张铎(Duo Zhang) <[email protected]> 于2022年1月22日周六 11:40写道： >> >>>> > >> >>>> >> Oh, just saw that we could do this by ourselves. >> >>>> >> >> >>>> >> Let me have a try. >> >>>> >> >> >>>> >> 张铎(Duo Zhang) <[email protected]> 于2022年1月22日周六 11:18写道： >> >>>> >> >> >>>> >>> OK, so while copying you could keep the old jenkins builds? >> >>>> >>> That's awesome. >> >>>> >>> >> >>>> >>> Then please help 'copying' HBase-Find-Flaky-Tests and >> >>>> HBase-Flaky-Tests >> >>>> >>> to ci-hbase too? We need the history builds to calucate the flaky >> >>>> test list >> >>>> >>> so if all the old build data can be kept then it will help us a >> lot. >> >>>> >>> >> >>>> >>> Thanks~ >> >>>> >>> >> >>>> >>> 张铎(Duo Zhang) <[email protected]> 于2022年1月22日周六 11:03写道： >> >>>> >>> >> >>>> >>>> Thank you Gavin. >> >>>> >>>> >> >>>> >>>> Let me first move some light weighted job first to see if >> >>>> everything is >> >>>> >>>> OK. >> >>>> >>>> >> >>>> >>>> HBase Nightly, some flaky related jobs, and also the main pre >> >>>> commit >> >>>> >>>> job for HBase are tied together. We need to move them at once. >> >>>> >>>> >> >>>> >>>> Gavin McDonald <[email protected]> 于2022年1月22日周六 02:44写道： >> >>>> >>>> >> >>>> >>>>> Thanks for your patience >> >>>> >>>>> >> >>>> >>>>> I have 'copied' over a job - >> >>>> >>>>> https://ci-hbase.apache.org/job/HBase%20Nightly/ >> >>>> >>>>> >> >>>> >>>>> which of course made me aware of what plugins I needed to >> install. >> >>>> >>>>> >> >>>> >>>>> Feel free to delete that, or use it and delete the old, and >> let me >> >>>> >>>>> know if have any issues >> >>>> >>>>> with any other missing plugins. >> >>>> >>>>> >> >>>> >>>>> >> >>>> >>>>> >> >>>> >>>>> On Thu, Jan 20, 2022 at 1:11 PM 张铎(Duo Zhang) < >> >>>> [email protected]> >> >>>> >>>>> wrote: >> >>>> >>>>> >> >>>> >>>>>> I’ve also replied on the jira issue. >> >>>> >>>>>> >> >>>> >>>>>> You can try to copy a job to the new Jenkins. >> >>>> >>>>>> >> >>>> >>>>>> Thanks. >> >>>> >>>>>> >> >>>> >>>>>> Gavin McDonald <[email protected]>于2022年1月20日 周四18:45写道： >> >>>> >>>>>> >> >>>> >>>>>>> Im taking a look ... >> >>>> >>>>>>> >> >>>> >>>>>>> On Thu, Jan 20, 2022 at 6:54 AM Sean Busbey < >> [email protected]> >> >>>> >>>>>>> wrote: >> >>>> >>>>>>> >> >>>> >>>>>>> > Hi Gavin >> >>>> >>>>>>> > >> >>>> >>>>>>> > I went to start migrating jobs to the ci-hbase controller, >> >>>> but the >> >>>> >>>>>>> new job >> >>>> >>>>>>> > workflow doesn't include pipelines or multibranch >> pipelines. >> >>>> We >> >>>> >>>>>>> use both of >> >>>> >>>>>>> > these job types for nearly all of our work. Can they be >> >>>> enabled? >> >>>> >>>>>>> > >> >>>> >>>>>>> > On Tue, Jan 18, 2022 at 5:04 AM Gavin McDonald < >> >>>> >>>>>>> [email protected]> >> >>>> >>>>>>> > wrote: >> >>>> >>>>>>> > >> >>>> >>>>>>> >> >> >>>> >>>>>>> >> >> >>>> >>>>>>> >> On 2022/01/18 09:20:24 "张铎(Duo Zhang)" wrote: >> >>>> >>>>>>> >> > Thank you Gavin. >> >>>> >>>>>>> >> > >> >>>> >>>>>>> >> > We have some urls in our jenkins files which references >> the >> >>>> >>>>>>> artifacts of >> >>>> >>>>>>> >> > the jenkins jobs, so a simple copy or move can not work. >> >>>> For >> >>>> >>>>>>> example, >> >>>> >>>>>>> >> here: >> >>>> >>>>>>> >> > >> >>>> >>>>>>> >> > >> >>>> >>>>>>> >> >> >>>> >>>>>>> >> >>>> >> https://github.com/apache/hbase/blob/c9bcd87b34a15d200a55ec7fdc2b1d86e3367a8c/dev-support/Jenkinsfile_GitHub#L47 >> >>>> >>>>>>> >> > >> >>>> >>>>>>> >> > So I prefer we migrate the jobs by ourselves. Will file >> a >> >>>> HBase >> >>>> >>>>>>> issue >> >>>> >>>>>>> >> to do >> >>>> >>>>>>> >> > this. >> >>>> >>>>>>> >> > >> >>>> >>>>>>> >> > And will report any problems on INFRA_22760 while >> >>>> migrating. >> >>>> >>>>>>> When the >> >>>> >>>>>>> >> > migration is half done, you can move for example, 5 >> hbase >> >>>> nodes >> >>>> >>>>>>> from >> >>>> >>>>>>> >> > ci-hadoop to ci-hbase, and once the migration is finally >> >>>> done, >> >>>> >>>>>>> you can >> >>>> >>>>>>> >> move >> >>>> >>>>>>> >> > all the remaining hbase nodes from ci-hadoop to >> ci-hbase. >> >>>> >>>>>>> >> > >> >>>> >>>>>>> >> > WDYT? >> >>>> >>>>>>> >> >> >>>> >>>>>>> >> However you want to do it is fine with me :) >> >>>> >>>>>>> >> >> >>>> >>>>>>> >> You have 10 nodes on there right now to get you started. >> >>>> >>>>>>> >> >> >>>> >>>>>>> >> Let me know how you get on >> >>>> >>>>>>> >> >> >>>> >>>>>>> >> > >> >>>> >>>>>>> >> > Thanks. >> >>>> >>>>>>> >> > >> >>>> >>>>>>> >> > Gavin McDonald <[email protected]> 于2022年1月18日周二 >> >>>> 16:16写道： >> >>>> >>>>>>> >> > >> >>>> >>>>>>> >> > > Hi All, >> >>>> >>>>>>> >> > > >> >>>> >>>>>>> >> > > I created an Infra ticket for tracking. >> >>>> >>>>>>> >> > > >> >>>> >>>>>>> >> > > https://issues.apache.org/jira/browse/INFRA-22760 >> >>>> >>>>>>> >> > > >> >>>> >>>>>>> >> > > For now, I left your current nodes hbase1 to hbase10 >> in >> >>>> >>>>>>> ci-hadoop.a.o >> >>>> >>>>>>> >> > > >> >>>> >>>>>>> >> > > I have created for you 10 new nodes so far out of our >> >>>> Hetzner >> >>>> >>>>>>> DC >> >>>> >>>>>>> >> > > of a similar specifications to the hbase1-10. >> >>>> >>>>>>> >> > > >> >>>> >>>>>>> >> > > Those are hbase11-20 and are available right now for >> >>>> testing >> >>>> >>>>>>> on >> >>>> >>>>>>> >> > > ci-hbase.apache.org >> >>>> >>>>>>> >> > > >> >>>> >>>>>>> >> > > I do not know your workflow well enough to decide the >> >>>> next >> >>>> >>>>>>> steps for >> >>>> >>>>>>> >> you >> >>>> >>>>>>> >> > > for testing. >> >>>> >>>>>>> >> > > >> >>>> >>>>>>> >> > > Options include: >> >>>> >>>>>>> >> > > >> >>>> >>>>>>> >> > > 1. I can 'move' all jobs out of ci-hadoop onto >> ci-hbase >> >>>> >>>>>>> >> > > 2. I can 'move' one or a number of jobs out of >> ci-hadoop >> >>>> onto >> >>>> >>>>>>> ci-hbase >> >>>> >>>>>>> >> > > 3. Similar to 1 or 2, but 'copy' any number of jobs >> over >> >>>> for >> >>>> >>>>>>> testing. >> >>>> >>>>>>> >> > > - I like copy, however quite a few projects jobs >> >>>> would >> >>>> >>>>>>> not be >> >>>> >>>>>>> >> > > appropriate >> >>>> >>>>>>> >> > > to have the same job running twice, including >> >>>> those that >> >>>> >>>>>>> >> deploy, or >> >>>> >>>>>>> >> > > those that >> >>>> >>>>>>> >> > > build from PRs and continue a workflow to report >> >>>> back, >> >>>> >>>>>>> or >> >>>> >>>>>>> >> approve >> >>>> >>>>>>> >> > > for merge. >> >>>> >>>>>>> >> > > So for a copy, would need to be a job that just >> >>>> 'builds >> >>>> >>>>>>> and >> >>>> >>>>>>> >> tests' >> >>>> >>>>>>> >> > > with no outflow. >> >>>> >>>>>>> >> > > 4. You guys create new jobs on ci-hbase.apache.org to >> >>>> test >> >>>> >>>>>>> the nodes >> >>>> >>>>>>> >> > > setup and the >> >>>> >>>>>>> >> > > main controller setup, without affecting current >> >>>> jobs. >> >>>> >>>>>>> >> > > >> >>>> >>>>>>> >> > > Obviously, you may to tweak version controlled script >> >>>> files >> >>>> >>>>>>> such as a >> >>>> >>>>>>> >> > > JenkinsFile, which makes the copy options harder >> unless >> >>>> >>>>>>> testing from a >> >>>> >>>>>>> >> > > branch created for the purpose. >> >>>> >>>>>>> >> > > >> >>>> >>>>>>> >> > > I may be overthinking things, and option 1 , the >> >>>> simplest and >> >>>> >>>>>>> most >> >>>> >>>>>>> >> > > straight forward from my end would also suit you guys >> >>>> and we >> >>>> >>>>>>> deal >> >>>> >>>>>>> >> with any >> >>>> >>>>>>> >> > > breakage as it happens (I'm thinking of integrations >> with >> >>>> >>>>>>> other >> >>>> >>>>>>> >> services, >> >>>> >>>>>>> >> > > Credentials/Secrets, Plugins I haven't installed that >> are >> >>>> >>>>>>> needed etc.) >> >>>> >>>>>>> >> > > >> >>>> >>>>>>> >> > > Let me know please ASAP how you want to proceed so we >> >>>> can get >> >>>> >>>>>>> using >> >>>> >>>>>>> >> these >> >>>> >>>>>>> >> > > nodes, after which I can then proceed to move >> hbase1-10 >> >>>> nodes >> >>>> >>>>>>> over >> >>>> >>>>>>> >> also. >> >>>> >>>>>>> >> > > And then start adding more into the mix. >> >>>> >>>>>>> >> > > >> >>>> >>>>>>> >> > > Oh. The 'git-websites' label should be a no-op. It is >> >>>> >>>>>>> available on >> >>>> >>>>>>> >> > > ci-hbase.a.o as the nodes with that label are shared >> and >> >>>> >>>>>>> leased out to >> >>>> >>>>>>> >> > > controllers as needed upon request of a job. >> >>>> >>>>>>> >> > > >> >>>> >>>>>>> >> > > Thanks >> >>>> >>>>>>> >> > > >> >>>> >>>>>>> >> > > Gav... >> >>>> >>>>>>> >> > > >> >>>> >>>>>>> >> > > >> >>>> >>>>>>> >> > > On 2021/12/09 15:07:26 "张铎(Duo Zhang)" wrote: >> >>>> >>>>>>> >> > > > Thanks you Gavin. >> >>>> >>>>>>> >> > > > >> >>>> >>>>>>> >> > > > You can do it in your convenience, just let me know >> so >> >>>> I >> >>>> >>>>>>> could >> >>>> >>>>>>> >> change the >> >>>> >>>>>>> >> > > > label in the jenkinsfile of our jenkins jobs, at >> least >> >>>> our >> >>>> >>>>>>> pre >> >>>> >>>>>>> >> commit job >> >>>> >>>>>>> >> > > > still uses the Hadoop label. >> >>>> >>>>>>> >> > > > >> >>>> >>>>>>> >> > > > And another thing is about the >> hbase_generate_website >> >>>> job, >> >>>> >>>>>>> it needs >> >>>> >>>>>>> >> to be >> >>>> >>>>>>> >> > > > executed on the nodes under the git-websites label. >> >>>> I'm not >> >>>> >>>>>>> sure if >> >>>> >>>>>>> >> the >> >>>> >>>>>>> >> > > > label is still available after the migration. >> >>>> >>>>>>> >> > > > >> >>>> >>>>>>> >> > > > Gavin McDonald <[email protected]> 于2021年12月8日周三 >> >>>> >>>>>>> 17:28写道： >> >>>> >>>>>>> >> > > > >> >>>> >>>>>>> >> > > > > Hi HBase devs. >> >>>> >>>>>>> >> > > > > >> >>>> >>>>>>> >> > > > > This is to let you know that Infra has created a >> >>>> >>>>>>> dedicated Jenkins >> >>>> >>>>>>> >> > > > > Controller at https://ci-hbase.apache.org . >> >>>> >>>>>>> >> > > > > >> >>>> >>>>>>> >> > > > > You have new incoming donated agents which will >> get >> >>>> >>>>>>> connected to >> >>>> >>>>>>> >> this >> >>>> >>>>>>> >> > > new >> >>>> >>>>>>> >> > > > > controller by Infra. >> >>>> >>>>>>> >> > > > > >> >>>> >>>>>>> >> > > > > You also have 10 or so agents connected to >> >>>> >>>>>>> >> > > https://ci-hadoop.apache.org - >> >>>> >>>>>>> >> > > > > I >> >>>> >>>>>>> >> > > > > would like to move these agents - and all of your >> >>>> jobs - >> >>>> >>>>>>> over to >> >>>> >>>>>>> >> the >> >>>> >>>>>>> >> > > new >> >>>> >>>>>>> >> > > > > controller as soon as possible. >> >>>> >>>>>>> >> > > > > >> >>>> >>>>>>> >> > > > > You may or may not need to reconfigure your jobs >> and >> >>>> or >> >>>> >>>>>>> >> JenkinsFile or >> >>>> >>>>>>> >> > > > > other. >> >>>> >>>>>>> >> > > > > >> >>>> >>>>>>> >> > > > > Can I get the go ahead to move these at my >> >>>> convenience or >> >>>> >>>>>>> suggest >> >>>> >>>>>>> >> > > please a >> >>>> >>>>>>> >> > > > > timeline for Infra to move these agents. >> >>>> >>>>>>> >> > > > > >> >>>> >>>>>>> >> > > > > Thanks! >> >>>> >>>>>>> >> > > > > >> >>>> >>>>>>> >> > > > > -- >> >>>> >>>>>>> >> > > > > >> >>>> >>>>>>> >> > > > > *Gavin McDonald* >> >>>> >>>>>>> >> > > > > Systems Administrator >> >>>> >>>>>>> >> > > > > ASF Infrastructure Team >> >>>> >>>>>>> >> > > > > >> >>>> >>>>>>> >> > > > >> >>>> >>>>>>> >> > > >> >>>> >>>>>>> >> > >> >>>> >>>>>>> >> >> >>>> >>>>>>> > >> >>>> >>>>>>> >> >>>> >>>>>>> -- >> >>>> >>>>>>> >> >>>> >>>>>>> *Gavin McDonald* >> >>>> >>>>>>> Systems Administrator >> >>>> >>>>>>> ASF Infrastructure Team >> >>>> >>>>>>> >> >>>> >>>>>> >> >>>> >>>>> >> >>>> >>>>> -- >> >>>> >>>>> >> >>>> >>>>> *Gavin McDonald* >> >>>> >>>>> Systems Administrator >> >>>> >>>>> ASF Infrastructure Team >> >>>> >>>>> >> >>>> >>>> >> >>>> >> >>>> -- >> >>>> >> >>>> *Gavin McDonald* >> >>>> Systems Administrator >> >>>> ASF Infrastructure Team >> >>>> >> >>> >> >> -- >> >> *Gavin McDonald* >> Systems Administrator >> ASF Infrastructure Team >> >
