Hi, Gavin, any luck on the ssh transfer plugin? Thanks.
张铎(Duo Zhang) <[email protected]> 于2022年1月29日周六 14:53写道： > So any updates here Gavin? > > Is the ssh transfer plugin available now? > > Thanks. > > Sean Busbey <[email protected]> 于2022年1月23日周日 09:04写道： > >> It looks like I currently do have access to manage the ci-hbase >> controller. >> >> >> Could you check that the git-websites label handling where a common >> machine gets leased to the ci-hbase controller is set up properly? or give >> me a pointer on how I would check this myself? >> >> I started trying to migrate our website generation job again and it's >> been waiting for a node with the git-websites label to show up for ~8 hours >> at this point. >> >> https://ci-hbase.apache.org/job/hbase_generate_website/3/ >> >> >> >> On Sat, Jan 22, 2022 at 2:42 AM Gavin McDonald <[email protected]> >> wrote: >> >>> Hi, >>> Timestamper plugin installed. >>> >>> By the way I thought I had given the PMC members permissions to manage >>> Jenkins >>> on ci-hbase including installing plugins etc, can someone check? >>> >>> >>> On Sat, Jan 22, 2022 at 7:34 AM 张铎(Duo Zhang) <[email protected]> >>> wrote: >>> >>> > We faced a strange issue.. >>> > >>> > rg.codehaus.groovy.control.MultipleCompilationErrorsException: startup >>> failed: >>> > WorkflowScript: 29: Invalid option type "timestamps". Valid option >>> types: [buildDiscarder, catchError, checkoutToSubdirectory, >>> disableConcurrentBuilds, disableResume, durabilityHint, >>> githubProjectProperty, newContainerPerStage, overrideIndexTriggers, >>> parallelsAlwaysFailFast, preserveStashes, quietPeriod, rateLimitBuilds, >>> retry, script, skipDefaultCheckout, skipStagesAfterUnstable, timeout, >>> waitUntil, warnError, withChecks, withContext, withCredentials, withEnv, >>> wrap, ws] @ line 29, column 5. >>> > timestamps() >>> > ^ >>> > >>> > 1 error >>> > >>> > at >>> org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:310) >>> > at >>> org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1085) >>> > at >>> org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:603) >>> > at >>> org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581) >>> > at >>> org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558) >>> > at >>> groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298) >>> > at >>> groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268) >>> > at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688) >>> > at groovy.lang.GroovyShell.parse(GroovyShell.java:700) >>> > at >>> org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:142) >>> > at >>> org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:127) >>> > at >>> org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:571) >>> > at >>> org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:523) >>> > at >>> org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:334) >>> > at >>> hudson.model.ResourceController.execute(ResourceController.java:97) >>> > at hudson.model.Executor.run(Executor.java:429) >>> > >>> > >>> > After googling, it seems a restart of jenkins can fix it, or we do not >>> > have timestamper plugin installed? Mind taking a look Gavin? >>> > >>> > Thanks. >>> > >>> > 张铎(Duo Zhang) <[email protected]> 于2022年1月22日周六 11:40写道： >>> > >>> >> Oh, just saw that we could do this by ourselves. >>> >> >>> >> Let me have a try. >>> >> >>> >> 张铎(Duo Zhang) <[email protected]> 于2022年1月22日周六 11:18写道： >>> >> >>> >>> OK, so while copying you could keep the old jenkins builds? >>> >>> That's awesome. >>> >>> >>> >>> Then please help 'copying' HBase-Find-Flaky-Tests and >>> HBase-Flaky-Tests >>> >>> to ci-hbase too? We need the history builds to calucate the flaky >>> test list >>> >>> so if all the old build data can be kept then it will help us a lot. >>> >>> >>> >>> Thanks~ >>> >>> >>> >>> 张铎(Duo Zhang) <[email protected]> 于2022年1月22日周六 11:03写道： >>> >>> >>> >>>> Thank you Gavin. >>> >>>> >>> >>>> Let me first move some light weighted job first to see if >>> everything is >>> >>>> OK. >>> >>>> >>> >>>> HBase Nightly, some flaky related jobs, and also the main pre commit >>> >>>> job for HBase are tied together. We need to move them at once. >>> >>>> >>> >>>> Gavin McDonald <[email protected]> 于2022年1月22日周六 02:44写道： >>> >>>> >>> >>>>> Thanks for your patience >>> >>>>> >>> >>>>> I have 'copied' over a job - >>> >>>>> https://ci-hbase.apache.org/job/HBase%20Nightly/ >>> >>>>> >>> >>>>> which of course made me aware of what plugins I needed to install. >>> >>>>> >>> >>>>> Feel free to delete that, or use it and delete the old, and let me >>> >>>>> know if have any issues >>> >>>>> with any other missing plugins. >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> On Thu, Jan 20, 2022 at 1:11 PM 张铎(Duo Zhang) < >>> [email protected]> >>> >>>>> wrote: >>> >>>>> >>> >>>>>> I’ve also replied on the jira issue. >>> >>>>>> >>> >>>>>> You can try to copy a job to the new Jenkins. >>> >>>>>> >>> >>>>>> Thanks. >>> >>>>>> >>> >>>>>> Gavin McDonald <[email protected]>于2022年1月20日 周四18:45写道： >>> >>>>>> >>> >>>>>>> Im taking a look ... >>> >>>>>>> >>> >>>>>>> On Thu, Jan 20, 2022 at 6:54 AM Sean Busbey <[email protected]> >>> >>>>>>> wrote: >>> >>>>>>> >>> >>>>>>> > Hi Gavin >>> >>>>>>> > >>> >>>>>>> > I went to start migrating jobs to the ci-hbase controller, but >>> the >>> >>>>>>> new job >>> >>>>>>> > workflow doesn't include pipelines or multibranch pipelines. We >>> >>>>>>> use both of >>> >>>>>>> > these job types for nearly all of our work. Can they be >>> enabled? >>> >>>>>>> > >>> >>>>>>> > On Tue, Jan 18, 2022 at 5:04 AM Gavin McDonald < >>> >>>>>>> [email protected]> >>> >>>>>>> > wrote: >>> >>>>>>> > >>> >>>>>>> >> >>> >>>>>>> >> >>> >>>>>>> >> On 2022/01/18 09:20:24 "张铎(Duo Zhang)" wrote: >>> >>>>>>> >> > Thank you Gavin. >>> >>>>>>> >> > >>> >>>>>>> >> > We have some urls in our jenkins files which references the >>> >>>>>>> artifacts of >>> >>>>>>> >> > the jenkins jobs, so a simple copy or move can not work. For >>> >>>>>>> example, >>> >>>>>>> >> here: >>> >>>>>>> >> > >>> >>>>>>> >> > >>> >>>>>>> >> >>> >>>>>>> >>> https://github.com/apache/hbase/blob/c9bcd87b34a15d200a55ec7fdc2b1d86e3367a8c/dev-support/Jenkinsfile_GitHub#L47 >>> >>>>>>> >> > >>> >>>>>>> >> > So I prefer we migrate the jobs by ourselves. Will file a >>> HBase >>> >>>>>>> issue >>> >>>>>>> >> to do >>> >>>>>>> >> > this. >>> >>>>>>> >> > >>> >>>>>>> >> > And will report any problems on INFRA_22760 while migrating. >>> >>>>>>> When the >>> >>>>>>> >> > migration is half done, you can move for example, 5 hbase >>> nodes >>> >>>>>>> from >>> >>>>>>> >> > ci-hadoop to ci-hbase, and once the migration is finally >>> done, >>> >>>>>>> you can >>> >>>>>>> >> move >>> >>>>>>> >> > all the remaining hbase nodes from ci-hadoop to ci-hbase. >>> >>>>>>> >> > >>> >>>>>>> >> > WDYT? >>> >>>>>>> >> >>> >>>>>>> >> However you want to do it is fine with me :) >>> >>>>>>> >> >>> >>>>>>> >> You have 10 nodes on there right now to get you started. >>> >>>>>>> >> >>> >>>>>>> >> Let me know how you get on >>> >>>>>>> >> >>> >>>>>>> >> > >>> >>>>>>> >> > Thanks. >>> >>>>>>> >> > >>> >>>>>>> >> > Gavin McDonald <[email protected]> 于2022年1月18日周二 >>> 16:16写道： >>> >>>>>>> >> > >>> >>>>>>> >> > > Hi All, >>> >>>>>>> >> > > >>> >>>>>>> >> > > I created an Infra ticket for tracking. >>> >>>>>>> >> > > >>> >>>>>>> >> > > https://issues.apache.org/jira/browse/INFRA-22760 >>> >>>>>>> >> > > >>> >>>>>>> >> > > For now, I left your current nodes hbase1 to hbase10 in >>> >>>>>>> ci-hadoop.a.o >>> >>>>>>> >> > > >>> >>>>>>> >> > > I have created for you 10 new nodes so far out of our >>> Hetzner >>> >>>>>>> DC >>> >>>>>>> >> > > of a similar specifications to the hbase1-10. >>> >>>>>>> >> > > >>> >>>>>>> >> > > Those are hbase11-20 and are available right now for >>> testing >>> >>>>>>> on >>> >>>>>>> >> > > ci-hbase.apache.org >>> >>>>>>> >> > > >>> >>>>>>> >> > > I do not know your workflow well enough to decide the next >>> >>>>>>> steps for >>> >>>>>>> >> you >>> >>>>>>> >> > > for testing. >>> >>>>>>> >> > > >>> >>>>>>> >> > > Options include: >>> >>>>>>> >> > > >>> >>>>>>> >> > > 1. I can 'move' all jobs out of ci-hadoop onto ci-hbase >>> >>>>>>> >> > > 2. I can 'move' one or a number of jobs out of ci-hadoop >>> onto >>> >>>>>>> ci-hbase >>> >>>>>>> >> > > 3. Similar to 1 or 2, but 'copy' any number of jobs over >>> for >>> >>>>>>> testing. >>> >>>>>>> >> > > - I like copy, however quite a few projects jobs would >>> >>>>>>> not be >>> >>>>>>> >> > > appropriate >>> >>>>>>> >> > > to have the same job running twice, including those >>> that >>> >>>>>>> >> deploy, or >>> >>>>>>> >> > > those that >>> >>>>>>> >> > > build from PRs and continue a workflow to report >>> back, >>> >>>>>>> or >>> >>>>>>> >> approve >>> >>>>>>> >> > > for merge. >>> >>>>>>> >> > > So for a copy, would need to be a job that just >>> 'builds >>> >>>>>>> and >>> >>>>>>> >> tests' >>> >>>>>>> >> > > with no outflow. >>> >>>>>>> >> > > 4. You guys create new jobs on ci-hbase.apache.org to >>> test >>> >>>>>>> the nodes >>> >>>>>>> >> > > setup and the >>> >>>>>>> >> > > main controller setup, without affecting current jobs. >>> >>>>>>> >> > > >>> >>>>>>> >> > > Obviously, you may to tweak version controlled script >>> files >>> >>>>>>> such as a >>> >>>>>>> >> > > JenkinsFile, which makes the copy options harder unless >>> >>>>>>> testing from a >>> >>>>>>> >> > > branch created for the purpose. >>> >>>>>>> >> > > >>> >>>>>>> >> > > I may be overthinking things, and option 1 , the simplest >>> and >>> >>>>>>> most >>> >>>>>>> >> > > straight forward from my end would also suit you guys and >>> we >>> >>>>>>> deal >>> >>>>>>> >> with any >>> >>>>>>> >> > > breakage as it happens (I'm thinking of integrations with >>> >>>>>>> other >>> >>>>>>> >> services, >>> >>>>>>> >> > > Credentials/Secrets, Plugins I haven't installed that are >>> >>>>>>> needed etc.) >>> >>>>>>> >> > > >>> >>>>>>> >> > > Let me know please ASAP how you want to proceed so we can >>> get >>> >>>>>>> using >>> >>>>>>> >> these >>> >>>>>>> >> > > nodes, after which I can then proceed to move hbase1-10 >>> nodes >>> >>>>>>> over >>> >>>>>>> >> also. >>> >>>>>>> >> > > And then start adding more into the mix. >>> >>>>>>> >> > > >>> >>>>>>> >> > > Oh. The 'git-websites' label should be a no-op. It is >>> >>>>>>> available on >>> >>>>>>> >> > > ci-hbase.a.o as the nodes with that label are shared and >>> >>>>>>> leased out to >>> >>>>>>> >> > > controllers as needed upon request of a job. >>> >>>>>>> >> > > >>> >>>>>>> >> > > Thanks >>> >>>>>>> >> > > >>> >>>>>>> >> > > Gav... >>> >>>>>>> >> > > >>> >>>>>>> >> > > >>> >>>>>>> >> > > On 2021/12/09 15:07:26 "张铎(Duo Zhang)" wrote: >>> >>>>>>> >> > > > Thanks you Gavin. >>> >>>>>>> >> > > > >>> >>>>>>> >> > > > You can do it in your convenience, just let me know so I >>> >>>>>>> could >>> >>>>>>> >> change the >>> >>>>>>> >> > > > label in the jenkinsfile of our jenkins jobs, at least >>> our >>> >>>>>>> pre >>> >>>>>>> >> commit job >>> >>>>>>> >> > > > still uses the Hadoop label. >>> >>>>>>> >> > > > >>> >>>>>>> >> > > > And another thing is about the hbase_generate_website >>> job, >>> >>>>>>> it needs >>> >>>>>>> >> to be >>> >>>>>>> >> > > > executed on the nodes under the git-websites label. I'm >>> not >>> >>>>>>> sure if >>> >>>>>>> >> the >>> >>>>>>> >> > > > label is still available after the migration. >>> >>>>>>> >> > > > >>> >>>>>>> >> > > > Gavin McDonald <[email protected]> 于2021年12月8日周三 >>> >>>>>>> 17:28写道： >>> >>>>>>> >> > > > >>> >>>>>>> >> > > > > Hi HBase devs. >>> >>>>>>> >> > > > > >>> >>>>>>> >> > > > > This is to let you know that Infra has created a >>> >>>>>>> dedicated Jenkins >>> >>>>>>> >> > > > > Controller at https://ci-hbase.apache.org . >>> >>>>>>> >> > > > > >>> >>>>>>> >> > > > > You have new incoming donated agents which will get >>> >>>>>>> connected to >>> >>>>>>> >> this >>> >>>>>>> >> > > new >>> >>>>>>> >> > > > > controller by Infra. >>> >>>>>>> >> > > > > >>> >>>>>>> >> > > > > You also have 10 or so agents connected to >>> >>>>>>> >> > > https://ci-hadoop.apache.org - >>> >>>>>>> >> > > > > I >>> >>>>>>> >> > > > > would like to move these agents - and all of your >>> jobs - >>> >>>>>>> over to >>> >>>>>>> >> the >>> >>>>>>> >> > > new >>> >>>>>>> >> > > > > controller as soon as possible. >>> >>>>>>> >> > > > > >>> >>>>>>> >> > > > > You may or may not need to reconfigure your jobs and >>> or >>> >>>>>>> >> JenkinsFile or >>> >>>>>>> >> > > > > other. >>> >>>>>>> >> > > > > >>> >>>>>>> >> > > > > Can I get the go ahead to move these at my >>> convenience or >>> >>>>>>> suggest >>> >>>>>>> >> > > please a >>> >>>>>>> >> > > > > timeline for Infra to move these agents. >>> >>>>>>> >> > > > > >>> >>>>>>> >> > > > > Thanks! >>> >>>>>>> >> > > > > >>> >>>>>>> >> > > > > -- >>> >>>>>>> >> > > > > >>> >>>>>>> >> > > > > *Gavin McDonald* >>> >>>>>>> >> > > > > Systems Administrator >>> >>>>>>> >> > > > > ASF Infrastructure Team >>> >>>>>>> >> > > > > >>> >>>>>>> >> > > > >>> >>>>>>> >> > > >>> >>>>>>> >> > >>> >>>>>>> >> >>> >>>>>>> > >>> >>>>>>> >>> >>>>>>> -- >>> >>>>>>> >>> >>>>>>> *Gavin McDonald* >>> >>>>>>> Systems Administrator >>> >>>>>>> ASF Infrastructure Team >>> >>>>>>> >>> >>>>>> >>> >>>>> >>> >>>>> -- >>> >>>>> >>> >>>>> *Gavin McDonald* >>> >>>>> Systems Administrator >>> >>>>> ASF Infrastructure Team >>> >>>>> >>> >>>> >>> >>> -- >>> >>> *Gavin McDonald* >>> Systems Administrator >>> ASF Infrastructure Team >>> >>
