Re: Splitting Jenkins pipelines - stop changes to Jenkinsfiles!

2018-11-21 Thread Marco de Abreu
Hi Anirudh, I have just deployed a fix for this. The status reports are now back to where they were: - "continuous-integration/jenkins/pr-merge" is the pipeline everybody is used and backed by the main Jenkinsfile -> This is the one you have to look at - "ci/jenkins/mxnet-validation/XXX" this is

Re: Splitting Jenkins pipelines - stop changes to Jenkinsfiles!

2018-11-21 Thread Anirudh
Hi Marco, Can you point out specifically which checks we have to make sure pass before merging PRs. Currently apart from the required one there are six steps added. Also, is the CI down currently : http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/PR-13324/17

Re: Splitting Jenkins pipelines - stop changes to Jenkinsfiles!

2018-11-21 Thread Marco de Abreu
Please notice that the "continuous-integration/jenkins/pr-merge" currently is overlapping with the new pipelines. Please make sure all checks pass (also the non-required ones) before merging the PRs. I will work on a fix for this overlap. -Marco On Wed, Nov 21, 2018 at 5:42 PM Anton Chernov wrot

Re: Splitting Jenkins pipelines - stop changes to Jenkinsfiles!

2018-11-21 Thread Anton Chernov
The ability to retrigger the pipelines separately is an amazing step forward. Great job Marco! ср, 21 нояб. 2018 г. в 15:03, Marco de Abreu : > Hello, > > the PR has been merged and I've created the new pipelines at [1]. You can > see the new reports if you have a look at this example PR at [2].

Re: Splitting Jenkins pipelines - stop changes to Jenkinsfiles!

2018-11-21 Thread Marco de Abreu
Hello, the PR has been merged and I've created the new pipelines at [1]. You can see the new reports if you have a look at this example PR at [2]. The new status messages will be the ones starting with "ci/jenkins/mxnet-validation/". This now allows you to retrigger specific pipelines if they fa

Re: Splitting Jenkins pipelines - stop changes to Jenkinsfiles!

2018-11-20 Thread Marco de Abreu
I have just submitted my PR at https://github.com/apache/incubator-mxnet/pull/13344. Test jobs are available at http://jenkins.mxnet-ci-dev.amazon-ml.com/view/test-marco-mxnet/. As soon as I'm done with my tests, I will mark it as ready for review. Best regards, Marco On Tue, Nov 20, 2018 at 9:0

Re: Splitting Jenkins pipelines - stop changes to Jenkinsfiles!

2018-11-20 Thread Marco de Abreu
Thanks, Pedro! I have also been looking into that issue, but it seems like this would require changes in the groovy interpreter of Jenkins. From what I can tell, a refactor will give us multiple benefits (clarity and speed) aside from resolving this issue. Best regards, Marco Am Di., 20. Nov. 20

Re: Splitting Jenkins pipelines - stop changes to Jenkinsfiles!

2018-11-20 Thread Pedro Larroy
I think this is a big problem, which has blocked us before. I want to point out that you are doing a great thing by avoiding everyone getting blocked by refactoring the pipelines. My concern is that we are kicking the can down the road and not addressing the root cause of the problem with is known

Re: Splitting Jenkins pipelines - stop changes to Jenkinsfiles!

2018-11-20 Thread Marco de Abreu
Hello Steffen, no, there won't be any impact on the PR process or nightly regressions. Only the reporting will have to be updated with the new job links, but that should be a minor issue. To avoid any outage, I have been thinking about running both versions in parallel. Best regards, Marco On

Re: Splitting Jenkins pipelines - stop changes to Jenkinsfiles!

2018-11-20 Thread Steffen Rochel
Hi Marco - is there any impact on reporting, the PR process or nightly regression beside reduction in TAT? If yes, please elaborate. Steffen On Tue, Nov 20, 2018 at 8:05 AM Marco de Abreu wrote: > Hello, > > we ran into issues around the maximum filesize of the Jenkinsfile a few > times already

Splitting Jenkins pipelines - stop changes to Jenkinsfiles!

2018-11-20 Thread Marco de Abreu
Hello, we ran into issues around the maximum filesize of the Jenkinsfile a few times already. In order to resolve this issue, I'd like to combine this with some refactors I have planned for quite some time. The idea is basically to move away from one big Jenkinsfile and instead split it into sepa