Re: Oak Jenkins build takes up too many resources

[Marcel Reutegger]

On 16.12.22, 20:01, "Konrad Windszus"  wrote:
> @Marcel: Are you still looking into optimizations?

No, I’m not.

Regards
Marcel

Re: Oak Jenkins build takes up too many resources

[Konrad Windszus]

Maybe adding some logic to only build modified modules helps like outlined in 
https://github.com/mrguamos/jenkins-pipeline-maven-multi-module/blob/master/Jenkinsfile
 
.
Hard though to figure out transitive dependencies with that.

Another approach is the new 
https://maven.apache.org/extensions/maven-build-cache-extension/ 
 (see also 
https://www.youtube.com/watch?v=DEQG4CNFMFE 
) which should also gain quite 
some improvements (I don’t have any experience though with it). 


> On 16. Dec 2022, at 19:59, Konrad Windszus  wrote:
> 
> The situation is still pretty bad. Right now Oak jobs take up roughly 300 
> jobs of total 450 in the queue. We should ASAP switch to incremental builds 
> (i.e. only trigger builds for modules which really have changed) and no 
> longer trigger 100s of build jobs in parallel for a single branch!
> 
> @Marcel: Are you still looking into optimizations?

Re: Oak Jenkins build takes up too many resources

[Konrad Windszus]

The situation is still pretty bad. Right now Oak jobs take up roughly 300 jobs 
of total 450 in the queue. We should ASAP switch to incremental builds (i.e. 
only trigger builds for modules which really have changed) and no longer 
trigger 100s of build jobs in parallel for a single branch!

@Marcel: Are you still looking into optimizations?

Re: Oak Jenkins build takes up too many resources

[Robert Munteanu]

Hi Marcel,
Hi Konrad,

Thanks for looking into this, some good improvements are on the way,
and it seems like the problem severity was inflated by the way Jenkins
reports build step duration.

Thanks,
Robert

On Tue, 2022-11-15 at 10:56 +0100, Konrad Windszus wrote:
> Probably it would also be fair to not parallelise all modules (i.e.
> reserve only handful number of slots instead of almost all
> available).
> Currently we trigger 35 builds at the same time:
> https://github.com/apache/jackrabbit-oak/blob/6c04951be723c4574d3a7c1bcc4ec04e9f9e7dd0/Jenkinsfile#L23
>  <
> https://github.com/apache/jackrabbit-oak/blob/6c04951be723c4574d3a7c1bcc4ec04e9f9e7dd0/Jenkinsfile#L23
> > while ASF Jenkins has only about the same amount of slots.
> I think we should block at most a handful of nodes in parallel
> (although this may lead to slightly slower build times).
> 
> Konrad
> 
> > On 14. Nov 2022, at 16:29, Konrad Windszus  wrote:
> > 
> > Seems this is pretty easy:
> > https://stackoverflow.com/a/70375236  > >
> > Can you come up with a PR?
> > Thanks,
> > Konrad
> > 
> > > On 14. Nov 2022, at 16:16, Marcel Reutegger
> > >  wrote:
> > > 
> > > Hi,
> > > 
> > > On 11.11.22, 15:11, "Robert Munteanu"  wrote:
> > > > I think it would be worthile to check if the build can be
> > > > optimised,
> > > > otherwise Oak builds are blocking many execution slots of the
> > > > Jenkins
> > > > ASF instance.
> > > 
> > > I did notice one thing. Our PRs may schedule a build each time
> > > commits
> > > are pushed to the branch. Is there a way to cancel an already
> > > running
> > > build when there are new changes coming in from a PR?
> > > 
> > > Regards
> > > Marcel
> > 
> 

Re: Oak Jenkins build takes up too many resources

[Marcel Reutegger]

Hi,

On 11.11.22, 15:11, "Robert Munteanu"  wrote:
> I think it would be worthile to check if the build can be optimised

FYI, I also changed the Job configuration to not build PRs in draft state.

Regards
Marcel

Re: Oak Jenkins build takes up too many resources

[Konrad Windszus]

Probably it would also be fair to not parallelise all modules (i.e. reserve 
only handful number of slots instead of almost all available).
Currently we trigger 35 builds at the same time: 
https://github.com/apache/jackrabbit-oak/blob/6c04951be723c4574d3a7c1bcc4ec04e9f9e7dd0/Jenkinsfile#L23
 

 while ASF Jenkins has only about the same amount of slots.
I think we should block at most a handful of nodes in parallel (although this 
may lead to slightly slower build times).

Konrad

> On 14. Nov 2022, at 16:29, Konrad Windszus  wrote:
> 
> Seems this is pretty easy: https://stackoverflow.com/a/70375236 
> 
> Can you come up with a PR?
> Thanks,
> Konrad
> 
>> On 14. Nov 2022, at 16:16, Marcel Reutegger  
>> wrote:
>> 
>> Hi,
>> 
>> On 11.11.22, 15:11, "Robert Munteanu"  wrote:
>>> I think it would be worthile to check if the build can be optimised,
>>> otherwise Oak builds are blocking many execution slots of the Jenkins
>>> ASF instance.
>> 
>> I did notice one thing. Our PRs may schedule a build each time commits
>> are pushed to the branch. Is there a way to cancel an already running
>> build when there are new changes coming in from a PR?
>> 
>> Regards
>> Marcel
> 

Re: Oak Jenkins build takes up too many resources

[Marcel Reutegger]

On 14.11.22, 16:29, "Konrad Windszus"  wrote:
> Can you come up with a PR?
See https://github.com/apache/jackrabbit-oak/pull/755
Though, I don’t think the build it triggers actually uses the updated 
Jenkinsfile.
Regards
Marcel

Re: Oak Jenkins build takes up too many resources

[Konrad Windszus]

Seems this is pretty easy: https://stackoverflow.com/a/70375236 

Can you come up with a PR?
Thanks,
Konrad

> On 14. Nov 2022, at 16:16, Marcel Reutegger  
> wrote:
> 
> Hi,
> 
> On 11.11.22, 15:11, "Robert Munteanu"  wrote:
>> I think it would be worthile to check if the build can be optimised,
>> otherwise Oak builds are blocking many execution slots of the Jenkins
>> ASF instance.
> 
> I did notice one thing. Our PRs may schedule a build each time commits
> are pushed to the branch. Is there a way to cancel an already running
> build when there are new changes coming in from a PR?
> 
> Regards
> Marcel

Re: Oak Jenkins build takes up too many resources

[Marcel Reutegger]

Hi,

On 11.11.22, 15:11, "Robert Munteanu"  wrote:
> I think it would be worthile to check if the build can be optimised,
> otherwise Oak builds are blocking many execution slots of the Jenkins
> ASF instance.

I did notice one thing. Our PRs may schedule a build each time commits
are pushed to the branch. Is there a way to cancel an already running
build when there are new changes coming in from a PR?

Regards
Marcel

Re: Oak Jenkins build takes up too many resources

[Marcel Reutegger]

Hi Robert,

On 11.11.22, 15:11, "Robert Munteanu"  wrote:
> I was waiting for a small PR of mine in Sling to be built by Jenkins and
> I saw that it was not picked for some time. Looking at the executing
> jobs I noticed that there were many, different, Oak stages being ran.
>
> A sample PR check [1] shows that there are many stages (one per module?)
> but each of them takes at least 20 minutes, usually over 30.

Yes, each stage builds a module.

I think the durations on this page are misleading. Yes, there are some
modules that take more than 30 minutes, but most are pretty quick.
While the page says the second execution of oak-api stage took 33
minutes, the log files shows it must have run for less than a minute [0].

Maybe the duration includes wait time?

Regards
Marcel

[0] 
https://ci-builds.apache.org/blue/rest/organizations/jenkins/pipelines/Jackrabbit/pipelines/oak-trunk-pr/branches/snfe_during_recovery/runs/2/nodes/54/log/?start=0