Re: RCA of memory conditions on Ubuntu EC2 agents on ci.jenkins.io causing test instability

Azure high mem: Standard_D16_v3 vcpu 16 memory 64 AWS: m5a.xlarge vcpu 4 memory 16 GiB I've changed it to: 'm5a.4xlarge' (16CPU, 64 GB ram) It's 4 times the cost so we'll need to keep an eye on it Thanks Tim On Fri, 12 Jun 2020 at 15:20, Jesse Glick wrote: > On Tue, Jun 9, 2020 at 3:59 AM

Re: RCA of memory conditions on Ubuntu EC2 agents on ci.jenkins.io causing test instability

On Tue, Jun 9, 2020 at 3:59 AM Tim Jacomb wrote: > High mem could possibly do with a change, the AWS ones are much lower spec > than the Azure ones, thoughts? Not sure but I just got an unexplained EC2 (aws) - High memory ubuntu 18.04 (i-0e7f3896526c7922e) was marked offline: Connection was

Re: RCA of memory conditions on Ubuntu EC2 agents on ci.jenkins.io causing test instability

Hi all, I've done the following: * linux docker - was t3.small, now t3a.large (2 core 8gb) * arm64 - was a1.medium, now t3a.large (2 core 8gb) ref: https://aws.amazon.com/ec2/pricing/on-demand/ (region is US East) Let's monitor and see how we go (pricing and performance wise) High mem could

Re: RCA of memory conditions on Ubuntu EC2 agents on ci.jenkins.io causing test instability

True, but I am not super sure that double the memory (e.g., m5a.large over t2.medium) would make a big enough difference for almost double the cost. I could be wrong though, I am definitely not an expert in java optimization, etc. On Fri, Jun 5, 2020 at 3:43 PM 'Gavin Mogan' via Jenkins

Re: RCA of memory conditions on Ubuntu EC2 agents on ci.jenkins.io causing test instability

Remember with more resources the tests can often run faster which reduces how much time the instance is needed for. It's never straight simple math On Fri., Jun. 5, 2020, 3:40 p.m. Slide, wrote: > Just for reference... > > [image: image.png] > [image: image.png] > [image: image.png] > >

Re: RCA of memory conditions on Ubuntu EC2 agents on ci.jenkins.io causing test instability

Just for reference... [image: image.png] [image: image.png] [image: image.png] t2.medium may be the way to go On Fri, Jun 5, 2020 at 2:32 PM Matt Sicker wrote: > Looks like m5a are AMD and t2 are Intel (and burstable). If they cost > similar, m5a sounds better. > > On Fri, Jun 5, 2020 at 4:19

Re: RCA of memory conditions on Ubuntu EC2 agents on ci.jenkins.io causing test instability

Looks like m5a are AMD and t2 are Intel (and burstable). If they cost similar, m5a sounds better. On Fri, Jun 5, 2020 at 4:19 PM Vlad Silverman wrote: > > I don't know what the cost difference is between the t2 and m5a instances. > > > I guess it depends on the region. > More details are at

Re: RCA of memory conditions on Ubuntu EC2 agents on ci.jenkins.io causing test instability

> I don't know what the cost difference is between the t2 and m5a instances. I guess it depends on the region. More details are at https://aws.amazon.com/ec2/pricing/on-demand/ > On Jun 5, 2020, at 2:02 PM, Slide wrote: > > We are currently

Re: RCA of memory conditions on Ubuntu EC2 agents on ci.jenkins.io causing test instability

Might be worth looking at OpenJ9. It has some nifty cloud native features for helping reduce JVM load. For example: * https://www.eclipse.org/openj9/docs/jitserver/ * https://www.eclipse.org/openj9/docs/shrc/ Disclaimer: I've only seen a talk about this; I've never tried configuring this in a

Re: RCA of memory conditions on Ubuntu EC2 agents on ci.jenkins.io causing test instability

On Fri, Jun 5, 2020 at 1:24 PM Jesse Glick wrote: > The next step would be PRs to infrastructure repositories. I agree. Unfortunately I have spent too much time on this issue already and cannot volunteer to become an infrastructure developer at present. > 256Mb seems low for a Surefire JVM—this

Re: RCA of memory conditions on Ubuntu EC2 agents on ci.jenkins.io causing test instability

Thank you for digging into this problem which has been plaguing us. (INFRA-2548?) Your analysis sounds right. The next step would be PRs to infrastructure repositories. 256Mb seems low for a Surefire JVM—this needs to run Jenkins and all plugins plus whatever your test code is doing. Of course

RCA of memory conditions on Ubuntu EC2 agents on ci.jenkins.io causing test instability

I recently stabilized my plugin's test suite on ci.jenkins.io. The following is my root cause analysis. At present there are eight online Ubuntu EC2 agents on ci.jenkins.io. Three of these are high memory and five of these are not: • EC2 (aws) - High memory ubuntu 18.04 (i-067cdb5c4dd6bbc66) •