Re: [JENKINS] - New Maven Controller for the project
Can you install AdoptOpenJdk for the Jenkins controller? It contains Eclipse OpenJ9 Garbage Collector and it significantly decreases memory consumption of the application due to the meta space goes to the disk. You should save 40 - 75% out of 3GB. I used G1, Shenandoah, ZGC and Eclipse OpenJ9 which saved the most memory. On Thu, Jul 22, 2021 at 9:23 AM Arnaud Héritier wrote: > yes for the controller it depends of its size (number of jobs and types of > jobs) but here we are fine it seems with our 3Gb > > * Java > - Version: 1.8.0292 > - Maximum memory: 3.00 GB (3221225472) > - Allocated memory: 3.00 GB (3221225472) > - Free memory: 750.15 MB (786591664) > - In-use memory: 2.27 GB (2434633808) > - GC strategy: G1 > - Available CPUs: 2 > > For agents I reduced the memory allocated to the agent process but it > doesn't help much (it seems - even if it is still a good thing to do) > > What is strange is that I see our agents sometimes disconnected even when > we have no activity on the jenkins controller > > Sadly jenkins is deployed on Apache Tomcat thus I cannot get access to its > logs > > In general the connection lost is detected by what we call the PingThread ( > > https://www.jenkins.io/doc/book/system-administration/monitoring/#ping-thread > ) but not only > > https://ci-maven.apache.org/log/all > > For example it was few minutes ago we got 3 agents disconnected while > nothing was running > > 2021-07-22 06:58:21.769+ [id=106291] INFO > hudson.slaves.ChannelPinger$1#onDead: > Ping failed. Terminating the channel maven4. > java.util.concurrent.TimeoutException: Ping started at 1626936861769 hasn't > completed by 1626937101769 > at hudson.remoting.PingThread.ping(PingThread.java:134) > at hudson.remoting.PingThread.run(PingThread.java:90) > 2021-07-22 06:58:21.778+ [id=106292] INFO > hudson.slaves.ChannelPinger$1#onDead: > Ping failed. Terminating the channel maven3. > java.util.concurrent.TimeoutException: Ping started at 1626936861777 hasn't > completed by 1626937101778 > at hudson.remoting.PingThread.ping(PingThread.java:134) > at hudson.remoting.PingThread.run(PingThread.java:90) > 2021-07-22 06:58:21.983+ [id=106295] INFO > hudson.slaves.ChannelPinger$1#onDead: > Ping failed. Terminating the channel maven5. > java.util.concurrent.TimeoutException: Ping started at 1626936861982 hasn't > completed by 1626937101983 > at hudson.remoting.PingThread.ping(PingThread.java:134) > at hudson.remoting.PingThread.run(PingThread.java:90) > > @Gavin McDonald In terms of network, is it the same > environment we use today compared to the ci-builds.apache.org environment > ? > > > On Wed, Jul 21, 2021 at 11:48 PM Tibor Digana > wrote: > > > In my company, I also used 1GB for Xmx of Java Heap for the Jenkins JVM > and > > it was enough. > > The subprocesses like Maven need to have much more memory to allocate for > > themself rather than Jenkins JVM. > > T > > > > On Wed, Jul 21, 2021 at 6:38 PM Arnaud Héritier > > wrote: > > > > > I am looking at our builds and I try to understand why our agents are > > often > > > disconnected during the builds. > > > We have in general a stacktrace like > > > > > > maven6 was marked offline: Connection was broken: java.io.IOException: > > > Pipe closed after 0 cycles > > > at > > > > > > org.apache.sshd.common.channel.ChannelPipedInputStream.read(ChannelPipedInputStream.java:118) > > > at > > > > > > org.apache.sshd.common.channel.ChannelPipedInputStream.read(ChannelPipedInputStream.java:101) > > > at > > > > > > hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:92) > > > at > > > > hudson.remoting.ChunkedInputStream.readHeader(ChunkedInputStream.java:73) > > > at > > > > > > hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:103) > > > at > > > > > > hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:39) > > > at > > > > > > hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34) > > > at > > > > > > hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63) > > > > > > > > > > > > As far I can see we are using 16Gb "hosts" for linux agents > > > > > > Something very strange is that the jenkins agent (this small component > > > doing the link between the build host and the controller) is configured > > > with `-Xms8g -Xmx8g` thus we are reserving for it 50% of the server mem > > > (even more because of the non-heap) > > > This one in general should require in general really less. 1Gb is > > already a > > > lot from my exp. > > > Due to this, the OS can see it has the biggest process on the host and > > > decide to kill it when the rest of the memory is used by the build. > > > I think we should decrease this value. > > > (I can do it but I don't know how was configured the ci.apache.org > > agents > > > and I would like
Re: Upcoming Maven 3.8.2
looking forward to the release. please ensure to have the latest snapshot deployed. Thank you for the awesome work. -D On Thu, Jul 22, 2021 at 11:37 AM Arnaud Héritier wrote: > Awesome work. > Well done team > > On Sat, Jul 3, 2021 at 4:31 PM Michael Osipov wrote: > > > Folks, > > > > I have ported approx. 50 non-invasive issues back to maven-3.8.x branch > > and it is in a very good shape. > > There are only two issues open (one blocker): > > * https://issues.apache.org/jira/browse/MNG-7161 > > * https://issues.apache.org/jira/browse/MNG-7034 > > (https://github.com/apache/maven/pull/484) > > > > I kindly asked gnodet@ for help to resolve them. > > > > Do you see any other issues either from 4.0.0-alpha-1 or others which > > need to be addressed in 3.8.2? > > > > M > > > > - > > To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org > > For additional commands, e-mail: dev-h...@maven.apache.org > > > > > > -- > Arnaud Héritier > Twitter/Skype : aheritier >
Re: Upcoming Maven 3.8.2
Awesome work. Well done team On Sat, Jul 3, 2021 at 4:31 PM Michael Osipov wrote: > Folks, > > I have ported approx. 50 non-invasive issues back to maven-3.8.x branch > and it is in a very good shape. > There are only two issues open (one blocker): > * https://issues.apache.org/jira/browse/MNG-7161 > * https://issues.apache.org/jira/browse/MNG-7034 > (https://github.com/apache/maven/pull/484) > > I kindly asked gnodet@ for help to resolve them. > > Do you see any other issues either from 4.0.0-alpha-1 or others which > need to be addressed in 3.8.2? > > M > > - > To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org > For additional commands, e-mail: dev-h...@maven.apache.org > > -- Arnaud Héritier Twitter/Skype : aheritier
Re: [JENKINS] - New Maven Controller for the project
yes for the controller it depends of its size (number of jobs and types of jobs) but here we are fine it seems with our 3Gb * Java - Version: 1.8.0292 - Maximum memory: 3.00 GB (3221225472) - Allocated memory: 3.00 GB (3221225472) - Free memory: 750.15 MB (786591664) - In-use memory: 2.27 GB (2434633808) - GC strategy: G1 - Available CPUs: 2 For agents I reduced the memory allocated to the agent process but it doesn't help much (it seems - even if it is still a good thing to do) What is strange is that I see our agents sometimes disconnected even when we have no activity on the jenkins controller Sadly jenkins is deployed on Apache Tomcat thus I cannot get access to its logs In general the connection lost is detected by what we call the PingThread ( https://www.jenkins.io/doc/book/system-administration/monitoring/#ping-thread ) but not only https://ci-maven.apache.org/log/all For example it was few minutes ago we got 3 agents disconnected while nothing was running 2021-07-22 06:58:21.769+ [id=106291] INFO hudson.slaves.ChannelPinger$1#onDead: Ping failed. Terminating the channel maven4. java.util.concurrent.TimeoutException: Ping started at 1626936861769 hasn't completed by 1626937101769 at hudson.remoting.PingThread.ping(PingThread.java:134) at hudson.remoting.PingThread.run(PingThread.java:90) 2021-07-22 06:58:21.778+ [id=106292] INFO hudson.slaves.ChannelPinger$1#onDead: Ping failed. Terminating the channel maven3. java.util.concurrent.TimeoutException: Ping started at 1626936861777 hasn't completed by 1626937101778 at hudson.remoting.PingThread.ping(PingThread.java:134) at hudson.remoting.PingThread.run(PingThread.java:90) 2021-07-22 06:58:21.983+ [id=106295] INFO hudson.slaves.ChannelPinger$1#onDead: Ping failed. Terminating the channel maven5. java.util.concurrent.TimeoutException: Ping started at 1626936861982 hasn't completed by 1626937101983 at hudson.remoting.PingThread.ping(PingThread.java:134) at hudson.remoting.PingThread.run(PingThread.java:90) @Gavin McDonald In terms of network, is it the same environment we use today compared to the ci-builds.apache.org environment ? On Wed, Jul 21, 2021 at 11:48 PM Tibor Digana wrote: > In my company, I also used 1GB for Xmx of Java Heap for the Jenkins JVM and > it was enough. > The subprocesses like Maven need to have much more memory to allocate for > themself rather than Jenkins JVM. > T > > On Wed, Jul 21, 2021 at 6:38 PM Arnaud Héritier > wrote: > > > I am looking at our builds and I try to understand why our agents are > often > > disconnected during the builds. > > We have in general a stacktrace like > > > > maven6 was marked offline: Connection was broken: java.io.IOException: > > Pipe closed after 0 cycles > > at > > > org.apache.sshd.common.channel.ChannelPipedInputStream.read(ChannelPipedInputStream.java:118) > > at > > > org.apache.sshd.common.channel.ChannelPipedInputStream.read(ChannelPipedInputStream.java:101) > > at > > > hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:92) > > at > > hudson.remoting.ChunkedInputStream.readHeader(ChunkedInputStream.java:73) > > at > > > hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:103) > > at > > > hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:39) > > at > > > hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34) > > at > > > hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63) > > > > > > > > As far I can see we are using 16Gb "hosts" for linux agents > > > > Something very strange is that the jenkins agent (this small component > > doing the link between the build host and the controller) is configured > > with `-Xms8g -Xmx8g` thus we are reserving for it 50% of the server mem > > (even more because of the non-heap) > > This one in general should require in general really less. 1Gb is > already a > > lot from my exp. > > Due to this, the OS can see it has the biggest process on the host and > > decide to kill it when the rest of the memory is used by the build. > > I think we should decrease this value. > > (I can do it but I don't know how was configured the ci.apache.org > agents > > and I would like to not add more issue if this setting was here in the > past > > > > I don't think it is the root cause of our instabilities (at least all) > and > > there is something else I have to find but it's a cheap fix to try > > > > FYI our agents VMs are ~like this today: > > > > - Java > > + Home: `/usr/local/asfpackages/java/oraclejdk-1.8.0-291/jre` > > + Vendor: Oracle Corporation > > + Version: 1.8.0291 > > + Maximum memory: 7.67 GB (8232370176) > > + Allocated memory: 7.67 GB (8232370176) > > + Free memory: 6.03 GB (6470953760) > > + In-use memory: 1.64 GB (1761416416) > > + GC strategy: ParallelGC > > + Available
