Thanks Marc , we will definitely update this settings and observe performance .
For now below fix worked for me , not observing any issues recently with my application . 1.upgrade jdk8 version to recommended version (1.8.0_222-b10 and above) . 2.Install monitoring plugin in Jenkins. 3.Udate NFS settings with recommended settings ( RPCNFSDCOUNT=16 (default is 8 sunrpc.tcp_slot_table_entries = 128) . 4.Auto clean up for sessions with scheduler ( https://wiki.jenkins.io/display/JENKINS/Invalidate+Jenkins+HTTP+sessions) 5.Add monitoring script to monitor active thread , JVM usage , thread deadlocks (https://wiki.jenkins.io/display/JENKINS/Monitoring+Scripts ) 6.Free cache memory whenever it reaches to MAX memory limit , below steps help me to clean cache memory for jenkins master pod. I am able to free cache memory for Jenkins master pod docker container . As per docker image behaviour its taking resources from k8s node where it is deployed . You can verify memory usage by below commands from docker container . bash-4.4$ cat /sys/fs/cgroup/memory/memory.limit_in_bytes bash-4.4$ cat /sys/fs/cgroup/memory/memory.max_usage_in_bytes bash-4.4$ cat /sys/fs/cgroup/memory/memory.stat | grep cache bash-4.4$ Solution , delete *cache memory for k8s node where your Jenkins master pod docker container is running* , there will be no downtime required for Jenkins service . Following document help me : https://www.tecmint.com/clear-ram-memory-cache-buffer-and-swap-space-on-linux/ Thanks , Mahesh On Monday, January 20, 2020 at 8:07:33 PM UTC+5:30, Mark Waite wrote: > > You might install the Jenkins Health Advisor by CloudBees and see if it > has suggestions for your instance. It will check for conditions that have > been found to cause problems in other installations and report them to you > in a daily e-mail message. > > https://jenkins.io/blog/2019/11/22/jenkins-health-advisor-by-cloudbees/ > > On Mon, Jan 20, 2020 at 6:49 AM Mahesh Wabale <[email protected] > <javascript:>> wrote: > >> We are using following JAVA_OPTS CONFIG in Jenkins deployment . >> >> "name": "JAVA_OPTS", >> "value": "-XX:+UseG1GC -XX:+ExplicitGCInvokesConcurrent >> -XX:+ParallelRefProcEnabled -XX:+UseStringDeduplication >> -XX:+UnlockExperimentalVMOptions -XX:G1NewSizePercent=20 >> -XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=summary >> -XX:G1SummarizeRSetStatsPeriod=1 -Xms2048m -Xmx8192m" , and 48 GB ram >> allocated for master Jenkins pod as per initial observation GC is working >> correctly its not going beyond 8 gb . But observing RAM USAGE of master pod >> is continusly increasing day by day . Some time it reaches to memory 48 GB >> seems like it taking memory for cache .How can we control ram memory size , >> or is it possible to free cache memory from Jenkins Docker image which can >> solve my problem . >> >> On Mon, Jan 20, 2020, 3:45 PM Mahesh Wabale <[email protected] >> <javascript:>> wrote: >> >>> Thanks Marc , >>> >>> We will check if it helps , Currently we have "maximum durability" >>> settings configured now , we are planning to make it to " >>> performance-optimized" . We have observed couple of time disc IO is >>> very high sometimes . >>> >>> On Monday, January 20, 2020 at 5:54:38 AM UTC+5:30, Mark Waite wrote: >>>> >>>> I'm not aware of an open bug in Jenkins related to being slow over NFS, >>>> though Jenkins is quite disc intensive and runs best with local disc >>>> drives. >>>> >>>> You might refer to the following articles for more information: >>>> >>>> - Jenkins pipeline durability settings documentation >>>> >>>> <https://www.google.com/url?q=https%3A%2F%2Fjenkins.io%2Fdoc%2Fbook%2Fpipeline%2Fscaling-pipeline%2F%23will-higher-performance-durability-settings-help-me&sa=D&sntz=1&usg=AFQjCNEtvgPugqm3XG5_99K4YYJi6vCf6Q> >>>> >>>> ("Will Higher-Performance Durability Settings Help Me? Yes, if your >>>> Jenkins >>>> instance uses NFS, magnetic storage, runs many Pipelines at once, or >>>> shows >>>> high iowait.") >>>> - CloudBees NFS guide >>>> <https://support.cloudbees.com/hc/en-us/articles/217479948-NFS-Guide> >>>> (multiple >>>> pages of tuning recommendations) >>>> >>>> >>>> >>>> On Fri, Jan 17, 2020 at 5:42 AM Mahesh Wabale <[email protected]> >>>> wrote: >>>> >>>>> Hi Team , >>>>> >>>>> We are using observing Jenkins slowness issues for jenkins version >>>>> 2.138 in multi branch pipeline project . >>>>> We have also used NFS 4.1 . >>>>> >>>>> After Jenkins restart it works for few days but at some stage it stuck >>>>> and becomes unresponsive . As per observation only restart will solve >>>>> this >>>>> issue for next few days . We require frequent restart due to this issue . >>>>> Have anyone observed similar issues , or this is open bug in Jenkins ? >>>>> >>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Jenkins Users" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/jenkinsci-users/319f90ff-5410-4495-bb46-51d61745b3e8%40googlegroups.com >>>>> >>>>> <https://groups.google.com/d/msgid/jenkinsci-users/319f90ff-5410-4495-bb46-51d61745b3e8%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> >>>> >>>> -- >>>> Thanks! >>>> Mark Waite >>>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "Jenkins Users" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/jenkinsci-users/8MnTICGxQ-o/unsubscribe >>> . >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected] <javascript:>. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/jenkinsci-users/8a711ba9-4433-4e62-9170-50e7f5c82bdd%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/jenkinsci-users/8a711ba9-4433-4e62-9170-50e7f5c82bdd%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Jenkins Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/jenkinsci-users/CABOY2ebxtXCGqDXKcfk7kzh0CgjvajTmyiOSni4EPF516gy9iA%40mail.gmail.com >> >> <https://groups.google.com/d/msgid/jenkinsci-users/CABOY2ebxtXCGqDXKcfk7kzh0CgjvajTmyiOSni4EPF516gy9iA%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> > > > -- > Thanks! > Mark Waite > -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/6f369000-dc41-4e16-b2e9-416625c49ebd%40googlegroups.com.
