Are you still facing this issue? On Tue, Oct 9, 2012 at 3:23 PM, icarusnine <[email protected]> wrote:
> > Hello. > > We have a very large Jenkins set up that includes on master node with 100+ > slaves and 1000+ jobs. > We have reasons for keeping just a single master node so it isn't possible > split our Hudson master. > > Now, we are experiencing performance problems(minor gc happens frequently > and it is performed over 1~2 minutes and it made stop-the-world.) > However, full gc is performed within 20~30 seconds. > Our heap size is over 100G so it is hard to generate and analysis heap > dump. > > Does anyone have any experience with very large Hudson installations like > this? > Is there any advice for tuning or recommendations for this issue? > > > Also, please do let me know if there is any other data that I can provide > that would help with analysis. > Thanks for any help you can provide. > > ---------------------------------------------------------------- > Jenkins info > ---------------------------------------------------------------- > Core ver : 1.424.6 > WAS : weblogic 10.3.2 > JAVA : jdk1.6.0.34 > JVM OPTION : > -Xms180g -Xmx180g -XX:NewSize=140g -XX:MaxNewSize=140g -XX:PermSize=1024m > -XX:MaxPermSize=1024m > -XX:-UseGCOverheadLimit -XX:+UseParallelGC -XX:SurvivorRatio=8 -verbosegc > -Xloggc:app_gc.log -XX:+PrintGCDetails > -XX:+PrintGCTimeStamps -Djava.awt.headless=true > > ---------------------------------------------------------------- > Server spec > ---------------------------------------------------------------- > CPU : Intel(R) Xeon(R) CPU E5-2690 2.90GHz * 4 (32 core) > RAM : 256GB > > ---------------------------------------------------------------- > Gc log > ---------------------------------------------------------------- > 60286.042: [GC [PSYoungGen: 8655292K->8714K(40587584K)] > 25909185K->17262608K(208359744K), 0.0248360 secs] [Times: user=0.38 > sys=0.01, real=0.03 secs] > 60286.067: [Full GC (System) [PSYoungGen: 8714K->0K(40587584K)] > [ParOldGen: 17253893K->17228395K(167772160K)] > 17262608K->17228395K(208359744K) [PSPermGen: 194622K->194622K(2097152K)], > 1.8638320 secs] [Times: user=33.26 sys=0.23, real=1.86 secs] > 60748.860: [GC [PSYoungGen: 39173056K->532623K(40528512K)] > 56401451K->17761019K(208300672K), 0.0837520 secs] [Times: user=1.19 > sys=0.00, real=0.08 secs] > 61243.483: [GC [PSYoungGen: 39705679K->29759K(40658432K)] > 56934075K->17272524K(208430592K), 0.0558890 secs] [Times: user=0.49 > sys=0.00, real=0.05 secs] > 61805.663: [GC [PSYoungGen: 39346943K->28331K(40601792K)] > 56589708K->17275705K(208373952K), 0.0544110 secs] [Times: user=0.49 > sys=0.01, real=0.06 secs] > 62383.664: [GC [PSYoungGen: 39345515K->33640K(40776640K)] > 56592889K->17284373K(208548800K), 0.0592330 secs] [Times: user=0.49 > sys=0.00, real=0.06 secs] > .......................... > 85842.953: [GC [PSYoungGen: 38973565K->1818337K(40038592K)] > 80709421K->44276973K(207810752K), 22.0442750 secs] [Times: user=2.44 > sys=503.41, real=22.04 secs] > 85976.095: [GC [PSYoungGen: 40038561K->1904445K(37126592K)] > 82497204K->46320890K(204898752K), 49.0663710 secs] [Times: user=2.88 > sys=1117.05, real=49.06 secs] > 86147.499: [GC [PSYoungGen: 37126456K->1721075K(38582592K)] > 81542901K->48037517K(206354752K), 39.6267960 secs] [Times: user=2.81 > sys=904.88, real=39.62 secs] > 86265.898: [GC [PSYoungGen: 36943219K->1147657K(38796608K)] > 83259661K->49166685K(206568768K), 43.2677960 secs] [Times: user=6.13 > sys=985.33, real=43.26 secs] > 86435.957: [GC [PSYoungGen: 36592456K->748179K(38591488K)] > 84611484K->49915859K(206363648K), 34.1037910 secs] [Times: user=2.48 > sys=780.02, real=34.10 secs] > 86560.263: [GC [PSYoungGen: 36192756K->448475K(38982464K)] > 85360436K->50343633K(206754624K), 27.3025220 secs] [Times: user=1.64 > sys=623.52, real=27.29 secs] > 86594.685: [GC [PSYoungGen: 36402298K->106372K(38914816K)] > 86297455K->50438940K(206686976K), 15.7548480 secs] [Times: user=1.88 > sys=359.23, real=15.76 secs] > >
