I ported over that memory tester I found for Java to groovy. Here are the results of that: ##### Heap utilization statistics [MB] ##### Used Memory:10.348419189453125 Free Memory:70.06784820556640625 Total Memory:81.0625 Max Memory:123.9375
Those mostly look the same as the results from the Java one except that groovy seems to take up a bit more memory as a baseline. But that's not an unsurprising result. I noticed that the job on ci.jenkins-ci.org finally got to actually outputting the json data... that only took most of the day. :-/ Is there a copy of JDK 6 available on ci.jenkins-ci.org that could be used in place of JDK 7? Or could we get the JDK 7u40 installed to try that out? Ben On Sun, Oct 13, 2013 at 1:51 PM, Benjamin Lau <[email protected]> wrote: > Yeah that might be an issue... think that would also affect sqlite? > Since once the JSON parsing is completed everything is moved to > working from that and it's also slow. Also... that post mentions that > it's not yet fixed in u17 but I noticed that u40 is the latest... > > What version of Java7 is running on the ci.jenkins-ci.org Jenkins? And > also interesting... way back at the top of the current run there's > some sort of error related to auto-installing the JDK: > > Unable to auto-install JDK until the license is accepted. > > I wonder what that's about? > > Ben > > On Sun, Oct 13, 2013 at 1:34 PM, Domi <[email protected]> wrote: >> This might be a shoot in the dark, but I also only used java6 and have a >> much better performance... Maybe we hit this: >> http://java-performance.info/changes-to-string-java-1-7-0_06/ >> ... we really are parsing a lot if Json, so this could be an issue with >> Java7... >> >> Am 13.10.2013 um 22:04 schrieb Benjamin Lau <[email protected]>: >> >> Groovy Version: 2.1.6 JVM: 1.6.0_51 Vendor: Apple Inc. OS: Mac OS X >> >> Java(TM) SE Runtime Environment (build 1.6.0_51-b11-457-11M4509) >> >> ##### Heap utilization statistics [MB] ##### [1] >> Used Memory:1 >> Free Memory:79 >> Total Memory:81 >> Max Memory:123 >> >> [1] >> http://viralpatel.net/blogs/getting-jvm-heap-size-used-memory-total-memory-using-java-runtime/ >> >> I'm not a groovy expert... is there a way to query it about what it >> thinks its heap size is? >> >> Ben >> >> On Sun, Oct 13, 2013 at 8:35 AM, domi <[email protected]> wrote: >> >> merged and started an other build: >> https://ci.jenkins-ci.org/view/Infrastructure/job/infra_statistics/95/ >> >> some other ideas/questions: >> >> - groovy version? >> >> - java version? >> >> - memory (heap)? >> >> regards Domi >> >> >> >> On 11.10.2013, at 00:13, Benjamin Lau <[email protected]> wrote: >> >> >> Some performance stats: >> >> With no indexes (didn't even finish running killed when it got to envfile): >> >> real 632m21.430s user 48m28.973s sys 84m20.053s >> >> >> + CREATE INDEX plugin_name on plugin (name); >> >> real 102m17.946s user 7m19.054s sys 15m54.729s >> >> >> + CREATE INDEX jenkins_version on jenkins (version); >> >> real 24m53.078s user 3m38.248s sys 3m38.441s >> >> >> + CREATE INDEX plugin_month on plugin (month); >> >> real 76m37.949s user 4m54.953s sys 10m2.820s >> >> Note: not exactly sure how adding an addition index winds up with the >> >> script running slower... >> >> >> + CREATE INDEX plugin_namemonth on plugin (name,month); >> >> real 1m13.944s user 0m44.397s sys 0m7.887s >> >> >> + DROP INDEX plugin_month; >> >> real 3m37.779s user 0m49.016s sys 0m28.856s >> >> >> So based on these results I've sent another pull request[1] which adds >> >> 3 additional indexes on top of the plugin.name one from before: >> >> CREATE INDEX jenkins_version on jenkins (version); >> >> CREATE INDEX plugin_month on plugin (month); >> >> CREATE INDEX plugin_namemonth on plugin (name,month); >> >> >> I ran collectNumbers.groovy after making the changes. Here's the >> >> performance results from that: >> >> mkdir -p target && time groovy collectNumbers.groovy >> >> ../jenkins-ci.org/census/20*.gz >> >> real 66m34.673s user 48m30.316s sys 19m25.083s >> >> >> time groovy createJson.groovy >> >> real 3m33.182s user 0m48.383s sys 0m24.900s >> >> >> Ben >> >> [1] https://github.com/jenkinsci/infra-statistics/pull/2 >> >> >> On Wed, Oct 9, 2013 at 4:05 PM, Benjamin Lau <[email protected]> >> wrote: >> >> I'll see if I can run a benchmark of the old version of the code vs >> >> the indexed version... I was getting the same runs forever behavior on >> >> my computer before I added the indexing and could run the whole job in >> >> a "reasonable"[1] amount of time after the change... I'm completely >> >> stumped as to why this remained broken. >> >> >> Ben >> >> >> On Wed, Oct 9, 2013 at 3:55 PM, Kohsuke Kawaguchi >> >> <[email protected]> wrote: >> >> On 10/08/2013 09:42 PM, Benjamin Lau wrote: >> >> >> It seems to at least be running... but the performance characteristics >> >> are nothing like what I'm seeing on the machine I have at home. >> >> >> Do you know anything about what kind of hardware ci.jenkins-ci.org is? >> >> >> >> I doubt if the slow down of this degree can be explained by hardware, but >> >> here it is: >> >> >> kohsuke@cucumber:~$ cat /proc/cpuinfo >> >> processor : 0 >> >> vendor_id : AuthenticAMD >> >> cpu family : 16 >> >> model : 2 >> >> model name : AMD Athlon(tm) 7850 Dual-Core Processor >> >> stepping : 3 >> >> cpu MHz : 1400.000 >> >> cache size : 512 KB >> >> physical id : 0 >> >> siblings : 2 >> >> core id : 0 >> >> cpu cores : 2 >> >> apicid : 0 >> >> initial apicid : 0 >> >> fpu : yes >> >> fpu_exception : yes >> >> cpuid level : 5 >> >> wp : yes >> >> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca >> >> cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt >> >> pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nonstop_tsc >> >> extd_apicid pni monitor cx16 popcnt lahf_lm cmp_legacy svm extapic >> >> cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs >> >> bogomips : 5599.89 >> >> TLB size : 1024 4K pages >> >> clflush size : 64 >> >> cache_alignment : 64 >> >> address sizes : 48 bits physical, 48 bits virtual >> >> power management: ts ttp tm stc 100mhzsteps hwpstate >> >> >> processor : 1 >> >> vendor_id : AuthenticAMD >> >> cpu family : 16 >> >> model : 2 >> >> model name : AMD Athlon(tm) 7850 Dual-Core Processor >> >> stepping : 3 >> >> cpu MHz : 1400.000 >> >> cache size : 512 KB >> >> physical id : 0 >> >> siblings : 2 >> >> core id : 1 >> >> cpu cores : 2 >> >> apicid : 1 >> >> initial apicid : 1 >> >> fpu : yes >> >> fpu_exception : yes >> >> cpuid level : 5 >> >> wp : yes >> >> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca >> >> cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt >> >> pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nonstop_tsc >> >> extd_apicid pni monitor cx16 popcnt lahf_lm cmp_legacy svm extapic >> >> cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs >> >> bogomips : 5600.08 >> >> TLB size : 1024 4K pages >> >> clflush size : 64 >> >> cache_alignment : 64 >> >> address sizes : 48 bits physical, 48 bits virtual >> >> power management: ts ttp tm stc 100mhzsteps hwpstate >> >> >> kohsuke@cucumber:~$ cat /proc/meminfo >> >> MemTotal: 8065552 kB >> >> MemFree: 982308 kB >> >> Buffers: 668732 kB >> >> Cached: 3066860 kB >> >> SwapCached: 39548 kB >> >> Active: 4339220 kB >> >> Inactive: 1716388 kB >> >> Active(anon): 1818532 kB >> >> Inactive(anon): 571804 kB >> >> Active(file): 2520688 kB >> >> Inactive(file): 1144584 kB >> >> Unevictable: 0 kB >> >> Mlocked: 0 kB >> >> SwapTotal: 23631572 kB >> >> SwapFree: 23049072 kB >> >> Dirty: 1412 kB >> >> Writeback: 0 kB >> >> AnonPages: 2299860 kB >> >> Mapped: 62588 kB >> >> Shmem: 70320 kB >> >> Slab: 754624 kB >> >> SReclaimable: 700616 kB >> >> SUnreclaim: 54008 kB >> >> KernelStack: 3848 kB >> >> PageTables: 48692 kB >> >> NFS_Unstable: 0 kB >> >> Bounce: 0 kB >> >> WritebackTmp: 0 kB >> >> CommitLimit: 27664348 kB >> >> Committed_AS: 3852228 kB >> >> VmallocTotal: 34359738367 kB >> >> VmallocUsed: 374416 kB >> >> VmallocChunk: 34359341476 kB >> >> HardwareCorrupted: 0 kB >> >> HugePages_Total: 0 >> >> HugePages_Free: 0 >> >> HugePages_Rsvd: 0 >> >> HugePages_Surp: 0 >> >> Hugepagesize: 2048 kB >> >> DirectMap4k: 9792 kB >> >> DirectMap2M: 1955840 kB >> >> DirectMap1G: 6291456 kB >> >> >> >> >> kohsuke@cucumber:~$ df -h >> >> Filesystem Size Used Avail Use% Mounted on >> >> /dev/sda1 895G 438G 412G 52% / >> >> none 3.9G 208K 3.9G 1% /dev >> >> none 3.9G 0 3.9G 0% /dev/shm >> >> none 3.9G 67M 3.8G 2% /var/run >> >> none 3.9G 0 3.9G 0% /var/lock >> >> none 3.9G 0 3.9G 0% /lib/init/rw >> >> >> >> >> -- >> >> Kohsuke Kawaguchi | CloudBees, Inc. | http://cloudbees.com/ >> >> Try Jenkins Enterprise, our professional version of Jenkins >> >> >> -- >> >> You received this message because you are subscribed to the Google Groups >> >> "Jenkins Developers" group. >> >> To unsubscribe from this group and stop receiving emails from it, send an >> >> email to [email protected]. >> >> For more options, visit https://groups.google.com/groups/opt_out. >> >> >> -- >> >> You received this message because you are subscribed to the Google Groups >> "Jenkins Developers" group. >> >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> >> For more options, visit https://groups.google.com/groups/opt_out. >> >> >> -- >> >> You received this message because you are subscribed to the Google Groups >> "Jenkins Developers" group. >> >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> >> For more options, visit https://groups.google.com/groups/opt_out. >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Jenkins Developers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> For more options, visit https://groups.google.com/groups/opt_out. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Jenkins Developers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
