Yeah that might be an issue... think that would also affect sqlite? Since once the JSON parsing is completed everything is moved to working from that and it's also slow. Also... that post mentions that it's not yet fixed in u17 but I noticed that u40 is the latest...
What version of Java7 is running on the ci.jenkins-ci.org Jenkins? And also interesting... way back at the top of the current run there's some sort of error related to auto-installing the JDK: Unable to auto-install JDK until the license is accepted. I wonder what that's about? Ben On Sun, Oct 13, 2013 at 1:34 PM, Domi <[email protected]> wrote: > This might be a shoot in the dark, but I also only used java6 and have a > much better performance... Maybe we hit this: > http://java-performance.info/changes-to-string-java-1-7-0_06/ > ... we really are parsing a lot if Json, so this could be an issue with > Java7... > > Am 13.10.2013 um 22:04 schrieb Benjamin Lau <[email protected]>: > > Groovy Version: 2.1.6 JVM: 1.6.0_51 Vendor: Apple Inc. OS: Mac OS X > > Java(TM) SE Runtime Environment (build 1.6.0_51-b11-457-11M4509) > > ##### Heap utilization statistics [MB] ##### [1] > Used Memory:1 > Free Memory:79 > Total Memory:81 > Max Memory:123 > > [1] > http://viralpatel.net/blogs/getting-jvm-heap-size-used-memory-total-memory-using-java-runtime/ > > I'm not a groovy expert... is there a way to query it about what it > thinks its heap size is? > > Ben > > On Sun, Oct 13, 2013 at 8:35 AM, domi <[email protected]> wrote: > > merged and started an other build: > https://ci.jenkins-ci.org/view/Infrastructure/job/infra_statistics/95/ > > some other ideas/questions: > > - groovy version? > > - java version? > > - memory (heap)? > > regards Domi > > > > On 11.10.2013, at 00:13, Benjamin Lau <[email protected]> wrote: > > > Some performance stats: > > With no indexes (didn't even finish running killed when it got to envfile): > > real 632m21.430s user 48m28.973s sys 84m20.053s > > > + CREATE INDEX plugin_name on plugin (name); > > real 102m17.946s user 7m19.054s sys 15m54.729s > > > + CREATE INDEX jenkins_version on jenkins (version); > > real 24m53.078s user 3m38.248s sys 3m38.441s > > > + CREATE INDEX plugin_month on plugin (month); > > real 76m37.949s user 4m54.953s sys 10m2.820s > > Note: not exactly sure how adding an addition index winds up with the > > script running slower... > > > + CREATE INDEX plugin_namemonth on plugin (name,month); > > real 1m13.944s user 0m44.397s sys 0m7.887s > > > + DROP INDEX plugin_month; > > real 3m37.779s user 0m49.016s sys 0m28.856s > > > So based on these results I've sent another pull request[1] which adds > > 3 additional indexes on top of the plugin.name one from before: > > CREATE INDEX jenkins_version on jenkins (version); > > CREATE INDEX plugin_month on plugin (month); > > CREATE INDEX plugin_namemonth on plugin (name,month); > > > I ran collectNumbers.groovy after making the changes. Here's the > > performance results from that: > > mkdir -p target && time groovy collectNumbers.groovy > > ../jenkins-ci.org/census/20*.gz > > real 66m34.673s user 48m30.316s sys 19m25.083s > > > time groovy createJson.groovy > > real 3m33.182s user 0m48.383s sys 0m24.900s > > > Ben > > [1] https://github.com/jenkinsci/infra-statistics/pull/2 > > > On Wed, Oct 9, 2013 at 4:05 PM, Benjamin Lau <[email protected]> > wrote: > > I'll see if I can run a benchmark of the old version of the code vs > > the indexed version... I was getting the same runs forever behavior on > > my computer before I added the indexing and could run the whole job in > > a "reasonable"[1] amount of time after the change... I'm completely > > stumped as to why this remained broken. > > > Ben > > > On Wed, Oct 9, 2013 at 3:55 PM, Kohsuke Kawaguchi > > <[email protected]> wrote: > > On 10/08/2013 09:42 PM, Benjamin Lau wrote: > > > It seems to at least be running... but the performance characteristics > > are nothing like what I'm seeing on the machine I have at home. > > > Do you know anything about what kind of hardware ci.jenkins-ci.org is? > > > > I doubt if the slow down of this degree can be explained by hardware, but > > here it is: > > > kohsuke@cucumber:~$ cat /proc/cpuinfo > > processor : 0 > > vendor_id : AuthenticAMD > > cpu family : 16 > > model : 2 > > model name : AMD Athlon(tm) 7850 Dual-Core Processor > > stepping : 3 > > cpu MHz : 1400.000 > > cache size : 512 KB > > physical id : 0 > > siblings : 2 > > core id : 0 > > cpu cores : 2 > > apicid : 0 > > initial apicid : 0 > > fpu : yes > > fpu_exception : yes > > cpuid level : 5 > > wp : yes > > flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca > > cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt > > pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nonstop_tsc > > extd_apicid pni monitor cx16 popcnt lahf_lm cmp_legacy svm extapic > > cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs > > bogomips : 5599.89 > > TLB size : 1024 4K pages > > clflush size : 64 > > cache_alignment : 64 > > address sizes : 48 bits physical, 48 bits virtual > > power management: ts ttp tm stc 100mhzsteps hwpstate > > > processor : 1 > > vendor_id : AuthenticAMD > > cpu family : 16 > > model : 2 > > model name : AMD Athlon(tm) 7850 Dual-Core Processor > > stepping : 3 > > cpu MHz : 1400.000 > > cache size : 512 KB > > physical id : 0 > > siblings : 2 > > core id : 1 > > cpu cores : 2 > > apicid : 1 > > initial apicid : 1 > > fpu : yes > > fpu_exception : yes > > cpuid level : 5 > > wp : yes > > flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca > > cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt > > pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nonstop_tsc > > extd_apicid pni monitor cx16 popcnt lahf_lm cmp_legacy svm extapic > > cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs > > bogomips : 5600.08 > > TLB size : 1024 4K pages > > clflush size : 64 > > cache_alignment : 64 > > address sizes : 48 bits physical, 48 bits virtual > > power management: ts ttp tm stc 100mhzsteps hwpstate > > > kohsuke@cucumber:~$ cat /proc/meminfo > > MemTotal: 8065552 kB > > MemFree: 982308 kB > > Buffers: 668732 kB > > Cached: 3066860 kB > > SwapCached: 39548 kB > > Active: 4339220 kB > > Inactive: 1716388 kB > > Active(anon): 1818532 kB > > Inactive(anon): 571804 kB > > Active(file): 2520688 kB > > Inactive(file): 1144584 kB > > Unevictable: 0 kB > > Mlocked: 0 kB > > SwapTotal: 23631572 kB > > SwapFree: 23049072 kB > > Dirty: 1412 kB > > Writeback: 0 kB > > AnonPages: 2299860 kB > > Mapped: 62588 kB > > Shmem: 70320 kB > > Slab: 754624 kB > > SReclaimable: 700616 kB > > SUnreclaim: 54008 kB > > KernelStack: 3848 kB > > PageTables: 48692 kB > > NFS_Unstable: 0 kB > > Bounce: 0 kB > > WritebackTmp: 0 kB > > CommitLimit: 27664348 kB > > Committed_AS: 3852228 kB > > VmallocTotal: 34359738367 kB > > VmallocUsed: 374416 kB > > VmallocChunk: 34359341476 kB > > HardwareCorrupted: 0 kB > > HugePages_Total: 0 > > HugePages_Free: 0 > > HugePages_Rsvd: 0 > > HugePages_Surp: 0 > > Hugepagesize: 2048 kB > > DirectMap4k: 9792 kB > > DirectMap2M: 1955840 kB > > DirectMap1G: 6291456 kB > > > > > kohsuke@cucumber:~$ df -h > > Filesystem Size Used Avail Use% Mounted on > > /dev/sda1 895G 438G 412G 52% / > > none 3.9G 208K 3.9G 1% /dev > > none 3.9G 0 3.9G 0% /dev/shm > > none 3.9G 67M 3.8G 2% /var/run > > none 3.9G 0 3.9G 0% /var/lock > > none 3.9G 0 3.9G 0% /lib/init/rw > > > > > -- > > Kohsuke Kawaguchi | CloudBees, Inc. | http://cloudbees.com/ > > Try Jenkins Enterprise, our professional version of Jenkins > > > -- > > You received this message because you are subscribed to the Google Groups > > "Jenkins Developers" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [email protected]. > > For more options, visit https://groups.google.com/groups/opt_out. > > > -- > > You received this message because you are subscribed to the Google Groups > "Jenkins Developers" group. > > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > > For more options, visit https://groups.google.com/groups/opt_out. > > > -- > > You received this message because you are subscribed to the Google Groups > "Jenkins Developers" group. > > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > > For more options, visit https://groups.google.com/groups/opt_out. > > > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/groups/opt_out. > > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
