Some performance stats: With no indexes (didn't even finish running killed when it got to envfile): real 632m21.430s user 48m28.973s sys 84m20.053s
+ CREATE INDEX plugin_name on plugin (name); real 102m17.946s user 7m19.054s sys 15m54.729s + CREATE INDEX jenkins_version on jenkins (version); real 24m53.078s user 3m38.248s sys 3m38.441s + CREATE INDEX plugin_month on plugin (month); real 76m37.949s user 4m54.953s sys 10m2.820s Note: not exactly sure how adding an addition index winds up with the script running slower... + CREATE INDEX plugin_namemonth on plugin (name,month); real 1m13.944s user 0m44.397s sys 0m7.887s + DROP INDEX plugin_month; real 3m37.779s user 0m49.016s sys 0m28.856s So based on these results I've sent another pull request[1] which adds 3 additional indexes on top of the plugin.name one from before: CREATE INDEX jenkins_version on jenkins (version); CREATE INDEX plugin_month on plugin (month); CREATE INDEX plugin_namemonth on plugin (name,month); I ran collectNumbers.groovy after making the changes. Here's the performance results from that: mkdir -p target && time groovy collectNumbers.groovy ../jenkins-ci.org/census/20*.gz real 66m34.673s user 48m30.316s sys 19m25.083s time groovy createJson.groovy real 3m33.182s user 0m48.383s sys 0m24.900s Ben [1] https://github.com/jenkinsci/infra-statistics/pull/2 On Wed, Oct 9, 2013 at 4:05 PM, Benjamin Lau <[email protected]> wrote: > I'll see if I can run a benchmark of the old version of the code vs > the indexed version... I was getting the same runs forever behavior on > my computer before I added the indexing and could run the whole job in > a "reasonable"[1] amount of time after the change... I'm completely > stumped as to why this remained broken. > > Ben > > On Wed, Oct 9, 2013 at 3:55 PM, Kohsuke Kawaguchi > <[email protected]> wrote: >> On 10/08/2013 09:42 PM, Benjamin Lau wrote: >>> >>> It seems to at least be running... but the performance characteristics >>> are nothing like what I'm seeing on the machine I have at home. >>> >>> Do you know anything about what kind of hardware ci.jenkins-ci.org is? >> >> >> I doubt if the slow down of this degree can be explained by hardware, but >> here it is: >> >>> kohsuke@cucumber:~$ cat /proc/cpuinfo >>> processor : 0 >>> vendor_id : AuthenticAMD >>> cpu family : 16 >>> model : 2 >>> model name : AMD Athlon(tm) 7850 Dual-Core Processor >>> stepping : 3 >>> cpu MHz : 1400.000 >>> cache size : 512 KB >>> physical id : 0 >>> siblings : 2 >>> core id : 0 >>> cpu cores : 2 >>> apicid : 0 >>> initial apicid : 0 >>> fpu : yes >>> fpu_exception : yes >>> cpuid level : 5 >>> wp : yes >>> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca >>> cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt >>> pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nonstop_tsc >>> extd_apicid pni monitor cx16 popcnt lahf_lm cmp_legacy svm extapic >>> cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs >>> bogomips : 5599.89 >>> TLB size : 1024 4K pages >>> clflush size : 64 >>> cache_alignment : 64 >>> address sizes : 48 bits physical, 48 bits virtual >>> power management: ts ttp tm stc 100mhzsteps hwpstate >>> >>> processor : 1 >>> vendor_id : AuthenticAMD >>> cpu family : 16 >>> model : 2 >>> model name : AMD Athlon(tm) 7850 Dual-Core Processor >>> stepping : 3 >>> cpu MHz : 1400.000 >>> cache size : 512 KB >>> physical id : 0 >>> siblings : 2 >>> core id : 1 >>> cpu cores : 2 >>> apicid : 1 >>> initial apicid : 1 >>> fpu : yes >>> fpu_exception : yes >>> cpuid level : 5 >>> wp : yes >>> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca >>> cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt >>> pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nonstop_tsc >>> extd_apicid pni monitor cx16 popcnt lahf_lm cmp_legacy svm extapic >>> cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs >>> bogomips : 5600.08 >>> TLB size : 1024 4K pages >>> clflush size : 64 >>> cache_alignment : 64 >>> address sizes : 48 bits physical, 48 bits virtual >>> power management: ts ttp tm stc 100mhzsteps hwpstate >>> >>> kohsuke@cucumber:~$ cat /proc/meminfo >>> MemTotal: 8065552 kB >>> MemFree: 982308 kB >>> Buffers: 668732 kB >>> Cached: 3066860 kB >>> SwapCached: 39548 kB >>> Active: 4339220 kB >>> Inactive: 1716388 kB >>> Active(anon): 1818532 kB >>> Inactive(anon): 571804 kB >>> Active(file): 2520688 kB >>> Inactive(file): 1144584 kB >>> Unevictable: 0 kB >>> Mlocked: 0 kB >>> SwapTotal: 23631572 kB >>> SwapFree: 23049072 kB >>> Dirty: 1412 kB >>> Writeback: 0 kB >>> AnonPages: 2299860 kB >>> Mapped: 62588 kB >>> Shmem: 70320 kB >>> Slab: 754624 kB >>> SReclaimable: 700616 kB >>> SUnreclaim: 54008 kB >>> KernelStack: 3848 kB >>> PageTables: 48692 kB >>> NFS_Unstable: 0 kB >>> Bounce: 0 kB >>> WritebackTmp: 0 kB >>> CommitLimit: 27664348 kB >>> Committed_AS: 3852228 kB >>> VmallocTotal: 34359738367 kB >>> VmallocUsed: 374416 kB >>> VmallocChunk: 34359341476 kB >>> HardwareCorrupted: 0 kB >>> HugePages_Total: 0 >>> HugePages_Free: 0 >>> HugePages_Rsvd: 0 >>> HugePages_Surp: 0 >>> Hugepagesize: 2048 kB >>> DirectMap4k: 9792 kB >>> DirectMap2M: 1955840 kB >>> DirectMap1G: 6291456 kB >> >>> >>> >>> kohsuke@cucumber:~$ df -h >>> Filesystem Size Used Avail Use% Mounted on >>> /dev/sda1 895G 438G 412G 52% / >>> none 3.9G 208K 3.9G 1% /dev >>> none 3.9G 0 3.9G 0% /dev/shm >>> none 3.9G 67M 3.8G 2% /var/run >>> none 3.9G 0 3.9G 0% /var/lock >>> none 3.9G 0 3.9G 0% /lib/init/rw >> >> >> >> -- >> Kohsuke Kawaguchi | CloudBees, Inc. | http://cloudbees.com/ >> Try Jenkins Enterprise, our professional version of Jenkins >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Jenkins Developers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
