Re: [Rpm-maint] [rpm-software-management/rpm] Run binary package generation via thread pools (#226)
Closed #226. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/226#event-2345662694___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Run binary package generation via thread pools (#226)
As #695 supersedes this I am closing this PRs. Thanks for your work and pushing this issue forwards. Those PRs clearly wouldn't be there if it wasn't for you! -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/226#issuecomment-492973864___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Run binary package generation via thread pools (#226)
Yes, as already mentioned in an earlier comment to this PR: https://github.com/rpm-software-management/rpm/pull/226#issuecomment-425422760 -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/226#issuecomment-490763686___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Run binary package generation via thread pools (#226)
@pmatilai I trust that the last two commits in my set (make string pool operations thread-safe; avoid use of static data) are already merged into master in some form? Don't see those fixes in #695. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/226#issuecomment-490518945___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Run binary package generation via thread pools (#226)
For interested parties, I submitted a more elaborate version as #695 with further code cleanups and additional parallelization of file classifier. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/226#issuecomment-490507280___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Run binary package generation via thread pools (#226)
FWIW, thread protection added to the string pool as of commit 7ffc4d17ffa8f87bd9107b5dc4d9e25daadd14ae so that part of these patches can be dropped. Brief testing showed no problems with the rest applied, but I don't have any real testcase for parallelism at hand. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/226#issuecomment-425422760___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Run binary package generation via thread pools (#226)
Is it possible to get this accepted? I've been testing it on openSUSE's build services, with some very rough (semi controlled) benchmarking the speedup's aren't huge a medium size package that splits out into multiple packages might save 10 seconds off a 5-6 minute build but when you take that across a full distro rebuild of 10,000+ packages it will be significant. Then there are things like libreoffice which is a silly long build, where the building the binary process used to take 900 seconds and is now cut down to 125 seconds, which equates to around an 8 minutes quicker although in the context of a almost 4 hr build time. I am now looking into making a similar change for the processing binaries stage. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/226#issuecomment-420852642___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Run binary package generation via thread pools (#226)
Never mind it seems that the patch is not being applied correctly here -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/226#issuecomment-419012669___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Run binary package generation via thread pools (#226)
@ignatenkobrain testing your patches in the Yocto project (with some extra debugging statements) on openSUSE's open build service instance its clear that the packages are still being built sequentially I'm not sure why yet, but here's some debug output below. Some of us at SUSE are quite keen to get this working and I have time allocated to it primarily because it should speed up our kernel builds significantly but also because it will reduce the load on our build service significantly. `START:packageBinaries` is printed at the start of the packageBinaries function, `START:writerpm` and `END:writerpm` are called either side of the `writeRPM` function the number being printed is just the address of the `pkg` pointer so I could keep track if the write's were actually being done in parallel `[ 457s]` is the timestamp of the current build in seconds ``` [ 457s] START:packageSources [ 457s] ~~~## START:writesourcerpm:37583280 ##~~~ [ 457s] Wrote: /home/abuild/rpmbuild/SRPMS/efl-1.20.7-260.7.src.rpm [ 458s] ~~~## END:writesourcerpm:37583280 ##~~~ [ 458s] END:packageSources [ 458s] START:packageBinaries [ 458s] ~~~## START:writerpm:37582688 ##~~~ [ 458s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/efl-1.20.7-260.7.x86_64.rpm [ 470s] ~~~## END:writerpm:37582688 ##~~~ [ 470s] ~~~## START:writerpm:37627248 ##~~~ [ 470s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/efl-debugsource-1.20.7-260.7.x86_64.rpm [ 470s] ~~~## END:writerpm:37627248 ##~~~ [ 470s] ~~~## START:writerpm:37630240 ##~~~ [ 470s] Wrote: /home/abuild/rpmbuild/RPMS/noarch/efl-lang-1.20.7-260.7.noarch.rpm [ 475s] ~~~## END:writerpm:37630240 ##~~~ [ 475s] ~~~## START:writerpm:37631968 ##~~~ [ 475s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/efl-devel-1.20.7-260.7.x86_64.rpm [ 475s] ~~~## END:writerpm:37631968 ##~~~ [ 475s] ~~~## START:writerpm:37642944 ##~~~ [ 475s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libecore1-1.20.7-260.7.x86_64.rpm [ 475s] ~~~## END:writerpm:37642944 ##~~~ [ 475s] ~~~## START:writerpm:37608864 ##~~~ [ 475s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libector1-1.20.7-260.7.x86_64.rpm [ 476s] ~~~## END:writerpm:37608864 ##~~~ [ 476s] ~~~## START:writerpm:37610224 ##~~~ [ 476s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libedje1-1.20.7-260.7.x86_64.rpm [ 476s] ~~~## END:writerpm:37610224 ##~~~ [ 476s] ~~~## START:writerpm:37612560 ##~~~ [ 476s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libeldbus1-1.20.7-260.7.x86_64.rpm [ 476s] ~~~## END:writerpm:37612560 ##~~~ [ 476s] ~~~## START:writerpm:37614288 ##~~~ [ 476s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libeet1-1.20.7-260.7.x86_64.rpm [ 476s] ~~~## END:writerpm:37614288 ##~~~ [ 476s] ~~~## START:writerpm:37665200 ##~~~ [ 476s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libeeze1-1.20.7-260.7.x86_64.rpm [ 476s] ~~~## END:writerpm:37665200 ##~~~ [ 476s] ~~~## START:writerpm:3788 ##~~~ [ 476s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libefl1-1.20.7-260.7.x86_64.rpm [ 476s] ~~~## END:writerpm:3788 ##~~~ [ 476s] ~~~## START:writerpm:37669168 ##~~~ [ 476s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libefreet1-1.20.7-260.7.x86_64.rpm [ 476s] ~~~## END:writerpm:37669168 ##~~~ [ 476s] ~~~## START:writerpm:37671472 ##~~~ [ 476s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libefreet_mime1-1.20.7-260.7.x86_64.rpm [ 476s] ~~~## END:writerpm:37671472 ##~~~ [ 476s] ~~~## START:writerpm:37774416 ##~~~ [ 476s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libefreet_trash1-1.20.7-260.7.x86_64.rpm [ 476s] ~~~## END:writerpm:37774416 ##~~~ [ 476s] ~~~## START:writerpm:37774000 ##~~~ [ 476s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libeina1-1.20.7-260.7.x86_64.rpm [ 477s] ~~~## END:writerpm:37774000 ##~~~ [ 477s] ~~~## START:writerpm:37775984 ##~~~ [ 477s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libeio1-1.20.7-260.7.x86_64.rpm [ 478s] ~~~## END:writerpm:37775984 ##~~~ [ 478s] ~~~## START:writerpm:3648 ##~~~ [ 478s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libelementary1-1.20.7-260.7.x86_64.rpm [ 478s] ~~~## END:writerpm:3648 ##~~~ [ 478s] ~~~## START:writerpm:37779280 ##~~~ [ 478s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libelocation1-1.20.7-260.7.x86_64.rpm [ 478s] ~~~## END:writerpm:37779280 ##~~~ [ 478s] ~~~## START:writerpm:37781248 ##~~~ [ 478s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libelput1-1.20.7-260.7.x86_64.rpm [ 478s] ~~~## END:writerpm:37781248 ##~~~ [ 478s] ~~~## START:writerpm:37782784 ##~~~ [ 478s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libelua1-1.20.7-260.7.x86_64.rpm [ 478s] ~~~## END:writerpm:37782784 ##~~~ [ 478s] ~~~## START:writerpm:37876144 ##~~~ [ 478s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libembryo1-1.20.7-260.7.x86_64.rpm [ 478s] ~~~## END:writerpm:37876144 ##~~~ [ 478s] ~~~## START:writerpm:37877696 ##~~~ [ 478s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libemotion1-1.20.7-260.7.x86_64.rpm [ 478s] ~~~## END:writerpm:37877696 ##~~~ [ 478s] ~~~##
Re: [Rpm-maint] [rpm-software-management/rpm] Run binary package generation via thread pools (#226)
Ok, I pushed the first and the last patch as a sign of good will. The problem here is that librpm (which rpmstrPool is part of) but also librpmbuild are libraries that might be used in applications running other threading implementations. Also rpm already uses pthread - although not very consistently. So yes, rpmstrPool needs some thread proofing. Still I am for now not comfortable to push these remaining patches without further looking into possible implications - especially on the librpm side. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/226#issuecomment-370439774___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Run binary package generation via thread pools (#226)
Shouldn't the time have come to review this again for finally merging this one after more than a half year now? With rpm 4.14 out, it should be the right time to merge this one now, no? :) -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/226#issuecomment-366881541___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Run binary package generation via thread pools (#226)
@ignatenkobrain: tex (with many packages all with few files) isn't the best benchmark to illustrate a parallel packaging speedup. There are a finite number of threads in the thread pool (which you seem not to have reported or controlled for, sigh). Try a benchmark with a few packages with lots of files, perhaps kernel or gcc, where the no. of packages is about the same as the no. of threads in the pool. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/226#issuecomment-326036010___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Run binary package generation via thread pools (#226)
@kanavin, even 8 minutes speedup is good ;) -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/226#issuecomment-325989203___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Run binary package generation via thread pools (#226)
@ignatenkobrain The reason we see a lot of speed up in Yocto with this patchset is that building the sources happens outside of rpm, and we only use rpm to do packaging of binaries. If the bulk of the above time is spend building texlive from source then of course the final difference won't be as much. Can you investigate a little bit where the time goes? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/226#issuecomment-325977144___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Run binary package generation via thread pools (#226)
It would be nice to run dependency generators in parallel as well (although it might disallow us in future to do some advanced generators which would take subpackages into account, but AFAIK no one is planning to work on something like this in foreseeable future so doesn't seem to be a problem). This is what taking quite a lot of time for some of subpackages, so if they would run in parallel, this would speedup build. # Results ## Before ``` real56m45.083s user39m34.072s sys 29m33.089s ``` ## After ``` real48m16.123s user51m54.226s sys 31m9.800s ``` ## Hardware ``` $ lscpu Architecture:x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 144 On-line CPU(s) list: 0-143 Thread(s) per core: 2 Core(s) per socket: 18 Socket(s): 4 NUMA node(s):4 Vendor ID: GenuineIntel CPU family: 6 Model: 79 Model name: Intel(R) Xeon(R) CPU E7-8860 v4 @ 2.20GHz Stepping:1 CPU MHz: 1200.305 CPU max MHz: 3200. CPU min MHz: 1200. BogoMIPS:4389.14 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache:256K L3 cache:46080K NUMA node0 CPU(s): 0,4,8,12,16,20,24,28,32,36,40,44,48,52,56,60,64,68,72,76,80,84,88,92,96,100,104,108,112,116,120,124,128,132,136,140 NUMA node1 CPU(s): 1,5,9,13,17,21,25,29,33,37,41,45,49,53,57,61,65,69,73,77,81,85,89,93,97,101,105,109,113,117,121,125,129,133,137,141 NUMA node2 CPU(s): 2,6,10,14,18,22,26,30,34,38,42,46,50,54,58,62,66,70,74,78,82,86,90,94,98,102,106,110,114,118,122,126,130,134,138,142 NUMA node3 CPU(s): 3,7,11,15,19,23,27,31,35,39,43,47,51,55,59,63,67,71,75,79,83,87,91,95,99,103,107,111,115,119,123,127,131,135,139,143 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a rdseed adx smap xsaveopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts ``` ## Comments It didn't really seem to generate binary packages in parallel, but probably it worked, not sure ;) ``` Wrote: /home/brain/rpmbuild/RPMS/noarch/texlive-xespotcolor-svn40118-36.fc26.5.noarch.rpm Wrote: /home/brain/rpmbuild/RPMS/noarch/texlive-xesearch-svn16041.0-36.fc26.5.noarch.rpm Wrote: /home/brain/rpmbuild/RPMS/noarch/texlive-uothesis-doc-svn25355.2.5.6-36.fc26.5.noarch.rpm Wrote: /home/brain/rpmbuild/RPMS/noarch/texlive-kdgdocs-doc-svn24498.1.0-36.fc26.5.noarch.rpm Wrote: /home/brain/rpmbuild/RPMS/noarch/texlive-polyglossia-svn40138-36.fc26.5.noarch.rpm Finished binary package job, result 0, filename /home/brain/rpmbuild/RPMS/noarch/texlive-xespotcolor-svn40118-36.fc26.5.noarch.rpm Finished binary package job, result 0, filename /home/brain/rpmbuild/RPMS/noarch/texlive-xesearch-svn16041.0-36.fc26.5.noarch.rpm Finished binary package job, result 0, filename /home/brain/rpmbuild/RPMS/noarch/texlive-uothesis-doc-svn25355.2.5.6-36.fc26.5.noarch.rpm Wrote: /home/brain/rpmbuild/RPMS/noarch/texlive-unicode-math-doc-svn38462-36.fc26.5.noarch.rpm Finished binary package job, result 0, filename /home/brain/rpmbuild/RPMS/noarch/texlive-unicode-math-doc-svn38462-36.fc26.5.noarch.rpm Finished binary package job, result 0, filename /home/brain/rpmbuild/RPMS/noarch/texlive-kdgdocs-doc-svn24498.1.0-36.fc26.5.noarch.rpm Finished binary package job, result 0, filename /home/brain/rpmbuild/RPMS/noarch/texlive-polyglossia-svn40138-36.fc26.5.noarch.rpm ``` -- You are receiving this because you are subscribed to this thread.
Re: [Rpm-maint] [rpm-software-management/rpm] Run binary package generation via thread pools (#226)
> Finished binary package job, result 0, filename > /home/brain/rpmbuild/RPMS/x86_64/hello-debugsource-1-1.fc28.x86_64.rpm not sure if we should print those messages... -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/226#issuecomment-325759830___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Run binary package generation via thread pools (#226)
P.S. it has thousands of subpackages -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/226#issuecomment-325729112___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Run binary package generation via thread pools (#226)
I will try to build texlive with and without PR and will report results :) -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/226#issuecomment-325729031___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Run binary package generation via thread pools (#226)
@ignatenkobrain Sorry, forgot to push the fix. Now fixed. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/226#issuecomment-325659110___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Run binary package generation via thread pools (#226)
@kanavin ```c spec.c: In function 'getBuildTime': spec.c:214:9: error: 'errno' undeclared (first use in this function); did you mean 'h_errno'? errno = 0; ^ h_errno ``` -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/226#issuecomment-325658362___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Run binary package generation via thread pools (#226)
Reopened #226. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/226#event-1225606832___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Run binary package generation via thread pools (#226)
Closed #226. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/226#event-1225606736___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Run binary package generation via thread pools (#226)
let me reopen PR to run CI tests -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/226#issuecomment-325657326___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Run binary package generation via thread pools (#226)
@Conan-Kudo @ignatenkobrain I have rebased this against master now. I checked that it builds, but did not run it against any spec files. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/226#issuecomment-325649868___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Run binary package generation via thread pools (#226)
@kanavin Now that rpm 4.14 has been branched, could you rebase them for review to merge into master? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/226#issuecomment-322755160___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Run binary package generation via thread pools (#226)
@kanavin any news here? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/226#issuecomment-318925604___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Run binary package generation via thread pools (#226)
BTW it'd probably be easier and less intrusive to make buildtime and buildhost part of the spec struct, initialized in newSpec() or so. AFAICS all the relevant places are receiving spec as the argument already, with the exception of writeRPM(). -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/226#issuecomment-308068265___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Run binary package generation via thread pools (#226)
This seems to work correctly on my Mac OS X 10.10 system, though admittedly I don't have a great test case with lots of subpackages. With ~3 subpackages (progs, libs, devel), it seems to work fine. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/226#issuecomment-308068138___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Run binary package generation via thread pools (#226)
I've no prior knowledge of OMP but doesn't look half bad on first sight. Cosmetics aside, we can't really have half the codebase doing half-assed manual pthread locking here and there and another half using OMP, AIUI this is undefined behavior and the string pool is heavily used by non-build code as well. So to set expectations straight: we'll look at merging after rpm 4.14 is branched off, which will be at least couple of months away still. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/226#issuecomment-308062041___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Run binary package generation via thread pools (#226)
I have now updated the patchset again, addressing the crashes with old gcc, and Jeff's comment above. And this time they should be rebased correctly :) -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/226#issuecomment-308014402___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Run binary package generation via thread pools (#226)
You need to make sure that RPMTAG_BUILD{TIME,HOST} in both the *.src.rpm and
the binary packages is identical. This was done originally with static
variables, but can also be done by setting these variables in one place once,
and then using the stored values where necessary.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/226#issuecomment-307575313___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Run binary package generation via thread pools (#226)
There you go, I force pushed the reworked patches. It is 100% pure openmp now :) -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/226#issuecomment-307121770___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Run binary package generation via thread pools (#226)
kanavin commented on this pull request. > @@ -14,6 +14,8 @@ #include #include +#include/* NSPR thread pools */ Alright, I'll rework the patches with openmp, and re-do the pull request. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/226#discussion_r120109813___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Run binary package generation via thread pools (#226)
proyvind commented on this pull request. > @@ -14,6 +14,8 @@ #include #include +#include/* NSPR thread pools */ openmp, unless for some reason would turn out to be unfeasiable, would be the definitive candidate IMO given it's standardization and implementation available from all compilers of relevance. :+1: -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/226#discussion_r119729749___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Run binary package generation via thread pools (#226)
ignatenkobrain commented on this pull request. > @@ -14,6 +14,8 @@ #include #include +#include/* NSPR thread pools */ openmp? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/226#discussion_r119714973___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Run binary package generation via thread pools (#226)
Conan-Kudo commented on this pull request. > @@ -14,6 +14,8 @@ #include #include +#include/* NSPR thread pools */ It'd be pretty hard for me to compile this on Mac OS now, as the Netscape stuff is difficult to bootstrap. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/226#discussion_r119712781___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Run binary package generation via thread pools (#226)
ignatenkobrain commented on this pull request. > @@ -14,6 +14,8 @@ #include #include +#include/* NSPR thread pools */ @kanavin NSS is not hard dependency, it has support for openssl and other crypto backend and I would say that NSS is hardest to bootstrap so we swithced to OpenSSL in Fedora not so long time ago... So assuming NSS to be present is very bad idea. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/226#discussion_r119710543___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Run binary package generation via thread pools (#226)
kanavin commented on this pull request. > @@ -14,6 +14,8 @@ #include #include +#include/* NSPR thread pools */ rpm already has a hard dependency on nss, and nss has a hard dependency on nspr, so it seemed obvious to utilize the thread pool API from nspr. Why you are reluctant? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/226#discussion_r119710103___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Run binary package generation via thread pools (#226)
ignatenkobrain commented on this pull request. > @@ -14,6 +14,8 @@ #include #include +#include/* NSPR thread pools */ I'm kinda reluctant for such external dependencies -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/226#pullrequestreview-41614362___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Run binary package generation via thread pools (#226)
cool! I'll try get around reviewing it myself at least when I find the time, then provide feedback and/or my approval of (FWIW). :) -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/226#issuecomment-305581868___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
