Re: [Rpm-maint] [rpm-software-management/rpm] Create binary rpm packages in parallel? (better multiple CPU core utilization) (#211)

2019-05-23 Thread Panu Matilainen
Forgot there's a also a ticket for this in addition to the PR. This is now solved as of commit 18c92e6fe76df334fed8d5f04b5f40b776daba8c, thanks for your efforts to drive this! -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on

Re: [Rpm-maint] [rpm-software-management/rpm] Create binary rpm packages in parallel? (better multiple CPU core utilization) (#211)

2019-05-23 Thread Panu Matilainen
Closed #211. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/issues/211#event-2361403405___ Rpm-maint mailing list

Re: [Rpm-maint] [rpm-software-management/rpm] Create binary rpm packages in parallel? (better multiple CPU core utilization) (#211)

2017-06-01 Thread Jeff Johnson
You need to handle the static buildtime/buildhost more carefully: the intent with build time was to have one constant and consistent timestamp everywhere during the build no matter where accessed (first access timestamps the entire build). Similarly, the build host (which is supposed to be the

Re: [Rpm-maint] [rpm-software-management/rpm] Create binary rpm packages in parallel? (better multiple CPU core utilization) (#211)

2017-06-01 Thread Jeff Johnson
I'm definitely interested in the patches @RPM5 (even if rather uninterested in a NSPR thread pool: I believe there are better alternatives, like OMP: YMMV). Let me look at what was needed. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view

Re: [Rpm-maint] [rpm-software-management/rpm] Create binary rpm packages in parallel? (better multiple CPU core utilization) (#211)

2017-06-01 Thread Alexander Kanavin
@proyvind @simotek There you go, pull request created :) -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

Re: [Rpm-maint] [rpm-software-management/rpm] Create binary rpm packages in parallel? (better multiple CPU core utilization) (#211)

2017-05-31 Thread proyvind
Best way would be to simply create a pull request from a finalized branch of yours (without unecessary amounts of WIP commits) pushed to a forked rpm repo of yours. Then it would make sense to rather continue discuss the subject and also the implementation of. Regardless of upstream

Re: [Rpm-maint] [rpm-software-management/rpm] Create binary rpm packages in parallel? (better multiple CPU core utilization) (#211)

2017-05-31 Thread Alexander Kanavin
I actually have this working now. The improvements for us (the Yocto project) are quite dramatic, together with the switchover to multithreaded xz compression the process takes several times less for many bigger packages. I had to do these things: 1) split the binary packaging loop body into

Re: [Rpm-maint] [rpm-software-management/rpm] Create binary rpm packages in parallel? (better multiple CPU core utilization) (#211)

2017-05-31 Thread Simon Lees
I've also just started looking into this as well. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

Re: [Rpm-maint] [rpm-software-management/rpm] Create binary rpm packages in parallel? (better multiple CPU core utilization) (#211)

2017-05-24 Thread Jeff Johnson
Again ... OPENMP is the least intrusive (but perhaps the most abstract) implementation. OPENMP has queues and pools and is viable even in 2017. I parallelized the rpm --verify loop over files a few years back using OPENMP. Here are the top level annotations that were needed: `#if

Re: [Rpm-maint] [rpm-software-management/rpm] Create binary rpm packages in parallel? (better multiple CPU core utilization) (#211)

2017-05-17 Thread Alexander Kanavin
NSPR includes thread pools, so I can make an experimental implementation using that: https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSPR/Reference/Thread_Pools Not really keen on messing about with the low-level pthreads API, it's 2017. -- You are receiving this because you are

Re: [Rpm-maint] [rpm-software-management/rpm] Create binary rpm packages in parallel? (better multiple CPU core utilization) (#211)

2017-05-03 Thread Jeff Johnson
Yes its quite feasible to package binary *.rpm files in parallel. The hardest problem is choosing a framework for parallelization: OMP, or pthreads or (in RPM5) yarn. A general framework for parallelization is needed, with queues and thread pools and output serialization etc, is needed because

[Rpm-maint] [rpm-software-management/rpm] Create binary rpm packages in parallel? (better multiple CPU core utilization) (#211)

2017-05-03 Thread Alexander Kanavin
Hello, I'd like to ask your opinion on the idea of creating binary rpms in parallel. It seems possible: I looked at packageBinaries() and it has a straightforward for loop with no obvious dependencies between iterations. Each iteration could be in its own thread via pthreads. The reason I