Hi Ian, your patch looks great!
PS: I will use it as an inspiration when I will be adding support of deltarpm to the createrepo_c :) Tomas ----- Original Message ----- > Posting here at James Antill's suggestion. > > In his talk at devconf Dennis Gilmore discussed the current bottlenecks > in the Fedora compose/release process. One thing he mentioned was > deltarpm creation. > > The current upstream createrepo is single-threaded/single-process for > all deltarpm actions. I've written some code to allow parallel workers > for these tasks, similar to the multi-process workers that can be used > in the initial package XML parsing tasks. > > GIT - > https://github.com/imcleod/createrepo/tree/feature/parallel_deltas_full > > RPMS - > http://imcleod.fedorapeople.org/createrepo/ > > The patch adds two options to the command line createrepo and the > associated config object: > > --delta-workers - The number of worker processes to use for delta > related tasks > > --max-concurrent-delta-rpm-size - The maximum total size of uncompressed > rpm payloads that are actively being processed by makedeltarpm at any > given time. > > The deltarpm documentation suggests that its peak RAM use is typically > typically 4x the uncompressed RPM payload size. This is consistent with > my experience. So, a reasonable use case is to set --delta-workers to > the number of CPU cores and --max-concurrent-delta-rpm-size to ~25% of > RAM size (or whatever quantity of memory you want to devote to the > parallel deltas). > > For my development stress-test-case I re-created an F20 x86_64 > Everything repo with F19 Everything as the "old" rpm source for deltas. > On a 32 core test system this task ran in 8 hours with a single deltarpm > worker versus 20 minutes when all 32 cores were used with a concurrent > size limit of 16 GB. In total this creates about 32,000 drpms. So, > this helps. > > Thoughts? > > -Ian > > _______________________________________________ > Yum-devel mailing list > Yum-devel@lists.baseurl.org > http://lists.baseurl.org/mailman/listinfo/yum-devel > -- Tomas Mlcoch (irc: tmlcoch) Release Configuration Management Brno, Czech Republic _______________________________________________ Yum-devel mailing list Yum-devel@lists.baseurl.org http://lists.baseurl.org/mailman/listinfo/yum-devel
