Re: [Rpm-maint] [rpm-software-management/rpm] Parallelize file processing (#1185)
Closed #1185. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/issues/1185#event-3604860188___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Parallelize file processing (#1185)
So I did some more testing and it turns out, after all, that rpmbuild only spends a tiny fraction of time in the `processPackageFiles()` function; dependency generators (kmod.prov in particular) are a much bigger bottleneck but also vastly trickier to parallelize. The speed improvements that I observed a while ago in https://github.com/rpm-software-management/rpm/issues/1185#issuecomment-617147116 were most likely just a false positive due to some internal data inconsistency caused by the "naive" patch that I tested with. With that said, I'm closing this ticket as there's no point in putting much effort in this. OTOH, parallel execution of dependency generators (`rpmfcGenerateDepends()`) is something where real improvements could be made (in the case of kernel, the cumulative runtime of these is in the order of hunderds of seconds). But that would be tracked in a separate issue. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/issues/1185#issuecomment-666524760___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Parallelize file processing (#1185)
Good to know! No PR yet, but I'll create one soon-ish and reference this issue in it. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/issues/1185#issuecomment-623961021___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Parallelize file processing (#1185)
@dmnks this sounds very nice, is there some PR to follow? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/issues/1185#issuecomment-623931541___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Parallelize file processing (#1185)
Thanks for the feedback. I also noticed `readFilesManifest()` as being one potential source of problems here. Indeed, with a dirty patch that I'm currently testing out, I got a few random crashes (possibly due to that). However, in those cases where it worked, the build time of a kernel on remote 16-core system with a F31 went down from ~7m to ~3m (just the build phase, i.e. `rpmbuild -bb --short-circuit --noclean`). That makes it worthy a more thorough look at least, so I'll see what I can do :) There's nothing to lose; I'll learn something about the codebase either way. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/issues/1185#issuecomment-617147116___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Parallelize file processing (#1185)
Based on a quick look, much of it does in fact appear thread-safe. There are grues in the darker corners though :eyes: One certain problem is the global macro space, readFilesManifest() does push/pop on %license which is probably harmless, but further down the call chain there's readManifest() and specExpand() which do push/pop on current file name and line number, which is not. So at the very least the files manifest parsing would have to be lifted to a pre-parallel section. I'd be nothing short of shocked if this was the only gotcha there is, as processPackageFiles() ends up covering a scary amount of code. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/issues/1185#issuecomment-617127136___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
