Re: [Rpm-maint] [rpm-software-management/rpm] Enable thread autodetection for a parallel compression. (#1324)

2020-08-26 Thread marxin
> We'll need some sort of resource manager (no matter how crude) in rpm sooner > than later to handle this sort of stuff, but in the One simple possible solution can be using a semaphore that will be respected by a compression library. `zstd` issue about it lives here:

Re: [Rpm-maint] [rpm-software-management/rpm] Enable thread autodetection for a parallel compression. (#1324)

2020-08-19 Thread Panu Matilainen
Merged #1324 into master. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/1324#event-3669884791___ Rpm-maint mailing list

Re: [Rpm-maint] [rpm-software-management/rpm] Enable thread autodetection for a parallel compression. (#1324)

2020-08-19 Thread Panu Matilainen
Okay, the first commit could've used a more elaborate commit message but I what I missed I missed. I do blame the GH interface to some extent on that though :( Thanks for the patches! -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it

Re: [Rpm-maint] [rpm-software-management/rpm] Enable thread autodetection for a parallel compression. (#1324)

2020-08-19 Thread marxin
@marxin commented on this pull request. > @@ -523,6 +523,27 @@ fprintf(stderr, "*** ufdOpen(%s,0x%x,0%o)\n", url, > (unsigned)flags, (unsigned)mo return fd; } +/* Return number of threads ought to be used for compression based + on a parsed value threads (e.g. from w7T0.xzdio or

Re: [Rpm-maint] [rpm-software-management/rpm] Enable thread autodetection for a parallel compression. (#1324)

2020-08-19 Thread marxin
@marxin pushed 2 commits. 2c659f7ec7dea6fb1a4a25d13a91592c0acb7d97 Add get_compression_threads and refactor code. c771b3b43d05d76d0b19df9a2c25eba3c9f1e6a1 Enable thread autodetection for a parallel compression. -- You are receiving this because you are subscribed to this thread. View it on

Re: [Rpm-maint] [rpm-software-management/rpm] Enable thread autodetection for a parallel compression. (#1324)

2020-08-19 Thread Panu Matilainen
@pmatilai commented on this pull request. > @@ -523,6 +523,27 @@ fprintf(stderr, "*** ufdOpen(%s,0x%x,0%o)\n", url, > (unsigned)flags, (unsigned)mo return fd; } +/* Return number of threads ought to be used for compression based + on a parsed value threads (e.g. from w7T0.xzdio or

Re: [Rpm-maint] [rpm-software-management/rpm] Enable thread autodetection for a parallel compression. (#1324)

2020-08-19 Thread marxin
@marxin pushed 2 commits. 854bf2775fbaaa7b0621de46f3e42a165a5fd60c Add get_compression_threads and refactor code. b490f59832a056a897cd1835d7faed4a41585327 Enable thread autodetection for a parallel compression. -- You are receiving this because you are subscribed to this thread. View it on

Re: [Rpm-maint] [rpm-software-management/rpm] Enable thread autodetection for a parallel compression. (#1324)

2020-08-19 Thread Panu Matilainen
Oops sorry, closing was accidental. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/1324#issuecomment-675888299___ Rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Enable thread autodetection for a parallel compression. (#1324)

2020-08-19 Thread Panu Matilainen
Reopened #1324. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/1324#event-3669326724___ Rpm-maint mailing list

Re: [Rpm-maint] [rpm-software-management/rpm] Enable thread autodetection for a parallel compression. (#1324)

2020-08-19 Thread Panu Matilainen
Closed #1324. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/1324#event-3669326500___ Rpm-maint mailing list

Re: [Rpm-maint] [rpm-software-management/rpm] Enable thread autodetection for a parallel compression. (#1324)

2020-08-19 Thread Panu Matilainen
Yeah, overall looks like intended now, just minor nits/cosmetics. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

Re: [Rpm-maint] [rpm-software-management/rpm] Enable thread autodetection for a parallel compression. (#1324)

2020-08-19 Thread Panu Matilainen
@pmatilai commented on this pull request. > @@ -1143,9 +1132,11 @@ static rpmzstd rpmzstdNew(int fdno, const char *fmode) goto err; } - if (threads > 0) + threads = get_compression_threads (threads); As noted above, there's an extra space before (threads)

Re: [Rpm-maint] [rpm-software-management/rpm] Enable thread autodetection for a parallel compression. (#1324)

2020-08-19 Thread Panu Matilainen
@pmatilai commented on this pull request. > @@ -797,8 +822,7 @@ static LZFILE *lzopen_internal(const char *mode, int fd, > int xz) ret = lzma_easy_encoder(>strm, level, LZMA_CHECK_SHA256); #ifdef HAVE_LZMA_MT } else { - if (threads == -1) -

Re: [Rpm-maint] [rpm-software-management/rpm] Enable thread autodetection for a parallel compression. (#1324)

2020-08-19 Thread Panu Matilainen
@pmatilai commented on this pull request. > @@ -1143,9 +1132,11 @@ static rpmzstd rpmzstdNew(int fdno, const char *fmode) goto err; } - if (threads > 0) + threads = get_compression_threads (threads); + if (threads > 0) { if (ZSTD_isError

Re: [Rpm-maint] [rpm-software-management/rpm] Enable thread autodetection for a parallel compression. (#1324)

2020-08-19 Thread Panu Matilainen
@pmatilai commented on this pull request. > @@ -523,6 +523,28 @@ fprintf(stderr, "*** ufdOpen(%s,0x%x,0%o)\n", url, > (unsigned)flags, (unsigned)mo return fd; } +/* Return number of threads ought to be used for compression based + on a parsed value threads (e.g. from w7T0.xzdio or

Re: [Rpm-maint] [rpm-software-management/rpm] Enable thread autodetection for a parallel compression. (#1324)

2020-08-19 Thread Panu Matilainen
@pmatilai commented on this pull request. > +/* Return number of threads ought to be used for compression based + on a parsed value threads (e.g. from w7T0.xzdio or w7T16.xzdio). + Value -1 means automatic detection. */ + +static int +get_compression_threads (int threads) +{ +if

Re: [Rpm-maint] [rpm-software-management/rpm] Enable thread autodetection for a parallel compression. (#1324)

2020-08-12 Thread marxin
@pmatilai I hope I addressed all your comments. Feel free to look at it now.. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

Re: [Rpm-maint] [rpm-software-management/rpm] Enable thread autodetection for a parallel compression. (#1324)

2020-08-12 Thread Panu Matilainen
Mm, that's not quite what I meant: xz and zstd should use a common helper function for the thread number autodetection, and that function offers a nice place to restrict the number on 32bit architectures centrally (for rpmio purposes until we get something better). OTOH, didn't realize XZ has

Re: [Rpm-maint] [rpm-software-management/rpm] Enable thread autodetection for a parallel compression. (#1324)

2020-08-08 Thread marxin
@marxin pushed 2 commits. 9ab2a82e5e0663a86d1d489a7a6f8825da395b54 Disable threading for xz on 32-bit systems. 47ca1e51e5e113763cb13223f915b45b9bf894b6 Enable thread autodetection for a parallel compression. -- You are receiving this because you are subscribed to this thread. View it on

Re: [Rpm-maint] [rpm-software-management/rpm] Enable thread autodetection for a parallel compression. (#1324)

2020-08-08 Thread marxin
> I think it should at least follow the same convention as XZ: T with no > numbers mean autodetection (its okay if 0 means that too). This seems to > silently change XZ behavior too which is not okay. I'm going to update the documentation of my change. Yes, it was intentional to sync behavior

Re: [Rpm-maint] [rpm-software-management/rpm] Enable thread autodetection for a parallel compression. (#1324)

2020-08-04 Thread Panu Matilainen
I think it should at least follow the same convention as XZ: T with no numbers mean autodetection (its okay if 0 means that too). This seems to silently change XZ behavior too which is not okay. We'll need some sort of resource manager (no matter how crude) in rpm sooner than later to handle

Re: [Rpm-maint] [rpm-software-management/rpm] Enable thread autodetection for a parallel compression. (#1324)

2020-08-04 Thread marxin
> Right, this isn't any worse than what we already have for XZ, but then again > what is there for XZ is simply wrong in the face of concurrent package > generation as we have now. The lowly IO stream has no way of knowing how many > threads it can legitimately consume. That said, would it be

Re: [Rpm-maint] [rpm-software-management/rpm] Enable thread autodetection for a parallel compression. (#1324)

2020-08-03 Thread Panu Matilainen
Right, this isn't any worse than what we already have for XZ, but then again what is there for XZ is simply wrong in the face of concurrent package generation as we have now. The lowly IO stream has no way of knowing how many threads it can legitimately consume. -- You are receiving this

Re: [Rpm-maint] [rpm-software-management/rpm] Enable thread autodetection for a parallel compression. (#1324)

2020-07-31 Thread marxin
Thank you for your thoughts. Let's first summarize what tricks are done for `xz`: https://github.com/rpm-software-management/rpm/blob/28b3704034a9c772959485a130b918f414da9796/rpmio/rpmio.c#L811-L859 - the compression supports `%{_xz_memlimit}` limit - for 32-bit it calculates how much memory

Re: [Rpm-maint] [rpm-software-management/rpm] Enable thread autodetection for a parallel compression. (#1324)

2020-07-31 Thread Florian Festi
Thanks for the patch. Unfortunately auto-detecting the number of threads is a more complicated topic and any solution for zstd needs to have a look at the larger picture. We added support for building multiple package in parallel, so the threads need to be split between those concurrent runs.

Re: [Rpm-maint] [rpm-software-management/rpm] Enable thread autodetection for a parallel compression. (#1324)

2020-07-30 Thread marxin
@marxin pushed 1 commit. 34a78ee61a67a9ffe11142c045fd1fa01e44160e Enable thread autodetection for a parallel compression. -- You are receiving this because you are subscribed to this thread. View it on GitHub:

Re: [Rpm-maint] [rpm-software-management/rpm] Enable thread autodetection for a parallel compression. (#1324)

2020-07-30 Thread marxin
It's a follow up of #1303, so adding @ignatenkobrain as a reviewer ;) -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

[Rpm-maint] [rpm-software-management/rpm] Enable thread autodetection for a parallel compression. (#1324)

2020-07-30 Thread marxin
When using xz or zstd, it can be handy to automatically detect number of threads. We can use `%{getncpus}` for that. You can view, comment on, or merge this pull request online at: https://github.com/rpm-software-management/rpm/pull/1324 -- Commit Summary -- * Enable thread autodetection