> We'll need some sort of resource manager (no matter how crude) in rpm sooner
> than later to handle this sort of stuff, but in the
One simple possible solution can be using a semaphore that will be respected by
a compression library.
`zstd` issue about it lives here:
Merged #1324 into master.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/1324#event-3669884791___
Rpm-maint mailing list
Okay, the first commit could've used a more elaborate commit message but I what
I missed I missed. I do blame the GH interface to some extent on that though :(
Thanks for the patches!
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it
@marxin commented on this pull request.
> @@ -523,6 +523,27 @@ fprintf(stderr, "*** ufdOpen(%s,0x%x,0%o)\n", url,
> (unsigned)flags, (unsigned)mo
return fd;
}
+/* Return number of threads ought to be used for compression based
+ on a parsed value threads (e.g. from w7T0.xzdio or
@marxin pushed 2 commits.
2c659f7ec7dea6fb1a4a25d13a91592c0acb7d97 Add get_compression_threads and
refactor code.
c771b3b43d05d76d0b19df9a2c25eba3c9f1e6a1 Enable thread autodetection for a
parallel compression.
--
You are receiving this because you are subscribed to this thread.
View it on
@pmatilai commented on this pull request.
> @@ -523,6 +523,27 @@ fprintf(stderr, "*** ufdOpen(%s,0x%x,0%o)\n", url,
> (unsigned)flags, (unsigned)mo
return fd;
}
+/* Return number of threads ought to be used for compression based
+ on a parsed value threads (e.g. from w7T0.xzdio or
@marxin pushed 2 commits.
854bf2775fbaaa7b0621de46f3e42a165a5fd60c Add get_compression_threads and
refactor code.
b490f59832a056a897cd1835d7faed4a41585327 Enable thread autodetection for a
parallel compression.
--
You are receiving this because you are subscribed to this thread.
View it on
Oops sorry, closing was accidental.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/1324#issuecomment-675888299___
Rpm-maint
Reopened #1324.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/1324#event-3669326724___
Rpm-maint mailing list
Closed #1324.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/1324#event-3669326500___
Rpm-maint mailing list
Yeah, overall looks like intended now, just minor nits/cosmetics.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
@pmatilai commented on this pull request.
> @@ -1143,9 +1132,11 @@ static rpmzstd rpmzstdNew(int fdno, const char *fmode)
goto err;
}
- if (threads > 0)
+ threads = get_compression_threads (threads);
As noted above, there's an extra space before (threads)
@pmatilai commented on this pull request.
> @@ -797,8 +822,7 @@ static LZFILE *lzopen_internal(const char *mode, int fd,
> int xz)
ret = lzma_easy_encoder(>strm, level,
LZMA_CHECK_SHA256);
#ifdef HAVE_LZMA_MT
} else {
- if (threads == -1)
-
@pmatilai commented on this pull request.
> @@ -1143,9 +1132,11 @@ static rpmzstd rpmzstdNew(int fdno, const char *fmode)
goto err;
}
- if (threads > 0)
+ threads = get_compression_threads (threads);
+ if (threads > 0) {
if (ZSTD_isError
@pmatilai commented on this pull request.
> @@ -523,6 +523,28 @@ fprintf(stderr, "*** ufdOpen(%s,0x%x,0%o)\n", url,
> (unsigned)flags, (unsigned)mo
return fd;
}
+/* Return number of threads ought to be used for compression based
+ on a parsed value threads (e.g. from w7T0.xzdio or
@pmatilai commented on this pull request.
> +/* Return number of threads ought to be used for compression based
+ on a parsed value threads (e.g. from w7T0.xzdio or w7T16.xzdio).
+ Value -1 means automatic detection. */
+
+static int
+get_compression_threads (int threads)
+{
+if
@pmatilai I hope I addressed all your comments. Feel free to look at it now..
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
Mm, that's not quite what I meant: xz and zstd should use a common helper
function for the thread number autodetection, and that function offers a nice
place to restrict the number on 32bit architectures centrally (for rpmio
purposes until we get something better).
OTOH, didn't realize XZ has
> I think it should at least follow the same convention as XZ: T with no
> numbers mean autodetection (its okay if 0 means that too). This seems to
> silently change XZ behavior too which is not okay.
I'm going to update the documentation of my change. Yes, it was intentional to
sync behavior
I think it should at least follow the same convention as XZ: T with no numbers
mean autodetection (its okay if 0 means that too). This seems to silently
change XZ behavior too which is not okay.
We'll need some sort of resource manager (no matter how crude) in rpm sooner
than later to handle
> Right, this isn't any worse than what we already have for XZ, but then again
> what is there for XZ is simply wrong in the face of concurrent package
> generation as we have now. The lowly IO stream has no way of knowing how many
> threads it can legitimately consume.
That said, would it be
Right, this isn't any worse than what we already have for XZ, but then again
what is there for XZ is simply wrong in the face of concurrent package
generation as we have now. The lowly IO stream has no way of knowing how many
threads it can legitimately consume.
--
You are receiving this
Thanks for the patch. Unfortunately auto-detecting the number of threads is a
more complicated topic and any solution for zstd needs to have a look at the
larger picture. We added support for building multiple package in parallel, so
the threads need to be split between those concurrent runs.
@marxin pushed 1 commit.
34a78ee61a67a9ffe11142c045fd1fa01e44160e Enable thread autodetection for a
parallel compression.
--
You are receiving this because you are subscribed to this thread.
View it on GitHub:
It's a follow up of #1303, so adding @ignatenkobrain as a reviewer ;)
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
25 matches
Mail list logo
