Re: [OE-core] [PATCH 1/2] package_rpm.bbclass: use multithreaded xz compression

[Andre McCurdy]

On Fri, Jun 2, 2017 at 4:40 AM, Alexander Kanavin
 wrote:
> On 06/01/2017 10:02 PM, Andre McCurdy wrote:
>>>
>>> RPM's default is single-threaded gz; the change greatly helps with
>>> both buildtimes (when there is a small number of large-sized packages)
>>> and disk space taken by resulting rpms.
>>
>> Do you have any test results for that? Obviously build times will be
>> longer when building with a single CPU since xz compression needs more
>> CPU than gzip, but I'm curious how many CPUs are required before the
>> multiple threads win over and build times actually get shorter. 4? 8?
>
> Yes; it's in the cover letter.

Yes, I only saw that after sending my reply. I shouldn't read emails
in reverse order... sorry about that.

The cover letter benchmarks didn't mention the 22 CPU cores though, so
it's good to have that clarified. Seeing some results from a 4 or 8
core machine would be interesting too.

>> One other concern with parallel xz compression is memory usage. Single
>> threaded xz needs more memory than gzip and running multiple xz
>> compression threads together will push the memory usage up again.
>> Maybe not an issue if the peak is still acceptable, but something to
>> be aware of (I've given up trying to build in 4GB, but hopefully 8GB
>> build machines still have some life left in them...).>
>
> I haven't seen anything resembling low-memory or OOM in my testing, even
> when running 44 rpm instances at the same time, each with their own several
> threads, each thread handling a package, and compression for the package
> again being done in a multi-threaded way. But then I have 22 cores and 32G
> RAM here.
>
> So do try the patches please, and report the findings, YMMV.
>
> Alex
>
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 1/2] package_rpm.bbclass: use multithreaded xz compression

[Alexander Kanavin]

On 06/01/2017 10:02 PM, Andre McCurdy wrote:

RPM's default is single-threaded gz; the change greatly helps with
both buildtimes (when there is a small number of large-sized packages)
and disk space taken by resulting rpms.


Do you have any test results for that? Obviously build times will be
longer when building with a single CPU since xz compression needs more
CPU than gzip, but I'm curious how many CPUs are required before the
multiple threads win over and build times actually get shorter. 4? 8?


Yes; it's in the cover letter.


One other concern with parallel xz compression is memory usage. Single
threaded xz needs more memory than gzip and running multiple xz
compression threads together will push the memory usage up again.
Maybe not an issue if the peak is still acceptable, but something to
be aware of (I've given up trying to build in 4GB, but hopefully 8GB
build machines still have some life left in them...).


I haven't seen anything resembling low-memory or OOM in my testing, even 
when running 44 rpm instances at the same time, each with their own 
several threads, each thread handling a package, and compression for the 
package again being done in a multi-threaded way. But then I have 22 
cores and 32G RAM here.


So do try the patches please, and report the findings, YMMV.

Alex

--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 1/2] package_rpm.bbclass: use multithreaded xz compression

[Andre McCurdy]

On Thu, Jun 1, 2017 at 8:15 AM, Alexander Kanavin
 wrote:
> RPM's default is single-threaded gz; the change greatly helps with
> both buildtimes (when there is a small number of large-sized packages)
> and disk space taken by resulting rpms.

Do you have any test results for that? Obviously build times will be
longer when building with a single CPU since xz compression needs more
CPU than gzip, but I'm curious how many CPUs are required before the
multiple threads win over and build times actually get shorter. 4? 8?

One other concern with parallel xz compression is memory usage. Single
threaded xz needs more memory than gzip and running multiple xz
compression threads together will push the memory usage up again.
Maybe not an issue if the peak is still acceptable, but something to
be aware of (I've given up trying to build in 4GB, but hopefully 8GB
build machines still have some life left in them...).

> Signed-off-by: Alexander Kanavin 
> ---
>  meta/classes/package_rpm.bbclass | 2 ++
>  1 file changed, 2 insertions(+)
>
> diff --git a/meta/classes/package_rpm.bbclass 
> b/meta/classes/package_rpm.bbclass
> index a844c4d4360..dc241975c3b 100644
> --- a/meta/classes/package_rpm.bbclass
> +++ b/meta/classes/package_rpm.bbclass
> @@ -644,6 +644,8 @@ python do_package_rpm () {
>  cmd = cmd + " --define '_build_name_fmt 
> %%{NAME}-%%{VERSION}-%%{RELEASE}.%%{ARCH}.rpm'"
>  cmd = cmd + " --define '_use_internal_dependency_generator 0'"
>  cmd = cmd + " --define '_binaries_in_noarch_packages_terminate_build 0'"
> +cmd = cmd + " --define '_binary_payload w6T.xzdio'"
> +cmd = cmd + " --define '_source_payload w6T.xzdio'"
>  if perfiledeps:
>  cmd = cmd + " --define '__find_requires " + outdepends + "'"
>  cmd = cmd + " --define '__find_provides " + outprovides + "'"
> --
> 2.11.0
>
> --
> ___
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 1/2] package_rpm.bbclass: use multithreaded xz compression

[Alexander Kanavin]

On 06/01/2017 08:11 PM, Mark Hatle wrote:

On 6/1/17 10:15 AM, Alexander Kanavin wrote:

RPM's default is single-threaded gz; the change greatly helps with
both buildtimes (when there is a small number of large-sized packages)
and disk space taken by resulting rpms.


If you wnable XZ here, you need to make sure XZ compression is enabled within
rpm itself.  (I know how that worked in rpm5, not sure how rpm4 does it -- it
may be enable that the binary be present to call out to.)


rpm4 does not call out to any binary, it is linking against liblzma from 
xz package, if it can find it during configure. Xz was already in list 
of dependencies, so nothing needed to be done. Same for other 
compression schemes.



Finally I have not investigated parallel XZ, but in some cases the resulting
compressed file is different depending on compression artifacts due to
parallelization.  (The compressed file is different, not the uncompressed 
content.)

This can be a concern to some who want fully reproducible builds (from source).
So we need to make sure we have a way to disable this type of parallelization
--if-- it causes a problem with different contents.


Let's first see if it is an actual concern for anyone. If the 
uncompressed contents is exactly same, I have difficulty seeing what is 
the problem specifically.


Alex
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 1/2] package_rpm.bbclass: use multithreaded xz compression

[Mark Hatle]

On 6/1/17 10:15 AM, Alexander Kanavin wrote:
> RPM's default is single-threaded gz; the change greatly helps with
> both buildtimes (when there is a small number of large-sized packages)
> and disk space taken by resulting rpms.

If you wnable XZ here, you need to make sure XZ compression is enabled within
rpm itself.  (I know how that worked in rpm5, not sure how rpm4 does it -- it
may be enable that the binary be present to call out to.)

Similarly to XZ, you can use pigz or other parallel compression tools to do
similar with other schemes.

Finally I have not investigated parallel XZ, but in some cases the resulting
compressed file is different depending on compression artifacts due to
parallelization.  (The compressed file is different, not the uncompressed 
content.)

This can be a concern to some who want fully reproducible builds (from source).
So we need to make sure we have a way to disable this type of parallelization
--if-- it causes a problem with different contents.

(Note the fact RPM package metadata changes each time a new RPM is constructed,
this is far -less- of a concern to me.. but something to be aware of.)

--Mark

> Signed-off-by: Alexander Kanavin 
> ---
>  meta/classes/package_rpm.bbclass | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/meta/classes/package_rpm.bbclass 
> b/meta/classes/package_rpm.bbclass
> index a844c4d4360..dc241975c3b 100644
> --- a/meta/classes/package_rpm.bbclass
> +++ b/meta/classes/package_rpm.bbclass
> @@ -644,6 +644,8 @@ python do_package_rpm () {
>  cmd = cmd + " --define '_build_name_fmt 
> %%{NAME}-%%{VERSION}-%%{RELEASE}.%%{ARCH}.rpm'"
>  cmd = cmd + " --define '_use_internal_dependency_generator 0'"
>  cmd = cmd + " --define '_binaries_in_noarch_packages_terminate_build 0'"
> +cmd = cmd + " --define '_binary_payload w6T.xzdio'"
> +cmd = cmd + " --define '_source_payload w6T.xzdio'"
>  if perfiledeps:
>  cmd = cmd + " --define '__find_requires " + outdepends + "'"
>  cmd = cmd + " --define '__find_provides " + outprovides + "'"
> 

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 1/2] package_rpm.bbclass: use multithreaded xz compression

[Alexander Kanavin]

RPM's default is single-threaded gz; the change greatly helps with
both buildtimes (when there is a small number of large-sized packages)
and disk space taken by resulting rpms.

Signed-off-by: Alexander Kanavin 
---
 meta/classes/package_rpm.bbclass | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/meta/classes/package_rpm.bbclass b/meta/classes/package_rpm.bbclass
index a844c4d4360..dc241975c3b 100644
--- a/meta/classes/package_rpm.bbclass
+++ b/meta/classes/package_rpm.bbclass
@@ -644,6 +644,8 @@ python do_package_rpm () {
 cmd = cmd + " --define '_build_name_fmt 
%%{NAME}-%%{VERSION}-%%{RELEASE}.%%{ARCH}.rpm'"
 cmd = cmd + " --define '_use_internal_dependency_generator 0'"
 cmd = cmd + " --define '_binaries_in_noarch_packages_terminate_build 0'"
+cmd = cmd + " --define '_binary_payload w6T.xzdio'"
+cmd = cmd + " --define '_source_payload w6T.xzdio'"
 if perfiledeps:
 cmd = cmd + " --define '__find_requires " + outdepends + "'"
 cmd = cmd + " --define '__find_provides " + outprovides + "'"
-- 
2.11.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core