Re: [OE-core] [PATCH 0/2] Multi-threaded RPM support

[Alexander Kanavin]

On 06/05/2017 04:47 PM, Davis, Michael wrote:

I just wanted to note that in my quest to get newer chrome to build I have used 
centos7 with the official devtoolset-6 scl.
It was able to build my pyro based distro fine in gcc6.  So Centos7 has an 
officially supported path that may be viable if it comes down to being the last 
gcc 4.x distro.


I guess at some point we'll have to officially do this as well, as the 
upstream projects will adopt C/C++11 (or even newer) and otherwise 
become incompatible with 4.x.


Alex
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 0/2] Multi-threaded RPM support

[Davis, Michael]

I just wanted to note that in my quest to get newer chrome to build I have used 
centos7 with the official devtoolset-6 scl.
It was able to build my pyro based distro fine in gcc6.  So Centos7 has an 
officially supported path that may be viable if it comes down to being the last 
gcc 4.x distro.


-Original Message-
From: openembedded-core-boun...@lists.openembedded.org 
[mailto:openembedded-core-boun...@lists.openembedded.org] On Behalf Of 
Alexander Kanavin
Sent: Monday, June 05, 2017 8:35 AM
To: Joshua Lock; Richard Purdie; openembedded-core@lists.openembedded.org; 
Leonardo Sandoval
Subject: Re: [OE-core] [PATCH 0/2] Multi-threaded RPM support

On 06/05/2017 04:21 PM, Joshua Lock wrote:

> On the Yocto Autobuilder clusters we have:
> * centos7 / gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-11)
> * debian8 / gcc (Debian 4.9.2-10) 4.9.2
> * debian-testing / gcc (Debian 6.3.0-14) 6.3.0 20170415
> * fedora24 / gcc (GCC) 6.3.1 20161221 (Red Hat 6.3.1-1)
> * fedora25 / gcc (GCC) 6.3.1 20161221 (Red Hat 6.3.1-1)
> * opensuse132 / gcc (SUSE Linux) 4.8.3 20140627 [gcc-4_8-branch
> revision 212064]
> * opensuse422 / gcc (SUSE Linux) 4.8.5
> * ubuntu1604 / gcc (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609
> * ubuntu1610 / gcc (Ubuntu 6.2.0-5ubuntu12) 6.2.0 20161005
>
> We are aiming to replace openSUSE 13.2 (as it was discontinued in
> January of this year) but CentOS 7 has a good long lifetime ahead of
> it, therefore supporting it and RHEL 7 will require gcc 4.8.5 support.

Thanks; in that case using atomic types and other C11 features is not 
possible at all for quite some time, and openmp support, while 
available, is restricted to version 3.1 of the spec (which should be 
just about enough - earlier versions do not have task support).

I actually have gcc 4.8 here (on debian testing, which will become 
debian 9 in a couple of weeks), even though the default version is 6.3, 
so I can test 4.8 compatibility locally.


Alex
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 0/2] Multi-threaded RPM support

[Alexander Kanavin]

On 06/05/2017 04:21 PM, Joshua Lock wrote:


On the Yocto Autobuilder clusters we have:
* centos7 / gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-11)
* debian8 / gcc (Debian 4.9.2-10) 4.9.2
* debian-testing / gcc (Debian 6.3.0-14) 6.3.0 20170415
* fedora24 / gcc (GCC) 6.3.1 20161221 (Red Hat 6.3.1-1)
* fedora25 / gcc (GCC) 6.3.1 20161221 (Red Hat 6.3.1-1)
* opensuse132 / gcc (SUSE Linux) 4.8.3 20140627 [gcc-4_8-branch
revision 212064]
* opensuse422 / gcc (SUSE Linux) 4.8.5
* ubuntu1604 / gcc (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609
* ubuntu1610 / gcc (Ubuntu 6.2.0-5ubuntu12) 6.2.0 20161005

We are aiming to replace openSUSE 13.2 (as it was discontinued in
January of this year) but CentOS 7 has a good long lifetime ahead of
it, therefore supporting it and RHEL 7 will require gcc 4.8.5 support.


Thanks; in that case using atomic types and other C11 features is not 
possible at all for quite some time, and openmp support, while 
available, is restricted to version 3.1 of the spec (which should be 
just about enough - earlier versions do not have task support).


I actually have gcc 4.8 here (on debian testing, which will become 
debian 9 in a couple of weeks), even though the default version is 6.3, 
so I can test 4.8 compatibility locally.



Alex
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 0/2] Multi-threaded RPM support

[Joshua Lock]

On Mon, 2017-06-05 at 15:14 +0300, Alexander Kanavin wrote:
> On 06/04/2017 05:15 PM, Richard Purdie wrote:
> 
> > https://autobuilder.yoctoproject.org/main/builders/nightly-x86/buil
> > ds/1
> > 163/steps/BuildImages/logs/stdio
> > 
> > https://autobuilder.yoctoproject.org/main/builders/nightly-x86-64/b
> > uild
> > s/1203/steps/BuildImages/logs/stdio
> > 
> > https://autobuilder.yoctoproject.org/main/builders/nightly-ipk/buil
> > ds/1
> > 112/steps/BuildImages/logs/stdio
> > 
> > I suspect the same issue causing failures in multiple places.
> 
> The patches as they are now require that the C compiler on the host
> is 
> recent; specifically that it defaults to C11 mode. I'm not sure when
> GCC 
> made the switch, but it works that way with GCC 6.3.0.
> 
> In any case, I'll rework the patches to use openmp, as requested by 
> upstream - this may however cause a different set of compatibility 
> problems with older GCC versions. What are the oldest host compiler 
> versions/distro versions that we still need to support in master?

On the Yocto Autobuilder clusters we have:
* centos7 / gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-11)
* debian8 / gcc (Debian 4.9.2-10) 4.9.2
* debian-testing / gcc (Debian 6.3.0-14) 6.3.0 20170415
* fedora24 / gcc (GCC) 6.3.1 20161221 (Red Hat 6.3.1-1)
* fedora25 / gcc (GCC) 6.3.1 20161221 (Red Hat 6.3.1-1)
* opensuse132 / gcc (SUSE Linux) 4.8.3 20140627 [gcc-4_8-branch
revision 212064]
* opensuse422 / gcc (SUSE Linux) 4.8.5
* ubuntu1604 / gcc (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609
* ubuntu1610 / gcc (Ubuntu 6.2.0-5ubuntu12) 6.2.0 20161005

We are aiming to replace openSUSE 13.2 (as it was discontinued in
January of this year) but CentOS 7 has a good long lifetime ahead of
it, therefore supporting it and RHEL 7 will require gcc 4.8.5 support.

Joshua
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 0/2] Multi-threaded RPM support

[Alexander Kanavin]

On 06/05/2017 04:01 PM, Burton, Ross wrote:


On 5 June 2017 at 13:14, Alexander Kanavin
> wrote:

The patches as they are now require that the C compiler on the host
is recent; specifically that it defaults to C11 mode. I'm not sure
when GCC made the switch, but it works that way with GCC 6.3.0.


Isn't that just a compiler flag away from being fixed?


Not if the compiler is 4.8 or earlier; it has to support _Atomic (a C11 
feature), which only appeared in 4.9.



Alex

--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 0/2] Multi-threaded RPM support

[Burton, Ross]

On 5 June 2017 at 13:14, Alexander Kanavin <
alexander.kana...@linux.intel.com> wrote:

> The patches as they are now require that the C compiler on the host is
> recent; specifically that it defaults to C11 mode. I'm not sure when GCC
> made the switch, but it works that way with GCC 6.3.0.
>

Isn't that just a compiler flag away from being fixed?

Ross
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 0/2] Multi-threaded RPM support

[Alexander Kanavin]

On 06/05/2017 03:14 PM, Alexander Kanavin wrote:

The patches as they are now require that the C compiler on the host is
recent; specifically that it defaults to C11 mode. I'm not sure when GCC
made the switch, but it works that way with GCC 6.3.0.


I researched; the change happened in gcc 5 (released april 22 2015).


Alex

--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 0/2] Multi-threaded RPM support

[Alexander Kanavin]

On 06/04/2017 05:15 PM, Richard Purdie wrote:


https://autobuilder.yoctoproject.org/main/builders/nightly-x86/builds/1
163/steps/BuildImages/logs/stdio

https://autobuilder.yoctoproject.org/main/builders/nightly-x86-64/build
s/1203/steps/BuildImages/logs/stdio

https://autobuilder.yoctoproject.org/main/builders/nightly-ipk/builds/1
112/steps/BuildImages/logs/stdio

I suspect the same issue causing failures in multiple places.


The patches as they are now require that the C compiler on the host is 
recent; specifically that it defaults to C11 mode. I'm not sure when GCC 
made the switch, but it works that way with GCC 6.3.0.


In any case, I'll rework the patches to use openmp, as requested by 
upstream - this may however cause a different set of compatibility 
problems with older GCC versions. What are the oldest host compiler 
versions/distro versions that we still need to support in master?



Alex
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 0/2] Multi-threaded RPM support

[Richard Purdie]

On Thu, 2017-06-01 at 18:15 +0300, Alexander Kanavin wrote:
> These two patches add multi-threaded features to RPM to speed up the
> do_package_write_rpm task.
> 
> Specifically:
> 
> 1) Multi-threaded xz compressor is used instead of default single-
> threaded gz. This has the most
> dramatic effect when a recipe produces a smaller number of large-
> sized packages.
> 
> 2) Packages creation is run in multiple threads via thread pools.
> This is most beneficial when
> a recipe produces a large amount of small packages.
> 
> The following changes since commit
> bd063fa288b49b6e3ea77982d0ccc46610feb1ad:
> 
>   tune-mips32*.inc: use consistent comments across all three .inc
> files (2017-05-30 10:15:20 +0100)
> 
> are available in the git repository at:
> 
>   git://git.yoctoproject.org/poky-contrib akanavin/parallel-rpm
>   http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=akanavin/p
> arallel-rpm

https://autobuilder.yoctoproject.org/main/builders/nightly-x86/builds/1
163/steps/BuildImages/logs/stdio

https://autobuilder.yoctoproject.org/main/builders/nightly-x86-64/build
s/1203/steps/BuildImages/logs/stdio

https://autobuilder.yoctoproject.org/main/builders/nightly-ipk/builds/1
112/steps/BuildImages/logs/stdio

I suspect the same issue causing failures in multiple places.

Cheers,

Richard

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 0/2] Multi-threaded RPM support

[Richard Purdie]

I did get a laugh when this showed up in my mail client screen as:

"Multi-threaded RP" :)

Seriously, I am looking forward to trying and hopefully merging this,
should make for some nice speedups.

Cheers,

RP



-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 0/2] Multi-threaded RPM support

[Leonardo Sandoval]

Alex,

I get this when bitbake core-image-minimal

http://errors.yoctoproject.org/Errors/Build/38347/


HEAD is at

http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=akanavin/parallel-rpm


On Thu, 2017-06-01 at 20:37 +0300, Alexander Kanavin wrote:
> On 06/01/2017 08:13 PM, Mark Hatle wrote:
> >>> perl
> >>> before: 1m19s 63M
> >>> after: 55s 42M
> >>>
> >>> python3
> >>> before: 40s 38M
> >>> after: 28s 24M
> >>
> >> it is interesting to see that perl and python does not benefit 'much'.
> >> buildstats should give us a better picture per recipe.
> >
> > This may be due to how we process rpmdeps within OpenEmbedded.. but I'm not
> > sure.  I'd have expected a bigger time difference with large packages (lots 
> > of
> > sub packages) like perl and python3.
> 
> The numbers include the overhead of starting bitbake (some 10 seconds). 
> Subtract that (from both 'before' and 'after') and then it's more 
> impressive :) I tested via
> 
> time bitbake -f -c do_package_write_rpm perl
> 
> Alex
> 


-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 0/2] Multi-threaded RPM support

[Alexander Kanavin]

On 06/01/2017 08:13 PM, Mark Hatle wrote:

perl
before: 1m19s 63M
after: 55s 42M

python3
before: 40s 38M
after: 28s 24M


it is interesting to see that perl and python does not benefit 'much'.
buildstats should give us a better picture per recipe.


This may be due to how we process rpmdeps within OpenEmbedded.. but I'm not
sure.  I'd have expected a bigger time difference with large packages (lots of
sub packages) like perl and python3.


The numbers include the overhead of starting bitbake (some 10 seconds). 
Subtract that (from both 'before' and 'after') and then it's more 
impressive :) I tested via


time bitbake -f -c do_package_write_rpm perl

Alex

--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 0/2] Multi-threaded RPM support

[Mark Hatle]

On 6/1/17 11:52 AM, Leonardo Sandoval wrote:
> On Thu, 2017-06-01 at 18:15 +0300, Alexander Kanavin wrote:

...

>> perl
>> before: 1m19s 63M
>> after: 55s 42M
>>
>> python3
>> before: 40s 38M
>> after: 28s 24M
> 
> it is interesting to see that perl and python does not benefit 'much'.
> buildstats should give us a better picture per recipe.

This may be due to how we process rpmdeps within OpenEmbedded.. but I'm not
sure.  I'd have expected a bigger time difference with large packages (lots of
sub packages) like perl and python3.

(kernel-modules would be another to look at, as it can get quite large.)
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 0/2] Multi-threaded RPM support

[Leonardo Sandoval]

On Thu, 2017-06-01 at 18:15 +0300, Alexander Kanavin wrote:
> These two patches add multi-threaded features to RPM to speed up the 
> do_package_write_rpm task.
> 
> Specifically:
> 
> 1) Multi-threaded xz compressor is used instead of default single-threaded 
> gz. This has the most
> dramatic effect when a recipe produces a smaller number of large-sized 
> packages.
> 
> 2) Packages creation is run in multiple threads via thread pools. This is 
> most beneficial when
> a recipe produces a large amount of small packages.
> 
> Some not very scientific benchmarks for time and .rpm sizes
> (time is measured for do_package_write_rpm tasks only):
> 
> webkitgtk
> before: 9m22s 1550M
> after: 1m50s 858M
> 
> glibc-locale
> before: 3m2s 125M
> after: 40s 56M
> 
> glibc
> before: 56s 54M
> after: 23s 38M
> 
> perl
> before: 1m19s 63M
> after: 55s 42M
> 
> python3
> before: 40s 38M
> after: 28s 24M

it is interesting to see that perl and python does not benefit 'much'.
buildstats should give us a better picture per recipe.

Leo


> 
> The following changes since commit bd063fa288b49b6e3ea77982d0ccc46610feb1ad:
> 
>   tune-mips32*.inc: use consistent comments across all three .inc files 
> (2017-05-30 10:15:20 +0100)
> 
> are available in the git repository at:
> 
>   git://git.yoctoproject.org/poky-contrib akanavin/parallel-rpm
>   
> http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=akanavin/parallel-rpm
> 
> Alexander Kanavin (2):
>   package_rpm.bbclass: use multithreaded xz compression
>   rpm: run binary package generation via thread pools
> 
>  meta/classes/package_rpm.bbclass   |   2 +
>  ...y-package-building-into-a-separate-functi.patch |  83 
>  ...-binary-package-creation-via-thread-pools.patch | 137 +
>  ...c-make-operations-over-string-pools-threa.patch | 209 +++
>  ...c-remove-static-local-variables-from-buil.patch | 227 
> +
>  meta/recipes-devtools/rpm/rpm_git.bb   |   4 +
>  6 files changed, 662 insertions(+)
>  create mode 100644 
> meta/recipes-devtools/rpm/files/0001-Split-binary-package-building-into-a-separate-functi.patch
>  create mode 100644 
> meta/recipes-devtools/rpm/files/0002-Run-binary-package-creation-via-thread-pools.patch
>  create mode 100644 
> meta/recipes-devtools/rpm/files/0003-rpmstrpool.c-make-operations-over-string-pools-threa.patch
>  create mode 100644 
> meta/recipes-devtools/rpm/files/0004-build-pack.c-remove-static-local-variables-from-buil.patch
> 
> -- 
> 2.11.0
> 


-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 0/2] Multi-threaded RPM support

[Alexander Kanavin]

These two patches add multi-threaded features to RPM to speed up the 
do_package_write_rpm task.

Specifically:

1) Multi-threaded xz compressor is used instead of default single-threaded gz. 
This has the most
dramatic effect when a recipe produces a smaller number of large-sized packages.

2) Packages creation is run in multiple threads via thread pools. This is most 
beneficial when
a recipe produces a large amount of small packages.

Some not very scientific benchmarks for time and .rpm sizes
(time is measured for do_package_write_rpm tasks only):

webkitgtk
before: 9m22s 1550M
after: 1m50s 858M

glibc-locale
before: 3m2s 125M
after: 40s 56M

glibc
before: 56s 54M
after: 23s 38M

perl
before: 1m19s 63M
after: 55s 42M

python3
before: 40s 38M
after: 28s 24M

The following changes since commit bd063fa288b49b6e3ea77982d0ccc46610feb1ad:

  tune-mips32*.inc: use consistent comments across all three .inc files 
(2017-05-30 10:15:20 +0100)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib akanavin/parallel-rpm
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=akanavin/parallel-rpm

Alexander Kanavin (2):
  package_rpm.bbclass: use multithreaded xz compression
  rpm: run binary package generation via thread pools

 meta/classes/package_rpm.bbclass   |   2 +
 ...y-package-building-into-a-separate-functi.patch |  83 
 ...-binary-package-creation-via-thread-pools.patch | 137 +
 ...c-make-operations-over-string-pools-threa.patch | 209 +++
 ...c-remove-static-local-variables-from-buil.patch | 227 +
 meta/recipes-devtools/rpm/rpm_git.bb   |   4 +
 6 files changed, 662 insertions(+)
 create mode 100644 
meta/recipes-devtools/rpm/files/0001-Split-binary-package-building-into-a-separate-functi.patch
 create mode 100644 
meta/recipes-devtools/rpm/files/0002-Run-binary-package-creation-via-thread-pools.patch
 create mode 100644 
meta/recipes-devtools/rpm/files/0003-rpmstrpool.c-make-operations-over-string-pools-threa.patch
 create mode 100644 
meta/recipes-devtools/rpm/files/0004-build-pack.c-remove-static-local-variables-from-buil.patch

-- 
2.11.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core