Re: [OE-core] [PATCH 2/2] rpm: run binary package generation via thread pools

[Alexander Kanavin]

On 06/01/2017 07:02 PM, Alexander Kanavin wrote:

On 06/01/2017 06:15 PM, Alexander Kanavin wrote:

This greatly reduces build times when there is a large amount of small
rpm packages to produce. The patches are rather invasive,
and so will be submitted upstream.


And now the pull request is open:

https://github.com/rpm-software-management/rpm/pull/226
https://github.com/rpm-software-management/rpm/issues/211


So the upstream's reaction to the pull request is that they're not 
against this, but they want the multiprocessing to be done with openmp 
(and not nss/nspr), given that they now actually support openssl as an 
alternative to nss, and don't want any hard dependencies on netscape 
stuff. I'll spend some time studying that.


Alex
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 2/2] rpm: run binary package generation via thread pools

[Alexander Kanavin]

On 06/01/2017 06:15 PM, Alexander Kanavin wrote:

This greatly reduces build times when there is a large amount of small
rpm packages to produce. The patches are rather invasive,
and so will be submitted upstream.


And now the pull request is open:

https://github.com/rpm-software-management/rpm/pull/226
https://github.com/rpm-software-management/rpm/issues/211

Alex
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 2/2] rpm: run binary package generation via thread pools

[Alexander Kanavin]

This greatly reduces build times when there is a large amount of small
rpm packages to produce. The patches are rather invasive,
and so will be submitted upstream.

Signed-off-by: Alexander Kanavin 
---
 ...y-package-building-into-a-separate-functi.patch |  83 
 ...-binary-package-creation-via-thread-pools.patch | 137 +
 ...c-make-operations-over-string-pools-threa.patch | 209 +++
 ...c-remove-static-local-variables-from-buil.patch | 227 +
 meta/recipes-devtools/rpm/rpm_git.bb   |   4 +
 5 files changed, 660 insertions(+)
 create mode 100644 
meta/recipes-devtools/rpm/files/0001-Split-binary-package-building-into-a-separate-functi.patch
 create mode 100644 
meta/recipes-devtools/rpm/files/0002-Run-binary-package-creation-via-thread-pools.patch
 create mode 100644 
meta/recipes-devtools/rpm/files/0003-rpmstrpool.c-make-operations-over-string-pools-threa.patch
 create mode 100644 
meta/recipes-devtools/rpm/files/0004-build-pack.c-remove-static-local-variables-from-buil.patch

diff --git 
a/meta/recipes-devtools/rpm/files/0001-Split-binary-package-building-into-a-separate-functi.patch
 
b/meta/recipes-devtools/rpm/files/0001-Split-binary-package-building-into-a-separate-functi.patch
new file mode 100644
index 000..b21c5119041
--- /dev/null
+++ 
b/meta/recipes-devtools/rpm/files/0001-Split-binary-package-building-into-a-separate-functi.patch
@@ -0,0 +1,83 @@
+From b841b699e519438a66b661247c94efff63d0700e Mon Sep 17 00:00:00 2001
+From: Alexander Kanavin 
+Date: Thu, 25 May 2017 18:15:27 +0300
+Subject: [PATCH 01/14] Split binary package building into a separate function
+
+So that it can be run as a thread pool task.
+
+Upstream-Status: Pending
+Signed-off-by: Alexander Kanavin 
+---
+ build/pack.c | 33 +
+ 1 file changed, 21 insertions(+), 12 deletions(-)
+
+diff --git a/build/pack.c b/build/pack.c
+index 497300b96..891e6bdc3 100644
+--- a/build/pack.c
 b/build/pack.c
+@@ -546,18 +546,13 @@ static rpmRC checkPackages(char *pkgcheck)
+ return RPMRC_OK;
+ }
+ 
+-rpmRC packageBinaries(rpmSpec spec, const char *cookie, int cheating)
++static rpmRC packageBinary(rpmSpec spec, Package pkg, const char *cookie, int 
cheating, char** filename)
+ {
+-rpmRC rc;
+-const char *errorString;
+-Package pkg;
+-char *pkglist = NULL;
+-
+-for (pkg = spec->packages; pkg != NULL; pkg = pkg->next) {
+-  char *fn;
++const char *errorString;
++rpmRC rc = RPMRC_OK;
+ 
+   if (pkg->fileList == NULL)
+-  continue;
++  return rc;
+ 
+   if ((rc = processScriptFiles(spec, pkg)))
+   return rc;
+@@ -591,7 +586,7 @@ rpmRC packageBinaries(rpmSpec spec, const char *cookie, 
int cheating)
+headerGetString(pkg->header, RPMTAG_NAME), errorString);
+   return RPMRC_FAIL;
+   }
+-  fn = rpmGetPath("%{_rpmdir}/", binRpm, NULL);
++  *filename = rpmGetPath("%{_rpmdir}/", binRpm, NULL);
+   if ((binDir = strchr(binRpm, '/')) != NULL) {
+   struct stat st;
+   char *dn;
+@@ -613,14 +608,28 @@ rpmRC packageBinaries(rpmSpec spec, const char *cookie, 
int cheating)
+   free(binRpm);
+   }
+ 
+-  rc = writeRPM(pkg, NULL, fn, NULL);
++  rc = writeRPM(pkg, NULL, *filename, NULL);
+   if (rc == RPMRC_OK) {
+   /* Do check each written package if enabled */
+-  char *pkgcheck = rpmExpand("%{?_build_pkgcheck} ", fn, NULL);
++  char *pkgcheck = rpmExpand("%{?_build_pkgcheck} ", *filename, NULL);
+   if (pkgcheck[0] != ' ') {
+   rc = checkPackages(pkgcheck);
+   }
+   free(pkgcheck);
++  }
++return rc;
++}
++
++rpmRC packageBinaries(rpmSpec spec, const char *cookie, int cheating)
++{
++rpmRC rc;
++Package pkg;
++char *pkglist = NULL;
++
++for (pkg = spec->packages; pkg != NULL; pkg = pkg->next) {
++  char *fn = NULL;
++rc = packageBinary(spec, pkg, cookie, cheating, );
++  if (rc == RPMRC_OK) {
+   rstrcat(, fn);
+   rstrcat(, " ");
+   }
+-- 
+2.11.0
+
diff --git 
a/meta/recipes-devtools/rpm/files/0002-Run-binary-package-creation-via-thread-pools.patch
 
b/meta/recipes-devtools/rpm/files/0002-Run-binary-package-creation-via-thread-pools.patch
new file mode 100644
index 000..2987a979698
--- /dev/null
+++ 
b/meta/recipes-devtools/rpm/files/0002-Run-binary-package-creation-via-thread-pools.patch
@@ -0,0 +1,137 @@
+From 45d92254cea0bceff38c42cbad89b301a0c07cea Mon Sep 17 00:00:00 2001
+From: Alexander Kanavin 
+Date: Thu, 25 May 2017 19:30:20 +0300
+Subject: [PATCH 02/14] Run binary package creation via thread pools.
+
+Upstream-Status: Pending
+Signed-off-by: Alexander Kanavin 
+---
+ build/pack.c | 100