Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling from libc recipe.

2011-07-08 Thread Phil Blundell 
On Mon, 2011-06-27 at 16:37 +0800, Dongxiao Xu wrote:
 +do_install_locale () {
 + dest=${D}/${includedir}/glibc-locale-internal-${MULTIMACH_TARGET_SYS}
 + install -d ${dest} ${dest}${bindir}
 + cp -fpPR ${D}${base_libdir} ${dest}${base_prefix}
 + cp -fpPR ${D}${libdir} ${dest}${exec_prefix}
 + cp -fpPR ${D}${datadir} ${dest}${exec_prefix}
 + cp -fpPR ${D}${bindir}/localedef ${dest}${bindir}
 + cp -fpPR ${WORKDIR}/SUPPORTED ${dest}
 +}

This turns out to lose if you don't have libc-locale-code in
DISTRO_FEATURES, since then localedef isn't installed and it blows up
trying to copy that file.

p.



___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling from libc recipe.

2011-06-28 Thread Richard Purdie 
On Tue, 2011-06-28 at 10:07 +0100, Phil Blundell wrote:
 On Tue, 2011-06-28 at 08:51 +0800, Xu, Dongxiao wrote:
   -Original Message-
   From: openembedded-core-boun...@lists.openembedded.org
   [mailto:openembedded-core-boun...@lists.openembedded.org] On Behalf Of
   Phil Blundell
   Sent: Monday, June 27, 2011 4:58 PM
   To: Patches and discussions about the oe-core layer
   Subject: Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling 
   from libc
   recipe.
   
   On Mon, 2011-06-27 at 16:37 +0800, Dongxiao Xu wrote:
-PACKAGES = ${PN}-dbg ${PN} catchsegv${PKGSUFFIX} sln${PKGSUFFIX}
   nscd${PKGSUFFIX} ldd${PKGSUFFIX} localedef${PKGSUFFIX} libcidn ${PN}-utils
   ${PN}-pic ${PN}-dev eglibc-doc eglibc-locale libmemusage
   libsegfault${PKGSUFFIX} eglibc-extra-nss eglibc-thread-db${PKGSUFFIX}
   eglibc-pcprofile libsotruss${PKGSUFFIX}
-
-PACKAGES_DYNAMIC =  \
-   glibc-gconv-*${PKGSUFFIX} glibc-charmap-* 
glibc-localedata-*
   glibc-binary-localedata-* \
-   eglibc-gconv-* eglibc-charmap-* eglibc-localedata-*
   eglibc-binary-localedata-* \
-   locale-base-*${PKGSUFFIX}
+PACKAGES = ${PN}-dbg ${PN} catchsegv${PKGSUFFIX} sln${PKGSUFFIX}
   nscd${PKGSUFFIX} ldd${PKGSUFFIX} ${PN}-utils ${PN}-pic ${PN}-dev 
   eglibc-doc
   libcidn libmemusage libsegfault${PKGSUFFIX} eglibc-extra-nss
   eglibc-thread-db${PKGSUFFIX} eglibc-pcprofile
   
   You seem to have made a bunch of changes here that are not related to 
   locales.
   What are those about?
  
  They are locale related changes.
  Locale related stuffs in the above PACKAGES and PACKAGES_DYNAMIC are moved 
  to eglibc-locale recipe.
 
 Is libsotruss${PKGSUFFIX} (for example) really a locale related stuff?
 It's not obvious to me what the connection is.

Dongxiao: What Phil is referring to here is that libsotruss${PKGSUFFIX}
disappeared from PACKAGES and that also ${libdir}/audit/.debug
disappeared from FILES_${PN}-dbg.

Specifically these changes came in as part of:

http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=77c2dd944db42e40cc8467e6bc5a175fed90c251

which I don't think your patch accounted for.

Since this patch has been around for a while and it otherwise looks
good, I've fixed up these couple of issues and merged it though.

Cheers,

Richard


___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling from libc recipe.

2011-06-28 Thread Koen Kooi 

Op 28 jun 2011, om 13:07 heeft Richard Purdie het volgende geschreven:

 On Tue, 2011-06-28 at 10:07 +0100, Phil Blundell wrote:
 On Tue, 2011-06-28 at 08:51 +0800, Xu, Dongxiao wrote:
 -Original Message-
 From: openembedded-core-boun...@lists.openembedded.org
 [mailto:openembedded-core-boun...@lists.openembedded.org] On Behalf Of
 Phil Blundell
 Sent: Monday, June 27, 2011 4:58 PM
 To: Patches and discussions about the oe-core layer
 Subject: Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling from 
 libc
 recipe.
 
 On Mon, 2011-06-27 at 16:37 +0800, Dongxiao Xu wrote:
 -PACKAGES = ${PN}-dbg ${PN} catchsegv${PKGSUFFIX} sln${PKGSUFFIX}
 nscd${PKGSUFFIX} ldd${PKGSUFFIX} localedef${PKGSUFFIX} libcidn ${PN}-utils
 ${PN}-pic ${PN}-dev eglibc-doc eglibc-locale libmemusage
 libsegfault${PKGSUFFIX} eglibc-extra-nss eglibc-thread-db${PKGSUFFIX}
 eglibc-pcprofile libsotruss${PKGSUFFIX}
 -
 -PACKAGES_DYNAMIC =  \
 - glibc-gconv-*${PKGSUFFIX} glibc-charmap-* glibc-localedata-*
 glibc-binary-localedata-* \
 - eglibc-gconv-* eglibc-charmap-* eglibc-localedata-*
 eglibc-binary-localedata-* \
 - locale-base-*${PKGSUFFIX}
 +PACKAGES = ${PN}-dbg ${PN} catchsegv${PKGSUFFIX} sln${PKGSUFFIX}
 nscd${PKGSUFFIX} ldd${PKGSUFFIX} ${PN}-utils ${PN}-pic ${PN}-dev eglibc-doc
 libcidn libmemusage libsegfault${PKGSUFFIX} eglibc-extra-nss
 eglibc-thread-db${PKGSUFFIX} eglibc-pcprofile
 
 You seem to have made a bunch of changes here that are not related to 
 locales.
 What are those about?
 
 They are locale related changes.
 Locale related stuffs in the above PACKAGES and PACKAGES_DYNAMIC are moved 
 to eglibc-locale recipe.
 
 Is libsotruss${PKGSUFFIX} (for example) really a locale related stuff?
 It's not obvious to me what the connection is.
 
 Dongxiao: What Phil is referring to here is that libsotruss${PKGSUFFIX}
 disappeared from PACKAGES and that also ${libdir}/audit/.debug
 disappeared from FILES_${PN}-dbg.
 
 Specifically these changes came in as part of:
 
 http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=77c2dd944db42e40cc8467e6bc5a175fed90c251
 
 which I don't think your patch accounted for.
 
 Since this patch has been around for a while and it otherwise looks
 good, I've fixed up these couple of issues and merged it though.

This breaks when using eglibc 2.12 :(
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling from libc recipe.

2011-06-28 Thread Richard Purdie 
On Tue, 2011-06-28 at 14:17 +0200, Koen Kooi wrote:
 Op 28 jun 2011, om 13:07 heeft Richard Purdie het volgende geschreven:
  On Tue, 2011-06-28 at 10:07 +0100, Phil Blundell wrote:

  
  Dongxiao: What Phil is referring to here is that libsotruss${PKGSUFFIX}
  disappeared from PACKAGES and that also ${libdir}/audit/.debug
  disappeared from FILES_${PN}-dbg.
  
  Specifically these changes came in as part of:
  
  http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=77c2dd944db42e40cc8467e6bc5a175fed90c251
  
  which I don't think your patch accounted for.
  
  Since this patch has been around for a while and it otherwise looks
  good, I've fixed up these couple of issues and merged it though.
 
 This breaks when using eglibc 2.12 :(

Sorry, I've pushed some cleanup to resolve that.

Cheers,

Richard


___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling from libc recipe.

2011-06-28 Thread Koen Kooi 

Op 28 jun 2011, om 16:00 heeft Richard Purdie het volgende geschreven:

 On Tue, 2011-06-28 at 14:17 +0200, Koen Kooi wrote:
 Op 28 jun 2011, om 13:07 heeft Richard Purdie het volgende geschreven:
 On Tue, 2011-06-28 at 10:07 +0100, Phil Blundell wrote:
 
 
 Dongxiao: What Phil is referring to here is that libsotruss${PKGSUFFIX}
 disappeared from PACKAGES and that also ${libdir}/audit/.debug
 disappeared from FILES_${PN}-dbg.
 
 Specifically these changes came in as part of:
 
 http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=77c2dd944db42e40cc8467e6bc5a175fed90c251
 
 which I don't think your patch accounted for.
 
 Since this patch has been around for a while and it otherwise looks
 good, I've fixed up these couple of issues and merged it though.
 
 This breaks when using eglibc 2.12 :(
 
 Sorry, I've pushed some cleanup to resolve that.

So after my shlib renaming patch I still can't build any image, since 
locale-base-* has disappeared. Is there anything related to libc and 
libc-locales this patch *didn't* break?
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling from libc recipe.

2011-06-27 Thread Phil Blundell 
On Mon, 2011-06-27 at 16:37 +0800, Dongxiao Xu wrote:
 -PACKAGES = ${PN}-dbg ${PN} catchsegv${PKGSUFFIX} sln${PKGSUFFIX} 
 nscd${PKGSUFFIX} ldd${PKGSUFFIX} localedef${PKGSUFFIX} libcidn ${PN}-utils 
 ${PN}-pic ${PN}-dev eglibc-doc eglibc-locale libmemusage 
 libsegfault${PKGSUFFIX} eglibc-extra-nss eglibc-thread-db${PKGSUFFIX} 
 eglibc-pcprofile libsotruss${PKGSUFFIX}
 -
 -PACKAGES_DYNAMIC =  \
 - glibc-gconv-*${PKGSUFFIX} glibc-charmap-* 
 glibc-localedata-* glibc-binary-localedata-* \
 - eglibc-gconv-* eglibc-charmap-* eglibc-localedata-* 
 eglibc-binary-localedata-* \
 - locale-base-*${PKGSUFFIX}
 +PACKAGES = ${PN}-dbg ${PN} catchsegv${PKGSUFFIX} sln${PKGSUFFIX} 
 nscd${PKGSUFFIX} ldd${PKGSUFFIX} ${PN}-utils ${PN}-pic ${PN}-dev eglibc-doc 
 libcidn libmemusage libsegfault${PKGSUFFIX} eglibc-extra-nss 
 eglibc-thread-db${PKGSUFFIX} eglibc-pcprofile

You seem to have made a bunch of changes here that are not related to
locales.  What are those about?

 -FILES_${PN}-dbg += ${libexecdir}/*/.debug ${libdir}/gconv/.debug 
 ${libdir}/audit/.debug
 +FILES_${PN}-dbg += ${libexecdir}/*/.debug

Likewise.

Also, why is localedef getting packaged by libc-locale?  Not that I
think it's necessarily a problem but it seems a little bit strange.

p.


___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling from libc recipe.

2011-06-27 Thread Xu, Dongxiao 


 -Original Message-
 From: openembedded-core-boun...@lists.openembedded.org
 [mailto:openembedded-core-boun...@lists.openembedded.org] On Behalf Of
 Phil Blundell
 Sent: Monday, June 27, 2011 4:58 PM
 To: Patches and discussions about the oe-core layer
 Subject: Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling from 
 libc
 recipe.
 
 On Mon, 2011-06-27 at 16:37 +0800, Dongxiao Xu wrote:
  -PACKAGES = ${PN}-dbg ${PN} catchsegv${PKGSUFFIX} sln${PKGSUFFIX}
 nscd${PKGSUFFIX} ldd${PKGSUFFIX} localedef${PKGSUFFIX} libcidn ${PN}-utils
 ${PN}-pic ${PN}-dev eglibc-doc eglibc-locale libmemusage
 libsegfault${PKGSUFFIX} eglibc-extra-nss eglibc-thread-db${PKGSUFFIX}
 eglibc-pcprofile libsotruss${PKGSUFFIX}
  -
  -PACKAGES_DYNAMIC =  \
  -   glibc-gconv-*${PKGSUFFIX} glibc-charmap-* glibc-localedata-*
 glibc-binary-localedata-* \
  -   eglibc-gconv-* eglibc-charmap-* eglibc-localedata-*
 eglibc-binary-localedata-* \
  -   locale-base-*${PKGSUFFIX}
  +PACKAGES = ${PN}-dbg ${PN} catchsegv${PKGSUFFIX} sln${PKGSUFFIX}
 nscd${PKGSUFFIX} ldd${PKGSUFFIX} ${PN}-utils ${PN}-pic ${PN}-dev eglibc-doc
 libcidn libmemusage libsegfault${PKGSUFFIX} eglibc-extra-nss
 eglibc-thread-db${PKGSUFFIX} eglibc-pcprofile
 
 You seem to have made a bunch of changes here that are not related to locales.
 What are those about?

They are locale related changes.
Locale related stuffs in the above PACKAGES and PACKAGES_DYNAMIC are moved to 
eglibc-locale recipe.

 
  -FILES_${PN}-dbg += ${libexecdir}/*/.debug ${libdir}/gconv/.debug
 ${libdir}/audit/.debug
  +FILES_${PN}-dbg += ${libexecdir}/*/.debug
 
 Likewise.
 
 Also, why is localedef getting packaged by libc-locale?  Not that I think it's
 necessarily a problem but it seems a little bit strange.

Originally I thought of putting all locale related things to eglibc-locale.
Now actually virtual/libiconv belongs to eglibc.
Thus I am OK to move localedef from eglibc-locale to eglibc, and will include 
this in my next pull request.

Thanks,
Dongxiao

 
 p.
 
 
 ___
 Openembedded-core mailing list
 Openembedded-core@lists.openembedded.org
 http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling from libc recipe.

2011-06-27 Thread Xu, Dongxiao 
 -Original Message-
 From: Xu, Dongxiao
 Sent: Tuesday, June 28, 2011 8:52 AM
 To: Patches and discussions about the oe-core layer
 Subject: RE: [OE-core] [PATCH 1/1] libc-locale: split locale handling from 
 libc
 recipe.
 
 
 
  -Original Message-
  From: openembedded-core-boun...@lists.openembedded.org
  [mailto:openembedded-core-boun...@lists.openembedded.org] On Behalf
 Of
  Phil Blundell
  Sent: Monday, June 27, 2011 4:58 PM
  To: Patches and discussions about the oe-core layer
  Subject: Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling
  from libc recipe.
 
  On Mon, 2011-06-27 at 16:37 +0800, Dongxiao Xu wrote:
   -PACKAGES = ${PN}-dbg ${PN} catchsegv${PKGSUFFIX} sln${PKGSUFFIX}
  nscd${PKGSUFFIX} ldd${PKGSUFFIX} localedef${PKGSUFFIX} libcidn
  ${PN}-utils ${PN}-pic ${PN}-dev eglibc-doc eglibc-locale libmemusage
  libsegfault${PKGSUFFIX} eglibc-extra-nss eglibc-thread-db${PKGSUFFIX}
  eglibc-pcprofile libsotruss${PKGSUFFIX}
   -
   -PACKAGES_DYNAMIC =  \
   - glibc-gconv-*${PKGSUFFIX} glibc-charmap-* glibc-localedata-*
  glibc-binary-localedata-* \
   - eglibc-gconv-* eglibc-charmap-* eglibc-localedata-*
  eglibc-binary-localedata-* \
   - locale-base-*${PKGSUFFIX}
   +PACKAGES = ${PN}-dbg ${PN} catchsegv${PKGSUFFIX} sln${PKGSUFFIX}
  nscd${PKGSUFFIX} ldd${PKGSUFFIX} ${PN}-utils ${PN}-pic ${PN}-dev
  eglibc-doc libcidn libmemusage libsegfault${PKGSUFFIX}
  eglibc-extra-nss eglibc-thread-db${PKGSUFFIX} eglibc-pcprofile
 
  You seem to have made a bunch of changes here that are not related to
 locales.
  What are those about?
 
 They are locale related changes.
 Locale related stuffs in the above PACKAGES and PACKAGES_DYNAMIC are
 moved to eglibc-locale recipe.
 
 
   -FILES_${PN}-dbg += ${libexecdir}/*/.debug ${libdir}/gconv/.debug
  ${libdir}/audit/.debug
   +FILES_${PN}-dbg += ${libexecdir}/*/.debug
 
  Likewise.
 
  Also, why is localedef getting packaged by libc-locale?  Not that I
  think it's necessarily a problem but it seems a little bit strange.
 
 Originally I thought of putting all locale related things to eglibc-locale.
 Now actually virtual/libiconv belongs to eglibc.
 Thus I am OK to move localedef from eglibc-locale to eglibc, and will include 
 this
 in my next pull request.

Just did an experiment to move localedef in eglibc instead of eglibc-locale, 
however eglibc-locale's do_package met errors since it needs localedef to 
generate locales, see libc-packages.inc.

Therefore localedef still needs to be packaged in eglibc-locale recipe.

Thanks,
Dongxiao

 
 Thanks,
 Dongxiao
 
 
  p.
 
 
  ___
  Openembedded-core mailing list
  Openembedded-core@lists.openembedded.org
  http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling from libc recipe.

2011-06-26 Thread Xu, Dongxiao 
 -Original Message-
 From: openembedded-core-boun...@lists.openembedded.org
 [mailto:openembedded-core-boun...@lists.openembedded.org] On Behalf Of
 Khem Raj
 Sent: Friday, June 24, 2011 7:43 AM
 To: openembedded-core@lists.openembedded.org
 Subject: Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling from 
 libc
 recipe.
 
 On 06/23/2011 03:14 AM, Richard Purdie wrote:
  On Thu, 2011-06-23 at 10:40 +0100, Phil Blundell wrote:
  On Thu, 2011-06-23 at 12:08 +0800, Xu, Dongxiao wrote:
  -Original Message-
  From: openembedded-core-boun...@lists.openembedded.org
  [mailto:openembedded-core-boun...@lists.openembedded.org] On
 Behalf
  Of Phil Blundell
  Sent: Wednesday, June 22, 2011 7:45 PM
  To: Patches and discussions about the oe-core layer
  Subject: Re: [OE-core] [PATCH 1/1] libc-locale: split locale
  handling from libc recipe.
 
  I still don't quite understand why the virtual/libiconv provider
  has moved from eglibc to eglibc-locale.  Can you explain what's going on
 there?
 
  Generally, the purpose of this patch is to reduce eglibc's do_package 
  time.
 
  Since many other recipes' do_package have dependency on eglibc's
 do_package, thus the sooner eglibc could finish its do_package, the ealier can
 other do_packages be executed, which improves the parallelism.
 
   From our study, we found most of eglibc's do_package time is used to
 handle locale. Therefore we split all the locale related stuffs into a 
 separately
 recipe eglibc-locale.
 
  Yes, I understand that.  But I don't think this answers the question
  about why, specifically, glibc-locale is now PROVIDEing
  virtual/libiconv.  Does that recipe, in fact, implement libiconv?
 
  I must admit I was wondering about this too. Isn't the libiconv
  functionality still packaged as part of the libc recipe, not the
  locale part?
 
 yes it should be in libc

Oh, it should be a mistake. I have changed that in my patch and doing test for 
it now.

Thanks,
Dongxiao

 
  Cheers,
 
  Richard
 
 
  ___
  Openembedded-core mailing list
  Openembedded-core@lists.openembedded.org
  http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
 
 
 ___
 Openembedded-core mailing list
 Openembedded-core@lists.openembedded.org
 http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling from libc recipe.

2011-06-23 Thread Phil Blundell 
On Thu, 2011-06-23 at 12:08 +0800, Xu, Dongxiao wrote:
  -Original Message-
  From: openembedded-core-boun...@lists.openembedded.org
  [mailto:openembedded-core-boun...@lists.openembedded.org] On Behalf Of
  Phil Blundell
  Sent: Wednesday, June 22, 2011 7:45 PM
  To: Patches and discussions about the oe-core layer
  Subject: Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling from 
  libc
  recipe.
  
  I still don't quite understand why the virtual/libiconv provider has moved 
  from
  eglibc to eglibc-locale.  Can you explain what's going on there?
 
 Generally, the purpose of this patch is to reduce eglibc's do_package time.
 
 Since many other recipes' do_package have dependency on eglibc's do_package, 
 thus the sooner eglibc could finish its do_package, the ealier can other 
 do_packages be executed, which improves the parallelism. 
 
 From our study, we found most of eglibc's do_package time is used to handle 
 locale. Therefore we split all the locale related stuffs into a separately 
 recipe eglibc-locale.

Yes, I understand that.  But I don't think this answers the question
about why, specifically, glibc-locale is now PROVIDEing
virtual/libiconv.  Does that recipe, in fact, implement libiconv?

p.

 
 Thanks,
 Dongxiao
 
  
  p.
  
  On Wed, 2011-06-22 at 17:01 +0800, Dongxiao Xu wrote:
   *libc's do_package will cost a lot of time due to the locale handing,
   which may delay the other recipe's do_package task and affect the
   build performance.
  
   This commit moves locale handling into a separate recipe *libc-locale.
  
   Signed-off-by: Dongxiao Xu dongxiao...@intel.com
   ---
meta/classes/libc-common.bbclass   |   23 
meta/classes/libc-package.bbclass  |   24 
meta/conf/distro/include/tclibc-eglibc.inc |2 +-
meta/conf/distro/include/tclibc-glibc.inc  |2 +-
meta/recipes-core/eglibc/eglibc-initial_2.13.bb|4 ++
meta/recipes-core/eglibc/eglibc-locale_2.13.bb |   58
  
meta/recipes-core/eglibc/eglibc-package.inc|   32 +--
meta/recipes-core/eglibc/eglibc.inc|4 +-
meta/recipes-core/eglibc/eglibc_2.13.bb|   12 
meta/recipes-core/glibc/glibc-initial_2.10.1.bb|4 ++
meta/recipes-core/glibc/glibc-locale_2.10.1.bb |   50
  +
meta/recipes-core/glibc/glibc-package.inc  |   29 ++-
meta/recipes-core/glibc/glibc.inc  |2 +-
meta/recipes-core/glibc/glibc_2.10.1.bb|   12 
.../meta/external-csl-toolchain_2008q3-72.bb   |1 +
15 files changed, 176 insertions(+), 83 deletions(-)  create mode
   100644 meta/classes/libc-common.bbclass  create mode 100644
   meta/recipes-core/eglibc/eglibc-locale_2.13.bb
create mode 100644 meta/recipes-core/glibc/glibc-locale_2.10.1.bb
  
   diff --git a/meta/conf/distro/include/tclibc-eglibc.inc
   b/meta/conf/distro/include/tclibc-eglibc.inc
   index a4c648e..90de04f 100644
   --- a/meta/conf/distro/include/tclibc-eglibc.inc
   +++ b/meta/conf/distro/include/tclibc-eglibc.inc
   @@ -10,7 +10,7 @@ TARGET_OS_powerpc =
   linux${@['','-gnuspe'][bb.data.getVar('BASE_PACKAGE_ARCH',d
# Add glibc overrides to the overrides for eglibc.
OVERRIDES .= :libc-glibc
  
   -PREFERRED_PROVIDER_virtual/libiconv ?= eglibc
   +PREFERRED_PROVIDER_virtual/libiconv ?= eglibc-locale
PREFERRED_PROVIDER_virtual/libiconv-nativesdk ?= eglibc-nativesdk
PREFERRED_PROVIDER_virtual/libintl ?= eglibc
PREFERRED_PROVIDER_virtual/libc ?= eglibc
   diff --git a/meta/conf/distro/include/tclibc-glibc.inc
   b/meta/conf/distro/include/tclibc-glibc.inc
   index e5843b0..748c23f 100644
   --- a/meta/conf/distro/include/tclibc-glibc.inc
   +++ b/meta/conf/distro/include/tclibc-glibc.inc
   @@ -10,7 +10,7 @@ TARGET_OS_powerpc =
   linux${@['','-gnuspe'][bb.data.getVar('BASE_PACKAGE_ARCH',d
# Add glibc to the overrides.
OVERRIDES =. libc-glibc:
  
   -PREFERRED_PROVIDER_virtual/libiconv ?= glibc
   +PREFERRED_PROVIDER_virtual/libiconv ?= glibc-locale
PREFERRED_PROVIDER_virtual/libiconv-nativesdk ?= glibc-nativesdk
PREFERRED_PROVIDER_virtual/libintl ?= glibc
PREFERRED_PROVIDER_virtual/libc ?= glibc
  
  
  
  ___
  Openembedded-core mailing list
  Openembedded-core@lists.openembedded.org
  http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
 
 ___
 Openembedded-core mailing list
 Openembedded-core@lists.openembedded.org
 http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core



___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling from libc recipe.

2011-06-23 Thread Khem Raj 

On 06/23/2011 03:14 AM, Richard Purdie wrote:

On Thu, 2011-06-23 at 10:40 +0100, Phil Blundell wrote:

On Thu, 2011-06-23 at 12:08 +0800, Xu, Dongxiao wrote:

-Original Message-
From: openembedded-core-boun...@lists.openembedded.org
[mailto:openembedded-core-boun...@lists.openembedded.org] On Behalf Of
Phil Blundell
Sent: Wednesday, June 22, 2011 7:45 PM
To: Patches and discussions about the oe-core layer
Subject: Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling from libc
recipe.

I still don't quite understand why the virtual/libiconv provider has moved from
eglibc to eglibc-locale.  Can you explain what's going on there?


Generally, the purpose of this patch is to reduce eglibc's do_package time.

Since many other recipes' do_package have dependency on eglibc's do_package, 
thus the sooner eglibc could finish its do_package, the ealier can other 
do_packages be executed, which improves the parallelism.

 From our study, we found most of eglibc's do_package time is used to handle 
locale. Therefore we split all the locale related stuffs into a separately 
recipe eglibc-locale.


Yes, I understand that.  But I don't think this answers the question
about why, specifically, glibc-locale is now PROVIDEing
virtual/libiconv.  Does that recipe, in fact, implement libiconv?


I must admit I was wondering about this too. Isn't the libiconv
functionality still packaged as part of the libc recipe, not the locale
part?


yes it should be in libc


Cheers,

Richard


___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core



___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling from libc recipe.

2011-06-22 Thread Phil Blundell 
I still don't quite understand why the virtual/libiconv provider has
moved from eglibc to eglibc-locale.  Can you explain what's going on
there?

p.

On Wed, 2011-06-22 at 17:01 +0800, Dongxiao Xu wrote:
 *libc's do_package will cost a lot of time due to the locale handing,
 which may delay the other recipe's do_package task and affect the build
 performance.
 
 This commit moves locale handling into a separate recipe *libc-locale.
 
 Signed-off-by: Dongxiao Xu dongxiao...@intel.com
 ---
  meta/classes/libc-common.bbclass   |   23 
  meta/classes/libc-package.bbclass  |   24 
  meta/conf/distro/include/tclibc-eglibc.inc |2 +-
  meta/conf/distro/include/tclibc-glibc.inc  |2 +-
  meta/recipes-core/eglibc/eglibc-initial_2.13.bb|4 ++
  meta/recipes-core/eglibc/eglibc-locale_2.13.bb |   58 
 
  meta/recipes-core/eglibc/eglibc-package.inc|   32 +--
  meta/recipes-core/eglibc/eglibc.inc|4 +-
  meta/recipes-core/eglibc/eglibc_2.13.bb|   12 
  meta/recipes-core/glibc/glibc-initial_2.10.1.bb|4 ++
  meta/recipes-core/glibc/glibc-locale_2.10.1.bb |   50 +
  meta/recipes-core/glibc/glibc-package.inc  |   29 ++-
  meta/recipes-core/glibc/glibc.inc  |2 +-
  meta/recipes-core/glibc/glibc_2.10.1.bb|   12 
  .../meta/external-csl-toolchain_2008q3-72.bb   |1 +
  15 files changed, 176 insertions(+), 83 deletions(-)
  create mode 100644 meta/classes/libc-common.bbclass
  create mode 100644 meta/recipes-core/eglibc/eglibc-locale_2.13.bb
  create mode 100644 meta/recipes-core/glibc/glibc-locale_2.10.1.bb
 
 diff --git a/meta/conf/distro/include/tclibc-eglibc.inc 
 b/meta/conf/distro/include/tclibc-eglibc.inc
 index a4c648e..90de04f 100644
 --- a/meta/conf/distro/include/tclibc-eglibc.inc
 +++ b/meta/conf/distro/include/tclibc-eglibc.inc
 @@ -10,7 +10,7 @@ TARGET_OS_powerpc = 
 linux${@['','-gnuspe'][bb.data.getVar('BASE_PACKAGE_ARCH',d
  # Add glibc overrides to the overrides for eglibc.
  OVERRIDES .= :libc-glibc
  
 -PREFERRED_PROVIDER_virtual/libiconv ?= eglibc
 +PREFERRED_PROVIDER_virtual/libiconv ?= eglibc-locale
  PREFERRED_PROVIDER_virtual/libiconv-nativesdk ?= eglibc-nativesdk
  PREFERRED_PROVIDER_virtual/libintl ?= eglibc
  PREFERRED_PROVIDER_virtual/libc ?= eglibc
 diff --git a/meta/conf/distro/include/tclibc-glibc.inc 
 b/meta/conf/distro/include/tclibc-glibc.inc
 index e5843b0..748c23f 100644
 --- a/meta/conf/distro/include/tclibc-glibc.inc
 +++ b/meta/conf/distro/include/tclibc-glibc.inc
 @@ -10,7 +10,7 @@ TARGET_OS_powerpc = 
 linux${@['','-gnuspe'][bb.data.getVar('BASE_PACKAGE_ARCH',d
  # Add glibc to the overrides.
  OVERRIDES =. libc-glibc:
  
 -PREFERRED_PROVIDER_virtual/libiconv ?= glibc
 +PREFERRED_PROVIDER_virtual/libiconv ?= glibc-locale
  PREFERRED_PROVIDER_virtual/libiconv-nativesdk ?= glibc-nativesdk
  PREFERRED_PROVIDER_virtual/libintl ?= glibc
  PREFERRED_PROVIDER_virtual/libc ?= glibc



___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling from libc recipe.

2011-06-22 Thread Khem Raj 

On 06/22/2011 02:01 AM, Dongxiao Xu wrote:

*libc's do_package will cost a lot of time due to the locale handing,
which may delay the other recipe's do_package task and affect the build
performance.


you could also move the locale generation into a task of its own instead 
of new recipe




This commit moves locale handling into a separate recipe *libc-locale.

Signed-off-by: Dongxiao Xudongxiao...@intel.com
---
  meta/classes/libc-common.bbclass   |   23 
  meta/classes/libc-package.bbclass  |   24 
  meta/conf/distro/include/tclibc-eglibc.inc |2 +-
  meta/conf/distro/include/tclibc-glibc.inc  |2 +-
  meta/recipes-core/eglibc/eglibc-initial_2.13.bb|4 ++
  meta/recipes-core/eglibc/eglibc-locale_2.13.bb |   58 
  meta/recipes-core/eglibc/eglibc-package.inc|   32 +--
  meta/recipes-core/eglibc/eglibc.inc|4 +-
  meta/recipes-core/eglibc/eglibc_2.13.bb|   12 
  meta/recipes-core/glibc/glibc-initial_2.10.1.bb|4 ++
  meta/recipes-core/glibc/glibc-locale_2.10.1.bb |   50 +
  meta/recipes-core/glibc/glibc-package.inc  |   29 ++-
  meta/recipes-core/glibc/glibc.inc  |2 +-
  meta/recipes-core/glibc/glibc_2.10.1.bb|   12 
  .../meta/external-csl-toolchain_2008q3-72.bb   |1 +
  15 files changed, 176 insertions(+), 83 deletions(-)
  create mode 100644 meta/classes/libc-common.bbclass
  create mode 100644 meta/recipes-core/eglibc/eglibc-locale_2.13.bb
  create mode 100644 meta/recipes-core/glibc/glibc-locale_2.10.1.bb

diff --git a/meta/classes/libc-common.bbclass b/meta/classes/libc-common.bbclass
new file mode 100644
index 000..bae0ace
--- /dev/null
+++ b/meta/classes/libc-common.bbclass
@@ -0,0 +1,23 @@
+do_install() {
+   oe_runmake install_root=${D} install
+   for r in ${rpcsvc}; do
+   h=`echo $r|sed -e's,\.x$,.h,'`
+   install -m 0644 ${S}/sunrpc/rpcsvc/$h ${D}/${includedir}/rpcsvc/
+   done
+   install -m 0644 ${WORKDIR}/etc/ld.so.conf ${D}/${sysconfdir}/
+   install -d ${D}${libdir}/locale
+   make -f ${WORKDIR}/generate-supported.mk IN=${S}/localedata/SUPPORTED 
OUT=${WORKDIR}/SUPPORTED
+   # get rid of some broken files...
+   for i in ${GLIBC_BROKEN_LOCALES}; do
+   grep -v $i ${WORKDIR}/SUPPORTED  ${WORKDIR}/SUPPORTED.tmp
+   mv ${WORKDIR}/SUPPORTED.tmp ${WORKDIR}/SUPPORTED
+   done
+   rm -f ${D}{sysconfdir}/rpc
+   rm -rf ${D}${datadir}/zoneinfo
+   rm -rf ${D}${libexecdir}/getconf
+}
+
+def get_libc_fpu_setting(bb, d):
+if bb.data.getVar('TARGET_FPU', d, 1) in [ 'soft' ]:
+return --without-fp
+return 
diff --git a/meta/classes/libc-package.bbclass 
b/meta/classes/libc-package.bbclass
index 55e3d48..5737af4 100644
--- a/meta/classes/libc-package.bbclass
+++ b/meta/classes/libc-package.bbclass
@@ -37,11 +37,6 @@ python __anonymous () {
  break
  }

-def get_libc_fpu_setting(bb, d):
-if bb.data.getVar('TARGET_FPU', d, 1) in [ 'soft' ]:
-return --without-fp
-return 
-
  OVERRIDES_append = :${TARGET_ARCH}-${TARGET_OS}

  do_configure_prepend() {
@@ -84,25 +79,6 @@ rm -rf ${TMP_LOCALE}
  }


-do_install() {
-   oe_runmake install_root=${D} install
-   for r in ${rpcsvc}; do
-   h=`echo $r|sed -e's,\.x$,.h,'`
-   install -m 0644 ${S}/sunrpc/rpcsvc/$h ${D}/${includedir}/rpcsvc/
-   done
-   install -m 0644 ${WORKDIR}/etc/ld.so.conf ${D}/${sysconfdir}/
-   install -d ${D}${libdir}/locale
-   make -f ${WORKDIR}/generate-supported.mk IN=${S}/localedata/SUPPORTED 
OUT=${WORKDIR}/SUPPORTED
-   # get rid of some broken files...
-   for i in ${GLIBC_BROKEN_LOCALES}; do
-   grep -v $i ${WORKDIR}/SUPPORTED  ${WORKDIR}/SUPPORTED.tmp
-   mv ${WORKDIR}/SUPPORTED.tmp ${WORKDIR}/SUPPORTED
-   done
-   rm -f ${D}${sysconfdir}/rpc
-   rm -rf ${D}${datadir}/zoneinfo
-   rm -rf ${D}${libexecdir}/getconf
-}
-
  TMP_LOCALE=/tmp/locale${libdir}/locale

  do_prep_locale_tree() {
diff --git a/meta/conf/distro/include/tclibc-eglibc.inc 
b/meta/conf/distro/include/tclibc-eglibc.inc
index a4c648e..90de04f 100644
--- a/meta/conf/distro/include/tclibc-eglibc.inc
+++ b/meta/conf/distro/include/tclibc-eglibc.inc
@@ -10,7 +10,7 @@ TARGET_OS_powerpc = 
linux${@['','-gnuspe'][bb.data.getVar('BASE_PACKAGE_ARCH',d
  # Add glibc overrides to the overrides for eglibc.
  OVERRIDES .= :libc-glibc

-PREFERRED_PROVIDER_virtual/libiconv ?= eglibc
+PREFERRED_PROVIDER_virtual/libiconv ?= eglibc-locale
  PREFERRED_PROVIDER_virtual/libiconv-nativesdk ?= eglibc-nativesdk
  PREFERRED_PROVIDER_virtual/libintl ?= eglibc
  PREFERRED_PROVIDER_virtual/libc ?= eglibc
diff --git a/meta/conf/distro/include/tclibc-glibc.inc 
b/meta/conf/distro/include/tclibc-glibc.inc
index

Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling from libc recipe.

2011-06-22 Thread Phil Blundell 
On Wed, 2011-06-22 at 07:44 -0700, Khem Raj wrote:
 On 06/22/2011 02:01 AM, Dongxiao Xu wrote:
  *libc's do_package will cost a lot of time due to the locale handing,
  which may delay the other recipe's do_package task and affect the build
  performance.
 
 you could also move the locale generation into a task of its own instead 
 of new recipe

It'd be hard to get the dependencies to work out right if you did that,
though.  I think a new recipe probably is the right answer.

p.

[  10 quoted lines snipped ... ]



___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling from libc recipe.

2011-06-22 Thread Mark Hatle 
On 6/22/11 9:47 AM, Phil Blundell wrote:
 On Wed, 2011-06-22 at 07:44 -0700, Khem Raj wrote:
 On 06/22/2011 02:01 AM, Dongxiao Xu wrote:
 *libc's do_package will cost a lot of time due to the locale handing,
 which may delay the other recipe's do_package task and affect the build
 performance.

 you could also move the locale generation into a task of its own instead 
 of new recipe
 
 It'd be hard to get the dependencies to work out right if you did that,
 though.  I think a new recipe probably is the right answer.

And in the case of libc, the locales need to be done -after- the
populate_sysroot, or we don't get the parallelism improvement.

--Mark

 p.
 
 [  10 quoted lines snipped ... ]
 
 
 
 ___
 Openembedded-core mailing list
 Openembedded-core@lists.openembedded.org
 http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core


___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling from libc recipe.

2011-06-22 Thread Khem Raj 

On 06/22/2011 08:17 AM, Mark Hatle wrote:

On 6/22/11 9:47 AM, Phil Blundell wrote:

On Wed, 2011-06-22 at 07:44 -0700, Khem Raj wrote:

On 06/22/2011 02:01 AM, Dongxiao Xu wrote:

*libc's do_package will cost a lot of time due to the locale handing,
which may delay the other recipe's do_package task and affect the build
performance.


you could also move the locale generation into a task of its own instead
of new recipe


It'd be hard to get the dependencies to work out right if you did that,
though.  I think a new recipe probably is the right answer.


And in the case of libc, the locales need to be done -after- the
populate_sysroot, or we don't get the parallelism improvement.


ofcourse thats why I was suggesting a new task.


--Mark


p.

[  10 quoted lines snipped ... ]



___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core



___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core



___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling from libc recipe.

2011-06-09 Thread Richard Purdie 
On Thu, 2011-06-09 at 12:14 +0100, Phil Blundell wrote:
 On Wed, 2011-06-08 at 16:35 +0100, Richard Purdie wrote:
  I'm not sure how this would reduce performance of builds of a few
  threads, it should just make better use of any available spare
  processing capacity throughout the build.
 
 If I'm reading the patch right, it does involve a certain amount of
 extra copying around of the locale-related files since they need to be
 stashed away in some place where libc-locale.bb can find them later.  I
 don't think these files are especially big, but there are quite a few of
 them (which is the whole reason that libc's do_package was slow in the
 first place). 
 
 I must admit that I'm slightly surprised that libc's packaging stage is
 becoming a bottleneck for anything other than the very smallest builds.
 If there's any substantial amount of other material being compiled then
 I would expect that there would be enough do_compile() work to keep the
 other threads busy until libc was done packaging.  That's not to say
 that I think this change is a bad idea, but I would be interested to
 know what the actual impact is for real-world workloads.
 
 And, just on a point of principle, any time we're making a chance for
 performance-related reasons I think we should always have measurements
 to back it up. 

Totally agreed and this decision is being made on real world data even
if its perhaps not clearly being presented here.

$ cat 
buildstats/core-image-sato-qemux86/201106030912/eglibc-2.13-r1+svnr13356/do_package
 | grep time
eglibc-2.13-r1+svnr13356: do_package: Elapsed time: 828.01 seconds

As you can see, eglibc do_package takes about 14 minutes which is about
14% of our build time. That is a long time to block pretty much all
packaging activity, particularly if you have access to something with
several cores. When it does complete, even on my 4 core system you see a
stampeding herd of packaging happening on the build charts suggesting
a backlog does build up.

For those reasons I do think its a reasonable change.

Cheers,

Richard









___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling from libc recipe.

2011-06-09 Thread Richard Purdie 
On Thu, 2011-06-09 at 12:43 +0100, Phil Blundell wrote:
 On Thu, 2011-06-09 at 12:29 +0100, Richard Purdie wrote:
  As you can see, eglibc do_package takes about 14 minutes which is about
  14% of our build time. That is a long time to block pretty much all
  packaging activity, particularly if you have access to something with
  several cores. When it does complete, even on my 4 core system you see a
  stampeding herd of packaging happening on the build charts suggesting
  a backlog does build up.
 
 Yeah, I can imagine that a backlog of packaging activity does build up.
 The thing I'm not entirely clear on is whether this is actually causing
 some threads to get starved of work (and hence the total build time to
 be longer than it needs to be) or whether we're really just shifting
 things around in the timeline without making much/any difference to the
 overall build duration.

It will certainly be a net win on large core systems which I know of
people running builds on as task starvation happens there. Last time I
tested this on a 4 core I think we saw a couple of minutes improvement
in build time so it appears beneficial there too.

   (I'm not familiar enough with bitbake's
 scheduler to know whether it will schedule tasks as early as possible,
 or as late as possible, or something else.)

Roughly speaking, the default scheduler runs tasks with the most things
depending upon them first. This means it will fill any spare threads
with lower priority tasks such as packaging.

 Just as a matter of interest, are you using qemu-based locale generation
 or the cross localedef for your measurement?  14 minutes does sound like
 an awfully long time and I wonder whether there is anything we could do
 in absolute terms to just speed that process up.

I'm using cross localedef. Its faster than with qemu but still slow. I'm
very open to suggestions on how to speed it up as it does take an age.

Cheers,

Richard


___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling from libc recipe.

2011-06-09 Thread Richard Purdie 
On Thu, 2011-06-09 at 14:15 +0100, Richard Purdie wrote:
 On Thu, 2011-06-09 at 12:43 +0100, Phil Blundell wrote:
  On Thu, 2011-06-09 at 12:29 +0100, Richard Purdie wrote:
   As you can see, eglibc do_package takes about 14 minutes which is about
   14% of our build time. That is a long time to block pretty much all
   packaging activity, particularly if you have access to something with
   several cores. When it does complete, even on my 4 core system you see a
   stampeding herd of packaging happening on the build charts suggesting
   a backlog does build up.
  
  Yeah, I can imagine that a backlog of packaging activity does build up.
  The thing I'm not entirely clear on is whether this is actually causing
  some threads to get starved of work (and hence the total build time to
  be longer than it needs to be) or whether we're really just shifting
  things around in the timeline without making much/any difference to the
  overall build duration.
 
 It will certainly be a net win on large core systems which I know of
 people running builds on as task starvation happens there. Last time I
 tested this on a 4 core I think we saw a couple of minutes improvement
 in build time so it appears beneficial there too.

Just as another data point, I'm running builds on my 4 core machine here
and it is scheduling many do_package tasks including long running ones
such as perl's in parallel with eglibc-locale's do_package task. It
would appear the bitbake scheduler believes there is benefit in
splitting these out at least.

Cheers,

Richard




___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling from libc recipe.

2011-06-09 Thread Koen Kooi 

Op 9 jun 2011, om 15:15 heeft Richard Purdie het volgende geschreven:

 On Thu, 2011-06-09 at 12:43 +0100, Phil Blundell wrote:
 On Thu, 2011-06-09 at 12:29 +0100, Richard Purdie wrote:
 As you can see, eglibc do_package takes about 14 minutes which is about
 14% of our build time. That is a long time to block pretty much all
 packaging activity, particularly if you have access to something with
 several cores. When it does complete, even on my 4 core system you see a
 stampeding herd of packaging happening on the build charts suggesting
 a backlog does build up.
 
 Yeah, I can imagine that a backlog of packaging activity does build up.
 The thing I'm not entirely clear on is whether this is actually causing
 some threads to get starved of work (and hence the total build time to
 be longer than it needs to be) or whether we're really just shifting
 things around in the timeline without making much/any difference to the
 overall build duration.
 
 It will certainly be a net win on large core systems which I know of
 people running builds on as task starvation happens there. Last time I
 tested this on a 4 core I think we saw a couple of minutes improvement
 in build time so it appears beneficial there too.
 
  (I'm not familiar enough with bitbake's
 scheduler to know whether it will schedule tasks as early as possible,
 or as late as possible, or something else.)
 
 Roughly speaking, the default scheduler runs tasks with the most things
 depending upon them first. This means it will fill any spare threads
 with lower priority tasks such as packaging.
 
 Just as a matter of interest, are you using qemu-based locale generation
 or the cross localedef for your measurement?  14 minutes does sound like
 an awfully long time and I wonder whether there is anything we could do
 in absolute terms to just speed that process up.
 
 I'm using cross localedef. Its faster than with qemu but still slow. I'm
 very open to suggestions on how to speed it up as it does take an age.

Does cross localegen obey parallel make or does it only do a single locale at a 
time?

regards,

Koen
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling from libc recipe.

2011-06-09 Thread Phil Blundell 
On Thu, 2011-06-09 at 15:53 +0200, Koen Kooi wrote:
 Does cross localegen obey parallel make or does it only do a single locale at 
 a time?

Just one at a time.  It's run out of Python code in libc-package, not by
make, so there is no completely straightforward way to get it to respect
PARALLEL_MAKE.  I guess you could teach libc-package how to generate a
makefile and then run it; that'd probably be the easiest way to
parallelize that particular process.

p.



___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling from libc recipe.

2011-06-08 Thread Phil Blundell 
On Wed, 2011-06-08 at 17:08 +0800, Dongxiao Xu wrote:
 *libc's do_package will cost a lot of time due to the locale handing,
 which may delay the other recipe's do_package task and affect the build
 performance.
 
 This commit moves locale handling into a separate recipe *libc-locale.

Can you quantify the effect on build performance a bit?  If I understand
correctly, you're basically saying that the goal is to increase
parallelism.  Does that cause reduced performance for people running
with few threads?

For example, it would be interesting to know what the before and after
times are for some representative image build using (say)
BB_NUMBER_THREADS=1 and BB_NUMBER_THREADS=16.

Also, see below.

 diff --git a/meta/conf/distro/include/tclibc-eglibc.inc 
 b/meta/conf/distro/include/tclibc-eglibc.inc
 index a4c648e..8997d9e 100644
 --- a/meta/conf/distro/include/tclibc-eglibc.inc
 +++ b/meta/conf/distro/include/tclibc-eglibc.inc
 @@ -10,7 +10,7 @@ TARGET_OS_powerpc = 
 linux${@['','-gnuspe'][bb.data.getVar('BASE_PACKAGE_ARCH',d
  # Add glibc overrides to the overrides for eglibc.
  OVERRIDES .= :libc-glibc
  
 -PREFERRED_PROVIDER_virtual/libiconv ?= eglibc
 +PREFERRED_PROVIDER_virtual/libiconv ?= eglibc-locale

That looks a bit strange.  Is this really correct?

 -  eglibc-localedata-i18n \
 -  eglibc-gconv-ibm850 \
 -  eglibc-gconv-cp1252 \
 -  eglibc-gconv-iso8859-1 \
 -  eglibc-gconv-iso8859-15 \
 +  eglibc-locale-localedata-i18n \
 +  eglibc-locale-gconv-ibm850 \
 +  eglibc-locale-gconv-cp1252 \
 +  eglibc-locale-gconv-iso8859-1 \
 +  eglibc-locale-gconv-iso8859-15 \

This kind of thing is definitely not very desirable.  Can you find a way
to keep the output package names the same?

p.


___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling from libc recipe.

2011-06-08 Thread Richard Purdie 
On Wed, 2011-06-08 at 10:36 +0100, Phil Blundell wrote:
 On Wed, 2011-06-08 at 17:08 +0800, Dongxiao Xu wrote:
  *libc's do_package will cost a lot of time due to the locale handing,
  which may delay the other recipe's do_package task and affect the build
  performance.
  
  This commit moves locale handling into a separate recipe *libc-locale.
 
 Can you quantify the effect on build performance a bit?  If I understand
 correctly, you're basically saying that the goal is to increase
 parallelism.  Does that cause reduced performance for people running
 with few threads?

One of the side effects of debian.bbclass is that it requires all
dependencies to do_package before any package itself can do_package so
any renaming of dependencies can be accounted for.

There is an issue if do_package for libc takes an age as it holds up any
other tasks from writing out packages. Pretty much most things depend on
libc.

This patch therefore splits it into two stages and means that packaging
of things depending on libc can happen sooner thereby increasing the
potential parallelism of the packaging stages of builds.

There is a very clear step on the bootchart graphs of builds I made
showing this.

I'm not sure how this would reduce performance of builds of a few
threads, it should just make better use of any available spare
processing capacity throughout the build.

Cheers,

Richard



___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core