Re: [meta-ti] [PATCH] meta-ti: Set the --jobs based on host cpu_count
Hi Khem, On 02/12/19 18:09, Khem Raj wrote: [...] --- a/classes/ti-pdk.bbclass +++ b/classes/ti-pdk.bbclass @@ -68,7 +68,7 @@ TI_PDK_DOXYGEN_SUPPORT ?= "1" TI_PDK_XDC_ARGS ?= "${TI_PDK_LIMIT_SOCS}" -PARALLEL_XDC = "--jobs=${BB_NUMBER_THREADS}" +PARALLEL_XDC = "--jobs=${@oe.utils.cpu_count()}" >>> >>> this may create another variableto control when limting parallelism so >>> expressing it in >>> terms of BB_NUMBER_THREAD s desired >>> >>> perhaps you can try adding >>> >>> do_configure[vardepsexclude] = "PARALLEL_XDC" >>> do_compile[vardepsexclude] = "PARALLEL_XDC" >> >> Thank you for your review. >> >> I agree the solution in this patch is not ideal as it removes ability to >> control the build load. >> >> However using BB_NUMBER_THREADS (with our without vardepsexclude) is >> also not quite correct. PARALLEL_XDC should have to do with >> PARALLEL_MAKE, not BB_NUMBER_THREADS. >> >> But PARALLEL_MAKE is not a number, so using PARALLEL_MAKE would be >> correct but it's not doable, and using BB_NUMBER_THREADS is doable but >> not quite correct... :( >> >> Bottom line: for lack of better ideas, I'm sending a new patch that >> keeps using BB_NUMBER_THREADS and takes your suggestion above. I guess >> it should be OK for most users. At least it's not worse than the current >> behavior and can still be improved later. >> > > another option would be to use parallel_make_argument to contruct what you > want > e.g. see how boost does it > > https://git.openembedded.org/openembedded-core/tree/meta/recipes-support/boost/boost.inc#n141 Oh, good idea, thanks! It looks like the correct solution to use the proper variable (PARALLEL_MAKE) in place of BB_NUMBER_THREADS. But surprisingly it also fixes the "basehash value changed" problem. How can this happen? My patch is simply: --- a/classes/ti-pdk.bbclass +++ b/classes/ti-pdk.bbclass -PARALLEL_XDC = "--jobs=${BB_NUMBER_THREADS}" +PARALLEL_XDC = "${@oe.utils.parallel_make_argument(d, '--jobs=%d')}" and --- a/recipes-ti/ipc/ti-ipc-rtos.inc +++ b/recipes-ti/ipc/ti-ipc-rtos.inc +PARALLEL_XDC = "${@oe.utils.parallel_make_argument(d, '--jobs=%d')}" EXTRA_OEMAKE = "\ PLATFORM=${PLATFORM} \ XDC_INSTALL_DIR="${XDC_INSTALL_DIR}" \ BIOS_INSTALL_DIR="${SYSBIOS_INSTALL_DIR}" \ ${IPC_TARGETS} \ -JOBS="--jobs=${BB_NUMBER_THREADS}" \ +${PARALLEL_XDC} \ JAVA_TOOL_OPTIONS=-Xss2560k \ " So I wonder why the signature is not considered as changing since EXTRA_OEMAKE *is* changing from one build to another with a different PARALLEL_MAKE. Any hints? Whatever the reason, this version definitely supersedes v2, and after some more testing I'll send v3 with the above implementation. -- Luca -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#12544): https://lists.yoctoproject.org/g/meta-ti/message/12544 Mute This Topic: https://lists.yoctoproject.org/mt/64369908/21656 Group Owner: meta-ti+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/meta-ti/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [meta-ti] [PATCH] meta-ti: Set the --jobs based on host cpu_count
On Mon, Dec 2, 2019 at 4:20 AM Luca Ceresoli wrote: > > Hi Khem, > > On 29/11/19 17:17, Khem Raj wrote: > > On Fri, Nov 29, 2019 at 6:48 AM Luca Ceresoli wrote: > >> > >> From: "Kemal R. Shakir" > >> > >> Using BB_NUMBER_THREADS to set variables causes an "basehash value changed" > >> error when BB_NUMBER_THREADS is changed between successive builds. > >> > >> Fixes: > >> > >> ERROR: When reparsing > >> .../meta-ti/recipes-ti/ipc/ti-ipc-rtos_git.bb.do_configure, the basehash > >> value changed from 63897a188012cedde03f756b5dbbfa70 to > >> 13b80bf4f4327c367bd201eb4aa32ad7. The metadata is not deterministic and > >> this needs to be fixed. > >> ERROR: When reparsing > >> .../meta-ti/recipes-ti/ipc/ti-ipc-rtos_git.bb.do_compile, the basehash > >> value changed from 2fb59a7db2a2f69d2c25c06c1b6a4245 to > >> 314c664cc040afae2800fcc378dd9ae0. The metadata is not deterministic and > >> this needs to be fixed. > >> > >> Steps to reproduce: > >> > >> BB_NUMBER_THREADS="3" bitbake ti-ipc-rtos > >> BB_NUMBER_THREADS="4" bitbake -c cleansstate osal-rtos > >> BB_NUMBER_THREADS="5" bitbake ti-ipc-rtos > >> > >> Reported here: https://e2e.ti.com/support/processors/f/791/p/825142/3059419 > >> Patch from Kemal R. Shakir > >> Reported-by: Matthew McKee (e-mail unknown) > >> [Luca: add detailed commit message] > >> Tested-by: Luca Ceresoli > >> Signed-off-by: Luca Ceresoli > >> > >> --- > >> > >> I'm having the same problem and found this patch solving it on e2e.ti.com, > >> but it does not appear on any git branch nor in the meta-ti mailing list. I > >> tested it on the thud branch and fixes the problem. Can it be applied to > >> both thud and future branches? > >> > >> Thanks > >> --- > >> classes/ti-pdk.bbclass | 2 +- > >> recipes-ti/ipc/ti-ipc-rtos.inc | 2 +- > >> 2 files changed, 2 insertions(+), 2 deletions(-) > >> > >> diff --git a/classes/ti-pdk.bbclass b/classes/ti-pdk.bbclass > >> index f0e6f2d2738a..5f78bafa94b5 100644 > >> --- a/classes/ti-pdk.bbclass > >> +++ b/classes/ti-pdk.bbclass > >> @@ -68,7 +68,7 @@ TI_PDK_DOXYGEN_SUPPORT ?= "1" > >> > >> TI_PDK_XDC_ARGS ?= "${TI_PDK_LIMIT_SOCS}" > >> > >> -PARALLEL_XDC = "--jobs=${BB_NUMBER_THREADS}" > >> +PARALLEL_XDC = "--jobs=${@oe.utils.cpu_count()}" > > > > this may create another variableto control when limting parallelism so > > expressing it in > > terms of BB_NUMBER_THREAD s desired > > > > perhaps you can try adding > > > > do_configure[vardepsexclude] = "PARALLEL_XDC" > > do_compile[vardepsexclude] = "PARALLEL_XDC" > > Thank you for your review. > > I agree the solution in this patch is not ideal as it removes ability to > control the build load. > > However using BB_NUMBER_THREADS (with our without vardepsexclude) is > also not quite correct. PARALLEL_XDC should have to do with > PARALLEL_MAKE, not BB_NUMBER_THREADS. > > But PARALLEL_MAKE is not a number, so using PARALLEL_MAKE would be > correct but it's not doable, and using BB_NUMBER_THREADS is doable but > not quite correct... :( > > Bottom line: for lack of better ideas, I'm sending a new patch that > keeps using BB_NUMBER_THREADS and takes your suggestion above. I guess > it should be OK for most users. At least it's not worse than the current > behavior and can still be improved later. > another option would be to use parallel_make_argument to contruct what you want e.g. see how boost does it https://git.openembedded.org/openembedded-core/tree/meta/recipes-support/boost/boost.inc#n141 > -- > Luca -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#12538): https://lists.yoctoproject.org/g/meta-ti/message/12538 Mute This Topic: https://lists.yoctoproject.org/mt/64369908/21656 Group Owner: meta-ti+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/meta-ti/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [meta-ti] [PATCH] meta-ti: Set the --jobs based on host cpu_count
Hi Khem, On 29/11/19 17:17, Khem Raj wrote: > On Fri, Nov 29, 2019 at 6:48 AM Luca Ceresoli wrote: >> >> From: "Kemal R. Shakir" >> >> Using BB_NUMBER_THREADS to set variables causes an "basehash value changed" >> error when BB_NUMBER_THREADS is changed between successive builds. >> >> Fixes: >> >> ERROR: When reparsing >> .../meta-ti/recipes-ti/ipc/ti-ipc-rtos_git.bb.do_configure, the basehash >> value changed from 63897a188012cedde03f756b5dbbfa70 to >> 13b80bf4f4327c367bd201eb4aa32ad7. The metadata is not deterministic and this >> needs to be fixed. >> ERROR: When reparsing >> .../meta-ti/recipes-ti/ipc/ti-ipc-rtos_git.bb.do_compile, the basehash value >> changed from 2fb59a7db2a2f69d2c25c06c1b6a4245 to >> 314c664cc040afae2800fcc378dd9ae0. The metadata is not deterministic and this >> needs to be fixed. >> >> Steps to reproduce: >> >> BB_NUMBER_THREADS="3" bitbake ti-ipc-rtos >> BB_NUMBER_THREADS="4" bitbake -c cleansstate osal-rtos >> BB_NUMBER_THREADS="5" bitbake ti-ipc-rtos >> >> Reported here: https://e2e.ti.com/support/processors/f/791/p/825142/3059419 >> Patch from Kemal R. Shakir >> Reported-by: Matthew McKee (e-mail unknown) >> [Luca: add detailed commit message] >> Tested-by: Luca Ceresoli >> Signed-off-by: Luca Ceresoli >> >> --- >> >> I'm having the same problem and found this patch solving it on e2e.ti.com, >> but it does not appear on any git branch nor in the meta-ti mailing list. I >> tested it on the thud branch and fixes the problem. Can it be applied to >> both thud and future branches? >> >> Thanks >> --- >> classes/ti-pdk.bbclass | 2 +- >> recipes-ti/ipc/ti-ipc-rtos.inc | 2 +- >> 2 files changed, 2 insertions(+), 2 deletions(-) >> >> diff --git a/classes/ti-pdk.bbclass b/classes/ti-pdk.bbclass >> index f0e6f2d2738a..5f78bafa94b5 100644 >> --- a/classes/ti-pdk.bbclass >> +++ b/classes/ti-pdk.bbclass >> @@ -68,7 +68,7 @@ TI_PDK_DOXYGEN_SUPPORT ?= "1" >> >> TI_PDK_XDC_ARGS ?= "${TI_PDK_LIMIT_SOCS}" >> >> -PARALLEL_XDC = "--jobs=${BB_NUMBER_THREADS}" >> +PARALLEL_XDC = "--jobs=${@oe.utils.cpu_count()}" > > this may create another variableto control when limting parallelism so > expressing it in > terms of BB_NUMBER_THREAD s desired > > perhaps you can try adding > > do_configure[vardepsexclude] = "PARALLEL_XDC" > do_compile[vardepsexclude] = "PARALLEL_XDC" Thank you for your review. I agree the solution in this patch is not ideal as it removes ability to control the build load. However using BB_NUMBER_THREADS (with our without vardepsexclude) is also not quite correct. PARALLEL_XDC should have to do with PARALLEL_MAKE, not BB_NUMBER_THREADS. But PARALLEL_MAKE is not a number, so using PARALLEL_MAKE would be correct but it's not doable, and using BB_NUMBER_THREADS is doable but not quite correct... :( Bottom line: for lack of better ideas, I'm sending a new patch that keeps using BB_NUMBER_THREADS and takes your suggestion above. I guess it should be OK for most users. At least it's not worse than the current behavior and can still be improved later. -- Luca -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#12536): https://lists.yoctoproject.org/g/meta-ti/message/12536 Mute This Topic: https://lists.yoctoproject.org/mt/64369908/21656 Group Owner: meta-ti+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/meta-ti/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [meta-ti] [PATCH] meta-ti: Set the --jobs based on host cpu_count
On Fri, Nov 29, 2019 at 6:48 AM Luca Ceresoli wrote: > > From: "Kemal R. Shakir" > > Using BB_NUMBER_THREADS to set variables causes an "basehash value changed" > error when BB_NUMBER_THREADS is changed between successive builds. > > Fixes: > > ERROR: When reparsing > .../meta-ti/recipes-ti/ipc/ti-ipc-rtos_git.bb.do_configure, the basehash > value changed from 63897a188012cedde03f756b5dbbfa70 to > 13b80bf4f4327c367bd201eb4aa32ad7. The metadata is not deterministic and this > needs to be fixed. > ERROR: When reparsing > .../meta-ti/recipes-ti/ipc/ti-ipc-rtos_git.bb.do_compile, the basehash value > changed from 2fb59a7db2a2f69d2c25c06c1b6a4245 to > 314c664cc040afae2800fcc378dd9ae0. The metadata is not deterministic and this > needs to be fixed. > > Steps to reproduce: > > BB_NUMBER_THREADS="3" bitbake ti-ipc-rtos > BB_NUMBER_THREADS="4" bitbake -c cleansstate osal-rtos > BB_NUMBER_THREADS="5" bitbake ti-ipc-rtos > > Reported here: https://e2e.ti.com/support/processors/f/791/p/825142/3059419 > Patch from Kemal R. Shakir > Reported-by: Matthew McKee (e-mail unknown) > [Luca: add detailed commit message] > Tested-by: Luca Ceresoli > Signed-off-by: Luca Ceresoli > > --- > > I'm having the same problem and found this patch solving it on e2e.ti.com, > but it does not appear on any git branch nor in the meta-ti mailing list. I > tested it on the thud branch and fixes the problem. Can it be applied to > both thud and future branches? > > Thanks > --- > classes/ti-pdk.bbclass | 2 +- > recipes-ti/ipc/ti-ipc-rtos.inc | 2 +- > 2 files changed, 2 insertions(+), 2 deletions(-) > > diff --git a/classes/ti-pdk.bbclass b/classes/ti-pdk.bbclass > index f0e6f2d2738a..5f78bafa94b5 100644 > --- a/classes/ti-pdk.bbclass > +++ b/classes/ti-pdk.bbclass > @@ -68,7 +68,7 @@ TI_PDK_DOXYGEN_SUPPORT ?= "1" > > TI_PDK_XDC_ARGS ?= "${TI_PDK_LIMIT_SOCS}" > > -PARALLEL_XDC = "--jobs=${BB_NUMBER_THREADS}" > +PARALLEL_XDC = "--jobs=${@oe.utils.cpu_count()}" this may create another variableto control when limting parallelism so expressing it in terms of BB_NUMBER_THREAD s desired perhaps you can try adding do_configure[vardepsexclude] = "PARALLEL_XDC" do_compile[vardepsexclude] = "PARALLEL_XDC" > PARALLEL_MAKE = "" > > def get_doxygen_support(d): > diff --git a/recipes-ti/ipc/ti-ipc-rtos.inc b/recipes-ti/ipc/ti-ipc-rtos.inc > index dcd6309fe2ab..1b21375edd33 100644 > --- a/recipes-ti/ipc/ti-ipc-rtos.inc > +++ b/recipes-ti/ipc/ti-ipc-rtos.inc > @@ -89,7 +89,7 @@ EXTRA_OEMAKE = "\ > XDC_INSTALL_DIR="${XDC_INSTALL_DIR}" \ > BIOS_INSTALL_DIR="${SYSBIOS_INSTALL_DIR}" \ > ${IPC_TARGETS} \ > -JOBS="--jobs=${BB_NUMBER_THREADS}" \ > +JOBS="--jobs=${@oe.utils.cpu_count()}" \ something similar here. > JAVA_TOOL_OPTIONS=-Xss2560k \ > " > EXTRA_OEMAKE_append_am65xx = " PDK_INSTALL_DIR=${PDK_INSTALL_DIR}" > -- > 2.24.0 > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > > View/Reply Online (#12534): > https://lists.yoctoproject.org/g/meta-ti/message/12534 > Mute This Topic: https://lists.yoctoproject.org/mt/64369908/1997914 > Group Owner: meta-ti+ow...@lists.yoctoproject.org > Unsubscribe: https://lists.yoctoproject.org/g/meta-ti/unsub > [raj.k...@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#12535): https://lists.yoctoproject.org/g/meta-ti/message/12535 Mute This Topic: https://lists.yoctoproject.org/mt/64369908/21656 Group Owner: meta-ti+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/meta-ti/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[meta-ti] [PATCH] meta-ti: Set the --jobs based on host cpu_count
From: "Kemal R. Shakir" Using BB_NUMBER_THREADS to set variables causes an "basehash value changed" error when BB_NUMBER_THREADS is changed between successive builds. Fixes: ERROR: When reparsing .../meta-ti/recipes-ti/ipc/ti-ipc-rtos_git.bb.do_configure, the basehash value changed from 63897a188012cedde03f756b5dbbfa70 to 13b80bf4f4327c367bd201eb4aa32ad7. The metadata is not deterministic and this needs to be fixed. ERROR: When reparsing .../meta-ti/recipes-ti/ipc/ti-ipc-rtos_git.bb.do_compile, the basehash value changed from 2fb59a7db2a2f69d2c25c06c1b6a4245 to 314c664cc040afae2800fcc378dd9ae0. The metadata is not deterministic and this needs to be fixed. Steps to reproduce: BB_NUMBER_THREADS="3" bitbake ti-ipc-rtos BB_NUMBER_THREADS="4" bitbake -c cleansstate osal-rtos BB_NUMBER_THREADS="5" bitbake ti-ipc-rtos Reported here: https://e2e.ti.com/support/processors/f/791/p/825142/3059419 Patch from Kemal R. Shakir Reported-by: Matthew McKee (e-mail unknown) [Luca: add detailed commit message] Tested-by: Luca Ceresoli Signed-off-by: Luca Ceresoli --- I'm having the same problem and found this patch solving it on e2e.ti.com, but it does not appear on any git branch nor in the meta-ti mailing list. I tested it on the thud branch and fixes the problem. Can it be applied to both thud and future branches? Thanks --- classes/ti-pdk.bbclass | 2 +- recipes-ti/ipc/ti-ipc-rtos.inc | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/classes/ti-pdk.bbclass b/classes/ti-pdk.bbclass index f0e6f2d2738a..5f78bafa94b5 100644 --- a/classes/ti-pdk.bbclass +++ b/classes/ti-pdk.bbclass @@ -68,7 +68,7 @@ TI_PDK_DOXYGEN_SUPPORT ?= "1" TI_PDK_XDC_ARGS ?= "${TI_PDK_LIMIT_SOCS}" -PARALLEL_XDC = "--jobs=${BB_NUMBER_THREADS}" +PARALLEL_XDC = "--jobs=${@oe.utils.cpu_count()}" PARALLEL_MAKE = "" def get_doxygen_support(d): diff --git a/recipes-ti/ipc/ti-ipc-rtos.inc b/recipes-ti/ipc/ti-ipc-rtos.inc index dcd6309fe2ab..1b21375edd33 100644 --- a/recipes-ti/ipc/ti-ipc-rtos.inc +++ b/recipes-ti/ipc/ti-ipc-rtos.inc @@ -89,7 +89,7 @@ EXTRA_OEMAKE = "\ XDC_INSTALL_DIR="${XDC_INSTALL_DIR}" \ BIOS_INSTALL_DIR="${SYSBIOS_INSTALL_DIR}" \ ${IPC_TARGETS} \ -JOBS="--jobs=${BB_NUMBER_THREADS}" \ +JOBS="--jobs=${@oe.utils.cpu_count()}" \ JAVA_TOOL_OPTIONS=-Xss2560k \ " EXTRA_OEMAKE_append_am65xx = " PDK_INSTALL_DIR=${PDK_INSTALL_DIR}" -- 2.24.0 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#12534): https://lists.yoctoproject.org/g/meta-ti/message/12534 Mute This Topic: https://lists.yoctoproject.org/mt/64369908/21656 Group Owner: meta-ti+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/meta-ti/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-