Re: [OE-core] [PATCH] kernel.bbclass: Fix do_shared_workdir task ordering
On Tue, Aug 11, 2015 at 9:06 AM, Bruce Ashfield bruce.ashfi...@gmail.com wrote: On Tue, Aug 11, 2015 at 8:47 AM, Otavio Salvador otavio.salva...@ossystems.com.br wrote: On Tue, Aug 11, 2015 at 12:23 AM, Bruce Ashfield bruce.ashfi...@gmail.com wrote: On Mon, Aug 10, 2015 at 11:21 AM, Stefan Müller-Klieser s.mueller-klie...@phytec.de wrote: commit 02d0a003d60326 [kernel.bbclass: Fix race condition] has surfaced a bug in the generation of the shared_workdir. The task do_compile_kernelmodules adds the exported symbols of the kernel modules to the Module.symvers. By creating the shared_workdir before the modules are compiled, the symbols of the modules are missing in the shared_workdir. Subsequent external module builds will not include the ABI CRC of functions exported in modules. Modprobe will fail to load the external module if CONFIG_MODVERSIONS is enabled.\ Have you seen our bug: https://bugzilla.yoctoproject.org/show_bug.cgi?id=8127 ? It's new .. so probably not. The significant issue with this, is that we are now forcing anyone that needs the shared workdir artifacts to build kernel modules. That's performance issue for many workflows. I had some changes where I was working to short cut parts of the process, but they turned out to miss a few corner cases. We need to do more thinking on this one, before we can bring in a change like this .. since avoiding that overhead is something valuable. I agree that performance is important but correctness seems more valuable for me. I think the optimization can come as a subsequent patch ... Let's disagree on this point. There's time to get this right. We have a bug to track it, so we wont' release with the active bug, and this only hits a very tiny set of users. So we are going to step back and try and fix this right. I hit send too soon. I have a suggestion in the bug already, so it isn't like we are talking about letting this sit for weeks. History shows that we are very unlikely to loop back and fix the performance of perf or other builds once the change goes in. So in the absence of other concrete suggestions, looking into some other small changes is a good use of time. Cheers, Bruce Bruce -- Otavio Salvador O.S. Systems http://www.ossystems.com.brhttp://code.ossystems.com.br Mobile: +55 (53) 9981-7854Mobile: +1 (347) 903-9750 -- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end -- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH] kernel.bbclass: Fix do_shared_workdir task ordering
On Tue, Aug 11, 2015 at 8:47 AM, Otavio Salvador otavio.salva...@ossystems.com.br wrote: On Tue, Aug 11, 2015 at 12:23 AM, Bruce Ashfield bruce.ashfi...@gmail.com wrote: On Mon, Aug 10, 2015 at 11:21 AM, Stefan Müller-Klieser s.mueller-klie...@phytec.de wrote: commit 02d0a003d60326 [kernel.bbclass: Fix race condition] has surfaced a bug in the generation of the shared_workdir. The task do_compile_kernelmodules adds the exported symbols of the kernel modules to the Module.symvers. By creating the shared_workdir before the modules are compiled, the symbols of the modules are missing in the shared_workdir. Subsequent external module builds will not include the ABI CRC of functions exported in modules. Modprobe will fail to load the external module if CONFIG_MODVERSIONS is enabled.\ Have you seen our bug: https://bugzilla.yoctoproject.org/show_bug.cgi?id=8127 ? It's new .. so probably not. The significant issue with this, is that we are now forcing anyone that needs the shared workdir artifacts to build kernel modules. That's performance issue for many workflows. I had some changes where I was working to short cut parts of the process, but they turned out to miss a few corner cases. We need to do more thinking on this one, before we can bring in a change like this .. since avoiding that overhead is something valuable. I agree that performance is important but correctness seems more valuable for me. I think the optimization can come as a subsequent patch ... Let's disagree on this point. There's time to get this right. We have a bug to track it, so we wont' release with the active bug, and this only hits a very tiny set of users. So we are going to step back and try and fix this right. Bruce -- Otavio Salvador O.S. Systems http://www.ossystems.com.brhttp://code.ossystems.com.br Mobile: +55 (53) 9981-7854Mobile: +1 (347) 903-9750 -- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH] kernel.bbclass: Fix do_shared_workdir task ordering
On Tue, Aug 11, 2015 at 12:23 AM, Bruce Ashfield bruce.ashfi...@gmail.com wrote: On Mon, Aug 10, 2015 at 11:21 AM, Stefan Müller-Klieser s.mueller-klie...@phytec.de wrote: commit 02d0a003d60326 [kernel.bbclass: Fix race condition] has surfaced a bug in the generation of the shared_workdir. The task do_compile_kernelmodules adds the exported symbols of the kernel modules to the Module.symvers. By creating the shared_workdir before the modules are compiled, the symbols of the modules are missing in the shared_workdir. Subsequent external module builds will not include the ABI CRC of functions exported in modules. Modprobe will fail to load the external module if CONFIG_MODVERSIONS is enabled.\ Have you seen our bug: https://bugzilla.yoctoproject.org/show_bug.cgi?id=8127 ? It's new .. so probably not. The significant issue with this, is that we are now forcing anyone that needs the shared workdir artifacts to build kernel modules. That's performance issue for many workflows. I had some changes where I was working to short cut parts of the process, but they turned out to miss a few corner cases. We need to do more thinking on this one, before we can bring in a change like this .. since avoiding that overhead is something valuable. I agree that performance is important but correctness seems more valuable for me. I think the optimization can come as a subsequent patch ... -- Otavio Salvador O.S. Systems http://www.ossystems.com.brhttp://code.ossystems.com.br Mobile: +55 (53) 9981-7854Mobile: +1 (347) 903-9750 -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH] kernel.bbclass: Fix do_shared_workdir task ordering
On Tue, Aug 11, 2015 at 10:25 AM, Stefan Müller-Klieser s.mueller-klie...@phytec.de wrote: On 11.08.2015 15:11, Bruce Ashfield wrote: On Tue, Aug 11, 2015 at 9:06 AM, Bruce Ashfield bruce.ashfi...@gmail.com wrote: On Tue, Aug 11, 2015 at 8:47 AM, Otavio Salvador otavio.salva...@ossystems.com.br wrote: On Tue, Aug 11, 2015 at 12:23 AM, Bruce Ashfield bruce.ashfi...@gmail.com wrote: On Mon, Aug 10, 2015 at 11:21 AM, Stefan Müller-Klieser s.mueller-klie...@phytec.de wrote: commit 02d0a003d60326 [kernel.bbclass: Fix race condition] has surfaced a bug in the generation of the shared_workdir. The task do_compile_kernelmodules adds the exported symbols of the kernel modules to the Module.symvers. By creating the shared_workdir before the modules are compiled, the symbols of the modules are missing in the shared_workdir. Subsequent external module builds will not include the ABI CRC of functions exported in modules. Modprobe will fail to load the external module if CONFIG_MODVERSIONS is enabled.\ Have you seen our bug: https://bugzilla.yoctoproject.org/show_bug.cgi?id=8127 ? It's new .. so probably not. I did not. Thanks. The significant issue with this, is that we are now forcing anyone that needs the shared workdir artifacts to build kernel modules. That's by design. The artifacts are modified by the module build. As was the generating of shared workdir being before the module build .. but yes, we are aware of how and why the kernel build generates those artifacts. That's performance issue for many workflows. I had some changes where I was working to short cut parts of the process, but they turned out to miss a few corner cases. We need to do more thinking on this one, before we can bring in a change like this .. since avoiding that overhead is something valuable. So you are saying a fast build is more important than a correct build? That's quite a bold statement. That's not what I said. I agree that performance is important but correctness seems more valuable for me. I think the optimization can come as a subsequent patch ... Let's disagree on this point. There's time to get this right. We have a bug to track it, so we wont' release with the active bug, and this only hits a very tiny set of users. So we are going to step back and try and fix this right. Well, if you really want to do this then there should at least be a module-interdepend.bbclass not using the shared workdir and depending on the modules build. Fido and master are broken at the moment. There's multiple ways to consider here and not a single right way. But if you have an idea, send patch for review, or update the bug I referenced, that way we can consider them as well. I'm not saying this isn't a bug or issue, I'm saying that it hits a certain set of builds (and not all), so fixing this in a way that doesn't break the other design goals of the kernel build is important as well. I end up getting the bugs, and having to fix performance issues ... so I'm sensitive to all the use cases. We've been back and forth on the shared artefacts generation, with a lot of unintended side effects. Any changes in this area have to soak and be looked at by as many eyes as possible. RP may grab this as is, and that's also fine with me, I'm just on the record pointing out the side effects, so when someone notices incremental builds, or sstate, or updates taking longer .. we can point to the reason why. Bruce Regards, Stefan I hit send too soon. I have a suggestion in the bug already, so it isn't like we are talking about letting this sit for weeks. History shows that we are very unlikely to loop back and fix the performance of perf or other builds once the change goes in. So in the absence of other concrete suggestions, looking into some other small changes is a good use of time. Cheers, Bruce Bruce -- Otavio Salvador O.S. Systems http://www.ossystems.com.brhttp://code.ossystems.com.br Mobile: +55 (53) 9981-7854Mobile: +1 (347) 903-9750 -- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end -- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH] kernel.bbclass: Fix do_shared_workdir task ordering
On 11.08.2015 15:11, Bruce Ashfield wrote: On Tue, Aug 11, 2015 at 9:06 AM, Bruce Ashfield bruce.ashfi...@gmail.com wrote: On Tue, Aug 11, 2015 at 8:47 AM, Otavio Salvador otavio.salva...@ossystems.com.br wrote: On Tue, Aug 11, 2015 at 12:23 AM, Bruce Ashfield bruce.ashfi...@gmail.com wrote: On Mon, Aug 10, 2015 at 11:21 AM, Stefan Müller-Klieser s.mueller-klie...@phytec.de wrote: commit 02d0a003d60326 [kernel.bbclass: Fix race condition] has surfaced a bug in the generation of the shared_workdir. The task do_compile_kernelmodules adds the exported symbols of the kernel modules to the Module.symvers. By creating the shared_workdir before the modules are compiled, the symbols of the modules are missing in the shared_workdir. Subsequent external module builds will not include the ABI CRC of functions exported in modules. Modprobe will fail to load the external module if CONFIG_MODVERSIONS is enabled.\ Have you seen our bug: https://bugzilla.yoctoproject.org/show_bug.cgi?id=8127 ? It's new .. so probably not. I did not. Thanks. The significant issue with this, is that we are now forcing anyone that needs the shared workdir artifacts to build kernel modules. That's by design. The artifacts are modified by the module build. That's performance issue for many workflows. I had some changes where I was working to short cut parts of the process, but they turned out to miss a few corner cases. We need to do more thinking on this one, before we can bring in a change like this .. since avoiding that overhead is something valuable. So you are saying a fast build is more important than a correct build? That's quite a bold statement. I agree that performance is important but correctness seems more valuable for me. I think the optimization can come as a subsequent patch ... Let's disagree on this point. There's time to get this right. We have a bug to track it, so we wont' release with the active bug, and this only hits a very tiny set of users. So we are going to step back and try and fix this right. Well, if you really want to do this then there should at least be a module-interdepend.bbclass not using the shared workdir and depending on the modules build. Fido and master are broken at the moment. Regards, Stefan I hit send too soon. I have a suggestion in the bug already, so it isn't like we are talking about letting this sit for weeks. History shows that we are very unlikely to loop back and fix the performance of perf or other builds once the change goes in. So in the absence of other concrete suggestions, looking into some other small changes is a good use of time. Cheers, Bruce Bruce -- Otavio Salvador O.S. Systems http://www.ossystems.com.brhttp://code.ossystems.com.br Mobile: +55 (53) 9981-7854Mobile: +1 (347) 903-9750 -- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH] kernel.bbclass: Fix do_shared_workdir task ordering
commit 02d0a003d60326 [kernel.bbclass: Fix race condition] has surfaced
a bug in the generation of the shared_workdir. The task
do_compile_kernelmodules adds the exported symbols of the kernel modules
to the Module.symvers. By creating the shared_workdir before the modules
are compiled, the symbols of the modules are missing in the
shared_workdir. Subsequent external module builds will not include the
ABI CRC of functions exported in modules. Modprobe will fail to load the
external module if CONFIG_MODVERSIONS is enabled.
Signed-off-by: Stefan Müller-Klieser s.mueller-klie...@phytec.de
---
meta/classes/kernel.bbclass | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass
index d06f6cf..473f1f8 100644
--- a/meta/classes/kernel.bbclass
+++ b/meta/classes/kernel.bbclass
@@ -249,7 +249,7 @@ kernel_do_install() {
}
do_install[prefuncs] += package_get_auto_pr
-addtask shared_workdir after do_compile before do_compile_kernelmodules
+addtask shared_workdir after do_compile_kernelmodules before do_install
addtask shared_workdir_setscene
do_shared_workdir_setscene () {
--
1.9.1
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH] kernel.bbclass: Fix do_shared_workdir task ordering
On Mon, Aug 10, 2015 at 11:21 AM, Stefan Müller-Klieser
s.mueller-klie...@phytec.de wrote:
commit 02d0a003d60326 [kernel.bbclass: Fix race condition] has surfaced
a bug in the generation of the shared_workdir. The task
do_compile_kernelmodules adds the exported symbols of the kernel modules
to the Module.symvers. By creating the shared_workdir before the modules
are compiled, the symbols of the modules are missing in the
shared_workdir. Subsequent external module builds will not include the
ABI CRC of functions exported in modules. Modprobe will fail to load the
external module if CONFIG_MODVERSIONS is enabled.\
Have you seen our bug: https://bugzilla.yoctoproject.org/show_bug.cgi?id=8127 ?
It's new .. so probably not.
The significant issue with this, is that we are now forcing anyone
that needs the
shared workdir artifacts to build kernel modules.
That's performance issue for many workflows.
I had some changes where I was working to short cut parts of the process, but
they turned out to miss a few corner cases.
We need to do more thinking on this one, before we can bring in a change like
this .. since avoiding that overhead is something valuable.
Cheers,
Bruce
Signed-off-by: Stefan Müller-Klieser s.mueller-klie...@phytec.de
---
meta/classes/kernel.bbclass | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass
index d06f6cf..473f1f8 100644
--- a/meta/classes/kernel.bbclass
+++ b/meta/classes/kernel.bbclass
@@ -249,7 +249,7 @@ kernel_do_install() {
}
do_install[prefuncs] += package_get_auto_pr
-addtask shared_workdir after do_compile before do_compile_kernelmodules
+addtask shared_workdir after do_compile_kernelmodules before do_install
addtask shared_workdir_setscene
do_shared_workdir_setscene () {
--
1.9.1
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
--
Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
