Re: [OE-core] [PATCH v3 4/6] bmap-tools: generate standalone script
On Thu, May 19, 2016 at 2:50 PM, Alexander Kanevskiywrote: > On Thu, May 19, 2016 at 5:54 PM, Christopher Larson > wrote: > >> On Thu, May 19, 2016 at 7:19 AM, Alexander Kanevskiy >> wrote: >> >>> On Wed, May 4, 2016 at 9:55 PM, Christopher Larson >>> wrote: >>> > Yeah, putting it there does require some discussion in general. Personally, I'm in favor of such a thing, because we don't want to put generated convenience tools in the sysroot, since we tell people explicitly to not use sysroot contents directly. Regarding getting it built for the image, I'd suggest adding something like IMAGE_TASKDEPENDS to image_type.bbclass, which is basically like IMAGE_DEPENDS, but with the task included. I.e. IMAGE_TASKDEPENDS_bmap = "bmap-tools-native:do_deploy". Then you wouldn't need the 'before do_package', and it'd fix the from-sstate usage. >>> >>> We had some private conversation with Richard about deploy/tools, I'll >>> try to summarize results of our discussion here: >>> >>> Thing this, that content under deploy/tools/* supposed to be exported to >>> "release" directory, and should not be used by any other tool during normal >>> build/development process. >>> Those are for usage on another host, where bitbake and content of >>> sysroots are not usually available. >>> bmaptool is a good example as stand-alone binary that require only >>> python2 and nothing else to run. >>> Another example is dfu-util-static, which is also, not dependant even on >>> host's libusb even it runs on other host. >>> >>> So, in general, it's for static/standalone binaries that are supposed to >>> be published as part of "release" of the build, same like deploy/images/* >>> >>> For any classes, any internals of build process, they never should >>> depend on content of deploy/* and use *-native tools and binaries out of >>> sysroot. >>> E.g. wic require bmap-tool, it should be dependant on presence of >>> bmap-tools-native installed into sysroot, and should be using both modules >>> and binary out of this sysroot. >>> >> >> I don't really understand the point you're trying to make here. As far as >> I can tell, wic put bmaptool into deploy/tools/ as a convenience for the >> user, not for wic itself to use, so it already does what you're saying it >> should do. >> > > Yeah, to some extend I'm mixing two topics. Most of above is about putting > tools into depoloy/* (as Richard raised concern about that). > > > Second topic is about subject "which part of image or wic should depend on > which task". Both me and Ed are trying to understand exact scenario (or > even better test case) would trigger issue which lead to idea above of > putting dependencies like IMAGE_TASKDEPENDS_bmap = > "bmap-tools-native:do_deploy". > > My initial idea, while creating support for bmap-tools in Ostro, was that > image type would depend on bmap-tools-native. > This recipe is also responsible to handle deploy/*. Code or dependencies > of specific image type don't know anything about deploy part, and IMHO > should not know. > In case of support in OE-core, I had impression that Ed implemented it > same way. However, I didn't check myself part that he did in wic, so not > sure about part of your comment "wic put bmaptool into deploy/tools/". > > I also tried in my environment to reproduce scenario as you described in > some previous mails in this thread: > > > You're right that this is very handy, but we hit a problem when building > from shared state. This uses do_deploy to deploy bmaptool into > DEPLOY_DIR_IMAGE, but IMAGE_DEPENDS only depends > on do_populate_sysroot, > not do_deploy, so an image build from sstate won't write this script out. > > but don't see issue with it: executing image build where all objects are > available in sstate, but bmap-tools-native not present in sysroot leads to > execution of following tasks: > > $ cat tmp-glibc/work/x86_64-linux/bmap-tools-native/3.2/temp/log.task_order > do_populate_sysroot_setscene (29437): > log.do_populate_sysroot_setscene.29437 > do_deploy_setscene (29438): log.do_deploy_setscene.29438 > do_populate_lic_setscene (29439): log.do_populate_lic_setscene.29439 > $ > > if it for some reason not like that in code that Ed did for OE-core, we > need to look at test scenario, investigate and fix the issue. > For what it's worth, I'm no longer able to reproduce this. I did hit it at one point, but it seems fine now. *shrug* -- Christopher Larson clarson at kergoth dot com Founder - BitBake, OpenEmbedded, OpenZaurus Maintainer - Tslib Senior Software Engineer, Mentor Graphics -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH v3 4/6] bmap-tools: generate standalone script
On Thu, May 19, 2016 at 5:54 PM, Christopher Larsonwrote: > On Thu, May 19, 2016 at 7:19 AM, Alexander Kanevskiy wrote: > >> On Wed, May 4, 2016 at 9:55 PM, Christopher Larson >> wrote: >> >>> Yeah, putting it there does require some discussion in general. >>> Personally, I'm in favor of such a thing, because we don't want to put >>> generated convenience tools in the sysroot, since we tell people explicitly >>> to not use sysroot contents directly. >>> >>> Regarding getting it built for the image, I'd suggest adding something >>> like IMAGE_TASKDEPENDS to image_type.bbclass, which is basically like >>> IMAGE_DEPENDS, but with the task included. I.e. IMAGE_TASKDEPENDS_bmap = >>> "bmap-tools-native:do_deploy". Then you wouldn't need the 'before >>> do_package', and it'd fix the from-sstate usage. >>> >> >> We had some private conversation with Richard about deploy/tools, I'll >> try to summarize results of our discussion here: >> >> Thing this, that content under deploy/tools/* supposed to be exported to >> "release" directory, and should not be used by any other tool during normal >> build/development process. >> Those are for usage on another host, where bitbake and content of >> sysroots are not usually available. >> bmaptool is a good example as stand-alone binary that require only >> python2 and nothing else to run. >> Another example is dfu-util-static, which is also, not dependant even on >> host's libusb even it runs on other host. >> >> So, in general, it's for static/standalone binaries that are supposed to >> be published as part of "release" of the build, same like deploy/images/* >> >> For any classes, any internals of build process, they never should depend >> on content of deploy/* and use *-native tools and binaries out of sysroot. >> E.g. wic require bmap-tool, it should be dependant on presence of >> bmap-tools-native installed into sysroot, and should be using both modules >> and binary out of this sysroot. >> > > I don't really understand the point you're trying to make here. As far as > I can tell, wic put bmaptool into deploy/tools/ as a convenience for the > user, not for wic itself to use, so it already does what you're saying it > should do. > Yeah, to some extend I'm mixing two topics. Most of above is about putting tools into depoloy/* (as Richard raised concern about that). Second topic is about subject "which part of image or wic should depend on which task". Both me and Ed are trying to understand exact scenario (or even better test case) would trigger issue which lead to idea above of putting dependencies like IMAGE_TASKDEPENDS_bmap = "bmap-tools-native:do_deploy". My initial idea, while creating support for bmap-tools in Ostro, was that image type would depend on bmap-tools-native. This recipe is also responsible to handle deploy/*. Code or dependencies of specific image type don't know anything about deploy part, and IMHO should not know. In case of support in OE-core, I had impression that Ed implemented it same way. However, I didn't check myself part that he did in wic, so not sure about part of your comment "wic put bmaptool into deploy/tools/". I also tried in my environment to reproduce scenario as you described in some previous mails in this thread: > You're right that this is very handy, but we hit a problem when building from shared state. This uses do_deploy to deploy bmaptool into DEPLOY_DIR_IMAGE, but IMAGE_DEPENDS only depends > on do_populate_sysroot, not do_deploy, so an image build from sstate won't write this script out. but don't see issue with it: executing image build where all objects are available in sstate, but bmap-tools-native not present in sysroot leads to execution of following tasks: $ cat tmp-glibc/work/x86_64-linux/bmap-tools-native/3.2/temp/log.task_order do_populate_sysroot_setscene (29437): log.do_populate_sysroot_setscene.29437 do_deploy_setscene (29438): log.do_deploy_setscene.29438 do_populate_lic_setscene (29439): log.do_populate_lic_setscene.29439 $ if it for some reason not like that in code that Ed did for OE-core, we need to look at test scenario, investigate and fix the issue. -- br, Alexander Kanevskiy -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH v3 4/6] bmap-tools: generate standalone script
On Thu, May 19, 2016 at 7:19 AM, Alexander Kanevskiywrote: > On Wed, May 4, 2016 at 9:55 PM, Christopher Larson > wrote: > >> >> > >>> > Also, the convention is to add do_deploy before do_build, not before >>> > do_package, otherwise do_deploy's checksum is included int he >>> checksums of >>> > do_package and subsequent tasks, even though its output isn't used by >>> any >>> > subsequent tasks. >>> >>> I also found this a bit unusual to put the tool into deploy/tools. >>> That's why I isolated this change. I think it requires some more >>> testing, but the rest of patchset can be accepted without it. >> >> >> Yeah, putting it there does require some discussion in general. >> Personally, I'm in favor of such a thing, because we don't want to put >> generated convenience tools in the sysroot, since we tell people explicitly >> to not use sysroot contents directly. >> >> Regarding getting it built for the image, I'd suggest adding something >> like IMAGE_TASKDEPENDS to image_type.bbclass, which is basically like >> IMAGE_DEPENDS, but with the task included. I.e. IMAGE_TASKDEPENDS_bmap = >> "bmap-tools-native:do_deploy". Then you wouldn't need the 'before >> do_package', and it'd fix the from-sstate usage. >> > > We had some private conversation with Richard about deploy/tools, I'll try > to summarize results of our discussion here: > > Thing this, that content under deploy/tools/* supposed to be exported to > "release" directory, and should not be used by any other tool during normal > build/development process. > Those are for usage on another host, where bitbake and content of sysroots > are not usually available. > bmaptool is a good example as stand-alone binary that require only python2 > and nothing else to run. > Another example is dfu-util-static, which is also, not dependant even on > host's libusb even it runs on other host. > > So, in general, it's for static/standalone binaries that are supposed to > be published as part of "release" of the build, same like deploy/images/* > > For any classes, any internals of build process, they never should depend > on content of deploy/* and use *-native tools and binaries out of sysroot. > E.g. wic require bmap-tool, it should be dependant on presence of > bmap-tools-native installed into sysroot, and should be using both modules > and binary out of this sysroot. > I don't really understand the point you're trying to make here. As far as I can tell, wic put bmaptool into deploy/tools/ as a convenience for the user, not for wic itself to use, so it already does what you're saying it should do. -- Christopher Larson clarson at kergoth dot com Founder - BitBake, OpenEmbedded, OpenZaurus Maintainer - Tslib Senior Software Engineer, Mentor Graphics -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH v3 4/6] bmap-tools: generate standalone script
On Wed, May 4, 2016 at 9:55 PM, Christopher Larsonwrote: > > > >> > Also, the convention is to add do_deploy before do_build, not before >> > do_package, otherwise do_deploy's checksum is included int he checksums >> of >> > do_package and subsequent tasks, even though its output isn't used by >> any >> > subsequent tasks. >> >> I also found this a bit unusual to put the tool into deploy/tools. >> That's why I isolated this change. I think it requires some more >> testing, but the rest of patchset can be accepted without it. > > > Yeah, putting it there does require some discussion in general. > Personally, I'm in favor of such a thing, because we don't want to put > generated convenience tools in the sysroot, since we tell people explicitly > to not use sysroot contents directly. > > Regarding getting it built for the image, I'd suggest adding something > like IMAGE_TASKDEPENDS to image_type.bbclass, which is basically like > IMAGE_DEPENDS, but with the task included. I.e. IMAGE_TASKDEPENDS_bmap = > "bmap-tools-native:do_deploy". Then you wouldn't need the 'before > do_package', and it'd fix the from-sstate usage. > We had some private conversation with Richard about deploy/tools, I'll try to summarize results of our discussion here: Thing this, that content under deploy/tools/* supposed to be exported to "release" directory, and should not be used by any other tool during normal build/development process. Those are for usage on another host, where bitbake and content of sysroots are not usually available. bmaptool is a good example as stand-alone binary that require only python2 and nothing else to run. Another example is dfu-util-static, which is also, not dependant even on host's libusb even it runs on other host. So, in general, it's for static/standalone binaries that are supposed to be published as part of "release" of the build, same like deploy/images/* For any classes, any internals of build process, they never should depend on content of deploy/* and use *-native tools and binaries out of sysroot. E.g. wic require bmap-tool, it should be dependant on presence of bmap-tools-native installed into sysroot, and should be using both modules and binary out of this sysroot. This is the way how I'm using it inside Ostro Project: description in our image type class: https://github.com/ostroproject/meta-ostro/blob/master/meta-ostro/classes/image-dsk.bbclass#L32 https://github.com/ostroproject/meta-ostro/blob/master/meta-ostro/classes/image-dsk.bbclass#L131 actual usage of bmaptools modules from sysroot: https://github.com/ostroproject/meta-ostro/blob/master/meta-ostro/lib/image-dsk.py#L16 -- br, Alexander Kanevskiy -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH v3 4/6] bmap-tools: generate standalone script
On Mon, 2016-05-02 at 15:22 +0300, Ed Bartosh wrote: > Generated standalone bmaptool script that is more convenient > to use than native script. It can be run straight from > its location ./tmp/deploy/tools/bmaptool. The script doesn't > depend on anything except Python. > > Signed-off-by: Ed Bartosh> --- > meta/recipes-support/bmap-tools/bmap-tools_3.2.bb | 20 > +++- > 1 file changed, 19 insertions(+), 1 deletion(-) > > diff --git a/meta/recipes-support/bmap-tools/bmap-tools_3.2.bb > b/meta/recipes-support/bmap-tools/bmap-tools_3.2.bb > index e10f5fd..0f95788 100644 > --- a/meta/recipes-support/bmap-tools/bmap-tools_3.2.bb > +++ b/meta/recipes-support/bmap-tools/bmap-tools_3.2.bb > @@ -15,10 +15,28 @@ SRC_URI[sha256sum] = > "cc6c7f7dc0a37e2a32deb127308e24e6c4b80bfb54f3803c308efab02b > > RDEPENDS_${PN} = "python-core python-compression" > > -inherit setuptools > +inherit setuptools deploy > > BBCLASSEXTEND = "native" > > do_install_append_class-native() { > sed -i -e 's|^#!.*/usr/bin/env python|#! /usr/bin/env > nativepython|' ${D}${bindir}/bmaptool > } > + > +do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_TOOLS}" > +do_deploy[stamp-extra-info] = "" > +do_deploy_class-native() { > +cp bmaptool __main__.py > +python -m zipfile -c bmaptool.zip bmaptools __main__.py > +echo '#!/usr/bin/env python' | cat - bmaptool.zip > bmaptool > -standalone > +install -d ${DEPLOYDIR} > +install -m 0755 bmaptool-standalone ${DEPLOYDIR}/bmaptool-${PV} > +rm -f ${DEPLOYDIR}/bmaptool > +ln -sf ./bmaptool-${PV} ${DEPLOYDIR}/bmaptool > +} > + > +do_deploy() { > +: > +} > + > +addtask deploy before do_package after do_install I'm not sure if we want to do this in the first place. Regardless, the addtask here is incorrect too. Cheers, Richard -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH v3 4/6] bmap-tools: generate standalone script
On Wed, May 4, 2016 at 7:01 AM, Ed Bartoshwrote: > On Tue, May 03, 2016 at 07:33:58PM +, Christopher Larson wrote: > > On Tue, May 3, 2016 at 7:31 PM, Christopher Larson > > wrote: > > > > > > > > > > > On Mon, May 2, 2016 at 12:22 PM, Ed Bartosh < > ed.bart...@linux.intel.com> > > > wrote: > > > > > >> Generated standalone bmaptool script that is more convenient > > >> to use than native script. It can be run straight from > > >> its location ./tmp/deploy/tools/bmaptool. The script doesn't > > >> depend on anything except Python. > > >> > > >> Signed-off-by: Ed Bartosh > > >> > > > > > > You're right that this is very handy, but we hit a problem when > building > > > from shared state. This uses do_deploy to deploy bmaptool into > > > DEPLOY_DIR_IMAGE, but IMAGE_DEPENDS only depends on > do_populate_sysroot, > > > not do_deploy, so an image build from sstate won't write this script > out. > > > > > > I don't know the right way to fix this, but it's a problem. I just ran > > > into it :) > > > > > > > Also, the convention is to add do_deploy before do_build, not before > > do_package, otherwise do_deploy's checksum is included int he checksums > of > > do_package and subsequent tasks, even though its output isn't used by any > > subsequent tasks. > > I also found this a bit unusual to put the tool into deploy/tools. > That's why I isolated this change. I think it requires some more > testing, but the rest of patchset can be accepted without it. Yeah, putting it there does require some discussion in general. Personally, I'm in favor of such a thing, because we don't want to put generated convenience tools in the sysroot, since we tell people explicitly to not use sysroot contents directly. Regarding getting it built for the image, I'd suggest adding something like IMAGE_TASKDEPENDS to image_type.bbclass, which is basically like IMAGE_DEPENDS, but with the task included. I.e. IMAGE_TASKDEPENDS_bmap = "bmap-tools-native:do_deploy". Then you wouldn't need the 'before do_package', and it'd fix the from-sstate usage. -- Christopher Larson clarson at kergoth dot com Founder - BitBake, OpenEmbedded, OpenZaurus Maintainer - Tslib Senior Software Engineer, Mentor Graphics -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH v3 4/6] bmap-tools: generate standalone script
On Tue, May 03, 2016 at 07:33:58PM +, Christopher Larson wrote: > On Tue, May 3, 2016 at 7:31 PM, Christopher Larson> wrote: > > > > > > > On Mon, May 2, 2016 at 12:22 PM, Ed Bartosh > > wrote: > > > >> Generated standalone bmaptool script that is more convenient > >> to use than native script. It can be run straight from > >> its location ./tmp/deploy/tools/bmaptool. The script doesn't > >> depend on anything except Python. > >> > >> Signed-off-by: Ed Bartosh > >> > > > > You're right that this is very handy, but we hit a problem when building > > from shared state. This uses do_deploy to deploy bmaptool into > > DEPLOY_DIR_IMAGE, but IMAGE_DEPENDS only depends on do_populate_sysroot, > > not do_deploy, so an image build from sstate won't write this script out. > > > > I don't know the right way to fix this, but it's a problem. I just ran > > into it :) > > > > Also, the convention is to add do_deploy before do_build, not before > do_package, otherwise do_deploy's checksum is included int he checksums of > do_package and subsequent tasks, even though its output isn't used by any > subsequent tasks. I also found this a bit unusual to put the tool into deploy/tools. That's why I isolated this change. I think it requires some more testing, but the rest of patchset can be accepted without it. -- Regards, Ed -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH v3 4/6] bmap-tools: generate standalone script
On Tue, May 3, 2016 at 7:31 PM, Christopher Larsonwrote: > > > On Mon, May 2, 2016 at 12:22 PM, Ed Bartosh > wrote: > >> Generated standalone bmaptool script that is more convenient >> to use than native script. It can be run straight from >> its location ./tmp/deploy/tools/bmaptool. The script doesn't >> depend on anything except Python. >> >> Signed-off-by: Ed Bartosh >> > > You're right that this is very handy, but we hit a problem when building > from shared state. This uses do_deploy to deploy bmaptool into > DEPLOY_DIR_IMAGE, but IMAGE_DEPENDS only depends on do_populate_sysroot, > not do_deploy, so an image build from sstate won't write this script out. > > I don't know the right way to fix this, but it's a problem. I just ran > into it :) > Also, the convention is to add do_deploy before do_build, not before do_package, otherwise do_deploy's checksum is included int he checksums of do_package and subsequent tasks, even though its output isn't used by any subsequent tasks. -- Christopher Larson clarson at kergoth dot com Founder - BitBake, OpenEmbedded, OpenZaurus Maintainer - Tslib Senior Software Engineer, Mentor Graphics -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH v3 4/6] bmap-tools: generate standalone script
On Mon, May 2, 2016 at 12:22 PM, Ed Bartoshwrote: > Generated standalone bmaptool script that is more convenient > to use than native script. It can be run straight from > its location ./tmp/deploy/tools/bmaptool. The script doesn't > depend on anything except Python. > > Signed-off-by: Ed Bartosh > You're right that this is very handy, but we hit a problem when building from shared state. This uses do_deploy to deploy bmaptool into DEPLOY_DIR_IMAGE, but IMAGE_DEPENDS only depends on do_populate_sysroot, not do_deploy, so an image build from sstate won't write this script out. I don't know the right way to fix this, but it's a problem. I just ran into it :) -- Christopher Larson clarson at kergoth dot com Founder - BitBake, OpenEmbedded, OpenZaurus Maintainer - Tslib Senior Software Engineer, Mentor Graphics -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH v3 4/6] bmap-tools: generate standalone script
Generated standalone bmaptool script that is more convenient to use than native script. It can be run straight from its location ./tmp/deploy/tools/bmaptool. The script doesn't depend on anything except Python. Signed-off-by: Ed Bartosh--- meta/recipes-support/bmap-tools/bmap-tools_3.2.bb | 20 +++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/meta/recipes-support/bmap-tools/bmap-tools_3.2.bb b/meta/recipes-support/bmap-tools/bmap-tools_3.2.bb index e10f5fd..0f95788 100644 --- a/meta/recipes-support/bmap-tools/bmap-tools_3.2.bb +++ b/meta/recipes-support/bmap-tools/bmap-tools_3.2.bb @@ -15,10 +15,28 @@ SRC_URI[sha256sum] = "cc6c7f7dc0a37e2a32deb127308e24e6c4b80bfb54f3803c308efab02b RDEPENDS_${PN} = "python-core python-compression" -inherit setuptools +inherit setuptools deploy BBCLASSEXTEND = "native" do_install_append_class-native() { sed -i -e 's|^#!.*/usr/bin/env python|#! /usr/bin/env nativepython|' ${D}${bindir}/bmaptool } + +do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_TOOLS}" +do_deploy[stamp-extra-info] = "" +do_deploy_class-native() { +cp bmaptool __main__.py +python -m zipfile -c bmaptool.zip bmaptools __main__.py +echo '#!/usr/bin/env python' | cat - bmaptool.zip > bmaptool-standalone +install -d ${DEPLOYDIR} +install -m 0755 bmaptool-standalone ${DEPLOYDIR}/bmaptool-${PV} +rm -f ${DEPLOYDIR}/bmaptool +ln -sf ./bmaptool-${PV} ${DEPLOYDIR}/bmaptool +} + +do_deploy() { +: +} + +addtask deploy before do_package after do_install -- 2.1.4 -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core