Re: [OE-core] [PATCH 0/6] YOCTO #12937 - Consistent naming scheme for deployed artifacts
On Thu, 2019-10-03 at 10:27 +0200, Martin Jansa wrote: > Any feedback on this or does it have to wait for 3.1? This is the kind of change we need to make early in the release since it means the release scripts, release engineering and potentially QA all need to update. It was late enough in M3 and we had enough other issues that I didn't really want to go here then. Please do remind me early in 3.1 to get this sorted. > At least the last patch is a fix for regression already included in > 3.0. Thanks, it applied with offset fuzz so I'll queue it. Cheers, Richard -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 0/6] YOCTO #12937 - Consistent naming scheme for deployed artifacts
Any feedback on this or does it have to wait for 3.1? At least the last patch is a fix for regression already included in 3.0. On Fri, Aug 23, 2019 at 8:01 AM Martin Jansa wrote: > Let me explain a bit what these changes do for us in LGE. > > We have jenkins jobs for CI as well for official releases. > > All built artifacts are moved from jenkins builder to fileserver after > the build. > > Each jobs have some identifier which is then included in the filenames > of all relevant build artifacts, e.g. CI jobs will add e.g. lgsvl-verf-12 > to show where it was created and what build created it (12 is > BUILD_NUMBER from jenkins, verf is type of build, lgsvl is location). > > To do this you can already use IMAGE_VERSION_SUFFIX variable and add > this as a suffix to current artifact names. But that has some bad > limitations. > > A) If you keep IMAGE_VERSION_SUFFIX in do_deploy signatures, then > the artifacts will be rebuilt even when the deploy sstate archive in > cache is identical except tha filename. We had this for a while, but > all CI jobs were slow, because of rebuilding kernel every single time. > > B) If you vardepexclude IMAGE_VERSION_SUFFIX from the tasks which use > it, you get faster CI builds, but with inconsistent artifact names when > kernel deploy sstate is reused, e.g. image will have lgsvl-verf-12 > but all kernel artifacts will end with lgsvl-verf-11, when the kernel > do_deploy was reused from previously built sstate-cache > > It gets even worse with B) when you have some other tooling (like > runtime testing farm) which is tasked to flash image and kernel from > lgsvl-verf-12 and it fails to find kernel image to flash, because it's > named differently. > > C) Using version-less artifacts and just storing them in different > directories might work better, but then it would make sense to include > IMAGE_VERSION_SUFFIX in DEPLOY_DIR_IMAGE and remove it from the actual > files inside (with version-less symlinks pointing to them). But this is > a bit problematic when the individual images are usually downloaded by > BFUs over http (and they end with various identically named files for > which they don't remember from where they came). > > So this was the motivation why we have this in webOS. > > The difference for you (most people shouldn't even notice): > > 1) hard links instead of symlinks in DEPLOY_DIR_IMAGE, because now the >version is in the *_LINK_* variables and you don't want symlink with >version release-1 pointing to file created with release-2 build. > > 2) do_deploy, do_rootfs can still be reused from sstate, it will restore >the version-less artifact and then just quicky add another hardlink with >new filename (do_deploy_links task). > > 3) we're using this for couple years (badly hacked into OE, because we > didn't >want to overlay all relevant .bbclasses, but still needed to inject > do_deploy_links >task dependency in multiple places) and the only issue I've noticed was > with >one our proprietary IMAGE_FSTYPE format which was appending to image > file in >DEPLOY_DIR_IMAGE if it existed before, instead of overwritting it from > scratch - >to fix that I've just changed fstype function to remove the file before > creating >it again. > > 4) Examples: >All show "ls -laih DEPLOY_DIR_IMAGE | sort" to show the symlinks and > hardlinks > >TMPDIR is removed between each example, except the e) and f), but > sstate-cache >is kept in all cases and reused where possible > >MODULE_TARBALL_DEPLOY = "1" is added to local.conf to have more than > just 1 >kernel image as artifact from kernel (e.g. rpi MACHINEs have a lot of > them with >all the dtds). > > a) Current master with default values: > > $ ls -laih qemux86-64-master-default/ | sort > 47054849 drwxr-xr-x 13 bitbake bitbake 4.0K Aug 16 16:37 .. > 50735788 -rw-r--r-- 1 bitbake bitbake 612K Aug 15 17:16 > grub-efi-bootx64.efi > 50735796 drwxr-xr-x 2 bitbake bitbake 4.0K Aug 16 14:23 . > 50741892 -rwxr-xr-x 1 bitbake bitbake 95K Aug 15 17:16 > systemd-bootx64.efi > 52067796 -rw-r--r-- 1 bitbake bitbake 7.6M Aug 16 14:22 > bzImage--5.0.19+git2+c2e34d9ab2_00638cdd8f-r0.17-qemux86-64-20190816141126.bin > 52067797 lrwxrwxrwx 1 bitbake bitbake 78 Aug 16 14:22 > bzImage-qemux86-64.bin -> > bzImage--5.0.19+git2+c2e34d9ab2_00638cdd8f-r0.17-qemux86-64-20190816141126.bin > 52067798 lrwxrwxrwx 1 bitbake bitbake 78 Aug 16 14:22 bzImage -> > bzImage--5.0.19+git2+c2e34d9ab2_00638cdd8f-r0.17-qemux86-64-20190816141126.bin > 52067799 -rw-r--r-- 1 bitbake bitbake 7.0M Aug 16 14:22 > modules--5.0.19+git2+c2e34d9ab2_00638cdd8f-r0.17-qemux86-64-20190816141126.tgz > 52068116 lrwxrwxrwx 1 bitbake bitbake 78 Aug 16 14:22 > modules-qemux86-64.tgz -> > modules--5.0.19+git2+c2e34d9ab2_00638cdd8f-r0.17-qemux86-64-20190816141126.tgz > 52068127 -rw-r--r-- 1 bitbake bitbake 1.3K Aug 16 14:23 > core-image-minimal-qemux86-64-20190816141126.qemuboot.conf > 52068128 lrwxrwxrwx 1 bitbake bi
[OE-core] [PATCH 0/6] YOCTO #12937 - Consistent naming scheme for deployed artifacts
Let me explain a bit what these changes do for us in LGE. We have jenkins jobs for CI as well for official releases. All built artifacts are moved from jenkins builder to fileserver after the build. Each jobs have some identifier which is then included in the filenames of all relevant build artifacts, e.g. CI jobs will add e.g. lgsvl-verf-12 to show where it was created and what build created it (12 is BUILD_NUMBER from jenkins, verf is type of build, lgsvl is location). To do this you can already use IMAGE_VERSION_SUFFIX variable and add this as a suffix to current artifact names. But that has some bad limitations. A) If you keep IMAGE_VERSION_SUFFIX in do_deploy signatures, then the artifacts will be rebuilt even when the deploy sstate archive in cache is identical except tha filename. We had this for a while, but all CI jobs were slow, because of rebuilding kernel every single time. B) If you vardepexclude IMAGE_VERSION_SUFFIX from the tasks which use it, you get faster CI builds, but with inconsistent artifact names when kernel deploy sstate is reused, e.g. image will have lgsvl-verf-12 but all kernel artifacts will end with lgsvl-verf-11, when the kernel do_deploy was reused from previously built sstate-cache It gets even worse with B) when you have some other tooling (like runtime testing farm) which is tasked to flash image and kernel from lgsvl-verf-12 and it fails to find kernel image to flash, because it's named differently. C) Using version-less artifacts and just storing them in different directories might work better, but then it would make sense to include IMAGE_VERSION_SUFFIX in DEPLOY_DIR_IMAGE and remove it from the actual files inside (with version-less symlinks pointing to them). But this is a bit problematic when the individual images are usually downloaded by BFUs over http (and they end with various identically named files for which they don't remember from where they came). So this was the motivation why we have this in webOS. The difference for you (most people shouldn't even notice): 1) hard links instead of symlinks in DEPLOY_DIR_IMAGE, because now the version is in the *_LINK_* variables and you don't want symlink with version release-1 pointing to file created with release-2 build. 2) do_deploy, do_rootfs can still be reused from sstate, it will restore the version-less artifact and then just quicky add another hardlink with new filename (do_deploy_links task). 3) we're using this for couple years (badly hacked into OE, because we didn't want to overlay all relevant .bbclasses, but still needed to inject do_deploy_links task dependency in multiple places) and the only issue I've noticed was with one our proprietary IMAGE_FSTYPE format which was appending to image file in DEPLOY_DIR_IMAGE if it existed before, instead of overwritting it from scratch - to fix that I've just changed fstype function to remove the file before creating it again. 4) Examples: All show "ls -laih DEPLOY_DIR_IMAGE | sort" to show the symlinks and hardlinks TMPDIR is removed between each example, except the e) and f), but sstate-cache is kept in all cases and reused where possible MODULE_TARBALL_DEPLOY = "1" is added to local.conf to have more than just 1 kernel image as artifact from kernel (e.g. rpi MACHINEs have a lot of them with all the dtds). a) Current master with default values: $ ls -laih qemux86-64-master-default/ | sort 47054849 drwxr-xr-x 13 bitbake bitbake 4.0K Aug 16 16:37 .. 50735788 -rw-r--r-- 1 bitbake bitbake 612K Aug 15 17:16 grub-efi-bootx64.efi 50735796 drwxr-xr-x 2 bitbake bitbake 4.0K Aug 16 14:23 . 50741892 -rwxr-xr-x 1 bitbake bitbake 95K Aug 15 17:16 systemd-bootx64.efi 52067796 -rw-r--r-- 1 bitbake bitbake 7.6M Aug 16 14:22 bzImage--5.0.19+git2+c2e34d9ab2_00638cdd8f-r0.17-qemux86-64-20190816141126.bin 52067797 lrwxrwxrwx 1 bitbake bitbake 78 Aug 16 14:22 bzImage-qemux86-64.bin -> bzImage--5.0.19+git2+c2e34d9ab2_00638cdd8f-r0.17-qemux86-64-20190816141126.bin 52067798 lrwxrwxrwx 1 bitbake bitbake 78 Aug 16 14:22 bzImage -> bzImage--5.0.19+git2+c2e34d9ab2_00638cdd8f-r0.17-qemux86-64-20190816141126.bin 52067799 -rw-r--r-- 1 bitbake bitbake 7.0M Aug 16 14:22 modules--5.0.19+git2+c2e34d9ab2_00638cdd8f-r0.17-qemux86-64-20190816141126.tgz 52068116 lrwxrwxrwx 1 bitbake bitbake 78 Aug 16 14:22 modules-qemux86-64.tgz -> modules--5.0.19+git2+c2e34d9ab2_00638cdd8f-r0.17-qemux86-64-20190816141126.tgz 52068127 -rw-r--r-- 1 bitbake bitbake 1.3K Aug 16 14:23 core-image-minimal-qemux86-64-20190816141126.qemuboot.conf 52068128 lrwxrwxrwx 1 bitbake bitbake 58 Aug 16 14:23 core-image-minimal-qemux86-64.qemuboot.conf -> core-image-minimal-qemux86-64-20190816141126.qemuboot.conf 52068136 -rw-r--r-- 1 bitbake bitbake 162K Aug 16 14:23 core-image-minimal-qemux86-64-20190816141126.testdata.json 52068137 lrwxrwxrwx 1 bitbake bitbake 58 Aug 16 14:23 core-image-minimal-qemux86-64.testdata.json -> core