Thanks for the detailed explanation. I understand the hesitancy around adding an extra parameter, especially given the cost and maintenance implications for both testing and long-term support. If we decide to incorporate it, I agree that it would be essential to include it in our test matrix to ensure it works reliably and doesn’t degrade over time.
One possible race condition could occur when there’s a parallel build for different images with distinct sets of required packages. For example, if one process is running an opkg update while another is creating an index with an opkg make index at the same time, this could cause the root filesystem image installation to fail. Regarding your concern about potential misuse and the risk of race conditions like above, I share that reservation. Would it be helpful if we documented these concerns prominently or added clear guidance on how to use this parameter safely? Thanks again for the details, which help me better understand Yocto internals. Sreejith On Mon, Nov 11, 2024 at 1:08 PM Richard Purdie < [email protected]> wrote: > On Mon, 2024-11-11 at 09:53 +0000, Sreejith Ravi wrote: > > We are not changing the default behavior of Yocto but are instead > > using our own custom class to handle IPK. > > > > Since create_packages_dir is a function in rootfs.py, it would be > > beneficial to add an option allowing users to select either the > > deploy directory or oe-rootfs-repo. This would also save the copy > > time to oe-rootfs when it is not required for their projects. > > For reference, the copy time here is minimal since the copy is created > using hardlinks to the files. > > The reason this is done is to filter the packages to the ones directly > in the recipe's dependency chain. This is because files can be written > to tmp/deploy/ while an image build is running and this can cause build > failures. > > For example consider core-image-sato-sdk and core-image-minimal being > built in parallel. The minimal rootfs will happen before the sato-sdk > one but if they all used the same directory, the minimal image could > see pieces of the SDK image. The files may be seen partially through > the write too. > > To avoid this and more complex forms of the race (where things are > removed as they're rebuilt due to changes), we create the hardlinked > copy of only the packages a given image recipe should "see". > > > Currently, in create_packages_dir, the option to check whether to > > create "oe-rootfs-repo" or use a symlink from "DEPLOY_DIR_IPK" to > > "oe-rootfs-repo" is ineffective with filterbydependencies, as it is > > always set to True. Our request is to add a configuration option for > > this behavior; otherwise, this check serves no purpose when used with > > filterbydependencies. > > > > if nodeps or not filterbydependencies: > > for arch in d.getVar("ALL_MULTILIB_PACKAGE_ARCHS").split() + > > d.getVar("ALL_MULTILIB_PACKAGE_ARCHS").replace("-", "_").split(): > > target = os.path.join(deploydir + "/" + arch) > > if os.path.exists(target): > > oe.path.symlink(target, subrepo_dir + "/" + arch, > > True) > > return > > > > > > This patch does not change the existing functionality. By default, it > > is set to True (consistent with default Yocto behavior), but users > > can adjust it as needed. > > I'm not saying yes or no to the patch but you have to understand that > adding an extra parameter means we have to support that code path. > Should we include it in our test matrix? If we do, there is the cost of > testing it but it should work. If we don't, there is a significant risk > it might stop working at some point. There is therefore a cost/risk to > both directions if we do add it. > > For me, the bigger concern is that we may then see people using this > without understanding the race concerns above too. > > Cheers, > > Richard > > >
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#207040): https://lists.openembedded.org/g/openembedded-core/message/207040 Mute This Topic: https://lists.openembedded.org/mt/109210754/21656 Group Owner: [email protected] Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
