Re: [yocto] [RFC] devtool: support multiple sources
Le jeu. 23 nov. 2023 à 00:36, Richard Purdie a écrit : > > On Wed, 2023-11-22 at 13:53 +0100, Julien Stephan wrote: > > Le mar. 14 nov. 2023 à 12:21, Alexander Kanavin > > a écrit : > > > > > > I think no one actually anticipated all these corner cases when > > > devtool was written. In general it's simply not designed to handle > > > multiple sources in SRC_URI, it can do (tarball or git)+patches, and > > > not any of the other options. No one is expecting a fix for all of > > > them, perhaps it's best to keep the scope to plain gitsm://. > > > > > > > Hi Alex, > > > > I sent a series to add submodule support. As you suggested I focused > > myself on gitsm recipes, but I had to handle the situation of a git > > tree extracted into S because it was throwing an error now with my > > submodule implementation (while it was not working but not throwing > > any error on master). > > > > > I don't have specific advice for the broader issue; if you have ideas, > > > it would be good to at least document them. > > > > > > Alex > > > > > > For the other cases, I was thinking of using the newly merged API > > "unpack tracer" > > (https://git.yoctoproject.org/poky/commit/?id=ef3e46afd910d4b7727d42c4c18b501525c65695) > > (suggested by RP in this thread > > https://git.yoctoproject.org/poky/commit/?id=ef3e46afd910d4b7727d42c4c18b501525c65695). > > I really think that we could leverage this new api to create a devtool > > unpack tracer, so we can have a better idea of what is unpacked and > > where. This could help to solve the corner cases I described > > previously but also the bug 15162 > > (https://bugzilla.yoctoproject.org/show_bug.cgi?id=15162). I think it > > may also help to simplify devtool code in various places. > > Where should I document this? > > Should I open a new ticket? > > Yes please, it would be good to at least document the idea as I do > think it could help devtool with a number of the corner cases. A > bugzilla bug for it is the place to do it. > Hi Richard and Alexander, FYI, here is the bugzilla bug https://bugzilla.yoctoproject.org/show_bug.cgi?id=15294 Cheers Julien > Cheers, > > Richard -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#61733): https://lists.yoctoproject.org/g/yocto/message/61733 Mute This Topic: https://lists.yoctoproject.org/mt/102509834/21656 Group Owner: yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [yocto] [RFC] devtool: support multiple sources
On Wed, 2023-11-22 at 13:53 +0100, Julien Stephan wrote: > Le mar. 14 nov. 2023 à 12:21, Alexander Kanavin > a écrit : > > > > I think no one actually anticipated all these corner cases when > > devtool was written. In general it's simply not designed to handle > > multiple sources in SRC_URI, it can do (tarball or git)+patches, and > > not any of the other options. No one is expecting a fix for all of > > them, perhaps it's best to keep the scope to plain gitsm://. > > > > Hi Alex, > > I sent a series to add submodule support. As you suggested I focused > myself on gitsm recipes, but I had to handle the situation of a git > tree extracted into S because it was throwing an error now with my > submodule implementation (while it was not working but not throwing > any error on master). > > > I don't have specific advice for the broader issue; if you have ideas, > > it would be good to at least document them. > > > > Alex > > > For the other cases, I was thinking of using the newly merged API > "unpack tracer" > (https://git.yoctoproject.org/poky/commit/?id=ef3e46afd910d4b7727d42c4c18b501525c65695) > (suggested by RP in this thread > https://git.yoctoproject.org/poky/commit/?id=ef3e46afd910d4b7727d42c4c18b501525c65695). > I really think that we could leverage this new api to create a devtool > unpack tracer, so we can have a better idea of what is unpacked and > where. This could help to solve the corner cases I described > previously but also the bug 15162 > (https://bugzilla.yoctoproject.org/show_bug.cgi?id=15162). I think it > may also help to simplify devtool code in various places. > Where should I document this? > Should I open a new ticket? Yes please, it would be good to at least document the idea as I do think it could help devtool with a number of the corner cases. A bugzilla bug for it is the place to do it. Cheers, Richard -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#61732): https://lists.yoctoproject.org/g/yocto/message/61732 Mute This Topic: https://lists.yoctoproject.org/mt/102509834/21656 Group Owner: yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/yocto/leave/6691583/21656/737036229/xyzzy [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [yocto] [RFC] devtool: support multiple sources
Le mar. 14 nov. 2023 à 12:21, Alexander Kanavin
a écrit :
>
> I think no one actually anticipated all these corner cases when
> devtool was written. In general it's simply not designed to handle
> multiple sources in SRC_URI, it can do (tarball or git)+patches, and
> not any of the other options. No one is expecting a fix for all of
> them, perhaps it's best to keep the scope to plain gitsm://.
>
Hi Alex,
I sent a series to add submodule support. As you suggested I focused
myself on gitsm recipes, but I had to handle the situation of a git
tree extracted into S because it was throwing an error now with my
submodule implementation (while it was not working but not throwing
any error on master).
> I don't have specific advice for the broader issue; if you have ideas,
> it would be good to at least document them.
>
> Alex
For the other cases, I was thinking of using the newly merged API
"unpack tracer"
(https://git.yoctoproject.org/poky/commit/?id=ef3e46afd910d4b7727d42c4c18b501525c65695)
(suggested by RP in this thread
https://git.yoctoproject.org/poky/commit/?id=ef3e46afd910d4b7727d42c4c18b501525c65695).
I really think that we could leverage this new api to create a devtool
unpack tracer, so we can have a better idea of what is unpacked and
where. This could help to solve the corner cases I described
previously but also the bug 15162
(https://bugzilla.yoctoproject.org/show_bug.cgi?id=15162). I think it
may also help to simplify devtool code in various places.
Where should I document this?
Should I open a new ticket?
Cheers
Julien
>
> On Fri, 10 Nov 2023 at 17:24, Julien Stephan wrote:
> >
> > Hi all,
> >
> > Sorry for the long message here, but I would really appreciate
> > feedback from the community :)
> >
> > I am currently working on bug #14141
> > (https://bugzilla.yoctoproject.org/show_bug.cgi?id=14141). The main
> > goal of this bug is to correctly support git submodules with devtool.
> >
> > I currently have an almost ready branch here:
> > https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/devtool-submodule-fix
> >
> > I asked Jon Mason (reporter of the bug) to test the branch, he
> > confirmed that its working for git submodules (i.e recipes using gitsm
> > only) such as vulkan-samples or hafnium (meta-arm), but he also
> > pointed out a new issue highlighted by the recipe edk2-firmware: this
> > recipe is using gitsm for the primary source then extract another git
> > repository inside S.
> >
> > This issue is not related only to gitsm + git recipes, I reproduced
> > this kind of error with cross-localedef-native recipe, using git +
> > git or https + git. But I think the issue is even more global: as of
> > today devtool doesn't properly handle recipes that unpack secondary
> > sources inside S. Let suppose we have:
> >
> > SRC_URI = " \
> > http://.tar.gz \
> > http://.tar.gz;subdir=mysubdir \
> > "
> > If I need to patch something inside mysubdir I would expect to have
> > something like:
> > SRC_URI = " \
> > http://.tar.gz \
> > http://.tar.gz;subdir=mysubdir \
> > file://mypatch.patch;patchdir=mysubdir \
> > "
> > Of course this is working, but if I run devtool modify, create my
> > patch and do a devtool finish I would have something like
> > SRC_URI = " \
> > http://.tar.gz \
> > http://.tar.gz;subdir=mysubdir \
> > file://mypatch.patch \
> > "
> > where mypatch.patch will patch files inside mysubdir as if they were
> > originally part of the primary source.
> > That might be acceptable (but I personally would prefer to have a
> > patchir parameter matching the subdir parameter).
> > Besides that, there is actually a bug here: trying to modify inside
> > devtool a file that was added using file:// will add a new patch AND
> > modify the original patch... (Tested on master branch with bzip2 with
> > the following: devtool modify bzip2, modify Makefile.am, commit the
> > modification, devtool finish bzip2 will create a new patch file AND
> > also update the originale Makefile.am inside the recipe space)
> >
> > But things get really bad when using a git source as secondary source:
> > SRC_URI = " \
> > http://.tar.gz \
> > git ://;destsuffix=mysubdir \
> > "
> > mysubdir gets automatically committed inside the primary source
> > ("Commiting changes from do patch"). This is an issue because doing
> > "git add" on a git directory adds it as a submodule (in a wrong way,
> > the correct way of adding it would be "git submodule add url path").
> > Here we end up with a submodule which is not populated into
> > .gitmodules then "git submodule foreach" commands just fail.
> >
> > My hack is to detect that git directories were added after unpack and
> > properly add them as submodules (" wip: support git tree extracted
> > inside S" in my branch) but I am not 100% satisfied with this because
> > it implies that devtool is managing secondary source differently if
> > they are git directories or not..
> >
> > So the long story made short:
> > * current
Re: [yocto] [RFC] devtool: support multiple sources
I think no one actually anticipated all these corner cases when
devtool was written. In general it's simply not designed to handle
multiple sources in SRC_URI, it can do (tarball or git)+patches, and
not any of the other options. No one is expecting a fix for all of
them, perhaps it's best to keep the scope to plain gitsm://.
I don't have specific advice for the broader issue; if you have ideas,
it would be good to at least document them.
Alex
On Fri, 10 Nov 2023 at 17:24, Julien Stephan wrote:
>
> Hi all,
>
> Sorry for the long message here, but I would really appreciate
> feedback from the community :)
>
> I am currently working on bug #14141
> (https://bugzilla.yoctoproject.org/show_bug.cgi?id=14141). The main
> goal of this bug is to correctly support git submodules with devtool.
>
> I currently have an almost ready branch here:
> https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/devtool-submodule-fix
>
> I asked Jon Mason (reporter of the bug) to test the branch, he
> confirmed that its working for git submodules (i.e recipes using gitsm
> only) such as vulkan-samples or hafnium (meta-arm), but he also
> pointed out a new issue highlighted by the recipe edk2-firmware: this
> recipe is using gitsm for the primary source then extract another git
> repository inside S.
>
> This issue is not related only to gitsm + git recipes, I reproduced
> this kind of error with cross-localedef-native recipe, using git +
> git or https + git. But I think the issue is even more global: as of
> today devtool doesn't properly handle recipes that unpack secondary
> sources inside S. Let suppose we have:
>
> SRC_URI = " \
> http://.tar.gz \
> http://.tar.gz;subdir=mysubdir \
> "
> If I need to patch something inside mysubdir I would expect to have
> something like:
> SRC_URI = " \
> http://.tar.gz \
> http://.tar.gz;subdir=mysubdir \
> file://mypatch.patch;patchdir=mysubdir \
> "
> Of course this is working, but if I run devtool modify, create my
> patch and do a devtool finish I would have something like
> SRC_URI = " \
> http://.tar.gz \
> http://.tar.gz;subdir=mysubdir \
> file://mypatch.patch \
> "
> where mypatch.patch will patch files inside mysubdir as if they were
> originally part of the primary source.
> That might be acceptable (but I personally would prefer to have a
> patchir parameter matching the subdir parameter).
> Besides that, there is actually a bug here: trying to modify inside
> devtool a file that was added using file:// will add a new patch AND
> modify the original patch... (Tested on master branch with bzip2 with
> the following: devtool modify bzip2, modify Makefile.am, commit the
> modification, devtool finish bzip2 will create a new patch file AND
> also update the originale Makefile.am inside the recipe space)
>
> But things get really bad when using a git source as secondary source:
> SRC_URI = " \
> http://.tar.gz \
> git ://;destsuffix=mysubdir \
> "
> mysubdir gets automatically committed inside the primary source
> ("Commiting changes from do patch"). This is an issue because doing
> "git add" on a git directory adds it as a submodule (in a wrong way,
> the correct way of adding it would be "git submodule add url path").
> Here we end up with a submodule which is not populated into
> .gitmodules then "git submodule foreach" commands just fail.
>
> My hack is to detect that git directories were added after unpack and
> properly add them as submodules (" wip: support git tree extracted
> inside S" in my branch) but I am not 100% satisfied with this because
> it implies that devtool is managing secondary source differently if
> they are git directories or not..
>
> So the long story made short:
> * current implementation of devtool is buggy (at least with the
> scenario described above with bzip2), I may try to work on this latter
> * is the current behaviour of devtool acceptable ? (i.e not adding
> patchir parameter on patches)
> * is my "hack" acceptable to handle git secondary sources as a real
> git submodule?
>
> This is giving me headaches and I would really appreciate the feedback
> of the community on this.
>
> Cheers
> Julien
>
>
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#61678): https://lists.yoctoproject.org/g/yocto/message/61678
Mute This Topic: https://lists.yoctoproject.org/mt/102509834/21656
Group Owner: yocto+ow...@lists.yoctoproject.org
Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
