> Using the git:// SRC_URI did not populate the submodules' directories.
That's expected, you need to use gitsm:// to do that (at least the ones listed in https://github.com/pytorch/pytorch/blob/main/.gitmodules). I did not realize that the other gitsm:// you were added were actually submodules of pytorch itself, but Julien is right, it should work with one gitsm:// in SRC_URI (even with patches applied to various submodules in do_patch). On Wed, Oct 18, 2023 at 4:24 PM Julien Stephan <[email protected]> wrote: > Le mer. 18 oct. 2023 à 16:12, Böszörményi Zoltán <[email protected]> a > écrit : > > > > 2023. 10. 18. 15:28 keltezéssel, Julien Stephan írta: > > > Le mer. 18 oct. 2023 à 15:25, Zoltan Boszormenyi <[email protected]> > a écrit : > > >> Thanks, I will try it. > > >> > > >> 2023. 10. 18. 15:16 keltezéssel, Martin Jansa írta: > > >>> Just add the name to pytorch repo url and use only that in > SRCREV_FORMAT, you probably > > >>> don't care about representing the other repos in SRCPV if you're > updating them only > > >>> together with main pytorch update. > > >>> > > >>> On Wed, Oct 18, 2023 at 3:05 PM Zoltan Boszormenyi < > [email protected]> wrote: > > >>> > > >>> Hi, > > >>> > > >>> I have a working python3-pytorch recipe which originally used > this: > > >>> > > >>> ======================================================= > > >>> addtask do_git_submodules after do_unpack before do_patch > > >>> > > >>> do_git_submodules[network] = "1" > > >>> > > >>> do_git_submodules () { > > >>> cd ${S} > > >>> git submodule update --init --recursive > > >>> } > > >>> ======================================================= > > >>> > > >>> I would like to replace the above with a series of gitsm:// > SRC_URI lines. > > >>> Here's a subset of submodules that shows the problem I am > seeing: > > >>> > > >>> SRC_URI = " \ > > >>> git:// > github.com/pytorch/pytorch.git;protocol=https;branch=release/2.1 > > >>> < > http://github.com/pytorch/pytorch.git;protocol=https;branch=release/2.1> \ > > >>> ... > > >>> gitsm:// > github.com/pytorch/FBGEMM.git;protocol=https;name=fbgemm;nobranch=1;destsuffix=third_party/fbgemm > > >>> < > http://github.com/pytorch/FBGEMM.git;protocol=https;name=fbgemm;nobranch=1;destsuffix=third_party/fbgemm > > > > >>> > > >>> \ > > >>> gitsm:// > github.com/asmjit/asmjit.git;protocol=https;name=fbgemmasmjit;destsuffix=third_party/fbgemm/third_party/asmjit > > >>> < > http://github.com/asmjit/asmjit.git;protocol=https;name=fbgemmasmjit;destsuffix=third_party/fbgemm/third_party/asmjit > > > > >>> > > >>> \ > > >>> gitsm:// > github.com/pytorch/cpuinfo.git;protocol=https;name=fbgemmcpuinfo;destsuffix=third_party/fbgemm/third_party/cpuinfo > > >>> < > http://github.com/pytorch/cpuinfo.git;protocol=https;name=fbgemmcpuinfo;destsuffix=third_party/fbgemm/third_party/cpuinfo > > > > >>> > > >>> \ > > >>> gitsm:// > github.com/NVIDIA/cutlass.git;protocol=https;name=fbgemmcutlass;destsuffix=third_party/fbgemm/third_party/cutlass > > >>> < > http://github.com/NVIDIA/cutlass.git;protocol=https;name=fbgemmcutlass;destsuffix=third_party/fbgemm/third_party/cutlass > > > > >>> > > >>> \ > > >>> gitsm:// > github.com/google/googletest.git;protocol=https;name=fbgemmgtest;destsuffix=third_party/fbgemm/third_party/googletest > > >>> < > http://github.com/google/googletest.git;protocol=https;name=fbgemmgtest;destsuffix=third_party/fbgemm/third_party/googletest > > > > >>> > > >>> \ > > >>> gitsm:// > github.com/ROCmSoftwarePlatform/hipify_torch.git;protocol=https;name=fbgemmhiptorch;destsuffix=third_party/fbgemm/third_party/hipify_torch > > >>> < > http://github.com/ROCmSoftwarePlatform/hipify_torch.git;protocol=https;name=fbgemmhiptorch;destsuffix=third_party/fbgemm/third_party/hipify_torch > > > > >>> > > >>> \ > > >>> ... > > >>> " > > > Hi Zoltan > > > This is not how you are supposed to use gitsm.. This is much simpler: > > > > > > SRC_URI = " gitsm:// > github.com/pytorch/pytorch.git;protocol=https;branch=release/2.1 > > > " > > > > > > will do all the magic for you :) > > > > When I first created that recipe (Yocto 4.0, PyTorch 1.13.0), it didn't. > > > > The problems were: > > * PyTorch source releases are not on https://pypi.org/project/torch > > This would have been the most convenient. > > * Yocto complains about using the (unstable) release tarball URL from > Github. > > * Using the git:// SRC_URI did not populate the submodules' directories. > > This was noticed because some submodules need patches to build under > Yocto. > > Without checking out submodules manually, those patches fail. > > > > Just checked the last one on mickledore, still a problem. > > > > Have Yocto master changed this? > > > > I used to have a kirkstone recipe using gitsm properly to fetch the > submodules. Can you reproduce your issue on master? And maybe share it > to help debugging? > > Cheers > Julien > > > You can look at the vulkan-sample recipes in poky for an example of > > > how to use it: > https://git.openembedded.org/openembedded-core/tree/meta/recipes-graphics/vulkan/vulkan-samples_git.bb?h=master > > > > > > Cheers > > > Julien > > > > > > > > >>> As you can see, there are recursively placed git submodules > > >>> and do_fetch fails with an error message about SRCREV_FORMAT > > >>> having to be set. > > >>> > > >>> The above is a limited subset of the complete list and > > >>> some of the repositories (like googletest, gloo, glog and > pybind11) > > >>> occurs multiple times as child submodules of upper ones, > > >>> with different SRCREV values in different leaf submodules. > > >>> > > >>> I have not found a conclusive example to use SRCREV_FORMAT > > >>> with git:// + gitsm:// > > >>> > > >>> Can someone enlighten me how to use SRCREV_FORMAT properly > > >>> for this case? Should I stick to the proven working nonstandard > way > > >>> and just run git submodule update --init --recursive? > > >>> > > >>> As an easy way out, is there a flag to git:// to process its > > >>> submodules automatically? > > >>> > > >>> Thanks in advance, > > >>> Zoltán Böszörményi > > >>> > > >>> > > >>> > > >>> > > >> > > >> > > >> > > >
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#189395): https://lists.openembedded.org/g/openembedded-core/message/189395 Mute This Topic: https://lists.openembedded.org/mt/102038382/21656 Group Owner: [email protected] Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
