Re: [yocto] Downloading Git Archives
On 9/27/19 2:27 AM, Ross Burton wrote: > On 27/09/2019 05:33, Chuck Wolber wrote: >> I was hoping that there was an equally simple way of grabbing git >> repository packages from an Internet connected host without having a >> bitbake build cache them for me. > > The easy way is to get bitbake to do it for you. A git mirror tarball > is just a clone with a specific name and path, but the easy solution > is definitely to run bitbake --c fetchall on a machine with > connectivity and expose the DL_DIR to the builders. > > This is what the yocto autobuilder does, by the way. The workers > don't have connectivity to the outside internet, so the master does a > fetchall with the proxies configured to populate DL_DIR before the > workers do the actual build. > Same thing here. I run a build system instance that simply downloads all the sources and places them on a local mirror. The only thing to think about is that if you have different targets they might require different source packages albeit there is always a larger overlap of packages that are always required. However, it's not too big of a problem launching the 'fetcher' instance repeatedly with different target since only what's missing will of course be downloaded. Another layer of complexity is of course added if you work with different versions of YP and OE layers. I solve this with Gitlab and submodules. A target project is a repo in Gitlab and the YP/OE layers are submodules. Gitlab handles that quite nicely. :rjs - Rudolf J Streif CEO/CTO ibeeto +1.855.442.3386 x700 signature.asc Description: OpenPGP digital signature -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] Downloading Git Archives
On 27/09/2019 05:33, Chuck Wolber wrote: I was hoping that there was an equally simple way of grabbing git repository packages from an Internet connected host without having a bitbake build cache them for me. The easy way is to get bitbake to do it for you. A git mirror tarball is just a clone with a specific name and path, but the easy solution is definitely to run bitbake --c fetchall on a machine with connectivity and expose the DL_DIR to the builders. This is what the yocto autobuilder does, by the way. The workers don't have connectivity to the outside internet, so the master does a fetchall with the proxies configured to populate DL_DIR before the workers do the actual build. Ross -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] Downloading Git Archives
Thanks for replying!
That is effectively what we have to do with the VM clone of the build
machine. We spin up the VM, do the build, see what is new in the downloads
directory, do our due diligence, and push the new stuff into the local
mirror behind the firewall. The primary build server must remain "sterile",
so everything has to be "hoisted" over the fence manually.
[Side Note: We also maintain a local clone of the openembedded and poky
repositories. Our build pins to a specific hash, which gets advanced after
we do our due diligence. This is when we discover that certain packages
need to be added to our ${LOCAL_MIRROR}. My desire is to automate as much
as possible, while exposing as little as possible.]
Whenever the nightly build fails due to a missing package, and I see a URL
with a (for example) .tar.gz in the build error log, I know my job is a lot
easier. I can simply wget from any Internet connected host, instead of
spinning up the VM clone, waiting for a build to complete, and then
figuring out what is new in the downloads directory.
I was hoping that there was an equally simple way of grabbing git
repository packages from an Internet connected host without having a
bitbake build cache them for me.
I understand that these packages are effectively a bare clone, but there
seems to be some additional non-obvious (to me at least) "magic" that makes
it more complicated than doing a git repo bare clone and wrapping it up
into a .tar.gz.
I am happy to start trawling through source code and figure it out on my
own, but I figured this had to be common knowledge to someone on this list.
..Ch:W..
On Thu, Sep 26, 2019 at 8:50 PM Rudolf J Streif
wrote:
> Chuck,
>
> Unless I am missing something (which I cannot entirely rule out), the
> default PREMIRRORS should actually do what you want. For the poky distro
> configuration has this:
>
> PREMIRRORS ??= "\
> bzr://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n \
> cvs://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n \
> git://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n \
> gitsm://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n \
> hg://.*/.*http://downloads.yoctoproject.org/mirror/sources/ \n \
> osc://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n \
> p4://.*/.*http://downloads.yoctoproject.org/mirror/sources/ \n \
> svn://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n"
>
> This causes the fetcher to intercept and translate all the SCM protocols
> into http downloads from the YP mirror. Since you are saying wget works for
> you this should actually automatically work for you. If you build system
> can at least download using http then
>
> bitbake --runall=fetch
>
> should download all the sources for you. Then you can disconnect the box
> from the Internet and start the build offline.
>
> :rjs
> On 9/26/19 8:06 PM, Chuck Wolber wrote:
>
> Hi All,
>
> Our build environment is stuck behind a firewall with no access to the
> Internet. When a Yocto build requires a new package, we normally extract
> the URL from the failed build log and "wget" the package from the upstream
> source, do our due diligence (checksums, etc), and then add it to the
> server pointed to by ${LOCAL_MIRROR}.
>
> Obviously that procedure does not work so well when a git repository needs
> to be fetched. We have tried various methods, but nothing seems to work
> reliably. Our current method is an annoying hack involving a virtual
> machine clone of the build server.
>
> I have searched far and wide, and cannot find any sort of procedure. Can
> someone point me at a procedure for translating something like git://
> git.yoctoproject.org/pseudo into a file we can store in our
> ${LOCAL_MIRROR}?
>
> Thank you,
>
> ..Ch:W..
>
> --
> -
> Rudolf J Streif
> CEO/CTO ibeeto
> +1.855.442.3386 x700
>
>
--
“I would challenge anyone here to think of a question upon which we once
had a scientific answer, however inadequate, but for which now the best
answer is a religious one." -Sam Harris
--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] Downloading Git Archives
Chuck,
Unless I am missing something (which I cannot entirely rule out), the
default PREMIRRORS should actually do what you want. For the poky distro
configuration has this:
PREMIRRORS ??= "\
bzr://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n \
cvs://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n \
git://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n \
gitsm://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n \
hg://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n \
osc://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n \
p4://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n \
svn://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n"
This causes the fetcher to intercept and translate all the SCM protocols
into http downloads from the YP mirror. Since you are saying wget works
for you this should actually automatically work for you. If you build
system can at least download using http then
bitbake --runall=fetch
should download all the sources for you. Then you can disconnect the box
from the Internet and start the build offline.
:rjs
On 9/26/19 8:06 PM, Chuck Wolber wrote:
> Hi All,
>
> Our build environment is stuck behind a firewall with no access to the
> Internet. When a Yocto build requires a new package, we normally
> extract the URL from the failed build log and "wget" the package from
> the upstream source, do our due diligence (checksums, etc), and then
> add it to the server pointed to by ${LOCAL_MIRROR}.
>
> Obviously that procedure does not work so well when a git repository
> needs to be fetched. We have tried various methods, but nothing seems
> to work reliably. Our current method is an annoying hack involving a
> virtual machine clone of the build server.
>
> I have searched far and wide, and cannot find any sort of procedure.
> Can someone point me at a procedure for translating something like
> git://git.yoctoproject.org/pseudo
> into a file we can store in our ${LOCAL_MIRROR}?
>
> Thank you,
>
> ..Ch:W..
>
--
-
Rudolf J Streif
CEO/CTO ibeeto
+1.855.442.3386 x700
signature.asc
Description: OpenPGP digital signature
--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto
[yocto] Downloading Git Archives
Hi All,
Our build environment is stuck behind a firewall with no access to the
Internet. When a Yocto build requires a new package, we normally extract
the URL from the failed build log and "wget" the package from the upstream
source, do our due diligence (checksums, etc), and then add it to the
server pointed to by ${LOCAL_MIRROR}.
Obviously that procedure does not work so well when a git repository needs
to be fetched. We have tried various methods, but nothing seems to work
reliably. Our current method is an annoying hack involving a virtual
machine clone of the build server.
I have searched far and wide, and cannot find any sort of procedure. Can
someone point me at a procedure for translating something like git://
git.yoctoproject.org/pseudo into a file we can store in our ${LOCAL_MIRROR}?
Thank you,
..Ch:W..
--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto
