On 30/03/2022 11:42:46+0100, Richard Purdie wrote: > [list address fixed, sorry] > > We've been having bandwidth trouble with downloads.yoctoproject.org so we did > some quick analysis to see what the issue is. Basically in speeding up the > server which was the rate limit, we hit the limits of the hosting pipe. I'd > note > a few things: > > a) it isn't the sstate mirroring, it is nearly all being used by downloads. > > b) 25% of all our bandwidth is going on "git2_sourceware.org.git.binutils- > gdb.git.tar.gz" - i.e. downloading the source mirror binutils tarball > > c) 15% is on git2_sourceware.org.git.glibc.git.tar.gz i.e. glibc > > d) OE-Core has downloads.yoctoproject.org as a MIRROR > > e) poky has it as a PREMIRROR > > What are our options? As far as I can see we could: > > a) increase the pipe from downloads.yoctoproject.org but that does come at a > non-trivial cost to the project. > > b) Seek help with hosting some of the larger mirror tarballs from people > better > able to host them and have that as a first premirror? > > c) Switch the binutils and glibc recipes to tarballs and patches. I know Khem > finds this less convenient and they keep moving back and forward but we keep > running into this issue and having to switch back from git. > > d) To soften the blow of c) we could add devupstream support to the recipes? > We > could script updating the recipe to add the patches? > > e) We could drop the PREMIRRORS from poky. This would stop the SCM targets > from > hitting our mirrors first. That does transfer load to the upstream project > SCMs > though and I'm not sure that will be appreciated. I did sent that patch, I'm > not > sure about it though.
I would simply drop PREMIRRORS, this is actually a privacy concern for some of our customers that didn't realize they are leaking the names of their internal git repositories to downloads.yoctoproject.org. -- Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#56599): https://lists.yoctoproject.org/g/yocto/message/56599 Mute This Topic: https://lists.yoctoproject.org/mt/90128469/21656 Group Owner: yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
