Re: What do folks use to mirror repositories
On Tue, 19 Apr 2022 10:57:38 -0600 Charles Curley wrote: > On Tue, 19 Apr 2022 11:02:26 -0500 > David Wright wrote: > > > As for laptops, > > I handle laptops a bit differently (that's Linux for you). My proxy > statement is in its own file in /etc/apt. When the laptop is home, > there's a symlink in /etc/apt/apt.conf.d. The symlink gets removed or > made by a script NetworkManager executes at various times: > /etc/NetworkManager/dispatcher.d/50ifupdown > > Non-laptops get the same file and symlink. The just don't get > NetworkManager or that script. I just bring up a wireguard tunnel to my LAN before updating / upgrading. -- Celejar
Re: What do folks use to mirror repositories
On Mon, 18 Apr 2022, Celejar wrote: On Fri, 15 Apr 2022 22:15:36 -0600 Charles Curley wrote: ... Apt-cacher-ng (hereafter acng) also requires a change in client apt configurations. Put one line into apt.conf or a one-liner in apt.conf.d. I use the latter, 02proxy: Acquire::http::Proxy "http://aptcacherdeb.localdomain:3142";; There are further wrinkles for laptops and other traveling computers, and for https only repos, which I will leave as an exercise for the student. Yes. I use apt-cacher-ng, but having to manually add a workaround for every SSL-only repository I use is getting rather annoying: https://wiki.debian.org/AptCacherNg#HTTPS_repositories https://blog.packagecloud.io/using-apt-cacher-ng-with-ssl-tls/ One ends up with an ugly and not-very readable line like this: PassThroughPattern: (winehq\.org|xpra\.org|vscodium\.com|packagecloud\.io):443$ I haven't really thought about it but could you point at a 'fake' url that then gets proxied to the real one. My thought would be to use: http://http.ssl.only.repo/ in sources.list, point that name at a local apache server and then forward to https://ssl.only.repo/ I only suggest apache because I've done something like this before - I'd guess there are much leaner solutions, even stunnel might work. depending on name resolution it might have to be a suffix rather than a prefix. Tim.
Re: What do folks use to mirror repositories
I've tried many... apt-cache debmirror ftpsync I prefer debmirror You can also use aptly but it will be a non original mirror. On 2022-04-19 12:57, Charles Curley wrote: > On Tue, 19 Apr 2022 11:02:26 -0500 > David Wright wrote: > >> As for laptops, > > I handle laptops a bit differently (that's Linux for you). My proxy > statement is in its own file in /etc/apt. When the laptop is home, > there's a symlink in /etc/apt/apt.conf.d. The symlink gets removed or > made by a script NetworkManager executes at various times: > /etc/NetworkManager/dispatcher.d/50ifupdown > > Non-laptops get the same file and symlink. The just don't get > NetworkManager or that script. > -- Polyna-Maude R.-Summerside -Be smart, Be wise, Support opensource development
Re: What do folks use to mirror repositories
On Tue, 19 Apr 2022 11:02:26 -0500 David Wright wrote: > As for laptops, I handle laptops a bit differently (that's Linux for you). My proxy statement is in its own file in /etc/apt. When the laptop is home, there's a symlink in /etc/apt/apt.conf.d. The symlink gets removed or made by a script NetworkManager executes at various times: /etc/NetworkManager/dispatcher.d/50ifupdown Non-laptops get the same file and symlink. The just don't get NetworkManager or that script. -- Does anybody read signatures any more? https://charlescurley.com https://charlescurley.com/blog/
Re: What do folks use to mirror repositories
On Tue 19 Apr 2022 at 07:24:53 (+0200), to...@tuxteam.de wrote: > On Mon, Apr 18, 2022 at 09:08:08PM -0400, Celejar wrote: > > On Fri, 15 Apr 2022 22:15:36 -0600 > > Charles Curley wrote: > > > > ... > > > > > Apt-cacher-ng (hereafter acng) also requires a change in client apt > > > configurations. Put one line into apt.conf or a one-liner in > > > apt.conf.d. I use the latter, 02proxy: > > > > > > Acquire::http::Proxy "http://aptcacherdeb.localdomain:3142";; > > > > > > There are further wrinkles for laptops and other traveling computers, > > > and for https only repos, which I will leave as an exercise for the > > > student. > > > > Yes. I use apt-cacher-ng, but having to manually add a workaround for > > every SSL-only repository I use is getting rather annoying: > > That's true. Caching and HTTPS don't play well with each other. Why > a package repository has to be https-only is anyone's guess, but I > think apt caches should learn to do "man in the middle". As for laptops, I iron out one wrinkle by not setting my proxy under /etc/apt/, but in the command lines instead. So the cron command has: apt-get -qq -o Acquire::http::Proxy="…" update && apt-get -qq -d -o Acquire::http::Proxy="…" dist-upgrade && find /var/cache/apt/archives/ -name '*deb' which generates an email whenever there are packages in the cache, and I upgrade the system with: apt-get -o Acquire::http::Proxy="…" update && apt-get -d -o Acquire::http::Proxy="…" dist-upgrade; apt-get upgrade; read -p 'Ctrl-C to avoid clean' _; apt-get clean (I avoid cleaning the cache when packages are held back so that an immediate apt-get dist-upgrade doesn't have to refetch them.) Apart from making it easy to run the root command when not at home (just rubout the -o option), this command line also works around apt-cacher-ng's occasional inability to cope with wrinkles such as expired certificates: when apt-listbugs runs during the upgrade, it avoids going through the cache. Cheers, David.
Re: What do folks use to mirror repositories
On Mon, Apr 18, 2022 at 09:08:08PM -0400, Celejar wrote: > On Fri, 15 Apr 2022 22:15:36 -0600 > Charles Curley wrote: > > ... > > > Apt-cacher-ng (hereafter acng) also requires a change in client apt > > configurations. Put one line into apt.conf or a one-liner in > > apt.conf.d. I use the latter, 02proxy: > > > > Acquire::http::Proxy "http://aptcacherdeb.localdomain:3142";; > > > > There are further wrinkles for laptops and other traveling computers, > > and for https only repos, which I will leave as an exercise for the > > student. > > Yes. I use apt-cacher-ng, but having to manually add a workaround for > every SSL-only repository I use is getting rather annoying: That's true. Caching and HTTPS don't play well with each other. Why a package repository has to be https-only is anyone's guess, but I think apt caches should learn to do "man in the middle". Cheers -- t signature.asc Description: PGP signature
Re: What do folks use to mirror repositories
On Mon, 18 Apr 2022 21:08:08 -0400 Celejar wrote: > Yes. I use apt-cacher-ng, but having to manually add a workaround for > every SSL-only repository I use is getting rather annoying: > > https://wiki.debian.org/AptCacherNg#HTTPS_repositories > https://blog.packagecloud.io/using-apt-cacher-ng-with-ssl-tls/ Yup. Or something like: Acquire::https::Proxy::updates.example.com "DIRECT"; to bypass acng entirely. Either one is kludgey. -- Does anybody read signatures any more? https://charlescurley.com https://charlescurley.com/blog/
Re: What do folks use to mirror repositories
On Fri, 15 Apr 2022 22:15:36 -0600 Charles Curley wrote: ... > Apt-cacher-ng (hereafter acng) also requires a change in client apt > configurations. Put one line into apt.conf or a one-liner in > apt.conf.d. I use the latter, 02proxy: > > Acquire::http::Proxy "http://aptcacherdeb.localdomain:3142";; > > There are further wrinkles for laptops and other traveling computers, > and for https only repos, which I will leave as an exercise for the > student. Yes. I use apt-cacher-ng, but having to manually add a workaround for every SSL-only repository I use is getting rather annoying: https://wiki.debian.org/AptCacherNg#HTTPS_repositories https://blog.packagecloud.io/using-apt-cacher-ng-with-ssl-tls/ One ends up with an ugly and not-very readable line like this: PassThroughPattern: (winehq\.org|xpra\.org|vscodium\.com|packagecloud\.io):443$ -- Celejar
Re: What do folks use to mirror repositories
Am Fri, 15 Apr 2022 07:03:14 -0400 schrieb Sam : >That's pretty much it. I want to mirror all my updates to a single server on >my LAN and have everything on my LAN apt update from it. This seems more >efficient than having everyone download their own copies. For mirroring a repository I recommend the package 'debmirror'. Instead of mirroring a whole repository (e.g. for mirroring Debian11 you need about 70 GB) you can select a subset of the repository with the include and exclude options. Dieter
Re: What do folks use to mirror repositories
On Fri, 15 Apr 2022 19:06:35 -0700 David Christensen wrote: > When I had several Debian machines in the past, I used approx(8) [2]. > It was good software -- KISS, properly documented, easy to set up, > worked efficiently, and never failed. > > > But approx(8) had some operational features that apt-cacher-ng(8) [3] > seems to improve upon: > > 1. Each host needed its apt(8) [1] configuration to point at the > approx(8) server. Apt-cacher-ng (hereafter acng) also requires a change in client apt configurations. Put one line into apt.conf or a one-liner in apt.conf.d. I use the latter, 02proxy: Acquire::http::Proxy "http://aptcacherdeb.localdomain:3142";; There are further wrinkles for laptops and other traveling computers, and for https only repos, which I will leave as an exercise for the student. > > 3. I seem to recall that I was able to use approx(8) with non-Debian > repositories via creative configuration. acng does support some non-deb repos. For more details, see /etc/apt-cacher-ng/acng.conf. I think I'll stay with acng. -- Does anybody read signatures any more? https://charlescurley.com https://charlescurley.com/blog/
Re: What do folks use to mirror repositories
On 4/15/22 04:36, to...@tuxteam.de wrote: On Fri, Apr 15, 2022 at 07:03:14AM -0400, Sam wrote: That's pretty much it. I want to mirror all my updates to a single server on my LAN and have everything on my LAN apt update from it. This seems more efficient than having everyone download their own copies. If all you are looking for is efficiency, you might also consider an apt cache (I'm pretty happy with apt-cacher-ng). It is pretty low maintenance, as it decides itself when to throw out older entries. The big plus for me is that it can cache across multiple repos (I sometimes build "old" images from archives.d.o for some legacy hardware hidden away in some customer's closets, don't ask ;-) It just does so silently. Whenever the package isn't there, it fetches it, next time it's served from the cache. Another nice point is that you can just keep your /etc/apt/sources.list as it should be (with the "real" repo addresses out there) and the apt cache works as a proxy. An entry in (e.g.) /etc/apt/apt.conf.d/02cache like so: # /etc/apt/apt.conf.d/02cache # Acquire::http::proxy "http://localhost:3142"; (or whatever your apt cache's URL is) suffices -- something you can easily disable or remove should you ever "release" your pet machine "into the wild" :-) Google told me to use apt-mirror [...] A mirror has different characteristics from a cache. You decide yourself which packages go in. This may be an advantage, depending on your use case -- or a disadvantage. When I had several Debian machines in the past, I used approx(8) [2]. It was good software -- KISS, properly documented, easy to set up, worked efficiently, and never failed. But approx(8) had some operational features that apt-cacher-ng(8) [3] seems to improve upon: 1. Each host needed its apt(8) [1] configuration to point at the approx(8) server. 2. The approx(8) server needed its configuration to point at the remote repositories. 3. I seem to recall that I was able to use approx(8) with non-Debian repositories via creative configuration. 4. The approx(8) cache needed to be thinned periodically by running approx-gc(8) [2]. David References: [1] https://manpages.debian.org/buster/apt/apt.8.en.html [2] https://manpages.debian.org/stretch/approx/approx.8.en.html [3] https://manpages.debian.org/unstable/apt-cacher-ng/apt-cacher-ng.8.en.html
Re: What do folks use to mirror repositories
On Fri 15 Apr 2022 at 07:03:14 (-0400), Sam wrote: > That's pretty much it. I want to mirror all my updates to a single > server on my LAN and have everything on my LAN apt update from it. > This seems more efficient than having everyone download their own > copies. Same here, except that the "server" is just my main workstation, so it tends to be first on in the morning and last off at night. > I got aptly to work, after applying a reasonable amount of elbow > grease. The only unsolvable problem was that after taking a few > seconds to download a handful of updates it takes aptly well over an > hour to republish the local repository snapshot, to incorporate those > updates. The full repository adds up to about 18000 packages, and > looks like aptly recompresses each one of them, every time. I don't > know why but that's what it does. I see it launching xz over and over > again, while aptly keeps me in the loop of how many of those 18000 > packages are left to process. > > Although I don't need to worry about wearing any SSDs (since it's > spinning rust), this is still somewhat suboptimal. So, that brings me > to: what do folks use to mirror repositories? Like some others, I've run apt-cacher-ng for nearly 10 years. It is caching two Debian versions for two architectures at any one time, except when Debian releases a new version, when a third version will be added before the oldest starts expiring, so it swells and then shrinks again. It pretty much runs itself, apart from a couple of odd wrinkles that are of my own making. Pros: it builds up the cache itself, as packages get installed. It also expires packages itself, so you don't have to maintain it. (The expiry parameters can be configured if required.) At the moment, its size is 12GB, and it contains 6906 .deb files. I see that I posted it occupied 13GB in 2018. (I don't run any Desktop Environments.) Cons: I've only met these when I've continued running apt-cacher-ng on an oldstable machine for a lengthy period, so there's no need for you to hit these snags at all. 1. Back when I was running apt-cacher-ng on wheezy, changes to the Debian repository formats/methods in jessie prevented it from expiring packages, so the cache kept growing. Answer: I installed a wheezy-backports version of apt-cacher-ng until I upgraded the machine to jessie. 2. Currently, if you run apt-cacher-ng on buster, /and/ you have installed apt-listbugs on your machines, you may have to say no when it asks whether to check for listed bugs. Or, you can update the lists and upgrade download-only (which gets the package(s) through the cache as desired), but perform the actual upgrade without going through apt-cacher-ng (which costs no bandwidth as the .deb files are on the local machine now). But, as I said, there's no need to hit this snag if you run apt-cacher-ng on a machine running stable (or testing/unstable etc, I guess). Cheers, David.
Re: What do folks use to mirror repositories
On Fri, Apr 15, 2022 at 6:10 AM Sam wrote: > That's pretty much it. I want to mirror all my updates to a single server > on > my LAN and have everything on my LAN apt update from it. This seems more > efficient than having everyone download their own copies. > > Google told me to use apt-mirror. As Andrew mentions in his reply, rsync is a good choice for this. Or something based on it. It's smarter about what doesn't need to be copied so uses less aggregate bandwidth. Lots of use-cases around the data center :-)
Re: What do folks use to mirror repositories
On Fri, Apr 15, 2022 at 07:03:14AM -0400, Sam wrote: > That's pretty much it. I want to mirror all my updates to a single server on > my LAN and have everything on my LAN apt update from it. This seems more > efficient than having everyone download their own copies. > > Google told me to use apt-mirror. My updates repository turned out to have > some metadata filename with an @ in it, apt-mirror refuses to do anything > with this state of affairs. I learned (after poking apt-mirror's page( that > this bug was two years old, but the project is not maintained any more. > Digging through aptly-mirror's bug tracker I caught a reference to aptly. > It depends: if you've got a lot of machines, you could do worse than ftpsync - which, despite it's name, uses mostly rsync - and Apache. https://www.debian.org/mirror/ftpmirror.el.html tells you to use ftpsync or debmirror See, for example, my blog on Planet Debian from a while ago for fairly detailed instructions: http://flosslinuxblog.blogspot.com/2020/02/rebuilding-mirror-software-mirroring-of.html [ftpsync setup] and http://flosslinuxblog.blogspot.com/2020/02/rebuilding-mirror-fedora-epel-mirroring.html [which details the Apache setup] I don't exclude architectures or distributions - so I have a full Debian mirror with everything on (apart from anything served only by security.d.o). If you _only_ need amd64 and _only_ need the current distribution, for example, your mirror will be very significantly smaller. > So, that brings me to: what do > folks use to mirror repositories? > Hope this helps. With every good wish, as ever, Andy Cater
Re: What do folks use to mirror repositories
On Fri, Apr 15, 2022 at 01:36:43PM +0200, to...@tuxteam.de wrote: > On Fri, Apr 15, 2022 at 07:03:14AM -0400, Sam wrote: > > That's pretty much it. I want to mirror all my updates to a single server on > > my LAN and have everything on my LAN apt update from it. This seems more > > efficient than having everyone download their own copies. > > If all you are looking for is efficiency, you might also consider an > apt cache (I'm pretty happy with apt-cacher-ng). It is pretty low > maintenance, as it decides itself when to throw out older entries. > > The big plus for me is that it can cache across multiple repos (I > sometimes build "old" images from archives.d.o for some legacy > hardware hidden away in some customer's closets, don't ask ;-) > It just does so silently. Whenever the package isn't there, it > fetches it, next time it's served from the cache. > > Another nice point is that you can just keep your /etc/apt/sources.list > as it should be (with the "real" repo addresses out there) and the > apt cache works as a proxy. An entry in (e.g.) /etc/apt/apt.conf.d/02cache > like so: > > # /etc/apt/apt.conf.d/02cache > # > Acquire::http::proxy "http://localhost:3142"; > > (or whatever your apt cache's URL is) suffices -- something you can > easily disable or remove should you ever "release" your pet machine > "into the wild" :-) > I very much agree with Tomas. After having tried a few different solutions years ago, I have settled on this one and I am convinced it is still the best available solution to this particular problem. Regards, -Roberto -- Roberto C. Sánchez
Re: What do folks use to mirror repositories
On Fri, Apr 15, 2022 at 07:03:14AM -0400, Sam wrote: > That's pretty much it. I want to mirror all my updates to a single server on > my LAN and have everything on my LAN apt update from it. This seems more > efficient than having everyone download their own copies. If all you are looking for is efficiency, you might also consider an apt cache (I'm pretty happy with apt-cacher-ng). It is pretty low maintenance, as it decides itself when to throw out older entries. The big plus for me is that it can cache across multiple repos (I sometimes build "old" images from archives.d.o for some legacy hardware hidden away in some customer's closets, don't ask ;-) It just does so silently. Whenever the package isn't there, it fetches it, next time it's served from the cache. Another nice point is that you can just keep your /etc/apt/sources.list as it should be (with the "real" repo addresses out there) and the apt cache works as a proxy. An entry in (e.g.) /etc/apt/apt.conf.d/02cache like so: # /etc/apt/apt.conf.d/02cache # Acquire::http::proxy "http://localhost:3142"; (or whatever your apt cache's URL is) suffices -- something you can easily disable or remove should you ever "release" your pet machine "into the wild" :-) > Google told me to use apt-mirror [...] A mirror has different characteristics from a cache. You decide yourself which packages go in. This may be an advantage, depending on your use case -- or a disadvantage. Cheers -- t signature.asc Description: PGP signature
What do folks use to mirror repositories
That's pretty much it. I want to mirror all my updates to a single server on my LAN and have everything on my LAN apt update from it. This seems more efficient than having everyone download their own copies. Google told me to use apt-mirror. My updates repository turned out to have some metadata filename with an @ in it, apt-mirror refuses to do anything with this state of affairs. I learned (after poking apt-mirror's page( that this bug was two years old, but the project is not maintained any more. Digging through aptly-mirror's bug tracker I caught a reference to aptly. I got aptly to work, after applying a reasonable amount of elbow grease. The only unsolvable problem was that after taking a few seconds to download a handful of updates it takes aptly well over an hour to republish the local repository snapshot, to incorporate those updates. The full repository adds up to about 18000 packages, and looks like aptly recompresses each one of them, every time. I don't know why but that's what it does. I see it launching xz over and over again, while aptly keeps me in the loop of how many of those 18000 packages are left to process. Although I don't need to worry about wearing any SSDs (since it's spinning rust), this is still somewhat suboptimal. So, that brings me to: what do folks use to mirror repositories? pgpqL38pnY_vQ.pgp Description: PGP signature
