Re: [gentoo-user] Re: Switched from rsync to git, now emerge --sync fails
On 14/10/2021 09:28, Miles Malone wrote: So the default being for git to act like git, instead of git to act like an alternative to rsync, makes perfect sense in that context. And once it's sync'd the first time, I think sync depth is ignored. It normally doesn't make much difference anyway, because git stores the current version in full, and then just diffs going backwards. The difference between a sync=1 and sync=0 isn't that much for linux I gather - maybe doubles the size? For a normal user who just wants to USE portage, a sync of 1 makes sense. They can then always go back as far as when they first set it up, anyway. Cheers, Wol
Re: [gentoo-user] Re: Switched from rsync to git, now emerge --sync fails
Look I mean in any other context for a git repository sync-depth=0 would be the obvious choice, why have a version control system if you're limiting the version control. It was added to gentoo so that developers could work on their own branches of the git repo whilst still having it actually work with portage. And then the next logical application is making your own /usr/local/portage not suck. Using git as the main sync instead of rsync is something that's come about way later, and really is fairly niche, compared to what the git capability in portage was originally designed for. So the default being for git to act like git, instead of git to act like an alternative to rsync, makes perfect sense in that context. On Thu, 14 Oct 2021 at 18:22, Peter Humphrey wrote: > > On Thursday, 14 October 2021 03:50:59 BST Grant Edwards wrote: > > On 2021-10-13, Rich Freeman wrote: > > > On Wed, Oct 13, 2021 at 2:50 PM Grant Edwards > wrote: > > >> Is there some reason it should default > > >> to doing unlimited depth fetch operations? > > > > > > If all you want is a repo, no reason to set the depth higher. > > > > Then a default of 1 seems like the obvious "right" answer. > > Unfortunately, it defaults to "unlimited" according to > > https://wiki.gentoo.org/wiki//etc/portage/repos.conf > > > > sync-depth > > > > Specifies sync depth to use for DVCS repositories. If set to 0, > > the depth is unlimited. Defaults to 0. > > > > > If you want to see the history then you'll want it all. > > > > Apparently, that's the default. Without any sync-depth setting the > > fetch was stalling and then timing out after 60s. With sync-depth=1, > > the fetch completed in about 1s. > > I have it set to 8 here, for no reason I can remember. Is there a disadvantage > in that? > > -- > Regards, > Peter. > > > >
Re: [gentoo-user] Re: Switched from rsync to git, now emerge --sync fails
On Thursday, 14 October 2021 03:50:59 BST Grant Edwards wrote: > On 2021-10-13, Rich Freeman wrote: > > On Wed, Oct 13, 2021 at 2:50 PM Grant Edwards wrote: > >> Is there some reason it should default > >> to doing unlimited depth fetch operations? > > > > If all you want is a repo, no reason to set the depth higher. > > Then a default of 1 seems like the obvious "right" answer. > Unfortunately, it defaults to "unlimited" according to > https://wiki.gentoo.org/wiki//etc/portage/repos.conf > > sync-depth > > Specifies sync depth to use for DVCS repositories. If set to 0, > the depth is unlimited. Defaults to 0. > > > If you want to see the history then you'll want it all. > > Apparently, that's the default. Without any sync-depth setting the > fetch was stalling and then timing out after 60s. With sync-depth=1, > the fetch completed in about 1s. I have it set to 8 here, for no reason I can remember. Is there a disadvantage in that? -- Regards, Peter.
[gentoo-user] Re: Switched from rsync to git, now emerge --sync fails
On 2021-10-13, Rich Freeman wrote: > On Wed, Oct 13, 2021 at 2:50 PM Grant Edwards > wrote: >> >> Is there some reason it should default >> to doing unlimited depth fetch operations? >> > > If all you want is a repo, no reason to set the depth higher. Then a default of 1 seems like the obvious "right" answer. Unfortunately, it defaults to "unlimited" according to https://wiki.gentoo.org/wiki//etc/portage/repos.conf sync-depth Specifies sync depth to use for DVCS repositories. If set to 0, the depth is unlimited. Defaults to 0. > If you want to see the history then you'll want it all. Apparently, that's the default. Without any sync-depth setting the fetch was stalling and then timing out after 60s. With sync-depth=1, the fetch completed in about 1s. -- Grant
Re: [gentoo-user] Re: Switched from rsync to git, now emerge --sync fails
On Wed, Oct 13, 2021 at 2:50 PM Grant Edwards wrote: > > Is there some reason it should default > to doing unlimited depth fetch operations? > If all you want is a repo, no reason to set the depth higher. If you want to see the history then you'll want it all. However, once you have an initial sync, I don't think it should go back and fetch all the history unless you explicitly ask git to do so. I don't see why this would cause issues after the initial sync. If you were fetching all the history, it would be the FIRST sync that caused all the issues. Well, unless portage is going out and trying to pull it all in (and if so I'd think it would have done it from the start). Once you have the full repo then subsequent syncs should be very fast and don't use much CPU server-side. The git client sends the remote server its current head, and then the server walks back from its head until it finds yours, which will only be a short distance if you've synced recently. Then it is only the new objects in-between that have to be sent. The whole thing is de-duplicated and copy-on-write just due to its data structure. I'm suspecting some sort of server-side issue - maybe an intermittent one. Either that or portage is really trying to pull in that history after the initial sync. Another option is to do a pull from the github mirror. That same repo is hosted on both gentoo's server and github, and they're identical (the content hash tells you as much), so you should be able to do a pull from either seamlessly. The signatures/etc are applied to both as well. Some don't care for github not being FOSS but if you're just using it as a mirror I'd argue it is no different than if one of the gazillion distfile mirrors happened to run IIS or have a firmware that wasn't coreboot. It is just another mirror. -- Rich
Re: [gentoo-user] Re: Switched from rsync to git, now emerge --sync fails
On Wed, 13 Oct 2021 18:50:10 - (UTC), Grant Edwards wrote: > > emerge --sync works here. What do you have in repos.conf? > > I didn't have a sync-depth setting. Setting that to 1 fixed the > problem. It was apparently timing out because it takes too long for > the server to respond to a fetch with an unlimited depth. Seems like > maybe it ought to default to 1? Is there some reason it should default > to doing unlimited depth fetch operations? I don't have that either. -- Neil Bothwick IRQs? We don't need no stinking IRQs! pgpnsa5ECG93U.pgp Description: OpenPGP digital signature
[gentoo-user] Re: Switched from rsync to git, now emerge --sync fails
On 2021-10-13, Neil Bothwick wrote: > On Wed, 13 Oct 2021 18:25:12 - (UTC), Grant Edwards wrote: > >> Yesterday I switched my gentoo repo from rsync to git, and the initial >> --sync with an empty directory did a git clone successfully. >> >> Today, when I try to sync, it always fails: >> >> $ sudo emerge --sync >>> Syncing repository 'gentoo' into >> '/var/db/repos/gentoo'... /usr/bin/git fetch origin >> error: RPC failed; HTTP 504 curl 22 The requested URL returned error: >> 504 fatal: the remote end hung up unexpectedly [...] >> After the initial sync which does a git clone, how do you do >> subsequent sync operations? > > emerge --sync works here. What do you have in repos.conf? I didn't have a sync-depth setting. Setting that to 1 fixed the problem. It was apparently timing out because it takes too long for the server to respond to a fetch with an unlimited depth. Seems like maybe it ought to default to 1? Is there some reason it should default to doing unlimited depth fetch operations? -- Grant
