Re: [gentoo-alt] gemato in prefix

2018-03-05 Thread Fabian Groffen
On 05-03-2018 18:45:00 +0100, Michael Weiser wrote: > > > gemato is down to 24 seconds for verification, hashverify now takes 13 > > > seconds. > > Darn, and how many cpu cores does it use throughout? Just two? > > I think you can relax. :) After upgrade from 10.3 to 11.2 it now runs > more than

Re: [gentoo-alt] gemato in prefix

2018-03-05 Thread Michael Weiser
Hi Fabian, On Sun, Mar 04, 2018 at 09:20:31PM +0100, Fabian Groffen wrote: > > Were you able to pin down the actual cause of the verification failures? > Yes. > (BRACE BRACE) Now that's what I call a tough nut to crack. Awesome work to sort it out. > > gemato is down to 24 seconds for verificat

Re: [gentoo-alt] gemato in prefix

2018-03-04 Thread Fabian Groffen
On 04-03-2018 17:58:57 +0100, Michael Weiser wrote: > Hi Fabian, > > On Sun, Mar 04, 2018 at 10:34:27AM +0100, Fabian Groffen wrote: > > > I think I finally cracked it. I basically rewrote hashgen because it > > was needlessly complex and interwoven for some reason. I just did an > > emerge --s

Re: [gentoo-alt] gemato in prefix

2018-03-04 Thread Michael Weiser
Hi Fabian, On Sun, Mar 04, 2018 at 10:34:27AM +0100, Fabian Groffen wrote: > I think I finally cracked it. I basically rewrote hashgen because it > was needlessly complex and interwoven for some reason. I just did an > emerge --sync (against rsync1) and ran hashverify after it (emerge > hashgen

Re: [gentoo-alt] gemato in prefix

2018-03-04 Thread Fabian Groffen
I think I finally cracked it. I basically rewrote hashgen because it was needlessly complex and interwoven for some reason. I just did an emerge --sync (against rsync1) and ran hashverify after it (emerge hashgen). It fails because of found excess file: srf-ip-conn-srv.pid but that's the only co

Re: [gentoo-alt] gemato in prefix

2018-03-01 Thread Michael Weiser
Hi Fabian, On Thu, Mar 01, 2018 at 02:12:16PM +0100, Fabian Groffen wrote: > > I synced at 11:56 from rsync2. > By the way. Syncing at 11:56 means you need to finish within 25 seconds Good that I put the time in there. I totally failed to connect the sync time to the regen time. But: It was on

Re: [gentoo-alt] gemato in prefix

2018-03-01 Thread Fabian Groffen
On 01-03-2018 14:12:16 +0100, Fabian Groffen wrote: > So, for your purposes of getting a consistent sync, you better sync > around 50 and 20 (assuming you can get the sync to finish within 6 > minutes). > > > I get: > > > > # time /usr/local/gentoo/usr/portage/scripts/rsync-generation/hashgen >

Re: [gentoo-alt] gemato in prefix

2018-03-01 Thread Fabian Groffen
On 01-03-2018 12:13:14 +0100, Michael Weiser wrote: > Hi Fabian, > > On Thu, Mar 01, 2018 at 08:46:26AM +0100, Fabian Groffen wrote: > > > I ran a --sync on my macbook this morning: > > I synced at 11:56 from rsync2. By the way. Syncing at 11:56 means you need to finish within 25 seconds or so

Re: [gentoo-alt] gemato in prefix

2018-03-01 Thread Michael Weiser
Hi Fabian, On Thu, Mar 01, 2018 at 08:46:26AM +0100, Fabian Groffen wrote: > I ran a --sync on my macbook this morning: I synced at 11:56 from rsync2. > [febe:~/Gentoo-10.10/usr/portage] fabian% cd scripts/rsync-generation > [febe:portage/scripts/rsync-generation] fabian% clang -o hashgen > -fo

Re: [gentoo-alt] gemato in prefix

2018-03-01 Thread Fabian Groffen
On 28-02-2018 12:34:31 -0600, R0b0t1 wrote: > Can you not use webrsync-gpg for the time being? I'm affraid not, we do "sign" the snapshots, but they are just tarred up versions of the rsync tree as generated. The same tree we're talking about here. > Incremental updates of authenticated files wo

Re: [gentoo-alt] gemato in prefix

2018-02-28 Thread Fabian Groffen
I ran a --sync on my macbook this morning: [febe:~/Gentoo-10.10/usr/portage] fabian% cd scripts/rsync-generation [febe:portage/scripts/rsync-generation] fabian% clang -o hashgen -fopenmp -Wall -Werror -O3 -pipe -lssl -lcrypto -lb2 -lz `gpgme-config --libs` hashgen.c [febe:portage/scripts/rsync-gen

Re: [gentoo-alt] gemato in prefix

2018-02-28 Thread Fabian Groffen
Ok, the SphinxTrain you report, I had that too, and that was a bad sync. So it seems I managed to break everyone's trees, and I need to bump mtime to get this fixed. I'll see to incorporating your patch for Darwin. I've just added more parallelism, and I believe hashverify is too fast to be corr

Re: [gentoo-alt] gemato in prefix

2018-02-28 Thread Michael Weiser
Hi Fabian, On Wed, Feb 28, 2018 at 04:58:34PM +0100, Fabian Groffen wrote: > Can you give this another try now? Many Manifests were not in sync, I'm > trying to see if that is persistent or not, but I got a clean run here > now. I synced at about 19:05 CET from rsync2. gemato still fails: ERRO

Re: [gentoo-alt] gemato in prefix

2018-02-28 Thread R0b0t1
On Tue, Feb 20, 2018 at 2:24 PM, Michael Weiser wrote: > Hi Fabian, > > On Tue, Feb 20, 2018 at 08:41:57PM +0100, Fabian Groffen wrote: > >> Thing is I once believed Portage checked manifest and all, but it seems >> not to do anything any more, so my idea of things being OK may have been > > I als

Re: [gentoo-alt] gemato in prefix

2018-02-28 Thread Fabian Groffen
Can you give this another try now? Many Manifests were not in sync, I'm trying to see if that is persistent or not, but I got a clean run here now. Fabian On 20-02-2018 20:25:33 +0100, Michael Weiser wrote: > Hi Fabian, > > On Fri, Feb 02, 2018 at 09:06:34PM +0100, Fabian Groffen wrote: > > >

Re: [gentoo-alt] gemato in prefix

2018-02-21 Thread Michael Weiser
Hi Fabian, On Wed, Feb 21, 2018 at 10:04:42AM +0100, Fabian Groffen wrote: > To give you an idea; the current tree is getting its Manifests from > hashgen.c, which you can find in scripts/rsync-generation/hashgen.c. > The hashverify tool, which I'm currently working on, is basically an > addition

Re: [gentoo-alt] gemato in prefix

2018-02-21 Thread Fabian Groffen
Hi Michael, To give you an idea; the current tree is getting its Manifests from hashgen.c, which you can find in scripts/rsync-generation/hashgen.c. The hashverify tool, which I'm currently working on, is basically an addition to that file (doing argv[0] detection) to perform the verification. At

Re: [gentoo-alt] gemato in prefix

2018-02-20 Thread Michael Weiser
Hi Fabian, On Tue, Feb 20, 2018 at 08:41:57PM +0100, Fabian Groffen wrote: > Thing is I once believed Portage checked manifest and all, but it seems > not to do anything any more, so my idea of things being OK may have been I also was a bit surprised to find that portage didn't authenticate and

Re: [gentoo-alt] gemato in prefix

2018-02-20 Thread Fabian Groffen
Well, yeah, I have the feeling that until I'm done with the verification (it's a work in progress, the problem is mostly in walking the entire tree a bit efficient) I can see if what we have actually makes sense. Thing is I once believed Portage checked manifest and all, but it seems not to do any

Re: [gentoo-alt] gemato in prefix

2018-02-20 Thread Michael Weiser
Hi Fabian, On Fri, Feb 02, 2018 at 09:06:34PM +0100, Fabian Groffen wrote: > > does it make sense to look into using gemato for repo verification or is > > there a reason this cannot work currently? > It should, but I didn't get around to it. I finally got around to trying gemato on my Mac. It

Re: [gentoo-alt] gemato in prefix

2018-02-02 Thread Fabian Groffen
It should, but I didn't get around to it. Instead I want to use my own C-based tool, but I also didn't get around to getting it ready. Fabian On 02-02-2018 14:27:52 +0100, Michael Weiser wrote: > Hi, > > does it make sense to look into using gemato for repo verification or is > there a reason

[gentoo-alt] gemato in prefix

2018-02-02 Thread Michael Weiser
Hi, does it make sense to look into using gemato for repo verification or is there a reason this cannot work currently? -- Thanks, Michael