Hi Fabian,

On Thu, Feb 22, 2018 at 08:24:25AM +0100, Fabian Groffen wrote:

> Please excuse my snipping.

I was rambling, wasn't I. :) I'll snip some more for now.

> > > hashgen currently runs in 30s or so on the tree to generate
> > > manifests.                                                                
> > >    
I recompiled gcc overnight and got the following numbers today on one
of the ARM SBCs:

[root@linux:~] time ./hashgen /usr/portage/
real    7m28.071s
user    10m8.355s
sys     0m57.435s
[root@linux:~] time ./hashgen /usr/portage/
real    7m6.195s
user    9m54.715s
sys     0m54.123s

[root@linux:~] time gemato create -K 
/var/lib/gentoo/gkeys/keyrings/gentoo/release/pubring.gpg -f -S -t -H SHA512\ 
BLAKE2B  /usr/portage
INFO:root:Creating Manifests in /usr/portage...
INFO:root:/usr/portage updated in 599.46 seconds

real    10m5.431s
user    8m54.606s
sys     0m59.072s

So there is a 30% speedup at least.

> Odd, was this rsync1 or rsync2?
> path.  Instead rsync1 and rsync2 are doing full generation both of them.
> To make this be equal, I had to play a trick with timestamps, I
> basically set the timestamp to the git commit time.  Maybe this plays a
> role too?

I just retried: timestamp.chk seemed to be in sync between rsync1 and
rsync2 at first: A sync from rsync1 syncs some files and later syncs
even against rsync2 find timestamp.chk matching.

When I delete timestamp.chk and retry several times until the other
server round-robins in, emaint sync rsyncs what seems like all
second-level Manifest.gzs (net-fs, dev-libs, etc.) and what seems like
the whole metadata/md5-cache directory.

When having synced from rsync2, gemato verify failed on
dev-cpp/Manifest.gz. When having synced from rsync1, it failed on

After waiting for the next refresh, emaint synced selective 2nd level
manifests and md5-cache from rsync2 again. It still failed verification
on dev-cpp/Manifest.gz.

On rsync1, timestamp.chk still differed after that first sync and emaint
sync again synced selective 2nd and 3rd level manifests and again what
seemed like the whole md5-cache. Verification with gemato still failed
on net-fs/Manifest.gz.

So the hash discrepancies seem to survive regeneration.

Then I deleted the whole $PORTDIR and sync from scratch. This was
against rsync1. Verification still fails on net-fs/Manifest.gz.

Deleting timestamp.chk again and running against rsync2, it behaved as
before and again failed at dev-cpp/Manifest.gz.

$ gemato  verify -K /Users/michael/b/pubring.gpg -R -j1
INFO:root:Manifest timestamp: 2018-02-22 19:26:44 UTC
INFO:root:Valid OpenPGP signature found:
INFO:root:- primary key: 0204A8ABD003E57A9558850DBA08091EC6317B3C
INFO:root:- subkey: 0204A8ABD003E57A9558850DBA08091EC6317B3C
INFO:root:- timestamp: 2018-02-22 19:26:44 UTC
INFO:root:Verifying /usr/local/gentoo/usr/portage/...
ERROR:root:Manifest mismatch for dev-cpp/Manifest.gz
  BLAKE2B: expected:
  SHA512: expected:

Can you perhaps confirm these hash discrepancies on the servers?

> I only checked rsync1 last time, maybe rsync2 isn't as equal as I
> thought.

They both individually seem to produce hash discrepancies on different

Reply via email to