Hi Fabian, On Thu, Feb 22, 2018 at 08:24:25AM +0100, Fabian Groffen wrote:
> Please excuse my snipping. I was rambling, wasn't I. :) I'll snip some more for now. > > > hashgen currently runs in 30s or so on the tree to generate > > > manifests. > > > I recompiled gcc overnight and got the following numbers today on one of the ARM SBCs: [root@linux:~] time ./hashgen /usr/portage/ real 7m28.071s user 10m8.355s sys 0m57.435s [root@linux:~] time ./hashgen /usr/portage/ real 7m6.195s user 9m54.715s sys 0m54.123s [root@linux:~] time gemato create -K /var/lib/gentoo/gkeys/keyrings/gentoo/release/pubring.gpg -f -S -t -H SHA512\ BLAKE2B /usr/portage INFO:root:Creating Manifests in /usr/portage... INFO:root:/usr/portage updated in 599.46 seconds real 10m5.431s user 8m54.606s sys 0m59.072s So there is a 30% speedup at least. > Odd, was this rsync1 or rsync2? [...] > path. Instead rsync1 and rsync2 are doing full generation both of them. > To make this be equal, I had to play a trick with timestamps, I > basically set the timestamp to the git commit time. Maybe this plays a > role too? I just retried: timestamp.chk seemed to be in sync between rsync1 and rsync2 at first: A sync from rsync1 syncs some files and later syncs even against rsync2 find timestamp.chk matching. When I delete timestamp.chk and retry several times until the other server round-robins in, emaint sync rsyncs what seems like all second-level Manifest.gzs (net-fs, dev-libs, etc.) and what seems like the whole metadata/md5-cache directory. When having synced from rsync2, gemato verify failed on dev-cpp/Manifest.gz. When having synced from rsync1, it failed on net-fs/Manifest.gz. After waiting for the next refresh, emaint synced selective 2nd level manifests and md5-cache from rsync2 again. It still failed verification on dev-cpp/Manifest.gz. On rsync1, timestamp.chk still differed after that first sync and emaint sync again synced selective 2nd and 3rd level manifests and again what seemed like the whole md5-cache. Verification with gemato still failed on net-fs/Manifest.gz. So the hash discrepancies seem to survive regeneration. Then I deleted the whole $PORTDIR and sync from scratch. This was against rsync1. Verification still fails on net-fs/Manifest.gz. Deleting timestamp.chk again and running against rsync2, it behaved as before and again failed at dev-cpp/Manifest.gz. $ gemato verify -K /Users/michael/b/pubring.gpg -R -j1 /usr/local/gentoo/usr/portage/ INFO:root:Manifest timestamp: 2018-02-22 19:26:44 UTC INFO:root:Valid OpenPGP signature found: INFO:root:- primary key: 0204A8ABD003E57A9558850DBA08091EC6317B3C INFO:root:- subkey: 0204A8ABD003E57A9558850DBA08091EC6317B3C INFO:root:- timestamp: 2018-02-22 19:26:44 UTC INFO:root:Verifying /usr/local/gentoo/usr/portage/... ERROR:root:Manifest mismatch for dev-cpp/Manifest.gz BLAKE2B: expected: 64d7c4bd55a14e8c7296e8185ad9654db39cfc095107411c97b5f425856f780c0b2dffcef436c07bc07c8832506943e7d80ab5eaf2923eb4bc419dea3a8d071a, have: 6b698c9af8c1bf5012ee01ea308718b2f09330a181b48e663a27977885b75bd439a64d568d59de6a2a17bcad86cedf0cd8cda28361155c382badadc0d369843d SHA512: expected: a7d12f2653817a47cc76de6850f8a9ab22bb952f2df1d1029cb23805f868b1d6610a2bc35d1f13666890ed1c9648907e25e949c78f75e2318065b400872e719f, have: 070bb46740c8ecc565d23dcc35cedc3bb96d6014b083da8428be00e4008ef2e2d882d3b8e8047fe84e16542b090d02c881c08ce60b052de76406f65caf6dd893 Can you perhaps confirm these hash discrepancies on the servers? > I only checked rsync1 last time, maybe rsync2 isn't as equal as I > thought. They both individually seem to produce hash discrepancies on different files. -- Thanks, Michael
