On 9 May 2017 at 07:18, Rhythmic Fistman <[email protected]> wrote: > > > Il giorno domenica 7 maggio 2017 03:38:18 UTC+10, mpl ha scritto: >> >> On 6 May 2017 at 03:05, Rhythmic Fistman <[email protected]> wrote: >> > >> > >> > Il giorno venerdì 5 maggio 2017 23:27:55 UTC+10, mpl ha scritto: >> >> >> >> On 5 May 2017 at 09:00, Rhythmic Fistman <[email protected]> wrote: >> >> > I'm in a bind with this - I ctrl-C camlistored and the kvIndexFile >> >> > became >> >> > corrupted (how do I cleanly stop it?) & re-indexing is taking days. >> >> >> >> Hmm, how come you were on kvIndexFile? I thought we had made leveldb >> >> the default everywhere (because we indeed experienced corruption >> >> problems with kv iirc). Did you specify kvIndexFile yourself on >> >> purpose? >> > >> > >> > Not on purpose, it's because I come back to camlistore every year or two >> > to >> > try and backup and deduplicate my photos, >> > so I've got old configurations lying around. >> > >> > So I switched to levelDB. On my local ssd setup >> > >> > $ time camlistored -openbrowser=false -reindex >> > >> > >> > Starting camlistored version 8f1a7df176de21d0d052c04db45b373c5c282fb1; >> > Go >> > go1.8 (darwin/amd64) >> > >> > Starting to listen on http://localhost:3179 >> > >> > Index wiped. >> > >> > Error: blobpacked storage detects non-zero packed zips, but no metadata. >> > Please re-start in recovery mode with -recovery. >> > >> > >> > >> > $ time camlistored -openbrowser=false -recovery >> > >> > Starting camlistored version 8f1a7df176de21d0d052c04db45b373c5c282fb1; >> > Go >> > go1.8 (darwin/amd64) >> > >> > Starting to listen on http://localhost:3179 >> > >> > ui: serving Closure from embedded resources >> > >> > Starting recovery of blobpacked index >> > >> > Caught panic installer handlers: error instantiating storage for prefix >> > "/bs/", type "blobpacked": Sum of all zips (1055150 bytes) does not >> > match >> > manifest's WholeSize (527575 bytes) f >> > >> > [...] BACKTRACE ELIDED >> > >> > >> > This time reindex works in ~13 minutes >> >> wait, are you saying that you got a panic with serverinit, but the >> camlistored process didn't die, and the reindexing kept going >> successfully? > > > no - -reindex failed (and exited), then -recovery failed (and exited). After > that -reindex worked.
oh, that's somewhat reassuring then. But if -recovery was not successful, you might still have "invisible" blobs. I'll look into it get back to you. >> > time camlistored -openbrowser=false -reindex >> > >> > >> > but should I be worried that the recovery failed? >> >> Not worried that something is corrupted, but worried that not all your >> blobs are reachable (because the blobpacked index is not up to date), >> yes. > > > What should I do? You could try restarting with -recovery again. >> Can you please file a bug about it, so I don't forget to try and >> reproduce/fix it? > > > Ok - not sure how. Maybe I'll back up the index, then try re-indexing. You would usually go to https://github.com/camlistore/camlistore/issues and create a new issue. But nevermind for this time, I've done it already: https://github.com/camlistore/camlistore/issues/926 >> > similar story with the larger external non-ssd. >> > >> > recovery fails, now re-indexing, looks like it might take an hour - this >> > is >> > an improvement over kvindex I think. >> > >> > Strangely, local SSD had camlistored saturating the CPU (200%?) whereas >> > external rust is sitting at 6%... >> >> yeah I think that's pretty common, hence why I mentioned using an SSD >> before. afair it's partly due to HDDs having to wait for fsync or >> something like that. > > > I started over with SSD and the difference was like night and day. I > processed 200GB of photos in ~1hour, all the cores were saturated. Cool. > It didn't all go smoothly though, near the end camlistored hit the open > files limit and I had to up the ulimit -n from 255 (4096 worked) and start > over, which was even quicker. Ugh. afair we fixed some parts of camlistored so they would set the ulimit for you, but apparently not in all cases yet. > -- > You received this message because you are subscribed to the Google Groups > "Camlistore" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Camlistore" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
