Il giorno domenica 7 maggio 2017 03:38:18 UTC+10, mpl ha scritto: > > On 6 May 2017 at 03:05, Rhythmic Fistman <[email protected] <javascript:>> > wrote: > > > > > > Il giorno venerdì 5 maggio 2017 23:27:55 UTC+10, mpl ha scritto: > >> > >> On 5 May 2017 at 09:00, Rhythmic Fistman <[email protected]> wrote: > >> > I'm in a bind with this - I ctrl-C camlistored and the kvIndexFile > >> > became > >> > corrupted (how do I cleanly stop it?) & re-indexing is taking days. > >> > >> Hmm, how come you were on kvIndexFile? I thought we had made leveldb > >> the default everywhere (because we indeed experienced corruption > >> problems with kv iirc). Did you specify kvIndexFile yourself on > >> purpose? > > > > > > Not on purpose, it's because I come back to camlistore every year or two > to > > try and backup and deduplicate my photos, > > so I've got old configurations lying around. > > > > So I switched to levelDB. On my local ssd setup > > > > $ time camlistored -openbrowser=false -reindex > > > > > > Starting camlistored version 8f1a7df176de21d0d052c04db45b373c5c282fb1; > Go > > go1.8 (darwin/amd64) > > > > Starting to listen on http://localhost:3179 > > > > Index wiped. > > > > Error: blobpacked storage detects non-zero packed zips, but no metadata. > > Please re-start in recovery mode with -recovery. > > > > > > > > $ time camlistored -openbrowser=false -recovery > > > > Starting camlistored version 8f1a7df176de21d0d052c04db45b373c5c282fb1; > Go > > go1.8 (darwin/amd64) > > > > Starting to listen on http://localhost:3179 > > > > ui: serving Closure from embedded resources > > > > Starting recovery of blobpacked index > > > > Caught panic installer handlers: error instantiating storage for prefix > > "/bs/", type "blobpacked": Sum of all zips (1055150 bytes) does not > match > > manifest's WholeSize (527575 bytes) f > > > > [...] BACKTRACE ELIDED > > > > > > This time reindex works in ~13 minutes > > wait, are you saying that you got a panic with serverinit, but the > camlistored process didn't die, and the reindexing kept going > successfully? >
no - -reindex failed (and exited), then -recovery failed (and exited). After that -reindex worked. > > > time camlistored -openbrowser=false -reindex > > > > > > but should I be worried that the recovery failed? > > Not worried that something is corrupted, but worried that not all your > blobs are reachable (because the blobpacked index is not up to date), > yes. > What should I do? > Can you please file a bug about it, so I don't forget to try and > reproduce/fix it? > Ok - not sure how. Maybe I'll back up the index, then try re-indexing. > > > similar story with the larger external non-ssd. > > > > recovery fails, now re-indexing, looks like it might take an hour - this > is > > an improvement over kvindex I think. > > > > Strangely, local SSD had camlistored saturating the CPU (200%?) whereas > > external rust is sitting at 6%... > > yeah I think that's pretty common, hence why I mentioned using an SSD > before. afair it's partly due to HDDs having to wait for fsync or > something like that. > I started over with SSD and the difference was like night and day. I processed 200GB of photos in ~1hour, all the cores were saturated. It didn't all go smoothly though, near the end camlistored hit the open files limit and I had to up the ulimit -n from 255 (4096 worked) and start over, which was even quicker. -- You received this message because you are subscribed to the Google Groups "Camlistore" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
