Il giorno venerdì 5 maggio 2017 23:27:55 UTC+10, mpl ha scritto: > > On 5 May 2017 at 09:00, Rhythmic Fistman <[email protected] <javascript:>> > wrote: > > I'm in a bind with this - I ctrl-C camlistored and the kvIndexFile > became > > corrupted (how do I cleanly stop it?) & re-indexing is taking days. > > Hmm, how come you were on kvIndexFile? I thought we had made leveldb > the default everywhere (because we indeed experienced corruption > problems with kv iirc). Did you specify kvIndexFile yourself on > purpose? >
Not on purpose, it's because I come back to camlistore every year or two to try and backup and deduplicate my photos, so I've got old configurations lying around. So I switched to levelDB. On my local ssd setup $ time camlistored -openbrowser=false -reindex Starting camlistored version 8f1a7df176de21d0d052c04db45b373c5c282fb1; Go go1.8 (darwin/amd64) Starting to listen on http://localhost:3179 Index wiped. Error: blobpacked storage detects non-zero packed zips, but no metadata. Please re-start in recovery mode with -recovery. $ time camlistored -openbrowser=false -recovery Starting camlistored version 8f1a7df176de21d0d052c04db45b373c5c282fb1; Go go1.8 (darwin/amd64) Starting to listen on http://localhost:3179 ui: serving Closure from embedded resources Starting recovery of blobpacked index Caught panic installer handlers: error instantiating storage for prefix "/bs/", type "blobpacked": Sum of all zips (1055150 bytes) does not match manifest's WholeSize (527575 bytes) f [...] BACKTRACE ELIDED This time reindex works in ~13 minutes time camlistored -openbrowser=false -reindex but should I be worried that the recovery failed? similar story with the larger external non-ssd. recovery fails, now re-indexing, looks like it might take an hour - this is an improvement over kvindex I think. Strangely, local SSD had camlistored saturating the CPU (200%?) whereas external rust is sitting at 6%... > > This is 40GB of blobs on an external non-ssd drive. A previous > experiment on > > my internal SSD with a store of 60gb takes less than a minute. > > > > I'd like to start over on SSD, but this is going to be a large store. > Losing > > the index isn't supposed to be a serious problem, but if I can't > recalculate > > it in a reasonable amount of time then it is a problem. Maybe a mysql > index > > would be safer, but I still have the problem of the slow initial import. > > MySQL would definitely be more reliable than any of the file-based > indexes, but it would indeed probably not be much faster. Improving > the (re-)indexing speed has always been long overdue, but other things > got higher priority unfortunately. > > > Il giorno giovedì 4 maggio 2017 04:00:25 UTC+10, clive boulton ha > scritto: > >> > >> Using the GCE launcher version of Camlistore. Are there any steps > required > >> to use MySQL or another indexer? > >> > >> On Apr 30, 2017 7:52 AM, "Mathieu Lonjaret" <[email protected]> > wrote: > >>> > >>> On 30 April 2017 at 02:28, Rhythmic Fistman <[email protected]> > wrote: > >>> > I finally tried out camlistore by starting a local instance of > >>> > camlistored > >>> > with blob & index store on an external drive and running camput on > >>> > 200GB of > >>> > photos, of which are probably 50% duplicates: > >>> > > >>> > > >>> > camput file --permanode --title "some photos" --tag=photos > >>> > ~/backups/photos/ > >>> > > >>> > > >>> > That was four days ago. > >>> > > >>> > > >>> > I forgot to add any progress indicator to camput, so I've had to > >>> > estimate > >>> > progress. Judging from blobs written to disk (40GB) and bytes read > by > >>> > camput > >>> > (50GB) it looks like it's a quarter of the way through. > >>> > > >>> > > >>> > I now need to interrupt the operation - will the successfully stored > >>> > blobs > >>> > still be there? Can I re-run camput & and start from day 4? > >>> > >>> Yes, you can interrupt without any problem. > >>> When you restart the camput command, it will restart from the > >>> beginning. But there's a layer of caching on the client-side, and even > >>> if there weren't camput tries to stat the blob on the server before > >>> trying to upload it, so in effect camput should "skip" all the blobs > >>> that it already uploaded. > >>> > >>> > Is there something I can do to speed my configuration up? cpu usage > of > >>> > both > >>> > client and server is minimal as is disk IO and (local network). > >>> > >>> as usual, SSD vs regular HDD has an impact. > >>> Also, some indexer implementations are better than others. MySQL is > >>> most likely the most reliable, and I'd guess among the fastest ones. > >>> > >>> > Where's the time being spent? > >>> > > >>> > > >>> > Thanks, > >>> > > >>> > RF > >>> > > >>> > -- > >>> > You received this message because you are subscribed to the Google > >>> > Groups > >>> > "Camlistore" group. > >>> > To unsubscribe from this group and stop receiving emails from it, > send > >>> > an > >>> > email to [email protected]. > >>> > For more options, visit https://groups.google.com/d/optout. > >>> > >>> -- > >>> You received this message because you are subscribed to the Google > Groups > >>> "Camlistore" group. > >>> To unsubscribe from this group and stop receiving emails from it, send > an > >>> email to [email protected]. > >>> For more options, visit https://groups.google.com/d/optout. > > > > -- > > You received this message because you are subscribed to the Google > Groups > > "Camlistore" group. > > To unsubscribe from this group and stop receiving emails from it, send > an > > email to [email protected] <javascript:>. > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Camlistore" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
