On 6 May 2017 at 03:05, Rhythmic Fistman <[email protected]> wrote: > > > Il giorno venerdì 5 maggio 2017 23:27:55 UTC+10, mpl ha scritto: >> >> On 5 May 2017 at 09:00, Rhythmic Fistman <[email protected]> wrote: >> > I'm in a bind with this - I ctrl-C camlistored and the kvIndexFile >> > became >> > corrupted (how do I cleanly stop it?) & re-indexing is taking days. >> >> Hmm, how come you were on kvIndexFile? I thought we had made leveldb >> the default everywhere (because we indeed experienced corruption >> problems with kv iirc). Did you specify kvIndexFile yourself on >> purpose? > > > Not on purpose, it's because I come back to camlistore every year or two to > try and backup and deduplicate my photos, > so I've got old configurations lying around. > > So I switched to levelDB. On my local ssd setup > > $ time camlistored -openbrowser=false -reindex > > > Starting camlistored version 8f1a7df176de21d0d052c04db45b373c5c282fb1; Go > go1.8 (darwin/amd64) > > Starting to listen on http://localhost:3179 > > Index wiped. > > Error: blobpacked storage detects non-zero packed zips, but no metadata. > Please re-start in recovery mode with -recovery. > > > > $ time camlistored -openbrowser=false -recovery > > Starting camlistored version 8f1a7df176de21d0d052c04db45b373c5c282fb1; Go > go1.8 (darwin/amd64) > > Starting to listen on http://localhost:3179 > > ui: serving Closure from embedded resources > > Starting recovery of blobpacked index > > Caught panic installer handlers: error instantiating storage for prefix > "/bs/", type "blobpacked": Sum of all zips (1055150 bytes) does not match > manifest's WholeSize (527575 bytes) f > > [...] BACKTRACE ELIDED > > > This time reindex works in ~13 minutes
wait, are you saying that you got a panic with serverinit, but the camlistored process didn't die, and the reindexing kept going successfully? > time camlistored -openbrowser=false -reindex > > > but should I be worried that the recovery failed? Not worried that something is corrupted, but worried that not all your blobs are reachable (because the blobpacked index is not up to date), yes. Can you please file a bug about it, so I don't forget to try and reproduce/fix it? > similar story with the larger external non-ssd. > > recovery fails, now re-indexing, looks like it might take an hour - this is > an improvement over kvindex I think. > > Strangely, local SSD had camlistored saturating the CPU (200%?) whereas > external rust is sitting at 6%... yeah I think that's pretty common, hence why I mentioned using an SSD before. afair it's partly due to HDDs having to wait for fsync or something like that. >> > This is 40GB of blobs on an external non-ssd drive. A previous >> > experiment on >> > my internal SSD with a store of 60gb takes less than a minute. >> > >> > I'd like to start over on SSD, but this is going to be a large store. >> > Losing >> > the index isn't supposed to be a serious problem, but if I can't >> > recalculate >> > it in a reasonable amount of time then it is a problem. Maybe a mysql >> > index >> > would be safer, but I still have the problem of the slow initial import. >> >> MySQL would definitely be more reliable than any of the file-based >> indexes, but it would indeed probably not be much faster. Improving >> the (re-)indexing speed has always been long overdue, but other things >> got higher priority unfortunately. >> >> > Il giorno giovedì 4 maggio 2017 04:00:25 UTC+10, clive boulton ha >> > scritto: >> >> >> >> Using the GCE launcher version of Camlistore. Are there any steps >> >> required >> >> to use MySQL or another indexer? >> >> >> >> On Apr 30, 2017 7:52 AM, "Mathieu Lonjaret" <[email protected]> >> >> wrote: >> >>> >> >>> On 30 April 2017 at 02:28, Rhythmic Fistman <[email protected]> wrote: >> >>> > I finally tried out camlistore by starting a local instance of >> >>> > camlistored >> >>> > with blob & index store on an external drive and running camput on >> >>> > 200GB of >> >>> > photos, of which are probably 50% duplicates: >> >>> > >> >>> > >> >>> > camput file --permanode --title "some photos" --tag=photos >> >>> > ~/backups/photos/ >> >>> > >> >>> > >> >>> > That was four days ago. >> >>> > >> >>> > >> >>> > I forgot to add any progress indicator to camput, so I've had to >> >>> > estimate >> >>> > progress. Judging from blobs written to disk (40GB) and bytes read >> >>> > by >> >>> > camput >> >>> > (50GB) it looks like it's a quarter of the way through. >> >>> > >> >>> > >> >>> > I now need to interrupt the operation - will the successfully stored >> >>> > blobs >> >>> > still be there? Can I re-run camput & and start from day 4? >> >>> >> >>> Yes, you can interrupt without any problem. >> >>> When you restart the camput command, it will restart from the >> >>> beginning. But there's a layer of caching on the client-side, and even >> >>> if there weren't camput tries to stat the blob on the server before >> >>> trying to upload it, so in effect camput should "skip" all the blobs >> >>> that it already uploaded. >> >>> >> >>> > Is there something I can do to speed my configuration up? cpu usage >> >>> > of >> >>> > both >> >>> > client and server is minimal as is disk IO and (local network). >> >>> >> >>> as usual, SSD vs regular HDD has an impact. >> >>> Also, some indexer implementations are better than others. MySQL is >> >>> most likely the most reliable, and I'd guess among the fastest ones. >> >>> >> >>> > Where's the time being spent? >> >>> > >> >>> > >> >>> > Thanks, >> >>> > >> >>> > RF >> >>> > >> >>> > -- >> >>> > You received this message because you are subscribed to the Google >> >>> > Groups >> >>> > "Camlistore" group. >> >>> > To unsubscribe from this group and stop receiving emails from it, >> >>> > send >> >>> > an >> >>> > email to [email protected]. >> >>> > For more options, visit https://groups.google.com/d/optout. >> >>> >> >>> -- >> >>> You received this message because you are subscribed to the Google >> >>> Groups >> >>> "Camlistore" group. >> >>> To unsubscribe from this group and stop receiving emails from it, send >> >>> an >> >>> email to [email protected]. >> >>> For more options, visit https://groups.google.com/d/optout. >> > >> > -- >> > You received this message because you are subscribed to the Google >> > Groups >> > "Camlistore" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> > an >> > email to [email protected]. >> > For more options, visit https://groups.google.com/d/optout. > > -- > You received this message because you are subscribed to the Google Groups > "Camlistore" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Camlistore" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
