On 5 May 2017 at 09:00, Rhythmic Fistman <[email protected]> wrote: > I'm in a bind with this - I ctrl-C camlistored and the kvIndexFile became > corrupted (how do I cleanly stop it?) & re-indexing is taking days.
Hmm, how come you were on kvIndexFile? I thought we had made leveldb the default everywhere (because we indeed experienced corruption problems with kv iirc). Did you specify kvIndexFile yourself on purpose? > This is 40GB of blobs on an external non-ssd drive. A previous experiment on > my internal SSD with a store of 60gb takes less than a minute. > > I'd like to start over on SSD, but this is going to be a large store. Losing > the index isn't supposed to be a serious problem, but if I can't recalculate > it in a reasonable amount of time then it is a problem. Maybe a mysql index > would be safer, but I still have the problem of the slow initial import. MySQL would definitely be more reliable than any of the file-based indexes, but it would indeed probably not be much faster. Improving the (re-)indexing speed has always been long overdue, but other things got higher priority unfortunately. > Il giorno giovedì 4 maggio 2017 04:00:25 UTC+10, clive boulton ha scritto: >> >> Using the GCE launcher version of Camlistore. Are there any steps required >> to use MySQL or another indexer? >> >> On Apr 30, 2017 7:52 AM, "Mathieu Lonjaret" <[email protected]> wrote: >>> >>> On 30 April 2017 at 02:28, Rhythmic Fistman <[email protected]> wrote: >>> > I finally tried out camlistore by starting a local instance of >>> > camlistored >>> > with blob & index store on an external drive and running camput on >>> > 200GB of >>> > photos, of which are probably 50% duplicates: >>> > >>> > >>> > camput file --permanode --title "some photos" --tag=photos >>> > ~/backups/photos/ >>> > >>> > >>> > That was four days ago. >>> > >>> > >>> > I forgot to add any progress indicator to camput, so I've had to >>> > estimate >>> > progress. Judging from blobs written to disk (40GB) and bytes read by >>> > camput >>> > (50GB) it looks like it's a quarter of the way through. >>> > >>> > >>> > I now need to interrupt the operation - will the successfully stored >>> > blobs >>> > still be there? Can I re-run camput & and start from day 4? >>> >>> Yes, you can interrupt without any problem. >>> When you restart the camput command, it will restart from the >>> beginning. But there's a layer of caching on the client-side, and even >>> if there weren't camput tries to stat the blob on the server before >>> trying to upload it, so in effect camput should "skip" all the blobs >>> that it already uploaded. >>> >>> > Is there something I can do to speed my configuration up? cpu usage of >>> > both >>> > client and server is minimal as is disk IO and (local network). >>> >>> as usual, SSD vs regular HDD has an impact. >>> Also, some indexer implementations are better than others. MySQL is >>> most likely the most reliable, and I'd guess among the fastest ones. >>> >>> > Where's the time being spent? >>> > >>> > >>> > Thanks, >>> > >>> > RF >>> > >>> > -- >>> > You received this message because you are subscribed to the Google >>> > Groups >>> > "Camlistore" group. >>> > To unsubscribe from this group and stop receiving emails from it, send >>> > an >>> > email to [email protected]. >>> > For more options, visit https://groups.google.com/d/optout. >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Camlistore" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to [email protected]. >>> For more options, visit https://groups.google.com/d/optout. > > -- > You received this message because you are subscribed to the Google Groups > "Camlistore" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Camlistore" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
