Alok, that's the setup (home(local, s3), linode(s3)) I ran for a few years (just one instance at home now). It should work well.
One caveat: with two instances both writing to the same blob store (s3 in this case), each won't immediately index the other's changes. You'll need to periodically trigger a "full scan" with camtool index on each host (via crontab or similar) to see changes from the other instance. On Sun, May 8, 2016 at 7:44 PM, Alok Parlikar <[email protected]> wrote: > Oh. Great! I think I understand now. > > I'll install camlistore on a local server with large disks (and > redundancy) as the primary storage, with S3 as secondary storage. > I'll use a cloud VPS to use the same S3 as primary storage. (umm. > probabaly still use linode, maybe move to ec2 later). > > S3/GCS vs EC2/GCE I'll run some price comparison, but I'm suspecting it's > going to be about the same. > > Thanks so much! > > On Monday, May 9, 2016 at 1:11:45 AM UTC+5:30, mpl wrote: > >> On 8 May 2016 at 08:08, Alok Parlikar <[email protected]> wrote: >> > Thanks, Mathieu. Very helpful. >> > >> >> Or you can use it to share photos whenever you want. In that case, one >> >> workflow is to have a "staging" Camlistore instance (e.g. on your >> >> laptop) to receive the photos, decide which ones you want to keep, and >> >> send the result to your backup instance(s). >> > >> > If I delete things from the staging instance, the original blob would >> still >> > be copied to the backups, right, along with the metadata about it being >> > deleted? What advantage would the staging provide? >> >> Right. I meant a throwaway instance, which does not replicate to the >> other ones, you just use it for receiving the photos this one time. I >> typically do it by starting the "dev server" we use for hacking on >> Camlistore. But yeah, you could transfer the photos to the laptop by >> whatever other means you want. >> >> > Also, not sure about another thing: >> > >> > My linode has a small disk (20GB), my laptop a bigger disk (500GB). But >> the >> > data I'd like to ingest is currently sitting on bigger disks (lots of >> owned >> > media content) and it will certainly not fit on the linode. There's >> really >> > two sets of data --> regular use (docs, code, etc.) and archives >> (media). >> > >> > Right now, I use owncloud on linode (synced with my laptop) to store my >> > regular data, and archives are backed up on external disks. >> > >> > What's the best way to store this larger data when I move everything to >> > camlistore? >> >> I've just realized we've been focusing so much on GCE that I forgot to >> mention that you don't have to run Camlistore instances on >> GCE/EC2/wherever to store your blobs on the cloud (Google Cloud >> Storage, S3) sorry. >> >> So, you could just store your data in Camlistore on any local machine >> with a large enough disk, and you would define GCS as the secondary >> storage, so the blobs would get replicated there automatically. >> But, since I assume your local machine is not always reachable on the >> net, what you could additionally do is then set up another Camlistore >> on Linode, which would have no local storage, and which would use the >> above GCS as its primary. It would be slow, but it would give you a >> permanent view/access (through e.g. the web UI) of your blobs wherever >> your are. Otoh, if you don't care about giving up on your linode, and >> if GCE is not more expensive, you could run a GCE instance instead >> (because much faster access to GCS). >> >> > On Sunday, May 8, 2016 at 7:35:39 PM UTC+5:30, mpl wrote: >> >> >> >> On 7 May 2016 at 19:05, Alok Parlikar <[email protected]> wrote: >> >> > Probably an FAQ, and I did browse the group archives, but might >> still >> >> > need a >> >> > little handholding in setting up my workflow. >> >> > >> >> > I have two linux laptops, an android phone, a RaspberryPi and a host >> >> > with >> >> > linode. The linode box currently has owncloud installed. My backup >> >> > workflow >> >> > currently just has duplicity with external hdds. >> >> > >> >> > I'd like to switch to camlistore -- looking at all the docs and >> videos, >> >> > it >> >> > was easy to make up my mind. But I'm stuck as to how to actually set >> up >> >> > my >> >> > workflow. >> >> > >> >> > (1) Which machine(s) should run camlistored? One of my laptops? The >> >> > linode? >> >> > Or should I switch to GCE/AWS instead of linode and use cloud >> storage >> >> > for >> >> > backup? >> >> >> >> If you're really serious about backups, then you should definitely >> >> have at least two copies in separate locations. So yes, one remote >> >> instance on e.g. Linode, and another one could be your laptop. Or one >> >> on Linode, and one on GCE/AWS, etc. >> >> Whether to keep your Linode or switch to GCE is really up to you. >> >> cost, brand, preferences etc considerations. And maybe you have other >> >> services running on your Linode that you don't want to migrate. >> >> >> >> > (2) Can the raspberry pi also be a local cache of my store on my >> >> > network? >> >> >> >> I haven't tried in a while, but I think the Pi (at least the 1st >> >> model) is a bit weak for Camlistore, especially because of lack of >> >> memory afair. Maybe if one disabled most features and indexing, and >> >> only used it as a dumb blobstore... >> >> >> >> > (3) Any tips for how my backup workflow should look like -- if I use >> >> > s3/GCS, >> >> > should I still back up my laptop periodically on disks? If so, >> what's >> >> > the >> >> > best way to back up the camlistore data? >> >> >> >> You can set up remote instances to be replicas of your laptop's >> >> instance (in the laptop's Camlistore configuration), in which case you >> >> don't have anything to do as any blobs written to the laptop are >> >> automatically sent to the other instances. >> >> Or you can use camtool sync. >> >> >> >> > (4) I think my $HOME should still be on disk, and not use the fuse >> >> > mount; in >> >> > that case, how easy is it to keep it in sync with camlistore? >> >> >> >> Wait. Camlistore is not really meant to replace your whole day to day >> >> work filesystem, at least not right now. So yeah, you definitely still >> >> want a traditional filesystem with a $HOME and everything on your >> >> laptop. >> >> >> >> > (5) When I transfer pictures out of my camera, where should I put >> them? >> >> > Directly into my camlistore? Copy to disk and let the "$HOME sync" >> take >> >> > care >> >> > of it? That might not be ideal (small laptop disk). Put it on >> external >> >> > disk >> >> > and in camlistore? how?. >> >> >> >> We have an android client. >> >> You can set it to regularly upload your photos to your instance, in >> >> which case you probably want to send them to an "always on" instance >> >> like the one on Linode or GCE/AWS. >> >> Or you can use it to share photos whenever you want. In that case, one >> >> workflow is to have a "staging" Camlistore instance (e.g. on your >> >> laptop) to receive the photos, decide which ones you want to keep, and >> >> send the result to your backup instance(s). >> >> >> >> hth, >> >> Mathieu >> >> >> >> > -- >> >> > You received this message because you are subscribed to the Google >> >> > Groups >> >> > "Camlistore" group. >> >> > To unsubscribe from this group and stop receiving emails from it, >> send >> >> > an >> >> > email to [email protected]. >> >> > For more options, visit https://groups.google.com/d/optout. >> > >> > -- >> > You received this message because you are subscribed to the Google >> Groups >> > "Camlistore" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> an >> > email to [email protected]. >> > For more options, visit https://groups.google.com/d/optout. >> > -- > You received this message because you are subscribed to the Google Groups > "Camlistore" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- best, Eric eric.pdxhub.org <http://pdxhub.org/people/eric> -- You received this message because you are subscribed to the Google Groups "Camlistore" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
