a few weeks ago I tried to use camlistore to make backups. I tried like your example above to do it; but failed mutliples times and I thought I do not understand the concepts of camlistore good enough and that is the reason why I failed.
I use crashplan and some backblaze B2 buckets that get synced from a synology NAS and their B2 support. crashplan offers to store all my files forever; but the thing is not scriptable or does offer real ownership of my data. its at best a third option store store it in case of nearly complete disaster. How hard it can be to "store files, with infinite retention"? As camlistore advertises, I want to store "my" files in stores I control and I want to have forever control over "my" files, photos and stuff. Cloud services are nice, but when I find out that google photos (as it did for me) was unable to generate a complete takeout of my photos or iTunes with it iTunes Match, Apple Music and what else "features" delete actually Music I am allowed to "own" as a file on a Mac or PC, "I want my files back" and want to have contol over it. Another example: once I restored files from crashplan and something failed. "some files where not restore". Actually there where a few thousand of them not restored. The log file did not say which... and crashplan support asekd me what I am missing... whell with a TB or so of data, who knows which files one is missing? so as I wrote, how hard could it be? there is a nice list with plenty of options. https://wiki.archlinux.org/index.php/Synchronization_and_backup_programs but I did not found any that would so basic that in case of a disaster, I would be able to restore files with bare and basic tools. I look for: - index and backup (sync over to something that stores it) files every other minute/hour/day - mark every backup run with a tag, like timestamp or something useful else - infinite retention, all versions of a file kept. - data deduplication - have an index that I can compare easily (which files did I "loose" between day X and day Z) - know that my stored files are in consistent state. - storage should be fexible, so that it can be stored on a filesystem or any of the cloud "bucket" solutions like S3 or B2 I have the impression all this can be implemented what camlistore offers, but I did not get my head around yet how to do it :-) On Saturday, 1 October 2016 22:23:24 UTC+2, Theodore Ts'o wrote: > > I'm just curious then. Is anyone actually using camlistore to do > production backups? And if so, does anyone have some scripts to share? > The current raw low-level interface seems to be roughly equivalent to that > of "git plumbing" --- e.g., harking back to the the early days of git when > Linus was writing shell scripts that used "git write-tree", "git ls-tree", > "git commit-tree", "git update-ref", etc. when he was first boot-strapping > git. The thing is, git's "porcelain" came very quickly, and camilstore > has been around for while (people have been asking questions about workflow > for doing backups going back to 2014, according to searches on this mailing > list). So before I go and re-invent the wheel --- does anyone have any > scripts they could share about how they are actually using camilstore to do > their own backups? > > I just find it hard to believe people are just using things like "camput > file -permanode -title="home-backup-2016-10-1" -tag backups $HOME" and just > calling it a day? Or is that really all people are doing? > > Thanks, > > -- Ted > > On Saturday, October 1, 2016 at 1:48:57 PM UTC-4, mpl wrote: >> >> Hi, >> >> camtool is indeed not the most user-friendly of our tools. When I >> forget how it works, (as I just had now), I just "reverse-engineer" >> it. That's to say, if you create (mkdir, cp) new nodes in roots, >> you'll see (with some devcam get, devcam tool describe, or in the web >> UI) that the relation created is a camliPath. Same goes for the >> children of these nodes. So: >> >> devcam put permanode >> -> sha1-500520ae4269b7ac7b20f0a42ba57650450ddb02 >> devcam put attr camliRoot >> sha1-500520ae4269b7ac7b20f0a42ba57650450ddb02 someRootName >> devcam put permanode >> -> sha1-c8db36279ee03f5daf69a3d2885ef5d8c0111f05 >> devcam put attr sha1-500520ae4269b7ac7b20f0a42ba57650450ddb02 >> camliPath:TODO.txt sha1-c8db36279ee03f5daf69a3d2885ef5d8c0111f05 >> devcam put file /home/mpl/docs/TODO.txt >> -> sha1-4c8b980f2c5faa49e687c8f1235ed17b2818691c >> devcam put attr sha1-c8db36279ee03f5daf69a3d2885ef5d8c0111f05 >> camliContent sha1-4c8b980f2c5faa49e687c8f1235ed17b2818691c >> >> >> ll /tmp/cammount-dir/roots/someRootName/TODO.txt >> -rw------- 1 mpl mpl 1654 Oct 1 19:14 >> /tmp/cammount-dir/roots/someRootName/TODO.txt >> >> I'd have to recheck, but I don't think the navigation from the roots >> are following other relations than camliPath and camliContent, so >> they're not following the static-set relation that you get betwen a >> dir and its children when you camput a directory. And I don't know >> what's the roadmap on that. >> >> As (I think) you have found out, you can still browse a directory if >> you know its hash, as in: >> >> devcam put file /home/mpl/docs >> -> sha1-339cf73c02ea5981b71f3aeaaf8864dbfbedafa5 >> cd /tmp/cammount-dir/sha1-339cf73c02ea5981b71f3aeaaf8864dbfbedafa5 >> >> hth, >> Mathieu >> >> >> On 1 October 2016 at 18:52, Theodore Ts'o <[email protected]> wrote: >> > I'm really confused about how to take a backup and make it available in >> > Roots so I can browse it using the cammount. I don't know if this >> releated >> > to the fact that I'm using ./bin/devcam, but the documentation on how >> Roots >> > and work is very sparse. :-( >> > >> > OK, so starting with "./bin/devcam server --wipe" in one window, I then >> do a >> > backup: >> > >> > % ./bin/devcam put file /tmp/test-dir >> > >> > So I get back a sha1-sig which looks like this: >> > >> > >> > % ./bin/devcam tool describe >> sha1-0609bc86627d0fa3067d45c09ede479c7e8643f8 >> > { >> > "meta": { >> > "sha1-01b8821894b8de827984b1b9f59671bf0cffc036": { >> > "blobRef": "sha1-01b8821894b8de827984b1b9f59671bf0cffc036", >> > "camliType": "directory", >> > "size": 306, >> > "dir": { >> > "fileName": "subdir", >> > "size": 1, >> > "wholeRef": null >> > }, >> > "dirChildren": [ >> > "sha1-9999264baa2c936c30b509be0f0f77c1e39615f6" >> > ] >> > }, >> > "sha1-0609bc86627d0fa3067d45c09ede479c7e8643f8": { >> > "blobRef": "sha1-0609bc86627d0fa3067d45c09ede479c7e8643f8", >> > "camliType": "directory", >> > "size": 309, >> > "dir": { >> > "fileName": "test-dir", >> > "size": 3, >> > "wholeRef": null >> > }, >> > "dirChildren": [ >> > "sha1-01b8821894b8de827984b1b9f59671bf0cffc036", >> > "sha1-7ecba772756f1b4fbe0f3179cc5ad02e087aad75", >> > "sha1-8256107f4be52db012e4e80d2e43a71b542769d7" >> > ] >> > }, >> > "sha1-7ecba772756f1b4fbe0f3179cc5ad02e087aad75": { >> > "blobRef": "sha1-7ecba772756f1b4fbe0f3179cc5ad02e087aad75", >> > "camliType": "file", >> > "size": 352, >> > "file": { >> > "fileName": "issue", >> > "size": 36, >> > "time": "2016-10-01T14:21:26.091166128Z", >> > "wholeRef": "sha1-309e70c09b04eb5e144b29b21e5ecec2d3289628" >> > } >> > }, >> > "sha1-8256107f4be52db012e4e80d2e43a71b542769d7": { >> > "blobRef": "sha1-8256107f4be52db012e4e80d2e43a71b542769d7", >> > "camliType": "file", >> > "size": 352, >> > "file": { >> > "fileName": "motd", >> > "size": 286, >> > "time": "2016-10-01T14:21:29.771162244Z", >> > "wholeRef": "sha1-8b55aac644e9e6f2701805584cc391ff81d3ecec" >> > } >> > }, >> > "sha1-9999264baa2c936c30b509be0f0f77c1e39615f6": { >> > "blobRef": "sha1-9999264baa2c936c30b509be0f0f77c1e39615f6", >> > "camliType": "file", >> > "size": 350, >> > "file": { >> > "fileName": "date", >> > "size": 29, >> > "time": "2016-10-01T15:09:46.36563753Z", >> > "wholeRef": "sha1-fc866bc41a833bed829bf1580ce11113580fd4b0" >> > } >> > } >> > } >> > } >> > >> > so the first weird thing is that the docs imply that this should work: >> > >> > % ./bin/devcam mount sha1-0609bc86627d0fa3067d45c09ede479c7e8643f8 >> > Cammount running with mountpoint /tmp/cammount-dir. Press 'q' <enter> >> or >> > ctrl-c to shut down. >> > 2016/10/01 12:32:19 root specified is not a blobref or name of a root: >> > "/tmp/cammount-dir" >> > >> > Hmm... so it claims it wants the "name of a root". how about the >> > pre-created dev-pics-root? >> > >> > % ./bin/devcam mount /dev-pics-root >> > Cammount running with mountpoint /tmp/cammount-dir. Press 'q' <enter> >> or >> > ctrl-c to shut down. >> > 2016/10/01 12:33:45 root specified is not a blobref or name of a root: >> > "/tmp/cammount-dir" >> > >> > Nope. OK, so I'll just use "./bin/devcam mount" and get the full >> default >> > top-level: >> > >> > % ls /tmp/cammount-dir >> > total 0 >> > 0 at/ 0 recent/ 0 sha1-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/ 0 >> > WELCOME.txt >> > 0 date/ 0 roots/ 0 tag/ >> > % ls /tmp/cammount-dir/roots >> > total 0 >> > 0 dev-pics-root/ >> > >> > Let's create a new root: >> > >> > % ./bin/devcam put permanode 2> /dev/null >> > sha1-19d8e08e5660275357b072edae6741b14db0c882 >> > % ./bin/devcam put attr sha1-19d8e08e5660275357b072edae6741b14db0c882 >> title >> > testroot 2> /dev/null >> > sha1-608d3485eb205e41e3db9ba2c7b498eff1797f3f >> > % ./bin/devcam put attr sha1-19d8e08e5660275357b072edae6741b14db0c882 >> > camliRoot testroot 2> /dev/null >> > sha1-10b885cd8630331db50648cc10242cb6e0edadb1 >> > >> > % ls /tmp/cammount-dir/roots/ >> > total 0 >> > 0 dev-pics-root/ 0 testroot/ >> > >> > Yay! It works. Now I try connecting the backup I made above >> > (sha1-0609bc86627d0fa3067d45c09ede479c7e8643f8) with the testroot.... >> and >> > everything fails. My best guess, which I fully expected to work was: >> > >> > % ./bin/devcam put attr sha1-19d8e08e5660275357b072edae6741b14db0c882 >> > camliPath:testdir sha1-0609bc86627d0fa3067d45c09ede479c7e8643f8 >> > sha1-aaf244b34b412331789caa86eaf09443468a22ee >> > >> > But alas, no dice: >> > >> > % ./bin/devcam mount >> > Cammount running with mountpoint /tmp/cammount-dir. Press 'q' <enter> >> or >> > ctrl-c to shut down. >> > >> > <in another window> >> > % ls /tmp/cammount-dir/roots/testroot/ >> > total 0 >> > >> > Argh, so let's try creating via fuse: >> > >> > % mkdir -p /tmp/cammount-dir/roots/testroot/foo/bar/baz >> > >> > <back to the original window after shutting down cammount> >> > >> > % ./bin/devcam tool describe >> sha1-19d8e08e5660275357b072edae6741b14db0c882{ >> > "meta": { >> > "sha1-19d8e08e5660275357b072edae6741b14db0c882": { >> > "blobRef": "sha1-19d8e08e5660275357b072edae6741b14db0c882", >> > "camliType": "permanode", >> > "size": 562, >> > "permanode": { >> > "attr": { >> > "camliPath:foo": [ >> > "sha1-fc4e1bf59ba29d2e13f73d685c09d2cd26674bdc" >> > ], >> > "camliPath:testdir": [ >> > "sha1-0609bc86627d0fa3067d45c09ede479c7e8643f8" >> > ], >> > "camliRoot": [ >> > "testroot" >> > ], >> > "title": [ >> > "testroot" >> > ] >> > }, >> > "modtime": "2016-10-01T16:44:00.886392008Z" >> > } >> > } >> > } >> > } >> > >> > Hmm, let's look at what the references for "foo" and "testdir" look >> like: >> > >> > % ./bin/devcam tool describe >> sha1-fc4e1bf59ba29d2e13f73d685c09d2cd26674bdc >> > { >> > "meta": { >> > "sha1-fc4e1bf59ba29d2e13f73d685c09d2cd26674bdc": { >> > "blobRef": "sha1-fc4e1bf59ba29d2e13f73d685c09d2cd26674bdc", >> > "camliType": "permanode", >> > "size": 562, >> > "permanode": { >> > "attr": { >> > "camliNodeType": [ >> > "directory" >> > ], >> > "camliPath:bar": [ >> > "sha1-b0dc6213dfbc7b004304fe46ebd667a9529dd928" >> > ], >> > "title": [ >> > "foo" >> > ] >> > }, >> > "modtime": "2016-10-01T16:44:00.905961518Z" >> > } >> > } >> > } >> > } >> > >> > and the description of "testdir" is above. >> > >> > And the first thing I notice is they look totally difference. For the >> > subdir created via fuse, it uses camliPath to name the subdirectories. >> But >> > in testdir, the directory contents are described very differently: >> > >> > "sha1-0609bc86627d0fa3067d45c09ede479c7e8643f8": { >> > "blobRef": "sha1-0609bc86627d0fa3067d45c09ede479c7e8643f8", >> > "camliType": "directory", >> > "size": 309, >> > "dir": { >> > "fileName": "test-dir", >> > "size": 3, >> > "wholeRef": null >> > }, >> > "dirChildren": [ >> > "sha1-01b8821894b8de827984b1b9f59671bf0cffc036", >> > "sha1-7ecba772756f1b4fbe0f3179cc5ad02e087aad75", >> > "sha1-8256107f4be52db012e4e80d2e43a71b542769d7" >> > ] >> > }, >> > >> > It doesn't have the camliPath attributes at all! >> > >> > Yet cammount can understand the other scheme, so long as I remember the >> sha1 >> > hash: >> > >> > % ls /tmp/cammount-dir/sha1-0609bc86627d0fa3067d45c09ede479c7e8643f8 >> > total 0 >> > 0 issue 0 motd 0 subdir/ >> > >> > But of course I don't want to get into the business of memorizing sha1 >> > hashes. And if I create a permanode, e.g: >> > >> > % ./bin/devcam put file --permanode /tmp/test-dir >> > >> > The permanode doesn't show up in FUSE hierarchy exported by cammount >> > (although it does show up in the web ui). >> > >> > Argh!!!! Very frustrating. >> > >> > What am I missing? >> > >> > Thanks, >> > >> > -- Ted >> > >> > >> > >> > -- >> > You received this message because you are subscribed to the Google >> Groups >> > "Camlistore" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> an >> > email to [email protected]. >> > For more options, visit https://groups.google.com/d/optout. >> > -- You received this message because you are subscribed to the Google Groups "Camlistore" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
