On 9 August 2016 at 20:38, magge <[email protected]> wrote: > First post on the list!
Welcome. > I like the ideas behind camlistore and I'm toying with an idea where > camlistore might be the backend for my collection of photos/videos including > a representation of the process from originally snapped photos into whatever > version(s) I end up with as the final result. The process could be > combination of photos to HDR/panoramas, just tuning colors or GIMPing it up > in other ways. From what I understand camlistore will offer me the ability > to see all versions of a given file by means of claims to a permanode which > would actually replace the camliContent / bytes of the file. That helps a > lot for single file projects. Additionally I would need to make some sort of > grouping concept, which I guess I can do with attrs. Somehow. Anyway that's > what I'll try to accomplish, if you should have any suggestions feel free to > post them! :) > > I've followed camlistore for a while but it's only the last week that I've > actually installed it and it's now been working a few days while I'm trying > to dump my entire photos folder into it. I have some questions that have > crossed my mind as I'm seeing how it works. I'm using the 0.9 release now. We also have more recent releases if you feel a bit more adventurous: https://camlistore.org/doc/release/monthly I should have cut a new one by now, but there's one bug in particular I'd like to see fixed first. > - The blobs are adressed by their SHA1 (etc) digest. There is a theoretical > possibility that two different blobs have the same digest, right? Is that > handled somehow? Yes, but I don't think accidental collisions should be that much of a worry yet. https://en.wikipedia.org/wiki/SHA-1 https://github.com/camlistore/camlistore/issues/537 > - I'm using FUSE/cammount and my system seems to max out at about 700 KB/s > transferring to camlistore. It seems to be transferring the big blobs, I > guess the actual data blobs, quite fast, then it's spending some time > creating smaller blobs (seems to be the bytes-indexes) after that. If you're creating permanodes for your files, then there's at least one permanode + one additional claim that attaches the file to it, for each file you're writing. Plus the file schema blob anyway, and the bytes schema blobs too. > Seems > it's maxing out the CPU as it's being slow (quite old hardware). Finally it > spends some more time "packing file". I will try the monthly build too, but > are there any typical things I can do to speed it up? I think some of it is due to the extreme verbosity. If you see a ton of log messages in the terminal where you started cammount, you can look for the most recurrent message, comment it out in the code, and see if that has an effect. I thought I had dealt with that with some verbose flags, but maybe not completely. For example, I see this log.Printf("CAMLI nodeReader READ on %v: %#v", nr.n.blobref, req) from pkg/fs/fs.go showing up a lot. The packing you're seeing is because you're using the default disk blobserver: blobpacked. > - I see that most blobs from a big file will be around ~64KB but their sizes > vary a lot. Also among the first chunks it seems typical to see one of > 200KB+. What is the strategy to chunk up the file? I read somewhere that > it's up to the client so I guess maybe the question is how cammount does it. > With regard to deduplication I suppose the that strategy can be quite > important, like if some header was changed in length but the rest of the > file was the same, it could mean with ideal chunks only 1 chunk (the header > chunk) was new and the rest unchanged, but with less ideal chunking all > chunks would be new... I think the clients could in theory decide, but afaik, all of them end up letting https://camlistore.org/pkg/schema#WriteFileMap do that job one way or the other. You can see how it's done there. See in particular the comments at the top of https://camlistore.org/pkg/schema/filewriter.go > Sorry for the random ramble. I hope somewhere down the line I'll get more > into both go and camlistore and be able to contribute and do something with > my mentioned idea. > > > > > > > > > > -- > You received this message because you are subscribed to the Google Groups > "Camlistore" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Camlistore" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
