Hi. short answers/remarks first, until I understand better what you're talking about.
On 14 October 2017 at 12:32, Michael Farr <[email protected]> wrote: > > I am having some difficulty with getting all my files into Camlistore - > mainly because I don't have a reliable way to sorting through them. My > files are currently in few different folder hierarchies that are all > undergoing regular change. I have a few reasons for wanting to pre-sort > through the files before going into Camlistore: > > Some of the files are large and only relevant for a short period of time. > These files will are associated with and sit near other files in the > hierarchy do need to be synced with Camlistore. So I manually camput files > in these folders to save space and bandwidth*^ > Some files need to be encrypted with an external tool before being synced > *~. Given the current lack of delete** I am keen to avoid creating an > unencrypted blob that I can’t easily remove later. > Some files are being changed frequently (many times per minute) and I do not > want too many copies of them. (large design files, media renderings, > software project binaries etc) > Some files are already encrypted/zipped or otherwise packaged in a format > that Camlistore will not be able to extract meaningful metadata from. The > metadata for that file exists within another file. If the sync is > completely automatic, it will be hard to detect when this situation arises > to know when to attach metadata to files like this. > > My current pre-camput process is quite cumbersome and I am not particularly > reliable at it, but I have only made some very basic initial steps with > Camlistore as a whole at using it so there is a lot I can learn. Are there > any steps I can take to avoid the aforementioned pre-processing problems > neatly within Camlistore? I only have one internet connected laptop and my > local USB drives are almost at capacity. I would prefer to use B2 or Wasabi > for future my blob storage needs. I'm not sure I understand exactly what you want to achieve in the end. In particular, if your Camlistore is not on a server accessible from anywhere anytime, why do you want to store things in Camlistore at all? What is your end goal? > These are some ideas that might help with the problem, though I am not sure > if any are sensible: > > Buy another USB disk to act as the local Camlistore blob store. Perform all > my preparation there, and then selectively sync that blob store to the cloud > blob store with "cond". I do not like the sound of this solution for lots > of reasons, but maybe it is the most straightforward way. > Use a separate tool that lets me apply metadata to the files before they go > into Camlistore (tmsu). I can then build scripts that sweep the file system > looking for files that are ready to be "camput"ted. (this idea actually > sounds quite terrible, but I thought i'd mention it anyway). > Import the file metadata into Camlistore without actually importing the file > data itself (I’m not sure if this is either possible or sensible). In > theory, I could then manipulate metadata directly within Camlistore and > synchronize only blobs that meet certain criteria. > Wait for gc.Collect to be fully implemented. > > > > If you also have this kind of issue, how do you work with Camlistore now? > Maybe one or more of the following: > > Import everything as soon as it is found and just suck up the extra data > costs. > > Manually pre-process and upload files/folders on their individual merit (ie: > as I do) Without knowing exactly what you want to do, I can't say that I have the same issue you do, but yes, when it comes to e.g. my pictures I first sort through them before uploading them with camput. > Custom scripts for running imports on various schedules. > Multiple Camlistore servers catering to different storage problems > Manually deleting blobs from the cloud storage systems. > Use other non-Camlistore systems for tricky situations (google drive etc) > All of the above > > Notes: > > 1. ** My reading of this issue is that gc.Collect doesn't currently do > anything: https://github.com/camlistore/camlistore/issues/792. yes, the garbage collector is not fully implemented and therefore not usable atm. > 2. *^ right now I have a very small local blob store for evaluation, but I > am planning on switching to b2/wasabi for blob store once I get everything > working. > > 3. *~ Possibly I do not understand Camlistore’s approach to encryption key > management, so I dont know enough to use it properly. My solution is to > encrypt the most sensitive files before they go into Camlistore even though > they are all encrypted again in the blobstore. Fyi, by default, nothing is encrypted. you'd have to specifically use the encrypt blobserver (which is still a "NO GUARANTEES" in terms of crypto afair). > My local systems are not > physically secure, so I need to encrypt data with a key hosted on a remote > system or one that is derived from a password that I remember. The local > key will protect all my files in the cloud store. The other keys protect my > sensitive files from both internal and external attacks. > > 4. I do not have a local permanent server with internet access. I can only > run Camlistore on my laptop. > > -- > You received this message because you are subscribed to the Google Groups > "Camlistore" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Camlistore" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
