I am having some difficulty with getting all my files into Camlistore -
mainly because I don't have a reliable way to sorting through them. My
files are currently in few different folder hierarchies that are all
undergoing regular change. I have a few reasons for wanting to pre-sort
through the files before going into Camlistore:
1. Some of the files are large and only relevant for a short period of
time. These files will are associated with and sit near other files in the
hierarchy do need to be synced with Camlistore. So I manually camput files
in these folders to save space and bandwidth*^
2. Some files need to be encrypted with an external tool before being
synced *~. Given the current lack of delete** I am keen to avoid creating
an unencrypted blob that I can’t easily remove later.
3. Some files are being changed frequently (many times per minute) and I
do not want too many copies of them. (large design files, media renderings,
software project binaries etc)
4. Some files are already encrypted/zipped or otherwise packaged in a
format that Camlistore will not be able to extract meaningful metadata
from. The metadata for that file exists within another file. If the sync
is completely automatic, it will be hard to detect when this situation
arises to know when to attach metadata to files like this.
My current pre-camput process is quite cumbersome and I am not particularly
reliable at it, but I have only made some very basic initial steps with
Camlistore as a whole at using it so there is a lot I can learn. Are there
any steps I can take to avoid the aforementioned pre-processing problems
neatly within Camlistore? I only have one internet connected laptop and my
local USB drives are almost at capacity. I would prefer to use B2 or
Wasabi for future my blob storage needs.
These are some ideas that might help with the problem, though I am not sure
if any are sensible:
1. Buy another USB disk to act as the local Camlistore blob store.
Perform all my preparation there, and then selectively sync that blob store
to the cloud blob store with "cond". I do not like the sound of this
solution for lots of reasons, but maybe it is the most straightforward way.
2. Use a separate tool that lets me apply metadata to the files before
they go into Camlistore (tmsu). I can then build scripts that sweep the
file system looking for files that are ready to be "camput"ted. (this idea
actually sounds quite terrible, but I thought i'd mention it anyway).
3. Import the file metadata into Camlistore without actually importing
the file data itself (I’m not sure if this is either possible or
sensible). In theory, I could then manipulate metadata directly within
Camlistore and synchronize only blobs that meet certain criteria.
4. Wait for gc.Collect to be fully implemented.
If you also have this kind of issue, how do you work with Camlistore now?
Maybe one or more of the following:
1. Import everything as soon as it is found and just suck up the extra
data costs.
2. Manually pre-process and upload files/folders on their individual
merit (ie: as I do)
3. Custom scripts for running imports on various schedules.
4. Multiple Camlistore servers catering to different storage problems
5. Manually deleting blobs from the cloud storage systems.
6. Use other non-Camlistore systems for tricky situations (google drive
etc)
7. All of the above
Notes:
1. ** My reading of this issue is that gc.Collect doesn't currently do
anything: https://github.com/camlistore/camlistore/issues/792.
2. *^ right now I have a very small local blob store for evaluation, but I
am planning on switching to b2/wasabi for blob store once I get everything
working.
3. *~ Possibly I do not understand Camlistore’s approach to encryption key
management, so I dont know enough to use it properly. My solution is to
encrypt the most sensitive files before they go into Camlistore even though
they are all encrypted again in the blobstore. My local systems are not
physically secure, so I need to encrypt data with a key hosted on a remote
system or one that is derived from a password that I remember. The local
key will protect all my files in the cloud store. The other keys protect
my sensitive files from both internal and external attacks.
4. I do not have a local permanent server with internet access. I can
only run Camlistore on my laptop.
--
You received this message because you are subscribed to the Google Groups
"Camlistore" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to camlistore+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
