I've just opened a WIP PR <https://github.com/perkeep/perkeep/pull/1282> against Perkeep in the hope of motivating a design discussion (about local blob storage, max blob sizes, and perhaps other topics). Here's the PR description. Cheers!
This PR is a preliminary sketch for a new blobserver type that uses files uploaded to it as their own storage. When you add a file to an fsbacked.Storage that's within the directory tree it controls, an entry is added to a database that maps between files and blobrefs; but the file's contents are not copied anywhere. When fetching the file's content blob later, the database directs the Storage to the right local file and the data is served from there. Adding files *outside* the directory tree, or adding any other kind of blob, fails over to another blobserver nested inside the fsbacked.Storage. This solves the problem of wanting to add a tree of large files (e.g., videos of my kids growing up) to a local Perkeep instance without storing all the data twice. This should be used only on directory trees whose files do not change, lest the blobrefs in the database become mismatched to their corresponding files. A number of other changes throughout Perkeep would be needed to make this truly useful. The io.Reader presented to a blobserver's ReceiveBlob method is usually (always?) some wrapper object (like checkHashReader <https://github.com/perkeep/perkeep/blob/d342b0e26632217a93a7b9a2ce85acca0c5cd00b/pkg/blobserver/receive.go#L71-L79>) that conceals the underlying *os.File, without which fsbacked.Storage cannot detect that a file within its tree is being uploaded. And in any case, Perkeep imposes rather a low limit on blob sizes <https://github.com/perkeep/perkeep/blob/d342b0e26632217a93a7b9a2ce85acca0c5cd00b/pkg/constants/constants.go#L22-L23> for this purpose. Presented for further discussion. -- You received this message because you are subscribed to the Google Groups "Perkeep" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/perkeep/CAEf8c4-Vgb5XX9vSoB9wh8dBTu3d_TZcD%3DBA1fa1Q6nQvKE0PA%40mail.gmail.com.
