As I understand, puts of existing blobs don't actually transfer the bytes, but since most of the time (with local transfer) is taken by hashing that doesn't speed things up much.
The only way I can think of to speed that up would be to somehow cache the file hashes (doesn't zfs support storing hashes? maybe that could be used as a fast path for hashing?) On Fri, May 3, 2019 at 11:13 AM Ian Denhardt <[email protected]> wrote: > Hey All, > > I have about 2TB of files that I'm looking at importing into perkeep. I > have a couple questions. > > First, do others have experience they can share re: how perkeep performs > holding this much data? From what I've read it sounds like > architecturally it should be manageable, but I'd like to know if anyone > can say how that's worked out in practice for them. > > Assuming this is realistic, I have some logistical questions about > getting the data in there in the first place. > > I left a pk-put going on a large sub-tree last night, and came back to > it today. It had spent about 12 hours copying things, finally running in > to some hiccough uploading a particular file (I don't have the error > message recorded, but it was something along the lines of "server did > not receive blob"). Trying to upload that file again worked fine, so I > assume some transient thing. > > During the transfer, usage on the drives holding the blobs grew by about > 80 GiB. This is transferring data between two hard drives connected to > the same machine via USB 3.0. Questions: > > 1. Is that kind of performance normal for pk-put? > 2. Is there currently any way to do a "resumable" version of pk-put, > where it can quickly pick up where it left off? > > If the answer to (2) is no, I might be interested in contributing such a > feature, and would appreciate pointers as to where to start. > > Thanks. > > -Ian > > -- > You received this message because you are subscribed to the Google Groups > "Perkeep" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- best, Eric eric.pdxhub.org -- You received this message because you are subscribed to the Google Groups "Perkeep" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
