There's a local cache for the local hashing too, though. If the file's stat metadata doesn't change at all (inode, mtime, size, ctime, etc) then it's not re-digested.
On Fri, May 3, 2019 at 12:17 PM Eric Drechsel <[email protected]> wrote: > As I understand, puts of existing blobs don't actually transfer the bytes, > but since most of the time (with local transfer) is taken by hashing that > doesn't speed things up much. > > The only way I can think of to speed that up would be to somehow cache the > file hashes (doesn't zfs support storing hashes? maybe that could be used > as a fast path for hashing?) > > On Fri, May 3, 2019 at 11:13 AM Ian Denhardt <[email protected]> wrote: > >> Hey All, >> >> I have about 2TB of files that I'm looking at importing into perkeep. I >> have a couple questions. >> >> First, do others have experience they can share re: how perkeep performs >> holding this much data? From what I've read it sounds like >> architecturally it should be manageable, but I'd like to know if anyone >> can say how that's worked out in practice for them. >> >> Assuming this is realistic, I have some logistical questions about >> getting the data in there in the first place. >> >> I left a pk-put going on a large sub-tree last night, and came back to >> it today. It had spent about 12 hours copying things, finally running in >> to some hiccough uploading a particular file (I don't have the error >> message recorded, but it was something along the lines of "server did >> not receive blob"). Trying to upload that file again worked fine, so I >> assume some transient thing. >> >> During the transfer, usage on the drives holding the blobs grew by about >> 80 GiB. This is transferring data between two hard drives connected to >> the same machine via USB 3.0. Questions: >> >> 1. Is that kind of performance normal for pk-put? >> 2. Is there currently any way to do a "resumable" version of pk-put, >> where it can quickly pick up where it left off? >> >> If the answer to (2) is no, I might be interested in contributing such a >> feature, and would appreciate pointers as to where to start. >> >> Thanks. >> >> -Ian >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Perkeep" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> For more options, visit https://groups.google.com/d/optout. >> > > > -- > best, Eric > eric.pdxhub.org > > -- > You received this message because you are subscribed to the Google Groups > "Perkeep" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Perkeep" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
