Ah, good to know. So it should resume fast then? On Fri, May 3, 2019 at 12:18 PM Brad Fitzpatrick <[email protected]> wrote:
> There's a local cache for the local hashing too, though. If the file's > stat metadata doesn't change at all (inode, mtime, size, ctime, etc) then > it's not re-digested. > > > On Fri, May 3, 2019 at 12:17 PM Eric Drechsel <[email protected]> wrote: > >> As I understand, puts of existing blobs don't actually transfer the >> bytes, but since most of the time (with local transfer) is taken by hashing >> that doesn't speed things up much. >> >> The only way I can think of to speed that up would be to somehow cache >> the file hashes (doesn't zfs support storing hashes? maybe that could be >> used as a fast path for hashing?) >> >> On Fri, May 3, 2019 at 11:13 AM Ian Denhardt <[email protected]> wrote: >> >>> Hey All, >>> >>> I have about 2TB of files that I'm looking at importing into perkeep. I >>> have a couple questions. >>> >>> First, do others have experience they can share re: how perkeep performs >>> holding this much data? From what I've read it sounds like >>> architecturally it should be manageable, but I'd like to know if anyone >>> can say how that's worked out in practice for them. >>> >>> Assuming this is realistic, I have some logistical questions about >>> getting the data in there in the first place. >>> >>> I left a pk-put going on a large sub-tree last night, and came back to >>> it today. It had spent about 12 hours copying things, finally running in >>> to some hiccough uploading a particular file (I don't have the error >>> message recorded, but it was something along the lines of "server did >>> not receive blob"). Trying to upload that file again worked fine, so I >>> assume some transient thing. >>> >>> During the transfer, usage on the drives holding the blobs grew by about >>> 80 GiB. This is transferring data between two hard drives connected to >>> the same machine via USB 3.0. Questions: >>> >>> 1. Is that kind of performance normal for pk-put? >>> 2. Is there currently any way to do a "resumable" version of pk-put, >>> where it can quickly pick up where it left off? >>> >>> If the answer to (2) is no, I might be interested in contributing such a >>> feature, and would appreciate pointers as to where to start. >>> >>> Thanks. >>> >>> -Ian >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Perkeep" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> -- >> best, Eric >> eric.pdxhub.org >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Perkeep" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> For more options, visit https://groups.google.com/d/optout. >> > -- > You received this message because you are subscribed to the Google Groups > "Perkeep" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- best, Eric eric.pdxhub.org -- You received this message because you are subscribed to the Google Groups "Perkeep" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
