It still needs to do some stats and readdirs and local key/value database lookups to find out where to resume, but yeah.
On Fri, May 3, 2019 at 12:19 PM Eric Drechsel <[email protected]> wrote: > Ah, good to know. So it should resume fast then? > > On Fri, May 3, 2019 at 12:18 PM Brad Fitzpatrick <[email protected]> wrote: > >> There's a local cache for the local hashing too, though. If the file's >> stat metadata doesn't change at all (inode, mtime, size, ctime, etc) then >> it's not re-digested. >> >> >> On Fri, May 3, 2019 at 12:17 PM Eric Drechsel <[email protected]> wrote: >> >>> As I understand, puts of existing blobs don't actually transfer the >>> bytes, but since most of the time (with local transfer) is taken by hashing >>> that doesn't speed things up much. >>> >>> The only way I can think of to speed that up would be to somehow cache >>> the file hashes (doesn't zfs support storing hashes? maybe that could be >>> used as a fast path for hashing?) >>> >>> On Fri, May 3, 2019 at 11:13 AM Ian Denhardt <[email protected]> wrote: >>> >>>> Hey All, >>>> >>>> I have about 2TB of files that I'm looking at importing into perkeep. I >>>> have a couple questions. >>>> >>>> First, do others have experience they can share re: how perkeep performs >>>> holding this much data? From what I've read it sounds like >>>> architecturally it should be manageable, but I'd like to know if anyone >>>> can say how that's worked out in practice for them. >>>> >>>> Assuming this is realistic, I have some logistical questions about >>>> getting the data in there in the first place. >>>> >>>> I left a pk-put going on a large sub-tree last night, and came back to >>>> it today. It had spent about 12 hours copying things, finally running in >>>> to some hiccough uploading a particular file (I don't have the error >>>> message recorded, but it was something along the lines of "server did >>>> not receive blob"). Trying to upload that file again worked fine, so I >>>> assume some transient thing. >>>> >>>> During the transfer, usage on the drives holding the blobs grew by about >>>> 80 GiB. This is transferring data between two hard drives connected to >>>> the same machine via USB 3.0. Questions: >>>> >>>> 1. Is that kind of performance normal for pk-put? >>>> 2. Is there currently any way to do a "resumable" version of pk-put, >>>> where it can quickly pick up where it left off? >>>> >>>> If the answer to (2) is no, I might be interested in contributing such a >>>> feature, and would appreciate pointers as to where to start. >>>> >>>> Thanks. >>>> >>>> -Ian >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Perkeep" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> >>> -- >>> best, Eric >>> eric.pdxhub.org >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Perkeep" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Perkeep" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> For more options, visit https://groups.google.com/d/optout. >> > > > -- > best, Eric > eric.pdxhub.org > > -- > You received this message because you are subscribed to the Google Groups > "Perkeep" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Perkeep" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
