Hi Colin, Thank you for the helpful responses.
> <snip> Given the way that scrypt works, if any of the allocation ends up > paged out, the computation will effectively never finish; and (as mentioned > above) tarsnap tries to avoid using so much memory as to cause a heavy burden > on the system. Ah, that’s a crucial detail that I hadn’t realized. >> 7. Interactively reducing the size of the ZFS ARC with, for example, >> ‘sysctl vfs.zfs.arc_max=512000000‘ doesn’t help, perhaps because FreeBSD >> seems still to count the released ARC memory as wired. (top does show the >> ~700 MB reduction in ARC size, however.) > > I was about to suggest trying this. The fact that the released memory is not > unwired seems suspicious -- generally speaking, wired memory is not available > for any other purpose, so reducing the size of the ARC doesn't seem to have > the intended effect here. It's possible that I'm missing something -- I'm not > an expert on either ZFS or the FreeBSD VM system -- but it seems to me like > the wired memory should be released (perhaps over time) once arc_max is cut > down, since otherwise what's the point? Yes, it puzzled me too. I believe the option to reduce the ARC dynamically is new in FreeBSD 11, and it’s not something I’ve tried before. I didn’t wait around to see whether the memory would become unwired over time. > <snip> >> 8. However, if I reboot the host, and then immediately run the tarsnap >> --fsck command within the jail, it succeeds: >> [snip] >> >> 9. Hypothesis: Tarsnap/scrypt is erroneously deciding that there is >> insufficient memory available on the system in point (6) above. (Either >> that, or I am doing something very silly.) > > Yes, this is exactly what's happening -- the available-memory-detection is > very rough, in large part because the concept of "available memory" is not > even well defined. I did have a quick peek at the scrypt source yesterday, and saw that it’s trying to do its best with limited information. > Maybe we should add a --passphrase-force option which overrides tarsnap's > attempt to autodetect whether there's enough memory. (Does anyone have a > preference for what the option is named?) An option like that would be great. I had anticipated that perhaps one would need to specify the amount of memory for Tarsnap to use. In that case, borrowing the '--passphrase-mem maxmem' option from tarsnap-keygen would have worked nicely. Your suggestion is even easier. I have no preference for what the option is called, merely that it should exist :-) Thank you again for the assistance. -- Daniel
