On Fri, Aug 26, 2022 at 10:09 AM Mark Knecht <[email protected]> wrote: > > Obviously you can do what you are most comfortable with but to me a NAS > machine with a bunch of external drives does not sound very reliable. >
I would have thought the same, but messing around with LizardFS I've found that the USB3 hard drives never disconnect from their Pi4 hosts. I've had more issues with LSI HBAs dying. Of course I have host-level redundancy so if one Pi4 flakes out I can just reboot it with zero downtime - the master server is on an amd64 container. I only have about 2 drives per Pi right now as well - at this point I'd probably add more drives per host but I wanted to get out to 5-6 hosts first so that I get better performance especially during rebuilds. Gigabit networking is definitely a bottleneck, but with all the chunkservers on one switch they each get gigabit full duplex to all the others so rebuilds are still reasonably fast. To go with 10GbE you'd need hardware with better IO than a Pi4 I'd think, but the main bottleneck on the Pi4 I'm having is with encryption which hits the CPU. I am using dm-crypt for this which I think is hardware-optimized. I will say that zfs encryption is definitely not hardware-optimized and really gets CPU-bound, so I'm running zfs on top of dm-crypt. I should probably consider if dm-integrity makes more sense than zfs in this application. -- Rich
