Re: [gentoo-user] Suggestions for NAS appliance?
On 14/11/21 12:52 am, Rich Freeman wrote: > On Sat, Nov 13, 2021 at 12:35 AM William Kenworthy wrote: >> Look at the odroid HC4 - I am using 5x the older HC2 version for moosefs >> - they are USB3 based but work well in this application. They are arm32 >> but 64bit is not needed. > I like the idea behind the HC series Odroids, but being limited to 1-2 > drives per node seems a bit contraining. With the USB3 approach there > really is no limit to how many drives I can put on a node, as long as > I don't mind the performance drop. I'm more concerned with static > storage capacity in this case. If you're using 1Gbps ethernet then I > guess two drives is already going to saturate the network if they're > able to read sequentially. > The HC2 works fine with a USB3 disk added to an existing sata drive (I was doing that for awhile using SATA->USB3 adaptors) - however it looks like the HC4 has dropped USB3 and only USB2 is available - backward step in my view. BillK
Re: [gentoo-user] Suggestions for NAS appliance?
On 13/11/2021 16:52, Rich Freeman wrote: On Sat, Nov 13, 2021 at 12:35 AM William Kenworthy wrote: Look at the odroid HC4 - I am using 5x the older HC2 version for moosefs - they are USB3 based but work well in this application. They are arm32 but 64bit is not needed. I like the idea behind the HC series Odroids, but being limited to 1-2 drives per node seems a bit contraining. With the USB3 approach there really is no limit to how many drives I can put on a node, as long as I don't mind the performance drop. I'm more concerned with static storage capacity in this case. If you're using 1Gbps ethernet then I guess two drives is already going to saturate the network if they're able to read sequentially. Note that the md-raid advice is "do NOT use USB. We don't understand why, but bad things seem to happen". Even when explicitly designed for it, multiplexing drives on a single comms channel isn't always a good idea... Years ago I built a CD copy station. Can't remember the exact layout, but it was something like three CD-writers and a hard drive on PATA-0 and PATA-1. I think the CD-writers achieved something like 0.5x speeds! So we bought two add-in cards, with the hd and first CD-writer were on PATA-0, and one CD-writer each on PATA-1, PATA-2 and PATA-3. They FLEW! They were something like 32x drives and really achieved a full write in about 2 minutes! Cheers, Wol
Re: [gentoo-user] Suggestions for NAS appliance?
On Sat, Nov 13, 2021 at 12:35 AM William Kenworthy wrote: > > Look at the odroid HC4 - I am using 5x the older HC2 version for moosefs > - they are USB3 based but work well in this application. They are arm32 > but 64bit is not needed. I like the idea behind the HC series Odroids, but being limited to 1-2 drives per node seems a bit contraining. With the USB3 approach there really is no limit to how many drives I can put on a node, as long as I don't mind the performance drop. I'm more concerned with static storage capacity in this case. If you're using 1Gbps ethernet then I guess two drives is already going to saturate the network if they're able to read sequentially. -- Rich
Re: [gentoo-user] Suggestions for NAS appliance?
On 13/11/21 5:56 am, Rich Freeman wrote: > On Wed, Nov 10, 2021 at 11:06 PM Mark Knecht wrote: >> >> Not a recommendation precisely but there's a guy on YouTube named Jeff >> Geerling that's doing a lot of that sort of thing using a Raspberry Pi and >> multiple SATA drives. I've just built my first RP4 box aimed at >> astrophotography and I'm pretty impressed with how well the Pi works. My >> next project will likely be some sort of NAS box using a second Pi4 with an >> M.2 system drive. >> > I run LizardFS and at this point Pi4s are my preferred hardware for > storage nodes. However, I don't deal with much IOPS. I tend to use > USB3 hard drives for convenience/cost. Really though SATA on a Pi4 > wouldn't be super-ideal anyway due to the lack of PCIe (I think it > lacks it at least). You can find ARM SBCs that have PCIe capable of > handling an HBA which are probably better if you want a bunch of SATA > drives, though those have their downsides. If you're serious about > IOPS I'm not sure anything cheap will do the trick. Look at the odroid HC4 - I am using 5x the older HC2 version for moosefs - they are USB3 based but work well in this application. They are arm32 but 64bit is not needed. > I would definitely avoid Pi2/3 for this due to the combo of 100MBps > networking and USB2 and a lot of the IO goes through USB2 in the first > place. It is just not a very good setup for IO at all, and there are > much better alternatives. The Pi4 though is pretty solid as long as > you don't mind USB3 (and it has two hosts so you can basically run 4 > spinning disks all-out without a performance hit until you get to the > network at least). I have a pi3B - bad idea to use this for any DFS - I tried... I am using an Odroid C4 for directly connected USB3 disks - works well. > Gigabit network is its own bottleneck for any kind of storage. I'm > too cheap to try to use anything better, but anybody doing serious DFS > is going to want 10Gbps, or often dual 10Gbps.
Re: [gentoo-user] Suggestions for NAS appliance?
On Fri, Nov 12, 2021 at 4:55 PM Rich Freeman wrote: > > On Fri, Nov 12, 2021 at 5:57 PM Mark Knecht wrote: > > > > which I don't think is available yet but will run in the $250 > > range without the drives. It appears that the motherboard > > they designed takes the PCIe to a card with a PCIe-to-SATA > > controller which is how you get better performance. > > With the USB3 solution your $40 SBC can directly access a hard drive. > Now we're talking about adding $250 of hardware to get to SATA. > > Honestly, if you're going to start spending that much I'd really > question why we're not just using a PC. > > I would love to see more reasonably-priced ARM options with decent IO > and RAM. If you want ARM and 32GB of RAM good luck finding something > for under $1k. For anything that runs 24x7 the power savings are > quite significant with ARM, but you definitely give up performance and > IO and of course general flexibility, If you're going to start > spending hundreds of dollars on ARM unless it is as some kind of > prototyping setup I'd really question why you wouldn't just use > inexpensive PC hardware. Get something with integrated graphics and > you have lots of PCIe lanes available for IO. You can get used HBAs > fairly inexpensively as well, though I do question the reliability of > some of that stuff (used). > > One other thing, I've used LSI HBAs with Rockpro64 SBCs (which have > PCIe), and the HBAs use a ton of power. The RK3399 might use very > little power (esp idle), but that LSI HBA is practically a space > heater even when idle. Maybe their newer stuff is more efficient, but > you're not going to get those for $30 on eBay. > > -- > Rich > No disagreements at all. I was only providing 1) something for the OP to look at, and 2) info on your point/question/observation about RP4 having or not having PCIe. As for YouTube 'influencers' I take them with a grain of salt. This guy was given preproduction product which almost certainly was free and given so as to produce positive press. Mark
Re: [gentoo-user] Suggestions for NAS appliance?
On Fri, Nov 12, 2021 at 5:57 PM Mark Knecht wrote: > > which I don't think is available yet but will run in the $250 > range without the drives. It appears that the motherboard > they designed takes the PCIe to a card with a PCIe-to-SATA > controller which is how you get better performance. With the USB3 solution your $40 SBC can directly access a hard drive. Now we're talking about adding $250 of hardware to get to SATA. Honestly, if you're going to start spending that much I'd really question why we're not just using a PC. I would love to see more reasonably-priced ARM options with decent IO and RAM. If you want ARM and 32GB of RAM good luck finding something for under $1k. For anything that runs 24x7 the power savings are quite significant with ARM, but you definitely give up performance and IO and of course general flexibility, If you're going to start spending hundreds of dollars on ARM unless it is as some kind of prototyping setup I'd really question why you wouldn't just use inexpensive PC hardware. Get something with integrated graphics and you have lots of PCIe lanes available for IO. You can get used HBAs fairly inexpensively as well, though I do question the reliability of some of that stuff (used). One other thing, I've used LSI HBAs with Rockpro64 SBCs (which have PCIe), and the HBAs use a ton of power. The RK3399 might use very little power (esp idle), but that LSI HBA is practically a space heater even when idle. Maybe their newer stuff is more efficient, but you're not going to get those for $30 on eBay. -- Rich
Re: [gentoo-user] Suggestions for NAS appliance?
I'm certainly not interested in building Gentoo on an RP4 but I'd be happy to do some limited testing. Not a good target for cross-compiling. Way outside my comfort zone and all my machines are now Ubuntu anyway. I did not see an 'alpha9' so I downloaded and unzst-ed the 'latest' version. I'll have to sort through my pile of 32GB Micro SD's and pick one to burn the image file on, possibly this weekend. Thanks for the info. Good to see interesting stuff getting worked on. Mark On Fri, Nov 12, 2021 at 2:19 PM Michael Jones wrote: > > > > On Fri, Nov 12, 2021 at 11:31 AM Laurence Perkins > wrote: >> >> >> >> > -Original Message- >> > From: Peter Humphrey >> > Sent: Friday, November 12, 2021 12:19 AM >> > To: gentoo-user@lists.gentoo.org >> > Subject: Re: [gentoo-user] Suggestions for NAS appliance? >> > >> > On Thursday, 11 November 2021 22:43:17 GMT Mark Knecht wrote: >> > > On Thu, Nov 11, 2021 at 3:21 PM Laurence Perkins >> > > >> > wrote: >> > > > The genpi64 project is supposedly back up and running, so you can >> > > > get newer images that need fewer updates again. >> > > > >> > > > >> > > > >> > > > From: Mark Knecht >> > > > Sent: Wednesday, November 10, 2021 8:06 PM >> > > > To: Gentoo User >> > > > Subject: Re: [gentoo-user] Suggestions for NAS appliance? >> > > >> > > Do you have a better URL than I have? >> > > >> > > https://github.com/sakaki-/gentoo-on-rpi-64bit >> > >> > I gogled for 'genpi64 project' and got this: https://github.com/GenPi64 . >> > >> > -- >> > Regards, >> > Peter. >> >> Yes, that one. I think the build.dist is the repo you want. I don't know >> if they're providing finished images yet or not, but it should at least be >> capable of building them for you on something more powerful than the pi >> itself. >> >> LMP > > > I am one of the contributors to the new/reincarnated GenPi64 project. > > The build.dist repository contains our image builder, and you should be able > to run it on the raspberry pi itself, or on an x86_64 machine. I recommend > using the alpha9 branch of the repository, it has the latest improvements. > > It basically downloads an aarch64 stage3, and chroots into it and runs > various configuration commands, including emerge to install new packages. > > The builder doesn't currently setup a distcc cross-compiler for you, but > there are patches available for that if you really want to give that a try. > It's tricky to set up. > > You might have some trouble with the kernel build, or the final image > partition setup, but we're happy to help out if you just open an issue on > github. > > We do have hosted images here: https://packages.genpi64.com/ -- be warned, > these are not considered high quality yet. But they work well enough for the > several people who've reported back to us. > and our readme here: https://github.com/GenPi64/gentoo-on-rpi-64bit > > We're actively looking for more contributors. Currently it's just 3 very busy > people, so any help is good help. >
Re: [gentoo-user] Suggestions for NAS appliance?
On Fri, Nov 12, 2021 at 2:56 PM Rich Freeman wrote: > > On Wed, Nov 10, 2021 at 11:06 PM Mark Knecht wrote: > > > > > > Not a recommendation precisely but there's a guy on YouTube named Jeff > > Geerling that's doing a lot of that sort of thing using a Raspberry Pi and > > multiple SATA drives. I've just built my first RP4 box aimed at > > astrophotography and I'm pretty impressed with how well the Pi works. My > > next project will likely be some sort of NAS box using a second Pi4 with an > > M.2 system drive. > > > > I run LizardFS and at this point Pi4s are my preferred hardware for > storage nodes. However, I don't deal with much IOPS. I tend to use > USB3 hard drives for convenience/cost. Really though SATA on a Pi4 > wouldn't be super-ideal anyway due to the lack of PCIe (I think it > lacks it at least). You can find ARM SBCs that have PCIe capable of > handling an HBA which are probably better if you want a bunch of SATA > drives, though those have their downsides. If you're serious about > IOPS I'm not sure anything cheap will do the trick. > > I would definitely avoid Pi2/3 for this due to the combo of 100MBps > networking and USB2 and a lot of the IO goes through USB2 in the first > place. It is just not a very good setup for IO at all, and there are > much better alternatives. The Pi4 though is pretty solid as long as > you don't mind USB3 (and it has two hosts so you can basically run 4 > spinning disks all-out without a performance hit until you get to the > network at least). > > Gigabit network is its own bottleneck for any kind of storage. I'm > too cheap to try to use anything better, but anybody doing serious DFS > is going to want 10Gbps, or often dual 10Gbps. > > -- > Rich > My understanding (from Geerling's video as well as looking at the Pi Compute Module manual) is that the standard RP4 has PCI express but it's hooked to the USB3 chip. In the case of the Pi 4 Compute module the PCI express is available to be used by the motherboard that you plug the compute module into. https://datasheets.raspberrypi.com/cm4/cm4-datasheet.pdf The NAS box Geering was demoing in this video: https://www.youtube.com/watch?v=YtdVotS3018=1023s is a kickstarter project https://pibox.io/ which I don't think is available yet but will run in the $250 range without the drives. It appears that the motherboard they designed takes the PCIe to a card with a PCIe-to-SATA controller which is how you get better performance. This designed is GB Ethernet which is probably the actual performance bottleneck. Cheers, Mark
Re: [gentoo-user] Suggestions for NAS appliance?
On Wed, Nov 10, 2021 at 11:06 PM Mark Knecht wrote: > > > Not a recommendation precisely but there's a guy on YouTube named Jeff > Geerling that's doing a lot of that sort of thing using a Raspberry Pi and > multiple SATA drives. I've just built my first RP4 box aimed at > astrophotography and I'm pretty impressed with how well the Pi works. My next > project will likely be some sort of NAS box using a second Pi4 with an M.2 > system drive. > I run LizardFS and at this point Pi4s are my preferred hardware for storage nodes. However, I don't deal with much IOPS. I tend to use USB3 hard drives for convenience/cost. Really though SATA on a Pi4 wouldn't be super-ideal anyway due to the lack of PCIe (I think it lacks it at least). You can find ARM SBCs that have PCIe capable of handling an HBA which are probably better if you want a bunch of SATA drives, though those have their downsides. If you're serious about IOPS I'm not sure anything cheap will do the trick. I would definitely avoid Pi2/3 for this due to the combo of 100MBps networking and USB2 and a lot of the IO goes through USB2 in the first place. It is just not a very good setup for IO at all, and there are much better alternatives. The Pi4 though is pretty solid as long as you don't mind USB3 (and it has two hosts so you can basically run 4 spinning disks all-out without a performance hit until you get to the network at least). Gigabit network is its own bottleneck for any kind of storage. I'm too cheap to try to use anything better, but anybody doing serious DFS is going to want 10Gbps, or often dual 10Gbps. -- Rich
Re: [gentoo-user] Suggestions for NAS appliance?
On Fri, Nov 12, 2021 at 11:31 AM Laurence Perkins wrote: > > > > -Original Message- > > From: Peter Humphrey > > Sent: Friday, November 12, 2021 12:19 AM > > To: gentoo-user@lists.gentoo.org > > Subject: Re: [gentoo-user] Suggestions for NAS appliance? > > > > On Thursday, 11 November 2021 22:43:17 GMT Mark Knecht wrote: > > > On Thu, Nov 11, 2021 at 3:21 PM Laurence Perkins > > > > > wrote: > > > > The genpi64 project is supposedly back up and running, so you can > > > > get newer images that need fewer updates again. > > > > > > > > > > > > > > > > From: Mark Knecht > > > > Sent: Wednesday, November 10, 2021 8:06 PM > > > > To: Gentoo User > > > > Subject: Re: [gentoo-user] Suggestions for NAS appliance? > > > > > > Do you have a better URL than I have? > > > > > > https://github.com/sakaki-/gentoo-on-rpi-64bit > > > > I gogled for 'genpi64 project' and got this: https://github.com/GenPi64 > . > > > > -- > > Regards, > > Peter. > > Yes, that one. I think the build.dist is the repo you want. I don't know > if they're providing finished images yet or not, but it should at least be > capable of building them for you on something more powerful than the pi > itself. > > LMP > I am one of the contributors to the new/reincarnated GenPi64 project. The build.dist repository contains our image builder, and you should be able to run it on the raspberry pi itself, or on an x86_64 machine. I recommend using the alpha9 branch of the repository, it has the latest improvements. It basically downloads an aarch64 stage3, and chroots into it and runs various configuration commands, including emerge to install new packages. The builder doesn't currently setup a distcc cross-compiler for you, but there are patches available for that if you really want to give that a try. It's tricky to set up. You might have some trouble with the kernel build, or the final image partition setup, but we're happy to help out if you just open an issue on github. We do have hosted images here: https://packages.genpi64.com/ -- be warned, these are not considered high quality yet. But they work well enough for the several people who've reported back to us. and our readme here: https://github.com/GenPi64/gentoo-on-rpi-64bit We're actively looking for more contributors. Currently it's just 3 very busy people, so any help is good help.
RE: [gentoo-user] Suggestions for NAS appliance?
> -Original Message- > From: Peter Humphrey > Sent: Friday, November 12, 2021 12:19 AM > To: gentoo-user@lists.gentoo.org > Subject: Re: [gentoo-user] Suggestions for NAS appliance? > > On Thursday, 11 November 2021 22:43:17 GMT Mark Knecht wrote: > > On Thu, Nov 11, 2021 at 3:21 PM Laurence Perkins > > > wrote: > > > The genpi64 project is supposedly back up and running, so you can > > > get newer images that need fewer updates again. > > > > > > > > > > > > From: Mark Knecht > > > Sent: Wednesday, November 10, 2021 8:06 PM > > > To: Gentoo User > > > Subject: Re: [gentoo-user] Suggestions for NAS appliance? > > > > Do you have a better URL than I have? > > > > https://github.com/sakaki-/gentoo-on-rpi-64bit > > I gogled for 'genpi64 project' and got this: https://github.com/GenPi64 . > > -- > Regards, > Peter. Yes, that one. I think the build.dist is the repo you want. I don't know if they're providing finished images yet or not, but it should at least be capable of building them for you on something more powerful than the pi itself. LMP
Re: [gentoo-user] Suggestions for NAS appliance?
On Friday, 12 November 2021 08:19:15 GMT Peter Humphrey wrote: > On Thursday, 11 November 2021 22:43:17 GMT Mark Knecht wrote: > > On Thu, Nov 11, 2021 at 3:21 PM Laurence Perkins > > wrote: > > > The genpi64 project is supposedly back up and running, so you can get > > > newer images that need fewer updates again. > > > > > > > > > > > > From: Mark Knecht > > > Sent: Wednesday, November 10, 2021 8:06 PM > > > To: Gentoo User > > > Subject: Re: [gentoo-user] Suggestions for NAS appliance? > > > > Do you have a better URL than I have? > > > > https://github.com/sakaki-/gentoo-on-rpi-64bit > > I gogled for 'genpi64 project' and got this: https://github.com/GenPi64 . Still no mention of the Pi 400 though. -- Regards, Peter.
Re: [gentoo-user] Suggestions for NAS appliance?
On Thursday, 11 November 2021 22:43:17 GMT Mark Knecht wrote: > On Thu, Nov 11, 2021 at 3:21 PM Laurence Perkins wrote: > > The genpi64 project is supposedly back up and running, so you can get > > newer images that need fewer updates again. > > > > > > > > From: Mark Knecht > > Sent: Wednesday, November 10, 2021 8:06 PM > > To: Gentoo User > > Subject: Re: [gentoo-user] Suggestions for NAS appliance? > > Do you have a better URL than I have? > > https://github.com/sakaki-/gentoo-on-rpi-64bit I gogled for 'genpi64 project' and got this: https://github.com/GenPi64 . -- Regards, Peter.
Re: [gentoo-user] Suggestions for NAS appliance?
On Thu, Nov 11, 2021 at 3:21 PM Laurence Perkins wrote: > > The genpi64 project is supposedly back up and running, so you can get newer > images that need fewer updates again. > > > > From: Mark Knecht > Sent: Wednesday, November 10, 2021 8:06 PM > To: Gentoo User > Subject: Re: [gentoo-user] Suggestions for NAS appliance? > Do you have a better URL than I have? https://github.com/sakaki-/gentoo-on-rpi-64bit
RE: [gentoo-user] Suggestions for NAS appliance?
The genpi64 project is supposedly back up and running, so you can get newer images that need fewer updates again. From: Mark Knecht Sent: Wednesday, November 10, 2021 8:06 PM To: Gentoo User Subject: Re: [gentoo-user] Suggestions for NAS appliance? On Wed, Nov 10, 2021, 7:49 PM Alan Grimes mailto:alonz...@verizon.net>> wrote: Hey, my old NAS box croaked the other day. I had to spend $400 ond hardware and software to recover my data but the issue now is finding a good new NAS solution. Use is basically media server + backup provider. I was happy with my old Netgear ReadyNAS idiot-proof box until it died. Any experience with products on the market? Not a recommendation precisely but there's a guy on YouTube named Jeff Geerling that's doing a lot of that sort of thing using a Raspberry Pi and multiple SATA drives. I've just built my first RP4 box aimed at astrophotography and I'm pretty impressed with how well the Pi works. My next project will likely be some sort of NAS box using a second Pi4 with an M.2 system drive. There is a bootable Gentoo image for the Pi4. It was pretty zippy but it's a year and a half old so if you're fixated on Gentoo you'd have a lot of work to do vs using Ubuntu server or something prebuilt. HTH, Mark
Re: [gentoo-user] Suggestions for NAS appliance?
On Thu, 11 Nov 2021 at 03:48, Alan Grimes wrote: > > Hey, my old NAS box croaked the other day. I had to spend $400 ond > hardware and software to recover my data but the issue now is finding a > good new NAS solution. > > Use is basically media server + backup provider. > > I was happy with my old Netgear ReadyNAS idiot-proof box until it died. > Any experience with products on the market? I have good experience with both QNAP and Synology boxes, running both right now. I run regular HDD's in them, and have had them fail on both systems, but have never had to restore data from online backup, as their redundancy works very well (some sort of RAID solution). Regards, Arve
Re: [gentoo-user] Suggestions for NAS appliance?
On Wed, Nov 10, 2021, 7:49 PM Alan Grimes wrote: > Hey, my old NAS box croaked the other day. I had to spend $400 ond > hardware and software to recover my data but the issue now is finding a > good new NAS solution. > > Use is basically media server + backup provider. > > I was happy with my old Netgear ReadyNAS idiot-proof box until it died. > Any experience with products on the market? > Not a recommendation precisely but there's a guy on YouTube named Jeff Geerling that's doing a lot of that sort of thing using a Raspberry Pi and multiple SATA drives. I've just built my first RP4 box aimed at astrophotography and I'm pretty impressed with how well the Pi works. My next project will likely be some sort of NAS box using a second Pi4 with an M.2 system drive. There is a bootable Gentoo image for the Pi4. It was pretty zippy but it's a year and a half old so if you're fixated on Gentoo you'd have a lot of work to do vs using Ubuntu server or something prebuilt. HTH, Mark >
