Re: [gentoo-user] Rasp-Pi-4 Gentoo servers
On Mon, Mar 2, 2020 at 12:22 AM William Kenworthy wrote: > > I thought lizardfs was much more community minded > but you are characterising it as similar to moosefs - a taster offering > by a commercial company holding back some of the non-essential but > jucier features for the paid version - is that how you see them? I don't see much of an active community. It seems like most actual development happens outside of the public repo, with big code drops by the private team doing the work (which seems to be associated with a company). A bit like the Android model. I'm sure they'll accept pull requests, but that isn't how most of the work is getting done. It seems like the main difference between them and moosefs is that they're making more stuff FOSS to entice users over. Shadow masters are FOSS as opposed to just having metadata loggers. HA is FOSS in the latest RC. So, it seems like their model is to trickle out the non-free stuff and make it free after a delay. It really seems like Ceph is the best fully open platform out there, but the resource requirements just make it impractical. I have no doubt that it can scale FAR better with its design, but that design basically forces every node to be a bit of a powerhouse, versus Lizardfs where you just have one daemon with all the intelligence and the rest are just dumping files on disks. And you really don't need much CPU/RAM for the master if you're serving large files - the demands would go up with IOPS and number of files, and multimedia is low on both. > By the way, to keep to the rpi subject, I did have a rpi3B with a usb2 > sata drive attached but it was hopeless as a chunkserver impacting the > whole cluster. Having the usb data flow and network data flow through > the same hub just didn't go well Hard drives plus 100Mbps LAN sharing a single USB 2.0 hub is definitely not a recipe for NAS success... When I upgraded to UniFi switches I really only noticed for the first time how many hosts I have that aren't gigabit, and they're mostly Pis at this point. They're nice little project boards but for anything IO-intensive they're almost always the wrong choice. The RockPro64 I'm using has gigabit plus PCIe 3.0 x8 plus USB3 and as far as I can tell they don't have any contention. Maybe they're all on a PCIe bus or something but obviously that can handle quite a bit. Only issue was that the rk3399 PCIe drivers were not the most robust in the kernel, but ayufan and the IRC channel were both helpful and his kernel branch is actively maintained, so I was able to get everything sorted (some delays needed during training to allow boards to initialize and I was having power issues in the beginning). Much of the rk3399 support in the kernel was pushed by Google for Chromebooks and LSI HBAs weren't exactly on their list of things to test with those - not sure if the Chromebooks put much of anything on PCIe. -- Rich
Re: [gentoo-user] Rasp-Pi-4 Gentoo servers
Sorry for the presumption. On 2020-03-02 01:08, Daniel Frey wrote: On 3/1/20 7:40 AM, n952162 wrote: "within the country"? :-) You must be American? No. Dan
Re: [gentoo-user] Rasp-Pi-4 Gentoo servers
On 2/3/20 10:40 am, Rich Freeman wrote: On Sun, Mar 1, 2020 at 8:52 PM William Kenworthy wrote: For those wanting to run a lot of drives on a single host - that defeats the main advantage of using a chunkserver based filesystem - redundancy. Its far more common to have a host fail than a disk drive. Losing the major part of your storage in one go means the cluster is effectively dead - hence having a lot of completely separate systems is much more reliable Of course. You should have multiple hosts before you start putting multiple drives on a single host. However, once you have a few hosts the performance improves by adding more, but you're not really getting THAT much additional redundancy. You would get faster rebuild times by having more hosts since there would be less data to transfer when one fails and more hosts doing the work. So, it is about finding a balance. You probably don't want 30 drives on 2 hosts. However, you probably also don't need 15-30 hosts for that many drives either. I wouldn't be putting 16 drives onto a single host until I had a fair number of hosts. As far as the status of lizardfs goes - as far as I can tell it is mostly developed by a company and they've wavered a bit on support in the last year. I share your observation that they seem to be picking up again. In any case, I'm running the latest stable and it works just fine, but it lacks the high availability features. I can have shadow masters, but they won't automatically fail over, so maintenance on the master is still a pain. Recovery due to failure of the master should be pretty quick though even if manual - just have to run a command on each shadow to determine which has the most recent metadata, then adjust DNS for my master CNAME to point to the new master, and then edit config on the new master to tell it that it is the master and no longer a shadow, and after restarting the daemon the cluster should be online again. The latest release candidate has the high availability features (used to be paid, is now free), however it is still a release candidate and I'm not in that much of a rush. There was a lot of griping on the forums/etc by users who switched to the release candidate and ran into bugs that ate their data. IMO that is why you don't go running release candidates for distributed filesystems with a dozen hard drives on them - if you want to try them out just run them in VMs with a few GB of storage to play with and who cares if your test data is destroyed. It is usually wise to be conservative with your filesystems. Makes no difference to me if they take another year to do the next release - I'd like the HA features but it isn't like the old code goes stale. Actually, the one thing that it would be nice if they fixed is the FUSE client - it seems to leak RAM. Oh, and the docs seem to hint at a windows client somewhere which would be really nice to have, but I can't find any trace of it. I only normally run a single client but it would obviously perform well as a general-purpose fileserver. There has been talk of a substantial rewrite, though I'm not sure if that will actually happen now. If it does I hope they do keep the RAM requirements low on the chunkservers. That was the main thing that turned me off from ceph - it is a great platform in general but needing 1GB RAM per 1TB disk adds up really fast, and it basically precludes ARM SBCs as OSDs as you can't get those with that much RAM for any sane price - even if you were only running one drive per host good luck finding a SBC with 13GB+ of RAM. You can tune ceph to use less RAM but I've heard that bad things happen if you have some hosts shuffle during a rebuild and you don't have gobs of RAM - all the OSDs end up with an impossible backlog and they keep crashing until you run around like Santa Claus filling every stocking with a handful of $60 DIMMs. Right now lizardfs basically uses almost no ram at all on chunkservers, so an ARM SBC could run dozens of drives without an issue. Everything bad you hear about ceph is true ... and then some! I did try, but this was some years ago so hopefully its better now. The two biggies were excessive network requirements (bandwidth, separation) and recovery times with frequent crash and burn. There are ceph features I would really like to use (rbd, local copies with much simpler config, ...) but moosefs is a lot more bullet proof on lesser resource requirements though I did find properly pruned vlans on a smartswitch separating the intra-cluster from external requests made a noticeable difference. moosefs has a windows client but its only available with the paid version. The master/shadow-master and auto failover is only available through the paid version - for the community you have to stop the master, copy the files then change DNS etc. before restarting the new master - cant really do it online even when scripted - its painful with downtime and I had dns caching issues
Re: [gentoo-user] Rasp-Pi-4 Gentoo servers
On Sun, Mar 1, 2020 at 8:52 PM William Kenworthy wrote: > > For those wanting to run a lot of drives on a single host - that defeats > the main advantage of using a chunkserver based filesystem - > redundancy. Its far more common to have a host fail than a disk drive. > Losing the major part of your storage in one go means the cluster is > effectively dead - hence having a lot of completely separate systems is > much more reliable Of course. You should have multiple hosts before you start putting multiple drives on a single host. However, once you have a few hosts the performance improves by adding more, but you're not really getting THAT much additional redundancy. You would get faster rebuild times by having more hosts since there would be less data to transfer when one fails and more hosts doing the work. So, it is about finding a balance. You probably don't want 30 drives on 2 hosts. However, you probably also don't need 15-30 hosts for that many drives either. I wouldn't be putting 16 drives onto a single host until I had a fair number of hosts. As far as the status of lizardfs goes - as far as I can tell it is mostly developed by a company and they've wavered a bit on support in the last year. I share your observation that they seem to be picking up again. In any case, I'm running the latest stable and it works just fine, but it lacks the high availability features. I can have shadow masters, but they won't automatically fail over, so maintenance on the master is still a pain. Recovery due to failure of the master should be pretty quick though even if manual - just have to run a command on each shadow to determine which has the most recent metadata, then adjust DNS for my master CNAME to point to the new master, and then edit config on the new master to tell it that it is the master and no longer a shadow, and after restarting the daemon the cluster should be online again. The latest release candidate has the high availability features (used to be paid, is now free), however it is still a release candidate and I'm not in that much of a rush. There was a lot of griping on the forums/etc by users who switched to the release candidate and ran into bugs that ate their data. IMO that is why you don't go running release candidates for distributed filesystems with a dozen hard drives on them - if you want to try them out just run them in VMs with a few GB of storage to play with and who cares if your test data is destroyed. It is usually wise to be conservative with your filesystems. Makes no difference to me if they take another year to do the next release - I'd like the HA features but it isn't like the old code goes stale. Actually, the one thing that it would be nice if they fixed is the FUSE client - it seems to leak RAM. Oh, and the docs seem to hint at a windows client somewhere which would be really nice to have, but I can't find any trace of it. I only normally run a single client but it would obviously perform well as a general-purpose fileserver. There has been talk of a substantial rewrite, though I'm not sure if that will actually happen now. If it does I hope they do keep the RAM requirements low on the chunkservers. That was the main thing that turned me off from ceph - it is a great platform in general but needing 1GB RAM per 1TB disk adds up really fast, and it basically precludes ARM SBCs as OSDs as you can't get those with that much RAM for any sane price - even if you were only running one drive per host good luck finding a SBC with 13GB+ of RAM. You can tune ceph to use less RAM but I've heard that bad things happen if you have some hosts shuffle during a rebuild and you don't have gobs of RAM - all the OSDs end up with an impossible backlog and they keep crashing until you run around like Santa Claus filling every stocking with a handful of $60 DIMMs. Right now lizardfs basically uses almost no ram at all on chunkservers, so an ARM SBC could run dozens of drives without an issue. -- Rich
Re: [gentoo-user] Rasp-Pi-4 Gentoo servers
I am going to answer multiple points below: On 1/3/20 11:36 pm, Daniel Frey wrote: On 3/1/20 6:33 AM, Rich Freeman wrote: On Sun, Mar 1, 2020 at 2:13 AM William Kenworthy wrote: Keep in mind that rpi are not the only cheap, capable arm hardware out there. I am in Oz, delivery from HardKernel (the South Korean company behind the Odroid line) takes ~1 week. Shipping is mostly via FedEx, who are a bit pricy and for me means a 30m drive to get it from a delivery centre as I cant be home to take delivery - though the last delivery used a different, much more flexible delivery company. lizardfs and moosefs are very similar (originally a fork) - I went with moosefs as the community was better a few months ago - but lizardfs sounds like its getting back on track after some infighting (moosefs apparently went through the same thing) I find the moosefs documentation and community help are adequate, but ultimately they are are a commercial project with a free taster offering hence some limitations in design for the community version (such as single master/no shadow masters) that lizardfs doesn't have. I will move to lizardfs when I am satisfied they have their act together because of this - having a single master means taking the whole cluster offline when the master needs maintenance or fails which is painful For those wanting to run a lot of drives on a single host - that defeats the main advantage of using a chunkserver based filesystem - redundancy. Its far more common to have a host fail than a disk drive. Losing the major part of your storage in one go means the cluster is effectively dead - hence having a lot of completely separate systems is much more reliable - yes, I did try having 4 sata drives on an atom board and found it was easy to justify two more HC2's for the reliability. (note, its not the effect of the atom boards reliability I am pointing out, but the effect on the whole storage system of losing such a large percentage of capacity in one go - think maintenance, fail to startup etc. - its more common than you may think) Dale: there is no single reference - I just designed on the fly and did what was necessary to move from my existing KVM/Qemu architecture on a two powerfull intel systems, each with 8TB storage to an lxc container based system backed by moosefs. It uses an odroid arm based n2 for lxc, an arm based xu4 for management and software compilation (its not that powerfull, but is adequate), the HC2's (very similar to the xu4 - same arm architecture) and an H2 to run the mfsmaster and I am moving to ansible for management. Overall, I am finding less maintenance due to better backups and better reliability, considerably lower power consumption while overall performance seems better. Gotchas: 1. I originally got the 4gb ram N2 to run the mfsmaster software - tests were excellent - until I added multiple copies of a mailserver with nearly 20 years of history - hit swap and slowed to a crawl. So I now have an intel based Odroid H2 with 32GB ram - currently is using ~4.2GB ram for 7TB of 20TB used, but has hit over 32GbB and well into swap while converging. One admin in our local LUG is using 4xHC2 with the master running on one of the HC2's as a media server with no problems - its millions of small files added ina short time that causes the grief - there is a formula in the moosefs documentation allowing resources to be estimated - on the mailing list I saw a mention of a data centre using something like 150G ram and having problems! 2. The Odroid HC2 has a single sata port and a single usb port - I have 5 of them, two have a sata + a usb3 drive attached. I did try (on the N2) using multiple usb3 drives on the internal hub - disaster with way too much traffic through the hub - don't do it! 3. Storage options are an SD card (the faster the better - swap on an SD card ... sucks!) or eMMC (5xfaster than even a good SD card.) for the N2 and xu4 - HC2's can only do sdcard, or sata. The H2 can do sd card, sata, eMMC, or m2 NVME - this last really flies! Note that almost all arm system max out at 2 to 4GB of ram, so swap is usually needed for safety - depending on OOM killer resource management on a SAN type storage system is asking for corruption like I saw in one recent gentoo email. 4. xu4/HC2 are 32 bit arm v7, the N2 is 64 bit and runs aarch64 nicely - I copied my rpi images across repurposed them (hooray for emerge/portage!). I do not use, or have done any work on their graphics/multimedia capabilities 5. I want to move to all gentoo-sources kernels - the xu4/HC2's are still on the odroid kernel until I get around to it. The N2 was a lot of work, but ultimately successful, the H2 is standard amd64 EFI. Have fun! BillK
Re: [gentoo-user] Rasp-Pi-4 Gentoo servers
On 3/1/20 7:40 AM, n952162 wrote: "within the country"? :-) You must be American? No. Dan
Re: [gentoo-user] Rasp-Pi-4 Gentoo servers
"within the country"? :-) You must be American? On 2020-03-01 16:38, Daniel Frey wrote: On 2/29/20 11:13 PM, William Kenworthy wrote: Keep in mind that rpi are not the only cheap, capable arm hardware out there. I am using a number of odroid devices, including an N2 with a gentoo based kernel and a gentoo aarch64 userland. Its used for lxc containers for asterisk, dns, webdav, mail, calendaring and web running on the N2 backed by an Odroid HC2 moosefs cluster (though I am using an intel powered Odroid H2 for the master). Its all working rather well now the initial install/config stages are over. Part of the gain over the pi's is the use of eMMC storage over sdcards - almost 5 times faster in my tests. The 4G of ram has proven quite adequate so far - even the asterisk latency is better than my previous QEMU/KVM on intel setup. I have currently have rpi 1B, 3B and a zero and while the specs for the rpi4 are better ... its not the best out there. BillK I am aware of other devices but the RPi (afaict) is the only one sold within the country and not subject to import duty/fees/taxes and the related shipping delays because of those. :( Dan
Re: [gentoo-user] Rasp-Pi-4 Gentoo servers
On 2/29/20 11:13 PM, William Kenworthy wrote: Keep in mind that rpi are not the only cheap, capable arm hardware out there. I am using a number of odroid devices, including an N2 with a gentoo based kernel and a gentoo aarch64 userland. Its used for lxc containers for asterisk, dns, webdav, mail, calendaring and web running on the N2 backed by an Odroid HC2 moosefs cluster (though I am using an intel powered Odroid H2 for the master). Its all working rather well now the initial install/config stages are over. Part of the gain over the pi's is the use of eMMC storage over sdcards - almost 5 times faster in my tests. The 4G of ram has proven quite adequate so far - even the asterisk latency is better than my previous QEMU/KVM on intel setup. I have currently have rpi 1B, 3B and a zero and while the specs for the rpi4 are better ... its not the best out there. BillK I am aware of other devices but the RPi (afaict) is the only one sold within the country and not subject to import duty/fees/taxes and the related shipping delays because of those. :( Dan
Re: [gentoo-user] Rasp-Pi-4 Gentoo servers
On 3/1/20 6:33 AM, Rich Freeman wrote: On Sun, Mar 1, 2020 at 2:13 AM William Kenworthy wrote: Keep in mind that rpi are not the only cheap, capable arm hardware out there. I completely agree. Anytime I'm looking at an application I consider the SBCs available as options. Certainly the odroids are highly spoken of. Main advantage of the Pi is its ubiquity - just about anything you could want is already packaged and documented for it. It is also pretty cheap. backed by an Odroid HC2 moosefs cluster (though I am using an intel powered Odroid H2 for the master). I considered an HC2 for lizardfs. My problem with it is that it has a single SATA port, which means you're buying a $50 SBC for every hard drive in your cluster. For a single drive per node it is probably your best bet. However, my chunkservers are: ~$65 RockPro64 $20 used LSI HBA $5 wall wart $25 cheap ATX PSU $5 ATX power switch $5 extra SATA cables $5 powered 16x PCIe riser cable (these are a bit hard to find) That is ~$125, and will support 16 hard drives. You're saving money on the 3rd drive per node. If you want some kind of enclosure for the drives you'll pay maybe another $5/drive. The other option that might be worth considering if you don't mind losing some bandwidth to the drives is just using SATA3 and hubs/etc and external drives. I'm shucking external drives anyway. So, any SBC with a SATA3 port would work for that, with nothing else needed. I could see USB3 bandwidth (shared) being a constraint if you're rebuilding, but it would keep up with gigabit ethernet. Oh, and for any kind of NAS/etc solution make sure that whatever you get has gigabit ethernet. The Pi3s at least don't have that - not sure about the Pi4. Wouldn't help in a Pi3 anyway as I think the LAN goes through the internal USB2 bus - the Pi is pretty lousy for IO in general - at least conventional PC IO. That GPIO breakout is of course nice for projects. I was reading the Pi4 has true gigabit now, thanks to its USB3 ports. Dan
Re: [gentoo-user] Rasp-Pi-4 Gentoo servers
On Sun, Mar 1, 2020 at 2:13 AM William Kenworthy wrote: > > Keep in mind that rpi are not the only cheap, capable arm hardware out > there. > I completely agree. Anytime I'm looking at an application I consider the SBCs available as options. Certainly the odroids are highly spoken of. Main advantage of the Pi is its ubiquity - just about anything you could want is already packaged and documented for it. It is also pretty cheap. > backed by an Odroid HC2 moosefs cluster (though I am using an intel > powered Odroid H2 for the master). I considered an HC2 for lizardfs. My problem with it is that it has a single SATA port, which means you're buying a $50 SBC for every hard drive in your cluster. For a single drive per node it is probably your best bet. However, my chunkservers are: ~$65 RockPro64 $20 used LSI HBA $5 wall wart $25 cheap ATX PSU $5 ATX power switch $5 extra SATA cables $5 powered 16x PCIe riser cable (these are a bit hard to find) That is ~$125, and will support 16 hard drives. You're saving money on the 3rd drive per node. If you want some kind of enclosure for the drives you'll pay maybe another $5/drive. The other option that might be worth considering if you don't mind losing some bandwidth to the drives is just using SATA3 and hubs/etc and external drives. I'm shucking external drives anyway. So, any SBC with a SATA3 port would work for that, with nothing else needed. I could see USB3 bandwidth (shared) being a constraint if you're rebuilding, but it would keep up with gigabit ethernet. Oh, and for any kind of NAS/etc solution make sure that whatever you get has gigabit ethernet. The Pi3s at least don't have that - not sure about the Pi4. Wouldn't help in a Pi3 anyway as I think the LAN goes through the internal USB2 bus - the Pi is pretty lousy for IO in general - at least conventional PC IO. That GPIO breakout is of course nice for projects. -- Rich
Re: [gentoo-user] Rasp-Pi-4 Gentoo servers
Michael wrote: > On Sunday, 1 March 2020 10:46:17 GMT Dale wrote: >> >> Could you share some links to some of these things? As I mentioned >> earlier, I'm thinking about building a NAS system. Later, I may build a >> mythTV system. Then I can access the NAS from it or my desktop, or cell >> phone now that I am somewhat more updated, past the Motorola Razr stage. >> >> Thanks much. >> >> Dale >> >> :-) :-) > Android has its optimal use cases for sure, but I have found embedded x86 > APUs > to be more capable for network and server tasks. As an example: > > https://pcengines.ch/ Interesting. I was thinking at first about just buying a really small computer MB, like a micro ATX or something, but reading this thread made think more about something like this. For one, it pulls a very small amount of power. The 'computer' and hard drives would likely pull well under 50 watts. Also generate very little heat, more like warmth. ;-) Basically, I'm looking for something that I can build a NAS with. I'd need a way to hook 4, 6 maybe 8 hard drives and maybe a couple ethernet ports. It seems to be easy to find them with ethernet ports. It's the SATA ports that are harder to find. I found one for the Raspberry thing that had four ports. So far, it is the only one I found. I'm not sure if it can expand past that either. It seems the cards are stackable based on some pics but I'm not sure on that. This is interesting info tho. The link you shared lead me to a tiny board, maybe a little more powerful than the Raspberry but still low powered. Still, the SATA ports is a bit difficult to find. Maybe I'm looking for the wrong thing??? Either way, I left a map light, tiny light that's up there with the dome light, on a few weeks ago in my car. I pulled out my hydrometer battery tester. It's almost always accurate. I got a few week cells in my car's battery. After someone else posted about those AGM batteries, I'm thinking of going down that route even for my car. They ain't cheap by no stretch. That said, those Raspberry things are almost disposable if you get a Chinese model. lol Dale :-) :-)
Re: [gentoo-user] Rasp-Pi-4 Gentoo servers
n952162 wrote: > > On 2020-03-01 11:46, Dale wrote: >> ... now that I am somewhat more updated, past the Motorola Razr stage. >> Thanks much. >> >> Dale >> >> :-) :-) >> > > Which Razr do you mean? > > Are you 9 years (https://en.wikipedia.org/wiki/Droid_Razr) or 16 years > out of date? (https://en.wikipedia.org/wiki/Motorola_Razr) > > ;-) > > Although, it's amazing to think, today, that there was only 7 years > between those two... > > > I had the flip phone, bottom link I think. I wore out the charging port and it got to where it was hard to charge the battery. Otherwise, the thing still worked great. I admit, texting is much easier with the new Samsung. Also, circle a word puzzles when at the doctors office waiting helps pass the time too. :/ Dale :-) :-)
Re: [gentoo-user] Rasp-Pi-4 Gentoo servers
On Sunday, 1 March 2020 10:46:17 GMT Dale wrote: > William Kenworthy wrote: > > On 29/2/20 11:31 pm, Rich Freeman wrote: > >> On Sat, Feb 29, 2020 at 10:17 AM Daniel Frey wrote: > >>> Yes, I'm aware linux does VLANs... I set up netifrc to do this (I > >>> already have some "smart" switches set up - not full layer 3.) I thought > >>> about running containers but if I ever have to do something like > >>> emergency maintenance on my server the whole LAN would be down. Seems > >>> like a no-brainer to have a tiny device like an RPi to do this. > >> > >> Yup. It really depends on your requirements. > >> > >> My main LAN uses a Pi as a DHCP+DNS server, for exactly this reason. > >> I don't want to be replacing a hard drive in my server and now my > >> lights/TV/whatever don't work. OpenHab runs on a Pi for this reason > >> as well. > > > > Keep in mind that rpi are not the only cheap, capable arm hardware out > > there. > > > > I am using a number of odroid devices, including an N2 with a gentoo > > based kernel and a gentoo aarch64 userland. Its used for lxc containers > > for asterisk, dns, webdav, mail, calendaring and web running on the N2 > > backed by an Odroid HC2 moosefs cluster (though I am using an intel > > powered Odroid H2 for the master). > > > > Its all working rather well now the initial install/config stages are > > over. Part of the gain over the pi's is the use of eMMC storage over > > sdcards - almost 5 times faster in my tests. The 4G of ram has proven > > quite adequate so far - even the asterisk latency is better than my > > previous QEMU/KVM on intel setup. > > > > I have currently have rpi 1B, 3B and a zero and while the specs for the > > rpi4 are better ... its not the best out there. > > > > BillK > > Could you share some links to some of these things? As I mentioned > earlier, I'm thinking about building a NAS system. Later, I may build a > mythTV system. Then I can access the NAS from it or my desktop, or cell > phone now that I am somewhat more updated, past the Motorola Razr stage. > > Thanks much. > > Dale > > :-) :-) Android has its optimal use cases for sure, but I have found embedded x86 APUs to be more capable for network and server tasks. As an example: https://pcengines.ch/ signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] Rasp-Pi-4 Gentoo servers
On 2020-03-01 11:46, Dale wrote: ... now that I am somewhat more updated, past the Motorola Razr stage. Thanks much. Dale :-) :-) Which Razr do you mean? Are you 9 years (https://en.wikipedia.org/wiki/Droid_Razr) or 16 years out of date? (https://en.wikipedia.org/wiki/Motorola_Razr) ;-) Although, it's amazing to think, today, that there was only 7 years between those two...
Re: [gentoo-user] Rasp-Pi-4 Gentoo servers
William Kenworthy wrote: > On 29/2/20 11:31 pm, Rich Freeman wrote: >> On Sat, Feb 29, 2020 at 10:17 AM Daniel Frey wrote: >>> Yes, I'm aware linux does VLANs... I set up netifrc to do this (I >>> already have some "smart" switches set up - not full layer 3.) I thought >>> about running containers but if I ever have to do something like >>> emergency maintenance on my server the whole LAN would be down. Seems >>> like a no-brainer to have a tiny device like an RPi to do this. >> Yup. It really depends on your requirements. >> >> My main LAN uses a Pi as a DHCP+DNS server, for exactly this reason. >> I don't want to be replacing a hard drive in my server and now my >> lights/TV/whatever don't work. OpenHab runs on a Pi for this reason >> as well. > Keep in mind that rpi are not the only cheap, capable arm hardware out > there. > > I am using a number of odroid devices, including an N2 with a gentoo > based kernel and a gentoo aarch64 userland. Its used for lxc containers > for asterisk, dns, webdav, mail, calendaring and web running on the N2 > backed by an Odroid HC2 moosefs cluster (though I am using an intel > powered Odroid H2 for the master). > > Its all working rather well now the initial install/config stages are > over. Part of the gain over the pi's is the use of eMMC storage over > sdcards - almost 5 times faster in my tests. The 4G of ram has proven > quite adequate so far - even the asterisk latency is better than my > previous QEMU/KVM on intel setup. > > I have currently have rpi 1B, 3B and a zero and while the specs for the > rpi4 are better ... its not the best out there. > > BillK > Could you share some links to some of these things? As I mentioned earlier, I'm thinking about building a NAS system. Later, I may build a mythTV system. Then I can access the NAS from it or my desktop, or cell phone now that I am somewhat more updated, past the Motorola Razr stage. Thanks much. Dale :-) :-)
Re: [gentoo-user] Rasp-Pi-4 Gentoo servers
On 29/2/20 11:31 pm, Rich Freeman wrote: > On Sat, Feb 29, 2020 at 10:17 AM Daniel Frey wrote: >> Yes, I'm aware linux does VLANs... I set up netifrc to do this (I >> already have some "smart" switches set up - not full layer 3.) I thought >> about running containers but if I ever have to do something like >> emergency maintenance on my server the whole LAN would be down. Seems >> like a no-brainer to have a tiny device like an RPi to do this. > Yup. It really depends on your requirements. > > My main LAN uses a Pi as a DHCP+DNS server, for exactly this reason. > I don't want to be replacing a hard drive in my server and now my > lights/TV/whatever don't work. OpenHab runs on a Pi for this reason > as well. Keep in mind that rpi are not the only cheap, capable arm hardware out there. I am using a number of odroid devices, including an N2 with a gentoo based kernel and a gentoo aarch64 userland. Its used for lxc containers for asterisk, dns, webdav, mail, calendaring and web running on the N2 backed by an Odroid HC2 moosefs cluster (though I am using an intel powered Odroid H2 for the master). Its all working rather well now the initial install/config stages are over. Part of the gain over the pi's is the use of eMMC storage over sdcards - almost 5 times faster in my tests. The 4G of ram has proven quite adequate so far - even the asterisk latency is better than my previous QEMU/KVM on intel setup. I have currently have rpi 1B, 3B and a zero and while the specs for the rpi4 are better ... its not the best out there. BillK
Re: [gentoo-user] Rasp-Pi-4 Gentoo servers
On Sat, Feb 29, 2020 at 10:17 AM Daniel Frey wrote: > > Yes, I'm aware linux does VLANs... I set up netifrc to do this (I > already have some "smart" switches set up - not full layer 3.) I thought > about running containers but if I ever have to do something like > emergency maintenance on my server the whole LAN would be down. Seems > like a no-brainer to have a tiny device like an RPi to do this. Yup. It really depends on your requirements. My main LAN uses a Pi as a DHCP+DNS server, for exactly this reason. I don't want to be replacing a hard drive in my server and now my lights/TV/whatever don't work. OpenHab runs on a Pi for this reason as well. On the other hand, for my other VLANs DHCP+DNS is handled by stuff like my UniFi gateway or other embedded solutions. These don't have the same requirements as my main LAN and being mostly self-contained a more consumer-oriented solution is fine. I don't want to be doing security updates on a bazillion Pis either. I use VLAN on Linux more for providing services on the VLANs. Not that I have much of this. Don't think I'm running some kind of datacenter. I just have a typical home LAN, and I'm running AREDN which basically needs two more VLANs of its own (one for the network it serves, and one for backhaul to the internet for tunnels/etc - don't want that stuff getting into my LAN and the IP address space conflicts in any case). I could see adding an IOT VLAN maybe, but the problem is that so much of that stuff needs to interact. If I stuck my TVs/Chromecasts/etc on a separate VLAN, then I couldn't cast to them from my phone or anything else unless it was on that VLAN too. > I'm not so sure I'll try installing Gentoo on it though, it doesn't > really seem suitable for compiling tasks. I'm pretty sure the kit I > ordered has a card with Raspbian on it, I'll check that out first. I run Raspbian on my Pis for this reason. If I had some niche use where Gentoo added value I'd go with it, but otherwise it just seems too painful. As it is I have to compile kernel modules on my RockPro64 boards and that takes forever even without having to build the actual kernel. When I've built kernels on those while troubleshooting issues with PCIe it would literally take an hour or more. If you do want to run Gentoo on a Pi you really should be cross-compiling it. Something like Gentoo Reference Platform on steroids would certainly be nice for ARM. -- Rich
Re: [gentoo-user] Rasp-Pi-4 Gentoo servers
On 2/28/20 5:38 PM, Rich Freeman wrote: On Fri, Feb 28, 2020 at 8:11 PM Daniel Frey wrote: Thanks for the detail, I've just ordered an RPi4B to mess around with. It would be helpful to move DNS etc off my home server as I'm trying to separate everything into VLANs. Keep in mind that Linux supports VLAN tagging, so if you set up your switch to trunk your server you can have containers or even services on multiple VLANs on the same host. I have this configured via systemd-networkd - I'm sure you could do it with various other network managers as well. I just have a bridge for each VLAN and then I can attach container virtual ethernet interfaces to the appropriate VLAN bridge for each container. KVM uses bridges and it should be just as easy to put VMs on the appropriate bridges. If you assign IPs on the host to each VLAN interface then as long as the VLANs don't have conflicting IP addresses you can just attach services to the appropriate VLANs by binding to their addresses. A service that binds to 0.0.0.0 or to multiple addresses would listen on all of them. Now, if your VLANs have conflicting address spaces then I'd probably just stick to containers so that no host actually sees conflicting IPs, otherwise you're probably going to have to go crazy with iproute2 and netfilter to get all the packets going to the right places. And all of that should work from a Pi as well as long as long as you enable CONFIG_VLAN_8021Q. You also need to make sure the tagged VLAN traffic is passed from the switch (which is not what you normally want to do for a non-VLAN-aware host where you would filter out all but one VLAN and remove the tag). I run my DHCP server on a Pi so that it is more independent. Yes, I'm aware linux does VLANs... I set up netifrc to do this (I already have some "smart" switches set up - not full layer 3.) I thought about running containers but if I ever have to do something like emergency maintenance on my server the whole LAN would be down. Seems like a no-brainer to have a tiny device like an RPi to do this. I'm not so sure I'll try installing Gentoo on it though, it doesn't really seem suitable for compiling tasks. I'm pretty sure the kit I ordered has a card with Raspbian on it, I'll check that out first. Dan
Re: [gentoo-user] Rasp-Pi-4 Gentoo servers
On Fri, Feb 28, 2020 at 8:11 PM Daniel Frey wrote: > > Thanks for the detail, I've just ordered an RPi4B to mess around with. > It would be helpful to move DNS etc off my home server as I'm trying to > separate everything into VLANs. > Keep in mind that Linux supports VLAN tagging, so if you set up your switch to trunk your server you can have containers or even services on multiple VLANs on the same host. I have this configured via systemd-networkd - I'm sure you could do it with various other network managers as well. I just have a bridge for each VLAN and then I can attach container virtual ethernet interfaces to the appropriate VLAN bridge for each container. KVM uses bridges and it should be just as easy to put VMs on the appropriate bridges. If you assign IPs on the host to each VLAN interface then as long as the VLANs don't have conflicting IP addresses you can just attach services to the appropriate VLANs by binding to their addresses. A service that binds to 0.0.0.0 or to multiple addresses would listen on all of them. Now, if your VLANs have conflicting address spaces then I'd probably just stick to containers so that no host actually sees conflicting IPs, otherwise you're probably going to have to go crazy with iproute2 and netfilter to get all the packets going to the right places. And all of that should work from a Pi as well as long as long as you enable CONFIG_VLAN_8021Q. You also need to make sure the tagged VLAN traffic is passed from the switch (which is not what you normally want to do for a non-VLAN-aware host where you would filter out all but one VLAN and remove the tag). I run my DHCP server on a Pi so that it is more independent. -- Rich
Re: [gentoo-user] Rasp-Pi-4 Gentoo servers
On 2/27/20 1:49 PM, Rich Freeman wrote: On Thu, Feb 27, 2020 at 4:25 PM james wrote: Yea, I was not clear. I'd run the mail-server, on a 'cluster' (4 or more), not an individual pi-board unless it was beef up, processor and ram wise. Gig E would also be on my list. Unless you have some niche need I wouldn't generally run servers on Pis. The biggest issue with ARM is that all the cheap platforms are starved for RAM, and RAM is one of the biggest issues when running services. And of course the Pi in particular has IO issues (as do many other cheap SBCs but this is less of an ARM issue). The RAM issue isn't so many an ARM issue as a supply/demand thing - the only people asking for 64GB ARM boards are big companies that are willing to pay a lot for them. I do actually run a few services on Pis - DNS, DHCP, and a VPN gateway. That's about it. These are fairly non-demanding tasks that the hardware doesn't struggle with, and the data is almost entirely static so an occasional backup makes any kind of recovery trivial. The only reason I run these services on Pis is that they are fairly fundamental to having a working network. Most of my services are running in containers on a server, but I don't want to have to think about taking a server down for maintenance and then literally every IOT device in the house won't work. These particular services are also basically dependency-free which means I can just boot them up and they just do their jobs, while they remain a dependency for just about everything else on the network. When you start running DHCP in a container you have more complex dependency issues. A fairly cheap amd64 system can run a ton of services in containers though, and it is way simpler to maintain that way. I still get quick access to snapshots/etc, but now if I want to run a gentoo container it is no big deal if 99% of the time it uses 25MB of RAM and 1% of one core, but once a month it needs 4GB of RAM and 100% of 6 cores. As long as I'm not doing an emerge -u world on half a dozen containers at once it is no big deal at all. Now, if I needed some server in some niche application that needed to be able to operate off of a car battery for a few days, then sure I'd be looking at Pis and so on. Thanks for the detail, I've just ordered an RPi4B to mess around with. It would be helpful to move DNS etc off my home server as I'm trying to separate everything into VLANs. Dan
Re: [gentoo-user] Rasp-Pi-4 Gentoo servers
On Friday, 28 February 2020 13:28:53 GMT Wols Lists wrote: > On 28/02/20 11:45, Michael wrote: --->8 > > http://www.runmapglobal.com/blog/fault-tolerant-dedicated-servers/ > > Noted. That link pre-dates me working on the site - I haven't checked > all the old links - I guess I should ... If you run a KDE desktop, KDE link checker is the business. -- Regards, Peter.
Re: [gentoo-user] Rasp-Pi-4 Gentoo servers
On 28/02/20 11:45, Michael wrote: > On Friday, 28 February 2020 11:04:43 GMT Wols Lists wrote: >> On 28/02/20 05:07, james wrote: >>> For data storage, long term important stuff, you should employ RAID >>> (1-10). We can get into that later, duplication of important data, via >>> backups or extra storage is a good idea too. Backups are an old >>> technology, but may help, but backups do can get old too and fragmented. >>> For now, lets not worry so much about long term bit integrity, but focus >>> on your next FUN gentoo rig. I'm hoping other join in to so you have >>> more than my prospective on your solution. >> >> https://raid.wiki.kernel.org/index.php/Linux_Raid >> >> Sorry for the plug, I edit the site. Brickbats/bouquets welcome :-) >> >> Cheers, >> Wol > > Thanks for sharing page Wol! > > I tried this page from the links at the bottom and ended up in a 404 error: > > http://www.runmapglobal.com/blog/fault-tolerant-dedicated-servers/ > Noted. That link pre-dates me working on the site - I haven't checked all the old links - I guess I should ... Cheers, Wol
Re: [gentoo-user] Rasp-Pi-4 Gentoo servers
On Fri, Feb 28, 2020 at 6:09 AM Wols Lists wrote: > > On 27/02/20 21:49, Rich Freeman wrote: > > A fairly cheap amd64 system can run a ton of services in containers > > though, and it is way simpler to maintain that way. I still get quick > > access to snapshots/etc, but now if I want to run a gentoo container > > it is no big deal if 99% of the time it uses 25MB of RAM and 1% of one > > core, but once a month it needs 4GB of RAM and 100% of 6 cores. As > > long as I'm not doing an emerge -u world on half a dozen containers at > > once it is no big deal at all. > > Do all your containers have the same make options etc? Can't remember > which directory it is, but I had a shared emerge directory where it > stored this stuff and I emerged with -bk options (use binary if it's > there, create binary if it isn't). > They're probably not too far off in general, but not exact. I only run one instance of any particular container, so I haven't tried to do parallel builds. If portage had support for multiple binary packages co-existing with different build options I might. If I ever get really bored for a few weeks I could see playing around with that. It seems like it ought to be possible to content-hash the list of build options and stick that hash in the binary package filename, and then have portage search for suitable packages, using a binary package if one matches, and doing a new build if not. Many of my containers don't even run Gentoo. I have a few running Arch, Ubuntu Server, or Debian. If some service is well-supported in one of those and is poorly supported in Gentoo I will tend to go that route. I'll package it if reasonable but some upstreams are just not very conducive to this. There was a question about ARM-based NAS in this thread which I'll go ahead and tackle to save a reply. I'm actually playing around with lizardfs (I might consider moosefs instead if starting from scratch - or Ceph if I were scaling up but that wouldn't be practical on ARM). I have a mix of chunkservers but my target is to run new ones on ARM. I'm using RockPro64 SBCs with LSI HBAs (this SBC is fairly unique in having PCIe). There is some issue with the lizardfs code that causes performance issues on ARM though I understand they're working on this, so that could change. I'm using it for multimedia and I care more about static space than iops, so it is fine for me. The LSI HBA pulls more power than the SBC does, but overall the setup is very low-power and fairly inexpensive (used HBAs on ebay). I can in theory get up to 16 drives on one SBC this way. The SBC also supports USB3 so that is another option with a hub - in fact I'm mostly shucking USB3 drives anyway. Main issue with ARM SBCs in general is that they don't have much RAM, so IMO that makes Ceph a non-starter. Otherwise that would probably be my preferred option. Bad things can happen on rebuilds if you don't have 1GB/TB as they suggest, and even with the relatively under-utilized servers I have now that would be a LOT of RAM for ARM (really, it would be expensive even on amd64). Lizardfs/moosefs chunkservers barely use any RAM at all. The master server does need more - I have shadow masters running on the SBCs but since I'm using this for multimedia the metadata server only uses about 100MB of RAM and that includes processes, libraries, and random minimal service daemons like sshd. I'm running my master on amd64 though to get optimal performance, shadowed on the chunkservers so that I can failover if needed, though in truth the amd64 box with ECC is the least likely thing to die and runs all the stuff that uses the storage right now anyway. The other suggestion to consider USB3 instead of SATA for storage isn't a bad idea. Though going that route means wall warts and drives as far as the eye can see. Might still be less messy than my setup, which has a couple of cheap ATX PSUs with ATX power switches, 16x PCIe powered risers for the HBAs (they pull too much power for the SBC), and rockwell drive cages to stack the drives in (they're meant for a server chasis but they're reasonably priced and basically give you an open enclosure with a fan). I'd definitely have a lot fewer PCBs showing if I used USB3 instead. I'm not sure how well that would perform though - that HBA has a lot of bandwidth if the node got busy with PCIe v2 x4 connectivity (SAS9200-16E) and with USB3 it would all go through 1-2 ports. Though I doubt I'd ever get THAT many drives on a node and if I needed more space I'd probably expand up to 5 chunkservers before I'm putting more than about 3 drives on each - you get better performance and more fault-tolerance that way. One big reason I went the distributed filesystem approach was that I was getting tired of trying to cram as many drives as I could into a single host and then dealing with some of the inflexibilities of zfs. The inflexibility bit is improving somewhat with removable vdevs, though I'm not sure how much residue
Re: [gentoo-user] Rasp-Pi-4 Gentoo servers
On Friday, 28 February 2020 11:04:43 GMT Wols Lists wrote: > On 28/02/20 05:07, james wrote: > > For data storage, long term important stuff, you should employ RAID > > (1-10). We can get into that later, duplication of important data, via > > backups or extra storage is a good idea too. Backups are an old > > technology, but may help, but backups do can get old too and fragmented. > > For now, lets not worry so much about long term bit integrity, but focus > > on your next FUN gentoo rig. I'm hoping other join in to so you have > > more than my prospective on your solution. > > https://raid.wiki.kernel.org/index.php/Linux_Raid > > Sorry for the plug, I edit the site. Brickbats/bouquets welcome :-) > > Cheers, > Wol Thanks for sharing page Wol! I tried this page from the links at the bottom and ended up in a 404 error: http://www.runmapglobal.com/blog/fault-tolerant-dedicated-servers/ signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] Rasp-Pi-4 Gentoo servers
On 27/02/20 21:49, Rich Freeman wrote: > A fairly cheap amd64 system can run a ton of services in containers > though, and it is way simpler to maintain that way. I still get quick > access to snapshots/etc, but now if I want to run a gentoo container > it is no big deal if 99% of the time it uses 25MB of RAM and 1% of one > core, but once a month it needs 4GB of RAM and 100% of 6 cores. As > long as I'm not doing an emerge -u world on half a dozen containers at > once it is no big deal at all. Do all your containers have the same make options etc? Can't remember which directory it is, but I had a shared emerge directory where it stored this stuff and I emerged with -bk options (use binary if it's there, create binary if it isn't). That way, when I updated my systems, I updated the "big grunt" system first, then the smaller ones, so the little ones didn't have to emerge anything other than what was unique to them. Cheers, Wol
Re: [gentoo-user] Rasp-Pi-4 Gentoo servers
On 28/02/20 05:07, james wrote: > For data storage, long term important stuff, you should employ RAID > (1-10). We can get into that later, duplication of important data, via > backups or extra storage is a good idea too. Backups are an old > technology, but may help, but backups do can get old too and fragmented. > For now, lets not worry so much about long term bit integrity, but focus > on your next FUN gentoo rig. I'm hoping other join in to so you have > more than my prospective on your solution. https://raid.wiki.kernel.org/index.php/Linux_Raid Sorry for the plug, I edit the site. Brickbats/bouquets welcome :-) Cheers, Wol
Re: [gentoo-user] Rasp-Pi-4 Gentoo servers
On Friday, 28 February 2020 05:07:07 GMT james wrote: > On 2/27/20 9:53 PM, Dale wrote: > > james wrote: > >> 5G + gentoo + embedded toys, is going to be FUN FUN FUN. > >> > >> > >> Then I'll be off to other states, via a hacked out Redneck > >> camper.. and too many microProcessors > >> > >> > >> Thanks Rich, your insights and comments are always most welcome. > >> > >> > >> James > > > > Off topic a bit but a question.� Would one of these Rasp-Pi-4 thingys > > make a NAS hard drive server? > > Sure, but, there may be a better solution, something all ready out there > and it really depends on refining your needs, current and in the future. > So lets refine your specifications (centric to your needs + growth) and > figure out what and how much you need. Then we can survey the > embedded-thingies, that meet your specs, with a bit of room for growth, OK? > > >I have a Cooler Master HAF-932 case > > Wow, that's big. What the number and capacity (TB) of > your existing hard-drives? > > How much more storage do you want? Replacing drives with larger > capacity, might be all you need to do? > > > but > > even it is running out of hard drive space.� I'm thinking about building > > a NAS box, taking sheet metal and bending it until it looks like a box. > > OK, so we first spec out options, then let you decide. Then you can > 'bargain shop' for appropriate housing/rack/open chassis, etc. > > > Thing is, it needs a small puter to take data from the drives to the > > network and vice-versa. > > embedded are not only small, they can have extended temperature ranges > of tolerance, use drastically less power and many other features. If > it's purposed hardware, that is only a few things todo, then yes > embedded uP (abbrev for microProcessor) are the way to go. Running off > of 12VDC, means an old car battery and a connection to your solar panels > (assuming you have those) and it's zero on your electric bill. There is > usually a vast array of tax and other incentives, particularly with > solar in Ag businesses. > > > I've never even seen one of those things, except on my monitor, so I > > have no idea what all they are capable of. > > Dale, you are pretty strong with Gentoo Linux, so putting a stripped, > purposed, minimized gentoo derivative stack, with far less ebuilds, to > work for your operations, is going to be quite fun. On a farm or ranch, > there are a myriad of things you can do with embedded boards and > gentoo-stripped. You can replace many of those expensive (vendor) > systems with embedded boards +sensors +controls codes and lots of wires > to do most anything. Let's focus on your NAS for now. > > > I figure a lot of SATA connectors and a ethernet connection > > plus enough CPU power and memory to get the job done. > > SATA, was great years ago. Still it makes sense to use, if you already > have them. Storage going forward is the process of faster and cheaper > and leaving SATA behind, like ide. Still useful, but a power hog. So > we'll start out with interfacing your existing SATA drives to the > embedded board, and look/decide on options for newer Solid State Data > storage options. > > > � https://en.wikipedia.org/wiki/USB > > You might not even need many sata ports. usb3 and the upcoming usb4 have > tons of bandwidth (date/time). Mechanical Hard drives are on the way > out. Too expensive and failure prone. SSd and other types of storage, > might be right for you, or a mixture. USB stick memory > can be huge, very low power draw and very inexpensive. > > A hybrid of several types of memory storage may be useful to experiment > with. You may want to categorize your long term storage: some accessed > often, others maybe once a year? > > > For data storage, long term important stuff, you should employ RAID > (1-10). We can get into that later, duplication of important data, via > backups or extra storage is a good idea too. Backups are an old > technology, but may help, but backups do can get old too and fragmented. > For now, lets not worry so much about long term bit integrity, but focus > on your next FUN gentoo rig. I'm hoping other join in to so you have > more than my prospective on your solution. > > > If those things are capable of doing that fairly > > easily.� After all, I'm me.� :/ > > OK, so let's survey some system, you can just purchase > with gentoo preinstalled, or a very easy pathway to embedded gentoo. > Let's look at a few, have some of the other guys jump in, and find you a > solution, to start with. Most will be expandable, and you can figure out > the casing, mounting, power and such. > > At this stage, it mostly a research effort and then deciding your > features/price. If you do not have massive bandwitdh requirements, I'm > sure we can find you > a very cost effective, DC powered solution. > > Just so you know, I use that fancy $300 OPtima 12vdc charger, and Optima > batteries. the charger reconditions most batteries, if they
Re: [gentoo-user] Rasp-Pi-4 Gentoo servers
On 2/27/20 9:53 PM, Dale wrote: james wrote: 5G + gentoo + embedded toys, is going to be FUN FUN FUN. Then I'll be off to other states, via a hacked out Redneck camper.. and too many microProcessors Thanks Rich, your insights and comments are always most welcome. James Off topic a bit but a question.� Would one of these Rasp-Pi-4 thingys make a NAS hard drive server? Sure, but, there may be a better solution, something all ready out there and it really depends on refining your needs, current and in the future. So lets refine your specifications (centric to your needs + growth) and figure out what and how much you need. Then we can survey the embedded-thingies, that meet your specs, with a bit of room for growth, OK? I have a Cooler Master HAF-932 case Wow, that's big. What the number and capacity (TB) of your existing hard-drives? How much more storage do you want? Replacing drives with larger capacity, might be all you need to do? but even it is running out of hard drive space.� I'm thinking about building a NAS box, taking sheet metal and bending it until it looks like a box. OK, so we first spec out options, then let you decide. Then you can 'bargain shop' for appropriate housing/rack/open chassis, etc. Thing is, it needs a small puter to take data from the drives to the network and vice-versa. embedded are not only small, they can have extended temperature ranges of tolerance, use drastically less power and many other features. If it's purposed hardware, that is only a few things todo, then yes embedded uP (abbrev for microProcessor) are the way to go. Running off of 12VDC, means an old car battery and a connection to your solar panels (assuming you have those) and it's zero on your electric bill. There is usually a vast array of tax and other incentives, particularly with solar in Ag businesses. I've never even seen one of those things, except on my monitor, so I have no idea what all they are capable of. Dale, you are pretty strong with Gentoo Linux, so putting a stripped, purposed, minimized gentoo derivative stack, with far less ebuilds, to work for your operations, is going to be quite fun. On a farm or ranch, there are a myriad of things you can do with embedded boards and gentoo-stripped. You can replace many of those expensive (vendor) systems with embedded boards +sensors +controls codes and lots of wires to do most anything. Let's focus on your NAS for now. I figure a lot of SATA connectors and a ethernet connection plus enough CPU power and memory to get the job done. SATA, was great years ago. Still it makes sense to use, if you already have them. Storage going forward is the process of faster and cheaper and leaving SATA behind, like ide. Still useful, but a power hog. So we'll start out with interfacing your existing SATA drives to the embedded board, and look/decide on options for newer Solid State Data storage options. � https://en.wikipedia.org/wiki/USB You might not even need many sata ports. usb3 and the upcoming usb4 have tons of bandwidth (date/time). Mechanical Hard drives are on the way out. Too expensive and failure prone. SSd and other types of storage, might be right for you, or a mixture. USB stick memory can be huge, very low power draw and very inexpensive. A hybrid of several types of memory storage may be useful to experiment with. You may want to categorize your long term storage: some accessed often, others maybe once a year? For data storage, long term important stuff, you should employ RAID (1-10). We can get into that later, duplication of important data, via backups or extra storage is a good idea too. Backups are an old technology, but may help, but backups do can get old too and fragmented. For now, lets not worry so much about long term bit integrity, but focus on your next FUN gentoo rig. I'm hoping other join in to so you have more than my prospective on your solution. If those things are capable of doing that fairly easily.� After all, I'm me.� :/ OK, so let's survey some system, you can just purchase with gentoo preinstalled, or a very easy pathway to embedded gentoo. Let's look at a few, have some of the other guys jump in, and find you a solution, to start with. Most will be expandable, and you can figure out the casing, mounting, power and such. At this stage, it mostly a research effort and then deciding your features/price. If you do not have massive bandwitdh requirements, I'm sure we can find you a very cost effective, DC powered solution. Just so you know, I use that fancy $300 OPtima 12vdc charger, and Optima batteries. the charger reconditions most batteries, if they are not beyond saving, even cheap lead-acid batteries. Every Farmer should have one, imho. The Digital 1200 is just awesome. https://www.optimabatteries.com/en-us/battery-charger If you like, you can read up on blue, red and yellow top versions and
Re: [gentoo-user] Rasp-Pi-4 Gentoo servers
james wrote: > > > 5G + gentoo + embedded toys, is going to be FUN FUN FUN. > > > Then I'll be off to other states, via a hacked out Redneck > camper.. and too many microProcessors > > > Thanks Rich, your insights and comments are always most welcome. > > > James Off topic a bit but a question. Would one of these Rasp-Pi-4 thingys make a NAS hard drive server? I have a Cooler Master HAF-932 case but even it is running out of hard drive space. I'm thinking about building a NAS box, taking sheet metal and bending it until it looks like a box. Thing is, it needs a small puter to take data from the drives to the network and vice-versa. I've never even seen one of those things, except on my monitor, so I have no idea what all they are capable of. I figure a lot of SATA connectors and a ethernet connection plus enough CPU power and memory to get the job done. If those things are capable of doing that fairly easily. After all, I'm me. :/ Just curious. Dale :-) :-)
Re: [gentoo-user] Rasp-Pi-4 Gentoo servers
On 2/27/20 4:49 PM, Rich Freeman wrote: On Thu, Feb 27, 2020 at 4:25 PM james wrote: Yea, I was not clear. I'd run the mail-server, on a 'cluster' (4 or more), not an individual pi-board unless it was beef up, processor and ram wise. Gig E would also be on my list. Unless you have some niche need I wouldn't generally run servers on Pis. The biggest issue with ARM is that all the cheap platforms are starved for RAM, and RAM is one of the biggest issues when running services. And of course the Pi in particular has IO issues (as do many other cheap SBCs but this is less of an ARM issue). The RAM issue isn't so many an ARM issue as a supply/demand thing - the only people asking for 64GB ARM boards are big companies that are willing to pay a lot for them. I do actually run a few services on Pis - DNS, DHCP, and a VPN gateway. That's about it. These are fairly non-demanding tasks that the hardware doesn't struggle with, and the data is almost entirely static so an occasional backup makes any kind of recovery trivial. The only reason I run these services on Pis is that they are fairly fundamental to having a working network. Most of my services are running in containers on a server, but I don't want to have to think about taking a server down for maintenance and then literally every IOT device in the house won't work. These particular services are also basically dependency-free which means I can just boot them up and they just do their jobs, while they remain a dependency for just about everything else on the network. When you start running DHCP in a container you have more complex dependency issues. A fairly cheap amd64 system can run a ton of services in containers though, and it is way simpler to maintain that way. I still get quick access to snapshots/etc, but now if I want to run a gentoo container it is no big deal if 99% of the time it uses 25MB of RAM and 1% of one core, but once a month it needs 4GB of RAM and 100% of 6 cores. As long as I'm not doing an emerge -u world on half a dozen containers at once it is no big deal at all. Now, if I needed some server in some niche application that needed to be able to operate off of a car battery for a few days, then sure I'd be looking at Pis and so on. Exactly. It's going to be a small RV, basically a 4x4 with a campershell. 2 Laptops with AMD64 and ram, the newest ones, are the powerhouses. One multicore with Radeon Graphics with a stand for 4x32" 4K 120MHZ screens to be mounted on the 1/2 table in front of the bench seat (my new mobile office). I'm hoping to get gcc-9 (10?) happy with auto-compiling using the AMD graphics (radeon chipsets). And a plethora of small embedded boards for a wide variety of toy-interfaces. Biggest problem? The arrival of the new roof mounted, 12VDC 21 SEER AC is delayed, due to that virus. Trying not to crank a genset, just solar, and fast-charge 12VDC battery banks; but we'll see how that goes in the Texas summer. I'm going to map out some 5G hot-spots and encourage folks in the areas to jump on a gentoo (derivative?) OS. The greater Dallas area is a hotbed for 5G testing and development. Austin is on fire with tons of new technologies too. Texas is pumping serious monies into everything 5G. It'll be an addon package for guys with tractors.. Making all of this fun and easy with Gentoo, should help grow our distro. The Texas Universities are moving to a new multi-homed private fiber network, where each link is 100G fiber based. So every campus in Texas, will soon be hotbeds for 5G R and play. Since the state of Texas has many 5G chip manufactures and many custom Rf shops, it's gonna be the worlds hotspot for 5G, imho. There's talk of Texas sharing this 100G multi-route network with Oklahoma, Arkansas and Mississippi; a brilliant move imho. Gentoo could 'own' Texas, with just a wee bit of effort. IBM taking over CoreOS(new version of RedHat) is leaving a very foul taste in many circles. 5G + gentoo + embedded toys, is going to be FUN FUN FUN. Then I'll be off to other states, via a hacked out Redneck camper.. and too many microProcessors Thanks Rich, your insights and comments are always most welcome. James
Re: [gentoo-user] Rasp-Pi-4 Gentoo servers
On Thu, Feb 27, 2020 at 4:25 PM james wrote: > > Yea, I was not clear. I'd run the mail-server, on a 'cluster' (4 or > more), not an individual pi-board unless it was beef up, processor and > ram wise. Gig E would also be on my list. > Unless you have some niche need I wouldn't generally run servers on Pis. The biggest issue with ARM is that all the cheap platforms are starved for RAM, and RAM is one of the biggest issues when running services. And of course the Pi in particular has IO issues (as do many other cheap SBCs but this is less of an ARM issue). The RAM issue isn't so many an ARM issue as a supply/demand thing - the only people asking for 64GB ARM boards are big companies that are willing to pay a lot for them. I do actually run a few services on Pis - DNS, DHCP, and a VPN gateway. That's about it. These are fairly non-demanding tasks that the hardware doesn't struggle with, and the data is almost entirely static so an occasional backup makes any kind of recovery trivial. The only reason I run these services on Pis is that they are fairly fundamental to having a working network. Most of my services are running in containers on a server, but I don't want to have to think about taking a server down for maintenance and then literally every IOT device in the house won't work. These particular services are also basically dependency-free which means I can just boot them up and they just do their jobs, while they remain a dependency for just about everything else on the network. When you start running DHCP in a container you have more complex dependency issues. A fairly cheap amd64 system can run a ton of services in containers though, and it is way simpler to maintain that way. I still get quick access to snapshots/etc, but now if I want to run a gentoo container it is no big deal if 99% of the time it uses 25MB of RAM and 1% of one core, but once a month it needs 4GB of RAM and 100% of 6 cores. As long as I'm not doing an emerge -u world on half a dozen containers at once it is no big deal at all. Now, if I needed some server in some niche application that needed to be able to operate off of a car battery for a few days, then sure I'd be looking at Pis and so on. -- Rich
Re: [gentoo-user] Rasp-Pi-4 Gentoo servers
On 2/27/20 2:51 PM, Ralph Seichter wrote: * ai...@aisha.cc: I'm not too sure that running it as a mail server is impossible. I never wrote that it is impossible, only that "I would not use it as an Internet-facing production Mailserver". That's a huge difference. You are free to do as you wish, but I still consider it an unsuitable role for a wee Rasberry Pi, considering the I/O load I see on our production mail servers. SD-Cards really don't like this sort of thing. -Ralph Yea, I was not clear. I'd run the mail-server, on a 'cluster' (4 or more), not an individual pi-board unless it was beef up, processor and ram wise. Gig E would also be on my list. There are also embedded boards, that can run gentoo, with up to 16 Gigs of DDR4 ram and better internal hardware for threads and such. I certainly, did not mean to offend you, so apologies galore. I'm not into running a mail server for more than a dozen folks and an ity-bity company of just one SD card? I'd find an embedded-board that runs (8Gbyte)DDR4 ram, so writes to the storage is very fast. It's a bit too detailed to look at the plethora of hardware available, that one can get for embedded projects and the matching (sensitive) price points. No need to go there (its a morass). thanks, James
Re: [gentoo-user] Rasp-Pi-4 Gentoo servers
* ai...@aisha.cc: > I'm not too sure that running it as a mail server is impossible. I never wrote that it is impossible, only that "I would not use it as an Internet-facing production Mailserver". That's a huge difference. You are free to do as you wish, but I still consider it an unsuitable role for a wee Rasberry Pi, considering the I/O load I see on our production mail servers. SD-Cards really don't like this sort of thing. -Ralph
Re: [gentoo-user] Rasp-Pi-4 Gentoo servers
I'm not too sure that running it as a mail server is impossible. Depending on your expected traffic level, it should be more than capable enough to do it. My current server is only a 1 core + 1 GB VPS, which is much more lax than a pi-4. Depending on what guides you follow you can definitely set it up as a mail server. But I am curious how you are planning to do this, unless you have a static ip + reverse DNS configured? --- Aisha blog.aisha.cc On 2020-02-27 10:11, Ralph Seichter wrote: * james: I'm thinking about setting up a pair of Rasp-Pi-4 as DNS servers with 4GB of ram. Is that enough ram for a DNS server? For running the Nameservers, yes. Compiling Gentoo packages will likely put your SD-Card under stress, but that's just how it goes. My Model B Rev 2 of 2015 runs dnsmasq as DHCP server, NGINX, Postfix, Unbound and more for a bunch of clients in a LAN. It is quite nifty as a local DNS Resolver and DHCP server, because it is usually the fastest to boot after the occasional power outage. I would not use it as an Internet-facing production Mailserver, though, because that would generate a lot of I/O, which is not a Raspberry Pi strong suit. -Ralph
Re: [gentoo-user] Rasp-Pi-4 Gentoo servers
* james: > I'm thinking about setting up a pair of Rasp-Pi-4 as DNS servers with > 4GB of ram. Is that enough ram for a DNS server? For running the Nameservers, yes. Compiling Gentoo packages will likely put your SD-Card under stress, but that's just how it goes. My Model B Rev 2 of 2015 runs dnsmasq as DHCP server, NGINX, Postfix, Unbound and more for a bunch of clients in a LAN. It is quite nifty as a local DNS Resolver and DHCP server, because it is usually the fastest to boot after the occasional power outage. I would not use it as an Internet-facing production Mailserver, though, because that would generate a lot of I/O, which is not a Raspberry Pi strong suit. -Ralph
[gentoo-user] Rasp-Pi-4 Gentoo servers
I'm thinking about setting up a pair of Rasp-Pi-4 as DNS servers with 4GB of ram. Is that enough ram for a DNS server? https://www.amazon.com/CanaKit-Raspberry-4GB-Starter-MAX/dp/B07XPHWPRB https://wiki.gentoo.org/wiki/Embedded_systems/ARM_hardware_list If 4GB is not enough, there are some boards with 8GB of ram. What about running postfix on a third board? https://samhobbs.co.uk/2013/12/raspberry-pi-email-server-part-1-postfix All feedback is encouraged. It will not be a high volume postfix email system. As far as the DNS servers, it seems I vaguely remember some software or filters to prevent hack attacks, that just overwhelm DNS servers; or something like that. It just lowers the data flow rate. It seems this is popular, and which version of USB 3.? to use on a stick, as now usb sticks can have large capacities now (T+). Surely I'll back up the mail (spool & such) to a secondary hard drive on another system. It this all goes well, surely I put a web server on a fourth board. https://pimylifeup.com/raspberry-pi-nginx/ Granted those links are not centric to embedded gentoo, but they do cover a lot of what is needed. Further suggestions are most welcome. This is a fun to do project, useful as it would basically be easy to duplicate, so I can travel anywhere, get a few static IPs and deploy a small business, self-sufficient, network. So the Rpi4 boards will all need their own unique, static Ip. But, another questions is while they are running as 4 distinct gentoo servers, could they also double as a quad-gentoo cluster? https://www.picocluster.com/collections/raspberry-pi4 Security pointers are most welcome too. The eventual idea is to have a fixed home network, but be able to travel around in my pickup/cabovercamper and have what I need right there, or provide it to friends as a ready to go small network. Embedded Gentoo, low-power hardware and 100% (gentoo) source driven). The S20 (520G ram) phone would server as the router. (2) 5G cell phones running on (2) different 5G service vendor, could make the services and cluster multi-homed? Would all of this work with just IP6 and a fancy cell phone, running gentoo? We shall see. James
