Re: [ceph-users] Ceph for "home lab" / hobbyist use?
I run Ceph on both a home server and a personal offsite backup server (both single-host setups). It's definitely feasible and comes with a lot of advantages over traditional RAID and ZFS and the like. The main disadvantages are performance overhead and resource consumption. On 07/09/2019 06.16, William Ferrell wrote: They're about $50 each, can boot from MicroSD or eMMC flash (basically an SSD with a custom connector), and have one SATA port. They have 8-core 32-bit CPUs, 2GB of RAM and a gigabit ethernet port. Four of them (including disks) can run off a single 12V/8A power adapter (basically 100 watts per set of 4). The obvious appeal is price, plus they're stackable so they'd be easy to hide away in a closet. Is it feasible for these to work as OSDs at all? The Ceph hardware recommendations page suggests OSDs need 1GB per TB of space, so does this mean these wouldn't be suitable with, say, a 4TB or 8TB disk? Or would they work, but just more slowly? 2GB seems tight, but *should* work if you're literally running an OSD and only an OSD on the thing. I use a 32GB RAM server to run 64TB worth of raw storage (8x8TB SATA disks), plus mon/mds, plus a bunch of unrelated applications and servers, routing, and even a desktop UI (I'll soon be splitting off some of these duties, since this box has grown into way too much of a kitchen sink). It used to be 16GB when I first moved to Ceph, and that juuust about worked but it was tight. So the 1GB/1TB recommendation is ample, 1GB/2TB works well, and 1GB/4TB is tight. I configure my OSDs for a 1.7GB memory target, so that should work on your 2GB RAM board, but it doesn't give you much headroom for increased consumption during recovery. To be safe I'd set them up for 1.2GB or so target on a board like yours. I would recommend keeping your PG count low and relying on the Ceph balancer to even out your disk usage. Keep in mind that the true number of PGs is your PG count multiplied by your pool width: that's your replica count for replicated pools, or your erasure code k+m width for EC pools. I use 8 PGs for my metadata pool (x3 replication = 24 PGs, 3 per OSD) and 64 PGs for my data pool (x 5.2 RS profile = 448 PG, 56 per OSD). All my data is on CephFS on my home setup. If you use EC and since your memory usage is tight, you might want to drop that a bit, e.g. target something like 24 PGs per OSD. For utilization balance, as long as your overall PG count (multiplied by pool width) is a multiple of your OSD count, you should be able to achieve a "perfect" distribution when using the balancer (otherwise you'll be off by +/- 1 PG, where more PGs makes that a smaller imbalance). Pushing my luck further (assuming the HC2 can handle OSD duties at all), is that enough muscle to run the monitor and/or metadata servers? Should monitors and MDS's be run separately, or can/should they piggyback on hosts running OSDs? The MDS will need RAM to cache your metadata (and you want it to, because it makes performance *way* better). I would definitely keep it well away from such tight OSDs. In fact the MDS in my setup uses more RAM than any given OSD, about 2GB or so (the cache size is also configurable). If you have fewer large files then resource consumption will be lower (I have a mixed workload with most of the space taken up by large files, but I have a few trees full of tiny ones, totaling several million files). You might get away with dedicating a separate identical board to the MDS. Maybe consider multi-mds, but I've found that during recovery it adds some phases that eat up even more RAM, so that might not be a good idea. The mon isn't as bad, but I wouldn't push it and try to co-host it on such limited hosts. Mine's sitting at about ~1GB res. In either case, keep some larger x86 box that you have the software installed on and can plug in some disks into around. In the worst case, if you end up in an out-of-memory loop, you can move e.g. the MDS or some OSDs to that machine and bring the cluster back to health. I'd be perfectly happy with a setup like this even if it could only achieve speeds in the 20-30MB/sec range. I can't speak for small ARM boards, but my performance (on a quad-core Haswell i5 hosting everything) is somewhere around half of what I'd get on a RAID6 over the same storage, using ~equivalent RS 5.2 encoding and dm-crypt (AES-NI) under the OSDs. Since you'd be running a single OSD per host, I imagine you should be able to get reasonable aggregate performance out of the whole thing, but I've never tried a setup like that. I'm actually considering this kind of thing in the future (moving from one monolithic server to a more cluster-like setup) but it's just an idea for now. -- Hector Martin (hec...@marcansoft.com) Public Key: https://mrcn.st/pub ___ ceph-users mailing list ceph-users@lists.ceph.com
Re: [ceph-users] Ceph for "home lab" / hobbyist use?
Saturday, 7 September 2019, 15.25 +0200 from wil...@gmail.com >On a related note, I came across this hardware while searching around >on this topic: https://ambedded.com/ambedded_com/AR M Interesting to see the cost of those. 8 LFF drives in 1U is pretty dense. Anyone using similar concepts in enterprise environments? Jesper > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph for "home lab" / hobbyist use?
On Sat, Sep 7, 2019 at 7:47 AM Paul Emmerich wrote: > > On Sat, Sep 7, 2019 at 12:55 AM William Ferrell wrote: > > > > On Fri, Sep 6, 2019 at 6:37 PM Peter Woodman wrote: > > > > > > 2GB ram is gonna be really tight, probably. > > > > Bummer. So it won't be enough for any Ceph component to run reliably? > > 2 GB is tough for an OSD. The main problem is recovery. If you attempt > this: the trick is the use fewer PGs per OSD than recommended. Interesting. I take it that reduces the memory requirements? What are the consequences of using fewer PGs than recommended? On a related note, I came across this hardware while searching around on this topic: https://ambedded.com/ambedded_com/ARM_Server_platform.html They position these systems as easy-to-deploy Ceph systems. I'm afraid to even ask the manufacturer how much those things cost, but I noticed the specs of the individual nodes aren't really much different from the HC2, especially the "Mars 200" model. In that model, each node has a dual core CPU and only 2GB of RAM. The 400 model changes the CPU type (but stays dual-core) and bumps the memory up to 4GB. So would the individual nodes in those machines have the same limitations? I guess the chassis provides an internal network fabric so the nodes can talk to each other super fast, so maybe that makes a difference. BTW thanks everyone who's responded so far! You've all been really helpful clearing up my newbie questions :) ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph for "home lab" / hobbyist use?
On Sat, Sep 7, 2019 at 12:55 AM William Ferrell wrote: > > On Fri, Sep 6, 2019 at 6:37 PM Peter Woodman wrote: > > > > 2GB ram is gonna be really tight, probably. > > Bummer. So it won't be enough for any Ceph component to run reliably? 2 GB is tough for an OSD. The main problem is recovery. If you attempt this: the trick is the use fewer PGs per OSD than recommended. -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 > > Drat. The nice thing about the HC2 is the fact that it can power the > attached SATA disk and itself through one barrel connector and > includes a stackable case to mount the disk and act as a heat sink. > Would have been so easy to stack those up and use fewer cables/power > adapters to do it. I wish I could convince the Hardkernel guys to drop > a new model with this form factor that uses their 64-bit 4GB chipset. > > > However, I do something similar at home with a bunch of rock64 4gb boards, > > and it works well. > > Nice! Would you mind describing your setup in more detail? It's > encouraging to hear this works on arm (I wasn't sure if that was still > supported). I'm also curious how you manage all the power supplies, > cables and disks. Assuming you're using external 3.5" USB 3 disks that > require their own power, you're already at 8 power adapters for just 4 > nodes. > > > There are sometimes issues with the released ARM packages (frequently crc32 > > doesn;'t work, which isn't great), so you may have to build your own on the > > board you're targeting or on something like scaleway, YMMV. > > Thanks, that's good to know. > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph for "home lab" / hobbyist use?
On Fri, Sep 6, 2019 at 6:37 PM Peter Woodman wrote: > > 2GB ram is gonna be really tight, probably. Bummer. So it won't be enough for any Ceph component to run reliably? Drat. The nice thing about the HC2 is the fact that it can power the attached SATA disk and itself through one barrel connector and includes a stackable case to mount the disk and act as a heat sink. Would have been so easy to stack those up and use fewer cables/power adapters to do it. I wish I could convince the Hardkernel guys to drop a new model with this form factor that uses their 64-bit 4GB chipset. > However, I do something similar at home with a bunch of rock64 4gb boards, > and it works well. Nice! Would you mind describing your setup in more detail? It's encouraging to hear this works on arm (I wasn't sure if that was still supported). I'm also curious how you manage all the power supplies, cables and disks. Assuming you're using external 3.5" USB 3 disks that require their own power, you're already at 8 power adapters for just 4 nodes. > There are sometimes issues with the released ARM packages (frequently crc32 > doesn;'t work, which isn't great), so you may have to build your own on the > board you're targeting or on something like scaleway, YMMV. Thanks, that's good to know. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph for "home lab" / hobbyist use?
2GB ram is gonna be really tight, probably. However, I do something similar at home with a bunch of rock64 4gb boards, and it works well. There are sometimes issues with the released ARM packages (frequently crc32 doesn;'t work, which isn't great), so you may have to build your own on the board you're targeting or on something like scaleway, YMMV. On Fri, Sep 6, 2019 at 6:16 PM Cranage, Steve wrote: > I use those HC2 nodes for my home Ceph cluster, but my setup only has to > support the librados API, my software does HSM between regular XFS file > systems and the RADOS api so I don’t need the other MDS and the rest so I > can’t tell you if you’ll be happy in your configuration. > > > > Steve Cranage > > Principal Architect, Co-Founder > > DeepSpace Storage > > 719-930-6960 > > > -- > *From:* ceph-users on behalf of > William Ferrell > *Sent:* Friday, September 6, 2019 3:16:30 PM > *To:* ceph-users@lists.ceph.com > *Subject:* [ceph-users] Ceph for "home lab" / hobbyist use? > > Hello everyone! > > After years of running several ZFS pools on a home server and several > disk failures along the way, I've decided that my current home storage > setup stinks. So far there hasn't been any data loss, but > recovering/"resilvering" a ZFS pool after a disk failure is a > nail-biting experience. I also think the way things are set up now > isn't making the best use of all the disks attached to the server; > they were acquired over time instead of all at once, so I've got 4 > 4-disk raidz1 pools, each in their own enclosures. If any enclosure > dies, all that pool's data is lost. Despite having a total of 16 disks > in use for storage, the entire system can only "safely" lose one disk > before there's a risk of a second failure taking a bunch of data with > it. > > I'd like to ask the list's opinions on running a Ceph cluster in a > home environment as a filer using cheap, low-power systems. I don't > have any expectations for high performance (this will be built on a > gigabit network, and just used for backups and streaming videos, > music, etc. for two people); the main concern is resiliency if one or > two disks fail, and the secondary concern is having a decent usable > storage capacity. Being able to slowly add capacity to the cluster one > disk at a time is a very appealing bonus. > > I'm interested in using these things as OSDs (and hopefully monitors > and metadata servers): > https://www.hardkernel.com/shop/odroid-hc2-home-cloud-two/ > > They're about $50 each, can boot from MicroSD or eMMC flash (basically > an SSD with a custom connector), and have one SATA port. They have > 8-core 32-bit CPUs, 2GB of RAM and a gigabit ethernet port. Four of > them (including disks) can run off a single 12V/8A power adapter > (basically 100 watts per set of 4). The obvious appeal is price, plus > they're stackable so they'd be easy to hide away in a closet. > > Is it feasible for these to work as OSDs at all? The Ceph hardware > recommendations page suggests OSDs need 1GB per TB of space, so does > this mean these wouldn't be suitable with, say, a 4TB or 8TB disk? Or > would they work, but just more slowly? > > Pushing my luck further (assuming the HC2 can handle OSD duties at > all), is that enough muscle to run the monitor and/or metadata > servers? Should monitors and MDS's be run separately, or can/should > they piggyback on hosts running OSDs? > > I'd be perfectly happy with a setup like this even if it could only > achieve speeds in the 20-30MB/sec range. > > Is this a dumb idea, or could it actually work? Are there any other > recommendations among Ceph users for low-end hardware to cobble > together a working cluster? > > Any feedback is sincerely appreciated. > > Thanks! > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph for "home lab" / hobbyist use?
I use those HC2 nodes for my home Ceph cluster, but my setup only has to support the librados API, my software does HSM between regular XFS file systems and the RADOS api so I don’t need the other MDS and the rest so I can’t tell you if you’ll be happy in your configuration. Steve Cranage Principal Architect, Co-Founder DeepSpace Storage 719-930-6960 [cid:image001.png@01D3FCBC.58FDB6F0] From: ceph-users on behalf of William Ferrell Sent: Friday, September 6, 2019 3:16:30 PM To: ceph-users@lists.ceph.com Subject: [ceph-users] Ceph for "home lab" / hobbyist use? Hello everyone! After years of running several ZFS pools on a home server and several disk failures along the way, I've decided that my current home storage setup stinks. So far there hasn't been any data loss, but recovering/"resilvering" a ZFS pool after a disk failure is a nail-biting experience. I also think the way things are set up now isn't making the best use of all the disks attached to the server; they were acquired over time instead of all at once, so I've got 4 4-disk raidz1 pools, each in their own enclosures. If any enclosure dies, all that pool's data is lost. Despite having a total of 16 disks in use for storage, the entire system can only "safely" lose one disk before there's a risk of a second failure taking a bunch of data with it. I'd like to ask the list's opinions on running a Ceph cluster in a home environment as a filer using cheap, low-power systems. I don't have any expectations for high performance (this will be built on a gigabit network, and just used for backups and streaming videos, music, etc. for two people); the main concern is resiliency if one or two disks fail, and the secondary concern is having a decent usable storage capacity. Being able to slowly add capacity to the cluster one disk at a time is a very appealing bonus. I'm interested in using these things as OSDs (and hopefully monitors and metadata servers): https://www.hardkernel.com/shop/odroid-hc2-home-cloud-two/ They're about $50 each, can boot from MicroSD or eMMC flash (basically an SSD with a custom connector), and have one SATA port. They have 8-core 32-bit CPUs, 2GB of RAM and a gigabit ethernet port. Four of them (including disks) can run off a single 12V/8A power adapter (basically 100 watts per set of 4). The obvious appeal is price, plus they're stackable so they'd be easy to hide away in a closet. Is it feasible for these to work as OSDs at all? The Ceph hardware recommendations page suggests OSDs need 1GB per TB of space, so does this mean these wouldn't be suitable with, say, a 4TB or 8TB disk? Or would they work, but just more slowly? Pushing my luck further (assuming the HC2 can handle OSD duties at all), is that enough muscle to run the monitor and/or metadata servers? Should monitors and MDS's be run separately, or can/should they piggyback on hosts running OSDs? I'd be perfectly happy with a setup like this even if it could only achieve speeds in the 20-30MB/sec range. Is this a dumb idea, or could it actually work? Are there any other recommendations among Ceph users for low-end hardware to cobble together a working cluster? Any feedback is sincerely appreciated. Thanks! ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Ceph for "home lab" / hobbyist use?
Hello everyone! After years of running several ZFS pools on a home server and several disk failures along the way, I've decided that my current home storage setup stinks. So far there hasn't been any data loss, but recovering/"resilvering" a ZFS pool after a disk failure is a nail-biting experience. I also think the way things are set up now isn't making the best use of all the disks attached to the server; they were acquired over time instead of all at once, so I've got 4 4-disk raidz1 pools, each in their own enclosures. If any enclosure dies, all that pool's data is lost. Despite having a total of 16 disks in use for storage, the entire system can only "safely" lose one disk before there's a risk of a second failure taking a bunch of data with it. I'd like to ask the list's opinions on running a Ceph cluster in a home environment as a filer using cheap, low-power systems. I don't have any expectations for high performance (this will be built on a gigabit network, and just used for backups and streaming videos, music, etc. for two people); the main concern is resiliency if one or two disks fail, and the secondary concern is having a decent usable storage capacity. Being able to slowly add capacity to the cluster one disk at a time is a very appealing bonus. I'm interested in using these things as OSDs (and hopefully monitors and metadata servers): https://www.hardkernel.com/shop/odroid-hc2-home-cloud-two/ They're about $50 each, can boot from MicroSD or eMMC flash (basically an SSD with a custom connector), and have one SATA port. They have 8-core 32-bit CPUs, 2GB of RAM and a gigabit ethernet port. Four of them (including disks) can run off a single 12V/8A power adapter (basically 100 watts per set of 4). The obvious appeal is price, plus they're stackable so they'd be easy to hide away in a closet. Is it feasible for these to work as OSDs at all? The Ceph hardware recommendations page suggests OSDs need 1GB per TB of space, so does this mean these wouldn't be suitable with, say, a 4TB or 8TB disk? Or would they work, but just more slowly? Pushing my luck further (assuming the HC2 can handle OSD duties at all), is that enough muscle to run the monitor and/or metadata servers? Should monitors and MDS's be run separately, or can/should they piggyback on hosts running OSDs? I'd be perfectly happy with a setup like this even if it could only achieve speeds in the 20-30MB/sec range. Is this a dumb idea, or could it actually work? Are there any other recommendations among Ceph users for low-end hardware to cobble together a working cluster? Any feedback is sincerely appreciated. Thanks! ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
