Re: [57north-discuss] Hosting servers
On 25/07/2021 22:43, Andy Gaskell wrote: [...] Do all six of them need to be on all the time? If not, maybe they could run on a remotely operated power switch thing. Hi, Andy. I'm running LizardFS SDS, so they all need to be on 24/7, but the disks do spin down when not in use. However, I've decided not to pursue the idea of hosting them at 57North and will run them in my garage again. Thanks for your thoughts about the idea, Tony. -- Minke Informatics Limited, Registered in Scotland - Company No. SC419028 Registered Office: 3 Donview, Bridge of Alford, AB33 8QJ, Scotland (UK) tel. +44(0)19755 63548http://minke-informatics.co.uk mob. +44(0)7985 078324mailto:tony.tra...@minke-informatics.co.uk ___ 57north-discuss mailing list 57north-discuss@lists.57north.co http://lists.57north.co/listinfo/57north-discuss
Re: [57north-discuss] Hosting servers
On 25/07/2021 23:45, Alfie Pates wrote: When I was looking @ putting video encoding compute boxes in a colo with metered power, I played with wake-on-demand to only spin them up when they actually had jobs to do. I was using Proxmox' built in WoL nonsense along with some terrifying calendar automation to schedule boxes being alive. Someone who cared a little bit more could probably write quite a nice thing where you can "book" time and have a box/boxes spin up, and gracefully shut down when done. Hi, Alfie. These servers run the LizardFS SDS and need to be on 24/7 because I'm using ec(2,2) redundancy. The disks do spin down when not in use, but many of the jobs I run take several days anyway and from what I've read recently on the Beowulf mailing list, the consequences of switching systems off/on to conserver power are dramatically shortened lifetime for the hardware, so I'm not thinking of going down that route. Apropos of very little, I used to heat my old flat's bedroom w/ a box because the central heating in that flat was *so bad* that it was cheaper to get the company to pay me for colo! Lived under the bed - Terrible place for a computer. I'm presently heating our study a lot more than I really want to, so the systems are going into my garage again. I've now decided not to pursue the idea of 57North hosting my servers. Thanks for your thoughts, Tony. -- Minke Informatics Limited, Registered in Scotland - Company No. SC419028 Registered Office: 3 Donview, Bridge of Alford, AB33 8QJ, Scotland (UK) tel. +44(0)19755 63548http://minke-informatics.co.uk mob. +44(0)7985 078324mailto:tony.tra...@minke-informatics.co.uk ___ 57north-discuss mailing list 57north-discuss@lists.57north.co http://lists.57north.co/listinfo/57north-discuss
Re: [57north-discuss] Hosting servers
When I was looking @ putting video encoding compute boxes in a colo with metered power, I played with wake-on-demand to only spin them up when they actually had jobs to do. I was using Proxmox' built in WoL nonsense along with some terrifying calendar automation to schedule boxes being alive. Someone who cared a little bit more could probably write quite a nice thing where you can "book" time and have a box/boxes spin up, and gracefully shut down when done. Apropos of very little, I used to heat my old flat's bedroom w/ a box because the central heating in that flat was *so bad* that it was cheaper to get the company to pay me for colo! Lived under the bed - Terrible place for a computer. ~a___ 57north-discuss mailing list 57north-discuss@lists.57north.co http://lists.57north.co/listinfo/57north-discuss
Re: [57north-discuss] Hosting servers
Hiya I'm a bit late to this chat, sorry. Yea, power costs are pretty scary, my maths/bills pegs it at £2365 p/a inc vat, for 6x250w 24/7. Do all six of them need to be on all the time? If not, maybe they could run on a remotely operated power switch thing. I don't know if AAOD pay a fixed fee for utilities at the moment, I sort of got that impression before, though I'm not sure where from. Can you stash them in a cupboard in the uni somewhere? Like in https://www.theregister.com/2021/07/16/bofh_2021_episode_12/ I think all my office infrastructure runs on Arm, but it's not crunching numbers, so it's a different sort of gig. Also, I'm a cheap-ass when it comes to hardware. Cheers Andy On Sun, 25 Jul 2021 at 19:24, Tony Travis < tony.tra...@minke-informatics.co.uk> wrote: > On 25/07/2021 16:29, Robert McWilliam wrote: > > [...] > > I'd be opposed to 57North paying for the electricity for this. It'll be > in the ball park of all our income⁰. As much as I'd like to play with a > compute cluster I don't think it's worth (possibly more than) all of the > space's money. > > Hi, Robert. > > OK, fair enough - I'll put it in my garage. > > Thanks for the reality-check :-) > > Bye, > >Tony. > > -- > Minke Informatics Limited, Registered in Scotland - Company No. SC419028 > Registered Office: 3 Donview, Bridge of Alford, AB33 8QJ, Scotland (UK) > tel. +44(0)19755 63548http://minke-informatics.co.uk > mob. +44(0)7985 078324mailto:tony.tra...@minke-informatics.co.uk > ___ > 57north-discuss mailing list > 57north-discuss@lists.57north.co > http://lists.57north.co/listinfo/57north-discuss > -- Andy Gaskell Head Designer and Developer Software Systems: Open For Business Email: a...@ssofb.co.uk Mobile: 07745 924 449 Office: 01224 634360 Office: 3 Belvidere Street, Aberdeen, AB25 2QS Limited company: SC347800 ___ 57north-discuss mailing list 57north-discuss@lists.57north.co http://lists.57north.co/listinfo/57north-discuss
Re: [57north-discuss] Hosting servers
On 25/07/2021 16:29, Robert McWilliam wrote: [...] I'd be opposed to 57North paying for the electricity for this. It'll be in the ball park of all our income⁰. As much as I'd like to play with a compute cluster I don't think it's worth (possibly more than) all of the space's money. Hi, Robert. OK, fair enough - I'll put it in my garage. Thanks for the reality-check :-) Bye, Tony. -- Minke Informatics Limited, Registered in Scotland - Company No. SC419028 Registered Office: 3 Donview, Bridge of Alford, AB33 8QJ, Scotland (UK) tel. +44(0)19755 63548http://minke-informatics.co.uk mob. +44(0)7985 078324mailto:tony.tra...@minke-informatics.co.uk ___ 57north-discuss mailing list 57north-discuss@lists.57north.co http://lists.57north.co/listinfo/57north-discuss
Re: [57north-discuss] Hosting servers
On Sat, 24 Jul 2021, at 23:15, Tony Travis wrote: > On 24/07/2021 16:52, Robert McWilliam wrote: > > 6x250W comes to about £2400/year (with what I'm paying for electricity, I > > don't know how that compares to price at the hackspace). And north of > > £3k/year if they're mostly busy and at the 350W each end of the range. What > > were you thinking for metering/covering cost of the electricity usage? > > I was thinking about 57North paying for the electricity and Internet in > return for member access to the compute and storage of the six servers. I'd be opposed to 57North paying for the electricity for this. It'll be in the ball park of all our income⁰. As much as I'd like to play with a compute cluster I don't think it's worth (possibly more than) all of the space's money. Robert ⁰ Accounts for last year¹ give us ~£3k income and ~£2k surplus, membership graph definitely looks like we'd be down from that now.² ¹ https://github.com/hackerdeen/meetings/tree/master/2021 ² https://hub.57north.org.uk/api/membership/graph Robert McWilliam r...@allmail.netargh.technology What do you consider to be your median achievement? ___ 57north-discuss mailing list 57north-discuss@lists.57north.co http://lists.57north.co/listinfo/57north-discuss
Re: [57north-discuss] Hosting servers
On 25/07/2021 13:35, Tom Jones wrote: [...] I think this would be a great thing for the space to do, espectially considering the two ISP that are providing us service. Hi, Tom. Thanks for your support :-) I think that we would need to run this by Isobel at AAOD though. This is likely to be a substantial increase to the currect electricity bill. We would also need to figure out how to rack these machine reasonably securely. They are pedestal (tower) servers in (quiet) Fractal Design Define XL R2 cases, not rack-mounted servers and they are quiet enough to be run in an office environment taking into account what Rob said about the 2kW heat load (i.e. most of the power consumed ends up as heat). It might be worth mentioning to Isobel that in the winter this might actually reduce the heating bill. I'm coming into the University on Thu 29 Jul, to bring back one of our servers that I upgraded at home because of Covid-19 restrictions, if you want to meet up for a coffee? Bye, Tony. -- Minke Informatics Limited, Registered in Scotland - Company No. SC419028 Registered Office: 3 Donview, Bridge of Alford, AB33 8QJ, Scotland (UK) tel. +44(0)19755 63548http://minke-informatics.co.uk mob. +44(0)7985 078324mailto:tony.tra...@minke-informatics.co.uk ___ 57north-discuss mailing list 57north-discuss@lists.57north.co http://lists.57north.co/listinfo/57north-discuss
Re: [57north-discuss] Hosting servers
On Sat, Jul 24, 2021 at 12:58:09PM +0100, Tony Travis wrote: > Hi, > > Brexit and Covid-19 have wreaked havoc on my business plans with Luca in > Italy and I'm reconsidering if I should ship six servers that I've built > to Italy or host them here in Scotland? > > The servers are Dual 8-core (16 thread) Opteron 6380 + 128GiB RAM, OS on > a 500GB SSD RAID1 and 4@8TB on an LSI9211-4i with an LSI9211-8i to allow > for expansion of another 8@SATA3 disks. > > I discussed this briefly with Iain and Tom, but I would welcome any > ideas or suggestions: Plan 1 is to host them in my garage, which I've > done before - I've just contacted Converged to see if they can provide > FTTP here, but I'm not optimistic (currently I'm on PlusNet FTTC/VDSL). > > Is it a realistic proposition for me to stike a deal with the Hackspace, > where, for example, 57North hosts my servers in the Space in return for > me providing login accounts and free VM's to 57North members? > > I talked to Tom about running OpenStack on these servers as a Hackspace > project, but just now it runs Ubuntu-MATE 20.04 LTS and LizardFS SDS. > > Just asking ;-) Hi Tony, I think this would be a great thing for the space to do, espectially considering the two ISP that are providing us service. I think that we would need to run this by Isobel at AAOD though. This is likely to be a substantial increase to the currect electricity bill. We would also need to figure out how to rack these machine reasonably securely. - Tom ___ 57north-discuss mailing list 57north-discuss@lists.57north.co http://lists.57north.co/listinfo/57north-discuss
Re: [57north-discuss] Hosting servers
On 24/07/2021 16:52, Robert McWilliam wrote: [...] To be clear, they consume about 250W when idle and ~350W when working. FFS, Tony. Heh, yes I was just typing a message checking if you meant W but you noticed yourself before I got as far as sending :) Hi, Robert. Thanks :-) 6x250W comes to about £2400/year (with what I'm paying for electricity, I don't know how that compares to price at the hackspace). And north of £3k/year if they're mostly busy and at the 350W each end of the range. What were you thinking for metering/covering cost of the electricity usage? I was thinking about 57North paying for the electricity and Internet in return for member access to the compute and storage of the six servers. Couple of kW is enough to probably have noticeable impact on room temp. Extra heating might be welcome in winter but not when weather is like it has been recently. Do you know if the units in the roof do cooling as well as heating? Might need to account for extra load on them. What were you thinking for cooling? Passive cooling (windows open...) Yeah, I'd agree with you that EC2 is expensive for stuff running 24/7 (for a decent length of time). For stuff I'm responsible for we've got a mix of bought and rented dedicated servers for things that run 24/7 and use EC2/Azure for intermittent tasks. I didn't mean to suggest that these servers weren't right choice for your needs - I expected you would have done appropriate analysis. Was just noting they're maybe not a good fit for more "general" loads. I don't mind if it's not of interest to 57North, but I thought it might be useful to have our own compute/storage cluster and experiment with different approaches to HPC. This is what I was setting up initially when we were in Union Street with the RackableSystems 2U Opteron 200's, which I replaced with Opteron 2,000's that are still running in Italy. Bye, Tony. -- Minke Informatics Limited, Registered in Scotland - Company No. SC419028 Registered Office: 3 Donview, Bridge of Alford, AB33 8QJ, Scotland (UK) tel. +44(0)19755 63548http://minke-informatics.co.uk mob. +44(0)7985 078324mailto:tony.tra...@minke-informatics.co.uk ___ 57north-discuss mailing list 57north-discuss@lists.57north.co http://lists.57north.co/listinfo/57north-discuss
Re: [57north-discuss] Hosting servers
On 24/07/2021 15:53, Tony Travis wrote: On 24/07/2021 15:38, Tony Travis wrote: [...] They idle at about 250Kw/h and use about 350Kw/h working (no GPGPU). Hi, Robert. You knew, of course that I meant 250W/h ;-) Hi, Robert. Sorry, again, I really hate making silly mistakes in public :-( To be clear, they consume about 250W when idle and ~350W when working. FFS, Tony. -- Minke Informatics Limited, Registered in Scotland - Company No. SC419028 Registered Office: 3 Donview, Bridge of Alford, AB33 8QJ, Scotland (UK) tel. +44(0)19755 63548http://minke-informatics.co.uk mob. +44(0)7985 078324mailto:tony.tra...@minke-informatics.co.uk ___ 57north-discuss mailing list 57north-discuss@lists.57north.co http://lists.57north.co/listinfo/57north-discuss
Re: [57north-discuss] Hosting servers
On 24/07/2021 15:38, Tony Travis wrote: [...] They idle at about 250Kw/h and use about 350Kw/h working (no GPGPU). Hi, Robert. You knew, of course that I meant 250W/h ;-) Head full of KW/h just now working out my bill! Sorry about that, Tony. -- Minke Informatics Limited, Registered in Scotland - Company No. SC419028 Registered Office: 3 Donview, Bridge of Alford, AB33 8QJ, Scotland (UK) tel. +44(0)19755 63548http://minke-informatics.co.uk mob. +44(0)7985 078324mailto:tony.tra...@minke-informatics.co.uk ___ 57north-discuss mailing list 57north-discuss@lists.57north.co http://lists.57north.co/listinfo/57north-discuss
Re: [57north-discuss] Hosting servers
On 24/07/2021 15:10, Robert McWilliam wrote: [...] ¹ Though they've got pretty high RAM to compute ratio so they could win out for tasks with high RAM needs, or working on more data than you can upload in a reasonable time etc. Hi, Robert. They also have ECC registered RAM, which is important for my work. Bye, Tony. -- Minke Informatics Limited, Registered in Scotland - Company No. SC419028 Registered Office: 3 Donview, Bridge of Alford, AB33 8QJ, Scotland (UK) tel. +44(0)19755 63548http://minke-informatics.co.uk mob. +44(0)7985 078324mailto:tony.tra...@minke-informatics.co.uk ___ 57north-discuss mailing list 57north-discuss@lists.57north.co http://lists.57north.co/listinfo/57north-discuss
Re: [57north-discuss] Hosting servers
On 24/07/2021 15:10, Robert McWilliam wrote: On Sat, 24 Jul 2021, at 12:58, Tony Travis wrote: The servers are Dual 8-core (16 thread) Opteron 6380 + 128GiB RAM, OS on a 500GB SSD RAID1 and 4@8TB on an LSI9211-4i with an LSI9211-8i to allow for expansion of another 8@SATA3 disks. What does power usage of those systems look like? Would they be always on or some kind of boot as needed set up? Hi, Robert. Forgot to answer this question: They have to run 24/7 if using LizardFS SDS configured for erasure-coding redundancy. Bye, Tony. -- Minke Informatics Limited, Registered in Scotland - Company No. SC419028 Registered Office: 3 Donview, Bridge of Alford, AB33 8QJ, Scotland (UK) tel. +44(0)19755 63548http://minke-informatics.co.uk mob. +44(0)7985 078324mailto:tony.tra...@minke-informatics.co.uk ___ 57north-discuss mailing list 57north-discuss@lists.57north.co http://lists.57north.co/listinfo/57north-discuss
Re: [57north-discuss] Hosting servers
On 24/07/2021 15:10, Robert McWilliam wrote: On Sat, 24 Jul 2021, at 12:58, Tony Travis wrote: The servers are Dual 8-core (16 thread) Opteron 6380 + 128GiB RAM, OS on a 500GB SSD RAID1 and 4@8TB on an LSI9211-4i with an LSI9211-8i to allow for expansion of another 8@SATA3 disks. What does power usage of those systems look like? Would they be always on or some kind of boot as needed set up? Hi, Robert. They idle at about 250Kw/h and use about 350Kw/h working (no GPGPU). I think they're on the cusp of cost of electricity to run them being more than costs for an EC2⁰ instance with the same performance¹. Comparison of them to some modern chips: https://www.cpubenchmark.net/compare/AMD-Opteron-6380-vs-Intel-i7-11850H-vs-AMD-Ryzen-5-5600X-vs-AMD-Ryzen-9-5950X-vs-AMD-Ryzen-9-5900X/2498vs4342vs3859vs3862vs3870 I'm using second-hand Opteron 6380's because their memory management is better than equivalently priced Xeons and I don't care about their FP performance. When I do care about FP performance, I use GPGPU not Xeons. In my experience, EC2 is much more expensive than running your own system iff you use it a lot - Remember my "Zen and the art of computer maintenance talk": Amazon EC2 is more cost-effective the less you use it. When I started out on this path, I was using EC2 and it cost a lot to have a 4-core + 8GB RAM + 20GB disk VM and my work required 16-cores + 32GiB RAM + 1TB disk at the time and now requires much more than that. https://aws.amazon.com/ec2/pricing/on-demand/ Anyway, thanks for your feedback, Tony. -- Minke Informatics Limited, Registered in Scotland - Company No. SC419028 Registered Office: 3 Donview, Bridge of Alford, AB33 8QJ, Scotland (UK) tel. +44(0)19755 63548http://minke-informatics.co.uk mob. +44(0)7985 078324mailto:tony.tra...@minke-informatics.co.uk ___ 57north-discuss mailing list 57north-discuss@lists.57north.co http://lists.57north.co/listinfo/57north-discuss
Re: [57north-discuss] Hosting servers
On Sat, 24 Jul 2021, at 12:58, Tony Travis wrote: > The servers are Dual 8-core (16 thread) Opteron 6380 + 128GiB RAM, OS on > a 500GB SSD RAID1 and 4@8TB on an LSI9211-4i with an LSI9211-8i to allow > for expansion of another 8@SATA3 disks. What does power usage of those systems look like? Would they be always on or some kind of boot as needed set up? I think they're on the cusp of cost of electricity to run them being more than costs for an EC2⁰ instance with the same performance¹. Comparison of them to some modern chips: https://www.cpubenchmark.net/compare/AMD-Opteron-6380-vs-Intel-i7-11850H-vs-AMD-Ryzen-5-5600X-vs-AMD-Ryzen-9-5950X-vs-AMD-Ryzen-9-5900X/2498vs4342vs3859vs3862vs3870 Robert ⁰ Other cloud providers are available. ¹ Though they've got pretty high RAM to compute ratio so they could win out for tasks with high RAM needs, or working on more data than you can upload in a reasonable time etc. Robert McWilliam r...@allmail.netargh.technology What do you consider to be your median achievement? ___ 57north-discuss mailing list 57north-discuss@lists.57north.co http://lists.57north.co/listinfo/57north-discuss
