Re: Arm ports build machines (was Re: Arch qualification for buster: call for DSA, Security, toolchain concerns)
spoke again to TL and asked if pine64 would be willing to look at sponsorship witn rockpro64 boards (the ones that take 4x PCIe): if someone from debian were to contact him direct he would happily consider it. i then asked him if i could cc him into this discussion and he said he was way *way* too busy to enter into another mailing list discussion, so if someone from debian emails me privately, off-list, i will then cc him and/or put them in touch with him on irc. i can also be reached on freenode and oftc as "lkcl", if that is easier. l.
Re: Arm ports build machines (was Re: Arch qualification for buster: call for DSA, Security, toolchain concerns)
On Fri, Jun 29, 2018 at 8:13 PM, Luke Kenneth Casson Leighton wrote: > On Fri, Jun 29, 2018 at 6:59 PM, Jonathan Wiltshire wrote: > >>> also worth noting, they're working on a 2U rackmount server which >>> will have i think something insane like 48 Rock64Pro boards in one >>> full-length case. > >> None of this addresses the basic DSA requirement of remote management. >> Troubling local hands to change a disk once in a while is reasonable; being >> blocked waiting for a power cycle on a regular basis is not (and I can't >> imagine hosting sponsors are wild about their employees' time being used >> for that either). > > i know exactly what you mean, i've had to deal with data centres. > i'll make sure that TL Lim is aware of this, and will ask him if > there's a way to include remote power-management / power-cycling of > boards in the planned product or if it's already been thought of. TL informs me that all the power and reset signals for all 48 of the RockPro64s tucked into the full-length 2U case are brought out to the back panel. an MCU (or MCUs) or SBC (or SBCs) may therefore be connected directly to those in order to provide *individual* remote power / reset management of all 48 RockPro64s. DIY remote power management, but it is actual remote power management. l.
Re: Arm ports build machines (was Re: Arch qualification for buster: call for DSA, Security, toolchain concerns)
--- crowd-funded eco-conscious hardware: https://www.crowdsupply.com/eoma68 On Fri, Jun 29, 2018 at 8:31 PM, Florian Weimer wrote: > * Luke Kenneth Casson Leighton: > >> that is not a surprise to hear: the massive thrashing caused by the >> linker phase not being possible to be RAM-resident will be absolutely >> hammering the drives beyond reasonable wear-and-tear limits. which is >> why i'm recommending people try "-Wl,--no-keep-memory". > > Note that ld will sometimes stuff everything into a single RWX segment > as a result, which is not desirable. florian, thank you for responding: i've put a copy of the insights that you give into the bugreport at https://sourceware.org/bugzilla/show_bug.cgi?id=22831#c16 > Unfortunately, without significant investment into historic linker > technologies (with external sorting and that kind of stuff), yes, ah ha! funnily enough the algorithm that i was asked to create back in 1988 was an external matrix-multiply, i take it you are talking about the same thing, where linking is done using explicit load-process-save cycles rather than relying on swap. > I don't > think it is viable to build 32-bit software natively in the near > future. i noted an alternative strategy in the bugreport: if binutils *at the very least* were to look at the available resident RAM and only malloc'd and used up to that amount, and kept only a few (or even just one) object file in memory at a time and did all the linking for that, it would be infinitely better than the current situation which is *only going to get worse*. > Maybe next year only a few packages will need exceptions, but > the number will grow with each month. well... that ignores the fact that at some point in the next few years there will be a package that needs 16 GB of resident RAM to link. and a few years after that it will be 32 GB. and that's on 64-bit systems. the package's name will probably be "firefox", given the current growth rate. does debian *really* want to have to upgrade all 64-bit systems in the build farm first to 16 GB RAM and then to 32 GB and then to 64 GB?? can the powerpc64 systems and all other 64-bit architectures even *be* upgraded to 16 GB then 32 GB then 64 GB of RAM?? basically the problems faced by 32-bit systems are a warning shot across the bows about ld not really being kept up-to-date with the increases in software complexity that's being thrown at it. it's *NOT* just about 32-bit. this problem can basically be faced REactively... or it can be faced PROactively: the historic linker strategies that you mention are i feel going to be needed in some years' time *even for 64-bit*. l.
Re: Arm ports build machines (was Re: Arch qualification for buster: call for DSA, Security, toolchain concerns)
* Luke Kenneth Casson Leighton: > that is not a surprise to hear: the massive thrashing caused by the > linker phase not being possible to be RAM-resident will be absolutely > hammering the drives beyond reasonable wear-and-tear limits. which is > why i'm recommending people try "-Wl,--no-keep-memory". Note that ld will sometimes stuff everything into a single RWX segment as a result, which is not desirable. Unfortunately, without significant investment into historic linker technologies (with external sorting and that kind of stuff), I don't think it is viable to build 32-bit software natively in the near future. Maybe next year only a few packages will need exceptions, but the number will grow with each month. Building on 64-bit kernels will delay the inevitable because more address space is available to user space, but that's probably 12 to 18 month extended life-time for native building.
Re: Arm ports build machines (was Re: Arch qualification for buster: call for DSA, Security, toolchain concerns)
On Fri, Jun 29, 2018 at 6:59 PM, Jonathan Wiltshire wrote: >> also worth noting, they're working on a 2U rackmount server which >> will have i think something insane like 48 Rock64Pro boards in one >> full-length case. > None of this addresses the basic DSA requirement of remote management. > Troubling local hands to change a disk once in a while is reasonable; being > blocked waiting for a power cycle on a regular basis is not (and I can't > imagine hosting sponsors are wild about their employees' time being used > for that either). i know exactly what you mean, i've had to deal with data centres. i'll make sure that TL Lim is aware of this, and will ask him if there's a way to include remote power-management / power-cycling of boards in the planned product or if it's already been thought of. l.
Re: Arm ports build machines (was Re: Arch qualification for buster: call for DSA, Security, toolchain concerns)
On Fri, Jun 29, 2018 at 06:05:55PM +0100, Luke Kenneth Casson Leighton wrote: > apologies for repeating it again: this is why i'm recommending people > try "-Wl,--no-keep-memory" on the linker phase as if it works as > intended it will almost certainly drastically reduce memory usage to > the point where it will stay, for the majority of packages, well > within the 2GB limit i.e. within resident RAM. [...] > most of them won't have native SATA: very few 32-bit ARM systems do. > GbE is not that common either (so decent-speed network drives are > challenging, as well). so they'll almost certainly be USB-based > (USB-to-SATA, which is known-unreliable), and putting such vast > amounts of drive-hammering through USB-to-SATA due to thrashing isn't > going to help :) [...] > > the allwinner A20 and R40 are the two low-cost ARM systems that i'm > aware of that have native SATA. > > > there is however a new devboard that is reasonably cheap and should > be available really soon: the Rock64Pro (not to be confused with the > Rock64, which does NOT have PCie), from pine64: > https://www.pine64.org/?page_id=61454 > > it's one of the first *low-cost* ARM dev-boards that i've seen which > has 4GB of RAM and has a 4x PCIe slot. the team have tested it out > with an NVMe SSD and also 4x SATA PCIe cards: they easily managed to > hit 20 Gigabits per second on the NVMe drive (2500 mbytes/sec). > > also worth noting, they're working on a 2U rackmount server which > will have i think something insane like 48 Rock64Pro boards in one > full-length case. > > the Rock64Pro uses the RK3399 which is a 4-core CortexA53 plus 2-core > CortexA72 for a total 6-core SMP system, all 64-bit. > > if anyone would like me to have a word with TL Lim (the CEO of > pine64) i can see if he is willing and able to donate some Rock64Pro > boards to the debian farm, let me know. None of this addresses the basic DSA requirement of remote management. Troubling local hands to change a disk once in a while is reasonable; being blocked waiting for a power cycle on a regular basis is not (and I can't imagine hosting sponsors are wild about their employees' time being used for that either). Development boards just don't cut it any longer. -- Jonathan Wiltshire j...@debian.org Debian Developer http://people.debian.org/~jmw 4096R: 0xD3524C51 / 0A55 B7C5 1223 3942 86EC 74C3 5394 479D D352 4C51
Re: Arm ports build machines (was Re: Arch qualification for buster: call for DSA, Security, toolchain concerns)
On Fri, Jun 29, 2018 at 5:21 PM, Steve McIntyre wrote: >>2G is also way too little memory these days for a new buildd. > > Nod - lots of packages are just too big for that now. apologies for repeating it again: this is why i'm recommending people try "-Wl,--no-keep-memory" on the linker phase as if it works as intended it will almost certainly drastically reduce memory usage to the point where it will stay, for the majority of packages, well within the 2GB limit i.e. within resident RAM. i'm really not sure why the discussions continue not to take this into account, repeating the status-quo and accepting "packages are too big" as if there is absolutely no possible way that this problem may be solved or even attempted to be solved... ever. i am very confused by this. perhaps it is down to latency in discussions as new people contribute (but have signficant delay on reading), i don't know. > Future options > == > > I understand DSA's reluctance to continue supporting dev boards as > build platforms - I've been the one working on some of these machines > in the machine room at Arm, and it's painful when you can't reliably > reboot or get onto the console of crashed machines. We've also had a > spate of disk failures recently which has caused extended downtime. that is not a surprise to hear: the massive thrashing caused by the linker phase not being possible to be RAM-resident will be absolutely hammering the drives beyond reasonable wear-and-tear limits. which is why i'm recommending people try "-Wl,--no-keep-memory". ... oh, i have an idea which people might like to consider trying. it's to use "-Wl,--no-keep-memory" on the linker phase of 32-bit builds. did i mention that already? :) it might save some build hardware from being destroyed if people try using "-Wl,--no-keep-memory"! > I'm just in the middle of switching the arm64 machines here to using > SW RAID to mitigate that in future, and that's just not an option on > the dev boards. We want to move away from dev boards for these > reasons, at the very least. most of them won't have native SATA: very few 32-bit ARM systems do. GbE is not that common either (so decent-speed network drives are challenging, as well). so they'll almost certainly be USB-based (USB-to-SATA, which is known-unreliable), and putting such vast amounts of drive-hammering through USB-to-SATA due to thrashing isn't going to help :) the allwinner A20 and R40 are the two low-cost ARM systems that i'm aware of that have native SATA. there is however a new devboard that is reasonably cheap and should be available really soon: the Rock64Pro (not to be confused with the Rock64, which does NOT have PCie), from pine64: https://www.pine64.org/?page_id=61454 it's one of the first *low-cost* ARM dev-boards that i've seen which has 4GB of RAM and has a 4x PCIe slot. the team have tested it out with an NVMe SSD and also 4x SATA PCIe cards: they easily managed to hit 20 Gigabits per second on the NVMe drive (2500 mbytes/sec). also worth noting, they're working on a 2U rackmount server which will have i think something insane like 48 Rock64Pro boards in one full-length case. the Rock64Pro uses the RK3399 which is a 4-core CortexA53 plus 2-core CortexA72 for a total 6-core SMP system, all 64-bit. if anyone would like me to have a word with TL Lim (the CEO of pine64) i can see if he is willing and able to donate some Rock64Pro boards to the debian farm, let me know. l.
Arm ports build machines (was Re: Arch qualification for buster: call for DSA, Security, toolchain concerns)
On Fri, Jun 29, 2018 at 11:23:25AM +0200, Julien Cristau wrote: >On 06/29/2018 09:16 AM, Uwe Kleine-König wrote: >>> >>> [DSA Sprint report]: >>> https://lists.debian.org/debian-project/2018/02/msg4.html >> >> In this report Julien Cristau wrote: >> >>> In short, the hardware (development boards) we're currently using to >>> build armel and armhf packages aren't up to our standards, and we >>> really, really want them to go away when stretch goes EOL (expected in >>> 2020). We urge arm porters to find a way to build armhf packages in >>> VMs or chroots on server-class arm64 hardware. >> >> If the concerns are mostly about the hardware not being rackable, there >> is a rackable NAS by Netgear: >> >> >> https://www.netgear.com/business/products/storage/readynas/RN2120.aspx#tab-techspecs >> >> with an armhf cpu. Not sure if cpu speed (1.2 GHz) and available RAM (2 >> GiB) are good enough. The machine can run mainline Linux[1]. I think >> U-Boot doesn't support this machine in mainline though. >> >Rackable, while good, is only part of it. The main part is remote >management. I'm not seeing any mention of ipmi or anything like that in >the datasheet? > >2G is also way too little memory these days for a new buildd. Nod - lots of packages are just too big for that now. So, here's a summary of the current build/porter machines we have for the arm ports. armel/armhf *only* build machines = We're mostly using dev boards to build 32-bit arm stuff yet. * 7 Marvell Armada XP machines: dev boards supplied in a box, 4GiB RAM - see http://photos.einval.com/gallery/2014_marvell_buildd These are nice powerful little machines, but they're a PITA to manage for power (won't cold boot without a button push) and serial (USB serial exposed only). We have 6 of these as build boxes and 1 porter box, and I have a spare ready to go into service if desired. They *don't* have NEON, so we also still have: * 1 Freescale imx53 dev board as a porter box - old, slow Cortex A8 machine with only 1GiB of RAM. This works better for serial, but remote power issues again - needs a button push to cold boot. Will happily retire this once we have NEON available by default. Each of these dev boards only has support for 1 disk, so disk failures are painful. arm64 build machines These are all more normal computers, with better support for remote power and serial, DIMM slots, multiple disks, etc. * APM Mustang (X-Gene 1): officially EOL, but working fine for now. Normal server-class machine (although supplied in a small desktop case!) with some onboard server management stuff. We currently have one of these. We used to have more loaned/hosted by Linaro, and I've had an offer of more of these if we're interested. They'll build and run A32 (32-bit instruction set) as well as A64. * Gigabyte MP30-AR0 (X-Gene 1): server systems based on the Mustang core - see https://b2b.gigabyte.com/Server-Motherboard/MP30-AR0-rev-11#ov Capable of building/running A32 and A64. * AMD Seattle (Opteron A1100): officially EOL too, but working fine. Same as the Softiron 3000, 2U rackmount case. Capable of building/running A32 and A64. One of these has just been configured to build armhf only. Future options == I understand DSA's reluctance to continue supporting dev boards as build platforms - I've been the one working on some of these machines in the machine room at Arm, and it's painful when you can't reliably reboot or get onto the console of crashed machines. We've also had a spate of disk failures recently which has caused extended downtime. I'm just in the middle of switching the arm64 machines here to using SW RAID to mitigate that in future, and that's just not an option on the dev boards. We want to move away from dev boards for these reasons, at the very least. So, at the moment as far as I can see we're happy with our current arm64 build machines. They are ageing, so obviously I'm continuing to look out for new options there as well. *But* my priority is new options for 32-bit building too. Following standard Debian practice, we want to build *natively* (i.e. not using cross-building or using hardware emulation). Building 32-bit code on a 64-bit platform should not be an issue so long as the platform can also execute that 32-bit code directly. I am not aware of any 32-bit Arm *server* platforms shipping today. Some have existed in the past (e.g. Calxeda), but almost universally people have moved on to 64-bit now. The awkward thing that is now becoming more common in the arm64 server world is that quite a few of the vendors are not seeing any value in A32 support so they're leaving it out of their CPUs. We'll need to be careful about that. Options I can see today for new publically available machines are here. I'm sure I'll have missed something obvious - please feel free to improve on this list! *
