Hi all!
Sorry for delay.
Well we currently have few systems at SPbSTU (and yes some of them runs
gentoo, but currently not that one that is in top500)
So about systems:
One old called Sigma
8 nodes:
cpu: 4 x AMD Opteron 6174, 12 core, 2.2GHz
mem: 128G RAM
net: 40Gbps QDR Infiniband, 10Gbps Ethernet
Storage FS: Lustre
Storage size: 13.5Tb
Second one is slightly exotic system called ccNUMA (actualy part of HPC
complex installed in 2014)
64 nodes:
cpu: 3 x AMD Opteron 6380, 16 core, 2.5GHz
mem: 96G RAM
net: 56Gbps FDR Infiniband, NumaScale
Storage FS: Lustre
Storage size: 1Pb
ccNUMA actuly uses NumaScale interconnect and bootloader so it can be
used as Jumbo ccNUMA Single System Image
with ~3000 cores and 12Tb RAM that will be seen by single linux kernel
(and yes it runs gentoo)
Other 3 partitions of our HPC systems currently runs CentOS 7 (but may
be migrated to gentoo in near future). All this partitions shares same
Lustre FS as ccNUMA
* tornado : nodes: 612
cpu: 2 x Intel Xeon CPU E5-2697 v3 @ 2.60GHz
cores/hwthreads: 28 / 28
mem: 64G
net: 56Gbps FDR Infiniband
* tornado-k40 : nodes: 56
cpu: 2 x Intel Xeon CPU E5-2697 v3 @ 2.60GHz
cores/hwthreads: 28 / 28
co-processor: 2 x Nvidia Tesla K40x
co-porcessor mem: 12G
mem: 64G
net: 56Gbps FDR Infiniband
* mic : nodes: 288
cpu: 1 x Intel Xeon Phi 5120D @ 1.10GHz
cores/hwthreads: 60 / 240
mem: 8G
net: 56Gbps FDR Infiniband
Storage: 1 PB Lustre FS
PS also we have some exotic storage system used for backup (400+ disks
attached via SAS and runs single ZFS filesystem)
Andrew Savchenko писал 05-01-2018 02:18:
Hi Alexxy and all!
While preparing Gentoo flyer[1,2] on upcoming FOSDEM event it was
suggested to include some words about Gentoo in HPC. So I'm asking
community about recent stats of using Gentoo on HPC clusters.
I ran several clusters at a university on Gentoo, but they are weak
by modern standards (<10 TFlops CPU, no GPU for the best one, though
with IB QDR interconnect), so I'm reluctant to add them in the
flyer. If someone can provide us with better and actual numbers,
this will be nice.
IIRC Alexxy is running some HPC setups. Are they on Gentoo? If yes,
please provide some stats (Flops, number of cores, nodes, storage,
interconnect and so on).
If someone can provide any other relevant info, we'll be grateful.
Years ago there was a Gentoo cluster project page with list of
active setups, but I can't find it any more and it is likely to be
outdated anyway.
[1]
https://download.sumptuouscapital.com/gentoo/fosdem2018/gentoo-fosdem2018-flyer.pdf
[2] git://git.sumptuouscapital.com/gentoo/pr-fosdem2018-flyer.git
Best regards,
Andrew Savchenko
--
Best Regards,
Alexey 'Alexxy' Shvetsov
Best Regards,
Alexey 'Alexxy' Shvetsov, PhD
Department of Molecular and Radiation Biophysics
FSBI Petersburg Nuclear Physics Institute, NRC Kurchatov Institute,
Leningrad region, Gatchina, Russia
Gentoo Team Ru
Gentoo Linux Dev
mailto:alexx...@gmail.com
mailto:ale...@gentoo.org
mailto:ale...@omrb.pnpi.spb.ru
