[Beowulf] EasyBuild User Meeting (EUM'24) announcement

[Jörg Saßmannshausen]

Dear all,

it gives me great pleasure to announce that this years EasyBuild User 
Meeting (EUM'24) will take place at 23-25 April 2024 @ Umeå, Sweden.

Like last year, it will be a hybrid meeting again.
Registration is free and still open, please follow this link:
https://easybuild.io/eum24

EasyBuild is software which is installing software from source, 
including all of the dependencies, in a reproducible manner. In case you 
want to know more about why reproducible software installation is 
important, please listen to the Code for Thought podcast entitled 
‘ByteSized RSE: Easy Does It – with easybuild’

https://codeforthought.buzzsprout.com/1326658/14361557-en-bytesized-rse-easy-does-it-with-easybuild-jorg-sassmannshausen

or watch the tutorial video, aimed for users, here:
https://www.youtube.com/watch?v=aYCb3rZApTM

Places for physical attendance are limited but then we got nearly 
unlimited Zoom places if you want to ask question, or watch it live or 
later on YouTube if you want to watch it with your favourite drink.


Please fell free to disseminate the information further.

All the best from London

Jörg
___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
https://beowulf.org/cgi-bin/mailman/listinfo/beowulf

[Beowulf] 19.01.24, 23:06, RSE Byte-sized introduction to EasyBuild (virtual, 23.1.2024 11:00 GMT (UCT), recorded)

[Jörg Saßmannshausen]

Dear all,

it gives me great pleasure to announce the

RSE Byte-sized introduction to EasyBuild (virtual, 23.1.2024 11:00 GMT 
(UCT), recorded)


This byte-sized session will provide a high-level introduction to the 
software build and installation framework EasyBuild 
(https://easybuild.io/). EasyBuild is focused on supporting the 
installation of software on High Performance Computing platforms and is 
increasingly widely used as a tool to simplify this process. Following a 
short introductory presentation, there will be an opportunity to see how 
EasyBuild works as part of a hands-on interactive tutorial. This event 
will be recorded and registration can be found here:

https://lnkd.in/ey6ZamW3
Please register, so you can get the link to the recording.
Feel free to share around, not only to system admins but interested 
users as well.


I am looking forward to see you there.

Jörg
___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
https://beowulf.org/cgi-bin/mailman/listinfo/beowulf

Re: [Beowulf] Fwd: [EESSI] "Best Practices for CernVM-FS in HPC" online tutorial on Mon 4 Dec 2023 (13:30-17:00 CET)

[Jörg Saßmannshausen]

Dear all,

we released quite a larger number of seats again, which does explain why 
some of you could and some of you could not register.


If you are interested, please try again.

Apologies for the inconvenience.

Regards

Jörg

Am 13.11.23 um 21:02 schrieb Michael DiDomenico:
looks like they opened it up, it definitely said "full" when i checked 
earlier


On Mon, Nov 13, 2023 at 1:02 PM leo camilo <mailto:lhcam...@gmail.com>> wrote:


I have just registered and last I saw it there were still vacancies
opened.

Maybe the registration was not working correctly?

On Mon, 13 Nov 2023 at 15:35, Michael DiDomenico
mailto:mdidomeni...@gmail.com>> wrote:

unfortunately, it looks like registration is full... :(


On Mon, Nov 13, 2023 at 4:34 AM Jörg Saßmannshausen
mailto:sassy-w...@sassy.formativ.net>> wrote:

Dear all,

just in case you are interested, there is a EESSI online
tutorial coming
up. EESSI is a way to share microarchitecture-specific and
thus highly
optimised software and the good thing here is: it is already
build, so
we don't need to build that for ourselves. As it is
basically mounted,
and due to the OS abstraction layer, it is meant to run on
any Linux
system, as well as Windows (Windows Subsystem for Linux).

Regards

Jörg

 Forwarded Message 
Subject: [EESSI] "Best Practices for CernVM-FS in HPC"
online tutorial
on Mon 4 Dec 2023 (13:30-17:00 CET)
Date: Mon, 13 Nov 2023 09:04:02 +0100
From: Kenneth Hoste
<02e68c3bfd33-dmarc-requ...@list.rug.nl
<mailto:02e68c3bfd33-dmarc-requ...@list.rug.nl>>
Reply to: European Environment for Scientific Soft
mailto:ee...@list.rug.nl>>
To: ee...@list.rug.nl <mailto:ee...@list.rug.nl>

Dear EESSI community,

On Monday 4 December 2023 (13:30-17:00 CET) we are
organising a half-day
online introductory tutorial on CernVM-FS
(https://cernvm.cern.ch/fs <https://cernvm.cern.ch/fs>),
via Zoom, in collaboration with the CernVM-FS development team.

CernVM-FS, the CernVM File System (also known as CVMFS), is
a file
distribution service that is particularly well suited to
distribute
software installations across a large number of systems
world-wide in an
efficient way.

The focus of this tutorial is on explaining what CernVM-FS
is, how to
access existing CernVM-FS repositories (like EESSI, see
https://eessi.io/docs <https://eessi.io/docs>), and covering
some aspects specific to using
CernVM-FS on HPC systems.
It is intended for people with a background in HPC (system
administrators, support team members, end users, etc.) who
are new to
CernVM-FS: no specific prior knowledge or experience with it
is required.

High-level preliminary agenda (subject to minor changes):

- What is CernVM-FS?
- European Environment for Scientific Software Installations
(EESSI)
- Accessing a CernVM-FS repository
- Configuring CernVM-FS on HPC infrastructure
- Troubleshooting and debugging CernVM-FS
- Performance aspects of CernVM-FS
- Different storage backends for CernVM-FS
- Containers and CernVM-FS
- Getting started with CernVM-FS (from scratch)

If you would like to attend this tutorial, please register via:

https://event.ugent.be/registration/cvmfshpc202312
<https://event.ugent.be/registration/cvmfshpc202312>


Please help us promote this event, by sharing it with
colleagues and
friends, and/or posting this event announcement on
appropriate channels.


regards,

Kenneth
on behalf of the tutorial organisers



___
Beowulf mailing list, Beowulf@beowulf.org
<mailto:Beowulf@beowulf.org> sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe)
visit https://beowulf.org/cgi-bin/mailman/listinfo/beowulf
<https://beowulf.org/cgi-bin/mailman/listinfo/beowulf>

___
Beowulf mailing list, Beowulf@beowulf.org
<mail

[Beowulf] Fwd: [EESSI] "Best Practices for CernVM-FS in HPC" online tutorial on Mon 4 Dec 2023 (13:30-17:00 CET)

[Jörg Saßmannshausen]

Dear all,

just in case you are interested, there is a EESSI online tutorial coming 
up. EESSI is a way to share microarchitecture-specific and thus highly 
optimised software and the good thing here is: it is already build, so 
we don't need to build that for ourselves. As it is basically mounted, 
and due to the OS abstraction layer, it is meant to run on any Linux 
system, as well as Windows (Windows Subsystem for Linux).


Regards

Jörg

 Forwarded Message 
Subject: [EESSI] "Best Practices for CernVM-FS in HPC" online tutorial 
on Mon 4 Dec 2023 (13:30-17:00 CET)

Date: Mon, 13 Nov 2023 09:04:02 +0100
From: Kenneth Hoste <02e68c3bfd33-dmarc-requ...@list.rug.nl>
Reply to: European Environment for Scientific Soft 
To: ee...@list.rug.nl

Dear EESSI community,

On Monday 4 December 2023 (13:30-17:00 CET) we are organising a half-day 
online introductory tutorial on CernVM-FS (https://cernvm.cern.ch/fs), 
via Zoom, in collaboration with the CernVM-FS development team.


CernVM-FS, the CernVM File System (also known as CVMFS), is a file 
distribution service that is particularly well suited to distribute 
software installations across a large number of systems world-wide in an 
efficient way.


The focus of this tutorial is on explaining what CernVM-FS is, how to 
access existing CernVM-FS repositories (like EESSI, see 
https://eessi.io/docs), and covering some aspects specific to using 
CernVM-FS on HPC systems.
It is intended for people with a background in HPC (system 
administrators, support team members, end users, etc.) who are new to 
CernVM-FS: no specific prior knowledge or experience with it is required.


High-level preliminary agenda (subject to minor changes):

- What is CernVM-FS?
- European Environment for Scientific Software Installations (EESSI)
- Accessing a CernVM-FS repository
- Configuring CernVM-FS on HPC infrastructure
- Troubleshooting and debugging CernVM-FS
- Performance aspects of CernVM-FS
- Different storage backends for CernVM-FS
- Containers and CernVM-FS
- Getting started with CernVM-FS (from scratch)

If you would like to attend this tutorial, please register via:

https://event.ugent.be/registration/cvmfshpc202312


Please help us promote this event, by sharing it with colleagues and 
friends, and/or posting this event announcement on appropriate channels.



regards,

Kenneth
on behalf of the tutorial organisers


___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
https://beowulf.org/cgi-bin/mailman/listinfo/beowulf

[Beowulf] EasyBuild Tech Talk “AVX10 for HPC - A reasonable solution to the 7 levels of AVX-512 folly”

[Jörg Saßmannshausen]

Dear all

In case you have not seen this before there is an upcoming EasyBuild 
Tech Talk “AVX10 for HPC - A reasonable solution to the 7 levels of 
AVX-512 folly” by Felix LeClair on Fri 13 Oct 2023 at 13:30 UTC.

More information is available at
https://easybuild.io/tech-talks/008_avx10.html .
If you would like to attend the Zoom session, please register via:
https://event.ugent.be/registration/ebtechtalk008avx10

The Tech Talks are not the usual sales-pit talks from vendors, they are 
literally technical focused and often very informativ I found.

The previous ones can be found here:
https://easybuild.io/tech-talks

Please spread out the word.

All the best from a very mild London!

Jörg
___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
https://beowulf.org/cgi-bin/mailman/listinfo/beowulf

[Beowulf] EUM'23 keynote

[Jörg Saßmannshausen]

Dear all,

further my last message, it gives me great pleasure to announce that one 
of our keynote speakers for the EUM'23 (https://easybuild.io/eum23) is:


Martyn Guest: "Performance of Community Codes on Multi-core Processors - 
An Analysis of Computational Chemistry and Ocean Modelling Applications"


https://easybuild.io/eum23/#keynote-martyn-guest

Please register so you can participate via Zoom which will allow you to 
ask questions. Registration will be open until Sunday 2 April 2023.


I am looking forward to welcome you at the EUM'23 in April at Imperial.

Jörg
___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
https://beowulf.org/cgi-bin/mailman/listinfo/beowulf

[Beowulf] EasyBuild User Meeting (EUM'23) announcement

[Jörg Saßmannshausen]

Dear all,

it gives me great pleasure to announce that this years EasyBuild User 
Meeting (EUM'23) will take place at Imperial College London, UK, and it 
will be a hybrid meeting again. Registration is free and open, please 
follow this link:

https://easybuild.io/eum23

EasyBuild is software which is installing software from source, 
including all of the dependencies, in a reproducible manner. The user 
meeting will bring together the community and we will have Keynote 
speakers like Ian Cutress ("Emerging Technologies and Silicon for HPC: 
Not all Cores are Equal") and TechTalks like from Jan-Patrick Lehr 
("AOMP: OpenMP Target Offloading for AMD GPUs"), next to user 
experiences like for example from Israel Vieira ("User experience with 
EasyBuild deploying Health-GPS Microsimulation to Imperial HPC"). Places 
for physical attendance are limited but then we got nearly unlimited 
Zoom places if you want to ask question, or watch it live or later on 
YouTube if you want to watch it with your favorite drink.


Please fell free to disseminate the information further.

All the best from London

Jörg

___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
https://beowulf.org/cgi-bin/mailman/listinfo/beowulf

Re: [Beowulf] Understanding environments and libraries caching on a beowulf cluster

[Jörg Saßmannshausen]

Dear all,

what we are doing is we are exporting our software stack via a shared file 
system like, for example NFS (not a good idea for larger clusters), 
SpectrumScale (formerly known as GPFS), Lustre, Ceph... the list is long.

To ease your pain with building architecture specific software and all the 
dependencies, I highly recommend EasyBuild for that:

https://easybuild.io/

It is a bit of a learning curve but then what is not. The big advantage here 
is there are currently over 2700 *individual* packages supported, not 
including extensions. Also, if you got a new cluster and you want to say 
install GROMACS from scratch, you can do it like this:

$ eb --rd GROMACS-2021.5-foss-2021b.eb

and that version is installed. You want CUDA support? Here it is:
$ eb --rd GROMACS-2021.5-foss-2021b-CUDA-11.4.1.eb

The community is constantly adding new software packages to it, so with each 
release there will be more. 

Also, unless you are already doing so, I recommend the use of Lmod instead of 
EnvironmentModules. 

I hope that helps a bit regarding taking care of libraries. 

All the best from a overcast and a bit windy London

Jörg 

Am Dienstag, 28. Juni 2022, 10:32:17 BST schrieb leo camilo:
> # Background
> 
> So, I am building this small beowulf cluster for my department. I have it
> running on ubuntu servers, a front node and at the moment 7 x 16 core
> nodes. I have installed SLURM as the scheduler and I have been
> procrastinating to setup environment modules.
> 
> In any case, I ran in this particular scenario where I was trying to
> schedule a few jobs in slurm, but for some reason slurm would not find this
> library (libgsl). But it was in fact installed in the frontnode, I checked
> the path with ldd and I even exported the LD_LIBRARY_PATH .
> 
> Oddly, if I ran the application directly in the frontnode, it would work
> fine.,
> 
> Though it occured to me that the computational nodes might not have this
> library and surely once I installed this library in the nodes the problem
> went away.
> 
> # Question:
> 
> So here is the question, is there a way to cache the frontnode's libraries
> and environment onto the computational nodes when a slurm job is created?
> 
> Will environment modules do that? If so, how?
> 
> Thanks in advance,
> 
> Cheers



___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
https://beowulf.org/cgi-bin/mailman/listinfo/beowulf

Re: [Beowulf] likwid vs stream (after HPCG discussion)

[Jörg Saßmannshausen]

Dear all,

reading through this made me realise how difficult things are these days. 
If you have a HPC cluster for just a few applications, you find it probably 
easier to build the software and replace the hardware as you really could go 
down to the assembler level and have really highly optimised code and hardware 
for exactly these few jobs.
If, as we are, you have to support anything from a single core job to say a 
few hundred cores, MPI and threaded jobs, memory bandwidth extensive to heave 
IO ones, all of that goes out the chimney. I reckon it will be very difficult 
to then find the only 'right' benchmark for you as your applications vary so 
much. So the trick is probably to find the sweet spot for your cluster, which 
might be a different setup for other sites. 

As always, thanks for sharing your thoughts. 

All the best from a sunny and mild London

Jörg

Am Montag, 21. März 2022, 09:46:31 GMT schrieb Mikhail Kuzminsky:
> In message from Scott Atchley  (Sun, 20 Mar
> 
> 2022 14:52:10 -0400):
> > On Sat, Mar 19, 2022 at 6:29 AM Mikhail Kuzminsky 
> >
> >wrote:
> >> If so, it turns out that for the HPC user, stream gives a more
> >> important estimate - the application is translated by the compiler
> >> (they do not write in assembler - except for modules from
> >>
> >>mathematical
> >>
> >> libraries), and stream will give a real estimate of what will be
> >> received in the application.
> > 
> > When vendors advertise STREAM results, they compile the application
> >
> >with
> >
> > non-temporal loads and stores. This means that all memory accesses
> >
> >bypass
> >
> > the processor's caches. If your application of interest does a random
> >
> >walk
> >
> > through memory and there is neither temporal or spatial locality,
> >
> >then
> >
> > using non-temporal loads and stores makes sense and STREAM
> >
> >irrelevant.
> 
> STREAM is not initially oriented to random access to memory. In this
> case, memory latencies are important, and it makes more sense to get a
> bandwidth estimate in the mega-sweep
> (https://github.com/UK-MAC/mega-stream).
> ___
> Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> https://beowulf.org/cgi-bin/mailman/listinfo/beowulf



___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
https://beowulf.org/cgi-bin/mailman/listinfo/beowulf

Re: [Beowulf] HPCG benchmark, again

[Jörg Saßmannshausen]

t; >>> HPCG measures memory bandwidth, the FLOPS capability of the chip is
> >>> completely irrelevant. Pretty much all the vendor implementations reach
> >>> very similar efficiency if you compare them to the available memory
> >>> bandwidth. There is some effect of the network at scale, but you need
> >>> to have a really large  system to see it in play.
> >>> 
> >>> M
> >>> 
> >>>> On Fri, Mar 18, 2022 at 5:20 PM Brian Dobbins mailto:bdobb...@gmail.com>> wrote:
> >>>Hi Jorg,
> >>>
> >>>  We (NCAR - weather/climate applications) tend to find that HPCG
> >>>
> >>>more closely tracks the performance we see from hardware than
> >>>Linpack, so it definitely is of interest and watched, but our
> >>>procurements tend to use actual code that vendors run as part of
> >>>the process, so we don't 'just' use published HPCG numbers.
> >>>Still, I'd say it's still very much a useful number, though.>>>
> >>>  As one example, while I haven't seen HPCG numbers for the MI250x
> >>>
> >>>accelerators, Prof. Matuoka of RIKEN tweeted back in November that
> >>>he anticipated that to score around 0.4% of peak on HPCG, vs 2% on
> >>>the NVIDIA A100 (while the A64FX they use hits an impressive 3%):
> >>>https://twitter.com/ProfMatsuoka/status/1458159517590384640
> >>><https://twitter.com/ProfMatsuoka/status/1458159517590384640>
> >>>
> >>>  Why is that relevant?  Well, /on paper/, the MI250X has ~96 TF
> >>>
> >>>FP64 w/ Matrix operations, vs 19.5 TF on the A100.  So, 5x in
> >>>theory, but Prof Matsuoka anticipated a ~5x differential in HPCG,
> >>>/erasing/ that differential.  Now, surely /someone/ has HPCG
> >>>numbers on the MI250X, but I've not yet seen any.  Would love to
> >>>know what they are.  But absent that information I tend to bet
> >>>Matsuoka isn't far off the mark.
> >>>
> >>>  Ultimately, it may help knowing more about what kind of
> >>>
> >>>applications you run - for memory bound CFD-like codes, HPCG tends
> >>>to be pretty representative.
> >>>
> >>>  Maybe it's time to update the saying that 'numbers never lie' to
> >>>
> >>>something more accurate - 'numbers never lie, but they also rarely
> >>>tell the whole story'.
> >>>
> >>>  Cheers,
> >>>  - Brian
> >>>
> >>>On Fri, Mar 18, 2022 at 5:08 PM Jörg Saßmannshausen
> >>> >>>
> >>><mailto:sassy-w...@sassy.formativ.net>> wrote:
> >>>Dear all,
> >>>
> >>>further the emails back in 2020 around the HPCG benchmark
> >>>test, as we are in
> >>>the process of getting a new cluster I was wondering if
> >>>somebody else in the
> >>>meantime has used that test to benchmark the particular
> >>>performance of the
> >>>cluster.
> >>>From what I can see, the latest HPCG version is 3.1 from
> >>>August 2019. I also
> >>>have noticed that their website has a link to download a
> >>>version which
> >>>includes the latest A100 GPUs from nVidia.
> >>>https://www.hpcg-benchmark.org/software/view.html?id=280
> >>><https://www.hpcg-benchmark.org/software/view.html?id=280>
> >>>
> >>>What I was wondering is: has anybody else apart from Prentice
> >>>tried that test
> >>>and is it somehow useful, or does it just give you another set
> >>>of numbers?
> >>>
> >>>Our new cluster will not be at the same league as the
> >>>supercomputers, but we
> >>>would like to have at least some kind of handle so we can
> >>>compare the various
> >>>offers from vendors. My hunch is the benchmark will somehow
> >>>(strongly?) depend
> >>>on how it is tuned. As my former colleague used to say: I am
> >>>looking for some
> >>>wa

[Beowulf] HPCG benchmark, again

[Jörg Saßmannshausen]

Dear all,

further the emails back in 2020 around the HPCG benchmark test, as we are in 
the process of getting a new cluster I was wondering if somebody else in the 
meantime has used that test to benchmark the particular performance of the 
cluster. 
From what I can see, the latest HPCG version is 3.1 from August 2019. I also 
have noticed that their website has a link to download a version which 
includes the latest A100 GPUs from nVidia. 
https://www.hpcg-benchmark.org/software/view.html?id=280

What I was wondering is: has anybody else apart from Prentice tried that test 
and is it somehow useful, or does it just give you another set of numbers?

Our new cluster will not be at the same league as the supercomputers, but we 
would like to have at least some kind of handle so we can compare the various 
offers from vendors. My hunch is the benchmark will somehow (strongly?) depend 
on how it is tuned. As my former colleague used to say: I am looking for some 
war stories (not very apt to say these days!).

Either way, I hope you are all well given the strange new world we are living 
in right now.

All the best from a spring like dark London

Jörg



___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
https://beowulf.org/cgi-bin/mailman/listinfo/beowulf

Re: [Beowulf] Data Destruction

[Jörg Saßmannshausen]

Hi Ellis,

interesting concept. I did not know about the Lustre fsencrypt but then, I am 
less the in-detail expert in PFS.

Just to make sure I get the concept of that correct: Basically Lustre is 
providing projects which itself are encrypted, similar to the encrypted 
containers I mentioned before. So in order to access the project folder, you 
would need some kind of encryption key. Without that, you only have 
meaningless data in front of you. Is that understanding correct?

Does anybody happen to know if a similar system like the one Lustre is 
offering is possible on Ceph?

The only problem I have with all these things is: at one point you will need 
to access the decrypted data. Then you need to make sure that this data is not 
leaving your system. So for that reason we are using a Data Safe Haven where 
data ingress and egress is done via a staging system. 

Some food for thought.

Thanks

All the best

Jörg

Am Mittwoch, 29. September 2021, 16:42:46 BST schrieb Ellis Wilson:
> Apologies in advance for the top-post -- too many interleaved streams
> here to sanely bottom-post appropriately.
> 
> SED drives, which are a reasonably small mark-up for both HDDs and SSDs,
> provide full drive or per-band solutions to "wipe" the drive by revving
> the key associated with the band or drive.  For enterprise HDDs the
> feature is extremely common -- for enterprise SSDs it is hit or miss
> (NVMe tend to have it, SATA infrequently do).  This is your best bet for
> a solution where you're a-ok with wiping the entire system.  Note
> there's non-zero complexity here usually revolving around a non-zero
> price KMIP server, but it's (usually) not terrible.  My old employ
> (Panasas) supports this level of encryption in their most recent release.
> 
> Writing zeros over HDDs or SSDs today is an extremely dubious solution.
>   SSDs will just write the zeros elsewhere (or more commonly, not write
> them at all) and HDDs are far more complex than the olden days so you're
> still given no hard guarantees there that writing to LBA X is actually
> writing to LBA X.  Add a PFS and then local FS in front of this and
> forget about it.  You're just wasting bandwidth.
> 
> If you have a multi-tenant system and cannot just wipe the whole system
> by revving encryption keys on the drives, you're options are static
> partitioning of the drives into SED bands per tenant and a rather
> complex setup with a KMIP server and parallel parallel file systems to
> support that, or client-side encryption.  Lustre 2.14 provides this via
> fsencrypt for data, which is actually pretty slick.  This is your best
> bet to cryptographically shred the data for individual users.  I have no
> experience with other commercial file systems so cannot comment on who
> does or doesn't support client-side encryption, but whoever does should
> allow you to fairly trivially shred the bits associated with that
> user/project/org by discarding/revving the corresponding keys.  If you
> go the client-side encryption route and shred the keys, snapshots, PFS,
> local FS, RAID, and all of the other factors here play no role and you
> can safely promise the data is mathematically "gone" to the end-user.
> 
> Best,
> 
> ellis
> 
> On 9/29/21 10:52 AM, Paul Edmon via Beowulf wrote:
> > I guess the question is for a parallel filesystem how do you make sure
> > you have 0'd out the file with out borking the whole filesystem since
> > you are spread over a RAID set and could be spread over multiple hosts.
> > 
> > -Paul Edmon-
> > 
> > On 9/29/2021 10:32 AM, Scott Atchley wrote:
> >> For our users that have sensitive data, we keep it encrypted at rest
> >> and in movement.
> >> 
> >> For HDD-based systems, you can perform a secure erase per NIST
> >> standards. For SSD-based systems, the extra writes from the secure
> >> erase will contribute to the wear on the drives and possibly their
> >> eventually wearing out. Most SSDs provide an option to mark blocks as
> >> zero without having to write the zeroes. I do not think that it is
> >> exposed up to the PFS layer (Lustre, GPFS, Ceph, NFS) and is only
> >> available at the ext4 or XFS layer.
> >> 
> >> On Wed, Sep 29, 2021 at 10:15 AM Paul Edmon  >> 
> >> > wrote:
> >> The former.  We are curious how to selectively delete data from a
> >> parallel filesystem.  For example we commonly use Lustre, ceph,
> >> and Isilon in our environment.  That said if other types allow for
> >> easier destruction of selective data we would be interested in
> >> hearing about it.
> >> 
> >> -Paul Edmon-
> >> 
> >> On 9/29/2021 10:06 AM, Scott Atchley wrote:
> >>> Are you asking about selectively deleting data from a parallel
> >>> file system (PFS) or destroying drives after removal from the
> >>> system either due to failure or system decommissioning?
> >>> 
> >>> For the latter, DOE does not allow us to send any non-volatile
> >>> media offsite 

Re: [Beowulf] [EXTERNAL] Re: Data Destruction

[Jörg Saßmannshausen]

Dear all,

back at a previous work place the shredder did come to our place and like Jim 
said: loud, not much to look at other than a heap of shredded metal and 
plastic the other end. 

There is an active volcano around right now:
https://www.spiegel.de/wissenschaft/natur/vulkanausbruch-auf-la-palma-lavastrom-fliesst-in-den-atlantik-a-9b86cd64-d9b6-4566-9a41-06a1c0ac4a47?
jwsource=cl
(Sorry, German text but the pictures are quite spectacular)
I think the only flipside is that usually the volcano does not come to you! :D

Melting it all down is a good idea from the data destruction point of view. 
However, it makes it much more difficult to separate the more precious 
elements later on. Here taking out the platters first and manually disassemble 
the rest might be a better way forward. You can shredder the platters and 
recycle them and recycle the rest as require. Ok, I take off my chemists hat. 
:D 

All the best

Jörg

Am Mittwoch, 29. September 2021, 20:00:08 BST schrieb Lux, Jim (US 7140) via 
Beowulf:
> There are special purpose drive shredders - they'll even come out to your
> facility with such a device mounted on a truck. It's not as exciting as you
> might think. Throw stuff in, makes horrible noise, done.
> 
> For entertainment, the truck sized wood chipper used when clearing old
> orchards is much more fun. Another unit rips the tree out of the ground and
> drops it into the giant feed hopper on top. Entire lemon or avocado trees
> turned into mulch in a very short time.
> 
> This isn't the one I saw, but it's the same idea, except horizontal feed.
> https://www.youtube.com/watch?v=BCdmO6WvBYk
> 
> 
> We had a lengthy discussion on Slack at work (JPL) about ways to destroy
> drives - melting them in some sort of forge seems to be the most fun,
> unless you have a convenient volcano with flowing lava handy.
> 
> On 9/29/21, 8:47 AM, "Beowulf on behalf of Ellis Wilson"
>  wrote:
> On 9/29/21 11:41 AM, Jörg Saßmannshausen wrote:
> > If you still need more, don't store the data at all but print it out
> > on paper and destroy it by means of incineration. :D
> 
> I have heard stories from past colleagues of one large US Lab putting
> their HDDs through wood chippers with magnets on the chipped side to
> kill the bits good and dead.  As a storage fanatic that always struck me
> as something I'd have loved to see.
> 
> Best,
> 
> ellis
> ___
> Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> https://urldefense.us/v3/__https://beowulf.org/cgi-bin/mailman/listinfo/beo
> wulf__;!!PvBDto6Hs4WbVuu7!Yrfky03OZZ3wo1YOCW77WLjeKXbglr4IVVHlYwAqOLxxgkK95r
> TaAlheRxZy6Ww0M92RGxs$
> 
> ___
> Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> https://beowulf.org/cgi-bin/mailman/listinfo/beowulf



___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
https://beowulf.org/cgi-bin/mailman/listinfo/beowulf

Re: [Beowulf] Data Destruction

[Jörg Saßmannshausen]

Dear all,

interesting discussion and very timely for me as well as we are currently 
setting up a new HPC facility, using OpenStack throughout so we can build a 
Data Safe Haven with it as well.
The question about data security came up too in various conversations, both 
internal and with industrial partners. 
Here I actually asked one collaboration partner what they understand about 
"data at rest":
- the drive has been turned off
- the data is not being accessed

For the former, that is easy we simply encrypt all drives, one way or another. 
However, that means when the drive is on, the data is not encrypted. 

For the latter that is a bit more complicated as you need to decrypt the 
files/folder when you want to access them. This, however, in addition to the 
drive encryption itself, should give you potentially the maximum security. 
When you want to destroy that data, deleting the encrypted container *and* the 
access key, i.e. the piece you need to decrypt it, like a Yubikey, should in 
my humble opinion being enough for most data. 
If you need more, shred the drive and don't use fancy stuff like RAID or PFS. 

If you still need more, don't store the data at all but print it out on paper 
and destroy it by means of incineration. :D

How about that?

All the best from a sunny London

Jörg

Am Mittwoch, 29. September 2021, 15:57:17 BST schrieb Skylar Thompson:
> In this case, we've successfully pushed back with the granting agency (US
> NIH, generally, for us) that it's just not feasible to guarantee that the
> data are truly gone on a production parallel filesystem. The data are
> encrypted at rest (including offsite backups), which has been sufficient
> for our purposes. We'll then just use something like GNU shred(1) to do a
> best-effort secure delete.
> 
> In addition to RAID, other confounding factors to be aware of are snapshots
> and cached data.
> 
> On Wed, Sep 29, 2021 at 10:52:33AM -0400, Paul Edmon via Beowulf wrote:
> > I guess the question is for a parallel filesystem how do you make sure you
> > have 0'd out the file with out borking the whole filesystem since you are
> > spread over a RAID set and could be spread over multiple hosts.
> > 
> > -Paul Edmon-
> > 
> > On 9/29/2021 10:32 AM, Scott Atchley wrote:
> > > For our users that have sensitive data, we keep it encrypted at rest and
> > > in movement.
> > > 
> > > For HDD-based systems, you can perform a secure erase per NIST
> > > standards. For SSD-based systems, the extra writes from the secure erase
> > > will contribute to the wear on the drives and possibly their eventually
> > > wearing out. Most SSDs provide an option to mark blocks as zero without
> > > having to write the zeroes. I do not think that it is exposed up to the
> > > PFS layer (Lustre, GPFS, Ceph, NFS) and is only available at the ext4 or
> > > XFS layer.
> > > 
> > > On Wed, Sep 29, 2021 at 10:15 AM Paul Edmon  > > 
> > > > wrote:
> > > The former.  We are curious how to selectively delete data from a
> > > parallel filesystem.  For example we commonly use Lustre, ceph,
> > > and Isilon in our environment.  That said if other types allow for
> > > easier destruction of selective data we would be interested in
> > > hearing about it.
> > > 
> > > -Paul Edmon-
> > > 
> > > On 9/29/2021 10:06 AM, Scott Atchley wrote:
> > > > Are you asking about selectively deleting data from a parallel
> > > > file system (PFS) or destroying drives after removal from the
> > > > system either due to failure or system decommissioning?
> > > > 
> > > > For the latter, DOE does not allow us to send any non-volatile
> > > > media offsite once it has had user data on it. When we are done
> > > > with drives, we have a very big shredder.
> > > > 
> > > > On Wed, Sep 29, 2021 at 9:59 AM Paul Edmon via Beowulf
> > > > 
> > > > mailto:beowulf@beowulf.org>> wrote:
> > > > Occassionally we get DUA (Data Use Agreement) requests for
> > > > sensitive
> > > > data that require data destruction (e.g. NIST 800-88). We've
> > > > been
> > > > struggling with how to handle this in an era of distributed
> > > > filesystems
> > > > and disks.  We were curious how other people handle requests
> > > > like this?
> > > > What types of filesystems to people generally use for this
> > > > and how do
> > > > people ensure destruction?  Do these types of DUA's preclude
> > > > certain
> > > > storage technologies from consideration or are there creative
> > > > ways to
> > > > comply using more common scalable filesystems?
> > > > 
> > > > Thanks in advance for the info.
> > > > 
> > > > -Paul Edmon-
> > > > 
> > > > ___
> > > > Beowulf mailing list, Beowulf@beowulf.org
> > > >

Re: [Beowulf] List archives

[Jörg Saßmannshausen]

Dear all,

never been to Rome, let alone Milan, but I been to this talk on Tuesday:
https://github.com/easybuilders/easybuild/wiki/EasyBuild-tech-talks-IV:-AMD-Rome-&-Milan

There are links for the pdf-files and the video as well. I hope that counts as 
(useful) comment. :-)

All the best

Jörg

Am Mittwoch, 18. August 2021, 09:45:41 BST schrieb John Hearns:
> I plead an advanced case of not keeping up with technology.
> I not this is for Ryzen - anyone care to comment on Rome/Milan?
> 
> On Wed, 18 Aug 2021 at 08:56, Jim Cownie  wrote:
> > John may have been looking for Doug’s tweet and just confused the delivery
> > medium...
> > 
> > https://twitter.com/thedeadline/status/1424833944000909313
> > 
> > On 17 Aug 2021, at 07:16, Chris Samuel  wrote:
> > 
> > Hi John,
> > 
> > On Monday, 16 August 2021 12:57:20 AM PDT John Hearns wrote:
> > 
> > The Beowulf list archives seem to end in July 2021.
> > I was looking for Doug Eadline's post on limiting AMD power and the
> > results
> > on performance.
> > 
> > 
> > I just went through the archives for July and compared them with what I
> > have
> > in my inpile and as far as I can tell there's nothing missing. There was a
> > thread from June with the subject "AMD and AVX512", perhaps that's what
> > you're
> > thinking of?
> > 
> > https://www.beowulf.org/pipermail/beowulf/2021-June/thread.html
> > 
> > Your email from today & my earlier reply are in the archives for August.
> > 
> > All the best!
> > Chris
> > --
> > 
> >  Chris Samuel  :  http://www.csamuel.org/  :  Berkeley, CA, USA
> > 
> > ___
> > Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
> > To change your subscription (digest mode or unsubscribe) visit
> > https://beowulf.org/cgi-bin/mailman/listinfo/beowulf
> > 
> > 
> > -- Jim
> > James Cownie 
> > Mob: +44 780 637 7146
> > 
> > 
> > 
> > 
> > ___
> > Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
> > To change your subscription (digest mode or unsubscribe) visit
> > https://beowulf.org/cgi-bin/mailman/listinfo/beowulf



___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
https://beowulf.org/cgi-bin/mailman/listinfo/beowulf

Re: [Beowulf] Power Cycling Question

[Jörg Saßmannshausen]

Hi Doug,

interesting topic and quite apt when I look at the flooding in Germany, 
Belgian and The Netherlands. 

I guess there are a number of reasons why people are not doing it. Discarding 
the usual "we never done that" one, I guess the main problem is: when do you 
want  to turn it off? After 5 mins being idle? Maybe 10 mins? One hour? How 
often do you then need to boot them up again and how much energy does that 
cost? From chatting to a few people who tried it in the past it somehow 
transpired that you do not save as much energy as you were hoping for. 

However, on thing came to my mind: is it possible to simply suspend it to disc 
and then let it be sleeping? That way, you wake the node up quicker and 
probably need less power when it is suspended. Think of laptops. 

The other way around would simply be: we know in say the summer, there is less 
demand so we simply turn X number of nodes off and might do some maintenance 
on them. So you are running the whole cluster for say 6 weeks with limited 
capacity. That might mean a few jobs are queuing but that also will give us a 
window to do things. Once people are coming back, the maintenance is done and 
the cluster can run at full capacity again. 

Just some (crazy?) ideas.

All the best

Jörg

Am Freitag, 16. Juli 2021, 20:35:11 BST schrieb Douglas Eadline:
> Hi everyone:
> 
> Reducing power use has become an important topic. One
> of the questions I always wondered about is
> why more cluster do not turn off unused nodes. Slurm
> has hooks to turn nodes off when not in use and
> turn them on when resources are needed.
> 
> My understanding is that power cycling creates
> temperature cycling, that then leads to premature node
> failure. Makes sense and has anyone ever studied/tested
> this ?
> 
> The only other reason I can think of is that the delay
> in server boot time makes job starts slow or power
> surge issues.
> 
> I'm curious about other ideas or experiences.
> 
> Thanks
> 
> --
> Doug



___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
https://beowulf.org/cgi-bin/mailman/listinfo/beowulf

Re: [Beowulf] AMD and AVX512

[Jörg Saßmannshausen]

Dear all

> System architecture has been a problem ... making a processing unit
> 10-100x as fast as its support components means you have to code with
> that in mind.  A simple `gfortran -O3 mycode.f` won't necessarily
> generate optimal code for the system ( but I swear ... -O3 ... it says
> it on the package!)

From a computational Chemist perspective I agree. In an ideal world, you want 
to get the right hardware for the program you want to use. Some of the code is 
running entirely in memory, others is using disc space for offloading files. 

This is, in my humble opinion, also the big problem CPUs are facing. They are 
build to tackle all possible scenarios, from simple integer to floating point, 
from in-memory to disc I/O. In some respect it would have been better to stick 
with a separate math unit which then could be selected according to your 
workload you want to run on that server. I guess this is where the GPUs are 
trying to fit in here, or maybe ARM. 

I also agree with the compiler "problem". If you are starting to push some 
compilers too much, the code is running very fast but the results are simply 
wrong. Again, in an ideal world we have a compiler for the job for the given 
hardware which also depends on the job you want to run. 

The problem here is not: is that possible, the problem is more: how much does 
it cost? From what I understand, some big server farms are actually not using 
commodity HPC stuff but they are designing what they need themselves. 

Maybe the whole climate problem will finally push HPC into the more bespoken 
system where the components are fit for the job in question, say weather 
modeling for example, simply as that would be more energy efficient and 
faster. 
Before somebody comes along with: but but but it costs! Think about how much 
money is being spent simply to kill people, or at other wasteful project like 
Brexit etc. 

My 2 shillings for what it is worth! :D

Jörg

Am Montag, 21. Juni 2021, 14:46:30 BST schrieb Joe Landman:
> On 6/21/21 9:20 AM, Jonathan Engwall wrote:
> > I have followed this thinking "square peg, round hole."
> > You have got it again, Joe. Compilers are your problem.
> 
> Erp ... did I mess up again?
> 
> System architecture has been a problem ... making a processing unit
> 10-100x as fast as its support components means you have to code with
> that in mind.  A simple `gfortran -O3 mycode.f` won't necessarily
> generate optimal code for the system ( but I swear ... -O3 ... it says
> it on the package!)
> 
> Way back at Scalable, our secret sauce was largely increasing IO
> bandwidth and lowering IO latency while coupling computing more tightly
> to this massive IO/network pipe set, combined with intelligence in the
> kernel on how to better use the resources.  It was simply a better
> architecture.  We used the same CPUs.  We simply exploited the design
> better.
> 
> End result was codes that ran on our systems with off-cpu work (storage,
> networking, etc.) could push our systems far harder than competitors. 
> And you didn't have to use a different ISA to get these benefits.  No
> recompilation needed, though we did show the folks who were interested,
> how to get even better performance.
> 
> Architecture matters, as does implementation of that architecture. 
> There are costs to every decision within an architecture.  For AVX512,
> along comes lots of other baggage associated with downclocking, etc. 
> You have to do a cost-benefit analysis on whether or not it is worth
> paying for that baggage, with the benefits you get from doing so.  Some
> folks have made that decision towards AVX512, and have been enjoying the
> benefits of doing so (e.g. willing to pay the costs).  For the general
> audience, these costs represent a (significant) hurdle one must overcome.
> 
> Here's where awesome compiler support would help.  FWIW, gcc isn't that
> great a compiler.  Its not performance minded for HPC. Its a reasonable
> general purpose standards compliant (for some subset of standards)
> compilation system.  LLVM is IMO a better compiler system, and its
> clang/flang are developing nicely, albeit still not really HPC focused. 
> Then you have variants built on that.  Like the Cray compiler, Nvidia
> compiler and AMD compiler. These are HPC focused, and actually do quite
> well with some codes (though the AMD version lags the Cray and Nvidia
> compilers). You've got the Intel compiler, which would be a good general
> compiler if it wasn't more of a marketing vehicle for Intel processors
> and their features (hey you got an AMD chip?  you will take the slowest
> code path even if you support the features needed for the high
> performance code path).
> 
> Maybe, someday, we'll get a great HPC compiler for C/Fortran.



___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 

Re: [Beowulf] AMD and AVX512

[Jörg Saßmannshausen]

Dear all,

same here, I should have joined the discussion earlier but currently I am 
recovering from a trapped ulnaris nerve OP, so long typing is something I need 
to avoid.
As it is quite apt I think, I would like to inform you about this upcoming 
talk (copy):

**
*Performance Optimizations & Best Practices for AMD Rome and Milan CPUs in HPC 
Environments*
- date & time: Fri July 2nd 2021 - 16:00-17:30 UTC
- speakers: Evan Burness and Jithin Jose (Principal Program Managers for High-
Performance Computing in Microsoft Azure)

More information available at https://github.com/easybuilders/easybuild/wiki/
EasyBuild-tech-talks-IV:-AMD-Rome-&-Milan

The talk will be presented via a Zoom session, which registered attendees can 
join, and will be streamed (+ recorded) via the EasyBuild YouTube channel.
Q via the #tech-talks channel in the EasyBuild Slack.

Please register (free or charge) if you plan to attend, via:
https://webappsx.ugent.be/eventManager/events/ebtechtalkamdromemilan
The Zoom link will only be shared with registered attendees.
**

These talks are really tech talks and not sales talks and all of the ones I 
been to were very informative and friendly. So that might be a good idea to 
ask some questions there?

All the best

Jörg

Am Sonntag, 20. Juni 2021, 18:28:25 BST schrieb Mikhail Kuzminsky:
> I apologize - I should have written earlier, but I don't always work
> with my broken right hand. It seems to me that a reasonable basis for
> discussing AMD EPYC performance could be the specified performance
> data in the Daresburg University benchmark from M.Guest. Yes, newer
> versions of AMD EPYC and Xeon Scalable processors have appeared since
> then, and new compiler versions. However, Intel already had AVX-512
> support, and AMD - AVX-256.
> Of course, peak performanceis is not so important as application
> performance. There are applications where performance is not limited
> to working with vectors - there AVX-512 may not be needed. And in AI
> tasks, working with vectors is actual - and GPUs are often used there.
> For AI, the Daresburg benchmark, on the other hand, is less relevant.
> And in Zen 4, AMD seemed to be going to support 512 bit vectors. But
> performance of linear algebra does not always require work with GPU.
> In quantum chemistry, you can get acceleration due to vectors on the
> V100, let's say a 2 times - how much more expensive is the GPU?
> Of course, support for 512 bit vectors is a plus, but you really need
> to look to application performance and cost (including power
> consumption). I prefer to see to the A64FX now, although there may
> need to be rebuild applications. Servers w/A64FX sold now, but the
> price is very important.
> 
> In message from John Hearns  (Sun, 20 Jun 2021
> 
> 06:38:06 +0100):
> > Regarding benchmarking real world codes on AMD , every year Martyn
> >
> >Guest
> >
> > presents a comprehensive set of benchmark studies to the UK Computing
> > Insights Conference.
> > I suggest a Sunday afternoon with the beverage of your choice is a
> >
> >good
> >
> > time to settle down and take time to read these or watch the
> >
> >presentation.
> >
> > 2019
> > https://www.scd.stfc.ac.uk/SiteAssets/Pages/CIUK-2019-Presentations/Martyn
> > _Guest.pdf
> > 
> > 
> > 2020 Video session
> > https://ukri.zoom.us/rec/share/ajvsxdJ8RM1wzpJtnlcypw4OyrZ9J27nqsfAG7eW49E
> > hq_Z5igat_7gj21Ge8gWu.78Cd9I1DNIjVViPV?startTime=1607008552000
> > 
> > Skylake / Cascade Lake / AMD Rome
> > 
> > The slides for 2020 do exist - as I remember all the slides from all
> >
> >talks
> >
> > are grouped together, but I cannot find them.
> > Watch the video - it is an excellent presentation.
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > On Sat, 19 Jun 2021 at 16:49, Gerald Henriksen 
> >
> >wrote:
> >> On Wed, 16 Jun 2021 13:15:40 -0400, you wrote:
> >> >The answer given, and I'm
> >> >not making this up, is that AMD listens to their users and gives the
> >> >users what they want, and right now they're not hearing any demand
> >>
> >>for
> >>
> >> >AVX512.
> >> >
> >> >Personally, I call BS on that one. I can't imagine anyone in the HPC
> >> >community saying "we'd like processors that offer only 1/2 the
> >>
> >>floating
> >>
> >> >point performance of Intel processors".
> >> 
> >> I suspect that is marketing speak, which roughly translates to not
> >> that no one has asked for it, but rather requests haven't reached a
> >> threshold where the requests are viewed as significant enough.
> >> 
> >> > Sure, AMD can offer more cores,
> >> >
> >> >but with only AVX2, you'd need twice as many cores as Intel
> >>
> >>processors,
> >>
> >> >all other things being equal.
> >> 
> >> But of course all other things aren't equal.
> >> 
> >> AVX512 is a mess.
> >> 
> >> Look at the Wikipedia page(*) and note that AVX512 means different
> >> things depending on the processor implementing it.
> >> 
> >> So what does the poor software 

Re: [Beowulf] Project Heron at the Sanger Institute [EXT]

[Jörg Saßmannshausen]

Dear all,

chiming in here, as I am supporting some of the Covid-19 sequencing at my 
current work place:

We got much smaller sequencing machines and here you can do the analysis on 
the machine as well. The problem we are facing is simply the storage capacity. 
It probably does not come to a surprise this project kicked off during 
lockdown and I only got involved in it end of last year. We actually shuffle 
our data off to a QNAP and then do the analysis on a directly attached Linux 
machine. That means, the sequencing machine has always enough capacity to 
store more raw data. 

One of the things I heard a few times is the use of GPUs for the analysis. Is 
that something you are doing as well? Also, on the topic of GPUs (and being a 
bit controversial): are there actually any programs out there which are not 
using nVidia GPUs and use the AMD ones for example?

> Scientists are conservative folks though, they sometimes get a bit nervous
> at the thought of discarding the raw sequence data.

Wearing my science-hat: I can see why. Traditionally, things like sequencing 
(or NMR files) are expensive to do: you need to have the right sample and the 
right machine for it. I am sometimes coming back to NMR spectra and have a new 
look at them, simply to see if there is something I missed the first time 
around as I did not know about it. With the Covid-19 variants: I guess it is 
the same here. If you only keep the result: yes, it contains Covid-19 or not, 
you cannot go back and re-do it for a different strain/variant. I also dare to 
say that we might need to re-analyse the raw data to make sure a new pipeline 
is producing the same results for example. So for me, there are good reasons 
to keep at least the relevant raw data. 

On a related subject: do you attach meta-data to your data so you can find 
relevant things quicker? I am trying to encourage my users to think about that 
so we can use things like IRODS for example to get a better data management. 
If I may ask: how does the Sanger does that and would be some kind of best 
practice, so it already not exist, be a good idea here?

All the best from a wet London

Jörg

Am Donnerstag, 4. Februar 2021, 10:35:22 GMT schrieb Tim Cutts:
> Compute capacity is not generally the issue.  For this pipeline, we only
> need about 200 cores to keep up with each sequencer, so a couple of
> servers.   Genomics has not, historically, been a good fit for SETI@home
> style cycle-stealing, because the amount of compute you perform on a given
> unit of data is quite low.  A lot of genomics is already I/O bound even
> when the compute is right next to the data, so you don’t gain much by
> shipping it off to cycle-stealing desktops.
 
> In fact, the direction most sequencing instrument suppliers are going is
> embedding the compute in the sequencer itself, at least for use cases where
> you don’t really need the sequence at all, you just need to know how it
> varies from a reference genome.  In such cases, it’s much more sensible to
> run the pipeline on or right next to the sequencer and just spit out the
> (very small) diffs.
 
> Scientists are conservative folks though, they sometimes get a bit nervous
> at the thought of discarding the raw sequence data.
 
> Tim
> 
> On 4 Feb 2021, at 10:27, Jonathan Aquilina
> mailto:jaquil...@eagleeyet.net>> wrote:
 
> Would love to help you guys out in anyway i can in terms of hardware
> processing.
 
> Have you guys thought of doing something like SETI@home and those projects
> to get idle compute power to help churn through the massive amounts of
> data?
 
> Regards,
> Jonathan
> 
> From: Tim Cutts mailto:t...@sanger.ac.uk>>
> Sent: 04 February 2021 11:26
> To: Jonathan Aquilina
> mailto:jaquil...@eagleeyet.net>>
 Cc: Beowulf
> mailto:beowulf@beowulf.org>>
> Subject: Re: [Beowulf] Project Heron at the Sanger Institute [EXT]
> 
> 
> 
> On 4 Feb 2021, at 10:14, Jonathan Aquilina via Beowulf
> mailto:beowulf@beowulf.org>> wrote:
 
> I am curious though to chunk out such large data is something like
> hadoop/HBase and the like of those platforms, are those whats being used?
 
> 
> It’s a combination of our home-grown sequencing pipeline which we use across
> the board, and then a specific COG-UK analysis of the genomes themselves. 
> This pipeline is common to all consortium members who are contributing
> sequence data.  It’s a Nextflow pipeline, and the code is here:
 
> https://github.com/connor-lab/ncov2019-artic-nf
> [github.com] m_connor-2Dlab_ncov2019-2Dartic-2Dnf=DwMF-g=D7ByGjS34AllFgecYw0iC6Zq7qlm
> 8uclZFI0SqQnqBo=gSesY1AbeTURZwExR_OGFZlp9YUzrLWyYpGmwAw4Q50=jJhOeZORmye7
> vKliXyqrCd2Kvbe5xu9pHhLw4rNQmHM=lSbHd9Jxd4Dy9P7rosnrdgOmieVt-yzUuVI-MPK7TM
> 0=>
 
> Being nextflow, you can run it on anything for which nextflow has a backend
> scheduler.   It supports data from both Illumina and Oxford Nanopore
> sequencers.
 
> Tim
> -- 

Re: [Beowulf] Project Heron at the Sanger Institute

[Jörg Saßmannshausen]

Hi John,

interesting stuff and good reading. 

For the IT interests on here: these sequencing machine are chucking out large 
amount of data per day. The project I am involved in can chew out 400 GB or so 
on raw data per day. That is a small machine. That then needs to be processed 
before you actually can analyze it. So there is quite some data movement etc 
involved here. 

All the best

Jörg

Am Mittwoch, 3. Februar 2021, 14:06:36 GMT schrieb John Hearns:
> https://edition.cnn.com/2021/02/03/europe/tracing-uk-variant-origins-gbr-int
> l/index.html
> 
> Dressed in white lab coats and surgical masks, staff here scurry from
> machine to machine -- robots and giant computers that are so heavy, they're
> placed on solid steel plates to support their weight.
> Heavy metal!



___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
https://beowulf.org/cgi-bin/mailman/listinfo/beowulf

[Beowulf] Singularity with Easybuild talk and tutorial

[Jörg Saßmannshausen]

Dear all,

I am a bit shamelessly would like to bring you the upcoming online Easybuild 
User Meeting from Jan 25-29 2021 to your attention. In particular, I am giving 
a presentation of how to easy-build Singularity containers and there will be 
also a brief tutorial of how to do that. The whole event is free and the 
videos will be made available later on YouTube. 

My talk is aimed more towards the users, in particular those who are not IT 
savvy but know enough Linux to get by. Obviously sys-admins are welcome to 
join as well. 

Obviously the meeting is not only about that, so please check out their 
homepage: https://easybuild.io/eum/

I hope you will find that useful. 

Regards

Jörg



___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
https://beowulf.org/cgi-bin/mailman/listinfo/beowulf

Re: [Beowulf] RoCE vs. InfiniBand

[Jörg Saßmannshausen]

Hi Gilad,

thanks for the feedback, much appreciated. 
In an ideal world, you are right of course. OpenStack is supported natively on 
InfiniBand, and you can get the MetroX system to connect between two different 
sites (I leave it open of how to read that) etc. 

However, in the real world all of that needs to fit into a budget. From what I 
can see on the cluster, most jobs are in the region between 64 and 128 cores. 
So, that raises the question for that rather small amount of cores, do we 
really need InfiniBand or can we do what we need to do with RoCE v2?

In other words, for the same budget, does it make sense to remove the 
InfiniBand part of the design and get say one GPU box in instead?

What I want to avoid is to make the wrong decision (cheap and cheerful) and 
ending up with a badly designed cluster later. 

As you mentioned MetroX: remind me please, what kind of cable does it need? Is 
that something special or can we use already existing cables, whatever is used 
between data centre sites (sic!)?

We had a chat with Darren about that which was, as always talking to your 
colleague Darren, very helpful. I remember very distinct there was a reason 
why we went for the InfiniBand/RoCE solution but I cannot really remember it. 
It was something with the GPU boxes we want to buy as well. 

I will pass your comments on to my colleague next week when I am back at work 
and see what they say. So many thanks for your sentiments here which are much 
appreciated from me!

All the best from a cold London

Jörg

Am Donnerstag, 26. November 2020, 12:51:55 GMT schrieb Gilad Shainer:
> Let me try to help:
> 
> -  OpenStack is supported natively on InfiniBand already, therefore
> there is no need to go to Ethernet for that
 
> -  File system wise, you can have IB file system, and connect
> directly to IB system.
 
> -  Depends on the distance, you can run 2Km IB between switches, or
> use Mellanox MetroX for connecting over 40Km. VicinityIO have system that
> go over thousands of miles…
 
> -  IB advantages are with much lower latency (switches alone are 3X
> lower latency), cost effectiveness (for the same speed, IB switches are
> more cost effective than Ethernet) and the In-Network Computing engines
> (MPI reduction operations, Tag Matching run on the network)
 
> If you need help, feel free to contact directly.
> 
> Regards,
> Gilad Shainer
> 
> From: Beowulf [mailto:beowulf-boun...@beowulf.org] On Behalf Of John Hearns
> Sent: Thursday, November 26, 2020 3:42 AM
> To: Jörg Saßmannshausen ; Beowulf Mailing
> List 
 Subject: Re: [Beowulf] RoCE vs. InfiniBand
> 
> External email: Use caution opening links or attachments
> 
> Jorg, I think I might know where the Lustre storage is !
> It is possible to install storage routers, so you could route between
> ethernet and infiniband.
 It is also worth saying that Mellanox have Metro
> Infiniband switches - though I do not think they go as far as the west of
> London! 
> Seriously though , you ask about RoCE. I will stick my neck out and say yes,
> if you are planning an Openstack cluster
 with the intention of having
> mixed AI and 'traditional' HPC workloads I would go for a RoCE style setup.
> In fact I am on a discussion about a new project for a customer with
> similar aims in an hours time. 
> I could get some benchmarking time if you want to do a direct comparison of
> Gromacs on IB / RoCE
 
> 
> 
> 
> 
> 
> 
> 
> 
> On Thu, 26 Nov 2020 at 11:14, Jörg Saßmannshausen
> mailto:sassy-w...@sassy.formativ.net>>
> wrote:
 Dear all,
> 
> as the DNS problems have been solve (many thanks for doing this!), I was
> wondering if people on the list have some experiences with this question:
> 
> We are currently in the process to purchase a new cluster and we want to
> use
 OpenStack for the whole management of the cluster. Part of the cluster
> will run HPC applications like GROMACS for example, other parts typical
> OpenStack applications like VM. We also are implementing a Data Safe Haven
> for the more sensitive data we are aiming to process. Of course, we want to
> have a decent size GPU partition as well!
> 
> Now, traditionally I would say that we are going for InfiniBand. However,
> for
 reasons I don't want to go into right now, our existing file storage
> (Lustre) will be in a different location. Thus, we decided to go for RoCE
> for the file storage and InfiniBand for the HPC applications.
> 
> The point I am struggling is to understand if this is really the best of
> the
 solution or given that we are not building a 100k node cluster, we
> could use RoCE for the few nodes which are doing parallel, read MPI, jobs
> too. I have a nagging feeling that I am missing something if we are moving
> to pure RoCE and ditch 

Re: [Beowulf] [External] RIP CentOS 8 [EXT]

[Jörg Saßmannshausen]

Dear all,

maybe we should take that as an opportunity to gently move vendors in the 
direction to not only support rpm based but also deb based distributions?
In the end, if I really only can get hold of a rpm package, I still can use 
the Debian 'alien' command to convert it to a deb package. 

I agree with the Debian 'puristic' stance, I found that sometimes a bit 
irritating. However, since I really moved into HPC in 2010, I never had a 
problem with that on either Desktops or Cluster hardware. 
Problem as in: I did not manage to get it running at all. 

I guess your mileage will vary. 

All the best

Jörg

Am Dienstag, 8. Dezember 2020, 22:05:21 GMT schrieben Sie:
> > On Dec 8, 2020, at 4:59 PM, Tim Cutts  wrote:
> >> On 8 Dec 2020, at 21:52, Ryan Novosielski  wrote:
> >> 
> >> It’s pretty common that if something supports only one distribution, it’s
> >> RedHat-based. That’s also true of hardware vendors.> 
> > True, officially, but often not officially.  Again, back around 2008 I
> > found it hilariously irritating.  HP supported a lot of Debian activity
> > in the background, and hosted quite a lot of the infrastructure.  
> > Officially, they only supported Red Hat, but I discovered on a visit to
> > Colorado Springs that they actually developed all their drivers using
> > Debian, and then ported them to Red Hat!
> > 
> > A year or two later, I remember noticing that the firmware update ISOs
> > that HP distributed were also Debian-based.  Even though they only
> > distributed Red Hat RPMs on them.  Duh!
> > 
> > Tim
> > -- The Wellcome Sanger Institute is operated by Genome Research Limited, a
> > charity registered in England with number 1021457 and a company
> > registered in England with number 2742969, whose registered office is 215
> > Euston Road, London, NW1 2BE.
> I should have been clearer too: this goes beyond just what they say they
> will support, or what sorts of commands they’ll ask you to run to prove a
> hardware problem, etc., but to your point, frequently stuff will be
> distributed only in RPM, or only yum repositories will be provided, or
> whatever else I might not be thinking of. And sure, I can work around that,
> etc., until one comes along that’s a huge pain in the neck for some reason,
> or it’s based on a different set of libraries, that don’t work well on the
> other distro, etc. There’s just a limit to how much time I want to spend on
> stuff like that, especially if we’re talking about stuff that really isn’t
> the focus of your business, like firmware or something. On my own
> equipment, I run either Debian or Ubuntu, and we do some Debian/Ubuntu in
> Singularity containers for the relatively few cases where they are
> supported and RHEL/CentOS are not.
> 
> That said, if CentOS truly goes away, isn’t seamlessly replaced, a lot of
> that could change.
> 
> --
> #BlackLivesMatter
> 
> 
> || \\UTGERS,   |---
*O*---
> ||
> ||_// the State| Ryan Novosielski - novos...@rutgers.edu
> ||
> || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
> || 
> ||  \\of NJ| Office of Advanced Research Computing - MSB C630, 
> Newark
> 
>  `'



___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
https://beowulf.org/cgi-bin/mailman/listinfo/beowulf

Re: [Beowulf] [External] RIP CentOS 8

[Jörg Saßmannshausen]

Dear all,

what I never understood is: why are people not using Debian?

I done some cluster installation (up to 100 or so nodes) with Debian, more or 
less out of the box, and I did not have any issue with it. I admit, I might 
have missed out something I don't know about, the famous unkown-unkowns, but 
by enlarge the clusters were running rock solid with no unusual problem. 
I did not use Lustre or GPFS etc. on it, I only played around a bit with BeeFS 
and some GlusterFS in a small scale. 

Just wondering, as people mentioned Ubuntu.

All the best from a dark London

Jörg

Am Dienstag, 8. Dezember 2020, 21:12:02 GMT schrieb Christopher Samuel:
> On 12/8/20 1:06 pm, Prentice Bisbal via Beowulf wrote:
> > I wouldn't be surprised if this causes Scientific Linux to come back
> > into existence.
> 
> It sounds like Greg K is already talking about CentOS-NG (via the ACM
> SIGHPC syspro Slack):
> 
> https://www.linkedin.com/posts/gmkurtzer_centos-project-shifts-focus-to-cent
> os-stream-activity-6742165208107761664-Ng4C
> 
> All the best,
> Chris



___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
https://beowulf.org/cgi-bin/mailman/listinfo/beowulf

Re: [Beowulf] RoCE vs. InfiniBand

[Jörg Saßmannshausen]

Hi Chris,

that is basically what we are planning to do: using RoCE v2 for the Lnet 
routers and use InfiniBand for the HPC part of the cluster. As I have 
mentioned before, the problem is that the existing Lustre will be in a 
different location to where the new facility will be. At least that was the 
latest we got told. To make things a bit more interesting: currently we *are* 
using InfiniBand (Mellanox and the Intel one) to connect from the compute 
nodes to Lustre. 
So we have 2 problems: how to connect the new with the old world and what to 
do with the HPC workload we are having? This is where that mixed RoCE/
InfiniBand design came up. 

I hope this, with what I wrote in the other replies, makes sense.

All the best from London, still cold and dark. :-) 

Jörg

Am Freitag, 27. November 2020, 07:13:46 GMT schrieb Chris Samuel:
> On Thursday, 26 November 2020 3:14:05 AM PST Jörg Saßmannshausen wrote:
> > Now, traditionally I would say that we are going for InfiniBand. However,
> > for reasons I don't want to go into right now, our existing file storage
> > (Lustre) will be in a different location. Thus, we decided to go for RoCE
> > for the file storage and InfiniBand for the HPC applications.
> 
> I think John hinted at this, but is there a reason for not going for IB for
> the cluster and then using Lnet routers to connect out to the Lustre storage
> via ethernet (with RoCE) ?
> 
> https://wiki.lustre.org/LNet_Router_Config_Guide
> 
> We use Lnet routers on our Cray system to bridge between the Aries
> interconnect inside the XC to the IB fabric our Lustre storage sits on.
> 
> All the best,
> Chris



___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
https://beowulf.org/cgi-bin/mailman/listinfo/beowulf

Re: [Beowulf] RoCE vs. InfiniBand

[Jörg Saßmannshausen]

Hi Lance,

greetings to Monash! It was in discussion with you the idea of RoCE was 
formed, initially for the files but very recently we got made aware you can 
replace InfiniBand with RoCE for *small* number of cores and basically this is 
what we got shown:

https://github.com/stackhpc/hpc-tests

This is where the idea came from that for *our* < 500 cores/job community we 
might be able to get away without InfiniBand. 

Thanks for letting us know about the teething problems you had. I am sure we 
will touch base here at one point again, with the rest of the team. 

All the best from a still cold and dark London

Jörg

Am Freitag, 27. November 2020, 00:07:25 GMT schrieb Lance Wilson:
> We are a ROCE shop with HPC on private cloud and have had a mostly good
> experience. We have had quite a number of issues over time that were bugs
> and needed vendor support to resolve. Some of which have taken a long time.
> So from a maturity perspective IB is definitely much better. As a matter of
> priority ensure that all the kit is RoCE V2, V1 kit has a number of
> unresolved issues.
> 
> I do like that it is ethernet though, especially with all of the cloud kit.
> If your workloads are heavy on the MPI end you might be better off with IB,
> but my communities are <500 core jobs.
> 
> Very happy to answer questions on our experience.
> 
> 
> Cheers,
> 
> Lance
> --
> Dr Lance Wilson
> Technical Lead ACCS Characterisation Virtual Laboratory (CVL) &
> Activity Lead HPC
> Ph: 03 99055942 (+61 3 99055942)
> Mobile: 0437414123 (+61 4 3741 4123)
> Multi-modal Australian ScienceS Imaging and Visualisation Environment
> (www.massive.org.au)
> Monash University
> 
> On Thu, 26 Nov 2020 at 23:52, Gilad Shainer  wrote:
> > Let me try to help:
> > 
> > -  OpenStack is supported natively on InfiniBand already,
> > therefore there is no need to go to Ethernet for that
> > 
> > -  File system wise, you can have IB file system, and connect
> > directly to IB system.
> > 
> > -  Depends on the distance, you can run 2Km IB between switches,
> > or use Mellanox MetroX for connecting over 40Km. VicinityIO have system
> > that go over thousands of miles…
> > 
> > -  IB advantages are with much lower latency (switches alone are
> > 3X lower latency), cost effectiveness (for the same speed, IB switches are
> > more cost effective than Ethernet) and the In-Network Computing engines
> > (MPI reduction operations, Tag Matching run on the network)
> > 
> > 
> > 
> > If you need help, feel free to contact directly.
> > 
> > 
> > 
> > Regards,
> > 
> > Gilad Shainer
> > 
> > 
> > 
> > *From:* Beowulf [mailto:beowulf-boun...@beowulf.org] *On Behalf Of *John
> > Hearns
> > *Sent:* Thursday, November 26, 2020 3:42 AM
> > *To:* Jörg Saßmannshausen ; Beowulf
> > Mailing List 
> > *Subject:* Re: [Beowulf] RoCE vs. InfiniBand
> > 
> > 
> > 
> > *External email: Use caution opening links or attachments*
> > 
> > 
> > 
> > Jorg, I think I might know where the Lustre storage is !
> > 
> > It is possible to install storage routers, so you could route between
> > ethernet and infiniband.
> > 
> > It is also worth saying that Mellanox have Metro Infiniband switches -
> > though I do not think they go as far as the west of London!
> > 
> > 
> > 
> > Seriously though , you ask about RoCE. I will stick my neck out and say
> > yes, if you are planning an Openstack cluster
> > 
> > with the intention of having mixed AI and 'traditional' HPC workloads I
> > would go for a RoCE style setup.
> > 
> > In fact I am on a discussion about a new project for a customer with
> > similar aims in an hours time.
> > 
> > 
> > 
> > I could get some benchmarking time if you want to do a direct comparison
> > of Gromacs on IB / RoCE
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > On Thu, 26 Nov 2020 at 11:14, Jörg Saßmannshausen <
> > sassy-w...@sassy.formativ.net> wrote:
> > 
> > Dear all,
> > 
> > as the DNS problems have been solve (many thanks for doing this!), I was
> > wondering if people on the list have some experiences with this question:
> > 
> > We are currently in the process to purchase a new cluster and we want to
> > use
> > OpenStack for the whole management of the cluster. Part of the cluster
> > will
> &

[Beowulf] RoCE vs. InfiniBand

[Jörg Saßmannshausen]

Dear all,

as the DNS problems have been solve (many thanks for doing this!), I was 
wondering if people on the list have some experiences with this question:
 
We are currently in the process to purchase a new cluster and we want to use 
OpenStack for the whole management of the cluster. Part of the cluster will 
run HPC applications like GROMACS for example, other parts typical OpenStack 
applications like VM. We also are implementing a Data Safe Haven for the more 
sensitive data we are aiming to process. Of course, we want to have a decent 
size GPU partition as well!

Now, traditionally I would say that we are going for InfiniBand. However, for 
reasons I don't want to go into right now, our existing file storage (Lustre) 
will be in a different location. Thus, we decided to go for RoCE for the file 
storage and InfiniBand for the HPC applications. 

The point I am struggling is to understand if this is really the best of the 
solution or given that we are not building a 100k node cluster, we could use 
RoCE for the few nodes which are doing parallel, read MPI, jobs too. 
I have a nagging feeling that I am missing something if we are moving to pure 
RoCE and ditch the InfiniBand. We got a mixed workload, from ML/AI to MPI 
applications like GROMACS to pipelines like they are used in the bioinformatic 
corner. We are not planning to partition the GPUs, the current design model is 
to have only 2 GPUs in a chassis. 
So, is there something I am missing or is the stomach feeling I have really a 
lust for some sushi? :-)

Thanks for your sentiments here, much welcome!

All the best from a dull London

Jörg



___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
https://beowulf.org/cgi-bin/mailman/listinfo/beowulf

Re: [Beowulf] Tech talk: Yes! You Can Run Your Software on Arm.

[Jörg Saßmannshausen]

Hi Darren,

many thanks for doing this!

The tech-talks are really good as they are from the experts for the experts, 
which makes them different to the usual sales-type talk. 
I had a sneak preview and I found it very interesting to be honest. 

All the best from London

Jörg

Am Montag, 28. September 2020, 00:36:29 BST schrieb Darren Wise:
> I've shared it via LinkedIn for you too Jörg as I've a few connections that
> might take the opportunity too :)
> 
> Kind regards,
> Darren Wise
> wisecorp.co.uk
> 
> On 27 September 2020 23:27:18 BST, "Jörg Saßmannshausen"  wrote:
> >Dear all,
> >
> >it gives me great pleasure to announce the upcoming tech-talk:
> >"Yes! You Can Run Your Software on Arm."
> >given by Chris Edsall (University of Bristol, UK) on Wednesday, 30th of
> >
> >September. It is free and no registration is required.
> >
> >More information can be found here:
> >
> >https://github.com/easybuilders/easybuild/wiki/EasyBuild-Tech-Talks-II:-Arm
> >
> >I hope I will find this talk interesting and exciting.
> >Please feel free to distribute it further.
> >
> >All the best
> >
> >Jörg
> >
> >
> >
> >___
> >Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin
> >Computing
> >To change your subscription (digest mode or unsubscribe) visit
> >https://beowulf.org/cgi-bin/mailman/listinfo/beowulf



___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
https://beowulf.org/cgi-bin/mailman/listinfo/beowulf

[Beowulf] Tech talk: Yes! You Can Run Your Software on Arm.

[Jörg Saßmannshausen]

Dear all,

it gives me great pleasure to announce the upcoming tech-talk:
"Yes! You Can Run Your Software on Arm."
given by Chris Edsall (University of Bristol, UK) on Wednesday, 30th of 
September. It is free and no registration is required. 

More information can be found here:

https://github.com/easybuilders/easybuild/wiki/EasyBuild-Tech-Talks-II:-Arm

I hope I will find this talk interesting and exciting. 
Please feel free to distribute it further. 

All the best

Jörg



___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
https://beowulf.org/cgi-bin/mailman/listinfo/beowulf

Re: [Beowulf] experience with HPC running on OpenStack

[Jörg Saßmannshausen]

Hi Chris,

thanks for your sentiments. Like most things, you have two sides of a coin: 
flexibility, which is what we all want, and complexity, which is the price you 
need to pay for the flexibility. 

This is why I thought it is best to ask the community for first hand 
experiences. One thing we also want to address is the use of GPUs, so we can 
use them a bit more efficient than we seem to do right now. 

Regards

Jörg


Am Mittwoch, 1. Juli 2020, 06:05:11 BST schrieb Chris Samuel:
> On 29/6/20 5:09 pm, Jörg Saßmannshausen wrote:
> > we are currently planning a new cluster and this time around the idea was
> > to use OpenStack for the HPC part of the cluster as well.
> > 
> > I was wondering if somebody has some first hand experiences on the list
> > here.
> At $JOB-2 I helped a group set up a cluster on OpenStack (they were
> resource constrained, they had access to OpenStack nodes and that was
> it).  In my experience it was just another added layer of complexity for
> no added benefit and resulted in a number of outages due to failures in
> the OpenStack layers underneath.
> 
> Given that Slurm which was being used there already had mature cgroups
> support there really was no advantage to them to having a layer of
> virtualisation on top of the hardware, especially as (if I'm remembering
> properly) in the early days the virtualisation layer didn't properly
> understand the Intel CPUs we had and so didn't reflect the correct
> capabilities to the VM.
> 
> All that said, these days it's likely improved, and I know then people
> were thinking about OpenStack "Ironic" which was a way for it to manage
> bare metal nodes.
> 
> But I do know the folks in question eventually managed to go to purely a
> bare metal solution and seemed a lot happier for it.
> 
> As for IB, I suspect that depends on the capabilities of your
> virtualisation layer, but I do believe that is quite possible. This
> cluster didn't have IB (when they started getting bare metal nodes they
> went RoCE instead).
> 
> All the best,
> Chris



___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
https://beowulf.org/cgi-bin/mailman/listinfo/beowulf

Re: [Beowulf] experience with HPC running on OpenStack [EXT]

[Jörg Saßmannshausen]

Hi Tim,

many thanks for sharing your experiences here and sorry for my slow reply, I 
am currently on annual leave and thus don't check emails on a daily base. 

The additional layer of complexity is probably the price you have to pay for 
the flexibility. What we are aiming for is having a more flexible system so we 
can move things around, similar to what you are doing. 

I might come back to this later if you don't mind.

Regards

Jörg

Am Mittwoch, 1. Juli 2020, 12:13:05 BST schrieb Tim Cutts:
> Here, we deploy some clusters on OpenStack, and some traditionally as bare
> metal.   Our largest cluster is actually a mixture of both, so we can
> dynamically expand it from the OpenStack service when needed.
 
> Our aim eventually is to use OpenStack as a common deployment layer, even
> for the bare metal cluster nodes, but we’re not quite there yet.
 
> The main motivation for this was to have a common hardware and deployment
> platform, and have flexibility for VM and batch workloads.  We have needed
> to dynamically change workloads (for example in the current COVID-19
> crisis, our human sequencing has largely stopped and we’ve been
> predominantly COVID-19 sequencing, using an imported pipeline from the
> consortium we’re part of).  Using OpenStack we could get that new pipeline
> running in under a week, and later moved it from the research to the
> production environment, reallocating research resources back to their
> normal workload.
 
> There certainly are downsides; OpenStack is a considerable layer of
> complexity, and we have had occasional issues, although those rarely affect
> established running VMs (such as batch clusters).  Those occasional
> problems are usually in the services for dynamically creating and
> destroying resources, so they don’t have immediate impact on batch
> clusters.  Plus, we tend to use fairly static provider networks to connect
> the Lustre systems to virtual clusters, which removes another layer of
> OpenStack complexity.
 
> Generally speaking it’s working pretty well, and we have uptimes of in
> excess of 99.5%
 
> Tim
> 
> On 1 Jul 2020, at 05:09, John Hearns
> mailto:hear...@gmail.com>> wrote:
 
> Jorg, I would back up what Matt Wallis says. What benefits would Openstack
> bring you ?
 Do you need to set up a flexible infrastructure where clusters
> can be created on demand for specific projects? 
> Regarding Infiniband the concept is SR-IOV. This article is worth reading:
> https://docs.openstack.org/neutron/pike/admin/config-sriov.html
> [docs.openstack.org]<https://urldefense.proofpoint.com/v2/url?u=https-3A__d
> ocs.openstack.org_neutron_pike_admin_config-2Dsriov.html=DwMFaQ=D7ByGjS3
> 4AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo=gSesY1AbeTURZwExR_OGFZlp9YUzrLWyYpGmwA
> w4Q50=T0asmfOta_bLT2cXWrpERYigde5lOqHx2vVIH2WSIOw=VMHyCkd1eb1ztnzu4i617z
> rYxnddfDUUEkn1u45xQq0=>
 
> I would take a step back and look at your storage technology and which is
> the best one to be going forward with.
 Also look at the proceeding sof the
> last STFC Computing Insights where Martyn Guest presented  a lot of
> benchmarking results   on AMD Rome
> Page 103 onwards in this report
> http://purl.org/net/epubs/manifestation/46387165/DL-CONF-2020-001.pdf
> [purl.org]<https://urldefense.proofpoint.com/v2/url?u=http-3A__purl.org_net
> _epubs_manifestation_46387165_DL-2DCONF-2D2020-2D001.pdf=DwMFaQ=D7ByGjS3
> 4AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo=gSesY1AbeTURZwExR_OGFZlp9YUzrLWyYpGmwA
> w4Q50=T0asmfOta_bLT2cXWrpERYigde5lOqHx2vVIH2WSIOw=GNtI2S6yacqAS4bpUYbfq4
> bDe8nv9gXksMXaqCqgbro=>
 
> 
> 
> 
> On Tue, 30 Jun 2020 at 12:21, Jörg Saßmannshausen
> mailto:sassy-w...@sassy.formativ.net>>
> wrote:
 Dear all,
> 
> we are currently planning a new cluster and this time around the idea was
> to
 use OpenStack for the HPC part of the cluster as well.
> 
> I was wondering if somebody has some first hand experiences on the list
> here.
 One of the things we currently are not so sure about it is
> InfiniBand (or another low latency network connection but not ethernet):
> Can you run HPC jobs on OpenStack which require more than the number of
> cores within a box? I am thinking of programs like CP2K, GROMACS, NWChem
> (if that sounds familiar to you) which utilise these kind of networks very
> well.
> 
> I cam across things like MagicCastle from Computing Canada but as far as I
> understand it, they are not using it for production (yet).
> 
> Is anybody on here familiar with this?
> 
> All the best from London
> 
> Jörg
> 
> 
> 
> ___
> Beowulf mailing list, Beowulf@beowulf.org<mailto:Beowulf@beowulf.org>
> sponsored by Penguin Computing
 To change your su

Re: [Beowulf] experience with HPC running on OpenStack

[Jörg Saßmannshausen]

Hi John,

thanks for the links. I know Martyn personally and of course I know his 
(in)famous talks at the CIUK workshops. 
The InfiniBand is not for storage (I still consider using IB for storage a 
waste) it is for parallel computing where you need the low latency. As far as 
I can see even the latest Ethernet technology is not as fast as IB is (fast as 
in low latency). 
We are currently working out how many cores users are using so we might get 
away with a few multi core machines. 

All the best

Jörg

Am Mittwoch, 1. Juli 2020, 05:09:17 BST schrieb John Hearns:
> Jorg, I would back up what Matt Wallis says. What benefits would Openstack
> bring you ?
> Do you need to set up a flexible infrastructure where clusters can be
> created on demand for specific projects?
> 
> Regarding Infiniband the concept is SR-IOV. This article is worth reading:
> https://docs.openstack.org/neutron/pike/admin/config-sriov.html
> 
> I would take a step back and look at your storage technology and which is
> the best one to be going forward with.
> Also look at the proceeding sof the last STFC Computing Insights where
> Martyn Guest presented  a lot of
> benchmarking results   on AMD Rome
> Page 103 onwards in this report
> http://purl.org/net/epubs/manifestation/46387165/DL-CONF-2020-001.pdf
> 
> 
> 
> 
> On Tue, 30 Jun 2020 at 12:21, Jörg Saßmannshausen <
> 
> sassy-w...@sassy.formativ.net> wrote:
> > Dear all,
> > 
> > we are currently planning a new cluster and this time around the idea was
> > to
> > use OpenStack for the HPC part of the cluster as well.
> > 
> > I was wondering if somebody has some first hand experiences on the list
> > here.
> > One of the things we currently are not so sure about it is InfiniBand (or
> > another low latency network connection but not ethernet): Can you run HPC
> > jobs
> > on OpenStack which require more than the number of cores within a box? I
> > am
> > thinking of programs like CP2K, GROMACS, NWChem (if that sounds familiar
> > to
> > you) which utilise these kind of networks very well.
> > 
> > I cam across things like MagicCastle from Computing Canada but as far as I
> > understand it, they are not using it for production (yet).
> > 
> > Is anybody on here familiar with this?
> > 
> > All the best from London
> > 
> > Jörg
> > 
> > 
> > 
> > ___
> > Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
> > To change your subscription (digest mode or unsubscribe) visit
> > https://beowulf.org/cgi-bin/mailman/listinfo/beowulf



___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
https://beowulf.org/cgi-bin/mailman/listinfo/beowulf

[Beowulf] experience with HPC running on OpenStack

[Jörg Saßmannshausen]

Dear all,

we are currently planning a new cluster and this time around the idea was to 
use OpenStack for the HPC part of the cluster as well. 

I was wondering if somebody has some first hand experiences on the list here. 
One of the things we currently are not so sure about it is InfiniBand (or 
another low latency network connection but not ethernet): Can you run HPC jobs 
on OpenStack which require more than the number of cores within a box? I am 
thinking of programs like CP2K, GROMACS, NWChem (if that sounds familiar to 
you) which utilise these kind of networks very well. 

I cam across things like MagicCastle from Computing Canada but as far as I 
understand it, they are not using it for production (yet). 

Is anybody on here familiar with this?

All the best from London

Jörg



___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
https://beowulf.org/cgi-bin/mailman/listinfo/beowulf

Re: [Beowulf] Lmod question

[Jörg Saßmannshausen]

Hi David,

that looks like a caching problem to me. I came across some oddities with Lmod 
before and most of the time regenerating the cache helps. 

If I recall correctly, there is a parameter for how long the cache is valid. 
So you either set it very low, and the cache will be regenerated very often, 
or very high, so the cache will be generated once and stays then like this.

Maybe that is something you want to look into?

I don't have access to a machine using Lmod right now else I would dig for the 
name of the cache-timeout. 

All the best from a dark London

Jörg


Am Freitag, 19. Juni 2020, 00:18:53 BST schrieb David Mathog:
> Calling all Lmod gurus...
> 
> I have been migrating packages from one machine to another, adding their
> Lmod lua files as I go using this script:
> 
> https://saf.bio.caltech.edu/pub/software/linux_or_unix_tools/module_generate
> _from_directory.sh
> 
> This is done from an account "modules" and normally as soon as that script
> has completed doing this:
> 
> module avail 2>&1 | grep newpackagename
> 
> will find the new package.  For some reason today that stopped working,
> or at least working immediately for the modules account.  But it still
> worked immediately for root.  File protections were all good, Selinux was
> ruled out.  After poking around this appears to be related to
> cache files, because when "modules" does:
> 
> module --ignore_cache avail 2>&1 | grep newpackagename
> 
> it sees the new entry.   The cache files were indeed located:
> 
> #as modules
> ls -al ~/.lmod.d/.cache
> total 424
> drwxrwxr-x. 2 modules modules 80 Jun 18 14:37 .
> drwxrwxr-x. 3 modules modules 20 Nov  1  2019 ..
> -rw-rw-r--. 1 modules modules 200725 Jun 18 10:55
> spiderT.x86_64_Linux.luac_5.3
> 
> and 10:55 is about the time things started to go wrong.  On my other
> machines "~/.lmod.d" does not exist.  On this machine root has these files
> too, in fact two of them,  and it seems to have updated recently:
> 
> -rw-r--r--. 1 root root 269155 Jun 18 14:17 spiderT.x86_64_Linux.lua
> -rw-r--r--. 1 root root 204618 Jun 18 14:17 spiderT.x86_64_Linux.luac_5.3
> 
> On the problem machine and the ones which do not have this problem the
> cache configuration is the same:
> 
> module --config 2>&1 | grep -i cache
> number of cache dirs   0
> Ignore Cache   no
> Cached loads   no
> User cache valid time(sec) 86400
> Write cache after (sec)2
> 
> Another oddity - if a new "modules" session is started on the problem
> machine, suddenly the missing modules are listed in
> "module avail", not only in that old session, but also in the new one.
> But the time stamps on the .cache files do not change.
> 
> I have not done anything explicitly to generate those cache files.  The
> only thing modified in the files Lmod installed where the paths in
> 
> /etc/profile.d/00-modulepath.sh (and .csh)
> 
> The output of "module --config" is the same on the bad machine as on the
> good ones (other than things like differences in some version numbers,
> TCL's for instance.)
> 
> Any idea what might be going on here?  My best guess is that
> some side effect is running the spider process and generated those
> cache files, and then "module" checked them, even though it was
> configured not to (I think, see above.)  Ideally I would like to prevent
> it from making those cache files again, kind of hard to do not knowing what
> made them in the first place!
> 
> This is on CentOS 8, Lmod 8.2.7-1.
> 
> Thanks,
> 
> David Mathog
> 
> 
> ___
> Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> https://beowulf.org/cgi-bin/mailman/listinfo/beowulf



___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
https://beowulf.org/cgi-bin/mailman/listinfo/beowulf

Re: [Beowulf] Power per area

[Jörg Saßmannshausen]

Dear all,

yes, there is a max load of the floor and you really should stick to that, 
even though that might open a hole, pardon, a door for a new data centre. :-)

There are various way of getting more cooling done. You can use doors which 
are cooled, the already described plates on the CPU, you can basically use a 
large trough and put your nodes in there (the trough is filled with oil, heat 
it up enough and you can fry your chips (not)), you can do that smaller as 
Iceotop demonstrated:
https://www.iceotope.com/

I guess there are a number of ways you can address the problem. Multi-core 
CPUs, like the new AMD ones, might also be a solution as you can get more 
cores per area. 

I hope that helps a bit.

Jörg

Am Dienstag, 10. März 2020, 20:26:18 GMT schrieb David Mathog:
> On Tue, 10 Mar 2020 15:36:42 -0400 Scott Atchley wrote:
> > To make the exercise even more fun, what is the weight per square foot
> > for
> > immersion systems? Our data centers have a limit of 250 or 500 pounds
> > per
> > square foot.
> 
> I am not an architect but...
> 
> Aren't there two load values for a floor?  The one I think you are
> citing is the amount of weight which can safely be placed in a "small"
> floor area without punching through or causing other localized damage,
> the other is the total weight that can be placed on that floor without
> the building collapsing.  If the whole data center is on the ground
> floor sitting right on a concrete slab with no voids beneath it I would
> expect the latter value to be huge and not a real concern, but it might
> be less than (500 pounds per square foot) X (total area) on the 2nd or
> higher floors.
> 
> Regards,
> 
> David Mathog
> mat...@caltech.edu
> Manager, Sequence Analysis Facility, Biology Division, Caltech
> ___
> Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> https://beowulf.org/cgi-bin/mailman/listinfo/beowulf



___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
https://beowulf.org/cgi-bin/mailman/listinfo/beowulf

Re: [Beowulf] [External] Re: First cluster in 20 years - questions about today

[Jörg Saßmannshausen]

Dear Prentice

yes and no. The issue here is open access, here reputable journals are 
charging as well. 
However, in general you are right, most journals, apart from open access, do 
not charge you. They charge the libraries which are stocking their issues.

Predatory publishers is a big problem though. They are masking themselves with 
name similar to reputable journals but ask you to pay for their open access. 
Their impact factor is very low as well. 

My rule is to stick to journals which are published by the learned societies. 

All the best

Jörg

Am Freitag, 7. Februar 2020, 11:40:41 GMT schrieb Prentice Bisbal via Beowulf:
> On 2/6/20 8:36 PM, Gerald Henriksen wrote:
> > On Tue, 4 Feb 2020 21:27:51 -0500, you wrote:
> >> Assuming my work and writing is acceptable quality, how likely will I be
> >> to
> >> get published with just a master degree?
> > 
> > Can't answer that, but my understanding is that publishing in academic
> > style journals costs money so that may also be a consideration for you
> > even if you create something of interest and can work past the
> > education/lack of institution.
> 
> A legitimate journal does not charge you to have your article published.
> Journals that do are known as "predatory publishers" and usually are of
> low-reputation. We recently had a seminar here at work on predatory
> publishing and how to avoid it, which is how I know this.
> 
> Prentice
> 
> 
> ___
> Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> https://beowulf.org/cgi-bin/mailman/listinfo/beowulf

___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
https://beowulf.org/cgi-bin/mailman/listinfo/beowulf

Re: [Beowulf] First cluster in 20 years - questions about today

[Jörg Saßmannshausen]

Hi Gerald

that is not correct. You only have to pay for it if you are going for open 
access. If you are not, then you do not have to pay for it. At least that is 
the case in Chemistry as else I would not be able to publish my work in highly 
rated journals. I guess other subjects have similar arrangements.

All the best

Jörg

Am Donnerstag, 6. Februar 2020, 20:36:18 GMT schrieb Gerald Henriksen:
> On Tue, 4 Feb 2020 21:27:51 -0500, you wrote:
> >Assuming my work and writing is acceptable quality, how likely will I be to
> >get published with just a master degree?
> 
> Can't answer that, but my understanding is that publishing in academic
> style journals costs money so that may also be a consideration for you
> even if you create something of interest and can work past the
> education/lack of institution.
> ___
> Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> https://beowulf.org/cgi-bin/mailman/listinfo/beowulf

___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
https://beowulf.org/cgi-bin/mailman/listinfo/beowulf

Re: [Beowulf] First cluster in 20 years - questions about today

[Jörg Saßmannshausen]

Hi Mark,

good to know you contributed to the COMD code at the time, so you are familiar 
with what I was referring too.

Regarding publishing: Good question, I don't have the answer for that. In 
theory if the work is up to the standard of the journal, you should be able to 
publish it. However, if you are not associated with an university or research 
centre, I don't know the answer here. That was one of the things I had to look 
into recently for one reason or another I don't want to go in here. 

I guess you should address this when you got something to publish. There are 
still the preprint servers for physics and chemistry.

All the best and good luck!

Jörg

Am Dienstag, 4. Februar 2020, 21:27:51 GMT schrieb Mark Kosmowski:
> Thank you for your reply.  I actually contributed a little bit of code to
> CPMD back in the day.
> 
> I'm going to start by trying to learn abinit.  They have experimental, CUDA
> only, GPU support, so I may save up for some used nVidia cards at some
> point, maybe I can find a deal on P106 class cards.
> 
> I already have the three Opteron 940 boxes; I've kept them since buying
> them in grad school.  Having said this, you remind me that my laptop is
> probably more powerful than those old machines.  I'll use the laptop to
> learn abinit on and then to do small system calculations while I'm (likely
> slowly) getting other equipment up and running.
> 
> Assuming my work and writing is acceptable quality, how likely will I be to
> get published with just a master degree?
> 
> > Message: 1
> > Date: Sun, 02 Feb 2020 23:40:50 +
> > From: Jörg Saßmannshausen 
> > To: beowulf@beowulf.org
> > Subject: Re: [Beowulf] First cluster in 20 years - questions about
> > 
> > today
> > 
> > Message-ID: <2382819.MDnfneh6fb@deepblue>
> > Content-Type: text/plain; charset="utf-8"
> > 
> > Hi Mark,
> > 
> > being a chemist and working in HPC for some years now, for a change I can
> > make
> > some contribution to the list as well.
> > 
> > I would not advice to use hardware which is over 5 years old, unless
> > somebody
> > else is footing the electricity bill. The new AMDs are much faster and
> > also as
> > you have more cores per node, you can run larger simulations without
> > having
> > InfiniBand interconnections. The next question would be which programs do
> > you
> > want to use? ORCA? NWChem? Gamess-US? CP2K/Castep? They all have different
> > requirements and the list is by no means exhaustive. Do you just want to
> > stick
> > to DFT calculations or wavefunction ones as well (like CASSCF, CASPT2)?
> > The
> > bottom line is you want to have something which is efficient and tailored
> > to the
> > program(s) you want to use.
> > 
> > Forget about Solaris. I don't know any code other than Gamess-US which is
> > supporting Solaris. Stick to Linux. From what you said I guess you want to
> > use
> > code like CP2K which requires large memory. Again the latest AMD can
> > address
> > really large memory so I would suggest to go for that, if you really want
> > to
> > be productive. You might want to consider using NVMe as scratch/swap or
> > even
> > OS drive and, if you want to use CP2K, make sure you got enough memory and
> > cores.
> > If you just want to toy around then by all means use old hardware but you
> > will
> > have more frustration than fun.
> > 
> > For your information: I am a 'gentleman' scientist, i.e. I do my research,
> > chemistry in my case, like most respectable scientist in the evening or
> > weekend and I still got a daytime job to attend to. By enlarge I get one
> > publication out per year in highly cited journals. Right now, as until
> > recently I had some clusters at my disposal, I got an old 8 core box with
> > 42
> > GB or RAM which I am planning to replace this year with an AMD one for
> > reasons
> > already mentioned on the list. I wanted to do that last year but for one
> > reason or another that did not work out. My desktop is a Intel(R) Core(TM)
> > i7-4770 CPU @ 3.40GHz machine which also does calculations and post-
> > processing. My bottle neck right now is the time I need to write up stuff,
> > another reason why I am still using the old server. At least it is heating
> > my
> > dining room. :-)
> > 
> > Let me know if you got any more questions, happy to help out a colleague!
> > 
> > All the best
> > 
> > Jörg
> > 
> > Am Samstag, 1. Februar 2020, 22:21:09 GMT schrieb Mark Kosmowski:
> > > I've been out of computation for about 2

Re: [Beowulf] First cluster in 20 years - questions about today

[Jörg Saßmannshausen]

Dear all,

my saying as well. See my previous message.

All the best

Jörg

Am Montag, 3. Februar 2020, 09:31:41 GMT schrieb David Mathog:
> On 2020-02-02 19:25, beowulf-requ...@beowulf.org wrote:
> >> On Sat, 1 Feb 2020 22:21:09 -0500, you wrote:
> >> >Should I consider Solaris or illumos?
> >> 
> >> Unless you are absolutely sure the software you want to run works on
> >> Solaris, using Solaris/Illumos is likely asking for trouble.
> > 
> > I'm testing FreeBSD right now and will test *SunOS before commiting a
> > full
> > deployment.
> 
> Stop wasting time on that - support for OS's other than linux for
> academically maintained scientific software is spotty at best.   The
> only one that even comes close is OS X, because a lot of academics have
> Macs, but it would be super expensive to build a cluster out of those.
> Moreover, many developers, in biology at least, have pretty much given
> up on portability and focus on providing docker images.  In those cases
> you may find that building from source (needed for maximum performance)
> on even common variants like CentOS can be difficult.
> 
> > My initial 3 nodes are socket 940 Opteron based.
> 
> Fine chip in its time, which is however long past.  Even a 5 year old
> Xeon will run rings around.  I say that having just thrown out the last
> of my Opterons and replaced them with 5 year old Xeons!
> 
> A bit of research up front on memory and cpu requirments for the
> software you want to run should pay off performance wise in the final
> cluster. Some code demands a lot of memory but can barely make use of a
> second core while other software is exactly the other way around.  If
> the processing is split between different machines the speed and type of
> network connection will be an issue.  You need some handle on this
> before you start buying hardware because both "a lot of memory" and "a
> lot of cores" ups the cost.
> 
> Consider also that you might want to run GPU versions of some programs.
> When that is available it can be a significant performance enhancement.
> Will each of your nodes be able to support a GPU?   Plan ahead as these
> often have quite large power requirements which may overload your supply
> circuits or overwhelm a room's A/C.
> 
> Regards,
> 
> David Mathog
> mat...@caltech.edu
> Manager, Sequence Analysis Facility, Biology Division, Caltech
> ___
> Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> https://beowulf.org/cgi-bin/mailman/listinfo/beowulf

___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
https://beowulf.org/cgi-bin/mailman/listinfo/beowulf

Re: [Beowulf] First cluster in 20 years - questions about today

[Jörg Saßmannshausen]

Hi Mark,

being a chemist and working in HPC for some years now, for a change I can make 
some contribution to the list as well.

I would not advice to use hardware which is over 5 years old, unless somebody 
else is footing the electricity bill. The new AMDs are much faster and also as 
you have more cores per node, you can run larger simulations without having 
InfiniBand interconnections. The next question would be which programs do you 
want to use? ORCA? NWChem? Gamess-US? CP2K/Castep? They all have different 
requirements and the list is by no means exhaustive. Do you just want to stick 
to DFT calculations or wavefunction ones as well (like CASSCF, CASPT2)? The 
bottom line is you want to have something which is efficient and tailored to 
the 
program(s) you want to use. 

Forget about Solaris. I don't know any code other than Gamess-US which is 
supporting Solaris. Stick to Linux. From what you said I guess you want to use 
code like CP2K which requires large memory. Again the latest AMD can address 
really large memory so I would suggest to go for that, if you really want to 
be productive. You might want to consider using NVMe as scratch/swap or even 
OS drive and, if you want to use CP2K, make sure you got enough memory and 
cores. 
If you just want to toy around then by all means use old hardware but you will 
have more frustration than fun.

For your information: I am a 'gentleman' scientist, i.e. I do my research, 
chemistry in my case, like most respectable scientist in the evening or 
weekend and I still got a daytime job to attend to. By enlarge I get one 
publication out per year in highly cited journals. Right now, as until 
recently I had some clusters at my disposal, I got an old 8 core box with 42 
GB or RAM which I am planning to replace this year with an AMD one for reasons 
already mentioned on the list. I wanted to do that last year but for one 
reason or another that did not work out. My desktop is a Intel(R) Core(TM) 
i7-4770 CPU @ 3.40GHz machine which also does calculations and post-
processing. My bottle neck right now is the time I need to write up stuff, 
another reason why I am still using the old server. At least it is heating my 
dining room. :-)

Let me know if you got any more questions, happy to help out a colleague!

All the best

Jörg

Am Samstag, 1. Februar 2020, 22:21:09 GMT schrieb Mark Kosmowski:
> I've been out of computation for about 20 years since my master degree.
> I'm getting into the game again as a private individual.  When I was active
> Opteron was just launched - I was an early adopter of amd64 because I
> needed the RAM (maybe more accurately I needed to thoroughly thrash my swap
> drives).  I never needed any cluster management software with my 3 node,
> dual socket, single core little baby Beowulf.  (My planned domain is
> computational chemistry and I'm hoping to get to a point where I can do ab
> initio catalyst surface reaction modeling of small molecules (not
> biomolecules).)
> 
> I'm planning to add a few nodes and it will end up being fairly
> heterogenous.  My initial plan is to add two or three multi-socket,
> multi-core nodes as well as a 48 port gigabit switch.  How should I assess
> whether to have one big heterogenous cluster vs. two smaller
> quasi-homogenous clusters?
> 
> Will it be worthwhile to learn a cluster management software?  If so,
> suggestions?
> 
> Should I consider Solaris or illumos?  I do plan on using ZFS, especially
> for the data node, but I want as much redundancy as I can get, since I'm
> going to be using used hardware.  Will the fancy Solaris cluster tools be
> useful?
> 
> Also, once I get running, while I'm getting current with theory and
> software may I inquire here about taking on a small, low priority academic
> project to make sure the cluster side is working good?
> 
> Thank you all for still being here!

___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
https://beowulf.org/cgi-bin/mailman/listinfo/beowulf

Re: [Beowulf] software for activating one of many programs but not the others?

[Jörg Saßmannshausen]

Dear all,

we are using Easybuild at the Crick and for us it is working well. We got the 
flexibility we need, it is all compiled from source and if you want to use a 
different compiler, just use a different tool chain. This and the flexibility 
of lmod are really good. This also means we can maintain a central software 
stack without too many issues. 
The only time we are using anaconda is when there are python software is 
involved and it is too difficult for us to build. 

My two pennies from Little Britain!

Jörg

Am Dienstag, 20. August 2019, 10:45:52 BST schrieb Christopher Samuel:
> On 8/20/19 10:40 AM, Alex Chekholko via Beowulf wrote:
> > Other examples include RPM or EasyBuild+Lmod or less common tools like
> > Singularity or Snap/Snappy or Flatpak.
> 
> +1 for Easybuild from me.
> 
> https://easybuilders.github.io/easybuild/
> 
> There's also Spack (I really don't like the name, it's too close to a
> really offensive term for people with disabilities from the UK when I
> was in school) here:
> 
> https://spack.io/
> 
> I've not used it but I hear it's pretty good.
> 
> All the best,
> Chris

___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
https://beowulf.org/cgi-bin/mailman/listinfo/beowulf

Re: [Beowulf] Lustre on google cloud

[Jörg Saßmannshausen]

Dear all, dear Chris,

thanks for the detailed explanation. We are currently looking into cloud-
bursting so your email was very timely for me as I am suppose to look into it. 

One of the issues I can see with our workload is simply getting data into the 
cloud and back out again. We are not talking about a few Gigs here, we are 
talking up to say 1 or more TB. For reference: we got 9 PB of storage (GPFS) 
of which we are currently using 7 PB and there are around 1000+ users 
connected to the system. So cloud bursting would only be possible in some 
cases. 
Do you happen to have a feeling of how to handle the issue with the file sizes 
sensibly? 

Sorry for hijacking the thread here a bit.

All the best from a hot London

Jörg

Am Montag, 22. Juli 2019, 14:14:13 BST schrieb Chris Dagdigian:
> A lot of production HPC runs on cloud systems.
> 
> AWS is big for this via their AWS Parallelcluster stack which does
> include lustre support via vfXT for lustre service although they are
> careful to caveat it as staging/scratch space not suitable for
> persistant storage.  AWS has some cool node types now with 25gig, 50gig
> and 100-gigabit network support.
> 
> Microsoft Azure is doing amazing things now that they have the
> cyclecomputing folks on board, integrated and able to call shots within
> the product space. They actually offer bare metal HPC and infiniband
> SKUs now and have some interesting parallel filesystem offerings as well.
> 
> Can't comment on google as I've not touched or used it professionally
> but AWS and Azure for sure are real players now to consider if you have
> an HPC requirement.
> 
> 
> That said, however, a sober cost accounting still shows on-prem or
> "owned' HPC is best from a financial perspective if your workload is
> 24x7x365 constant.  The cloud based HPC is best for capability,  bursty
> workloads, temporary workloads, auto-scaling, computing against
> cloud-resident data sets or the neat new model where instead of on-prem
> multi-user shared HPC you go out and decide to deliver individual
> bespoke HPC clusters to each user or team on the cloud.
> 
> The big paradigm shift for cloud HPC is that it does not make a lot of
> sense to make a monolithic stack shared by multiple competing users and
> groups. The automated provisioning and elasticity of the cloud make it
> more sensible to build many clusters so that you can tune each cluster
> specifically for the cluster or workload and then blow it up when the
> work is done.
> 
> My $.02 of course!
> 
> Chris
> 
> > Jonathan Aquilina 
> > July 22, 2019 at 1:48 PM
> > 
> > Hi Guys,
> > 
> > I am looking at
> > https://cloud.google.com/blog/products/storage-data-transfer/introducing-l
> > ustre-file-system-cloud-deployment-manager-scripts
> > 
> > This basically allows you to deploy a lustre cluster on google cloud.
> > In your HPC setups have you considered moving towards cloud based
> > clusters?
> > 
> > Regards,
> > 
> > Jonathan
> > 
> > 
> > 
> > ___
> > Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
> > To change your subscription (digest mode or unsubscribe) visit
> > https://beowulf.org/cgi-bin/mailman/listinfo/beowulf

___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
https://beowulf.org/cgi-bin/mailman/listinfo/beowulf

Re: [Beowulf] A careful exploit?

[Jörg Saßmannshausen]

Dear all,

why port 23? Was that a typo and you mean port 22 (ssh)?

All the best

Jörg

Am Donnerstag, 13. Juni 2019, 11:09:21 BST schrieb Robert G. Brown:
> On Thu, 13 Jun 2019, Jonathan Engwall wrote:
> > It was an actual machine I could ping but I could not connect. It was
> > there
> > at start up.
> 
> If it is an actual machine, hang a console on it and see what is
> happening.  If you can ping it, its network is up.  But to be able to
> connect to it, you have to have a bunch of stuff configured to allow
> connection.  These problems all live at a higher level than the physical
> transport levels.
> 
> Personally, I'd start by killing selinux, as it is notorious for
> nearly randomly deciding that this or that connection is not secure and
> blocking it with no (EXTERNAL) warning -- it would show up in logs.  If
> you prefer, master selinux and figure out how to configure it for the
> specific ports you are trying to connect to.  Then I'd check the
> firewall.  Are you trying to ssh in?  Make sure that port 23 is open and
> not firewalled off in the default installation image.  Then check
> services.  Are you trying to ssh in?  Well, is sshd installed and
> running?  If it isn't, you have to install it, configure it, make sure
> the firewall passes it, and make sure selinux isn't going to come in and
> override the firewall and refuse to pass it after all.  And so on, for
> any port(s) you wish to access.  Most linuxes these days install in a
> default "secure" mode with no open ports and firewalled up pretty tight,
> assuming that the installer is a normal human who has no idea how to
> offer services or secure them, but if you run a cluster you really need
> to be at least on the road to being an abnormal person who does.
> 
> If you're trying to build a cluster that automagically installs with all
> of this stuff up, well, then you'll need to read the manual(s) or
> whatever documentation they provide to see what you didn't preconfigure
> on the install host.
> 
> Hopefully you're getting the idea that debugging networking problems
> requires a) a pretty good knowledge of networking from the wire on up to
> the network application; b) a pretty good knowledge of systems
> administration and how to set up, start, manage, debug applications,
> read logs (know where the logs are to read, for starters) etc; c) a very
> patient and systematic approach.  As Chris says, start at the wire up,
> if it is wired, look at the wireless router tables of connected hosts if
> it is wireless, etc.  See if it pings.  If it pings, see what's
> wrong with the ports/services you're trying to connect to.  Read logs.
> Try experiments.  Compare a working host to the one that isn't working.
> Read the logs some more.
> 
> It's all in there, if you know how to get it out.
> 
> And again, if you really want our help, repost a DETAILED DESCRIPTION OF
> WHAT IS WRONG.  I'd wager 90% or more of the people on this list could
> debug your problem from a sufficiently detailed description alone, but
> so far we know next to nothing about what you are trying to do, what
> your network looks like, what version of Linux (or other operating
> system!) you are using, what tools you're talking about.  I don't even
> know if you are really trying to build or work with a cluster or are
> just trying to figure out why ssh doesn't work out of the box on hosts
> in an office.
> 
> Details, please!
> 
>  rgb
> 
> > On Tue, Jun 11, 2019, 9:49 PM Chris Samuel  wrote:
> >   On 11/6/19 8:18 pm, Robert G. Brown wrote:
> >   > * Are these real hosts, each with their own network interface
> >   
> >   (wired or
> >   
> >   > wireless), or are these virtual hosts?
> >   
> >   In addendum to RGB's excellent advice and questions I would add
> >   to this
> >   question the network engineers maxim of "start at layer 1 and
> >   work up".
> >   
> >   In other words, first check your physical connectivity and then
> >   head up
> >   the layers.
> >   
> >   Best of luck!
> >   Chris
> >   --
> >   ? Chris Samuel? :?http://www.csamuel.org/? :?Berkeley, CA, USA
> >   ___
> >   Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin
> >   Computing
> >   To change your subscription (digest mode or unsubscribe) visit
> >   https://beowulf.org/cgi-bin/mailman/listinfo/beowulf
> 
> Robert G. Brown  http://www.phy.duke.edu/~rgb/
> Duke University Dept. of Physics, Box 90305
> Durham, N.C. 27708-0305
> Phone: 1-919-660-2567  Fax: 919-660-2525 email:r...@phy.duke.edu
> 
> 
> ___
> Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> https://beowulf.org/cgi-bin/mailman/listinfo/beowulf

___
Beowulf 

Re: [Beowulf] Dual IB Cards in a Single Box

[Jörg Saßmannshausen]

Hi Paul,

if that is a Mellanox network I would have thought that putting in a single, 
dual port HDR InfiniBand card would do the trick as well. From what I 
understand, the HDR will use the slower FDR speed when the other end is only 
EDR. 
I done a similar setup with a mixed network where the switch is FDR but some 
of the nodes are only QDR and here the connection is throttled down to QDR.

I might, of course, be wrong here. 

All the best from a sunny London

Jörg

Am Freitag, 10. Mai 2019, 12:13:42 BST schrieb Paul Edmon:
> We are looking to bridge our FDR fabric to our HDR fabric via LNET and
> we had a question about whether a single machine can have 2 different IB
> cards simultaneously. Does anyone have any experience with that or know
> if there are any hitches with having a single box with a FDR card and a
> HDR card running simultaneously?
> 
> -Paul Edmon-
> 
> ___
> Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> https://beowulf.org/cgi-bin/mailman/listinfo/beowulf

___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
https://beowulf.org/cgi-bin/mailman/listinfo/beowulf

Re: [Beowulf] Frontier Announcement

[Jörg Saßmannshausen]

Dear all,

I think the answer to the question lies here:

https://en.wikipedia.org/wiki/OpenACC

As I follow these things rather loosely, my understanding was that OpenACC 
should run on both nVidia and other GPUs. So maybe that is the reason why it 
is a 'pure' AMD cluster where both GPUs and CPUs are from the same supplier?
IF all of that is working out and if it is really true that you can compile 
and run OpenACC code on both types of GPUs, it would a be big win for AMD.

Time will tell!

All the best from my TARDIS!

Jörg

Am Dienstag, 7. Mai 2019, 16:59:48 BST schrieben Sie:
> >   I think it is interesting that they are using AMD for
> > 
> > both the CPUs and GPUs
> 
> I agree. That means a LOT of codes will have to be ported from CUDA to
> whatever AMD uses. I know AMD announced their HIP interface to convert
> CUDA code into something that will run on AMD processors, but I don't
> know how well that works in theory. Frankly, I haven't heard anything
> about it since it was announced at SC a few years ago.
> 
> I would not be surprised if AMD pursued this bid quite agressively,
> possibly at a significant loss, for the opportunity to prove their GPUs
> can compete with NVIDIA and demonstrate that codes can be successfully
> converted from CUDA to something AMD GPUs can use to demonstrate GPU
> users don't need to be locked in to a single vendor. If so, this could
> be a costly gamble for the DOE and AMD, but if it pays off, I imagine it
> could change AMD's fortunes in HPC.
> 
>   "Win on Sunday, sell on Monday" doesn't apply just to cars.
> 
> Prentice
> 
> On 5/7/19 4:43 PM, Jörg Saßmannshausen wrote:
> > Hi Prentice,
> > 
> > that looks interesting and I hope it means I will finally get the neutron
> > structure which was measured last year there! :-)
> > 
> > On a more serious note: I think it is interesting that they are using AMD
> > for both the CPUs and GPUs. It sounds at least very fast of what they
> > want to build, lets hope their design will work as planned as well.
> > 
> > All the best from London
> > 
> > Jörg
> > 
> > Am Dienstag, 7. Mai 2019, 10:32:42 BST schrieb Prentice Bisbal via 
Beowulf:
> >> ORNL's Frontier System has been announced:
> >> 
> >> https://www.hpcwire.com/2019/05/07/cray-amd-exascale-frontier-at-oak-ridg
> >> e/

___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
https://beowulf.org/cgi-bin/mailman/listinfo/beowulf

Re: [Beowulf] Frontier Announcement

[Jörg Saßmannshausen]

Hi Prentice,

that looks interesting and I hope it means I will finally get the neutron 
structure which was measured last year there! :-)

On a more serious note: I think it is interesting that they are using AMD for 
both the CPUs and GPUs. It sounds at least very fast of what they want to 
build, lets hope their design will work as planned as well. 

All the best from London

Jörg

Am Dienstag, 7. Mai 2019, 10:32:42 BST schrieb Prentice Bisbal via Beowulf:
> ORNL's Frontier System has been announced:
> 
> https://www.hpcwire.com/2019/05/07/cray-amd-exascale-frontier-at-oak-ridge/

___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
https://beowulf.org/cgi-bin/mailman/listinfo/beowulf

Re: [Beowulf] GPFS question

[Jörg Saßmannshausen]

Dear all,

many thanks for all the emails, both on- and offline, which was a great help 
for 
me to get some idea how long that might take. 

All the best from a sunny London!

Jörg

Am Dienstag, 30. April 2019, 07:58:20 BST schrieb John Hearns via Beowulf:
> Hi Jorg. I will mail you offline.
> IBM support for GPFS is excellent - so if they advise a check like that it
> is needed.
> 
> On Tue, 30 Apr 2019 at 04:53, Chris Samuel  wrote:
> > On Monday, 29 April 2019 3:47:10 PM PDT Jörg Saßmannshausen wrote:
> > > thanks for the feedback. I guess it also depends how much meta-data you
> > 
> > have
> > 
> > > and whether or not you have zillions of small or larger files.
> > > At least I got an idea how long it might take.
> > 
> > This thread might also be useful, it is a number of years old but it does
> > have some
> > advice on placement of the filesystem manager before the scan and also on
> > their
> > experience scanning a ~1PB filesystem.
> > 
> > 
> > https://www.ibm.com/developerworks/community/forums/html/topic?id=
> > ----14834266
> > 
> > All the best,
> > Chris
> > --
> > 
> >   Chris Samuel  :  http://www.csamuel.org/  :  Berkeley, CA, USA
> > 
> > ___
> > Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
> > To change your subscription (digest mode or unsubscribe) visit
> > https://beowulf.org/cgi-bin/mailman/listinfo/beowulf

___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
https://beowulf.org/cgi-bin/mailman/listinfo/beowulf

Re: [Beowulf] GPFS question

[Jörg Saßmannshausen]

Hi Lance,

thanks for the feedback. I guess it also depends how much meta-data you have 
and whether or not you have zillions of small or larger files. 
At least I got an idea how long it might take.

All the best

Jörg

Am Dienstag, 30. April 2019, 07:59:06 BST schrieb Lance Wilson:
> Hi Jörg,
> Our 1Pb array took up to 3 days when we did this. It might be faster for
> you but it took a very long time with little to no indication of how long
> it would take. Just a word of caution though, we didn’t do an offline scan
> once too long and that scan took much longer than previously. Good luck!
> 
> Lance
> 
> On Tue, 30 Apr 2019 at 7:34 am, Jörg Saßmannshausen <
> 
> sassy-w...@sassy.formativ.net> wrote:
> > Dear all,
> > 
> > just a quick question regarding GPFS:
> > we are running a 9 PB GPFS storage space at work of which around 6 -7 PB
> > are
> > used. It is a single file system but with different file-sets installed on
> > it.
> > During our routine checks we found that:
> > $ mmhealth node show -n all
> > reports this problem:
> > 
> > fserrinvalid(FOO)
> > 
> > (where FOO is being the file system).
> > 
> > Our vendor suggested to do an online check:
> > 
> > $ mmfsck FOO -o -y
> > 
> > which is still running.
> > Today the vendor suggested to take the GPFS file system offline and run
> > the above
> > command without the -o option, which would lead to an outage.
> > 
> > So my simply question is: has anybody ever done that on such a large file
> > set
> > and how long roughly would that take? Every time I am asking this question
> > I
> > get told: a long time!
> > Our vendor told us we could use for example
> > --threads 128
> > as oppose to the normally used 16 threads, so I am aware my mileage will
> > vary
> > here a bit, but I would just like a guestimate of the time.
> > 
> > Many thanks for your help here!
> > 
> > All the best from London
> > 
> > Jörg
> > 
> > ___
> > Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
> > To change your subscription (digest mode or unsubscribe) visit
> > https://beowulf.org/cgi-bin/mailman/listinfo/beowulf

___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
https://beowulf.org/cgi-bin/mailman/listinfo/beowulf

[Beowulf] GPFS question

[Jörg Saßmannshausen]

Dear all,

just a quick question regarding GPFS:
we are running a 9 PB GPFS storage space at work of which around 6 -7 PB are 
used. It is a single file system but with different file-sets installed on it.
During our routine checks we found that:
$ mmhealth node show -n all
reports this problem:

fserrinvalid(FOO)

(where FOO is being the file system).

Our vendor suggested to do an online check:

$ mmfsck FOO -o -y

which is still running. 
Today the vendor suggested to take the GPFS file system offline and run the 
above 
command without the -o option, which would lead to an outage. 

So my simply question is: has anybody ever done that on such a large file set 
and how long roughly would that take? Every time I am asking this question I 
get told: a long time! 
Our vendor told us we could use for example 
--threads 128
as oppose to the normally used 16 threads, so I am aware my mileage will vary 
here a bit, but I would just like a guestimate of the time. 

Many thanks for your help here!

All the best from London

Jörg

___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
https://beowulf.org/cgi-bin/mailman/listinfo/beowulf

Re: [Beowulf] Thoughts on EasyBuild?

[Jörg Saßmannshausen]

Dear all,

we are using Easybuild at work. Like most projects, you mileage will vary. I 
am using it since October last year so my experiences might be a bit limited 
here. What I found is that in most cases it really does what it says on the 
tin. You can simply say 'build cp2k with the Intel compiler' which is a rather 
big build if you put in all its dependencies, and it does it. You issue one 
command and when you come back from lunch it has done it all for you. I 
recently noticed it even downloads the Intel compiler it needs, next to all 
the dependencies, installed them and created the necessary modules. 
Like most things, I had a few issues with it as well. Some where around 
downloading the source code. If the URL changed or there is a network problem, 
then it obviously fails. That can be easily corrected. 
I did run in some more complicated problems but I have to say the people on 
the Slack channel are very helpful, friendly and responsive. 

I guess is it depends on what you want to do. If you only have a small 
software stack, it might be a bit of an overkill. If you happen to support a 
large institution with big demands, it helps. Due to the use of modules, you 
can provided different versions of one software as well. They are providing a 
large set of Easybuild files and you can modify and create your owns as well. 
We are not using the Python ones for one reason or another, here we are using 
the conda environment. Else everything is done in Easybuild. 
We also have a few different versions of Easybuild in our software stack, next 
to some home brew Easybuild build files. 

I would say give it a go, install a few complex systems you need on a test 
machine, see how you find it, and decide. 

My $2.00 here.

All the best from a cold London

Jörg

Am Donnerstag, 17. Januar 2019, 08:32:36 GMT schrieb Faraz Hussain:
> Some folks I work with are using EasyBuild but I am not sure what to
> make of it. I am little skeptical of tools that promise one-command
> installations of complex software. On the other hand I can see its
> benefits by saving a lot of "re-inventing the wheel".
> 
> What do people think? Like everything, the answer is probably it
> depends on what you use it for? Or is it another layer of complexity
> that just makes everything more complex?
> 
> ___
> Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf

___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
http://www.beowulf.org/mailman/listinfo/beowulf

Re: [Beowulf] USB flash drive bootable distro to check cluster health.

[Jörg Saßmannshausen]

Hi,

why USB? Why not PXE boot?

Back at the old workplace I had a PXE boot image installed on the headnode 
which I used to install the new compute node and did debugging or testing. 
If you don't want to install anything on the cluster: put that image on a USB 
stick, boot it on the headnode and then boot the compute nodes. So everything 
should run of the USB stick (headnode) and via the network (compute nodes in 
PXE boot).

Depending on the cluster size I guess there are limits of how many compute 
nodes you could run simultaneously. 

Just an idea.

All the best from a dark London

Jörg


Am Freitag, 11. Januar 2019, 18:29:28 GMT schrieb Richard Chang:
> Hi,
> I would like to know if we have or can make( or prepare) a USB bootable OS
> that we can boot in a cluster and its nodes to test all its functionality.
> 
> The purpose of this is to boot a new or existing cluster to check its
> health, including Infiniband network,  any cards, local hard disks, memory
> etc, so that I don't have to disturb the existing OS and its configuration.
> 
> If possible, it would be nice to boot the compute nodes from the master
> node.
> 
> Anyone knows of any pre-existing distribution that will do the job ? Or know
> how to do it with Centos or Ubuntu ?
> 
> Thanks,
> Richard.
> ___
> Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf

___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
http://www.beowulf.org/mailman/listinfo/beowulf

Re: [Beowulf] Oh.. IBM eats Red Hat

[Jörg Saßmannshausen]

Hi all,

it is not only that, but I saw something about Mellanox today as well. 

Not good news in my humble opinion. :-(

All the best

Jörg


Am Montag, 29. Oktober 2018, 07:42:48 GMT schrieb Tony Brian Albers:
> https://www.reuters.com/article/us-red-hat-m-a-ibm/ibm-to-acquire-softw
> are-company-red-hat-for-34-billion-idUSKCN1N20N3
> 
> I wonder where that places us in the not too distant future..
> 
> I've worked for Big Blue, and I'm not sure the company cultures are
> compatible to say the least.
> 
> /tony

___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
http://www.beowulf.org/mailman/listinfo/beowulf

Re: [Beowulf] Contents of Compute Nodes Images vs. Login Node Images

[Jörg Saßmannshausen]

Hi Prentice,

that sounds somehow similar of what I have done back at my days at UCL:
- login node with development packages
- compute node only what is really needed in terms of software and services

However, if you are removing packages from the compx.xml file manually, how can 
you be sure you are not breaking dependencies?

As I was using Debian, I simply did bare installation and then installed what 
I needed. Once I got a running OS, I rsynced that to a folder on the headnode 
which was the 'image' of the compute nodes. Depending on the cluster, I build 
the software on the login node and copied it to the folder where the image 
was. So during installation, that image folder was copied to the compute nodes 
and I only had to install the boot-loader (I never really looking into how to 
script that as well) using PXE boot. It worked quite well.
Upgrading a software package simply means installing it inside the image 
foldeer (either chroot if it was a .deb package or just copie the files over) 
and rsynced it to the compute nodes. 


It was a robust system and I managed to handle th 112 compute nodes quite well 
I had. I even could take care of older and newer nodes and can install highly 
optimised packages on them as well. So nodes which only got avx got only the 
avx enabled software and the ones which had avx2 got these ones. 

It might not be the most flashy solution but it was KIS: Kepp It Simple!

All the best from a rainy London

Jörg

Am Dienstag, 23. Oktober 2018, 13:43:43 GMT schrieb Prentice Bisbal via 
Beowulf:
> Ryan,
> 
> When I was at IAS, I pared down what was on the compute nodes
> tremendously. I went through the comps.xml file practically line-by-line
> and reduced the number of packages installed on the compute nodes to
> only about 500 RPMs. I can't remember all the details, but I remember
> omitting the following groups of packages:
> 
> 1. Anything related to desktop environments, graphics, etc.
> 2. -devel packages
> 3. Any RPMS for wireless or bluetooth support.
> 4. Any kind of service that wasn't strictly needed by the compute nodes.
> 
> In this case, the user's desktops mounted the same home and project
> directories and shared application directory (/usr/local), so the user's
> had all the the GUI, post-processing, and devel packages they needed 
> right on their desktop, so the cluster was used purely for running
> non-interactive batch jobs. In fact, there was no way for a user to even
> get an interactive session on the cluster.  IAS  was a small environment
> where I had complete control over the desktops and the cluster, so I was
> able to this. I would do it all again just like that, given as similar
> environment.
> 
> I'm currently managing a cluster with PU, and PU only puts the -devel
> packages, etc. on the the login nodes so users can compile there apps
> there.
> 
> So yes, this is still being done.
> 
> There are definitely benefits to providing specialized packages lists
> like this:
> 
> 1. On the IAS cluster, a kickstart installation, including configuration
> with the post-install script, was very quick - I think it was 5 minutes
> at most.
> 2. You generally want as few services running on your compute nodes as
> possible. The easiest way to keep services from running on your cluster
> nodes is to not install those services in the first place.
> 3. Less software installed = smaller attack surface for security exploits.
> 
> Does this mean you are moving away from Warewulf, or are you creating
> different Warewulf images for login vs. compute nodes?
> 
> 
> Prentice
> 
> On 10/23/2018 12:15 PM, Ryan Novosielski wrote:
> > Hi there,
> > 
> > I realize this may not apply to all cluster setups, but I’m curious what
> > other sites do with regard to software (specifically distribution
> > packages, not a shared software tree that might be remote mounted) for
> > their login nodes vs. their compute nodes. From what I knew/conventional
> > wisdom, sites generally place pared down node images on compute nodes,
> > only containing the runtime. I’m curious to see if that’s still true, or
> > if there are people doing something else entirely, etc.
> > 
> > Thanks.
> > 
> > --
> > 
> > 
> > || \\UTGERS, |---
*O*---
> > ||
> > ||_// the State  | Ryan Novosielski - novos...@rutgers.edu
> > ||
> > || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS
> > || Campus
> > || 
> > ||  \\of NJ  | Office of Advanced Research Computing - MSB C630,
> > ||  Newark
> > ||  
> >   `'
> > 
> > ___
> > Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
> > To change your subscription (digest mode or unsubscribe) visit
> > http://www.beowulf.org/mailman/listinfo/beowulf
> ___
> Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
> To change your 

Re: [Beowulf] RHEL7 kernel update for L1TF vulnerability breaks RDMA

[Jörg Saßmannshausen]

Hi Ade,

thanks for this. I will give it a spin.

So far I only done a simple ping-pong test but never done a RDMA test.

All the best and thanks!

Jörg

Am Dienstag, 2. Oktober 2018, 21:33:09 BST schrieb Ade Fewings:
> Hello from Wales
> 
> Red Hat quoted just a simple ib_write_bw test as indicating the broken state
> of IB RDMA (https://access.redhat.com/solutions/3568891):
> 
> Run a RDMA write bandwidth test. ib_write_bw is provided by the package
> perftest.
> 
> On target node run :
> # ib_write_bw
> 
> On client side run :
> # ib_write_bw 
> 
> The test should fail.
> 
> Hope that helps
> Ade
> 
> 
> 
> -Original Message-
> From: Beowulf  On Behalf Of Jörg Saßmannshausen
> Sent: 02 October 2018 22:20
> To: beowulf@beowulf.org
> Subject: Re: [Beowulf] RHEL7 kernel update for L1TF vulnerability breaks
> RDMA
> 
> Dear all,
> 
> is there some kind of quick test to demonstrate the patch does have or does
> not cause a problem with RDMA? I have been asked to look into that but I
> don't really want to use a large cp2k calculation which, I believe, makes
> use of RDMA.
> 
> All the best from London
> 
> Jörg
> 
> Am Mittwoch, 12. September 2018, 18:02:06 BST schrieb John Hearns via 
Beowulf:
> > Regarding CentOS, Karanbir Singh is the leader of the project and has
> > a job at Redhat
> > https://www.linuxfoundation.org/blog/2014/01/centos-project-leader-kar
> > anbir-> singh-opens-up-on-red-hat-deal/ On Tue, 11 Sep 2018 at 18:03,
> > Peter St. John 
> 
> wrote:
> > > I mean the RH QA that tests RH products isn't the same team as tests
> > > (or
> > > not) CentOS, but I only know from the wiki that RH has an expanding
> > > agreement with CentOS so may be this is all merging. As I said, my
> > > buddy doesn't work in this area, and I sure don't. Probably all you
> > > guys are more up to date on the merging than either of us.> On Tue,
> > > 
> > > Sep 11, 2018 at 12:50 PM, Peter Kjellström  wrote:
> > >> On Tue, 11 Sep 2018 12:37:18 -0400
> > >> 
> > >> "Peter St. John"  wrote:
> > >> > A friend at RH (who works in a different area) tells me RH does
> > >> > not themselves test the downstream CentOS.
> > >> > Peter
> > >> 
> > >> That isn't surprising is it? But in this case we're talking about
> > >> them not testing their own product.. :-D
> > >> 
> > >> /Peter K
> > >> 
> > >> 
> > >> --
> > >> Sent from my Android device with K-9 Mail. Please excuse my brevity.
> > > 
> > > ___
> > > Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin
> > > Computing To change your subscription (digest mode or unsubscribe)
> > > visit http://www.beowulf.org/mailman/listinfo/beowulf
> > 
> > ___
> > Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin
> > Computing To change your subscription (digest mode or unsubscribe)
> > visit http://www.beowulf.org/mailman/listinfo/beowulf
> 
> ___
> Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing To
> change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
> 
> 
>[HPC Wales - www.hpcwales.co.uk] <http://www.hpcwales.co.uk>
> 
> 
> 
> The contents of this email and any files transmitted with it are
> confidential and intended solely for the named addressee only.  Unless you
> are the named addressee (or authorised to receive this on their behalf) you
> may not copy it or use it, or disclose it to anyone else.  If you have
> received this email in error, please notify the sender by email or
> telephone.  All emails sent by High Performance Computing Wales have been
> checked using an Anti-Virus system.  We would advise you to run your own
> virus check before opening any attachments received as we will not in any
> event accept any liability whatsoever, once an email and/or attachment is
> received.
> 
> High Performance Computing Wales is a private limited company incorporated
> in Wales on 8 March 2010 as company number 07181701.
> 
> Our registered office is at Finance Office, Bangor University, Cae Derwen,
> College Road, Bangor, Gwynedd. LL57 2DG. UK.
> 
> High Performance Computing Wales is part funded by the European Regional
> Development Fund through the Welsh Government.

___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
http://www.beowulf.org/mailman/listinfo/beowulf

Re: [Beowulf] RHEL7 kernel update for L1TF vulnerability breaks RDMA

[Jörg Saßmannshausen]

Dear all,

is there some kind of quick test to demonstrate the patch does have or does 
not cause a problem with RDMA? I have been asked to look into that but I don't 
really want to use a large cp2k calculation which, I believe, makes use of 
RDMA.

All the best from London

Jörg

Am Mittwoch, 12. September 2018, 18:02:06 BST schrieb John Hearns via Beowulf:
> Regarding CentOS, Karanbir Singh is the leader of the project and has
> a job at Redhat
> https://www.linuxfoundation.org/blog/2014/01/centos-project-leader-karanbir-> 
> singh-opens-up-on-red-hat-deal/
> On Tue, 11 Sep 2018 at 18:03, Peter St. John  
wrote:
> > I mean the RH QA that tests RH products isn't the same team as tests (or
> > not) CentOS, but I only know from the wiki that RH has an expanding
> > agreement with CentOS so may be this is all merging. As I said, my buddy
> > doesn't work in this area, and I sure don't. Probably all you guys are
> > more up to date on the merging than either of us.> 
> > On Tue, Sep 11, 2018 at 12:50 PM, Peter Kjellström  wrote:
> >> On Tue, 11 Sep 2018 12:37:18 -0400
> >> 
> >> "Peter St. John"  wrote:
> >> > A friend at RH (who works in a different area) tells me RH does not
> >> > themselves test the downstream CentOS.
> >> > Peter
> >> 
> >> That isn't surprising is it? But in this case we're talking about them
> >> not testing their own product.. :-D
> >> 
> >> /Peter K
> >> 
> >> 
> >> --
> >> Sent from my Android device with K-9 Mail. Please excuse my brevity.
> > 
> > ___
> > Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
> > To change your subscription (digest mode or unsubscribe) visit
> > http://www.beowulf.org/mailman/listinfo/beowulf
> ___
> Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf

___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
http://www.beowulf.org/mailman/listinfo/beowulf

Re: [Beowulf] RHEL7 kernel update for L1TF vulnerability breaks RDMA

[Jörg Saßmannshausen]

Dear all,

whereas I am accepting that no system is 100% secure ans bug-free, I am 
beginning to wonder whether the current problems we are having are actually 
design flaws and whether, and that is the more important bit, Intel and other 
vendors did know about it. I am thinking of the famous 'diesel-engine' scandal 
and, continuing this line of thought, dragging the vendors into the limelight 
and get them to pay for this. 
I mean, we have to sort out the mess the company was making in the first place, 
have to judge whether to apply a patch which might decrease the performance of 
our systems (I am doing HPC, hence my InfiniBand question) versus security. 
Where will it stop?

Given the current and previous 'bugs' are clearly design flaws IMHO, what are 
the chances of a law suite? The any compensation here should go to Open Source 
projects, in my opinion, which are making software more secure. 

Any comments here?

All the best

Jörg

Am Sonntag, 19. August 2018, 06:11:16 BST schrieb John Hearns via Beowulf:
> Rather more seriously, this is a topic which is well worth discussing,
> What are best practices on patching HPC systems?
> Perhaps we need a separate thread here.
> 
> I will throw in one thought, which I honestly do not want to see happening.
> I recently took a trip to Bletchley Park in the UK. On display there was an
> IBM punch card machine and sample punch cards Back in the day one prepared
> a 'job deck' which was collected by an operator in a metal hopper then
> wheeled off to the mainframe. You did not ever touch the mainframe. So
> effectively an air gapped system. A system like that would in these days
> kill productivity.
> However should there be 'virus checking' of executables  before they are
> run on compute nodes.
> One of the advantages lauded for Linux systems is of course that anti-virus
> programs are not needed.
> 
> Also I should ask - in the jargon of anti-virus is there a 'signature' for
> any of these exploit codes? One would guess that bad actors copy the
> example codes already published and use these almost in a cut and paste
> fashion. So the signature would be tight loops repeatedly reading or
> writing to the same memory locations. Can that be distinguished from
> innocent code?
> 
> On Sun, 19 Aug 2018 at 05:59, John Hearns  wrote:
> > *To patch, or not to patch, that is the question:* Whether 'tis nobler in
> > the mind to suffer
> > The loops and branches of speculative execution,
> > Or to take arms against a sea of exploits
> > And by opposing end them. To die—to sleep,
> > No more; and by a sleep to say we end
> > The heart-ache and the thousand natural shocks
> > That HPC is heir to: 'tis a consummation
> > Devoutly to be wish'd. To die, to sleep
> > 
> > On Sun, 19 Aug 2018 at 02:31, Chris Samuel  wrote:
> >> On Sunday, 19 August 2018 5:19:07 AM AEST Jeff Johnson wrote:
> >> > With the spate of security flaws over the past year and the impacts
> >> 
> >> their
> >> 
> >> > fixes have on performance and functionality it might be worthwhile to
> >> 
> >> just
> >> 
> >> > run airgapped.
> >> 
> >> For me none of the HPC systems I've been involved with here in Australia
> >> would
> >> have had that option.  Virtually all have external users and/or reliance
> >> on
> >> external data for some of the work they are used for (and the sysadmins
> >> don't
> >> usually have control over the projects & people who get to use them).
> >> 
> >> All the best,
> >> Chris
> >> --
> >> 
> >>  Chris Samuel  :  http://www.csamuel.org/  :  Melbourne, VIC
> >> 
> >> ___
> >> Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
> >> To change your subscription (digest mode or unsubscribe) visit
> >> http://www.beowulf.org/mailman/listinfo/beowulf

___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
http://www.beowulf.org/mailman/listinfo/beowulf

Re: [Beowulf] RHEL7 kernel update for L1TF vulnerability breaks RDMA

[Jörg Saßmannshausen]

Hi Chris,

unless there is something I miss but I read about that in 'Der Spiegel Online' 
on Wednesday 

http://www.spiegel.de/netzwelt/gadgets/foreshadow-neue-angriffsmethode-trifft-intel-chips-und-cloud-dienste-a-1223289.html

and the link was to this page here:

https://foreshadowattack.eu/

So I don't really understand about "Cannot make this public, as the patch that 
caused it was due to embargo'd security fix." issue. The problem is known and I 
also noticed that Debian issued some Intel microcode patches which raised my 
awareness about a potential problem again.

Sorry, maybe I miss out something here.

All the best

Jörg

Am Samstag, 18. August 2018, 21:56:52 BST schrieb Chris Samuel:
> On 18/8/18 8:47 pm, Jörg Saßmannshausen wrote:
> > Hi Chris,
> 
> Hiya,
> 
> > these are bad news if InfiniBand will be affected here as well as
> > that is what we need to use for parallel calculations. They make use
> > of RMDA and if that has a problem. well, you get the idea I
> > guess.
> 
> Oh yes, this is why I wanted to bring it to everyones attention, this
> isn't just about Lustre, it's much more widespread.
> 
> > Has anybody contacted the vendors like Mellanox or Intel regarding
> > this?
> 
> As Kilian wrote in the Lustre bug quoting his RHEL bug:
> 
>  https://bugzilla.redhat.com/show_bug.cgi?id=1618452
> 
>  — Comment #3 from Don Dutile  —
>  Already reported and being actively fixed.
> 
>  Cannot make this public, as the patch that caused it was due to
> embargo'd
>  security fix.
> 
>  This issue has highest priority for resolution.
>  Revert to 3.10.0-862.11.5.el7 in the mean time.
> 
>  This bug has been marked as a duplicate of bug 1616346

___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
http://www.beowulf.org/mailman/listinfo/beowulf

Re: [Beowulf] RHEL7 kernel update for L1TF vulnerability breaks RDMA

[Jörg Saßmannshausen]

Hi Chris,

these are bad news if InfiniBand will be affected here as well as that is what 
we need to use for parallel calculations. They make use of RMDA and if that 
has a problem. well, you get the idea I guess.

Has anybody contacted the vendors like Mellanox or Intel regarding this?

All the beset

Jörg

Am Samstag, 18. August 2018, 18:31:35 BST schrieb Christopher Samuel:
> On 18/08/18 17:22, Jörg Saßmannshausen wrote:
> > if the problem is RMDA, how about InfiniBand? Will that be broken as
> > well?
> 
> For RDMA it appears yes, though IPoIB still works for us (though ours is
> OPA rather than IB Kilian reported the same).
> 
> All the best,
> Chris

___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
http://www.beowulf.org/mailman/listinfo/beowulf

Re: [Beowulf] RHEL7 kernel update for L1TF vulnerability breaks RDMA

[Jörg Saßmannshausen]

Dear all,

if the problem is RMDA, how about InfiniBand? Will that be broken as well?

All the best

Jörg

Am Samstag, 18. August 2018, 13:33:55 BST schrieb Chris Samuel:
> On Saturday, 18 August 2018 12:54:03 AM AEST Kilian Cavalotti wrote:
> > That's true: RH mentioned an "embargo'd security fix" but didn't refer
> > to L1TF explicitly (which I think is not under embargo anymore).
> 
> Agreed, though I'm not sure any of the listed fixes are embargoed now.
> 
> > As the reporter of the issue on the Whamcloud JIRA, I also have to
> > apologize for initially pointing fingers at Lustre, it didn't cross my
> > mind that this kind of whole RDMA stack breakage would have slipped
> > past Red Hat's QA.
> 
> Oh I didn't read that as pointing any fingers at Lustre at all, just that
> the kernel update broke Lustre for you (and for us!).
> 
> All the best,
> Chris

___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
http://www.beowulf.org/mailman/listinfo/beowulf

Re: [Beowulf] RHEL7 kernel update for L1TF vulnerability breaks RDMA

[Jörg Saßmannshausen]

Hi all,

I came across the 'foreshadow' problem 2 days ago. 

This is what I got back from my colleagues:

https://access.redhat.com/security/vulnerabilities/L1TF-perf

This is more a performance investigation though but I thought I might add a 
bit more information to the whole problem.

All the best

Jörg

Am Freitag, 17. August 2018, 07:54:03 BST schrieb Kilian Cavalotti:
> Hi Chris,
> 
> On Thu, Aug 16, 2018 at 10:05 PM, Chris Samuel  wrote:
> > There's 6 CVE's addressed in that update from the look of it, so it might
> > not be the L1TF fix itself that has triggered it.
> > 
> > https://access.redhat.com/errata/RHSA-2018:2384
> 
> That's true: RH mentioned an "embargo'd security fix" but didn't refer
> to L1TF explicitly (which I think is not under embargo anymore).
> 
> As the reporter of the issue on the Whamcloud JIRA, I also have to
> apologize for initially pointing fingers at Lustre, it didn't cross my
> mind that this kind of whole RDMA stack breakage would have slipped
> past Red Hat's QA.
> 
> Cheers,

___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
http://www.beowulf.org/mailman/listinfo/beowulf

[Beowulf] Fwd: Questionnaire on the value of newer Fortran standards

[Jörg Saßmannshausen]

Dear all,

it might be a bit off-topic, so please forgive me, but I have been asked to 
circulate that as widely as possible. 
I would appreciate it if you could circulate the below on behalf of Anton 
Shterenlikht (University of Bristol/IK) to Fortran users at your institution.

All the best

Jörg

--  Weitergeleitete Nachricht  --

***

Benefits of continuing Fortran standardisation

If you use Fortran, directly or indirectly, please will you complete
this survey
  https://goo.gl/forms/JUFUReOoVUin2m8D2
It has been developed by the committee of the BCS Fortran Group to
quantify the value of modern Fortran standards to organisations and
individuals.  The results of the survey will help the Group justify
continuing involvement in Fortran standardisation efforts.

The results of the survey will be shared with the ISO Fortran
standardisation committee, so your responses will help shape the future
of the Fortran language.

The survey will run until 31 Dec. 2018.

The Fortran language has been steadily developing since its origins in
1957.  Many people have been working on revising the Fortran
specification, resulting in Fortran 77, 90, 95, 2003, 2008 and 2018
standards.  This survey is designed to find out exactly what benefits
newer Fortran standards bring to the community.

We would like to know how newer Fortran standards have increased the
quality of your code, cut development costs, increased portability or
performance of your code, or whether you can attach any monetary value
to the benefits enabled by modern Fortran standards.

This questionnaire contains 4 sections.
All questions are optional.
It will take no more than 10 minutes to complete.

Anton Shterenlikht, BCS Fortran Standards Officer





-
___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
http://www.beowulf.org/mailman/listinfo/beowulf

Re: [Beowulf] Lustre Upgrades

[Jörg Saßmannshausen]

Hi John,

good idea! Specially as the ship has been restaurated and is in my 
neighbourhood. The only flip side here might be that some tourists might not 
like the idea and it might be a wee bit difficult to get it back into the 
Thames. :-)

All the best

Jörg

Am Freitag, 27. Juli 2018, 10:01:53 BST schrieb John Hearns:
> Jörg,  then the days of the Tea Clipper Races should be revived. We have
> just the ship for it already. Powered by green energy, and built in
> Scotland of course.
> https://en.wikipedia.org/wiki/Cutty_Sark
> 
> Just fill her hold with hard drives and set sail. Aaar me hearties.
> I can just see HPC types being made to climb the rigging in a gale...

___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
http://www.beowulf.org/mailman/listinfo/beowulf

Re: [Beowulf] Lustre Upgrades

[Jörg Saßmannshausen]

Hi all,

Jim: the flip side of the cable is: once it is installed you still can use it, 
whereas with the snow mobile you have to pay for every use.

So in the long run the cable is cheaper, specially as we do need fast 
connection for scientific purposes. 

I was at a talk here in London not so long ago when they were talking about 
data transfer of the very large telescope. As that is generating a huge amount 
of data a week, say, a snow mobile would simply not be practical here. 
Besides, the data is generated on  literally the other side of the world. 

All the best from a hot, sunny London

Jörg


Am Freitag, 27. Juli 2018, 00:47:18 BST schrieb Lux, Jim (337K):
> A quick calculation shows that the bandwidth is on the order of single digit
> Tbps, depending on the link length and road conditions.  Pasadena to Ann
> Arbor works out to 7.6 Tbps on I-80
> 
> If they charge by fractional months - it's about a 33 hour drive, so call
> that 1/15th of a month.  So about $35k to do the transport. Significantly
> cheaper than 4000 km of fiber, coax, or cat 5 cable.
> 
> 
> Jim Lux
> (818)354-2075 (office)
> (818)395-2714 (cell)
> 
> 
> -Original Message-
> From: Beowulf [mailto:beowulf-boun...@beowulf.org] On Behalf Of Fred
> Youhanaie Sent: Thursday, July 26, 2018 5:10 PM
> To: beowulf@beowulf.org
> Subject: Re: [Beowulf] Lustre Upgrades
> 
> Yep, this could be considered as a form of COTS high volume data transfer
> ;-)
> 
> from https://aws.amazon.com/snowmobile/faqs/ (the very last item)
> 
> "Q: How much does a Snowmobile job cost?
> 
> "Snowmobile provides a practical solution to exabyte-scale data migration
> and is significantly faster and cheaper than any network-based solutions,
> which can take decades and millions of dollars of investment in networking
> and logistics. Snowmobile jobs cost $0.005/GB/month based on the amount of
> provisioned Snowmobile storage capacity and the end to end duration of the
> job, which starts when a Snowmobile departs an AWS data center for delivery
> to the time when data ingestion into AWS is complete. Please see AWS
> Snowmobile pricing or contact AWS Sales for an evaluation."
> 
> So it seems a fully loaded snowmobile, 100PB at 0.005/GB/month, would cost
> $524,288.00/month!
> 
> Cheers,
> Fred.
> 
> On 26/07/18 21:49, Lux, Jim (337K) wrote:
> > SO this is the modern equivalent of "nothing beats the bandwidth of a
> > station wagon full of mag tapes" It *is* a clever idea - I'm sure all the
> > big cloud providers have figured out how to do a "data center in shipping
> > container", and that's basically what this is.
> > 
> > I wonder what it costs (yeah, I know I can "Contact Sales to order a
> > AWS Snowmobile"... but...)
> > 
> > 
> > Jim Lux
> > (818)354-2075 (office)
> > (818)395-2714 (cell)
> > 
> > -Original Message-
> > From: Beowulf [mailto:beowulf-boun...@beowulf.org] On Behalf Of Fred
> > Youhanaie
> > Sent: Tuesday, July 24, 2018 11:21 AM
> > To: beowulf@beowulf.org
> > Subject: Re: [Beowulf] Lustre Upgrades
> > 
> > Nah, that ain't large scale ;-) If you want large scale have a look at 
snowmobile:
> > https://aws.amazon.com/snowmobile/
> > 
> > They drive a 45-foot truck to your data centre, fill it up with your
> > data bits, then drive it back to their data centre :-()
> > 
> > Cheers,
> > Fred
> > 
> > On 24/07/18 19:04, Jonathan Engwall wrote:
> >> Snowball is the very large scale AWS data service.
> >> 
> >> 
> >> On July 24, 2018, at 8:35 AM, Joe Landman  wrote:
> >> 
> >> On 07/24/2018 11:06 AM, John Hearns via Beowulf wrote:
> >>> Joe, sorry to split the thread here. I like BeeGFS and have set it up.
> >>> I have worked for two companies now who have sites around the world,
> >>> those sites being independent research units. But HPC facilities are
> >>> in headquarters.
> >>> The sites want to be able to drop files onto local storage yet have
> >>> it magically appear on HPC storage, and same with the results going
> >>> back the other way.
> >>> 
> >>> One company did this well with GPFS and AFM volumes.
> >>> For the current company, I looked at gluster and Gluster
> >>> geo-replication is one way only.
> >>> What do you know of the BeeGFS mirroring? Will it work over long
> >>> distances? (Note to me - find out yourself you lazy besom)
> >> 
> >> This isn't the use case for most/all cluster file systems.   This is
> >> where distributed object systems and buckets rule.
> >> 
> >> Take your file, dump it into an S3 like bucket on one end, pull it
> >> out of the S3 like bucket on the other.  If you don't want to use
> >> get/put operations, then use s3fs/s3ql.  You can back this up with
> >> replicating EC minio stores (will take a few minutes to set up ...
> >> compare that to others).
> >> 
> >> The down side to this is that minio has limits of about 16TiB last I
> >> checked.   If you need more, replace minio with another system
> >> (igneous, ceph, etc.).  Ping me offline if you want to talk more.
> >> 
> >> [...]
> 

Re: [Beowulf] Lustre Upgrades

[Jörg Saßmannshausen]

Hi John,

thanks. I should have said that this was one of the reasons I became 
interested in BeeGFS and this experience is some years ago. I believe at the 
time I was not aware of BeeGFS.
In any case, that was at the old workplace and at the current one we don't 
have these demands on the hardware. 

All the best

Jörg

Am Donnerstag, 26. Juli 2018, 09:53:35 BST schrieb John Hearns:
> Jorg,
> you should look at BeeGFS and BeeOnDemand  https://www.beegfs.io/wiki/BeeOND
> 
> On Thu, 26 Jul 2018 at 09:15, Jörg Saßmannshausen <
> 
> sassy-w...@sassy.formativ.net> wrote:
> > Dear all,
> > 
> > I once had this idea as well: using the spinning discs which I have in the
> > compute nodes as part of a distributed scratch space. I was using
> > glusterfs
> > for that as I thought it might be a good idea. It was not. The reason
> > behind
> > it is that as soon as a job is creating say 700 GB of scratch data (real
> > job
> > not some fictional one!), the performance of the node which is hosting
> > part of
> > that data approaches zero due to the high disc IO. This meant that the job
> > which was running there was affected. So in the end this led to an
> > installation which got a separate file server for the scratch space.
> > I also should add that this was a rather small setup of 8 nodes and it was
> > a
> > few years back.
> > The problem I found in computational chemistry is that some jobs require
> > either large amount of memory, i.e. significantly more than the usual 2 GB
> > per
> > core, or large amount of scratch space (if there is insufficient memory).
> > You
> > are in trouble if it requires both. :-)
> > 
> > All the best from a still hot London
> > 
> > Jörg
> > 
> > Am Dienstag, 24. Juli 2018, 17:02:43 BST schrieb John Hearns via Beowulf:
> > > Paul, thanks for the reply.
> > > I would like to ask, if I may. I rather like Glustre, but have not
> > 
> > deployed
> > 
> > > it in HPC. I have heard a few people comment about Gluster not working
> > 
> > well
> > 
> > > in HPC. Would you be willing to be more specific?
> > > 
> > > One research site I talked to did the classic 'converged infrastructure'
> > > idea of attaching storage drives to their compute nodes and distributing
> > > Glustre storage. They were not happy with that IW as told, and I can
> > > very
> > > much understand why. But Gluster on dedicated servers I would be
> > 
> > interested
> > 
> > > to hear about.
> > > 
> > > On Tue, 24 Jul 2018 at 16:41, Paul Edmon  wrote:
> > > > While I agree with you in principle, one also has to deal with the
> > 
> > reality
> > 
> > > > as you find yourself in.  In our case we have more experience with
> > 
> > Lustre
> > 
> > > > than Ceph in an HPC and we got burned pretty badly by Gluster.  While
> > > > I
> > > > like Ceph in principle I haven't seen it do what Lustre can do in a
> > > > HPC
> > > > setting over IB.  Now it may be able to do that, which is great.
> > 
> > However
> > 
> > > > then you have to get your system set up to do that and prove that it
> > 
> > can.
> > 
> > > > After all users have a funny way of breaking things that work
> > > > amazingly
> > > > well in controlled test environs, especially when you have no control
> > 
> > how
> > 
> > > > they will actually use the system (as in a research environment).
> > > > Certainly we are working on exploring this option too as it would be
> > > > awesome and save many headaches.
> > > > 
> > > > Anyways no worries about you being a smartarse, it is a valid point.
> > 
> > One
> > 
> > > > just needs to consider the realities on the ground in ones own
> > > > environment.
> > > > 
> > > > -Paul Edmon-
> > > > 
> > > > On 07/24/2018 10:31 AM, John Hearns via Beowulf wrote:
> > > > 
> > > > Forgive me for saying this, but the philosophy for software defined
> > > > storage such as CEPH and Gluster is that forklift style upgrades
> > > > should
> > > > not
> > > > be necessary.
> > > > When a storage server is to be retired the data is copied onto the new
> > > > server then the old one taken out of service. Well, copied is not the
> > > > correct word, as there are erasure-coded copies

Re: [Beowulf] Lustre Upgrades

[Jörg Saßmannshausen]

Dear all,

I once had this idea as well: using the spinning discs which I have in the 
compute nodes as part of a distributed scratch space. I was using glusterfs 
for that as I thought it might be a good idea. It was not. The reason behind 
it is that as soon as a job is creating say 700 GB of scratch data (real job 
not some fictional one!), the performance of the node which is hosting part of 
that data approaches zero due to the high disc IO. This meant that the job 
which was running there was affected. So in the end this led to an 
installation which got a separate file server for the scratch space. 
I also should add that this was a rather small setup of 8 nodes and it was a 
few years back. 
The problem I found in computational chemistry is that some jobs require 
either large amount of memory, i.e. significantly more than the usual 2 GB per 
core, or large amount of scratch space (if there is insufficient memory). You 
are in trouble if it requires both. :-)

All the best from a still hot London

Jörg

Am Dienstag, 24. Juli 2018, 17:02:43 BST schrieb John Hearns via Beowulf:
> Paul, thanks for the reply.
> I would like to ask, if I may. I rather like Glustre, but have not deployed
> it in HPC. I have heard a few people comment about Gluster not working well
> in HPC. Would you be willing to be more specific?
> 
> One research site I talked to did the classic 'converged infrastructure'
> idea of attaching storage drives to their compute nodes and distributing
> Glustre storage. They were not happy with that IW as told, and I can very
> much understand why. But Gluster on dedicated servers I would be interested
> to hear about.
> 
> On Tue, 24 Jul 2018 at 16:41, Paul Edmon  wrote:
> > While I agree with you in principle, one also has to deal with the reality
> > as you find yourself in.  In our case we have more experience with Lustre
> > than Ceph in an HPC and we got burned pretty badly by Gluster.  While I
> > like Ceph in principle I haven't seen it do what Lustre can do in a HPC
> > setting over IB.  Now it may be able to do that, which is great.  However
> > then you have to get your system set up to do that and prove that it can.
> > After all users have a funny way of breaking things that work amazingly
> > well in controlled test environs, especially when you have no control how
> > they will actually use the system (as in a research environment).
> > Certainly we are working on exploring this option too as it would be
> > awesome and save many headaches.
> > 
> > Anyways no worries about you being a smartarse, it is a valid point.  One
> > just needs to consider the realities on the ground in ones own
> > environment.
> > 
> > -Paul Edmon-
> > 
> > On 07/24/2018 10:31 AM, John Hearns via Beowulf wrote:
> > 
> > Forgive me for saying this, but the philosophy for software defined
> > storage such as CEPH and Gluster is that forklift style upgrades should
> > not
> > be necessary.
> > When a storage server is to be retired the data is copied onto the new
> > server then the old one taken out of service. Well, copied is not the
> > correct word, as there are erasure-coded copies of the data. Rebalanced is
> > probaby a better word.
> > 
> > Sorry if I am seeming to be a smartarse. I have gone through the pain of
> > forklift style upgrades in the past when storage arrays reach End of Life.
> > I just really like the Software Defined Storage mantra - no component
> > should be a point of failure.
> > 
> > 
> > ___
> > Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
> > To change your subscription (digest mode or unsubscribe) visit
> > http://www.beowulf.org/mailman/listinfo/beowulf
> > 
> > 
> > ___
> > Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
> > To change your subscription (digest mode or unsubscribe) visit
> > http://www.beowulf.org/mailman/listinfo/beowulf

___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
http://www.beowulf.org/mailman/listinfo/beowulf

Re: [Beowulf] Lustre Upgrades

[Jörg Saßmannshausen]

Hi Paul,

with a file system being 93% full, in my humble opinion it would make sense to 
increase the underlying hardware capacity as well. The reasoning behind it is 
that usually over time there will be more data on any given file system and 
thus if there is already a downtime, I would increase the size of it as well. 
I rather have a bit of a longer down time and then I got a new version of 
Luster (of which I know little about) and more capacity which will last for 
longer than only upgrading Luster and then run out of disc capacity a bit 
later. 
It also means that in your case you could simply install the new system, test 
it, and then migrate the data over. Depending how it is set up you even could 
do that at stages. 
As you mentioned 3 different Luster servers, you could for example start with 
the biggest one and use new hardware here. The freed capacity of the now 
obsolete hardware could for example being utilized for the other systems. 
Of course, I don't know your hardware etc. 

Just some ideas from a hot London 8-)

Jörg


Am Montag, 23. Juli 2018, 14:11:40 BST schrieb Paul Edmon:
> Yeah we've pinged Intel/Whamcloud to find out upgrade paths as we wanted
> to know what the recommended procedure is.
> 
> Sure. So we have 3 systems that we want to upgrade 1 that is a PB and 2
> that are 5 PB each.  I will just give you a description of one and
> assume that everything would scale linearly with size. They all have the
> same hardware.
> 
> The head nodes are Dell R620's while the shelves are M3420 (mds) and
> M3260 (oss).  The MDT is 2.2T with 466G used and 268M inodes used.  Each
> OST is 30T with each OSS hosting 6.  The filesystem itself is 93% full.
> 
> -Paul Edmon-
> 
> On 07/23/2018 01:58 PM, Jeff Johnson wrote:
> > Paul,
> > 
> > How big are your ldiskfs volumes? What type of underlying hardware are
> > they? Running e2fsck (ldiskfs aware) is wise and can be done in
> > parallel. It could be within a couple of days, the time all depends on
> > the size and underlying hardware.
> > 
> > Going from 2.5.34 to 2.10.4 is a significant jump. I would be sure
> > there isn't a step upgrade advised. I know there has been step
> > upgrades in the past, not sure about going to/from these two versions.
> > 
> > --Jeff
> > 
> > On Mon, Jul 23, 2018 at 10:34 AM, Paul Edmon  > 
> > > wrote:
> > Yeah we've found out firsthand that its problematic as we have
> > been seeing issues :).  Hence the urge to upgrade.
> > 
> > We've begun exploring this but we wanted to reach out to other
> > people who may have gone through the same thing to get their
> > thoughts.  We also need to figure out how significant an outage
> > this will be.  As if it takes a day or two of full outage to do
> > the upgrade that is more acceptable than a week.  We also wanted
> > to know if people had experienced data loss/corruption in the
> > process and any other kinks.
> > 
> > We were planning on playing around on VM's to test the upgrade
> > path before committing to upgrading our larger systems.  One of
> > the questions we had though was if we needed to run e2fsck
> > before/after the upgrade as that could add significant time to the
> > outage for that to complete.
> > 
> > -Paul Edmon-
> > 
> > On 07/23/2018 01:18 PM, Jeff Johnson wrote:
> >> You're running 2.10.4 clients against 2.5.34 servers? I believe
> >> there are notable lnet attrs that don't exist in 2.5.34. Maybe a
> >> Whamcloud wiz might chime in but I think that version mismatch
> >> might be problematic.
> >> 
> >> You can do a testbed upgrade to test taking a ldiskfs volume from
> >> 2.5.34 to 2.10.4, just to be conservative.
> >> 
> >> --Jeff
> >> 
> >> 
> >> On Mon, Jul 23, 2018 at 10:05 AM, Paul Edmon
> >> 
> >> mailto:ped...@cfa.harvard.edu>> wrote:
> >> My apologies I meant 2.5.34 not 2.6.34. We'd like to get up
> >> to 2.10.4 which is what our clients are running.  Recently we
> >> upgraded our cluster to CentOS7 which necessitated the client
> >> upgrade.  Our storage servers though stayed behind on 2.5.34.
> >> 
> >> -Paul Edmon-
> >> 
> >> On 07/23/2018 01:00 PM, Jeff Johnson wrote:
> >>> Paul,
> >>> 
> >>> 2.6.34 is a kernel version. What version of Lustre are you
> >>> at now? Some updates are easier than others.
> >>> 
> >>> --Jeff
> >>> 
> >>> On Mon, Jul 23, 2018 at 8:59 AM, Paul Edmon
> >>> 
> >>> mailto:ped...@cfa.harvard.edu>> wrote:
> >>> We have some old large scale Lustre installs that are
> >>> running 2.6.34 and we want to get these up to the latest
> >>> version of Lustre.  I was curious if people in this
> >>> group have any experience with doing this and if they
> >>>   

Re: [Beowulf] memory usage

[Jörg Saßmannshausen]

Hi Michael,

thanks. 

Sorry I am a bit slow with replying right now.

All the best

Jörg

Am Montag, 25. Juni 2018, 08:35:09 BST schrieb Michael Di Domenico:
> On Sun, Jun 24, 2018 at 7:14 AM, Jörg Saßmannshausen
> 
>  wrote:
> > out of curiosity: why do you want to / need to know which in numa domain
> > the allocated memory is sitting?
> > I simply try to understand what you are after here.
> 
> I'm trying to prove that when a process is locked to a domain via some
> mechanism, either numactl, mpi binding, etc, that it is in fact only
> allocating memory on that domain.  It's a trust but verify thing.
> ___
> Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf

___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
http://www.beowulf.org/mailman/listinfo/beowulf

Re: [Beowulf] memory usage

[Jörg Saßmannshausen]

Hi Michael,

out of curiosity: why do you want to / need to know which in numa domain the 
allocated memory is sitting?
I simply try to understand what you are after here.

All the best from a sunny London!

Jörg

Am Freitag, 22. Juni 2018, 13:58:24 BST schrieb Michael Di Domenico:
> does anyone know of a tool that looks at a process
> (single/multi-threaded) and tells you how much memory it's using and
> in which numa domain the allocated memory is sitting.
> ___
> Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf

___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
http://www.beowulf.org/mailman/listinfo/beowulf

Re: [Beowulf] Intel motherboard BMC

[Jörg Saßmannshausen]

Hi,

I am not sure why, if it is a java-script problem, it would work with the 
older Supermicro nodes but not with the new Intel ones. 

I got more and more the feeling it is probably a simple configuration somewhere 
but we are looking at the wrong place here.

All the best

Jörg

Am Donnerstag, 21. Juni 2018, 08:19:00 BST schrieb Jonathan Engwall:
> This sounds so much like a common JavaScript problem. I thought this might
> be valuable here:
> https://www.gnu.org/software/freeipmi/freeipmi-hostrange.txt
> When test a JavaScript file the browser will shortcut to running the file
> not the browser itself via a "server" resulting in s omething like this:
> ///localhost/3000/myNewApp
> For what it's worth.
> All meaning according to the rest of your shop the cluster is dead.
> 
> On Jun 21, 2018 4:49 AM, "John Hearns via Beowulf" 
> 
> wrote:
> > Jorg, the notes I have for setting up Intel BMCs are not of any use to
> > you, sorry. See below.
> > 
> > Regarding the PXE booting, t is easy to use the syscfg utility to print
> > out and to set the boot order.
> > But I do not think this is your problem.
> > https://www.intel.com/content/dam/support/us/en/documents/
> > server-products/server-boards/Intel_Syscfg_UserGuide_V1.02.pdf
> > 
> > The IPMI interfaces have three LAN Channels – 1, 2, 3
> > 
> > LAN Channel 3 is the one used by the AXXRMM4LITE add-on management module
> > 
> > http://ark.intel.com/products/55168/Remote-Management-Module-AXXRMM4LITE
> > 
> > To configure LAN channel 3:
> > 
> > *module load ipmitool *
> > 
> > *ipmitool lan print 1* (to show the configuration of Channel 1)
> > 
> > *ipmitool lan set 3 ipaddr 172.16.202.XXX * (to set the IP address the
> > same as Channel 1)
> > 
> > *ipmitool lan set 3 netmask 255.255.255.0*
> > 
> > *ipmitool lan print 1* (to show the configuration of Channel 1)
> > 
> > 
> > The Bright Cluster manager uses user number 4 to access the IPMI
> > interface. This user is named ‘bright’. You should set the level of access
> > for user 4 on LAN Channel 3:
> > 
> > *module load ipmitool ; ipmitool user priv 4 0x4 3*
> > 
> > *module load ipmitool ; ipmitool channel setaccess 3 4 callin=on ipmi=on
> > link=on privilege=4*
> > 
> > *ipmitool channel getaccess 3 4*
> > 
> > On 21 June 2018 at 12:25, Jörg Saßmannshausen <
> > 
> > sassy-w...@sassy.formativ.net> wrote:
> >> Hi John,
> >> 
> >> thanks for the reply. I am aware you can install user and admin level,
> >> erm,
> >> users on the BMC. I only install admin-level users as there is only a
> >> need for
> >> an admin to access the BMC GUI.
> >> However, that does not explain why the IP address is working and the
> >> hostname
> >> is not.
> >> 
> >> The only way to get the users installed was using the ipmitool command
> >> once
> >> the node was up and running. I tried in the BIOS but that was not
> >> working. I
> >> got told you will need to enable the 'root' user first before you can
> >> install
> >> users but in the BIOS there was no option for that or we did not find it.
> >> 
> >> I guess it is the usual learning initial curve here for me. I actually
> >> did
> >> look into the documentation but I could not find anything we were doing
> >> wrong
> >> at the time. Quite often I found it is a really minor, but important bit
> >> one
> >> is missing out and later you think: why did I miss that.
> >> 
> >> Thanks for your suggestions!
> >> 
> >> Jörg
> >> 
> >> Am Donnerstag, 21. Juni 2018, 12:07:48 BST schrieb John Hearns via
> >> 
> >> Beowulf:
> >> > Jorg, recalling my experience with Intel. I did not come across the
> >> 
> >> problem
> >> 
> >> > with IP address versus Hostname which you have.
> >> > However I do recall that I had to configure the Admin user and the
> >> > privilege level for that user on the LAN interface. In that case the
> >> > additional BMC modules were being used.
> >> > 
> >> > I might have the commands written up somewhere.
> >> > 
> >> > On 21 June 2018 at 12:02, John Hearns  wrote:
> >> > > Hello Jorg. As you know I have worked a lot with Supermicro machines.
> >> > > I also installed Intel machines for Greenwich University, so I have
> >> > > experience of setting up IPMI

Re: [Beowulf] Intel motherboard BMC

[Jörg Saßmannshausen]

Hi John,

interesting idea, but how does it work when the IP address has not been set 
yet?

Regarding hostname/IP for the GUI: see my email to Chris.

Thanks!

Jörg

Am Donnerstag, 21. Juni 2018, 12:55:07 BST schrieben Sie:
> It is worth saying that Intel have an excellent free to download tool
> called syscfg which lets you set BIOS and IPMI from the command line
> To get BIOS settings the same on all nodes on a cluster you just cget the
> correct settings on one node, copy this file to all nodes,
> syscfg then reboot. Simples.
> 
> https://downloadcenter.intel.com/download/26971/Save-and-Restore-System-Conf
> iguration-Utility-syscfg-
> 
> 
> Regarding your web server problem, is your DHCP server giving the correct
> hostnames to the IPMI cards when they request an address?
> I do not see how that could matter, but maybe?
> 
> 
> 
> On 21 June 2018 at 12:29, Jörg Saßmannshausen  > wrote:
> > 
> > Hi John,
> > 
> > further the last email: I am using the onboard interface for the BMC/IPMI
> > (shared link) so there is only one physical network cable between the
> > switch
> > and the compute node.
> > 
> > Regarding PXE boot: I don't want to boot over IB. Right now I even don't
> > have
> > an IB network. I was just wondering whether for some reason at one stage
> > of
> > the boot process the kernel recognises the IB card and then tries to boot
> > from
> > there and shuts off the other NIC which could explain that behaviour.
> > However, even disabling it in the BIOS did not solve the problem.
> > 
> > I guess I will need to do some debugging here but without some good ideas
> > I am
> > a bit stuck as I already have tried the usual suspects.
> > 
> > All the best
> > 
> > Jörg
> > 
> > Am Donnerstag, 21. Juni 2018, 12:02:40 BST schrieb John Hearns via 
Beowulf:
> > > Hello Jorg. As you know I have worked a lot with Supermicro machines.
> > > I also installed Intel machines for Greenwich University, so I have
> > > experience of setting up IPMI on them.
> > > I will take time to try to understand your problem!
> > > Also Intel provides excellent documentation for all its products.
> > > Really.
> > > But you must get the correct part number and search for it.
> > > I really recommend finding the BMC manual, as I recall that made things
> > > a
> > > lot clearer.
> > > 
> > > One quick question - are you using the on-board ethernet interface for
> > 
> > IPMI
> > 
> > > or are you using the additional hardware module which has its own
> > 
> > ethernet
> > 
> > > port?
> > > 
> > > > It also has a InfiniBand card which does allow booting from it.
> > > 
> > > You can PXE boot over a Mellanox Infiniband card. As you probably know
> > 
> > this
> > 
> > > involves installing extra firmware on the card.
> > > In my last job we had an IB only cluster, so booting over IB had to
> > > work!
> > > I guess you do nto need to flash the card, but to be honest running the
> > > utility is not scary. You just have to get the exact firmware for your
> > 
> > card.
> > 
> > > On 21 June 2018 at 11:20, Tony Brian Albers  wrote:
> > > > Does the BMC itself know its own hostname?
> > > > 
> > > > /tony
> > > > 
> > > > On 21/06/18 11:13, Jörg Saßmannshausen wrote:
> > > > > Dear all,
> > > > > 
> > > > > I got a bit of a confusing situation with the BMC of some Intel
> > > > 
> > > > motherboards
> > > > 
> > > > > which we recently purchased and I am not quite sure what to make out
> > 
> > of
> > 
> > > > it.
> > > > 
> > > > > We have install a generic user via the IPMI commands on the compute
> > > > 
> > > > nodes and
> > > > 
> > > > > I can access the BMC remotely, again via the IPMI command like this:
> > > > > 
> > > > > $ ipmitool -H node105-bmc -U username -P xxx power status
> > > > > 
> > > > > This is working, Also, this works:
> > > > > 
> > > > > $ ipmitool -H 10.0.1.105 -U username -P xxx power status
> > > > > 
> > > > > A nslookup of node105-bmc gives the right IP address as well.
> > > > > 
> > > > > However, if I want to use the GUI for the BMC, i.e. opening my
> > 
> > browser
> > 
&

Re: [Beowulf] Intel motherboard BMC

[Jörg Saßmannshausen]

Hi Chris,

ok, you can change the 'hostname' of the BMC in the BIOS, but not in the GUI, 
which I found interesting but here you go. 
I changed it and I still got the same issue: I can login using the IP address 
but not the hostname. Here login, with the correct credentials, will be 
refused.
Also, on the host which is running firefox, the hostname and its IP address are 
in /etc/hosts and both the forward and backward IP lookup work correctly. 

So it was a good idea, but does not solve the problem.

Thanks for your help!

Jörg

Am Donnerstag, 21. Juni 2018, 20:26:42 BST schrieb Chris Samuel:
> On Thursday, 21 June 2018 8:17:15 PM AEST Jörg Saßmannshausen wrote:
> > However, given I can go to the login page and given I then get rejected by
> > the webserver and redirected to another page, I am not quite sure how that
> > could affect my ability to log in. Something I am missing here?
> 
> A web browser will send an HTTP "Host:" header for the website it is looking
> for, so perhaps the web app on the BMC will only honour logins where that
> matches what it thinks its hostname is (or its IP address).
> 
> You could test that by adding the hostname it thinks it has to your
> /etc/hosts and then using that hostname to access it.   I'd be a little
> surprised if it worked, but not too much given the general quality of
> BMCs...
> 
> cheers,
> Chris

___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
http://www.beowulf.org/mailman/listinfo/beowulf

Re: [Beowulf] Intel motherboard BMC

[Jörg Saßmannshausen]

Hi John,

thanks for the reply. I am aware you can install user and admin level, erm, 
users on the BMC. I only install admin-level users as there is only a need for 
an admin to access the BMC GUI. 
However, that does not explain why the IP address is working and the hostname 
is not. 

The only way to get the users installed was using the ipmitool command once 
the node was up and running. I tried in the BIOS but that was not working. I 
got told you will need to enable the 'root' user first before you can install 
users but in the BIOS there was no option for that or we did not find it. 

I guess it is the usual learning initial curve here for me. I actually did 
look into the documentation but I could not find anything we were doing wrong 
at the time. Quite often I found it is a really minor, but important bit one 
is missing out and later you think: why did I miss that.

Thanks for your suggestions!

Jörg

Am Donnerstag, 21. Juni 2018, 12:07:48 BST schrieb John Hearns via Beowulf:
> Jorg, recalling my experience with Intel. I did not come across the problem
> with IP address versus Hostname which you have.
> However I do recall that I had to configure the Admin user and the
> privilege level for that user on the LAN interface. In that case the
> additional BMC modules were being used.
> 
> I might have the commands written up somewhere.
> 
> On 21 June 2018 at 12:02, John Hearns  wrote:
> > Hello Jorg. As you know I have worked a lot with Supermicro machines.
> > I also installed Intel machines for Greenwich University, so I have
> > experience of setting up IPMI on them.
> > I will take time to try to understand your problem!
> > Also Intel provides excellent documentation for all its products. Really.
> > But you must get the correct part number and search for it.
> > I really recommend finding the BMC manual, as I recall that made things a
> > lot clearer.
> > 
> > One quick question - are you using the on-board ethernet interface for
> > IPMI or are you using the additional hardware module which has its own
> > ethernet port?
> > 
> > > It also has a InfiniBand card which does allow booting from it.
> > 
> > You can PXE boot over a Mellanox Infiniband card. As you probably know
> > this involves installing extra firmware on the card.
> > In my last job we had an IB only cluster, so booting over IB had to work!
> > I guess you do nto need to flash the card, but to be honest running the
> > utility is not scary. You just have to get the exact firmware for your
> > card.> 
> > On 21 June 2018 at 11:20, Tony Brian Albers  wrote:
> >> Does the BMC itself know its own hostname?
> >> 
> >> /tony
> >> 
> >> On 21/06/18 11:13, Jörg Saßmannshausen wrote:
> >> > Dear all,
> >> > 
> >> > I got a bit of a confusing situation with the BMC of some Intel
> >> 
> >> motherboards
> >> 
> >> > which we recently purchased and I am not quite sure what to make out of
> >> 
> >> it.
> >> 
> >> > We have install a generic user via the IPMI commands on the compute
> >> 
> >> nodes and
> >> 
> >> > I can access the BMC remotely, again via the IPMI command like this:
> >> > 
> >> > $ ipmitool -H node105-bmc -U username -P xxx power status
> >> > 
> >> > This is working, Also, this works:
> >> > 
> >> > $ ipmitool -H 10.0.1.105 -U username -P xxx power status
> >> > 
> >> > A nslookup of node105-bmc gives the right IP address as well.
> >> > 
> >> > However, if I want to use the GUI for the BMC, i.e. opening my browser
> >> 
> >> and
> >> 
> >> > put:
> >> > 
> >> > https://node105-bmc
> >> > 
> >> > in the URL, I get the loging page When I enter my login credentials
> >> 
> >> then,
> >> 
> >> > which are the same as above, I have a problem to log in *IF* I am using
> >> 
> >> the
> >> 
> >> > hostname as address but not *IF* I am using the IP address. Just to add
> >> 
> >> to the
> >> 
> >> > confusion more, on one node the hostname was working.
> >> > With problems I mean the browser tells me my login credentials are
> >> 
> >> wrong which
> >> 
> >> > does not happen when I am using the IP address.
> >> > Also, I can only use https and not http and for now I got the generic
> >> 
> >> self
> >> 
> >> > signed certificates. I want to change

Re: [Beowulf] Intel motherboard BMC

[Jörg Saßmannshausen]

Hi Tony,

nope. The GUI gives me a hostname, something like BMC23453234 and it is 
greyed-out so I cannot change it. The string after BMC is basically the MAC 
address without any hyphens or so. 

However, given I can go to the login page and given I then get rejected by the 
webserver and redirected to another page, I am not quite sure how that could 
affect my ability to log in. Something I am missing here?

All the best

Jörg

Am Donnerstag, 21. Juni 2018, 09:20:51 BST schrieb Tony Brian Albers:
> Does the BMC itself know its own hostname?
> 
> /tony
> 
> On 21/06/18 11:13, Jörg Saßmannshausen wrote:
> > Dear all,
> > 
> > I got a bit of a confusing situation with the BMC of some Intel
> > motherboards which we recently purchased and I am not quite sure what to
> > make out of it.
> > 
> > We have install a generic user via the IPMI commands on the compute nodes
> > and I can access the BMC remotely, again via the IPMI command like this:
> > 
> > $ ipmitool -H node105-bmc -U username -P xxx power status
> > 
> > This is working, Also, this works:
> > 
> > $ ipmitool -H 10.0.1.105 -U username -P xxx power status
> > 
> > A nslookup of node105-bmc gives the right IP address as well.
> > 
> > However, if I want to use the GUI for the BMC, i.e. opening my browser and
> > put:
> > 
> > https://node105-bmc
> > 
> > in the URL, I get the loging page When I enter my login credentials then,
> > which are the same as above, I have a problem to log in *IF* I am using
> > the
> > hostname as address but not *IF* I am using the IP address. Just to add to
> > the confusion more, on one node the hostname was working.
> > With problems I mean the browser tells me my login credentials are wrong
> > which does not happen when I am using the IP address.
> > Also, I can only use https and not http and for now I got the generic self
> > signed certificates. I want to change them at one point but right now that
> > is more on the bottom of my to-do list.
> > 
> > I find that really odd and I am not quite sure what is going on here. With
> > all the Supermicro kit I once had I never had these issues before. I was
> > able to log in regardless of using the hostname or IP address.
> > So clearly Intel does something here Supermicro did not (at the time).
> > 
> > The boards in question are Intel S2600BPB ones.
> > 
> > Has anybody seen this before?
> > 
> > I got a second issue with these boards. I usually do the normal PXE/NFS
> > boot and the setup is working well for the other, older Supermicro
> > machines. However, with the new Intel ones, this is crashing.
> > The procedure is you are selecting in the boot-menu you want to do a PXE
> > boot and not boot from the local hard drive.
> > It then boots the initramfs which seems to be fine. From what I can see,
> > both during the boot process and from the log files of the DHCP-server,
> > it is getting the right IP address.
> > However, when the initramfs hands over to the kernel, it crashes with:
> > kernel panic! attempt to kill init
> > and you literally have to pull the plug on the machine, i.e. a hard reset.
> > 
> > The only time I have seen that was when I did not specify the NIC and when
> > I had two NICs, it somehow decided to use the other one. I fixed that
> > problem by defining the interface in the boot-arguments and also the
> > second NIC is not connected anyway. It also has a InfiniBand card which
> > does allow booting from it. Again, it is not connected so in theory it
> > should not matter.
> > 
> > I am stuck here. I am using a 4.x kernel for the PXE boot, so a fairly
> > recent one. As I said, it works for the older machines but not for the
> > newer ones.> 
> >   I upgraded the whole PXE/NFS boot and that is not working too.
> > 
> > Does anybody have any ideas here?
> > 
> > Sorry for asking 2 questions in one email but as they are related I hope
> > that is ok.
> > 
> > All the best from a sunny London
> > 
> > Jörg
> > 
> > ___
> > Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
> > To change your subscription (digest mode or unsubscribe) visit
> > http://www.beowulf.org/mailman/listinfo/beowulf

___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
http://www.beowulf.org/mailman/listinfo/beowulf

[Beowulf] Intel motherboard BMC

[Jörg Saßmannshausen]

Dear all,

I got a bit of a confusing situation with the BMC of some Intel motherboards 
which we recently purchased and I am not quite sure what to make out of it. 

We have install a generic user via the IPMI commands on the compute nodes and 
I can access the BMC remotely, again via the IPMI command like this:

$ ipmitool -H node105-bmc -U username -P xxx power status

This is working, Also, this works:

$ ipmitool -H 10.0.1.105 -U username -P xxx power status

A nslookup of node105-bmc gives the right IP address as well. 

However, if I want to use the GUI for the BMC, i.e. opening my browser and 
put:

https://node105-bmc

in the URL, I get the loging page When I enter my login credentials then, 
which are the same as above, I have a problem to log in *IF* I am using the 
hostname as address but not *IF* I am using the IP address. Just to add to the 
confusion more, on one node the hostname was working. 
With problems I mean the browser tells me my login credentials are wrong which 
does not happen when I am using the IP address. 
Also, I can only use https and not http and for now I got the generic self 
signed certificates. I want to change them at one point but right now that is 
more on the bottom of my to-do list.

I find that really odd and I am not quite sure what is going on here. With all 
the Supermicro kit I once had I never had these issues before. I was able to 
log in regardless of using the hostname or IP address. 
So clearly Intel does something here Supermicro did not (at the time).

The boards in question are Intel S2600BPB ones. 

Has anybody seen this before? 

I got a second issue with these boards. I usually do the normal PXE/NFS boot 
and the setup is working well for the other, older Supermicro machines. 
However, with the new Intel ones, this is crashing. 
The procedure is you are selecting in the boot-menu you want to do a PXE boot 
and not boot from the local hard drive. 
It then boots the initramfs which seems to be fine. From what I can see, both 
during the boot process and from the log files of the DHCP-server, it is 
getting the right IP address. 
However, when the initramfs hands over to the kernel, it crashes with:
kernel panic! attempt to kill init
and you literally have to pull the plug on the machine, i.e. a hard reset.

The only time I have seen that was when I did not specify the NIC and when I 
had two NICs, it somehow decided to use the other one. I fixed that problem by 
defining the interface in the boot-arguments and also the second NIC is not 
connected anyway. It also has a InfiniBand card which does allow booting from 
it. Again, it is not connected so in theory it should not matter. 

I am stuck here. I am using a 4.x kernel for the PXE boot, so a fairly recent 
one. As I said, it works for the older machines but not for the newer ones. 

 I upgraded the whole PXE/NFS boot and that is not working too.

Does anybody have any ideas here?

Sorry for asking 2 questions in one email but as they are related I hope that 
is ok.

All the best from a sunny London

Jörg

___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
http://www.beowulf.org/mailman/listinfo/beowulf

Re: [Beowulf] Bright Cluster Manager

[Jörg Saßmannshausen]

Dear Chris,

further to your email:

> - And if miracles occur and they do have expert level linux people then
> more often than not these people are overworked or stretched in many
> directions

This is exactly what has happened to me at the old work place: pulled into too 
many different directions. 

I am a bit surprised about the ZFS experiences. Although I did not have 
petabyte of storage and I did not generate 300 TB per week, I did have a 
fairly large storage space running on xfs and ext4 for backups and 
provisioning of file space. Some of it was running on old hardware (please sit 
down, I am talking about me messing around with SCSI cables) and I gradually 
upgraded to newer one. So, I am not quite sure what went wrong with the ZFS 
storage here. 

However, there is a common trend, at least what I observe here in the UK, to 
out-source problems: pass the bucket to somebody else and we pay for it. 
I am personally still  more of an in-house expert than an out-sourced person 
who may or may not be able to understand what you are doing. 
I should add I am working in academia and I know little about the commercial 
world here. Having said that, my friends in commerce are telling me that the 
company likes to outsource as it is 'cheaper'. 
I agree with the Linux expertise. I think I am one of the two who are Linux 
admins in the present work place. The official line is: we do not support Linux 
(but we teach it). 

Anyhow, I don't want to digress here too much. However, "..do HPC work in 
commercial environments where the skills simply don't exist onsite."
Are we a dying art?

My 1 shilling here from a still cold and dark London.

Jörg



Am Mittwoch, 2. Mai 2018, 16:19:48 BST schrieb Chris Dagdigian:
> Jeff White wrote:
> > I never used Bright.  Touched it and talked to a salesperson at a
> > conference but I wasn't impressed.
> > 
> > Unpopular opinion: I don't see a point in using "cluster managers"
> > unless you have a very tiny cluster and zero Linux experience.  These
> > are just Linux boxes with a couple applications (e.g. Slurm) running
> > on them.  Nothing special. xcat/Warewulf/Scyld/Rocks just get in the
> > way more than they help IMO.  They are mostly crappy wrappers around
> > free software (e.g. ISC's dhcpd) anyway.  When they aren't it's
> > proprietary trash.
> > 
> > I install CentOS nodes and use
> > Salt/Chef/Puppet/Ansible/WhoCares/Whatever to plop down my configs and
> > software.  This also means I'm not suck with "node images" and can
> > instead build everything as plain old text files (read: write
> > SaltStack states), update them at will, and push changes any time.  My
> > "base image" is CentOS and I need no "baby's first cluster" HPC
> > software to install/PXEboot it.  YMMV
> 
> Totally legit opinion and probably not unpopular at all given the user
> mix on this list!
> 
> The issue here is assuming a level of domain expertise with Linux,
> bare-metal provisioning, DevOps and (most importantly) HPC-specific
> configStuff that may be pervasive or easily available in your
> environment but is often not easily available in a
> commercial/industrial  environment where HPC or "scientific computing"
> is just another business area that a large central IT organization must
> support.
> 
> If you have that level of expertise available then the self-managed DIY
> method is best. It's also my preference
> 
> But in the commercial world where HPC is becoming more and more
> important you run into stuff like:
> 
> - Central IT may not actually have anyone on staff who knows Linux (more
> common than you expect; I see this in Pharma/Biotech all the time)
> 
> - The HPC user base is not given budget or resource to self-support
> their own stack because of a drive to centralize IT ops and support
> 
> - And if they do have Linux people on staff they may be novice-level
> people or have zero experience with HPC schedulers, MPI fabric tweaking
> and app needs (the domain stuff)
> 
> - And if miracles occur and they do have expert level linux people then
> more often than not these people are overworked or stretched in many
> directions
> 
> 
> So what happens in these environments is that organizations will
> willingly (and happily) pay commercial pricing and adopt closed-source
> products if they can deliver a measurable reduction in administrative
> burden, operational effort or support burden.
> 
> This is where Bright, Univa etc. all come in -- you can buy stuff from
> them that dramatically reduces that onsite/local IT has to manage the
> care and feeding of.
> 
> Just having a vendor to call for support on Grid Engine oddities makes
> the cost of Univa licensing worthwhile. Just having a vendor like Bright
> be on the hook for "cluster operations" is a huge win for an overworked
> IT staff that does not have linux or HPC specialists on-staff or easily
> available.
> 
> My best example of "paying to reduce operational burden in HPC" comes
> from a massive well known genome 

Re: [Beowulf] Bright Cluster Manager

[Jörg Saßmannshausen]

Dear all,

at least something I can contribute here: at the new work place the small 
cluster I am looking after is using Bright Cluster Manager to manage the 20 
nodes and the 10 or so GPU nodes.

I was not around when it all got installed so I cannot comment on how quickly 
it can be done or how easily. 

I used to do larger installations with up to 112 compute nodes which have 
different physical hardware. So I needed at least 2 images. I done all of that 
with a bit of scripting and not with a GUI. I did not use LDAP and 
authentication was done locally. It all provided a robust system. Maybe not as 
easy to manage as a system which got a GUI which does it all for you but on 
the flip side I knew exactly what the scripts were doing and what I need to do 
if there was a problem. 

By enlarge I agree with what John  Hearns said for example. To be frank: I 
still consider the Bright Cluster Manager tool to be good for people who do 
not know about HPC (I stick to that for this argument), don't know much about 
Linux etc. So in my personal opinion it is good for those who's day-to-day job 
is not HPC but something different. People who are coming from a GUI world (I 
don't mean that nasty here). For situations where it does not reckon to have a 
dedicated support. So for this it is fantastic: it works, there is a good 
support if things go wrong. 
We are using SLURM and the only issue I had when I first started at the new 
place a year ago that during a routine update SLRUM got re-installed and all 
the configurations were gone. This could be as it was not installed properly in 
the first place, it could be a bug, we don't know as the support did not manage 
to reproduce this. 
I am having some other minor issues with the authentication (we are 
authenticating against external AD) but again that could be the way it was 
installed at the time. I don't know who done that. 

Having said all of that: I am personally more a hands-on person so I know what 
the system is doing. This usually gets obscured by a GUI which does things in 
the background you may or may not want it to do. I had some problems at the 
old work place with ROCKS which lead me to removing it and install Debian on 
the clusters. They were working rock solid, even on hardware which had issues 
with the ROCKS installation. 

So, for me the answer to the question is: it depends: If you got a capable HPC 
admin who is well networked and you got a larger, specialized cluster, you 
might be better off to use the money and buy some additional compute nodes. 
For a installation where you do not have a dedicated admin, and you might have 
a smaller, homogeneous installation, you might be better off with a cluster 
management tool light the one Bright is offering. 
If money is an issue, you need to carefully balance the two: a good HPC admin 
does more than installing software, they do user support as well for example 
and make sure users can work. If you are lucky, you get one who actually 
understands what the users are doing. 

I think that is basically what everybody here says in different words: your 
mileage will vary.

My to shillings from a rather cold London! :-)

Jörg

Am Dienstag, 1. Mai 2018, 16:57:40 BST schrieb Robert Taylor:
> Hi Beowulfers.
> Does anyone have any experience with Bright Cluster Manager?
> My boss has been looking into it, so I wanted to tap into the collective
> HPC consciousness and see
> what people think about it.
> It appears to do node management, monitoring, and provisioning, so we would
> still need a job scheduler like lsf, slurm,etc, as well. Is that correct?
> 
> If you have experience with Bright, let me know. Feel free to contact me
> off list or on.

___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
http://www.beowulf.org/mailman/listinfo/beowulf

Re: [Beowulf] Slow RAID reads, no errors logged, why?

[Jörg Saßmannshausen]

Hi Chris,

Am Dienstag, 20. März 2018, 16:44:48 GMT schrieb Chris Samuel:
> On Tuesday, 20 March 2018 11:55:01 AM AEDT Jörg Saßmannshausen wrote:
> > Thus, it could well be that the controller is slowing down the
> > rest of the RAID as there is a bottle neck.
> 
> As David is testing reading from a RAID-5 array then (IIRC) it needs to read
> the stripe from all active drives to retrieve the data, and if one drive is
> slower than the rest then the controller will end up waiting for that
> drive.

Well, that is basically just confirming my suspicion here. :-)

All the best

Jörg

___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
http://www.beowulf.org/mailman/listinfo/beowulf

Re: [Beowulf] Slow RAID reads, no errors logged, why?

[Jörg Saßmannshausen]

Hi David,

I am wondering if that is not what you would expect. 
The long SMART test is quite thorough so the disc will be subject to quite a 
bit of stress. Thus, it could well be that the controller is slowing down the 
rest of the RAID as there is a bottle neck. So from that angle it does make 
sense to me. 
This is, however, just speculation from my side.

All the best from a chilly London

Jörg


Am Montag, 19. März 2018, 16:50:51 GMT schrieb David Mathog:
> Found the problem.  Well, sort of.
> 
> The issue is that when a long SMART test runs on any disk on the system
> (A) which has this problem the IO goes down to 30Mb/s.  It doesn't
> matter which disk is running the test.  The system we have which is most
> like it (C) does not have this issue.
> 
> A   C
> Centos 6.7 6.9
> RAM512 512 Gb
> CPUs   56  40  (actually threads)
> PowerEdge  T630T630
> Xeon   E5-2695 E5-2650 (both v3)
> speed  2.30GHz 2.30Ghz
> cpufreq?   yes no
> PERC   H730H730P
> SAS disk   ST2000NM0023
> SAS disk   ST4000NM0005
> 
> There are a bunch of small differences between the two systems so it is
> hard to say for sure which is the actual culprit.
> 
> I will put this out on the smartmontools list and see if anybody has
> seen it before.
> 
> Regards,
> 
> David Mathog
> mat...@caltech.edu
> Manager, Sequence Analysis Facility, Biology Division, Caltech
> ___
> Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf

___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
http://www.beowulf.org/mailman/listinfo/beowulf

Re: [Beowulf] cluster authentication part II

[Jörg Saßmannshausen]

Hi Jonathan,

thanks for the link. That is basically what I was doing as there are many 
tutorials and how-to pages out there. Your page is quite nicely condensed 
which makes it easy to read. 

All the best

Jörg


Am Mittwoch, 17. Januar 2018, 16:36:57 GMT schrieben Sie:
> I don't want to bore anybody, this might be interesting. My parts are
> almost all in. This is a really great topic.
> https://arthurdejong.org/nss-pam-ldapd/setup
> And with several informative web page a.
> 
> On Jan 17, 2018 3:13 PM, "Jörg Saßmannshausen" <
> 
> sassy-w...@sassy.formativ.net> wrote:
> > Dear all,
> > 
> > thanks for all your useful comments.
> > In the end, and after some debugging, I found the culprit. For one reason
> > or
> > another I installed libpam-ldap instead of libpam-ldapd. I guess that was
> > a
> > typo as libpam-ldapd will be pulled automatically when you are installing
> > nslcd.
> > Once I corrected that, both su -l USER and ssh USER@localhost (or from a
> > remote host to the Ubuntu VDI) are working fast again.
> > Don't ask me what is the difference between the two, I don't know is the
> > short
> > answer here.
> > 
> > One question: when I was doing some research on the internet, I came
> > across
> > nslcd and sssd. Which one is 'better'? I know that is a bit of an
> > ambiguous
> > question to ask but I have not found a page telling me the difference
> > between
> > the two.
> > 
> > Regarding Ubuntu vs. other distros: that is not my choice. Personally I am
> > in
> > favour of Debian but that is my personal choice. At the workplace I have
> > to
> > work with what is their policy. I am not a great fan of having different
> > distributions floating around at one place as it make the administration a
> > nightmare (you will be never good at all of them) but we are where we are
> > here.
> > 
> > Regarding sudo: that is still a problem on one of the servers: it simply
> > does
> > not accept the password. Once I know more here I can report back to you
> > John.
> > 
> > Sorry for my slow response here. I am not looking at the email list when I
> > am
> > at work and thus it takes me a day or two to reply.
> > 
> > All the best from a cold London (storm about to come tonight)
> > 
> > Jörg
> > 
> > Am Mittwoch, 17. Januar 2018, 12:08:37 GMT schrieben Sie:
> > > I would switch to sssd. I had many problems with nslcd (connection,
> > > cache...).
> > > 
> > > Best regards
> > > 
> > > On 16/01/2018 00:35, Jörg Saßmannshausen wrote:
> > > > Dear all,
> > > > 
> > > > reading the Cluster Authentication (LDAP,AD) thread which was posted
> > > > at
> > > > the
> > > > end of last year reminds me of a problem we are having.
> > > > 
> > > > For our Ubuntu 14 virtual machines we are authenticating against AD
> > 
> > and I
> > 
> > > > am using the nslcd daemon to do that.
> > > > This is working very well in a shell, i.e. when I am doing this in a
> > > > shell:
> > > > 
> > > > $ su -l USER
> > > > 
> > > > It is fast, it is creating the home directory if I need it (or not if
> > > > I
> > > > want to mount the file space elsewhere and use a local home) and the
> > > > standard lookup tools like
> > > > 
> > > > $ getent password USER
> > > > 
> > > > are fast as well.
> > > > 
> > > > However, and here is where I am stuck: when I want to log in to the
> > > > machine
> > > > using the GUI, this takes forever. We measures it and it takes up to
> > > > 90
> > > > sec. until it finally works. I also noticed that it is not reading the
> > > > /etc/nslcd.conf file but either /etc/ldap.conf or /etc/ldap/ldap.conf.
> > 
> > The
> > 
> > > > content of the ldap.conf file is identical with the nslcd.conf file. I
> > 
> > am
> > 
> > > > using TLS and not SSL for the secure connection .
> > > > Furthermore, and here I am not sure whether it is the same problem or
> > > > a
> > > > different one, if I want to ssh into the Ubuntu VM, this also take a
> > 
> > very
> > 
> > > > long time (90 sec) until I can do that.
> > > > Strangely enough, our HPC cluster is using nslcd as well (I used that
> > > > nslcd.conf file as a template for 

Re: [Beowulf] cluster authentication part II

[Jörg Saßmannshausen]

Hi Jonathan,

it is. I tried both, wheel and in the sudoers file. It is asking for the 
password which I supply (it is my account), it is asking for the password 
again. The password is correct as I am using it. 
The interesting thing is: I only got this problem on the headnode but not on 
the compute nodes. Here it is working as expected. I *should* be the same 
setup regarding LDAP but obviously it is not. One of my problems is I did not 
install the cluster and I have already found a number of bugs on it. As it is 
a live system I cannot run nslcd in the debug mode. Having said that, I am 
currently installing a sandbox which is a copy of the headnode and I will try 
to reproduce it there and here I can run nslcd in the debug mode. Hopefully 
that gives me some ideas of what is going on there. 

Thanks for your suggestions.

Jörg

Am Donnerstag, 18. Januar 2018, 07:47:31 GMT schrieben Sie:
> Hi Jorg,
> 
> Is the user added either to the Wheel group or as a user in the sudoers
> file?
> 
> Regards
> Jonathan
> 
> On 2018-01-17 23:12, Jörg Saßmannshausen wrote:
> > Dear all,
> > 
> > thanks for all your useful comments.
> > In the end, and after some debugging, I found the culprit. For one
> > reason or
> > another I installed libpam-ldap instead of libpam-ldapd. I guess that
> > was a
> > typo as libpam-ldapd will be pulled automatically when you are
> > installing
> > nslcd.
> > Once I corrected that, both su -l USER and ssh USER@localhost (or from
> > a
> > remote host to the Ubuntu VDI) are working fast again.
> > Don't ask me what is the difference between the two, I don't know is
> > the short
> > answer here.
> > 
> > One question: when I was doing some research on the internet, I came
> > across
> > nslcd and sssd. Which one is 'better'? I know that is a bit of an
> > ambiguous
> > question to ask but I have not found a page telling me the difference
> > between
> > the two.
> > 
> > Regarding Ubuntu vs. other distros: that is not my choice. Personally I
> > am in
> > favour of Debian but that is my personal choice. At the workplace I
> > have to
> > work with what is their policy. I am not a great fan of having
> > different
> > distributions floating around at one place as it make the
> > administration a
> > nightmare (you will be never good at all of them) but we are where we
> > are
> > here.
> > 
> > Regarding sudo: that is still a problem on one of the servers: it
> > simply does
> > not accept the password. Once I know more here I can report back to you
> > John.
> > 
> > Sorry for my slow response here. I am not looking at the email list
> > when I am
> > at work and thus it takes me a day or two to reply.
> > 
> > All the best from a cold London (storm about to come tonight)
> > 
> > Jörg
> > 
> > Am Mittwoch, 17. Januar 2018, 12:08:37 GMT schrieben Sie:
> >> I would switch to sssd. I had many problems with nslcd (connection,
> >> cache...).
> >> 
> >> Best regards
> >> 
> >> On 16/01/2018 00:35, Jörg Saßmannshausen wrote:
> >> > Dear all,
> >> > 
> >> > reading the Cluster Authentication (LDAP,AD) thread which was posted at
> >> > the
> >> > end of last year reminds me of a problem we are having.
> >> > 
> >> > For our Ubuntu 14 virtual machines we are authenticating against AD and
> >> > I
> >> > am using the nslcd daemon to do that.
> >> > This is working very well in a shell, i.e. when I am doing this in a
> >> > shell:
> >> > 
> >> > $ su -l USER
> >> > 
> >> > It is fast, it is creating the home directory if I need it (or not if I
> >> > want to mount the file space elsewhere and use a local home) and the
> >> > standard lookup tools like
> >> > 
> >> > $ getent password USER
> >> > 
> >> > are fast as well.
> >> > 
> >> > However, and here is where I am stuck: when I want to log in to the
> >> > machine
> >> > using the GUI, this takes forever. We measures it and it takes up to 90
> >> > sec. until it finally works. I also noticed that it is not reading the
> >> > /etc/nslcd.conf file but either /etc/ldap.conf or /etc/ldap/ldap.conf.
> >> > The
> >> > content of the ldap.conf file is identical with the nslcd.conf file. I
> >> > am
> >> > using TLS and not SSL for the secure connection .
> >> > Furthermore, and here I am n

Re: [Beowulf] cluster authentication part II

[Jörg Saßmannshausen]

Dear all,

thanks for all your useful comments. 
In the end, and after some debugging, I found the culprit. For one reason or 
another I installed libpam-ldap instead of libpam-ldapd. I guess that was a 
typo as libpam-ldapd will be pulled automatically when you are installing 
nslcd.
Once I corrected that, both su -l USER and ssh USER@localhost (or from a 
remote host to the Ubuntu VDI) are working fast again. 
Don't ask me what is the difference between the two, I don't know is the short 
answer here. 

One question: when I was doing some research on the internet, I came across 
nslcd and sssd. Which one is 'better'? I know that is a bit of an ambiguous 
question to ask but I have not found a page telling me the difference between 
the two. 

Regarding Ubuntu vs. other distros: that is not my choice. Personally I am in 
favour of Debian but that is my personal choice. At the workplace I have to 
work with what is their policy. I am not a great fan of having different 
distributions floating around at one place as it make the administration a 
nightmare (you will be never good at all of them) but we are where we are 
here. 

Regarding sudo: that is still a problem on one of the servers: it simply does 
not accept the password. Once I know more here I can report back to you John.

Sorry for my slow response here. I am not looking at the email list when I am 
at work and thus it takes me a day or two to reply.

All the best from a cold London (storm about to come tonight)

Jörg


Am Mittwoch, 17. Januar 2018, 12:08:37 GMT schrieben Sie:
> I would switch to sssd. I had many problems with nslcd (connection,
> cache...).
> 
> Best regards
> 
> On 16/01/2018 00:35, Jörg Saßmannshausen wrote:
> > Dear all,
> > 
> > reading the Cluster Authentication (LDAP,AD) thread which was posted at
> > the
> > end of last year reminds me of a problem we are having.
> > 
> > For our Ubuntu 14 virtual machines we are authenticating against AD and I
> > am using the nslcd daemon to do that.
> > This is working very well in a shell, i.e. when I am doing this in a
> > shell:
> > 
> > $ su -l USER
> > 
> > It is fast, it is creating the home directory if I need it (or not if I
> > want to mount the file space elsewhere and use a local home) and the
> > standard lookup tools like
> > 
> > $ getent password USER
> > 
> > are fast as well.
> > 
> > However, and here is where I am stuck: when I want to log in to the
> > machine
> > using the GUI, this takes forever. We measures it and it takes up to 90
> > sec. until it finally works. I also noticed that it is not reading the
> > /etc/nslcd.conf file but either /etc/ldap.conf or /etc/ldap/ldap.conf. The
> > content of the ldap.conf file is identical with the nslcd.conf file. I am
> > using TLS and not SSL for the secure connection .
> > Furthermore, and here I am not sure whether it is the same problem or a
> > different one, if I want to ssh into the Ubuntu VM, this also take a very
> > long time (90 sec) until I can do that.
> > Strangely enough, our HPC cluster is using nslcd as well (I used that
> > nslcd.conf file as a template for the Ubuntu setup), authenticating
> > against the same AD and that works instantaneous.
> > 
> > Does anybody has some ideas of where to look at? It somehow puzzles me.
> > I am a bit inclined to say the problem is within Ubuntu 14 as the cluster
> > is running CentOS and my Debian chroot environment ist Stretch.
> > 
> > All the best from London
> > 
> > Jörg
> > 
> > ___
> > Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
> > To change your subscription (digest mode or unsubscribe) visit
> > http://www.beowulf.org/mailman/listinfo/beowulf

___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
http://www.beowulf.org/mailman/listinfo/beowulf

[Beowulf] cluster authentication part II

[Jörg Saßmannshausen]

Dear all,

reading the Cluster Authentication (LDAP,AD) thread which was posted at the 
end of last year reminds me of a problem we are having.

For our Ubuntu 14 virtual machines we are authenticating against AD and I am 
using the nslcd daemon to do that. 
This is working very well in a shell, i.e. when I am doing this in a shell:

$ su -l USER

It is fast, it is creating the home directory if I need it (or not if I want 
to mount the file space elsewhere and use a local home) and the standard lookup 
tools like 

$ getent password USER

are fast as well.

However, and here is where I am stuck: when I want to log in to the machine 
using the GUI, this takes forever. We measures it and it takes up to 90 sec. 
until it finally works. I also noticed that it is not reading the 
/etc/nslcd.conf file but either /etc/ldap.conf or /etc/ldap/ldap.conf. The 
content of the ldap.conf file is identical with the nslcd.conf file. I am using 
TLS and not SSL for the secure connection .
Furthermore, and here I am not sure whether it is the same problem or a 
different one, if I want to ssh into the Ubuntu VM, this also take a very long 
time (90 sec) until I can do that. 
Strangely enough, our HPC cluster is using nslcd as well (I used that 
nslcd.conf file as a template for the Ubuntu setup), authenticating against the 
same AD and that works instantaneous. 

Does anybody has some ideas of where to look at? It somehow puzzles me. 
I am a bit inclined to say the problem is within Ubuntu 14 as the cluster is 
running CentOS and my Debian chroot environment ist Stretch. 

All the best from London

Jörg

___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
http://www.beowulf.org/mailman/listinfo/beowulf

Re: [Beowulf] [upgrade strategy] Intel CPU design bug & security flaw - kernel fix imposes performance penalty

[Jörg Saßmannshausen]

Dear all,

Chris is right here. It depends on what is running on your HPC cluster. You 
might not see a performance degrade at all, or you might see one of 30% (just 
to stick with that number).
Also, if you got a cluster which is solely used by one research group the 
chances they are hacking each other are slim I would argue. That leave still 
the argument about a compromised user account.
If you are running a large, multi user, multi institutional cluster you might 
want to put security over performance. This might be especially true if you 
are using confidential data like patient data. 
So, you will need to set up your own risk matrix and hope you made the right 
decision. 
For me: we have decided to upgrade the headnode but for now leave the compute 
nodes untouched. We then can decide at a later state whether or not we want to 
upgrade the compute nodes, maybe after we done some testing of typical 
programs. It is not an ideal scenario but we are living in a real and not 
ideal world I guess.

All the best from London

Jörg


Am Montag, 8. Januar 2018, 09:24:12 GMT schrieb Christopher Samuel:
> On 08/01/18 09:18, Richard Walsh wrote:
> > Mmm ... maybe I am missing something, but for an HPC cluster-specific
> > solution ... how about skipping the fixes, and simply requiring all
> > compute node jobs to run in exclusive mode and then zero-ing out user
> > memory between jobs ... ??
> 
> If you are running other daemons with important content (say the munge
> service that Slurm uses for authentication) then you risk the user being
> able to steal the secret key from the daemon.
> 
> But it all depends on your risk analysis of course.
> 
> All the best!
> Chris

___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
http://www.beowulf.org/mailman/listinfo/beowulf

Re: [Beowulf] [upgrade strategy] Intel CPU design bug & security flaw - kernel fix imposes performance penalty

[Jörg Saßmannshausen]

Dear Chris,

the first court cases against Intel have been filed:

http://www.spiegel.de/netzwelt/web/spectre-meltdown-erste-us-verbraucher-verklagen-intel-wegen-chip-schwachstelle-a-1186595.html

http://docs.dpaq.de/13109-show_temp.pl-27.pdf

http://docs.dpaq.de/13111-show_temp.pl-28.pdf

http://docs.dpaq.de/13110-07316352607.pdf

So, lets hope others are joining in here to get the ball rolling.

Don't get me wrong here, this is nothing against Intel per se. However, and 
here I am talking wearing my HPC hat, a performance decrease of up to 30% is 
simply not tolerable for me. I am working hard to squeeze the last performance 
out of the CPU and using highly optimised libraries and then the hardware has 
a flaw which makes all of that useless. I am somewhat surprised that this has 
not discovered earlier (both bugs I mean). 

I am sure it will be interesting to see how it will be patched and what the 
performance penalty will be here. 

All the best

Jörg

Am Samstag, 6. Januar 2018, 03:27:33 GMT schrieb Christopher Samuel:
> On 05/01/18 10:48, Jörg Saßmannshausen wrote:
> > What I would like to know is: how about compensation? For me that is
> > the same as the VW scandal last year. We, the users, have been
> > deceived.
> 
> I think you would be hard pressed to prove that, especially as it seems
> that pretty much every mainstream CPU is affected (Intel, AMD, ARM, Power).
> 
> > Specially if the 30% performance loss which have been mooted are not
> > special corner cases but are seen often in HPC. Some of the chemistry
> > code I am supporting relies on disc I/O, others on InfiniBand and
> > again other is running entirely in memory.
> 
> For RDMA based networks like IB I would suspect that the impact will be
> far less as the system calls to set things up will be impacted but that
> after that it should be less of an issue (as the whole idea of RDMA was
> to get the kernel out of the way as much as possible).
> 
> But of course we need real benchmarks to gauge that impact.
> 
> Separating out the impact of various updates will also be important,
> I've heard that the SLES upgrade to their microcode package includes
> disabling branch prediction on AMD k17 family CPUs for instance.
> 
> All the best,
> Chris

___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
http://www.beowulf.org/mailman/listinfo/beowulf

Re: [Beowulf] [upgrade strategy] Intel CPU design bug & security flaw - kernel fix imposes performance penalty

[Jörg Saßmannshausen]

Dear all,

that was the question I was pondering about all day today and I tried to read 
and digest any information I could get. 

In the end, I contacted my friend at CERT and proposed the following:
- upgrade the heanode/login node (name it how you like) as that one is exposed 
to the outside world via ssh
- do not upgrade the compute nodes for now until we got more information about 
the impact of the patch(es). 

It would not be the first time a patch is opening up another can of worms. What 
I am hoping for is finding a middle way between security and performance. IF 
the patch(es) are save to apply, I still can roll them out to the compute 
nodes without loosing too much uptime. IF there is a problem regarding 
performance it only affects the headnode which I can ignore on that cluster.

As always, your mileage will vary, specially as different clusters have 
different purposes.

What I would like to know is: how about compensation? For me that is the same 
as the VW scandal last year. We, the users, have been deceived. Specially if 
the 30% performance loss which have been mooted are not special corner cases 
but are seen often in HPC. Some of the chemistry code I am supporting relies 
on disc I/O, others on InfiniBand and again other is running entirely in 
memory. 

These are my 2 cents. If somebody has a better idea, please let me know.

All the best from a rainy and windy London

Jörg


Am Mittwoch, 3. Januar 2018, 13:56:50 GMT schrieb Remy Dernat:
> Hi,
> I renamed that thread because IMHO there is a another issue related to that
> threat. Should we upgrade our system and lost a significant amount of
> XFlops... ? What should be consider :   - the risk  - your user population
> (size / type / average "knowledge" of hacking techs...)  - the isolation
> level from the outside (internet)
> 
> So here is me question : if this is not confidential, what will you do ?
> I would not patch our little local cluster, contrary to all of our other
> servers. Indeed, there is another "little" risk. If our strategy is to
> always upgrade/patch, in this particular case you can loose many users that
> will complain about perfs... So another question : what is your global
> strategy about upgrades on your clusters ? Do you upgrade it as often as
> you can ? One upgrade every X months (due to the downtime issue) ... ?
> 
> Thanks,
> Best regardsRémy.
> 
>  Message d'origine De : John Hearns via Beowulf
>  Date : 03/01/2018  09:48  (GMT+01:00) À : Beowulf
> Mailing List  Objet : Re: [Beowulf] Intel CPU design
> bug & security flaw - kernel fix imposes performance penalty Thanks Chris. 
> In the past there have been Intel CPU 'bugs' trumpeted, but generally these
> are fixed with a microcode update. This looks different, as it is a
> fundamental part of the chips architecture.However the Register article
> says: "It allows normal user programs – to discern to some extent the
> layout or contents of protected kernel memory areas" I guess the phrase "to
> some extent" is the vital one here. Are there any security exploits which
> use this information? I guess it is inevitable that one will be engineered
> now that this is known about. The question I am really asking is should we
> worry about this for real world systems. And I guess tha answer is that if
> the kernel developers are worried enough then yes we should be too.
> Comments please.
> 
> 
> 
> On 3 January 2018 at 06:56, Greg Lindahl  wrote:
> 
> On Wed, Jan 03, 2018 at 02:46:07PM +1100, Christopher Samuel wrote:
> > There appears to be no microcode fix possible and the kernel fix will
> > 
> > incur a significant performance penalty, people are talking about in the
> > 
> > range of 5%-30% depending on the generation of the CPU. :-(
> 
> The performance hit (at least for the current patches) is related to
> 
> system calls, which HPC programs using networking gear like OmniPath
> 
> or Infiniband don't do much of.
> 
> 
> 
> -- greg
> 
> 
> 
> 
> 
> ___
> 
> Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
> 
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf

___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
http://www.beowulf.org/mailman/listinfo/beowulf

Re: [Beowulf] Openlava down?

[Jörg Saßmannshausen]

Hi Jeff,

have you tried:

http://openlava.com/

I only get an error on this site:

http://www.openlava.org/

Now, it could well be that these are different sites. 

All the best from London and I wish you all a Merry Christmas!

Jörg



Am Samstag, 23. Dezember 2017, 15:44:49 GMT schrieb Jeffrey Layton:
> Good afternoon,
> 
> I've noticed that openlava.org is down and emails to openlava-users at
> Google groups bounce with a message that the address doesn't exist? Is
> Openlava still around?
> 
> Jeff

___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
http://www.beowulf.org/mailman/listinfo/beowulf

Re: [Beowulf] Troubleshooting NFS stale file handles

[Jörg Saßmannshausen]

Hi Prentice,

three questions (not necessarily to you and it can be dealt with in a different 
thread too):

- why automount and not a static mount?
- do I get that right that the nodes itself export shares to other nodes?
- has anything changed? I am thinking of something like more nodes added, new 
programs being installed, more users added, generally a higher load on the 
cluster.

One problem I had in the past with my 112 node cluster where I am exporting 
/home, /opt and one directory in /usr/local to all the nodes from the headnode 
was that the NFS-server on the headnode did not have enough spare servers 
assigned and thus was running out of capacity. That also lead to strange 
behaviour which I fixed by increasing the numbers of spare servers. 

The way I have done that was setting this in 
/etc/default/nfs-kernel-server

# Number of servers to start up
RPCNFSDCOUNT=32

That seems to provide the right amount of servers and spare ones for me. 
Like in your case, the cluster was running stable until I added more nodes 
*and* users decided to use them, i.e. the load of the cluster got up. A more 
idle cluster did not show any problems, a cluster under 80 % load suddenly had 
problem. 

I hope that helps a bit. I am not the expert in NFS as well and this is just 
my experience. I am also using Debian nfs-kernel-server 1:1.2.6-4 if that 
helps. 

All the best from a sunny London

Jörg

On Mittwoch 19 April 2017 Prentice Bisbal wrote:
> On 04/19/2017 02:17 PM, Ellis H. Wilson III wrote:
> > On 04/19/2017 02:11 PM, Prentice Bisbal wrote:
> >> Thanks for the suggestion(s). Just this morning I started considering
> >> the network as a possible source of error. My stale file handle errors
> >> are easily fixed by just restarting the nfs servers with 'service nfs
> >> restart', so they aren't as severe you describe.
> > 
> > If a restart on solely the /server-side/ gets you back into a good
> > state this is an interesting tidbit.
> 
> That is correct, restarting NFS on the server-side is all it takes to
> fix the problem
> 
> > Do you have some form of HA setup for NFS?  Automatic failover
> > (sometimes setup with IP aliasing) in the face of network hiccups can
> > occasionally goof the clients if they aren't setup properly to keep up
> > with the change.  A restart of the server will likely revert back to
> > using the primary, resulting in the clients thinking everything is
> > back up and healthy again.  This situation varies so much between
> > vendors it's hard to say much more without more details on your setup.
> 
> My setup isn't nearly that complicated. Every node in this cluster has a
> /local directory that is shared out to the other nodes in the cluster.
> The other nodes automount this by remote directory as /l/hostname, where
> "hostname" is the name of owner of the filesystem. For example, hostB
> will mount hostA:/local as /l/lhostA.
> 
> No fancy fail-over or anything like that.
> 
> > Best,
> > 
> > ellis
> > 
> > P.S., apologies for the top-post last time around.
> 
> NO worries. I'm so used to people doing that, in mailing lists that I've
> become numb to it.
> 
> Prentice
> 
> ___
> Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf


-- 
*
Dr. Jörg Saßmannshausen, MRSC
University College London
Department of Chemistry
20 Gordon Street
London
WC1H 0AJ 

email: j.sassmannshau...@ucl.ac.uk
web: http://sassy.formativ.net

Please avoid sending me Word or PowerPoint attachments.
See http://www.gnu.org/philosophy/no-word-attachments.html


signature.asc
Description: This is a digitally signed message part.
___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
http://www.beowulf.org/mailman/listinfo/beowulf

Re: [Beowulf] Suggestions to what DFS to use

[Jörg Saßmannshausen]

t; Hat if you like.
> >> 
> >> Take a look at the RH whitepapers about typical GlusterFS architecture.
> >> 
> >> CephFS, on the other hand, is not yet mature enough, IMHO.
> >> On Mon, Feb 13, 2017 at 8:31 AM Justin Y. Shi <s...@temple.edu> wrote:
> >> 
> >> Maybe you would consider Scality (http://www.scality.com/) for your
> >> growth concerns. If you need speed, DDN is faster in rapid data ingestion
> >> and for extreme HPC data needs.
> >> 
> >> Justin
> >> 
> >> On Mon, Feb 13, 2017 at 4:32 AM, Tony Brian Albers <t...@kb.dk> wrote:
> >> 
> >> On 2017-02-13 09:36, Benson Muite wrote:
> >> > Hi,
> >> > 
> >> > Do you have any performance requirements?
> >> > 
> >> > Benson
> >> > 
> >> > On 02/13/2017 09:55 AM, Tony Brian Albers wrote:
> >> >> Hi guys,
> >> >> 
> >> >> So, we're running a small(as in a small number of nodes(10), not
> >> >> storage(170TB)) hadoop cluster here. Right now we're on IBM Spectrum
> >> >> Scale(GPFS) which works fine and has POSIX support. On top of GPFS we
> >> >> have a GPFS transparency connector so that HDFS uses GPFS.
> >> >> 
> >> >> Now, if I'd like to replace GPFS with something else, what should I
> >> 
> >> use?
> >> 
> >> >> It needs to be a fault-tolerant DFS, with POSIX support(so that users
> >> >> can move data to and from it with standard tools).
> >> >> 
> >> >> I've looked at MooseFS which seems to be able to do the trick, but are
> >> >> there any others that might do?
> >> >> 
> >> >> TIA
> >> 
> >> Well, we're not going to be doing a huge amount of I/O. So performance
> >> requirements are not high. But ingest needs to be really fast, we're
> >> talking tens of terabytes here.
> >> 
> >> /tony
> >> 
> >> --
> >> Best regards,
> >> 
> >> Tony Albers
> >> Systems administrator, IT-development
> >> Royal Danish Library, Victor Albecks Vej 1, 8000 Aarhus C, Denmark.
> >> Tel: +45 2566 2383 / +45 8946 2316
> >> ___
> >> Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
> >> To change your subscription (digest mode or unsubscribe) visit
> >> http://www.beowulf.org/mailman/listinfo/beowulf
> >> 
> >> 
> >> ___
> >> Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
> >> To change your subscription (digest mode or unsubscribe) visit
> >> http://www.beowulf.org/mailman/listinfo/beowulf
> >> 
> >> ___
> >> Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
> >> To change your subscription (digest mode or unsubscribe) visit
> >> http://www.beowulf.org/mailman/listinfo/beowulf
> > 
> > --
> > ‘[A] talent for following the ways of yesterday, is not sufficient to
> > improve the world of today.’
> > 
> >  - King Wu-Ling, ruler of the Zhao state in northern China, 307 BC
> > 
> > ___
> > Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
> > To change your subscription (digest mode or unsubscribe) visit
> > http://www.beowulf.org/mailman/listinfo/beowulf
> > 
> > 
> > 
> > ___
> > Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
> > To change your subscription (digest mode or unsubscribe) visit
> > http://www.beowulf.org/mailman/listinfo/beowulf


-- 
*
Dr. Jörg Saßmannshausen, MRSC
University College London
Department of Chemistry
20 Gordon Street
London
WC1H 0AJ 

email: j.sassmannshau...@ucl.ac.uk
web: http://sassy.formativ.net

Please avoid sending me Word or PowerPoint attachments.
See http://www.gnu.org/philosophy/no-word-attachments.html

signature.asc
Description: This is a digitally signed message part.
___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
http://www.beowulf.org/mailman/listinfo/beowulf

Re: [Beowulf] cluster os

[Jörg Saßmannshausen]

Hi Jonathan,

the only thing I am compiling from source are the HPC programs we are using 
and also some of the HPC libraries. I want to get the best out of the system 
and hence I decided to do it this way. Also, some HPC programs are simply only 
available as source code. So depending how often there is an upgrade of the 
program and how it is distributed (source or binary) I do the upgrades. 

The libraries I usually do once. In particular I don't upgrade say ATLAS, fftw3 
etc until I upgrade the whole cluster. 

I hope that helps a bit.

All the best from London

Jörg

On Friday 20 May 2016 12:34:37 Jonathan Aquilina wrote:
> Going a bit off track from upgrades here.  In a cluster environment with
> debian do you spend a fair bit of time compiling anything from source?
> 
> On 2016-05-20 12:08, Tim Cutts wrote:
> > In practice, at Sanger we haven't made very heavy use of the ability of
> > Debian to upgrade from release to release.  We use FAI to install the
> > boxes, so frankly it's faster and less hassle to just reinstall them from
> > scratch when the next release comes out.
> > 
> > For some complicated bespoke systems, I have done the manual upgrade, and
> > I've even done that across debian-derived distros.
> > 
> > Officially, you're not supposed to be able to use apt to upgrade a system
> > from Debian to Ubuntu, for example, but I have done it in the past - I
> > worked out a documented procedure for doing a dist-upgrade from Debian
> > Lenny to Ubuntu Lucid, for example.  At one point, you do have to force
> > downgrade packages which are newer in the original os than they are in
> > the target, which led to the following one liner of which I am perversely
> > proud:
> > 
> > aptitude search -F '%p' ~i | xargs -n 1 apt-cache policy | sed -n -e '
> > /^[a-z0-9.-]*:$/{h;d;}
> > /\*\*\*/{
> > n
> > /^  *500 /d
> > /^  *100 /{
> > n
> > /^  *[^ ]/{x;s/:/\/lucid/p;x;}
> > }
> > /^[a-z0-9.-]*:$/{h;d;}
> > }
> > ' | xargs apt-get -y --force-yes install
> > 
> > Gotta love sed for resulting in completely impenetrable commands.  I did
> > try to do it in a more readable way using the python APT bindings, but
> > that was actually much harder.
> > 
> > Tim
> > ___
> > Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
> > To change your subscription (digest mode or unsubscribe) visit
> > http://www.beowulf.org/mailman/listinfo/beowulf


-- 
*
Dr. Jörg Saßmannshausen, MRSC
University College London
Department of Chemistry
20 Gordon Street
London
WC1H 0AJ 

email: j.sassmannshau...@ucl.ac.uk
web: http://sassy.formativ.net

Please avoid sending me Word or PowerPoint attachments.
See http://www.gnu.org/philosophy/no-word-attachments.html

signature.asc
Description: This is a digitally signed message part.
___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
http://www.beowulf.org/mailman/listinfo/beowulf

Re: [Beowulf] cluster os

[Jörg Saßmannshausen]

Dear Jonathan,

here at UCL we are running a number of different distros as it looks like.

I am using Debian for my clusters now for the last 5 years or so without any 
problems. In fact, I have upgraded from Debain etch up to jessie, the latest, 
over the time and I never had any real issues. Of course I had to recompile 
the HPC code which is not part of the distribution but that is something you 
have to do anyway. The clusters are running stable without any issues 
regarding OS (i.e. not hardware failures).
The upgrade in Debian is working really well. You really can install a machine 
once and then you can upgrade it to the latest OS without much issues. You 
might need to do some minor tweaks when things are changing, like going from 
NFS3 to NFS4, but that is in the nature of the upgrade. 
This experience seems to be in stark difference from the OS upgrade on the 
central cluster (RedHat) where the cluster needed to be installed from 
scratch. At least that was my understanding. 

A source based one like Gentoo might be good for a desktop but with a Cluster 
you don't want to spent all the time in compiling the OS. You do want to spent 
time to make sure the HPC programs are running with the best efficiency you can 
get. Thus, I rather spent time into compiling a proper BLAS instead of 
spending time to optimise say gcc.

Thus, I would recommend a package based distribution and not a source based 
one and I would recommend Debian as the distro of choice here.

My two pennies from London. :-)

Jörg

On Thursday 19 May 2016 09:08:22 remy. dernat wrote:
> Hi,
> If you need certified things like drivers or whatever (IB...), choose
> redhat-like. Otherwise, IMHO debian-like is a betteer choice.
> Gentoo could be use for development, or test purpose. I do not think it is
> really a good choice. Cheers.
> 
> 
> Envoyé depuis mon appareil Samsung
> 
>  Message d'origine 
> De : Jonathan Aquilina <jaquil...@eagleeyet.net>
> Date : 19/05/2016  06:20  (GMT+01:00)
> À : Beowulf Mailing List <beowulf@beowulf.org>
> Objet : [Beowulf] cluster os
> 
> 
> Good Morning,
> I am just wondering what distribution of choice would one use for their
> cluster? Would one go for a source based distro like gentoo or a
> precompiled one like Centos7? 


-- 
*
Dr. Jörg Saßmannshausen, MRSC
University College London
Department of Chemistry
20 Gordon Street
London
WC1H 0AJ 

email: j.sassmannshau...@ucl.ac.uk
web: http://sassy.formativ.net

Please avoid sending me Word or PowerPoint attachments.
See http://www.gnu.org/philosophy/no-word-attachments.html

signature.asc
Description: This is a digitally signed message part.
___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
http://www.beowulf.org/mailman/listinfo/beowulf

Re: [Beowulf] shared memory error

[Jörg Saßmannshausen]

Hi all,

sorry for the lack of reply but I done some testings and now I got some 
updates here.

I am using GAMESS version 5 DEC 2014 (R1) compiled with gfortran 4.9.2 on 
Debian Linux Jessie. ATLAS is used for BLAS/LAPACK. All the test jobs have 
passed.

What I have noticed is: the problem is not really reproducible. So, the same 
input file is running well on my machine at home (same GAMESS version but 
gfortran 4.7.2 and Debian Wheezy) but not on the machines at work. To make 
things more interesting:
- it might run perfectly ok on one machine but not on another one. They are 
identical nodes with identical OS. All the installation of the nodes are done 
from one master image.
- it might start and generates the error very soon
- it might run for ages and suddenly generates the error
- my binary from my machine does generate the error on the machines at work

- I am lost in the mist. :-)

I cannot see a pattern here. I am still wondering whether my settings of the 
shared memory might be correct as the only differences I can see between my 
machine at home (48 GB of RAM) and the machines at work (64 GB of RAM) is the 
memory. Having said that, as I got less RAM at home and it is working I would 
have thought that my settings are ok for less RAM and thus should work on more 
RAM as well.

Unfortunately I never got a reply from the GAMESS groups which usually means 
nobody knows the answer here. 

Any ideas?

All the best from a cold London

Jörg




On Dienstag 05 April 2016 Rafael R. Pappalardo wrote:
> Could you share with us the input file(s)? Which version of GAMESS-US?
> 
> On lunes, 4 de abril de 2016 22:29:11 (CEST) Jörg Saßmannshausen wrote:
> > Dear all,
> > 
> > I was wondering whether somebody might be able to shed some light on this
> > problem I am having with a chemistry code (GAMESS-US):
> > 
> > DDI Process 15: semop return an error performing 1 operation(s) on semid
> > 98307.
> > semop errno=EINVAL.
> > 
> > This sometimes happens when I need quite a bit of memory for the fortran
> > code (155000 words). Originally I thought it has to do with the
> > hardware I am running it on but meanwhile I found it all over the place,
> > i.e. on some older Opterons and on some newer Ivy and Haswell CPUs.
> > 
> > It is not quite reproducible, unfortunately. A run might work ok for a
> > few days and then the problem kicks in and the logfile explodes from
> > around 14 MB to 17 GB, or it might just work.
> > 
> > Some system informations: I am running Debian Jessie with gcc / gfortran
> > version 4.9.2-10. The nodes have 64 GB of RAM and 16 or 20 cores.  As the
> > shared memory default settings in Linux are not suitable for GAMESS
> > (there is a note in the documentation), I am using these settings on the
> > 64 GB RAM machines:
> > 
> > kernel.shmmax = 692300
> > kernel.shmall = 25165824
> > kernel.shmmni = 32768
> > 
> > I got the feeling the problem lies burried in these settings but my
> > knowledge here is not sufficient to solve the problem. Could somebody
> > point me in the right direction here?
> > 
> > All the best from London
> > 
> > Jörg


-- 
*
Dr. Jörg Saßmannshausen, MRSC
University College London
Department of Chemistry
20 Gordon Street
London
WC1H 0AJ 

email: j.sassmannshau...@ucl.ac.uk
web: http://sassy.formativ.net

Please avoid sending me Word or PowerPoint attachments.
See http://www.gnu.org/philosophy/no-word-attachments.html


signature.asc
Description: This is a digitally signed message part.
___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
http://www.beowulf.org/mailman/listinfo/beowulf

[Beowulf] shared memory error

[Jörg Saßmannshausen]

Dear all,

I was wondering whether somebody might be able to shed some light on this 
problem I am having with a chemistry code (GAMESS-US):

DDI Process 15: semop return an error performing 1 operation(s) on semid 
98307.
semop errno=EINVAL.

This sometimes happens when I need quite a bit of memory for the fortran code 
(155000 words). Originally I thought it has to do with the hardware I am 
running it on but meanwhile I found it all over the place, i.e. on some older 
Opterons and on some newer Ivy and Haswell CPUs.

It is not quite reproducible, unfortunately. A run might work ok for a few 
days and then the problem kicks in and the logfile explodes from around 14 MB 
to 17 GB, or it might just work.

Some system informations: I am running Debian Jessie with gcc / gfortran 
version 4.9.2-10. The nodes have 64 GB of RAM and 16 or 20 cores.  As the 
shared memory default settings in Linux are not suitable for GAMESS (there is 
a note in the documentation), I am using these settings on the 64 GB RAM 
machines:

kernel.shmmax = 692300
kernel.shmall = 25165824
kernel.shmmni = 32768

I got the feeling the problem lies burried in these settings but my knowledge 
here is not sufficient to solve the problem. Could somebody point me in the 
right direction here?

All the best from London

Jörg

-- 
*
Dr. Jörg Saßmannshausen, MRSC
University College London
Department of Chemistry
20 Gordon Street
London
WC1H 0AJ 

email: j.sassmannshau...@ucl.ac.uk
web: http://sassy.formativ.net

Please avoid sending me Word or PowerPoint attachments.
See http://www.gnu.org/philosophy/no-word-attachments.html


signature.asc
Description: This is a digitally signed message part.
___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
http://www.beowulf.org/mailman/listinfo/beowulf

Re: [Beowulf] modern batch management systems

[Jörg Saßmannshausen]

Hi Mikhail,

I am still using the last free SGE (6.2u5) and that is working for me well. I 
had to update a few scripts to it is recognising newer kernels. 

Having said that, I am aware there is the Son of Gridengine which is 
maintained by a colleague in Liverpool if I am not mistaken. As far as I know 
this one is free. The link is here:

https://arc.liv.ac.uk/trac/SGE

Whether it will be still free in 200 years time I cannot tell of course. ;-)

I hope that helps a bit.

All the best from a dull London

Jörg

On Tuesday 10 Nov 2015 12:49:57 Mikhail Kuzminsky wrote:
>  In our  more old clusters we used PBS and SGE as batch systems to run
> quantum-chemical applications. Now there are also commercial versions PBS
> Pro and Oracle Grid Engine, and other commercial batch management programs.
> But we are based on free open-source batch management systems. Which free
> (and potentially free in a few years) batch systems you recommend ?
> 
> Mikhail Kuzminsky
> Zelinsky Institute of Organic Chemistry RAS
> Moscow
> 
> 
> 
> 
> Mikhail Kuzminsky


-- 
*****
Dr. Jörg Saßmannshausen, MRSC
University College London
Department of Chemistry
20 Gordon Street
London
WC1H 0AJ 

email: j.sassmannshau...@ucl.ac.uk
web: http://sassy.formativ.net

Please avoid sending me Word or PowerPoint attachments.
See http://www.gnu.org/philosophy/no-word-attachments.html

signature.asc
Description: This is a digitally signed message part.
___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
http://www.beowulf.org/mailman/listinfo/beowulf

Re: [Beowulf] Mellanox UFM question

[Jörg Saßmannshausen]

Hi Jeff,

no, not yet. 
What I want to avoid is: I try the OFED subnet manager and it does not work 
and then I have to wait until I get the licence. This project has enough 
delays right now and I don't want to add to it. Hence my question.

Having said that: are you happy with the OFED one?

All the best

Jörg

On Dienstag 15 September 2015 Jeff Becker wrote:
> Hi Jörg,
> 
> Have you tried using the subnet manager from Mellanox OFED (which is
> free)? That's what we use on our big heterogeneous cluster at NASA.
> 
> HTH
> 
> -jeff
> 
> On 09/15/2015 08:55 AM, Jörg Saßmannshausen wrote:
> > Dear all,
> > 
> > I am a bit confused and I was wondering whether somebody on the list
> > could give me a bit of advice here.
> > 
> > I was previously using QLogic for my QDR InfiniBand network. I got one
> > master switch which got the licence for the InfiniBand installed and
> > things appear to work ok. At least I cannot detect any problems despite
> > adding switches and nodes to the fabric.
> > 
> > Now, we recently purchased a new cluster with 20 cores per node and here
> > I decided to go for FDR to be a bit more future proofed as well. So I
> > got the 'normal' licence from Mellanox for the cluster. I got one
> > licence per node so I assumed that was ok.
> > 
> > Now, we are in the process to set up another cluster with a mixture of
> > older and newer hardware. Again I have decided to opt for the FDR simply
> > to be a bit more future proofed. And this is where the confusion comes
> > in.
> > 
> > Apparently I do need now the UFM (Unified Fibre Manager) from Mellanox to
> > run the InfiniBand. However, the normal licence is only for up to 16
> > cores per node and I would need the more expensive exhanced licence.
> > 
> >  From what I and a colleague of mine can see the UFM is nothing more than
> >  the
> > 
> > requires subnet manager plus some diagnostic tools.
> > 
> > There are two questions here:
> > 
> > - do we really need the exhanced UFM licence or is that just a way to
> > make money?
> > 
> > - would the open source subnet manage work as well and would the open
> > source diagnostic tools be ok?
> > 
> > - why do I need to pay for a licence for each node? Somehow I cannot
> > recall having done that in the past.
> > 
> > Unfortunately, InfiniBand is not my strong side and thus I would
> > appreciate and advice here.
> > 
> > All the best from a meanwhile sunny London
> > 
> > Jörg
> > 
> > 
> > 
> > ___
> > Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
> > To change your subscription (digest mode or unsubscribe) visit
> > http://www.beowulf.org/mailman/listinfo/beowulf


-- 
*
Dr. Jörg Saßmannshausen, MRSC
University College London
Department of Chemistry
20 Gordon Street
London
WC1H 0AJ 

email: j.sassmannshau...@ucl.ac.uk
web: http://sassy.formativ.net

Please avoid sending me Word or PowerPoint attachments.
See http://www.gnu.org/philosophy/no-word-attachments.html


signature.asc
Description: This is a digitally signed message part.
___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
http://www.beowulf.org/mailman/listinfo/beowulf

[Beowulf] Mellanox UFM question

[Jörg Saßmannshausen]

Dear all,

I am a bit confused and I was wondering whether somebody on the list could 
give me a bit of advice here.

I was previously using QLogic for my QDR InfiniBand network. I got one master 
switch which got the licence for the InfiniBand installed and things appear to 
work ok. At least I cannot detect any problems despite adding switches and 
nodes to the fabric.

Now, we recently purchased a new cluster with 20 cores per node and here I 
decided to go for FDR to be a bit more future proofed as well. So I got the 
'normal' licence from Mellanox for the cluster. I got one licence per node so 
I assumed that was ok.

Now, we are in the process to set up another cluster with a mixture of older 
and newer hardware. Again I have decided to opt for the FDR simply to be a bit 
more future proofed. And this is where the confusion comes in.

Apparently I do need now the UFM (Unified Fibre Manager) from Mellanox to run 
the InfiniBand. However, the normal licence is only for up to 16 cores per node 
and I would need the more expensive exhanced licence.

From what I and a colleague of mine can see the UFM is nothing more than the 
requires subnet manager plus some diagnostic tools. 

There are two questions here:

- do we really need the exhanced UFM licence or is that just a way to make 
money?

- would the open source subnet manage work as well and would the open source 
diagnostic tools be ok?

- why do I need to pay for a licence for each node? Somehow I cannot recall 
having done that in the past.

Unfortunately, InfiniBand is not my strong side and thus I would appreciate and 
advice here. 

All the best from a meanwhile sunny London

Jörg

-- 
*
Dr. Jörg Saßmannshausen, MRSC
University College London
Department of Chemistry
20 Gordon Street
London
WC1H 0AJ 

email: j.sassmannshau...@ucl.ac.uk
web: http://sassy.formativ.net

Please avoid sending me Word or PowerPoint attachments.
See http://www.gnu.org/philosophy/no-word-attachments.html


signature.asc
Description: This is a digitally signed message part.
___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
http://www.beowulf.org/mailman/listinfo/beowulf

Re: [Beowulf] Haswell as supercomputer microprocessors

[Jörg Saßmannshausen]

Hi Mikhail,

I would guess your queueing system could take care of that. 

With SGE you can define how many cores each node has. Thus, if you only want to 
use 16 out of the 18 cores you simply define that.

Alternatively, at least OpenMPI allows you to underpopulate the nodes as well.

Having said that, is there a good reason why you want to purchase 18 cores and 
then only use 16? 
The only thing I can think of why one needs to / wants to do that is if your 
job requires more memory which you got on the node. For memory intensive work 
I am still thinking that less cores and more nodes are beneficial here.

My 2 cents from a sunny London

Jörg

On Monday 03 Aug 2015 10:06:27 Mikhail Kuzminsky wrote:
  New special supercomputer microprocessors (like IBM Power BQC and Fujitsu
 SPARC64 XIfx) have 2**N +2 cores (N=4 for 1st, N=5 for 2nd), where 2 last
 cores are redundant, not for computations, but only for other work w/Linux
 or even for replacing of failed computational core.
 
 Current Intel Haswell E5 v3 may also have 18 = 2**4 +2 cores.  Is there
 some sense to try POWER BQC or SPARC64 XIfx ideas (not exactly), and use
 only 16 Haswell cores for parallel computations ? If the answer is yes,
 then how to use this way under Linux ?
 
 Mikhail Kuzminsky,
 Zelinsky Institute of Organic Chemistry RAS,
 Moscow

-- 
*
Dr. Jörg Saßmannshausen, MRSC
University College London
Department of Chemistry
Gordon Street
London
WC1H 0AJ 

email: j.sassmannshau...@ucl.ac.uk
web: http://sassy.formativ.net

Please avoid sending me Word or PowerPoint attachments.
See http://www.gnu.org/philosophy/no-word-attachments.html


signature.asc
Description: This is a digitally signed message part.
___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
http://www.beowulf.org/mailman/listinfo/beowulf

Re: [Beowulf] Software installation across cluster

[Jörg Saßmannshausen]

Hi Trevor,

I got a simple solution for that:
I got all the software (OS, bootloader and programs) installed in one folder 
on the headnode and I am simply using rsync to copy it over to the nodes. So 
apart from the home directory, there is only /opt and the SGE NFS mounted. 

I am not running NIS or suchlike but stick to the old password, shadow, and 
group files in /etc.

This way around if the headnode crashes there is a good chance the program 
will continue to run until the headnode is back. From experiences that is 
working (strangely but true).

Also, as I got a heterogeneous environment in one cluster, i.e. Sandybridge, 
Ivy-Bridge and Haswell CPUs, I can distribute the highly optimised 
libraries/programs individually. This is a bit of manual work but given I 
don't need to re-install the nodes often the installation of the users and the 
programs are actually easy.

I know there are other ways of doing it and I think they all have their pros 
and cons. In the end, use something you feel comfortable with and you know how 
to manage. 

My 2 cents here :D

All the best from a dark London

Jörg


On Dienstag 26 Mai 2015 Trevor Gale wrote:
 Hello all,
 
 I was wondering what solutions other use for easy software installation
 across clusters. Is there any method that is generally accepted to be the
 most effective for making sure that all the software is consistent across
 each node?
 
 Thanks,
 Trevor
 ___
 Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
 To change your subscription (digest mode or unsubscribe) visit
 http://www.beowulf.org/mailman/listinfo/beowulf


-- 
*
Dr. Jörg Saßmannshausen, MRSC
University College London
Department of Chemistry
Gordon Street
London
WC1H 0AJ 

email: j.sassmannshau...@ucl.ac.uk
web: http://sassy.formativ.net

Please avoid sending me Word or PowerPoint attachments.
See http://www.gnu.org/philosophy/no-word-attachments.html


signature.asc
Description: This is a digitally signed message part.
___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
http://www.beowulf.org/mailman/listinfo/beowulf

Re: [Beowulf] Cluster networking/OSs

[Jörg Saßmannshausen]

Hi Trevor,

what kind of jobs do you want to run on the cluster?

What I am after is: is it mainly number crunching, large parallel jobs,
serial jobs, jobs which do a lot of disc IO, need much memory,
visualisation.

For example, I am not the 'master' of 8 clusters and most of them have
different purposes. The largest cluster has a fast InfiniBand network as
people are running parallel jobs there, sometimes up to 128 cores or so.
Large for that cluster but still tiny for others, I know.
A different cluster has a web interface for the users, so here they can
submit structures, see whether they have been calculated before, re-do
it or just get the results. So here I have installed a webserver and a
database (both in virtual machines, incidentally).

I am using Debian as the OS here, with wheezy and jessie for the
versions. I use PXE for the installation of the nodes which gives me the
advantage I got a rescue OS as well in case there is a disc problem. The
nodes get their IP from a DHCP server and I am running a local DNS
server so the command

$ ssh node1

does actually work throughout the cluster.

I hope that helps you a little bit.

All the best from a sunny Graz where I am on vacation!

Jörg

On 08/05/15 20:30, Trevor Gale wrote:
 Hey Everyone,
 
 I’m fairly new to linux clusters and am trying to learn as much as I can 
 about specifically networking on clusters and different operating systems run 
 across clusters. Does anyone know any good resources to learn from?
 
 Thanks,
 Trevor
 
 ___
 Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
 To change your subscription (digest mode or unsubscribe) visit 
 http://www.beowulf.org/mailman/listinfo/beowulf
 

-- 
*
Dr. Jörg Saßmannshausen, MRSC
University College London
Department of Chemistry
Gordon Street
London
WC1H 0AJ

email: j.sassmannshau...@ucl.ac.uk
web: http://sassy.formativ.net

Please avoid sending me Word or PowerPoint attachments.
See http://www.gnu.org/philosophy/no-word-attachments.html



signature.asc
Description: OpenPGP digital signature
___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
http://www.beowulf.org/mailman/listinfo/beowulf

Re: [Beowulf] RAID question

[Jörg Saßmannshausen]

Hi all,

you can access the disc if it is hooked up to (some) LSI controllers:

Try:

$ smartctl -i /dev/sda -d megaraid,X

You need to play around a bit with X as that is the port the controller is 
using. As you can see, I am using a megaraid controller card and that is 
working well here.

As always: your mileage may vary.

All the best from a sunny London

Jörg

On Wednesday 18 Mar 2015 00:34:16 you wrote:
 Dell controllers are (almost?) always rebranded LSI.  You can use the LSI
 tool megacli, or if you prefer a slightly less insane UI you can use the
 newer storcli. To see what happened during your consistency check you
 would need to check the logs that can be generated with these tools.  These
 logs will probably also help you determine what went wrong initially.
 Unfortunately, these RAID controllers do not allow you to access any SMART
 data unless they are in JBOD mode so you pretty much have to pull the disk
 and check it on another machine.
 
 Last I looked, the default behavior for a consistency check is to assume
 the primary copy is correct for any discrepancies and overwrite the
 parity/mirror versions with that.  Which is pretty dumb for a default
 setting (you can at least disable this automatic repair).  So your
 consistency check might have wiped the good data...
 
 
 
 On Tue, Mar 17, 2015 at 2:57 PM, Jörg Saßmannshausen 
 
 j.sassmannshau...@ucl.ac.uk wrote:
  Hi David,
  
  for me it looks like either a controller or disc issue.
  
  I have seen these problems before on SCSI discs when the controller had a
  problem. Depending on the manufacturer it might be a good idea to contact
  them
  and see if they got more informations here. I have had some problems in
  the past with RAID controllers and the manufacturer here was ever so
  helpful in the diagnosis and repair of a failed RAID5 for example.
  
  So it might be a good idea to try them.
  
  All the best from a cold London
  
  Jörg
  
  On Montag 16 März 2015 mathog wrote:
   Thanks for the feedback.
   
   After copying /boot and /bin from another machine and mucking about
   with grub for far too long (had to edit grub.conf to change virtual
   disk names, and in CentOS's rescue disk it saw the boot disk as hd1,
   but when grub actually started, it saw it as hd0) the system is back
   on line.
   
   The logs don't show a root command line that specifically took out
   those directories.  They do show a bunch of scripts being run.  My
   best guess
   
   is that one of them did something like this:
  AVAR=`command that failed and returned an empty string`
  rm -rf ${AVAR}/b*
   
   It seems unlikely that a low level controller failure would have
   snipped out those files/directories without resulting in a file system
   that was seen as corrupt by fsck.
   
   That said, there is something hardware related going on, since
   /var/log/messages has a lot of these (sorry about the wrap):
   
   Mar 16 12:37:27 mandolin kernel: sd 7:0:0:0: [sdb]  Sense Key :
   Recovered Error [current] [descriptor]
   Mar 16 12:37:27 mandolin kernel: Descriptor sense data with sense
   descriptors (in hex):
   Mar 16 12:37:27 mandolin kernel:72 01 04 1d 00 00 00 0e 09 0c
   00 00 00 00 00 00
   Mar 16 12:37:27 mandolin kernel:00 4f 00 c2 40 50
   Mar 16 12:37:27 mandolin kernel: sd 7:0:0:0: [sdb]  ASC=0x4 ASCQ=0x1d
   
   That group has several other similar Dell servers, and this is the only
   one logging these.  sdb1 holds /boot and sdb2 is where the lvm keeps
   its information.
   
   Regards,
   
   David Mathog
   mat...@caltech.edu
   Manager, Sequence Analysis Facility, Biology Division, Caltech
   ___
   Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin
   Computing To change your subscription (digest mode or unsubscribe)
   visit http://www.beowulf.org/mailman/listinfo/beowulf
  
  --
  *
  Dr. Jörg Saßmannshausen, MRSC
  University College London
  Department of Chemistry
  Gordon Street
  London
  WC1H 0AJ
  
  email: j.sassmannshau...@ucl.ac.uk
  web: http://sassy.formativ.net
  
  Please avoid sending me Word or PowerPoint attachments.
  See http://www.gnu.org/philosophy/no-word-attachments.html
  
  ___
  Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
  To change your subscription (digest mode or unsubscribe) visit
  http://www.beowulf.org/mailman/listinfo/beowulf

-- 
*
Dr. Jörg Saßmannshausen, MRSC
University College London
Department of Chemistry
Gordon Street
London
WC1H 0AJ 

email: j.sassmannshau...@ucl.ac.uk
web: http://sassy.formativ.net

Please avoid sending me Word or PowerPoint attachments.
See http://www.gnu.org/philosophy/no-word-attachments.html


signature.asc
Description: This is a digitally signed message part

Re: [Beowulf] RAID question

[Jörg Saßmannshausen]

Hi David,

for me it looks like either a controller or disc issue.

I have seen these problems before on SCSI discs when the controller had a 
problem. Depending on the manufacturer it might be a good idea to contact them 
and see if they got more informations here. I have had some problems in the 
past with RAID controllers and the manufacturer here was ever so helpful in 
the diagnosis and repair of a failed RAID5 for example.

So it might be a good idea to try them.

All the best from a cold London

Jörg


On Montag 16 März 2015 mathog wrote:
 Thanks for the feedback.
 
 After copying /boot and /bin from another machine and mucking about with
 grub for far too long (had to edit grub.conf to change virtual disk
 names, and in CentOS's rescue disk it saw the boot disk as hd1, but when
 grub actually started, it saw it as hd0) the system is back on line.
 
 The logs don't show a root command line that specifically took out those
 directories.  They do show a bunch of scripts being run.  My best guess
 is that one of them did something like this:
 
AVAR=`command that failed and returned an empty string`
rm -rf ${AVAR}/b*
 
 It seems unlikely that a low level controller failure would have snipped
 out those files/directories without resulting in a file system that was
 seen as corrupt by fsck.
 
 That said, there is something hardware related going on, since
 /var/log/messages has a lot of these (sorry about the wrap):
 
 Mar 16 12:37:27 mandolin kernel: sd 7:0:0:0: [sdb]  Sense Key :
 Recovered Error [current] [descriptor]
 Mar 16 12:37:27 mandolin kernel: Descriptor sense data with sense
 descriptors (in hex):
 Mar 16 12:37:27 mandolin kernel:72 01 04 1d 00 00 00 0e 09 0c 00
 00 00 00 00 00
 Mar 16 12:37:27 mandolin kernel:00 4f 00 c2 40 50
 Mar 16 12:37:27 mandolin kernel: sd 7:0:0:0: [sdb]  ASC=0x4 ASCQ=0x1d
 
 That group has several other similar Dell servers, and this is the only
 one logging these.  sdb1 holds /boot and sdb2 is where the lvm keeps its
 information.
 
 Regards,
 
 David Mathog
 mat...@caltech.edu
 Manager, Sequence Analysis Facility, Biology Division, Caltech
 ___
 Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
 To change your subscription (digest mode or unsubscribe) visit
 http://www.beowulf.org/mailman/listinfo/beowulf


-- 
*
Dr. Jörg Saßmannshausen, MRSC
University College London
Department of Chemistry
Gordon Street
London
WC1H 0AJ 

email: j.sassmannshau...@ucl.ac.uk
web: http://sassy.formativ.net

Please avoid sending me Word or PowerPoint attachments.
See http://www.gnu.org/philosophy/no-word-attachments.html


signature.asc
Description: This is a digitally signed message part.
___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
http://www.beowulf.org/mailman/listinfo/beowulf