Hello James.
Bravo and many thanks for this excellent overview of your activities. Of course the setup
(in your previous message) and the activities are very impressive by themselves.
But in addition, even though your message is not in itself a perl advocacy message, I feel
that it would have its right place in some perl/mod_perl advocacy forum, because it
touches on some general idea which are valid /also/ for perl and mod_perl.
It was very refreshing to read for once a clear exposé of why it is still important
nowadays to think before programming, to program efficiently, and to choose the right tool
for the job at hand (be it perl, mod_perl, or any other) without the kind of off-the-cuff
general a-priori which tend to plague these discussions.
And even though our own (commercial) activities and setups do not have anything even close
to the scope which you describe, I would like to say that the same basic principles which
you mention in your exposé are just as valid when you scale-down as when you scale-up.
("--you can’t just throw memory, CPUs, power at a problem – you have to
think – how can I do what I need to do with the least resources..")
Even when you think of a single server, or a single server rack, at any one period in time
there is always a practical limit as to how much memory or CPUs you can fit in a given
server, or how many servers you can fit in a rack, or how many additional Gb of bandwidth
you can allocate per server, beyond which there is a sudden "quantum jump" as to how
practical and cost-effective a whole project becomes.
In that sense, I particulary enjoyed your examples of the database and of the additional
power line.
On 24.12.2020 02:38, James Smith wrote:
We don’t use perl for everything, yes we use it for web data, yes we still use it as the
glue language in a lot of cases, the most complex stuff is done with C (not even C++ as
that is too slow). Others on site use Python, Java, Rust, Go, PHP, along with looking at
using GPUs in cases where code can be highly parallelised
It is not just one application – but many, many applications… All with a common goal of
understanding the human genome, and using it to assist in developing new understanding and
techniques which can advance health care.
We are a very large sequencing centre (one of the largest in the world) – what I was
pointing out is that you can’t just throw memory, CPUs, power at a problem – you have to
think – how can I do what I need to do with the least resources. Rather than what
resources can I throw at the problem.
Currently we are acting as the central repository for all COVID-19 sequencing in the UK,
along with one of the largest “wet” labs sequencing data for it – and that is half the
sequenced samples in the whole world. The UK is sequencing more COVID-19 genomes a day
than most other countries have sequenced since the start of the pandemic in Feb/Mar. This
has lead to us discovering a new more transmissible version of the virus, and it what part
of the country the different strains are present – no other country in the world has the
information, technology or infrastructure in place to achieve this.
But this is just a small part of the genomic sequencing we are looking at – we
work on:
* other pathogens – e.g. Plasmodium (Malaria);
* cancer genomes (and how effective drugs are);
* are a major part of the Human Cell Atlas which is looking at how the expression of genes
(in the simplest terms which ones are switched on and switched off) are different in
different tissues;
* sequencing the genomes of other animals to understand their evolution;
* and looking at some other species in detail, to see what we can learn from them when
they have defective genes;
Although all these are currently scaled back so that we can work relentlessly to support
the medical teams and other researchers get on top of COVID-19.
What is interesting is that many of the developers we have on campus (well all wfh at the
moment) are all (relatively) old as we learnt to develop code on machines with limited CPU
and limited memory – so that things had to be efficient, had to be compact…. And that is
as important now as it was 20 or 30 years ago – the data we handle is going up faster than
Moore’s Law! Many of us have pride in doing things as efficiently as possible.
It took around 10 years to sequence and assemble the first human genome {well we are still
tinkering with it and filling in the gaps} – now at the institute we can sequence and
assemble around 400 human genomes in a day – to the same quality!
So most of our issues are due to the scale of the problems we face – e.g. the human genome
has 3 billion base-pairs (A, C, G, Ts) , so normal solutions don’t scale to that (once
many years ago we looked at setting up an Oracle database where there was at least 1 row
for every base pair – recording all variants (think of them as spelling mistakes, for
example a T rather than an A, or an extra letter inserted or deleted) for that base pair…
The schema was set up – and then they realised it would take 12 months to load the data
which we had then (which is probably less than a millionth of what we have now)!
Moving compute off site is a problem as the transfer of the level of data we have would
cause a problem – you can’t easily move all the data to the compute – so you have to bring
the compute to the data.
The site I worked on before I became a more general developer was doing that – and the
code that was written 12-15 years ago is actually still going strong – it has seen a few
changes over the year – many displays have had to be redeveloped as the scale of the data
has got so big that even the summary pages we produced 10 years ago have to be summarised
because they are so large.
*From:*Mithun Bhattacharya <mit...@gmail.com>
*Sent:* 24 December 2020 00:06
*To:* mod_perl list <modperl@perl.apache.org>
*Subject:* Re: Confused about two development utils [EXT]
James would you be able to share more info about your setup ?
1. What exactly is your application doing which requires so much memory and CPU - is it
something like gene splicing (no i don't know much about it beyond Jurassic Park :D )
2. Do you feel Perl was the best choice for whatever you are doing and if yes then why ?
How much of your stuff is using mod_perl considering you mentioned not much is web related ?
3. What are the challenges you are currently facing with your implementation ?
On Wed, Dec 23, 2020 at 6:58 AM James Smith <j...@sanger.ac.uk <mailto:j...@sanger.ac.uk>>
wrote:
Oh but memory is a problem – but not if you have just a small cluster of
machines!
Our boxes are larger than that – but they all run virtual machine {only a
small
proportion web related} – machines/memory would rapidly become in our data
centre - we
run VMWARE [995 hosts] and openstack [10,000s of hosts] + a selection of
large memory
machines {measured in TBs of memory per machine }.
We would be looking at somewhere between 0.5 PB and 1 PB of memory – not
just the
price of buying that amount of memory - for many machines we need the
fastest memory
money can buy for the workload, but we would need a lot more CPUs then we
currently
have as we would need a larger amount of machines to have 64GB virtual
machines {we
would get 2 VMs per host. We currently have approx. 1-2000 CPUs running our
hardware
(last time I had a figure) – it would probably need to go to approximately
5-10,000!
It is not just the initial outlay but the environmental and financial cost
of running
that number of machines, and finding space to run them without putting the
cooling
costs through the roof!! That is without considering what additional
constraints on
storage having the extra machines may have (at the last count a year ago we
had over
30 PBytes of storage on side – and a large amount of offsite backup.
We would also stretch the amount of power we can get from the national grid
to power
it all - we currently have 3 feeds from different part of the national grid
(we are
fortunately in position where this is possible) and the dedicated link we
would need
to add more power would be at least 50 miles long!
So - managing cores/memory is vitally important to us – moving to the cloud
is an
option we are looking at – but that is more than 4 times the price of our
onsite
set-up (with substantial discounts from AWS) and would require an upgrade
of our
existing link to the internet – which is currently 40Gbit of data (I think).
Currently we are analysing a very large amounts of data directly linked to
the current
major world problem – this is why the UK is currently being isolated as we
have
discovered and can track a new strain, in near real time – other countries
have no
ability to do this – we in a day can and do handle, sequence and analyse
more samples
than the whole of France has sequenced since February. We probably don’t
have more of
the new variant strain than in other areas of the world – it is just that
we know we
have because of the amount of sequencing and analysis that we in the UK
have done.
*From:*Matthias Peng <pengmatth...@gmail.com
<mailto:pengmatth...@gmail.com>>
*Sent:* 23 December 2020 12:02
*To:* mod_perl list <modperl@perl.apache.org
<mailto:modperl@perl.apache.org>>
*Subject:* Re: Confused about two development utils [EXT]
Today memory is not serious problem, each of our server has 64GB memory.
Forgot to add - so our FCGI servers need a lot (and I mean a lot) more
memory than
the mod_perl servers to serve the same level of content (just in case
memory blows
up with FCGI backends)
-----Original Message-----
From: James Smith <j...@sanger.ac.uk <mailto:j...@sanger.ac.uk>>
Sent: 23 December 2020 11:34
To: André Warnier (tomcat/perl) <a...@ice-sa.com
<mailto:a...@ice-sa.com>>;
modperl@perl.apache.org <mailto:modperl@perl.apache.org>
Subject: RE: Confused about two development utils [EXT]
> This costs memory, and all the more since many perl modules are not
thread-safe, so if you use them in your code, at this moment the only
safe way to
do it is to use the Apache httpd prefork model. This means that each
Apache httpd
child process has its own copy of the perl interpreter, which means
that the
memory used by this embedded perl interpreter has to be counted n times
(as many
times as there are Apache httpd child processes running at any one
time).
This isn’t quite true - if you load modules before the process forks
then they can
cleverly share the same parts of memory. It is useful to be able to
"pre-load"
core functionality which is used across all functions {this is the case
in Linux
anyway}. It also speeds up child process generation as the modules are
already in
memory and converted to byte code.
One of the great advantages of mod_perl is Apache2::SizeLimit which can
blow away
large child process - and then if needed create new ones. This is not
the case
with some of the FCGI solutions as the individual processes can grow if
there is a
memory leak or a request that retrieves a large amount of content (even
if not
served), but perl can't give the memory back. So FCGI processes only
get bigger
and bigger and eventually blow up memory (or hit swap first)
--
The Wellcome Sanger Institute is operated by Genome Research
Limited, a charity
registered in England with number 1021457 and a company registered in
England
with number 2742969, whose registered office is 215 Euston Road,
London, NW1 2
[google.com]
<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.google.com_maps_search_s-2B215-2BEuston-2BRoad-2C-2BLondon-2C-2BNW1-2B2-3Fentry-3Dgmail-26source-3Dg&d=DwMFaQ&c=D7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=oH2yp0ge1ecj4oDX0XM7vQ&m=friR8ykiZ-NWYdX6SrbT_ogNXEVR-4ixdkrhy5khQjA&s=xU3F4xE2ugQuDWHZ4GtDn9mPBCKcJJOI0PYScsSNjSg&e=>BE.
--
The Wellcome Sanger Institute is operated by Genome Research
Limited, a charity registered in England with number 1021457 and a
company registered in England with number 2742969, whose registered
office is 215 Euston Road, London, NW1 2 [google.com]
<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.google.com_maps_search_s-2B215-2BEuston-2BRoad-2C-2BLondon-2C-2BNW1-2B2-3Fentry-3Dgmail-26source-3Dg&d=DwMFaQ&c=D7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=oH2yp0ge1ecj4oDX0XM7vQ&m=friR8ykiZ-NWYdX6SrbT_ogNXEVR-4ixdkrhy5khQjA&s=xU3F4xE2ugQuDWHZ4GtDn9mPBCKcJJOI0PYScsSNjSg&e=>BE.
-- The Wellcome Sanger Institute is operated by Genome Research Limited, a
charity
registered in England with number 1021457 and a company registered in
England with
number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.
-- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity
registered in England with number 1021457 and a company registered in England with number
2742969, whose registered office is 215 Euston Road, London, NW1 2BE.
