Thank you
Yours sincerely,
TAY wee-beng
On 3/11/2015 12:45 PM, Barry Smith wrote:
On Nov 2, 2015, at 10:37 PM, TAY wee-beng<[email protected]
<mailto:[email protected]>> wrote:
Hi,
I tried :
1. -poisson_pc_gamg_agg_nsmooths 1 -poisson_pc_type gamg
2. -poisson_pc_type gamg
Run with -poisson_ksp_monitor_true_residual
-poisson_ksp_monitor_converged_reason
Does your poisson have Neumann boundary conditions? Do you
have any zeros on the diagonal for the matrix (you shouldn't).
There may be something wrong with your poisson
discretization that was also messing up hypre
Both options give:
1 0.00150000 0.00000000 0.00000000
1.00000000 NaN NaN NaN
M Diverged but why?, time = 2
reason = -9
How can I check what's wrong?
Thank you
Yours sincerely,
TAY wee-beng
On 3/11/2015 3:18 AM, Barry Smith wrote:
hypre is just not scaling well here. I do not know
why. Since hypre is a block box for us there is no way
to determine why the poor scaling.
If you make the same two runs with -pc_type gamg
there will be a lot more information in the log
summary about in what routines it is scaling well or
poorly.
Barry
On Nov 2, 2015, at 3:17 AM, TAY
wee-beng<[email protected]
<mailto:[email protected]>> wrote:
Hi,
I have attached the 2 files.
Thank you
Yours sincerely,
TAY wee-beng
On 2/11/2015 2:55 PM, Barry Smith wrote:
Run (158/2)x(266/2)x(150/2) grid on 8
processes and then (158)x(266)x(150) on 64
processors and send the two -log_summary results
Barry
On Nov 2, 2015, at 12:19 AM, TAY
wee-beng<[email protected]
<mailto:[email protected]>> wrote:
Hi,
I have attached the new results.
Thank you
Yours sincerely,
TAY wee-beng
On 2/11/2015 12:27 PM, Barry Smith wrote:
Run without the -momentum_ksp_view
-poisson_ksp_view and send the new results
You can see from the log summary
that the PCSetUp is taking a much
smaller percentage of the time meaning
that it is reusing the preconditioner
and not rebuilding it each time.
Barry
Something makes no sense with the
output: it gives
KSPSolve 199 1.0
2.3298e+03 1.0 5.20e+09 1.8 3.8e+04
9.9e+05 5.0e+02 90100 66100 24 90100
66100 24 165
90% of the time is in the solve but
there is no significant amount of time
in other events of the code which is
just not possible. I hope it is due to
your IO.
On Nov 1, 2015, at 10:02 PM, TAY
wee-beng<[email protected]
<mailto:[email protected]>> wrote:
Hi,
I have attached the new run with
100 time steps for 48 and 96 cores.
Only the Poisson eqn 's RHS
changes, the LHS doesn't. So if I
want to reuse the preconditioner,
what must I do? Or what must I not do?
Why does the number of processes
increase so much? Is there
something wrong with my coding?
Seems to be so too for my new run.
Thank you
Yours sincerely,
TAY wee-beng
On 2/11/2015 9:49 AM, Barry Smith
wrote:
If you are doing many time
steps with the same linear
solver then you MUST do your
weak scaling studies with MANY
time steps since the setup
time of AMG only takes place
in the first stimestep. So run
both 48 and 96 processes with
the same large number of time
steps.
Barry
On Nov 1, 2015, at 7:35
PM, TAY
wee-beng<[email protected]
<mailto:[email protected]>>
wrote:
Hi,
Sorry I forgot and use the
old a.out. I have attached
the new log for 48cores
(log48), together with the
96cores log (log96).
Why does the number of
processes increase so
much? Is there something
wrong with my coding?
Only the Poisson eqn 's
RHS changes, the LHS
doesn't. So if I want to
reuse the preconditioner,
what must I do? Or what
must I not do?
Lastly, I only simulated 2
time steps previously. Now
I run for 10 timesteps
(log48_10). Is it building
the preconditioner at
every timestep?
Also, what about momentum
eqn? Is it working well?
I will try the gamg later too.
Thank you
Yours sincerely,
TAY wee-beng
On 2/11/2015 12:30 AM,
Barry Smith wrote:
You used gmres with
48 processes but
richardson with 96.
You need to be careful
and make sure you
don't change the
solvers when you
change the number of
processors since you
can get very different
inconsistent results
Anyways all the
time is being spent in
the BoomerAMG
algebraic multigrid
setup and it is is
scaling badly. When
you double the problem
size and number of
processes it went from
3.2445e+01 to
4.3599e+02 seconds.
PCSetUp
3 1.0 3.2445e+01 1.0
9.58e+06 2.0 0.0e+00
0.0e+00 4.0e+00 62 8
0 0 4 62 8 0 0
5 11
PCSetUp
3 1.0 4.3599e+02 1.0
9.58e+06 2.0 0.0e+00
0.0e+00 4.0e+00 85 18
0 0 6 85 18 0 0
6 2
Now is the Poisson
problem changing at
each timestep or can
you use the same
preconditioner built
with BoomerAMG for all
the time steps?
Algebraic multigrid
has a large set up
time that you often
doesn't matter if you
have many time steps
but if you have to
rebuild it each
timestep it is too large?
You might also try
-pc_type gamg and see
how PETSc's algebraic
multigrid scales for
your problem/machine.
Barry
On Nov 1, 2015, at
7:30 AM, TAY
wee-beng<[email protected]
<mailto:[email protected]>>
wrote:
On 1/11/2015 10:00
AM, Barry Smith wrote:
On Oct 31,
2015, at
8:43 PM,
TAY
wee-beng<[email protected]
<mailto:[email protected]>>
wrote:
On
1/11/2015
12:47 AM,
Matthew
Knepley wrote:
On
Sat,
Oct
31,
2015
at
11:34
AM,
TAY
wee-beng<[email protected]
<mailto:[email protected]>>
wrote:
Hi,
I
understand
that
as
mentioned
in the
faq,
due to
the
limitations
in
memory, the
scaling is
not
linear. So,
I am
trying
to
write
a
proposal
to use
a
supercomputer.
Its
specs are:
Compute nodes:
82,944
nodes
(SPARC64
VIIIfx; 16GB
of
memory
per node)
8
cores
/
processor
Interconnect:
Tofu
(6-dimensional
mesh/torus)
Interconnect
Each
cabinet contains
96
computing
nodes,
One of
the
requirement
is to
give
the
performance
of my
current code
with
my
current set
of
data,
and
there
is a
formula to
calculate
the
estimated
parallel
efficiency
when
using
the
new
large
set of
data
There
are 2
ways
to
give
performance:
1.
Strong
scaling,
which
is
defined as
how
the
elapsed time
varies
with
the
number
of
processors
for a
fixed
problem.
2.
Weak
scaling,
which
is
defined as
how
the
elapsed time
varies
with
the
number
of
processors
for a
fixed
problem size
per
processor.
I ran
my
cases
with
48 and
96
cores
with
my
current cluster,
giving
140
and 90
mins
respectively.
This
is
classified
as
strong
scaling.
Cluster specs:
CPU:
AMD
6234
2.4GHz
8
cores
/
processor
(CPU)
6 CPU
/ node
So 48
Cores
/ CPU
Not
sure
abt
the
memory
/ node
The
parallel
efficiency
‘En’
for a
given
degree
of
parallelism
‘n’
indicates
how
much
the
program is
efficiently
accelerated
by
parallel
processing.
‘En’
is
given
by the
following
formulae.
Although
their
derivation
processes
are
different
depending
on
strong
and
weak
scaling,
derived formulae
are the
same.
From
the
estimated
time,
my
parallel
efficiency
using
Amdahl's
law on
the
current old
cluster was
52.7%.
So is
my
results
acceptable?
For
the
large
data
set,
if
using
2205
nodes
(2205X8cores),
my
expected
parallel
efficiency
is
only
0.5%.
The
proposal
recommends
value
of > 50%.
The
problem with
this
analysis
is
that
the
estimated
serial
fraction
from
Amdahl's
Law
changes as
a function
of
problem size,
so you
cannot
take
the
strong
scaling from
one
problem and
apply
it to
another without
a
model
of
this
dependence.
Weak
scaling does
model
changes with
problem size,
so I
would
measure weak
scaling on
your
current
cluster,
and
extrapolate
to the
big
machine.
I
realize that
this
does
not
make
sense
for
many
scientific
applications,
but
neither does
requiring
a
certain parallel
efficiency.
Ok I check
the
results
for my
weak
scaling it
is even
worse for
the
expected
parallel
efficiency. From
the
formula
used, it's
obvious
it's doing
some sort
of
exponential
extrapolation
decrease.
So unless
I can
achieve a
near > 90%
speed up
when I
double the
cores and
problem
size for
my current
48/96
cores
setup,
extrapolating
from about
96 nodes
to 10,000
nodes will
give a
much lower
expected
parallel
efficiency
for the
new case.
However,
it's
mentioned
in the FAQ
that due
to memory
requirement,
it's
impossible
to get
>90% speed
when I
double the
cores and
problem
size (ie
linear
increase
in
performance),
which
means that
I can't
get >90%
speed up
when I
double the
cores and
problem
size for
my current
48/96
cores
setup. Is
that so?
What is the
output of
-ksp_view
-log_summary
on the problem
and then on
the problem
doubled in
size and
number of
processors?
Barry
Hi,
I have attached
the output
48 cores: log48
96 cores: log96
There are 2
solvers - The
momentum linear
eqn uses bcgs,
while the Poisson
eqn uses hypre
BoomerAMG.
Problem size
doubled from
158x266x150 to
158x266x300.
So is it
fair to
say that
the main
problem
does not
lie in my
programming skills,
but rather
the way
the linear
equations
are solved?
Thanks.
Thanks,
Matt
Is it
possible
for
this
type
of
scaling in
PETSc
(>50%), when
using
17640
(2205X8)
cores?
Btw, I
do not
have
access
to the
system.
Sent
using
CloudMagic
Email
--
What
most
experimenters
take
for
granted before
they
begin
their
experiments
is
infinitely
more
interesting
than
any
results to
which
their
experiments
lead.
--
Norbert Wiener
<log48.txt><log96.txt>
<log48_10.txt><log48.txt><log96.txt>
<log96_100.txt><log48_100.txt>
<log96_100_2.txt><log48_100_2.txt>
<log64_100.txt><log8_100.txt>
--
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which
their experiments lead.
-- Norbert Wiener
