[petsc-users] Segmentation Fault in ts tutorial ex23
Hello: I was trying to run example ts/examples/tutorial/ex23 and I seem to get an error similar to the one I've shown below no matter what options I use. For the error below, I simply ran ex23 with no arguments. I've tried running with the released petsc version and the development version from bitbucket. Any suggestions? Thanks Milad [0]PETSC ERROR: - Error Message [0]PETSC ERROR: Argument out of range! [0]PETSC ERROR: New nonzero at (2,9) caused a malloc! [0]PETSC ERROR: [0]PETSC ERROR: Petsc Development HG revision: f86d5620e977ae10ba0f45b74b5f06746e16ab61 HG Date: Mon Nov 19 18:09:55 2012 +0100 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: [0]PETSC ERROR: ./ex23 on a arch-linu named animal5 by milad Mon Nov 26 10:12:42 2012 [0]PETSC ERROR: Libraries linked from /home/milad/petsc/petsc-dev-bb/arch-linux2-c-debug/lib [0]PETSC ERROR: Configure run at Mon Nov 19 13:22:07 2012 [0]PETSC ERROR: Configure options --with-moab=1 --with-moab-dir=/home/milad/Downloads/moab/moab-opt-parallel/install --with-hdf5=1 --with-hdf5-dir=/usr/local/hdf5-1.8.8-parallel --with-netcdf=1 --with-netcdf-dir=/usr/local/netcdf-4.1.3-serial --with-clanguage=C++ [0]PETSC ERROR: [0]PETSC ERROR: MatSetValues_SeqAIJ() line 353 in /home/milad/petsc/petsc-dev-bb/src/mat/impls/aij/seq/aij.c [0]PETSC ERROR: MatSetValues() line 1080 in /home/milad/petsc/petsc-dev-bb/src/mat/interface/matrix.c [0]PETSC ERROR: MatSetValuesLocal() line 1941 in /home/milad/petsc/petsc-dev-bb/src/mat/interface/matrix.c [0]PETSC ERROR: SetUpMatrices() line 396 in src/ts/examples/tutorials/ex23.c [0]PETSC ERROR: main() line 76 in src/ts/examples/tutorials/ex23.c application called MPI_Abort(MPI_COMM_WORLD, 63) - process 0 -- next part -- An HTML attachment was scrubbed... URL: http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20121126/3e21bd85/attachment.html
Manually setting DA min and max indexes
Hello: Is it possible to set the min/max m,n,p indexes when creating a 3D distributed array? I'm writing a 3D code on a structured mesh, and the mesh is partitioned among the processes elsewhere. Assuming that each process already knows what chunk of the mesh it owns, is it possible to give Petsc this information when you create a DA so that the partitioning is consistent between PETSc and the rest of the code? Thank You Milad
Integer DA's
Hello: I was wondering if it was possible to have distributed arrays that contain integers, instead of floating point numbers? If so, is it possible to have DA's represent arrays of any type? Thanks Milad Fatenejad
Integer DA's
Thanks, that answers my question Milad On Mon, Jul 21, 2008 at 12:50 PM, Barry Smith bsmith at mcs.anl.gov wrote: You can compile PETSc so vectors can contain with double precision, single precision or double complex. There is no support for integers. If you are using these as just part of some code then you can just use the double vector and store integers in them. Unless the bulk of your code involves moving integers between processes then you wouldn't even notice any difference in performance. If you entire code involves only moving integers between processes then PETSc is likely not the right tool for you. Barry On Jul 21, 2008, at 11:39 AM, Milad Fatenejad wrote: Hello: I was wondering if it was possible to have distributed arrays that contain integers, instead of floating point numbers? If so, is it possible to have DA's represent arrays of any type? Thanks Milad Fatenejad
Question about DAVecGetArray and DAVecRestoreArray
Hello: The documentation states that to access elements of a vector created with a DA you should use the DAVecGetArray function and you should return the pointer when you are finished with the DAVecRestoreArray function. For my code, I would really like to call DAVecGetArray as soon as the vector is created, and call DAVecRestoreArray right before the vector is destroyed. In other words, I would like to be able to access the elements during the vector's entire life and not have to call these two functions every time I need to access individual elements. I would like to know if having access to the elements of the vector prevents me from using it in some way? I am concerned that there might be some operation that won't work while the array is being accessed, i.e. that DAVecGetArray somehow locks the vector so some other function cannot use it until DAVecRestoreArray is called. Is this the case? Thanks for your help. Milad
2 Questions about DAs
Hello:
Thanks for all of your help, this has helped me tremendously!
Milad
On Mon, May 12, 2008 at 7:22 PM, Barry Smith bsmith at mcs.anl.gov wrote:
A couple of items.
Overlapping communication and computation is pretty much a myth. The CPU
is used by MPI to pack
the messages and put them on the network so it is not available for
computation during this time. Usually
if you try to overlap communication and computation it will end up being
slower and I've never seen it faster.
Vendors will try to trick you into buying a machine by saying it does it,
but it really doesn't. Just forget about trying to do it.
Creating a DA involves a good amount of setup and some communication; it
is fine to use a few DA's
but setting up hundreds of DAs is not a good idea UNLESS YOU DO TONS OF
WORK for each DA.
In your case you are doing just a tiny amount of communication with each
DA so the DA setup time
is dominating.
If you have hundreds of vectors that you wish to communicate AT THE SAME
TIME (seems strange but
I suppose it is possible), then rather than having hundreds of
DAGlobalToLocalBegin/End() in a row
you will want to create an additional meta DA that has the same m,n,p as
the regular DA but has a
dof equal to the number of vectors you wish to communicate at the same
time. Use VecStrideScatterAll()
to get the individual vectors into a meta vector, do the
DAGlobalToLocalBegin/End() on the meta vector
to get the ghost values and then use DAStrideGatherAll() to get the values
into the 322 individual ghosted
vectors. The reason to do it this way is so the values in all the vectors
are all sent together in a single
MPI message instead of the separate message that would needed for each of
the small
DAGlobalToLocalBegin/End().
Barry
On May 12, 2008, at 6:21 PM, Milad Fatenejad wrote:
Hi:
I created a simple test problem that demonstrates the issue. In the
test problem, 100 vectors are created using:
single.cpp: a single distributed array and
multi.cpp: 100 distributed arrays
Some math is performed on the vectors, then they are scattered to
local vectors..
The log summary (running 2 processes) shows that multi.cpp uses more
memory and performs more reductions than single.cpp, which is similar
to the experience I had with my program...
I hope this helps
Milad
On Mon, May 12, 2008 at 3:15 PM, Matthew Knepley knepley at gmail.com
wrote:
On Mon, May 12, 2008 at 3:01 PM, Milad Fatenejad icksa1 at gmail.com
wrote:
Hello:
I've attached the result of two calculations. The file log-multi-da
uses 1 DA for each vector (322 in all) and the file log-single-da
using 1 DA for the entire calculation. When using 322 DA's, about 10x
more time is spent in VecScatterBegin and VecScatterEnd. Both were
running using two processes
I should mention that the source code for these two runs was exactly
the same, I didn't reorder the scatters differently. The only
difference was the number of DAs
Any suggestions? Do you think this is related to the number of DA's,
or something else?
There are vastly different numbers of reductions and much bigger memory
usage.
Please send the code and I will look at it.
Matt
Thanks for your help
Milad
On Mon, May 12, 2008 at 1:56 PM, Matthew Knepley knepley at gmail.com
wrote:
On Mon, May 12, 2008 at 11:02 AM, Milad Fatenejad
mfatenejad at wisc.edu wrote:
Hello:
I have two separate DA questions:
1) I am writing a large finite difference code and would like to
be
able to represent an array of vectors. I am currently doing this
by
creating a single DA and calling DACreateGlobalVector several
times,
but the manual also states that:
PETSc currently provides no container for multiple arrays sharing
the
same distributed array communication; note, however, that the dof
parameter handles many cases of interest.
I also found the following mailing list thread which describes how
to
use the dof parameter to represent several vectors:
http://www-unix.mcs.anl.gov/web-mail-archive/lists/petsc-users/2008/02/msg00040.html
Where the following solution is proposed:
The easiest thing to do in C is to declare a struct:
typedef struct {
PetscScalar v[3];
PetscScalar p;
} Space;
and then cast pointers
Space ***array;
DAVecGetArray(da, u, (void *) array);
array[k][j][i].v *= -1.0;
The problem with the proposed solution, is that they use a struct
to
get the individual values, but what if you don't know the number
of
degrees of freedom at compile time?
It would be nice to get variable structs in C. However, you can just
deference
2 Questions about DAs
Hello: First, I'm having some email problems, so sorry if this shows up a few times... I am using PETSc to write a large multi-physics finite difference code with a lot of opportunity for overlapping computation and communication. Right now, I have created ~100 petsc vectors for storing various quantities, which currently all share a single DA. The problem with this system is that I can only scatter one quantity at a time to update the values of the ghost points. If I try to scatter more than one object at a time, I get the following error: [0]PETSC ERROR: Object is in wrong state! [0]PETSC ERROR: Scatter ctx already in use! It would be really nice to be able to start scattering a vector whenever I am done with a computation, and just finish the scatter whenever I need the vector again. Again, this is impossible because all of the vectors share the same DA. I then reorganized my code, so that each vector had its own DA, however, this led to the program running significantly more slowly (I assume this is just because I have so many vectors). So my first question is: Is there a way to organize the code so I can overlap the scattering of vectors without having a significant performance hit? And on a related note, many times I need to create arrays of vectors. I just discovered the function VecDuplicateVecs (and related functions), which look like performs this operation. Is this the best way to create arrays of vectors? Is there a way to directly get the array from the DA without having to create a vector and duplicate it (I don't see a DACreateGlobalVectorS)? I know it is also possible to do something like this using the DOF parameter in the DACreate call as shown in: http://www-unix.mcs.anl.gov/web-mail-archive/lists/petsc-users/2008/02/msg00040.html Are there any advantages to using dof as opposed to VecDuplicateVecs, etc.? I'd appreciate any help Thank You Milad Fatenejad
2 Questions about DAs
Hi,
Thanks for the reply.
If I have two global vectors from the same DA, say global1 and global2
and I try to scatter to local vectors local1 and local2 using the
command DAGlobalToLocalBegin/End in the following manner:
DAGlobalToLocalBegin(da, global1, INSERT_VALUES, local1);
DAGlobalToLocalEnd(da, global1, INSERT_VALUES, local1);
DAGlobalToLocalBegin(da, global2, INSERT_VALUES, local2);
DAGlobalToLocalEnd(da, global2, INSERT_VALUES, local2);
Everything is fine. If instead I do:
DAGlobalToLocalBegin(da, global1, INSERT_VALUES, local1);
DAGlobalToLocalBegin(da, global2, INSERT_VALUES, local2);
DAGlobalToLocalEnd(da, global1, INSERT_VALUES, local1);
DAGlobalToLocalEnd(da, global2, INSERT_VALUES, local2);
I get the error:
[0]PETSC ERROR: Object is in wrong state!
[0]PETSC ERROR: Scatter ctx already in use!
What I would like to be able to do is overlap the scattering and that
produces the error.
Thanks
Milad
On Mon, May 12, 2008 at 1:43 PM, Amit.Itagi at seagate.com wrote:
Milad,
I have a DA with 6 vectors sharing the DA structure. I defined the DA to
have just 1 DOF. I generated the 6 vectors from that DA. Also, I scatter
the vectors separately. Things seem to work fine.
Thanks
Rgds,
Amit
Milad Fatenejad
mfatenejad at wisc.
edu To
Sent by: petsc-users at mcs.anl.gov
owner-petsc-users cc
@mcs.anl.gov
No Phone Info Subject
Available 2 Questions about DAs
05/12/2008 12:02
PM
Please respond to
petsc-users at mcs.a
nl.gov
Hello:
I have two separate DA questions:
1) I am writing a large finite difference code and would like to be
able to represent an array of vectors. I am currently doing this by
creating a single DA and calling DACreateGlobalVector several times,
but the manual also states that:
PETSc currently provides no container for multiple arrays sharing the
same distributed array communication; note, however, that the dof
parameter handles many cases of interest.
I also found the following mailing list thread which describes how to
use the dof parameter to represent several vectors:
http://www-unix.mcs.anl.gov/web-mail-archive/lists/petsc-users/2008/02/msg00040.html
Where the following solution is proposed:
The easiest thing to do in C is to declare a struct:
typedef struct {
PetscScalar v[3];
PetscScalar p;
} Space;
and then cast pointers
Space ***array;
DAVecGetArray(da, u, (void *) array);
array[k][j][i].v *= -1.0;
The problem with the proposed solution, is that they use a struct to
get the individual values, but what if you don't know the number of
degrees of freedom at compile time?
So my question is two fold:
a) Is there a problem with just having a single DA and calling
DACreateGlobalVector multiple times? Does this affect performance at
all (I have many different vectors)?
b) Is there a way to use the dof parameter when creating a DA when the
number of degrees of freedom is not known at compile time?
Specifically, I would like to be able to access the individual values
of the vector, just like the example shows...
2) The code I am writing has a lot of different parts which present a
lot of opportunities to overlap communication an computation when
scattering vectors to update values in the ghost points. Right now,
all of my vectors (there are ~50 of them) share a single DA because
they all have the same shape. However, by sharing a single DA, I can
only scatter one vector at a time. It would be nice to be able to
start scattering each vector right after I'm done computing it, and
finish scattering it right before I need it again but I can't because
other vectors might need to be scattered in between. I then re-wrote
part of my code so that each vector had its own DA object, but this
ended up being incredibly slow (I assume this is because I have so
many vectors).
My question is, is there a way to scatter multiple vectors
simultaneously without affecting the performance of the code? Does it
make sense to do this?
I'd really appreciate any help...
Thanks
Milad Fatenejad
2 Questions about DAs
Hello:
I've attached the result of two calculations. The file log-multi-da
uses 1 DA for each vector (322 in all) and the file log-single-da
using 1 DA for the entire calculation. When using 322 DA's, about 10x
more time is spent in VecScatterBegin and VecScatterEnd. Both were
running using two processes
I should mention that the source code for these two runs was exactly
the same, I didn't reorder the scatters differently. The only
difference was the number of DAs
Any suggestions? Do you think this is related to the number of DA's,
or something else?
Thanks for your help
Milad
On Mon, May 12, 2008 at 1:56 PM, Matthew Knepley knepley at gmail.com wrote:
On Mon, May 12, 2008 at 11:02 AM, Milad Fatenejad mfatenejad at wisc.edu
wrote:
Hello:
I have two separate DA questions:
1) I am writing a large finite difference code and would like to be
able to represent an array of vectors. I am currently doing this by
creating a single DA and calling DACreateGlobalVector several times,
but the manual also states that:
PETSc currently provides no container for multiple arrays sharing the
same distributed array communication; note, however, that the dof
parameter handles many cases of interest.
I also found the following mailing list thread which describes how to
use the dof parameter to represent several vectors:
http://www-unix.mcs.anl.gov/web-mail-archive/lists/petsc-users/2008/02/msg00040.html
Where the following solution is proposed:
The easiest thing to do in C is to declare a struct:
typedef struct {
PetscScalar v[3];
PetscScalar p;
} Space;
and then cast pointers
Space ***array;
DAVecGetArray(da, u, (void *) array);
array[k][j][i].v *= -1.0;
The problem with the proposed solution, is that they use a struct to
get the individual values, but what if you don't know the number of
degrees of freedom at compile time?
It would be nice to get variable structs in C. However, you can just
deference
the object directly. For example, for 50 degrees of freedom, you can do
array[k][j][i][47] *= -1.0;
So my question is two fold:
a) Is there a problem with just having a single DA and calling
DACreateGlobalVector multiple times? Does this affect performance at
all (I have many different vectors)?
These are all independent objects. Thus, by itself, creating any number of
Vecs does nothing to performance (unless you start to run out of memory).
b) Is there a way to use the dof parameter when creating a DA when the
number of degrees of freedom is not known at compile time?
Specifically, I would like to be able to access the individual values
of the vector, just like the example shows...
see above.
2) The code I am writing has a lot of different parts which present a
lot of opportunities to overlap communication an computation when
scattering vectors to update values in the ghost points. Right now,
all of my vectors (there are ~50 of them) share a single DA because
they all have the same shape. However, by sharing a single DA, I can
only scatter one vector at a time. It would be nice to be able to
start scattering each vector right after I'm done computing it, and
finish scattering it right before I need it again but I can't because
other vectors might need to be scattered in between. I then re-wrote
part of my code so that each vector had its own DA object, but this
ended up being incredibly slow (I assume this is because I have so
many vectors).
The problem here is that buffering will have to be done for each outstanding
scatter. Thus I see two resolutions:
1) Duplicate the DA scatter for as many Vecs as you wish to scatter at once.
This is essentially what you accomplish with separate DAs.
2) You the dof method. However, this scatter ALL the vectors every time.
I do not understand what performance problem you would have with multiple
DAs. With any performance questions, we suggest sending the output of
-log_summary so we have data to look at.
Matt
My question is, is there a way to scatter multiple vectors
simultaneously without affecting the performance of the code? Does it
make sense to do this?
I'd really appreciate any help...
Thanks
Milad Fatenejad
--
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which
their experiments lead.
-- Norbert Wiener
-- next part --
A non-text attachment was scrubbed...
Name: log-multi-da
Type: application/octet-stream
Size: 11372 bytes
Desc: not available
URL:
http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20080512/b45b0f2e/attachment.obj
-- next part --
A non-text attachment was scrubbed...
Name: log-single
2 Questions about DAs
Hi Amit: I was thinking of the following situation. There are two vectors global/local1 and global/local2. Function f1 modifies the values of global1 and global2. Function f3 requires the updated ghost point values. Function f2 doesn't depend on either vector at all, but is really computationally intensive I would like to do the following: f1(global1, global2)// Both vectors modified, I need to scatter before calling f3() DAGlobalToLocalBegin(da, global1, INSERT_VALUES, local1); DAGlobalToLocalBegin(da, global2, INSERT_VALUES, local2); f2() // function that takes a really long time DAGlobalToLocalEnd(da, global1, INSERT_VALUES, local1); DAGlobalToLocalEnd(da, global2, INSERT_VALUES, local2); f3(local1, local2) // Needs the ghost values If I don't overlap the scattering, I end up with something like this: f1(global1, global2)// Both vectors modified, I need to scatter before calling f3() DAGlobalToLocalBegin(da, global1, INSERT_VALUES, local1); f2() // function that takes a really long time DAGlobalToLocalEnd(da, global1, INSERT_VALUES, local1); DAGlobalToLocalBegin(da, global2, INSERT_VALUES, local2); // Nothing left to do, just wait for scattering to end DAGlobalToLocalEnd(da, global2, INSERT_VALUES, local2); f3(local1, local2) // Needs the ghost values In the second case, there is nothing left to do while the second vector scatters and it seems like I just have to wait for this to occur. Ideally, I would like to scatter both vectors while waiting for f2() to finish... I'm a little new to all of this, so let me know if my understanding is just wrong... Thanks Milad On Mon, May 12, 2008 at 2:51 PM, Amit.Itagi at seagate.com wrote: I don't understand your motivation for trying to have two consecutive scatterBegin before the scatterEnd of the first. I think that the scatterBegin and the corresponding scatterEnd need to thought of as a single scatter operation. Infact, I don't understand the concept of overlapping the scattering. Thanks Rgds, Amit Milad Fatenejad icksa1 at gmail.com To Sent by: petsc-users at mcs.anl.gov owner-petsc-users cc @mcs.anl.gov No Phone Info Subject Available Re: 2 Questions about DAs 05/12/2008 02:58 PM Please respond to petsc-users at mcs.a nl.gov Hi, Thanks for the reply. If I have two global vectors from the same DA, say global1 and global2 and I try to scatter to local vectors local1 and local2 using the command DAGlobalToLocalBegin/End in the following manner: DAGlobalToLocalBegin(da, global1, INSERT_VALUES, local1); DAGlobalToLocalEnd(da, global1, INSERT_VALUES, local1); DAGlobalToLocalBegin(da, global2, INSERT_VALUES, local2); DAGlobalToLocalEnd(da, global2, INSERT_VALUES, local2); Everything is fine. If instead I do: DAGlobalToLocalBegin(da, global1, INSERT_VALUES, local1); DAGlobalToLocalBegin(da, global2, INSERT_VALUES, local2); DAGlobalToLocalEnd(da, global1, INSERT_VALUES, local1); DAGlobalToLocalEnd(da, global2, INSERT_VALUES, local2); I get the error: [0]PETSC ERROR: Object is in wrong state! [0]PETSC ERROR: Scatter ctx already in use! What I would like to be able to do is overlap the scattering and that produces the error. Thanks Milad On Mon, May 12, 2008 at 1:43 PM, Amit.Itagi at seagate.com wrote: Milad, I have a DA with 6 vectors sharing the DA structure. I defined the DA to have just 1 DOF. I generated the 6 vectors from that DA. Also, I scatter the vectors separately. Things seem to work fine. Thanks Rgds, Amit Milad Fatenejad mfatenejad at wisc. edu To Sent by: petsc-users at mcs.anl.gov owner-petsc-users cc @mcs.anl.gov No Phone Info Subject Available 2 Questions about DAs 05/12/2008 12:02 PM Please respond to petsc-users at mcs.a nl.gov Hello: I have two separate DA questions: 1) I am writing a large finite difference code and would like to be able to represent an array of vectors. I am currently doing this by creating a single DA and calling DACreateGlobalVector several times, but the manual also states that: PETSc currently provides no container for multiple arrays sharing the same distributed array communication; note, however, that the dof parameter handles many cases of interest. I also
2 Questions about DAs
Hi Matt:
The code is several thousand lines long, requires many external
libraries and is generally very messy right now. I'd rather not send
it because I wouldn't want to take up too much of your time. I think I
will try to go back and try set up some simpler problems to test the
difference between 1 vs. many DA's, and will write back if I have the
same issue.
Thank you
Milad
On Mon, May 12, 2008 at 3:15 PM, Matthew Knepley knepley at gmail.com wrote:
On Mon, May 12, 2008 at 3:01 PM, Milad Fatenejad icksa1 at gmail.com wrote:
Hello:
I've attached the result of two calculations. The file log-multi-da
uses 1 DA for each vector (322 in all) and the file log-single-da
using 1 DA for the entire calculation. When using 322 DA's, about 10x
more time is spent in VecScatterBegin and VecScatterEnd. Both were
running using two processes
I should mention that the source code for these two runs was exactly
the same, I didn't reorder the scatters differently. The only
difference was the number of DAs
Any suggestions? Do you think this is related to the number of DA's,
or something else?
There are vastly different numbers of reductions and much bigger memory
usage.
Please send the code and I will look at it.
Matt
Thanks for your help
Milad
On Mon, May 12, 2008 at 1:56 PM, Matthew Knepley knepley at gmail.com
wrote:
On Mon, May 12, 2008 at 11:02 AM, Milad Fatenejad mfatenejad at
wisc.edu wrote:
Hello:
I have two separate DA questions:
1) I am writing a large finite difference code and would like to be
able to represent an array of vectors. I am currently doing this by
creating a single DA and calling DACreateGlobalVector several times,
but the manual also states that:
PETSc currently provides no container for multiple arrays sharing
the
same distributed array communication; note, however, that the dof
parameter handles many cases of interest.
I also found the following mailing list thread which describes how
to
use the dof parameter to represent several vectors:
http://www-unix.mcs.anl.gov/web-mail-archive/lists/petsc-users/2008/02/msg00040.html
Where the following solution is proposed:
The easiest thing to do in C is to declare a struct:
typedef struct {
PetscScalar v[3];
PetscScalar p;
} Space;
and then cast pointers
Space ***array;
DAVecGetArray(da, u, (void *) array);
array[k][j][i].v *= -1.0;
The problem with the proposed solution, is that they use a struct to
get the individual values, but what if you don't know the number of
degrees of freedom at compile time?
It would be nice to get variable structs in C. However, you can just
deference
the object directly. For example, for 50 degrees of freedom, you can do
array[k][j][i][47] *= -1.0;
So my question is two fold:
a) Is there a problem with just having a single DA and calling
DACreateGlobalVector multiple times? Does this affect performance at
all (I have many different vectors)?
These are all independent objects. Thus, by itself, creating any
number of
Vecs does nothing to performance (unless you start to run out of
memory).
b) Is there a way to use the dof parameter when creating a DA when
the
number of degrees of freedom is not known at compile time?
Specifically, I would like to be able to access the individual
values
of the vector, just like the example shows...
see above.
2) The code I am writing has a lot of different parts which present
a
lot of opportunities to overlap communication an computation when
scattering vectors to update values in the ghost points. Right now,
all of my vectors (there are ~50 of them) share a single DA because
they all have the same shape. However, by sharing a single DA, I can
only scatter one vector at a time. It would be nice to be able to
start scattering each vector right after I'm done computing it, and
finish scattering it right before I need it again but I can't
because
other vectors might need to be scattered in between. I then re-wrote
part of my code so that each vector had its own DA object, but this
ended up being incredibly slow (I assume this is because I have so
many vectors).
The problem here is that buffering will have to be done for each
outstanding
scatter. Thus I see two resolutions:
1) Duplicate the DA scatter for as many Vecs as you wish to scatter
at once.
This is essentially what you accomplish
2 Questions about DAs
Hi:
I created a simple test problem that demonstrates the issue. In the
test problem, 100 vectors are created using:
single.cpp: a single distributed array and
multi.cpp: 100 distributed arrays
Some math is performed on the vectors, then they are scattered to
local vectors..
The log summary (running 2 processes) shows that multi.cpp uses more
memory and performs more reductions than single.cpp, which is similar
to the experience I had with my program...
I hope this helps
Milad
On Mon, May 12, 2008 at 3:15 PM, Matthew Knepley knepley at gmail.com wrote:
On Mon, May 12, 2008 at 3:01 PM, Milad Fatenejad icksa1 at gmail.com wrote:
Hello:
I've attached the result of two calculations. The file log-multi-da
uses 1 DA for each vector (322 in all) and the file log-single-da
using 1 DA for the entire calculation. When using 322 DA's, about 10x
more time is spent in VecScatterBegin and VecScatterEnd. Both were
running using two processes
I should mention that the source code for these two runs was exactly
the same, I didn't reorder the scatters differently. The only
difference was the number of DAs
Any suggestions? Do you think this is related to the number of DA's,
or something else?
There are vastly different numbers of reductions and much bigger memory
usage.
Please send the code and I will look at it.
Matt
Thanks for your help
Milad
On Mon, May 12, 2008 at 1:56 PM, Matthew Knepley knepley at gmail.com
wrote:
On Mon, May 12, 2008 at 11:02 AM, Milad Fatenejad mfatenejad at
wisc.edu wrote:
Hello:
I have two separate DA questions:
1) I am writing a large finite difference code and would like to be
able to represent an array of vectors. I am currently doing this by
creating a single DA and calling DACreateGlobalVector several times,
but the manual also states that:
PETSc currently provides no container for multiple arrays sharing
the
same distributed array communication; note, however, that the dof
parameter handles many cases of interest.
I also found the following mailing list thread which describes how
to
use the dof parameter to represent several vectors:
http://www-unix.mcs.anl.gov/web-mail-archive/lists/petsc-users/2008/02/msg00040.html
Where the following solution is proposed:
The easiest thing to do in C is to declare a struct:
typedef struct {
PetscScalar v[3];
PetscScalar p;
} Space;
and then cast pointers
Space ***array;
DAVecGetArray(da, u, (void *) array);
array[k][j][i].v *= -1.0;
The problem with the proposed solution, is that they use a struct to
get the individual values, but what if you don't know the number of
degrees of freedom at compile time?
It would be nice to get variable structs in C. However, you can just
deference
the object directly. For example, for 50 degrees of freedom, you can do
array[k][j][i][47] *= -1.0;
So my question is two fold:
a) Is there a problem with just having a single DA and calling
DACreateGlobalVector multiple times? Does this affect performance at
all (I have many different vectors)?
These are all independent objects. Thus, by itself, creating any
number of
Vecs does nothing to performance (unless you start to run out of
memory).
b) Is there a way to use the dof parameter when creating a DA when
the
number of degrees of freedom is not known at compile time?
Specifically, I would like to be able to access the individual
values
of the vector, just like the example shows...
see above.
2) The code I am writing has a lot of different parts which present
a
lot of opportunities to overlap communication an computation when
scattering vectors to update values in the ghost points. Right now,
all of my vectors (there are ~50 of them) share a single DA because
they all have the same shape. However, by sharing a single DA, I can
only scatter one vector at a time. It would be nice to be able to
start scattering each vector right after I'm done computing it, and
finish scattering it right before I need it again but I can't
because
other vectors might need to be scattered in between. I then re-wrote
part of my code so that each vector had its own DA object, but this
ended up being incredibly slow (I assume this is because I have so
many vectors).
The problem here is that buffering will have to be done for each
outstanding
scatter. Thus I see two resolutions:
1) Duplicate the DA scatter
