Sometimes computing can be an experimental science. Run the same
size DA on 1, 2, 4, 8, 16, 32 processes and gather the information
about memory usage and make a little table.
Here is what you should find. The amount of memory depends on the
local size of the array, which for your example below is
1000*1000*1000 on one process. Thus you will see that as you increase
the number of processes you'll see the space needed per process for
the DA decreases. It increases from 1 process to 2 because it needs
all the ghost point data and VecScatter that are not needed on 1.
Note also on one process a SINGLE vector for this size mesh is 8
gigabytes so the DA is really not much of pig since it is less than
one vector.
Barry
On Nov 22, 2009, at 9:34 PM, Denis Teplyashin wrote:
> So this sort of memory consumption is expected? Is it possible to
> reduce is somehow? I'm not sure about underlying petsc object but it
> looks like these additional objects require more memory than the
> actual vector itself.
>
> Cheers,
> Denis
>
> Matthew Knepley wrote:
>>
>> It is not simple, but it is scalable, meaning in the limit of large
>> N, the memory will be constant on
>> each processor. When it is created, the VecScatter objects mapping
>> global to local vecs are created.
>>
>> Matt
>>
>> On Sun, Nov 22, 2009 at 8:29 PM, Denis Teplyashin
>> <denist at al.com.au> wrote:
>> Hi guys,
>>
>> I'm a bit confused with distributed array memory consumption. I did
>> a simple test like this one:
>> ierr = DACreate3d(PETSC_COMM_WORLD, DA_NONPERIODIC,
>> DA_STENCIL_BOX, 1000, 1000, 1000, PETSC_DECIDE, PETSC_DECIDE,
>> PETSC_DECIDE, 1, 1, PETSC_NULL, PETSC_NULL, PETSC_NULL , &da);
>> and then checked memory with PetscMemoryGetCurrentUsage and
>> PetscMemoryGetMaximumUsage. Running this test using mpi on one core
>> gives me this result: current usage 3818Mb and maximum usage
>> 7633Mb. And this is the result after creating just a DA without
>> actual vectors. Running the same test on two cores gives me even
>> more interesting result: rank 0 - 9552/11463Mb and rank 1 -
>> 5735/5732Mb.
>> Is it what i should expect in general or am i doing something
>> wrong? Is there a simple formula which could show how much memory i
>> would need to allocate and array with given resolution?
>>
>> Thanks in advance,
>> Denis
>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to
>> which their experiments lead.
>> -- Norbert Wiener
>
