With the current implementation the following can happen (v is of type
VECCUSP):
- Originally data on GPU, v.valid_GPU_array == PETSC_CUSP_GPU
- a call to VecPlaceArray(v, arr) unplaces the data on the host and sets
v.valid_CPU_array=CPU. Note that the GPU data does not get stashed.
- subsequent accesses of the GPU data will clobber the data that was
there before VecPlaceArray.
I think there are two possible solutions:
- In VecPlaceArray_SeqCUSP we allocate a new array on the GPU and stash
the current values.
- We do a GPU->CPU synchronization in VecPlaceArray_SeqCUSP to make sure
that the data on the CPU is up to date.
It's a space/time tradeoff. Also, the first option further complicates
the caching mechanism. I think the caching mechanism is already too
complicated (nearly every bug I encounter with the CUDA stuff is related
to caching). The second option allows us to more easily reuse
VecPlaceArray_Seq. And we don't have to juggle a GPU unplaced array in
addition to the host side unplaced array. I'd therefore propose that we
take the hit of a GPU->CPU data synchronization. Not that this
synchronization only incurs in a PCIe data transfer if the data on the
CPU is stale.
But all of this is only needed if the semantics of
VecPlaceArray/VecResetArray array is to preserve the contents of the
unplaced array.
Cheers,
Dominic
On 08/07/2015 09:23 PM, Barry Smith wrote:
On Aug 7, 2015, at 7:50 PM, Dominic Meiser <dmei...@txcorp.com> wrote:
FYI I've opened a pull request that addresses this issue.
While going through the code I ran into a question regarding the semantics of
VecPlaceArray and VecResetArray: Is the contents of the "unplaced" array
supposed to be preserved so that the vector is completely restored upon calling
VecResetArray? With the current implementation of VecPlaceArray_SeqCUSP and
VecResetArray_SeqCUSP a situation can occur where the contents of the unplaced array gets
clobbered. Does this need to be fixed?
Hmm, I think I always assumed the "unplaced" array was inaccessible during
the time it is unplaced (since there is no public pointer to the array), this would mean
that the values there shouldn't change. What are the exact details of how they can get
changed?
Thanks
Barry
Cheers,
Dominic
On 08/07/2015 02:42 PM, Barry Smith wrote:
Guardians of CUDA/GPUs
http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2015/08/06/examples_master_arch-cuda-double_bb-proxy.log
http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2015/08/06/examples_master_arch-cuda_bb-proxy.log
search for ex2_bjacobi
note that this example does not fail in non CUDA builds.
For some reason the iterative solver thinks it converges in 0 iterations but
the answer is completely wrong.
Barry
--
Dominic Meiser
Tech-X Corporation
5621 Arapahoe Avenue
Boulder, CO 80303
USA
Telephone: 303-996-2036
Fax: 303-448-7756
www.txcorp.com
--
Dominic Meiser
Tech-X Corporation
5621 Arapahoe Avenue
Boulder, CO 80303
USA
Telephone: 303-996-2036
Fax: 303-448-7756
www.txcorp.com
