Hi Satish,
thanks for the pull request. I approve the changes, improved appending the
-Wno-deprecated-gpu-targets to also work on my machine, and have merged
everything to next.
* My fixes should alleviate some of the CUSP installation issues. I
don't know enough about CUSP interface wrt the useful features vs
other burderns - and if its good to drop it or not. [If needed - we
can add in more version dependencies in configure]
This should be fine for now. In the long term CUSP may be completely
superseded by NVIDIA's AMGX. Let's see how things develop...
* Wrt CUDA - currently my test is with CUDA-7.5. I can try migrating a
couple of tests to CUDA-9.1 [on frog]. But what about older
releases? Any reason we should drop them? I.e any reason to up the
following values?
self.CUDAMinVersion = '5000' # Minimal cuda version is 5.0
self.CUSPMinVersion = '400' # Minimal cusp version is 0.4
See the answer here for a list of CUDA capabilities and defaults:
https://stackoverflow.com/questions/28932864/cuda-compute-capability-requirements
We definitely don't need to support compute architecture 1.x (~10 years old),
as there is no double precision support and hence fairly useless for our
purposes. Thus, we should be absolutely fine with requiring CUDA 7.0 or
higher.
We do change it for complex build [we don't have a test for this case]
if self.defaultScalarType.lower() == 'complex': self.CUDAMinVersion =
'7050'
I don't remember the exact reason, but I remember that there is one for
requiring CUDA 7.5 here. Let's use CUDA 7.5 as the minimum for both real and
complex then?
* Our test GUP is M2090 - with Compute capability (version) 2.0.
CUDA-7.5 works on it. CUDA-8 gives deprecated warnings. CUDA-9 does
not work? So what do we do for such old hardware? Do we keep
CUDA-7.5 is the minimum supported version for extended time? [At
some point we could switch to minimum version CUDA-8 - if we can get
rid of the warnings]
Your PR silences the deprecation warnings.
Compute capability 2.0 is fine for our tests for some time to come. We should
certainly upgrade at some point, yet my experience with GPUs is that older
GPUs are actually the better test environment, as they tend to reveal bugs
quicker than newer hardware.
Best regards,
Karli
* BTW: Wrt --with-cuda-arch, I'm hoping we can get rid of it in favor
of CUDAFLAGS [with defaults similar to CFLAGS defaults] - but its
not clear if I can easily untangle the dependencies we have [wrt CPP
- and others]
Or can we get rid of this default alltogether [currently
-arch=sm_20] - and expect nvcc to have sane defaults? Then we can
probably eliminate all this complicated code. [If cuda-7.5 and higer
do this properly - we could use that as the minimum supported version?]
Satish
On Sat, 10 Mar 2018, Karl Rupp wrote:
Hi all,
a couple of notes here, particularly for Manuel:
* CUSP is repeatedly causing such installation problems, hence we will
soon
drop it as a vector backend and instead only provide a native
CUBLAS/CUSPARSE-based backend.
* you can use this native CUDA backend already now. Just configure with
only
--with-cuda=1 --with-cuda-arch=sm_60 (sm_30 should also work and is
compatible
with Tesla K20 GPUs you may find on other clusters).
* The multigrid preconditioner from CUSP is selected via
-pc_type sa_cusp
Make sure you also use -vec_type cusp -mat_type aijcusp
If you don't need the multigrid preconditioner from CUSP, please
just reconfigure and use the native CUDA-backend with -vec_type cuda
-mat_type
aijcusparse
* Right now only one of {native CUDA, CUSP, ViennaCL} can be activated at
configure time. This will be fixed later this month.
If you're looking for a GPU-accelerated multigrid preconditioner: I just
heard
yesterday that NVIDIA's AMGX is now open source. I'll provide a wrapper
within
PETSc soon.
As Matt already said: Don't expect much more than a modest speedup over
your
existing CPU-based code - provided that your setup is GPU-friendly and your
problem size is appropriate.
Best regards,
Karli
On 03/10/2018 03:38 AM, Satish Balay wrote:
I've updated configure so that --download-cusp gets the
correct/compatible cusp version - for cuda 7,8 vs 9
The changes are in branch balay/cuda-cusp-cleanup - and merged to next.
Satish
On Wed, 7 Mar 2018, Satish Balay wrote:
--download-cusp gets hardly ever used so likely broken.
It needs to be updated to somehow use the correct cusp version based
on the cuda version thats being used.
[and since we can't easily check for cusp compatibility - we should
probably
remove checkCUSPVersion() code]
When using Cuda-9 - you can try options:
--download-cusp=1 --download-cusp-commit=116b090
