Dear Devs, first of all, even after the latest Karl's fix, my GPU-enabled solver (using PETSc's KSPGMRES and PCASM) still gives a substantially different residual after one time step in 1 or more GPUs:
+0.72 (1 GPU) against -1.00 (2-8 GPU). The situtation is even more dramatic for PCBJACOBI: in this case it works the same on 1 GPU and crashes on more GPUs ( "Null argument, when expecting valid pointer! [0]PETSC ERROR: Trying to zero at a null pointer!"). The only thing that could be wrong on my side, I think, is the way I construct the matrix sparsity pattern for the MPIAIJ matrix, as far as I see, since that is the only difference between a "normal"scenario using parallel BAIJ matrices (which perfectly works for me on multiple CPUs). BUT, surprise, if I use PCJACOBI (which is GPU-based as far as I understood), single and multi-GPU residuals are identical! so I start doubting that the problem is due to my sparsity pattern. Perhaps there is something wrong in the PCASM / PCBJACOBI preconditioners when used combined with the GPU-version of the KSP ... what do you think? I remind that I create the matrix for the multi-GPU case with: MatSetType(m_mat, MATMPIAIJCUSP); MatSetBlockSize(m_mat, blockSize); MatMPIAIJSetPreallocation(m_mat, dnz, dnnz, onz, onnz); please confirm if this is right ... I do have another problem though: if I compile the last petsc-dev master with debugging=0 (which is desirable in my case since I'd like to do benchmarks), the petsc tests fail with the messages in the enclosed text file. Any idea why this happens. Mind that I'm using openmpi version 1.6.5 (if this matters) and that by just configuring/compiling with debugging flag = 1, all tests passed with no problem. I guess this could potentially affect many users ... Thanks in advance Andrea -- Dr. Andrea Lani Senior Research Engineer, PhD Aeronautics & Aerospace dept., CFD group Von Karman Institute for Fluid Dynamics Chausse de Waterloo 72, B-1640, Rhode-Saint-Genese, Belgium fax : +32-2-3599600 work : +32-2-3599769 *[email protected] <[email protected]>*
file.err.nodebug_f77
Description: Binary data
