Hi Mani,I tested your code with branch `karlrupp/viennacl-getarray`, which is also in `next` and just got merged to `master`.
Initially I had to fix up a missing double precision pragma for OpenCL, two GCC warnings and a missing local work group size specification in your code to get it to run at all, but then I immediately got the correct results. The fixed code is attached, changes are tagged with '//FIXED'.
Except for possibly machine-specific issues, the problem was the missing local work size specification in your code. Always specify both the local and the global work sizes and make sure that the global sizes are divisible by the respective local work sizes. Good local work sizes are usually in the range 64-256 workers total, hence I set the two local work size dimensions to 8. Also, I recommend some additional checks for 'REAL' to be identical to 'PetscScalar', otherwise you will get garbage because of obvious float<->double incompatibilities at the binary level.
Best regards, Karli On 01/25/2014 10:22 PM, Mani Chandra wrote:
Hi Karl, Thanks for looking into it. Do let me know if there is anything I can do to help you debug this. Attached is the code. Cheers, Mani On Sat, Jan 25, 2014 at 3:14 PM, Karl Rupp <[email protected] <mailto:[email protected]>> wrote: Hi Mani, could you please send me the code including the error checks? Thanks and best regards, Karli On 01/25/2014 10:07 PM, Mani Chandra wrote: Hi Karl, I now checked the error flags of all petsc functions in all functions. I also recompiled petsc with debug. Here's the report: Case 1) Still works. All good. Case 2) This is the case with ComputeResidual using VecGetArrays but with DMSetVecType(da, VECVIENNACL) and DMSetMatType(da, MATAIJVIENNACL). The flags don't show any errors. *The programs proceeds smoothly but the solution is wrong even though SNES is converging.* Case 3) This is the case with ComputeResidualViennaCL using VecViennaCLGetArrays with and without DMSetVecType(da, VECVIENNACL) and DMSetMatType(da, MATAIJVIENNACL). In both cases I get the following error: [0]PETSC ERROR: TSComputeIFunction() line 676 in /home/mc/Downloads/petsc/src/__ts/interface/ts.c [0]PETSC ERROR: SNESTSFormFunction_Theta() line 284 in /home/mc/Downloads/petsc/src/__ts/impls/implicit/theta/theta.__c [0]PETSC ERROR: SNESTSFormFunction() line 3499 in /home/mc/Downloads/petsc/src/__ts/interface/ts.c [0]PETSC ERROR: SNESComputeFunction() line 2089 in /home/mc/Downloads/petsc/src/__snes/interface/snes.c [0]PETSC ERROR: SNESSolve_NEWTONLS() line 175 in /home/mc/Downloads/petsc/src/__snes/impls/ls/ls.c [0]PETSC ERROR: SNESSolve() line 3812 in /home/mc/Downloads/petsc/src/__snes/interface/snes.c [0]PETSC ERROR: TSStep_Theta() line 183 in /home/mc/Downloads/petsc/src/__ts/impls/implicit/theta/theta.__c [0]PETSC ERROR: TSStep() line 2625 in /home/mc/Downloads/petsc/src/__ts/interface/ts.c [0]PETSC ERROR: TSSolve() line 2741 in /home/mc/Downloads/petsc/src/__ts/interface/ts.c [0]PETSC ERROR: main() line 83 in /home/mc/PhD/opencl_tests/__petsc_opencl/petsc_opencl.cpp ------------------------------__------------------------------__-------------- MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD with errorcode -473550369. NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them. ------------------------------__------------------------------__-------------- The error occured at VecViennaCLGetArrayRead/Write in my ComputeResidualViennaCL function. Note that with the petsc debug mode, the code crashes with the above error even if I don't catch the error codes. Might be because of some VecViennaCLGetArray calls inside petsc. Cheers, Mani On Sat, Jan 25, 2014 at 2:48 AM, Karl Rupp <[email protected] <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>> wrote: Hi Mani, please check the return value of *all* function calls from PETSc, e.g. ierr = DMCreateGlobalVector(da, &soln);CHKERRQ(ierr); instead of just DMCreateGlobalVector(da, &soln); Most likely one of the routines threw an error, but your code just kept going, producing wrong results. Best regards, Karli On 01/25/2014 05:35 AM, Mani Chandra wrote: Hi Everyone, I'm trying to use TS with ViennaCL vecs/mats and residual evaluation on device and have encountered some problems. I have attached a small test code that illustrates the issue. The code simply advects a blob diagonally using TS. I have written the residual evaluation function using 1) the usual Petsc vectors (VecGetArray) and 2) using ViennaCL vectors (VecViennaCLGetArrayRead/____Write). Run the code using the following: ./petsc_opencl -ts_monitor -snes_monitor -ts_max_steps 1000 -ts_type theta -ts_dt 10 -snes_rtol 1e-4 -ts_final_time 1000 -ts_monitor_draw_solution Case 1) No ViennaCL anywhere. I simply use the usual Petsc vectors and set the residual evaluation function as ComputeResidual (line no. 55). This case works and the blob is indeed advected as can be seen. (I haven't bothered with the boundaries. The simulation just stops before the blob hits the boundaries). Case 2) We again use the ComputeResidual but now enable ViennaCL vecs and mats (line nos. 48, 49). This case does NOT work. The SNES monitor shows convergence but the solution makes no sense. Case 3) We now use ComputeResidualViennaCL (line no. 56). This does NOT work either with or without enabling the ViennaCL vecs (line nos. 48, 49). Cheers, Mani
petsc_opencl_fixed.tar.gz
Description: GNU Zip compressed data
