Dear Karli, Thanks for your suggestions! Right now I test 'src/ksp/ksp/examples/tutorials/ex11.c' with n=2 I rewrite the sample into a c++ class, and run it as mpirun -np 2 ..... I notice that the vector x is stored in 2 processors. But the matrix A seems not as you told me 'MatView should report your matrix R as being equally split over the MPI ranks.'
Norm of error 1.19381e-16 iterations 4 Vector Object: 2 MPI processes type: mpi Process [0] 4.16334e-17 0 + 1.38778e-17 i Process [1] 0 1.11022e-16 Matrix Object: 1 MPI processes type: mpiaij row 0: (0, -7.11111 + 0.0800036 i) (1, -1) (2, -1) row 1: (0, -1) (1, -7.11111 + 0.00111359 i) (3, -1) row 2: (0, -1) (2, -7.11111 + 0.0601271 i) (3, -1) row 3: (1, -1) (2, -1) (3, -7.11111 + 0.0943018 i) I thought Matrix Object should have 2 MPI processes. And again thanks for your help. Best regards, Yan On Sun, Jan 19, 2014 at 6:34 PM, Karl Rupp <[email protected]> wrote: > Hi, > > > On 01/19/2014 10:39 AM, yan chang wrote: > >> I have used Eigen (a C++ template library ) to set up a spin dynamics >> simulation program, and now I would like to speed up the simulation in >> parallel using PETSc, mostly use sparse Mat and Vec operations. After >> some rewrite work (just replace the matrix operations by PETSc), I can >> firstly set up a spin system and calculate its relaxation matrix R. >> >> PetscErrorCode ierr; >> PetscInitialize(&argc, &argv, (char*) 0, help); >> SpinSystem sys; >> sys.set_magnet(14.1); >> sys.set_isotopes("1H 13C"); >> sys.set_basis("sphten_liouv"); >> sys.set_zeeman_eigs("(7 15 -22) (11 18 -29)"); >> sys.set_zeeman_euler("(1.0472 0.7854 0.6283) (0.5236 0.4488 0.3927)"); >> sys.set_coordinates("1 (0 0 0) 2 (0 0 1.02)"); >> sys.create(); >> Relaxation relax(sys); >> Mat R = relax.Rmat(); >> ierr = MatView(R, PETSC_VIEWER_STDOUT_WORLD); >> ierr = MatDestroy(&R); >> CHKERRQ(ierr); >> ierr = PetscFinalize(); >> >> And run it with mpirun -np 1 ./Release/spin-scenarios-petsc it is ok. >> > > Please don't forget to check the error code through CHKERRQ after *each* > function call. > > > > However if I set the np to be 2 or more, the program would generate the >> same number of the spin system object and it is time consuming to get >> the result. I am very confused that in the implementation the Mat and >> Vec variables are not sequential like Eigen, so could you give me some >> advise how to organize my program based on PETSc? >> > > a) Did you verify that you are using the correct 'mpirun'? You need to use > the one you specified for the PETSc installation. Otherwise, the same > executable will be called multiple times in sequential mode. Hint: MatView > should report your matrix R as being equally split over the MPI ranks. > > b) To get reasonable performance and scalability you need to consider the > multiple MPI ranks in your SpinSystem and Relaxation objects already. Each > MPI rank should then set up its part of the global system. Have a look at > e.g. the examples in src/ksp/ksp/examples/tutorials/ or > src/snes/examples/tutorials to get an idea how this is done. Keep in mind > that the distributed matrix is decomposed by rows, i.e. each MPI rank > stores a horizontal strip of the full matrix R, but PETSc will take care of > the details if you set values in rows which are not locally owned. > > > > I am a beginner of both OPENMPI and PETSc. Also, after the simple set up >> part, there will be some more time consuming work to do, that's why I >> switch from Eigen to PETSc. >> Thanks a lot! >> > > My advice is to familiarize yourself with some of the simpler tutorials on > how simpler systems are set up in parallel. Then, incrementally migrate > these techniques over to your use case. > > Best regards, > Karli > >
