As Matt said, high- frequency Helmholtz is very hard, low-frequency is doable by using a larger coarse grid (you have a tiny coarse grid).
On Sat, Apr 13, 2024 at 3:04 PM Matthew Knepley <[email protected]> wrote: > On Fri, Apr 12, 2024 at 8:19 PM Ng, Cho-Kuen via petsc-users < > [email protected]> wrote: > >> Mark. The FEM problem is of high-frequency Helmholtz type for the Maxwell >> equation. The combination "-ksp_type groppcg -pc_type gamg" worked for this >> small problem and the results agreed well with those obtained from the >> direct solver >> ZjQcmQRYFpfptBannerStart >> This Message Is From an External Sender >> This message came from outside your organization. >> >> ZjQcmQRYFpfptBannerEnd >> Mark. >> >> The FEM problem is of high-frequency Helmholtz type for the Maxwell >> equation. The combination "-ksp_type groppcg -pc_type gamg" worked for this >> small problem and the results agreed well with those obtained from the >> direct solver MUMPS. However, this ksp-pc combination failed to converge >> for much larger problem sizes. What would be a good combination to solve >> this kind of large-scale problems where direct solvers would not be able to >> solve? >> > > Solving high frequency Helmholtz is a hard research problem. There are no > easy answers. I know of no > real solutions for C0 FEM. it seems to be a bad discretization for this > from the start. I have not seen anything reliable that is not mostly > direct, but this is not what I do for a living so something might exist, > although not in any publicly available software. > > Thanks, > > Matt > > >> Thanks, >> Cho >> ------------------------------ >> *From:* Mark Adams <[email protected]> >> *Sent:* Friday, April 12, 2024 4:52 PM >> *To:* Barry Smith <[email protected]> >> *Cc:* Ng, Cho-Kuen <[email protected]>; petsc-users < >> [email protected]>; Jacob Faibussowitsch <[email protected]> >> *Subject:* Re: [petsc-users] Using PETSc GPU backend >> >> As Barry said, this is a bit small but the performance looks reasonable. >> The solver does very badly, mathematically. >> I would try hypre to get another data point. >> You could also try 'cg' to check that the pipelined version is not a >> problem. >> Mark >> >> On Fri, Apr 12, 2024 at 3:54 PM Barry Smith <[email protected]> wrote: >> >> 800k is a pretty small problem for GPUs. We would need to see the runs >> with output from -ksp_view -log_view to see if the timing results are >> reasonable. On Apr 12, 2024, at 1: 48 PM, Ng, Cho-Kuen <cho@ slac. >> stanford. edu> wrote: I performed >> ZjQcmQRYFpfptBannerStart >> This Message Is From an External Sender >> This message came from outside your organization. >> >> ZjQcmQRYFpfptBannerEnd >> >> 800k is a pretty small problem for GPUs. >> >> We would need to see the runs with output from -ksp_view -log_view to >> see if the timing results are reasonable. >> >> On Apr 12, 2024, at 1:48 PM, Ng, Cho-Kuen <[email protected]> wrote: >> >> I performed tests on comparison using KSP with and without cuda backend >> on NERSC's Perlmutter. For a finite element solve with 800k degrees of >> freedom, the best times obtained using MPI and MPI+GPU were >> >> o MPI - 128 MPI tasks, 27 s >> >> o MPI+GPU - 4 MPI tasks, 4 GPU's, 32 s >> >> Is that the performance one would expect using the hybrid mode of >> computation. Attached image shows the scaling on a single node. >> >> Thanks, >> Cho >> ------------------------------ >> *From:* Ng, Cho-Kuen <[email protected]> >> *Sent:* Saturday, August 12, 2023 8:08 AM >> *To:* Jacob Faibussowitsch <[email protected]> >> *Cc:* Barry Smith <[email protected]>; petsc-users < >> [email protected]> >> *Subject:* Re: [petsc-users] Using PETSc GPU backend >> >> Thanks Jacob. >> ------------------------------ >> *From:* Jacob Faibussowitsch <[email protected]> >> *Sent:* Saturday, August 12, 2023 5:02 AM >> *To:* Ng, Cho-Kuen <[email protected]> >> *Cc:* Barry Smith <[email protected]>; petsc-users < >> [email protected]> >> *Subject:* Re: [petsc-users] Using PETSc GPU backend >> >> > Can petsc show the number of GPUs used? >> >> -device_view >> >> Best regards, >> >> Jacob Faibussowitsch >> (Jacob Fai - booss - oh - vitch) >> >> > On Aug 12, 2023, at 00:53, Ng, Cho-Kuen via petsc-users < >> [email protected]> wrote: >> > >> > Barry, >> > >> > I tried again today on Perlmutter and running on multiple GPU nodes >> worked. Likely, I had messed up something the other day. Also, I was able >> to have multiple MPI tasks on a GPU using Nvidia MPS. The petsc output >> shows the number of MPI tasks: >> > >> > KSP Object: 32 MPI processes >> > >> > Can petsc show the number of GPUs used? >> > >> > Thanks, >> > Cho >> > >> > From: Barry Smith <[email protected]> >> > Sent: Wednesday, August 9, 2023 4:09 PM >> > To: Ng, Cho-Kuen <[email protected]> >> > Cc: [email protected] <[email protected]> >> > Subject: Re: [petsc-users] Using PETSc GPU backend >> > >> > We would need more information about "hanging". Do PETSc examples and >> tiny problems "hang" on multiple nodes? If you run with -info what are the >> last messages printed? Can you run with a debugger to see where it is >> "hanging"? >> > >> > >> > >> >> On Aug 9, 2023, at 5:59 PM, Ng, Cho-Kuen <[email protected]> >> wrote: >> >> >> >> Barry and Matt, >> >> >> >> Thanks for your help. Now I can use petsc GPU backend on Perlmutter: 1 >> node, 4 MPI tasks and 4 GPUs. However, I ran into problems with multiple >> nodes: 2 nodes, 8 MPI tasks and 8 GPUs. The run hung on KSPSolve. How can I >> fix this? >> >> >> >> Best, >> >> Cho >> >> >> >> From: Barry Smith <[email protected]> >> >> Sent: Monday, July 17, 2023 6:58 AM >> >> To: Ng, Cho-Kuen <[email protected]> >> >> Cc: [email protected] <[email protected]> >> >> Subject: Re: [petsc-users] Using PETSc GPU backend >> >> >> >> The examples that use DM, in particular DMDA all trivially support >> using the GPU with -dm_mat_type aijcusparse -dm_vec_type cuda >> >> >> >> >> >> >> >>> On Jul 17, 2023, at 1:45 AM, Ng, Cho-Kuen <[email protected]> >> wrote: >> >>> >> >>> Barry, >> >>> >> >>> Thank you so much for the clarification. >> >>> >> >>> I see that ex104.c and ex300.c use MatXAIJSetPreallocation(). Are >> there other tutorials available? >> >>> >> >>> Cho >> >>> From: Barry Smith <[email protected]> >> >>> Sent: Saturday, July 15, 2023 8:36 AM >> >>> To: Ng, Cho-Kuen <[email protected]> >> >>> Cc: [email protected] <[email protected]> >> >>> Subject: Re: [petsc-users] Using PETSc GPU backend >> >>> >> >>> Cho, >> >>> >> >>> We currently have a crappy API for turning on GPU support, and >> our documentation is misleading in places. >> >>> >> >>> People constantly say "to use GPU's with PETSc you only need to >> use -mat_type aijcusparse (for example)" This is incorrect. >> >>> >> >>> This does not work with code that uses the convenience Mat >> constructors such as MatCreateAIJ(), MatCreateAIJWithArrays etc. It only >> works if you use the constructor approach of MatCreate(), MatSetSizes(), >> MatSetFromOptions(), MatXXXSetPreallocation(). ... Similarly you need to >> use VecCreate(), VecSetSizes(), VecSetFromOptions() and -vec_type cuda >> >>> >> >>> If you use DM to create the matrices and vectors then you can use >> -dm_mat_type aijcusparse -dm_vec_type cuda >> >>> >> >>> Sorry for the confusion. >> >>> >> >>> Barry >> >>> >> >>> >> >>> >> >>> >> >>>> On Jul 15, 2023, at 8:03 AM, Matthew Knepley <[email protected]> >> wrote: >> >>>> >> >>>> On Sat, Jul 15, 2023 at 1:44 AM Ng, Cho-Kuen <[email protected]> >> wrote: >> >>>> Matt, >> >>>> >> >>>> After inserting 2 lines in the code: >> >>>> >> >>>> ierr = >> MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr); >> >>>> ierr = MatSetFromOptions(A);CHKERRQ(ierr); >> >>>> ierr = MatCreateAIJ(PETSC_COMM_WORLD,mlocal,mlocal,m,n, >> >>>> >> d_nz,PETSC_NULL,o_nz,PETSC_NULL,&A);;CHKERRQ(ierr); >> >>>> >> >>>> "There are no unused options." However, there is no improvement on >> the GPU performance. >> >>>> >> >>>> 1. MatCreateAIJ() sets the type, and in fact it overwrites the Mat >> you created in steps 1 and 2. This is detailed in the manual. >> >>>> >> >>>> 2. You should replace MatCreateAIJ(), with MatSetSizes() before >> MatSetFromOptions(). >> >>>> >> >>>> THanks, >> >>>> >> >>>> Matt >> >>>> Thanks, >> >>>> Cho >> >>>> >> >>>> From: Matthew Knepley <[email protected]> >> >>>> Sent: Friday, July 14, 2023 5:57 PM >> >>>> To: Ng, Cho-Kuen <[email protected]> >> >>>> Cc: Barry Smith <[email protected]>; Mark Adams <[email protected]>; >> [email protected] <[email protected]> >> >>>> Subject: Re: [petsc-users] Using PETSc GPU backend >> >>>> On Fri, Jul 14, 2023 at 7:57 PM Ng, Cho-Kuen <[email protected]> >> wrote: >> >>>> I managed to pass the following options to PETSc using a GPU node on >> Perlmutter. >> >>>> >> >>>> -mat_type aijcusparse -vec_type cuda -log_view -options_left >> >>>> >> >>>> Below is a summary of the test using 4 MPI tasks and 1 GPU per task. >> >>>> >> >>>> o #PETSc Option Table entries: >> >>>> -log_view >> >>>> -mat_type aijcusparse >> >>>> -options_left >> >>>> -vec_type cuda >> >>>> #End of PETSc Option Table entries >> >>>> WARNING! There are options you set that were not used! >> >>>> WARNING! could be spelling mistake, etc! >> >>>> There is one unused database option. It is: >> >>>> Option left: name:-mat_type value: aijcusparse >> >>>> >> >>>> The -mat_type option has not been used. In the application code, we >> use >> >>>> >> >>>> ierr = MatCreateAIJ(PETSC_COMM_WORLD,mlocal,mlocal,m,n, >> >>>> d_nz,PETSC_NULL,o_nz,PETSC_NULL,&A);;CHKERRQ(ierr); >> >>>> >> >>>> >> >>>> If you create the Mat this way, then you need MatSetFromOptions() in >> order to set the type from the command line. >> >>>> >> >>>> Thanks, >> >>>> >> >>>> Matt >> >>>> o The percent flops on the GPU for KSPSolve is 17%. >> >>>> >> >>>> In comparison with a CPU run using 16 MPI tasks, the GPU run is an >> order of magnitude slower. How can I improve the GPU performance? >> >>>> >> >>>> Thanks, >> >>>> Cho >> >>>> From: Ng, Cho-Kuen <[email protected]> >> >>>> Sent: Friday, June 30, 2023 7:57 AM >> >>>> To: Barry Smith <[email protected]>; Mark Adams <[email protected]> >> >>>> Cc: Matthew Knepley <[email protected]>; [email protected] < >> [email protected]> >> >>>> Subject: Re: [petsc-users] Using PETSc GPU backend >> >>>> Barry, Mark and Matt, >> >>>> >> >>>> Thank you all for the suggestions. I will modify the code so we can >> pass runtime options. >> >>>> >> >>>> Cho >> >>>> From: Barry Smith <[email protected]> >> >>>> Sent: Friday, June 30, 2023 7:01 AM >> >>>> To: Mark Adams <[email protected]> >> >>>> Cc: Matthew Knepley <[email protected]>; Ng, Cho-Kuen < >> [email protected]>; [email protected] <[email protected]> >> >>>> Subject: Re: [petsc-users] Using PETSc GPU backend >> >>>> >> >>>> Note that options like -mat_type aijcusparse -vec_type cuda only >> work if the program is set up to allow runtime swapping of matrix and >> vector types. If you have a call to MatCreateMPIAIJ() or other specific >> types then then these options do nothing but because Mark had you use >> -options_left the program will tell you at the end that it did not use the >> option so you will know. >> >>>> >> >>>>> On Jun 30, 2023, at 9:30 AM, Mark Adams <[email protected]> wrote: >> >>>>> >> >>>>> PetscCall(PetscInitialize(&argc, &argv, NULL, help)); gives us the >> args and you run: >> >>>>> >> >>>>> a.out -mat_type aijcusparse -vec_type cuda -log_view -options_left >> >>>>> >> >>>>> Mark >> >>>>> >> >>>>> On Fri, Jun 30, 2023 at 6:16 AM Matthew Knepley <[email protected]> >> wrote: >> >>>>> On Fri, Jun 30, 2023 at 1:13 AM Ng, Cho-Kuen via petsc-users < >> [email protected]> wrote: >> >>>>> Mark, >> >>>>> >> >>>>> The application code reads in parameters from an input file, where >> we can put the PETSc runtime options. Then we pass the options to >> PetscInitialize(...). Does that sounds right? >> >>>>> >> >>>>> PETSc will read command line argument automatically in >> PetscInitialize() unless you shut it off. >> >>>>> >> >>>>> Thanks, >> >>>>> >> >>>>> Matt >> >>>>> Cho >> >>>>> From: Ng, Cho-Kuen <[email protected]> >> >>>>> Sent: Thursday, June 29, 2023 8:32 PM >> >>>>> To: Mark Adams <[email protected]> >> >>>>> Cc: [email protected] <[email protected]> >> >>>>> Subject: Re: [petsc-users] Using PETSc GPU backend >> >>>>> Mark, >> >>>>> >> >>>>> Thanks for the information. How do I put the runtime options for >> the executable, say, a.out, which does not have the provision to append >> arguments? Do I need to change the C++ main to read in the options? >> >>>>> >> >>>>> Cho >> >>>>> From: Mark Adams <[email protected]> >> >>>>> Sent: Thursday, June 29, 2023 5:55 PM >> >>>>> To: Ng, Cho-Kuen <[email protected]> >> >>>>> Cc: [email protected] <[email protected]> >> >>>>> Subject: Re: [petsc-users] Using PETSc GPU backend >> >>>>> Run with options: -mat_type aijcusparse -vec_type cuda -log_view >> -options_left >> >>>>> The last column of the performance data (from -log_view) will be >> the percent flops on the GPU. Check that that is > 0. >> >>>>> >> >>>>> The end of the output will list the options that were used and >> options that were _not_ used (if any). Check that there are no options left. >> >>>>> >> >>>>> Mark >> >>>>> >> >>>>> On Thu, Jun 29, 2023 at 7:50 PM Ng, Cho-Kuen via petsc-users < >> [email protected]> wrote: >> >>>>> I installed PETSc on Perlmutter using "spack install >> petsc+cuda+zoltan" and used it by "spack load petsc/fwge6pf". Then I >> compiled the application code (purely CPU code) linking to the petsc >> package, hoping that I can get performance improvement using the petsc GPU >> backend. However, the timing was the same using the same number of MPI >> tasks with and without GPU accelerators. Have I missed something in the >> process, for example, setting up PETSc options at runtime to use the GPU >> backend? >> >>>>> >> >>>>> Thanks, >> >>>>> Cho >> >>>>> >> >>>>> >> >>>>> -- >> >>>>> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> >>>>> -- Norbert Wiener >> >>>>> >> >>>>> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fzprTMUfKGkoQcPM_9rzpAwqm9RSmA7djOy8cPIpHuaXVzOoeyoCY7H375lUNWy-az1Fwc6MzSzjF96EH-7LV7k$ >> >>>>> >> <https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fMikK_wvRIVkv5jLV6EHt_rPhWLibqlxAAYjRVMbAEGOUp417LWCH59TvzCtcD3j4dOd4xR_tUy2MRnqU1N7kew$> >> >>>> >> >>>> >> >>>> >> >>>> -- >> >>>> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> >>>> -- Norbert Wiener >> >>>> >> >>>> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fzprTMUfKGkoQcPM_9rzpAwqm9RSmA7djOy8cPIpHuaXVzOoeyoCY7H375lUNWy-az1Fwc6MzSzjF96EH-7LV7k$ >> >>>> >> <https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fMikK_wvRIVkv5jLV6EHt_rPhWLibqlxAAYjRVMbAEGOUp417LWCH59TvzCtcD3j4dOd4xR_tUy2MRnqU1N7kew$> >> >>>> >> >>>> >> >>>> -- >> >>>> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> >>>> -- Norbert Wiener >> >>>> >> >>>> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fzprTMUfKGkoQcPM_9rzpAwqm9RSmA7djOy8cPIpHuaXVzOoeyoCY7H375lUNWy-az1Fwc6MzSzjF96EH-7LV7k$ >> >>>> >> <https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fMikK_wvRIVkv5jLV6EHt_rPhWLibqlxAAYjRVMbAEGOUp417LWCH59TvzCtcD3j4dOd4xR_tUy2MRnqU1N7kew$> >> >> >> >> >> >> >> >> From: Barry Smith <[email protected]> >> >> Sent: Monday, July 17, 2023 6:58 AM >> >> To: Ng, Cho-Kuen <[email protected]> >> >> Cc: [email protected] <[email protected]> >> >> Subject: Re: [petsc-users] Using PETSc GPU backend >> >> >> >> The examples that use DM, in particular DMDA all trivially support >> using the GPU with -dm_mat_type aijcusparse -dm_vec_type cuda >> >> >> >> >> >> >> >>> On Jul 17, 2023, at 1:45 AM, Ng, Cho-Kuen <[email protected]> >> wrote: >> >>> >> >>> Barry, >> >>> >> >>> Thank you so much for the clarification. >> >>> >> >>> I see that ex104.c and ex300.c use MatXAIJSetPreallocation(). Are >> there other tutorials available? >> >>> >> >>> Cho >> >>> From: Barry Smith <[email protected]> >> >>> Sent: Saturday, July 15, 2023 8:36 AM >> >>> To: Ng, Cho-Kuen <[email protected]> >> >>> Cc: [email protected] <[email protected]> >> >>> Subject: Re: [petsc-users] Using PETSc GPU backend >> >>> >> >>> Cho, >> >>> >> >>> We currently have a crappy API for turning on GPU support, and >> our documentation is misleading in places. >> >>> >> >>> People constantly say "to use GPU's with PETSc you only need to >> use -mat_type aijcusparse (for example)" This is incorrect. >> >>> >> >>> This does not work with code that uses the convenience Mat >> constructors such as MatCreateAIJ(), MatCreateAIJWithArrays etc. It only >> works if you use the constructor approach of MatCreate(), MatSetSizes(), >> MatSetFromOptions(), MatXXXSetPreallocation(). ... Similarly you need to >> use VecCreate(), VecSetSizes(), VecSetFromOptions() and -vec_type cuda >> >>> >> >>> If you use DM to create the matrices and vectors then you can use >> -dm_mat_type aijcusparse -dm_vec_type cuda >> >>> >> >>> Sorry for the confusion. >> >>> >> >>> Barry >> >>> >> >>> >> >>> >> >>> >> >>>> On Jul 15, 2023, at 8:03 AM, Matthew Knepley <[email protected]> >> wrote: >> >>>> >> >>>> On Sat, Jul 15, 2023 at 1:44 AM Ng, Cho-Kuen <[email protected]> >> wrote: >> >>>> Matt, >> >>>> >> >>>> After inserting 2 lines in the code: >> >>>> >> >>>> ierr = >> MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr); >> >>>> ierr = MatSetFromOptions(A);CHKERRQ(ierr); >> >>>> ierr = MatCreateAIJ(PETSC_COMM_WORLD,mlocal,mlocal,m,n, >> >>>> >> d_nz,PETSC_NULL,o_nz,PETSC_NULL,&A);;CHKERRQ(ierr); >> >>>> >> >>>> "There are no unused options." However, there is no improvement on >> the GPU performance. >> >>>> >> >>>> 1. MatCreateAIJ() sets the type, and in fact it overwrites the Mat >> you created in steps 1 and 2. This is detailed in the manual. >> >>>> >> >>>> 2. You should replace MatCreateAIJ(), with MatSetSizes() before >> MatSetFromOptions(). >> >>>> >> >>>> THanks, >> >>>> >> >>>> Matt >> >>>> Thanks, >> >>>> Cho >> >>>> >> >>>> From: Matthew Knepley <[email protected]> >> >>>> Sent: Friday, July 14, 2023 5:57 PM >> >>>> To: Ng, Cho-Kuen <[email protected]> >> >>>> Cc: Barry Smith <[email protected]>; Mark Adams <[email protected]>; >> [email protected] <[email protected]> >> >>>> Subject: Re: [petsc-users] Using PETSc GPU backend >> >>>> On Fri, Jul 14, 2023 at 7:57 PM Ng, Cho-Kuen <[email protected]> >> wrote: >> >>>> I managed to pass the following options to PETSc using a GPU node on >> Perlmutter. >> >>>> >> >>>> -mat_type aijcusparse -vec_type cuda -log_view -options_left >> >>>> >> >>>> Below is a summary of the test using 4 MPI tasks and 1 GPU per task. >> >>>> >> >>>> o #PETSc Option Table entries: >> >>>> -log_view >> >>>> -mat_type aijcusparse >> >>>> -options_left >> >>>> -vec_type cuda >> >>>> #End of PETSc Option Table entries >> >>>> WARNING! There are options you set that were not used! >> >>>> WARNING! could be spelling mistake, etc! >> >>>> There is one unused database option. It is: >> >>>> Option left: name:-mat_type value: aijcusparse >> >>>> >> >>>> The -mat_type option has not been used. In the application code, we >> use >> >>>> >> >>>> ierr = MatCreateAIJ(PETSC_COMM_WORLD,mlocal,mlocal,m,n, >> >>>> d_nz,PETSC_NULL,o_nz,PETSC_NULL,&A);;CHKERRQ(ierr); >> >>>> >> >>>> >> >>>> If you create the Mat this way, then you need MatSetFromOptions() in >> order to set the type from the command line. >> >>>> >> >>>> Thanks, >> >>>> >> >>>> Matt >> >>>> o The percent flops on the GPU for KSPSolve is 17%. >> >>>> >> >>>> In comparison with a CPU run using 16 MPI tasks, the GPU run is an >> order of magnitude slower. How can I improve the GPU performance? >> >>>> >> >>>> Thanks, >> >>>> Cho >> >>>> From: Ng, Cho-Kuen <[email protected]> >> >>>> Sent: Friday, June 30, 2023 7:57 AM >> >>>> To: Barry Smith <[email protected]>; Mark Adams <[email protected]> >> >>>> Cc: Matthew Knepley <[email protected]>; [email protected] < >> [email protected]> >> >>>> Subject: Re: [petsc-users] Using PETSc GPU backend >> >>>> Barry, Mark and Matt, >> >>>> >> >>>> Thank you all for the suggestions. I will modify the code so we can >> pass runtime options. >> >>>> >> >>>> Cho >> >>>> From: Barry Smith <[email protected]> >> >>>> Sent: Friday, June 30, 2023 7:01 AM >> >>>> To: Mark Adams <[email protected]> >> >>>> Cc: Matthew Knepley <[email protected]>; Ng, Cho-Kuen < >> [email protected]>; [email protected] <[email protected]> >> >>>> Subject: Re: [petsc-users] Using PETSc GPU backend >> >>>> >> >>>> Note that options like -mat_type aijcusparse -vec_type cuda only >> work if the program is set up to allow runtime swapping of matrix and >> vector types. If you have a call to MatCreateMPIAIJ() or other specific >> types then then these options do nothing but because Mark had you use >> -options_left the program will tell you at the end that it did not use the >> option so you will know. >> >>>> >> >>>>> On Jun 30, 2023, at 9:30 AM, Mark Adams <[email protected]> wrote: >> >>>>> >> >>>>> PetscCall(PetscInitialize(&argc, &argv, NULL, help)); gives us the >> args and you run: >> >>>>> >> >>>>> a.out -mat_type aijcusparse -vec_type cuda -log_view -options_left >> >>>>> >> >>>>> Mark >> >>>>> >> >>>>> On Fri, Jun 30, 2023 at 6:16 AM Matthew Knepley <[email protected]> >> wrote: >> >>>>> On Fri, Jun 30, 2023 at 1:13 AM Ng, Cho-Kuen via petsc-users < >> [email protected]> wrote: >> >>>>> Mark, >> >>>>> >> >>>>> The application code reads in parameters from an input file, where >> we can put the PETSc runtime options. Then we pass the options to >> PetscInitialize(...). Does that sounds right? >> >>>>> >> >>>>> PETSc will read command line argument automatically in >> PetscInitialize() unless you shut it off. >> >>>>> >> >>>>> Thanks, >> >>>>> >> >>>>> Matt >> >>>>> Cho >> >>>>> From: Ng, Cho-Kuen <[email protected]> >> >>>>> Sent: Thursday, June 29, 2023 8:32 PM >> >>>>> To: Mark Adams <[email protected]> >> >>>>> Cc: [email protected] <[email protected]> >> >>>>> Subject: Re: [petsc-users] Using PETSc GPU backend >> >>>>> Mark, >> >>>>> >> >>>>> Thanks for the information. How do I put the runtime options for >> the executable, say, a.out, which does not have the provision to append >> arguments? Do I need to change the C++ main to read in the options? >> >>>>> >> >>>>> Cho >> >>>>> From: Mark Adams <[email protected]> >> >>>>> Sent: Thursday, June 29, 2023 5:55 PM >> >>>>> To: Ng, Cho-Kuen <[email protected]> >> >>>>> Cc: [email protected] <[email protected]> >> >>>>> Subject: Re: [petsc-users] Using PETSc GPU backend >> >>>>> Run with options: -mat_type aijcusparse -vec_type cuda -log_view >> -options_left >> >>>>> The last column of the performance data (from -log_view) will be >> the percent flops on the GPU. Check that that is > 0. >> >>>>> >> >>>>> The end of the output will list the options that were used and >> options that were _not_ used (if any). Check that there are no options left. >> >>>>> >> >>>>> Mark >> >>>>> >> >>>>> On Thu, Jun 29, 2023 at 7:50 PM Ng, Cho-Kuen via petsc-users < >> [email protected]> wrote: >> >>>>> I installed PETSc on Perlmutter using "spack install >> petsc+cuda+zoltan" and used it by "spack load petsc/fwge6pf". Then I >> compiled the application code (purely CPU code) linking to the petsc >> package, hoping that I can get performance improvement using the petsc GPU >> backend. However, the timing was the same using the same number of MPI >> tasks with and without GPU accelerators. Have I missed something in the >> process, for example, setting up PETSc options at runtime to use the GPU >> backend? >> >>>>> >> >>>>> Thanks, >> >>>>> Cho >> >>>>> >> >>>>> >> >>>>> -- >> >>>>> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> >>>>> -- Norbert Wiener >> >>>>> >> >>>>> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fzprTMUfKGkoQcPM_9rzpAwqm9RSmA7djOy8cPIpHuaXVzOoeyoCY7H375lUNWy-az1Fwc6MzSzjF96EH-7LV7k$ >> >>>>> >> <https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fMikK_wvRIVkv5jLV6EHt_rPhWLibqlxAAYjRVMbAEGOUp417LWCH59TvzCtcD3j4dOd4xR_tUy2MRnqU1N7kew$> >> >>>> >> >>>> >> >>>> >> >>>> -- >> >>>> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> >>>> -- Norbert Wiener >> >>>> >> >>>> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fzprTMUfKGkoQcPM_9rzpAwqm9RSmA7djOy8cPIpHuaXVzOoeyoCY7H375lUNWy-az1Fwc6MzSzjF96EH-7LV7k$ >> >>>> >> <https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fMikK_wvRIVkv5jLV6EHt_rPhWLibqlxAAYjRVMbAEGOUp417LWCH59TvzCtcD3j4dOd4xR_tUy2MRnqU1N7kew$> >> >>>> >> >>>> >> >>>> -- >> >>>> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> >>>> -- Norbert Wiener >> >>>> >> >>>> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fzprTMUfKGkoQcPM_9rzpAwqm9RSmA7djOy8cPIpHuaXVzOoeyoCY7H375lUNWy-az1Fwc6MzSzjF96EH-7LV7k$ >> >>>> >> <https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fMikK_wvRIVkv5jLV6EHt_rPhWLibqlxAAYjRVMbAEGOUp417LWCH59TvzCtcD3j4dOd4xR_tUy2MRnqU1N7kew$> >> >> >> <Untitled.png> >> >> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fzprTMUfKGkoQcPM_9rzpAwqm9RSmA7djOy8cPIpHuaXVzOoeyoCY7H375lUNWy-az1Fwc6MzSzjF96EH-7LV7k$ > > <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fzprTMUfKGkoQcPM_9rzpAwqm9RSmA7djOy8cPIpHuaXVzOoeyoCY7H375lUNWy-az1Fwc6MzSzjF96Ertj1RsU$ > > >
