Could you send the configure.log file?
--Junchao Zhang
On Fri, Sep 20, 2024 at 1:33 PM Qiyue Lu wrote:
> Thanks for your response.
> Here is my module loaded:
>
>
>
> *Currently Loaded Modules: 1) craype-arm-grace 4)
> perftools-base/24.03.0 7)
--Junchao Zhang
On Fri, Sep 20, 2024 at 11:42 AM Qiyue Lu wrote:
> Hello,
> I am trying to build the PETSc version 3.21.5 (the latest, cloned from
> git) version on a Grace Hopper node with this configuration options:
> *--with-cc=cc --with-cxx=CC --with-fc=ftn --download-f2cblaslapac
Pierre, thanks for the additional information.
This is scary. If this is really an OpenMPI bug, we don't have a petsc test
to catch it and currently have no clue what went wrong.
--Junchao Zhang
On Wed, Sep 18, 2024 at 5:09 AM LEDAC Pierre wrote:
> Junchao, I just tried PETSc last ve
Did you "fix" the problem with OpenMPI 5, but keep petsc unchanged (ie.,
still 3.20)?
--Junchao Zhang
On Tue, Sep 17, 2024 at 9:47 AM LEDAC Pierre wrote:
> Thanks Satish, and nice guess for OpenMPI 5 !
>
>
> It seems it solves the issue (at least on my GPU box where I
Could you try petsc/main to see if the problem persists?
--Junchao Zhang
On Mon, Sep 16, 2024 at 10:51 AM LEDAC Pierre wrote:
> Hi all,
>
>
> We are using PETSc 3.20 in our code and running succesfully several
> solvers on Nvidia GPU with OpenMPI library which are not GPU awa
has 25.
Generally, VecDuplicate(par_xcoord,&par_xvec) is wrong since you assume x,
y have the same size.
--Junchao Zhang
On Thu, Sep 5, 2024 at 4:20 PM Karthikeyan Chockalingam - STFC UKRI via
petsc-users wrote:
> Hello,
>
>
>
> I am unsure why the program crashes even whi
Hi, Ling,
MatMult almost degraded 20%, which is a lot. Do you have configure.log
for the two builds? We might find compiler discrepancies from it.
--Junchao Zhang
On Wed, Aug 21, 2024 at 8:57 AM Zou, Ling wrote:
> Hi Junchao,
>
>
>
> Yeah, I have part of the log_view, fo
Do you have -log_view report so that we can know which petsc functions
degraded? Or is it because compilers were different?
--Junchao Zhang
On Sun, Aug 18, 2024 at 6:04 PM Zou, Ling via petsc-users <
petsc-users@mcs.anl.gov> wrote:
> Hi all,
>
>
>
> After updating M
The error message is a little confusing. It says the indices should be in
[0, 200), but you used an out of range index 200 in VecSetValues.
--Junchao Zhang
On Fri, Jul 26, 2024 at 10:24 PM Ivan Luthfi wrote:
> Hi friend, I am trying to try my second Petsc exercise in a lecture f
Currently we don't support Kokkos dense matrix and its solvers. You can
use MATSEQDENSECUDA/HIP
--Junchao Zhang
On Wed, Jul 24, 2024 at 2:08 PM Barry Smith wrote:
>
>For one MPI rank, it looks like you can use -pc_type cholesky
> -pc_factor_mat_solver_type cupm th
Let me run some examples on our end to see whether the code calls expected
functions.
--Junchao Zhang
On Mon, Jun 24, 2024 at 10:46 AM Matthew Knepley wrote:
> On Mon, Jun 24, 2024 at 11: 21 AM Yongzhong Li utoronto. ca> wrote: Thank you Pierre for your information. Do we have a
> c
Could you send your petsc configure.log?
--Junchao Zhang
On Sat, Jun 22, 2024 at 9:07 PM Yongzhong Li
wrote:
> Yeah, I ran my program again using -mat_view::ascii_info and set
> MKL_VERBOSE to be 1, then I noticed the outputs suggested that the matrix
> to be seqaijmkl type (I’ve a
No, you don't. It is strange. Perhaps you can you run a petsc example
first and see if MKL is really used
$ cd src/mat/tests
$ make ex1
$ MKL_VERBOSE=1 ./ex1
--Junchao Zhang
On Fri, Jun 21, 2024 at 4:03 PM Yongzhong Li
wrote:
> I am using
>
> export MKL_VERBOSE=1
>
>
I remember there are some MKL env vars to print MKL routines called. Maybe
we can try it to see what MKL routines are really used and then we can
understand why some petsc functions did not speed up
--Junchao Zhang
On Thu, Jun 20, 2024 at 10:39 PM Yongzhong Li
wrote:
> Hi Barry, sorry for
Arpan,
Nice to meet you. -log_view in a petsc option, so I think you can add it
to your fvSolution_petsc_pKok_Uof at location like
mat_type mpiaijkokkos;
vec_type kokkos;
log_view;
--Junchao Zhang
On Fri, Jun 14, 2024 at 2:44 PM Sircar, Arpan wrote:
> Hi Junchao and Barry,
&g
Arpan,
Did you add -log_view ?
--Junchao Zhang
On Fri, Jun 14, 2024 at 2:00 PM Sircar, Arpan via petsc-users <
petsc-users@mcs.anl.gov> wrote:
> Hi Barry, Thanks for your prompt response. These are run with with the
> same PETSc solvers but the one on GPUs (log_pKok) uses mataij
I haven't used CUDA graph with PETSc. Do you happen to have a working
example so we can debug?
--Junchao Zhang
On Tue, May 14, 2024 at 6:08 PM Sreeram R Venkat
wrote:
> I have a MatShell object that I want to convert to a MATDENSECUDA.
> Normally, I use MatComputeOperator for th
I used cudatoolkit-standalone/12.4.1 and gcc-12.3.
Be sure to use the latest petsc/main or petsc/release, which contains fixes
for Polaris.
--Junchao Zhang
On Thu, May 2, 2024 at 10:23 AM Satish Balay via petsc-users <
petsc-users@mcs.anl.gov> wrote:
> Try:
>
> module use /s
No, I think sparse matrix-vector products (MatMult in petsc) can be
accelerated with multithreading. petsc does not do that, but one can use
other libraries, such as MKL for that.
--Junchao Zhang
On Tue, Apr 23, 2024 at 3:00 PM Yongzhong Li
wrote:
> Thanks Barry! Does this mean that
Thanks for the trick. We can prepare the example script for Lonestar6 and
mention it.
--Junchao Zhang
On Fri, Apr 19, 2024 at 11:55 AM Sreeram R Venkat
wrote:
> I talked to the MVAPICH people, and they told me to try adding
> /path/to/mvapich2-gdr/lib64/libmpi.so to LD_PRELOAD (appa
pFWw9ViCb$
>
> that mentioned it, and I heard in Barry's talk at SIAM PP this year about
> the need for stream-aware MPI, so I was wondering if NCCL would be used in
> PETSc to do GPU-GPU communication.
>
> On Wed, Apr 17, 2024, 7:58 AM Junchao Zhang
> wrote:
>
>>
On Wed, Apr 17, 2024 at 7:51 AM Sreeram R Venkat
wrote:
> Do you know if there are plans for NCCL support in PETSc?
>
What is your need? Do you mean using NCCL for the MPI communication?
>
> On Tue, Apr 16, 2024, 10:41 PM Junchao Zhang
> wrote:
>
>> Glad to hear yo
Victor, through the SMART PETSc project, I do have access to Frontera and
Lonestar6.
--Junchao Zhang
On Wed, Apr 17, 2024 at 3:55 AM Victor Eijkhout
wrote:
>
>
>
>
>- Did you use Frontera at TACC? If yes, I could have a try.
>
>
>
> If you’re interested in ac
Glad to hear you found a way. Did you use Frontera at TACC? If yes, I
could have a try.
--Junchao Zhang
On Tue, Apr 16, 2024 at 8:35 PM Sreeram R Venkat
wrote:
> I finally figured out a way to make it work. I had to build PETSc and my
> application using the (non GPU-aware) Intel MPI
gpu_aware_mpi 0". Without it, I saw code crashes.
From what I can see, the gpu-aware mpi on Summit is an unusable and
unmaintained state.
--Junchao Zhang
On Fri, Mar 1, 2024 at 3:58 PM Junchao Zhang
wrote:
> It is weird, with
> jsrun --smpiargs "-gpu" -n 6 -a 1
XnfuQMkmJQ$
, I
think your flags were right.
I just got my Summit account reactivated today. I will give it a try.
--Junchao Zhang
On Fri, Mar 1, 2024 at 3:32 PM Blondel, Sophie wrote:
> I have been using --smpiargs "-gpu".
>
> I tried the benchmark with "jsrun --s
On Fri, Mar 1, 2024 at 11:10 AM Fabian Wermelinger
wrote:
> On Fri, 01 Mar 2024 10:09:17 -0600, Junchao Zhang wrote:
> >No. MatMPIBAIJSetPreallocationCSR() copies the data, but
> >VecCreateMPIWithArray() does not copy (only use pointers user provided).
>
> OK, my u
On Fri, Mar 1, 2024 at 9:28 AM Fabian Wermelinger wrote:
> Dear All, I am implementing a linear solver interface in a flow solver
> with support for PETSc. My application uses a parallel CSR representation
> and it manages the memory for it. I would like to wrap PETSc matrices (and
> vectors) aro
to jsrun
--Junchao Zhang
On Thu, Feb 29, 2024 at 3:22 PM Blondel, Sophie via petsc-users <
petsc-users@mcs.anl.gov> wrote:
> I still get the same error when deactivating GPU-aware MPI. I also tried
> unloading spectrum MPI and using openMPI instead (recompiling everything)
> and
Yes, it is a runtime option. No need to reconfigure petsc. Just add
"-use_gpu_aware_mpi 0" to your test's command line.
--Junchao Zhang
On Thu, Feb 29, 2024 at 10:07 AM Matthew Knepley wrote:
> On Thu, Feb 29, 2024 at 11:03 AM Blondel, Sophie via petsc-users <
>
try on other GPU
machines to see if it is just an IBM Spectrum MPI problem.
Thanks.
--Junchao Zhang
On Thu, Feb 29, 2024 at 9:17 AM Blondel, Sophie via petsc-users <
petsc-users@mcs.anl.gov> wrote:
> Hi, I am using PETSc build with the Kokkos CUDA backend on Summit but when
> I run
OffsetView, MatSetValuesCOO etc.
--Junchao Zhang
On Wed, Feb 14, 2024 at 11:23 AM Satish Balay via petsc-users <
petsc-users@mcs.anl.gov> wrote:
> Looks like ex55 is the one to use - that links in with ex55k
>
> But it needs a fix for a build from 'make'
>
> >>>>>
On Thu, Feb 8, 2024 at 3:15 AM Maruthi NH wrote:
> Hi Barry,
> Thanks. Yes, the global column index was wrong. I have one more question
> regarding MatCreateMPIAIJWithArrays. If I have 100 elements in rank 0 and
> 50 in rank 1, does PETSc redistribute equally among procs before solving?
>
No, th
FYI, Argonne Training Program on Extreme-Scale Computing (ATPESC)
2024, "There are no fees to participate in ATPESC. Domestic airfare, meals,
and lodging are also provided."
Join us February 8 for a webinar over the 2024 ATPESC program and
application process.
*ALCF WEBINAR*
-
Try to use Valgrind to detect memory errors?
--Junchao Zhang
On Mon, Feb 5, 2024 at 3:29 AM TAY Wee Beng wrote:
> Hi,
>
> I have a code which uses PETSc and it was working fine. However, I
> haven't used the code for a while and now we've migrated to a new cray
> clus
On Sun, Feb 4, 2024 at 9:57 PM Runfeng Jin wrote:
> Hi,
>
> I see in document the mat status in hip is still in development, but I
> see some hip interface in MAT. I want to use petsc with primme(eigenvalue
> solver) to diagonalize the matrix in AMD GPU. The primme need user to
> provide MV (
Glad you figured it out!
--Junchao Zhang
On Sun, Feb 4, 2024 at 7:56 PM Yesypenko, Anna wrote:
> Hi Junchao, Victor,
>
> I fixed the issue! The issue was with the CPU bindings. Python has a
> limitation that it only runs on one core.
> I had to modify the MPI thread launch scri
Hi, Anna,
Do you have other CUDA machines to try? If you can share your test, then
I will run on Polaris@Argonne to see if it is a petsc/hypre issue. If not,
then it must be a GPU-MPI binding problem on TACC.
Thanks
--Junchao Zhang
On Thu, Feb 1, 2024 at 5:31 PM Yesypenko, Anna wrote
Cc Victor@TACC, who might have some ideas.
--Junchao Zhang
On Thu, Feb 1, 2024 at 9:28 AM Yesypenko, Anna wrote:
> Hi Junchao,
>
> Thank you for your suggestion, you're right that binding MPI ranks to GPUs
> seems to be the issue.
> I looked at the TACC documentation, a
Hi, Anna,
Could you attach your petsc configure.log?
--Junchao Zhang
On Thu, Feb 1, 2024 at 9:28 AM Yesypenko, Anna wrote:
> Hi Junchao,
>
> Thank you for your suggestion, you're right that binding MPI ranks to GPUs
> seems to be the issue.
> I looked at the TACC docume
LE_DEVICES
environment variable)
Please follow up if you could not solve it.
Thanks.
--Junchao Zhang
On Wed, Jan 31, 2024 at 4:07 PM Yesypenko, Anna
wrote:
> Dear Petsc devs,
>
> I'm encountering an error running hypre on a single node with multiple
> GPUs.
> The issue is in
the right VecGet/RestoreArrayWrite(), the code worked.
--Junchao Zhang
On Fri, Jan 26, 2024 at 8:40 AM Pierre Jolivet wrote:
>
> On 26 Jan 2024, at 3:11 PM, Pierre Jolivet wrote:
>
>
> On 26 Jan 2024, at 3:03 PM, mich...@paraffinalia.co.uk wrote:
>
> On 2024-01-23 18:09, Ju
Do you have an example to reproduce it?
--Junchao Zhang
On Tue, Jan 23, 2024 at 10:49 AM wrote:
> Hello,
>
> I have used the GMRES solver in PETSc successfully up to now, but on
> installing the most recent release, 3.20.3, the solver fails by exiting
> early. Output from the
9, 2024, at 3:02 PM, Dave May wrote:
>
> Thank you Barry and Junchao for these explanations. I'll turn on
> -log_view_gpu_time.
>
> Do either of you have any thoughts regarding why the percentage of flop's
> being reported on the GPU is not 100% for MGSmooth Level {0,1,2} f
Try to also add -log_view_gpu_time,
https://petsc.org/release/manualpages/Profiling/PetscLogGpuTime/
--Junchao Zhang
On Fri, Jan 19, 2024 at 11:35 AM Dave May wrote:
> Hi all,
>
> I am trying to understand the logging information associated with the
> %flops-performed-on-the-gpu
On Thu, Jan 18, 2024 at 1:47 AM 袁煕 wrote:
> Dear PETSc Experts,
>
> My FEM program works well generally, but in some specific cases with
> multiple CPUs are used, it freezes when calling MatAssemblyBegin where
> PMPI_Allreduce is called (see attached file).
>
> After some investigation, I found t
mpirun -n 8 --bind-to numa --map-by numa ./MPIVersion
8 173565.3584 Rate (MB/s) 7.68175
On this kind of machine, you can expect constant time of your test up to
8 MPI ranks.
--Junchao Zhang
On Fri, Jan 12, 2024 at 11:13 AM Steffen Wilksen | Universitaet Bremen <
swilk...@itp.uni-bre
l be fixed. The code could run to
gw_deallocate () before abort. There are other memory errors. You can
install/use valgrind to fix them. Run it with valgrind ./GW.exe and look
through the output
Thanks.
--Junchao Zhang
On Thu, Jan 11, 2024 at 10:49 PM Shatanawi, Sawsan Muhammad <
Hi, Steffen,
Would it be an MPI process binding issue? Could you try running with
mpiexec --bind-to core -n N python parallel_example.py
--Junchao Zhang
On Fri, Jan 12, 2024 at 8:52 AM Steffen Wilksen | Universitaet Bremen <
swilk...@itp.uni-bremen.de> wrote:
> Thank you
Sawsan,
Also, another error at
https://github.com/sshatanawi/SS_GWM/blob/main/GW_solver_try.F90#L170. You
can not destroy the vector BEFORE calling VecRestoreArray(),
--Junchao Zhang
On Wed, Jan 10, 2024 at 5:49 PM Junchao Zhang
wrote:
> Hi, Sawsan,
> I could build your code and
_GWM/blob/main/GW_solver_try.F90#L169 has
"call VecGetArray(temp_solution, H_vector, ierr)".I don't find petsc
Fortran examples doing VecGetArray. Do we still support it?
--Junchao Zhang
On Wed, Jan 10, 2024 at 2:38 PM Shatanawi, Sawsan Muhammad via petsc-users <
petsc-users@
On Fri, Jan 5, 2024 at 9:33 AM Edoardo Centofanti <
edoardo.centofant...@universitadipavia.it> wrote:
>
>
> Il giorno ven 5 gen 2024 alle ore 16:21 Junchao Zhang <
> junchao.zh...@gmail.com> ha scritto:
>
>>
>>
>>
>> On Fri, Jan 5, 2024 a
On Fri, Jan 5, 2024 at 4:40 AM Edoardo Centofanti <
edoardo.centofant...@universitadipavia.it> wrote:
> Dear all,
>
> I have a code running both on GPU and CPU. This code has both cuda kernels
> and calls to PETSc KSP and related PC. In particular, I am trying to
> perform tests with Hypre BoomerA
On Thu, Dec 21, 2023 at 5:54 AM Matthew Knepley wrote:
> On Thu, Dec 21, 2023 at 6:46 AM Sreeram R Venkat
> wrote:
>
>> Ok, I think the error I'm getting has something to do with how the
>> multiple solves are being done in succession. I'll try to see if there's
>> anything I'm doing wrong there
MPICH folks confirmed it's an MPICH problem and an issue is created at
https://github.com/pmodels/mpich/issues/6843
--Junchao Zhang
On Tue, Dec 12, 2023 at 7:53 PM Junchao Zhang
wrote:
> I was able to reproduce it. Let me ask MPICH developers.
>
> --Junchao Zhang
>
>
&g
I was able to reproduce it. Let me ask MPICH developers.
--Junchao Zhang
On Tue, Dec 12, 2023 at 3:06 PM Randall Mackie
wrote:
> It now seems to me that petsc+mpich is no longer valgrind clean, or I am
> doing something wrong.
>
> A simple program:
>
>
> Program tes
Glad it worked!
--Junchao Zhang
On Wed, Dec 6, 2023 at 1:20 PM Sreeram R Venkat wrote:
> Thank you for your help. It turned out the problem was that I forgot to
> assemble the "x" vector before doing the scatter. It seems to be working
> now.
>
> Thanks,
> Sreeram
&
rocess [9]
Process [10]
Process [11]
Vec Object: Vec Y 12 MPI processes
type: mpi
Process [0]
0.
12.
24.
Process [1]
1.
13.
25.
Process [2]
2.
14.
26.
Process [3]
3.
15.
27.
Process [4]
4.
16.
28.
Process [5]
5.
17.
29.
Process [6]
6.
18.
30.
Process [7]
7.
19.
31.
Process [8]
8.
20.
32.
Process [9
I think your approach is correct. Do you have an example code?
--Junchao Zhang
On Tue, Dec 5, 2023 at 5:15 PM Sreeram R Venkat wrote:
> Hi, I have a follow up question on this.
>
> Now, I'm trying to do a scatter and permutation of the vector. Under the
> same setup as the
I was wondering if you can use
https://petsc.org/release/manualpages/Sys/PetscObjectCompose/ to attach the
DM to the Vec.
--Junchao Zhang
On Thu, Nov 16, 2023 at 8:06 AM Carl-Johan Thore
wrote:
> Hi,
>
> Given a Vec (or Mat) created at some point with DMCreate...,
> is it possible
Hi, Sajid,
Do you have a test example to reproduce the error?
--Junchao Zhang
On Thu, Nov 2, 2023 at 3:37 PM Sajid Ali
wrote:
> Hi PETSc-developers,
>
> I had posted about crashes within PETScSF when using GPU-aware MPI on
> Perlmutter a while ago (
> https://lists.mcs.anl.go
On Thu, Oct 26, 2023 at 8:21 AM Qiyue Lu wrote:
> Hello,
> I am trying to incorporate PETSc as a linear solver to compute Ax=b in my
> code. Currently, the sequential version works.
> 1) I have the global matrix A in CSR format and they are stored in three
> 1-dimensional arrays: row_ptr[ ], col_
Copy the code block of MatSetValues_MPIAIJ() in 3.17 to 3.20. If it works,
then it is possible :)
--Junchao Zhang
On Mon, Oct 23, 2023 at 7:16 AM Mark Adams wrote:
> I have a Fortran user that is getting a segv in MatSetValues_MPIAIJ in
> v3.19 and v3.20 and it works with v3.17.
>
Hi, Philip,
That branch was merged to petsc/main today. Let me know once you have
new profiling results.
Thanks.
--Junchao Zhang
On Mon, Oct 16, 2023 at 9:33 AM Fackler, Philip wrote:
> Junchao,
>
> I've attached updated timing plots (red and blue are swapped from befor
Could you just use "--with-cc=mpiicx --with-cxx=mpiicpx" ? In addition,
you can export environment vars I_MPI_CC=icx and I_MPI_CXX=icpx to
specify the underlying compilers.
--Junchao Zhang
On Mon, Oct 9, 2023 at 7:33 AM Richter, Roland
wrote:
> Hei,
>
> I'm currently
mp;rstart, &rend);
VecCreate(comm, &y);
VecSetSizes(y, PETSC_DECIDE, N);
VecSetType(y, type);
ISCreateStride(PetscObjectComm((PetscObject)x), rend - rstart, rstart, 1,
&ix);
VecScatterCreate(x, ix, y, ix, &vscat);
--Junchao Zhang
On Wed, Oct 4, 2023 at 6:03 PM Sreeram R Venkat wrot
Hi, Philip,
I will look into the tarballs and get back to you.
Thanks.
--Junchao Zhang
On Mon, Oct 2, 2023 at 9:41 AM Fackler, Philip via petsc-users <
petsc-users@mcs.anl.gov> wrote:
> We finally have xolotl ported to use the new COO interface and the
> aijkokkos implementa
found) PetscCall(PetscDLLibraryAppend(PETSC_COMM_WORLD,
&PetscDLLibrariesLoaded, dlib));
}
PetscFunctionReturn(PETSC_SUCCESS);
}
--Junchao Zhang
On Thu, Sep 21, 2023 at 4:59 PM Di Miao via petsc-users <
petsc-users@mcs.anl.gov> wrote:
> Hi Barry,
>
>
>
> I found the following sn
So it means when you require --with-threadsafety, your system needs to have
std atomic; otherwise you need to add --download-concurrencykit or
--with-openmp
--Junchao Zhang
On Thu, Sep 21, 2023 at 9:39 AM Di Miao via petsc-users <
petsc-users@mcs.anl.gov> wrote:
> To whom it may concern,
>
at_type value: aijcusparse
>
> Using nvprof does not include kernels, only API use.
>
> Ramoni Z. S. Azevedo
>
> Em qua., 20 de set. de 2023 às 12:31, Junchao Zhang <
> junchao.zh...@gmail.com> escreveu:
>
>> Try to also add *-dm_mat_type aijcusparse -dm_ve
Try to also add *-dm_mat_type aijcusparse -dm_vec_type cuda*
--Junchao Zhang
On Wed, Sep 20, 2023 at 10:21 AM Ramoni Z. Sedano Azevedo <
ramoni.zsed...@gmail.com> wrote:
>
> Hey!
>
> I am using PETSc in a Fortran code and we use MPI parallelization. We
> would like to us
Hi, Maruthi,
I could not reproduce it. I used the attached slightly modified code
(note the added PetscFunctionBeginUser;
PetscFunctionReturn(PETSC_SUCCESS); in bc_const_temp_both_sides)
Could you try it?
--Junchao Zhang
On Wed, Sep 13, 2023 at 1:07 AM Maruthi NH wrote:
> Hi Junc
es of mat for each rank: 0 100
Total time taken for KSP solve: rank: 0 0.0112464
--Junchao Zhang
On Tue, Sep 12, 2023 at 10:15 AM Maruthi NH wrote:
> Hi Junchao Zhang,
>
> Thanks for the help. Updating PETSc fixed the problem. However, if I use
> boomeramg from hypre as
maybe the problem is already fixed by MR
https://gitlab.com/petsc/petsc/-/merge_requests/6828
You can use petsc/main branch to try. Note your petsc version is from
Date: 2023-08-13
Thanks.
--Junchao Zhang
On Mon, Sep 11, 2023 at 12:10 PM Maruthi NH wrote:
> Hi Barry Smith,
>
> Thank
$ mpirun -n 4 ./main
Solve duration: 22693
It suggests binding is important.
--Junchao Zhang
On Fri, Sep 8, 2023 at 4:53 PM Barry Smith wrote:
>
> For the size problem you are running this is very unexpected. The
> reductions should only start to dominate for thousands of MPI ran
On Wed, Aug 30, 2023 at 8:46 PM Barry Smith wrote:
>
>What convergence do you get without the GPU matrix and vector
> operations?
Barry, that was in the original email
>
>
>Can you try the GPU run with -ksp_type gmres -ksp_pc_side right ?
>
>For certain problems, ILU can produce cat
Hi, Ramoni
Do you have a reproducible example? Usually it is because the cpu and
gpu are out of synchronization. It could be a user's problem or petsc's.
Thanks.
--Junchao Zhang
On Wed, Aug 30, 2023 at 4:13 PM Ramoni Z. Sedano Azevedo <
ramoni.zsed...@gmail.com> wrote:
ine_always -mat_type aijcusparse
-vec_type cuda
args: -ksp_view
--Junchao Zhang
On Wed, Aug 23, 2023 at 9:27 AM TROTIGNON Laurent
wrote:
> Hello all,
>
> In the on line docs of PETSc, I found this paragraph :
>
>
>
> « PETSc uses a single source programming model where sol
-mat_type aijcusparse -vec_type cuda
Then, check again with nvidia-smi to see if GPU memory is evenly
allocated.
--Junchao Zhang
On Tue, Aug 22, 2023 at 3:03 PM Matthew Knepley wrote:
> On Tue, Aug 22, 2023 at 2:54 PM Vanella, Marcos (Fed) via petsc-users <
> petsc-users@mcs.anl.gov> wr
That is a good question. Looking at
https://slurm.schedmd.com/gres.html#GPU_Management, I was wondering if you
can share the output of your job so we can search CUDA_VISIBLE_DEVICES and
see how GPUs were allocated.
--Junchao Zhang
On Mon, Aug 21, 2023 at 2:38 PM Vanella, Marcos (Fed
Hi, Macros,
If you look at the PIDs of the nvidia-smi output, you will only find 8
unique PIDs, which is expected since you allocated 8 MPI ranks per node.
The duplicate PIDs are usually for threads spawned by the MPI runtime
(for example, progress threads in MPI implementation). So your job
I don't see a problem in the matrix assembly.
If you point me to your repo and show me how to build it, I can try to
reproduce.
--Junchao Zhang
On Mon, Aug 14, 2023 at 2:53 PM Vanella, Marcos (Fed) <
marcos.vane...@nist.gov> wrote:
> Hi Junchao, I've tried for my case using
Yeah, it looks like ex60 was run correctly.
Double check your code again and if you still run into errors, we can try
to reproduce on our end.
Thanks.
--Junchao Zhang
On Mon, Aug 14, 2023 at 1:05 PM Vanella, Marcos (Fed) <
marcos.vane...@nist.gov> wrote:
> Hi Junchao, I compiled and
-sub_pc_factor_mat_solver_type
cusparse
--Junchao Zhang
On Fri, Aug 11, 2023 at 4:36 PM Vanella, Marcos (Fed) <
marcos.vane...@nist.gov> wrote:
> Hi Junchao, thank you for the info. I compiled the main branch of PETSc in
> another machine that has the openmpi/4.1.4/gcc-11.2.1-cuda-11.7 toolchain
> and don&#x
ers should know how to fix them.
Thanks.
--Junchao Zhang
On Fri, Aug 11, 2023 at 2:22 PM Vanella, Marcos (Fed) <
marcos.vane...@nist.gov> wrote:
> Hi Junchao, thanks for the explanation. Is there some development
> documentation on the GPU work? I'm interested learning about it.
THRUSTINTARRAY32. We try to support generations of
cuda toolkits and thus have the current code.
Anyway, this should be fixed.
--Junchao Zhang
On Fri, Aug 11, 2023 at 1:07 PM Barry Smith wrote:
>
>We do not currently have any code for using 64 bit integer sizes on the
> GPUs.
>
&g
; guide to see
how to bind MPI processes to GPUs. If the job scheduler has done that, the
number of visible CUDA devices to a process might just appear to be 1,
making petsc's own mapping void.
Thanks.
--Junchao Zhang
On Fri, Aug 11, 2023 at 12:43 PM Vanella, Marcos (Fed) <
marcos.vane.
Hi, Marcos,
Could you build petsc in debug mode and then copy and paste the whole
error stack message?
Thanks
--Junchao Zhang
On Thu, Aug 10, 2023 at 5:51 PM Vanella, Marcos (Fed) via petsc-users <
petsc-users@mcs.anl.gov> wrote:
> Hi, I'm trying to run a parallel matrix ve
Hi, Philip,
It's my fault. I should follow up early that this problem was fixed by
https://gitlab.com/petsc/petsc/-/merge_requests/6586.
Could you try petsc/main?
Thanks.
--Junchao Zhang
On Tue, Jun 27, 2023 at 9:30 AM Fackler, Philip wrote:
> Good morning Junchao! I'm
Hi, Philip,
Thanks for reporting. I will have a look at the issue.
--Junchao Zhang
On Wed, Jun 7, 2023 at 9:30 AM Fackler, Philip via petsc-users <
petsc-users@mcs.anl.gov> wrote:
> I'm encountering a problem in xolotl. We initialize kokkos before
> initializing petsc. The
Hi, Maxime,
Thanks for the introduction. It sounds to me it makes sense to share the
COO information in MatDuplicate. The data structure consumes a lot of
memory, making it more justified. I will add it.
--Junchao Zhang
On Thu, Apr 27, 2023 at 2:15 PM Maxime Bouyges
wrote:
> Thanks
It sounds like we should do reference counting on the internal data
structures used by COO>.
--Junchao Zhang
On Wed, Apr 26, 2023 at 3:59 PM Barry Smith wrote:
>
>Yes, it looks like a bug since no one tested this situation.
>
>MatSetPreallocationCOO() is pretty heavy
I don't know OpenMP, but I saw these in your configure
OMP_PROC_BIND = 'TRUE'
OMP_PLACES = '{0:24}'
Try not to do any binding and let OS freely schedule threads.
--Junchao Zhang
On Fri, Apr 7, 2023 at 7:17 PM Astor Piaz wrote:
> Thanks for your reply Matt.
> OpenMP threads are able to spawn MPI processes
I am curious why you have this usage. Is it because that you want a pure
OpenMP code (i.e., not MPI capable) to call petsc?
--Junchao Zhang
On Fri, Apr 7, 2023 at 9:06 AM Astor Piaz wrote:
> Hello petsc-users,
> I am trying to use a
On Fri, Feb 10, 2023 at 8:16 AM Ces VLC wrote:
> Hi!
>
> I searched if it’s supported to link an application to two different
> builds of PETSc (one configured as float precision, and the other as
> double). The last post I found on that topic was from 2016 and it said it’s
> not recommended.
>
>
Hi, Philip,
I believe this MR https://gitlab.com/petsc/petsc/-/merge_requests/6030
would fix the problem. It is a fix to petsc/release, but you can
cherry-pick it to petsc/main.
Could you try that in your case?
Thanks.
--Junchao Zhang
On Fri, Jan 20, 2023 at 11:31 AM Junchao Zhang
wrote
.
--Junchao Zhang
On Fri, Jan 20, 2023 at 10:55 AM Fackler, Philip via petsc-users <
petsc-users@mcs.anl.gov> wrote:
> The following is the log_view output for the ported case using 4
Sorry, no progress. I guess that is because a vector was gotten but not
restored (e.g., VecRestoreArray() etc), causing host and device data not
synced. Maybe in your code, or in petsc code.
After the ECP AM, I will have more time on this bug.
Thanks.
--Junchao Zhang
On Fri, Jan 20, 2023 at 11
I try and output
> remoteoffsets between the Distribute Section and Create Section it throws
> the same segmentation fault.
>
> Thanks for the help
> Nicholas
>
> On Thu, Jan 12, 2023 at 11:56 PM Junchao Zhang
> wrote:
>
>>
>>
>> On Thu, Jan 12, 2023
eoffsets
call PetscSFCreateSectionSFF90(distributionSF, section_filt_l,
remoteoffsets, leafSection, distributionSF_dof, ierr)
call PetscIntArray1dDestroyF90(remoteOffsets,ierr) // free remoteoffsets
when not needed
Could you try it?
Sincerely
> Nicholas
>
> On Tue, Jan 10, 2023 at 4:42
1 - 100 of 366 matches
Mail list logo
