Remove your option '-vecload_block_size 10'. Hong
On Wed, May 24, 2017 at 3:06 PM, Danyang Su <danyang...@gmail.com> wrote: > Dear Hong, > > I just tested with different number of processors for the same matrix. It > sometimes got "ERROR: Arguments are incompatible" for different number of > processors. It works fine using 4, 8, or 24 processors, but failed with > "ERROR: Arguments are incompatible" using 16 or 48 processors. The error > information is attached. I tested this on my local computer with 6 cores 12 > threads. Any suggestion on this? > > Thanks, > > Danyang > > On 17-05-24 12:28 PM, Danyang Su wrote: > > Hi Hong, > > Awesome. Thanks for testing the case. I will try your options for the code > and get back to you later. > > Regards, > > Danyang > > On 17-05-24 12:21 PM, Hong wrote: > > Danyang : > I tested your data. > Your matrices encountered zero pivots, e.g. > petsc/src/ksp/ksp/examples/tutorials (master) > $ mpiexec -n 24 ./ex10 -f0 a_react_in_2.bin -rhs b_react_in_2.bin > -ksp_monitor -ksp_error_if_not_converged > > [15]PETSC ERROR: Zero pivot in LU factorization: > http://www.mcs.anl.gov/petsc/documentation/faq.html#zeropivot > [15]PETSC ERROR: Zero pivot row 1249 value 2.05808e-14 tolerance > 2.22045e-14 > ... > > Adding option '-sub_pc_factor_shift_type nonzero', I got > mpiexec -n 24 ./ex10 -f0 a_react_in_2.bin -rhs b_react_in_2.bin > -ksp_monitor -ksp_error_if_not_converged -sub_pc_factor_shift_type nonzero > -mat_view ascii::ascii_info > > Mat Object: 24 MPI processes > type: mpiaij > rows=450000, cols=450000 > total: nonzeros=6991400, allocated nonzeros=6991400 > total number of mallocs used during MatSetValues calls =0 > not using I-node (on process 0) routines > 0 KSP Residual norm 5.849777711755e+01 > 1 KSP Residual norm 6.824179430230e-01 > 2 KSP Residual norm 3.994483555787e-02 > 3 KSP Residual norm 6.085841461433e-03 > 4 KSP Residual norm 8.876162583511e-04 > 5 KSP Residual norm 9.407780665278e-05 > Number of iterations = 5 > Residual norm 0.00542891 > > Hong > >> Hi Matt, >> >> Yes. The matrix is 450000x450000 sparse. The hypre takes hundreds of >> iterates, not for all but in most of the timesteps. The matrix is not well >> conditioned, with nonzero entries range from 1.0e-29 to 1.0e2. I also made >> double check if there is anything wrong in the parallel version, however, >> the matrix is the same with sequential version except some round error >> which is relatively very small. Usually for those not well conditioned >> matrix, direct solver should be faster than iterative solver, right? But >> when I use the sequential iterative solver with ILU prec developed almost >> 20 years go by others, the solver converge fast with appropriate >> factorization level. In other words, when I use 24 processor using hypre, >> the speed is almost the same as as the old sequential iterative solver >> using 1 processor. >> >> I use most of the default configuration for the general case with pretty >> good speedup. And I am not sure if I miss something for this problem. >> >> Thanks, >> >> Danyang >> >> On 17-05-24 11:12 AM, Matthew Knepley wrote: >> >> On Wed, May 24, 2017 at 12:50 PM, Danyang Su <danyang...@gmail.com> >> wrote: >> >>> Hi Matthew and Barry, >>> >>> Thanks for the quick response. >>> >>> I also tried superlu and mumps, both work but it is about four times >>> slower than ILU(dt) prec through hypre, with 24 processors I have tested. >>> >> You mean the total time is 4x? And you are taking hundreds of iterates? >> That seems hard to believe, unless you are dropping >> a huge number of elements. >> >>> When I look into the convergence information, the method using ILU(dt) >>> still takes 200 to 3000 linear iterations for each newton iteration. One >>> reason is this equation is hard to solve. As for the general cases, the >>> same method works awesome and get very good speedup. >>> >> I do not understand what you mean here. >> >>> I also doubt if I use hypre correctly for this case. Is there anyway to >>> check this problem, or is it possible to increase the factorization level >>> through hypre? >>> >> I don't know. >> >> Matt >> >>> Thanks, >>> >>> Danyang >>> >>> On 17-05-24 04:59 AM, Matthew Knepley wrote: >>> >>> On Wed, May 24, 2017 at 2:21 AM, Danyang Su <danyang...@gmail.com> >>> wrote: >>> >>>> Dear All, >>>> >>>> I use PCFactorSetLevels for ILU and PCFactorSetFill for other >>>> preconditioning in my code to help solve the problems that the default >>>> option is hard to solve. However, I found the latter one, PCFactorSetFill >>>> does not take effect for my problem. The matrices and rhs as well as the >>>> solutions are attached from the link below. I obtain the solution using >>>> hypre preconditioner and it takes 7 and 38 iterations for matrix 1 and >>>> matrix 2. However, if I use other preconditioner, the solver just failed at >>>> the first matrix. I have tested this matrix using the native sequential >>>> solver (not PETSc) with ILU preconditioning. If I set the incomplete >>>> factorization level to 0, this sequential solver will take more than 100 >>>> iterations. If I increase the factorization level to 1 or more, it just >>>> takes several iterations. This remind me that the PC factor for this >>>> matrices should be increased. However, when I tried it in PETSc, it just >>>> does not work. >>>> >>>> Matrix and rhs can be obtained from the link below. >>>> >>>> https://eilinator.eos.ubc.ca:8443/index.php/s/CalUcq9CMeblk4R >>>> >>>> Would anyone help to check if you can make this work by increasing the >>>> PC factor level or fill? >>>> >>> >>> We have ILU(k) supported in serial. However ILU(dt) which takes a >>> tolerance only works through Hypre >>> >>> http://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html >>> >>> I recommend you try SuperLU or MUMPS, which can both be downloaded >>> automatically by configure, and >>> do a full sparse LU. >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Thanks and regards, >>>> >>>> Danyang >>>> >>>> >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> http://www.caam.rice.edu/~mk51/ >>> >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> http://www.caam.rice.edu/~mk51/ >> >> >> > > >