jifeng : > "The matrices come from Jifeng Zhao, Northwestern University, > [email protected]. The stiffness and mass matrices arise from a extended > finite element model to predict phononic band gap of a three-dimensional > two-phase composite with periodic SCC crystal structure. In addition, > periodic boundary conditions are applied on the primary unit cell by > employing Bloch wave analysis."
Thanks. I used your description, but removed your email address: ============================= The matrices come from Jifeng Zhao, Northwestern University. The stiffness and mass matrices arise from a extended finite element model to predict phononic band gap of a three-dimensional two-phase composite with periodic SCC crystal structure. In addition, periodic boundary conditions are applied on the primary unit cell by employing Bloch wave analysis. Compute first 40 eigensolutions K x = lamda M x K and M are real and symmetric (Aug. 2014) ----------------------------------------------- petsc_stiff20.dat: type: seqaij rows=1591812, cols=1591812, total: nonzeros=6.93869e+07 petsc_mass20.dat type: seqaij rows=1591812, cols=1591812, total: nonzeros=4.09661e+07, --------------------------------- petsc_stiff33.dat: type: seqaij rows=1591812, cols=1591812, total: nonzeros=7.05468e+07, petsc_mass33.dat: type: seqaij rows=1591812, cols=1591812, total: nonzeros=4.13649e+07 ============================== As you see, K and M have different data structures (different non zeros) which makes MatAXPY() less efficient than the same data structure. You may create K and M with same data structure, i.e., inserting additional zeros to K or M. Then tells slepc that your matrices K and M have same non-zero structure. > > ---------------------------- > > In addition, do want to let you know that I tried the runtime command you > gave me using PETSC/3.4.4 (3.5 is not ready yet for me), it failed to > converge. > > PETSC ERROR: ! > PETSC ERROR: KSP did not converge (reason=DIVERGED_ITS)! I used minres/bjacobi/ icc with np=32 CPUs and got very slow convergence. How many CPUs do you use? When I run sequential minres/ icc, I got better convergence (use '-st_ksp_monitor to monitor the convergence). If you use more than 32 CPUs, then your bjacobi has more blocks than 32, thus a weaker preconditioner than the one I used. Finding an efficient preconditioner seems the key for your application. > > I also tried to different preconditioners: > minres/bjacobi/ icc > minres/bjacobi/ asm This should be minres/asm/icc Use '-st_ksp_monitor', do you see improvement of asm over bjacobi? You may also increase levels of icc. > minres/bjacobi/ gamg Do not do this. > > All of them failed by the same error. I can try to increase the maximum > iteration number to remove this error message, but it will converge very > slow for sure. > > Since I am already using -aij command, I am not sure if these slow > convergence performance will disappear if I upgrade to PETSC/3.5. Is there > anything else I can try on? The bug in sbaij format is irrelevant to the convergence problems. I would suggest staying with aij format until we patch the petsc-3.5. When I find time, I may help you explore better preconditioners. Which matrices are you interested, petsc_mass20.dat and petsc_stiff20.dat, or something more challenging? I would suggest you sending email to petsc-maint instead of petsc-users. I guess Jose also on the mailing list of petsc-maint. Hong > > On Mon, Aug 4, 2014 at 11:18 PM, [email protected] <[email protected]> > wrote: >> >> jifeng : >> > >> > I just want to check with you if there is any updates on why the code >> > crashes under -sbaij? >> >> It is a bug in petsc. I've fixed it in the brach hzhang/mataxpy-bugfix: >> https://bitbucket.org/petsc/petsc/branch/hzhang/mataxpy-bugfix >> >> which will be merged to 'next' branch for tests, then patch to petsc-3.5. >> You can use this branch if you like. >> >> Again, I like to add your matrices into petsc matrix collection. >> Can you give me a short sentence describing your matrices, e.g., >> "The following matrices come from Rick Dean, Arco E&P Technology, >> [email protected]. They arise from multiphase flow modeling of oil >> reservoirs,..." >> >> Thanks, >> Hong >> >> >> > >> > At the meanwhile, unfortunately PETSC-3.5 hasn't been installed on my >> > server >> > due to some technical problems. I hope technicians can get it done as >> > soon >> > as possible. I will update you when I have any new results. Thanks so >> > much. >> > >> > Best regards, >> > Jifeng >> > >> > >> > On Wed, Jul 30, 2014 at 2:53 PM, jifeng zhao >> > <[email protected]> wrote: >> >> >> >> Hello Hong, >> >> >> >> Thanks a lot. The result makes sense, but I think I will need higher >> >> accuracy (smaller tolerance). I will take a try myself at the >> >> meanwhile. >> >> >> >> Best regards, >> >> Jifeng Zhao >> >> >> >> >> >> On Wed, Jul 30, 2014 at 1:16 PM, [email protected] >> >> <[email protected]> >> >> wrote: >> >>> >> >>> Jifeng, >> >>> Using petsc-3.5 and slepc-3.5 (maint branch), I tested your matrices >> >>> >> >>> mpiexec -n 32 ./ex7 -f1 $data/zhao/petsc_stiff20.dat -f2 >> >>> $data/zhao/petsc_mass20.dat -mat_view ascii::ascii_info -mat_type aij >> >>> -eps_gen_hermitian -eps_type krylovschur -eps_target -0.1 -st_ksp_type >> >>> minres -st_pc_type bjacobi -st_sub_pc_type icc -st_ksp_rtol 1.e-2 >> >>> -eps_tol 1.e-2 -eps_nev 40 -st_type sinvert -log_summary |& tee >> >>> log.mat20.ex7.np32.tol-2 >> >>> >> >>> The output is attached. >> >>> >> >>> When using '-mat_type sbaij', I got same error as reported, which >> >>> might be a bug in our libraries. I'll investigate it. >> >>> Switching to '-mat_type aij' works (I'm not sure if the computed >> >>> eigenvalues are meaningful though), but the linear solver >> >>> minres/bjacobi/icc converges very slow, takes around 500 - 1000 linear >> >>> iterations for each eps iteration. You may experiment with stronger >> >>> pc_type, e.g., '-pc_type asm' etc. >> >>> >> >>> I'll let you know why sbaij crashes. >> >>> >> >>> Hong >> >>> >> >>> On Mon, Jul 28, 2014 at 4:52 PM, Jose E. Roman <[email protected]> >> >>> wrote: >> >>> > >> >>> > El 28/07/2014, a las 23:48, jifeng zhao escribió: >> >>> > >> >>> >> Hi Barry, >> >>> >> >> >>> >> Thanks for the reply. I see. I am still testing it and looking for >> >>> >> the >> >>> >> bug. >> >>> >> >> >>> >> It is pretty weird that the error appears, since the code I am >> >>> >> running >> >>> >> is ex7 and I didn't change anything. >> >>> >> >> >>> >> My matrices are assembled in binary files separately. As I >> >>> >> explained >> >>> >> earlier, I have a series of matrices to solve, each of them have >> >>> >> the same >> >>> >> patterns but with different values. The way I assembly the matrix >> >>> >> has proven >> >>> >> to be correct for all the other cases. Only for this one, when the >> >>> >> matrix >> >>> >> has an zero eigenvalue, ex7 failed to solve it. >> >>> >> >> >>> >> In fact, I don't understand why this "Out of range" error could >> >>> >> possibly appear? 1. my matrices have the correct parrellel lay up. >> >>> >> 2. In >> >>> >> ex7, I am not explicitly accessing to any elements at all! All I >> >>> >> did is >> >>> >> EPSSolve(); >> >>> >> >> >>> >> Any ideas? >> >>> >> >> >>> >> Best regards, >> >>> >> Jifeng Zhao >> >>> >> >> >>> > >> >>> > Tomorrow we will release slepc-3.5. I would suggest trying with the >> >>> > new >> >>> > version. If the problem persist we can help to track down the >> >>> > problem. >> >>> > >> >>> > Jose >> >>> > >> >> >> >> >> >> >> >> >> >> -- >> >> Jifeng Zhao >> >> PhD candidate at Northwestern University, US >> >> Theoretical and Applied Mechanics Program >> > >> > >> > >> > >> > -- >> > Jifeng Zhao >> > PhD candidate at Northwestern University, US >> > Theoretical and Applied Mechanics Program > > > > > -- > Jifeng Zhao > PhD candidate at Northwestern University, US > Theoretical and Applied Mechanics Program
