Hello Hong, Thank you so much.
I was using 24 processors currently and may considering using more. For this application, the fewer processors, the more efficient bjacobi will be, Is that right? As you suggest, these days I will start trying these combinations: minres/icc or minres/asm/icc or minres/icc/others ... What is more, how about jacobi? the most simplest preconditioner? It gives me the best performance actually when I was trying to use JD solver before. petsc_mass20.dat and petsc_stiff20.dat are the most challenging one for now. I have another set of matrices which have very similar eigenvalues distribution with ***20.dat. I think they should work under the same conditions. It will be fantastic, if you will have time to work out a better preconditioner for them. I deeply appreciate your kind help. Best regards, Jifeng Zhao On Tue, Aug 5, 2014 at 11:15 AM, [email protected] <[email protected]> wrote: > jifeng : > > > "The matrices come from Jifeng Zhao, Northwestern University, > > [email protected]. The stiffness and mass matrices arise from a > extended > > finite element model to predict phononic band gap of a three-dimensional > > two-phase composite with periodic SCC crystal structure. In addition, > > periodic boundary conditions are applied on the primary unit cell by > > employing Bloch wave analysis." > > Thanks. I used your description, but removed your email address: > ============================= > The matrices come from Jifeng Zhao, Northwestern University. The > stiffness and mass matrices arise from a extended finite element model > to predict phononic band gap of a three-dimensional two-phase > composite with periodic SCC crystal structure. In addition, periodic > boundary conditions are applied on the primary unit cell by employing > Bloch wave analysis. > > Compute first 40 eigensolutions > K x = lamda M x > K and M are real and symmetric (Aug. 2014) > ----------------------------------------------- > petsc_stiff20.dat: > type: seqaij > rows=1591812, cols=1591812, total: nonzeros=6.93869e+07 > petsc_mass20.dat > type: seqaij > rows=1591812, cols=1591812, total: nonzeros=4.09661e+07, > --------------------------------- > petsc_stiff33.dat: > type: seqaij > rows=1591812, cols=1591812, total: nonzeros=7.05468e+07, > petsc_mass33.dat: > type: seqaij > rows=1591812, cols=1591812, total: nonzeros=4.13649e+07 > ============================== > > As you see, K and M have different data structures (different non > zeros) which makes > MatAXPY() less efficient than the same data structure. You may create > K and M with same data structure, i.e., inserting additional zeros to > K or M. Then tells slepc that your matrices K and M have same non-zero > structure. > > > > ---------------------------- > > > > In addition, do want to let you know that I tried the runtime command you > > gave me using PETSC/3.4.4 (3.5 is not ready yet for me), it failed to > > converge. > > > > PETSC ERROR: ! > > PETSC ERROR: KSP did not converge (reason=DIVERGED_ITS)! > > I used minres/bjacobi/ icc with np=32 CPUs and got very slow > convergence. How many CPUs do you use? > When I run sequential minres/ icc, I got better convergence > (use '-st_ksp_monitor to monitor the convergence). If you use more > than 32 CPUs, then your bjacobi has more blocks than 32, thus a weaker > preconditioner than the one I used. Finding an efficient > preconditioner seems the key for your application. > > > > I also tried to different preconditioners: > > minres/bjacobi/ icc > > minres/bjacobi/ asm > > This should be minres/asm/icc > Use '-st_ksp_monitor', do you see improvement of asm over bjacobi? > You may also increase levels of icc. > > > minres/bjacobi/ gamg > Do not do this. > > > > All of them failed by the same error. I can try to increase the maximum > > iteration number to remove this error message, but it will converge very > > slow for sure. > > > > Since I am already using -aij command, I am not sure if these slow > > convergence performance will disappear if I upgrade to PETSC/3.5. Is > there > > anything else I can try on? > > The bug in sbaij format is irrelevant to the convergence problems. I > would suggest staying with aij format until we patch the petsc-3.5. > > When I find time, I may help you explore better preconditioners. Which > matrices are you interested, > petsc_mass20.dat and petsc_stiff20.dat, or something more challenging? > > I would suggest you sending email to petsc-maint instead of > petsc-users. I guess Jose also on the mailing list of petsc-maint. > > Hong > > > > > On Mon, Aug 4, 2014 at 11:18 PM, [email protected] < > [email protected]> > > wrote: > >> > >> jifeng : > >> > > >> > I just want to check with you if there is any updates on why the code > >> > crashes under -sbaij? > >> > >> It is a bug in petsc. I've fixed it in the brach hzhang/mataxpy-bugfix: > >> https://bitbucket.org/petsc/petsc/branch/hzhang/mataxpy-bugfix > >> > >> which will be merged to 'next' branch for tests, then patch to > petsc-3.5. > >> You can use this branch if you like. > >> > >> Again, I like to add your matrices into petsc matrix collection. > >> Can you give me a short sentence describing your matrices, e.g., > >> "The following matrices come from Rick Dean, Arco E&P Technology, > >> [email protected]. They arise from multiphase flow modeling of oil > >> reservoirs,..." > >> > >> Thanks, > >> Hong > >> > >> > >> > > >> > At the meanwhile, unfortunately PETSC-3.5 hasn't been installed on my > >> > server > >> > due to some technical problems. I hope technicians can get it done as > >> > soon > >> > as possible. I will update you when I have any new results. Thanks so > >> > much. > >> > > >> > Best regards, > >> > Jifeng > >> > > >> > > >> > On Wed, Jul 30, 2014 at 2:53 PM, jifeng zhao > >> > <[email protected]> wrote: > >> >> > >> >> Hello Hong, > >> >> > >> >> Thanks a lot. The result makes sense, but I think I will need higher > >> >> accuracy (smaller tolerance). I will take a try myself at the > >> >> meanwhile. > >> >> > >> >> Best regards, > >> >> Jifeng Zhao > >> >> > >> >> > >> >> On Wed, Jul 30, 2014 at 1:16 PM, [email protected] > >> >> <[email protected]> > >> >> wrote: > >> >>> > >> >>> Jifeng, > >> >>> Using petsc-3.5 and slepc-3.5 (maint branch), I tested your matrices > >> >>> > >> >>> mpiexec -n 32 ./ex7 -f1 $data/zhao/petsc_stiff20.dat -f2 > >> >>> $data/zhao/petsc_mass20.dat -mat_view ascii::ascii_info -mat_type > aij > >> >>> -eps_gen_hermitian -eps_type krylovschur -eps_target -0.1 > -st_ksp_type > >> >>> minres -st_pc_type bjacobi -st_sub_pc_type icc -st_ksp_rtol 1.e-2 > >> >>> -eps_tol 1.e-2 -eps_nev 40 -st_type sinvert -log_summary |& tee > >> >>> log.mat20.ex7.np32.tol-2 > >> >>> > >> >>> The output is attached. > >> >>> > >> >>> When using '-mat_type sbaij', I got same error as reported, which > >> >>> might be a bug in our libraries. I'll investigate it. > >> >>> Switching to '-mat_type aij' works (I'm not sure if the computed > >> >>> eigenvalues are meaningful though), but the linear solver > >> >>> minres/bjacobi/icc converges very slow, takes around 500 - 1000 > linear > >> >>> iterations for each eps iteration. You may experiment with stronger > >> >>> pc_type, e.g., '-pc_type asm' etc. > >> >>> > >> >>> I'll let you know why sbaij crashes. > >> >>> > >> >>> Hong > >> >>> > >> >>> On Mon, Jul 28, 2014 at 4:52 PM, Jose E. Roman <[email protected]> > >> >>> wrote: > >> >>> > > >> >>> > El 28/07/2014, a las 23:48, jifeng zhao escribió: > >> >>> > > >> >>> >> Hi Barry, > >> >>> >> > >> >>> >> Thanks for the reply. I see. I am still testing it and looking > for > >> >>> >> the > >> >>> >> bug. > >> >>> >> > >> >>> >> It is pretty weird that the error appears, since the code I am > >> >>> >> running > >> >>> >> is ex7 and I didn't change anything. > >> >>> >> > >> >>> >> My matrices are assembled in binary files separately. As I > >> >>> >> explained > >> >>> >> earlier, I have a series of matrices to solve, each of them have > >> >>> >> the same > >> >>> >> patterns but with different values. The way I assembly the matrix > >> >>> >> has proven > >> >>> >> to be correct for all the other cases. Only for this one, when > the > >> >>> >> matrix > >> >>> >> has an zero eigenvalue, ex7 failed to solve it. > >> >>> >> > >> >>> >> In fact, I don't understand why this "Out of range" error could > >> >>> >> possibly appear? 1. my matrices have the correct parrellel lay > up. > >> >>> >> 2. In > >> >>> >> ex7, I am not explicitly accessing to any elements at all! All I > >> >>> >> did is > >> >>> >> EPSSolve(); > >> >>> >> > >> >>> >> Any ideas? > >> >>> >> > >> >>> >> Best regards, > >> >>> >> Jifeng Zhao > >> >>> >> > >> >>> > > >> >>> > Tomorrow we will release slepc-3.5. I would suggest trying with > the > >> >>> > new > >> >>> > version. If the problem persist we can help to track down the > >> >>> > problem. > >> >>> > > >> >>> > Jose > >> >>> > > >> >> > >> >> > >> >> > >> >> > >> >> -- > >> >> Jifeng Zhao > >> >> PhD candidate at Northwestern University, US > >> >> Theoretical and Applied Mechanics Program > >> > > >> > > >> > > >> > > >> > -- > >> > Jifeng Zhao > >> > PhD candidate at Northwestern University, US > >> > Theoretical and Applied Mechanics Program > > > > > > > > > > -- > > Jifeng Zhao > > PhD candidate at Northwestern University, US > > Theoretical and Applied Mechanics Program > -- Jifeng Zhao PhD candidate at Northwestern University, US Theoretical and Applied Mechanics Program
