On Tue, Oct 25, 2016 at 7:57 PM, Olivier Mesnard <[email protected] > wrote:
> On 25 October 2016 at 19:59, Matthew Knepley <[email protected]> wrote: > >> On Tue, Oct 25, 2016 at 6:22 PM, Barry Smith <[email protected]> wrote: >> >>> >>> > On Oct 25, 2016, at 5:39 PM, Olivier Mesnard < >>> [email protected]> wrote: >>> > >>> > On 25 October 2016 at 17:51, Barry Smith <[email protected]> wrote: >>> > >>> > Olivier, >>> > >>> > In theory you do not need to change anything else. Are you using a >>> different matrix object for the velocity_ksp object than the poisson_ksp >>> object? >>> > >>> > The matrix is different for the velocity_ksp and the poisson_ksp. >>> > >>> > The code change in PETSc is very little but we have a report from >>> another CFD user who also had problems with the change so there may be some >>> subtle bug that we can't figure out causing things to not behave properly. >>> > >>> > First run the 3.7.4 code with -poisson_ksp_view and verify that >>> when it prints the matrix information it prints something like has attached >>> null space if it does not print that it means that somehow the matrix is >>> not properly getting the matrix attached. >>> > >>> > When running with 3.7.4 and -poisson_ksp_view, the output shows that >>> the nullspace is not attached to the KSP (as it was with 3.5.4); however >>> the print statement is now under the Mat info (which is expected when >>> moving from KSPSetNullSpace to MatSetNullSpace?). >>> >>> Good, this is how it should be. >>> > >>> > Though older versions had MatSetNullSpace() they didn't >>> necessarily associate it with the KSP so it was not expected to work as a >>> replacement for KSPSetNullSpace() with older versions. >>> > >>> > Because our other user had great difficulty trying to debug the >>> issue feel free to send us at [email protected] your code with >>> instructions on building and running and we can try to track down the >>> problem. Better than hours and hours spent with fruitless email. We will, >>> of course, not distribute the code and will delete in when we are finished >>> with it. >>> > >>> > The code is open-source and hosted on GitHub ( >>> https://github.com/barbagroup/PetIBM). >>> > I just pushed the branches `feature-compatible-petsc-3.7` and >>> `revert-compatible-petsc-3.5` that I used to observe this problem. >>> > >>> Thanks, I'll get back to you if I discover anything >> >> >> Obviously GAMG is behaving quite differently (1 vs 2 levels and a much >> sparser coarse problem in 3.7). >> >> Could you try one thing for me before we start running it? Run with >> >> -poisson_mg_coarse_sub_pc_type svd >> >> and see what happens on 2 procs for 3.7? >> >> Hi Matt, > > With > -poisson_mg_coarse_sub_pc_type svd > , > it ran normally on 1 proc but not on 2 procs the end of the output says: > > "** On entry to DGESVD parameter number 6 had an illegal value > " > Something is wrong with your 3.7 installation. That parameter is a simple size, so I suspect memory corruption. Run with valgrind. Thanks, Matt > I attached the log file. > > > >> Thanks, >> >> Matt >> >> >>> >>> > PETSc (both 3.5.4 and 3.7.4) was configured as follow: >>> > export PETSC_ARCH="linux-gnu-dbg" >>> > ./configure --PETSC_ARCH=$PETSC_ARCH \ >>> > --with-cc=gcc \ >>> > --with-cxx=g++ \ >>> > --with-fc=gfortran \ >>> > --COPTFLAGS="-O0" \ >>> > --CXXOPTFLAGS="-O0" \ >>> > --FOPTFLAGS="-O0" \ >>> > --with-debugging=1 \ >>> > --download-fblaslapack \ >>> > --download-mpich \ >>> > --download-hypre \ >>> > --download-yaml \ >>> > --with-x=1 >>> > >>> > Our code was built using the following commands: >>> > mkdir petibm-build >>> > cd petibm-build >>> > export PETSC_DIR=<directory of PETSc> >>> > export PETSC_ARCH="linux-gnu-dbg" >>> > export PETIBM_DIR=<directory of PetIBM git repo> >>> > $PETIBM_DIR/configure --prefix=$PWD \ >>> > CXX=$PETSC_DIR/$PETSC_ARCH/bin/mpicxx \ >>> > CXXFLAGS="-g -O0 -std=c++11" >>> > make all >>> > make install >>> > >>> > Then >>> > cd examples >>> > make examples >>> > >>> > The example of the lid-driven cavity I was talking about can be found >>> in the folder `examples/2d/convergence/lidDrivenCavity20/20/` >>> > >>> > To run it: >>> > mpiexec -n N <path-to-petibm-build>/bin/petibm2d -directory >>> <path-to-example> >>> > >>> > Let me know if you need more info. Thank you. >>> > >>> > Barry >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > > On Oct 25, 2016, at 4:38 PM, Olivier Mesnard < >>> [email protected]> wrote: >>> > > >>> > > Hi all, >>> > > >>> > > We develop a CFD code using the PETSc library that solves the >>> Navier-Stokes equations using the fractional-step method from Perot (1993). >>> > > At each time-step, we solve two systems: one for the velocity field, >>> the other, a Poisson system, for the pressure field. >>> > > One of our test-cases is a 2D lid-driven cavity flow (Re=100) on a >>> 20x20 grid using 1 or 2 procs. >>> > > For the Poisson system, we usually use CG preconditioned with GAMG. >>> > > >>> > > So far, we have been using PETSc-3.5.4, and we would like to update >>> the code with the latest release: 3.7.4. >>> > > >>> > > As suggested in the changelog of 3.6, we replaced the routine >>> `KSPSetNullSpace()` with `MatSetNullSpace()`. >>> > > >>> > > Here is the list of options we use to configure the two solvers: >>> > > * Velocity solver: prefix `-velocity_` >>> > > -velocity_ksp_type bcgs >>> > > -velocity_ksp_rtol 1.0E-08 >>> > > -velocity_ksp_atol 0.0 >>> > > -velocity_ksp_max_it 10000 >>> > > -velocity_pc_type jacobi >>> > > -velocity_ksp_view >>> > > -velocity_ksp_monitor_true_residual >>> > > -velocity_ksp_converged_reason >>> > > * Poisson solver: prefix `-poisson_` >>> > > -poisson_ksp_type cg >>> > > -poisson_ksp_rtol 1.0E-08 >>> > > -poisson_ksp_atol 0.0 >>> > > -poisson_ksp_max_it 20000 >>> > > -poisson_pc_type gamg >>> > > -poisson_pc_gamg_type agg >>> > > -poisson_pc_gamg_agg_nsmooths 1 >>> > > -poissonksp_view >>> > > -poisson_ksp_monitor_true_residual >>> > > -poisson_ksp_converged_reason >>> > > >>> > > With 3.5.4, the case runs normally on 1 or 2 procs. >>> > > With 3.7.4, the case runs normally on 1 proc but not on 2. >>> > > Why? The Poisson solver diverges because of an indefinite >>> preconditioner (only with 2 procs). >>> > > >>> > > We also saw that the routine `MatSetNullSpace()` was already >>> available in 3.5.4. >>> > > With 3.5.4, replacing `KSPSetNullSpace()` with `MatSetNullSpace()` >>> led to the Poisson solver diverging because of an indefinite matrix (on 1 >>> and 2 procs). >>> > > >>> > > Thus, we were wondering if we needed to update something else for >>> the KSP, and not just modifying the name of the routine? >>> > > >>> > > I have attached the output files from the different cases: >>> > > * `run-petsc-3.5.4-n1.log` (3.5.4, `KSPSetNullSpace()`, n=1) >>> > > * `run-petsc-3.5.4-n2.log` >>> > > * `run-petsc-3.5.4-nsp-n1.log` (3.5.4, `MatSetNullSpace()`, n=1) >>> > > * `run-petsc-3.5.4-nsp-n2.log` >>> > > * `run-petsc-3.7.4-n1.log` (3.7.4, `MatSetNullSpace()`, n=1) >>> > > * `run-petsc-3.7.4-n2.log` >>> > > >>> > > Thank you for your help, >>> > > Olivier >>> > > <run-petsc-3.5.4-n1.log><run-petsc-3.5.4-n2.log><run-petsc-3 >>> .5.4-nsp-n1.log><run-petsc-3.5.4-nsp-n2.log><run-petsc-3.7.4 >>> -n1.log><run-petsc-3.7.4-n2.log> >>> > >>> > >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
