We should be able to overlap PetscSFReduce. I will have a look. Thanks. --Junchao Zhang
On Mon, Jul 22, 2019 at 5:23 AM Stefano Zampini <[email protected]<mailto:[email protected]>> wrote: Junchao, I found an issue with PetscSFReduceBegin/End. It seems that we can no longer do (always worked before) ierr = PetscSFReduceBegin(sf,MPIU_INT,leaf,root,MPI_MAX);CHKERRQ(ierr); ierr = PetscSFReduceBegin(sf,MPIU_INT,leaf2,root2,MPI_MAX);CHKERRQ(ierr); ierr = PetscSFReduceEnd(sf,MPIU_INT,leaf,root,MPI_MAX);CHKERRQ(ierr); ierr = PetscSFReduceEnd(sf,MPIU_INT,leaf2,root2,MPI_MAX);CHKERRQ(ierr); You can reproduce with the currrent master $ cd $PETSC_DIR/src/ksp/ksp/examples/tutorials $ make ex71 $ mpiexec -n 4 ./ex71 -pde_type Elasticity -cells 7,9 -dim 2 -pc_bddc_levels 1 -pc_bddc_coarsening_ratio 2 -ksp_error_if_not_converged -pc_bddc_monolithic -pc_bddc_use_faces -pc_bddc_coarse_pc_bddc_corner_selection -pc_bddc_coarse_l1_pc_bddc_corner_selection -mat_partitioning_type average -pc_bddc_coarse_pc_bddc_use_deluxe_scaling -pc_bddc_coarse_sub_schurs_mat_solver_type petsc The attached patch shows that if we instead do ierr = PetscSFReduceBegin(sf,MPIU_INT,leaf,root,MPI_MAX);CHKERRQ(ierr); ierr = PetscSFReduceEnd(sf,MPIU_INT,leaf,root,MPI_MAX);CHKERRQ(ierr); ierr = PetscSFReduceBegin(sf,MPIU_INT,leaf2,root2,MPI_MAX);CHKERRQ(ierr); ierr = PetscSFReduceEnd(sf,MPIU_INT,leaf2,root2,MPI_MAX);CHKERRQ(ierr); everything works properly. I think the issue is with the SF GetPack mechanism, but I couldn't figure out where. -- Stefano
