I am runing open-mpi 1.1.1-1 compiled from OFED1.1 which I downloaded from their website.
I am using SGE installed via OSCAR 5.0 and when running under SGE I get the "mca_mpool_openib_register: ibv_reg_mr(0x590000,528384) failed with error: Cannot allocate memory" error discussed at length in your FAQ. When I run from the command line using mpirun, I don't get the errors. Of course, I don't know how to tell if the code is actually using the IB interface instead of the GigE network... I tried the suggestions in the FAQ regarding setting the memlock parameter in /etc/security/limits.conf: and all the nodes return "unlimited" in response to "ulimit -l" after rebooting the nodes. The problem persists under SGE and still does not appear when simply using mpirun. I assumed it would work since openmpi 1.1.1 was included as working with SGE in OSCAR 5.0, but I don't know how different that version and the one included with OFED is. Any suggestions would be appreciated.
