Hello Ralph,

Attached below is the verbose output for a failing machine and a passing
machine.

Thanks,
Adam LeBlanc

On Thu, Nov 1, 2018 at 1:41 PM Adam LeBlanc <alebl...@iol.unh.edu> wrote:

>
>
> ---------- Forwarded message ---------
> From: Ralph H Castain <r...@open-mpi.org>
> Date: Thu, Nov 1, 2018 at 1:07 PM
> Subject: Re: [OMPI users] Bug with Open-MPI Processor Count
> To: Open MPI Users <users@lists.open-mpi.org>
>
>
> Set rmaps_base_verbose=10 for debugging output
>
> Sent from my iPhone
>
> On Nov 1, 2018, at 9:31 AM, Adam LeBlanc <alebl...@iol.unh.edu> wrote:
>
> The version by the way for Open-MPI is 3.1.2.
>
> -Adam LeBlanc
>
> On Thu, Nov 1, 2018 at 12:05 PM Adam LeBlanc <alebl...@iol.unh.edu> wrote:
>
>> Hello, I am an employee of the UNH InterOperability Lab, and we are in
>> the process of testing OFED-4.17-RC1 for the OpenFabrics Alliance. We have
>> purchased some new hardware that has one processor, and noticed an issue
>> when running mpi jobs on nodes that do not have similar processor counts.
>> If we launch the MPI job from a node that has 2 processors, it will fail
>> and stating there are not enough resources and will not start the run, like
>> so:
>> --------------------------------------------------------------------------
>> There are not enough slots available in the system to satisfy the 14 slots
>> that were requested by the application:   IMB-MPI1 Either request fewer
>> slots for your application, or make more slots available for use.
>> --------------------------------------------------------------------------
>> If we launch the MPI job from the node with one processor, without changing
>> the mpirun command at all, it runs as expected. Here is the command being
>> run: mpirun --mca btl_openib_warn_no_device_params_found 0 --mca
>> orte_base_help_aggregate 0 --mca btl openib,vader,self --mca pml ob1 --mca
>> btl_openib_receive_queues P,65536,120,64,32 -hostfile
>> /home/soesterreich/ce-mpi-hosts IMB-MPI1 Here is the hostfile being used:
>> farbauti-ce.ofa.iol.unh.edu slots=1 hyperion-ce.ofa.iol.unh.edu slots=1
>> io-ce.ofa.iol.unh.edu slots=1 jarnsaxa-ce.ofa.iol.unh.edu slots=1
>> rhea-ce.ofa.iol.unh.edu slots=1 tarqeq-ce.ofa.iol.unh.edu slots=1
>> tarvos-ce.ofa.iol.unh.edu slots=1 This seems like a bug and we would
>> like some help to explain and fix what is happening. The IBTA plugfest saw
>> similar behaviours, so this should be reproduceable. Thanks, Adam LeBlanc
>>
> _______________________________________________
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
>
> _______________________________________________
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
>
[0_01:49:28_aleblanc@hyperion]{~}$ > mpirun --mca 
btl_openib_warn_no_device_params_found 0 --mca orte_base_help_aggregate 0 --mca 
btl openib,vader,self --mca pml ob1 --mca btl_openib_receive_queues 
P,65536,120,64,32 -hostfile /home/soesterreich/ce-mpi-hosts --mca 
rmaps_base_verbose 10 IMB-MPI1
[hyperion.ofa.iol.unh.edu:05190] mca: base: components_register: registering 
framework rmaps components
[hyperion.ofa.iol.unh.edu:05190] mca: base: components_register: found loaded 
component resilient
[hyperion.ofa.iol.unh.edu:05190] mca: base: components_register: component 
resilient register function successful
[hyperion.ofa.iol.unh.edu:05190] mca: base: components_register: found loaded 
component seq
[hyperion.ofa.iol.unh.edu:05190] mca: base: components_register: component seq 
register function successful
[hyperion.ofa.iol.unh.edu:05190] mca: base: components_register: found loaded 
component ppr
[hyperion.ofa.iol.unh.edu:05190] mca: base: components_register: component ppr 
register function successful
[hyperion.ofa.iol.unh.edu:05190] mca: base: components_register: found loaded 
component mindist
[hyperion.ofa.iol.unh.edu:05190] mca: base: components_register: component 
mindist register function successful
[hyperion.ofa.iol.unh.edu:05190] mca: base: components_register: found loaded 
component round_robin
[hyperion.ofa.iol.unh.edu:05190] mca: base: components_register: component 
round_robin register function successful
[hyperion.ofa.iol.unh.edu:05190] mca: base: components_register: found loaded 
component rank_file
[hyperion.ofa.iol.unh.edu:05190] mca: base: components_register: component 
rank_file register function successful
[hyperion.ofa.iol.unh.edu:05190] [[63394,0],0] rmaps:base set policy with NULL 
device NONNULL
[hyperion.ofa.iol.unh.edu:05190] mca: base: components_open: opening rmaps 
components
[hyperion.ofa.iol.unh.edu:05190] mca: base: components_open: found loaded 
component resilient
[hyperion.ofa.iol.unh.edu:05190] mca: base: components_open: component 
resilient open function successful
[hyperion.ofa.iol.unh.edu:05190] mca: base: components_open: found loaded 
component seq
[hyperion.ofa.iol.unh.edu:05190] mca: base: components_open: component seq open 
function successful
[hyperion.ofa.iol.unh.edu:05190] mca: base: components_open: found loaded 
component ppr
[hyperion.ofa.iol.unh.edu:05190] mca: base: components_open: component ppr open 
function successful
[hyperion.ofa.iol.unh.edu:05190] mca: base: components_open: found loaded 
component mindist
[hyperion.ofa.iol.unh.edu:05190] mca: base: components_open: component mindist 
open function successful
[hyperion.ofa.iol.unh.edu:05190] mca: base: components_open: found loaded 
component round_robin
[hyperion.ofa.iol.unh.edu:05190] mca: base: components_open: component 
round_robin open function successful
[hyperion.ofa.iol.unh.edu:05190] mca: base: components_open: found loaded 
component rank_file
[hyperion.ofa.iol.unh.edu:05190] mca: base: components_open: component 
rank_file open function successful
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps:select: checking available component 
resilient
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps:select: Querying component 
[resilient]
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps:select: checking available component 
seq
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps:select: Querying component [seq]
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps:select: checking available component 
ppr
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps:select: Querying component [ppr]
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps:select: checking available component 
mindist
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps:select: Querying component [mindist]
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps:select: checking available component 
round_robin
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps:select: Querying component 
[round_robin]
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps:select: checking available component 
rank_file
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps:select: Querying component 
[rank_file]
[hyperion.ofa.iol.unh.edu:05190] [[63394,0],0]: Final mapper priorities
[hyperion.ofa.iol.unh.edu:05190]        Mapper: ppr Priority: 90
[hyperion.ofa.iol.unh.edu:05190]        Mapper: seq Priority: 60
[hyperion.ofa.iol.unh.edu:05190]        Mapper: resilient Priority: 40
[hyperion.ofa.iol.unh.edu:05190]        Mapper: mindist Priority: 20
[hyperion.ofa.iol.unh.edu:05190]        Mapper: round_robin Priority: 10
[hyperion.ofa.iol.unh.edu:05190]        Mapper: rank_file Priority: 0
[tarqeq.ofa.iol.unh.edu:04778] mca: base: components_register: registering 
framework rmaps components
[tarqeq.ofa.iol.unh.edu:04778] mca: base: components_register: found loaded 
component seq
[tarqeq.ofa.iol.unh.edu:04778] mca: base: components_register: component seq 
register function successful
[tarqeq.ofa.iol.unh.edu:04778] mca: base: components_register: found loaded 
component ppr
[tarqeq.ofa.iol.unh.edu:04778] mca: base: components_register: component ppr 
register function successful
[tarqeq.ofa.iol.unh.edu:04778] mca: base: components_register: found loaded 
component round_robin
[tarqeq.ofa.iol.unh.edu:04778] mca: base: components_register: component 
round_robin register function successful
[tarqeq.ofa.iol.unh.edu:04778] mca: base: components_register: found loaded 
component rank_file
[tarqeq.ofa.iol.unh.edu:04778] mca: base: components_register: component 
rank_file register function successful
[tarqeq.ofa.iol.unh.edu:04778] mca: base: components_register: found loaded 
component resilient
[tarqeq.ofa.iol.unh.edu:04778] mca: base: components_register: component 
resilient register function successful
[tarqeq.ofa.iol.unh.edu:04778] mca: base: components_register: found loaded 
component mindist
[tarqeq.ofa.iol.unh.edu:04778] mca: base: components_register: component 
mindist register function successful
[tarqeq.ofa.iol.unh.edu:04778] [[63394,0],5] rmaps:base set policy with NULL 
device NONNULL
[tarqeq.ofa.iol.unh.edu:04778] mca: base: components_open: opening rmaps 
components
[tarqeq.ofa.iol.unh.edu:04778] mca: base: components_open: found loaded 
component seq
[tarqeq.ofa.iol.unh.edu:04778] mca: base: components_open: component seq open 
function successful
[tarqeq.ofa.iol.unh.edu:04778] mca: base: components_open: found loaded 
component ppr
[tarqeq.ofa.iol.unh.edu:04778] mca: base: components_open: component ppr open 
function successful
[tarqeq.ofa.iol.unh.edu:04778] mca: base: components_open: found loaded 
component round_robin
[tarqeq.ofa.iol.unh.edu:04778] mca: base: components_open: component 
round_robin open function successful
[tarqeq.ofa.iol.unh.edu:04778] mca: base: components_open: found loaded 
component rank_file
[tarqeq.ofa.iol.unh.edu:04778] mca: base: components_open: component rank_file 
open function successful
[tarqeq.ofa.iol.unh.edu:04778] mca: base: components_open: found loaded 
component resilient
[tarqeq.ofa.iol.unh.edu:04778] mca: base: components_open: component resilient 
open function successful
[tarqeq.ofa.iol.unh.edu:04778] mca: base: components_open: found loaded 
component mindist
[tarqeq.ofa.iol.unh.edu:04778] mca: base: components_open: component mindist 
open function successful
[tarqeq.ofa.iol.unh.edu:04778] mca:rmaps:select: checking available component 
seq
[tarqeq.ofa.iol.unh.edu:04778] mca:rmaps:select: Querying component [seq]
[tarqeq.ofa.iol.unh.edu:04778] mca:rmaps:select: checking available component 
ppr
[tarqeq.ofa.iol.unh.edu:04778] mca:rmaps:select: Querying component [ppr]
[tarqeq.ofa.iol.unh.edu:04778] mca:rmaps:select: checking available component 
round_robin
[tarqeq.ofa.iol.unh.edu:04778] mca:rmaps:select: Querying component 
[round_robin]
[tarqeq.ofa.iol.unh.edu:04778] mca:rmaps:select: checking available component 
rank_file
[tarqeq.ofa.iol.unh.edu:04778] mca:rmaps:select: Querying component [rank_file]
[tarqeq.ofa.iol.unh.edu:04778] mca:rmaps:select: checking available component 
resilient
[tarqeq.ofa.iol.unh.edu:04778] mca:rmaps:select: Querying component [resilient]
[tarqeq.ofa.iol.unh.edu:04778] mca:rmaps:select: checking available component 
mindist
[tarqeq.ofa.iol.unh.edu:04778] mca:rmaps:select: Querying component [mindist]
[tarqeq.ofa.iol.unh.edu:04778] [[63394,0],5]: Final mapper priorities
[tarqeq.ofa.iol.unh.edu:04778]  Mapper: ppr Priority: 90
[tarqeq.ofa.iol.unh.edu:04778]  Mapper: seq Priority: 60
[tarqeq.ofa.iol.unh.edu:04778]  Mapper: resilient Priority: 40
[tarqeq.ofa.iol.unh.edu:04778]  Mapper: mindist Priority: 20
[tarqeq.ofa.iol.unh.edu:04778]  Mapper: round_robin Priority: 10
[tarqeq.ofa.iol.unh.edu:04778]  Mapper: rank_file Priority: 0
[rhea.ofa.iol.unh.edu:05289] mca: base: components_register: registering 
framework rmaps components
[rhea.ofa.iol.unh.edu:05289] mca: base: components_register: found loaded 
component ppr
[rhea.ofa.iol.unh.edu:05289] mca: base: components_register: component ppr 
register function successful
[rhea.ofa.iol.unh.edu:05289] mca: base: components_register: found loaded 
component resilient
[rhea.ofa.iol.unh.edu:05289] mca: base: components_register: component 
resilient register function successful
[rhea.ofa.iol.unh.edu:05289] mca: base: components_register: found loaded 
component rank_file
[rhea.ofa.iol.unh.edu:05289] mca: base: components_register: component 
rank_file register function successful
[rhea.ofa.iol.unh.edu:05289] mca: base: components_register: found loaded 
component round_robin
[rhea.ofa.iol.unh.edu:05289] mca: base: components_register: component 
round_robin register function successful
[rhea.ofa.iol.unh.edu:05289] mca: base: components_register: found loaded 
component seq
[rhea.ofa.iol.unh.edu:05289] mca: base: components_register: component seq 
register function successful
[rhea.ofa.iol.unh.edu:05289] mca: base: components_register: found loaded 
component mindist
[rhea.ofa.iol.unh.edu:05289] mca: base: components_register: component mindist 
register function successful
[rhea.ofa.iol.unh.edu:05289] [[63394,0],4] rmaps:base set policy with NULL 
device NONNULL
[rhea.ofa.iol.unh.edu:05289] mca: base: components_open: opening rmaps 
components
[rhea.ofa.iol.unh.edu:05289] mca: base: components_open: found loaded component 
ppr
[rhea.ofa.iol.unh.edu:05289] mca: base: components_open: component ppr open 
function successful
[rhea.ofa.iol.unh.edu:05289] mca: base: components_open: found loaded component 
resilient
[rhea.ofa.iol.unh.edu:05289] mca: base: components_open: component resilient 
open function successful
[rhea.ofa.iol.unh.edu:05289] mca: base: components_open: found loaded component 
rank_file
[rhea.ofa.iol.unh.edu:05289] mca: base: components_open: component rank_file 
open function successful
[rhea.ofa.iol.unh.edu:05289] mca: base: components_open: found loaded component 
round_robin
[rhea.ofa.iol.unh.edu:05289] mca: base: components_open: component round_robin 
open function successful
[rhea.ofa.iol.unh.edu:05289] mca: base: components_open: found loaded component 
seq
[rhea.ofa.iol.unh.edu:05289] mca: base: components_open: component seq open 
function successful
[rhea.ofa.iol.unh.edu:05289] mca: base: components_open: found loaded component 
mindist
[rhea.ofa.iol.unh.edu:05289] mca: base: components_open: component mindist open 
function successful
[rhea.ofa.iol.unh.edu:05289] mca:rmaps:select: checking available component ppr
[rhea.ofa.iol.unh.edu:05289] mca:rmaps:select: Querying component [ppr]
[rhea.ofa.iol.unh.edu:05289] mca:rmaps:select: checking available component 
resilient
[rhea.ofa.iol.unh.edu:05289] mca:rmaps:select: Querying component [resilient]
[rhea.ofa.iol.unh.edu:05289] mca:rmaps:select: checking available component 
rank_file
[rhea.ofa.iol.unh.edu:05289] mca:rmaps:select: Querying component [rank_file]
[rhea.ofa.iol.unh.edu:05289] mca:rmaps:select: checking available component 
round_robin
[rhea.ofa.iol.unh.edu:05289] mca:rmaps:select: Querying component [round_robin]
[rhea.ofa.iol.unh.edu:05289] mca:rmaps:select: checking available component seq
[rhea.ofa.iol.unh.edu:05289] mca:rmaps:select: Querying component [seq]
[rhea.ofa.iol.unh.edu:05289] mca:rmaps:select: checking available component 
mindist
[rhea.ofa.iol.unh.edu:05289] mca:rmaps:select: Querying component [mindist]
[rhea.ofa.iol.unh.edu:05289] [[63394,0],4]: Final mapper priorities
[rhea.ofa.iol.unh.edu:05289]    Mapper: ppr Priority: 90
[rhea.ofa.iol.unh.edu:05289]    Mapper: seq Priority: 60
[rhea.ofa.iol.unh.edu:05289]    Mapper: resilient Priority: 40
[rhea.ofa.iol.unh.edu:05289]    Mapper: mindist Priority: 20
[rhea.ofa.iol.unh.edu:05289]    Mapper: round_robin Priority: 10
[rhea.ofa.iol.unh.edu:05289]    Mapper: rank_file Priority: 0
[jarnsaxa.ofa.iol.unh.edu:05118] mca: base: components_register: registering 
framework rmaps components
[jarnsaxa.ofa.iol.unh.edu:05118] mca: base: components_register: found loaded 
component seq
[jarnsaxa.ofa.iol.unh.edu:05118] mca: base: components_register: component seq 
register function successful
[jarnsaxa.ofa.iol.unh.edu:05118] mca: base: components_register: found loaded 
component ppr
[jarnsaxa.ofa.iol.unh.edu:05118] mca: base: components_register: component ppr 
register function successful
[jarnsaxa.ofa.iol.unh.edu:05118] mca: base: components_register: found loaded 
component resilient
[jarnsaxa.ofa.iol.unh.edu:05118] mca: base: components_register: component 
resilient register function successful
[jarnsaxa.ofa.iol.unh.edu:05118] mca: base: components_register: found loaded 
component round_robin
[jarnsaxa.ofa.iol.unh.edu:05118] mca: base: components_register: component 
round_robin register function successful
[jarnsaxa.ofa.iol.unh.edu:05118] mca: base: components_register: found loaded 
component mindist
[jarnsaxa.ofa.iol.unh.edu:05118] mca: base: components_register: component 
mindist register function successful
[jarnsaxa.ofa.iol.unh.edu:05118] mca: base: components_register: found loaded 
component rank_file
[jarnsaxa.ofa.iol.unh.edu:05118] mca: base: components_register: component 
rank_file register function successful
[jarnsaxa.ofa.iol.unh.edu:05118] [[63394,0],3] rmaps:base set policy with NULL 
device NONNULL
[jarnsaxa.ofa.iol.unh.edu:05118] mca: base: components_open: opening rmaps 
components
[jarnsaxa.ofa.iol.unh.edu:05118] mca: base: components_open: found loaded 
component seq
[jarnsaxa.ofa.iol.unh.edu:05118] mca: base: components_open: component seq open 
function successful
[jarnsaxa.ofa.iol.unh.edu:05118] mca: base: components_open: found loaded 
component ppr
[jarnsaxa.ofa.iol.unh.edu:05118] mca: base: components_open: component ppr open 
function successful
[jarnsaxa.ofa.iol.unh.edu:05118] mca: base: components_open: found loaded 
component resilient
[jarnsaxa.ofa.iol.unh.edu:05118] mca: base: components_open: component 
resilient open function successful
[jarnsaxa.ofa.iol.unh.edu:05118] mca: base: components_open: found loaded 
component round_robin
[jarnsaxa.ofa.iol.unh.edu:05118] mca: base: components_open: component 
round_robin open function successful
[jarnsaxa.ofa.iol.unh.edu:05118] mca: base: components_open: found loaded 
component mindist
[jarnsaxa.ofa.iol.unh.edu:05118] mca: base: components_open: component mindist 
open function successful
[jarnsaxa.ofa.iol.unh.edu:05118] mca: base: components_open: found loaded 
component rank_file
[jarnsaxa.ofa.iol.unh.edu:05118] mca: base: components_open: component 
rank_file open function successful
[jarnsaxa.ofa.iol.unh.edu:05118] mca:rmaps:select: checking available component 
seq
[jarnsaxa.ofa.iol.unh.edu:05118] mca:rmaps:select: Querying component [seq]
[jarnsaxa.ofa.iol.unh.edu:05118] mca:rmaps:select: checking available component 
ppr
[jarnsaxa.ofa.iol.unh.edu:05118] mca:rmaps:select: Querying component [ppr]
[jarnsaxa.ofa.iol.unh.edu:05118] mca:rmaps:select: checking available component 
resilient
[jarnsaxa.ofa.iol.unh.edu:05118] mca:rmaps:select: Querying component 
[resilient]
[jarnsaxa.ofa.iol.unh.edu:05118] mca:rmaps:select: checking available component 
round_robin
[jarnsaxa.ofa.iol.unh.edu:05118] mca:rmaps:select: Querying component 
[round_robin]
[jarnsaxa.ofa.iol.unh.edu:05118] mca:rmaps:select: checking available component 
mindist
[jarnsaxa.ofa.iol.unh.edu:05118] mca:rmaps:select: Querying component [mindist]
[jarnsaxa.ofa.iol.unh.edu:05118] mca:rmaps:select: checking available component 
rank_file
[jarnsaxa.ofa.iol.unh.edu:05118] mca:rmaps:select: Querying component 
[rank_file]
[jarnsaxa.ofa.iol.unh.edu:05118] [[63394,0],3]: Final mapper priorities
[jarnsaxa.ofa.iol.unh.edu:05118]        Mapper: ppr Priority: 90
[jarnsaxa.ofa.iol.unh.edu:05118]        Mapper: seq Priority: 60
[jarnsaxa.ofa.iol.unh.edu:05118]        Mapper: resilient Priority: 40
[jarnsaxa.ofa.iol.unh.edu:05118]        Mapper: mindist Priority: 20
[jarnsaxa.ofa.iol.unh.edu:05118]        Mapper: round_robin Priority: 10
[jarnsaxa.ofa.iol.unh.edu:05118]        Mapper: rank_file Priority: 0
[farbauti.ofa.iol.unh.edu:04673] mca: base: components_register: registering 
framework rmaps components
[farbauti.ofa.iol.unh.edu:04673] mca: base: components_register: found loaded 
component ppr
[farbauti.ofa.iol.unh.edu:04673] mca: base: components_register: component ppr 
register function successful
[farbauti.ofa.iol.unh.edu:04673] mca: base: components_register: found loaded 
component mindist
[farbauti.ofa.iol.unh.edu:04673] mca: base: components_register: component 
mindist register function successful
[farbauti.ofa.iol.unh.edu:04673] mca: base: components_register: found loaded 
component round_robin
[farbauti.ofa.iol.unh.edu:04673] mca: base: components_register: component 
round_robin register function successful
[farbauti.ofa.iol.unh.edu:04673] mca: base: components_register: found loaded 
component resilient
[farbauti.ofa.iol.unh.edu:04673] mca: base: components_register: component 
resilient register function successful
[farbauti.ofa.iol.unh.edu:04673] mca: base: components_register: found loaded 
component rank_file
[farbauti.ofa.iol.unh.edu:04673] mca: base: components_register: component 
rank_file register function successful
[farbauti.ofa.iol.unh.edu:04673] mca: base: components_register: found loaded 
component seq
[farbauti.ofa.iol.unh.edu:04673] mca: base: components_register: component seq 
register function successful
[farbauti.ofa.iol.unh.edu:04673] [[63394,0],1] rmaps:base set policy with NULL 
device NONNULL
[farbauti.ofa.iol.unh.edu:04673] mca: base: components_open: opening rmaps 
components
[farbauti.ofa.iol.unh.edu:04673] mca: base: components_open: found loaded 
component ppr
[farbauti.ofa.iol.unh.edu:04673] mca: base: components_open: component ppr open 
function successful
[farbauti.ofa.iol.unh.edu:04673] mca: base: components_open: found loaded 
component mindist
[farbauti.ofa.iol.unh.edu:04673] mca: base: components_open: component mindist 
open function successful
[farbauti.ofa.iol.unh.edu:04673] mca: base: components_open: found loaded 
component round_robin
[farbauti.ofa.iol.unh.edu:04673] mca: base: components_open: component 
round_robin open function successful
[farbauti.ofa.iol.unh.edu:04673] mca: base: components_open: found loaded 
component resilient
[farbauti.ofa.iol.unh.edu:04673] mca: base: components_open: component 
resilient open function successful
[farbauti.ofa.iol.unh.edu:04673] mca: base: components_open: found loaded 
component rank_file
[farbauti.ofa.iol.unh.edu:04673] mca: base: components_open: component 
rank_file open function successful
[farbauti.ofa.iol.unh.edu:04673] mca: base: components_open: found loaded 
component seq
[farbauti.ofa.iol.unh.edu:04673] mca: base: components_open: component seq open 
function successful
[farbauti.ofa.iol.unh.edu:04673] mca:rmaps:select: checking available component 
ppr
[farbauti.ofa.iol.unh.edu:04673] mca:rmaps:select: Querying component [ppr]
[farbauti.ofa.iol.unh.edu:04673] mca:rmaps:select: checking available component 
mindist
[farbauti.ofa.iol.unh.edu:04673] mca:rmaps:select: Querying component [mindist]
[farbauti.ofa.iol.unh.edu:04673] mca:rmaps:select: checking available component 
round_robin
[farbauti.ofa.iol.unh.edu:04673] mca:rmaps:select: Querying component 
[round_robin]
[farbauti.ofa.iol.unh.edu:04673] mca:rmaps:select: checking available component 
resilient
[farbauti.ofa.iol.unh.edu:04673] mca:rmaps:select: Querying component 
[resilient]
[farbauti.ofa.iol.unh.edu:04673] mca:rmaps:select: checking available component 
rank_file
[farbauti.ofa.iol.unh.edu:04673] mca:rmaps:select: Querying component 
[rank_file]
[farbauti.ofa.iol.unh.edu:04673] mca:rmaps:select: checking available component 
seq
[farbauti.ofa.iol.unh.edu:04673] mca:rmaps:select: Querying component [seq]
[farbauti.ofa.iol.unh.edu:04673] [[63394,0],1]: Final mapper priorities
[farbauti.ofa.iol.unh.edu:04673]        Mapper: ppr Priority: 90
[farbauti.ofa.iol.unh.edu:04673]        Mapper: seq Priority: 60
[farbauti.ofa.iol.unh.edu:04673]        Mapper: resilient Priority: 40
[farbauti.ofa.iol.unh.edu:04673]        Mapper: mindist Priority: 20
[farbauti.ofa.iol.unh.edu:04673]        Mapper: round_robin Priority: 10
[farbauti.ofa.iol.unh.edu:04673]        Mapper: rank_file Priority: 0
[tarvos.ofa.iol.unh.edu:04698] mca: base: components_register: registering 
framework rmaps components
[tarvos.ofa.iol.unh.edu:04698] mca: base: components_register: found loaded 
component mindist
[tarvos.ofa.iol.unh.edu:04698] mca: base: components_register: component 
mindist register function successful
[tarvos.ofa.iol.unh.edu:04698] mca: base: components_register: found loaded 
component seq
[tarvos.ofa.iol.unh.edu:04698] mca: base: components_register: component seq 
register function successful
[tarvos.ofa.iol.unh.edu:04698] mca: base: components_register: found loaded 
component rank_file
[tarvos.ofa.iol.unh.edu:04698] mca: base: components_register: component 
rank_file register function successful
[tarvos.ofa.iol.unh.edu:04698] mca: base: components_register: found loaded 
component round_robin
[tarvos.ofa.iol.unh.edu:04698] mca: base: components_register: component 
round_robin register function successful
[tarvos.ofa.iol.unh.edu:04698] mca: base: components_register: found loaded 
component resilient
[tarvos.ofa.iol.unh.edu:04698] mca: base: components_register: component 
resilient register function successful
[tarvos.ofa.iol.unh.edu:04698] mca: base: components_register: found loaded 
component ppr
[tarvos.ofa.iol.unh.edu:04698] mca: base: components_register: component ppr 
register function successful
[tarvos.ofa.iol.unh.edu:04698] [[63394,0],6] rmaps:base set policy with NULL 
device NONNULL
[tarvos.ofa.iol.unh.edu:04698] mca: base: components_open: opening rmaps 
components
[tarvos.ofa.iol.unh.edu:04698] mca: base: components_open: found loaded 
component mindist
[tarvos.ofa.iol.unh.edu:04698] mca: base: components_open: component mindist 
open function successful
[tarvos.ofa.iol.unh.edu:04698] mca: base: components_open: found loaded 
component seq
[tarvos.ofa.iol.unh.edu:04698] mca: base: components_open: component seq open 
function successful
[tarvos.ofa.iol.unh.edu:04698] mca: base: components_open: found loaded 
component rank_file
[tarvos.ofa.iol.unh.edu:04698] mca: base: components_open: component rank_file 
open function successful
[tarvos.ofa.iol.unh.edu:04698] mca: base: components_open: found loaded 
component round_robin
[tarvos.ofa.iol.unh.edu:04698] mca: base: components_open: component 
round_robin open function successful
[tarvos.ofa.iol.unh.edu:04698] mca: base: components_open: found loaded 
component resilient
[tarvos.ofa.iol.unh.edu:04698] mca: base: components_open: component resilient 
open function successful
[tarvos.ofa.iol.unh.edu:04698] mca: base: components_open: found loaded 
component ppr
[tarvos.ofa.iol.unh.edu:04698] mca: base: components_open: component ppr open 
function successful
[tarvos.ofa.iol.unh.edu:04698] mca:rmaps:select: checking available component 
mindist
[tarvos.ofa.iol.unh.edu:04698] mca:rmaps:select: Querying component [mindist]
[tarvos.ofa.iol.unh.edu:04698] mca:rmaps:select: checking available component 
seq
[tarvos.ofa.iol.unh.edu:04698] mca:rmaps:select: Querying component [seq]
[tarvos.ofa.iol.unh.edu:04698] mca:rmaps:select: checking available component 
rank_file
[tarvos.ofa.iol.unh.edu:04698] mca:rmaps:select: Querying component [rank_file]
[tarvos.ofa.iol.unh.edu:04698] mca:rmaps:select: checking available component 
round_robin
[tarvos.ofa.iol.unh.edu:04698] mca:rmaps:select: Querying component 
[round_robin]
[tarvos.ofa.iol.unh.edu:04698] mca:rmaps:select: checking available component 
resilient
[tarvos.ofa.iol.unh.edu:04698] mca:rmaps:select: Querying component [resilient]
[tarvos.ofa.iol.unh.edu:04698] mca:rmaps:select: checking available component 
ppr
[tarvos.ofa.iol.unh.edu:04698] mca:rmaps:select: Querying component [ppr]
[tarvos.ofa.iol.unh.edu:04698] [[63394,0],6]: Final mapper priorities
[tarvos.ofa.iol.unh.edu:04698]  Mapper: ppr Priority: 90
[tarvos.ofa.iol.unh.edu:04698]  Mapper: seq Priority: 60
[tarvos.ofa.iol.unh.edu:04698]  Mapper: resilient Priority: 40
[tarvos.ofa.iol.unh.edu:04698]  Mapper: mindist Priority: 20
[tarvos.ofa.iol.unh.edu:04698]  Mapper: round_robin Priority: 10
[tarvos.ofa.iol.unh.edu:04698]  Mapper: rank_file Priority: 0
[io.ofa.iol.unh.edu:05186] mca: base: components_register: registering 
framework rmaps components
[io.ofa.iol.unh.edu:05186] mca: base: components_register: found loaded 
component round_robin
[io.ofa.iol.unh.edu:05186] mca: base: components_register: component 
round_robin register function successful
[io.ofa.iol.unh.edu:05186] mca: base: components_register: found loaded 
component ppr
[io.ofa.iol.unh.edu:05186] mca: base: components_register: component ppr 
register function successful
[io.ofa.iol.unh.edu:05186] mca: base: components_register: found loaded 
component seq
[io.ofa.iol.unh.edu:05186] mca: base: components_register: component seq 
register function successful
[io.ofa.iol.unh.edu:05186] mca: base: components_register: found loaded 
component resilient
[io.ofa.iol.unh.edu:05186] mca: base: components_register: component resilient 
register function successful
[io.ofa.iol.unh.edu:05186] mca: base: components_register: found loaded 
component rank_file
[io.ofa.iol.unh.edu:05186] mca: base: components_register: component rank_file 
register function successful
[io.ofa.iol.unh.edu:05186] mca: base: components_register: found loaded 
component mindist
[io.ofa.iol.unh.edu:05186] mca: base: components_register: component mindist 
register function successful
[io.ofa.iol.unh.edu:05186] [[63394,0],2] rmaps:base set policy with NULL device 
NONNULL
[io.ofa.iol.unh.edu:05186] mca: base: components_open: opening rmaps components
[io.ofa.iol.unh.edu:05186] mca: base: components_open: found loaded component 
round_robin
[io.ofa.iol.unh.edu:05186] mca: base: components_open: component round_robin 
open function successful
[io.ofa.iol.unh.edu:05186] mca: base: components_open: found loaded component 
ppr
[io.ofa.iol.unh.edu:05186] mca: base: components_open: component ppr open 
function successful
[io.ofa.iol.unh.edu:05186] mca: base: components_open: found loaded component 
seq
[io.ofa.iol.unh.edu:05186] mca: base: components_open: component seq open 
function successful
[io.ofa.iol.unh.edu:05186] mca: base: components_open: found loaded component 
resilient
[io.ofa.iol.unh.edu:05186] mca: base: components_open: component resilient open 
function successful
[io.ofa.iol.unh.edu:05186] mca: base: components_open: found loaded component 
rank_file
[io.ofa.iol.unh.edu:05186] mca: base: components_open: component rank_file open 
function successful
[io.ofa.iol.unh.edu:05186] mca: base: components_open: found loaded component 
mindist
[io.ofa.iol.unh.edu:05186] mca: base: components_open: component mindist open 
function successful
[io.ofa.iol.unh.edu:05186] mca:rmaps:select: checking available component 
round_robin
[io.ofa.iol.unh.edu:05186] mca:rmaps:select: Querying component [round_robin]
[io.ofa.iol.unh.edu:05186] mca:rmaps:select: checking available component ppr
[io.ofa.iol.unh.edu:05186] mca:rmaps:select: Querying component [ppr]
[io.ofa.iol.unh.edu:05186] mca:rmaps:select: checking available component seq
[io.ofa.iol.unh.edu:05186] mca:rmaps:select: Querying component [seq]
[io.ofa.iol.unh.edu:05186] mca:rmaps:select: checking available component 
resilient
[io.ofa.iol.unh.edu:05186] mca:rmaps:select: Querying component [resilient]
[io.ofa.iol.unh.edu:05186] mca:rmaps:select: checking available component 
rank_file
[io.ofa.iol.unh.edu:05186] mca:rmaps:select: Querying component [rank_file]
[io.ofa.iol.unh.edu:05186] mca:rmaps:select: checking available component 
mindist
[io.ofa.iol.unh.edu:05186] mca:rmaps:select: Querying component [mindist]
[io.ofa.iol.unh.edu:05186] [[63394,0],2]: Final mapper priorities
[io.ofa.iol.unh.edu:05186]      Mapper: ppr Priority: 90
[io.ofa.iol.unh.edu:05186]      Mapper: seq Priority: 60
[io.ofa.iol.unh.edu:05186]      Mapper: resilient Priority: 40
[io.ofa.iol.unh.edu:05186]      Mapper: mindist Priority: 20
[io.ofa.iol.unh.edu:05186]      Mapper: round_robin Priority: 10
[io.ofa.iol.unh.edu:05186]      Mapper: rank_file Priority: 0
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps: mapping job [63394,1]
[hyperion.ofa.iol.unh.edu:05190] AVAILABLE NODES FOR MAPPING:
[hyperion.ofa.iol.unh.edu:05190]     node: hyperion daemon: 0
[hyperion.ofa.iol.unh.edu:05190]     node: farbauti-ce daemon: 1
[hyperion.ofa.iol.unh.edu:05190]     node: io-ce daemon: 2
[hyperion.ofa.iol.unh.edu:05190]     node: jarnsaxa-ce daemon: 3
[hyperion.ofa.iol.unh.edu:05190]     node: rhea-ce daemon: 4
[hyperion.ofa.iol.unh.edu:05190]     node: tarqeq-ce daemon: 5
[hyperion.ofa.iol.unh.edu:05190]     node: tarvos-ce daemon: 6
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps: setting mapping policies for job 
[63394,1] nprocs 7
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps[181] mapping not set by user and no 
NUMA - using bysocket
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps[321] binding not given and no NUMA - 
using bysocket
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps:ppr: job [63394,1] not using ppr 
mapper PPR NULL policy PPR NOTSET
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps:seq: job [63394,1] not using seq 
mapper
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps:resilient: cannot perform initial 
map of job [63394,1] - no fault groups
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps:mindist: job [63394,1] not using 
mindist mapper
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps:rr: mapping job [63394,1]
[hyperion.ofa.iol.unh.edu:05190] AVAILABLE NODES FOR MAPPING:
[hyperion.ofa.iol.unh.edu:05190]     node: hyperion daemon: 0
[hyperion.ofa.iol.unh.edu:05190]     node: farbauti-ce daemon: 1
[hyperion.ofa.iol.unh.edu:05190]     node: io-ce daemon: 2
[hyperion.ofa.iol.unh.edu:05190]     node: jarnsaxa-ce daemon: 3
[hyperion.ofa.iol.unh.edu:05190]     node: rhea-ce daemon: 4
[hyperion.ofa.iol.unh.edu:05190]     node: tarqeq-ce daemon: 5
[hyperion.ofa.iol.unh.edu:05190]     node: tarvos-ce daemon: 6
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps:rr: mapping no-span by Package for 
job [63394,1] slots 7 num_procs 7
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps:rr: found 1 Package objects on node 
hyperion
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps:rr: calculated nprocs 1
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps:rr: assigning nprocs 1
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps:rr: found 2 Package objects on node 
farbauti-ce
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps:rr: calculated nprocs 1
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps:rr: assigning nprocs 1
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps:rr: found 2 Package objects on node 
io-ce
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps:rr: calculated nprocs 1
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps:rr: assigning nprocs 1
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps:rr: found 2 Package objects on node 
jarnsaxa-ce
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps:rr: calculated nprocs 1
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps:rr: assigning nprocs 1
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps:rr: found 2 Package objects on node 
rhea-ce
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps:rr: calculated nprocs 1
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps:rr: assigning nprocs 1
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps:rr: found 1 Package objects on node 
tarqeq-ce
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps:rr: calculated nprocs 1
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps:rr: assigning nprocs 1
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps:rr: found 2 Package objects on node 
tarvos-ce
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps:rr: calculated nprocs 1
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps:rr: assigning nprocs 1
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps: assigning locations for job 
[63394,1]
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps:ppr: job [63394,1] not using ppr 
assign: round_robin
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps:resilient: job [63394,1] not using 
resilient assign: round_robin
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps:mindist: job [63394,1] not using 
mindist mapper
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps:rr: assign locations for job 
[63394,1]
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps:rr: assigning locations by Package 
for job [63394,1]
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps:rr: found 1 Package objects on node 
hyperion
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps:rr: found 2 Package objects on node 
farbauti-ce
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps:rr: found 2 Package objects on node 
io-ce
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps:rr: found 2 Package objects on node 
jarnsaxa-ce
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps:rr: found 2 Package objects on node 
rhea-ce
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps:rr: found 1 Package objects on node 
tarqeq-ce
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps:rr: found 2 Package objects on node 
tarvos-ce
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps:base: computing vpids by slot for 
job [63394,1]
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps:base: assigning rank 0 to node 
hyperion
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps:base: assigning rank 1 to node 
farbauti-ce
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps:base: assigning rank 2 to node io-ce
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps:base: assigning rank 3 to node 
jarnsaxa-ce
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps:base: assigning rank 4 to node 
rhea-ce
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps:base: assigning rank 5 to node 
tarqeq-ce
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps:base: assigning rank 6 to node 
tarvos-ce
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps: compute bindings for job [63394,1] 
with policy SOCKET:IF-SUPPORTED[1004]
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps: bindings for job [63394,1] - bind 
in place
[hyperion.ofa.iol.unh.edu:05190] mca:rmaps: bind in place for job [63394,1] 
with bindings SOCKET:IF-SUPPORTED
[hyperion.ofa.iol.unh.edu:05190] [[63394,0],0] reset_usage: node hyperion has 1 
procs on it
[hyperion.ofa.iol.unh.edu:05190] [[63394,0],0] reset_usage: ignoring proc 
[[63394,1],0]
[hyperion.ofa.iol.unh.edu:05190] BINDING PROC [[63394,1],0] TO Package NUMBER 0
[hyperion.ofa.iol.unh.edu:05190] [[63394,0],0] BOUND PROC [[63394,1],0] TO 
0-7[Package:0] on node hyperion
[rhea.ofa.iol.unh.edu:05289] mca:rmaps: assigning locations for job [63394,1]
[rhea.ofa.iol.unh.edu:05289] mca:rmaps:ppr: job [63394,1] not using ppr assign: 
round_robin
[rhea.ofa.iol.unh.edu:05289] mca:rmaps:resilient: job [63394,1] not using 
resilient assign: round_robin
[rhea.ofa.iol.unh.edu:05289] mca:rmaps:mindist: job [63394,1] not using mindist 
mapper
[rhea.ofa.iol.unh.edu:05289] mca:rmaps:rr: assign locations for job [63394,1]
[rhea.ofa.iol.unh.edu:05289] mca:rmaps:rr: assigning locations by Package for 
job [63394,1]
[rhea.ofa.iol.unh.edu:05289] mca:rmaps:rr: found 1 Package objects on node 
hyperion
[rhea.ofa.iol.unh.edu:05289] mca:rmaps:rr: found 2 Package objects on node 
farbauti-ce
[rhea.ofa.iol.unh.edu:05289] mca:rmaps:rr: found 2 Package objects on node io-ce
[rhea.ofa.iol.unh.edu:05289] mca:rmaps:rr: found 2 Package objects on node 
jarnsaxa-ce
[rhea.ofa.iol.unh.edu:05289] mca:rmaps:rr: found 2 Package objects on node 
rhea-ce
[rhea.ofa.iol.unh.edu:05289] mca:rmaps:rr: found 1 Package objects on node 
tarqeq-ce
[rhea.ofa.iol.unh.edu:05289] mca:rmaps:rr: found 2 Package objects on node 
tarvos-ce
[rhea.ofa.iol.unh.edu:05289] mca:rmaps:base: computing vpids by slot for job 
[63394,1]
[rhea.ofa.iol.unh.edu:05289] mca:rmaps:base: assigning rank 0 to node hyperion
[rhea.ofa.iol.unh.edu:05289] mca:rmaps:base: assigning rank 1 to node 
farbauti-ce
[rhea.ofa.iol.unh.edu:05289] mca:rmaps:base: assigning rank 2 to node io-ce
[rhea.ofa.iol.unh.edu:05289] mca:rmaps:base: assigning rank 3 to node 
jarnsaxa-ce
[rhea.ofa.iol.unh.edu:05289] mca:rmaps:base: assigning rank 4 to node rhea-ce
[rhea.ofa.iol.unh.edu:05289] mca:rmaps:base: assigning rank 5 to node tarqeq-ce
[rhea.ofa.iol.unh.edu:05289] mca:rmaps:base: assigning rank 6 to node tarvos-ce
[rhea.ofa.iol.unh.edu:05289] mca:rmaps: compute bindings for job [63394,1] with 
policy SOCKET:IF-SUPPORTED[1004]
[rhea.ofa.iol.unh.edu:05289] mca:rmaps: bindings for job [63394,1] - bind in 
place
[rhea.ofa.iol.unh.edu:05289] mca:rmaps: bind in place for job [63394,1] with 
bindings SOCKET:IF-SUPPORTED
[rhea.ofa.iol.unh.edu:05289] [[63394,0],4] reset_usage: node rhea-ce has 1 
procs on it
[rhea.ofa.iol.unh.edu:05289] [[63394,0],4] reset_usage: ignoring proc 
[[63394,1],4]
[rhea.ofa.iol.unh.edu:05289] BINDING PROC [[63394,1],4] TO Package NUMBER 0
[rhea.ofa.iol.unh.edu:05289] [[63394,0],4] BOUND PROC [[63394,1],4] TO 
0-15,32-47[Package:0] on node rhea-ce
[io.ofa.iol.unh.edu:05186] mca:rmaps: assigning locations for job [63394,1]
[io.ofa.iol.unh.edu:05186] mca:rmaps:ppr: job [63394,1] not using ppr assign: 
round_robin
[io.ofa.iol.unh.edu:05186] mca:rmaps:resilient: job [63394,1] not using 
resilient assign: round_robin
[io.ofa.iol.unh.edu:05186] mca:rmaps:mindist: job [63394,1] not using mindist 
mapper
[io.ofa.iol.unh.edu:05186] mca:rmaps:rr: assign locations for job [63394,1]
[io.ofa.iol.unh.edu:05186] mca:rmaps:rr: assigning locations by Package for job 
[63394,1]
[io.ofa.iol.unh.edu:05186] mca:rmaps:rr: found 1 Package objects on node 
hyperion
[io.ofa.iol.unh.edu:05186] mca:rmaps:rr: found 2 Package objects on node 
farbauti-ce
[io.ofa.iol.unh.edu:05186] mca:rmaps:rr: found 2 Package objects on node io-ce
[io.ofa.iol.unh.edu:05186] mca:rmaps:rr: found 2 Package objects on node 
jarnsaxa-ce
[io.ofa.iol.unh.edu:05186] mca:rmaps:rr: found 2 Package objects on node rhea-ce
[io.ofa.iol.unh.edu:05186] mca:rmaps:rr: found 1 Package objects on node 
tarqeq-ce
[io.ofa.iol.unh.edu:05186] mca:rmaps:rr: found 2 Package objects on node 
tarvos-ce
[io.ofa.iol.unh.edu:05186] mca:rmaps:base: computing vpids by slot for job 
[63394,1]
[io.ofa.iol.unh.edu:05186] mca:rmaps:base: assigning rank 0 to node hyperion
[io.ofa.iol.unh.edu:05186] mca:rmaps:base: assigning rank 1 to node farbauti-ce
[io.ofa.iol.unh.edu:05186] mca:rmaps:base: assigning rank 2 to node io-ce
[io.ofa.iol.unh.edu:05186] mca:rmaps:base: assigning rank 3 to node jarnsaxa-ce
[io.ofa.iol.unh.edu:05186] mca:rmaps:base: assigning rank 4 to node rhea-ce
[io.ofa.iol.unh.edu:05186] mca:rmaps:base: assigning rank 5 to node tarqeq-ce
[io.ofa.iol.unh.edu:05186] mca:rmaps:base: assigning rank 6 to node tarvos-ce
[io.ofa.iol.unh.edu:05186] mca:rmaps: compute bindings for job [63394,1] with 
policy SOCKET:IF-SUPPORTED[1004]
[io.ofa.iol.unh.edu:05186] mca:rmaps: bindings for job [63394,1] - bind in place
[io.ofa.iol.unh.edu:05186] mca:rmaps: bind in place for job [63394,1] with 
bindings SOCKET:IF-SUPPORTED
[io.ofa.iol.unh.edu:05186] [[63394,0],2] reset_usage: node io-ce has 1 procs on 
it
[io.ofa.iol.unh.edu:05186] [[63394,0],2] reset_usage: ignoring proc 
[[63394,1],2]
[io.ofa.iol.unh.edu:05186] BINDING PROC [[63394,1],2] TO Package NUMBER 0
[io.ofa.iol.unh.edu:05186] [[63394,0],2] BOUND PROC [[63394,1],2] TO 
0-15,32-47[Package:0] on node io-ce
[jarnsaxa.ofa.iol.unh.edu:05118] mca:rmaps: assigning locations for job 
[63394,1]
[jarnsaxa.ofa.iol.unh.edu:05118] mca:rmaps:ppr: job [63394,1] not using ppr 
assign: round_robin
[jarnsaxa.ofa.iol.unh.edu:05118] mca:rmaps:resilient: job [63394,1] not using 
resilient assign: round_robin
[jarnsaxa.ofa.iol.unh.edu:05118] mca:rmaps:mindist: job [63394,1] not using 
mindist mapper
[jarnsaxa.ofa.iol.unh.edu:05118] mca:rmaps:rr: assign locations for job 
[63394,1]
[jarnsaxa.ofa.iol.unh.edu:05118] mca:rmaps:rr: assigning locations by Package 
for job [63394,1]
[jarnsaxa.ofa.iol.unh.edu:05118] mca:rmaps:rr: found 1 Package objects on node 
hyperion
[jarnsaxa.ofa.iol.unh.edu:05118] mca:rmaps:rr: found 2 Package objects on node 
farbauti-ce
[jarnsaxa.ofa.iol.unh.edu:05118] mca:rmaps:rr: found 2 Package objects on node 
io-ce
[jarnsaxa.ofa.iol.unh.edu:05118] mca:rmaps:rr: found 2 Package objects on node 
jarnsaxa-ce
[jarnsaxa.ofa.iol.unh.edu:05118] mca:rmaps:rr: found 2 Package objects on node 
rhea-ce
[jarnsaxa.ofa.iol.unh.edu:05118] mca:rmaps:rr: found 1 Package objects on node 
tarqeq-ce
[jarnsaxa.ofa.iol.unh.edu:05118] mca:rmaps:rr: found 2 Package objects on node 
tarvos-ce
[jarnsaxa.ofa.iol.unh.edu:05118] mca:rmaps:base: computing vpids by slot for 
job [63394,1]
[jarnsaxa.ofa.iol.unh.edu:05118] mca:rmaps:base: assigning rank 0 to node 
hyperion
[jarnsaxa.ofa.iol.unh.edu:05118] mca:rmaps:base: assigning rank 1 to node 
farbauti-ce
[jarnsaxa.ofa.iol.unh.edu:05118] mca:rmaps:base: assigning rank 2 to node io-ce
[jarnsaxa.ofa.iol.unh.edu:05118] mca:rmaps:base: assigning rank 3 to node 
jarnsaxa-ce
[jarnsaxa.ofa.iol.unh.edu:05118] mca:rmaps:base: assigning rank 4 to node 
rhea-ce
[jarnsaxa.ofa.iol.unh.edu:05118] mca:rmaps:base: assigning rank 5 to node 
tarqeq-ce
[jarnsaxa.ofa.iol.unh.edu:05118] mca:rmaps:base: assigning rank 6 to node 
tarvos-ce
[jarnsaxa.ofa.iol.unh.edu:05118] mca:rmaps: compute bindings for job [63394,1] 
with policy SOCKET:IF-SUPPORTED[1004]
[jarnsaxa.ofa.iol.unh.edu:05118] mca:rmaps: bindings for job [63394,1] - bind 
in place
[jarnsaxa.ofa.iol.unh.edu:05118] mca:rmaps: bind in place for job [63394,1] 
with bindings SOCKET:IF-SUPPORTED
[jarnsaxa.ofa.iol.unh.edu:05118] [[63394,0],3] reset_usage: node jarnsaxa-ce 
has 1 procs on it
[jarnsaxa.ofa.iol.unh.edu:05118] [[63394,0],3] reset_usage: ignoring proc 
[[63394,1],3]
[jarnsaxa.ofa.iol.unh.edu:05118] BINDING PROC [[63394,1],3] TO Package NUMBER 0
[jarnsaxa.ofa.iol.unh.edu:05118] [[63394,0],3] BOUND PROC [[63394,1],3] TO 
0-5,12-17[Package:0] on node jarnsaxa-ce
[farbauti.ofa.iol.unh.edu:04673] mca:rmaps: assigning locations for job 
[63394,1]
[farbauti.ofa.iol.unh.edu:04673] mca:rmaps:ppr: job [63394,1] not using ppr 
assign: round_robin
[farbauti.ofa.iol.unh.edu:04673] mca:rmaps:resilient: job [63394,1] not using 
resilient assign: round_robin
[farbauti.ofa.iol.unh.edu:04673] mca:rmaps:mindist: job [63394,1] not using 
mindist mapper
[farbauti.ofa.iol.unh.edu:04673] mca:rmaps:rr: assign locations for job 
[63394,1]
[farbauti.ofa.iol.unh.edu:04673] mca:rmaps:rr: assigning locations by Package 
for job [63394,1]
[farbauti.ofa.iol.unh.edu:04673] mca:rmaps:rr: found 1 Package objects on node 
hyperion
[farbauti.ofa.iol.unh.edu:04673] mca:rmaps:rr: found 2 Package objects on node 
farbauti-ce
[farbauti.ofa.iol.unh.edu:04673] mca:rmaps:rr: found 2 Package objects on node 
io-ce
[farbauti.ofa.iol.unh.edu:04673] mca:rmaps:rr: found 2 Package objects on node 
jarnsaxa-ce
[farbauti.ofa.iol.unh.edu:04673] mca:rmaps:rr: found 2 Package objects on node 
rhea-ce
[farbauti.ofa.iol.unh.edu:04673] mca:rmaps:rr: found 1 Package objects on node 
tarqeq-ce
[farbauti.ofa.iol.unh.edu:04673] mca:rmaps:rr: found 2 Package objects on node 
tarvos-ce
[farbauti.ofa.iol.unh.edu:04673] mca:rmaps:base: computing vpids by slot for 
job [63394,1]
[farbauti.ofa.iol.unh.edu:04673] mca:rmaps:base: assigning rank 0 to node 
hyperion
[farbauti.ofa.iol.unh.edu:04673] mca:rmaps:base: assigning rank 1 to node 
farbauti-ce
[farbauti.ofa.iol.unh.edu:04673] mca:rmaps:base: assigning rank 2 to node io-ce
[farbauti.ofa.iol.unh.edu:04673] mca:rmaps:base: assigning rank 3 to node 
jarnsaxa-ce
[farbauti.ofa.iol.unh.edu:04673] mca:rmaps:base: assigning rank 4 to node 
rhea-ce
[farbauti.ofa.iol.unh.edu:04673] mca:rmaps:base: assigning rank 5 to node 
tarqeq-ce
[farbauti.ofa.iol.unh.edu:04673] mca:rmaps:base: assigning rank 6 to node 
tarvos-ce
[farbauti.ofa.iol.unh.edu:04673] mca:rmaps: compute bindings for job [63394,1] 
with policy SOCKET:IF-SUPPORTED[1004]
[farbauti.ofa.iol.unh.edu:04673] mca:rmaps: bindings for job [63394,1] - bind 
in place
[farbauti.ofa.iol.unh.edu:04673] mca:rmaps: bind in place for job [63394,1] 
with bindings SOCKET:IF-SUPPORTED
[farbauti.ofa.iol.unh.edu:04673] [[63394,0],1] reset_usage: node farbauti-ce 
has 1 procs on it
[farbauti.ofa.iol.unh.edu:04673] [[63394,0],1] reset_usage: ignoring proc 
[[63394,1],1]
[farbauti.ofa.iol.unh.edu:04673] BINDING PROC [[63394,1],1] TO Package NUMBER 0
[farbauti.ofa.iol.unh.edu:04673] [[63394,0],1] BOUND PROC [[63394,1],1] TO 
0-3,8-11[Package:0] on node farbauti-ce
[tarvos.ofa.iol.unh.edu:04698] mca:rmaps: assigning locations for job [63394,1]
[tarvos.ofa.iol.unh.edu:04698] mca:rmaps:ppr: job [63394,1] not using ppr 
assign: round_robin
[tarvos.ofa.iol.unh.edu:04698] mca:rmaps:resilient: job [63394,1] not using 
resilient assign: round_robin
[tarvos.ofa.iol.unh.edu:04698] mca:rmaps:mindist: job [63394,1] not using 
mindist mapper
[tarvos.ofa.iol.unh.edu:04698] mca:rmaps:rr: assign locations for job [63394,1]
[tarvos.ofa.iol.unh.edu:04698] mca:rmaps:rr: assigning locations by Package for 
job [63394,1]
[tarvos.ofa.iol.unh.edu:04698] mca:rmaps:rr: found 1 Package objects on node 
hyperion
[tarvos.ofa.iol.unh.edu:04698] mca:rmaps:rr: found 2 Package objects on node 
farbauti-ce
[tarvos.ofa.iol.unh.edu:04698] mca:rmaps:rr: found 2 Package objects on node 
io-ce
[tarvos.ofa.iol.unh.edu:04698] mca:rmaps:rr: found 2 Package objects on node 
jarnsaxa-ce
[tarvos.ofa.iol.unh.edu:04698] mca:rmaps:rr: found 2 Package objects on node 
rhea-ce
[tarvos.ofa.iol.unh.edu:04698] mca:rmaps:rr: found 1 Package objects on node 
tarqeq-ce
[tarvos.ofa.iol.unh.edu:04698] mca:rmaps:rr: found 2 Package objects on node 
tarvos-ce
[tarvos.ofa.iol.unh.edu:04698] mca:rmaps:base: computing vpids by slot for job 
[63394,1]
[tarvos.ofa.iol.unh.edu:04698] mca:rmaps:base: assigning rank 0 to node hyperion
[tarvos.ofa.iol.unh.edu:04698] mca:rmaps:base: assigning rank 1 to node 
farbauti-ce
[tarvos.ofa.iol.unh.edu:04698] mca:rmaps:base: assigning rank 2 to node io-ce
[tarvos.ofa.iol.unh.edu:04698] mca:rmaps:base: assigning rank 3 to node 
jarnsaxa-ce
[tarvos.ofa.iol.unh.edu:04698] mca:rmaps:base: assigning rank 4 to node rhea-ce
[tarvos.ofa.iol.unh.edu:04698] mca:rmaps:base: assigning rank 5 to node 
tarqeq-ce
[tarvos.ofa.iol.unh.edu:04698] mca:rmaps:base: assigning rank 6 to node 
tarvos-ce
[tarvos.ofa.iol.unh.edu:04698] mca:rmaps: compute bindings for job [63394,1] 
with policy SOCKET:IF-SUPPORTED[1004]
[tarvos.ofa.iol.unh.edu:04698] mca:rmaps: bindings for job [63394,1] - bind in 
place
[tarvos.ofa.iol.unh.edu:04698] mca:rmaps: bind in place for job [63394,1] with 
bindings SOCKET:IF-SUPPORTED
[tarvos.ofa.iol.unh.edu:04698] [[63394,0],6] reset_usage: node tarvos-ce has 1 
procs on it
[tarvos.ofa.iol.unh.edu:04698] [[63394,0],6] reset_usage: ignoring proc 
[[63394,1],6]
[tarvos.ofa.iol.unh.edu:04698] BINDING PROC [[63394,1],6] TO Package NUMBER 0
[tarvos.ofa.iol.unh.edu:04698] [[63394,0],6] BOUND PROC [[63394,1],6] TO 
0-3,8-11[Package:0] on node tarvos-ce
[tarqeq.ofa.iol.unh.edu:04778] mca:rmaps: assigning locations for job [63394,1]
[tarqeq.ofa.iol.unh.edu:04778] mca:rmaps:ppr: job [63394,1] not using ppr 
assign: round_robin
[tarqeq.ofa.iol.unh.edu:04778] mca:rmaps:resilient: job [63394,1] not using 
resilient assign: round_robin
[tarqeq.ofa.iol.unh.edu:04778] mca:rmaps:mindist: job [63394,1] not using 
mindist mapper
[tarqeq.ofa.iol.unh.edu:04778] mca:rmaps:rr: assign locations for job [63394,1]
[tarqeq.ofa.iol.unh.edu:04778] mca:rmaps:rr: assigning locations by Package for 
job [63394,1]
[tarqeq.ofa.iol.unh.edu:04778] mca:rmaps:rr: found 1 Package objects on node 
hyperion
[tarqeq.ofa.iol.unh.edu:04778] mca:rmaps:rr: found 2 Package objects on node 
farbauti-ce
[tarqeq.ofa.iol.unh.edu:04778] mca:rmaps:rr: found 2 Package objects on node 
io-ce
[tarqeq.ofa.iol.unh.edu:04778] mca:rmaps:rr: found 2 Package objects on node 
jarnsaxa-ce
[tarqeq.ofa.iol.unh.edu:04778] mca:rmaps:rr: found 2 Package objects on node 
rhea-ce
[tarqeq.ofa.iol.unh.edu:04778] mca:rmaps:rr: found 1 Package objects on node 
tarqeq-ce
[tarqeq.ofa.iol.unh.edu:04778] mca:rmaps:rr: found 2 Package objects on node 
tarvos-ce
[tarqeq.ofa.iol.unh.edu:04778] mca:rmaps:base: computing vpids by slot for job 
[63394,1]
[tarqeq.ofa.iol.unh.edu:04778] mca:rmaps:base: assigning rank 0 to node hyperion
[tarqeq.ofa.iol.unh.edu:04778] mca:rmaps:base: assigning rank 1 to node 
farbauti-ce
[tarqeq.ofa.iol.unh.edu:04778] mca:rmaps:base: assigning rank 2 to node io-ce
[tarqeq.ofa.iol.unh.edu:04778] mca:rmaps:base: assigning rank 3 to node 
jarnsaxa-ce
[tarqeq.ofa.iol.unh.edu:04778] mca:rmaps:base: assigning rank 4 to node rhea-ce
[tarqeq.ofa.iol.unh.edu:04778] mca:rmaps:base: assigning rank 5 to node 
tarqeq-ce
[tarqeq.ofa.iol.unh.edu:04778] mca:rmaps:base: assigning rank 6 to node 
tarvos-ce
[tarqeq.ofa.iol.unh.edu:04778] mca:rmaps: compute bindings for job [63394,1] 
with policy SOCKET:IF-SUPPORTED[1004]
[tarqeq.ofa.iol.unh.edu:04778] mca:rmaps: bindings for job [63394,1] - bind in 
place
[tarqeq.ofa.iol.unh.edu:04778] mca:rmaps: bind in place for job [63394,1] with 
bindings SOCKET:IF-SUPPORTED
[tarqeq.ofa.iol.unh.edu:04778] [[63394,0],5] reset_usage: node tarqeq-ce has 1 
procs on it
[tarqeq.ofa.iol.unh.edu:04778] [[63394,0],5] reset_usage: ignoring proc 
[[63394,1],5]
[tarqeq.ofa.iol.unh.edu:04778] BINDING PROC [[63394,1],5] TO Package NUMBER 0
[tarqeq.ofa.iol.unh.edu:04778] [[63394,0],5] BOUND PROC [[63394,1],5] TO 
0-7[Package:0] on node tarqeq-ce
--------------------------------------------------------------------------
WARNING: rd_win specification is non optimal. For maximum performance it is
advisable to configure rd_win bigger than (rd_num - rd_low), but currently
rd_win = 32 and (rd_num - rd_low) = 56.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
WARNING: rd_win specification is non optimal. For maximum performance it is
advisable to configure rd_win bigger than (rd_num - rd_low), but currently
rd_win = 32 and (rd_num - rd_low) = 56.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
WARNING: rd_win specification is non optimal. For maximum performance it is
advisable to configure rd_win bigger than (rd_num - rd_low), but currently
rd_win = 32 and (rd_num - rd_low) = 56.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
WARNING: rd_win specification is non optimal. For maximum performance it is
advisable to configure rd_win bigger than (rd_num - rd_low), but currently
rd_win = 32 and (rd_num - rd_low) = 56.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
WARNING: rd_win specification is non optimal. For maximum performance it is
advisable to configure rd_win bigger than (rd_num - rd_low), but currently
rd_win = 32 and (rd_num - rd_low) = 56.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
WARNING: rd_win specification is non optimal. For maximum performance it is
advisable to configure rd_win bigger than (rd_num - rd_low), but currently
rd_win = 32 and (rd_num - rd_low) = 56.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
WARNING: rd_win specification is non optimal. For maximum performance it is
advisable to configure rd_win bigger than (rd_num - rd_low), but currently
rd_win = 32 and (rd_num - rd_low) = 56.
--------------------------------------------------------------------------
#------------------------------------------------------------
#    Intel (R) MPI Benchmarks 2018 Update 1, MPI-1 part    
#------------------------------------------------------------
# Date                  : Thu Nov  1 13:49:31 2018
# Machine               : x86_64
# System                : Linux
# Release               : 4.17.14
# Version               : #1 SMP Thu Aug 9 16:04:40 EDT 2018
# MPI Version           : 3.1
# MPI Thread Environment: 


# Calling sequence was: 

# IMB-MPI1

# Minimum message length in bytes:   0
# Maximum message length in bytes:   4194304
#
# MPI_Datatype                   :   MPI_BYTE 
# MPI_Datatype for reductions    :   MPI_FLOAT
# MPI_Op                         :   MPI_SUM  
#
#

# List of Benchmarks to run:

# PingPong
# PingPing
# Sendrecv
# Exchange
# Allreduce
# Reduce
# Reduce_scatter
# Allgather
# Allgatherv
# Gather
# Gatherv
# Scatter
# Scatterv
# Alltoall
# Alltoallv
# Bcast
# Barrier

#---------------------------------------------------
# Benchmarking PingPong 
# #processes = 2 
# ( 5 additional processes waiting in MPI_Barrier)
#---------------------------------------------------
       #bytes #repetitions      t[usec]   Mbytes/sec
            0         1000         2.88         0.00
            1         1000         2.96         0.34
            2         1000         2.94         0.68
            4         1000         2.94         1.36
            8         1000         2.96         2.70
           16         1000         2.97         5.38
           32         1000         2.98        10.73
           64         1000         3.09        20.73
          128         1000         3.62        35.39
          256         1000         3.97        64.52
          512         1000         4.20       121.83
         1024         1000         4.74       215.87
         2048         1000         5.76       355.52
         4096         1000         7.95       515.26
         8192         1000        11.14       735.22
        16384         1000        16.00      1024.17
        32768         1000        21.49      1524.65
        65536          640        32.19      2036.19
       131072          320        53.46      2451.69
       262144          160        96.44      2718.14
       524288           80       182.11      2879.04
      1048576           40       353.37      2967.37
      2097152           20       695.81      3013.97
      4194304           10      1382.61      3033.62

#---------------------------------------------------
# Benchmarking PingPing 
# #processes = 2 
# ( 5 additional processes waiting in MPI_Barrier)
#---------------------------------------------------
       #bytes #repetitions      t[usec]   Mbytes/sec
            0         1000         2.91         0.00
            1         1000         2.98         0.34
            2         1000         2.98         0.67
            4         1000         3.00         1.33
            8         1000         3.00         2.66
           16         1000         3.01         5.32
           32         1000         3.04        10.51
           64         1000         3.14        20.40
          128         1000         3.69        34.64
          256         1000         4.09        62.67
          512         1000         4.40       116.30
         1024         1000         4.94       207.39
         2048         1000         5.85       350.21
         4096         1000         7.99       512.52
         8192         1000        10.87       753.38
        16384         1000        26.31       622.83
        32768         1000        37.16       881.93
        65536          640        50.63      1294.30
       131072          320        72.41      1810.03
       262144          160       114.81      2283.36
       524288           80       201.10      2607.14
      1048576           40       371.17      2825.09
      2097152           20       712.27      2944.33
      4194304           10      1400.13      2995.66

#-----------------------------------------------------------------------------
# Benchmarking Sendrecv 
# #processes = 2 
# ( 5 additional processes waiting in MPI_Barrier)
#-----------------------------------------------------------------------------
       #bytes #repetitions  t_min[usec]  t_max[usec]  t_avg[usec]   Mbytes/sec
            0         1000         2.97         2.97         2.97         0.00
            1         1000         3.03         3.03         3.03         0.66
            2         1000         3.03         3.03         3.03         1.32
            4         1000         3.06         3.06         3.06         2.61
            8         1000         3.04         3.04         3.04         5.27
           16         1000         3.06         3.06         3.06        10.44
           32         1000         3.07         3.07         3.07        20.85
           64         1000         3.16         3.16         3.16        40.45
          128         1000         3.77         3.77         3.77        67.84
          256         1000         4.13         4.13         4.13       123.90
          512         1000         4.46         4.46         4.46       229.51
         1024         1000         4.94         4.95         4.95       413.69
         2048         1000         5.88         5.88         5.88       696.06
         4096         1000         8.21         8.22         8.21       996.95
         8192         1000        11.19        11.20        11.20      1463.08
        16384         1000        26.45        26.45        26.45      1238.94
        32768         1000        37.25        37.25        37.25      1759.28
        65536          640        48.71        48.71        48.71      2691.00
       131072          320        69.80        69.80        69.80      3755.75
       262144          160       112.68       112.69       112.68      4652.66
       524288           80       198.16       198.16       198.16      5291.49
      1048576           40       369.81       369.81       369.81      5670.94
      2097152           20       712.38       712.39       712.39      5887.62
      4194304           10      1396.38      1396.43      1396.40      6007.18

#-----------------------------------------------------------------------------
# Benchmarking Sendrecv 
# #processes = 4 
# ( 3 additional processes waiting in MPI_Barrier)
#-----------------------------------------------------------------------------
       #bytes #repetitions  t_min[usec]  t_max[usec]  t_avg[usec]   Mbytes/sec
            0         1000         4.53         4.54         4.54         0.00
            1         1000         4.56         4.57         4.57         0.44
            2         1000         4.57         4.58         4.58         0.87
            4         1000         4.53         4.53         4.53         1.76
            8         1000         4.60         4.61         4.61         3.47
           16         1000         4.62         4.63         4.63         6.91
           32         1000         4.60         4.61         4.61        13.87
           64         1000         4.68         4.69         4.68        27.31
          128         1000         5.40         5.42         5.41        47.27
          256         1000         5.67         5.68         5.67        90.18
          512         1000         5.96         5.98         5.97       171.15
         1024         1000         6.54         6.56         6.55       312.27
         2048         1000         7.35         7.36         7.35       556.30
         4096         1000         9.42         9.44         9.43       868.12
         8192         1000        11.95        11.97        11.96      1368.90
        16384         1000        29.03        29.05        29.04      1128.10
^C
[1_01:49:08_aleblanc@farbauti]{~}$ > mpirun --mca 
btl_openib_warn_no_device_params_found 0 --mca orte_base_help_aggregate 0 --mca 
btl openib,vader,self --mca pml ob1 --mca btl_openib_receive_queues 
P,65536,120,64,32 -hostfile /home/soesterreich/ce-mpi-hosts --mca 
rmaps_base_verbose 10 IMB-MPI1
[farbauti.ofa.iol.unh.edu:04649] mca: base: components_register: registering 
framework rmaps components
[farbauti.ofa.iol.unh.edu:04649] mca: base: components_register: found loaded 
component ppr
[farbauti.ofa.iol.unh.edu:04649] mca: base: components_register: component ppr 
register function successful
[farbauti.ofa.iol.unh.edu:04649] mca: base: components_register: found loaded 
component mindist
[farbauti.ofa.iol.unh.edu:04649] mca: base: components_register: component 
mindist register function successful
[farbauti.ofa.iol.unh.edu:04649] mca: base: components_register: found loaded 
component round_robin
[farbauti.ofa.iol.unh.edu:04649] mca: base: components_register: component 
round_robin register function successful
[farbauti.ofa.iol.unh.edu:04649] mca: base: components_register: found loaded 
component resilient
[farbauti.ofa.iol.unh.edu:04649] mca: base: components_register: component 
resilient register function successful
[farbauti.ofa.iol.unh.edu:04649] mca: base: components_register: found loaded 
component rank_file
[farbauti.ofa.iol.unh.edu:04649] mca: base: components_register: component 
rank_file register function successful
[farbauti.ofa.iol.unh.edu:04649] mca: base: components_register: found loaded 
component seq
[farbauti.ofa.iol.unh.edu:04649] mca: base: components_register: component seq 
register function successful
[farbauti.ofa.iol.unh.edu:04649] [[27543,0],0] rmaps:base set policy with NULL 
device NONNULL
[farbauti.ofa.iol.unh.edu:04649] mca: base: components_open: opening rmaps 
components
[farbauti.ofa.iol.unh.edu:04649] mca: base: components_open: found loaded 
component ppr
[farbauti.ofa.iol.unh.edu:04649] mca: base: components_open: component ppr open 
function successful
[farbauti.ofa.iol.unh.edu:04649] mca: base: components_open: found loaded 
component mindist
[farbauti.ofa.iol.unh.edu:04649] mca: base: components_open: component mindist 
open function successful
[farbauti.ofa.iol.unh.edu:04649] mca: base: components_open: found loaded 
component round_robin
[farbauti.ofa.iol.unh.edu:04649] mca: base: components_open: component 
round_robin open function successful
[farbauti.ofa.iol.unh.edu:04649] mca: base: components_open: found loaded 
component resilient
[farbauti.ofa.iol.unh.edu:04649] mca: base: components_open: component 
resilient open function successful
[farbauti.ofa.iol.unh.edu:04649] mca: base: components_open: found loaded 
component rank_file
[farbauti.ofa.iol.unh.edu:04649] mca: base: components_open: component 
rank_file open function successful
[farbauti.ofa.iol.unh.edu:04649] mca: base: components_open: found loaded 
component seq
[farbauti.ofa.iol.unh.edu:04649] mca: base: components_open: component seq open 
function successful
[farbauti.ofa.iol.unh.edu:04649] mca:rmaps:select: checking available component 
ppr
[farbauti.ofa.iol.unh.edu:04649] mca:rmaps:select: Querying component [ppr]
[farbauti.ofa.iol.unh.edu:04649] mca:rmaps:select: checking available component 
mindist
[farbauti.ofa.iol.unh.edu:04649] mca:rmaps:select: Querying component [mindist]
[farbauti.ofa.iol.unh.edu:04649] mca:rmaps:select: checking available component 
round_robin
[farbauti.ofa.iol.unh.edu:04649] mca:rmaps:select: Querying component 
[round_robin]
[farbauti.ofa.iol.unh.edu:04649] mca:rmaps:select: checking available component 
resilient
[farbauti.ofa.iol.unh.edu:04649] mca:rmaps:select: Querying component 
[resilient]
[farbauti.ofa.iol.unh.edu:04649] mca:rmaps:select: checking available component 
rank_file
[farbauti.ofa.iol.unh.edu:04649] mca:rmaps:select: Querying component 
[rank_file]
[farbauti.ofa.iol.unh.edu:04649] mca:rmaps:select: checking available component 
seq
[farbauti.ofa.iol.unh.edu:04649] mca:rmaps:select: Querying component [seq]
[farbauti.ofa.iol.unh.edu:04649] [[27543,0],0]: Final mapper priorities
[farbauti.ofa.iol.unh.edu:04649]        Mapper: ppr Priority: 90
[farbauti.ofa.iol.unh.edu:04649]        Mapper: seq Priority: 60
[farbauti.ofa.iol.unh.edu:04649]        Mapper: resilient Priority: 40
[farbauti.ofa.iol.unh.edu:04649]        Mapper: mindist Priority: 20
[farbauti.ofa.iol.unh.edu:04649]        Mapper: round_robin Priority: 10
[farbauti.ofa.iol.unh.edu:04649]        Mapper: rank_file Priority: 0
[rhea.ofa.iol.unh.edu:05232] mca: base: components_register: registering 
framework rmaps components
[rhea.ofa.iol.unh.edu:05232] mca: base: components_register: found loaded 
component ppr
[rhea.ofa.iol.unh.edu:05232] mca: base: components_register: component ppr 
register function successful
[rhea.ofa.iol.unh.edu:05232] mca: base: components_register: found loaded 
component resilient
[rhea.ofa.iol.unh.edu:05232] mca: base: components_register: component 
resilient register function successful
[rhea.ofa.iol.unh.edu:05232] mca: base: components_register: found loaded 
component rank_file
[rhea.ofa.iol.unh.edu:05232] mca: base: components_register: component 
rank_file register function successful
[rhea.ofa.iol.unh.edu:05232] mca: base: components_register: found loaded 
component round_robin
[rhea.ofa.iol.unh.edu:05232] mca: base: components_register: component 
round_robin register function successful
[rhea.ofa.iol.unh.edu:05232] mca: base: components_register: found loaded 
component seq
[rhea.ofa.iol.unh.edu:05232] mca: base: components_register: component seq 
register function successful
[rhea.ofa.iol.unh.edu:05232] mca: base: components_register: found loaded 
component mindist
[rhea.ofa.iol.unh.edu:05232] mca: base: components_register: component mindist 
register function successful
[rhea.ofa.iol.unh.edu:05232] [[27543,0],4] rmaps:base set policy with NULL 
device NONNULL
[rhea.ofa.iol.unh.edu:05232] mca: base: components_open: opening rmaps 
components
[rhea.ofa.iol.unh.edu:05232] mca: base: components_open: found loaded component 
ppr
[rhea.ofa.iol.unh.edu:05232] mca: base: components_open: component ppr open 
function successful
[rhea.ofa.iol.unh.edu:05232] mca: base: components_open: found loaded component 
resilient
[rhea.ofa.iol.unh.edu:05232] mca: base: components_open: component resilient 
open function successful
[rhea.ofa.iol.unh.edu:05232] mca: base: components_open: found loaded component 
rank_file
[rhea.ofa.iol.unh.edu:05232] mca: base: components_open: component rank_file 
open function successful
[rhea.ofa.iol.unh.edu:05232] mca: base: components_open: found loaded component 
round_robin
[rhea.ofa.iol.unh.edu:05232] mca: base: components_open: component round_robin 
open function successful
[rhea.ofa.iol.unh.edu:05232] mca: base: components_open: found loaded component 
seq
[rhea.ofa.iol.unh.edu:05232] mca: base: components_open: component seq open 
function successful
[rhea.ofa.iol.unh.edu:05232] mca: base: components_open: found loaded component 
mindist
[rhea.ofa.iol.unh.edu:05232] mca: base: components_open: component mindist open 
function successful
[rhea.ofa.iol.unh.edu:05232] mca:rmaps:select: checking available component ppr
[rhea.ofa.iol.unh.edu:05232] mca:rmaps:select: Querying component [ppr]
[rhea.ofa.iol.unh.edu:05232] mca:rmaps:select: checking available component 
resilient
[rhea.ofa.iol.unh.edu:05232] mca:rmaps:select: Querying component [resilient]
[rhea.ofa.iol.unh.edu:05232] mca:rmaps:select: checking available component 
rank_file
[rhea.ofa.iol.unh.edu:05232] mca:rmaps:select: Querying component [rank_file]
[rhea.ofa.iol.unh.edu:05232] mca:rmaps:select: checking available component 
round_robin
[rhea.ofa.iol.unh.edu:05232] mca:rmaps:select: Querying component [round_robin]
[rhea.ofa.iol.unh.edu:05232] mca:rmaps:select: checking available component seq
[rhea.ofa.iol.unh.edu:05232] mca:rmaps:select: Querying component [seq]
[rhea.ofa.iol.unh.edu:05232] mca:rmaps:select: checking available component 
mindist
[rhea.ofa.iol.unh.edu:05232] mca:rmaps:select: Querying component [mindist]
[rhea.ofa.iol.unh.edu:05232] [[27543,0],4]: Final mapper priorities
[rhea.ofa.iol.unh.edu:05232]    Mapper: ppr Priority: 90
[rhea.ofa.iol.unh.edu:05232]    Mapper: seq Priority: 60
[rhea.ofa.iol.unh.edu:05232]    Mapper: resilient Priority: 40
[rhea.ofa.iol.unh.edu:05232]    Mapper: mindist Priority: 20
[rhea.ofa.iol.unh.edu:05232]    Mapper: round_robin Priority: 10
[rhea.ofa.iol.unh.edu:05232]    Mapper: rank_file Priority: 0
[tarvos.ofa.iol.unh.edu:04643] mca: base: components_register: registering 
framework rmaps components
[tarvos.ofa.iol.unh.edu:04643] mca: base: components_register: found loaded 
component mindist
[tarvos.ofa.iol.unh.edu:04643] mca: base: components_register: component 
mindist register function successful
[tarvos.ofa.iol.unh.edu:04643] mca: base: components_register: found loaded 
component seq
[tarvos.ofa.iol.unh.edu:04643] mca: base: components_register: component seq 
register function successful
[tarvos.ofa.iol.unh.edu:04643] mca: base: components_register: found loaded 
component rank_file
[tarvos.ofa.iol.unh.edu:04643] mca: base: components_register: component 
rank_file register function successful
[tarvos.ofa.iol.unh.edu:04643] mca: base: components_register: found loaded 
component round_robin
[tarvos.ofa.iol.unh.edu:04643] mca: base: components_register: component 
round_robin register function successful
[tarvos.ofa.iol.unh.edu:04643] mca: base: components_register: found loaded 
component resilient
[tarvos.ofa.iol.unh.edu:04643] mca: base: components_register: component 
resilient register function successful
[tarvos.ofa.iol.unh.edu:04643] mca: base: components_register: found loaded 
component ppr
[tarvos.ofa.iol.unh.edu:04643] mca: base: components_register: component ppr 
register function successful
[tarvos.ofa.iol.unh.edu:04643] [[27543,0],6] rmaps:base set policy with NULL 
device NONNULL
[tarvos.ofa.iol.unh.edu:04643] mca: base: components_open: opening rmaps 
components
[tarvos.ofa.iol.unh.edu:04643] mca: base: components_open: found loaded 
component mindist
[tarvos.ofa.iol.unh.edu:04643] mca: base: components_open: component mindist 
open function successful
[tarvos.ofa.iol.unh.edu:04643] mca: base: components_open: found loaded 
component seq
[tarvos.ofa.iol.unh.edu:04643] mca: base: components_open: component seq open 
function successful
[tarvos.ofa.iol.unh.edu:04643] mca: base: components_open: found loaded 
component rank_file
[tarvos.ofa.iol.unh.edu:04643] mca: base: components_open: component rank_file 
open function successful
[tarvos.ofa.iol.unh.edu:04643] mca: base: components_open: found loaded 
component round_robin
[tarvos.ofa.iol.unh.edu:04643] mca: base: components_open: component 
round_robin open function successful
[tarvos.ofa.iol.unh.edu:04643] mca: base: components_open: found loaded 
component resilient
[tarvos.ofa.iol.unh.edu:04643] mca: base: components_open: component resilient 
open function successful
[tarvos.ofa.iol.unh.edu:04643] mca: base: components_open: found loaded 
component ppr
[tarvos.ofa.iol.unh.edu:04643] mca: base: components_open: component ppr open 
function successful
[tarvos.ofa.iol.unh.edu:04643] mca:rmaps:select: checking available component 
mindist
[tarvos.ofa.iol.unh.edu:04643] mca:rmaps:select: Querying component [mindist]
[tarvos.ofa.iol.unh.edu:04643] mca:rmaps:select: checking available component 
seq
[tarvos.ofa.iol.unh.edu:04643] mca:rmaps:select: Querying component [seq]
[tarvos.ofa.iol.unh.edu:04643] mca:rmaps:select: checking available component 
rank_file
[tarvos.ofa.iol.unh.edu:04643] mca:rmaps:select: Querying component [rank_file]
[tarvos.ofa.iol.unh.edu:04643] mca:rmaps:select: checking available component 
round_robin
[tarvos.ofa.iol.unh.edu:04643] mca:rmaps:select: Querying component 
[round_robin]
[tarvos.ofa.iol.unh.edu:04643] mca:rmaps:select: checking available component 
resilient
[tarvos.ofa.iol.unh.edu:04643] mca:rmaps:select: Querying component [resilient]
[tarvos.ofa.iol.unh.edu:04643] mca:rmaps:select: checking available component 
ppr
[tarvos.ofa.iol.unh.edu:04643] mca:rmaps:select: Querying component [ppr]
[tarvos.ofa.iol.unh.edu:04643] [[27543,0],6]: Final mapper priorities
[tarvos.ofa.iol.unh.edu:04643]  Mapper: ppr Priority: 90
[tarvos.ofa.iol.unh.edu:04643]  Mapper: seq Priority: 60
[tarvos.ofa.iol.unh.edu:04643]  Mapper: resilient Priority: 40
[tarvos.ofa.iol.unh.edu:04643]  Mapper: mindist Priority: 20
[tarvos.ofa.iol.unh.edu:04643]  Mapper: round_robin Priority: 10
[tarvos.ofa.iol.unh.edu:04643]  Mapper: rank_file Priority: 0
[io.ofa.iol.unh.edu:05130] mca: base: components_register: registering 
framework rmaps components
[io.ofa.iol.unh.edu:05130] mca: base: components_register: found loaded 
component round_robin
[io.ofa.iol.unh.edu:05130] mca: base: components_register: component 
round_robin register function successful
[io.ofa.iol.unh.edu:05130] mca: base: components_register: found loaded 
component ppr
[io.ofa.iol.unh.edu:05130] mca: base: components_register: component ppr 
register function successful
[io.ofa.iol.unh.edu:05130] mca: base: components_register: found loaded 
component seq
[io.ofa.iol.unh.edu:05130] mca: base: components_register: component seq 
register function successful
[io.ofa.iol.unh.edu:05130] mca: base: components_register: found loaded 
component resilient
[io.ofa.iol.unh.edu:05130] mca: base: components_register: component resilient 
register function successful
[io.ofa.iol.unh.edu:05130] mca: base: components_register: found loaded 
component rank_file
[io.ofa.iol.unh.edu:05130] mca: base: components_register: component rank_file 
register function successful
[io.ofa.iol.unh.edu:05130] mca: base: components_register: found loaded 
component mindist
[io.ofa.iol.unh.edu:05130] mca: base: components_register: component mindist 
register function successful
[io.ofa.iol.unh.edu:05130] [[27543,0],2] rmaps:base set policy with NULL device 
NONNULL
[io.ofa.iol.unh.edu:05130] mca: base: components_open: opening rmaps components
[io.ofa.iol.unh.edu:05130] mca: base: components_open: found loaded component 
round_robin
[io.ofa.iol.unh.edu:05130] mca: base: components_open: component round_robin 
open function successful
[io.ofa.iol.unh.edu:05130] mca: base: components_open: found loaded component 
ppr
[io.ofa.iol.unh.edu:05130] mca: base: components_open: component ppr open 
function successful
[io.ofa.iol.unh.edu:05130] mca: base: components_open: found loaded component 
seq
[io.ofa.iol.unh.edu:05130] mca: base: components_open: component seq open 
function successful
[io.ofa.iol.unh.edu:05130] mca: base: components_open: found loaded component 
resilient
[io.ofa.iol.unh.edu:05130] mca: base: components_open: component resilient open 
function successful
[io.ofa.iol.unh.edu:05130] mca: base: components_open: found loaded component 
rank_file
[io.ofa.iol.unh.edu:05130] mca: base: components_open: component rank_file open 
function successful
[io.ofa.iol.unh.edu:05130] mca: base: components_open: found loaded component 
mindist
[io.ofa.iol.unh.edu:05130] mca: base: components_open: component mindist open 
function successful
[io.ofa.iol.unh.edu:05130] mca:rmaps:select: checking available component 
round_robin
[io.ofa.iol.unh.edu:05130] mca:rmaps:select: Querying component [round_robin]
[io.ofa.iol.unh.edu:05130] mca:rmaps:select: checking available component ppr
[io.ofa.iol.unh.edu:05130] mca:rmaps:select: Querying component [ppr]
[io.ofa.iol.unh.edu:05130] mca:rmaps:select: checking available component seq
[io.ofa.iol.unh.edu:05130] mca:rmaps:select: Querying component [seq]
[io.ofa.iol.unh.edu:05130] mca:rmaps:select: checking available component 
resilient
[io.ofa.iol.unh.edu:05130] mca:rmaps:select: Querying component [resilient]
[io.ofa.iol.unh.edu:05130] mca:rmaps:select: checking available component 
rank_file
[io.ofa.iol.unh.edu:05130] mca:rmaps:select: Querying component [rank_file]
[io.ofa.iol.unh.edu:05130] mca:rmaps:select: checking available component 
mindist
[io.ofa.iol.unh.edu:05130] mca:rmaps:select: Querying component [mindist]
[io.ofa.iol.unh.edu:05130] [[27543,0],2]: Final mapper priorities
[io.ofa.iol.unh.edu:05130]      Mapper: ppr Priority: 90
[io.ofa.iol.unh.edu:05130]      Mapper: seq Priority: 60
[io.ofa.iol.unh.edu:05130]      Mapper: resilient Priority: 40
[io.ofa.iol.unh.edu:05130]      Mapper: mindist Priority: 20
[io.ofa.iol.unh.edu:05130]      Mapper: round_robin Priority: 10
[io.ofa.iol.unh.edu:05130]      Mapper: rank_file Priority: 0
[hyperion.ofa.iol.unh.edu:05082] mca: base: components_register: registering 
framework rmaps components
[hyperion.ofa.iol.unh.edu:05082] mca: base: components_register: found loaded 
component resilient
[hyperion.ofa.iol.unh.edu:05082] mca: base: components_register: component 
resilient register function successful
[hyperion.ofa.iol.unh.edu:05082] mca: base: components_register: found loaded 
component seq
[hyperion.ofa.iol.unh.edu:05082] mca: base: components_register: component seq 
register function successful
[hyperion.ofa.iol.unh.edu:05082] mca: base: components_register: found loaded 
component ppr
[hyperion.ofa.iol.unh.edu:05082] mca: base: components_register: component ppr 
register function successful
[hyperion.ofa.iol.unh.edu:05082] mca: base: components_register: found loaded 
component mindist
[hyperion.ofa.iol.unh.edu:05082] mca: base: components_register: component 
mindist register function successful
[hyperion.ofa.iol.unh.edu:05082] mca: base: components_register: found loaded 
component round_robin
[hyperion.ofa.iol.unh.edu:05082] mca: base: components_register: component 
round_robin register function successful
[hyperion.ofa.iol.unh.edu:05082] mca: base: components_register: found loaded 
component rank_file
[hyperion.ofa.iol.unh.edu:05082] mca: base: components_register: component 
rank_file register function successful
[hyperion.ofa.iol.unh.edu:05082] [[27543,0],1] rmaps:base set policy with NULL 
device NONNULL
[hyperion.ofa.iol.unh.edu:05082] mca: base: components_open: opening rmaps 
components
[hyperion.ofa.iol.unh.edu:05082] mca: base: components_open: found loaded 
component resilient
[hyperion.ofa.iol.unh.edu:05082] mca: base: components_open: component 
resilient open function successful
[hyperion.ofa.iol.unh.edu:05082] mca: base: components_open: found loaded 
component seq
[hyperion.ofa.iol.unh.edu:05082] mca: base: components_open: component seq open 
function successful
[hyperion.ofa.iol.unh.edu:05082] mca: base: components_open: found loaded 
component ppr
[hyperion.ofa.iol.unh.edu:05082] mca: base: components_open: component ppr open 
function successful
[hyperion.ofa.iol.unh.edu:05082] mca: base: components_open: found loaded 
component mindist
[hyperion.ofa.iol.unh.edu:05082] mca: base: components_open: component mindist 
open function successful
[hyperion.ofa.iol.unh.edu:05082] mca: base: components_open: found loaded 
component round_robin
[hyperion.ofa.iol.unh.edu:05082] mca: base: components_open: component 
round_robin open function successful
[hyperion.ofa.iol.unh.edu:05082] mca: base: components_open: found loaded 
component rank_file
[hyperion.ofa.iol.unh.edu:05082] mca: base: components_open: component 
rank_file open function successful
[hyperion.ofa.iol.unh.edu:05082] mca:rmaps:select: checking available component 
resilient
[hyperion.ofa.iol.unh.edu:05082] mca:rmaps:select: Querying component 
[resilient]
[hyperion.ofa.iol.unh.edu:05082] mca:rmaps:select: checking available component 
seq
[hyperion.ofa.iol.unh.edu:05082] mca:rmaps:select: Querying component [seq]
[hyperion.ofa.iol.unh.edu:05082] mca:rmaps:select: checking available component 
ppr
[hyperion.ofa.iol.unh.edu:05082] mca:rmaps:select: Querying component [ppr]
[hyperion.ofa.iol.unh.edu:05082] mca:rmaps:select: checking available component 
mindist
[hyperion.ofa.iol.unh.edu:05082] mca:rmaps:select: Querying component [mindist]
[hyperion.ofa.iol.unh.edu:05082] mca:rmaps:select: checking available component 
round_robin
[hyperion.ofa.iol.unh.edu:05082] mca:rmaps:select: Querying component 
[round_robin]
[hyperion.ofa.iol.unh.edu:05082] mca:rmaps:select: checking available component 
rank_file
[hyperion.ofa.iol.unh.edu:05082] mca:rmaps:select: Querying component 
[rank_file]
[hyperion.ofa.iol.unh.edu:05082] [[27543,0],1]: Final mapper priorities
[hyperion.ofa.iol.unh.edu:05082]        Mapper: ppr Priority: 90
[hyperion.ofa.iol.unh.edu:05082]        Mapper: seq Priority: 60
[hyperion.ofa.iol.unh.edu:05082]        Mapper: resilient Priority: 40
[hyperion.ofa.iol.unh.edu:05082]        Mapper: mindist Priority: 20
[hyperion.ofa.iol.unh.edu:05082]        Mapper: round_robin Priority: 10
[hyperion.ofa.iol.unh.edu:05082]        Mapper: rank_file Priority: 0
[jarnsaxa.ofa.iol.unh.edu:05064] mca: base: components_register: registering 
framework rmaps components
[jarnsaxa.ofa.iol.unh.edu:05064] mca: base: components_register: found loaded 
component seq
[jarnsaxa.ofa.iol.unh.edu:05064] mca: base: components_register: component seq 
register function successful
[jarnsaxa.ofa.iol.unh.edu:05064] mca: base: components_register: found loaded 
component ppr
[jarnsaxa.ofa.iol.unh.edu:05064] mca: base: components_register: component ppr 
register function successful
[jarnsaxa.ofa.iol.unh.edu:05064] mca: base: components_register: found loaded 
component resilient
[jarnsaxa.ofa.iol.unh.edu:05064] mca: base: components_register: component 
resilient register function successful
[jarnsaxa.ofa.iol.unh.edu:05064] mca: base: components_register: found loaded 
component round_robin
[jarnsaxa.ofa.iol.unh.edu:05064] mca: base: components_register: component 
round_robin register function successful
[jarnsaxa.ofa.iol.unh.edu:05064] mca: base: components_register: found loaded 
component mindist
[jarnsaxa.ofa.iol.unh.edu:05064] mca: base: components_register: component 
mindist register function successful
[jarnsaxa.ofa.iol.unh.edu:05064] mca: base: components_register: found loaded 
component rank_file
[jarnsaxa.ofa.iol.unh.edu:05064] mca: base: components_register: component 
rank_file register function successful
[jarnsaxa.ofa.iol.unh.edu:05064] [[27543,0],3] rmaps:base set policy with NULL 
device NONNULL
[jarnsaxa.ofa.iol.unh.edu:05064] mca: base: components_open: opening rmaps 
components
[jarnsaxa.ofa.iol.unh.edu:05064] mca: base: components_open: found loaded 
component seq
[jarnsaxa.ofa.iol.unh.edu:05064] mca: base: components_open: component seq open 
function successful
[jarnsaxa.ofa.iol.unh.edu:05064] mca: base: components_open: found loaded 
component ppr
[jarnsaxa.ofa.iol.unh.edu:05064] mca: base: components_open: component ppr open 
function successful
[jarnsaxa.ofa.iol.unh.edu:05064] mca: base: components_open: found loaded 
component resilient
[jarnsaxa.ofa.iol.unh.edu:05064] mca: base: components_open: component 
resilient open function successful
[jarnsaxa.ofa.iol.unh.edu:05064] mca: base: components_open: found loaded 
component round_robin
[jarnsaxa.ofa.iol.unh.edu:05064] mca: base: components_open: component 
round_robin open function successful
[jarnsaxa.ofa.iol.unh.edu:05064] mca: base: components_open: found loaded 
component mindist
[jarnsaxa.ofa.iol.unh.edu:05064] mca: base: components_open: component mindist 
open function successful
[jarnsaxa.ofa.iol.unh.edu:05064] mca: base: components_open: found loaded 
component rank_file
[jarnsaxa.ofa.iol.unh.edu:05064] mca: base: components_open: component 
rank_file open function successful
[jarnsaxa.ofa.iol.unh.edu:05064] mca:rmaps:select: checking available component 
seq
[jarnsaxa.ofa.iol.unh.edu:05064] mca:rmaps:select: Querying component [seq]
[jarnsaxa.ofa.iol.unh.edu:05064] mca:rmaps:select: checking available component 
ppr
[jarnsaxa.ofa.iol.unh.edu:05064] mca:rmaps:select: Querying component [ppr]
[jarnsaxa.ofa.iol.unh.edu:05064] mca:rmaps:select: checking available component 
resilient
[jarnsaxa.ofa.iol.unh.edu:05064] mca:rmaps:select: Querying component 
[resilient]
[jarnsaxa.ofa.iol.unh.edu:05064] mca:rmaps:select: checking available component 
round_robin
[jarnsaxa.ofa.iol.unh.edu:05064] mca:rmaps:select: Querying component 
[round_robin]
[jarnsaxa.ofa.iol.unh.edu:05064] mca:rmaps:select: checking available component 
mindist
[jarnsaxa.ofa.iol.unh.edu:05064] mca:rmaps:select: Querying component [mindist]
[jarnsaxa.ofa.iol.unh.edu:05064] mca:rmaps:select: checking available component 
rank_file
[jarnsaxa.ofa.iol.unh.edu:05064] mca:rmaps:select: Querying component 
[rank_file]
[jarnsaxa.ofa.iol.unh.edu:05064] [[27543,0],3]: Final mapper priorities
[jarnsaxa.ofa.iol.unh.edu:05064]        Mapper: ppr Priority: 90
[jarnsaxa.ofa.iol.unh.edu:05064]        Mapper: seq Priority: 60
[jarnsaxa.ofa.iol.unh.edu:05064]        Mapper: resilient Priority: 40
[jarnsaxa.ofa.iol.unh.edu:05064]        Mapper: mindist Priority: 20
[jarnsaxa.ofa.iol.unh.edu:05064]        Mapper: round_robin Priority: 10
[jarnsaxa.ofa.iol.unh.edu:05064]        Mapper: rank_file Priority: 0
[tarqeq.ofa.iol.unh.edu:04724] mca: base: components_register: registering 
framework rmaps components
[tarqeq.ofa.iol.unh.edu:04724] mca: base: components_register: found loaded 
component seq
[tarqeq.ofa.iol.unh.edu:04724] mca: base: components_register: component seq 
register function successful
[tarqeq.ofa.iol.unh.edu:04724] mca: base: components_register: found loaded 
component ppr
[tarqeq.ofa.iol.unh.edu:04724] mca: base: components_register: component ppr 
register function successful
[tarqeq.ofa.iol.unh.edu:04724] mca: base: components_register: found loaded 
component round_robin
[tarqeq.ofa.iol.unh.edu:04724] mca: base: components_register: component 
round_robin register function successful
[tarqeq.ofa.iol.unh.edu:04724] mca: base: components_register: found loaded 
component rank_file
[tarqeq.ofa.iol.unh.edu:04724] mca: base: components_register: component 
rank_file register function successful
[tarqeq.ofa.iol.unh.edu:04724] mca: base: components_register: found loaded 
component resilient
[tarqeq.ofa.iol.unh.edu:04724] mca: base: components_register: component 
resilient register function successful
[tarqeq.ofa.iol.unh.edu:04724] mca: base: components_register: found loaded 
component mindist
[tarqeq.ofa.iol.unh.edu:04724] mca: base: components_register: component 
mindist register function successful
[tarqeq.ofa.iol.unh.edu:04724] [[27543,0],5] rmaps:base set policy with NULL 
device NONNULL
[tarqeq.ofa.iol.unh.edu:04724] mca: base: components_open: opening rmaps 
components
[tarqeq.ofa.iol.unh.edu:04724] mca: base: components_open: found loaded 
component seq
[tarqeq.ofa.iol.unh.edu:04724] mca: base: components_open: component seq open 
function successful
[tarqeq.ofa.iol.unh.edu:04724] mca: base: components_open: found loaded 
component ppr
[tarqeq.ofa.iol.unh.edu:04724] mca: base: components_open: component ppr open 
function successful
[tarqeq.ofa.iol.unh.edu:04724] mca: base: components_open: found loaded 
component round_robin
[tarqeq.ofa.iol.unh.edu:04724] mca: base: components_open: component 
round_robin open function successful
[tarqeq.ofa.iol.unh.edu:04724] mca: base: components_open: found loaded 
component rank_file
[tarqeq.ofa.iol.unh.edu:04724] mca: base: components_open: component rank_file 
open function successful
[tarqeq.ofa.iol.unh.edu:04724] mca: base: components_open: found loaded 
component resilient
[tarqeq.ofa.iol.unh.edu:04724] mca: base: components_open: component resilient 
open function successful
[tarqeq.ofa.iol.unh.edu:04724] mca: base: components_open: found loaded 
component mindist
[tarqeq.ofa.iol.unh.edu:04724] mca: base: components_open: component mindist 
open function successful
[tarqeq.ofa.iol.unh.edu:04724] mca:rmaps:select: checking available component 
seq
[tarqeq.ofa.iol.unh.edu:04724] mca:rmaps:select: Querying component [seq]
[tarqeq.ofa.iol.unh.edu:04724] mca:rmaps:select: checking available component 
ppr
[tarqeq.ofa.iol.unh.edu:04724] mca:rmaps:select: Querying component [ppr]
[tarqeq.ofa.iol.unh.edu:04724] mca:rmaps:select: checking available component 
round_robin
[tarqeq.ofa.iol.unh.edu:04724] mca:rmaps:select: Querying component 
[round_robin]
[tarqeq.ofa.iol.unh.edu:04724] mca:rmaps:select: checking available component 
rank_file
[tarqeq.ofa.iol.unh.edu:04724] mca:rmaps:select: Querying component [rank_file]
[tarqeq.ofa.iol.unh.edu:04724] mca:rmaps:select: checking available component 
resilient
[tarqeq.ofa.iol.unh.edu:04724] mca:rmaps:select: Querying component [resilient]
[tarqeq.ofa.iol.unh.edu:04724] mca:rmaps:select: checking available component 
mindist
[tarqeq.ofa.iol.unh.edu:04724] mca:rmaps:select: Querying component [mindist]
[tarqeq.ofa.iol.unh.edu:04724] [[27543,0],5]: Final mapper priorities
[tarqeq.ofa.iol.unh.edu:04724]  Mapper: ppr Priority: 90
[tarqeq.ofa.iol.unh.edu:04724]  Mapper: seq Priority: 60
[tarqeq.ofa.iol.unh.edu:04724]  Mapper: resilient Priority: 40
[tarqeq.ofa.iol.unh.edu:04724]  Mapper: mindist Priority: 20
[tarqeq.ofa.iol.unh.edu:04724]  Mapper: round_robin Priority: 10
[tarqeq.ofa.iol.unh.edu:04724]  Mapper: rank_file Priority: 0
[farbauti.ofa.iol.unh.edu:04649] mca:rmaps: mapping job [27543,1]
[farbauti.ofa.iol.unh.edu:04649] AVAILABLE NODES FOR MAPPING:
[farbauti.ofa.iol.unh.edu:04649]     node: farbauti daemon: 0
[farbauti.ofa.iol.unh.edu:04649]     node: hyperion-ce daemon: 1
[farbauti.ofa.iol.unh.edu:04649]     node: io-ce daemon: 2
[farbauti.ofa.iol.unh.edu:04649]     node: jarnsaxa-ce daemon: 3
[farbauti.ofa.iol.unh.edu:04649]     node: rhea-ce daemon: 4
[farbauti.ofa.iol.unh.edu:04649]     node: tarqeq-ce daemon: 5
[farbauti.ofa.iol.unh.edu:04649]     node: tarvos-ce daemon: 6
[farbauti.ofa.iol.unh.edu:04649] mca:rmaps: setting mapping policies for job 
[27543,1] nprocs 7
[farbauti.ofa.iol.unh.edu:04649] mca:rmaps[177] mapping not set by user - using 
bynuma
[farbauti.ofa.iol.unh.edu:04649] mca:rmaps[317] binding not given - using bynuma
[farbauti.ofa.iol.unh.edu:04649] mca:rmaps:ppr: job [27543,1] not using ppr 
mapper PPR NULL policy PPR NOTSET
[farbauti.ofa.iol.unh.edu:04649] mca:rmaps:seq: job [27543,1] not using seq 
mapper
[farbauti.ofa.iol.unh.edu:04649] mca:rmaps:resilient: cannot perform initial 
map of job [27543,1] - no fault groups
[farbauti.ofa.iol.unh.edu:04649] mca:rmaps:mindist: job [27543,1] not using 
mindist mapper
[farbauti.ofa.iol.unh.edu:04649] mca:rmaps:rr: mapping job [27543,1]
[farbauti.ofa.iol.unh.edu:04649] AVAILABLE NODES FOR MAPPING:
[farbauti.ofa.iol.unh.edu:04649]     node: farbauti daemon: 0
[farbauti.ofa.iol.unh.edu:04649]     node: hyperion-ce daemon: 1
[farbauti.ofa.iol.unh.edu:04649]     node: io-ce daemon: 2
[farbauti.ofa.iol.unh.edu:04649]     node: jarnsaxa-ce daemon: 3
[farbauti.ofa.iol.unh.edu:04649]     node: rhea-ce daemon: 4
[farbauti.ofa.iol.unh.edu:04649]     node: tarqeq-ce daemon: 5
[farbauti.ofa.iol.unh.edu:04649]     node: tarvos-ce daemon: 6
[farbauti.ofa.iol.unh.edu:04649] mca:rmaps:rr: mapping no-span by NUMANode for 
job [27543,1] slots 7 num_procs 7
[farbauti.ofa.iol.unh.edu:04649] mca:rmaps:rr: found 2 NUMANode objects on node 
farbauti
[farbauti.ofa.iol.unh.edu:04649] mca:rmaps:rr: calculated nprocs 1
[farbauti.ofa.iol.unh.edu:04649] mca:rmaps:rr: assigning nprocs 1
[farbauti.ofa.iol.unh.edu:04649] mca:rmaps:rr: found 2 NUMANode objects on node 
io-ce
[farbauti.ofa.iol.unh.edu:04649] mca:rmaps:rr: calculated nprocs 1
[farbauti.ofa.iol.unh.edu:04649] mca:rmaps:rr: assigning nprocs 1
[farbauti.ofa.iol.unh.edu:04649] mca:rmaps:rr: found 2 NUMANode objects on node 
jarnsaxa-ce
[farbauti.ofa.iol.unh.edu:04649] mca:rmaps:rr: calculated nprocs 1
[farbauti.ofa.iol.unh.edu:04649] mca:rmaps:rr: assigning nprocs 1
[farbauti.ofa.iol.unh.edu:04649] mca:rmaps:rr: found 2 NUMANode objects on node 
rhea-ce
[farbauti.ofa.iol.unh.edu:04649] mca:rmaps:rr: calculated nprocs 1
[farbauti.ofa.iol.unh.edu:04649] mca:rmaps:rr: assigning nprocs 1
[farbauti.ofa.iol.unh.edu:04649] mca:rmaps:rr: found 2 NUMANode objects on node 
tarvos-ce
[farbauti.ofa.iol.unh.edu:04649] mca:rmaps:rr: calculated nprocs 1
[farbauti.ofa.iol.unh.edu:04649] mca:rmaps:rr: assigning nprocs 1
[farbauti.ofa.iol.unh.edu:04649] mca:rmaps:rr: found 2 NUMANode objects on node 
farbauti
[farbauti.ofa.iol.unh.edu:04649] mca:rmaps:rr: calculated nprocs 0
[farbauti.ofa.iol.unh.edu:04649] mca:rmaps:rr: assigning nprocs 1
--------------------------------------------------------------------------
There are not enough slots available in the system to satisfy the 7 slots
that were requested by the application:
  IMB-MPI1

Either request fewer slots for your application, or make more slots available
for use.
--------------------------------------------------------------------------
[farbauti.ofa.iol.unh.edu:04649] mca: base: close: component ppr closed
[farbauti.ofa.iol.unh.edu:04649] mca: base: close: unloading component ppr
[farbauti.ofa.iol.unh.edu:04649] mca: base: close: component mindist closed
[farbauti.ofa.iol.unh.edu:04649] mca: base: close: unloading component mindist
[farbauti.ofa.iol.unh.edu:04649] mca: base: close: component round_robin closed
[farbauti.ofa.iol.unh.edu:04649] mca: base: close: unloading component 
round_robin
[farbauti.ofa.iol.unh.edu:04649] mca: base: close: component resilient closed
[farbauti.ofa.iol.unh.edu:04649] mca: base: close: unloading component resilient
[farbauti.ofa.iol.unh.edu:04649] mca: base: close: component rank_file closed
[farbauti.ofa.iol.unh.edu:04649] mca: base: close: unloading component rank_file
[farbauti.ofa.iol.unh.edu:04649] mca: base: close: component seq closed
[farbauti.ofa.iol.unh.edu:04649] mca: base: close: unloading component seq
_______________________________________________
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Reply via email to