Re: [OMPI users] Sending large boradcasts
Hi Brock, That message should only be 2MB. Are you sure its not a mismatch of message lengths in MPI_Bcast calls? David On 01/04/2011 03:47 AM, Brock Palen wrote: I have a user who reports that sending a broadcast of 540*1080 of reals (just over 2GB) fails with this: *** An error occurred in MPI_Bcast *** on communicator MPI_COMM_WORLD *** MPI_ERR_TRUNCATE: message truncated *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort) I was reading the archives and there appears to be an issue with large messages. I was a little confused, is there a way to send messages larger than 2GB? The user has access to some IB machines, per a note in the archives there was an issue with writev() would this issue only be related to messages over ethernet? Thanks just trying to get some clarification. Brock Palen www.umich.edu/~brockp Center for Advanced Computing bro...@umich.edu (734)936-1985
Re: [OMPI users] Sending large boradcasts
Hi Brock He's probably hitting the MPI address boundary of 2GB. A workaround is to declare a user defined type (MPI_TYPE_CONTIGUOUS, or MPI_TYPE_VECTOR), to bundle a bunch of primitive data (e.g. reals), then send (broadcast for him/her) a smaller number of those types. See this thread: http://www.open-mpi.org/community/lists/users/2010/11/14792.php http://www.open-mpi.org/community/lists/users/2010/11/14816.php I hope this helps, Gus Correa On Jan 3, 2011, at 11:47 AM, Brock Palen wrote: > I have a user who reports that sending a broadcast of > > 540*1080 of reals (just over 2GB) fails with this: > > > *** An error occurred in MPI_Bcast > *** on communicator MPI_COMM_WORLD > *** MPI_ERR_TRUNCATE: message truncated > *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort) > > I was reading the archives and there appears to be an issue with large > messages. I was a little confused, is there a way to send messages larger > than 2GB? > > The user has access to some IB machines, per a note in the archives there was > an issue with writev() would this issue only be related to messages over > ethernet? > > Thanks just trying to get some clarification. > > Brock Palen > www.umich.edu/~brockp > Center for Advanced Computing > bro...@umich.edu > (734)936-1985 > > > > > ___ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users
Re: [OMPI users] Granular locks?
On Oct 2, 2010, at 10:54 , Gijsbert Wiesenekker wrote: > > On Oct 1, 2010, at 23:24 , Gijsbert Wiesenekker wrote: > >> I have a large array that is shared between two processes. One process >> updates array elements randomly, the other process reads array elements >> randomly. Most of the time these writes and reads do not overlap. >> The current version of the code uses Linux shared memory with NSEMS >> semaphores. When array element i has to be read or updated semaphore (i % >> NSEMS) is used. if NSEMS = 1 the entire array will be locked which leads to >> unnecessary waits because reads and writes do not overlap most of the time. >> Performance increases as NSEMS increases, and flattens out at NSEMS = 32, at >> which point the code runs twice as fast when compared to NSEMS = 1. >> I want to change the code to use OpenMPI RMA, but MPI_Win_lock locks the >> entire array, which is similar to NSEMS = 1. Is there a way to have more >> granular locks? >> >> Gijsbert >> > > Also, is there an MPI_Win_lock equavalent for IPC_NOWAIT? > > Gijsbert > FYI, as in my case the performance penalty by using OpenMPI RMA instead of shared memory was too large I have written a couple of wrapper functions that use OpenMPI to gracefully allocate and release shared memory: //mpi_alloc_shm is a collective operation that allocates arg_nrecords of arg_record_size each in the shared memory segment identified by arg_key with arg_nsems semaphores to control access. //arg_key is the shared memory key. //arg_nrecords is the number of records. //arg_record_size is the size of a record. //arg_default is the default record value. If not equal to NULL all arg_nrecord records will be initialized to *arg_default. //arg_nsems is the number of semaphores that will be used to control access. If record irecord has to be updated or read, semaphore (irecord % arg_nsems) will be used for exclusive access. //arg_mpi_id is the mpi_id of the process that will create the shared memory segment. If the mpi_id of the calling process is not equal to arg_mpi_id the process will not create but try to open it. void mpi_alloc_shm(key_t arg_key, i64_t arg_nrecords, i64_t arg_record_size, void *arg_default, int arg_nsems, int arg_mpi_id, MPI_Comm comm); //mpi_shm_put updates record irecord in the shared memory segment identified by shm_key with value *source. void mpi_shm_put(key_t shm_key, void *source, i64_t irecord); //mpi_shm_get tries to read record irecord in the shared memory segment identified by shm_key using IPC_NO_WAIT to request a lock. //FALSE is returned if the lock could not be obtained, else TRUE and the record in *dest. //as in my case only the creator of the shared memory segment will update it, a lock is not used if the creator tries to read record irecord. int mpi_shm_get(key_t shm_key, i64_t irecord, void *dest); //mpi_free_shm is a collective operation that deallocates the shared memory segment identified by shm_key void mpi_free_shm(key_t shm_key, MPI_Comm comm); Please feel free to contact me if you would like to have a copy of the source code of these routines. Regards, Gijsbert
[OMPI users] Sending MPI-message from master to master
I've not found any helpful information about possibility of sending
messages from a node of cluster to the same node (for example
MPI_Send(, 1, MPI_UNSIGNED_LONG_LONG, 0, 0, MPI_COMM_WORLD) from
node #0).
I wrote program with two threads and one thread MPI_Send message to
another thread that must it MPI_Recv. And such program freezes because
interlock occurs sometime (not always).
If it's possible I've attached cpp-file with example of problem.
Thanks a lot for your attantion.
#include
#include
#include
#include
#include "tbb/spin_mutex.h"
#include "tbb/mutex.h"
#include
#include
#include
#include
#include
#include
#include
#include
#include "LibraryInterface.h"
bool continueProcessing = true;
struct MPITags {
typedef int type;
enum {
NoCommand,
FinishProcessing,
NewTask,
ResultSize
};
};
class SystemPars {
public:
friend class boost::serialization::access;
int MPI_size;
SystemPars() {}
template
void serialize(Archive & ar, const unsigned int version) {
ar & MPI_size;
}
} parObject = SystemPars();
bool yesno(std::string text) {
std::cout << "Are you sure " << text << "? (y/n): ";
char c;
std::cin >> c;
if (c == 'y') return true;
else return false;
}
class ConsoleHandlerThread {
public:
void operator ()() {
std::string s;
while(false) {
std::cin >> s;
if (s == "exit") {
if (yesno("want to exit")) {
break;
}
} else {
std::cout << "Unknown command.\n";
}
}
}
ConsoleHandlerThread() { }
};
class ConcurentTasksContainer {
private:
std::vector minTasks;
size_t minTasksNum;
std::vector commands;
size_t commandsNum;
tbb::spin_mutex qMutex;
std::vector haveSendedFP;
size_t nFinishedNodes;
public:
ConcurentTasksContainer() {
minTasksNum = 0;
qMutex = tbb::spin_mutex();
nFinishedNodes = 0;
}
void resize(size_t size) {
minTasks.resize(size);
commands.resize(size);
haveSendedFP.resize(size);
nFinishedNodes = size;
}
bool hasCommands() {
tbb::spin_mutex::scoped_lock l(qMutex);
return commandsNum > 0;
}
bool hasCommand(size_t node) {
tbb::spin_mutex::scoped_lock l(qMutex);
return commands[node] != MPITags::NoCommand;
}
void setCommand(size_t node, MPITags::type com) {
tbb::spin_mutex::scoped_lock l(qMutex);
if (!haveSendedFP[node] && commands[node] == MPITags::NoCommand) {
commands[node] = com;
commandsNum++;
}
if (com == MPITags::FinishProcessing) {
haveSendedFP[node] = true;
}
}
MPITags::type getCommand(size_t node) {
tbb::spin_mutex::scoped_lock l(qMutex);
MPITags::type com = commands[node];
if (commands[node] == MPITags::FinishProcessing) {
nFinishedNodes--;
}
if (commands[node] != MPITags::NoCommand) {
commands[node] = MPITags::NoCommand;
commandsNum--;
}
return com;
}
void newTask(size_t node, UInt64 task) {
tbb::spin_mutex::scoped_lock l(qMutex);
if (!haveSendedFP[node]) {
minTasks[node] = task;
minTasksNum++;
}
}
void endTask(size_t node) {
tbb::spin_mutex::scoped_lock l(qMutex);
minTasks[node] = 0;
minTasksNum--;
}
bool isEmpty() {
tbb::spin_mutex::scoped_lock l(qMutex);
return minTasksNum == 0 && commandsNum == 0;
}
bool isProcessing() {
tbb::spin_mutex::scoped_lock l(qMutex);
return nFinishedNodes != 0;
}
} taskContainer;
UInt8 stMutex = true;
class ServerThread {
public:
void operator ()() {
std::cout << "ServerThread: begin\n";
srand(time(NULL));
taskContainer.resize(parObject.MPI_size);
MPI_Status state;
UInt64 f;
UInt32 FunctionsNum;
stMutex = false;
for(size_t i = parObject.MPI_size; i--; ) {
f = (rand() << 32);
MPI_Send(, 1, MPI_UNSIGNED_LONG_LONG, i, MPITags::NewTask, MPI_COMM_WORLD);
taskContainer.newTask(i, f);
std::cout << "ServerThread: present to " << i << std::endl;
}
while(!taskContainer.isEmpty()) {
MPI_Recv(, 1, MPI_UNSIGNED_LONG, MPI_ANY_SOURCE, MPITags::ResultSize, MPI_COMM_WORLD, );
std::cout << "ServerThread: postreceived from " << state.MPI_SOURCE << "\n";
taskContainer.endTask(state.MPI_SOURCE);
MPI_Send(, 1, MPI_UNSIGNED_LONG_LONG, state.MPI_SOURCE, MPITags::FinishProcessing, MPI_COMM_WORLD);
}
std::cout << "ServerThread: end\n";
}
ServerThread() { }
};
void tasksProcessing(const int rank) {
srand(time(NULL));
[OMPI users] Sending large boradcasts
I have a user who reports that sending a broadcast of 540*1080 of reals (just over 2GB) fails with this: *** An error occurred in MPI_Bcast *** on communicator MPI_COMM_WORLD *** MPI_ERR_TRUNCATE: message truncated *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort) I was reading the archives and there appears to be an issue with large messages. I was a little confused, is there a way to send messages larger than 2GB? The user has access to some IB machines, per a note in the archives there was an issue with writev() would this issue only be related to messages over ethernet? Thanks just trying to get some clarification. Brock Palen www.umich.edu/~brockp Center for Advanced Computing bro...@umich.edu (734)936-1985
Re: [OMPI users] Using MPI_Put/Get correctly?
I'm using Open MPI 1.4.3, is the bug in that version as well? -Original Message- From: users-boun...@open-mpi.org [mailto:users-boun...@open-mpi.org] On Behalf Of Barrett, Brian W Sent: Monday, January 03, 2011 11:35 AM To: Open MPI Users Subject: Re: [OMPI users] Using MPI_Put/Get correctly? Matt - There's a known bug in the datatype engine of Open MPI 1.5 that breaks MPI One-sided when used with user-defined datatypes. Unfortunately, I don't have a timetable as to when it will be fixed. Brian On Jan 3, 2011, at 9:18 AM, Grismer,Matthew J Civ USAF AFMC AFRL/RBAT wrote: > Unfortunately correcting the integer type for the displacement does not fix > the problem in my code, argh! So, thinking this might have something to do > with the large arrays and amount of data being passed in the actual code, I > modified my example (attached putbothways2.f90) so that the array sizes and > amount of data swapped are nearly identical to the code giving me the issue. > I also filled the array that is shared with random data, instead of 0's and > 1's, to ensure nothing special was happening due to the simple, uniform > data. Unfortunately, the example works great, but my actual code still seg > faults. > > So, the summary is the example code that uses MPI_Put calls with indexed > datatypes to swap data between 2 processors works without issue, while the > actual code that communicates in the same manner fails. The only difference > is the actual code allocates many other arrays, which are communicated in > various ways (sends, puts, broadcasts, etc). I checked and re-checked all > the argument lists associated with the indexed data, window, and puts; > everything looks correct. Any thoughts or suggestions on how to proceed? > > Matt > > -Original Message- > From: users-boun...@open-mpi.org [mailto:users-boun...@open-mpi.org] On > Behalf Of Grismer,Matthew J Civ USAF AFMC AFRL/RBAT > Sent: Wednesday, December 29, 2010 1:42 PM > To: Open MPI Users > Subject: Re: [OMPI users] Using MPI_Put/Get correctly? > > Someone correctly pointed out the bug in my examples. In the MPI_Put I > pass a 0 as the displacement, however, the argument must be of type > integer (kind=MPI_ADDRESS_KIND), which is NOT the default integer type. > Replacing the 0 with the correct integer type fixes both examples. Now > to see if it fixes the actual code I am having difficulty with... > > -Original Message- > From: users-boun...@open-mpi.org [mailto:users-boun...@open-mpi.org] On > Behalf Of Grismer,Matthew J Civ USAF AFMC AFRL/RBAT > Sent: Monday, December 27, 2010 5:33 PM > To: Open MPI Users > Subject: Re: [OMPI users] Using MPI_Put/Get correctly? > > I decided to try and isolate the issue, and created two example test > programs > that appear to highlight an issue > with Open MPI; both die when I run them on 2 processors. I am pretty > certain > the first (putoneway.f90) should work, as I am only doing a single put > from > one processor to a second processor; the target processor is doing > nothing > with the window'ed array that is receiving the data. My guess is the > problem > lies in the indexed datatypes that I am using for both the origin and > target. > > The second case (putbothways.f90) closely mirrors what I am actually > trying > to do in my code, that is have each processor put into the other > processors > window'ed array at the same time. So, each process is sending from and > receiving into the same array at the same time, with no overlap in the > sent > and received data. Once again I'm using indexed data types for both the > origin and target. > > To build: mpif90 putoneway.f90 > To run: mpiexec -np 2 a.out > > Matt > > -Original Message- > From: users-boun...@open-mpi.org [mailto:users-boun...@open-mpi.org] On > Behalf Of James Dinan > Sent: Thursday, December 16, 2010 10:09 AM > To: Open MPI Users > Subject: Re: [OMPI users] Using MPI_Put/Get correctly? > > On 12/16/2010 08:34 AM, Jeff Squyres wrote: >> Additionally, since MPI-3 is updating the semantics of the one-sided >> stuff, it might be worth waiting for all those clarifications before >> venturing into the MPI one-sided realm. One-sided semantics are much >> more subtle and complex than two-sided semantics. > > Hi Jeff, > > I don't think we should give users the hope that MPI-3 RMA will be out > tomorrow. The RMA revisions are still in proposal form and need work. > Realistically speaking, we might be able to get this accepted into the > standard within a year and it will be another year before > implementations catch up. If users need one-sided now, they should use > the MPI-2 one-sided API. > > MPI-3 RMA extends MPI-2 RMA and will be backward compatible, so anything > you write now will still work. It's still unclear to me whether MPI-3's > RMA semantics will be the leap forward in usability we have hoped for. > We are trying to make it more flexible, but there will likely still be > tricky parts due
Re: [OMPI users] Using MPI_Put/Get correctly?
Matt - There's a known bug in the datatype engine of Open MPI 1.5 that breaks MPI One-sided when used with user-defined datatypes. Unfortunately, I don't have a timetable as to when it will be fixed. Brian On Jan 3, 2011, at 9:18 AM, Grismer,Matthew J Civ USAF AFMC AFRL/RBAT wrote: > Unfortunately correcting the integer type for the displacement does not fix > the problem in my code, argh! So, thinking this might have something to do > with the large arrays and amount of data being passed in the actual code, I > modified my example (attached putbothways2.f90) so that the array sizes and > amount of data swapped are nearly identical to the code giving me the issue. > I also filled the array that is shared with random data, instead of 0's and > 1's, to ensure nothing special was happening due to the simple, uniform > data. Unfortunately, the example works great, but my actual code still seg > faults. > > So, the summary is the example code that uses MPI_Put calls with indexed > datatypes to swap data between 2 processors works without issue, while the > actual code that communicates in the same manner fails. The only difference > is the actual code allocates many other arrays, which are communicated in > various ways (sends, puts, broadcasts, etc). I checked and re-checked all > the argument lists associated with the indexed data, window, and puts; > everything looks correct. Any thoughts or suggestions on how to proceed? > > Matt > > -Original Message- > From: users-boun...@open-mpi.org [mailto:users-boun...@open-mpi.org] On > Behalf Of Grismer,Matthew J Civ USAF AFMC AFRL/RBAT > Sent: Wednesday, December 29, 2010 1:42 PM > To: Open MPI Users > Subject: Re: [OMPI users] Using MPI_Put/Get correctly? > > Someone correctly pointed out the bug in my examples. In the MPI_Put I > pass a 0 as the displacement, however, the argument must be of type > integer (kind=MPI_ADDRESS_KIND), which is NOT the default integer type. > Replacing the 0 with the correct integer type fixes both examples. Now > to see if it fixes the actual code I am having difficulty with... > > -Original Message- > From: users-boun...@open-mpi.org [mailto:users-boun...@open-mpi.org] On > Behalf Of Grismer,Matthew J Civ USAF AFMC AFRL/RBAT > Sent: Monday, December 27, 2010 5:33 PM > To: Open MPI Users > Subject: Re: [OMPI users] Using MPI_Put/Get correctly? > > I decided to try and isolate the issue, and created two example test > programs > that appear to highlight an issue > with Open MPI; both die when I run them on 2 processors. I am pretty > certain > the first (putoneway.f90) should work, as I am only doing a single put > from > one processor to a second processor; the target processor is doing > nothing > with the window'ed array that is receiving the data. My guess is the > problem > lies in the indexed datatypes that I am using for both the origin and > target. > > The second case (putbothways.f90) closely mirrors what I am actually > trying > to do in my code, that is have each processor put into the other > processors > window'ed array at the same time. So, each process is sending from and > receiving into the same array at the same time, with no overlap in the > sent > and received data. Once again I'm using indexed data types for both the > origin and target. > > To build: mpif90 putoneway.f90 > To run: mpiexec -np 2 a.out > > Matt > > -Original Message- > From: users-boun...@open-mpi.org [mailto:users-boun...@open-mpi.org] On > Behalf Of James Dinan > Sent: Thursday, December 16, 2010 10:09 AM > To: Open MPI Users > Subject: Re: [OMPI users] Using MPI_Put/Get correctly? > > On 12/16/2010 08:34 AM, Jeff Squyres wrote: >> Additionally, since MPI-3 is updating the semantics of the one-sided >> stuff, it might be worth waiting for all those clarifications before >> venturing into the MPI one-sided realm. One-sided semantics are much >> more subtle and complex than two-sided semantics. > > Hi Jeff, > > I don't think we should give users the hope that MPI-3 RMA will be out > tomorrow. The RMA revisions are still in proposal form and need work. > Realistically speaking, we might be able to get this accepted into the > standard within a year and it will be another year before > implementations catch up. If users need one-sided now, they should use > the MPI-2 one-sided API. > > MPI-3 RMA extends MPI-2 RMA and will be backward compatible, so anything > you write now will still work. It's still unclear to me whether MPI-3's > RMA semantics will be the leap forward in usability we have hoped for. > We are trying to make it more flexible, but there will likely still be > tricky parts due to portability and performance concerns. > > So, my advice: don't be scared of MPI-2. I agree, it's complicated, but > once you get acclimated it's not that bad. Really. :) > > Best, > ~Jim. > ___ > users mailing list > us...@open-mpi.org >
Re: [OMPI users] Using MPI_Put/Get correctly?
Unfortunately correcting the integer type for the displacement does not fix the problem in my code, argh! So, thinking this might have something to do with the large arrays and amount of data being passed in the actual code, I modified my example (attached putbothways2.f90) so that the array sizes and amount of data swapped are nearly identical to the code giving me the issue. I also filled the array that is shared with random data, instead of 0's and 1's, to ensure nothing special was happening due to the simple, uniform data. Unfortunately, the example works great, but my actual code still seg faults. So, the summary is the example code that uses MPI_Put calls with indexed datatypes to swap data between 2 processors works without issue, while the actual code that communicates in the same manner fails. The only difference is the actual code allocates many other arrays, which are communicated in various ways (sends, puts, broadcasts, etc). I checked and re-checked all the argument lists associated with the indexed data, window, and puts; everything looks correct. Any thoughts or suggestions on how to proceed? Matt -Original Message- From: users-boun...@open-mpi.org [mailto:users-boun...@open-mpi.org] On Behalf Of Grismer,Matthew J Civ USAF AFMC AFRL/RBAT Sent: Wednesday, December 29, 2010 1:42 PM To: Open MPI Users Subject: Re: [OMPI users] Using MPI_Put/Get correctly? Someone correctly pointed out the bug in my examples. In the MPI_Put I pass a 0 as the displacement, however, the argument must be of type integer (kind=MPI_ADDRESS_KIND), which is NOT the default integer type. Replacing the 0 with the correct integer type fixes both examples. Now to see if it fixes the actual code I am having difficulty with... -Original Message- From: users-boun...@open-mpi.org [mailto:users-boun...@open-mpi.org] On Behalf Of Grismer,Matthew J Civ USAF AFMC AFRL/RBAT Sent: Monday, December 27, 2010 5:33 PM To: Open MPI Users Subject: Re: [OMPI users] Using MPI_Put/Get correctly? I decided to try and isolate the issue, and created two example test programs that appear to highlight an issue with Open MPI; both die when I run them on 2 processors. I am pretty certain the first (putoneway.f90) should work, as I am only doing a single put from one processor to a second processor; the target processor is doing nothing with the window'ed array that is receiving the data. My guess is the problem lies in the indexed datatypes that I am using for both the origin and target. The second case (putbothways.f90) closely mirrors what I am actually trying to do in my code, that is have each processor put into the other processors window'ed array at the same time. So, each process is sending from and receiving into the same array at the same time, with no overlap in the sent and received data. Once again I'm using indexed data types for both the origin and target. To build: mpif90 putoneway.f90 To run: mpiexec -np 2 a.out Matt -Original Message- From: users-boun...@open-mpi.org [mailto:users-boun...@open-mpi.org] On Behalf Of James Dinan Sent: Thursday, December 16, 2010 10:09 AM To: Open MPI Users Subject: Re: [OMPI users] Using MPI_Put/Get correctly? On 12/16/2010 08:34 AM, Jeff Squyres wrote: > Additionally, since MPI-3 is updating the semantics of the one-sided > stuff, it might be worth waiting for all those clarifications before > venturing into the MPI one-sided realm. One-sided semantics are much > more subtle and complex than two-sided semantics. Hi Jeff, I don't think we should give users the hope that MPI-3 RMA will be out tomorrow. The RMA revisions are still in proposal form and need work. Realistically speaking, we might be able to get this accepted into the standard within a year and it will be another year before implementations catch up. If users need one-sided now, they should use the MPI-2 one-sided API. MPI-3 RMA extends MPI-2 RMA and will be backward compatible, so anything you write now will still work. It's still unclear to me whether MPI-3's RMA semantics will be the leap forward in usability we have hoped for. We are trying to make it more flexible, but there will likely still be tricky parts due to portability and performance concerns. So, my advice: don't be scared of MPI-2. I agree, it's complicated, but once you get acclimated it's not that bad. Really. :) Best, ~Jim. ___ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users ___ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users putbothways2.f90 Description: Binary data smime.p7s Description: S/MIME cryptographic signature
Re: [OMPI users] Windows installers of 1.5.1 - No Fortan ?
Hi Damien, Unfortunately, we don't have a valid license for Intel Fortran compiler at moment on the machine that we built this installer. Regards, Shiqing On 12/29/2010 6:47 AM, Damien Hocking wrote: Jeff, Shiqing, anyone... I notice there's no Fortan support in the Windows binary versions of 1.5.1 on the website. Is that a deliberate decision? Damien ___ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users
Re: [OMPI users] memory consumption on rank 0 and btl_openib_receive_queues use
hi, i'd like to know if someone had a chance to check at the issue I reported. thanks and happy new year ! éloi On 12/21/2010 10:58 AM, Eloi Gaudry wrote: hi, when launching a parallel computation on 128 nodes using openib and the "-mca btl_openib_receive_queues P,65536,256,192,128" option, i observe a rather large memory consumption (2GB: 65336*256*128) on the process with rank 0 (only). this memory seems to be initialized and in use as the resident memory indicator matches the virtual one. at the time of observation, a call to MPI_Init has been done, but no communication (user side) has been done. i'd like to know why the other processes doesn't behave the same: - other processes located on the same nodes don't use that amount of memory - all others processes (i.e. located on any other nodes) neither i'm using OpenMPI-1.4.2, built with gcc-4.3.4 and '--enable-cxx-exceptions --with-pic --with-threads=posix' options. the cluster is based on eight-core nodes using mellanox hca. thanks for your help, éloi -- Eloi Gaudry Senior Product Development Engineer Free Field Technologies Company Website: http://www.fft.be Direct Phone Number: +32 10 495 147
