Re: [OMPI users] alltoall messages > 2^26

2011-04-11 Thread Michael Di Domenico
Here's a chunk of code that reproduces the error everytime on my cluster

If you call it with $((2**24)) as a parameter it should run fine, change it
to $((2**27)) and it will stall

On Tue, Apr 5, 2011 at 11:24 AM, Terry Dontje wrote:

>  It was asked during the community concall whether the below may be related
> to ticket #2722 https://svn.open-mpi.org/trac/ompi/ticket/2722?
>
> --td
>
> On 04/04/2011 10:17 PM, David Zhang wrote:
>
> Any error messages?  Maybe the nodes ran out of memory?  I know MPI
> implement some kind of buffering under the hood, so even though you're
> sending array's over 2^26 in size, it may require more than that for MPI to
> actually send it.
>
> On Mon, Apr 4, 2011 at 2:16 PM, Michael Di Domenico <
> mdidomeni...@gmail.com> wrote:
>
>> Has anyone seen an issue where OpenMPI/Infiniband hangs when sending
>> messages over 2^26 in size?
>>
>> For a reason i have not determined just yet machines on my cluster
>> (OpenMPI v1.5 and Qlogic Stack/QDR IB Adapters) is failing to send
>> array's over 2^26 in size via the AllToAll collective. (user code)
>>
>> Further testing seems to indicate that an MPI message over 2^26 fails
>> (tested with IMB-MPI)
>>
>> Running the same test on a different older IB connected cluster seems
>> to work, which would seem to indicate a problem with the infiniband
>> drivers of some sort rather then openmpi (but i'm not sure).
>>
>> Any thoughts, directions, or tests?
>> ___
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>
>
>
> --
> David Zhang
> University of California, San Diego
>
>
> ___
> users mailing 
> listusers@open-mpi.orghttp://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
>
> --
> [image: Oracle]
> Terry D. Dontje | Principal Software Engineer
> Developer Tools Engineering | +1.781.442.2631
>  Oracle * - Performance Technologies*
>  95 Network Drive, Burlington, MA 01803
> Email terry.don...@oracle.com
>
>
>
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>

#include 
#include 
#include 
#include 
#include 

typedef signed char	int8;
typedef unsigned char	uint8;
typedef short		int16;
typedef unsigned short	uint16;
typedef int		int32;
typedef unsigned int	uint32;
typedef long		int64;
typedef unsigned long	uint64;

#define	I64(c)		(c##L)
#define	UI64(c)		(c##uL)

#define	_BR_RUNUP_	128
#define	_BR_LG_TABSZ_	7
#define	_BR_TABSZ_	(I64(1) << _BR_LG_TABSZ_)

#define	_ZERO64		UI64(0x0)

#define	_maskl(x)	(((x) == 0) ? _ZERO64 : ((~_ZERO64) << (64-(x
#define	_maskr(x)	(((x) == 0) ? _ZERO64 : ((~_ZERO64) >> (64-(x

#define	_BR_64STEP_(H,L,A,B) {\
	uint64	x;\
	x = H ^ (H << A) ^ (L >> (64 - A));\
	H = L | (x >> (B - 64));\
	L = x << (128 - B);\
}

static uint64_t _rtc()
{
	unsigned hi, lo, tmp;
	asm volatile ("rdtsc" : "=a" (lo), "=d" (hi));
	return (uint64_t)hi << 32 | lo;
}

typedef struct
{
	uint64	hi, lo, ind;
	uint64	tab[_BR_TABSZ_];
} brand_t;

static uint64 brand (brand_t *p)
{
	uint64	hi=p->hi, lo=p->lo, i=p->ind, ret;
	
	ret = p->tab[i];

	// 64-step a primitive trinomial LRS:  0, 45, 118   
	_BR_64STEP_(hi,lo,45,118);

	p->tab[i] = ret + hi;
	p->hi  = hi;
	p->lo  = lo;
	p->ind = hi & _maskr(_BR_LG_TABSZ_);

	return ret;
}

static void brand_init (brand_t *p, uint64 val)
{
	int64	i;
	uint64	hi, lo;

	hi = UI64(0x9ccae22ed2c6e578) ^ val;
	lo = UI64(0xce4db5d70739bd22) & _maskl(118-64);

	// we 64-step 0, 33, 118 in the initialization   
	for (i = 0; i < 64; i++)
		_BR_64STEP_(hi,lo,33,118);

	for (i = 0; i < _BR_TABSZ_; i++) {
		_BR_64STEP_(hi,lo,33,118);
		p->tab[i] = hi;
	}
	p->ind = _BR_TABSZ_/2;
	p->hi  = hi;
	p->lo  = lo;

	for (i = 0; i < _BR_RUNUP_; i++)
		brand(p);
}

void rubbish(brand_t* state, uint64_t n_words, uint64_t array[])
{
	uint64_t	i;

	for (i = 0; i < n_words; i++)
		array[i] = brand(state);
}

void usage(const char* prog)
{
	int	me;

	MPI_Comm_rank(MPI_COMM_WORLD, );
	if (me == 0)
		fprintf(stderr, "usage: %s #bytes/process\n", prog);

	exit(2);
}

int main(int argc, char* argv[])
{
	brand_t		state;
	int		i_proc, n_procs, words_per_chunk, loop;
	size_t		array_size;
	uint64_t*	source;
	uint64_t*	dest;

	MPI_Init(, );
	MPI_Comm_size(MPI_COMM_WORLD, _procs);
	MPI_Comm_rank(MPI_COMM_WORLD, _proc);
	MPI_Datatype uint64_type = MPI_LONG;
	MPI_Aint foo = 0;
	MPI_Type_extent(uint64_type, );

	assert(foo == 8);

	if (argc < 2)
		usage(argv[0]);

	array_size = strtoull(argv[1], NULL, 0);
	words_per_chunk = (array_size >> 3) / n_procs;

	if (i_proc == 0)
		printf("array size is 0x%zx bytes (%g)\n", array_size,

	(double)array_size);
	source = malloc(array_size);
	dest = malloc(array_size);
	
	if (source == NULL || dest == NULL) {
		fprintf(stderr, "process %d out of memory\n", i_proc);
		exit(1);
	}

	brand_init(, i_proc+1);
	rubbish(, array_size >> 3, source);
	

[OMPI users] OpenMPI 1.4.2 Hangs When Using More Than 1 Proc

2011-04-11 Thread Stergiou, Jonathan C CIV NSWCCD West Bethesda, 6640
I am running OpenMPI 1.4.2 under RHEL 5.5.  After install, I tested with 
"mpirun -np 4 date"; the command returned four "date" outputs. 

Then I tried running two different MPI programs, "geminimpi" and "salinas".  
Both run correctly with "mpirun -np 1 $prog".  However, both hang indefinitely 
when I use anything other than "-np 1".  

Next, I ran "mpirun --debug-daemons -np 1 geminimpi" and got the following:  
(this looks good, and is what I would expect)

[code]
[xxx@XXX_TUX01 ~]$ mpirun --debug-daemons -np 1 geminimpi
[XXX_TUX01:06558] [[15027,0],0] node[0].name XXX_TUX01 daemon 0 arch ffc91200
[XXX_TUX01:06558] [[15027,0],0] orted_cmd: received add_local_procs
[XXX_TUX01:06558] [[15027,0],0] orted_recv: received sync+nidmap from local 
proc [[15027,1],0]
[XXX_TUX01:06558] [[15027,0],0] orted_cmd: received collective data cmd
[XXX_TUX01:06558] [[15027,0],0] orted_cmd: received message_local_procs
[XXX_TUX01:06558] [[15027,0],0] orted_cmd: received collective data cmd
[XXX_TUX01:06558] [[15027,0],0] orted_cmd: received message_local_procs
Fluid Proc Ready: ID, FluidMaster,LagMaster = 001
 Checking license for Gemini
 Checking license for Linux OS
 Checking internal license list
 License valid

 GEMINI Startup
 Gemini +++ Version 5.1.00  20110501 +++

 + ERROR MESSAGE +
 FILE MISSING (Input): name = gemini.inp
[XXX_TUX01:06558] [[15027,0],0] orted_cmd: received waitpid_fired cmd
[XXX_TUX01:06558] [[15027,0],0] orted_cmd: received iof_complete cmd
--
mpirun has exited due to process rank 0 with PID 6559 on
node XXX_TUX01 exiting without calling "finalize". This may
have caused other processes in the application to be
terminated by signals sent by mpirun (as reported here).
--
[XXX_TUX01:06558] [[15027,0],0] orted_cmd: received exit
[/code]

With "mpirun --debug-daemons -np 2 geminimpi", it hangs like this: (hangs 
indefinitely)

[code]
[xxx@XXX_TUX01 ~]$ mpirun --debug-daemons -np 2 geminimpi
[XXX_TUX01:06570] [[14983,0],0] node[0].name XXX_TUX01 daemon 0 arch ffc91200
[XXX_TUX01:06570] [[14983,0],0] orted_cmd: received add_local_procs
[XXX_TUX01:06570] [[14983,0],0] orted_recv: received sync+nidmap from local 
proc [[14983,1],1]
[XXX_TUX01:06570] [[14983,0],0] orted_recv: received sync+nidmap from local 
proc [[14983,1],0]
[XXX_TUX01:06570] [[14983,0],0] orted_cmd: received collective data cmd
[XXX_TUX01:06570] [[14983,0],0] orted_cmd: received collective data cmd
[XXX_TUX01:06570] [[14983,0],0] orted_cmd: received message_local_procs
[/code]


I cloned my entire installation to a number of other machines to test.  On all 
the other workstations, everything behaves correctly and various regression 
suites return good results. 

Any ideas? 

--
Jon Stergiou
Engineer
NSWC Carderock




[OMPI users] "Value out of bounds in file" error

2011-04-11 Thread hi
Hi,

For the attached test program (reference:
http://www.netlib.org/blacs/BLACS/Examples.html#HELLO), I am observing
following message and binary halts...

[:03864] [[16991,0],0] ORTE_ERROR_LOG: Value out of bounds in
file ..\..\orte\mca\oob\tcp\oob_tcp.c at line 1193


*Command used to compile is as follow:
1) ifort blacs_ex01.f -c -I"C:\Program Files
(x86)\Intel\Compiler\11.1\065\mkl/include"

2) link /SUBSYSTEM:CONSOLE -NODEFAULTLIB /INCREMENTAL:NO /MAP
/MAPINFO:EXPORTS /DEBUG -opt:ref,icf /out:blacs_ex01.exe blacs_ex01.obj
/LIBPATH:"C:/Program Files (x86)/Intel/Compiler/11.1/065\\Lib\ia32"
libirc.lib libifcorert.lib /LIBPATH:"C:\Program Files
(x86)\Intel\Compiler\11.1\065\lib\ia32" ifqwin.lib /LIBPATH:"C:\Program
Files (x86)\Intel\Compiler\11.1\065\mkl/ia32/lib" mkl_blacs_intelmpi.lib
mkl_core.lib mkl_core_dll.lib /LIBPATH:"C:/Program Files
(x86)/Intel/Compiler/11.1/065/mkl\\ia32\\lib" mkl_lapack95.lib
mkl_core_dll.lib mkl_intel_c_dll.lib mkl_intel_thread_dll.lib
/LIBPATH:"C:/Program Files (x86)/Intel/Compiler/11.1/065/mkl/ia32/lib/"
mkl_scalapack_core.lib mkl_blacs_dll.lib
/LIBPATH:"C:\openmpi-1.5.2\installed_vs2008\lib" libmpi_f77d.lib libmpid.lib
libopen-pald.lib libopen-rted.lib /LIBPATH:"C:\Program Files\Microsoft
SDKs\Windows\v6.1\\\Lib" /LIBPATH:"C:/Program Files (x86)/Microsoft Visual
Studio 9.0/VC\\Lib" msvcrt.lib msvcprt.lib kernel32.lib

* Environment:
OS: Windows 7 (64-bit)
Compiler: cl.exe (32-bit) and ifort (32-bit)
MPI: openmpi-1.5.2 (local build)

Is it the case that I am combining mkl_intelmpi with locally built openmpi
and hence this problem???

Thank you in advance.
-Hiral


blacs_ex01.f
Description: Binary data