Hi Anatoly,
The OpenSM that you sent has many different SM executions, so I'm
not sure what execution should I refer to. However, I see a lot of
the following errors:
[23:28:03:918][152C] -> osm_vendor_send: ERR 3B28: Unable to create address
vector (IB_INSUFFICIENT_MEMORY).
[23:28:03:918][152C] -> __osm_pr_rcv_respond: ERR 1F15: Unable to send MAD
(IB_INSUFFICIENT_MEMORY)
Is there a problem with memory on the host?
This would explain why the host freezes.
I'm just trying to understand whether the problem is in OpenSM
(in which case, as we already mentioned, the right solution
would be updating/upgrading WinSM), or in the specific setup.
> usage of OpenSM goes to ~30MB, cpu usage to 100% and eventually
> the node freezes and needs to be reset.
30MB is peanuts. Who is eating all the memory?
Do these MPI jobs run from the SM node too?
Also, how do the MPI tasks communicate? What do they use to perform
all-to-all path resolution? IPoIB? Just a thought:
(32 Computing nodes, Quadcore Dualsocket) = 2^5 MPI tasks in each
MPI job, which is 2^10 path resolution queries.
256 concurrent jobs make it 2^16 path resolution queries (~65K),
so if they all try to do path resolution w/o some kind of queries
aggregation/caching, it will keep SM busy with 100% CPU for some time.
-- Yevgeny
Anatoly Greenblatt wrote:
Hi,
Information I have so far:
32 Computing nodes, Quadcore Dualsocket
1 Head node, Quadcore Dualsocket 64GB RAM
1 Control node, Dualcore Dualsocket
3 edge switches
2 core switches
OpenSM running either on Head node or control node, in both cases the
system freezes when running over 192 concurrent jobs.
They need to run 256 MPI concurrent jobs continuously.
Regards,
Anatoly.
-----Original Message-----
From: Yevgeny Kliteynik [mailto:[EMAIL PROTECTED]
Sent: Thursday, October 02, 2008 22:54
To: Anatoly Greenblatt
Cc: [email protected]
Subject: Re: [ofw] OpenSM with HPC
Hi Anatoly,
I need more details:
Anatoly Greenblatt wrote:
Hi,
Our client reported problems running over 192 concurrent jobs with
OpenSM.
What kind of cluster does your client have?
How many hosts? How many switches?
What do these jobs do?
Are these MPI jobs? Do they use/create multicast groups? Something else?
How many processes each job has?
The jobs are executed several times. After a while the memory
usage of OpenSM goes to ~30MB, cpu usage to 100% and eventually the
node
freezes and needs to be reset.
Is the problem reproducible?
Can you send me SM log?
-- Yevgeny
Configuration:
Winof rev 1596 (~rc1)
ConnectX HCA
Windows 2008 x64 with HPC pack rc2
NetworkDirect is installed
OpenSM is running as a service on the head node.
About a hundred nodes are used (maybe more, I don't have exact number
yet)
Has anyone any thoughts about this?
Thanks,
Anatoly.
------------------------------------------------------------------------
_______________________________________________
ofw mailing list
[email protected]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw
_______________________________________________
ofw mailing list
[email protected]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw
