"john t" <[EMAIL PROTECTED]> wrote on 10/05/2006 08:18:31 AM:
> Hi Bernard,
>
> I had a configuration issue. I fixed it and now I get same BW (i.e.
> around 10 Gb/sec) on each port provided I use ports on different HCA
> cards. If I use two ports of the same HCA card then BW gets divided
> between these two ports. I am using Mellanox HCA cards and doing
> simple send/recv using uverbs.
>
> Do you think it could be an issue with Mallanox driver or could it
> be due to system/PCI-E limitation.
I haven't looked closely at the Mellanox driver, but I suspect it is there. When we were looking at the EHCA driver we found that if you have two links using the same event queue ( one per adapter ), and if the completion queue handling is serialized off the one interrupt per adapter, then one and two links get the same aggregate bandwidth. The problem is if you have two completion queues for the two links, they are handled serially. You need separate threads off the interrupt, each processing one of the completion queues in parallel on different CPU's to get the scaling you expect for 2 links.
I don't think it is the PCI-e bus because it can handle much more than 20 Gb/s. However, I also don't know how the Mellanox chip works for 2 ports.
>
> Regards,
> John T.
>
Bernie King-Smith
IBM Corporation
Server Group
Cluster System Performance
[EMAIL PROTECTED] (845)433-8483
Tie. 293-8483 or wombat2 on NOTES
"We are not responsible for the world we are born into, only for the world we leave when we die.
So we have to accept what has gone before us and work to change the only thing we can,
-- The Future." William Shatner
_______________________________________________ openib-general mailing list [email protected] http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
