Re: yet again the atomic operations
On Tue, 2010-08-10 at 04:46 -0700, Rui Machado wrote: > > There are two kinds supported. QLogic's driver does them in > > the host driver so they are atomic with respect to all the CPUs > > in the host. > > I'm just curious about this: how does this work? Is the CPU getting > interrupted and doing the operation while the Mellanox HCA does > everything in hardware? Yes. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: yet again the atomic operations
> > You can work around this by creating a loopback connection (ie an RC > connection from the local HCA to itself) and post atomic operations to > that QP instead of accessing the memory directly with the CPU. > Right but that's really slow, specially if you're implementing some sort of synchronization scheme that happens really often. Isn't it possible to do something such as interrupting the CPU and have it do the operation? For an atomic operation (small data) maybe the overhead is not that large?! Cheers, Rui -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: yet again the atomic operations
> There are two kinds supported. QLogic's driver does them in > the host driver so they are atomic with respect to all the CPUs > in the host. I'm just curious about this: how does this work? Is the CPU getting interrupted and doing the operation while the Mellanox HCA does everything in hardware? -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: yet again the atomic operations
On Fri, 2010-08-06 at 04:43 -0700, Rui Machado wrote: > Hi there, > > > There are two kinds supported. QLogic's driver does them in > > the host driver so they are atomic with respect to all the CPUs > > in the host. Mellanox uses HCA wide atomic which means the > > HCA will do a memory read/write without allowing other reads > > or writes from different QP operations passing through that > > HCA to get in between. The CPUs on the host won't see > > atomic operations since from their perspective, it looks > > like a normal read and write from the PCIe bus. > > So if the CPU writes/reads to/from the same address, even atomically > (lock), there might be room for some inconsistency on the values? It > is not really atomic from the whole system point of view, just for the > HCA? If so, is there any possibility to make the whole operation > 'system-wide' atomic? Correct. It won't be consistent from the HCA's point of view if other HCAs or CPUs are modifying the memory - even if they do it atomically. It is only consistent if a single HCA is doing atomic ops to the memory. There is no possibility to change this unless PCIe atomic operations are used by the HCA and if the root complex supports atomic operations. I don't know of any HCAs or root complex chips which have this support yet. > > You can see what type the HCA supports with "ibv_devinfo -v" > > and look for "atomic_cap: ATOMIC_HCA (1)" or > > "atomic_cap: ATOMIC_GLOB (2)". > > ATOMIC_HCA (1) is what I see in my Mellanox hardware. This is the case > you mentioned, "without allowing other reads or writes from different > QP operations passing through that HCA to get in between" > ATOMIC_GLOB (2) means with respect to all HCAs and even the CPU? Correct. > Cheers, > Rui > -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: yet again the atomic operations
> So if the CPU writes/reads to/from the same address, even atomically > (lock), there might be room for some inconsistency on the values? It > is not really atomic from the whole system point of view, just for the > HCA? If so, is there any possibility to make the whole operation > 'system-wide' atomic? PCI does not have any capability for atomic operations until PCI Express 3.0 (not available in any real devices yet). So any current HCA performing atomic operations across a PCI bus will always have to do read-modify-write which leaves a window for the CPU to mess things up if it accesses the same location. You can work around this by creating a loopback connection (ie an RC connection from the local HCA to itself) and post atomic operations to that QP instead of accessing the memory directly with the CPU. - R. -- Roland Dreier || For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/index.html -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: yet again the atomic operations
Hi there, > There are two kinds supported. QLogic's driver does them in > the host driver so they are atomic with respect to all the CPUs > in the host. Mellanox uses HCA wide atomic which means the > HCA will do a memory read/write without allowing other reads > or writes from different QP operations passing through that > HCA to get in between. The CPUs on the host won't see > atomic operations since from their perspective, it looks > like a normal read and write from the PCIe bus. So if the CPU writes/reads to/from the same address, even atomically (lock), there might be room for some inconsistency on the values? It is not really atomic from the whole system point of view, just for the HCA? If so, is there any possibility to make the whole operation 'system-wide' atomic? > You can see what type the HCA supports with "ibv_devinfo -v" > and look for "atomic_cap: ATOMIC_HCA (1)" or > "atomic_cap: ATOMIC_GLOB (2)". ATOMIC_HCA (1) is what I see in my Mellanox hardware. This is the case you mentioned, "without allowing other reads or writes from different QP operations passing through that HCA to get in between" ATOMIC_GLOB (2) means with respect to all HCAs and even the CPU? Cheers, Rui -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: yet again the atomic operations
Atomic ops are optional. Mellanox and QLogic HCAs support them, I don't know about the other HCAs. There are two kinds supported. QLogic's driver does them in the host driver so they are atomic with respect to all the CPUs in the host. Mellanox uses HCA wide atomic which means the HCA will do a memory read/write without allowing other reads or writes from different QP operations passing through that HCA to get in between. The CPUs on the host won't see atomic operations since from their perspective, it looks like a normal read and write from the PCIe bus. You can see what type the HCA supports with "ibv_devinfo -v" and look for "atomic_cap: ATOMIC_HCA (1)" or "atomic_cap: ATOMIC_GLOB (2)". On Wed, 2010-08-04 at 05:32 -0700, Rui Machado wrote: > Hi all, > > I would like to know how do the IB atomic operations work and how much > can one rely on them? Particularly, how does the interaction with the > CPU takes place when for example the CPU is referencing and possibly > modifying the same address, atomically . Does one need to take care of > memory say, with memory barriers? > Does anyone has experience with this? There must be also different > hardware support from the vendors, right? > > I hope I'm not sounding too vague, I'm not such a kernel or hw guy. I > tried to look for information on this but found pretty much nothing. > Can the experts shed some light on the problem? > > Cheers, > Rui > -- > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html