> From: Roland Dreier [mailto:[EMAIL PROTECTED] > Sent: Tuesday, October 11, 2005 8:39 PM > > On mainstream architectures, it turns out that we can get away with > violating this rule. However, on non-cache-coherent architectures > like PowerPC 4xx, dma_map_single(..., DMA_TO_DEVICE) does a cache > flush, which makes sure that the contents of the CPU's cache are > really written to memory. If a driver then changes the contents of > the buffer after the call to dma_map_single(), then it's quite likely > that the change will be made only in the CPU's cache and the device > will end up DMA-ing the old data. > > The problem I hit is in ib_post_send_mad(), specifically: > > smp = (struct ib_smp *)send_wr->wr.ud.mad_hdr; > if (smp->mgmt_class == IB_MGMT_CLASS_SUBN_DIRECTED_ROUTE) { > ret = handle_outgoing_dr_smp(mad_agent_priv, smp, > send_wr); > > basically, when the MAD layer goes to send a directed route reply, it > changes the MAD buffer after the DMA mapping is done. The HCA > doesn't see the change, the wrong packet gets sent and the SM never > sees replies to its queries. > > Adding a PPC-specific cache flush call after the call to > handle_outgoing_dr_smp() fixes things to the point that the port can > be brought to ACTIVE, and in fact IPoIB works as well. However, this > is just a cludge -- the real fix will need to be more invasive. It > seems that the whole interface to the MAD layer may need to be > reorganized to avoid doing this.
Why not just use inline sends for the special QPs and remove the need to perform any DMA mappings on the send side altogether? - Fab _______________________________________________ openib-general mailing list [email protected] http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
