> I'm trying out FI_MORE with the verbs;ofi_rxm provider, but I'm not seeing any
> performance benefit from it. ...
The verbs provider ignores the FI_MORE flag. With FI_MORE, you would lose
overlapping the transfer of the first write with the posting of the next, with
a potential gain of limiting PCI transactions. The GNI and EFA providers
support FI_MORE, and might have details of the benefits they see.
Is there a way to tell from the information returned by fi_getinfo() whether
one should expect any benefit from FI_MORE with the provider one is using? I
don't see anything, and none of the provider man pages seem to mention it, at
least in the v1.8.x branch I've built from. I'd prefer to avoid doing the
effective equivalent of 'if (!usingVerbsProvider) { /* do FI_MORE stuff */
...}', unless I absolutely have to. (I'm writing middleware, a multinode
communication implementation for the Chapel language runtime, which I want to
work over as wide a range of networks and providers as possible.)
I would expect to see a benefit batching writes, and waiting for completions at
the end, versus delaying each write until the previous one completed. If
you're not seeing this, can you try decreasing the size of the writes and see
if the results are unchanged?
Sure can, although there will be a delay because right now the initiator for my
fi_writemsg() calls seems to be concluding that all of them have arrived at the
target before that's actually happened, so I'll have to fix that first. I
don't know if that's because I'm mis-counting transactions (doesn't look like
it) or have misunderstood the completion semantics. I'm throwing
FI_DELIVERY_COMPLETE|FI_COMPLETION in my hints->tx_attr->op_flags, and believe
that means that once the initiator sees the CQ event from a fi_writemsg() the
processor on the target side must see the written bytes if it loads from the
target memory. (In some cases it's seeing the old memory contents.)
greg
_______________________________________________
ofiwg mailing list
[email protected]
https://lists.openfabrics.org/mailman/listinfo/ofiwg
