with the help of the great folks at myricom, we've been able to find a couple of performance limitations and fix a few bugs. we were able to transfer aoe at >1GB/s between two modest (3 year old) 1.86ghz intel 5000 xeons. contrib quanstro/etherm10g
this patch uses the vmappat patch which is now out-of-date. a redo is available at /n/sources/contrib/quanstro/patredux. this patch is not required but helps if you are using "small" packets. by small, i mean < 8k. to remove, just turn vmappat to vmap without the 3d arg. since we are not using msi interrupts, the coal parameter (interrupt coalesing) is fairly sensitve. this is because the dance even mp interrupts need to do takes many cycles. it's amazing how many apics, i/o apics and other nonsense is involved on a modern pcie machine. i have used values from 15-50µs with the current code. ymmv. - erik
