Hi Ethan,
Thanks for sharing your overview on multi-threading, which is quite
helpful for us to understand the big picture. It is fairly reasonable
to ensure correctness first and then optimizing it.
On Wednesday, December 04, 2013 4:55 AM, Ethan Jackson wrote:
>
> Specifically we're going to look at structural changes to
> further improve the performance and it's ability so scale to multiple
> threads. Also we'll invest in improving the efficiency of each
> thread, possibly by implementing something like RCU.
It is great that you are considering lock-free optimizations.
Apart of that, with regard to structural changes for salability, I would
still prefer a structure without the centralized dispatcher. But as
also mentioned by Alex, this is not urgent.
>
> With regards to the fmbs. These are a temporary solution which
> clearly aren't ideally. I'm working on some patches which should be
> ready this week which remove them entirely, and instead handle flow
> installation from the miss handler threads directly. Obviously, this
> is going to have a significant impact on the performance
> characteristics of the switch, so it may be worth watching out for
> these patches to be merged.
This is great!
>
> There's a tradeoff between CPU utilization and latency.
I cannot agree more on your view on this. And you are so right that
my patch performs much worse under TCP_CRR test. I agree that it's not a
problem for a dedicated TOR switch to have high CPU as long as it helps on
throughput. So our goal is to reduce the unnecessary contention which is
especially sensitive in our many core hosts as shown by profile from
Chengyuan that ticket spinlocks introduced by futex calls occupied the
CPU.
> Also, as the
> system becomes more loaded, batching will happen naturally anyways so
> the cost of this system is relatively low when under load.
>
I am kind of worried about this point, because in our testing we
observed that when system is loaded even with ~90% CPU for
dispatcher thread, it cannot handle upcalls in batch in most cases.
Here is the data collected by Chengyuan:
n_upcalls total 1076606
n_upcalls in a batch > 5: 4537
n_upcalls in a batch > 10: 825
n_upcalls in a batch > 20: 103
n_upcalls in a batch >= 50: 0
Less than 5% upcalls were handled in a batch bigger than 5, and
the MAX_BATCH number 50 was never hit.
When we increased hping speed above this threshold batch handling
did increase a lot but fastpath already started dropping packets because
dispatcher was now bottleneck and cannot handle fast enough:
lookups: hit:165917 missed:8042062 lost:168576
This means batching will not happen naturally: it is either no batching,
or batching but overloaded.
Best regards,
Han
_______________________________________________
discuss mailing list
discuss@openvswitch.org
http://openvswitch.org/mailman/listinfo/discuss
