Garrett,
Garrett D'Amore wrote:
> I've been thinking about hardware that has multiple transmit rings ("tx
> resources").
>
> We really should have a way to expose this up to the stack. And
> ideally, the stack should guarantee that a given flow will always be
> sent down using the same hardware tx resource.
>
> I've heard that crossbow will deliver this, but I can't find evidence of
> it in the crossbow gate. Am I missing something? Is it functionality
> yet to be added, or is it not planned?
Its designed in but code is yet to make it in Crossbow gate. I think
parts of it are sitting in Roamer and Gopi's workspaces.
> The other problem I've heard from PAE, which is that one potential
> approach drivers could use today, which is to map the flow by hashing
> the sending CPU (which one would expect not to change for a given flow)
> is doomed to suffer packet reordering. Apparently the problem is that
> application threads can get get bounced around between CPUs by the
> scheduler pretty freely (more so than one would thing), and the result
> is that you can't assume that the sending CPU will be reasonably static
> for a given flow. (I gotta think this wreaks havoc on the caches
> involved... but that's a different problem.)
>
> _If_ transmitted packets are sent to the stack and always land in a
> delivery queue, then perhaps the outbound queue (squeue?) can have a
> worker thread that doesn't migrate around. But in order for that to
> happen, we have to stop having sending threads deliver right to the
> driver driver when intervening queues are empty.
This doesn't really apply to forwarding traffic and in case of traffic
terminating on the host, the application thread very rarely is able to
reach the driver directly (its about 17-18% of the time on web
workloads). The times it does means that there was nothing else to do
anyway and its better to let the thread go through instead of doing
a context switch.
> I _think_ this will work better for throughput. It may hurt latency
> slightly though. I haven't measured the latencies involved with queuing
> as opposed to direct delivery through the driver's xxx_send/xxx_start
> routine, but I'd be curious to know if others here have.
Yes, you are discussing FireEngine design here. The ARC case has a
detailed document which discusses all these things. Can't remember the
case number but search for FireEngine.
Cheers,
Sunay
>
> Anyway, let me know your thoughts.
>
> -- Garrett
>
> _______________________________________________
> crossbow-discuss mailing list
> crossbow-discuss at opensolaris.org
> http://opensolaris.org/mailman/listinfo/crossbow-discuss
--
Sunay Tripathi
Distinguished Engineer
Solaris Core Operating System
Sun MicroSystems Inc.
Solaris Networking: http://www.opensolaris.org/os/community/networking
Project Crossbow: http://www.opensolaris.org/os/project/crossbow
