Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers

[Jesus Sanchez-Palencia]

Hi,

On 10/19/2017 01:39 PM, Richard Cochran wrote:
> On Wed, Oct 18, 2017 at 03:37:35PM -0700, Jesus Sanchez-Palencia wrote:
>> I also did some tests with when you don't set valid launch times, but here 
>> using
>> your idea from above, so with the driver calculating a valid launch time 
>> (i.e.
>> current NIC time + X ns, varying X across tests) for packets that didn't 
>> have it
>> set by the user, and I wasn't too happy with its reliability. It could
>> definitely be improved, but it has left me wondering: instead, what about
>> documenting that if you enable TXTIME, then you *must* provide a valid Launch
>> time for all packets on traffic classes that are affected?
> 
> If txtime is enabled, then CBS is pointless because the txtime already
> specifies the bandwidth implicitly.


Assuming there is no "interfering" traffic on that traffic class, yes.
Otherwise, CBS could be configured just to avoid that outbound traffic ever goes
beyond the reserved bandwidth.


> 
> The problem is when one program uses txtime and another uses CBS, then
> the CBS user will experience really wrong performance.


Good point. We'll need to adjust the launch time for controllers that behave
like the i210 then, imo.


Thanks,
Jesus


> 
> Thanks,
> Richard
> 

Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers

[Richard Cochran]

On Wed, Oct 18, 2017 at 03:37:35PM -0700, Jesus Sanchez-Palencia wrote:
> I also did some tests with when you don't set valid launch times, but here 
> using
> your idea from above, so with the driver calculating a valid launch time (i.e.
> current NIC time + X ns, varying X across tests) for packets that didn't have 
> it
> set by the user, and I wasn't too happy with its reliability. It could
> definitely be improved, but it has left me wondering: instead, what about
> documenting that if you enable TXTIME, then you *must* provide a valid Launch
> time for all packets on traffic classes that are affected?

If txtime is enabled, then CBS is pointless because the txtime already
specifies the bandwidth implicitly.

The problem is when one program uses txtime and another uses CBS, then
the CBS user will experience really wrong performance.

Thanks,
Richard

Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers

[Jesus Sanchez-Palencia]

Hi Richard,


On 09/19/2017 10:25 PM, Richard Cochran wrote:
(...)
>  
>> I have a question, what about a controller that doesn't provide a way to
>> set a per-packet transmission time, but it supports Qbv/Qbu. What would
>> be your proposal to configure it?
> 
> SO_TXTIME will have a generic SW fallback.
> 
> BTW, regarding the i210, there is no sensible way to configure both
> CBS and time based transmission at the same time.  The card performs a
> logical AND to make the launch decision.  The effect of this is that
> each and every packet needs a LaunchTime, and the driver would be
> forced to guess the time for a packet before entering it into its
> queue.
> 
> So if we end up merging CBS and SO_TXTIME, then we'll have to make
> them exclusive of each other (in the case of the i210) and manage the
> i210 queue configurations correctly.
> 

I've ran some quick tests here having launch time enabled on i210 + our cbs
patchset. When valid Launch times are set on each packet you still get the
expected behavior, so I'm not sure we should just make them exclusive of each 
other.

I also did some tests with when you don't set valid launch times, but here using
your idea from above, so with the driver calculating a valid launch time (i.e.
current NIC time + X ns, varying X across tests) for packets that didn't have it
set by the user, and I wasn't too happy with its reliability. It could
definitely be improved, but it has left me wondering: instead, what about
documenting that if you enable TXTIME, then you *must* provide a valid Launch
time for all packets on traffic classes that are affected?

With the SO_TXTIME qdisc idea in place, that could even be enforced before
packets were enqueued into the netdevice.


Regards,
Jesus

Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers

[Guedes, Andre]

Hi all,

On Mon, 2017-10-02 at 12:45 -0600, Levi Pearson wrote:
> Hi Rodney,
> 
> Some archives seem to have threaded it, but I have CC'd the
> participants I saw in the original discussion thread since they may
> not otherwise notice it amongst the normal traffic.
> 
> On Fri, Sep 29, 2017 at 2:44 PM, Rodney Cummings 
> wrote:

[...]

> > 1. Question: From an 802.1 perspective, is this RFC intended to support
> > end-station (e.g. NIC in host), bridges (i.e. DSA), or both?
> > 
> > This is very important to clarify, because the usage of this interface
> > will be very different for one or the other.
> > 
> > For a bridge, the user code typically represents a remote management
> > protocol (e.g. SNMP, NETCONF, RESTCONF), and this interface is
> > expected to align with the specifications of 802.1Q clause 12,
> > which serves as the information model for management. Historically,
> > a standard kernel interface for management hasn't been viewed as
> > essential, but I suppose it wouldn't hurt.
> 
> I don't think the proposal was meant to cover the case of non-local
> switch hardware, but in addition to dsa and switchdev switch ICs
> managed by embedded Linux-running SoCs, there are SoCs with embedded
> small port count switches or even plain multiple NICs with software
> bridging. Many of these embedded small port count switches have FQTSS
> hardware that could potentially be configured by the proposed cbs
> qdisc. This blurs the line somewhat between what is a "bridge" and
> what is an "end-station" in 802.1Q terminology, but nevertheless these
> devices exist, sometimes acting as an endpoint + a real bridge and
> sometimes as just a system with multiple network interfaces.

During the development of this proposal, we were most focused on end-station
use-cases. We considered some bridge use-cases as well just to verify that the
proposed design wouldn't be an issue if someone else goes for it.

We agree that the line between end-station and bridge can be a bit blurred (in
this case). Even though we designed this interface with end-station use-cases
in mind, if the proposed infrastructure could be used as is in bridge use-
cases, good.

> > For an end station, the user code can be an implementation of SRP
> > (802.1Q clause 35), or it can be an application-specific
> > protocol (e.g. industrial fieldbus) that exchanges data according
> > to P802.1Qcc clause 46. Either way, the top-level user interface
> > is designed for individual streams, not queues and shapers. That
> > implies some translation code between that top-level interface
> > and this sort of kernel interface.

Yes, you're right. Our understanding is that the top-level interfaces should be
implemented at user space as well as any stream management functionality. The
idea here is to keep the kernel-side as simple as possible. The kernel handles
hardware configuration (via Traffic Control interface) while the user space
handles TSN streams i.e. the kernel provides the mechanism and the user space
provides the policy.

> > As a specific end-station example, for CBS, 802.1Q-2014 subclause
> > 34.6.1 requires "per-stream queues" in the Talker end-station.
> > I don't see 34.6.1 represented in the proposed RFC, but that's
> > okay... maybe per-stream queues are implemented in user code.
> > Nevertheless, if that is the assumption, I think we need to
> > clarify, especially in examples.
> 
> You're correct that the FQTSS credit-based shaping algorithm requires
> per-stream shaping by Talker endpoints as well, but this is in
> addition to the per-class shaping provided by most hardware shaping
> implementations that I'm aware of in endpoint network hardware. I
> agree that we need to document the need to provide this, but it can
> definitely be built on top of the current proposal.
> 
> I believe the per-stream shaping could be managed either by a user
> space application that manages all use of a streaming traffic class,
> or through an additional qdisc module that performs per-stream
> management on top of the proposed cbs qdisc, ensuring that the
> frames-per-observation interval aspect of each stream's reservation is
> obeyed. This becomes a fairly simple qdisc to implement on top of a
> per-traffic class shaper, and could even be implemented with the help
> of the timestamp that the SO_TXTIME proposal adds to skbuffs, but I
> think keeping the layers separate provides more flexibility to
> implementations and keeps management of various kinds of hardware
> offload support simpler as well.

Indeed, 'per-stream queue' is not covered in this RFC. For now, we expect it to
be implemented in user code. We believe the proposed CBS qdisc could be
extended to support a full software-based implementation which would be used to
implement 'per-stream queue' support. This functionality should be addressed by
a separated series.

Anyways, we're about to send the v3 patchset implementing this proposal and
we'll make it clear.

> > 2. 

RE: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers

[Rodney Cummings]

Thanks Levi,

Great discussion. It sounds like we're aligned.

Qbv is essential in switches. My concern was the end-station side only.

You're absolutely right that i210 LaunchTime doesn't do everything that is 
needed 
for scheduling. It still requires quite a bit of frame-juggling to ensure that
transmit descriptors are ordered by time, and that the control system's cyclic
traffic is repeated as expected. Like you, I'm not sure where that code is best
located in the long run.

For a hardware design that connects an FPGA to one of the internal ports of a
Qbv-capable switch, the preferable scheduling design can be done there.
The FPGA IP essentially implements a cyclic schedule of transmit descriptors,
each with its own buffer that can be written atomically by application code.
For a control system (i.e. industrial field-level or automotive control),
each buffer (stream) can hold a single frame. If you take a look at the
"Network Interface" specs for FlexRay in automotive, they went so far as to
mandate that design for a NIC. The IP for that sort of thing
doesn't take alot of gates or memory, so I can see why FlexRay went that way.

That sort of design wasn't destined to happen as part of Qbv, because the switch
vendors are worried about things like 64-port switches. Also, if we treat Qbv
as applying to switches only, FlexRay-style IP isn't needed. For the switch,
we only need it to "get out of the way", and Qbv is fine for that.

I may be naive and idealistic, but in the long run, I'm still hopeful that an
Ethernet NIC vendor will implement Flexray-style scheduling for end-station. 
We need a NIC trend-setter to do it as a differentiator, and the rest will 
probably follow along. That completely eliminates code in Linux for the 
frame-juggling. We'll need to provide a way to open a socket for a specific 
stream, but that doesn't seem too challenging.

Rodney




> -Original Message-
> From: Levi Pearson [mailto:levipear...@gmail.com]
> Sent: Monday, October 2, 2017 4:49 PM
> To: Rodney Cummings <rodney.cummi...@ni.com>
> Cc: Linux Kernel Network Developers <netdev@vger.kernel.org>; Vinicius
> Costa Gomes <vinicius.go...@intel.com>; Henrik Austad <hen...@austad.us>;
> richardcoch...@gmail.com; jesus.sanchez-palen...@intel.com;
> andre.gue...@intel.com
> Subject: Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for
> traffic shapers
> 
> Hi Rodney,
> 
> On Mon, Oct 2, 2017 at 1:40 PM, Rodney Cummings <rodney.cummi...@ni.com>
> wrote:
> 
> > It's a shame that someone built such hardware. Speaking as a
> manufacturer
> > of daisy-chainable products for industrial/automotive applications, I
> > wouldn't use that hardware in my products. The whole point of scheduling
> > is to obtain close-to-optimal network determinism. If the hardware
> > doesn't schedule properly, the product is probably better off using CBS.
> 
> I should note that the hardware I'm using was not designed to be a TSN
> endpoint. It just happens that the hardware, designed for other
> reasons, happens to have hardware support for Qbv and thus makes it an
> interesting and inexpensive platform on which to do Qbv experiments.
> Nevertheless, I believe the right combination of driver support for
> the hardware shaping and appropriate software in the network stack
> would allow it to schedule things fairly precisely in the
> application-centric manner you've described. I will ultimately defer
> to your judgement on that, though, and spend some time with the Qcc
> draft to see where I may be wrong.
> 
> > It would be great if all of the user-level application code was
> scheduled
> > as tightly as the network, but the reality is that we aren't there yet.
> > Also, even if the end-station has a separately timed RT Linux thread for
> > each stream, the order of those threads can vary. Therefore, when the
> > end-station (SoC) has more than one talker, the frames for each talker
> > can be written into the Qbv queue at any time, in any order.
> > There is no scheduling of frames (i.e. streams)... only the queue.
> 
> Frames are presented to *the kernel* in any order. The qdisc
> infrastructure provides mechanisms by which other scheduling policy
> can be added between presentation to the kernel and presentation to
> hardware queues.
> 
> Even with i210-style LaunchTime functionality, this must also be
> provided. The hardware cannot re-order time-scheduled frames, and
> reviewers of the SO_TXTIME proposal rejected the idea of having the
> driver try to re-order scheduled frames already submitted to the
> hardware-visible descriptor chains, and rightly so I think.
> 
> So far, I have thought of 3 places this ordering constraint might be
> provided:
> 
> 1. Built int

Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers

[Levi Pearson]

Hi Rodney,

On Mon, Oct 2, 2017 at 1:40 PM, Rodney Cummings  wrote:

> It's a shame that someone built such hardware. Speaking as a manufacturer
> of daisy-chainable products for industrial/automotive applications, I
> wouldn't use that hardware in my products. The whole point of scheduling
> is to obtain close-to-optimal network determinism. If the hardware
> doesn't schedule properly, the product is probably better off using CBS.

I should note that the hardware I'm using was not designed to be a TSN
endpoint. It just happens that the hardware, designed for other
reasons, happens to have hardware support for Qbv and thus makes it an
interesting and inexpensive platform on which to do Qbv experiments.
Nevertheless, I believe the right combination of driver support for
the hardware shaping and appropriate software in the network stack
would allow it to schedule things fairly precisely in the
application-centric manner you've described. I will ultimately defer
to your judgement on that, though, and spend some time with the Qcc
draft to see where I may be wrong.

> It would be great if all of the user-level application code was scheduled
> as tightly as the network, but the reality is that we aren't there yet.
> Also, even if the end-station has a separately timed RT Linux thread for
> each stream, the order of those threads can vary. Therefore, when the
> end-station (SoC) has more than one talker, the frames for each talker
> can be written into the Qbv queue at any time, in any order.
> There is no scheduling of frames (i.e. streams)... only the queue.

Frames are presented to *the kernel* in any order. The qdisc
infrastructure provides mechanisms by which other scheduling policy
can be added between presentation to the kernel and presentation to
hardware queues.

Even with i210-style LaunchTime functionality, this must also be
provided. The hardware cannot re-order time-scheduled frames, and
reviewers of the SO_TXTIME proposal rejected the idea of having the
driver try to re-order scheduled frames already submitted to the
hardware-visible descriptor chains, and rightly so I think.

So far, I have thought of 3 places this ordering constraint might be provided:

1. Built into each driver with offload support for SO_TXTIME
2. A qdisc module that will re-order scheduled skbuffs within a certain window
3. Left to userspace to coordinate

I think 1 is particularly bad due to duplication of code. 3 is the
easy default from the point of view of the kernel, but not attractive
from the point of view of precise scheduling and low latency to
egress. Unless someone has a better idea, I think 2 would be the most
appropriate choice. I can think of a number of ways one might be
designed, but I have not thought them through fully yet.

> From the perspective of the CNC, that means that a Qbv-only end-station is
> sloppy. In contrast, an end-station using per-stream scheduling
> (e.g. i210 with LaunchTime) is precise, because each frame has a known time.
> It's true that P802.1Qcc supports both, so the CNC might support both.
> Nevertheless, the CNC is likely to compute a network-wide schedule
> that optimizes for the per-stream end-stations, and locates the windows
> for the Qbv-only end-stations in a slower interval.

Because out-of-order submission of frames to LaunchTime hardware would
result in the frame that is enqueued later but scheduled earlier only
transmitting once the later-scheduled frame completes, enqueuing
frames immediately with LaunchTime information results in even *worse*
sloppiness if two applications don't always submit their frames in the
correct order to the kernel during a cycle. If your queue is simply
gated at the time of the first scheduled launch, the frame that should
have been first will only be late by the length of the queued-first
frame. If they both have precise launch times, the frame that should
have been first will have to wait until the normal launch time of the
later-scheduled frame plus the time it takes for it to egress!

In either case, re-ordering is required if it is not controlled for in
userspace. In other words, I am not disagreeing with you about the
insufficiency of a Qbv shaper for an endpoint! I am just pointing out
that neither hardware LaunchTime support nor hardware Qbv window
support are sufficient by themselves. Nor, for that case, is the CBS
offload support sufficient to meet multi-stream requirements from SRP.
All of these provide a low-layer mechanism by which appropriate
scheduling of a multi-stream or multi-application endpoint can be
enhanced, but none of them provide the complete story by themselves.

> I realize that we are only doing CBS for now. I guess my main point is that
> if/when we add scheduling, it cannot be limited to Qbv. Per-stream scheduling
> is essential, because that is the assumption for most CNC designs that
> I'm aware of.

I understand and completely agree. I have believed that per-stream
shaping/scheduling in 

RE: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers

[Rodney Cummings]

Thanks Levi,

> One particular device I am working with now provides all network
> access through a DSA switch chip with hardware Qbv support in addtion
> to hardware Qav support. The SoC attached to it has no hardware timed
> launch (SO_TXTIME) support. In this case, although the proposed
> interface for Qbv is not *sufficient* to make a working time-aware end
> station, it does provide a usable building block to provide one. As
> with the credit-based shaping system, Talkers must provide an
> additional level of per-stream shaping as well, but this is largely
> (absent the jitter calculations, which are sort of a middle-level
> concern) independent of what sort of hardware offload of the
> scheduling is provided.

It's a shame that someone built such hardware. Speaking as a manufacturer
of daisy-chainable products for industrial/automotive applications, I
wouldn't use that hardware in my products. The whole point of scheduling
is to obtain close-to-optimal network determinism. If the hardware
doesn't schedule properly, the product is probably better off using CBS.

It would be great if all of the user-level application code was scheduled
as tightly as the network, but the reality is that we aren't there yet.
Also, even if the end-station has a separately timed RT Linux thread for
each stream, the order of those threads can vary. Therefore, when the 
end-station (SoC) has more than one talker, the frames for each talker 
can be written into the Qbv queue at any time, in any order. 
There is no scheduling of frames (i.e. streams)... only the queue.

From the perspective of the CNC, that means that a Qbv-only end-station is
sloppy. In contrast, an end-station using per-stream scheduling 
(e.g. i210 with LaunchTime) is precise, because each frame has a known time.
It's true that P802.1Qcc supports both, so the CNC might support both.
Nevertheless, the CNC is likely to compute a network-wide schedule
that optimizes for the per-stream end-stations, and locates the windows
for the Qbv-only end-stations in a slower interval.

I realize that we are only doing CBS for now. I guess my main point is that
if/when we add scheduling, it cannot be limited to Qbv. Per-stream scheduling
is essential, because that is the assumption for most CNC designs that 
I'm aware of.

Rodney

> -Original Message-
> From: Levi Pearson [mailto:levipear...@gmail.com]
> Sent: Monday, October 2, 2017 1:46 PM
> To: Rodney Cummings <rodney.cummi...@ni.com>
> Cc: Linux Kernel Network Developers <netdev@vger.kernel.org>; Vinicius
> Costa Gomes <vinicius.go...@intel.com>; Henrik Austad <hen...@austad.us>;
> richardcoch...@gmail.com; jesus.sanchez-palen...@intel.com;
> andre.gue...@intel.com
> Subject: Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for
> traffic shapers
> 
> Hi Rodney,
> 
> Some archives seem to have threaded it, but I have CC'd the
> participants I saw in the original discussion thread since they may
> not otherwise notice it amongst the normal traffic.
> 
> On Fri, Sep 29, 2017 at 2:44 PM, Rodney Cummings <rodney.cummi...@ni.com>
> wrote:
> > Hi,
> >
> > I am posting my reply to this thread after subscribing, so I apologize
> > if the archive happens to attach it to the wrong thread.
> >
> > First, I'd like to say that I strongly support this RFC.
> > We need Linux interfaces for IEEE 802.1 TSN features.
> >
> > Although I haven't looked in detail, the proposal for CBS looks good.
> > My questions/concerns are more related to future work, such for 802.1Qbv
> > (scheduled traffic).
> >
> > 1. Question: From an 802.1 perspective, is this RFC intended to support
> > end-station (e.g. NIC in host), bridges (i.e. DSA), or both?
> >
> > This is very important to clarify, because the usage of this interface
> > will be very different for one or the other.
> >
> > For a bridge, the user code typically represents a remote management
> > protocol (e.g. SNMP, NETCONF, RESTCONF), and this interface is
> > expected to align with the specifications of 802.1Q clause 12,
> > which serves as the information model for management. Historically,
> > a standard kernel interface for management hasn't been viewed as
> > essential, but I suppose it wouldn't hurt.
> 
> I don't think the proposal was meant to cover the case of non-local
> switch hardware, but in addition to dsa and switchdev switch ICs
> managed by embedded Linux-running SoCs, there are SoCs with embedded
> small port count switches or even plain multiple NICs with software
> bridging. Many of these embedded small port count switches have FQTSS
> hardware that could potentially be configured by the proposed cbs
> qdisc. This blurs the line somewhat between what is a &q

Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers

[Levi Pearson]

Hi Rodney,

Some archives seem to have threaded it, but I have CC'd the
participants I saw in the original discussion thread since they may
not otherwise notice it amongst the normal traffic.

On Fri, Sep 29, 2017 at 2:44 PM, Rodney Cummings  wrote:
> Hi,
>
> I am posting my reply to this thread after subscribing, so I apologize
> if the archive happens to attach it to the wrong thread.
>
> First, I'd like to say that I strongly support this RFC.
> We need Linux interfaces for IEEE 802.1 TSN features.
>
> Although I haven't looked in detail, the proposal for CBS looks good.
> My questions/concerns are more related to future work, such for 802.1Qbv
> (scheduled traffic).
>
> 1. Question: From an 802.1 perspective, is this RFC intended to support
> end-station (e.g. NIC in host), bridges (i.e. DSA), or both?
>
> This is very important to clarify, because the usage of this interface
> will be very different for one or the other.
>
> For a bridge, the user code typically represents a remote management
> protocol (e.g. SNMP, NETCONF, RESTCONF), and this interface is
> expected to align with the specifications of 802.1Q clause 12,
> which serves as the information model for management. Historically,
> a standard kernel interface for management hasn't been viewed as
> essential, but I suppose it wouldn't hurt.

I don't think the proposal was meant to cover the case of non-local
switch hardware, but in addition to dsa and switchdev switch ICs
managed by embedded Linux-running SoCs, there are SoCs with embedded
small port count switches or even plain multiple NICs with software
bridging. Many of these embedded small port count switches have FQTSS
hardware that could potentially be configured by the proposed cbs
qdisc. This blurs the line somewhat between what is a "bridge" and
what is an "end-station" in 802.1Q terminology, but nevertheless these
devices exist, sometimes acting as an endpoint + a real bridge and
sometimes as just a system with multiple network interfaces.

> For an end station, the user code can be an implementation of SRP
> (802.1Q clause 35), or it can be an application-specific
> protocol (e.g. industrial fieldbus) that exchanges data according
> to P802.1Qcc clause 46. Either way, the top-level user interface
> is designed for individual streams, not queues and shapers. That
> implies some translation code between that top-level interface
> and this sort of kernel interface.
>
> As a specific end-station example, for CBS, 802.1Q-2014 subclause
> 34.6.1 requires "per-stream queues" in the Talker end-station.
> I don't see 34.6.1 represented in the proposed RFC, but that's
> okay... maybe per-stream queues are implemented in user code.
> Nevertheless, if that is the assumption, I think we need to
> clarify, especially in examples.

You're correct that the FQTSS credit-based shaping algorithm requires
per-stream shaping by Talker endpoints as well, but this is in
addition to the per-class shaping provided by most hardware shaping
implementations that I'm aware of in endpoint network hardware. I
agree that we need to document the need to provide this, but it can
definitely be built on top of the current proposal.

I believe the per-stream shaping could be managed either by a user
space application that manages all use of a streaming traffic class,
or through an additional qdisc module that performs per-stream
management on top of the proposed cbs qdisc, ensuring that the
frames-per-observation interval aspect of each stream's reservation is
obeyed. This becomes a fairly simple qdisc to implement on top of a
per-traffic class shaper, and could even be implemented with the help
of the timestamp that the SO_TXTIME proposal adds to skbuffs, but I
think keeping the layers separate provides more flexibility to
implementations and keeps management of various kinds of hardware
offload support simpler as well.

> 2. Suggestion: Do not assume that a time-aware (i.e. scheduled)
> end-station will always use 802.1Qbv.
>
> For those who are subscribed to the 802.1 mailing list,
> I'd suggest a read of draft P802.1Qcc/D1.6, subclause U.1
> of Annex U. Subclause U.1 assumes that bridges in the network use
> 802.1Qbv, and then it poses the question of what an end-station
> Talker should do. If the end-station also uses 802.1Qbv,
> and that end-station transmits multiple streams, 802.1Qbv is
> a bad implementation. The reason is that the scheduling
> (i.e. order in time) of each stream cannot be controlled, which
> in turn means that the CNC (network manager) cannot optimize
> the 802.1Qbv schedules in bridges. The preferred technique
> is to use "per-stream scheduling" in each Talker, so that
> the CNC can create an optimal schedules (i.e. best determinism).
>
> I'm aware of a small number of proprietary CNC implementations for
> 802.1Qbv in bridges, and they are generally assuming per-stream
> scheduling in end-stations (Talkers).
>
> The i210 NIC's LaunchTime can be used to implement 

Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers

[Rodney Cummings]

Hi,

I am posting my reply to this thread after subscribing, so I apologize
if the archive happens to attach it to the wrong thread.

First, I'd like to say that I strongly support this RFC. 
We need Linux interfaces for IEEE 802.1 TSN features.

Although I haven't looked in detail, the proposal for CBS looks good. 
My questions/concerns are more related to future work, such for 802.1Qbv
(scheduled traffic).

1. Question: From an 802.1 perspective, is this RFC intended to support 
end-station (e.g. NIC in host), bridges (i.e. DSA), or both?

This is very important to clarify, because the usage of this interface
will be very different for one or the other. 

For a bridge, the user code typically represents a remote management 
protocol (e.g. SNMP, NETCONF, RESTCONF), and this interface is 
expected to align with the specifications of 802.1Q clause 12, 
which serves as the information model for management. Historically,
a standard kernel interface for management hasn't been viewed as
essential, but I suppose it wouldn't hurt.

For an end station, the user code can be an implementation of SRP
(802.1Q clause 35), or it can be an application-specific 
protocol (e.g. industrial fieldbus) that exchanges data according
to P802.1Qcc clause 46. Either way, the top-level user interface
is designed for individual streams, not queues and shapers. That
implies some translation code between that top-level interface 
and this sort of kernel interface.

As a specific end-station example, for CBS, 802.1Q-2014 subclause 
34.6.1 requires "per-stream queues" in the Talker end-station.
I don't see 34.6.1 represented in the proposed RFC, but that's
okay... maybe per-stream queues are implemented in user code.
Nevertheless, if that is the assumption, I think we need to 
clarify, especially in examples.

2. Suggestion: Do not assume that a time-aware (i.e. scheduled) 
end-station will always use 802.1Qbv.

For those who are subscribed to the 802.1 mailing list, 
I'd suggest a read of draft P802.1Qcc/D1.6, subclause U.1 
of Annex U. Subclause U.1 assumes that bridges in the network use 
802.1Qbv, and then it poses the question of what an end-station
Talker should do. If the end-station also uses 802.1Qbv, 
and that end-station transmits multiple streams, 802.1Qbv is
a bad implementation. The reason is that the scheduling
(i.e. order in time) of each stream cannot be controlled, which
in turn means that the CNC (network manager) cannot optimize
the 802.1Qbv schedules in bridges. The preferred technique
is to use "per-stream scheduling" in each Talker, so that
the CNC can create an optimal schedules (i.e. best determinism).

I'm aware of a small number of proprietary CNC implementations for 
802.1Qbv in bridges, and they are generally assuming per-stream
scheduling in end-stations (Talkers).

The i210 NIC's LaunchTime can be used to implement per-stream 
scheduling. I haven't looked at SO_TXTIME in detail, but it sounds
like per-stream scheduling. If so, then we already have the
fundamental building blocks for a complete implementation
of a time-aware end-station.

If we answer the preceding question #1 as "end-station only",
I would recommend avoiding 802.1Qbv in this interface. There
isn't really anything wrong with it per-se, but it would lead
developers down the wrong path.

Rodney Cummings (National Instruments)
Editor, IEEE P802.1Qcc

---
Hi,

This patchset is an RFC on a proposal of how the Traffic Control subsystem can
be used to offload the configuration of traffic shapers into network devices
that provide support for them in HW. Our goal here is to start upstreaming
support for features related to the Time-Sensitive Networking (TSN) set of
standards into the kernel.

As part of this work, we've assessed previous public discussions related to TSN
enabling: patches from Henrik Austad (Cisco), the presentation from Eric Mann
at Linux Plumbers 2012, patches from Gangfeng Huang (National Instruments) and
the current state of the OpenAVNU project (https://github.com/AVnu/OpenAvnu/).

Please note that the patches provided as part of this RFC are implementing what
is needed only for 802.1Qav (FQTSS) only, but we'd like to take advantage of
this discussion and share our WIP ideas for the 802.1Qbv and 802.1Qbu interfaces
as well. The current patches are only providing support for HW offload of the
configs.


Overview


Time-sensitive Networking (TSN) is a set of standards that aim to address
resources availability for providing bandwidth reservation and bounded latency
on Ethernet based LANs. The proposal described here aims to cover mainly what is
needed to enable the following standards: 802.1Qat, 802.1Qav, 802.1Qbv and
802.1Qbu.

The initial target of this work is the Intel i210 NIC, but other controllers'
datasheet were also taken into account, like the Renesas RZ/A1H RZ/A1M group and
the Synopsis DesignWare Ethernet QoS controller.


Proposal


Feature-wise, what is covered here are configuration interfaces for 

Re: TSN Scorecard, was Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers

[Jesus Sanchez-Palencia]

Hi,


On 09/19/2017 10:49 PM, Richard Cochran wrote:
(...)

> 
> No, that is not what I meant.  We need some minimal additional kernel
> support in order to fully implement the TSN family of standards.  Of
> course, the bulk will have to be done in user space.  It would be a
> mistake to cram the stuff that belongs in userland into the kernel.
> 
> Looking at the table, and reading your descriptions of the state of
> OpenAVB, I remained convinced that the kernel needs only three
> additions:
> 
> 1. SO_TXTIME
> 2. CBS Qdisc
> 3. ALSA support for DAC clock control (but that is another story)


We'll be posting the CBS v1 series for review soon.

The current SO_TXTIME RFC for the purpose of Launchtime looks great, and we are
looking forward for the v1 + its companion qdisc so we can test / review and
provide feedback.

We are still under the impression that a config interface for HW offload of Qbv
/ Qbu config will be needed, but we'll be deferring the 'taprio' proposal until
there are NICs (end stations) that support these standards available. We can
revisit it if that ever happens, and if it's still needed, but then taking into
account SO_TXTIME (and its related qdisc).

Thanks everyone for all the feedback so far.

Regards,
Jesus

Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers

[Richard Cochran]

On Tue, Sep 19, 2017 at 05:19:18PM -0700, Vinicius Costa Gomes wrote:
> (I think LaunchTime is something specific to the i210, right?)

Levi just told us:

   Recent SoCs from NXP (the i.MX 6 SoloX, and all the i.MX 7 and 8
   parts) support Qav shaping as well as scheduled launch
   functionality;

Thanks,
Richard

Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers

[Richard Cochran]

On Tue, Sep 19, 2017 at 07:59:11PM -0600, levipear...@gmail.com wrote:
> If some endpoint device shows up with direct Qbv support, this interface would
> probably work well there too, although a talker would need to be able to
> schedule its transmits pretty precisely to achieve the lowest possible 
> latency.

This is an argument for SO_TXTIME.

> One concern here is calling the base-time parameter an interval; it's really
> an absolute time with respect to the PTP timescale. Good documentation will
> be important to this one, since the specification discusses some subtleties
> regarding the impact of different time values chosen here.
> 
> The format for specifying the actual intervals such as cycle-time could prove
> to be an important detail as well; Qbv specifies cycle-time as a ratio of two
> integers expressed in seconds, while extension-time is specified as an integer
> number of nanoseconds.
> 
> Precision with the cycle-time is especially important, since base-time can be
> almost arbitrarily far in the past or future, and any given cycle start should
> be calculable from the base-time plus/minus some integer multiple of cycle-
> time.

The above three points also.

Thanks,
Richard

Re: TSN Scorecard, was Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers

[Richard Cochran]

On Tue, Sep 19, 2017 at 11:17:54PM -0600, levipear...@gmail.com wrote:
> In addition to OpenAvnu, Renesas has a number of github repositories with 
> what looks like a fairly
> complete media streaming system:

Is it a generic stack or a set of hacks for their HW?

> Although your SO_TXTIME proposal could certainly form the basis of an 
> endpoint's implementation of Qbv, I
> think it is a stretch to consider it a Qbv implementation in itself, if 
> that's what you mean by this.

No, that is not what I meant.  We need some minimal additional kernel
support in order to fully implement the TSN family of standards.  Of
course, the bulk will have to be done in user space.  It would be a
mistake to cram the stuff that belongs in userland into the kernel.

Looking at the table, and reading your descriptions of the state of
OpenAVB, I remained convinced that the kernel needs only three
additions:

1. SO_TXTIME
2. CBS Qdisc
3. ALSA support for DAC clock control (but that is another story)

> The proper interfaces for the Qbv configuration and managing of switch-level 
> PTP timestamps are not yet
> in place, so there's nothing even at RFC stage to present yet, but 
> Qbv-capable Linux-managed switch
> hardware is available and we hope to get some reusable code published even if 
> it's not yet ready to be
> integrated in the kernel.

Right, configuring Qbv in an attached DSA switch needs its own
interface.

Regarding PHC support for DSA switches, I have something in the works
to be published soon.

> A bit of progress has been made since that was written, although it is true 
> that it's still not
> quite complete and certainly not turnkey.

So OpenAVB is neither complete nor turnkey.  That was my impression,
too.

> Things are maybe a bit farther along than they seemed, but there is still 
> important kernel work to be
> done to reduce the need for out-of-tree drivers and to get everyone on the 
> same interfaces. I plan
> to be an active participant going forward.

You mentioned a couple of different kernel things you implemented.
I would encourage you to post the work already done.

Thanks,
Richard

Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers

[Richard Cochran]

On Tue, Sep 19, 2017 at 05:19:18PM -0700, Vinicius Costa Gomes wrote:
> One of the problems with OpenAVNU is that it's too coupled with the i210
> NIC. One of the things we want is to decouple OpenAVNU from the
> controller.

Yes, I want that, too.

> The way we thought best was to propose interfaces (that
> would work along side to the Linux networking stack) as close as
> possible to what the current standards define, that means the IEEE
> 802.1Q family of specifications, in the hope that network controller
> vendors would also look at the specifications when designing their
> controllers.

These standard define the *behavior*, not the programming APIs.  Our
task as kernel developers is to invent the best interfaces for
supporting 802.1Q and other standards, the hardware capabilities, and
the widest range of applications (not jut AVB).

> Our objective with the Qdiscs we are proposing (both cbs and taprio) is
> to provide a sane way to configure controllers that support TSN features
> (we were looking specifically at the IEEE specs).

I can see how your proposed Qdiscs are inspired by the IEEE standards.
However, in the case of time based transmission, I think there is a
better way to do it, namely with SO_TXTIME (which BTW was originally
proposed by Eric Mann).
 
> After we have some rough consensus on the interfaces to use, then we can
> start working on OpenAVNU.

Did you see my table in the other mail?  Any comments?

> (Sorry if I am being annoying here, but the idea of an opaque schedule
> is not ours, that comes from the people who wrote the Qbv specification)

The schedule is easy to implement using SO_TXTIME.
 
> I have a question, what about a controller that doesn't provide a way to
> set a per-packet transmission time, but it supports Qbv/Qbu. What would
> be your proposal to configure it?

SO_TXTIME will have a generic SW fallback.

BTW, regarding the i210, there is no sensible way to configure both
CBS and time based transmission at the same time.  The card performs a
logical AND to make the launch decision.  The effect of this is that
each and every packet needs a LaunchTime, and the driver would be
forced to guess the time for a packet before entering it into its
queue.

So if we end up merging CBS and SO_TXTIME, then we'll have to make
them exclusive of each other (in the case of the i210) and manage the
i210 queue configurations correctly.

> (I think LaunchTime is something specific to the i210, right?)

To my knowledge yes.  However, if TSN does take hold, then other MAC
vendors will copy it.

Thanks,
Richard

TSN Scorecard, was Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers

[levipearson]

On Mon, Sep 18, 2017, Richard Cochran wrote:
> Just for the record, here is my score card showing the current status
> of TSN support in Linux.  Comments and corrections are more welcome.
> 
> Thanks,
> Richard
> 
> 
>  | FEATURE| STANDARD| 
> STATUS   |
>  
> |+-+--|
>  | Synchronization| 802.1AS-2011| 
> Implemented in   |
>  || | - 
> Linux kernel PHC subsystem |
>  || | - 
> linuxptp (userspace)   |
>  
> |+-+--|

An alternate implementation of the userspace portion of gPTP is also available 
at [1]

>  | Forwarding and Queuing Enhancements| 802.1Q-2014 sec. 34 | RFC 
> posted (this thread) |
>  | for Time-Sensitive Streams (FQTSS) | | 
>  |
>  
> |+-+--|
>  | Stream Reservation Protocol (SRP)  | 802.1Q-2014 sec. 35 | in 
> Open-AVB [1]  |
>  
> |+-+--|
>  | Audio Video Transport Protocol (AVTP)  | IEEE 1722-2011  | DNE 
>  |
>  
> |+-+--|
>  | Audio/Video Device Discovery, Enumeration, | IEEE 1722.1-2013| 
> jdksavdecc-c [2] |
>  | Connection Management and Control (AVDECC) | | 
>  |
>  | AVDECC Connection Management Protocol (ACMP)   | | 
>  |
>  | AVDECC Enumeration and Control Protocol (AECP) | | 
>  |
>  | MAC Address Acquisition Protocol (MAAP)| | in 
> Open-AVB  |
>  
> |+-+--|

All of the above are available to some degree in the AVTP Pipeline part of [1], 
specifically at this
location: https://github.com/AVnu/OpenAvnu/tree/master/lib/avtp_pipeline

The code is very modular and configurable, although some parts are in better 
shape than others. The AVTP
portion can use the custom userspace driver for the i210, which can be 
configured to use launch scheduling,
or it can use standard kernel interfaces via sendmsg or PACKET_MMAP. It runs 
as-is when configured for
standard interfaces with any network hardware that supports gPTP. I previously 
implemented a CMSG-based
launch time scheduling mechanism like the one you have proposed, and I have a 
socket backend for it that
could easily be ported to your proposal. It is not part of the repository yet 
since there's no kernel
support for it outside of my prototype and your RFC.

It is currently tied to the OpenAvnu gPTP daemon rather than linuxptp, as it 
uses a shared memory interface
to get the current rate-ratio and offset information between the various 
clocks. There may be better ways
to do this, but that's how the initial port of the codebase was done. It would 
be nice to get it working
with linuxptp's userspace tools at some point as well, though.

The libraries under avtp_pipeline are designed to be used separately, but a 
simple integrated application
is provided and is built by the CI system.

In addition to OpenAvnu, Renesas has a number of github repositories with what 
looks like a fairly
complete media streaming system:

https://github.com/renesas-rcar/avb-mse
https://github.com/renesas-rcar/avb-streaming
https://github.com/renesas-rcar/avb-applications

I haven't examined them in great detail yet, though.


>  | Frame Preemption   | P802.1Qbu   | DNE 
>  |
>  | Scheduled Traffic  | P802.1Qbv   | RFC 
> posted (SO_TXTIME)   |
>  | SRP Enhancements and Performance Improvements  | P802.1Qcc   | DNE 
>  |
> 
>  DNE = Does Not Exist (to my knowledge)

Although your SO_TXTIME proposal could certainly form the basis of an 
endpoint's implementation of Qbv, I
think it is a stretch to consider it a Qbv implementation in itself, if that's 
what you mean by this.

I have been working with colleagues on some experiments relating to a 
Linux-controlled DSN switch
(a Marvell Topaz) that are a part of this effort in TSN: 

http://ieee802.org/1/files/public/docs2017/tsn-cgunther-802-3cg-multidrop-0917-v01.pdf

The proper interfaces for the Qbv 

Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers

[levipearson]

On Thu, Aug 31, 2017 at 06:26:20PM -0700, Vinicius Costa Gomes wrote:
> Hi,
> 
> This patchset is an RFC on a proposal of how the Traffic Control subsystem can
> be used to offload the configuration of traffic shapers into network devices
> that provide support for them in HW. Our goal here is to start upstreaming
> support for features related to the Time-Sensitive Networking (TSN) set of
> standards into the kernel.

I'm very excited to see these features moving into the kernel! I am one of the
maintainers of the OpenAvnu project and I've been involved in building AVB/TSN
systems and working on the standards for around 10 years, so the support that's
been slowly making it into more silicon and now Linux drivers is very
encouraging.

My team at Harman is working on endpoint code based on what's in the OpenAvnu
project and a few Linux-based platforms. The Qav interface you've proposed will
fit nicely with our traffic shaper management daemon, which already uses mqprio
as a base but uses the htb shaper to approximate the Qav credit-based shaper on
platforms where launch time scheduling isn't available.

I've applied your patches and plan on testing them in conjunction with our
shaper manager to see if we run into any hitches, but I don't expect any
problems.

> As part of this work, we've assessed previous public discussions related to 
> TSN
> enabling: patches from Henrik Austad (Cisco), the presentation from Eric Mann
> at Linux Plumbers 2012, patches from Gangfeng Huang (National Instruments) and
> the current state of the OpenAVNU project (https://github.com/AVnu/OpenAvnu/).
> 
> Please note that the patches provided as part of this RFC are implementing 
> what
> is needed only for 802.1Qav (FQTSS) only, but we'd like to take advantage of
> this discussion and share our WIP ideas for the 802.1Qbv and 802.1Qbu 
> interfaces
> as well. The current patches are only providing support for HW offload of the
> configs.
> 
> 
> Overview
> 
> 
> Time-sensitive Networking (TSN) is a set of standards that aim to address
> resources availability for providing bandwidth reservation and bounded latency
> on Ethernet based LANs. The proposal described here aims to cover mainly what 
> is
> needed to enable the following standards: 802.1Qat, 802.1Qav, 802.1Qbv and
> 802.1Qbu.
> 
> The initial target of this work is the Intel i210 NIC, but other controllers'
> datasheet were also taken into account, like the Renesas RZ/A1H RZ/A1M group 
> and
> the Synopsis DesignWare Ethernet QoS controller.

Recent SoCs from NXP (the i.MX 6 SoloX, and all the i.MX 7 and 8 parts) support
Qav shaping as well as scheduled launch functionality; these are the parts I 
have been mostly working with. Marvell silicon (some subset of Armada processors
and Link Street DSA switches) generally supports traffic shaping as well.

I think a lack of an interface like this has probably slowed upstream driver
support for this functionality where it exists; most vendors have an out-of-
tree version of their driver with TSN functionality enabled via non-standard
interfaces. Hopefully making it available will encourage vendors to upstream
their driver support!

> Proposal
> 
> 
> Feature-wise, what is covered here are configuration interfaces for HW
> implementations of the Credit-Based shaper (CBS, 802.1Qav), Time-Aware shaper
> (802.1Qbv) and Frame Preemption (802.1Qbu). CBS is a per-queue shaper, while
> Qbv and Qbu must be configured per port, with the configuration covering all
> queues. Given that these features are related to traffic shaping, and that the
> traffic control subsystem already provides a queueing discipline that offloads
> config into the device driver (i.e. mqprio), designing new qdiscs for the
> specific purpose of offloading the config for each shaper seemed like a good
> fit.

This makes sense to me too. The 802.1Q standards are all based on the sort of
mappings between priority, traffic class, and hardware queues that the existing
tc infrastructure seems to be modeling. I believe the mqprio module's mapping
scheme is flexible enough to meet any TSN needs in conjunction with the other
parts of the kernel qdisc system.

> For steering traffic into the correct queues, we use the socket option
> SO_PRIORITY and then a mechanism to map priority to traffic classes / 
> Txqueues.
> The qdisc mqprio is currently used in our tests.
> 
> As for the shapers config interface:
> 
>  * CBS (802.1Qav)
> 
>This patchset is proposing a new qdisc called 'cbs'. Its 'tc' cmd line is:
>$ tc qdisc add dev IFACE parent ID cbs locredit N hicredit M sendslope S \
>  idleslope I
> 
>Note that the parameters for this qdisc are the ones defined by the
>802.1Q-2014 spec, so no hardware specific functionality is exposed here.

These parameters look good to me as a baseline; some additional optional
parameters may be useful for software-based implementations--such as setting an
interval at which to recalculate queues--but 

Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers

[Vinicius Costa Gomes]

Hi Richard,

Richard Cochran  writes:

> On Mon, Sep 18, 2017 at 04:06:28PM -0700, Vinicius Costa Gomes wrote:
>> That's the point, the application does not need to know that, and asking
>> that would be stupid.
>
> On the contrary, this information is essential to the application.
> Probably you have never seen an actual Ethernet field bus in
> operation?  In any case, you are missing the point.
>
>> (And that's another nice point of how 802.1Qbv works, applications do
>> not need to be changed to use it, and I think we should work to achieve
>> this on the Linux side)
>
> Once you start to care about real time performance, then you need to
> consider the applications.  This is industrial control, not streaming
> your tunes from your ipod.
>
>> That being said, that only works for kinds of traffic that maps well to
>> this configuration in advance model, which is the model that the IEEE
>> (see 802.1Qcc) and the AVNU Alliance[1] are pushing for.
>
> Again, you are missing the point of what they aiming for.  I have
> looked at a number of production systems, and in each case the
> developers want total control over the transmission, in order to
> reduce latency to an absolute minimum.  Typically the data to be sent
> are available only microseconds before the transmission deadline.
>
> Consider OpenAVB on github that people are already using.  Take a look
> at simple_talker.c and explain how "applications do not need to be
> changed to use it."

Just let me use the mention of OpenAVNU as a hook to explain what we
(the team I am part of) are working to do, perhaps it will make our
choices and designs clearer.

One of the problems with OpenAVNU is that it's too coupled with the i210
NIC. One of the things we want is to decouple OpenAVNU from the
controller. The way we thought best was to propose interfaces (that
would work along side to the Linux networking stack) as close as
possible to what the current standards define, that means the IEEE
802.1Q family of specifications, in the hope that network controller
vendors would also look at the specifications when designing their
controllers.

Our objective with the Qdiscs we are proposing (both cbs and taprio) is
to provide a sane way to configure controllers that support TSN features
(we were looking specifically at the IEEE specs).

After we have some rough consensus on the interfaces to use, then we can
start working on OpenAVNU.

>
>> [1]
>> http://avnu.org/theory-of-operation-for-tsn-enabled-industrial-systems/
>
> Did you even read this?
>
> [page 24]
>
> As described in section 2, some industrial control systems require
> predictable, very low latency and cycle-to-cycle variation to meet
> hard real-time application requirements. In these systems,
> multiple distributed controllers commonly synchronize their
> sensor/actuator operations with other controllers by scheduling
> these operations in time, typically using a repeating control
> cycle.
> ...
> The gate control mechanism is itself a time-aware PTP application
> operating within a bridge or end station port.
>
> It is an application, not a "god box."
>
>> In short, I see a per-packet transmission time and a per-queue schedule
>> as solutions to different problems.
>
> Well, I can agree with that.  For some non real-time applications,
> bandwidth shaping is enough, and your Qdisc idea is sufficient.  For
> the really challenging TSN targets (industrial control, automotive),
> your idea of an opaque schedule file won't fly.

(Sorry if I am being annoying here, but the idea of an opaque schedule
is not ours, that comes from the people who wrote the Qbv specification)

I have a question, what about a controller that doesn't provide a way to
set a per-packet transmission time, but it supports Qbv/Qbu. What would
be your proposal to configure it?

(I think LaunchTime is something specific to the i210, right?)


Cheers,
--
Vinicius

Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers

[Henrik Austad]

Hi all,

On Tue, Sep 19, 2017 at 07:22:44AM +0200, Richard Cochran wrote:
> On Mon, Sep 18, 2017 at 04:06:28PM -0700, Vinicius Costa Gomes wrote:
> > That's the point, the application does not need to know that, and asking
> > that would be stupid.
> 
> On the contrary, this information is essential to the application.
> Probably you have never seen an actual Ethernet field bus in
> operation?  In any case, you are missing the point.
> 
> > (And that's another nice point of how 802.1Qbv works, applications do
> > not need to be changed to use it, and I think we should work to achieve
> > this on the Linux side)
> 
> Once you start to care about real time performance, then you need to
> consider the applications.  This is industrial control, not streaming
> your tunes from your ipod.

Do not underestimate the need for media over TSN. I fully see your point of 
real-time systems, but they are not the only valid use-cases for TSN.

> > That being said, that only works for kinds of traffic that maps well to
> > this configuration in advance model, which is the model that the IEEE
> > (see 802.1Qcc) and the AVNU Alliance[1] are pushing for.
> 
> Again, you are missing the point of what they aiming for.  I have
> looked at a number of production systems, and in each case the
> developers want total control over the transmission, in order to
> reduce latency to an absolute minimum.  Typically the data to be sent
> are available only microseconds before the transmission deadline.
> 
> Consider OpenAVB on github that people are already using.  Take a look
> at simple_talker.c and explain how "applications do not need to be
> changed to use it."

I do not think simple-talker was everintended to be how users of AVB should 
be implemented, but as a demonstration of what the protocol could do.

ALSA/V4L2 should supply some interface to this so that you can attach 
media-applications to it without the application itself having to be "TSN 
aware".

> > [1]
> > http://avnu.org/theory-of-operation-for-tsn-enabled-industrial-systems/
> 
> Did you even read this?
> 
> [page 24]
> 
> As described in section 2, some industrial control systems require
> predictable, very low latency and cycle-to-cycle variation to meet
> hard real-time application requirements. In these systems,
> multiple distributed controllers commonly synchronize their
> sensor/actuator operations with other controllers by scheduling
> these operations in time, typically using a repeating control
> cycle.
> ...
> The gate control mechanism is itself a time-aware PTP application
> operating within a bridge or end station port.
> 
> It is an application, not a "god box."
>
> > In short, I see a per-packet transmission time and a per-queue schedule
> > as solutions to different problems.
> 
> Well, I can agree with that.  For some non real-time applications,
> bandwidth shaping is enough, and your Qdisc idea is sufficient.  For
> the really challenging TSN targets (industrial control, automotive),
> your idea of an opaque schedule file won't fly.

Would it make sense to adapt the proposed Qdisc here as well as the 
back-o-the-napkin idea in the other thread to to a per-socket queue for 
each priority and then sort those sockets based on SO_TXTIME?

TSN operates on a per-StreamID basis, and that should map fairly well to a 
per-socket approach I think (let us just assume that an application that 
sends TSN traffic will open up a separate socket for each stream.

This should allow a userspace application that is _very_ aware of its 
timing constraints to send frames exactly when it needs to as you have 
SO_TXTIME available. It would also let applications that basically want a 
fine-grained rate control (audio and video comes to mind) to use the same 
qdisc.

For those sockets that do not support SO_TXTIME, but still map to a 
priority handled by sch_cbs (or whatever it'll end up being called) you can 
set the transmit-time to be the time of the last skb in the queue + an 
delta which will give you the correct rate (TSN operates on observation 
intervals which you can specify via tc when you create the queues).

Then you can have, as you propose in your other series, a hrtimer that is 
being called when the next SO_TXTIME enters, grab a skb and move it to the 
hw-queue. This should also allow you to keep a sorted per-socket queue 
should an application send frames in the wrong order, without having to 
rearrange descriptors for the DMA machinery.

If this makes sense, I am more than happy to give it a stab and see how it 
goes.

-Henrik


signature.asc
Description: PGP signature

Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers

[Richard Cochran]

On Mon, Sep 18, 2017 at 04:06:28PM -0700, Vinicius Costa Gomes wrote:
> That's the point, the application does not need to know that, and asking
> that would be stupid.

On the contrary, this information is essential to the application.
Probably you have never seen an actual Ethernet field bus in
operation?  In any case, you are missing the point.

> (And that's another nice point of how 802.1Qbv works, applications do
> not need to be changed to use it, and I think we should work to achieve
> this on the Linux side)

Once you start to care about real time performance, then you need to
consider the applications.  This is industrial control, not streaming
your tunes from your ipod.
 
> That being said, that only works for kinds of traffic that maps well to
> this configuration in advance model, which is the model that the IEEE
> (see 802.1Qcc) and the AVNU Alliance[1] are pushing for.

Again, you are missing the point of what they aiming for.  I have
looked at a number of production systems, and in each case the
developers want total control over the transmission, in order to
reduce latency to an absolute minimum.  Typically the data to be sent
are available only microseconds before the transmission deadline.

Consider OpenAVB on github that people are already using.  Take a look
at simple_talker.c and explain how "applications do not need to be
changed to use it."

> [1]
> http://avnu.org/theory-of-operation-for-tsn-enabled-industrial-systems/

Did you even read this?

[page 24]

As described in section 2, some industrial control systems require
predictable, very low latency and cycle-to-cycle variation to meet
hard real-time application requirements. In these systems,
multiple distributed controllers commonly synchronize their
sensor/actuator operations with other controllers by scheduling
these operations in time, typically using a repeating control
cycle.
...
The gate control mechanism is itself a time-aware PTP application
operating within a bridge or end station port.

It is an application, not a "god box."

> In short, I see a per-packet transmission time and a per-queue schedule
> as solutions to different problems.

Well, I can agree with that.  For some non real-time applications,
bandwidth shaping is enough, and your Qdisc idea is sufficient.  For
the really challenging TSN targets (industrial control, automotive),
your idea of an opaque schedule file won't fly.

Thanks,
Richard

Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers

[Vinicius Costa Gomes]

Hi Richard,

Richard Cochran  writes:

> On Thu, Aug 31, 2017 at 06:26:20PM -0700, Vinicius Costa Gomes wrote:
>>  * Time-aware shaper (802.1Qbv):
>
> I just posted a working alternative showing how to handle 802.1Qbv and
> many other Ethernet field buses.
>
>>The idea we are currently exploring is to add a "time-aware", priority 
>> based
>>qdisc, that also exposes the Tx queues available and provides a mechanism 
>> for
>>mapping priority <-> traffic class <-> Tx queues in a similar fashion as
>>mqprio. We are calling this qdisc 'taprio', and its 'tc' cmd line would 
>> be:
>>
>>$ $ tc qdisc add dev ens4 parent root handle 100 taprio num_tc 4\
>> map 2 2 1 0 3 3 3 3 3 3 3 3 3 3 3 3 \
>> queues 0 1 2 3  \
>> sched-file gates.sched [base-time ]   \
>>[cycle-time ] [extension-time ]
>>
>> is multi-line, with each line being of the following format:
>>  
>>
>>Qbv only defines one : "S" for 'SetGates'
>>
>>For example:
>>
>>S 0x01 300
>>S 0x03 500
>>
>>This means that there are two intervals, the first will have the gate
>>for traffic class 0 open for 300 nanoseconds, the second will have
>>both traffic classes open for 500 nanoseconds.
>
> The idea of the schedule file will not work in practice.  Consider the
> fact that the application wants to deliver time critical data in a
> particular slot.  How can it find out a) what the time slots are and
> b) when the next slot is scheduled?  With this Qdisc, it cannot do
> this, AFAICT.  The admin might delete the file after configuring the
> Qdisc!

That's the point, the application does not need to know that, and asking
that would be stupid. From the point of view of the Qbv specification,
applications only need to care about its basic bandwidth requirements:
its interval, frame size, frames per interval (using the terms of the
SRP section of 802.1Q). The traffic schedule is provided (off band) by a
"god box" which knows all the requirements of all applications in all
the nodes and how they are connected.

(And that's another nice point of how 802.1Qbv works, applications do
not need to be changed to use it, and I think we should work to achieve
this on the Linux side)

That being said, that only works for kinds of traffic that maps well to
this configuration in advance model, which is the model that the IEEE
(see 802.1Qcc) and the AVNU Alliance[1] are pushing for.

In the real world, I can see multiple types of applications, some using
something like TXTIME, and some configured in advance.

>
> Using the SO_TXTIME option, the application has total control over the
> scheduling.  The great advantages of this approach is that we can
> support any possible combination of periodic or aperiodic scheduling
> and we can support any priority scheme user space dreams up.

It has the disavantage of that the scheduling information has to be
in-band with the data. I *really* think that for scheduled traffic,
there should be a clear separation, we should not mix the dataflow with
scheduling. In short, an application in the network don't need to have
all the information necessary to schedule its own traffic well.

I have two points here: 1. I see both "solutions" (taprio and SO_TXTIME)
as being ortoghonal and useful, both; 2. trying to make one do the job
of the other, however, looks like "If all I have is a hammer, everything
looks like a nail".

In short, I see a per-packet transmission time and a per-queue schedule
as solutions to different problems.

>
> For example, one can imaging running two or more loops that only
> occasionally collide.  When they do collide, which packet should be
> sent first?  Just let user space decide.
>
> Thanks,
> Richard

Cheers,
--
Vinicius

[1]
http://avnu.org/theory-of-operation-for-tsn-enabled-industrial-systems/

Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers

[Henrik Austad]

On Mon, Sep 18, 2017 at 10:02:14AM +0200, Richard Cochran wrote:
> On Thu, Aug 31, 2017 at 06:26:20PM -0700, Vinicius Costa Gomes wrote:
> >  * Time-aware shaper (802.1Qbv):
> 
> I just posted a working alternative showing how to handle 802.1Qbv and
> many other Ethernet field buses.

Yes, I saw them, grabbing them for testing now - thanks!

> >The idea we are currently exploring is to add a "time-aware", priority 
> > based
> >qdisc, that also exposes the Tx queues available and provides a 
> > mechanism for
> >mapping priority <-> traffic class <-> Tx queues in a similar fashion as
> >mqprio. We are calling this qdisc 'taprio', and its 'tc' cmd line would 
> > be:
> > 
> >$ $ tc qdisc add dev ens4 parent root handle 100 taprio num_tc 4\
> >map 2 2 1 0 3 3 3 3 3 3 3 3 3 3 3 3 \
> >queues 0 1 2 3  \
> >sched-file gates.sched [base-time ]   \
> >[cycle-time ] [extension-time ]
> > 
> > is multi-line, with each line being of the following format:
> >  
> > 
> >Qbv only defines one : "S" for 'SetGates'
> > 
> >For example:
> > 
> >S 0x01 300
> >S 0x03 500
> > 
> >This means that there are two intervals, the first will have the gate
> >for traffic class 0 open for 300 nanoseconds, the second will have
> >both traffic classes open for 500 nanoseconds.
> 
> The idea of the schedule file will not work in practice.  Consider the
> fact that the application wants to deliver time critical data in a
> particular slot.  How can it find out a) what the time slots are and
> b) when the next slot is scheduled?  With this Qdisc, it cannot do
> this, AFAICT.  The admin might delete the file after configuring the
> Qdisc!
> 
> Using the SO_TXTIME option, the application has total control over the
> scheduling.  The great advantages of this approach is that we can
> support any possible combination of periodic or aperiodic scheduling
> and we can support any priority scheme user space dreams up.

Using SO_TXTIME makes a lot of sense. TSN has a presentation_time, which 
you can use to deduce the time it should be transmitted (Class A has a 2ms 
latency guarantee, B has 50), but given how TSN uses the timestamp, it will 
wrap every 4.3 seconds, using SO_TXTIME allows you to schedule transmission 
at a much later time. It should also lessen the dependency on a specific 
protocol, which is also good.

> For example, one can imaging running two or more loops that only
> occasionally collide.  When they do collide, which packet should be
> sent first?  Just let user space decide.

If 2 userspace apps send to the same Tx-queue with the same priority, would 
it not make sense to just do FIFO? For all practical purposes, they have 
the same importance (same SO_PRIORITY, same SO_TXTIME). If the priority 
differs, then they would be directed to different queues, where one queue 
will take presedence anyway.

How far into the future would it make sense to schedule packets anyway?

I'll have a look at the other series you just posted!

-- 
Henrik Austad


signature.asc
Description: PGP signature

Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers

[Richard Cochran]

On Thu, Aug 31, 2017 at 06:26:20PM -0700, Vinicius Costa Gomes wrote:
> This patchset is an RFC on a proposal of how the Traffic Control subsystem can
> be used to offload the configuration of traffic shapers into network devices
> that provide support for them in HW. Our goal here is to start upstreaming
> support for features related to the Time-Sensitive Networking (TSN) set of
> standards into the kernel.

Just for the record, here is my score card showing the current status
of TSN support in Linux.  Comments and corrections are more welcome.

Thanks,
Richard


 | FEATURE| STANDARD| 
STATUS   |
 
|+-+--|
 | Synchronization| 802.1AS-2011| 
Implemented in   |
 || | - 
Linux kernel PHC subsystem |
 || | - 
linuxptp (userspace)   |
 
|+-+--|
 | Forwarding and Queuing Enhancements| 802.1Q-2014 sec. 34 | RFC 
posted (this thread) |
 | for Time-Sensitive Streams (FQTSS) | |   
   |
 
|+-+--|
 | Stream Reservation Protocol (SRP)  | 802.1Q-2014 sec. 35 | in 
Open-AVB [1]  |
 
|+-+--|
 | Audio Video Transport Protocol (AVTP)  | IEEE 1722-2011  | DNE   
   |
 
|+-+--|
 | Audio/Video Device Discovery, Enumeration, | IEEE 1722.1-2013| 
jdksavdecc-c [2] |
 | Connection Management and Control (AVDECC) | |   
   |
 | AVDECC Connection Management Protocol (ACMP)   | |   
   |
 | AVDECC Enumeration and Control Protocol (AECP) | |   
   |
 | MAC Address Acquisition Protocol (MAAP)| | in 
Open-AVB  |
 
|+-+--|
 | Frame Preemption   | P802.1Qbu   | DNE   
   |
 | Scheduled Traffic  | P802.1Qbv   | RFC 
posted (SO_TXTIME)   |
 | SRP Enhancements and Performance Improvements  | P802.1Qcc   | DNE   
   |

 DNE = Does Not Exist (to my knowledge)

1. https://github.com/Avnu/OpenAvnu

   (DISCLAIMER from the website:)

   It is planned to eventually include the various packet encapsulation types,
   protocol discovery daemons, libraries to convert media clocks to AVB clocks
   and vice versa, and drivers.

   This repository does not include all components required to build a full
   production AVB/TSN system (e.g. a turnkey solution to stream stored or live 
audio
   or video content). Some simple example applications are provided which
   illustrate the flow - but a professional Audio/Video system requires a full 
media stack
   - including audio and video inputs and outputs, media processing elements, 
and
   various graphical user interfaces. Various companies provide such integrated
   solutions.

2. https://github.com/jdkoftinoff/jdksavdecc-c

Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers

[Richard Cochran]

On Thu, Aug 31, 2017 at 06:26:20PM -0700, Vinicius Costa Gomes wrote:
>  * Time-aware shaper (802.1Qbv):

I just posted a working alternative showing how to handle 802.1Qbv and
many other Ethernet field buses.
 
>The idea we are currently exploring is to add a "time-aware", priority 
> based
>qdisc, that also exposes the Tx queues available and provides a mechanism 
> for
>mapping priority <-> traffic class <-> Tx queues in a similar fashion as
>mqprio. We are calling this qdisc 'taprio', and its 'tc' cmd line would be:
> 
>$ $ tc qdisc add dev ens4 parent root handle 100 taprio num_tc 4\
>  map 2 2 1 0 3 3 3 3 3 3 3 3 3 3 3 3 \
>  queues 0 1 2 3  \
>  sched-file gates.sched [base-time ]   \
>[cycle-time ] [extension-time ]
> 
> is multi-line, with each line being of the following format:
>  
> 
>Qbv only defines one : "S" for 'SetGates'
> 
>For example:
> 
>S 0x01 300
>S 0x03 500
> 
>This means that there are two intervals, the first will have the gate
>for traffic class 0 open for 300 nanoseconds, the second will have
>both traffic classes open for 500 nanoseconds.

The idea of the schedule file will not work in practice.  Consider the
fact that the application wants to deliver time critical data in a
particular slot.  How can it find out a) what the time slots are and
b) when the next slot is scheduled?  With this Qdisc, it cannot do
this, AFAICT.  The admin might delete the file after configuring the
Qdisc!

Using the SO_TXTIME option, the application has total control over the
scheduling.  The great advantages of this approach is that we can
support any possible combination of periodic or aperiodic scheduling
and we can support any priority scheme user space dreams up.

For example, one can imaging running two or more loops that only
occasionally collide.  When they do collide, which packet should be
sent first?  Just let user space decide.

Thanks,
Richard

Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers

[Richard Cochran]

On Thu, Sep 07, 2017 at 06:29:00PM -0700, Vinicius Costa Gomes wrote:
> >>  * Time-aware shaper (802.1Qbv):
> >>
> >>The idea we are currently exploring is to add a "time-aware", priority 
> >> based
> >>qdisc, that also exposes the Tx queues available and provides a 
> >> mechanism for
> >>mapping priority <-> traffic class <-> Tx queues in a similar fashion as
> >>mqprio. We are calling this qdisc 'taprio', and its 'tc' cmd line would 
> >> be:
> >
> > As far as I know, this is not supported by i210, and if time-aware shaping
> > is enabled in the network - you'll be queued on a bridge until the window
> > opens as time-aware shaping is enforced on the tx-port and not on rx. Is
> > this required in this driver?
> 
> Yeah, i210 doesn't support the time-aware shaper. I think the second
> part of your question doesn't really apply, then.

Actually, you can implement 802.1Qbv (as an end station) quite easily
using the i210.  I'll show how by posting a series after net-next
opens up again.

Thanks,
Richard

Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers

[Henrik Austad]

On Thu, Sep 07, 2017 at 07:58:53PM +, Guedes, Andre wrote:
> Hi Henrik,
> 
> Thanks for your feedback! I'll address some of your comments below.
> 
> On Thu, 2017-09-07 at 07:34 +0200, Henrik Austad wrote:
> > > As for the shapers config interface:
> > > 
> > >  * CBS (802.1Qav)
> > > 
> > >    This patchset is proposing a new qdisc called 'cbs'. Its 'tc' cmd line
> > > is:
> > >    $ tc qdisc add dev IFACE parent ID cbs locredit N hicredit M sendslope 
> > > S
> > > \
> > >  idleslope I
> > 
> > So this confuses me a bit, why specify sendSlope?
> > 
> > sendSlope = portTransmitRate - idleSlope
> > 
> > and portTransmitRate is the speed of the MAC (which you get from the 
> > driver). Adding sendSlope here is just redundant I think.
> 
> Yes, this was something we've spent quite a few time discussing before this 
> RFC
> series. After reading the Annex L from 802.1Q-2014 (operation of CBS 
> algorithm)
> so many times, we've came up with the rationale explained below.
> 
> The rationale here is that sendSlope is just another parameter from CBS
> algorithm like idleSlope, hiCredit and loCredit. As such, its calculation
> should be done at the same "layer" as the others parameters (in this case, 
> user
> space) in order to keep consistency. Moreover, in this design, the driver 
> layer
> is dead simple: all the device driver has to do is applying CBS parameters to
> hardware. Having any CBS parameter calculation in the driver layer means all
> device drivers must implement that calculation.

Ok, that actually makes a lot of sense, and anything that keeps this kind 
of arithmetic outside the kernel is a good thing!

Thanks for the clarification!

> > Also, does this mean that when you create the qdisc, you have locked the 
> > bandwidth for the scheduler? Meaning, if I later want to add another 
> > stream that requires more bandwidth, I have to close all active streams, 
> > reconfigure the qdisc and then restart?
> 
> If we want to reserve more bandwidth to "accommodate" a new stream, we don't
> need to close all active streams. All we have to do is changing the CBS qdisc
> and pass the new CBS parameters. Here is what the command-line would look 
> like:
> 
> $ tc qdisc change dev enp0s4 parent 8001:5 cbs locredit -1470 hicredit 30
> sendslope -98 idleslope 2
> 
> No application/stream is interrupted while new CBS parameters are applied.

Ah, good.

> > >    Note that the parameters for this qdisc are the ones defined by the
> > >    802.1Q-2014 spec, so no hardware specific functionality is exposed 
> > > here.
> > 
> > You do need to know if the link is brought up as 100 or 1000 though - which 
> > the driver already knows.
> 
> User space knows that information via ethtool or /sys.

Fair point.

> > > Testing this RFC
> > > 
> > > 
> > > For testing the patches of this RFC only, you can refer to the samples and
> > > helper script being added to samples/tsn/ and the use the 'mqprio' qdisc 
> > > to
> > > setup the priorities to Tx queues mapping, together with the 'cbs' qdisc 
> > > to
> > > configure the HW shaper of the i210 controller:
> > 
> > I will test it, feedback will be provided soon! :)
> 
> That's great! Please let us know if you find any issue and thanks for you
> support.
> 
> > > 8) You can also run a Talker for class B (prio 2 here)
> > > $ ./talker -i enp3s0 -p 2
> > > 
> > >  * The bandwidth displayed on the listener output now should increase to
> > > very
> > >    close to the one configured for class A + class B.
> > 
> > Because you grab both class A *and* B, or because B will eat what A does 
> > not use?
> 
> Because the listener application grabs both class A and B traffic.

Right, got it.

Thanks for the feedback, I'm getting really excited about this! :D

-- 
Henrik Austad


signature.asc
Description: PGP signature

Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers

[Vinicius Costa Gomes]

Henrik Austad  writes:

> On Thu, Aug 31, 2017 at 06:26:20PM -0700, Vinicius Costa Gomes wrote:
>> Hi,
>>
>> This patchset is an RFC on a proposal of how the Traffic Control subsystem 
>> can
>> be used to offload the configuration of traffic shapers into network devices
>> that provide support for them in HW. Our goal here is to start upstreaming
>> support for features related to the Time-Sensitive Networking (TSN) set of
>> standards into the kernel.
>
> Nice to see that others are working on this as well! :)
>
> A short disclaimer; I'm pretty much anchored in the view "linux is the
> end-station in a TSN domain", is this your approach as well, or are you
> looking at this driver to be used in bridges as well? (because that will
> affect the comments on time-aware shaper and frame preemption)
>
> Yet another disclaimer; I am not a linux networking subsystem expert. Not
> by a long shot! There are black magic happening in the internals of the
> networking subsystem that I am not even aware of. So if something I say or
> ask does not make sense _at_all_, that's probably why..
>
> I do know a tiny bit about TSN though, and I have been messing around
> with it for a little while, hence my comments below
>
>> As part of this work, we've assessed previous public discussions related to 
>> TSN
>> enabling: patches from Henrik Austad (Cisco), the presentation from Eric Mann
>> at Linux Plumbers 2012, patches from Gangfeng Huang (National Instruments) 
>> and
>> the current state of the OpenAVNU project 
>> (https://github.com/AVnu/OpenAvnu/).
>
> /me eyes Cc ;p
>
>> Overview
>> 
>>
>> Time-sensitive Networking (TSN) is a set of standards that aim to address
>> resources availability for providing bandwidth reservation and bounded 
>> latency
>> on Ethernet based LANs. The proposal described here aims to cover mainly 
>> what is
>> needed to enable the following standards: 802.1Qat, 802.1Qav, 802.1Qbv and
>> 802.1Qbu.
>>
>> The initial target of this work is the Intel i210 NIC, but other controllers'
>> datasheet were also taken into account, like the Renesas RZ/A1H RZ/A1M group 
>> and
>> the Synopsis DesignWare Ethernet QoS controller.
>
> NXP has a TSN aware chip on the i.MX7 sabre board as well 

Cool. Will take a look.

>
>> Proposal
>> 
>>
>> Feature-wise, what is covered here are configuration interfaces for HW
>> implementations of the Credit-Based shaper (CBS, 802.1Qav), Time-Aware shaper
>> (802.1Qbv) and Frame Preemption (802.1Qbu). CBS is a per-queue shaper, while
>> Qbv and Qbu must be configured per port, with the configuration covering all
>> queues. Given that these features are related to traffic shaping, and that 
>> the
>> traffic control subsystem already provides a queueing discipline that 
>> offloads
>> config into the device driver (i.e. mqprio), designing new qdiscs for the
>> specific purpose of offloading the config for each shaper seemed like a good
>> fit.
>
> just to be clear, you register sch_cbs as a subclass to mqprio, not as a
> root class?

That's right.

>
>> For steering traffic into the correct queues, we use the socket option
>> SO_PRIORITY and then a mechanism to map priority to traffic classes / Tx 
>> queues.
>> The qdisc mqprio is currently used in our tests.
>
> Right, fair enough, I'd prefer the TSN qdisc to be the root-device and
> rather have mqprio for high priority traffic and another for 'everything
> else'', but this would work too. This is not that relevant at this stage I
> guess :)

That's a scenario I haven't considered, will give it some thought.

>
>> As for the shapers config interface:
>>
>>  * CBS (802.1Qav)
>>
>>This patchset is proposing a new qdisc called 'cbs'. Its 'tc' cmd line is:
>>$ tc qdisc add dev IFACE parent ID cbs locredit N hicredit M sendslope S \
>>  idleslope I
>
> So this confuses me a bit, why specify sendSlope?
>
> sendSlope = portTransmitRate - idleSlope
>
> and portTransmitRate is the speed of the MAC (which you get from the
> driver). Adding sendSlope here is just redundant I think.
>
> Also, does this mean that when you create the qdisc, you have locked the
> bandwidth for the scheduler? Meaning, if I later want to add another
> stream that requires more bandwidth, I have to close all active streams,
> reconfigure the qdisc and then restart?
>
>>Note that the parameters for this qdisc are the ones defined by the
>>802.1Q-2014 spec, so no hardware specific functionality is exposed here.
>
> You do need to know if the link is brought up as 100 or 1000 though - which
> the driver already knows.
>
>>  * Time-aware shaper (802.1Qbv):
>>
>>The idea we are currently exploring is to add a "time-aware", priority 
>> based
>>qdisc, that also exposes the Tx queues available and provides a mechanism 
>> for
>>mapping priority <-> traffic class <-> Tx queues in a similar fashion as
>>mqprio. We are calling this qdisc 'taprio', and its 'tc' cmd line would 
>> be:
>
> As 

Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers

[Guedes, Andre]

On Thu, 2017-09-07 at 18:18 +0200, Henrik Austad wrote:
> On Thu, Sep 07, 2017 at 05:53:15PM +0200, Richard Cochran wrote:
> > On Thu, Sep 07, 2017 at 05:27:51PM +0200, Henrik Austad wrote:
> > > On Thu, Sep 07, 2017 at 02:40:18PM +0200, Richard Cochran wrote:
> > > And if you want to this driver to act as a bridge, how do you accomodate 
> > > change in network requirements? (i.e. how does this work with switchdev?)
> > 
> > To my understanding, this Qdisc idea provides QoS for the host's
> > transmitted traffic, and nothing more.
> 
> Ok, then we're on the same page.
> 
> > > - Or am I overthinking this?
> > 
> > Being able to configure the external ports of a switchdev is probably
> > a nice feature, but that is another story.  (But maybe I misunderstood
> > the authors' intent!)
> 
> ok, chalk that one up for later perhaps

Just to clarify, we've been most focused on end-station use-cases. We've
considered some bridge use-cases as well just to verify that the proposed
design won't be an issue if someone else goes for it.

- Andre

smime.p7s
Description: S/MIME cryptographic signature

Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers

[Guedes, Andre]

Hi Henrik,

Thanks for your feedback! I'll address some of your comments below.

On Thu, 2017-09-07 at 07:34 +0200, Henrik Austad wrote:
> > As for the shapers config interface:
> > 
> >  * CBS (802.1Qav)
> > 
> >    This patchset is proposing a new qdisc called 'cbs'. Its 'tc' cmd line
> > is:
> >    $ tc qdisc add dev IFACE parent ID cbs locredit N hicredit M sendslope S
> > \
> >  idleslope I
> 
> So this confuses me a bit, why specify sendSlope?
> 
> sendSlope = portTransmitRate - idleSlope
> 
> and portTransmitRate is the speed of the MAC (which you get from the 
> driver). Adding sendSlope here is just redundant I think.

Yes, this was something we've spent quite a few time discussing before this RFC
series. After reading the Annex L from 802.1Q-2014 (operation of CBS algorithm)
so many times, we've came up with the rationale explained below.

The rationale here is that sendSlope is just another parameter from CBS
algorithm like idleSlope, hiCredit and loCredit. As such, its calculation
should be done at the same "layer" as the others parameters (in this case, user
space) in order to keep consistency. Moreover, in this design, the driver layer
is dead simple: all the device driver has to do is applying CBS parameters to
hardware. Having any CBS parameter calculation in the driver layer means all
device drivers must implement that calculation.

> Also, does this mean that when you create the qdisc, you have locked the 
> bandwidth for the scheduler? Meaning, if I later want to add another 
> stream that requires more bandwidth, I have to close all active streams, 
> reconfigure the qdisc and then restart?

If we want to reserve more bandwidth to "accommodate" a new stream, we don't
need to close all active streams. All we have to do is changing the CBS qdisc
and pass the new CBS parameters. Here is what the command-line would look like:

$ tc qdisc change dev enp0s4 parent 8001:5 cbs locredit -1470 hicredit 30
sendslope -98 idleslope 2

No application/stream is interrupted while new CBS parameters are applied.

> >    Note that the parameters for this qdisc are the ones defined by the
> >    802.1Q-2014 spec, so no hardware specific functionality is exposed here.
> 
> You do need to know if the link is brought up as 100 or 1000 though - which 
> the driver already knows.

User space knows that information via ethtool or /sys.

> > Testing this RFC
> > 
> > 
> > For testing the patches of this RFC only, you can refer to the samples and
> > helper script being added to samples/tsn/ and the use the 'mqprio' qdisc to
> > setup the priorities to Tx queues mapping, together with the 'cbs' qdisc to
> > configure the HW shaper of the i210 controller:
> 
> I will test it, feedback will be provided soon! :)

That's great! Please let us know if you find any issue and thanks for you
support.

> > 8) You can also run a Talker for class B (prio 2 here)
> > $ ./talker -i enp3s0 -p 2
> > 
> >  * The bandwidth displayed on the listener output now should increase to
> > very
> >    close to the one configured for class A + class B.
> 
> Because you grab both class A *and* B, or because B will eat what A does 
> not use?

Because the listener application grabs both class A and B traffic.

Regards,

Andre

smime.p7s
Description: S/MIME cryptographic signature

Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers

[Henrik Austad]

On Thu, Sep 07, 2017 at 05:53:15PM +0200, Richard Cochran wrote:
> On Thu, Sep 07, 2017 at 05:27:51PM +0200, Henrik Austad wrote:
> > On Thu, Sep 07, 2017 at 02:40:18PM +0200, Richard Cochran wrote:
> > And if you want to this driver to act as a bridge, how do you accomodate 
> > change in network requirements? (i.e. how does this work with switchdev?)
> 
> To my understanding, this Qdisc idea provides QoS for the host's
> transmitted traffic, and nothing more.

Ok, then we're on the same page.

> > - Or am I overthinking this?
> 
> Being able to configure the external ports of a switchdev is probably
> a nice feature, but that is another story.  (But maybe I misunderstood
> the authors' intent!)

ok, chalk that one up for later perhaps

> > If you have more than 1 application in userspace that wants to send data 
> > using this scheduler, how do you ensure fair transmission of frames? (both 
> > how much bandwidth they use,
> 
> There are many ways to handle this, and we shouldn't put any of that
> policy into the kernel.  For example, there might be a monolithic
> application with configurable threads, or an allocation server that
> grants bandwidth to applications via IPC, or a multiplexing stream
> server like jack, pulse, etc, and so on...

true

> > but also ordering of frames from each application)
> 
> Not sure what you mean by this.

Fair enough, I'm not that good at making myself clear :)

Let's see if I can make a better attempt:

If you have 2 separate applications that have their own streams going to 
different endpoints - but both are in the same class, then they will 
share the qdisc bandwidth.

So application 
- A sends frame A1, A2, A3, .. An
- B sends B1, B2, .. Bn

What I was trying to describe was: if application A send 2 frames, and B 
sends 2 frames at the same time, then you would hope that the order would 
be A1, B1, A2, B2, and not A1, A2, B1, B2.

None of this would be a problem if you expect a *single* user, like the 
allocation server you described above. Again, I think this is just me 
overthinking the problem right now :)

> > Do you expect all of this to be handled in userspace?
> 
> Yes, I do.

ok, fair enough

Thanks for answering my questions!

-- 
Henrik Austad


signature.asc
Description: PGP signature

Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers

[Richard Cochran]

On Thu, Sep 07, 2017 at 05:27:51PM +0200, Henrik Austad wrote:
> On Thu, Sep 07, 2017 at 02:40:18PM +0200, Richard Cochran wrote:
> And if you want to this driver to act as a bridge, how do you accomodate 
> change in network requirements? (i.e. how does this work with switchdev?)

To my understanding, this Qdisc idea provides QoS for the host's
transmitted traffic, and nothing more.

> - Or am I overthinking this?

Being able to configure the external ports of a switchdev is probably
a nice feature, but that is another story.  (But maybe I misunderstood
the authors' intent!)

> If you have more than 1 application in userspace that wants to send data 
> using this scheduler, how do you ensure fair transmission of frames? (both 
> how much bandwidth they use,

There are many ways to handle this, and we shouldn't put any of that
policy into the kernel.  For example, there might be a monolithic
application with configurable threads, or an allocation server that
grants bandwidth to applications via IPC, or a multiplexing stream
server like jack, pulse, etc, and so on...

> but also ordering of frames from each application)

Not sure what you mean by this.

> Do you expect all of this to be handled in userspace?

Yes, I do.

Thanks,
Richard

Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers

[Henrik Austad]

On Thu, Sep 07, 2017 at 02:40:18PM +0200, Richard Cochran wrote:
> On Thu, Sep 07, 2017 at 07:34:11AM +0200, Henrik Austad wrote:
> > Also, does this mean that when you create the qdisc, you have locked the 
> > bandwidth for the scheduler? Meaning, if I later want to add another 
> > stream that requires more bandwidth, I have to close all active streams, 
> > reconfigure the qdisc and then restart?
> 
> No, just allocate enough bandwidth to accomodate all of the expected
> streams.  The streams can start and stop at will.

Sure, that'll work.

And if you want to this driver to act as a bridge, how do you accomodate 
change in network requirements? (i.e. how does this work with switchdev?)
- Or am I overthinking this?

> > So my understanding of all of this is that you configure the *total* 
> > bandwith for each class when you load the qdisc and then let userspace 
> > handle the rest. Is this correct?
> 
> Nothing wrong with that.

Didn't mean to say it was wrong, just making sure I've understood the 
concept.

> > In my view, it would be nice if the qdisc had some notion about streams so 
> > that you could create a stream, feed frames to it and let the driver pace 
> > them out. (The fewer you queue, the shorter the delay). This will also 
> > allow you to enforce per-stream bandwidth restrictions. I don't see how you 
> > can do this here unless you want to do this in userspace.
> > 
> > Do you have any plans for adding support for multiplexing streams? If you 
> > have multiple streams, how do you enforce that one stream does not eat into 
> > the bandwidth of another stream? AFAIK, this is something the network must 
> > enforce, but I see no option of doing som here.
> 
> Please, lets keep this simple. 

Simple is always good

> Today we have exactly zero user space
> applications using this kind of bandwidth reservation.  The case of
> wanting the kernel to police individual stream usage does not exist,
> and probably never will.

That we have *zero* userspace applications today is probably related to the 
fact that we have exacatly *zero* drivers in the kernel that talks TSN :)

To rephrase a bit, what I'm worried about:

If you have more than 1 application in userspace that wants to send data 
using this scheduler, how do you ensure fair transmission of frames? (both 
how much bandwidth they use, but also ordering of frames from each 
application) Do you expect all of this to be handled in userspace?

> For serious TSN use cases, the bandwidth needed by each system and
> indeed the entire network will be engineered, and we can reasonably
> expect applications to cooperate in this regard.

yes.. that'll happen ;)

> Thanks,
> Richard

Don't get me wrong, I think it is great that others are working on this!
I'm just trying to fully understand the thought that have gone into this 
and how it is inteded to be used.

I'll get busy testing the code and wrapping my head around the different 
parameters.

-- 
Henrik Austad


signature.asc
Description: PGP signature

Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers

[Richard Cochran]

On Thu, Sep 07, 2017 at 07:34:11AM +0200, Henrik Austad wrote:
> Also, does this mean that when you create the qdisc, you have locked the 
> bandwidth for the scheduler? Meaning, if I later want to add another 
> stream that requires more bandwidth, I have to close all active streams, 
> reconfigure the qdisc and then restart?

No, just allocate enough bandwidth to accomodate all of the expected
streams.  The streams can start and stop at will.

> So my understanding of all of this is that you configure the *total* 
> bandwith for each class when you load the qdisc and then let userspace 
> handle the rest. Is this correct?

Nothing wrong with that.
 
> In my view, it would be nice if the qdisc had some notion about streams so 
> that you could create a stream, feed frames to it and let the driver pace 
> them out. (The fewer you queue, the shorter the delay). This will also 
> allow you to enforce per-stream bandwidth restrictions. I don't see how you 
> can do this here unless you want to do this in userspace.
> 
> Do you have any plans for adding support for multiplexing streams? If you 
> have multiple streams, how do you enforce that one stream does not eat into 
> the bandwidth of another stream? AFAIK, this is something the network must 
> enforce, but I see no option of doing som here.

Please, lets keep this simple.  Today we have exactly zero user space
applications using this kind of bandwidth reservation.  The case of
wanting the kernel to police individual stream usage does not exist,
and probably never will.

For serious TSN use cases, the bandwidth needed by each system and
indeed the entire network will be engineered, and we can reasonably
expect applications to cooperate in this regard.

Thanks,
Richard

Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers

[Henrik Austad]

On Thu, Aug 31, 2017 at 06:26:20PM -0700, Vinicius Costa Gomes wrote:
> Hi,
> 
> This patchset is an RFC on a proposal of how the Traffic Control subsystem can
> be used to offload the configuration of traffic shapers into network devices
> that provide support for them in HW. Our goal here is to start upstreaming
> support for features related to the Time-Sensitive Networking (TSN) set of
> standards into the kernel.

Nice to see that others are working on this as well! :)

A short disclaimer; I'm pretty much anchored in the view "linux is the 
end-station in a TSN domain", is this your approach as well, or are you 
looking at this driver to be used in bridges as well? (because that will 
affect the comments on time-aware shaper and frame preemption)

Yet another disclaimer; I am not a linux networking subsystem expert. Not 
by a long shot! There are black magic happening in the internals of the 
networking subsystem that I am not even aware of. So if something I say or 
ask does not make sense _at_all_, that's probably why..

I do know a tiny bit about TSN though, and I have been messing around 
with it for a little while, hence my comments below

> As part of this work, we've assessed previous public discussions related to 
> TSN
> enabling: patches from Henrik Austad (Cisco), the presentation from Eric Mann
> at Linux Plumbers 2012, patches from Gangfeng Huang (National Instruments) and
> the current state of the OpenAVNU project (https://github.com/AVnu/OpenAvnu/).

/me eyes Cc ;p

> Overview
> 
> 
> Time-sensitive Networking (TSN) is a set of standards that aim to address
> resources availability for providing bandwidth reservation and bounded latency
> on Ethernet based LANs. The proposal described here aims to cover mainly what 
> is
> needed to enable the following standards: 802.1Qat, 802.1Qav, 802.1Qbv and
> 802.1Qbu.
> 
> The initial target of this work is the Intel i210 NIC, but other controllers'
> datasheet were also taken into account, like the Renesas RZ/A1H RZ/A1M group 
> and
> the Synopsis DesignWare Ethernet QoS controller.

NXP has a TSN aware chip on the i.MX7 sabre board as well 

> Proposal
> 
> 
> Feature-wise, what is covered here are configuration interfaces for HW
> implementations of the Credit-Based shaper (CBS, 802.1Qav), Time-Aware shaper
> (802.1Qbv) and Frame Preemption (802.1Qbu). CBS is a per-queue shaper, while
> Qbv and Qbu must be configured per port, with the configuration covering all
> queues. Given that these features are related to traffic shaping, and that the
> traffic control subsystem already provides a queueing discipline that offloads
> config into the device driver (i.e. mqprio), designing new qdiscs for the
> specific purpose of offloading the config for each shaper seemed like a good
> fit.

just to be clear, you register sch_cbs as a subclass to mqprio, not as a 
root class?

> For steering traffic into the correct queues, we use the socket option
> SO_PRIORITY and then a mechanism to map priority to traffic classes / Tx 
> queues.
> The qdisc mqprio is currently used in our tests.

Right, fair enough, I'd prefer the TSN qdisc to be the root-device and 
rather have mqprio for high priority traffic and another for 'everything 
else'', but this would work too. This is not that relevant at this stage I 
guess :)

> As for the shapers config interface:
> 
>  * CBS (802.1Qav)
> 
>This patchset is proposing a new qdisc called 'cbs'. Its 'tc' cmd line is:
>$ tc qdisc add dev IFACE parent ID cbs locredit N hicredit M sendslope S \
>  idleslope I

So this confuses me a bit, why specify sendSlope?

sendSlope = portTransmitRate - idleSlope

and portTransmitRate is the speed of the MAC (which you get from the 
driver). Adding sendSlope here is just redundant I think.

Also, does this mean that when you create the qdisc, you have locked the 
bandwidth for the scheduler? Meaning, if I later want to add another 
stream that requires more bandwidth, I have to close all active streams, 
reconfigure the qdisc and then restart?

>Note that the parameters for this qdisc are the ones defined by the
>802.1Q-2014 spec, so no hardware specific functionality is exposed here.

You do need to know if the link is brought up as 100 or 1000 though - which 
the driver already knows.

>  * Time-aware shaper (802.1Qbv):
> 
>The idea we are currently exploring is to add a "time-aware", priority 
> based
>qdisc, that also exposes the Tx queues available and provides a mechanism 
> for
>mapping priority <-> traffic class <-> Tx queues in a similar fashion as
>mqprio. We are calling this qdisc 'taprio', and its 'tc' cmd line would be:

As far as I know, this is not supported by i210, and if time-aware shaping 
is enabled in the network - you'll be queued on a bridge until the window 
opens as time-aware shaping is enforced on the tx-port and not on rx. Is 
this required in this driver?

>$ $ tc qdisc add dev ens4 parent root 

Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers

[Richard Cochran]

On Fri, Sep 01, 2017 at 09:12:17AM -0700, Jesus Sanchez-Palencia wrote:
> On 09/01/2017 06:03 AM, Richard Cochran wrote:
> > The timing of this RFC is good, as I am just finishing up an RFC that
> > implements time-based transmit using the i210.  I'll try and get that
> > out ASAP.

I have an RFC series ready for net-next, but the the merge window just
started.  I'll post it when the window closes again...

Thanks,
Richard

Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers

[Richard Cochran]

On Fri, Sep 01, 2017 at 09:12:17AM -0700, Jesus Sanchez-Palencia wrote:
> Is it correct to assume you are referring to an interface for Launchtime here?

Yes.

Thanks,
Richard

Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers

[Jesus Sanchez-Palencia]

Hi Richard,


On 09/01/2017 06:03 AM, Richard Cochran wrote:
> 
> I happy to see this posted.  At first glance, it seems like a step in
> the right direction.
> 
> On Thu, Aug 31, 2017 at 06:26:20PM -0700, Vinicius Costa Gomes wrote:
>>  * Time-aware shaper (802.1Qbv):
> ...
>>S 0x01 300
>>S 0x03 500
>>
>>This means that there are two intervals, the first will have the gate
>>for traffic class 0 open for 300 nanoseconds, the second will have
>>both traffic classes open for 500 nanoseconds.
> 
> The i210 doesn't support this in HW, or does it?


No, it does not. i210 only provides support for a per-packet feature called
LaunchTime that can be used control both the fetch and the transmission time of
packets.


> 
>>  * Frame Preemption (802.1Qbu):
>>
>>To control even further the latency, it may prove useful to signal which
>>traffic classes are marked as preemptable. For that, 'taprio' provides the
>>preemption command so you set each traffic class as preemptable or not:
>>
>>$ tc qdisc (...) \
>> preemption 0 1 1 1
> 
> Neither can the i210 preempt frames, or what am I missing?

No, it does not.

But when we started working on the shapers we decided to look ahead and try to
come up with interfaces that could cover beyond 802.1Qav. These are just some
ideas we've been prototyping here together with the 'cbs' qdisc.


> 
> The timing of this RFC is good, as I am just finishing up an RFC that
> implements time-based transmit using the i210.  I'll try and get that
> out ASAP.


Is it correct to assume you are referring to an interface for Launchtime here?


Thanks,
Jesus

Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers

[Richard Cochran]

I happy to see this posted.  At first glance, it seems like a step in
the right direction.

On Thu, Aug 31, 2017 at 06:26:20PM -0700, Vinicius Costa Gomes wrote:
>  * Time-aware shaper (802.1Qbv):
...
>S 0x01 300
>S 0x03 500
> 
>This means that there are two intervals, the first will have the gate
>for traffic class 0 open for 300 nanoseconds, the second will have
>both traffic classes open for 500 nanoseconds.

The i210 doesn't support this in HW, or does it?

>  * Frame Preemption (802.1Qbu):
> 
>To control even further the latency, it may prove useful to signal which
>traffic classes are marked as preemptable. For that, 'taprio' provides the
>preemption command so you set each traffic class as preemptable or not:
> 
>$ tc qdisc (...) \
> preemption 0 1 1 1

Neither can the i210 preempt frames, or what am I missing?

The timing of this RFC is good, as I am just finishing up an RFC that
implements time-based transmit using the i210.  I'll try and get that
out ASAP.

Thanks,
Richard

[RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers

[Vinicius Costa Gomes]

Hi,

This patchset is an RFC on a proposal of how the Traffic Control subsystem can
be used to offload the configuration of traffic shapers into network devices
that provide support for them in HW. Our goal here is to start upstreaming
support for features related to the Time-Sensitive Networking (TSN) set of
standards into the kernel.

As part of this work, we've assessed previous public discussions related to TSN
enabling: patches from Henrik Austad (Cisco), the presentation from Eric Mann
at Linux Plumbers 2012, patches from Gangfeng Huang (National Instruments) and
the current state of the OpenAVNU project (https://github.com/AVnu/OpenAvnu/).

Please note that the patches provided as part of this RFC are implementing what
is needed only for 802.1Qav (FQTSS) only, but we'd like to take advantage of
this discussion and share our WIP ideas for the 802.1Qbv and 802.1Qbu interfaces
as well. The current patches are only providing support for HW offload of the
configs.


Overview


Time-sensitive Networking (TSN) is a set of standards that aim to address
resources availability for providing bandwidth reservation and bounded latency
on Ethernet based LANs. The proposal described here aims to cover mainly what is
needed to enable the following standards: 802.1Qat, 802.1Qav, 802.1Qbv and
802.1Qbu.

The initial target of this work is the Intel i210 NIC, but other controllers'
datasheet were also taken into account, like the Renesas RZ/A1H RZ/A1M group and
the Synopsis DesignWare Ethernet QoS controller.


Proposal


Feature-wise, what is covered here are configuration interfaces for HW
implementations of the Credit-Based shaper (CBS, 802.1Qav), Time-Aware shaper
(802.1Qbv) and Frame Preemption (802.1Qbu). CBS is a per-queue shaper, while
Qbv and Qbu must be configured per port, with the configuration covering all
queues. Given that these features are related to traffic shaping, and that the
traffic control subsystem already provides a queueing discipline that offloads
config into the device driver (i.e. mqprio), designing new qdiscs for the
specific purpose of offloading the config for each shaper seemed like a good
fit.

For steering traffic into the correct queues, we use the socket option
SO_PRIORITY and then a mechanism to map priority to traffic classes / Tx queues.
The qdisc mqprio is currently used in our tests.

As for the shapers config interface:

 * CBS (802.1Qav)

   This patchset is proposing a new qdisc called 'cbs'. Its 'tc' cmd line is:
   $ tc qdisc add dev IFACE parent ID cbs locredit N hicredit M sendslope S \
 idleslope I

   Note that the parameters for this qdisc are the ones defined by the
   802.1Q-2014 spec, so no hardware specific functionality is exposed here.


 * Time-aware shaper (802.1Qbv):

   The idea we are currently exploring is to add a "time-aware", priority based
   qdisc, that also exposes the Tx queues available and provides a mechanism for
   mapping priority <-> traffic class <-> Tx queues in a similar fashion as
   mqprio. We are calling this qdisc 'taprio', and its 'tc' cmd line would be:

   $ $ tc qdisc add dev ens4 parent root handle 100 taprio num_tc 4\
   map 2 2 1 0 3 3 3 3 3 3 3 3 3 3 3 3 \
   queues 0 1 2 3  \
   sched-file gates.sched [base-time ]   \
   [cycle-time ] [extension-time ]

is multi-line, with each line being of the following format:
 

   Qbv only defines one : "S" for 'SetGates'

   For example:

   S 0x01 300
   S 0x03 500

   This means that there are two intervals, the first will have the gate
   for traffic class 0 open for 300 nanoseconds, the second will have
   both traffic classes open for 500 nanoseconds.

   Additionally, an option to set just one entry of the gate control list will
   also be provided by 'taprio':

   $ tc qdisc (...) \
sched-row  \
[base-time ] [cycle-time ] \
[extension-time ]


 * Frame Preemption (802.1Qbu):

   To control even further the latency, it may prove useful to signal which
   traffic classes are marked as preemptable. For that, 'taprio' provides the
   preemption command so you set each traffic class as preemptable or not:

   $ tc qdisc (...) \
preemption 0 1 1 1


 * Time-aware shaper + Preemption:

   As an example of how Qbv and Qbu can be used together, we may specify
   both the schedule and the preempt-mask, and this way we may also
   specify the Set-Gates-and-Hold and Set-Gates-and-Release commands as
   specified in the Qbu spec:

   $ tc qdisc add dev ens4 parent root handle 100 taprio num_tc 4 \
   map 2 2 1 0 3 3 3 3 3 3 3 3 3 3 3 3\
   queues 0 1 2 3 \
   preemption 0 1 1 1 \
   sched-file preempt_gates.sched

 is multi-line, with each line being of the following format: