Narayan,

Thanks for the response. A general comment before diving into the  
details:

In general it seems that the issues being discussed are related to  
either (1) the port of the existing LDOM functionality to the updated  
MAC client interfaces introduced by Crossbow, or (2) the addition of  
new LDOM functionality based on these new APIs.

It's critical that we resolve the issues falling into the first  
category as soon as possible. This will allow the port of LDOM to the  
new APIs to start as soon as possible, maintain LDOM functionality,  
and allow Crossbow to meet its integration schedule. For the class of  
issues related to future LDOM functionality, this discussion will  
have to take place with a clear understanding of the new functional  
requirements.

So given my latest response below, what are in your opinion the  
remaining issues which are directly related to the port of the  
existing LDOM functionality to the new MAC API?

On Oct 11, 2007, at 3:13 PM, Narayan Venkat wrote:

> Hi Nicolas
>
> Sorry for the late response. Please see some more questions/ 
> comments below ..
>
> <..snip..>
>
>>
>>>
>>>     Q1.2) On pg 38, there is a reference to the following flags,  
>>> but which
>>>          interface takes them as an argument?
>>>
>>>      MAC_OPEN_FLAGS_FORCE_MULTI_RINGS
>>>      MAC_OPEN_FLAGS_FORCE_ONE_RING
>>>
>>>      It seems like these are an argument to mac_client_open(),
>>>      but there is a reference mac_open() in the description see below:
>>>
>>>      "If MAC_OPEN_FLAGS_FORCE_MULTI_RINGS flag is set and it is not
>>>       possible to allocate mbc_ncpus hardware rings, the mac_open()
>>>       call will fail, otherwise the MAC layer will attempt to reserve
>>>       one hardware ring for the MAC client."
>>
>> These flags are specified when calling mac_client_open(), not  
>> mac_open().
>
> I guess the above text will get fixed in a subsequent revision.

Yes.

>
>
>>>
>>>     Q1.3) Are there any other flags other than the following ones?
>>>
>>>      MAC_OPEN_FLAGS_FORCE_MULTI_RINGS
>>>      MAC_OPEN_FLAGS_FORCE_ONE_RING
>>
>> No.
>
> Is there a reason this is tied to hardware rings. We would like the  
> mac client
> open request be extended so that it can get either all software  
> rings, a mix of
> hardware and software or HW rings in order to match the number of  
> cpus specified
> in the client_open call .. A flag can be specified for this ..

Why do you want to do this? What is the functional requirement here?

>
> Also according to the explanation in the doc at page 38, there is
> also a case where no flags is specified. It seems like, if no flags
> specified, then it will attempt to reserve one hardware ring.
> It seems to not to fail even if such reservation fails, but it is not
> clearly specified.

If you don't specify the ONE_RING flag, and a hardware ring cannot be  
reserved for the MAC client, then the MAC client will share the  
default ring with other MAC clients.

>
>>>  - Is there a way to force a software ring?
>>
>> Do you mean not assign a hardware ring? I think this is something  
>> we could add, yes.
>
> This is related to the above. Can you add a flag that we can use
> to indicate that client wants to use a single ring or multiple rings
> but not force hardware rings. That way even when the underlying
> device does not have enough hardware rings, a client can get a soft
> ring per CPU.

How is that different than the currently documented behavior?

> Above comment applies to this one also. The behavior without any flags
> seem to attempt to reserve one h/w ring. What is the failure case ?

Without any flag, we try to allocate N hardware rings. If that fails,  
then we try to allocate 1 hardware ring and do fanout to N soft  
rings, if that fails, then we share the default hardware ring, and do  
fanout to N soft rings.

>
>>>
>>>     Q1.4) Is the mbc_cpus in mac_bind_cpus_t an array of CPU ids?
>>
>> Yes.
>>
>>>
>>>     Q1.5) The following description of mbc_cpus on pg 37 is not  
>>> clear,
>>>          especially for the non-NULL case.
>>>
>>>              "If mbc_cpus is NULL, the MAC layer will pick the CPUs.
>>>      If mbc_cpus is non-NULL, the MAC layer will chose the CPUs.".
>>
>> The first one is correct. If mbc_cpus is non-NULL, the MAC layer  
>> will assign the CPUs provided by the caller.
>
> When the mbc_cpus is NULL what determines how many CPUs and hence
> the number of rings available to this client.

mbc_ncpus.

>
>>
>>>
>>>     Q1.6) What is the relationship between Unicast addresses 
>>> (multiple
>>>          unicast set via mac_unicast_add()), Rings and CPUs?
>>>             
>>>              - Is there a 1:1 relation between a unicast address and a  
>>> ring?
>>>      - Is there a 1:1 relation between a ring and CPU?
>>
>> Neither. The MAC addresses will share the same rings and CPUs.
>
> But since you are allowing multiple mac addresses to be associated
> with a client, can we add support as part of the unicast_add call
> to indicate that each of these addresses should be associated with
> ring (either HW or SW).

No, each client is associated with a group of hardware rings or soft  
rings. Each group or set of rings corresponds to a set of unicast MAC  
addresses. The bandwidth limits are set on a per MAC client basis.  
This maps to how hardware NICs do their classification and fanout.

If you want a separate set of rings for different MAC addresses, then  
you create a new MAC client.

>>>         -  The Rings and CPUs are tightly coupled in this interface.
>>>        How can allocate multiple rings even when there is one CPU(or  
>>> less
>>>        number of CPUs).
>>
>> You don't allocate rings explicitly, you express a level of  
>> parallelism instead, the framework distributes the hardware rings  
>> transparently.
>
> But the only way we can control this parallelism is by specifying
> the number of CPUs in the domain. But in a system capable of adding
> and removing CPUs dynamically, we might want to change the parallelism
> level too. The current APIs dont allow changing this. We will need
> a way to specify this as an extension to the client_open or a via
> a new API call.

So you want an API which allows you to change the actual  
mac_bind_cpu_t for a client which has been already opened? I think we  
can do that.

>
> Also the document states that if mbc_ncpus HW rings the open will  
> fail.
> As I mentioned earlier it would be nice if we can get software rings
> in this case.

See Q1.3 above, you can fallback to using software rings depending on  
the flags set.

> Also, in terms of parallelism is this specified by the no. of CPUs
> or by unique CPUIDs in the array. What happens if I specify ncpus
> where all IDs are the same - do I get ncpus HW rings if they are
> available. Also can we then change the ring to cpu mapping when
> more CPUs are added/removed to/from the domain ?

There should be no duplicate CPU ids in the array.

>
>>
>>>      - When there are multiple CPUs and multiple unicast addresses,
>>>        is there address fanout per CPU?
>>
>> See 2 answers above.
>
> This will be a very useful feature as it will allow clients to  
> associate
> each ring with a mac address. Currently the only way to do this is
> to do separate mac_client_open calls associate it with a ring and then
> bind it to a mac address.

Yes, you have to do a mac_client_open() if you want to assign a  
different set of rings to a MAC address, see also above.

>>>     Q1.7) How is the binding of CPUs via mac_bind_cpus_t is co- 
>>> ordinated
>>>          with CPU DR(on the platforms that support them)?
>>
>> The MAC layer will be notified of the removal of the CPU and will  
>> stop using it for its worker threads and interrupts.
>
> That is purely error handling. We need the ability to be able to use
> more CPUs and improve the level of parallelism when CPUs are added.
> The reverse is true when the CPUs are removed. When the MAC layer is
> notified about CPUs going away does it remove the rings associated
> with the CPUs ?

I was not talking specifically about error handling. If the MAC layer  
bound a ring worker thread or interrupt to a CPU and that CPU is  
going away, the MAC layer will move that thread or interrupt to a  
different CPU.

The API discussed in Q1.6 above would allow a MAC client to increase  
the number of CPUs if it detects that CPUs were added to the system.

<snip>

>
>>>
>>>     Q1.10) Can the mac client interface be extended to support  
>>> creating
>>>            a client based on ether_type? This is required for mac  
>>> clients
>>>        like fiberchannel over ethernet.
>>
>> No, each MAC client corresponds to a MAC level entity which is  
>> defined by its MAC address. Multiple ether types can be supported  
>> on top of a MAC client.
>
> Devices like the Niagara2 NIU allow classification of packets using
> parameters like the ether_type. How can a mac_client take advantage
> of such a functionality.

The fact that a particular hardware implementation can do  
classification on a specific header field of a packet doesn't  
necessarily mean that a MAC client needs to be associated with that  
field.

Today the SAP demultiplexing is done by DLS on top of MAC clients. At  
some point in the future we may make use of hardware classification  
to offload that demultiplexing, but can be done at a level above the  
MAC layer, and maintain the separation between MAC clients and what  
defines them (MAC addresses and VLANs), from SAP demultiplexing.

>>>
>>>     Q2.1) The section 4.5 describes "By value" type which is used
>>>          to set a specific MAC address by the MAC client. But there
>>>          is no equivalent addr_type definition under mac_unicast_add()
>>>          interface.
>>
>> MAC_UNICAST_VALUE is missing from the list, this is what you are  
>> looking for.
>
> I presume this will be documented in the next revision.

Correct.

>>>
>>>     Q2.2) Is there an impact to the multiaddress_capab_t.maddr_add 
>>> ()/
>>>          maddr_remove() interfaces? Are these being obsoleted or
>>>          going away?
>>
>> The capability will stay, and the framework will continue to use  
>> that capability to query and control the allocation of MAC address  
>> slots. However that interface is not intended to be used by  
>> drivers which should use the MAC client interfaces instead.
>
> OK.

Since my last reply Kais and Roamer have been working on the design  
for the new driver interface. Their proposal removes the multiple MAC  
address capability as it is known today. You should read their design  
document, which is available at http://www.opensolaris.org/os/project/ 
crossbow/Docs/virtual_resources.pdf

>>>
>>>     Q2.3) A system with many domains (aka LDoms) with virtual network
>>>               devices, it requires the use of a large number  
>>> layer2 addresses,
>>>           this will exhaust h/w slots available on most standard NICs.
>>>           How can a client take advantage of layer2 filtering  
>>> provided by
>>>           NICs like NII-NIU/Neptune. Specifically, this will help in
>>>               avoiding the programming of the device into  
>>> PROMISCous mode
>>>               etc.  Currently there are no interfaces that seem  
>>> to provide
>>>               such ability.
>>
>> Yes, this is a situation we are aware of. We've talked on this  
>> list about having multiple VNICs sharing the same MAC address, and  
>> identified by their IP address instead. However this needs to be  
>> scoped and defined further before we can commit on providing that  
>> functionality.
>>
>
> The current APIs only allow adding as many addresses as the
> number of slots available. Following this it will put the adapter
> in promisc mode. Instead can you add the capability to specify
> when to use a filter and when to take up a slot in the HW.

Do you mean that you want to be able to specify that a mac_unicast_add 
() should put the NIC in promiscuous mode even though there are MAC  
address slots available? What is the use case for this?

<snip>

>>>
>>>     Q2.6) Can it be assumed that every address added to a client is
>>>              processed in a separate ring (either h/w ring or s/w  
>>> ring)?
>>
>> No, all the MAC addresses for a client will share the same ring 
>> (s). If there's a need to have a different set of rings associated  
>> with a MAC address, then a different MAC client should be created.
>
> What happens when a single client has multiple rings and multiple
> mac addresses. How is the mapping done in that case ? Would it
> be possible to in that case request a 1-to-1 mapping and reserve
> a ring for each address ?

If you need a 1-1 mapping between a MAC address and a hardware ring,  
then you need to use multiple MAC clients, and assign a ring and a  
unicast address to each client.

>>>     Q2.7) How are the multiple addresses per client maintained, is  
>>> it done
>>>          in the MAC layer or does it bybpass the MAC layer and passed
>>>          to h/w directly.
>>
>> Since the action of reserving the MAC address is triggered by a  
>> call to the MAC layer, the MAC layer cannot be bypassed. The MAC  
>> layer will use the multiple MAC address capability exposed by the  
>> driver to reserve a new MAC address slot.
>
> What if the driver does not expose that capability. Will the  
> unicast_add
> call fail ? Is the MAC layer essentially reflecting the capability of
> the underlying hardware or does it provide the ability to have  
> multiple
> addresses irrespective of whether the HW has multiple slots or not ?

The request will still succeed if the number of MAC address slots is  
exhausted, or if the underlying NIC doesn't support the multiple MAC  
address capability. However in these cases, the MAC layer will  
transparently put the NIC in promiscuous mode in order to receive  
traffic for that new MAC unicast address.

>>>     Q2.8) Can unlimited number of mac addresses be assigned to a MAC
>>>          client? What are the software/hardware features that limit  
>>> this?
>>
>> Memory that can be allocated by the kernel.
>
> So even if the underlying device runs out of slots the MAC layer will
> maintain all the addresses associated with that client. How does it  
> then
> manage and associate these addresses with the rings allocated for
> this client ? What does it do in both software and hardware to
> filter the addresses for this client ? Also which addresses get HW  
> slots
> and which dont ? Also if you run out of slots does the HW go to  
> promisc mode ..

Each MAC client is associated with a group of rings. Each group of  
rings is therefore associated with a set of MAC addresses. If a  
client needs to be associated with more than one MAC address, then  
the corresponding group needs to be associated with the same set of  
addresses. If the hardware runs out of MAC addresses, then the NIC is  
put in promiscuous mode. The allocation of slots is on a first come  
first served basis.

>>> 3) Rings related:
>>>     (Crossbow-virt.pdf  Section 5.3 Pg 43)
>>>       mac_rint_t *mac_rings_list_get(mac_client_handle_t mch,
>>>               uint_t nrings);
>>>       void mac_rings_list_free(mac_rings_t *rings, uint_t nrings);
>>>       uint16_t mac_ring_get_flags(mac_ring_t ring);
>>>
>>>
>>>     QUESTIONS:
>>>
>>>             Q3.1) All of these interfaces are now categorized as project- 
>>> private
>>>           API. What motivated this change. These interfaces need to be
>>>               more open.
>>
>> The MAC layer will do the allocation of hardware resources to the  
>> various MAC clients and their flows. Instead of having each MAC  
>> client manage its own set of resources, the resources are  
>> allocated to MAC clients based on their needs, for example the  
>> degree of parallelism expressed through mac_client_open(). If you  
>> have specific functional requirements that are not satisfied by  
>> the current document, please list them.
>
> Currently rings are hidden resources entirely managed by the mac layer
> and clients have no visiblity. All the client gets to do is request a
> degree of parallelism. Providing APIs that allow clients to see how
> rings were allocated will be useful.

Why? What is the functional requirement?

>>>         Q3.2) The mac_rings_list_get() is only  for h/w rings, is  
>>> there
>>>          an equivalent interface to obtain s/w ring information.
>>>          Or this interface can be extended return both h/w ring
>>>          or s/w ring information.
>>
>> The interface will evolve to provide that information, but it will  
>> remain project private. It is provided here FYI but will change in  
>> future revisions of the document.
>
> So the expectation is that ring APIs should not be used by clients  
> and it
> is only an internal MAC layer resource managed by it ?

Yes, the MAC layer does the allocation of resources to MAC clients  
and their flows.

>>>             Q3.3) Are the mac_resource_set() and mac_resources() interfaces
>>>          going away?        
>>
>> Yes, they will be replaced by different interfaces. But note that  
>> they are already project private in Nevada and were not supposed  
>> to be used by other ON components.
>
> Agreed, but there is no other way in the new Crossbow API a way
> to take advantage of multiple rings. There is one generic RX callback,
> but no other way to associate a callback with a specific ring. This
> is a limitation of the existing API and support should be added to
> open API list, so that we can process traffic independently. So having
> some additional APIs to expose this will be very useful.

But you can process the traffic independently since packets will be  
sent up to the MAC client concurrently for the multiple rings  
associated with the client. What else do you need to do here  
specifically that is prevented by the API?

>>>     Q3.4) What is the action taken when no free h/w ring available.
>>>          As per the documentation of mac_rings_list_get(), if no h/w
>>>          ring available, it returns NULL. In such case, how does
>>>          mac_unicast_add() behave when NULL is passed for rings?
>>
>> mac_unicast_add() no longer takes rings. This will be handled  
>> transparently to the MAC clients by using a default ring and  
>> falling back to software classification.
>
> So when there are multiple rings -- and these rings are associated
> with all the addresses - there is no pre-defined mapping ? That can
> be inefficient as the hardware has the ability to associate each ring
> with an address. Can we extend the MAC apis to allow a client to  
> choose
> between the default behavior and binding addresses to rings ?

I think I already answered this question multiple times above.

>>>     Q3.5) Are there any interfaces other than the above mac_rings_xxx
>>>          interfaces that are available to deal with MAC rings?
>>
>> Not available to MAC clients. The set of project private  
>> interfaces might evolve as we refine the design.
>
> I would like to see some of the functionality exported via these
> private mac interfaces promoted to a open API. Even if the API
> cannot be moved over, can we extend APIs to provide hints ..

Again, what is the functional requirement? What functionality do you  
need to provide with this information? What hints do you think are  
missing from the API?

>>>     Q3.6) Is the mac_rings_list_get() returns the list of mac rings
>>>          assigned to the client at the time of client open. How can
>>>          this be changed after the client is open.
>>
>> The set of assigned rings may change. The details on the APIs  
>> needed to support this still need to be defined, but they will  
>> remain project private.
>
> So you are saying there is no way to rely on how many rings are
> available to a particular client. This will change without the
> client's control ? Is CPUs being removed from the system a case
> under which this will happen ?

The flags taken by mac_client_open() allows some control by the MAC  
client, see Q1.3. If the client specified that a given CPU be  
assigned to the client, we could block the DR'ing out the CPU until  
the MAC client releases that CPU, what is your requirement here?

>>>     Q3.7) Assigning h/w rings to a specific MAC address limits the
>>>          bandwidth to the number of rings that are assigned to that
>>>          address. Is there a way to not to bind h/w rings specific
>>>          to MAC address so that the bandwidth could be used by
>>>          any mac client depending on the traffic?
>>
>> See Q1.3.
>
> Not sure what you mean. Are you suggesting that some mac addresses
> will have SW rings and others will be associated to HW rings ?

Between different MAC clients, that's possible. But within the same  
MAC client, all unicast addresses of that client will share the same  
group of hardware rings or SRS.

>>> 4) Receive callback related:
>>>     (Crossbow-virt.pdf  Section 5.2.5 Pg 40)
>>>     int mac_rx_set(mac_client_handle_t mch, mac_rx_fn_t rx_fn,
>>>         void *arg);
>>>     int mac_rx_clear(mac_client_handle_t mch);
>>>
>>>     QUESTIONS:
>>>
>>>             Q4.1) How can a client get rx callback  per ring that is  
>>> assigned
>>>          to the mac client? This will allow parallel processing
>>>          and improve the performance. Such a feature is already
>>>          being used in the current implementation of LDoms vSwitch
>>>          driver and the mac_xxx interfaces should support such an
>>>              ability.
>>
>> The parallel processing will still happen. I.e. if multiple  
>> hardware rings or software rings are assigned to a MAC clients,  
>> multiple connections associated with that MAC client will be  
>> spread across these rings.
>
> So with multiple rings, there will be concurrent callbacks to the
> rx_fn, each with packets in the corresponding ring ? Also will each
> callback be able to determine what ring did the callback ?

Why do you need that information, what is your functional requirement?

>>>     Q4.2) How can a client get a separate callback for a defined  
>>> type of
>>>          traffic, such as different SAP numbers etc. This will
>>>          be useful to provide out of the band type packet processing
>>>              or related services.
>>
>> This will be supported by a MAC flow API built on top of the MAC  
>> client API. The flow API will be described by a separate document.
>
> So if a client wants to use the flow API will it need to layer  
> itself on
> the flow API and not the mac client API directly. Can you give me more
> information on what this layering will look like. Also, when do you
> expect the flow API doc to be available ?

The flow API will be an addition to the MAC client API. A MAC client  
will be able to use that flow API. Such a flow operation would be of  
the form mac_flow_xxx(mac_client_handle_t mch, <flow description>,  
<bandwidth properties>, etc). Kais is working on defining that API,  
I'll let him comment on expected availability.

>>>
>>> 5) Transmit related:
>>>
>>>     (Crossbow-virt.pdf  Section 5.2.7 Pg 41)
>>>     mblk_t *mac_tx(mac_client_handle_t *mch, mblk_t *mp, uint64_t  
>>> hint);
>>>
>>>    QUESTIONS:
>>>
>>>     Q5.1) What are the valid values for the 'hint' argument?
>>>           From the description on pg 42, NULL seems to be
>>>           a valid value. Is it safe to assume that the 'hint' is a
>>>               ring-id, if so, a NULL value of 0 will conflict with a
>>>               ring-id of 0.
>>
>> The hint can be any 64 bit value, but it must always be the same  
>> value for the packets corresponding to the same connection to  
>> avoid reordering. TCP and UDP for example pass the connection  
>> pointer as the hint, which allows us to avoid packet inspection  
>> for these protocols.
>
> Can you clarify what the hint is being used for. Is it similar
> to the case below -- where a hash will be applied on hint value
> to pick up a TX ring ?

Basically yes, a hash function is applied to the hint to select the  
outbound TX ring.

>>>     Q5.2) If NULL specified as a 'hint', how is the tx ring
>>>           selected?
>>
>> In this case mac_tx() will parse the packet headers and hash on  
>> the header information to select a transmit ring.
>>
>
> Is the goal here to somehow bifurcate traffic being sent by a client
> via the interface ?

The goal is to spread the traffic among the transmit rings assigned  
to the client while maintaining packet ordering for individual  
connections, without exposing the details of assignment of transmit  
rings to MAC clients.

> The algorithm is pre-determined by the mac layer
> and either hint or misc headed + hash will be used to determine the
> Tx ring for transmit ?

Yes.

> It is possible for a client to request a specific
> ring -- is the only way to do this is pick a unique hint ?

No, this is done transparently to the MAC client, which doesn't need  
to know which transmit rings are assigned to it.

<snip>

>>
>>> 6) Multicast addresses related:
>>>     (Crossbow-virt.pdf  Section 5.2.6 Pg 41)
>>>     int mac_multicast_add(mac_client_handle_t mch, const uint8_t  
>>> *addr);
>>>     int mac_multicast_remove(mac_client_handle_t mch, const uint8_t  
>>> *addr);
>>>
>>>
>>>         No comments at this point.
>>>
>>> 7) Promiscous mode realted:
>>>
>>>     (Crossbow-virt.pdf  Section 5.2.8 Pg 42)
>>>     Its not clear if the above interface will be available or not,
>>>     but two new intefaces are added:
>>>
>>>     int mac_promisc_add(mac_client_handle_t mch, mac_promisc_type
>>>         promisc_type, mac_promisc_fn_t promisc_fn, void *arg,
>>>         mac_promisc_handle_t *php);
>>>     int mac_promisc_remove(mac_client_handle_t mch,
>>>         mac_promisc_handle_t *ph);
>>>
>>>             MAC_PROMISC_ALL - send all packets
>>>             MAC_PROMISC_MULTI - only broadcast and multicast
>>>
>>>     May be the mac_promisc_add(MAC_PROMISC_ALL) will force device
>>>     to operate in the promiscous mode.
>>
>> Both need to, since the device needs to be in promiscuous mode  
>> also to receive all multicast traffic.
>
> What do mean by "Both need to." ? In addition to above interface
> to enable promisc mode, will the existing promisc_set() interface
> be removed ? Also, PROMISC_ALL is unicast+multicast, whereas MULTI
> is only all multicast traffic ?

Both MAC_PROMISC_* modes will need to cause the device to be put in  
promiscuous mode. The existing mac_promisc_set() interface is being  
removed. Yes, PROMISC_ALL will be for all traffic, MULTI only for  
multicast (and broadcast) traffic.

>
>>
>>>
>>>     QUESTIONS:
>>>
>>>             Q7.1) According to the section 4.6, the promiscuous mode  
>>> operates
>>>          in the layer2 switch model. When choosing the promiscuous mode
>>>              model can it be either layer2 switch model or shared
>>>          ethernet model?
>>
>
> Comments ?

The answer was included with my answer to Q7.2.

>>>     Q7.2) From the explanation of mac_promisc_add(), it seems like
>>>          the mac_promisc_add() could be called without setting
>>>          MAC address via mac_unicast_add(). Is this correct?
>>>          If so, what is the expected behaviour?
>>
>> Currently we provide the same semantics as a switched environment,  
>> i.e. a MAC client will see the same traffic that would be seen by  
>> a NIC connected to a switch.
>>
>
> Is there a way to see only the multicast traffic associated with  
> all mac
> clients - union of all mac_client multicast_add addresses. The MULTI
> promisc option seems more a way to weed to unicast and broadcast  
> traffic
> on the wire and pass all wire multicast traffic up - including ones  
> the
> system may not be interested in ? Is this the case ?

These broadcast flags apply not only to the incoming received traffic  
but also the traffic sent my MAC clients of the same underlying MAC.  
I.e. a MAC client PROMISC_MULTI callback will also see all multicast  
traffic sent by the other MAC clients defined on top of the same MAC.  
In order to preserve the semantics that are implemented by a real  
physical switch, this applies to *all* multicast traffic, not just  
the multicast groups that were "joined" by the individual MAC clients.

>> What we would also like to provide is the ability to for a MAC  
>> client to obtain all the traffic going in and out of the box, as  
>> well as the traffic exchanged between MAC clients. The non-unicast  
>> address was part of that solution.
>>
>
> OK ..
>
>> Another option would be to generalize this with the shared  
>> ethernet model, and allow a MAC client to specify that it wants to  
>> observe all traffic via a separate promiscuous type. I need to see  
>> how this can be added to the API.
>
> This will be very useful. How about something MAC_PROMISC_CLIENTS ?

The new modes will be:

* ALL: all traffic, including all traffic sent by MAC clients and  
traffic seen by the hardware
* FILTERED: all multicast and broadcast traffic, plus the traffic to  
the unicast MAC addresses associated with the MAC client.
* MULTI: all multicast and broadcast traffic received on the wire and  
sent by the MAC clients

>
>>
>>>
>>> 8) Statistics related:
>>>
>>>             Q8.1) Is the mac_stat_get() interface being obsoleted or  
>>> changed?
>>>           If so, what is the new equivalent interface?
>>
>> Yes, there will be a new MAC client interface. The MAC layer will  
>> also maintain per-MAC client statistics for MAC client specific  
>> statistics such as number of packets sent/received, etc. I need to  
>> add that interface to the document.
>
> OK -- when will this be available. Next doc update ? Until that  
> point in time
> should we continue to use the mac_stat_get() interface ?

Yes, this will be documented in the next doc update. You will need to  
use this new interface as part of the port to the new MAC API  
introduced by Crossbow.

>
>>> GENERAL QUESTONS:
>>> ================
>>>
>>>     Qg.1) Are there any GLDv3 MAC client interfaces that are being
>>>           obsoleted(provided by the Nemo framework) but not documented
>>>           in this doc?
>>
>> The MAC client interface was project private, and most of the  
>> interface is being completely revamped by Crossbow. The set of MAC  
>> client API available to ON consolidation components is described  
>> by section 5.2 of the document. Any other MAC client API are still  
>> project private.
>
> So all interfaces that were part of GLDv3 will be replaced with the
> interfaces specified here. Anything that is not specified here should
> not be used moving forward ?

Yes.

>>>     Qg.2) Are there any changes to the MAC driver interfaces or being
>>>          obsoleted?
>>
>> The changes made to the driver API will be published as part of a  
>> separate forthcoming document.
>
> How soon will this doc become available ? I see in the latest doc  
> Kais published
> there are some new interfaces for ring support. I presume there  
> will be a separate
> doc for the deneric mac driver interfaces ?

That's the document. It describes the changes to the driver interface  
that are made by Crossbow.

>>>     Qg.3) There are no MAC client interfaces to specify bandwidth
>>>          attributes. From the section 4.7, it seems like they are
>>>          implemented as part of VNIC and not as MAC client interfaces.
>>>          If this is the case, how can the bandwidth attributes be
>>>              specified?
>>
>> They are not documented yet, but will be specified as arguments to  
>> mac_client_open().
>
> Can we expect to see this in the next revision of this document ?

Yes, I need to formalize the interface and document it. The changes  
will consist of new arguments to the mac_client_open() call  
specifying the bandwidth limit and priority.

>>>     Qg.4) When will the classification interface be fully documented
>>>           for review?
>>
>> There will be separate documents for the MAC driver classification  
>> interfaces, and for the MAC client flow APIs.
>
> When will this be available ?

This was already discussed in Q4.2 and Qg.2.

Thanks,
Nicolas.

-- 
Nicolas Droux - Solaris Core OS - Sun Microsystems, Inc.
droux at sun.com - http://blogs.sun.com/droux




Reply via email to