Erik Nordmark wrote:
Cathy Zhou wrote:
This is a reminder that this design review is still ongoing. Also, I
am extending this design review to Jan. 5, 2006. As before, the
document is available here:
http://opensolaris.org/os/community/networking/uv-design.pdf
A few late comments and questions.
Overall the document is quite good and quite clear, but I think it is
hard to find a good order in which to explain things due to the
interdependencies. I had several questions that came up while I was
reading that were answered before I got to the end of the document.
(I've suggested on how to address some of that in down below.)
High-level
----------
The legacy support seems to assume that everything that is DDI_NT_NET is
a DLPI provider. At least in the past there has been random stuff
labeled as DDI_NT_NET (I forgot what it was; but ifconfig -a plumb tried
to use some strange device in the past). It might make sense to verify
(with a DL_INFO_REQ) that the device is a DLPI provider before softmac
is applied to the device.
Is the intent that the PPP devices (sppp0 etc) appear under softmac?
Vanity naming in combination with VLANs might cause some potential
confusion with snoop. Today if I snoop on bge0 I untagged packets as
well as all tagged packets. But if I snoop a VLAN (bge1000), I only see
packets with that VLAN tag. With vanity naming we'll have the ability to
name bge1000 as net1, and later change things so that bge0 is named
net1. Unless we do something, a side effect of such a change is that the
packets one can see with snoop changes. This might be confusing to
administrators, and a security issue if we decide it is safe for
somebody to snoop on net1 that shouldn't be allowed to see packets with
any VLAN tag. But I don't know 1) how critical this issue is, and 2) a
way to fix it without introducing a lot more complexity.
Has the designed considered diskless boot? I recall some dependency on
user level setup to do some operations, but that wouldn't be present in
the diskless boot case. (The case I saw was in section 6.2.5 where there
is a door upcall to devfsadmd; not clear what would happen early during
diskless boot and there is no devfsadmd running.)
The APIs in section 7.2.5 explicitly expose that M_CTL messages are used
to carry IPsec information. I think we want to retain the flexibility to
change the way the IPsec information is carried to a NIC, so I would
recommend that the APIs be changed so that the da_ipsec_t is carried as
a separate argument to the send and receive functions. This adds a bit
of extra overhead in softmac, since it needs to convert between this
extra argument and the pre-pended M_CTL, but it means that GLDv3 drivers
with IPsec acceleration don't constrain our ability to replace the M_CTL
hack down the road. (For example, adding a m_tx_ipsec() function that
takes a dl_ipsec_t as an extra argument, and similarly on the receive side.)
A thing that isn't changed in this project, but where this project might
be a good opportunity to improve things, is the mapping from names like
ce17, bge17, and net0 to the actual NIC and port on the system.
With DLPI style2 devices it is hard to find this mapping, since /dev/ce
is just a pointer to a pseudo node. With DLPI style 1 devices there is a
bit more of a hint, because /dev/bge0 is a symlink to a /devices path
which describe the bus on which the device appears.
With Vanity naming, will any of that be easier or harder?
If I know I'm using link name net1, then I can find the device name
using dladm. And then can I use 'ls -l /dev/<devicename> to see the
physical path name?
Page 38 talks pf disabling the send-side loopback mechanism when using
softmac. An other approach which has been used for similar purposes in
the past, is to define a new b_flag value (MSGLOOP) which is set by
softmac on transmit and checked by softmac on receipt. Since both
copymsg and dupmsg preserve b_flag this would be in indication of a
message looped back by software in the DLPI driver. The scheme suggested
in the document (filtering based on the source MAC address) is both a
performance issue and disables some aspects of IPv6 Duplicate Address
Detection.
Minor:
------
In section 3.1.4, does the order between the cfgadm -c configure PCI8,
and dladm rename-link ce0 net0, matter?
Is there an assumption that when the ce0 replaces bge0, that the system
automatically propagate information to the new one? By this I mean the
fact that the users of net0 might have joined some set of multicast
groups, others might have the device in promiscuous mode, and we might
have IPsec state in the driver.
Table 1 shows N/A for Generic VLAN support on top of VLAN. Is this an
implementation constraint or an architectural constraint? The reason I'm
asking is that there are proposal for Q-in-Q encapsulations (Ethernet
packets with multiple VLAN headers in them). I don't know if Solaris
would ever be directly connected to such a thing, but it might make
sense to not have an architectural limitations preventing us to add
support for this should it become needed in the future.
Section 4.1.1. Is there some option to show-link -s that, akin to
'netstat -i 10', would show delta values for the counters?
Section 4.3.4 autopush. It isn't clear from the document whether this is
"just" a way to specify autopush using a link name, but that it
otherwise has all the limitations of autopush. For instance, a user
could reasonably ask whether autopush would apply to a currently used
link name, or what it would take for the new autopush configuration to
be applied. I'm assuming one would need to do an ifconfig unplumb and
ifconfig plumb for the new information to be applied. If that's the
case, would it make sense for autopush -m to fail if the link name is in
use?
Page 20 footnote. Will CR 6292043 be fixed as part of this project?
Is it a requirement that it be fixed before this project integrates?
Page 22 interrupt coalescing. What timer granularity do you need to do
effective coalescing? Normally the devices have microsecond timers for
this, which can't (effectively) be done in software. So it isn't clear
to me that coalescing makes much sense in software. Polling mode support
might make sense, because it doesn't have a need for a fine grain timer.
Page 22 kstats for softmac. If softmac is going to prefix "net/" to the
kstat names, wouldn't it be simpler to apply this to the GLD kstat names
overall? With the specified approach any application that wants to look
at the standard kstats would need to be aware of softmac so that the
application can prefix "net/". Or is the intent that applications see
the kstats from the legacy driver?
Page 23 autopush backward compatibility.
The section ends with an assumption that autopushed modules not walk
q_next upstream to look for IP. Do we know if this is true for common
modules like vpnmod and fw-1?
Section 6.1.4: DL_CAPAB_UNBOUNDED_PKT
The frame size isn't really unbounded, so a more accurate name might
make more sense.
Section 6.2.2. In the new devfsadmd support, is there an implication
that there is a new persistent devfs repository for <link name, linkid>,
or will this use some existing repository (like path_to_inst)?
Section 6.2.4 last paragraph says that "administrators will run dladm
create-vlan -l net0". But is this the case even if a VLAN was explicitly
created before the rename? For example, if the administrator does
dladm create-vlan -l bge0 -v 23 net12
dladm rename-link -l bge0 foo3
does the administrator have to do something to (re)create net12, or does
net12's existence survive the rename?
Section 6.3.1 talks about how softmac generates devices. But it doesn't
fully specify how different legacy devices are handled. In terms of
presence in /dev, we have at least 3 different types of legacy drivers:
1. Those that only do style 2, hence have only a /dev/foo which is a
like to a pseudo entry in /devices
2. Those that support both style 1 and style 2, with a /dev/foo as well
as /dev/foo<N>.
3. Those that only support style 1. (I haven't see any such devices, but
I haven't looked for them either).
In particular, for #1, I don't see how softmac would know what instances
exist, since there is only a pseudo node. How would softmac know to
create /dev/net/ce0 and /dev/net/ce1 when two ce cards are in the system?
Section 6.3.2 talks about the need to support DL_ADDR_REQ etc. I didn't
think Sun Trunking was using that set of primitives. (They were added to
dlpi.h with the intent of folding IPMP and link aggregation together,
but that folding was never completed.) It makes sense to check the Sun
Trunking code whether it uses these primitives, or drop the paragraph
from the document.
Section 7.1.4 talks about transforming an M_MULTIDATA message into an
M_DATA message. It isn't clear to me that this provides much performance
benefits if any. Wouldn't it make more sense to add some m_multitx()
function by which multiple packets can be passed to the mac driver in
one call? That way the mac driver can use a single or a few ddi_dma
calls for the multiple packets.
Editorial:
---------
Page 3 says "all link configuration and kernel data structures will be
updated to refer to linkids". I assuming you are not updating IP, ipf,
etc, but that this change is limited to the GLD layers. Same issue in
first paragraph in section 6.2.2.
In section 3.1.7, which is the first time you talk about /dev and
/dev/net, it would be quite useful if you could explain what NICs appear
where (legacy using softmac, vanity names, VLANs). I found that
information much later in the document.
Also in section 3.1.7, it would be helpful if you stated the persistence
of the VLAN PPA hack names. Does ls show them in the directory? Do the
disappear after an ifconfig unplumb? (I think I found that information
later in the document as well.)
Section 6.1.3. When I read this it wasn't clear whether softmac would
replace the /dev/ce, or whether it was left in place. (The question was
answered as I read deeper into the document.)
Section 6.2.2 link configuration file.
I think it makes sense to state in the design document that this file
are not for editing, and the administrator must use dladm to manipulate
the content.
Section 6.2.3 starts with "In the future". When do we expect this future
to arrive? If it is part of this project it makes sense to explicitly
state so. Same wording appears in section 6.2.5, 6.2.6, 6.2.7,
Erik
_______________________________________________
networking-discuss mailing list
[email protected]
