Hi Peter,

>  > Yes, IPMP is IP Multipathing. Even if you have multiple addresses on
>  > the same subnet on different interfaces, as you do, it does not
>  > automatically enable IPMP.
> Though the resulting configuration is unsupported and broken.  That
> is, if you have multiple IP interfaces on the same link, they *must*
> be configured into an IPMP group.  If you don't do this, you will see
> duplicate broadcast and multicast traffic, among other problems.

Unfortunately, I guess this is exactly my setup (multiple interfaces
on the same link). Just to be sure, here it is:

                                   10/100 MBit Miniswitch w/ Auto-uplink
Galaxy X4200 +                            |                   |
             +- ILOM/SMC ----- smc ------ + port 1            |
             |                            |                   |
             +- Mainboard -+              |                   |
                           |              |                   |
                           +-- e1000g0 -- + port 2            |
                           |              |                   |
                           +-- e1000g1 -- + port 3            |
                           |              |                   |
                           +-- e1000g2 -- + port 4     port 5 + -- ISP switch 
port -- ISP router
                           |              +-------------------+
                           +-- e1000g3 -- not connected

So, from your reply, I guess having e1000g0-2 on the same link
without having them in one IPMP group is evil.

Having the smc on the same link should do no harm, since the ILOM/SMC
is an embedded Linux-computer, right?

What if I replaced the 10/100 MBit Miniswitch with a VLAN capable and
configured ports 2-4 so that they each be on their own virtual LAN?
Whould this still require having e1000g0-2 in one IPMP group?

What if I ordered 2 ports at the one ISP switch? Would the ISP switch
need to be VLAN capable/setup to NOT require IPMP group?

And 2 ports from two different ISP switches each uplinked to a router?
Then really not I guess .., right?

The reason I'm asking: I would really like to have IPMP, but I can't
have it together with stateful packet filtering:

Solaris 10 System Administrator Collection
>> System Administration Guide: IP Services
>> Chapter 25 Solaris IP Filter (Overview)

"IP Network Multipathing (IPMP) supports stateless filtering only."


I NEED stateful filtering. So I can't eat the cake;(

>  > And I still get confused by that selection.... Back with interface
>  > groups (pre IPMP), outbound traffic on would leave on the same
>  > interface a connection came in on, but I'm not sure that is still the
>  > case.
> With IPMP, outbound interface selection is determined in a round-robin
> fashion the first time an application attempts to send to a particular
> destination that does not yet have an IRE cache entry.
> See illgrp_scheduler() for details.

Not sure if I got you right:

A 3-way TCP handshake for an incoming TCP connection involves:

1. incoming SYN packet
2. outgoing SYN-ACK packet
3. incoming ACK packet

With IPMP, the outbound interface is selected RR with packet 2, right?

Thus, packet 1 could come in e1000g2 and packet 2 could leave on e1000g0.
Then, all subsequent outgoing packets would leave on e1000g0?
What about the subsequent incoming packets?

What I saw was this:

Packet 1 on e1000g2, Packet 2 on e1000g0, Packet 3 and all subsequent
incoming/outgoing packet on e1000g2. But this was with no IPMP configured
and thus bogus anyway.

Problem is: from my understanding of stateful IPF, i.e. a rule

pass in quick proto tcp from any to port = 80  keep state

IPF builds an state table entry from the very first packet, that is the
incoming SYN. I guess the state entry includes source/dest IP/port plus
interface. If the SYN comes in on e1000g2, any packet trying to leave
over e1000g0 will be blocked. This is what I saw: the incoming SYN passed,
the outgoing SYN-ACK was blocked (since on a different ifc) and all subsequent
outgoing packets blocked since the state entry could not get into the final
handshaked 4/4 state.

"Once the first SYN packet hits the ssh server, state is created and the 
remainder of the ssh session is allowed to take place without interference from 
the firewall."

"Once the 3-way handshake has been witness by the state engine, it is marked in 
4/4 mode, which means it's setup for long-term data exchange until such time as 
the connection is torn down (wherein the mode changes again."


With IPMP not the first incoming SYN packet's interface, but the second
outoing SYN-ACK packet's interface enters the IRE cache.

If this is all correct, then I can understand why IPMP and stateful IPF
are incompatible.

The design of RR on the first outgoing packet with IPMP is understandable:
balance load. Perhaps Solaris IPF could be modified to build it's state
entry from the 2nd packet .. the outoing SYN-ACK?

For stateful outgoing TCP there should be no problem anyway:

pass out quick proto tcp from to any keep state

1. outgoing SYN packet
2. incoming SYN-ACK packet
3. outgoing ACK packet

IPMP would build IRE cache entry on the first packet (outgoing SYN).
IPF would initiate state from the same packet.


All in all, this is my resumee:

- having multiple interfaces on the same link requires IPMP
- IPMP alongside stateful IPF does not work

Consequently, I will back up and bring all interfaces but e1000g0 down and
reconfigure the global and all non-global zones to use e1000g0.

Thanks alot for your hints Peter, Steffen,
Obviously, I would have been pretty much lost without them,
This message posted from opensolaris.org
zones-discuss mailing list

Reply via email to