Am Thu, 8 Feb 2018 09:31:15 +0100
"O. Hartmann" <> schrieb:

> Hello,
> I fight with the following problem without any kind of success and I need some
> help and/or advice.
> We are running several CURRENT and 11.1-RELENG-p6 boxes. CURRENT is at the 
> most
> recent version as of today.
> VIMAGE is compiled in into all kernels.
> IPFW is compiled into all kernels and is the one and only firewall used.
> On CURRENT, the host's ipfw is set to "OPEN" (using the rc.-scripts so far). 
> By
> convention, I address the host running the kernel by "host".
> Every jail is created/configured with its own "vnet" cloned network stack
> (vnet=new).
> All hosts do have at least three physical NICs. The host itself is supposed to
> be member of the "friendly" network via a dedicated NIC. The two remaining 
> NICs
> are split into fractions belonging to an "hostile" network on which I'd like 
> to
> place exposed jails (for now), and to the "friendly" network, on which also
> jails will be hosted, but via a dedicated NIC.
> Inbetween those two networks, the host will have a third, intermediate,
> network, call it the "service" network.
> The following will be true for ALL jails created, including the host itself:
> First, I clone/create three bridge(4) devices, bridge0 (considered to be the
> "glue" between the "service" jails), bridge1 (considered to be the glue 
> between
> the jails on the friednly network side) and bridge2, which is the glue between
> the jails on the hostile side. bridge1 has eth1 as a member, which provides 
> the
> physical access to the friendly network, eth2 is member of bridge2, which
> provides access to the hostile network.
> By convention, when creating epair(4), the a-portion belongs to the jail 
> itself
> and is assigned with an IPv6 address. The b-portion of the epair(4) is member
> of its bridge according to its realm (friendly, service or hostile network). 
> Additionally, there is a special jail, the router, which has three epair(4)
> devices, the b-portion of the epair is member of the appropriate bridge(4) and
> this router jail has static routes assigned, pointing to the appropriate
> epairXXXa that is suppoesd to be the link into the correct bridge/network. 
> is set to open on this jail (for now). On this special
> jail it is set: net.inet.ip.forwarding=1.
> I hope, the topology is clear so far. All epairs or epair endpoints as well as
> the bridges are UP! Double checked this.
> Jails on bridge0 (service net) have IPs in the range, the
> b-portion of the routing jail's epair is member of bridge0, as described 
> above,
> and the a-portion of the epair has IP Default route on each jeail
> on bridge0 is set to accordingly.
> Consider a similar setup on the other jails on the friendly and hostile
> network, except the fact that their bridges do have a physical NIC to which
> they may have access to a real network.
> The setup might not be ideal and/or applicable for the purpose of separartion
> of networks virtually, but that shouldn't be the subject here. More important
> is that I assume that I haven't understood some essentials, because the setup
> doens't work as expected. Furthermore, it behaves on FreeBSD 11.1-RELENG-p6
> sometimes completely unpredictable - but in that special case, I think I ran
> IPFW on the host as "WORKSTATION" and dynamic rules may play an important role
> here. But focussing on the CURRENT box, the host's IPFW is set to OPEN.
> With jexec -l hostA I gain access to host A on the "service" bridge0 and I
> want to ping its neighbour, hostB, on the same bridge and in the same net. It
> doesn't work! From the routing jail, I CAN NOT ping any host on bridge0. The
> routing jail has these network settings:
> [... routing jail ...]
>  lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
>         options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
>         inet netmask 0xff000000 
>         groups: lo
> [epair to bridge0 - service net] 
> epair4000a: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 
> 1500
>         options=8<VLAN_MTU>
>         ether 02:57:d0:00:07:0a
>         inet netmask 0xffffff00 broadcast 
>         media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
>         status: active
>         groups: epair
> [epair to bridge1, friendly net] 
> epair4001a: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 
> 1500
>         options=8<VLAN_MTU>
>         ether 02:57:d0:00:09:0a
>         inet netmask 0xffffff00 broadcast 
>         media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
>         status: active
>         groups: epair
> [epair to bridge2, hostile net] 
> epair4002a: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 
> 1500
>         options=8<VLAN_MTU>
>         ether 02:57:d0:00:0b:0a
>         inet netmask 0xfffffc00 broadcast 
>         media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
>         status: active
>         groups: epair 
> routing:
> netstat -Warn
> Routing tables
> Internet:
> Destination        Gateway            Flags       Use    Mtu      Netif Expire
>       link#2             U            11   1500 epair4000a
>          link#2             UHS           4  16384        lo0
>      link#4             U           210   1500 epair4002a
>         link#4             UHS          44  16384        lo0
>          link#1             UH            0  16384        lo0
>    link#3             U             9   1500 epair4001a
>       link#3             UHS           0  16384        lo0
> Consider a jail hostCC on bridge2 in the hostile network, IP 
> I can ping that jail, although it has conceptionally the very same setup as 
> the
> unreachable jails on bridge0!
> It is weird. On bridge0, no jail can be pinged, it looks like the ethernet is
> somehwo down on that bridge. I would expect to ping each host member of the
> very same bridge! On 11.1-RELENG-p6, there are other weird issues, I was able
> to ping those jails, even ssh to them, but that vanished after several
> restarts of the jails system (each bridge, epair is created by jail.conf and
> destroyed after the jails has been deactivated and doing so a considerable
> amount brings down the FreeBSD 11.1-RELENG-p6 host verys successfully - it
> crashes!).
> So, since VIMAGE is now default in CURRENT's GENERIC, I consider its
> functionality at least "predictable", but I fail somehow here.
> Does someone have a deeper insight or realise the mistake I'm celebrating 
> here?
> Thanks in adavnce,
> Oliver 

Is this problem to trivial?


Attachment: pgplC7wyH9MUQ.pgp
Description: OpenPGP digital signature

Reply via email to