Re: VIMAGE: vnet, epair and lots of jails on bridgeX - routing
On Sat, 10 Feb 2018 09:54:49 +0100 Marko Zec wrote: > On Sat, 10 Feb 2018 08:52:21 +0100 > "O. Hartmann" wrote: > > > Am Fri, 09 Feb 2018 16:43:17 + > > "Bjoern A. Zeeb" schrieb: > > > > > On 9 Feb 2018, at 16:22, O. Hartmann wrote: > > > > > > > Am Thu, 8 Feb 2018 09:31:15 +0100 > > > > "O. Hartmann" schrieb: > > > > > > > > Is this problem to trivial? > > > > > > I read through it yesterday and found myself in the position that I > > > need a whiteboard or paper and pencil or an ASCII art of your > > > situation. But by the time I made it to the question I was > > > basically lost. Could you massively simplify this and maybe > > > produce the ASCII art? > > > > > > /bz > > > ___ > > > freebsd-current@freebsd.org mailing list > > > https://lists.freebsd.org/mailman/listinfo/freebsd-current > > > To unsubscribe, send any mail to > > > "freebsd-current-unsubscr...@freebsd.org" > > > > All right. > > > > I'm not much of an artist and at this very moment, I haven't much > > experience with neat ASCII art tools. But I'll provide a sketch > > later, but I also will simplify the situation. > > > > Consider three "vswitches", basically based on the creation of > > bridges, bridge0, bridge1, bridge2. Create at least three individual > > vnet-jails attached to each vbridge. Those jails have epair pseudo > > devices. The jail itself owns the "a-part" of the epair and the > > b-part is "member of the bridge". Each jail's epairXXXa has an IP > > assigned of the network the vswitch is part of. I mention a- and > > b-part of the epair here, because I thought it could matter, but I > > think for symmetry reasons it doesn't. > > > > Now consider a further, special jail. This jail is supposed to have > > three epair devices, each one is reaching into one of the vbridges. > > This jail is the router/routing jail. Later, this jail should filter > > via IPFW the traffic between the three vbridges according to rules, > > but this doesn't matter here, beacuase the basics are not working as > > expected. > > > > Now the problems. It doesn't matter on which jail of the three > > vswitches I login, the moment a vbridge has more than two member > > epairs (one is alway member of the routing jail, now consider a > > database jail and a webserver jail), pinging each jail or the routing > > jail fails. It works sometimes for a couple of ICMP packets and then > > stops. > > > > If each vbridge has only one member jail, I have NO PROBLEMS > > traversing accordingly to the static routing rules from one vbridge > > to any other, say from vbridge1 to vbridge0 or vbridge2 and any > > permutation of that. > > > > The moment any of the bridges gets an additional member epair > > interface (so the bridge has at least three members including the on > > reaching into the virtual router jail) the vbridge seems to operate > > unpredictable (to me). Pinging jails memeber of that vbridge are > > unreachable. > > > > Technical information: > > > > The kernel has options IPFIREWALL, VIMAGE. The host's ipfw (kernel) > > declines packets by default. Each jail is configured to have ipfw > > "open". > > > > Thanks for the patience. > > If you could provide a script which sets up the topology you described > in two lengthy posts then others could reproduce this, and your chances > of getting useful feedback would certainly increase. > > We also have a graphical tool (https://github.com/imunes/imunes) which > can set up a topology like you described in a few clicks of a mouse, > albeit using netgraph and ng_eiface instead of epairs, but I assume this > is irellevant as long as you are not aiming for maximum packet > throughputs. If you attempt to use this tool, note that selecting > "stpswitch" will create if_bridge instances, whereas "lanswitch" > creates ng_bridge instances. > > Good luck, > > Marko > > > > > > Kind regards, > > > > O. Hartmann > > ___ > freebsd-current@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org" Hello Marko, thanks for your response. First of all: I looked at "imunes". From the first glimpse it looks great! Something really usefull; I regret not having a port for this tool or the chance to package it via poudriere. The problems I faced seem to be related to a bug Olivier Cochard-Labbe pointed me at: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=176671 Checking the MAC of the epairs created revealed, that either doubles or even more occur on the host side of the epair (in my case, all the b-parts of an epair), or, if there is nothing irregular, then the a-parts (owned by the VIMAGE jail) have same MAC. The more jails I create, the more ambiguous MACs are present. It is the first time I ran into a more complex network topology and thanks for the hint using netgraph.
Re: VIMAGE: vnet, epair and lots of jails on bridgeX - routing
On Mon, 12 Feb 2018 09:37:47 +0100
"O. Hartmann" wrote:
> On Sat, 10 Feb 2018 11:52:18 +0100
> Olivier Cochard-Labbé wrote:
>
> > On Sat, Feb 10, 2018 at 8:52 AM, O. Hartmann wrote:
> >
> > >
> > > The moment any of the bridges gets an additional member epair interface
> > > (so the bridge
> > > has at least three members including the on reaching into the virtual
> > > router jail) the
> > > vbridge seems to operate unpredictable (to me). Pinging jails memeber of
> > > that vbridge
> > > are unreachable.
> > >
> > >
> > First idea:
> > Did you try with a more simple setup, like with just 3 jails members of one
> > bridge ?
> > => I've tried it on a -head, and all 4 members (3 jails and the host)
> > reach to communicate.
> >
> > Second idea:
> > Can you check that all epairs have different MAC address each ?
> > I hit this bug: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=176671
>
> Wow, that PR is from 2013(!) and it is still not solved?
>
> >
> > Regards,
> >
> > Olivier
> > ___
> > freebsd-current@freebsd.org mailing list
> > https://lists.freebsd.org/mailman/listinfo/freebsd-current
> > To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
>
> ___
> freebsd-current@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
After rebooting recent CURRENT, the view with "ifconfig -a ether" looked good
so far, each epair/bridge has its unique MAC.
But then, login on the jails and checking the epair's counterpart owned by the
VIMAGE jail, I found almost EVERY jail has the same MAC, even those jails
members of the same bridge:
[...]
jail 11 on bridge2
epair20129a: flags=8843 metric 0 mtu
1500 options=8
ether 02:68:d0:00:07:0a
jail 10 on bridge2
epair20128a: flags=8843 metric 0 mtu
1500 options=8
ether 02:68:d0:00:07:0a
jail 9 on bridge 1
epair10250a: flags=8843 metric 0 mtu
1500 options=8
ether 02:68:d0:00:07:0a
jail 8 on bridge1
epair10238a: flags=8843 metric 0 mtu
1500 options=8
ether 02:68:d0:00:07:0a
jail 7 on bridge0
epair238a: flags=8843 metric 0 mtu 1500
options=8
ether 02:68:d0:00:07:0a
epair251a: flags=8843 metric 0 mtu 1500
options=8
ether 02:68:d0:00:07:0a
The way I create epairs and put them into a jail's context/domain is straight
forward. In jail.conf, I have more generic setup with variables like:
# DNS Master
ns1 {
$if = "2";
$ip4_addr = "10.10.0.${if}";
$ip4_cidr = "24";
$ip4_my_default_route = "10.10.0.1";
$vnet_if = "epair${if}";
$home_bridge = "${if_bridge_core}";
depend= "vrouter";
allow.raw_sockets= "1";
}
and in the common portion of jail.conf definitions, I use this:
[...]
vnet = "new";
vnet.interface ="${vnet_if}a";
persist;
exec.clean;
exec.start= "";
exec.start+="/sbin/ifconfig ${vnet_if}a inet
${ip4_addr}/${ip4_cidr} up"; exec.start+="/bin/sh /etc/rc";
exec.start+="/sbin/route add default ${ip4_my_default_route}";
exec.start+="/sbin/sysctl net.link.bridge.pfil_member=0";
exec.start+="/sbin/sysctl net.link.bridge.pfil_bridge=0";
exec.start+="/sbin/sysctl net.link.bridge.pfil_onlyip=0";
exec.stop= "/bin/sh /etc/rc.shutdown";
exec.prestart= "";
exec.prestart+= "ifconfig ${vnet_if} create";
exec.prestart+= "ifconfig ${vnet_if}b up";
exec.prestart+= "ifconfig ${home_bridge} addm ${vnet_if}b up";
exec.poststop= "ifconfig ${home_bridge} deletem ${vnet_if}b";
exec.poststop+= "ifconfig ${vnet_if}b destroy";
exec.consolelog="/var/log/jail_${name}_console.log";
The big question here is: when a jail with VIMAGE kernel "swallows" a
epair-pseudo device, it leaves the ciontext or visibility of the host. How can
the FreeBSD VIMAGE kernel then know about what former epair's MAC was? Is this
mechanism maybe the culprit? It is just a thought, so I do not want to be
beheaded - I'm not much into system development.
Kind regards,
O. Hartmann
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: VIMAGE: vnet, epair and lots of jails on bridgeX - routing
On Sat, 10 Feb 2018 11:52:18 +0100 Olivier Cochard-Labbé wrote: > On Sat, Feb 10, 2018 at 8:52 AM, O. Hartmann wrote: > > > > > The moment any of the bridges gets an additional member epair interface > > (so the bridge > > has at least three members including the on reaching into the virtual > > router jail) the > > vbridge seems to operate unpredictable (to me). Pinging jails memeber of > > that vbridge > > are unreachable. > > > > > First idea: > Did you try with a more simple setup, like with just 3 jails members of one > bridge ? > => I've tried it on a -head, and all 4 members (3 jails and the host) > reach to communicate. > > Second idea: > Can you check that all epairs have different MAC address each ? > I hit this bug: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=176671 Wow, that PR is from 2013(!) and it is still not solved? > > Regards, > > Olivier > ___ > freebsd-current@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org" ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: VIMAGE: vnet, epair and lots of jails on bridgeX - routing
On Sat, 10 Feb 2018 11:52:18 +0100 Olivier Cochard-Labbé wrote: > On Sat, Feb 10, 2018 at 8:52 AM, O. Hartmann wrote: > > > > > The moment any of the bridges gets an additional member epair interface > > (so the bridge > > has at least three members including the on reaching into the virtual > > router jail) the > > vbridge seems to operate unpredictable (to me). Pinging jails memeber of > > that vbridge > > are unreachable. > > > > > First idea: > Did you try with a more simple setup, like with just 3 jails members of one > bridge ? > => I've tried it on a -head, and all 4 members (3 jails and the host) > reach to communicate. Yes, I did. I used to setup one bridge (bridge0) and three jails. Each jail owns its epairXXa device with IP assigned. epairXXb of each jail is member of the bridge. Bridge is up, epairXXb is up. > > Second idea: > Can you check that all epairs have different MAC address each ? > I hit this bug: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=176671 Very good shot! Thanks. No, they do not have all of them unique MAC adrresses and in some occassions members of the very same bridge have the very same MAC addresses, mostly the a-part of the epair: From console: [...] ng_ether_ifnet_arrival_event: can't re-name node epair10250a ==>> epair20128a: Ethernet address: 02:ce:d0:00:07:0a epair20128b: Ethernet address: 02:ce:d0:00:13:0b epair20128a: link state changed to UP epair20128b: link state changed to UP epair20128b: promiscuous mode enabled ng_ether_ifnet_arrival_event: can't re-name node epair20128a ==>> epair20129a: Ethernet address: 02:ce:d0:00:07:0a epair20129b: Ethernet address: 02:ce:d0:00:14:0b epair20129a: link state changed to UP epair20129b: link state changed to UP epair20129b: promiscuous mode enabled ng_ether_ifnet_arrival_event: can't re-name node epair20129a epair20XXX are member of bridge2 and dedicated epairs of jails. The following is the desastrous picture of bridge1: [...] ==>> epair235a: Ethernet address: 02:ce:d0:00:07:0a epair235b: Ethernet address: 02:ce:d0:00:0d:0b epair235a: link state changed to UP epair235b: link state changed to UP epair235b: promiscuous mode enabled ng_ether_ifnet_arrival_event: can't re-name node epair235a ==>> epair237a: Ethernet address: 02:ce:d0:00:07:0a epair237b: Ethernet address: 02:ce:d0:00:0e:0b epair237a: link state changed to UP epair237b: link state changed to UP epair237b: promiscuous mode enabled ng_ether_ifnet_arrival_event: can't re-name node epair237a ==>> epair251a: Ethernet address: 02:ce:d0:00:07:0a epair251b: Ethernet address: 02:ce:d0:00:0f:0b epair251a: link state changed to UP epair251b: link state changed to UP epair251b: promiscuous mode enabled ng_ether_ifnet_arrival_event: can't re-name node epair251a ==>> epair238a: Ethernet address: 02:ce:d0:00:07:0a epair238b: Ethernet address: 02:ce:d0:00:10:0b epair238a: link state changed to UP epair238b: link state changed to UP epair238b: promiscuous mode enabled ng_ether_ifnet_arrival_event: can't re-name node epair238a [...] This is on CURRENT (FreeBSD 12.0-CURRENT #36 r329150: Mon Feb 12 06:30:47 CET 2018 amd64). I did check on Friday in the bureau and didn't catch it since I was checking on each jail, but the console log accessed via dmesg revealed the problem very easily. I did a check on the 11.1-RELENG-p6 box and I got the same picture there, different, but very similar setup. So I didn't see it in the masses of epairs our setup requires :-( I'll go now for setting MAC addresses manually and check functionality again. > > Regards, > > Olivier Kind regards, Oliver ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: VIMAGE: vnet, epair and lots of jails on bridgeX - routing
On Sat, Feb 10, 2018 at 8:52 AM, O. Hartmann wrote: > > The moment any of the bridges gets an additional member epair interface > (so the bridge > has at least three members including the on reaching into the virtual > router jail) the > vbridge seems to operate unpredictable (to me). Pinging jails memeber of > that vbridge > are unreachable. > > First idea: Did you try with a more simple setup, like with just 3 jails members of one bridge ? => I've tried it on a -head, and all 4 members (3 jails and the host) reach to communicate. Second idea: Can you check that all epairs have different MAC address each ? I hit this bug: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=176671 Regards, Olivier ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: VIMAGE: vnet, epair and lots of jails on bridgeX - routing
On Sat, 10 Feb 2018 08:52:21 +0100 "O. Hartmann" wrote: > Am Fri, 09 Feb 2018 16:43:17 + > "Bjoern A. Zeeb" schrieb: > > > On 9 Feb 2018, at 16:22, O. Hartmann wrote: > > > > > Am Thu, 8 Feb 2018 09:31:15 +0100 > > > "O. Hartmann" schrieb: > > > > > > Is this problem to trivial? > > > > I read through it yesterday and found myself in the position that I > > need a whiteboard or paper and pencil or an ASCII art of your > > situation. But by the time I made it to the question I was > > basically lost. Could you massively simplify this and maybe > > produce the ASCII art? > > > > /bz > > ___ > > freebsd-current@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-current > > To unsubscribe, send any mail to > > "freebsd-current-unsubscr...@freebsd.org" > > All right. > > I'm not much of an artist and at this very moment, I haven't much > experience with neat ASCII art tools. But I'll provide a sketch > later, but I also will simplify the situation. > > Consider three "vswitches", basically based on the creation of > bridges, bridge0, bridge1, bridge2. Create at least three individual > vnet-jails attached to each vbridge. Those jails have epair pseudo > devices. The jail itself owns the "a-part" of the epair and the > b-part is "member of the bridge". Each jail's epairXXXa has an IP > assigned of the network the vswitch is part of. I mention a- and > b-part of the epair here, because I thought it could matter, but I > think for symmetry reasons it doesn't. > > Now consider a further, special jail. This jail is supposed to have > three epair devices, each one is reaching into one of the vbridges. > This jail is the router/routing jail. Later, this jail should filter > via IPFW the traffic between the three vbridges according to rules, > but this doesn't matter here, beacuase the basics are not working as > expected. > > Now the problems. It doesn't matter on which jail of the three > vswitches I login, the moment a vbridge has more than two member > epairs (one is alway member of the routing jail, now consider a > database jail and a webserver jail), pinging each jail or the routing > jail fails. It works sometimes for a couple of ICMP packets and then > stops. > > If each vbridge has only one member jail, I have NO PROBLEMS > traversing accordingly to the static routing rules from one vbridge > to any other, say from vbridge1 to vbridge0 or vbridge2 and any > permutation of that. > > The moment any of the bridges gets an additional member epair > interface (so the bridge has at least three members including the on > reaching into the virtual router jail) the vbridge seems to operate > unpredictable (to me). Pinging jails memeber of that vbridge are > unreachable. > > Technical information: > > The kernel has options IPFIREWALL, VIMAGE. The host's ipfw (kernel) > declines packets by default. Each jail is configured to have ipfw > "open". > > Thanks for the patience. If you could provide a script which sets up the topology you described in two lengthy posts then others could reproduce this, and your chances of getting useful feedback would certainly increase. We also have a graphical tool (https://github.com/imunes/imunes) which can set up a topology like you described in a few clicks of a mouse, albeit using netgraph and ng_eiface instead of epairs, but I assume this is irellevant as long as you are not aiming for maximum packet throughputs. If you attempt to use this tool, note that selecting "stpswitch" will create if_bridge instances, whereas "lanswitch" creates ng_bridge instances. Good luck, Marko > > Kind regards, > > O. Hartmann ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: VIMAGE: vnet, epair and lots of jails on bridgeX - routing
Am Fri, 09 Feb 2018 16:43:17 + "Bjoern A. Zeeb" schrieb: > On 9 Feb 2018, at 16:22, O. Hartmann wrote: > > > Am Thu, 8 Feb 2018 09:31:15 +0100 > > "O. Hartmann" schrieb: > > > > Is this problem to trivial? > > I read through it yesterday and found myself in the position that I need > a whiteboard or paper and pencil or an ASCII art of your situation. But > by the time I made it to the question I was basically lost. Could you > massively simplify this and maybe produce the ASCII art? > > /bz > ___ > freebsd-current@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org" All right. I'm not much of an artist and at this very moment, I haven't much experience with neat ASCII art tools. But I'll provide a sketch later, but I also will simplify the situation. Consider three "vswitches", basically based on the creation of bridges, bridge0, bridge1, bridge2. Create at least three individual vnet-jails attached to each vbridge. Those jails have epair pseudo devices. The jail itself owns the "a-part" of the epair and the b-part is "member of the bridge". Each jail's epairXXXa has an IP assigned of the network the vswitch is part of. I mention a- and b-part of the epair here, because I thought it could matter, but I think for symmetry reasons it doesn't. Now consider a further, special jail. This jail is supposed to have three epair devices, each one is reaching into one of the vbridges. This jail is the router/routing jail. Later, this jail should filter via IPFW the traffic between the three vbridges according to rules, but this doesn't matter here, beacuase the basics are not working as expected. Now the problems. It doesn't matter on which jail of the three vswitches I login, the moment a vbridge has more than two member epairs (one is alway member of the routing jail, now consider a database jail and a webserver jail), pinging each jail or the routing jail fails. It works sometimes for a couple of ICMP packets and then stops. If each vbridge has only one member jail, I have NO PROBLEMS traversing accordingly to the static routing rules from one vbridge to any other, say from vbridge1 to vbridge0 or vbridge2 and any permutation of that. The moment any of the bridges gets an additional member epair interface (so the bridge has at least three members including the on reaching into the virtual router jail) the vbridge seems to operate unpredictable (to me). Pinging jails memeber of that vbridge are unreachable. Technical information: The kernel has options IPFIREWALL, VIMAGE. The host's ipfw (kernel) declines packets by default. Each jail is configured to have ipfw "open". Thanks for the patience. Kind regards, O. Hartmann pgpruzqWVMaOU.pgp Description: OpenPGP digital signature
Re: VIMAGE: vnet, epair and lots of jails on bridgeX - routing
On 9 Feb 2018, at 16:22, O. Hartmann wrote: Am Thu, 8 Feb 2018 09:31:15 +0100 "O. Hartmann" schrieb: Is this problem to trivial? I read through it yesterday and found myself in the position that I need a whiteboard or paper and pencil or an ASCII art of your situation. But by the time I made it to the question I was basically lost. Could you massively simplify this and maybe produce the ASCII art? /bz ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: VIMAGE: vnet, epair and lots of jails on bridgeX - routing
Am Thu, 8 Feb 2018 09:31:15 +0100 "O. Hartmann" schrieb: > Hello, > > I fight with the following problem without any kind of success and I need some > help and/or advice. > > We are running several CURRENT and 11.1-RELENG-p6 boxes. CURRENT is at the > most > recent version as of today. > > VIMAGE is compiled in into all kernels. > IPFW is compiled into all kernels and is the one and only firewall used. > On CURRENT, the host's ipfw is set to "OPEN" (using the rc.-scripts so far). > By > convention, I address the host running the kernel by "host". > > Every jail is created/configured with its own "vnet" cloned network stack > (vnet=new). > > All hosts do have at least three physical NICs. The host itself is supposed to > be member of the "friendly" network via a dedicated NIC. The two remaining > NICs > are split into fractions belonging to an "hostile" network on which I'd like > to > place exposed jails (for now), and to the "friendly" network, on which also > jails will be hosted, but via a dedicated NIC. > > Inbetween those two networks, the host will have a third, intermediate, > network, call it the "service" network. > > The following will be true for ALL jails created, including the host itself: > > net.link.bridge.pfil_member=0 > net.link.bridge.pfil_bridge=0 > net.link.bridge.pfil_onlyip=0 > > First, I clone/create three bridge(4) devices, bridge0 (considered to be the > "glue" between the "service" jails), bridge1 (considered to be the glue > between > the jails on the friednly network side) and bridge2, which is the glue between > the jails on the hostile side. bridge1 has eth1 as a member, which provides > the > physical access to the friendly network, eth2 is member of bridge2, which > provides access to the hostile network. > > By convention, when creating epair(4), the a-portion belongs to the jail > itself > and is assigned with an IPv6 address. The b-portion of the epair(4) is member > of its bridge according to its realm (friendly, service or hostile network). > > Additionally, there is a special jail, the router, which has three epair(4) > devices, the b-portion of the epair is member of the appropriate bridge(4) and > this router jail has static routes assigned, pointing to the appropriate > epairXXXa that is suppoesd to be the link into the correct bridge/network. > IPFW > is set to open on this jail (for now). On this special > jail it is set: net.inet.ip.forwarding=1. > > I hope, the topology is clear so far. All epairs or epair endpoints as well as > the bridges are UP! Double checked this. > > Jails on bridge0 (service net) have IPs in the range 10.10.0.0/24, the > b-portion of the routing jail's epair is member of bridge0, as described > above, > and the a-portion of the epair has IP 10.10.0.1. Default route on each jeail > on bridge0 is set to 10.10.0.1 accordingly. > > Consider a similar setup on the other jails on the friendly and hostile > network, except the fact that their bridges do have a physical NIC to which > they may have access to a real network. > > The setup might not be ideal and/or applicable for the purpose of separartion > of networks virtually, but that shouldn't be the subject here. More important > is that I assume that I haven't understood some essentials, because the setup > doens't work as expected. Furthermore, it behaves on FreeBSD 11.1-RELENG-p6 > sometimes completely unpredictable - but in that special case, I think I ran > IPFW on the host as "WORKSTATION" and dynamic rules may play an important role > here. But focussing on the CURRENT box, the host's IPFW is set to OPEN. > > With jexec -l hostA I gain access to host A on the "service" bridge0 and I > want to ping its neighbour, hostB, on the same bridge and in the same net. It > doesn't work! From the routing jail, I CAN NOT ping any host on bridge0. The > routing jail has these network settings: > > [... routing jail ...] > lo0: flags=8049 metric 0 mtu 16384 > options=63 > inet 127.0.0.1 netmask 0xff00 > groups: lo > [epair to bridge0 - service net] > epair4000a: flags=8843 metric 0 mtu > 1500 > options=8 > ether 02:57:d0:00:07:0a > inet 10.10.0.1 netmask 0xff00 broadcast 10.10.0.255 > media: Ethernet 10Gbase-T (10Gbase-T ) > status: active > groups: epair > [epair to bridge1, friendly net] > epair4001a: flags=8843 metric 0 mtu > 1500 > options=8 > ether 02:57:d0:00:09:0a > inet 192.168.11.1 netmask 0xff00 broadcast 192.168.11.255 > media: Ethernet 10Gbase-T (10Gbase-T ) > status: active > groups: epair > [epair to bridge2, hostile net] > epair4002a: flags=8843 metric 0 mtu > 1500 > options=8 > ether 02:57:d0:00:0b:0a > inet 10.10.10.1 netmask 0xfc00 broadcast 10.10.10.255 > media: Ethernet 10Gbase-T (10Gbase-T ) > status: active > groups: epair > > routing: > netstat -War
