Re: bridge(4) Problems when running under ESXi ?

2020-11-30 Thread Tom Smyth
Hello Heinrich,
as another hack you can setup virtual switches (separate ones for any given
link between two VMs  )

eg vm1--vswitch2---vm2---vswitch3--vm4
so if you have promiscuous enabled and you only have two vms attached to
the vswitch is not so bad...
but if you have 100x vms on a port with 100mb/s of traffic then you  can
generate a lot of traffic copies to other vms...

so what you can do is run  a virtual switch per vlan and then attach
individual vms to  a dedicated vswitch per vlan (kind of like private
vlans)  it sucks wbut will work...  a
and then they can be isolated on layer 2...

and then have the switch port configured as a tagged port facing the
physical port on the server...
 and  have an OpenBSD Box as a default gateway with a separate ip on each
vlan...

that way you can avoid nat nastiness on vmware...and have decent layer2 /
Layer3 separation controlled by OpenBSD

All the Best
Tom Smyth





On Mon, 30 Nov 2020 at 18:28, Heinrich Rebehn 
wrote:

> Hello Tom,
>
> Thank you very much for your in-depth explanations.
>
> Actually enabling mac changes and forged transmits did the trick. A HUGE
> trick:
>
> While A was pinging R, I tried to look at the icmp requests and replies on
> B’s vmx1 interface. But they did not show. Neither bridge0 or vmx0 showed
> anything from or to A. I then blocked all traffic in B’s pf. A still kept
> on pinging successfully. I then shut down B. A was still happily pinging R.
> This is really scary! I intended to protect a Linux host whose firewall I
> don’t trust, but now it seems that I can trust VMware’s vmswitch even less.
>
> I also love VMware, it is fine for playing with networks, subnetting,
> IPSec etc.. but I never used virtual switches before.
>
> If there isn’t any way to firewall another host without doing NAT (both in
> the same subnet’s IP range), then I am afraid the Linux firewall will have
> to do.
>
> With kind greetings,
>
> Heinrich
>
> On 29. Nov 2020, at 23:26, Tom Smyth  wrote:
>
> Hello Heinrich,
> it is not OpenBSD  it is a Vmware issue ...
>
> virtualnets / vswitches in ESXI are not proper switches... they forward
> packets based on static mac- virtual port entries.   (they do not do proper
> mac learning)
>
> you can set the vwswitch in the networking configuration section ... there
> are 2 places  you can set it ... in the vmnet and the vswitch setup in the
> vmnet setup config  in vsphere
>
> there are 3 workarounds
>
> 1) use promiscuous mode (you can set the promiscuous setting on the
> vswitch)  you will also need to allow mac changes and forged transmits
> (from memory)
> Upside (it works) and is Free
>
> downside each vm on that vswitch receives a copy of the frames sent and
> received   ...  promiscuous makes a vhub rather than a vswitch
> so it is slower than one would like
>
> 2) there is a lab test switch (it was in vmware labs I think)  that does
> mac learning however it does not do mac aging
> upside it works and is faster than promiscuous
> downside not againg out macs is just f**king dumb ...
>
> 3) get the enterprise enterprise enterprise +  licence and they will give
> you proper mac learning on the virtual switches
>
> and that is the reason I migrated to a different Virtual machine solution
> ...
>
> I love Vmware but they are optimistic when they call their vswitches
> switches ...  they are efficeint for non forwarding workloads and I can
> understand why they do the static map by default
> but for networking  (they dont even give you LACP on their enterprise
> licence you have to go for their top line license enterprise Plus (last
> time i checked)
>
> it is a pitty because I do like Vmware and moving off it was tough as
> breaking an addiction...
>
> Hope this helps
>
> Tom Smyth
>
>
>
> On Sun, 29 Nov 2020 at 22:10, Heinrich Rebehn 
> wrote:
>
>> Unfortunately, switching to vmx(4) did *not* do the trick
>>
>> -Heinrich
>>
>>
>> > On 29. Nov 2020, at 22:38, Heinrich Rebehn 
>> wrote:
>> >
>> > Some things I forgot:
>> >
>> > All interfaces are UP
>> > pf(4) ist disabled
>> > bridge0 sees a bunch of lladdrs on em0 and one on em1, which is that of
>> “A”
>> >
>> > -Heinrich
>> >
>> >
>> >> On 29. Nov 2020, at 22:29, Heinrich Rebehn > > wrote:
>> >>
>> >> Hi all,
>> >>
>> >> I am trying to setup an OpenBSD 6.7 virtual machine under VMware ESXi
>> 6.7 to use as a filtering bridge between two virtual networks. I enabled
>> promiscuous mode for both virtual switches.
>> >> One network is the VMnet network, which is connected to the “outside
>> world”.
>> >>
>> >> “A” ——> “B” ——> “R”
>> >>
>> >> “A” is a test machine192.168.1.152
>> >> “B” is the bridgeNo IP. em0 connects to R, em1 connects to
>> A
>> >> “R” is the router provided by the hoster 192.168.1.1
>> >>
>> >> The addresses are only examples, the actual addresses a public IPs.
>> >>
>> >> When A tries to ping R, ist sends an arp request for R’s lladdr. R
>> responds with its lladdr. Tcpdump on R’s 

Re: bridge(4) Problems when running under ESXi ?

2020-11-30 Thread Heinrich Rebehn
Hello Tom,

Thank you very much for your in-depth explanations.

Actually enabling mac changes and forged transmits did the trick. A HUGE trick:

While A was pinging R, I tried to look at the icmp requests and replies on B’s 
vmx1 interface. But they did not show. Neither bridge0 or vmx0 showed anything 
from or to A. I then blocked all traffic in B’s pf. A still kept on pinging 
successfully. I then shut down B. A was still happily pinging R.
This is really scary! I intended to protect a Linux host whose firewall I don’t 
trust, but now it seems that I can trust VMware’s vmswitch even less.

I also love VMware, it is fine for playing with networks, subnetting, IPSec 
etc.. but I never used virtual switches before.

If there isn’t any way to firewall another host without doing NAT (both in the 
same subnet’s IP range), then I am afraid the Linux firewall will have to do.

With kind greetings,

Heinrich

> On 29. Nov 2020, at 23:26, Tom Smyth  wrote:
> 
> Hello Heinrich,
> it is not OpenBSD  it is a Vmware issue ...
> 
> virtualnets / vswitches in ESXI are not proper switches... they forward 
> packets based on static mac- virtual port entries.   (they do not do proper 
> mac learning)
> 
> you can set the vwswitch in the networking configuration section ... there 
> are 2 places  you can set it ... in the vmnet and the vswitch setup in the 
> vmnet setup config  in vsphere
> 
> there are 3 workarounds
> 
> 1) use promiscuous mode (you can set the promiscuous setting on the vswitch)  
> you will also need to allow mac changes and forged transmits (from memory)
> Upside (it works) and is Free 
> 
> downside each vm on that vswitch receives a copy of the frames sent and 
> received   ...  promiscuous makes a vhub rather than a vswitch  
> so it is slower than one would like 
> 
> 2) there is a lab test switch (it was in vmware labs I think)  that does mac 
> learning however it does not do mac aging
> upside it works and is faster than promiscuous 
> downside not againg out macs is just f**king dumb ... 
> 
> 3) get the enterprise enterprise enterprise +  licence and they will give you 
> proper mac learning on the virtual switches 
> 
> and that is the reason I migrated to a different Virtual machine solution ...
> 
> I love Vmware but they are optimistic when they call their vswitches  
> switches ...  they are efficeint for non forwarding workloads and I can 
> understand why they do the static map by default
> but for networking  (they dont even give you LACP on their enterprise licence 
> you have to go for their top line license enterprise Plus (last time i 
> checked) 
> 
> it is a pitty because I do like Vmware and moving off it was tough as 
> breaking an addiction...
> 
> Hope this helps
> 
> Tom Smyth
> 
> 
> 
> On Sun, 29 Nov 2020 at 22:10, Heinrich Rebehn  > wrote:
> Unfortunately, switching to vmx(4) did *not* do the trick
> 
> -Heinrich
> 
> 
> > On 29. Nov 2020, at 22:38, Heinrich Rebehn  > > wrote:
> > 
> > Some things I forgot:
> > 
> > All interfaces are UP
> > pf(4) ist disabled
> > bridge0 sees a bunch of lladdrs on em0 and one on em1, which is that of “A”
> > 
> > -Heinrich
> > 
> > 
> >> On 29. Nov 2020, at 22:29, Heinrich Rebehn  >>   >> >> wrote:
> >> 
> >> Hi all,
> >> 
> >> I am trying to setup an OpenBSD 6.7 virtual machine under VMware ESXi 6.7 
> >> to use as a filtering bridge between two virtual networks. I enabled 
> >> promiscuous mode for both virtual switches.
> >> One network is the VMnet network, which is connected to the “outside 
> >> world”.
> >> 
> >> “A” ——> “B” ——> “R”
> >> 
> >> “A” is a test machine192.168.1.152
> >> “B” is the bridgeNo IP. em0 connects to R, em1 connects to A
> >> “R” is the router provided by the hoster 192.168.1.1
> >> 
> >> The addresses are only examples, the actual addresses a public IPs.
> >> 
> >> When A tries to ping R, ist sends an arp request for R’s lladdr. R 
> >> responds with its lladdr. Tcpdump on R’s em1 suggests that it is sent out 
> >> on the virtual network. However, A does not see the arp reply, hence 
> >> ping(8) fails.
> >> 
> >> What am I missing? While browsing the mailing list archive, I just saw 
> >> that vmx(4) might be a better choice, but I had not yet time to try it out.
> >> 
> >> 
> >> Any other known issues around bridge(4) or promiscuous mode under ESXi ?
> >> 
> >> Thanks for any insights,
> >> 
> >>  Heinrich
> 
> 
> 
> -- 
> Kindest regards,
> Tom Smyth.



bridge(4) Problems when running under ESXi ?

2020-11-29 Thread Heinrich Rebehn
Hi all,

I am trying to setup an OpenBSD 6.7 virtual machine under VMware ESXi 6.7 to 
use as a filtering bridge between two virtual networks. I enabled promiscuous 
mode for both virtual switches.
One network is the VMnet network, which is connected to the “outside world”.

“A” ——> “B” ——> “R”

“A” is a test machine   192.168.1.152
“B” is the bridge   No IP. em0 connects to R, em1 connects to A
“R” is the router provided by the hoster192.168.1.1

The addresses are only examples, the actual addresses a public IPs.

When A tries to ping R, ist sends an arp request for R’s lladr. R responds with 
its lladr. Tcpdump on R’s em1 suggests that it is sent out on the virtual 
network. However, A does not see the arp reply, hence ping(8) fails.

What am I missing? While browsing the mailing list archive, I just saw that 
vmx(4) might be a better choice, but I had not yet time to try it out.


Any other known issues around bridge(4) or promiscuous mode under ESXi ?

Thanks for any insights,

Heinrich





Re: bridge(4) Problems when running under ESXi ?

2020-11-29 Thread Tom Smyth
Hello Heinrich,
it is not OpenBSD  it is a Vmware issue ...

virtualnets / vswitches in ESXI are not proper switches... they forward
packets based on static mac- virtual port entries.   (they do not do proper
mac learning)

you can set the vwswitch in the networking configuration section ... there
are 2 places  you can set it ... in the vmnet and the vswitch setup in the
vmnet setup config  in vsphere

there are 3 workarounds

1) use promiscuous mode (you can set the promiscuous setting on the
vswitch)  you will also need to allow mac changes and forged transmits
(from memory)
Upside (it works) and is Free

downside each vm on that vswitch receives a copy of the frames sent and
received   ...  promiscuous makes a vhub rather than a vswitch
so it is slower than one would like

2) there is a lab test switch (it was in vmware labs I think)  that does
mac learning however it does not do mac aging
upside it works and is faster than promiscuous
downside not againg out macs is just f**king dumb ...

3) get the enterprise enterprise enterprise +  licence and they will give
you proper mac learning on the virtual switches

and that is the reason I migrated to a different Virtual machine solution
...

I love Vmware but they are optimistic when they call their vswitches
switches ...  they are efficeint for non forwarding workloads and I can
understand why they do the static map by default
but for networking  (they dont even give you LACP on their enterprise
licence you have to go for their top line license enterprise Plus (last
time i checked)

it is a pitty because I do like Vmware and moving off it was tough as
breaking an addiction...

Hope this helps

Tom Smyth



On Sun, 29 Nov 2020 at 22:10, Heinrich Rebehn 
wrote:

> Unfortunately, switching to vmx(4) did *not* do the trick
>
> -Heinrich
>
>
> > On 29. Nov 2020, at 22:38, Heinrich Rebehn 
> wrote:
> >
> > Some things I forgot:
> >
> > All interfaces are UP
> > pf(4) ist disabled
> > bridge0 sees a bunch of lladdrs on em0 and one on em1, which is that of
> “A”
> >
> > -Heinrich
> >
> >
> >> On 29. Nov 2020, at 22:29, Heinrich Rebehn  > wrote:
> >>
> >> Hi all,
> >>
> >> I am trying to setup an OpenBSD 6.7 virtual machine under VMware ESXi
> 6.7 to use as a filtering bridge between two virtual networks. I enabled
> promiscuous mode for both virtual switches.
> >> One network is the VMnet network, which is connected to the “outside
> world”.
> >>
> >> “A” ——> “B” ——> “R”
> >>
> >> “A” is a test machine192.168.1.152
> >> “B” is the bridgeNo IP. em0 connects to R, em1 connects to A
> >> “R” is the router provided by the hoster 192.168.1.1
> >>
> >> The addresses are only examples, the actual addresses a public IPs.
> >>
> >> When A tries to ping R, ist sends an arp request for R’s lladdr. R
> responds with its lladdr. Tcpdump on R’s em1 suggests that it is sent out
> on the virtual network. However, A does not see the arp reply, hence
> ping(8) fails.
> >>
> >> What am I missing? While browsing the mailing list archive, I just saw
> that vmx(4) might be a better choice, but I had not yet time to try it out.
> >>
> >>
> >> Any other known issues around bridge(4) or promiscuous mode under ESXi ?
> >>
> >> Thanks for any insights,
> >>
> >>  Heinrich
>
>

-- 
Kindest regards,
Tom Smyth.


Re: bridge(4) Problems when running under ESXi ?

2020-11-29 Thread Heinrich Rebehn
Some things I forgot:

All interfaces are UP
pf(4) ist disabled
bridge0 sees a bunch of lladdrs on em0 and one on em1, which is that of “A”

-Heinrich


> On 29. Nov 2020, at 22:29, Heinrich Rebehn  wrote:
> 
> Hi all,
> 
> I am trying to setup an OpenBSD 6.7 virtual machine under VMware ESXi 6.7 to 
> use as a filtering bridge between two virtual networks. I enabled promiscuous 
> mode for both virtual switches.
> One network is the VMnet network, which is connected to the “outside world”.
> 
> “A” ——> “B” ——> “R”
> 
> “A” is a test machine 192.168.1.152
> “B” is the bridge No IP. em0 connects to R, em1 connects to A
> “R” is the router provided by the hoster  192.168.1.1
> 
> The addresses are only examples, the actual addresses a public IPs.
> 
> When A tries to ping R, ist sends an arp request for R’s lladdr. R responds 
> with its lladdr. Tcpdump on R’s em1 suggests that it is sent out on the 
> virtual network. However, A does not see the arp reply, hence ping(8) fails.
> 
> What am I missing? While browsing the mailing list archive, I just saw that 
> vmx(4) might be a better choice, but I had not yet time to try it out.
> 
> 
> Any other known issues around bridge(4) or promiscuous mode under ESXi ?
> 
> Thanks for any insights,
> 
>   Heinrich
> 
> 
> 



Re: bridge(4) Problems when running under ESXi ?

2020-11-29 Thread Heinrich Rebehn
Unfortunately, switching to vmx(4) did *not* do the trick

-Heinrich


> On 29. Nov 2020, at 22:38, Heinrich Rebehn  wrote:
> 
> Some things I forgot:
> 
> All interfaces are UP
> pf(4) ist disabled
> bridge0 sees a bunch of lladdrs on em0 and one on em1, which is that of “A”
> 
> -Heinrich
> 
> 
>> On 29. Nov 2020, at 22:29, Heinrich Rebehn > > wrote:
>> 
>> Hi all,
>> 
>> I am trying to setup an OpenBSD 6.7 virtual machine under VMware ESXi 6.7 to 
>> use as a filtering bridge between two virtual networks. I enabled 
>> promiscuous mode for both virtual switches.
>> One network is the VMnet network, which is connected to the “outside world”.
>> 
>> “A” ——> “B” ——> “R”
>> 
>> “A” is a test machine192.168.1.152
>> “B” is the bridgeNo IP. em0 connects to R, em1 connects to A
>> “R” is the router provided by the hoster 192.168.1.1
>> 
>> The addresses are only examples, the actual addresses a public IPs.
>> 
>> When A tries to ping R, ist sends an arp request for R’s lladdr. R responds 
>> with its lladdr. Tcpdump on R’s em1 suggests that it is sent out on the 
>> virtual network. However, A does not see the arp reply, hence ping(8) fails.
>> 
>> What am I missing? While browsing the mailing list archive, I just saw that 
>> vmx(4) might be a better choice, but I had not yet time to try it out.
>> 
>> 
>> Any other known issues around bridge(4) or promiscuous mode under ESXi ?
>> 
>> Thanks for any insights,
>> 
>>  Heinrich