Re: Cannot access internet with virtual switch

[Aham Brahmasmi]

Thank you Koshibe-san for your reply.

Here is the output of ping, after the steps:
$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
ping: sendmsg: Network is down
ping: wrote 8.8.8.8 64 chars, ret=-1
...

So, it seems the ping fails, except, this time there is some output.


> > Interestingly, while searching for addlocal, I encountered a
> > presentation on switch [1]. On page 13 of that pdf, there is mention of
> > the switch sharing the STP code with bridge. Would it be correct to
> > assume that there would be no loops if there was STP in the switch?
> 
> Even with a bridge, you'd need to enable STP and set priority values
> on the ports for it to work, so you're correct - if there were any
> loops, the bridge probably wouldn't have worked either. But you've
> also seen that, for switch(4), the STP-related options aren't
> available in ifconfig, and as far as I can tell switchd doesn't do
> topology/loop detection (and probably won't want to rely on (R)STP to
> do so). So, the code might be shared, but is likely not used.

I do not know much about the network stack, but I went grepping in the
source, and I encountered a bstp_create function call in the
switch_clone_create function within sys/net/if_switch.c file. This
bstp_create function seems to be defined in sys/net/bridgestp.c [2].
I may be wrong here, but the bridge seems to have some kind of STP,
since there is the "rstp" in the ifconfig output. Whether the switch
does indeed use STP, and if it does, does it work, is something that I
unfortunately cannot determine.

For tap0, I ran "tcpdump -nettti tap0" on a normal machine, without
the above vether. I saw a large stream of messages. Then I ran it on
em0 interface. The "tcpdump -nettti em0" closely matched the output of
tap0 interface output. This led me to try to understand tap interfaces.
I encountered an article [3] and an image [4]. My hunch is that tap0
is sort of the mirror equivalent of em0. In terms of that image, em0
is the green physical NIC and tap0 is the Virtual Uplink Port. And since
they are connected to two ends of the same (imaginary) cable, they will
have the same set of messages. This leads me to believe that I should
pass all traffic on tap0. Again, this is based on my uneducated guess
and searching, so I could be wrong.

Passing all traffic on tap0 still does not lead to the ping reply.
However, doing a tcpdump on em0 shows an echo request and echo reply
whiile tcpdump on tap0 only shows an echo reply. This is irrespective of
addlocal vether0. Also, this is irrespective of the pf configuration.
Now, why does the echo request not show up on tap0 and what is exactly
stopping the echo reply to reach the ping command - I cannot determine.
I have done the usual block log, tcpdump -netti pflog0 routine. There
is infrequent igmp2 output, but nothing related to icmp. In fact, I
have also used a one line "pass" file for pf configuration, still no
ping reply.

Regards,
ab
[1] - 
http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/sys/net/if_switch.c?rev=1.23=text/x-cvsweb-markup
[2] - 
http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/sys/net/bridgestp.c?rev=1.65=text/x-cvsweb-markup
[3] - www.innervoice.in/blogs/2013/12/08/tap-interfaces-linux-bridge/
[4] - 
https://i2.wp.com/www.innervoice.in/blogs/wp-content/uploads/2013/12/VirtualNetwotk.png
-|-|-|-|-|-|-|--

Re: Cannot access internet with virtual switch

[Ayaka Koshibe]

On Mon, May 14, 2018 at 1:01 PM, Aham Brahmasmi  wrote:
> Thank you Koshibe-san for your reply.
>
>> I've actually held back on that diff since it's a bit insufficient by itself.
>
> Ok.
>
>> Actually, you said that you had just em0 on that switch. Can you try
>> adding a local port (addlocal instead of add) alongside em0? It will
>> be a vether(4) interface that needs to be given em0's current address,
>> in its place.
>
> Should I be doing the following? And if yes, what address should em0
> have?
>
> $ cat /etc/hostname.vether0
> inet 1.2.3.4 255.255.255.0
> $ cat /etc/hostname.em0
> inet ?.?.?.? 255.255.255.0
> $ doas ifconfig switch0 create
> $ doas ifconfig switch0 add em0
> $ doas ifconfig switch0 addlocal vether0
> $ doas ifconfig switch0 up
>
> Here, 1.2.3.4 is the external public IP address of the machine
> originally assigned to em0.

em0 shouldn't have an address, and you'll also want to explicitly
enable vether0. Otherwise that looks fine.

>> > There is a continuous stream of messages when running "switchd -dvv":
>> > ...
>>
>> I can't say what they are without the full output, but you will tend
>> to see broadcasts (periodic or otherwise) like your second example
>> even on your bridge. From a second look at your earlier logs, it seems
>> the 1->1 'loops' are generated by the switch seeing VLAN traffic in
>> other parts of the network.
>
> Ok. Would sharing the full output of "switchd -dvv" help?

I wouldn't worry much about it, unless adding the local port doesn't
work for you.

> Interestingly, while searching for addlocal, I encountered a
> presentation on switch [1]. On page 13 of that pdf, there is mention of
> the switch sharing the STP code with bridge. Would it be correct to
> assume that there would be no loops if there was STP in the switch?

Even with a bridge, you'd need to enable STP and set priority values
on the ports for it to work, so you're correct - if there were any
loops, the bridge probably wouldn't have worked either. But you've
also seen that, for switch(4), the STP-related options aren't
available in ifconfig, and as far as I can tell switchd doesn't do
topology/loop detection (and probably won't want to rely on (R)STP to
do so). So, the code might be shared, but is likely not used.


> Regards,
> ab
> [1] - https://www.openbsd.org/papers/bsdcan2016-switchd.pdf
> -|-|-|-|-|-|-|--

Re: Cannot access internet with virtual switch

[Aham Brahmasmi]

Thank you Koshibe-san for your reply.

> I've actually held back on that diff since it's a bit insufficient by itself.

Ok.
 
> Actually, you said that you had just em0 on that switch. Can you try
> adding a local port (addlocal instead of add) alongside em0? It will
> be a vether(4) interface that needs to be given em0's current address,
> in its place.

Should I be doing the following? And if yes, what address should em0
have?

$ cat /etc/hostname.vether0
inet 1.2.3.4 255.255.255.0
$ cat /etc/hostname.em0
inet ?.?.?.? 255.255.255.0
$ doas ifconfig switch0 create
$ doas ifconfig switch0 add em0
$ doas ifconfig switch0 addlocal vether0
$ doas ifconfig switch0 up

Here, 1.2.3.4 is the external public IP address of the machine
originally assigned to em0.

> > There is a continuous stream of messages when running "switchd -dvv":
> > ...
> 
> I can't say what they are without the full output, but you will tend
> to see broadcasts (periodic or otherwise) like your second example
> even on your bridge. From a second look at your earlier logs, it seems
> the 1->1 'loops' are generated by the switch seeing VLAN traffic in
> other parts of the network.

Ok. Would sharing the full output of "switchd -dvv" help?

Interestingly, while searching for addlocal, I encountered a
presentation on switch [1]. On page 13 of that pdf, there is mention of
the switch sharing the STP code with bridge. Would it be correct to
assume that there would be no loops if there was STP in the switch?

Regards,
ab
[1] - https://www.openbsd.org/papers/bsdcan2016-switchd.pdf
-|-|-|-|-|-|-|--

Re: Cannot access internet with virtual switch

[Ayaka Koshibe]

> tap0 is a control interface created by switchd to communicate with
> switches. It won't carry normal network traffic.

(Last bit of noise, and I'm done)
Actually, I think I'm wrong here. I'll need to dig a bit more to
answer correctly.

Re: Cannot access internet with virtual switch

[Ayaka Koshibe]

On Sat, May 12, 2018 at 8:54 PM, Ayaka Koshibe  wrote:
>> Currently, I am using a bridge. From what I understand from the patch
>> and the cvsweb history, that patch is waiting for ok and commit.
>
> I've actually held back on that diff since it's a bit insufficient by itself.

Sorry, I should elaborate. If you plan to try the local port, you may
still want the patch in place to prevent the switch from
disconnecting.

Re: Cannot access internet with virtual switch

[Ayaka Koshibe]

> Currently, I am using a bridge. From what I understand from the patch
> and the cvsweb history, that patch is waiting for ok and commit.

I've actually held back on that diff since it's a bit insufficient by itself.

> This time, the switch does not close. However, I am still unable to
> ping 8.8.8.8 with the switch. As usual, I am able to ping 8.8.8.8
> using a bridge.

Actually, you said that you had just em0 on that switch. Can you try
adding a local port (addlocal instead of add) alongside em0? It will
be a vether(4) interface that needs to be given em0's current address,
in its place.

> There is a continuous stream of messages when running "switchd -dvv":
> ...

I can't say what they are without the full output, but you will tend
to see broadcasts (periodic or otherwise) like your second example
even on your bridge. From a second look at your earlier logs, it seems
the 1->1 'loops' are generated by the switch seeing VLAN traffic in
other parts of the network.

> Finally, I have a query - What does the tap0 interface do? I ask this
> because currently I do not pass in/out tap0 traffic in pf. This is
> because I do not know what tap0 is and what it does.

tap0 is a control interface created by switchd to communicate with
switches. It won't carry normal network traffic.

Re: Cannot access internet with virtual switch

[Aham Brahmasmi]

> > $ cat /etc/hostname.switch0
> > add em0
> > up
> >
> > Here, em0 is the egress interface connected to the dedicated/bare-metal
> > machine provider's network. This provider's network is beyond my
> > control. As such, there might be a loop in the provider's network.
> 
> (Sorry, was meaning to respond but got busy)
> That's my suspicion, the network you're connected to, rather than your
> switch. Say, if you are still trying to use switch(4), there's a
> chance that this might help:
> 
> https://marc.info/?l=openbsd-tech=152424475112610=2

Thank you Koshibe-san for your reply.

Please, no sorry. I am grateful for your help.

Currently, I am using a bridge. From what I understand from the patch
and the cvsweb history, that patch is waiting for ok and commit.

This is my first time running a patch, so I may have messed up the
steps. Based on some inspiration from [1], I have used the following
steps for running the patch on -current:

$ doas user mod -G wsrc 
$ exit

$ cd /usr
$ cvs -qd anon...@anoncvs.ca.openbsd.org:/cvs checkout -P src
$ cd /usr/src
$ cvs -q up -Pd -A
$ cd /usr/src/usr.sbin/switchd
$ vi ofp13.c

$ doas user mod -G wobj 
$ exit

$ cd /usr/src/usr.sbin/switchd
$ make obj
$ make
$ doas make install

Then the usual switch0, switchd and switchctl steps, as mentioned in
my previous mails.

This time, the switch does not close. However, I am still unable to
ping 8.8.8.8 with the switch. As usual, I am able to ping 8.8.8.8
using a bridge.

There is a continuous stream of messages when running "switchd -dvv":
...
switch_learn: updated mac 11:11:11:11:11:11 on switch 1 port 1
packet_input: 11:11:11:11:11:11 -> 22:22:22:22:22:22, port 1 -> 1
ofrelay_input_done: connection 1.1: 1528 bytes from switch 1
/dev/switch0 > any: version 1_3 type PACKET_IN length 1528 xid 11362
buffer NO_BUFFER length 1478 reason REASON_NO_MATCH table <0> cookie 0x0
000
match type OXM length 24 (padded to 26)
ox match class OPENFLOW_BASIC type IN_PORT hasmask no length 4
1
ox match class OPENFLOW_BASIC type META hasmask no length 8
0


...
switch_learn: updated mac 22:22:22:22:22:22 on switch 1 port 1
packet_input: 22:22:22:22:22:22 -> ff:ff:ff:ff:ff:ff, port 1 -> 4294967295
any > /dev/switch0: version 1_3 type PACKET_OUT length 100 xid 10
buffer NO_BUFFER in_port <1> actions_len 16
action OUTPUT len 16 port FLOOD max_len NO_BUFFER
ofrelay_input_done: connection 1.1: 110 bytes from switch 1
...
 1 messages with incrementing xids 11366...>
 4294967295 messages with incrementing xids 11...>

I have noticed in the ifconfig output that the bridge has "rstp" while
there is no "rstp" or "stp" for the switch. It may be possible that the
provider has redundant internet routes, and that the bridge with the
spanning tree protocol might be able to handle it while the switch
may not be. This is just my guess as a layman user and I may be wrong
since I do not know whether spanning tree protocol is applicable for
the switch. I base this on the absence of stp options in the man pages
for switch/switchd/switchctl/switchd.conf.

Furthermore, the output of "switchctl show summary" displays a large
number of macs with ages within 10 seconds, although the switch has
been up for upwards of 1000 seconds. These macs seem to never go above
10 seconds. Two macs in particular are always age 0s. There is only one
switch - /dev/switch0.

Finally, I have a query - What does the tap0 interface do? I ask this
because currently I do not pass in/out tap0 traffic in pf. This is
because I do not know what tap0 is and what it does. With a "block log"
in pf and "tcpdump -nettti pflog0", I get the following output:

tcpdump: WARNING: snaplen raised from 116 to 160
tcpdump: listening on pflog0, link-type PFLOG
 rule 1/(match) block in on tap0: >
239.255.255.250.1900: udp 94 (DF) [tos 0x38] [ttl 1]
...


If I run a ping 8.8.8.8, the ping packets do not show up in the above
output.

Regards,
ab
[1] - https://ftp.openbsd.org/pub/OpenBSD/patches/6.3/common/005_httpd.patch.sig
-|-|-|-|-|-|-|--

Re: Cannot access internet with virtual switch

[Ayaka Koshibe]

> $ cat /etc/hostname.switch0
> add em0
> up
>
> Here, em0 is the egress interface connected to the dedicated/bare-metal
> machine provider's network. This provider's network is beyond my
> control. As such, there might be a loop in the provider's network.

(Sorry, was meaning to respond but got busy)
That's my suspicion, the network you're connected to, rather than your
switch. Say, if you are still trying to use switch(4), there's a
chance that this might help:

https://marc.info/?l=openbsd-tech=152424475112610=2

Re: Cannot access internet with virtual switch

[Aham Brahmasmi]

> Sent: Thursday, April 12, 2018 at 11:24 AM
> From: "Ayaka Koshibe" <akosh...@gmail.com>
> To: misc@openbsd.org
> Subject: Re: Cannot access internet with virtual switch
>
> On Wed, Apr 11, 2018 at 6:25 AM, Aham Brahmasmi <aham.brahma...@gmx.com> 
> wrote:
> >> Sent: Wednesday, April 11, 2018 at 10:18 AM
> >> From: "Ayaka Koshibe" <akosh...@gmail.com>
> >> To: misc@openbsd.org
> >> Subject: Re: Cannot access internet with virtual switch
> >>
> >> > This informs us that for a PACKET_OUT with action OUTPUT, it cannot
> >> > have its port as ANY. Now, I do not know why for a PACKET_OUT message,
> >> > an action OUTPUT cannot have port as ANY. More importantly, I do not
> >> > know why the controller seems to be sending the PACKET_OUT with action
> >> > OUTPUT and port ANY.
> >>
> >> A PACKET_OUT is usually a response to some message e.g. a PACKET_IN,
> >> so it would probably help to see which message (if any) the switch
> >> sent to the controller to receive that PACKET_OUT.
> >
> > Thank you Koshibe-san for your reply.
> >
> > From what I understand, the PACKET_IN for that PACKET_OUT seems to be
> > the following:
> >
> > ofrelay_input_done: connection 1.1: 179 bytes from switch 1
> > /dev/switch0 > any: version 1_3 type PACKET_IN length 179 xid 81972
> > buffer NO_BUFFER length 129 reason REASON_NO_MATCH table <0> cookie 
> > 0x
> > match type OXM length 24 (padded to 26)
> > ox match class OPENFLOW_BASIC type IN_PORT hasmask no length 4
> > 1
> > ox match class OPENFLOW_BASIC type META hasmask no length 8
> > 0
> > switch_learn: updated mac ac:1f:6b:2e:22:ce on switch 1 port 1
> > packet_input: ac:1f:6b:2e:22:ce -> 00:c8:8b:e2:d6:87, port 1 -> 1
> 
> This seems to be the right message. It looks like switchd will set the
> output port to ANY if it sees a loop (which port 1 -> 1 suggests),
> intended for dropping the packet. Do you have loops?

Thank you Koshibe-san for your reply.

I looked up switch loops in order to understand your insight. Based on
the configuration of hostname.switch0, I do not think there is a loop
in the virtual switch. However, I may be wrong since I am not good at
networks.

$ cat /etc/hostname.switch0
add em0
up

Here, em0 is the egress interface connected to the dedicated/bare-metal
machine provider's network. This provider's network is beyond my
control. As such, there might be a loop in the provider's network.

Is there a way that I can verify if there is a loop in the network?

If it helps in any way, a bridge works on the same network.

$ cat /etc/hostname.bridge0
add em0

Regards,
ab
-|-|-|-|-|-|-|--

Re: Cannot access internet with virtual switch

[Ayaka Koshibe]

On Wed, Apr 11, 2018 at 6:25 AM, Aham Brahmasmi <aham.brahma...@gmx.com> wrote:
>> Sent: Wednesday, April 11, 2018 at 10:18 AM
>> From: "Ayaka Koshibe" <akosh...@gmail.com>
>> To: misc@openbsd.org
>> Subject: Re: Cannot access internet with virtual switch
>>
>> > This informs us that for a PACKET_OUT with action OUTPUT, it cannot
>> > have its port as ANY. Now, I do not know why for a PACKET_OUT message,
>> > an action OUTPUT cannot have port as ANY. More importantly, I do not
>> > know why the controller seems to be sending the PACKET_OUT with action
>> > OUTPUT and port ANY.
>>
>> A PACKET_OUT is usually a response to some message e.g. a PACKET_IN,
>> so it would probably help to see which message (if any) the switch
>> sent to the controller to receive that PACKET_OUT.
>
> Thank you Koshibe-san for your reply.
>
> From what I understand, the PACKET_IN for that PACKET_OUT seems to be
> the following:
>
> ofrelay_input_done: connection 1.1: 179 bytes from switch 1
> /dev/switch0 > any: version 1_3 type PACKET_IN length 179 xid 81972
> buffer NO_BUFFER length 129 reason REASON_NO_MATCH table <0> cookie 
> 0x
> match type OXM length 24 (padded to 26)
> ox match class OPENFLOW_BASIC type IN_PORT hasmask no length 4
> 1
> ox match class OPENFLOW_BASIC type META hasmask no length 8
> 0
> switch_learn: updated mac ac:1f:6b:2e:22:ce on switch 1 port 1
> packet_input: ac:1f:6b:2e:22:ce -> 00:c8:8b:e2:d6:87, port 1 -> 1

This seems to be the right message. It looks like switchd will set the
output port to ANY if it sees a loop (which port 1 -> 1 suggests),
intended for dropping the packet. Do you have loops?

> I have also attached the complete output of "doas switchd -d". This
> is because I do not know whether the above message is the correct
> PACKET_OUT message corresponding to the PACKET_IN message.
>
> Regards,
> ab
> -|-|-|-|-|-|-|--

Re: Cannot access internet with virtual switch

[Aham Brahmasmi]

> Sent: Wednesday, April 11, 2018 at 10:18 AM
> From: "Ayaka Koshibe" <akosh...@gmail.com>
> To: misc@openbsd.org
> Subject: Re: Cannot access internet with virtual switch
>
> > This informs us that for a PACKET_OUT with action OUTPUT, it cannot
> > have its port as ANY. Now, I do not know why for a PACKET_OUT message,
> > an action OUTPUT cannot have port as ANY. More importantly, I do not
> > know why the controller seems to be sending the PACKET_OUT with action
> > OUTPUT and port ANY.
> 
> A PACKET_OUT is usually a response to some message e.g. a PACKET_IN,
> so it would probably help to see which message (if any) the switch
> sent to the controller to receive that PACKET_OUT.

Thank you Koshibe-san for your reply.

>From what I understand, the PACKET_IN for that PACKET_OUT seems to be
the following:

ofrelay_input_done: connection 1.1: 179 bytes from switch 1
/dev/switch0 > any: version 1_3 type PACKET_IN length 179 xid 81972
buffer NO_BUFFER length 129 reason REASON_NO_MATCH table <0> cookie 
0x
match type OXM length 24 (padded to 26)
ox match class OPENFLOW_BASIC type IN_PORT hasmask no length 4
1
ox match class OPENFLOW_BASIC type META hasmask no length 8
0
switch_learn: updated mac ac:1f:6b:2e:22:ce on switch 1 port 1
packet_input: ac:1f:6b:2e:22:ce -> 00:c8:8b:e2:d6:87, port 1 -> 1

I have also attached the complete output of "doas switchd -d". This
is because I do not know whether the above message is the correct
PACKET_OUT message corresponding to the PACKET_IN message.

Regards,
ab
-|-|-|-|-|-|-|--
listen on 0.0.0.0:6633
ofrelay_attach: new connection 1.1
ofp_open: new connection 1.1 from switch 0
any > /dev/switch0: version 1_3 type HELLO length 16 xid 0
	version bitmap:
		version 1_0
		version 1_3
0408 00014025
ofrelay_input_done: connection 1.1: 8 bytes from switch 0
0408 00014025
/dev/switch0 > any: version 1_3 type HELLO length 8 xid 81957
any > /dev/switch0: version 1_3 type FEATURES_REQUEST length 8 xid 1
04060020 0001
55330647 2d30f2ae  fe00 000f 
ofrelay_input_done: connection 1.1: 32 bytes from switch 1
04060020 0001 55330647 2d30f2ae  fe00 000f 
/dev/switch0 > any: version 1_3 type FEATURES_REPLY length 32 xid 1
	datapath_id 0x553306472d30f2ae nbuffers 0 ntables 254 aux_id 0 capabilities 0x0f
any > /dev/switch0: version 1_3 type MULTIPART_REQUEST length 16 xid 2
	type TABLE_FEATURES flags 
	empty table properties request
any > /dev/switch0: version 1_3 type SET_CONFIG length 12 xid 3
	flags  miss_send_len NO_BUFFER
04130418 0002
000c  0408     
       
 2710 0018 00010004 00020004 00030004 00040004 00050004
00010018 00010004 00020004 00030004 00040004 00050004 00060018 00120004
00110004 00190004 00160004 0004 00070018 00120004 00110004 00190004
00160004 0004 00040018 00120004 00110004 00190004 00160004 0004
00050018 00120004 00110004 00190004 00160004 0004 0008008c 8004
8508 8706 8906 8a02 8d02 8e01 80001001 80001201
80001401 80001704 80001904 80001a02 80001c02 80001e02 80002002 80002202
80002402 80002601 80002801 80002a02 80002d04 80002f04 80003106 80003306
80003510 80003710 80003904 80003a01 80003c01 80003e10 80004006 80004206
80004c08  000a0088 8408 8606 8806 8a02 8c02
8e01 80001001 80001201 80001401 80001604 80001804 80001a02 80001c02
80001e02 80002002 80002202 80002402 80002601 80002801 80002a02 80002c04
80002e04 80003006 80003206 80003410 80003610 80003804 80003a01 80003c01
80003e10 80004006 80004206 80004c08 000c0084 8606 8806 8a02
8c02 8e01 80001001 80001201 80001401 80001604 80001804 80001a02
80001c02 80001e02 80002002 80002202 80002402 80002601 80002801 80002a02
80002c04 80002e04 80003006 80003206 80003410 80003610 80003804 80003a01
80003c01 80003e10 80004006 80004206 80004c08  000d0084 8606
8806 8a02 8c02 8e01 80001001 80001201 80001401 80001604
80001804 80001a02 80001c02 80001e02 80002002 80002202 80002402 80002601
80002801 80002a02 80002c04 80002e04 80003006 80003206 80003410 80003610
80003804 80003a01 80003c01 80003e10 80004006 80004206 80004c08 
000e0084 8606 8806 8a02 8c02 8e01 80001001 80001201
80001401 80001604 80001804 80001a02 80001c02 80001e02 80002002 80002202
80002402 80002601 80002801 80002a02 80002c04 80002e04 80003006 80003206
80003410 80003610 80003804 80003a01 80003c01 80003e10 80004006 80004206
80004c08  000f0084 8606 8806 8a02 8c02 8e01
80001001 80001201 80001401 80001604 80001804 80001a02 80001c02 80001e

Re: Cannot access internet with virtual switch

[Ayaka Koshibe]

> This informs us that for a PACKET_OUT with action OUTPUT, it cannot
> have its port as ANY. Now, I do not know why for a PACKET_OUT message,
> an action OUTPUT cannot have port as ANY. More importantly, I do not
> know why the controller seems to be sending the PACKET_OUT with action
> OUTPUT and port ANY.

A PACKET_OUT is usually a response to some message e.g. a PACKET_IN,
so it would probably help to see which message (if any) the switch
sent to the controller to receive that PACKET_OUT.

Re: Cannot access internet with virtual switch

[Aham Brahmasmi]

> Sent: Monday, April 09, 2018 at 6:50 PM
> From: "Aham Brahmasmi" <aham.brahma...@gmx.com>
> To: misc@openbsd.org
> Subject: Re: Cannot access internet with virtual switch
>
> > Sent: Saturday, April 07, 2018 at 5:02 AM
> > From: "Ayaka Koshibe" <akosh...@gmail.com>
> > To: "Aham Brahmasmi" <aham.brahma...@gmx.com>
> > Cc: misc@openbsd.org
> > Subject: Re: Cannot access internet with virtual switch
> >
> > On Fri, Apr 6, 2018 at 4:40 PM, Aham Brahmasmi <aham.brahma...@gmx.com> 
> > wrote:
> > > Hello misc,
> > >
> > > Problem
> > > A physical server with a switch (add em0 up) cannot access the internet.
> > > However, the same host with a bridge (add em0 up) can access the
> > > internet.
> > >
> > > Steps
> > > $ ifconfig
> > > em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
> > > lladdr 22:22:22:22:22:22
> > > index 1 priority 0 llprio 3
> > > groups: egress
> > > media: Ethernet autoselect (1000baseT full-duplex,master)
> > > status: active
> > > inet 20.20.20.20 netmask 0xff00 broadcast 20.20.20.255
> > > ...
> > > $ doas route -n show
> > > Routing tables
> > >
> > > Internet:
> > > Destination GatewayFlags   Refs  Use   Mtu  Prio Iface
> > > default 20.20.20.1 UGS0 1XXX - 8 em0
> > > 224/4   127.0.0.1  URS00 32768 8 lo0
> > > 127/8   127.0.0.1  UGRS   00 32768 8 lo0
> > > 127.0.0.1   127.0.0.1  UHhl   1X 32768 1 lo0
> > > 20.20.20/24 20.20.20.20UCn1  9XX - 4 em0
> > > 20.20.20.1  33:33:33:33:33:33  UHLch  1 1XXX - 3 em0
> > > 20.20.20.20 44:44:44:44:44:44  UHLl   0X - 1 em0
> > > 20.20.20.25520.20.20.20UHb00 - 1 em0
> > > $ ping 8.8.8.8
> > > PING 8.8.8.8 (8.8.8.8): 56 data bytes
> > > 64 bytes from 8.8.8.8: icmp_seq=0 ttl=61 time=x.xxx ms
> > > ...
> > > $ doas ifconfig switch0 create
> > > $ doas ifconfig switch0 add em0
> > > $ doas ifconfig switch0 up
> > > $ ping 8.8.8.8
> > > PING 8.8.8.8 (8.8.8.8): 56 data bytes
> > > ^C
> > > --- 8.8.8.8 ping statistics ---
> > > 31 packets transmitted, 0 packets received, 100.0% packet loss
> > 
> > Hi,
> > 
> > Seems you haven't started switchd(8), or connected your switch to it
> > -- it shouldn't forward traffic until you do so.
> 
> 
> Hi Koshibe-san,
> 
> Thank you for your reply.
> 
> I have started switchd and connected to it. However, I still cannot
> ping 8.8.8.8. Starting switchd in debug mode results in output which
> broadly says error and closes the switch.
> 
> Steps (after the above switch0 up)
> $ cat /etc/switchd.conf
> listen on 0.0.0.0 tls port 6633
> $ doas switchd -d
> listen on 0.0.0.0 6633
> 
> (On another session)
> $ switchctl connect /dev/switch0
> 
> (Back to main session)
> ofrelay_input_done: ...
> /dev/switch0 > any: ...
> switch_learn: ...
> packet_input: ...
> any > /dev/switch0: ...
> (above block repeated multiple times)
> ...
> ofrelay_input_done: connection 1.1: 76 bytes from switch 1
> 0401004c 0013 00020004 040d00a9 0013  0001 0010
>  0010    00c88be2 d687ac1f 6b2e22ce
> 8100026f 08004500 006f42d2
> /dev/switch0 > any: version 1_3 type ERROR length 76 xid 19
> error type BAD_ACTION code 4
> ofp13_input: message not supported: ERROR
> ofrelay_close: connection 1.1 closed
> switch_remove: switch 1 removed.
> 
> (Another session)
> $ tail -10 /var/log/messages
> Apr 9 XX:XX:XX MachineName /bsd: arp: attempt to add entry for GATEWAY_IP
> on em0 by XX:XX:XX:XX:XX:XX on tap0
> (above message repeated infrequently)
> 
> If it helps in any way, this machine is a dedicated/bare-metal machine
> on a large dedicated/bare-metal machine provider's network. The em0
> interface is in the egress group, has a public IP and is connected to
> the internet via the provider's network equipment.
> 
> The end goal in using the switch is to enable multiple OpenBSD VM's with
> with non-contiguous public IPs to be connected to the Internet as real
> hosts. In https://www.openbsd.org/faq/faq6.html#VMMnet, this is the
> Option 4, except using a switch

Re: Cannot access internet with virtual switch

[Aham Brahmasmi]

> Sent: Saturday, April 07, 2018 at 5:02 AM
> From: "Ayaka Koshibe" <akosh...@gmail.com>
> To: "Aham Brahmasmi" <aham.brahma...@gmx.com>
> Cc: misc@openbsd.org
> Subject: Re: Cannot access internet with virtual switch
>
> On Fri, Apr 6, 2018 at 4:40 PM, Aham Brahmasmi <aham.brahma...@gmx.com> wrote:
> > Hello misc,
> >
> > Problem
> > A physical server with a switch (add em0 up) cannot access the internet.
> > However, the same host with a bridge (add em0 up) can access the
> > internet.
> >
> > Steps
> > $ ifconfig
> > em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
> > lladdr 22:22:22:22:22:22
> > index 1 priority 0 llprio 3
> > groups: egress
> > media: Ethernet autoselect (1000baseT full-duplex,master)
> > status: active
> > inet 20.20.20.20 netmask 0xff00 broadcast 20.20.20.255
> > ...
> > $ doas route -n show
> > Routing tables
> >
> > Internet:
> > Destination GatewayFlags   Refs  Use   Mtu  Prio Iface
> > default 20.20.20.1 UGS0 1XXX - 8 em0
> > 224/4   127.0.0.1  URS00 32768 8 lo0
> > 127/8   127.0.0.1  UGRS   00 32768 8 lo0
> > 127.0.0.1   127.0.0.1  UHhl   1X 32768 1 lo0
> > 20.20.20/24 20.20.20.20UCn1  9XX - 4 em0
> > 20.20.20.1  33:33:33:33:33:33  UHLch  1 1XXX - 3 em0
> > 20.20.20.20 44:44:44:44:44:44  UHLl   0X - 1 em0
> > 20.20.20.25520.20.20.20UHb00 - 1 em0
> > $ ping 8.8.8.8
> > PING 8.8.8.8 (8.8.8.8): 56 data bytes
> > 64 bytes from 8.8.8.8: icmp_seq=0 ttl=61 time=x.xxx ms
> > ...
> > $ doas ifconfig switch0 create
> > $ doas ifconfig switch0 add em0
> > $ doas ifconfig switch0 up
> > $ ping 8.8.8.8
> > PING 8.8.8.8 (8.8.8.8): 56 data bytes
> > ^C
> > --- 8.8.8.8 ping statistics ---
> > 31 packets transmitted, 0 packets received, 100.0% packet loss
> 
> Hi,
> 
> Seems you haven't started switchd(8), or connected your switch to it
> -- it shouldn't forward traffic until you do so.


Hi Koshibe-san,

Thank you for your reply.

I have started switchd and connected to it. However, I still cannot
ping 8.8.8.8. Starting switchd in debug mode results in output which
broadly says error and closes the switch.

Steps (after the above switch0 up)
$ cat /etc/switchd.conf
listen on 0.0.0.0 tls port 6633
$ doas switchd -d
listen on 0.0.0.0 6633

(On another session)
$ switchctl connect /dev/switch0

(Back to main session)
ofrelay_input_done: ...
/dev/switch0 > any: ...
switch_learn: ...
packet_input: ...
any > /dev/switch0: ...
(above block repeated multiple times)
...
ofrelay_input_done: connection 1.1: 76 bytes from switch 1
0401004c 0013 00020004 040d00a9 0013  0001 0010
 0010    00c88be2 d687ac1f 6b2e22ce
8100026f 08004500 006f42d2
/dev/switch0 > any: version 1_3 type ERROR length 76 xid 19
error type BAD_ACTION code 4
ofp13_input: message not supported: ERROR
ofrelay_close: connection 1.1 closed
switch_remove: switch 1 removed.

(Another session)
$ tail -10 /var/log/messages
Apr 9 XX:XX:XX MachineName /bsd: arp: attempt to add entry for GATEWAY_IP
on em0 by XX:XX:XX:XX:XX:XX on tap0
(above message repeated infrequently)

If it helps in any way, this machine is a dedicated/bare-metal machine
on a large dedicated/bare-metal machine provider's network. The em0
interface is in the egress group, has a public IP and is connected to
the internet via the provider's network equipment.

The end goal in using the switch is to enable multiple OpenBSD VM's with
with non-contiguous public IPs to be connected to the Internet as real
hosts. In https://www.openbsd.org/faq/faq6.html#VMMnet, this is the
Option 4, except using a switch instead of a bridge and public IPs
on the host network.

Regards,
ab
-|-|-|-|-|-|-|--

> 
> > $ ifconfig
> > em0: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> 
> > mtu 1500
> > lladdr 22:22:22:22:22:22
> > index 1 priority 0 llprio 3
> > groups: egress
> > media: Ethernet autoselect (1000baseT full-duplex,master)
> > status: active
> > inet 20.20.20.20 netmask 0xff00 broadcast 20.20.20.255
> > switch0: flags=41<UP,RUNNING>
> > index 6 llprio 3
> > groups: switch
> > datapath xx maxflow 1 m

Re: Cannot access internet with virtual switch

[Ayaka Koshibe]

On Fri, Apr 6, 2018 at 4:40 PM, Aham Brahmasmi  wrote:
> Hello misc,
>
> Problem
> A physical server with a switch (add em0 up) cannot access the internet.
> However, the same host with a bridge (add em0 up) can access the
> internet.
>
> Steps
> $ ifconfig
> em0: flags=8843 mtu 1500
> lladdr 22:22:22:22:22:22
> index 1 priority 0 llprio 3
> groups: egress
> media: Ethernet autoselect (1000baseT full-duplex,master)
> status: active
> inet 20.20.20.20 netmask 0xff00 broadcast 20.20.20.255
> ...
> $ doas route -n show
> Routing tables
>
> Internet:
> Destination GatewayFlags   Refs  Use   Mtu  Prio Iface
> default 20.20.20.1 UGS0 1XXX - 8 em0
> 224/4   127.0.0.1  URS00 32768 8 lo0
> 127/8   127.0.0.1  UGRS   00 32768 8 lo0
> 127.0.0.1   127.0.0.1  UHhl   1X 32768 1 lo0
> 20.20.20/24 20.20.20.20UCn1  9XX - 4 em0
> 20.20.20.1  33:33:33:33:33:33  UHLch  1 1XXX - 3 em0
> 20.20.20.20 44:44:44:44:44:44  UHLl   0X - 1 em0
> 20.20.20.25520.20.20.20UHb00 - 1 em0
> $ ping 8.8.8.8
> PING 8.8.8.8 (8.8.8.8): 56 data bytes
> 64 bytes from 8.8.8.8: icmp_seq=0 ttl=61 time=x.xxx ms
> ...
> $ doas ifconfig switch0 create
> $ doas ifconfig switch0 add em0
> $ doas ifconfig switch0 up
> $ ping 8.8.8.8
> PING 8.8.8.8 (8.8.8.8): 56 data bytes
> ^C
> --- 8.8.8.8 ping statistics ---
> 31 packets transmitted, 0 packets received, 100.0% packet loss

Hi,

Seems you haven't started switchd(8), or connected your switch to it
-- it shouldn't forward traffic until you do so.

> $ ifconfig
> em0: flags=8b43 mtu 
> 1500
> lladdr 22:22:22:22:22:22
> index 1 priority 0 llprio 3
> groups: egress
> media: Ethernet autoselect (1000baseT full-duplex,master)
> status: active
> inet 20.20.20.20 netmask 0xff00 broadcast 20.20.20.255
> switch0: flags=41
> index 6 llprio 3
> groups: switch
> datapath xx maxflow 1 maxgroup 1000
> em0 flags=0<>
> port 1 ifpriority 0 ifcost 0
> ...
> $ doas route -n show
> Routing tables
>
> Internet:
> Destination GatewayFlags   Refs  Use   Mtu  Prio Iface
> default 20.20.20.1 UGS0 1XXX - 8 em0
> 224/4   127.0.0.1  URS00 32768 8 lo0
> 127/8   127.0.0.1  UGRS   00 32768 8 lo0
> 127.0.0.1   127.0.0.1  UHhl   1X 32768 1 lo0
> 20.20.20/24 20.20.20.20UCn1  9XX - 4 em0
> 20.20.20.1  33:33:33:33:33:33  UHLch  1 1XXX - 3 em0
> 20.20.20.20 44:44:44:44:44:44  UHLl   0X - 1 em0
> 20.20.20.25520.20.20.20UHb00 - 1 em0
> $ doas ifconfig switch0 destroy
> $ ping 8.8.8.8
> PING 8.8.8.8 (8.8.8.8): 56 data bytes
> 64 bytes from 8.8.8.8: icmp_seq=0 ttl=61 time=x.xxx ms
>
> Repeating the above steps with bridge0 does let the ping pass through
> after the bridge is brought up. The only delta between the switch and
> bridge output is in the ifconfig.
> $ ifconfig
> bridge0: flags=41
> index 8 llprio 3
> groups: bridge
> priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rtsp
> em0 flags=3
> port 1 ifpriority 0 ifcost 0
> ...
>
> I have run "doas route -n monitor" in a separate session while doing
> this. However, I cannot comprehend the output. pf is not involved -
> running tcpdump -nettti pflog0 with the catchall "block log" produces
> the normal output of blocked packets with the bridge. However, it stops
> producing the normal output of blocked packets with the switch. Once the
> switch is destroyed, it is back to normal blocked packets output.
>
> What am I doing wrong/missing? The only thing that stands out to me is
> the em0 flags=0<> line in the ifconfig for the switch. And I do not know
> what to make of it.
>
> Regards,
> ab
> -|-|-|-|-|-|-|--
>

Cannot access internet with virtual switch

[Aham Brahmasmi]

Hello misc,

Problem
A physical server with a switch (add em0 up) cannot access the internet.
However, the same host with a bridge (add em0 up) can access the
internet.

Steps
$ ifconfig
em0: flags=8843 mtu 1500
lladdr 22:22:22:22:22:22
index 1 priority 0 llprio 3
groups: egress
media: Ethernet autoselect (1000baseT full-duplex,master)
status: active
inet 20.20.20.20 netmask 0xff00 broadcast 20.20.20.255
...
$ doas route -n show
Routing tables

Internet:
Destination GatewayFlags   Refs  Use   Mtu  Prio Iface
default 20.20.20.1 UGS0 1XXX - 8 em0
224/4   127.0.0.1  URS00 32768 8 lo0
127/8   127.0.0.1  UGRS   00 32768 8 lo0
127.0.0.1   127.0.0.1  UHhl   1X 32768 1 lo0
20.20.20/24 20.20.20.20UCn1  9XX - 4 em0
20.20.20.1  33:33:33:33:33:33  UHLch  1 1XXX - 3 em0
20.20.20.20 44:44:44:44:44:44  UHLl   0X - 1 em0
20.20.20.25520.20.20.20UHb00 - 1 em0
$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: icmp_seq=0 ttl=61 time=x.xxx ms
...
$ doas ifconfig switch0 create
$ doas ifconfig switch0 add em0
$ doas ifconfig switch0 up
$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
^C
--- 8.8.8.8 ping statistics ---
31 packets transmitted, 0 packets received, 100.0% packet loss
$ ifconfig
em0: flags=8b43 mtu 
1500
lladdr 22:22:22:22:22:22
index 1 priority 0 llprio 3
groups: egress
media: Ethernet autoselect (1000baseT full-duplex,master)
status: active
inet 20.20.20.20 netmask 0xff00 broadcast 20.20.20.255
switch0: flags=41
index 6 llprio 3
groups: switch
datapath xx maxflow 1 maxgroup 1000
em0 flags=0<>
port 1 ifpriority 0 ifcost 0
...
$ doas route -n show
Routing tables

Internet:
Destination GatewayFlags   Refs  Use   Mtu  Prio Iface
default 20.20.20.1 UGS0 1XXX - 8 em0
224/4   127.0.0.1  URS00 32768 8 lo0
127/8   127.0.0.1  UGRS   00 32768 8 lo0
127.0.0.1   127.0.0.1  UHhl   1X 32768 1 lo0
20.20.20/24 20.20.20.20UCn1  9XX - 4 em0
20.20.20.1  33:33:33:33:33:33  UHLch  1 1XXX - 3 em0
20.20.20.20 44:44:44:44:44:44  UHLl   0X - 1 em0
20.20.20.25520.20.20.20UHb00 - 1 em0
$ doas ifconfig switch0 destroy
$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: icmp_seq=0 ttl=61 time=x.xxx ms

Repeating the above steps with bridge0 does let the ping pass through
after the bridge is brought up. The only delta between the switch and
bridge output is in the ifconfig.
$ ifconfig
bridge0: flags=41
index 8 llprio 3
groups: bridge
priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rtsp
em0 flags=3
port 1 ifpriority 0 ifcost 0
...

I have run "doas route -n monitor" in a separate session while doing
this. However, I cannot comprehend the output. pf is not involved -
running tcpdump -nettti pflog0 with the catchall "block log" produces
the normal output of blocked packets with the bridge. However, it stops
producing the normal output of blocked packets with the switch. Once the
switch is destroyed, it is back to normal blocked packets output.

What am I doing wrong/missing? The only thing that stands out to me is
the em0 flags=0<> line in the ifconfig for the switch. And I do not know
what to make of it.

Regards,
ab
-|-|-|-|-|-|-|--