Re: permanent ARP being overwritten by ISP

[Martin Pieuchot]

On 25/10/16(Tue) 03:27, Doug Moss wrote:
> On Wednesday, January 20, 2016 1:37 PM, Martin Pieuchot  
> wrote:
> >If you're referring to my reply, I was interested in the behavior in a
> >-current kernel, what will be 5.9 soon.  A lot of changes happened
> >since 5.8.
> >
> >It would be nice if you could also post the output of "route -n show -inet"
> 
> >with such kernel.
> 
> I have brought up this issue before, but I think I have narrowed down the 
> possible causes/scenario,
> and I have my machine up to date with the current release.
> 
> I think my question is:
> on my openbsd 6.0 amd64 machine, where I already have an accurate ARP entry 
> that has been manually
> set as permanent, when the sshd daemon receives a connection, why does that 
> machine broadcast
> an ARP 'who-has' for the IP address of the SSH client machine?

Good question.  Could you rebuild arp(8) with the diff attached and show
me the output of "arp -an" after triggering the 'who-has'?  I'd like to
know if some code path set the expiration timer of your permanent entry.

Index: arp.c
===
RCS file: /cvs/src/usr.sbin/arp/arp.c,v
retrieving revision 1.76
diff -u -p -r1.76 arp.c
--- arp.c   27 Aug 2016 04:15:52 -  1.76
+++ arp.c   4 Nov 2016 10:03:34 -
@@ -556,7 +556,7 @@ print_entry(struct sockaddr_dl *sdl, str
 
if (rtm->rtm_flags & (RTF_PERMANENT_ARP|RTF_LOCAL))
printf(" %-10.10s", "permanent");
-   else if (rtm->rtm_rmx.rmx_expire == 0)
+   if (rtm->rtm_rmx.rmx_expire == 0)
printf(" %-10.10s", "static");
else if (rtm->rtm_rmx.rmx_expire > now.tv_sec)
printf(" %-10.10s",

Re: permanent ARP being overwritten by ISP

[Mihai Popescu]

> My question is, why?

Since that is a machine controlled by your ISP, they can do whatever
they want or do not want. Do not believe all ISP are respecting
Internet standards. Are there standards? Maybe it is a mistake in
configuration. If I remember correctly from some time ago when I read
tcp/ip illustrated but not, some kind of ARP server can be set up,
maybe bridge related, but i'm not totally sure.

So, try to bug your ISP with that ARP overwrite.

Re: permanent ARP being overwritten by ISP

[Doug Moss]

On Wednesday, January 20, 2016 1:37 PM, Martin Pieuchot  
wrote:
>If you're referring to my reply, I was interested in the behavior in a
>-current kernel, what will be 5.9 soon.  A lot of changes happened
>since 5.8.
>
>It would be nice if you could also post the output of "route -n show -inet"

>with such kernel.

I have brought up this issue before, but I think I have narrowed down the 
possible causes/scenario,
and I have my machine up to date with the current release.

I think my question is:
on my openbsd 6.0 amd64 machine, where I already have an accurate ARP entry 
that has been manually
set as permanent, when the sshd daemon receives a connection, why does that 
machine broadcast
an ARP 'who-has' for the IP address of the SSH client machine?

This was sparked by seeing entries in my /var/log/messages like:
Oct 22 23:50:00 www /bsd: arp: attempt to overwrite permanent entry for 
70.20.25.26 by fa:c0:01:75:98:cd on em0


The details are, as best I can summarize:

network diagram:

   

-- |  
|192.168.1.x

--ISP 70.20.25.1|switch  | |  |
|SG200-18| router  |  | home LAN
||70.20.25.26 / 192.168.1.1|switch|
|| 
||70.20.25.28
||
||70.20.25.29
||
||70.20.25.30
--  OBSD 6.0 amd64 release



ethernet IDs:
fa:c0:01:75:98:cd   70.20.25.1  FIOS gateway
00:25:90:0A:69:B6   70.20.25.26 my router - external
00:25:90:0A:69:B7   192.168.1.1 my router - internal
00:25:90:EA:52:9C   70.20.25.30
00:30:48:DC:1E:35   70.20.25.28
00:30:48:DC:75:DF   70.20.25.29


I have wanted to leave nothing to chance regarding IP to ethernet ID mapping so,
on 70.20.25.30
rc.local has:
arp -F -s 70.20.25.26 00:25:90:0a:69:b6 permanent
arp -F -s 70.20.25.28 00:30:48:dc:1e:35 permanent
arp -F -s 70.20.25.29 00:30:48:dc:75:df permanent

on 70.20.25.30
# route -n show -inet
Routing tables
Internet:
Destination Gateway FlagsRefs   UseMtu   Prio 
Iface
default 70.20.25.1  UGS 4 16498  -  8 
em0 
224/4   127.0.0.1   URS 0 0  32768  8 
lo0 
70.20.25/24 70.20.25.30 UC  1 47033  -  4 
em0 
70.20.25.1  fa:c0:01:75:98:cd   UHLc1 24835  -  4 
em0 
70.20.25.26 00:25:90:0a:69:b6   UHLS3   1   657  - L8 
em0 
70.20.25.28 00:30:48:dc:1e:35   UHLS3   0   590  -  8 
em0 
70.20.25.29 00:30:48:dc:75:df   UHLS3   0   592  -  8 
em0 
70.20.25.30 00:25:90:ea:52:9c   UHLl0  7904  -  1 
em0 
70.20.25.25570.20.25.30 UHb 0 0  -  1 
em0 
127/8   127.0.0.1   UGRS0 0  32768  8 
lo0 
127.0.0.1   127.0.0.1   UHl 1   393  32768  1 
lo0 


When I initiate an SSH client connection from 70.20.25.26 to 70.20.25.30:

at 22 Oct 23:50, ssh from 70.20.25.26 to www..org (70.20.25.30)
#ssh user1@www..org


Then, having logged into 70.20.25.30
/var/log/authlog shows:
Oct 22 23:50:04 www sshd[5107]: Accepted password for user1 from 70.20.25.26 
port 8477 ssh2

/var/log/messages shows:
Oct 22 23:50:00 www /bsd: arp: attempt to overwrite permanent entry for 
70.20.25.26 by fa:c0:01:75:98:cd on em0


tcpdump log of arp traffic shows:
Oct 22 23:50:00.885770 00:25:90:ea:52:9c ff:ff:ff:ff:ff:ff 0806 42: arp who-has 
70.20.25.26 tell 70.20.25.30
Oct 22 23:50:00.885893 00:25:90:0a:69:b6 00:25:90:ea:52:9c 0806 60: arp reply 
70.20.25.26 is-at 00:25:90:0a:69:b6
Oct 22 23:50:00.886738 fa:c0:01:75:98:cd 00:25:90:ea:52:9c 0806 60: arp reply 
70.20.25.26 is-at fa:c0:01:75:98:cd


This looks to me like 00:25:90:ea:52:9c (which is 70.20.25.30) is broadcasting 
an arp 'who-has' requesting the
ethernet ID for 70.20.25.26 (SSH client)
Apropriately, 00:25:90:0a:69:b6 replies, but the FIOS gateway fa:c0:01:75:98:cd 
also replies,
and tries to pretend it is 70.20.25.26

My question is, why?
I have put the ethernet/IP address in as permanent in the arp table, the 
routing table shows it is there.
So why, when I open an SSH connection, is the 70.20.25.30 machine asking for 
the ethernet address of
the client machine?

If I didn't have the rc.local arp commands, there might have been the situation 
where the FIOS gateway
interposes itself.

I should point out that this is not exactly reproducible - this is after 
several weeks of running the machine,
and I looked for 

Re: permanent ARP being overwritten by ISP

[Martin Pieuchot]

On 20/01/16(Wed) 00:11, Doug Moss wrote:
> [...]
> Second - per other reply. I upgraded from OpenBSD 5.7 amd64 to OpenBSD 5.8 
> amd64 yesterday

If you're referring to my reply, I was interested in the behavior in a
-current kernel, what will be 5.9 soon.  A lot of changes happened
since 5.8.

It would be nice if you could also post the output of "route -n show -inet"
with such kernel.

Re: permanent ARP being overwritten by ISP

[Doug Moss]

>On Wednesday, January 20, 2016 1:37 PM, Martin Pieuchot  
>wrote:

>On 20/01/16(Wed) 00:11, Doug Moss wrote:
>> Second - per other reply. I upgraded from OpenBSD 5.7 amd64 to OpenBSD 5.8 
>> amd64 yesterday>
>If you're referring to my reply, I was interested in the behavior in a
>-current kernel, what will be 5.9 soon.  A lot of changes happened
>since 5.8.
>
>It would be nice if you could also post the output of "route -n show -inet"
>with such kernel.

I cannot go to -current, but here is from 5.8amd64:

$ uname -a
OpenBSD www..org 5.8 GENERIC.MP#1236 amd64
$ route -n show -inet
Routing tables

Internet:
DestinationGatewayFlags   Refs  Use   Mtu  Prio Iface
default70.20.25.1 UGS414448 - 8 em0 
70.20.25/2470.20.25.30UC 20 - 8 em0 
70.20.25.1 fa:c0:01:75:98:cd  UHLc   10 - 8 em0 
70.20.25.26fa:c0:01:75:98:cd  UHLc   1  144 - 8 em0 
70.20.25.3000:25:90:ea:52:9c  HLl00 - 1 lo0 
70.20.25.255   70.20.25.30UHb00 - 1 em0 
127/8  127.0.0.1  UGRS   00 32768 8 lo0 
127.0.0.1  127.0.0.1  UHl1 7899 32768 1 lo0 
224/4  127.0.0.1  URS00 32768 8 lo0 
$ arp -an
Host Ethernet Address   Netif Expire Flags
70.20.25.1   fa:c0:01:75:98:cdem0 19m49s 
70.20.25.26  fa:c0:01:75:98:cdem0 18m42s 
70.20.25.30  00:25:90:ea:52:9cem0 permanent  l

Re: permanent ARP being overwritten by ISP

[Stuart Henderson]

On 2016-01-20, Doug Moss  wrote:
> Second - per other reply. I upgraded from OpenBSD 5.7 amd64 to OpenBSD 5.8 
> amd64 yesterday
> This broke other things/packages
> (OpenLDAP 2.4 to OpenLDAP 3.0, doesn't seem to like slapd.conf 
> password-hash={CRYPT} )
> setting me back a day, but

There is no OpenLDAP 3.0. Could you describe the problem in more
detail please? slappasswd is known not to work properly on OpenBSD
(same for 5.7 as 5.8) but other than that 'password-hash {CRYPT}'
is expected to work and is working for me with OpenLDAP 2.4.

Re: permanent ARP being overwritten by ISP

[Doug Moss]

>On Sunday, January 17, 2016 2:23 PM, Vijay Sankar  wrote:

>Not clear from your message so I was wondering if you have all the following 
>on the same switch
>ISP interface
>External interface of your firewall
>Internal interface of your firewall
>Interfaces of your other systems
>I noticed behaviour similar to what you described when I did something like 
>the above. 
>The arp rewrite attempts stopped when I separated the Internet connection and 
>the external
>interface of the firewall on one switch and all the internal systems on 
>another switch. 

Yes - for my situation, one switch handles the external interfaces 
(ISP=70.20.25.1 and
my router=70.20.25.26 and my webserver=70.20.25.30)
and the other ethernet port of my router (192.168.1.x) goes to a physically 
separate other switch

Second - per other reply. I upgraded from OpenBSD 5.7 amd64 to OpenBSD 5.8 
amd64 yesterday
This broke other things/packages
(OpenLDAP 2.4 to OpenLDAP 3.0, doesn't seem to like slapd.conf 
password-hash={CRYPT} )
setting me back a day, but

the problem still occurs on OpenBSD 5.8 amd64

/var/log/messages from today:
Jan 19 05:44:42 www httpd[27728]: server_accept_tls: TLS accept failed - accept 
failed: error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol
Jan 19 07:53:54 www /bsd: arp: attempt to overwrite permanent entry for 
70.20.25.26 by fa:c0:01:75:98:cd on em0
Jan 19 08:13:59 www /bsd: arp: attempt to overwrite permanent entry for 
70.20.25.26 by fa:c0:01:75:98:cd on em0
Jan 19 09:58:46 www httpd[27728]: server_accept_tls: TLS accept failed - accept 
failed: error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol
Jan 19 15:00:01 www syslogd: restart
Jan 19 18:27:05 www /bsd: arp info overwritten for 70.20.25.26 by 
fa:c0:01:75:98:cd on em0


$ arp -an
Host Ethernet Address   Netif Expire Flags
70.20.25.1   fa:c0:01:75:98:cdem0 19m59s 
70.20.25.26  fa:c0:01:75:98:cdem0 20m0s 
70.20.25.30  00:25:90:ea:52:9cem0 permanent  l


If people would like, I can send my dmesg.
I'd be happy to try other debugging methods.
With all the warnings about -current on http://www.openbsd.org/faq/faq5.html
I'm leary of doing that - sorry.

Out of curiousity - these changes to the routing tables
visible with 'arp -an' and 'route -n show'
I imagine these can happen through more than one mechanism, and happen at the
network stack or kernel level?
Is there another mechanism that I should pay attention to?




>> On Jan 16, 2016, at 12:40, Doug Moss  wrote:
>> 
>> (my apologies for last message - unfamiliar with Yahoo and forcing plain 
>> text email)
>> 
>> Why is a manually entered permanent arp entry being overwritten?
>> 
>> 
>> At my home, I have an ISP from which I have 5 static IPv4 addresses.
>> I use these for my home network, a home email server, jabber server for 
>> family/friends,
>> website related to my academic work, etc, with different domains.
>> 
>> 
>> The ISP service comes into my home via an ethernet cable which I connect to 
>> a switch
>> (Cisco gigabit)
>> 
>> Connected to the switch are:
>> (A) router to my home network (behind which are desktops, a wireless access 
>> point, kids laptops, etc)
>> a low-power, dual NIC OpenBSD amd64 running NAT and unbound (caching)
>> with IP address 70.20.25.26
>> (B) the academic website
>> a low-power, OpenBSD 5.7 amd64
>> with IP address 70.20.25.30
>> (plus other servers)
>> 
>> The ISP gateway/router is IP address 70.20.25.1
>> 
>> On the academic website, I noticed that the arp table
>> showed 70.20.25.26 with the MAC of the ISP gateway
>> 
>> I thought - why should my private traffic from my personal webserver be 
>> routed
>> through the ISP gateway - why not go directly to my home network on the same 
>> switch?
>> 
>> So on my webserver, I did this:
>> # sudo arp -s 70.20.25.26 00:25:90:0A:69:B6 permanent
>> 
>> Then I checked:
>> # arp -an
>> Host Ethernet Address   Netif Expire 
>> Flags
>> 70.20.25.1   fa:c0:01:75:98:cdem0 19m59s 
>> 70.20.25.26  00:25:90:0a:69:b6em0 permanent 
>> 70.20.25.30  00:25:90:ea:52:9cem0 permanent  l
>> 
>> The next day, I found this is the logs:
>> Jan 12 08:17:54 www /bsd: arp info overwritten for 70.20.25.26 by 
>> 00:25:90:0a:69:b6 on em0
>> Jan 12 08:17:54 www /bsd: arp info overwritten for 70.20.25.26 by 
>> fa:c0:01:75:98:cd on em0
>> Jan 12 08:37:54 www /bsd: arp info overwritten for 70.20.25.26 by 
>> 00:25:90:0a:69:b6 on em0
>> Jan 12 08:37:54 www /bsd: arp info overwritten for 70.20.25.26 by 
>> fa:c0:01:75:98:cd on em0
>> Jan 12 08:57:54 www /bsd: arp info overwritten for 70.20.25.26 by 
>> 00:25:90:0a:69:b6 on em0
>> Jan 12 08:57:54 www /bsd: arp info overwritten for 70.20.25.26 by 
>> fa:c0:01:75:98:cd on em0
>> (repeated 

Re: permanent ARP being overwritten by ISP

[Vijay Sankar]

Not clear from your message so I was wondering if you have all the following
on the same switch

ISP interface
External interface of your firewall
Internal interface of your firewall
Interfaces of your other systems

I noticed behaviour similar to what you described when I did something like
the above.

The arp rewrite attempts stopped when I separated the Internet connection and
the external interface of the firewall on one switch and all the internal
systems on another switch.

Vijay

Sent from my iPhone

> On Jan 16, 2016, at 12:40, Doug Moss  wrote:
>
> (my apologies for last message - unfamiliar with Yahoo and forcing plain
text email)
>
> Why is a manually entered permanent arp entry being overwritten?
>
>
> At my home, I have an ISP from which I have 5 static IPv4 addresses.
> I use these for my home network, a home email server, jabber server for
family/friends,
> website related to my academic work, etc, with different domains.
>
>
> The ISP service comes into my home via an ethernet cable which I connect to
a switch
> (Cisco gigabit)
>
> Connected to the switch are:
> (A) router to my home network (behind which are desktops, a wireless access
point, kids laptops, etc)
> a low-power, dual NIC OpenBSD amd64 running NAT and unbound (caching)
> with IP address 70.20.25.26
> (B) the academic website
> a low-power, OpenBSD 5.7 amd64
> with IP address 70.20.25.30
> (plus other servers)
>
> The ISP gateway/router is IP address 70.20.25.1
>
> On the academic website, I noticed that the arp table
> showed 70.20.25.26 with the MAC of the ISP gateway
>
> I thought - why should my private traffic from my personal webserver be
routed
> through the ISP gateway - why not go directly to my home network on the same
switch?
>
> So on my webserver, I did this:
> # sudo arp -s 70.20.25.26 00:25:90:0A:69:B6 permanent
>
> Then I checked:
> # arp -an
> Host Ethernet Address   Netif Expire
Flags
> 70.20.25.1   fa:c0:01:75:98:cdem0 19m59s
> 70.20.25.26  00:25:90:0a:69:b6em0 permanent
> 70.20.25.30  00:25:90:ea:52:9cem0 permanent  l
>
> The next day, I found this is the logs:
> Jan 12 08:17:54 www /bsd: arp info overwritten for 70.20.25.26 by
00:25:90:0a:69:b6 on em0
> Jan 12 08:17:54 www /bsd: arp info overwritten for 70.20.25.26 by
fa:c0:01:75:98:cd on em0
> Jan 12 08:37:54 www /bsd: arp info overwritten for 70.20.25.26 by
00:25:90:0a:69:b6 on em0
> Jan 12 08:37:54 www /bsd: arp info overwritten for 70.20.25.26 by
fa:c0:01:75:98:cd on em0
> Jan 12 08:57:54 www /bsd: arp info overwritten for 70.20.25.26 by
00:25:90:0a:69:b6 on em0
> Jan 12 08:57:54 www /bsd: arp info overwritten for 70.20.25.26 by
fa:c0:01:75:98:cd on em0
> (repeated a couple hundred times)
>
> $ arp -an
> Host Ethernet Address   Netif Expire
Flags
> 70.20.25.1   fa:c0:01:75:98:cdem0 19m54s
> 70.20.25.26  fa:c0:01:75:98:cdem0 17m15s
> 70.20.25.30  00:25:90:ea:52:9cem0 permanent  l
>
> and
> $ traceroute 70.20.25.26
> traceroute to 70.20.25.26 (70.20.25.26), 64 hops max, 40 byte packets
> 1  lo0-100.BSTNMA-VFTTP-308.verizon-gni.net (70.20.25.1)  2.841 ms  0.594 ms
3.724 ms
> 2  static-70-20-25-26.bstnma.fios.verizon.net (70.20.25.26)  3.544 ms  1.255
ms  3.593 ms
>
> Am I understanding this correctly?
> Is the ISP gateway continuing to try to re-direct the arp table on my home
router
> to route traffic out to its gateway before coming back to my home network,
instead of
> directly from my router to the other server connected to ports on the same
switch?
>
>
> Have I done something wrong in my configuration?
>
> Is this (a) expected (b) strange but innocent (c) nefarious, or (d)
something else?

Re: permanent ARP being overwritten by ISP

[Martin Pieuchot]

On 16/01/16(Sat) 18:40, Doug Moss wrote:
> (my apologies for last message - unfamiliar with Yahoo and forcing plain text 
> email)
> 
> Why is a manually entered permanent arp entry being overwritten?

It should not, are you running -current?  If not could you try?

> 
> At my home, I have an ISP from which I have 5 static IPv4 addresses.
> I use these for my home network, a home email server, jabber server for 
> family/friends,
> website related to my academic work, etc, with different domains.
> 
> 
> The ISP service comes into my home via an ethernet cable which I connect to a 
> switch
> (Cisco gigabit)
> 
> Connected to the switch are:
> (A) router to my home network (behind which are desktops, a wireless access 
> point, kids laptops, etc)
>  a low-power, dual NIC OpenBSD amd64 running NAT and unbound (caching)
>  with IP address 70.20.25.26
> (B) the academic website
>  a low-power, OpenBSD 5.7 amd64
>  with IP address 70.20.25.30
> (plus other servers)
> 
> The ISP gateway/router is IP address 70.20.25.1
> 
> On the academic website, I noticed that the arp table
> showed 70.20.25.26 with the MAC of the ISP gateway
> 
> I thought - why should my private traffic from my personal webserver be routed
> through the ISP gateway - why not go directly to my home network on the same 
> switch?
> 
> So on my webserver, I did this:
> # sudo arp -s 70.20.25.26 00:25:90:0A:69:B6 permanent
> 
> Then I checked:
> # arp -an
> Host Ethernet Address   Netif Expire Flags
> 70.20.25.1   fa:c0:01:75:98:cdem0 19m59s 
> 70.20.25.26  00:25:90:0a:69:b6em0 permanent 
> 70.20.25.30  00:25:90:ea:52:9cem0 permanent  l
> 
> The next day, I found this is the logs:
> Jan 12 08:17:54 www /bsd: arp info overwritten for 70.20.25.26 by 
> 00:25:90:0a:69:b6 on em0
> Jan 12 08:17:54 www /bsd: arp info overwritten for 70.20.25.26 by 
> fa:c0:01:75:98:cd on em0
> Jan 12 08:37:54 www /bsd: arp info overwritten for 70.20.25.26 by 
> 00:25:90:0a:69:b6 on em0
> Jan 12 08:37:54 www /bsd: arp info overwritten for 70.20.25.26 by 
> fa:c0:01:75:98:cd on em0
> Jan 12 08:57:54 www /bsd: arp info overwritten for 70.20.25.26 by 
> 00:25:90:0a:69:b6 on em0
> Jan 12 08:57:54 www /bsd: arp info overwritten for 70.20.25.26 by 
> fa:c0:01:75:98:cd on em0
> (repeated a couple hundred times)
> 
> $ arp -an
> Host Ethernet Address   Netif Expire Flags
> 70.20.25.1   fa:c0:01:75:98:cdem0 19m54s 
> 70.20.25.26  fa:c0:01:75:98:cdem0 17m15s 
> 70.20.25.30  00:25:90:ea:52:9cem0 permanent  l
> 
> and
> $ traceroute 70.20.25.26
> traceroute to 70.20.25.26 (70.20.25.26), 64 hops max, 40 byte packets
> 1  lo0-100.BSTNMA-VFTTP-308.verizon-gni.net (70.20.25.1)  2.841 ms  0.594 ms  
> 3.724 ms
> 2  static-70-20-25-26.bstnma.fios.verizon.net (70.20.25.26)  3.544 ms  1.255 
> ms  3.593 ms
> 
> Am I understanding this correctly?
> Is the ISP gateway continuing to try to re-direct the arp table on my home 
> router
> to route traffic out to its gateway before coming back to my home network, 
> instead of
> directly from my router to the other server connected to ports on the same 
> switch?
> 
> 
> Have I done something wrong in my configuration?
> 
> Is this (a) expected (b) strange but innocent (c) nefarious, or (d) something 
> else?

permanent ARP being overwritten by ISP

[Doug Moss]

(my apologies for last message - unfamiliar with Yahoo and forcing plain text 
email)

Why is a manually entered permanent arp entry being overwritten?


At my home, I have an ISP from which I have 5 static IPv4 addresses.
I use these for my home network, a home email server, jabber server for 
family/friends,
website related to my academic work, etc, with different domains.


The ISP service comes into my home via an ethernet cable which I connect to a 
switch
(Cisco gigabit)

Connected to the switch are:
(A) router to my home network (behind which are desktops, a wireless access 
point, kids laptops, etc)
 a low-power, dual NIC OpenBSD amd64 running NAT and unbound (caching)
 with IP address 70.20.25.26
(B) the academic website
 a low-power, OpenBSD 5.7 amd64
 with IP address 70.20.25.30
(plus other servers)

The ISP gateway/router is IP address 70.20.25.1

On the academic website, I noticed that the arp table
showed 70.20.25.26 with the MAC of the ISP gateway

I thought - why should my private traffic from my personal webserver be routed
through the ISP gateway - why not go directly to my home network on the same 
switch?

So on my webserver, I did this:
# sudo arp -s 70.20.25.26 00:25:90:0A:69:B6 permanent

Then I checked:
# arp -an
Host Ethernet Address   Netif Expire Flags
70.20.25.1   fa:c0:01:75:98:cdem0 19m59s 
70.20.25.26  00:25:90:0a:69:b6em0 permanent 
70.20.25.30  00:25:90:ea:52:9cem0 permanent  l

The next day, I found this is the logs:
Jan 12 08:17:54 www /bsd: arp info overwritten for 70.20.25.26 by 
00:25:90:0a:69:b6 on em0
Jan 12 08:17:54 www /bsd: arp info overwritten for 70.20.25.26 by 
fa:c0:01:75:98:cd on em0
Jan 12 08:37:54 www /bsd: arp info overwritten for 70.20.25.26 by 
00:25:90:0a:69:b6 on em0
Jan 12 08:37:54 www /bsd: arp info overwritten for 70.20.25.26 by 
fa:c0:01:75:98:cd on em0
Jan 12 08:57:54 www /bsd: arp info overwritten for 70.20.25.26 by 
00:25:90:0a:69:b6 on em0
Jan 12 08:57:54 www /bsd: arp info overwritten for 70.20.25.26 by 
fa:c0:01:75:98:cd on em0
(repeated a couple hundred times)

$ arp -an
Host Ethernet Address   Netif Expire Flags
70.20.25.1   fa:c0:01:75:98:cdem0 19m54s 
70.20.25.26  fa:c0:01:75:98:cdem0 17m15s 
70.20.25.30  00:25:90:ea:52:9cem0 permanent  l

and
$ traceroute 70.20.25.26
traceroute to 70.20.25.26 (70.20.25.26), 64 hops max, 40 byte packets
1  lo0-100.BSTNMA-VFTTP-308.verizon-gni.net (70.20.25.1)  2.841 ms  0.594 ms  
3.724 ms
2  static-70-20-25-26.bstnma.fios.verizon.net (70.20.25.26)  3.544 ms  1.255 ms 
 3.593 ms

Am I understanding this correctly?
Is the ISP gateway continuing to try to re-direct the arp table on my home 
router
to route traffic out to its gateway before coming back to my home network, 
instead of
directly from my router to the other server connected to ports on the same 
switch?


Have I done something wrong in my configuration?

Is this (a) expected (b) strange but innocent (c) nefarious, or (d) something 
else?

permanent ARP being overwritten by ISP

[Doug Moss]

Why is a manually entered permanent arp entry being overwritten?

At my home, I have an ISP from which I have 5 static IPv4 addresses.I use
these for my home network, a home email server, jabber server for
family/friends,website related to my academic work, etc, with different
domains.

The ISP service comes into my home via an ethernet cable which I connect to a
switch (Cisco gigabit)

Connected to the switch are:(A) router to my home network (behind which are
desktops, a wireless access point, kids laptops, etc) a low-power, dual NIC
OpenBSD amd64 running NAT and unbound (caching) with IP address
70.20.25.26(B) the academic website a low-power, OpenBSD 5.7 amd64 with IP
address 70.20.25.30(plus other servers)
The ISP gateway/router is IP address 70.20.25.1

On the academic website, I noticed that the arp table showed 70.20.25.26 with
the MAC of the ISP gateway
I thought - why should my private traffic from my personal webserver be
routed through the ISP gateway - why not go directly to my home network on
the same switch?
So on my webserver, I did this:# sudo arp -s 70.20.25.26 00:25:90:0A:69:B6
permanent
Then I checked:# arp -anHost                                
Ethernet Address   Netif Expire     Flags70.20.25.1                
          fa:c0:01:75:98:cd    em0 19m59s     70.20.25.26      
                   00:25:90:0a:69:b6    em0 permanent
 70.20.25.30                          00:25:90:ea:52:9c    em0
permanent  l
The next day, I found this is the logs:Jan 12 08:17:54 www /bsd: arp info
overwritten for 70.20.25.26 by 00:25:90:0a:69:b6 on em0Jan 12 08:17:54 www
/bsd: arp info overwritten for 70.20.25.26 by fa:c0:01:75:98:cd on em0Jan 12
08:37:54 www /bsd: arp info overwritten for 70.20.25.26 by 00:25:90:0a:69:b6
on em0Jan 12 08:37:54 www /bsd: arp info overwritten for 70.20.25.26 by
fa:c0:01:75:98:cd on em0Jan 12 08:57:54 www /bsd: arp info overwritten for
70.20.25.26 by 00:25:90:0a:69:b6 on em0Jan 12 08:57:54 www /bsd: arp info
overwritten for 70.20.25.26 by fa:c0:01:75:98:cd on em0(repeated a couple
hundred times)
$ arp -anHost                                 Ethernet Address
  Netif Expire     Flags70.20.25.1                          
fa:c0:01:75:98:cd    em0 19m54s     70.20.25.26                
         fa:c0:01:75:98:cd    em0 17m15s     70.20.25.30        
                 00:25:90:ea:52:9c    em0 permanent  l
and$ traceroute 70.20.25.26traceroute to 70.20.25.26 (70.20.25.26), 64 hops
max, 40 byte packets 1  lo0-100.BSTNMA-VFTTP-308.verizon-gni.net
(70.20.25.1)  2.841 ms  0.594 ms  3.724 ms 2
 static-70-20-25-26.bstnma.fios.verizon.net (70.20.25.26)  3.544 ms  1.255
ms  3.593 ms
Am I understanding this correctly?Is the ISP gateway continuing to try to
re-direct the arp table on my personal serverto route traffic out to its
gateway before coming back to my home network, instead of directlyfrom my
server to my router connected to ports on the same switch?
Have I done something wrong in my configuration?

Since on my webserver (70.20.25.30) I use the ISP's provided name servers,
does the name-mapping-to-IP(in /etc/resolv.conf) impact the IP-mapping-to-MAC
of the local ARP tables?
Is this (a) expected (b) strange but innocent (c) nefarious, or (d) something
else?
thanks in advance for considering this.