Re: high sofnet load with gif(4) and icmp

[Marc Peters]

On Thu, Apr 05, 2018 at 02:38:36PM +0200, Alexander Bluhm wrote:
> OK bluhm@
> 
> On Thu, Apr 05, 2018 at 09:14:32AM +1000, David Gwynne wrote:
> > Index: if_gif.c
> > ===
> > RCS file: /cvs/src/sys/net/if_gif.c,v
> > retrieving revision 1.113
> > diff -u -p -r1.113 if_gif.c
> > --- if_gif.c15 Mar 2018 21:01:18 -  1.113
> > +++ if_gif.c4 Apr 2018 23:12:02 -
> > @@ -403,6 +403,8 @@ gif_output(struct ifnet *ifp, struct mbu
> > error = ENOBUFS;
> > goto drop;
> > }
> > +   memcpy((caddr_t)(mtag + 1), >if_index, sizeof(ifp->if_index));
> > +   m_tag_prepend(m, mtag);
> >  
> > m->m_pkthdr.ph_family = dst->sa_family;
> >  

Is this considered a regression, which is worth an errata? We have quite a
few gif tunnels.

Marc

Re: high sofnet load with gif(4) and icmp

[Job Snijders]

Hi,

I'm optimistic about this patch: where previously running traffic
through this router over gif
tunnels would result in memory exhaustion, the problem now seems gone.
memory graph: http://instituut.net/~job/screenshots/ee7f0fa5304032a2.png

Should perhaps an errata / syspatch blob be prepared for the 6.3 users
that rely on gif(4)?
I can attempt to create one for amd64.

Kind regards,

Job

On Thu, Apr 5, 2018 at 2:38 PM, Alexander Bluhm  wrote:
> OK bluhm@
>
> On Thu, Apr 05, 2018 at 09:14:32AM +1000, David Gwynne wrote:
>> Index: if_gif.c
>> ===
>> RCS file: /cvs/src/sys/net/if_gif.c,v
>> retrieving revision 1.113
>> diff -u -p -r1.113 if_gif.c
>> --- if_gif.c  15 Mar 2018 21:01:18 -  1.113
>> +++ if_gif.c  4 Apr 2018 23:12:02 -
>> @@ -403,6 +403,8 @@ gif_output(struct ifnet *ifp, struct mbu
>>   error = ENOBUFS;
>>   goto drop;
>>   }
>> + memcpy((caddr_t)(mtag + 1), >if_index, sizeof(ifp->if_index));
>> + m_tag_prepend(m, mtag);
>>
>>   m->m_pkthdr.ph_family = dst->sa_family;
>>
>

Re: high sofnet load with gif(4) and icmp

[Alexander Bluhm]

OK bluhm@

On Thu, Apr 05, 2018 at 09:14:32AM +1000, David Gwynne wrote:
> Index: if_gif.c
> ===
> RCS file: /cvs/src/sys/net/if_gif.c,v
> retrieving revision 1.113
> diff -u -p -r1.113 if_gif.c
> --- if_gif.c  15 Mar 2018 21:01:18 -  1.113
> +++ if_gif.c  4 Apr 2018 23:12:02 -
> @@ -403,6 +403,8 @@ gif_output(struct ifnet *ifp, struct mbu
>   error = ENOBUFS;
>   goto drop;
>   }
> + memcpy((caddr_t)(mtag + 1), >if_index, sizeof(ifp->if_index));
> + m_tag_prepend(m, mtag);
>  
>   m->m_pkthdr.ph_family = dst->sa_family;
>  

Re: high sofnet load with gif(4) and icmp

[Christian Barthel]

David Gwynne  writes:

> On Wed, Apr 04, 2018 at 08:27:52PM +0200, Christian Barthel wrote:
>> Hello, 
>> 
>> After upgrading to 6.3, I've encountered a problem while using gif(4): 
>> 
>> Pre-Cond: fresh install of OpenBSD 6.3
>> # cat /etc/hostname.gif0
>> tunnel 10.0.0.1 10.0.0.2
>> 10.0.0.1 10.0.0.2
>> # sh /etc/netstart.sh gif0
>> # ping 10.0.0.2
>> 
>> 
>> Result & Behavior: the kernel thread softnet starts running on the cpu
>> (STATE: run) and consumes 100% cpu time spent in system mode.  The
>> memory consumption increases until there is no free memory (mbufpl,
>> mtagpl seem to grow). Even upon stopping/suspending the ping(1) process,
>> the sofnet thread continues to consume cpu time.  Only destroying the
>> gif0 interface changes the softnet STATE to sleep (memory seems still
>> allocated).  
>> 
>> I've discovered this on a Hetzner virtual machine (VPS) System (dmesg
>> and sysinfo attached) but i also reproduced the behavior on a Thinkpad
>> X200 (dmesg not attached). 
>> 
>> Has anyone seen something similar or is there something I am missing?  I
>> tried to backtrack (at least a bit) and I can reproduce the behavior at
>> least up to revision 1.109 (if_gif.c). Not sure if my config is absolute
>> correct but in anyway, I guess softnet shouldn't consume so much cpu
>> time in that case.  
>
> can you try this diff?

Thanks for your reply and having looked at it.  I applied the patch and
it solved the above described issue.  

Christian

>
> Index: if_gif.c
> ===
> RCS file: /cvs/src/sys/net/if_gif.c,v
> retrieving revision 1.113
> diff -u -p -r1.113 if_gif.c
> --- if_gif.c  15 Mar 2018 21:01:18 -  1.113
> +++ if_gif.c  4 Apr 2018 23:12:02 -
> @@ -403,6 +403,8 @@ gif_output(struct ifnet *ifp, struct mbu
>   error = ENOBUFS;
>   goto drop;
>   }
> + memcpy((caddr_t)(mtag + 1), >if_index, sizeof(ifp->if_index));
> + m_tag_prepend(m, mtag);
>  
>   m->m_pkthdr.ph_family = dst->sa_family;
>  
>

Re: high sofnet load with gif(4) and icmp

[David Gwynne]

On Wed, Apr 04, 2018 at 08:27:52PM +0200, Christian Barthel wrote:
> Hello, 
> 
> After upgrading to 6.3, I've encountered a problem while using gif(4): 
> 
> Pre-Cond: fresh install of OpenBSD 6.3
> # cat /etc/hostname.gif0
> tunnel 10.0.0.1 10.0.0.2
> 10.0.0.1 10.0.0.2
> # sh /etc/netstart.sh gif0
> # ping 10.0.0.2
> 
> 
> Result & Behavior: the kernel thread softnet starts running on the cpu
> (STATE: run) and consumes 100% cpu time spent in system mode.  The
> memory consumption increases until there is no free memory (mbufpl,
> mtagpl seem to grow). Even upon stopping/suspending the ping(1) process,
> the sofnet thread continues to consume cpu time.  Only destroying the
> gif0 interface changes the softnet STATE to sleep (memory seems still
> allocated).  
> 
> I've discovered this on a Hetzner virtual machine (VPS) System (dmesg
> and sysinfo attached) but i also reproduced the behavior on a Thinkpad
> X200 (dmesg not attached). 
> 
> Has anyone seen something similar or is there something I am missing?  I
> tried to backtrack (at least a bit) and I can reproduce the behavior at
> least up to revision 1.109 (if_gif.c). Not sure if my config is absolute
> correct but in anyway, I guess softnet shouldn't consume so much cpu
> time in that case.  

can you try this diff?

Index: if_gif.c
===
RCS file: /cvs/src/sys/net/if_gif.c,v
retrieving revision 1.113
diff -u -p -r1.113 if_gif.c
--- if_gif.c15 Mar 2018 21:01:18 -  1.113
+++ if_gif.c4 Apr 2018 23:12:02 -
@@ -403,6 +403,8 @@ gif_output(struct ifnet *ifp, struct mbu
error = ENOBUFS;
goto drop;
}
+   memcpy((caddr_t)(mtag + 1), >if_index, sizeof(ifp->if_index));
+   m_tag_prepend(m, mtag);
 
m->m_pkthdr.ph_family = dst->sa_family;