Re: FreeBSD 11 : running blacklistd needed for 520.pfdenied?

2016-10-13 Thread Michael Grimm 
Hi -

On 15.08.2016, at 19:01, Michael Grimm  wrote:

> I recently upgraded from 10.3-STABLE to 11.0-PRERELEASE. Now, I am missing 
> those parts in my daily security report regarding pf, e.g.:
> 
>   example.private pf denied packets:
>   +block drop in on ix0 all [ Evaluations: 12757684 Packets: 133590 
> Bytes: 7477681 States: 0 ]
>   +block drop in log quick on ix0 from  to any [ 
> Evaluations: 12754165 Packets: 3753 Bytes: 269612 States: 0 ]
>   +block drop quick on ix0 from any to  [ Evaluations: 790740 
> Packets: 873 Bytes: 295032 States: 0 ]
> 
> I do believe that those lines should be generated by 
> /etc/periodic/security/520.pfdenied (stripped to the relevant part):
> 
>   TMP=`mktemp -t security` 
>   touch ${TMP} 
>   for _a in "" blacklistd 
>   do 
>   pfctl -a ${_a} -sr -v -z 2>/dev/null | \ 
>   nawk '{if (/^block/) {buf=$0; getline; gsub(" +"," ",$0); if 
> ($5 > 0) print buf$0;} }' >> ${TMP} 
>   done 

Well, one needs to add the "old" functionality of 10.3-STABLE's 
/etc/periodic/security/520.pfdenied to get those lines reappear again. The new 
script in 11-STABLE (and presumably 11-RELEASE) assumes a running blacklistd 
which isn't necessarily the case in every installation running pf firewalls.

Patch:
++ SNIP 
++
--- 520.pfdenied2016-08-15 18:59:11.532831000 +0200
+++ 520.pfdenied.new2016-10-13 20:03:28.891362000 +0200
@@ -50,6 +50,8 @@
pfctl -a ${_a} -sr -v -z 2>/dev/null | \
nawk '{if (/^block/) {buf=$0; getline; gsub(" +"," ",$0); if 
($5 > 0) print buf$0;} }' >> ${TMP}
done
+   pfctl -sr -v 2>/dev/null | \
+   nawk '{if (/^block/) {buf=$0; getline; gsub(" +"," ",$0); if ($5 > 0) 
print buf$0;} }' >> ${TMP}
if [ -s ${TMP} ]; then
check_diff new_only pf ${TMP} "${host} pf denied packets:"
fi
++ SNAP 
++

Regards,
Michael

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: 11.0 stuck on high network load

2016-10-13 Thread Julien Charbon 
On 10/13/16 5:17 PM, Slawa Olhovchenkov wrote:
> On Thu, Oct 13, 2016 at 05:06:00PM +0200, Julien Charbon wrote:
> 
 will give you that trace in the core, and without INVARIANT then it is
 better to use dtrace:

 $ cat tcp-twstart-dropped.d
 fbt::tcp_twstart:entry
 /args[0]->t_inpcb->inp_flags & 0x0400/
 {
   stack();
   printf("INP_DROPPED in tcp_twstart: %x", args[0]->t_inpcb->inp_flags);
 }
>>>
>>> Same code may be insert there too, IMHO.
>>
>>  Hmm, I don't think so:
>>
>>  - If you have INVARIANT, the kernel will panic in tcp_twstart() or
>> tcp_detach() and you will have everything you need to debug.
>>  - If you don't, dtrace is the right tool to use in all cases anyway.
> 
> dtrace don't executed in may case w/ diagnostic "dtrace: processing
> aborted: Abort due to systemic unresponsiveness". This is for
> tcp_close. May be tcp_twstart will be more successuful, may be not.

 It does and will.

> Also, using dtrace too complex in production (need complex startup
> under screen and capture output) and for many peoples.
> kdb_backtrace() have too less administrative overhead.

 I still think it is overkill.  The main goal of this change is to fix a
quite tricky and old TCP stack locking issue.  Let's try to do that
first, it is complex enough by itself.

 Once the fix is validated and pushed, feel free to propose your own
patch/review to add kdb_backtrace(), log(), etc.. to get other devs
point of view.

 I don't remember who said: "Never ever optimize error cases"...

--
Julien



signature.asc
Description: OpenPGP digital signature

Re: 11.0 stuck on high network load

2016-10-13 Thread Slawa Olhovchenkov 
On Thu, Oct 13, 2016 at 05:06:00PM +0200, Julien Charbon wrote:

> >> will give you that trace in the core, and without INVARIANT then it is
> >> better to use dtrace:
> >>
> >> $ cat tcp-twstart-dropped.d
> >> fbt::tcp_twstart:entry
> >> /args[0]->t_inpcb->inp_flags & 0x0400/
> >> {
> >>   stack();
> >>   printf("INP_DROPPED in tcp_twstart: %x", args[0]->t_inpcb->inp_flags);
> >> }
> > 
> > Same code may be insert there too, IMHO.
> 
>  Hmm, I don't think so:
> 
>  - If you have INVARIANT, the kernel will panic in tcp_twstart() or
> tcp_detach() and you will have everything you need to debug.
>  - If you don't, dtrace is the right tool to use in all cases anyway.

dtrace don't executed in may case w/ diagnostic "dtrace: processing
aborted: Abort due to systemic unresponsiveness". This is for
tcp_close. May be tcp_twstart will be more successuful, may be not.
Also, using dtrace too complex in production (need complex startup
under screen and capture output) and for many peoples.
kdb_backtrace() have too less administrative overhead.
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: 11.0 stuck on high network load

2016-10-13 Thread Julien Charbon 

 Hi Slawa,

On 10/13/16 4:38 PM, Slawa Olhovchenkov wrote:
> On Thu, Oct 13, 2016 at 01:56:21PM +0200, Julien Charbon wrote:
  Something like:
>>>
>>> Yes, thanks!
>>
>>  Proposed changes added in the review:
>>
>> https://reviews.freebsd.org/D8211
>>
>>  tell me when you have three days without issue with this change.
>>
 tcp_detach() {

   ...
   if (inp->inp_flags & INP_TIMEWAIT) {

 ...
 if (inp->inp_flags & INP_DROPPED) {

   in_pcbdetach(inp);
   if (__predict_true(tp == NULL)) {
   in_pcbfree(inp);
   } else {
 #ifdef INVARIANTS
 panic("tcp_detach: tp != NULL, That's not good because 'blah'\n");
 #else
 log(LOG_ERR, "tcp_detach: tp != NULL, That's not good because
 'blah'\n");
>>>
>>> May be some more info in log can help to detect root cause of issuse?
>>> I am don't know what info, may be flags or number of references?
>>
>>  For this kind of issue, the useful part is the stacktrace.  INVARIANT
> 
> Like this?
> 
> #ifdef KDB
> kdb_backtrace();
> #endif
> 
> as found in sys/netgraph/ng_base.c

 It is overkill dtrace can do that.

>> will give you that trace in the core, and without INVARIANT then it is
>> better to use dtrace:
>>
>> $ cat tcp-twstart-dropped.d
>> fbt::tcp_twstart:entry
>> /args[0]->t_inpcb->inp_flags & 0x0400/
>> {
>>   stack();
>>   printf("INP_DROPPED in tcp_twstart: %x", args[0]->t_inpcb->inp_flags);
>> }
> 
> Same code may be insert there too, IMHO.

 Hmm, I don't think so:

 - If you have INVARIANT, the kernel will panic in tcp_twstart() or
tcp_detach() and you will have everything you need to debug.
 - If you don't, dtrace is the right tool to use in all cases anyway.

--
Julien



signature.asc
Description: OpenPGP digital signature

Re: 11.0 stuck on high network load

2016-10-13 Thread Slawa Olhovchenkov 
On Thu, Oct 13, 2016 at 01:56:21PM +0200, Julien Charbon wrote:

> >>  Something like:
> > 
> > Yes, thanks!
> 
>  Proposed changes added in the review:
> 
> https://reviews.freebsd.org/D8211
> 
>  tell me when you have three days without issue with this change.
> 
> >> tcp_detach() {
> >>
> >>   ...
> >>   if (inp->inp_flags & INP_TIMEWAIT) {
> >>
> >> ...
> >> if (inp->inp_flags & INP_DROPPED) {
> >>
> >>   in_pcbdetach(inp);
> >>   if (__predict_true(tp == NULL)) {
> >>   in_pcbfree(inp);
> >>   } else {
> >> #ifdef INVARIANTS
> >> panic("tcp_detach: tp != NULL, That's not good because 'blah'\n");
> >> #else
> >> log(LOG_ERR, "tcp_detach: tp != NULL, That's not good because
> >> 'blah'\n");
> > 
> > May be some more info in log can help to detect root cause of issuse?
> > I am don't know what info, may be flags or number of references?
> 
>  For this kind of issue, the useful part is the stacktrace.  INVARIANT

Like this?

#ifdef KDB
kdb_backtrace();
#endif

as found in sys/netgraph/ng_base.c

> will give you that trace in the core, and without INVARIANT then it is
> better to use dtrace:
> 
> $ cat tcp-twstart-dropped.d
> fbt::tcp_twstart:entry
> /args[0]->t_inpcb->inp_flags & 0x0400/
> {
>   stack();
>   printf("INP_DROPPED in tcp_twstart: %x", args[0]->t_inpcb->inp_flags);
> }

Same code may be insert there too, IMHO.

> --
> Julien
> 
> 



___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: ZFS l2arc broken in 10.3

2016-10-13 Thread Peter 

Pete French wrote:


Ok, thats a bit worry if true - but I can confirm that l2arc works fine
under 10.3 on amd64, so what you say about cross-compling might be true.
Am taking an inetrest in this as I have just dpeloyed a lot of machines
which are going to be relying on l2arc working to get reasobale performance.


Sure on my amd64 it also works fine. AFAIK such things are tolerated 
when compiling in 64bit.


But I was pointed to another point interim: my source is from STABLE 
branch; in the 10.3 RELEASE the code is different. Obviousely there were

recent changes, and that explains why the problem was not yet detected.

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: 11.0 stuck on high network load

2016-10-13 Thread Julien Charbon 

 Hi Slawa,

On 10/12/16 5:42 PM, Slawa Olhovchenkov wrote:
> On Wed, Oct 12, 2016 at 05:17:35PM +0200, Julien Charbon wrote:
> 
>>  I see, thus just for the context:  The TCP stack in sys/dev/cxgb* 
>> is a
>> TOE (TCP Offload Engine?) TCP stack for Chelsio NICs, it is a
>> separate/side TCP stack that is used only with TCP_OFFLOAD option.
>>
>>  This TOE TCP stack actually has its own set of detach()/input()
>> functions and seems to check INP_DROPPED flag properly.  I guess @np
>> check fixes in socket TCP stack and decides which one can also impact
>> the Chelsio TOE TCP stack.  Some bugs are only in socket TCP stack, 
>> some
>> are only in TOE TCP stack.
>
> I am fear about other direction -- setting INP_TIMEWAIT in Chelsio TOE
> TCP stack and impact this to
> tcp_timer_2msl()/tcp_close()/sofree()/tcp_usr_detach() path.

  I see, I expect no problem on this side as tcp_timer_2msl() checks the
 INP_TIMEWAIT flag and do not call tcp_close() if set.
>>>
>>> I am about case when at time of first INP_WUNLOCK() tcp_timer_2msl()
>>> don't see INP_TIMEWAIT, call tcp_close(), tcp_close() do INP_WUNLOCK()
>>> and now Chelsio TOE take INP_WLOCK, do tcp_twstart() and set
>>> INP_TIMEWAIT. After this tcp_timer_2msl resume and have unexpected
>>> INP_TIMEWAIT in tcp_usr_detach().
>>
>>  Sure, basically the same bug that in classic TCP stack.  If you think
>> it can happen, send an email describing that to np@ and he will check
>> and fix that.  He is a TOE TCP stack expert and I am not.  In all cases,
>> if this issue is possible in TOE TCP stack context, the patch will be
>> straightforward:  If the INP_DROPPED flag is set do not call 
>> tcp_twstart().
> 
> I am email to np@
> 
>>  The current patch focuses only on the classic TCP stack.
>
> May be current workaround (with logging) in tcp_usr_detach() is good
> solutuion for preventing system lockout by similar bugs?

  Good question, the quick workaround in tcp_usr_detach() does not handle
 all the cases.  If it reduces the number of crashes you can still find
 scenarios where it can have unexpected side effect.
>>>
>>> This is best then guaranted lockout.
>>>
  Long term solution is to enforce:  If the inp has the INP_DROPPED flag
 just stop processing it and return.  If you grep the INP_DROPPED flag in
 kernel sources, you can see that this test is already done in almost all
 tcp_*() processing functions but tcp_input().

  I would say that even without this issue tcp_input() should check
 INP_DROPPED flags after INP_WLOCK anyway.  Same for the TOE TCP stack,
 you are simply not supposed to process a inp with INP_DROPPED flag.
>>>
>>> Absolutly acceptant!
>>> May point is: more check and good handling of check result is best for
>>> stability.
>>>
>>> I.e. AND check INP_DROPPED in tcp_input AND workaroud INP_TIMEWAIT in
>>> tcp_usr_detach (with logging) and check of some posible cases in XXX TOE.
>>>
>>> Current TCP stack too complex and have many corner cases. This is need
>>> additional guards where posible (not caused kernel panic).
>>
>>  I see your point:  Even if this issue is caught by this assert:
>>
>> KASSERT(tp == NULL, ("tcp_detach: INP_TIMEWAIT && "
>> "INP_DROPPED && tp != NULL"));
>> https://github.com/freebsd/freebsd/blob/release/11.0.0/sys/netinet/tcp_usrreq.c#L213
>>
>>  you might not have INVARIANT option, then you will get a lockout quite
>> difficult to debug.  Thus what we can do is:
>>
>>  - If INVARIANT is set:  kernel panic to get all the details in the core.
>>  - If INVARIANT is not set:  Log this error with an explicit kernel
>> log(LOG_ERR) describing the issue, and then use the workaround to avoid
>> the double-free to let the system to good enough state.
>>
>>  Something like:
> 
> Yes, thanks!

 Proposed changes added in the review:

https://reviews.freebsd.org/D8211

 tell me when you have three days without issue with this change.

>> tcp_detach() {
>>
>>   ...
>>   if (inp->inp_flags & INP_TIMEWAIT) {
>>
>> ...
>> if (inp->inp_flags & INP_DROPPED) {
>>
>>   in_pcbdetach(inp);
>>   if (__predict_true(tp == NULL)) {
>>   in_pcbfree(inp);
>>   } else {
>> #ifdef INVARIANTS
>> panic("tcp_detach: tp != NULL, That's not good because 'blah'\n");
>> #else
>> log(LOG_ERR, "tcp_detach: tp != NULL, That's not good because
>> 'blah'\n");
> 
> May be some more info in log can help to detect root cause of issuse?
> I am don't know what info, may be flags or number of references?

 For this kind of issue, the useful part is the stacktrace.  INVARIANT
will give you that trace in the core, and without INVARIANT then it is
better to use dtrace:

$ cat tcp-twstart-dropped.d
fbt::tcp_twstart:entry
/args[0]->t_inpcb->inp_flags & 0x0400/
{

Re: ZFS l2arc broken in 10.3

2016-10-13 Thread Andriy Gapon 
On 12/10/2016 23:18, Peter wrote:
> Details:
> After upgrading 2 machines from 9.3 to 10.3-STABLE, on one of them the
> l2arc stays empty (capacity alloc = 0), although it is online and gets
> accessed. It did work well on 9.3.
> 
> I did the following tests:
>  * Create a zpool on a stick, with two volumes: one filesystem and one
>cache. The cache stays with alloc=0.
>Export it and move it into the other machine. The cache immediately
>fills.
>Move it back, the cache stays with alloc=0.
>-> this rules out all zpool/zfs get/set options, as they should
>   walk with the pool.
>  * Boot the GENERIC kernel. l2arc stays with alloc=0.
>-> this rules out all my nonstandard kernel options.
>  * Boot in single user mode. l2arc stays with alloc=0.
>-> this rules out all /etc/* config files.
>  * Delete the zpool.cache and reimport pools. l2arc stays with alloc=0.
>  * Copy the /boot/loader.conf settings to the other machine. The l2arc
>still works there.
> 
> I could not think of any remaining place where this could come from,
> except the kernel code itself.
> From there, I found these counters nicely incrementing each second:
>   kstat.zfs.misc.arcstats.l2_write_buffer_list_iter: 50758
>   kstat.zfs.misc.arcstats.l2_write_buffer_list_null_iter: 27121
>   kstat.zfs.misc.arcstats.l2_write_buffer_bytes_scanned: 40589375488
> But also this counter incrementing:
>   kstat.zfs.misc.arcstats.l2_write_full: 14604
> 
> Then with some printf in the code I saw these values provided:
>  buf_sz = hdr->b_size;
> align = (size_t)1 << dev->l2ad_vdev->vdev_ashift;
> buf_a_sz = P2ROUNDUP(buf_sz, align);
> if ((write_asize + buf_a_sz) > target_sz) {
>full = B_TRUE;
>mutex_exit(hash_lock);
>ARCSTAT_BUMP(arcstat_l2_write_full);
>break;
> }
> 
> buf_sz =1536
> align =512
> buf_a_sz =18446744069414585856
> write_asize =0
> target_sz =16777216
> 
> where buf_a_sz is obviousely off by (2^64 - 2^32).
> 
> Maybe this is an effect of crosscompiling i386 on amd64.

Yes, the problem is specific to 32-bit platforms where size_t is 32-bit.

> But anyway, as long as
> i386 is still supported, it should not happen.

Certainly.

> Now, my real concern is: if this really obvious ... made it undetected until
> 10.3, how many other missing typecasts are still in the code??

No need to be dramatic here.  That particular piece code is very new.
I committed it to head in April (r297848), MFC-ed even later.
Apparently no one else who uses 32-bit systems and has L2ARC configured had a
chance to run into the bug.

Thank you very much for discovering and analyzing the bug and providing a fix
for it!


-- 
Andriy Gapon
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: ZFS l2arc broken in 10.3

2016-10-13 Thread Pete French 

Ok, thats a bit worry if true - but I can confirm that l2arc works fine
under 10.3 on amd64, so what you say about cross-compling might be true.
Am taking an inetrest in this as I have just dpeloyed a lot of machines
which are going to be relying on l2arc working to get reasobale performance.

-pete.

On 10/12/16 21:18, Peter wrote:
> Details:
> After upgrading 2 machines from 9.3 to 10.3-STABLE, on one of them the
> l2arc stays empty (capacity alloc = 0), although it is online and gets
> accessed. It did work well on 9.3.
> 
> I did the following tests:
>  * Create a zpool on a stick, with two volumes: one filesystem and one
>cache. The cache stays with alloc=0.
>Export it and move it into the other machine. The cache immediately
>fills.
>Move it back, the cache stays with alloc=0.
>-> this rules out all zpool/zfs get/set options, as they should
>   walk with the pool.
>  * Boot the GENERIC kernel. l2arc stays with alloc=0.
>-> this rules out all my nonstandard kernel options.
>  * Boot in single user mode. l2arc stays with alloc=0.
>-> this rules out all /etc/* config files.
>  * Delete the zpool.cache and reimport pools. l2arc stays with alloc=0.
>  * Copy the /boot/loader.conf settings to the other machine. The l2arc
>still works there.
> 
> I could not think of any remaining place where this could come from,
> except the kernel code itself.
> From there, I found these counters nicely incrementing each second:
>   kstat.zfs.misc.arcstats.l2_write_buffer_list_iter: 50758
>   kstat.zfs.misc.arcstats.l2_write_buffer_list_null_iter: 27121
>   kstat.zfs.misc.arcstats.l2_write_buffer_bytes_scanned: 40589375488
> But also this counter incrementing:
>   kstat.zfs.misc.arcstats.l2_write_full: 14604
> 
> Then with some printf in the code I saw these values provided:
>  buf_sz = hdr->b_size;
> align = (size_t)1 << dev->l2ad_vdev->vdev_ashift;
> buf_a_sz = P2ROUNDUP(buf_sz, align);
> if ((write_asize + buf_a_sz) > target_sz) {
>full = B_TRUE;
>mutex_exit(hash_lock);
>ARCSTAT_BUMP(arcstat_l2_write_full);
>break;
> }
> 
> buf_sz =1536
> align =512
> buf_a_sz =18446744069414585856
> write_asize =0
> target_sz =16777216
> 
> where buf_a_sz is obviousely off by (2^64 - 2^32).
> 
> Maybe this is an effect of crosscompiling i386 on amd64. But anyway, as
> long as i386 is still supported, it should not happen.
> 
> 
> Now, my real concern is: if this really obvious ... made it undetected
> until 10.3, how many other missing typecasts are still in the code??
> 
> ___
> freebsd-stable@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Fwd: 11-RELEASE PL1 doesn't load kernel modules with ZFS-on-Root

2016-10-13 Thread Aryeh Friedman 
-- Forwarded message --
From: Aryeh Friedman 
Date: Thu, Oct 13, 2016 at 1:59 AM
Subject: Re: 11-RELEASE PL1 doesn't load kernel modules with ZFS-on-Root
To: Kurt Jaeger 


On Thu, Oct 13, 2016 at 1:39 AM, Kurt Jaeger  wrote:

> Hi!
>
> > root@thin:~ # kldload linux
> > kldload: an error occurred while loading the module. Please check
> dmesg(8)
> > for more details.
>
> What does the dmesg log show ?
>

KLD linux.ko: depends on kernel - not available or version mismatch
linker_load_file: Unsupported file type

Fresh install from FreeBSD-11.0-RELEASE-amd64-bootonly.iso.   Source tree
generated with a svn checkout

aryehl@thin:~ % ls -lt /boot/kernel/linux.ko
-r-xr-xr-x  1 root  wheel  582728 Oct 11 11:26 /boot/kernel/linux.ko
aryehl@thin:~ % uname -a

FreeBSD thin.lan.fnwe.net 11.0-RELEASE-p1 FreeBSD 11.0-RELEASE-p1 #0
r306420: Thu Sep 29 01:43:23 UTC 2016
r...@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC
amd64


On Thu, Oct 13, 2016 at 2:31 AM, Kurt Jaeger  wrote:
Hi!

> KLD linux.ko: depends on kernel - not available or version mismatch
> linker_load_file: Unsupported file type

What does uname -a say ?

What does

file /boot/kernel/linux.ko

say ?

What does

ls -lt /boot/kernel/

show ? Maybe the file was not replaced during upgrade and is still on some
previous version ?

-- 
Aryeh M. Friedman, Lead Developer, http://www.PetiteCloud.org
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: 11-RELEASE PL1 doesn't load kernel modules with ZFS-on-Root

2016-10-13 Thread Andriy Gapon 
On 13/10/2016 07:04, Aryeh Friedman wrote:
> Please check dmesg(8)
> for more details.


-- 
Andriy Gapon
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"