FreeBSD_STABLE_11-arm64 - Build #155 - Fixed
FreeBSD_STABLE_11-arm64 - Build #155 - Fixed: Build information: https://jenkins.FreeBSD.org/job/FreeBSD_STABLE_11-arm64/155/ Full change log: https://jenkins.FreeBSD.org/job/FreeBSD_STABLE_11-arm64/155/changes Full build log: https://jenkins.FreeBSD.org/job/FreeBSD_STABLE_11-arm64/155/console Change summaries: 306060 by karels: MFC r304713: Fix L2 caching for UDP over IPv6 ip6_output() was missing cache invalidation code analougous to ip_output.c. r304545 disabled L2 caching for UDP/IPv6 as a workaround. This change adds the missing cache invalidation code and reverts r304545. Reviewed by:gnn Approved by:gnn (mentor) Tested by: peter@, Mike Andrews Differential Revision: https://reviews.freebsd.org/D7591 ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: nginx and FreeBSD11
On Tue, Sep 20, 2016 at 04:00:10PM -0600, Warner Losh wrote: > >> > > Is this sandy bridge ? > >> > > >> > Sandy Bridge EP > >> > > >> > > Show me first 100 lines of the verbose dmesg, > >> > > >> > After day or two, after end of this test run -- I am need to enable > >> > verbose. > >> > > >> > > I want to see cpu features lines. In particular, does you CPU support > >> > > the INVPCID feature. > >> > > >> > CPU: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz (2000.05-MHz K8-class CPU) > >> > Origin="GenuineIntel" Id=0x206d7 Family=0x6 Model=0x2d Stepping=7 > >> > > >> > Features=0xbfebfbff > >> > > >> > Features2=0x1fbee3ff > >> > AMD Features=0x2c100800 > >> > AMD Features2=0x1 > >> > XSAVE Features=0x1 > >> > VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID > >> > TSC: P-state invariant, performance statistics > >> > > >> > I am don't see this feature before E5v3: > >> > > >> > CPU: Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz (2600.06-MHz K8-class CPU) > >> > Origin="GenuineIntel" Id=0x306e4 Family=0x6 Model=0x3e Stepping=4 > >> > > >> > Features=0xbfebfbff > >> > > >> > Features2=0x7fbee3ff > >> > AMD Features=0x2c100800 > >> > AMD Features2=0x1 > >> > Structured Extended Features=0x281 > >> > XSAVE Features=0x1 > >> > VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr > >> > TSC: P-state invariant, performance statistics > >> > > >> > (don't run 11.0 on this CPU) > >> Ok. > >> > >> > > >> > CPU: Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz (2600.05-MHz K8-class CPU) > >> > Origin="GenuineIntel" Id=0x306f2 Family=0x6 Model=0x3f Stepping=2 > >> > > >> > Features=0xbfebfbff > >> > > >> > Features2=0x7ffefbff > >> > AMD Features=0x2c100800 > >> > AMD Features2=0x21 > >> > Structured Extended > >> > Features=0x37ab > >> > XSAVE Features=0x1 > >> > VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr > >> > TSC: P-state invariant, performance statistics > >> > > >> > (11.0 run w/o this issuse) > >> Do you mean that similarly configured nginx+aio do not demonstrate the > >> corruption on this machine ? > > > > Yes. > > But different storage configuration and different pattern load. > > > > Also 11.0 run w/o this issuse on > > > > CPU: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz (2200.04-MHz K8-class CPU) > > Origin="GenuineIntel" Id=0x406f1 Family=0x6 Model=0x4f Stepping=1 > > > > Features=0xbfebfbff > > > > Features2=0x7ffefbff > > AMD Features=0x2c100800 > > AMD Features2=0x121 > > Structured Extended > > Features=0x21cbfbb > > XSAVE Features=0x1 > > VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr > > TSC: P-state invariant, performance statistics > > > > PS: all systems is dual-cpu. > > Does this mean 2 cores or two sockets? We've seen a similar hang with > the following CPU: two sockets. not sure how this impotant, just for record. you system also w/o INVPCID feature (as kib question). may be you case also will be resolved by vm.pmap.pcid_enabled=0? > CPU: Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz (2700.06-MHz K8-class CPU) > Origin="GenuineIntel" Id=0x306e4 Family=0x6 Model=0x3e Stepping=4 > > Features=0xbfebfbff > > Features2=0x7fbee3ff > AMD Features=0x2c100800 > AMD Features2=0x1 > Structured Extended Features=0x281 > XSAVE Features=0x1 > VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr > TSC: P-state invariant, performance statistics > real memory = 274877906944 (262144 MB) > avail memory = 267146330112 (254770 MB) > > 12 cores x 2 SMT x 1 socket > > Warner ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
FreeBSD_STABLE_11-arm64 - Build #154 - Failure
FreeBSD_STABLE_11-arm64 - Build #154 - Failure: Build information: https://jenkins.FreeBSD.org/job/FreeBSD_STABLE_11-arm64/154/ Full change log: https://jenkins.FreeBSD.org/job/FreeBSD_STABLE_11-arm64/154/changes Full build log: https://jenkins.FreeBSD.org/job/FreeBSD_STABLE_11-arm64/154/console Change summaries: 306025 by ae: MFC r305778: Fix swap tables between sets when this functional is enabled. We have 6 opcode rewriters for table opcodes. When `set swap' command invoked, it is called for each rewriter, so at the end we get the same result, because opcode rewriter uses ETLV type to match opcode. And all tables opcodes have the same ETLV type. To solve this problem, use separate sets handler for one opcode rewriter. Use it to handle TEST_ALL, SWAP_ALL and MOVE_ALL commands. PR: 212630 305971 by bde: MFC r305380: Fix missing fmodl() on arches with 53-bit long doubles. PR: 199422, 211965 The end of the build log: [...truncated 21808 lines...] ===> share/doc/papers/relengr (includes) --- includes_subdir_share/doc/papers/sysperf --- ===> share/doc/papers/sysperf (includes) --- includes_subdir_share/doc/papers/timecounter --- ===> share/doc/papers/timecounter (includes) --- includes_subdir_share/doc/psd --- ===> share/doc/psd (includes) --- includes_subdir_share/doc/psd/title --- ===> share/doc/psd/title (includes) --- includes_subdir_share/doc/psd/contents --- ===> share/doc/psd/contents (includes) --- includes_subdir_share/doc/psd/01.cacm --- ===> share/doc/psd/01.cacm (includes) --- includes_subdir_share/doc/psd/02.implement --- ===> share/doc/psd/02.implement (includes) --- includes_subdir_share/doc/psd/03.iosys --- ===> share/doc/psd/03.iosys (includes) --- includes_subdir_share/doc/psd/04.uprog --- ===> share/doc/psd/04.uprog (includes) --- includes_subdir_share/doc/psd/05.sysman --- ===> share/doc/psd/05.sysman (includes) --- includes_subdir_share/doc/psd/06.Clang --- ===> share/doc/psd/06.Clang (includes) --- includes_subdir_share/doc/psd/12.make --- ===> share/doc/psd/12.make (includes) --- includes_subdir_share/doc/psd/13.rcs --- ===> share/doc/psd/13.rcs (includes) --- includes_subdir_share/doc/psd/13.rcs/rcs --- ===> share/doc/psd/13.rcs/rcs (includes) --- includes_subdir_share/doc/psd/13.rcs/rcs_func --- ===> share/doc/psd/13.rcs/rcs_func (includes) --- includes_subdir_share/doc/psd/15.yacc --- ===> share/doc/psd/15.yacc (includes) --- includes_subdir_share/doc/psd/16.lex --- ===> share/doc/psd/16.lex (includes) --- includes_subdir_share/doc/psd/17.m4 --- ===> share/doc/psd/17.m4 (includes) --- includes_subdir_share/doc/psd/18.gprof --- ===> share/doc/psd/18.gprof (includes) --- includes_subdir_share/doc/psd/20.ipctut --- ===> share/doc/psd/20.ipctut (includes) --- includes_subdir_share/doc/psd/21.ipc --- ===> share/doc/psd/21.ipc (includes) --- includes_subdir_share/doc/psd/22.rpcgen --- ===> share/doc/psd/22.rpcgen (includes) --- includes_subdir_share/doc/psd/23.rpc --- ===> share/doc/psd/23.rpc (includes) --- includes_subdir_share/doc/psd/24.xdr --- ===> share/doc/psd/24.xdr (includes) --- includes_subdir_share/doc/psd/25.xdrrfc --- ===> share/doc/psd/25.xdrrfc (includes) --- includes_subdir_share/doc/psd/26.rpcrfc --- ===> share/doc/psd/26.rpcrfc (includes) --- includes_subdir_share/doc/psd/27.nfsrpc --- ===> share/doc/psd/27.nfsrpc (includes) --- includes_subdir_share/doc/smm --- ===> share/doc/smm (includes) --- includes_subdir_share/doc/smm/title --- ===> share/doc/smm/title (includes) --- includes_subdir_share/doc/smm/contents --- ===> share/doc/smm/contents (includes) --- includes_subdir_share/doc/smm/01.setup --- ===> share/doc/smm/01.setup (includes) --- includes_subdir_share/doc/smm/02.config --- ===> share/doc/smm/02.config (includes) --- includes_subdir_share/doc/smm/03.fsck --- ===> share/doc/smm/03.fsck (includes) --- includes_subdir_share/doc/smm/04.quotas --- ===> share/doc/smm/04.quotas (includes) --- includes_subdir_share/doc/smm/05.fastfs --- ===> share/doc/smm/05.fastfs (includes) --- includes_subdir_share/doc/smm/06.nfs --- ===> share/doc/smm/06.nfs (includes) --- includes_subdir_share/doc/smm/07.lpd --- ===> share/doc/smm/07.lpd (includes) --- includes_subdir_share/doc/smm/08.sendmailop --- ===> share/doc/smm/08.sendmailop (includes) --- includes_subdir_share/doc/smm/11.timedop --- ===> share/doc/smm/11.timedop (includes) --- includes_subdir_share/doc/smm/12.timed --- ===> share/doc/smm/12.timed (includes) --- includes_subdir_share/doc/smm/18.net --- ===> share/doc/smm/18.net (includes) --- includes_subdir_share/doc/usd --- ===> share/doc/usd (includes) --- includes_subdir_share/doc/usd/title --- ===> share/doc/usd/title (includes) --- includes_subdir_share/doc/usd/contents --- ===> share/doc/usd/contents (includes) --- includes_subdir_share/doc/usd/04.csh --- ===> share/doc/usd/04.csh (includes) --- includes_subdir_share/doc/usd/05.dc --- ===> share/doc/usd/05.dc (includes) --- includes_subdir_share/doc/usd/06.bc ---
Re: nginx and FreeBSD11
On Tue, Sep 20, 2016 at 3:47 PM, Slawa Olhovchenkov wrote:
> On Wed, Sep 21, 2016 at 12:15:17AM +0300, Konstantin Belousov wrote:
>
>> On Tue, Sep 20, 2016 at 11:38:54PM +0300, Slawa Olhovchenkov wrote:
>> > On Tue, Sep 20, 2016 at 11:19:25PM +0300, Konstantin Belousov wrote:
>> >
>> > > On Tue, Sep 20, 2016 at 10:20:53PM +0300, Slawa Olhovchenkov wrote:
>> > > > On Tue, Sep 20, 2016 at 09:52:44AM +0300, Slawa Olhovchenkov wrote:
>> > > >
>> > > > > On Mon, Sep 19, 2016 at 06:05:46PM -0700, John Baldwin wrote:
>> > > > >
>> > > > > > > > If this panics, then vmspace_switch_aio() is not working for
>> > > > > > > > some reason.
>> > > > > > >
>> > > > > > > I am try using next DTrace script:
>> > > > > > >
>> > > > > > > #pragma D option dynvarsize=64m
>> > > > > > >
>> > > > > > > int req[struct vmspace *, void *];
>> > > > > > > self int trace;
>> > > > > > >
>> > > > > > > syscall:freebsd:aio_read:entry
>> > > > > > > {
>> > > > > > > this->aio = *(struct aiocb *)copyin(arg0, sizeof(struct
>> > > > > > > aiocb));
>> > > > > > > req[curthread->td_proc->p_vmspace, this->aio.aio_buf] =
>> > > > > > > curthread->td_proc->p_pid;
>> > > > > > > }
>> > > > > > >
>> > > > > > > fbt:kernel:aio_process_rw:entry
>> > > > > > > {
>> > > > > > > self->job = args[0];
>> > > > > > > self->trace = 1;
>> > > > > > > }
>> > > > > > >
>> > > > > > > fbt:kernel:aio_process_rw:return
>> > > > > > > /self->trace/
>> > > > > > > {
>> > > > > > > req[self->job->userproc->p_vmspace,
>> > > > > > > self->job->uaiocb.aio_buf] = 0;
>> > > > > > > self->job = 0;
>> > > > > > > self->trace = 0;
>> > > > > > > }
>> > > > > > >
>> > > > > > > fbt:kernel:vn_io_fault:entry
>> > > > > > > /self->trace && !req[curthread->td_proc->p_vmspace,
>> > > > > > > args[1]->uio_iov[0].iov_base]/
>> > > > > > > {
>> > > > > > > this->buf = args[1]->uio_iov[0].iov_base;
>> > > > > > > printf("%Y vn_io_fault %p:%p pid %d\n", walltimestamp,
>> > > > > > > curthread->td_proc->p_vmspace, this->buf,
>> > > > > > > req[curthread->td_proc->p_vmspace, this->buf]);
>> > > > > > > }
>> > > > > > > ===
>> > > > > > >
>> > > > > > > And don't got any messages near nginx core dump.
>> > > > > > > What I can check next?
>> > > > > > > May be check context/address space switch for kernel process?
>> > > > > >
>> > > > > > Which CPU are you using?
>> > > > >
>> > > > > CPU: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz (2000.04-MHz K8-class
>> > > > > CPU)
>> > > Is this sandy bridge ?
>> >
>> > Sandy Bridge EP
>> >
>> > > Show me first 100 lines of the verbose dmesg,
>> >
>> > After day or two, after end of this test run -- I am need to enable
>> > verbose.
>> >
>> > > I want to see cpu features lines. In particular, does you CPU support
>> > > the INVPCID feature.
>> >
>> > CPU: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz (2000.05-MHz K8-class CPU)
>> > Origin="GenuineIntel" Id=0x206d7 Family=0x6 Model=0x2d Stepping=7
>> >
>> > Features=0xbfebfbff
>> >
>> > Features2=0x1fbee3ff
>> > AMD Features=0x2c100800
>> > AMD Features2=0x1
>> > XSAVE Features=0x1
>> > VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
>> > TSC: P-state invariant, performance statistics
>> >
>> > I am don't see this feature before E5v3:
>> >
>> > CPU: Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz (2600.06-MHz K8-class CPU)
>> > Origin="GenuineIntel" Id=0x306e4 Family=0x6 Model=0x3e Stepping=4
>> >
>> > Features=0xbfebfbff
>> >
>> > Features2=0x7fbee3ff
>> > AMD Features=0x2c100800
>> > AMD Features2=0x1
>> > Structured Extended Features=0x281
>> > XSAVE Features=0x1
>> > VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
>> > TSC: P-state invariant, performance statistics
>> >
>> > (don't run 11.0 on this CPU)
>> Ok.
>>
>> >
>> > CPU: Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz (2600.05-MHz K8-class CPU)
>> > Origin="GenuineIntel" Id=0x306f2 Family=0x6 Model=0x3f Stepping=2
>> >
>> > Features=0xbfebfbff
>> >
>> > Features2=0x7ffefbff
>> > AMD Features=0x2c100800
>> > AMD Features2=0x21
>> > Structured Extended
>> > Features=0x37ab
>> > XSAVE Features=0x1
>> > VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
>> > TSC: P-state invariant, performance statistics
>> >
>> > (11.0 run w/o this issuse)
>> Do you mean that similarly configured nginx+aio do not demonstrate the
>> corruption on this machine ?
>
> Yes.
> But different storage configuration and different pattern load.
>
> Also 11.0 run w/o this issuse on
>
> CPU: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz (2200.04-MHz K8-class CPU)
> Origin="GenuineIntel" Id=0x406f1 Family=0x6 Model=0x4f Stepping=1
>
> Features=0xbfebfbff
>
> Features2=0x7ffefbff
> AMD Features=0x2c100800
> AMD Features2=0x121
> Structured Extended
> Features=0x21cbfbb
> XSAVE Features=0x1
> VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
> TSC: P-state invariant, performance statistics
>
> PS: all systems is dual-cpu.
D
Re: nginx and FreeBSD11
On Wed, Sep 21, 2016 at 12:15:17AM +0300, Konstantin Belousov wrote:
> On Tue, Sep 20, 2016 at 11:38:54PM +0300, Slawa Olhovchenkov wrote:
> > On Tue, Sep 20, 2016 at 11:19:25PM +0300, Konstantin Belousov wrote:
> >
> > > On Tue, Sep 20, 2016 at 10:20:53PM +0300, Slawa Olhovchenkov wrote:
> > > > On Tue, Sep 20, 2016 at 09:52:44AM +0300, Slawa Olhovchenkov wrote:
> > > >
> > > > > On Mon, Sep 19, 2016 at 06:05:46PM -0700, John Baldwin wrote:
> > > > >
> > > > > > > > If this panics, then vmspace_switch_aio() is not working for
> > > > > > > > some reason.
> > > > > > >
> > > > > > > I am try using next DTrace script:
> > > > > > >
> > > > > > > #pragma D option dynvarsize=64m
> > > > > > >
> > > > > > > int req[struct vmspace *, void *];
> > > > > > > self int trace;
> > > > > > >
> > > > > > > syscall:freebsd:aio_read:entry
> > > > > > > {
> > > > > > > this->aio = *(struct aiocb *)copyin(arg0, sizeof(struct
> > > > > > > aiocb));
> > > > > > > req[curthread->td_proc->p_vmspace, this->aio.aio_buf] =
> > > > > > > curthread->td_proc->p_pid;
> > > > > > > }
> > > > > > >
> > > > > > > fbt:kernel:aio_process_rw:entry
> > > > > > > {
> > > > > > > self->job = args[0];
> > > > > > > self->trace = 1;
> > > > > > > }
> > > > > > >
> > > > > > > fbt:kernel:aio_process_rw:return
> > > > > > > /self->trace/
> > > > > > > {
> > > > > > > req[self->job->userproc->p_vmspace,
> > > > > > > self->job->uaiocb.aio_buf] = 0;
> > > > > > > self->job = 0;
> > > > > > > self->trace = 0;
> > > > > > > }
> > > > > > >
> > > > > > > fbt:kernel:vn_io_fault:entry
> > > > > > > /self->trace && !req[curthread->td_proc->p_vmspace,
> > > > > > > args[1]->uio_iov[0].iov_base]/
> > > > > > > {
> > > > > > > this->buf = args[1]->uio_iov[0].iov_base;
> > > > > > > printf("%Y vn_io_fault %p:%p pid %d\n", walltimestamp,
> > > > > > > curthread->td_proc->p_vmspace, this->buf,
> > > > > > > req[curthread->td_proc->p_vmspace, this->buf]);
> > > > > > > }
> > > > > > > ===
> > > > > > >
> > > > > > > And don't got any messages near nginx core dump.
> > > > > > > What I can check next?
> > > > > > > May be check context/address space switch for kernel process?
> > > > > >
> > > > > > Which CPU are you using?
> > > > >
> > > > > CPU: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz (2000.04-MHz K8-class
> > > > > CPU)
> > > Is this sandy bridge ?
> >
> > Sandy Bridge EP
> >
> > > Show me first 100 lines of the verbose dmesg,
> >
> > After day or two, after end of this test run -- I am need to enable verbose.
> >
> > > I want to see cpu features lines. In particular, does you CPU support
> > > the INVPCID feature.
> >
> > CPU: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz (2000.05-MHz K8-class CPU)
> > Origin="GenuineIntel" Id=0x206d7 Family=0x6 Model=0x2d Stepping=7
> >
> > Features=0xbfebfbff
> >
> > Features2=0x1fbee3ff
> > AMD Features=0x2c100800
> > AMD Features2=0x1
> > XSAVE Features=0x1
> > VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
> > TSC: P-state invariant, performance statistics
> >
> > I am don't see this feature before E5v3:
> >
> > CPU: Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz (2600.06-MHz K8-class CPU)
> > Origin="GenuineIntel" Id=0x306e4 Family=0x6 Model=0x3e Stepping=4
> >
> > Features=0xbfebfbff
> >
> > Features2=0x7fbee3ff
> > AMD Features=0x2c100800
> > AMD Features2=0x1
> > Structured Extended Features=0x281
> > XSAVE Features=0x1
> > VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
> > TSC: P-state invariant, performance statistics
> >
> > (don't run 11.0 on this CPU)
> Ok.
>
> >
> > CPU: Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz (2600.05-MHz K8-class CPU)
> > Origin="GenuineIntel" Id=0x306f2 Family=0x6 Model=0x3f Stepping=2
> >
> > Features=0xbfebfbff
> >
> > Features2=0x7ffefbff
> > AMD Features=0x2c100800
> > AMD Features2=0x21
> > Structured Extended
> > Features=0x37ab
> > XSAVE Features=0x1
> > VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
> > TSC: P-state invariant, performance statistics
> >
> > (11.0 run w/o this issuse)
> Do you mean that similarly configured nginx+aio do not demonstrate the
> corruption on this machine ?
Yes.
But different storage configuration and different pattern load.
Also 11.0 run w/o this issuse on
CPU: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz (2200.04-MHz K8-class CPU)
Origin="GenuineIntel" Id=0x406f1 Family=0x6 Model=0x4f Stepping=1
Features=0xbfebfbff
Features2=0x7ffefbff
AMD Features=0x2c100800
AMD Features2=0x121
Structured Extended
Features=0x21cbfbb
XSAVE Features=0x1
VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
TSC: P-state invariant, performance statistics
PS: all systems is dual-cpu.
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr
Re: nginx and FreeBSD11
On Tue, Sep 20, 2016 at 11:38:54PM +0300, Slawa Olhovchenkov wrote:
> On Tue, Sep 20, 2016 at 11:19:25PM +0300, Konstantin Belousov wrote:
>
> > On Tue, Sep 20, 2016 at 10:20:53PM +0300, Slawa Olhovchenkov wrote:
> > > On Tue, Sep 20, 2016 at 09:52:44AM +0300, Slawa Olhovchenkov wrote:
> > >
> > > > On Mon, Sep 19, 2016 at 06:05:46PM -0700, John Baldwin wrote:
> > > >
> > > > > > > If this panics, then vmspace_switch_aio() is not working for
> > > > > > > some reason.
> > > > > >
> > > > > > I am try using next DTrace script:
> > > > > >
> > > > > > #pragma D option dynvarsize=64m
> > > > > >
> > > > > > int req[struct vmspace *, void *];
> > > > > > self int trace;
> > > > > >
> > > > > > syscall:freebsd:aio_read:entry
> > > > > > {
> > > > > > this->aio = *(struct aiocb *)copyin(arg0, sizeof(struct
> > > > > > aiocb));
> > > > > > req[curthread->td_proc->p_vmspace, this->aio.aio_buf] =
> > > > > > curthread->td_proc->p_pid;
> > > > > > }
> > > > > >
> > > > > > fbt:kernel:aio_process_rw:entry
> > > > > > {
> > > > > > self->job = args[0];
> > > > > > self->trace = 1;
> > > > > > }
> > > > > >
> > > > > > fbt:kernel:aio_process_rw:return
> > > > > > /self->trace/
> > > > > > {
> > > > > > req[self->job->userproc->p_vmspace,
> > > > > > self->job->uaiocb.aio_buf] = 0;
> > > > > > self->job = 0;
> > > > > > self->trace = 0;
> > > > > > }
> > > > > >
> > > > > > fbt:kernel:vn_io_fault:entry
> > > > > > /self->trace && !req[curthread->td_proc->p_vmspace,
> > > > > > args[1]->uio_iov[0].iov_base]/
> > > > > > {
> > > > > > this->buf = args[1]->uio_iov[0].iov_base;
> > > > > > printf("%Y vn_io_fault %p:%p pid %d\n", walltimestamp,
> > > > > > curthread->td_proc->p_vmspace, this->buf,
> > > > > > req[curthread->td_proc->p_vmspace, this->buf]);
> > > > > > }
> > > > > > ===
> > > > > >
> > > > > > And don't got any messages near nginx core dump.
> > > > > > What I can check next?
> > > > > > May be check context/address space switch for kernel process?
> > > > >
> > > > > Which CPU are you using?
> > > >
> > > > CPU: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz (2000.04-MHz K8-class CPU)
> > Is this sandy bridge ?
>
> Sandy Bridge EP
>
> > Show me first 100 lines of the verbose dmesg,
>
> After day or two, after end of this test run -- I am need to enable verbose.
>
> > I want to see cpu features lines. In particular, does you CPU support
> > the INVPCID feature.
>
> CPU: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz (2000.05-MHz K8-class CPU)
> Origin="GenuineIntel" Id=0x206d7 Family=0x6 Model=0x2d Stepping=7
>
> Features=0xbfebfbff
>
> Features2=0x1fbee3ff
> AMD Features=0x2c100800
> AMD Features2=0x1
> XSAVE Features=0x1
> VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
> TSC: P-state invariant, performance statistics
>
> I am don't see this feature before E5v3:
>
> CPU: Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz (2600.06-MHz K8-class CPU)
> Origin="GenuineIntel" Id=0x306e4 Family=0x6 Model=0x3e Stepping=4
>
> Features=0xbfebfbff
>
> Features2=0x7fbee3ff
> AMD Features=0x2c100800
> AMD Features2=0x1
> Structured Extended Features=0x281
> XSAVE Features=0x1
> VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
> TSC: P-state invariant, performance statistics
>
> (don't run 11.0 on this CPU)
Ok.
>
> CPU: Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz (2600.05-MHz K8-class CPU)
> Origin="GenuineIntel" Id=0x306f2 Family=0x6 Model=0x3f Stepping=2
>
> Features=0xbfebfbff
>
> Features2=0x7ffefbff
> AMD Features=0x2c100800
> AMD Features2=0x21
> Structured Extended
> Features=0x37ab
> XSAVE Features=0x1
> VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
> TSC: P-state invariant, performance statistics
>
> (11.0 run w/o this issuse)
Do you mean that similarly configured nginx+aio do not demonstrate the
corruption on this machine ?
>
> > Also you may show me the 'sysctl vm.pmap' output.
>
> # sysctl vm.pmap
> vm.pmap.pdpe.demotions: 3
> vm.pmap.pde.promotions: 172495
> vm.pmap.pde.p_failures: 2119294
> vm.pmap.pde.mappings: 1927
> vm.pmap.pde.demotions: 126192
> vm.pmap.pcid_save_cnt: 0
> vm.pmap.invpcid_works: 0
> vm.pmap.pcid_enabled: 0
> vm.pmap.pg_ps_enabled: 1
> vm.pmap.pat_works: 1
>
> This is after vm.pmap.pcid_enabled=0 in loader.conf
>
> > > >
> > > > > Perhaps try disabling PCID support (I think vm.pmap.pcid_enabled=0
> > > > > from
> > > > > loader prompt or loader.conf)? (Wondering if pmap_activate() is
> > > > > somehow not switching)
> > >
> > > I am need some more time to test (day or two), but now this is like
> > > workaround/solution: 12h runtime and peak hour w/o nginx crash.
> > > (vm.pmap.pcid_enabled=0 in loader.conf).
> >
> > Please try this variation of the previous patch.
>
> and remove vm.pmap.pcid_enabled=0?
Definitely.
>
> > diff --git a/sys/vm/vm_map.c b/sys/vm/vm_map.c
> > index a23468e..f754652 100644
> > --- a/sys/vm/vm_map.c
Re: 11.0 stuck on high network load
Hi Slawa,
On 9/19/16 10:43 PM, Slawa Olhovchenkov wrote:
> On Mon, Sep 19, 2016 at 10:32:13PM +0200, Julien Charbon wrote:
>>
>>> @ CPU_CLK_UNHALTED_CORE [4653445 samples]
>>>
>>> 51.86% [2413083] lock_delay @ /boot/kernel.VSTREAM/kernel
>>> 100.0% [2413083] __rw_wlock_hard
>>> 100.0% [2413083]tcp_tw_2msl_scan
>>>99.99% [2412958] pfslowtimo
>>> 100.0% [2412958] softclock_call_cc
>>> 100.0% [2412958] softclock
>>> 100.0% [2412958]intr_event_execute_handlers
>>>100.0% [2412958] ithread_loop
>>> 100.0% [2412958] fork_exit
>>>00.01% [125] tcp_twstart
>>> 100.0% [125] tcp_do_segment
>>> 100.0% [125] tcp_input
>>> 100.0% [125]ip_input
>>>100.0% [125] swi_net
>>> 100.0% [125] intr_event_execute_handlers
>>> 100.0% [125] ithread_loop
>>> 100.0% [125]fork_exit
>>
>> The only write lock tcp_tw_2msl_scan() tries to get is a
>> INP_WLOCK(inp). Thus here, tcp_tw_2msl_scan() seems to be stuck
>> spinning on INP_WLOCK (or pfslowtimo() is going crazy and calls
>> tcp_tw_2msl_scan() at high rate but this will be quite unexpected).
>>
>> Thus my hypothesis is that something is holding the INP_WLOCK and not
>> releasing it, and tcp_tw_2msl_scan() is spinning on it.
>>
>> If you can, could you compile the kernel with below options:
>>
>> optionsDDB # Support DDB.
>> optionsDEADLKRES # Enable the deadlock resolver
>> optionsINVARIANTS # Enable calls of extra sanity
>> checking
>> optionsINVARIANT_SUPPORT # Extra sanity checks of internal
>> structures, required by INVARIANTS
>> optionsWITNESS # Enable checks to detect
>> deadlocks and cycles
>> optionsWITNESS_SKIPSPIN# Don't run witness on spinlocks
>> for speed
>
> Currently this host run with 100% CPU load (on all cores), i.e.
> enabling WITNESS will be significant drop performance.
> Can I use only some subset of options?
>
> Also, I can some troubles to DDB enter in this case.
> May be kgdb will be success (not tryed yet)?
If these kernel options will certainly slow down your kernel, they also
might found the root cause of your issue before reaching the point where
you have 100% cpu load on all cores (thanks to INVARIANTS). I would
suggest:
#1. Try above kernel options at least once, and see what you can get.
#2. If #1 is a total failure try below patch: It won't solve anything,
it just makes tcp_tw_2msl_scan() less greedy when there is contention on
the INP write lock. If it makes the debugging more feasible, continue
to #3.
diff --git a/sys/netinet/tcp_timewait.c b/sys/netinet/tcp_timewait.c
index a8b78f9..4206ea3 100644
--- a/sys/netinet/tcp_timewait.c
+++ b/sys/netinet/tcp_timewait.c
@@ -701,34 +701,42 @@ tcp_tw_2msl_scan(int reuse)
in_pcbref(inp);
TW_RUNLOCK(V_tw_lock);
+retry:
if (INP_INFO_TRY_RLOCK(&V_tcbinfo)) {
- INP_WLOCK(inp);
- tw = intotw(inp);
- if (in_pcbrele_wlocked(inp)) {
- KASSERT(tw == NULL, ("%s: held last inp "
- "reference but tw not NULL", __func__));
- INP_INFO_RUNLOCK(&V_tcbinfo);
- continue;
- }
+ if (INP_TRY_WLOCK(inp)) {
+ tw = intotw(inp);
+ if (in_pcbrele_wlocked(inp)) {
+ KASSERT(tw == NULL, ("%s: held
last inp "
+ "reference but tw not NULL",
__func__));
+ INP_INFO_RUNLOCK(&V_tcbinfo);
+ continue;
+ }
- if (tw == NULL) {
- /* tcp_twclose() has already been called */
- INP_WUNLOCK(inp);
- INP_INFO_RUNLOCK(&V_tcbinfo);
- continue;
- }
+ if (tw == NULL) {
+ /* tcp_twclose() has already
been called */
+ INP_WUNLOCK(inp);
+ INP_INFO_RUNLOCK(&V_tcbinfo);
+ continue;
+ }
- tcp_twclose(tw, reuse);
- INP_INFO_RUNLOCK(&V_tcbinfo);
- if (reuse)
- return tw;
+ tcp_twclose(tw, reuse);
+ INP_INFO_RUNLOCK(&V_tcbinfo);
+
Re: nginx and FreeBSD11
On Tue, Sep 20, 2016 at 11:19:25PM +0300, Konstantin Belousov wrote:
> On Tue, Sep 20, 2016 at 10:20:53PM +0300, Slawa Olhovchenkov wrote:
> > On Tue, Sep 20, 2016 at 09:52:44AM +0300, Slawa Olhovchenkov wrote:
> >
> > > On Mon, Sep 19, 2016 at 06:05:46PM -0700, John Baldwin wrote:
> > >
> > > > > > If this panics, then vmspace_switch_aio() is not working for
> > > > > > some reason.
> > > > >
> > > > > I am try using next DTrace script:
> > > > >
> > > > > #pragma D option dynvarsize=64m
> > > > >
> > > > > int req[struct vmspace *, void *];
> > > > > self int trace;
> > > > >
> > > > > syscall:freebsd:aio_read:entry
> > > > > {
> > > > > this->aio = *(struct aiocb *)copyin(arg0, sizeof(struct
> > > > > aiocb));
> > > > > req[curthread->td_proc->p_vmspace, this->aio.aio_buf] =
> > > > > curthread->td_proc->p_pid;
> > > > > }
> > > > >
> > > > > fbt:kernel:aio_process_rw:entry
> > > > > {
> > > > > self->job = args[0];
> > > > > self->trace = 1;
> > > > > }
> > > > >
> > > > > fbt:kernel:aio_process_rw:return
> > > > > /self->trace/
> > > > > {
> > > > > req[self->job->userproc->p_vmspace,
> > > > > self->job->uaiocb.aio_buf] = 0;
> > > > > self->job = 0;
> > > > > self->trace = 0;
> > > > > }
> > > > >
> > > > > fbt:kernel:vn_io_fault:entry
> > > > > /self->trace && !req[curthread->td_proc->p_vmspace,
> > > > > args[1]->uio_iov[0].iov_base]/
> > > > > {
> > > > > this->buf = args[1]->uio_iov[0].iov_base;
> > > > > printf("%Y vn_io_fault %p:%p pid %d\n", walltimestamp,
> > > > > curthread->td_proc->p_vmspace, this->buf,
> > > > > req[curthread->td_proc->p_vmspace, this->buf]);
> > > > > }
> > > > > ===
> > > > >
> > > > > And don't got any messages near nginx core dump.
> > > > > What I can check next?
> > > > > May be check context/address space switch for kernel process?
> > > >
> > > > Which CPU are you using?
> > >
> > > CPU: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz (2000.04-MHz K8-class CPU)
> Is this sandy bridge ?
Sandy Bridge EP
> Show me first 100 lines of the verbose dmesg,
After day or two, after end of this test run -- I am need to enable verbose.
> I want to see cpu features lines. In particular, does you CPU support
> the INVPCID feature.
CPU: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz (2000.05-MHz K8-class CPU)
Origin="GenuineIntel" Id=0x206d7 Family=0x6 Model=0x2d Stepping=7
Features=0xbfebfbff
Features2=0x1fbee3ff
AMD Features=0x2c100800
AMD Features2=0x1
XSAVE Features=0x1
VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
TSC: P-state invariant, performance statistics
I am don't see this feature before E5v3:
CPU: Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz (2600.06-MHz K8-class CPU)
Origin="GenuineIntel" Id=0x306e4 Family=0x6 Model=0x3e Stepping=4
Features=0xbfebfbff
Features2=0x7fbee3ff
AMD Features=0x2c100800
AMD Features2=0x1
Structured Extended Features=0x281
XSAVE Features=0x1
VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
TSC: P-state invariant, performance statistics
(don't run 11.0 on this CPU)
CPU: Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz (2600.05-MHz K8-class CPU)
Origin="GenuineIntel" Id=0x306f2 Family=0x6 Model=0x3f Stepping=2
Features=0xbfebfbff
Features2=0x7ffefbff
AMD Features=0x2c100800
AMD Features2=0x21
Structured Extended
Features=0x37ab
XSAVE Features=0x1
VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
TSC: P-state invariant, performance statistics
(11.0 run w/o this issuse)
> Also you may show me the 'sysctl vm.pmap' output.
# sysctl vm.pmap
vm.pmap.pdpe.demotions: 3
vm.pmap.pde.promotions: 172495
vm.pmap.pde.p_failures: 2119294
vm.pmap.pde.mappings: 1927
vm.pmap.pde.demotions: 126192
vm.pmap.pcid_save_cnt: 0
vm.pmap.invpcid_works: 0
vm.pmap.pcid_enabled: 0
vm.pmap.pg_ps_enabled: 1
vm.pmap.pat_works: 1
This is after vm.pmap.pcid_enabled=0 in loader.conf
> > >
> > > > Perhaps try disabling PCID support (I think vm.pmap.pcid_enabled=0 from
> > > > loader prompt or loader.conf)? (Wondering if pmap_activate() is
> > > > somehow not switching)
> >
> > I am need some more time to test (day or two), but now this is like
> > workaround/solution: 12h runtime and peak hour w/o nginx crash.
> > (vm.pmap.pcid_enabled=0 in loader.conf).
>
> Please try this variation of the previous patch.
and remove vm.pmap.pcid_enabled=0?
> diff --git a/sys/vm/vm_map.c b/sys/vm/vm_map.c
> index a23468e..f754652 100644
> --- a/sys/vm/vm_map.c
> +++ b/sys/vm/vm_map.c
> @@ -481,6 +481,7 @@ vmspace_switch_aio(struct vmspace *newvm)
> if (oldvm == newvm)
> return;
>
> + spinlock_enter();
> /*
>* Point to the new address space and refer to it.
>*/
> @@ -489,6 +490,7 @@ vmspace_switch_aio(struct vmspace *newvm)
>
> /* Activate the new mapping. */
> pmap_activate(curthread);
> + spinlock_exit();
>
> /* Remove the daemon's reference to the old address space. *
Re: 11.0 stuck on high network load
On Tue, Sep 20, 2016 at 10:00:25PM +0200, Julien Charbon wrote:
>
> Hi Slawa,
>
> On 9/19/16 10:43 PM, Slawa Olhovchenkov wrote:
> > On Mon, Sep 19, 2016 at 10:32:13PM +0200, Julien Charbon wrote:
> >>
> >>> @ CPU_CLK_UNHALTED_CORE [4653445 samples]
> >>>
> >>> 51.86% [2413083] lock_delay @ /boot/kernel.VSTREAM/kernel
> >>> 100.0% [2413083] __rw_wlock_hard
> >>> 100.0% [2413083]tcp_tw_2msl_scan
> >>>99.99% [2412958] pfslowtimo
> >>> 100.0% [2412958] softclock_call_cc
> >>> 100.0% [2412958] softclock
> >>> 100.0% [2412958]intr_event_execute_handlers
> >>>100.0% [2412958] ithread_loop
> >>> 100.0% [2412958] fork_exit
> >>>00.01% [125] tcp_twstart
> >>> 100.0% [125] tcp_do_segment
> >>> 100.0% [125] tcp_input
> >>> 100.0% [125]ip_input
> >>>100.0% [125] swi_net
> >>> 100.0% [125] intr_event_execute_handlers
> >>> 100.0% [125] ithread_loop
> >>> 100.0% [125]fork_exit
> >>
> >> The only write lock tcp_tw_2msl_scan() tries to get is a
> >> INP_WLOCK(inp). Thus here, tcp_tw_2msl_scan() seems to be stuck
> >> spinning on INP_WLOCK (or pfslowtimo() is going crazy and calls
> >> tcp_tw_2msl_scan() at high rate but this will be quite unexpected).
> >>
> >> Thus my hypothesis is that something is holding the INP_WLOCK and not
> >> releasing it, and tcp_tw_2msl_scan() is spinning on it.
> >>
> >> If you can, could you compile the kernel with below options:
> >>
> >> optionsDDB # Support DDB.
> >> optionsDEADLKRES # Enable the deadlock resolver
> >> optionsINVARIANTS # Enable calls of extra sanity
> >> checking
> >> optionsINVARIANT_SUPPORT # Extra sanity checks of internal
> >> structures, required by INVARIANTS
> >> optionsWITNESS # Enable checks to detect
> >> deadlocks and cycles
> >> optionsWITNESS_SKIPSPIN# Don't run witness on spinlocks
> >> for speed
> >
> > Currently this host run with 100% CPU load (on all cores), i.e.
> > enabling WITNESS will be significant drop performance.
> > Can I use only some subset of options?
> >
> > Also, I can some troubles to DDB enter in this case.
> > May be kgdb will be success (not tryed yet)?
>
> If these kernel options will certainly slow down your kernel, they also
> might found the root cause of your issue before reaching the point where
> you have 100% cpu load on all cores (thanks to INVARIANTS). I would
> suggest:
Hmmm, may be I am not clarified.
This host run at peak hours with 100% CPU load as normal operation,
this is for servering 2x10G, this is CPU load not result of lock
issuse, this is not us case. And this is because I am fear to enable
WITNESS -- I am fear drop performance.
This lock issuse happen irregulary and may be caused by other issuse
(nginx crashed). In this case about 1/3 cores have 100% cpu load,
perhaps by this lock -- I am can trace only from one core and need
more then hour for this (may be on other cores different trace, I
can't guaranted anything).
> #1. Try above kernel options at least once, and see what you can get.
OK, I am try this after some time.
> #2. If #1 is a total failure try below patch: It won't solve anything,
> it just makes tcp_tw_2msl_scan() less greedy when there is contention on
> the INP write lock. If it makes the debugging more feasible, continue
> to #3.
OK, thanks.
What purpose to not skip locked tcptw in this loop?
> diff --git a/sys/netinet/tcp_timewait.c b/sys/netinet/tcp_timewait.c
> index a8b78f9..4206ea3 100644
> --- a/sys/netinet/tcp_timewait.c
> +++ b/sys/netinet/tcp_timewait.c
> @@ -701,34 +701,42 @@ tcp_tw_2msl_scan(int reuse)
> in_pcbref(inp);
> TW_RUNLOCK(V_tw_lock);
>
> +retry:
> if (INP_INFO_TRY_RLOCK(&V_tcbinfo)) {
>
> - INP_WLOCK(inp);
> - tw = intotw(inp);
> - if (in_pcbrele_wlocked(inp)) {
> - KASSERT(tw == NULL, ("%s: held last inp "
> - "reference but tw not NULL", __func__));
> - INP_INFO_RUNLOCK(&V_tcbinfo);
> - continue;
> - }
> + if (INP_TRY_WLOCK(inp)) {
> + tw = intotw(inp);
> + if (in_pcbrele_wlocked(inp)) {
> + KASSERT(tw == NULL, ("%s: held
> last inp "
> + "reference but tw not NULL",
> __func__));
> + INP_INFO_RUNLOCK(&V_tcbinfo);
> + continue;
> + }
>
> - i
Re: LAGG and Jumbo Frames
If your lag interface is up, what’s not working? Does something like this work? ping -D -s 8972 and then this not? ping -D -s 8972 If your firewall is on the LAN side supporting jumbo frames ok, but not WAN side, then the router will have to fragment all of the packets. (unless DF bit is set of course). -- Robert inoc.net!rblayzor XMPP: rblayzor.AT.inoc.net PGP Key: 78BEDCE1 @ pgp.mit.edu > On Sep 19, 2016, at 3:23 PM, Dean E. Weimer wrote: > > Does anyone see an issue with the Jumbo Frames setup above, or are Jumbo > Frames not supported correctly in a LACP Aggregate configuration. ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: nginx and FreeBSD11
On Tue, Sep 20, 2016 at 10:20:53PM +0300, Slawa Olhovchenkov wrote:
> On Tue, Sep 20, 2016 at 09:52:44AM +0300, Slawa Olhovchenkov wrote:
>
> > On Mon, Sep 19, 2016 at 06:05:46PM -0700, John Baldwin wrote:
> >
> > > > > If this panics, then vmspace_switch_aio() is not working for
> > > > > some reason.
> > > >
> > > > I am try using next DTrace script:
> > > >
> > > > #pragma D option dynvarsize=64m
> > > >
> > > > int req[struct vmspace *, void *];
> > > > self int trace;
> > > >
> > > > syscall:freebsd:aio_read:entry
> > > > {
> > > > this->aio = *(struct aiocb *)copyin(arg0, sizeof(struct aiocb));
> > > > req[curthread->td_proc->p_vmspace, this->aio.aio_buf] =
> > > > curthread->td_proc->p_pid;
> > > > }
> > > >
> > > > fbt:kernel:aio_process_rw:entry
> > > > {
> > > > self->job = args[0];
> > > > self->trace = 1;
> > > > }
> > > >
> > > > fbt:kernel:aio_process_rw:return
> > > > /self->trace/
> > > > {
> > > > req[self->job->userproc->p_vmspace, self->job->uaiocb.aio_buf]
> > > > = 0;
> > > > self->job = 0;
> > > > self->trace = 0;
> > > > }
> > > >
> > > > fbt:kernel:vn_io_fault:entry
> > > > /self->trace && !req[curthread->td_proc->p_vmspace,
> > > > args[1]->uio_iov[0].iov_base]/
> > > > {
> > > > this->buf = args[1]->uio_iov[0].iov_base;
> > > > printf("%Y vn_io_fault %p:%p pid %d\n", walltimestamp,
> > > > curthread->td_proc->p_vmspace, this->buf,
> > > > req[curthread->td_proc->p_vmspace, this->buf]);
> > > > }
> > > > ===
> > > >
> > > > And don't got any messages near nginx core dump.
> > > > What I can check next?
> > > > May be check context/address space switch for kernel process?
> > >
> > > Which CPU are you using?
> >
> > CPU: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz (2000.04-MHz K8-class CPU)
Is this sandy bridge ? Show me first 100 lines of the verbose dmesg,
I want to see cpu features lines. In particular, does you CPU support
the INVPCID feature.
Also you may show me the 'sysctl vm.pmap' output.
> >
> > > Perhaps try disabling PCID support (I think vm.pmap.pcid_enabled=0 from
> > > loader prompt or loader.conf)? (Wondering if pmap_activate() is somehow
> > > not switching)
>
> I am need some more time to test (day or two), but now this is like
> workaround/solution: 12h runtime and peak hour w/o nginx crash.
> (vm.pmap.pcid_enabled=0 in loader.conf).
Please try this variation of the previous patch.
diff --git a/sys/vm/vm_map.c b/sys/vm/vm_map.c
index a23468e..f754652 100644
--- a/sys/vm/vm_map.c
+++ b/sys/vm/vm_map.c
@@ -481,6 +481,7 @@ vmspace_switch_aio(struct vmspace *newvm)
if (oldvm == newvm)
return;
+ spinlock_enter();
/*
* Point to the new address space and refer to it.
*/
@@ -489,6 +490,7 @@ vmspace_switch_aio(struct vmspace *newvm)
/* Activate the new mapping. */
pmap_activate(curthread);
+ spinlock_exit();
/* Remove the daemon's reference to the old address space. */
KASSERT(oldvm->vm_refcnt > 1,
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: nginx and FreeBSD11
On Tue, Sep 20, 2016 at 09:52:44AM +0300, Slawa Olhovchenkov wrote:
> On Mon, Sep 19, 2016 at 06:05:46PM -0700, John Baldwin wrote:
>
> > > > If this panics, then vmspace_switch_aio() is not working for
> > > > some reason.
> > >
> > > I am try using next DTrace script:
> > >
> > > #pragma D option dynvarsize=64m
> > >
> > > int req[struct vmspace *, void *];
> > > self int trace;
> > >
> > > syscall:freebsd:aio_read:entry
> > > {
> > > this->aio = *(struct aiocb *)copyin(arg0, sizeof(struct aiocb));
> > > req[curthread->td_proc->p_vmspace, this->aio.aio_buf] =
> > > curthread->td_proc->p_pid;
> > > }
> > >
> > > fbt:kernel:aio_process_rw:entry
> > > {
> > > self->job = args[0];
> > > self->trace = 1;
> > > }
> > >
> > > fbt:kernel:aio_process_rw:return
> > > /self->trace/
> > > {
> > > req[self->job->userproc->p_vmspace, self->job->uaiocb.aio_buf] =
> > > 0;
> > > self->job = 0;
> > > self->trace = 0;
> > > }
> > >
> > > fbt:kernel:vn_io_fault:entry
> > > /self->trace && !req[curthread->td_proc->p_vmspace,
> > > args[1]->uio_iov[0].iov_base]/
> > > {
> > > this->buf = args[1]->uio_iov[0].iov_base;
> > > printf("%Y vn_io_fault %p:%p pid %d\n", walltimestamp,
> > > curthread->td_proc->p_vmspace, this->buf,
> > > req[curthread->td_proc->p_vmspace, this->buf]);
> > > }
> > > ===
> > >
> > > And don't got any messages near nginx core dump.
> > > What I can check next?
> > > May be check context/address space switch for kernel process?
> >
> > Which CPU are you using?
>
> CPU: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz (2000.04-MHz K8-class CPU)
>
> > Perhaps try disabling PCID support (I think vm.pmap.pcid_enabled=0 from
> > loader prompt or loader.conf)? (Wondering if pmap_activate() is somehow
> > not switching)
I am need some more time to test (day or two), but now this is like
workaround/solution: 12h runtime and peak hour w/o nginx crash.
(vm.pmap.pcid_enabled=0 in loader.conf).
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: LAGG and Jumbo Frames
On Mon, Sep 19, 2016 at 03:59:20PM -0700, Lyndon Nerenberg wrote: > > > On Sep 19, 2016, at 3:08 PM, Slawa Olhovchenkov wrote: > > > > This is because RTT of this link for jumbo frames higher 1500 bytes > > frame for store-and-forward switch chain. > > For TCP, RTT isn't really a factor (in this scenario), I am don't see scenario in first message. For may scenario this is limiting factor > as the windowing and congestion avoidance algorithms will adapt to the actual > bandwidth-delay product of the link, and the delays in each direction will be > symmetrical. > > Now the ack for a single 9000 octet packet will take longer than > that for a 1500 octet one, but that's because you're sending six > times as many octets before the ACK can be generated. The time to > send six 1500 octet packets and receive the ACK from sixth packet is > going to be comparable to that of receiving the ack from a single > 9000 octet packet. It's simple arithmetic to calculate the extra > protocol header overhead for 6x1500 vs 1x9000. Time to send send six 1500 octet packets significant less then for send one 9000 octet packet over multiple switch: H1-[S1]-[S2]-[S3]-H2 Sendig single 1500 octet packet from H1 to S1 over 1Gbit link: (1500+14+4+12+8)*8/10^9 = 12us switch delayed for 3us same for s1-s2, s2-s3, s3-h2. 2'nd packet delayed for 12us. 3..6 -- same. Sending all six packets (5 inter packets over 4 hop): (12+3)*4 + 12*5 = 120us. Sending single 9000 octet packet from H1 to S1 over 1Gbit link: (9000+14+4+12+8)*8/10^9 = 72us switch delayed for 3us Sending single 9000 octet packet over 4 hop: (72+3)*4 = 300us. 300/120 = 2.5 time slower > If there *is* a significant difference (beyond the extra protocol header > overhead), it's time to take a very close look at the NICs you are using in > the end hosts. A statistically significant difference would hint at poor > interrupt handling performance on the part of one or more of the NICs and > their associated device drivers. > > The intermediate switch overhead will be a constant (unless the switch > backplane becomes saturated from unrelated traffic). You lost serelisation time. ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
