Re: Process stuck in "vnread"

2016-03-29 Thread Maxim Sobolev
Hey Adrian, unfortunately, this is kinda our production build/test box, it's not particularly beefy and we also run some heavily contended apps on it too just to see how it performs on this size of machine, so we would not really want to put any heavy debug into kernel unless it's some issue that

Re: Process stuck in "vnread"

2016-03-28 Thread Maxim Sobolev
Andriy, this is file that gets copies from md(4)-baked UFS to file that is located on ZFS zraid volume that is mounted via NULLFS. The file that backs up md(4) is located on ZFS, so in a sense we have full cycle with the backing block starting on ZFS and ending up on ZFS too. cp(1) calls write(),

Re: Process stuck in "vnread"

2016-03-28 Thread Andriy Gapon
On 28/03/2016 19:23, Konstantin Belousov wrote: > On Mon, Mar 28, 2016 at 08:52:03AM -0700, Maxim Sobolev wrote: >> Done some head scratching, it looks like it's got page fault in the >> copyin() (cp(1) AFAIK mmaps source file). There might be some interlock >> issue between competing write to the

Re: Process stuck in "vnread"

2016-03-28 Thread Konstantin Belousov
On Mon, Mar 28, 2016 at 08:52:03AM -0700, Maxim Sobolev wrote: > Done some head scratching, it looks like it's got page fault in the > copyin() (cp(1) AFAIK mmaps source file). There might be some interlock > issue between competing write to the same ZFS, the md0 device is locked > forever waiting

Re: Process stuck in "vnread"

2016-03-28 Thread Adrian Chadd
Hi, I think a bunch of the lock order checks with witness get disabled for ZFS ? Maybe compile up a witness kernel so you can get 'show alllocks' in ddb. That should help narrow down exactly what's going on. Thanks! -a ___ freebsd-stable@freebsd.org

Re: Process stuck in "vnread"

2016-03-28 Thread Maxim Sobolev
r/home/sobomax/projects/freebsd103/sys/kern/vfs_vnops.c:1225 >> #23 0x80981937 in dofilewrite (td=0xf803320b84b0, fd=4, >> fp=0xf8004bc0ddc0, auio=0xfe0468c4f960, offset=> out>, >> flags=0) at file.h:305 >> #24 0x80981668 in kern_writev (

Re: Process stuck in "vnread"

2016-03-28 Thread Maxim Sobolev
x80cbb559 in amd64_syscall (td=0xf803320b84b0, traced=0) > at subr_syscall.c:141 > #27 0x80ca087b in Xfast_syscall () at > /usr/home/sobomax/projects/freebsd103/sys/amd64/amd64/exception.S:396 > #28 0x000800968f5a in ?? () > > > On Wed, Mar 2, 2016 at 1:

Re: Process stuck in "vnread"

2016-03-28 Thread Maxim Sobolev
0) at subr_syscall.c:141 #27 0x80ca087b in Xfast_syscall () at /usr/home/sobomax/projects/freebsd103/sys/amd64/amd64/exception.S:396 #28 0x000800968f5a in ?? () On Wed, Mar 2, 2016 at 1:12 AM, Maxim Sobolev <sobo...@freebsd.org> wrote: > Hi, I've encountered cp(1) process stu

Re: Process stuck in "vnread"

2016-03-02 Thread Maxim Sobolev
02 Mar 2016 10:12:31 +0100, Maxim Sobolev <sobo...@freebsd.org> > wrote: > > Hi, I've encountered cp(1) process stuck in the vnread state on one of my >> build machines that got recently upgraded to 10.3. >> >>0 79596 1 0 20 0 170921396 wait I

Re: Process stuck in "vnread"

2016-03-02 Thread Maxim Sobolev
nullfs.ko > 31 0x819c8000 2fd2f0 zfs.ko > > > > > On Wed, Mar 2, 2016 at 1:53 AM, Konstantin Belousov <kostik...@gmail.com> > wrote: > >> On Wed, Mar 02, 2016 at 01:12:31AM -0800, Maxim Sobolev wrote: >> > Hi, I've encountered cp(1) pr

Re: Process stuck in "vnread"

2016-03-02 Thread Konstantin Belousov
On Wed, Mar 02, 2016 at 03:02:02AM -0800, Maxim Sobolev wrote: > Thanks, Konstantin. > > Re: md(4) state: > >0 88688 0 0 -8 0 0 16 tx->tx_s DL- 0:45.43 > [md0] > > Its backtrace: So md is stuck in ZFS. ___

Re: Process stuck in "vnread"

2016-03-02 Thread Maxim Sobolev
On Wed, Mar 2, 2016 at 1:53 AM, Konstantin Belousov <kostik...@gmail.com> wrote: > On Wed, Mar 02, 2016 at 01:12:31AM -0800, Maxim Sobolev wrote: > > Hi, I've encountered cp(1) process stuck in the vnread state on one of my > > build machines that got recently upgraded to 10.3

Re: Process stuck in "vnread"

2016-03-02 Thread Konstantin Belousov
On Wed, Mar 02, 2016 at 01:12:31AM -0800, Maxim Sobolev wrote: > Hi, I've encountered cp(1) process stuck in the vnread state on one of my > build machines that got recently upgraded to 10.3. > >0 79596 1 0 20 0 170921396 wait I 1 0:00.00 > /bin/sh

Re: Process stuck in "vnread"

2016-03-02 Thread Ronald Klop
org> wrote: Hi, I've encountered cp(1) process stuck in the vnread state on one of my build machines that got recently upgraded to 10.3. 0 79596 1 0 20 0 170921396 wait I 1 0:00.00 /bin/sh /usr/local/bin/autoreconf -f -i 0 79602 79596 0 52 0 41488903

Process stuck in "vnread"

2016-03-02 Thread Maxim Sobolev
Hi, I've encountered cp(1) process stuck in the vnread state on one of my build machines that got recently upgraded to 10.3. 0 79596 1 0 20 0 170921396 wait I 1 0:00.00 /bin/sh /usr/local/bin/autoreconf -f -i 0 79602 79596 0 52 0 414889036 wait I