FreeBSD_STABLE_10-i386 - Build #1504 - Failure

2016-11-10 Thread jenkins-admin 
FreeBSD_STABLE_10-i386 - Build #1504 - Failure:

Build information: https://jenkins.FreeBSD.org/job/FreeBSD_STABLE_10-i386/1504/
Full change log: 
https://jenkins.FreeBSD.org/job/FreeBSD_STABLE_10-i386/1504/changes
Full build log: 
https://jenkins.FreeBSD.org/job/FreeBSD_STABLE_10-i386/1504/console

Change summaries:

308503 by sephe:
MFC 307985-307988

307985
hyperv/hn: Nuke unnecessary M_NETVSC

Sponsored by:   Microsoft
Differential Revision:  https://reviews.freebsd.org/D8340

307986
hyperv/hn: Move %b format string for capabilities near their definition.

Sponsored by:   Microsoft
Differential Revision:  https://reviews.freebsd.org/D8341

307987
hyperv/hn: Define empty packet filter.

Sponsored by:   Microsoft
Differential Revision:  https://reviews.freebsd.org/D8342

307988
hyperv/hn: Shuffle chimney sending buffer alloc/free around.

This paves way for more chimney sending buffer reorganization.

Sponsored by:   Microsoft
Differential Revision:  https://reviews.freebsd.org/D8343

308502 by sephe:
MFC 307983

hyperv/hn: Properly configure RSS according to RSS capabilities

Sponsored by:   Microsoft
Differential Revision:  https://reviews.freebsd.org/D8338

308501 by sephe:
MFC 307952,307953,308278

307952
hyperv/vmbus: Add missing white space.

Submitted by:   QianYue You 
Sponsored by:   Microsoft

307953
hyperv/vmbus: Implement vmbus_chan_printf.

And use it for vmbus channel logging, which can log the channel
owner's name properly, instead of vmbus0.

Submitted by:   QianYue You 
Sponsored by:   Microsoft

308278
hyperv/vmbus: Reset ch_dev, once the child is deleted.

So it will not be mis-used later on, e.g. in vmbus_chan_printf().

Submitted by:   dexuan
Reported by:dexuan
Sponsored by:   Microsoft

308500 by sephe:
MFC 307893

hyperv/hn: Set baudrate properly

PR: 208931
Submitted by:   Eugene Grosbein 
Reported by:Eugene Grosbein 
Sponsored by:   Microsoft

308499 by sephe:
MFC 307845

hyperv/ic: Rework framework/message version negotiation.

Submitted by:   Hongjiang Zhang 
Modified by:sephe
Sponsored by:   Microsoft
Differential Revision:  https://reviews.freebsd.org/D8280

308498 by sephe:
MFC 307844

hyperv/hn: Nuke unused forward declaration.

Sponsored by:   Microsoft
Differential Revision:  https://reviews.freebsd.org/D8314



The end of the build log:

[...truncated 172586 lines...]
--- all_subdir_vmbus ---
===> hyperv/vmbus (all)
--- hyperv_machdep.o ---
cc  -O2 -pipe  -fno-strict-aliasing -Werror -D_KERNEL -DKLD_MODULE -nostdinc  
-I/usr/src/sys/modules/hyperv/vmbus/../../../dev/hyperv/include 
-I/usr/src/sys/modules/hyperv/vmbus/../../../dev/hyperv/vmbus 
-DHAVE_KERNEL_OPTION_HEADERS -include /usr/obj/usr/src/sys/GENERIC/opt_global.h 
-I. -I@ -I@/contrib/altq -fno-common -g -I/usr/obj/usr/src/sys/GENERIC  
-mno-aes -mno-avx -mno-mmx -mno-sse -msoft-float -ffreestanding 
-fstack-protector -gdwarf-2 -std=iso9899:1999 -Qunused-arguments  
-fstack-protector -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes  
-Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual  -Wundef 
-Wno-pointer-sign -fformat-extensions  -Wmissing-include-dirs 
-fdiagnostics-show-option  -Wno-error-tautological-compare 
-Wno-error-empty-body  -Wno-error-parentheses-equality 
-Wno-error-unused-function-c 
/usr/src/sys/modules/hyperv/vmbus/../../../dev/hyperv/vmbus/i386/hyperv_machdep.c
 -o hyperv_machdep.o
--- all_subdir_geom ---
--- tr_raid1e.o ---
ctfconvert -L VERSION -g tr_raid1e.o
--- all_subdir_hyperv ---
ctfconvert -L VERSION -g hyperv_machdep.o
--- hyperv.o ---
cc  -O2 -pipe  -fno-strict-aliasing -Werror -D_KERNEL -DKLD_MODULE -nostdinc  
-I/usr/src/sys/modules/hyperv/vmbus/../../../dev/hyperv/include 
-I/usr/src/sys/modules/hyperv/vmbus/../../../dev/hyperv/vmbus 
-DHAVE_KERNEL_OPTION_HEADERS -include /usr/obj/usr/src/sys/GENERIC/opt_global.h 
-I. -I@ -I@/contrib/altq -fno-common -g -I/usr/obj/usr/src/sys/GENERIC  
-mno-aes -mno-avx -mno-mmx -mno-sse -msoft-float -ffreestanding 
-fstack-protector -gdwarf-2 -std=iso9899:1999 -Qunused-arguments  
-fstack-protector -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes  
-Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual  -Wundef 
-Wno-pointer-sign -fformat-extensions  -Wmissing-include-dirs 
-fdiagnostics-show-option  -Wno-error-tautological-compare 
-Wno-error-empty-body  -Wno-error-parentheses-equality 
-Wno-error-unused-function-c 
/usr/src/sys/modules/hyperv/vmbus/../../../dev/hyperv/vmbus/hyperv.c -o hyperv.o
--- all_subdir_hwpmc ---
ctfconvert -L VERSION -g hwpmc_soft.o
--- all_subdir_hptrr ---
--- hptrr_osm_bsd.o ---
ctfconvert -L VERSION -g hptrr_osm_bsd.o
--- all_subdir_hwpmc ---
WARNING: hwpmc_soft.c: enum pmc_event has too many values: 1760 > 1023
--- all_subdir_geom ---
--- tr_raid5.o ---
--- all_subdir_hwpmc ---
---

Re: Freebsd 11.0 RELEASE - ZFS deadlock

2016-11-10 Thread Henri Hennebert 

On 11/10/2016 19:40, Andriy Gapon wrote:

On 10/11/2016 19:55, Henri Hennebert wrote:



On 11/10/2016 18:33, Andriy Gapon wrote:

On 10/11/2016 18:12, Henri Hennebert wrote:

On 11/10/2016 16:54, Andriy Gapon wrote:

On 10/11/2016 17:20, Henri Hennebert wrote:

On 11/10/2016 15:00, Andriy Gapon wrote:

Interesting.  I can not spot any suspicious thread that would hold the vnode
lock.  Could you please run kgdb (just like that, no arguments), then execute
'bt' command and then select a frame when _vn_lock is called with 'fr N'
command.  Then please 'print *vp' and share the result.


I Think I miss something in your request:


Oh, sorry!  The very first step should be 'tid 101112' to switch to the correct
context.



(kgdb) fr 7
#7  0x8063c5b3 in _vn_lock (vp=, flags=2121728,


"value optimized out" - not good


file=,
line=) at vnode_if.h:859
859vnode_if.h: No such file or directory.
in vnode_if.h
(kgdb) print *vp


I am not sure if this output is valid, because of the message above.
Could you please try to navigate to nearby frames and see if vp itself has a
valid value there.  If you can find such a frame please do *vp  there.



Does this seems better?


Yes!


(kgdb) fr 8
#8  0x8062a5f7 in vget (vp=0xf80049c2c000, flags=2121728,
td=0xf80009ba0500) at /usr/src/sys/kern/vfs_subr.c:2523
2523if ((error = vn_lock(vp, flags)) != 0) {
(kgdb) print *vp
$1 = {v_tag = 0x813be535 "zfs", v_op = 0x813d0f70, v_data =
0xf80049c1f420, v_mount = 0xf800093aa660,
  v_nmntvnodes = {tqe_next = 0xf80049c2c938, tqe_prev = 0xf80049c2bb30},
v_un = {vu_mount = 0x0, vu_socket = 0x0,
vu_cdev = 0x0, vu_fifoinfo = 0x0}, v_hashlist = {le_next = 0x0, le_prev =
0x0}, v_cache_src = {lh_first = 0x0}, v_cache_dst = {
tqh_first = 0xf800bfc8e3f0, tqh_last = 0xf800bfc8e410}, v_cache_dd =
0x0, v_lock = {lock_object = {
  lo_name = 0x813be535 "zfs", lo_flags = 117112832, lo_data = 0,
lo_witness = 0x0}, lk_lock = 23, lk_exslpfail = 0,
lk_timo = 51, lk_pri = 96}, v_interlock = {lock_object = {lo_name =
0x8099e9e0 "vnode interlock", lo_flags = 16973824,
  lo_data = 0, lo_witness = 0x0}, mtx_lock = 4}, v_vnlock =
0xf80049c2c068, v_actfreelist = {
tqe_next = 0xf80049c2c938, tqe_prev = 0xf80049ae9bd0}, v_bufobj =
{bo_lock = {lock_object = {
lo_name = 0x8099e9f0 "bufobj interlock", lo_flags = 86179840,
lo_data = 0, lo_witness = 0x0}, rw_lock = 1},
bo_ops = 0x80c4bf70, bo_object = 0xf800b62e9c60, bo_synclist =
{le_next = 0x0, le_prev = 0x0},
bo_private = 0xf80049c2c000, __bo_vnode = 0xf80049c2c000, bo_clean =
{bv_hd = {tqh_first = 0x0,
tqh_last = 0xf80049c2c120}, bv_root = {pt_root = 0}, bv_cnt = 0},
bo_dirty = {bv_hd = {tqh_first = 0x0,
tqh_last = 0xf80049c2c140}, bv_root = {pt_root = 0}, bv_cnt = 0},
bo_numoutput = 0, bo_flag = 0, bo_bsize = 131072},
  v_pollinfo = 0x0, v_label = 0x0, v_lockf = 0x0, v_rl = {rl_waiters =
{tqh_first = 0x0, tqh_last = 0xf80049c2c188},
rl_currdep = 0x0}, v_cstart = 0, v_lasta = 0, v_lastw = 0, v_clen = 0,
v_holdcnt = 9, v_usecount = 6, v_iflag = 512,
  v_vflag = 32, v_writecount = 0, v_hash = 4833984, v_type = VREG}
(kgdb)


flags=2121728 = 0x206000 = LK_SHARED | LK_VNHELD | LK_NODDLKTREAT
lk_lock = 23 = 0x17 = LK_ONE_SHARER | LK_EXCLUSIVE_WAITERS | LK_SHARED_WAITERS |
LK_SHARE

So, here's what we have here: this thread tries to get a shared lock on the
vnode, the vnode is already locked in shared mode, but there is an exclusive
waiter (or, perhaps, multiple waiters).  So, this thread can not get the lock
because of the exclusive waiter.  And I do not see an easy way to identify that
waiter.

In the procstat output that you provided earlier there was no other thread in
vn_lock.  Hmm, I see this:
procstat: sysctl: kern.proc.kstack: 14789: Device busy
procstat: sysctl: kern.proc.kstack: 82034: Device busy

Could you please check what those two processes are (if they are still running)?
Perhaps try procstat for each of the pids several times.



This 2 processes are the 2 instances of the innd daemon (news server) 
which seems in accordance with the directory /usr/local/news/bin.


[root@avoriaz ~]# procstat 14789
  PID  PPID  PGID   SID  TSID THR LOGINWCHAN EMUL  COMM
14789 29165 29165 29165 0   1 root zfs   FreeBSD ELF64 innd
[root@avoriaz ~]# procstat 82034
  PID  PPID  PGID   SID  TSID THR LOGINWCHAN EMUL  COMM
82034 29165 29165 29165 0   1 root zfs   FreeBSD ELF64 innd
[root@avoriaz ~]# procstat -f 14789
procstat: kinfo_getfile(): Device busy
  PID COMMFD T V FLAGSREF  OFFSET PRO NAME
[root@avoriaz ~]# procstat -f 14789
procstat: kinfo_getfile(): Device busy
  PID COMMFD T V FLAGSREF  OFFSET PRO NAME
[root@avoriaz ~]# procstat -f 14789
procstat: kinfo_getfile(): Device busy
  PID COMMFD T V FLAGS

Vestibular SENAI 2017/1 | Inscreva-se e garanta sua vaga

2016-11-10 Thread FIERGS | FATEC - Faculdade SENAI de Tecnologia 

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: Freebsd 11.0 RELEASE - ZFS deadlock

2016-11-10 Thread Andriy Gapon 
On 10/11/2016 19:55, Henri Hennebert wrote:
> 
> 
> On 11/10/2016 18:33, Andriy Gapon wrote:
>> On 10/11/2016 18:12, Henri Hennebert wrote:
>>> On 11/10/2016 16:54, Andriy Gapon wrote:
 On 10/11/2016 17:20, Henri Hennebert wrote:
> On 11/10/2016 15:00, Andriy Gapon wrote:
>> Interesting.  I can not spot any suspicious thread that would hold the 
>> vnode
>> lock.  Could you please run kgdb (just like that, no arguments), then 
>> execute
>> 'bt' command and then select a frame when _vn_lock is called with 'fr N'
>> command.  Then please 'print *vp' and share the result.
>>
> I Think I miss something in your request:

 Oh, sorry!  The very first step should be 'tid 101112' to switch to the 
 correct
 context.

>>>
>>> (kgdb) fr 7
>>> #7  0x8063c5b3 in _vn_lock (vp=, flags=2121728,
>>
>> "value optimized out" - not good
>>
>>> file=,
>>> line=) at vnode_if.h:859
>>> 859vnode_if.h: No such file or directory.
>>> in vnode_if.h
>>> (kgdb) print *vp
>>
>> I am not sure if this output is valid, because of the message above.
>> Could you please try to navigate to nearby frames and see if vp itself has a
>> valid value there.  If you can find such a frame please do *vp  there.
>>
> 
> Does this seems better?

Yes!

> (kgdb) fr 8
> #8  0x8062a5f7 in vget (vp=0xf80049c2c000, flags=2121728,
> td=0xf80009ba0500) at /usr/src/sys/kern/vfs_subr.c:2523
> 2523if ((error = vn_lock(vp, flags)) != 0) {
> (kgdb) print *vp
> $1 = {v_tag = 0x813be535 "zfs", v_op = 0x813d0f70, v_data =
> 0xf80049c1f420, v_mount = 0xf800093aa660,
>   v_nmntvnodes = {tqe_next = 0xf80049c2c938, tqe_prev = 
> 0xf80049c2bb30},
> v_un = {vu_mount = 0x0, vu_socket = 0x0,
> vu_cdev = 0x0, vu_fifoinfo = 0x0}, v_hashlist = {le_next = 0x0, le_prev =
> 0x0}, v_cache_src = {lh_first = 0x0}, v_cache_dst = {
> tqh_first = 0xf800bfc8e3f0, tqh_last = 0xf800bfc8e410}, 
> v_cache_dd =
> 0x0, v_lock = {lock_object = {
>   lo_name = 0x813be535 "zfs", lo_flags = 117112832, lo_data = 0,
> lo_witness = 0x0}, lk_lock = 23, lk_exslpfail = 0,
> lk_timo = 51, lk_pri = 96}, v_interlock = {lock_object = {lo_name =
> 0x8099e9e0 "vnode interlock", lo_flags = 16973824,
>   lo_data = 0, lo_witness = 0x0}, mtx_lock = 4}, v_vnlock =
> 0xf80049c2c068, v_actfreelist = {
> tqe_next = 0xf80049c2c938, tqe_prev = 0xf80049ae9bd0}, v_bufobj =
> {bo_lock = {lock_object = {
> lo_name = 0x8099e9f0 "bufobj interlock", lo_flags = 86179840,
> lo_data = 0, lo_witness = 0x0}, rw_lock = 1},
> bo_ops = 0x80c4bf70, bo_object = 0xf800b62e9c60, bo_synclist =
> {le_next = 0x0, le_prev = 0x0},
> bo_private = 0xf80049c2c000, __bo_vnode = 0xf80049c2c000, 
> bo_clean =
> {bv_hd = {tqh_first = 0x0,
> tqh_last = 0xf80049c2c120}, bv_root = {pt_root = 0}, bv_cnt = 0},
> bo_dirty = {bv_hd = {tqh_first = 0x0,
> tqh_last = 0xf80049c2c140}, bv_root = {pt_root = 0}, bv_cnt = 0},
> bo_numoutput = 0, bo_flag = 0, bo_bsize = 131072},
>   v_pollinfo = 0x0, v_label = 0x0, v_lockf = 0x0, v_rl = {rl_waiters =
> {tqh_first = 0x0, tqh_last = 0xf80049c2c188},
> rl_currdep = 0x0}, v_cstart = 0, v_lasta = 0, v_lastw = 0, v_clen = 0,
> v_holdcnt = 9, v_usecount = 6, v_iflag = 512,
>   v_vflag = 32, v_writecount = 0, v_hash = 4833984, v_type = VREG}
> (kgdb)

flags=2121728 = 0x206000 = LK_SHARED | LK_VNHELD | LK_NODDLKTREAT
lk_lock = 23 = 0x17 = LK_ONE_SHARER | LK_EXCLUSIVE_WAITERS | LK_SHARED_WAITERS |
LK_SHARE

So, here's what we have here: this thread tries to get a shared lock on the
vnode, the vnode is already locked in shared mode, but there is an exclusive
waiter (or, perhaps, multiple waiters).  So, this thread can not get the lock
because of the exclusive waiter.  And I do not see an easy way to identify that
waiter.

In the procstat output that you provided earlier there was no other thread in
vn_lock.  Hmm, I see this:
procstat: sysctl: kern.proc.kstack: 14789: Device busy
procstat: sysctl: kern.proc.kstack: 82034: Device busy

Could you please check what those two processes are (if they are still running)?
Perhaps try procstat for each of the pids several times.

-- 
Andriy Gapon
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: Freebsd 11.0 RELEASE - ZFS deadlock

2016-11-10 Thread Henri Hennebert 



On 11/10/2016 18:33, Andriy Gapon wrote:

On 10/11/2016 18:12, Henri Hennebert wrote:

On 11/10/2016 16:54, Andriy Gapon wrote:

On 10/11/2016 17:20, Henri Hennebert wrote:

On 11/10/2016 15:00, Andriy Gapon wrote:

Interesting.  I can not spot any suspicious thread that would hold the vnode
lock.  Could you please run kgdb (just like that, no arguments), then execute
'bt' command and then select a frame when _vn_lock is called with 'fr N'
command.  Then please 'print *vp' and share the result.


I Think I miss something in your request:


Oh, sorry!  The very first step should be 'tid 101112' to switch to the correct
context.



(kgdb) fr 7
#7  0x8063c5b3 in _vn_lock (vp=, flags=2121728,


"value optimized out" - not good


file=,
line=) at vnode_if.h:859
859vnode_if.h: No such file or directory.
in vnode_if.h
(kgdb) print *vp


I am not sure if this output is valid, because of the message above.
Could you please try to navigate to nearby frames and see if vp itself has a
valid value there.  If you can find such a frame please do *vp  there.



Does this seems better?

(kgdb) fr 8
#8  0x8062a5f7 in vget (vp=0xf80049c2c000, flags=2121728, 
td=0xf80009ba0500) at /usr/src/sys/kern/vfs_subr.c:2523

2523if ((error = vn_lock(vp, flags)) != 0) {
(kgdb) print *vp
$1 = {v_tag = 0x813be535 "zfs", v_op = 0x813d0f70, 
v_data = 0xf80049c1f420, v_mount = 0xf800093aa660,
  v_nmntvnodes = {tqe_next = 0xf80049c2c938, tqe_prev = 
0xf80049c2bb30}, v_un = {vu_mount = 0x0, vu_socket = 0x0,
vu_cdev = 0x0, vu_fifoinfo = 0x0}, v_hashlist = {le_next = 0x0, 
le_prev = 0x0}, v_cache_src = {lh_first = 0x0}, v_cache_dst = {
tqh_first = 0xf800bfc8e3f0, tqh_last = 0xf800bfc8e410}, 
v_cache_dd = 0x0, v_lock = {lock_object = {
  lo_name = 0x813be535 "zfs", lo_flags = 117112832, lo_data 
= 0, lo_witness = 0x0}, lk_lock = 23, lk_exslpfail = 0,
lk_timo = 51, lk_pri = 96}, v_interlock = {lock_object = {lo_name = 
0x8099e9e0 "vnode interlock", lo_flags = 16973824,
  lo_data = 0, lo_witness = 0x0}, mtx_lock = 4}, v_vnlock = 
0xf80049c2c068, v_actfreelist = {
tqe_next = 0xf80049c2c938, tqe_prev = 0xf80049ae9bd0}, 
v_bufobj = {bo_lock = {lock_object = {
lo_name = 0x8099e9f0 "bufobj interlock", lo_flags = 
86179840, lo_data = 0, lo_witness = 0x0}, rw_lock = 1},
bo_ops = 0x80c4bf70, bo_object = 0xf800b62e9c60, 
bo_synclist = {le_next = 0x0, le_prev = 0x0},
bo_private = 0xf80049c2c000, __bo_vnode = 0xf80049c2c000, 
bo_clean = {bv_hd = {tqh_first = 0x0,
tqh_last = 0xf80049c2c120}, bv_root = {pt_root = 0}, bv_cnt 
= 0}, bo_dirty = {bv_hd = {tqh_first = 0x0,
tqh_last = 0xf80049c2c140}, bv_root = {pt_root = 0}, bv_cnt 
= 0}, bo_numoutput = 0, bo_flag = 0, bo_bsize = 131072},
  v_pollinfo = 0x0, v_label = 0x0, v_lockf = 0x0, v_rl = {rl_waiters = 
{tqh_first = 0x0, tqh_last = 0xf80049c2c188},
rl_currdep = 0x0}, v_cstart = 0, v_lasta = 0, v_lastw = 0, v_clen = 
0, v_holdcnt = 9, v_usecount = 6, v_iflag = 512,

  v_vflag = 32, v_writecount = 0, v_hash = 4833984, v_type = VREG}
(kgdb)

Henri
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: Freebsd 11.0 RELEASE - ZFS deadlock

2016-11-10 Thread Andriy Gapon 
On 10/11/2016 18:12, Henri Hennebert wrote:
> On 11/10/2016 16:54, Andriy Gapon wrote:
>> On 10/11/2016 17:20, Henri Hennebert wrote:
>>> On 11/10/2016 15:00, Andriy Gapon wrote:
 Interesting.  I can not spot any suspicious thread that would hold the 
 vnode
 lock.  Could you please run kgdb (just like that, no arguments), then 
 execute
 'bt' command and then select a frame when _vn_lock is called with 'fr N'
 command.  Then please 'print *vp' and share the result.

>>> I Think I miss something in your request:
>>
>> Oh, sorry!  The very first step should be 'tid 101112' to switch to the 
>> correct
>> context.
>>
> 
> (kgdb) fr 7
> #7  0x8063c5b3 in _vn_lock (vp=, flags=2121728,

"value optimized out" - not good

> file=,
> line=) at vnode_if.h:859
> 859vnode_if.h: No such file or directory.
> in vnode_if.h
> (kgdb) print *vp

I am not sure if this output is valid, because of the message above.
Could you please try to navigate to nearby frames and see if vp itself has a
valid value there.  If you can find such a frame please do *vp  there.

> $1 = {v_tag = 0x80faeb78 "â~\231\200", v_op = 0xf80009a41000,
> v_data = 0x0, v_mount = 0xf80009a41010,
>   v_nmntvnodes = {tqe_next = 0x0, tqe_prev = 0x80edc088}, v_un =
> {vu_mount = 0x0, vu_socket = 0x0, vu_cdev = 0x0,
> vu_fifoinfo = 0x0}, v_hashlist = {le_next = 0xf80009466e90, le_prev =
> 0x0}, v_cache_src = {lh_first = 0xfe010186d768},
>   v_cache_dst = {tqh_first = 0x0, tqh_last = 0xfeb8a7c0}, v_cache_dd =
> 0xf8000284f000, v_lock = {lock_object = {
>   lo_name = 0xf8002c00ee80 "", lo_flags = 0, lo_data = 0, lo_witness =
> 0xf800068bd480},
> lk_lock = 1844673520268056, lk_exslpfail = 153715840, lk_timo = -2048,
> lk_pri = 0}, v_interlock = {lock_object = {
>   lo_name = 0x18af8  address>, lo_flags = 0, lo_data = 0,
>   lo_witness = 0x0}, mtx_lock = 0}, v_vnlock = 0x0, v_actfreelist =
> {tqe_next = 0x0, tqe_prev = 0xf80009ba05c0},
>   v_bufobj = {bo_lock = {lock_object = {lo_name = 0xf80009a41000 "",
> lo_flags = 1, lo_data = 0, lo_witness = 0x400ff},
>   rw_lock = 2}, bo_ops = 0x1, bo_object = 0xf80049c2c068,
> bo_synclist = {le_next = 0x813be535,
>   le_prev = 0x1}, bo_private = 0x0, __bo_vnode = 0x0, 
> bo_clean =
> {bv_hd = {tqh_first = 0x0, tqh_last = 0x0},
>   bv_root = {pt_root = 0}, bv_cnt = 0}, bo_dirty = {bv_hd = {tqh_first =
> 0xf80088ac8d00, tqh_last = 0xf8003cc5b600},
>   bv_root = {pt_root = 2553161591}, bv_cnt = -1741805705}, bo_numoutput =
> 31, bo_flag = 0, bo_bsize = 0}, v_pollinfo = 0x0,
>   v_label = 0x0, v_lockf = 0x0, v_rl = {rl_waiters = {tqh_first = 0xf88,
> tqh_last = 0x19cc}, rl_currdep = 0x3f8},
>   v_cstart = 16256, v_lasta = 679, v_lastw = 0, v_clen = 0, v_holdcnt = 0,
> v_usecount = 2369, v_iflag = 0, v_vflag = 0,
>   v_writecount = 0, v_hash = 0, v_type = VNON}
> (kgdb)
> 
> Thanks for your time
> 
> Henri


-- 
Andriy Gapon
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: Freebsd 11.0 RELEASE - ZFS deadlock

2016-11-10 Thread Henri Hennebert 

On 11/10/2016 16:54, Andriy Gapon wrote:

On 10/11/2016 17:20, Henri Hennebert wrote:

On 11/10/2016 15:00, Andriy Gapon wrote:

Interesting.  I can not spot any suspicious thread that would hold the vnode
lock.  Could you please run kgdb (just like that, no arguments), then execute
'bt' command and then select a frame when _vn_lock is called with 'fr N'
command.  Then please 'print *vp' and share the result.


I Think I miss something in your request:


Oh, sorry!  The very first step should be 'tid 101112' to switch to the correct
context.



(kgdb) fr 7
#7  0x8063c5b3 in _vn_lock (vp=, 
flags=2121728, file=,

line=) at vnode_if.h:859
859 vnode_if.h: No such file or directory.
in vnode_if.h
(kgdb) print *vp
$1 = {v_tag = 0x80faeb78 "â~\231\200", v_op = 
0xf80009a41000, v_data = 0x0, v_mount = 0xf80009a41010,
  v_nmntvnodes = {tqe_next = 0x0, tqe_prev = 0x80edc088}, v_un 
= {vu_mount = 0x0, vu_socket = 0x0, vu_cdev = 0x0,
vu_fifoinfo = 0x0}, v_hashlist = {le_next = 0xf80009466e90, 
le_prev = 0x0}, v_cache_src = {lh_first = 0xfe010186d768},
  v_cache_dst = {tqh_first = 0x0, tqh_last = 0xfeb8a7c0}, 
v_cache_dd = 0xf8000284f000, v_lock = {lock_object = {
  lo_name = 0xf8002c00ee80 "", lo_flags = 0, lo_data = 0, 
lo_witness = 0xf800068bd480},
lk_lock = 1844673520268056, lk_exslpfail = 153715840, lk_timo = 
-2048, lk_pri = 0}, v_interlock = {lock_object = {
  lo_name = 0x18af8 Bad address>, lo_flags = 0, lo_data = 0,
  lo_witness = 0x0}, mtx_lock = 0}, v_vnlock = 0x0, v_actfreelist = 
{tqe_next = 0x0, tqe_prev = 0xf80009ba05c0},
  v_bufobj = {bo_lock = {lock_object = {lo_name = 0xf80009a41000 
"", lo_flags = 1, lo_data = 0, lo_witness = 0x400ff},
  rw_lock = 2}, bo_ops = 0x1, bo_object = 
0xf80049c2c068, bo_synclist = {le_next = 0x813be535,
  le_prev = 0x1}, bo_private = 0x0, __bo_vnode = 0x0, 
bo_clean = {bv_hd = {tqh_first = 0x0, tqh_last = 0x0},
  bv_root = {pt_root = 0}, bv_cnt = 0}, bo_dirty = {bv_hd = 
{tqh_first = 0xf80088ac8d00, tqh_last = 0xf8003cc5b600},
  bv_root = {pt_root = 2553161591}, bv_cnt = -1741805705}, 
bo_numoutput = 31, bo_flag = 0, bo_bsize = 0}, v_pollinfo = 0x0,
  v_label = 0x0, v_lockf = 0x0, v_rl = {rl_waiters = {tqh_first = 
0xf88, tqh_last = 0x19cc}, rl_currdep = 0x3f8},
  v_cstart = 16256, v_lasta = 679, v_lastw = 0, v_clen = 0, v_holdcnt = 
0, v_usecount = 2369, v_iflag = 0, v_vflag = 0,

  v_writecount = 0, v_hash = 0, v_type = VNON}
(kgdb)

Thanks for your time

Henri
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: Freebsd 11.0 RELEASE - ZFS deadlock

2016-11-10 Thread Andriy Gapon 
On 10/11/2016 17:20, Henri Hennebert wrote:
> On 11/10/2016 15:00, Andriy Gapon wrote:
>> Interesting.  I can not spot any suspicious thread that would hold the vnode
>> lock.  Could you please run kgdb (just like that, no arguments), then execute
>> 'bt' command and then select a frame when _vn_lock is called with 'fr N'
>> command.  Then please 'print *vp' and share the result.
>>
> I Think I miss something in your request:

Oh, sorry!  The very first step should be 'tid 101112' to switch to the correct
context.

> [root@avoriaz ~]# kgdb
> GNU gdb 6.1.1 [FreeBSD]
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you are
> welcome to change it and/or distribute copies of it under certain conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB.  Type "show warranty" for details.
> This GDB was configured as "amd64-marcel-freebsd"...
> Reading symbols from /boot/kernel/zfs.ko...Reading symbols from
> /usr/lib/debug//boot/kernel/zfs.ko.debug...done.
> done.
> Loaded symbols for /boot/kernel/zfs.ko
> Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from
> /usr/lib/debug//boot/kernel/opensolaris.ko.debug...done.
> done.
> 
> --- clip ---
> 
> Loaded symbols for /boot/kernel/accf_data.ko
> Reading symbols from /boot/kernel/daemon_saver.ko...Reading symbols from
> /usr/lib/debug//boot/kernel/daemon_saver.ko.debug...done.
> done.
> Loaded symbols for /boot/kernel/daemon_saver.ko
> #0  sched_switch (td=0xf8001131da00, newtd=0xf800762a8500, 
> flags= optimized out>)
> at /usr/src/sys/kern/sched_ule.c:1973
> 1973cpuid = PCPU_GET(cpuid);
> (kgdb) bt
> #0  sched_switch (td=0xf8001131da00, newtd=0xf800762a8500, 
> flags= optimized out>)
> at /usr/src/sys/kern/sched_ule.c:1973
> #1  0x80566b15 in tc_fill_vdso_timehands32 (vdso_th32=0x0) at
> /usr/src/sys/kern/kern_tc.c:2121
> #2  0x80555227 in timekeep_push_vdso () at
> /usr/src/sys/kern/kern_sharedpage.c:174
> #3  0x80566226 in tc_windup () at /usr/src/sys/kern/kern_tc.c:1426
> #4  0x804eaa41 in hardclock_cnt (cnt=1, usermode= out>)
> at /usr/src/sys/kern/kern_clock.c:589
> #5  0x808fac74 in handleevents (now=, fake=0) at
> /usr/src/sys/kern/kern_clocksource.c:223
> #6  0x808fb1d7 in timercb (et=0x8100cf20, arg= out>) at /usr/src/sys/kern/kern_clocksource.c:352
> #7  0xf800b6429a00 in ?? ()
> #8  0x81051080 in vm_page_array ()
> #9  0x81051098 in vm_page_queue_free_mtx ()
> #10 0xfe0101818920 in ?? ()
> #11 0x805399c0 in __mtx_lock_sleep (c=, tid=Error
> accessing memory address 0xffac: Bad add\
> ress.
> ) at /usr/src/sys/kern/kern_mutex.c:590
> Previous frame inner to this frame (corrupt stack?)
> Current language:  auto; currently minimal
> (kgdb) q
> [root@avoriaz ~]#
> 
> I don't find the requested frame
> 
> Henri


-- 
Andriy Gapon
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: Freebsd 11.0 RELEASE - ZFS deadlock

2016-11-10 Thread Henri Hennebert 

On 11/10/2016 15:00, Andriy Gapon wrote:

On 10/11/2016 12:30, Henri Hennebert wrote:

On 11/10/2016 11:21, Andriy Gapon wrote:

On 09/11/2016 15:58, Eric van Gyzen wrote:

On 11/09/2016 07:48, Henri Hennebert wrote:

I encounter a strange deadlock on

FreeBSD avoriaz.restart.bel 11.0-RELEASE-p3 FreeBSD 11.0-RELEASE-p3 #0 r308260:
Fri Nov  4 02:51:33 CET 2016
r...@avoriaz.restart.bel:/usr/obj/usr/src/sys/AVORIAZ  amd64

This system is exclusively running on zfs.

After 3 or 4 days, `periodic daily` is locked in the directory
/usr/local/news/bin

[root@avoriaz ~]# ps xa|grep find
85656  -  D0:01.13 find / ( ! -fstype local -o -fstype rdonly ) -prune
-o ( -name [#,]* -o -name .#* -o -name a.out -o -nam
  462  1  S+   0:00.00 grep find
[root@avoriaz ~]# procstat -f 85656
  PID COMMFD T V FLAGSREF  OFFSET PRO NAME
85656 find  text v r r---   -   - - /usr/bin/find
85656 find   cwd v d r---   -   - - /usr/local/news/bin
85656 find  root v d r---   -   - - /
85656 find 0 v c r---   3   0 - /dev/null
85656 find 1 p - rw--   1   0 - -
85656 find 2 v r -w--   7  17 - -
85656 find 3 v d r---   1   0 - /home/root
85656 find 4 v d r---   1   0 - /home/root
85656 find 5 v d rn--   1 533545184 - /usr/local/news/bin
[root@avoriaz ~]#

If I try `ls /usr/local/news/bin` it is also locked.

After `shutdown -r now` the system remain locked after the line '0 0 0 0 0 0'

After a reset and reboot  I can access /usr/local/news/bin.

I delete this directory and reinstall the package `portupgrade -fu news/inn`

5 days later `periodic daily`is locked on the same directory :-o

Any idea?


I can't help with the deadlock, but someone who _can_ help will probably ask for
the output of "procstat -kk PID" with the PID of the "find" process.


In fact, it's procstat -kk -a.  With just one thread we would see that a thread
is blocked on something, but we won't see why that something can not be 
acquired.



I attach the result,


Interesting.  I can not spot any suspicious thread that would hold the vnode
lock.  Could you please run kgdb (just like that, no arguments), then execute
'bt' command and then select a frame when _vn_lock is called with 'fr N'
command.  Then please 'print *vp' and share the result.


I Think I miss something in your request:

[root@avoriaz ~]# kgdb
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain 
conditions.

Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...
Reading symbols from /boot/kernel/zfs.ko...Reading symbols from 
/usr/lib/debug//boot/kernel/zfs.ko.debug...done.

done.
Loaded symbols for /boot/kernel/zfs.ko
Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from 
/usr/lib/debug//boot/kernel/opensolaris.ko.debug...done.

done.

--- clip ---

Loaded symbols for /boot/kernel/accf_data.ko
Reading symbols from /boot/kernel/daemon_saver.ko...Reading symbols from 
/usr/lib/debug//boot/kernel/daemon_saver.ko.debug...done.

done.
Loaded symbols for /boot/kernel/daemon_saver.ko
#0  sched_switch (td=0xf8001131da00, newtd=0xf800762a8500, 
flags=)

at /usr/src/sys/kern/sched_ule.c:1973
1973cpuid = PCPU_GET(cpuid);
(kgdb) bt
#0  sched_switch (td=0xf8001131da00, newtd=0xf800762a8500, 
flags=)

at /usr/src/sys/kern/sched_ule.c:1973
#1  0x80566b15 in tc_fill_vdso_timehands32 (vdso_th32=0x0) at 
/usr/src/sys/kern/kern_tc.c:2121
#2  0x80555227 in timekeep_push_vdso () at 
/usr/src/sys/kern/kern_sharedpage.c:174

#3  0x80566226 in tc_windup () at /usr/src/sys/kern/kern_tc.c:1426
#4  0x804eaa41 in hardclock_cnt (cnt=1, usermode=optimized out>) at /usr/src/sys/kern/kern_clock.c:589
#5  0x808fac74 in handleevents (now=, 
fake=0) at /usr/src/sys/kern/kern_clocksource.c:223
#6  0x808fb1d7 in timercb (et=0x8100cf20, arg=optimized out>) at /usr/src/sys/kern/kern_clocksource.c:352

#7  0xf800b6429a00 in ?? ()
#8  0x81051080 in vm_page_array ()
#9  0x81051098 in vm_page_queue_free_mtx ()
#10 0xfe0101818920 in ?? ()
#11 0x805399c0 in __mtx_lock_sleep (c=, 
tid=Error accessing memory address 0xffac: Bad add\

ress.
) at /usr/src/sys/kern/kern_mutex.c:590
Previous frame inner to this frame (corrupt stack?)
Current language:  auto; currently minimal
(kgdb) q
[root@avoriaz ~]#

I don't find the requested frame

Henri
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To

Re: Freebsd 11.0 RELEASE - ZFS deadlock

2016-11-10 Thread Andriy Gapon 
On 10/11/2016 12:30, Henri Hennebert wrote:
> On 11/10/2016 11:21, Andriy Gapon wrote:
>> On 09/11/2016 15:58, Eric van Gyzen wrote:
>>> On 11/09/2016 07:48, Henri Hennebert wrote:
 I encounter a strange deadlock on

 FreeBSD avoriaz.restart.bel 11.0-RELEASE-p3 FreeBSD 11.0-RELEASE-p3 #0 
 r308260:
 Fri Nov  4 02:51:33 CET 2016
 r...@avoriaz.restart.bel:/usr/obj/usr/src/sys/AVORIAZ  amd64

 This system is exclusively running on zfs.

 After 3 or 4 days, `periodic daily` is locked in the directory
 /usr/local/news/bin

 [root@avoriaz ~]# ps xa|grep find
 85656  -  D0:01.13 find / ( ! -fstype local -o -fstype rdonly ) 
 -prune
 -o ( -name [#,]* -o -name .#* -o -name a.out -o -nam
   462  1  S+   0:00.00 grep find
 [root@avoriaz ~]# procstat -f 85656
   PID COMMFD T V FLAGSREF  OFFSET PRO NAME
 85656 find  text v r r---   -   - - /usr/bin/find
 85656 find   cwd v d r---   -   - - /usr/local/news/bin
 85656 find  root v d r---   -   - - /
 85656 find 0 v c r---   3   0 - /dev/null
 85656 find 1 p - rw--   1   0 - -
 85656 find 2 v r -w--   7  17 - -
 85656 find 3 v d r---   1   0 - /home/root
 85656 find 4 v d r---   1   0 - /home/root
 85656 find 5 v d rn--   1 533545184 - 
 /usr/local/news/bin
 [root@avoriaz ~]#

 If I try `ls /usr/local/news/bin` it is also locked.

 After `shutdown -r now` the system remain locked after the line '0 0 0 0 0 
 0'

 After a reset and reboot  I can access /usr/local/news/bin.

 I delete this directory and reinstall the package `portupgrade -fu 
 news/inn`

 5 days later `periodic daily`is locked on the same directory :-o

 Any idea?
>>>
>>> I can't help with the deadlock, but someone who _can_ help will probably 
>>> ask for
>>> the output of "procstat -kk PID" with the PID of the "find" process.
>>
>> In fact, it's procstat -kk -a.  With just one thread we would see that a 
>> thread
>> is blocked on something, but we won't see why that something can not be 
>> acquired.
>>
>>
> I attach the result,

Interesting.  I can not spot any suspicious thread that would hold the vnode
lock.  Could you please run kgdb (just like that, no arguments), then execute
'bt' command and then select a frame when _vn_lock is called with 'fr N'
command.  Then please 'print *vp' and share the result.



-- 
Andriy Gapon
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

RES: Mês de Black Friday

2016-11-10 Thread Loja Catholicus 
Loja Catholicus

 
[http://trk.promocat.expvtinboxonline.com/index.dma/DmaClick?20214,864,194181,13298,22949d61fc9f335749fb76cc9c59e18b,aHR0cDovL2xvamFjYXRob2xpY3VzLmNvbS5ici8/dXRtX3NvdXJjZT1WaXJ0dWFsX1RhcmdldCZ1dG1fbWVkaXVtPUVtYWlsJnV0bV9jb250ZW50PSZ1dG1fY2FtcGFpZ249Y2F0aG9saWN1cyZ1dG1fdGVybT0=,1,ZnJlZWJzZC5vcmc=]

 
[http://trk.promocat.expvtinboxonline.com/index.dma/DmaClick?20214,864,194181,13299,22949d61fc9f335749fb76cc9c59e18b,aHR0cDovL3BhaW5lbC52aXJ0dWFsdGFyZ2V0LmNvbS5ici9pbmRleC5kbWEvU29jaWFsU2hhcmU/cD0yMDIxNCw4NjQsMSZ1cmw9aHR0cDovL3Ryay5wcm9tb2NhdC5leHB2dGluYm94b25saW5lLmNvbS9pbmRleC5kbWEvRG1hUHJldmlldz8yMDIxNC44NjQuMTk0MTgxLjIyOTQ5ZDYxZmM5ZjMzNTc0OWZiNzZjYzljNTllMThiLjEmdXRtX3NvdXJjZT1WaXJ0dWFsX1RhcmdldCZ1dG1fbWVkaXVtPUVtYWlsJnV0bV9jb250ZW50PSZ1dG1fY2FtcGFpZ249Y2F0aG9saWN1cyZ1dG1fdGVybT0=,1,ZnJlZWJzZC5vcmc=]
Twitter
[http://trk.promocat.expvtinboxonline.com/index.dma/DmaClick?20214,864,194181,13299,22949d61fc9f335749fb76cc9c59e18b,aHR0cDovL3BhaW5lbC52aXJ0dWFsdGFyZ2V0LmNvbS5ici9pbmRleC5kbWEvU29jaWFsU2hhcmU/cD0yMDIxNCw4NjQsMSZ1cmw9aHR0cDovL3Ryay5wcm9tb2NhdC5leHB2dGluYm94b25saW5lLmNvbS9pbmRleC5kbWEvRG1hUHJldmlldz8yMDIxNC44NjQuMTk0MTgxLjIyOTQ5ZDYxZmM5ZjMzNTc0OWZiNzZjYzljNTllMThiLjEmdXRtX3NvdXJjZT1WaXJ0dWFsX1RhcmdldCZ1dG1fbWVkaXVtPUVtYWlsJnV0bV9jb250ZW50PSZ1dG1fY2FtcGFpZ249Y2F0aG9saWN1cyZ1dG1fdGVybT0=,1,ZnJlZWJzZC5vcmc=]
 
[http://trk.promocat.expvtinboxonline.com/index.dma/DmaClick?20214,864,194181,13300,22949d61fc9f335749fb76cc9c59e18b,aHR0cDovL3BhaW5lbC52aXJ0dWFsdGFyZ2V0LmNvbS5ici9pbmRleC5kbWEvU29jaWFsU2hhcmU/cD0yMDIxNCw4NjQsMiZ1cmw9aHR0cDovL3Ryay5wcm9tb2NhdC5leHB2dGluYm94b25saW5lLmNvbS9pbmRleC5kbWEvRG1hUHJldmlldz8yMDIxNC44NjQuMTk0MTgxLjIyOTQ5ZDYxZmM5ZjMzNTc0OWZiNzZjYzljNTllMThiLjEmdXRtX3NvdXJjZT1WaXJ0dWFsX1RhcmdldCZ1dG1fbWVkaXVtPUVtYWlsJnV0bV9jb250ZW50PSZ1dG1fY2FtcGFpZ249Y2F0aG9saWN1cyZ1dG1fdGVybT0=,1,ZnJlZWJzZC5vcmc=]
Facebook
[http://trk.promocat.expvtinboxonline.com/index.dma/DmaClick?20214,864,194181,13300,22949d61fc9f335749fb76cc9c59e18b,aHR0cDovL3BhaW5lbC52aXJ0dWFsdGFyZ2V0LmNvbS5ici9pbmRleC5kbWEvU29jaWFsU2hhcmU/cD0yMDIxNCw4NjQsMiZ1cmw9aHR0cDovL3Ryay5wcm9tb2NhdC5leHB2dGluYm94b25saW5lLmNvbS9pbmRleC5kbWEvRG1hUHJldmlldz8yMDIxNC44NjQuMTk0MTgxLjIyOTQ5ZDYxZmM5ZjMzNTc0OWZiNzZjYzljNTllMThiLjEmdXRtX3NvdXJjZT1WaXJ0dWFsX1RhcmdldCZ1dG1fbWVkaXVtPUVtYWlsJnV0bV9jb250ZW50PSZ1dG1fY2FtcGFpZ249Y2F0aG9saWN1cyZ1dG1fdGVybT0=,1,ZnJlZWJzZC5vcmc=]
 
[http://trk.promocat.expvtinboxonline.com/index.dma/DmaClick?20214,864,194181,13301,22949d61fc9f335749fb76cc9c59e18b,aHR0cDovL3BhaW5lbC52aXJ0dWFsdGFyZ2V0LmNvbS5ici9pbmRleC5kbWEvU29jaWFsU2hhcmU/cD0yMDIxNCw4NjQsMyZ1cmw9aHR0cDovL3Ryay5wcm9tb2NhdC5leHB2dGluYm94b25saW5lLmNvbS9pbmRleC5kbWEvRG1hUHJldmlldz8yMDIxNC44NjQuMTk0MTgxLjIyOTQ5ZDYxZmM5ZjMzNTc0OWZiNzZjYzljNTllMThiLjEmdXRtX3NvdXJjZT1WaXJ0dWFsX1RhcmdldCZ1dG1fbWVkaXVtPUVtYWlsJnV0bV9jb250ZW50PSZ1dG1fY2FtcGFpZ249Y2F0aG9saWN1cyZ1dG1fdGVybT0=,1,ZnJlZWJzZC5vcmc=]
Delicious
[http://trk.promocat.expvtinboxonline.com/index.dma/DmaClick?20214,864,194181,13301,22949d61fc9f335749fb76cc9c59e18b,aHR0cDovL3BhaW5lbC52aXJ0dWFsdGFyZ2V0LmNvbS5ici9pbmRleC5kbWEvU29jaWFsU2hhcmU/cD0yMDIxNCw4NjQsMyZ1cmw9aHR0cDovL3Ryay5wcm9tb2NhdC5leHB2dGluYm94b25saW5lLmNvbS9pbmRleC5kbWEvRG1hUHJldmlldz8yMDIxNC44NjQuMTk0MTgxLjIyOTQ5ZDYxZmM5ZjMzNTc0OWZiNzZjYzljNTllMThiLjEmdXRtX3NvdXJjZT1WaXJ0dWFsX1RhcmdldCZ1dG1fbWVkaXVtPUVtYWlsJnV0bV9jb250ZW50PSZ1dG1fY2FtcGFpZ249Y2F0aG9saWN1cyZ1dG1fdGVybT0=,1,ZnJlZWJzZC5vcmc=]
 
[http://trk.promocat.expvtinboxonline.com/index.dma/DmaClick?20214,864,194181,13302,22949d61fc9f335749fb76cc9c59e18b,aHR0cDovL3BhaW5lbC52aXJ0dWFsdGFyZ2V0LmNvbS5ici9pbmRleC5kbWEvU29jaWFsU2hhcmU/cD0yMDIxNCw4NjQsNCZ1cmw9aHR0cDovL3Ryay5wcm9tb2NhdC5leHB2dGluYm94b25saW5lLmNvbS9pbmRleC5kbWEvRG1hUHJldmlldz8yMDIxNC44NjQuMTk0MTgxLjIyOTQ5ZDYxZmM5ZjMzNTc0OWZiNzZjYzljNTllMThiLjEmdXRtX3NvdXJjZT1WaXJ0dWFsX1RhcmdldCZ1dG1fbWVkaXVtPUVtYWlsJnV0bV9jb250ZW50PSZ1dG1fY2FtcGFpZ249Y2F0aG9saWN1cyZ1dG1fdGVybT0=,1,ZnJlZWJzZC5vcmc=]
Digg
[http://trk.promocat.expvtinboxonline.com/index.dma/DmaClick?20214,864,194181,13302,22949d61fc9f335749fb76cc9c59e18b,aHR0cDovL3BhaW5lbC52aXJ0dWFsdGFyZ2V0LmNvbS5ici9pbmRleC5kbWEvU29jaWFsU2hhcmU/cD0yMDIxNCw4NjQsNCZ1cmw9aHR0cDovL3Ryay5wcm9tb2NhdC5leHB2dGluYm94b25saW5lLmNvbS9pbmRleC5kbWEvRG1hUHJldmlldz8yMDIxNC44NjQuMTk0MTgxLjIyOTQ5ZDYxZmM5ZjMzNTc0OWZiNzZjYzljNTllMThiLjEmdXRtX3NvdXJjZT1WaXJ0dWFsX1RhcmdldCZ1dG1fbWVkaXVtPUVtYWlsJnV0bV9jb250ZW50PSZ1dG1fY2FtcGFpZ249Y2F0aG9saWN1cyZ1dG1fdGVybT0=,1,ZnJlZWJzZC5vcmc=]
 
[http://trk.promocat.expvtinboxonline.com/index.dma/DmaClick?20214,864,194181,13303,22949d61fc9f335749fb76cc9c59e18b,aHR0cDovL3BhaW5lbC52aXJ0dWFsdGFyZ2V0LmNvbS5ici9pbmRleC5kbWEvU29jaWFsU2hhcmU/cD0yMDIxNCw4NjQsNSZ1cmw9aHR0cDovL3Ryay5wcm9tb2NhdC5leHB2dGluYm94b25saW5lLmNvbS9pbmRleC5kbWEvRG1hUHJldmlldz8yMDIxNC44NjQuMTk0MTgxLjIyOTQ5ZDYxZmM5ZjMzNTc0OWZiNzZjYzljNTllMThiLjEmdXRtX3NvdXJjZT1WaXJ0dWFsX1RhcmdldCZ1dG1fbWVkaXVtPUVtYWlsJnV0bV9jb250ZW50PSZ1dG1fY2FtcGFpZ249Y2F0aG9saWN1cyZ1dG1fdGVybT0=,1,ZnJlZWJzZC5vcmc=]
Blogger

Palestra exclusiva com o Sr. Gentileza no CONASPAR!

2016-11-10 Thread Conaspar 
Revista Paróquiasa & Casas Religiosas

 
[http://trk.promocat.expvtinboxonline.com/index.dma/DmaClick?20214,863,194181,13277,8080a84c06a50672aec05972677c9749,aHR0cDovL2NvbmFzcGFyLmNhdGhvbGljdXMub3JnLmJyL2luc2NyaWNhby8/dXRtX3NvdXJjZT1WaXJ0dWFsX1RhcmdldCZ1dG1fbWVkaXVtPUVtYWlsJnV0bV9jb250ZW50PSZ1dG1fY2FtcGFpZ249Y29uYXNwYXImdXRtX3Rlcm09,1,ZnJlZWJzZC5vcmc=]

 
[http://trk.promocat.expvtinboxonline.com/index.dma/DmaClick?20214,863,194181,13278,8080a84c06a50672aec05972677c9749,aHR0cDovL3BhaW5lbC52aXJ0dWFsdGFyZ2V0LmNvbS5ici9pbmRleC5kbWEvU29jaWFsU2hhcmU/cD0yMDIxNCw4NjMsMSZ1cmw9aHR0cDovL3Ryay5wcm9tb2NhdC5leHB2dGluYm94b25saW5lLmNvbS9pbmRleC5kbWEvRG1hUHJldmlldz8yMDIxNC44NjMuMTk0MTgxLjgwODBhODRjMDZhNTA2NzJhZWMwNTk3MjY3N2M5NzQ5LjEmdXRtX3NvdXJjZT1WaXJ0dWFsX1RhcmdldCZ1dG1fbWVkaXVtPUVtYWlsJnV0bV9jb250ZW50PSZ1dG1fY2FtcGFpZ249Y29uYXNwYXImdXRtX3Rlcm09,1,ZnJlZWJzZC5vcmc=]
Twitter
[http://trk.promocat.expvtinboxonline.com/index.dma/DmaClick?20214,863,194181,13278,8080a84c06a50672aec05972677c9749,aHR0cDovL3BhaW5lbC52aXJ0dWFsdGFyZ2V0LmNvbS5ici9pbmRleC5kbWEvU29jaWFsU2hhcmU/cD0yMDIxNCw4NjMsMSZ1cmw9aHR0cDovL3Ryay5wcm9tb2NhdC5leHB2dGluYm94b25saW5lLmNvbS9pbmRleC5kbWEvRG1hUHJldmlldz8yMDIxNC44NjMuMTk0MTgxLjgwODBhODRjMDZhNTA2NzJhZWMwNTk3MjY3N2M5NzQ5LjEmdXRtX3NvdXJjZT1WaXJ0dWFsX1RhcmdldCZ1dG1fbWVkaXVtPUVtYWlsJnV0bV9jb250ZW50PSZ1dG1fY2FtcGFpZ249Y29uYXNwYXImdXRtX3Rlcm09,1,ZnJlZWJzZC5vcmc=]
 
[http://trk.promocat.expvtinboxonline.com/index.dma/DmaClick?20214,863,194181,13279,8080a84c06a50672aec05972677c9749,aHR0cDovL3BhaW5lbC52aXJ0dWFsdGFyZ2V0LmNvbS5ici9pbmRleC5kbWEvU29jaWFsU2hhcmU/cD0yMDIxNCw4NjMsMiZ1cmw9aHR0cDovL3Ryay5wcm9tb2NhdC5leHB2dGluYm94b25saW5lLmNvbS9pbmRleC5kbWEvRG1hUHJldmlldz8yMDIxNC44NjMuMTk0MTgxLjgwODBhODRjMDZhNTA2NzJhZWMwNTk3MjY3N2M5NzQ5LjEmdXRtX3NvdXJjZT1WaXJ0dWFsX1RhcmdldCZ1dG1fbWVkaXVtPUVtYWlsJnV0bV9jb250ZW50PSZ1dG1fY2FtcGFpZ249Y29uYXNwYXImdXRtX3Rlcm09,1,ZnJlZWJzZC5vcmc=]
Facebook
[http://trk.promocat.expvtinboxonline.com/index.dma/DmaClick?20214,863,194181,13279,8080a84c06a50672aec05972677c9749,aHR0cDovL3BhaW5lbC52aXJ0dWFsdGFyZ2V0LmNvbS5ici9pbmRleC5kbWEvU29jaWFsU2hhcmU/cD0yMDIxNCw4NjMsMiZ1cmw9aHR0cDovL3Ryay5wcm9tb2NhdC5leHB2dGluYm94b25saW5lLmNvbS9pbmRleC5kbWEvRG1hUHJldmlldz8yMDIxNC44NjMuMTk0MTgxLjgwODBhODRjMDZhNTA2NzJhZWMwNTk3MjY3N2M5NzQ5LjEmdXRtX3NvdXJjZT1WaXJ0dWFsX1RhcmdldCZ1dG1fbWVkaXVtPUVtYWlsJnV0bV9jb250ZW50PSZ1dG1fY2FtcGFpZ249Y29uYXNwYXImdXRtX3Rlcm09,1,ZnJlZWJzZC5vcmc=]
 
[http://trk.promocat.expvtinboxonline.com/index.dma/DmaClick?20214,863,194181,13280,8080a84c06a50672aec05972677c9749,aHR0cDovL3BhaW5lbC52aXJ0dWFsdGFyZ2V0LmNvbS5ici9pbmRleC5kbWEvU29jaWFsU2hhcmU/cD0yMDIxNCw4NjMsMyZ1cmw9aHR0cDovL3Ryay5wcm9tb2NhdC5leHB2dGluYm94b25saW5lLmNvbS9pbmRleC5kbWEvRG1hUHJldmlldz8yMDIxNC44NjMuMTk0MTgxLjgwODBhODRjMDZhNTA2NzJhZWMwNTk3MjY3N2M5NzQ5LjEmdXRtX3NvdXJjZT1WaXJ0dWFsX1RhcmdldCZ1dG1fbWVkaXVtPUVtYWlsJnV0bV9jb250ZW50PSZ1dG1fY2FtcGFpZ249Y29uYXNwYXImdXRtX3Rlcm09,1,ZnJlZWJzZC5vcmc=]
Delicious
[http://trk.promocat.expvtinboxonline.com/index.dma/DmaClick?20214,863,194181,13280,8080a84c06a50672aec05972677c9749,aHR0cDovL3BhaW5lbC52aXJ0dWFsdGFyZ2V0LmNvbS5ici9pbmRleC5kbWEvU29jaWFsU2hhcmU/cD0yMDIxNCw4NjMsMyZ1cmw9aHR0cDovL3Ryay5wcm9tb2NhdC5leHB2dGluYm94b25saW5lLmNvbS9pbmRleC5kbWEvRG1hUHJldmlldz8yMDIxNC44NjMuMTk0MTgxLjgwODBhODRjMDZhNTA2NzJhZWMwNTk3MjY3N2M5NzQ5LjEmdXRtX3NvdXJjZT1WaXJ0dWFsX1RhcmdldCZ1dG1fbWVkaXVtPUVtYWlsJnV0bV9jb250ZW50PSZ1dG1fY2FtcGFpZ249Y29uYXNwYXImdXRtX3Rlcm09,1,ZnJlZWJzZC5vcmc=]
 
[http://trk.promocat.expvtinboxonline.com/index.dma/DmaClick?20214,863,194181,13281,8080a84c06a50672aec05972677c9749,aHR0cDovL3BhaW5lbC52aXJ0dWFsdGFyZ2V0LmNvbS5ici9pbmRleC5kbWEvU29jaWFsU2hhcmU/cD0yMDIxNCw4NjMsNCZ1cmw9aHR0cDovL3Ryay5wcm9tb2NhdC5leHB2dGluYm94b25saW5lLmNvbS9pbmRleC5kbWEvRG1hUHJldmlldz8yMDIxNC44NjMuMTk0MTgxLjgwODBhODRjMDZhNTA2NzJhZWMwNTk3MjY3N2M5NzQ5LjEmdXRtX3NvdXJjZT1WaXJ0dWFsX1RhcmdldCZ1dG1fbWVkaXVtPUVtYWlsJnV0bV9jb250ZW50PSZ1dG1fY2FtcGFpZ249Y29uYXNwYXImdXRtX3Rlcm09,1,ZnJlZWJzZC5vcmc=]
Digg
[http://trk.promocat.expvtinboxonline.com/index.dma/DmaClick?20214,863,194181,13281,8080a84c06a50672aec05972677c9749,aHR0cDovL3BhaW5lbC52aXJ0dWFsdGFyZ2V0LmNvbS5ici9pbmRleC5kbWEvU29jaWFsU2hhcmU/cD0yMDIxNCw4NjMsNCZ1cmw9aHR0cDovL3Ryay5wcm9tb2NhdC5leHB2dGluYm94b25saW5lLmNvbS9pbmRleC5kbWEvRG1hUHJldmlldz8yMDIxNC44NjMuMTk0MTgxLjgwODBhODRjMDZhNTA2NzJhZWMwNTk3MjY3N2M5NzQ5LjEmdXRtX3NvdXJjZT1WaXJ0dWFsX1RhcmdldCZ1dG1fbWVkaXVtPUVtYWlsJnV0bV9jb250ZW50PSZ1dG1fY2FtcGFpZ249Y29uYXNwYXImdXRtX3Rlcm09,1,ZnJlZWJzZC5vcmc=]
 
[http://trk.promocat.expvtinboxonline.com/index.dma/DmaClick?20214,863,194181,13282,8080a84c06a50672aec05972677c9749,aHR0cDovL3BhaW5lbC52aXJ0dWFsdGFyZ2V0LmNvbS5ici9pbmRleC5kbWEvU29jaWFsU2hhcmU/cD0yMDIxNCw4NjMsNSZ1cmw9aHR0cDovL3Ryay5wcm9tb2NhdC5leHB2dGluYm94b25saW5lLmNvbS9pbmRleC5kbWEvRG1hUHJldmlldz8yMDIxNC44NjMuMTk0MTgxLjgwODBhODRjMDZhNTA2NzJhZWMwNTk3MjY3N2M5NzQ5LjEmdXRtX3NvdXJjZT1WaXJ0dWFsX1RhcmdldCZ1dG1fbWVkaXVtPUVtYWlsJnV0bV9jb250ZW50PSZ1dG1fY2FtcGFpZ249Y29uYXNwYXImdXRtX3Rlcm09,1,ZnJlZWJzZC5vcmc=]
Blogger

Mês de Black Friday

2016-11-10 Thread Loja Catholicus 
Loja Catholicus

 
[http://trk.promocat.expvtinboxonline.com/index.dma/DmaClick?20214,862,194181,13263,9511a83b0e21e4458aba0b9096b11e0e,aHR0cDovL2NhdGhvbGljdXMuY29tLmJyL2h0dHA6Ly9sb2phY2F0aG9saWN1cy5jb20uYnIvLz91dG1fc291cmNlPVZpcnR1YWxfVGFyZ2V0JnV0bV9tZWRpdW09RW1haWwmdXRtX2NvbnRlbnQ9JnV0bV9jYW1wYWlnbj1jYXRob2xpY3VzJnV0bV90ZXJtPQ==,1,ZnJlZWJzZC5vcmc=]

 
[http://trk.promocat.expvtinboxonline.com/index.dma/DmaClick?20214,862,194181,13264,9511a83b0e21e4458aba0b9096b11e0e,aHR0cDovL3BhaW5lbC52aXJ0dWFsdGFyZ2V0LmNvbS5ici9pbmRleC5kbWEvU29jaWFsU2hhcmU/cD0yMDIxNCw4NjIsMSZ1cmw9aHR0cDovL3Ryay5wcm9tb2NhdC5leHB2dGluYm94b25saW5lLmNvbS9pbmRleC5kbWEvRG1hUHJldmlldz8yMDIxNC44NjIuMTk0MTgxLjk1MTFhODNiMGUyMWU0NDU4YWJhMGI5MDk2YjExZTBlLjEmdXRtX3NvdXJjZT1WaXJ0dWFsX1RhcmdldCZ1dG1fbWVkaXVtPUVtYWlsJnV0bV9jb250ZW50PSZ1dG1fY2FtcGFpZ249Y2F0aG9saWN1cyZ1dG1fdGVybT0=,1,ZnJlZWJzZC5vcmc=]
Twitter
[http://trk.promocat.expvtinboxonline.com/index.dma/DmaClick?20214,862,194181,13264,9511a83b0e21e4458aba0b9096b11e0e,aHR0cDovL3BhaW5lbC52aXJ0dWFsdGFyZ2V0LmNvbS5ici9pbmRleC5kbWEvU29jaWFsU2hhcmU/cD0yMDIxNCw4NjIsMSZ1cmw9aHR0cDovL3Ryay5wcm9tb2NhdC5leHB2dGluYm94b25saW5lLmNvbS9pbmRleC5kbWEvRG1hUHJldmlldz8yMDIxNC44NjIuMTk0MTgxLjk1MTFhODNiMGUyMWU0NDU4YWJhMGI5MDk2YjExZTBlLjEmdXRtX3NvdXJjZT1WaXJ0dWFsX1RhcmdldCZ1dG1fbWVkaXVtPUVtYWlsJnV0bV9jb250ZW50PSZ1dG1fY2FtcGFpZ249Y2F0aG9saWN1cyZ1dG1fdGVybT0=,1,ZnJlZWJzZC5vcmc=]
 
[http://trk.promocat.expvtinboxonline.com/index.dma/DmaClick?20214,862,194181,13265,9511a83b0e21e4458aba0b9096b11e0e,aHR0cDovL3BhaW5lbC52aXJ0dWFsdGFyZ2V0LmNvbS5ici9pbmRleC5kbWEvU29jaWFsU2hhcmU/cD0yMDIxNCw4NjIsMiZ1cmw9aHR0cDovL3Ryay5wcm9tb2NhdC5leHB2dGluYm94b25saW5lLmNvbS9pbmRleC5kbWEvRG1hUHJldmlldz8yMDIxNC44NjIuMTk0MTgxLjk1MTFhODNiMGUyMWU0NDU4YWJhMGI5MDk2YjExZTBlLjEmdXRtX3NvdXJjZT1WaXJ0dWFsX1RhcmdldCZ1dG1fbWVkaXVtPUVtYWlsJnV0bV9jb250ZW50PSZ1dG1fY2FtcGFpZ249Y2F0aG9saWN1cyZ1dG1fdGVybT0=,1,ZnJlZWJzZC5vcmc=]
Facebook
[http://trk.promocat.expvtinboxonline.com/index.dma/DmaClick?20214,862,194181,13265,9511a83b0e21e4458aba0b9096b11e0e,aHR0cDovL3BhaW5lbC52aXJ0dWFsdGFyZ2V0LmNvbS5ici9pbmRleC5kbWEvU29jaWFsU2hhcmU/cD0yMDIxNCw4NjIsMiZ1cmw9aHR0cDovL3Ryay5wcm9tb2NhdC5leHB2dGluYm94b25saW5lLmNvbS9pbmRleC5kbWEvRG1hUHJldmlldz8yMDIxNC44NjIuMTk0MTgxLjk1MTFhODNiMGUyMWU0NDU4YWJhMGI5MDk2YjExZTBlLjEmdXRtX3NvdXJjZT1WaXJ0dWFsX1RhcmdldCZ1dG1fbWVkaXVtPUVtYWlsJnV0bV9jb250ZW50PSZ1dG1fY2FtcGFpZ249Y2F0aG9saWN1cyZ1dG1fdGVybT0=,1,ZnJlZWJzZC5vcmc=]
 
[http://trk.promocat.expvtinboxonline.com/index.dma/DmaClick?20214,862,194181,13266,9511a83b0e21e4458aba0b9096b11e0e,aHR0cDovL3BhaW5lbC52aXJ0dWFsdGFyZ2V0LmNvbS5ici9pbmRleC5kbWEvU29jaWFsU2hhcmU/cD0yMDIxNCw4NjIsMyZ1cmw9aHR0cDovL3Ryay5wcm9tb2NhdC5leHB2dGluYm94b25saW5lLmNvbS9pbmRleC5kbWEvRG1hUHJldmlldz8yMDIxNC44NjIuMTk0MTgxLjk1MTFhODNiMGUyMWU0NDU4YWJhMGI5MDk2YjExZTBlLjEmdXRtX3NvdXJjZT1WaXJ0dWFsX1RhcmdldCZ1dG1fbWVkaXVtPUVtYWlsJnV0bV9jb250ZW50PSZ1dG1fY2FtcGFpZ249Y2F0aG9saWN1cyZ1dG1fdGVybT0=,1,ZnJlZWJzZC5vcmc=]
Delicious
[http://trk.promocat.expvtinboxonline.com/index.dma/DmaClick?20214,862,194181,13266,9511a83b0e21e4458aba0b9096b11e0e,aHR0cDovL3BhaW5lbC52aXJ0dWFsdGFyZ2V0LmNvbS5ici9pbmRleC5kbWEvU29jaWFsU2hhcmU/cD0yMDIxNCw4NjIsMyZ1cmw9aHR0cDovL3Ryay5wcm9tb2NhdC5leHB2dGluYm94b25saW5lLmNvbS9pbmRleC5kbWEvRG1hUHJldmlldz8yMDIxNC44NjIuMTk0MTgxLjk1MTFhODNiMGUyMWU0NDU4YWJhMGI5MDk2YjExZTBlLjEmdXRtX3NvdXJjZT1WaXJ0dWFsX1RhcmdldCZ1dG1fbWVkaXVtPUVtYWlsJnV0bV9jb250ZW50PSZ1dG1fY2FtcGFpZ249Y2F0aG9saWN1cyZ1dG1fdGVybT0=,1,ZnJlZWJzZC5vcmc=]
 
[http://trk.promocat.expvtinboxonline.com/index.dma/DmaClick?20214,862,194181,13267,9511a83b0e21e4458aba0b9096b11e0e,aHR0cDovL3BhaW5lbC52aXJ0dWFsdGFyZ2V0LmNvbS5ici9pbmRleC5kbWEvU29jaWFsU2hhcmU/cD0yMDIxNCw4NjIsNCZ1cmw9aHR0cDovL3Ryay5wcm9tb2NhdC5leHB2dGluYm94b25saW5lLmNvbS9pbmRleC5kbWEvRG1hUHJldmlldz8yMDIxNC44NjIuMTk0MTgxLjk1MTFhODNiMGUyMWU0NDU4YWJhMGI5MDk2YjExZTBlLjEmdXRtX3NvdXJjZT1WaXJ0dWFsX1RhcmdldCZ1dG1fbWVkaXVtPUVtYWlsJnV0bV9jb250ZW50PSZ1dG1fY2FtcGFpZ249Y2F0aG9saWN1cyZ1dG1fdGVybT0=,1,ZnJlZWJzZC5vcmc=]
Digg
[http://trk.promocat.expvtinboxonline.com/index.dma/DmaClick?20214,862,194181,13267,9511a83b0e21e4458aba0b9096b11e0e,aHR0cDovL3BhaW5lbC52aXJ0dWFsdGFyZ2V0LmNvbS5ici9pbmRleC5kbWEvU29jaWFsU2hhcmU/cD0yMDIxNCw4NjIsNCZ1cmw9aHR0cDovL3Ryay5wcm9tb2NhdC5leHB2dGluYm94b25saW5lLmNvbS9pbmRleC5kbWEvRG1hUHJldmlldz8yMDIxNC44NjIuMTk0MTgxLjk1MTFhODNiMGUyMWU0NDU4YWJhMGI5MDk2YjExZTBlLjEmdXRtX3NvdXJjZT1WaXJ0dWFsX1RhcmdldCZ1dG1fbWVkaXVtPUVtYWlsJnV0bV9jb250ZW50PSZ1dG1fY2FtcGFpZ249Y2F0aG9saWN1cyZ1dG1fdGVybT0=,1,ZnJlZWJzZC5vcmc=]
 
[http://trk.promocat.expvtinboxonline.com/index.dma/DmaClick?20214,862,194181,13268,9511a83b0e21e4458aba0b9096b11e0e,aHR0cDovL3BhaW5lbC52aXJ0dWFsdGFyZ2V0LmNvbS5ici9pbmRleC5kbWEvU29jaWFsU2hhcmU/cD0yMDIxNCw4NjIsNSZ1cmw9aHR0cDovL3Ryay5wcm9tb2NhdC5leHB2dGluYm94b25saW5lLmNvbS9pbmRleC5kbWEvRG1hUHJldmlldz8yMDIxNC44NjIuMTk0MTgxLjk1MTFhODNiMGUyMWU0NDU4YWJhMGI5MDk2YjExZTBlLjEmdXRtX3NvdXJjZT1WaXJ0dWFsX1RhcmdldCZ1dG1fbWVkaXVtPUVtYWlsJnV0bV9jb250ZW50PSZ1dG1fY2FtcGFpZ249Y2F0aG9saWN1cyZ1dG1fdGVybT0=,1,ZnJlZWJzZC5vcmc=]
Blogger

Re: Freebsd 11.0 RELEASE - ZFS deadlock

2016-11-10 Thread Henri Hennebert 

On 11/10/2016 11:21, Andriy Gapon wrote:

On 09/11/2016 15:58, Eric van Gyzen wrote:

On 11/09/2016 07:48, Henri Hennebert wrote:

I encounter a strange deadlock on

FreeBSD avoriaz.restart.bel 11.0-RELEASE-p3 FreeBSD 11.0-RELEASE-p3 #0 r308260:
Fri Nov  4 02:51:33 CET 2016
r...@avoriaz.restart.bel:/usr/obj/usr/src/sys/AVORIAZ  amd64

This system is exclusively running on zfs.

After 3 or 4 days, `periodic daily` is locked in the directory 
/usr/local/news/bin

[root@avoriaz ~]# ps xa|grep find
85656  -  D0:01.13 find / ( ! -fstype local -o -fstype rdonly ) -prune
-o ( -name [#,]* -o -name .#* -o -name a.out -o -nam
  462  1  S+   0:00.00 grep find
[root@avoriaz ~]# procstat -f 85656
  PID COMMFD T V FLAGSREF  OFFSET PRO NAME
85656 find  text v r r---   -   - - /usr/bin/find
85656 find   cwd v d r---   -   - - /usr/local/news/bin
85656 find  root v d r---   -   - - /
85656 find 0 v c r---   3   0 - /dev/null
85656 find 1 p - rw--   1   0 - -
85656 find 2 v r -w--   7  17 - -
85656 find 3 v d r---   1   0 - /home/root
85656 find 4 v d r---   1   0 - /home/root
85656 find 5 v d rn--   1 533545184 - /usr/local/news/bin
[root@avoriaz ~]#

If I try `ls /usr/local/news/bin` it is also locked.

After `shutdown -r now` the system remain locked after the line '0 0 0 0 0 0'

After a reset and reboot  I can access /usr/local/news/bin.

I delete this directory and reinstall the package `portupgrade -fu news/inn`

5 days later `periodic daily`is locked on the same directory :-o

Any idea?


I can't help with the deadlock, but someone who _can_ help will probably ask for
the output of "procstat -kk PID" with the PID of the "find" process.


In fact, it's procstat -kk -a.  With just one thread we would see that a thread
is blocked on something, but we won't see why that something can not be 
acquired.



I attach the result,

Henri
[root@avoriaz ~]# procstat -kk -a
  PIDTID COMM TDNAME   KSTACK   
0 10 kernel   swapper  mi_switch+0xd2 
sleepq_timedwait+0x3a _sleep+0x281 swapper+0x464 btext+0x2c 
0 19 kernel   kqueue_ctx taskq mi_switch+0xd2 sleepq_wait+0x3a 
_sleep+0x2a1 taskqueue_thread_loop+0x141 fork_exit+0x85 fork_trampoline+0xe 
0 100012 kernel   aiod_kick taskq  mi_switch+0xd2 sleepq_wait+0x3a 
_sleep+0x2a1 taskqueue_thread_loop+0x141 fork_exit+0x85 fork_trampoline+0xe 
0 100013 kernel   thread taskq mi_switch+0xd2 sleepq_wait+0x3a 
_sleep+0x2a1 taskqueue_thread_loop+0x141 fork_exit+0x85 fork_trampoline+0xe 
0 100018 kernel   firmware taskq   mi_switch+0xd2 sleepq_wait+0x3a 
_sleep+0x2a1 taskqueue_thread_loop+0x141 fork_exit+0x85 fork_trampoline+0xe 
0 100022 kernel   acpi_task_0  mi_switch+0xd2 sleepq_wait+0x3a 
msleep_spin_sbt+0x1bd taskqueue_thread_loop+0x113 fork_exit+0x85 
fork_trampoline+0xe 
0 100023 kernel   acpi_task_1  mi_switch+0xd2 sleepq_wait+0x3a 
msleep_spin_sbt+0x1bd taskqueue_thread_loop+0x113 fork_exit+0x85 
fork_trampoline+0xe 
0 100024 kernel   acpi_task_2  mi_switch+0xd2 sleepq_wait+0x3a 
msleep_spin_sbt+0x1bd taskqueue_thread_loop+0x113 fork_exit+0x85 
fork_trampoline+0xe 
0 100025 kernel   em0 que  mi_switch+0xd2 sleepq_wait+0x3a 
msleep_spin_sbt+0x1bd taskqueue_thread_loop+0x113 fork_exit+0x85 
fork_trampoline+0xe 
0 100026 kernel   em0 txq  mi_switch+0xd2 sleepq_wait+0x3a 
msleep_spin_sbt+0x1bd taskqueue_thread_loop+0x113 fork_exit+0x85 
fork_trampoline+0xe 
0 100027 kernel   em1 taskqmi_switch+0xd2 sleepq_wait+0x3a 
msleep_spin_sbt+0x1bd taskqueue_thread_loop+0x113 fork_exit+0x85 
fork_trampoline+0xe 
0 100060 kernel   mca taskqmi_switch+0xd2 sleepq_wait+0x3a 
msleep_spin_sbt+0x1bd taskqueue_thread_loop+0x113 fork_exit+0x85 
fork_trampoline+0xe 
0 100061 kernel   system_taskq_0   mi_switch+0xd2 sleepq_wait+0x3a 
_sleep+0x2a1 taskqueue_thread_loop+0x141 fork_exit+0x85 fork_trampoline+0xe 
0 100062 kernel   system_taskq_1   mi_switch+0xd2 sleepq_wait+0x3a 
_sleep+0x2a1 taskqueue_thread_loop+0x141 fork_exit+0x85 fork_trampoline+0xe 
0 100063 kernel   dbu_evictmi_switch+0xd2 sleepq_wait+0x3a 
_sleep+0x2a1 taskqueue_thread_loop+0x141 fork_exit+0x85 fork_trampoline+0xe 
0 100072 kernel   CAM taskqmi_switch+0xd2 sleepq_wait+0x3a 
_sleep+0x2a1 taskqueue_thread_loop+0x141 fork_exit+0x85 fork_trampoline+0xe 
0 100086 kernel   if_config_tqg_0  mi_switch+0xd2 sleepq_wait+0x3a 
msleep_spin_sbt+0x1bd gtaskqueue_thread_loop+0x113 fork_exit+0x85 
fork_trampoline+0xe 
0 100087 kernel   if_io_tqg_0  mi_switch+0xd2 sleepq_wait+0x3a

Re: Freebsd 11.0 RELEASE - ZFS deadlock

2016-11-10 Thread Andriy Gapon 
On 09/11/2016 15:58, Eric van Gyzen wrote:
> On 11/09/2016 07:48, Henri Hennebert wrote:
>> I encounter a strange deadlock on
>>
>> FreeBSD avoriaz.restart.bel 11.0-RELEASE-p3 FreeBSD 11.0-RELEASE-p3 #0 
>> r308260:
>> Fri Nov  4 02:51:33 CET 2016
>> r...@avoriaz.restart.bel:/usr/obj/usr/src/sys/AVORIAZ  amd64
>>
>> This system is exclusively running on zfs.
>>
>> After 3 or 4 days, `periodic daily` is locked in the directory 
>> /usr/local/news/bin
>>
>> [root@avoriaz ~]# ps xa|grep find
>> 85656  -  D0:01.13 find / ( ! -fstype local -o -fstype rdonly ) 
>> -prune
>> -o ( -name [#,]* -o -name .#* -o -name a.out -o -nam
>>   462  1  S+   0:00.00 grep find
>> [root@avoriaz ~]# procstat -f 85656
>>   PID COMMFD T V FLAGSREF  OFFSET PRO NAME
>> 85656 find  text v r r---   -   - - /usr/bin/find
>> 85656 find   cwd v d r---   -   - - /usr/local/news/bin
>> 85656 find  root v d r---   -   - - /
>> 85656 find 0 v c r---   3   0 - /dev/null
>> 85656 find 1 p - rw--   1   0 - -
>> 85656 find 2 v r -w--   7  17 - -
>> 85656 find 3 v d r---   1   0 - /home/root
>> 85656 find 4 v d r---   1   0 - /home/root
>> 85656 find 5 v d rn--   1 533545184 - /usr/local/news/bin
>> [root@avoriaz ~]#
>>
>> If I try `ls /usr/local/news/bin` it is also locked.
>>
>> After `shutdown -r now` the system remain locked after the line '0 0 0 0 0 0'
>>
>> After a reset and reboot  I can access /usr/local/news/bin.
>>
>> I delete this directory and reinstall the package `portupgrade -fu news/inn`
>>
>> 5 days later `periodic daily`is locked on the same directory :-o
>>
>> Any idea?
> 
> I can't help with the deadlock, but someone who _can_ help will probably ask 
> for
> the output of "procstat -kk PID" with the PID of the "find" process.

In fact, it's procstat -kk -a.  With just one thread we would see that a thread
is blocked on something, but we won't see why that something can not be 
acquired.


-- 
Andriy Gapon
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: Freebsd 11.0 RELEASE - ZFS deadlock

2016-11-10 Thread Henri Hennebert 

On 11/09/2016 19:23, Thierry Thomas wrote:

Le mer.  9 nov. 16 à 15:03:49 +0100, Henri Hennebert 
 écrivait :


[root@avoriaz ~]# procstat -kk 85656
   PIDTID COMM TDNAME KSTACK
85656 101112 find -mi_switch+0xd2
sleepq_wait+0x3a sleeplk+0x1b4 __lockmgr_args+0x356 vop_stdlock+0x3c
VOP_LOCK1_APV+0x8d _vn_lock+0x43 vget+0x47 cache_lookup+0x679
vfs_cache_lookup+0xac VOP_LOOKUP_APV+0x87 lookup+0x591 namei+0x572
kern_statat+0xa8 sys_fstatat+0x2c amd64_syscall+0x4ce Xfast_syscall+0xfb


It looks similar to the problem reportes in PR 205163
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=205163

May be causes by too small values for some vfs.zfs.arc*.
Could you please list sysctl for vfs.zfs.arc_max and others?

Regards,


[root@avoriaz ~]# sysctl vfs.zfs
vfs.zfs.trim.max_interval: 1
vfs.zfs.trim.timeout: 30
vfs.zfs.trim.txg_delay: 32
vfs.zfs.trim.enabled: 1
vfs.zfs.vol.unmap_enabled: 1
vfs.zfs.vol.recursive: 0
vfs.zfs.vol.mode: 1
vfs.zfs.version.zpl: 5
vfs.zfs.version.spa: 5000
vfs.zfs.version.acl: 1
vfs.zfs.version.ioctl: 6
vfs.zfs.debug: 0
vfs.zfs.super_owner: 0
vfs.zfs.sync_pass_rewrite: 2
vfs.zfs.sync_pass_dont_compress: 5
vfs.zfs.sync_pass_deferred_free: 2
vfs.zfs.zio.exclude_metadata: 0
vfs.zfs.zio.use_uma: 1
vfs.zfs.cache_flush_disable: 0
vfs.zfs.zil_replay_disable: 0
vfs.zfs.min_auto_ashift: 9
vfs.zfs.max_auto_ashift: 13
vfs.zfs.vdev.trim_max_pending: 1
vfs.zfs.vdev.bio_delete_disable: 0
vfs.zfs.vdev.bio_flush_disable: 0
vfs.zfs.vdev.write_gap_limit: 4096
vfs.zfs.vdev.read_gap_limit: 32768
vfs.zfs.vdev.aggregation_limit: 131072
vfs.zfs.vdev.trim_max_active: 64
vfs.zfs.vdev.trim_min_active: 1
vfs.zfs.vdev.scrub_max_active: 2
vfs.zfs.vdev.scrub_min_active: 1
vfs.zfs.vdev.async_write_max_active: 10
vfs.zfs.vdev.async_write_min_active: 1
vfs.zfs.vdev.async_read_max_active: 3
vfs.zfs.vdev.async_read_min_active: 1
vfs.zfs.vdev.sync_write_max_active: 10
vfs.zfs.vdev.sync_write_min_active: 10
vfs.zfs.vdev.sync_read_max_active: 10
vfs.zfs.vdev.sync_read_min_active: 10
vfs.zfs.vdev.max_active: 1000
vfs.zfs.vdev.async_write_active_max_dirty_percent: 60
vfs.zfs.vdev.async_write_active_min_dirty_percent: 30
vfs.zfs.vdev.mirror.non_rotating_seek_inc: 1
vfs.zfs.vdev.mirror.non_rotating_inc: 0
vfs.zfs.vdev.mirror.rotating_seek_offset: 1048576
vfs.zfs.vdev.mirror.rotating_seek_inc: 5
vfs.zfs.vdev.mirror.rotating_inc: 0
vfs.zfs.vdev.trim_on_init: 1
vfs.zfs.vdev.cache.bshift: 16
vfs.zfs.vdev.cache.size: 0
vfs.zfs.vdev.cache.max: 16384
vfs.zfs.vdev.metaslabs_per_vdev: 200
vfs.zfs.txg.timeout: 5
vfs.zfs.space_map_blksz: 4096
vfs.zfs.spa_slop_shift: 5
vfs.zfs.spa_asize_inflation: 24
vfs.zfs.deadman_enabled: 1
vfs.zfs.deadman_checktime_ms: 5000
vfs.zfs.deadman_synctime_ms: 100
vfs.zfs.debug_flags: 0
vfs.zfs.recover: 0
vfs.zfs.spa_load_verify_data: 1
vfs.zfs.spa_load_verify_metadata: 1
vfs.zfs.spa_load_verify_maxinflight: 1
vfs.zfs.ccw_retry_interval: 300
vfs.zfs.check_hostid: 1
vfs.zfs.mg_fragmentation_threshold: 85
vfs.zfs.mg_noalloc_threshold: 0
vfs.zfs.condense_pct: 200
vfs.zfs.metaslab.bias_enabled: 1
vfs.zfs.metaslab.lba_weighting_enabled: 1
vfs.zfs.metaslab.fragmentation_factor_enabled: 1
vfs.zfs.metaslab.preload_enabled: 1
vfs.zfs.metaslab.preload_limit: 3
vfs.zfs.metaslab.unload_delay: 8
vfs.zfs.metaslab.load_pct: 50
vfs.zfs.metaslab.min_alloc_size: 33554432
vfs.zfs.metaslab.df_free_pct: 4
vfs.zfs.metaslab.df_alloc_threshold: 131072
vfs.zfs.metaslab.debug_unload: 0
vfs.zfs.metaslab.debug_load: 0
vfs.zfs.metaslab.fragmentation_threshold: 70
vfs.zfs.metaslab.gang_bang: 16777217
vfs.zfs.free_bpobj_enabled: 1
vfs.zfs.free_max_blocks: 18446744073709551615
vfs.zfs.no_scrub_prefetch: 0
vfs.zfs.no_scrub_io: 0
vfs.zfs.resilver_min_time_ms: 3000
vfs.zfs.free_min_time_ms: 1000
vfs.zfs.scan_min_time_ms: 1000
vfs.zfs.scan_idle: 50
vfs.zfs.scrub_delay: 4
vfs.zfs.resilver_delay: 2
vfs.zfs.top_maxinflight: 32
vfs.zfs.zfetch.array_rd_sz: 1048576
vfs.zfs.zfetch.max_distance: 8388608
vfs.zfs.zfetch.min_sec_reap: 2
vfs.zfs.zfetch.max_streams: 8
vfs.zfs.prefetch_disable: 1
vfs.zfs.delay_scale: 50
vfs.zfs.delay_min_dirty_percent: 60
vfs.zfs.dirty_data_sync: 67108864
vfs.zfs.dirty_data_max_percent: 10
vfs.zfs.dirty_data_max_max: 4294967296
vfs.zfs.dirty_data_max: 373664153
vfs.zfs.max_recordsize: 1048576
vfs.zfs.mdcomp_disable: 0
vfs.zfs.nopwrite_enabled: 1
vfs.zfs.dedup.prefetch: 1
vfs.zfs.l2c_only_size: 0
vfs.zfs.mfu_ghost_data_lsize: 24202240
vfs.zfs.mfu_ghost_metadata_lsize: 136404992
vfs.zfs.mfu_ghost_size: 160607232
vfs.zfs.mfu_data_lsize: 449569280
vfs.zfs.mfu_metadata_lsize: 102724608
vfs.zfs.mfu_size: 714202624
vfs.zfs.mru_ghost_data_lsize: 874834432
vfs.zfs.mru_ghost_metadata_lsize: 387692032
vfs.zfs.mru_ghost_size: 1262526464
vfs.zfs.mru_data_lsize: 151275008
vfs.zfs.mru_metadata_lsize: 13547008
vfs.zfs.mru_size: 322614272
vfs.zfs.anon_data_lsize: 0
vfs.zfs.anon_metadata_lsize: 0
vfs.zfs.anon_size: 2916352
vfs.zfs.l2arc_norw: 1