Re: unionfs bugs, a partial patch and some comments [Was: Re: 1-BETA3 Panic: __lockmgr_args: downgrade a recursed lockmgr nfs @ /usr/local/share/deploy-tools/RELENG_11/src/sys/fs/unionfs/union_vnops.c

2017-03-10 Thread Konstantin Belousov 
On Thu, Mar 09, 2017 at 10:49:01PM +, Rick Macklem wrote:
> Konstantin Belousov wrote:
> > I did not touched unionfs, and have no plans to.  It is equally broken in
> > all relevant versions of FreeBSD.
> Heh, heh. I chuckled when I read this. I think he's trying to say
> "it probably won't ever be fixed". My understanding is that it would
> require a major redesign of the FreeBSD VFS to make it fully stackable
> to fix unionfs and that isn't happening anytime soon...
>

I mean that there is nobody who is both interested in unionfs and capable
of fixing it, and it was so at least from the 8.x times.

Also, I do not think that VFS needs a redesign to support unionfs-like
operations.  The problems that current unionfs has are its local problems.
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: unionfs bugs, a partial patch and some comments [Was: Re: 1-BETA3 Panic: __lockmgr_args: downgrade a recursed lockmgr nfs @ /usr/local/share/deploy-tools/RELENG_11/src/sys/fs/unionfs/union_vnops.c

2017-03-09 Thread Rick Macklem 
Konstantin Belousov wrote:
> I did not touched unionfs, and have no plans to.  It is equally broken in
> all relevant versions of FreeBSD.
Heh, heh. I chuckled when I read this. I think he's trying to say "it probably
won't ever be fixed". My understanding is that it would require a major redesign
of the FreeBSD VFS to make it fully stackable to fix unionfs and that isn't 
happening
anytime soon...

Harry Schmalzbauer wrote:
> In 2016 I did some tests with 11.0-Beta1, where this thread origins, and
> Rick kindly looked into it and provided the following patch:
> https://lists.freebsd.org/pipermail/freebsd-stable/attachments/20160818/d1d1691d/attachment.obj
> (Explanation:
> https://lists.freebsd.org/pipermail/freebsd-stable/2016-August/085294.html)
Yep, I am guilty of creating this patch. All it tried to do was fix the crash.
I don't know why a debug kernel with the patch crashes, but I might try and
reproduce that.

> Picture has changed, the machine daedlocked over night. So it does have
> a significant influence, but unfortunately isn't the real solution.
So, do you mean there is no longer any unionfs mount on the machine and it
still hangs?
If yes, then this should be looked at.

rick
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: unionfs bugs, a partial patch and some comments [Was: Re: 1-BETA3 Panic: __lockmgr_args: downgrade a recursed lockmgr nfs @ /usr/local/share/deploy-tools/RELENG_11/src/sys/fs/unionfs/union_vnops.c

2017-03-08 Thread Harry Schmalzbauer 
Bezüglich Konstantin Belousov's Nachricht vom 08.03.2017 00:55 (localtime):
> On Tue, Mar 07, 2017 at 10:49:01PM +, Rick Macklem wrote:
>> Hmm, this is going to sound dumb, but I don't recall generating any
>> unionfs patch;-)
>> I'll go look for it. Maybe it was Kostik's?
> I did not touched unionfs, and have no plans to.  It is equally broken in
> all relevant versions of FreeBSD.

ACK.

While this is no good news, I have more bad news: deadlock came back…

I'd like to summarize in case anybody else is interested in uninionfs,
maybe at any time in the future:

I observed locking problems back in 2012 and Attilio Rao's final attempt
was this: https://people.freebsd.org/~attilio/unionfs_nodeget4.patch
I never used it, most likely because it didn't work even back with
RELENG_9. It applies to stable/11, but has no effect besides panicing
KDB kernels.
What I used up to 10.3 was the following simple patch:
--- src/sys/fs/unionfs/union_subr.c (revision 231702)
+++ src/sys/fs/unionfs/union_subr.c (working copy)
@@ -261,7 +261,9 @@ unionfs_nodeget(struct mount *mp, struct vnode *up
free(unp, M_UNIONFSNODE);
return (error);
}
+   vn_lock(vp, LK_EXCLUSIVE | LK_RETRY);
error = insmntque(vp, mp);  /* XXX: Too early for mpsafe fs */
+   VOP_UNLOCK(vp, 0);
if (error != 0) {
free(unp, M_UNIONFSNODE);
return (error);

This hasn't lead to any panic or deadlock during the last 5 years on ~50
machines, up to 10.3.

In 2016 I did some tests with 11.0-Beta1, where this thread origins, and
Rick kindly looked into it and provided the following patch:
https://lists.freebsd.org/pipermail/freebsd-stable/attachments/20160818/d1d1691d/attachment.obj
(Explanation:
https://lists.freebsd.org/pipermail/freebsd-stable/2016-August/085294.html)

This also panics KDB-kernel (and works without KDB) but at least does
have influence on the dedalock, in case symlinks are involved, where
deadlocks are significantly postponed.

…

 db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
 0xfe00982220e0
 vpanic() at vpanic+0x186/frame 0xfe0098222160
 kassert_panic() at kassert_panic+0x126/frame 0xfe00982221d0
 witness_assert() at witness_assert+0x35a/frame 0xfe009830
 __lockmgr_args() at __lockmgr_args+0x517/frame 0xfe0098d0
 vop_stdunlock() at vop_stdunlock+0x3b/frame 0xfe0098f0
 VOP_UNLOCK_APV() at VOP_UNLOCK_APV+0xe0/frame 0xfe0098222320
 unionfs_unlock() at unionfs_unlock+0x112/frame 0xfe0098222390
 VOP_UNLOCK_APV() at VOP_UNLOCK_APV+0xe0/frame 0xfe00982223c0
 unionfs_nodeget() at unionfs_nodeget+0x3ef/frame 0xfe0098222470
 unionfs_domount() at unionfs_domount+0x518/frame 0xfe00982226b0
 vfs_donmount() at vfs_donmount+0xe37/frame 0xfe00982228f0
 sys_nmount() at sys_nmount+0x72/frame 0xfe0098222930
 amd64_syscall() at amd64_syscall+0x2f9/frame 0xfe0098222ab0
 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfe0098222ab0
 --- syscall (378, FreeBSD ELF64, sys_nmount), rip = 0x80086ecea, rsp =
 0x7fffe318, rbp = 0x7fffeca0 ---
>>> New discovery:
>>> Rick's latest patch casues panic only with KDB. If I compile a kernel
>>> without witenss and KDB, the machine boots fine!
>>> Also, it's at least not so easy anymore to trigger the deadlock :-) . I
>>> need to do more testing but until now Rick's approach seems very
>>> promising :-) .
>>
>> My unionfs deadlock problem isn't really solved with Rick's latest
>> patch, I still can reproduce it: krb5.conf and krb5.keytab are files on
>> unionfs referenced by /etc.  libexec/negotiate_kerberos_auth reads these
>> and if I have enough helper processes handling requests, the deadlock
>> occurs.
>>
>> _But_: If I move the files outside the unionfs and create a symlink, I
>> cannot reproduce the deadlock anymore, which was similar easily
>> reproducable without it or any of the other workarounds.

Picture has changed, the machine daedlocked over night. So it does have
a significant influence, but unfortunately isn't the real solution.

Thanks for any help,

-harry
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: unionfs bugs, a partial patch and some comments [Was: Re: 1-BETA3 Panic: __lockmgr_args: downgrade a recursed lockmgr nfs @ /usr/local/share/deploy-tools/RELENG_11/src/sys/fs/unionfs/union_vnops.c

2017-03-07 Thread Konstantin Belousov 
On Tue, Mar 07, 2017 at 10:49:01PM +, Rick Macklem wrote:
> Hmm, this is going to sound dumb, but I don't recall generating any
> unionfs patch;-)
> I'll go look for it. Maybe it was Kostik's?
I did not touched unionfs, and have no plans to.  It is equally broken in
all relevant versions of FreeBSD.

> 
> rick
> 
> From: Harry Schmalzbauer 
> Sent: Tuesday, March 7, 2017 2:45:40 PM
> To: Rick Macklem
> Cc: Konstantin Belousov; FreeBSD Stable; Mark Johnston; k...@freebsd.org
> Subject: Re: unionfs bugs, a partial patch and some comments [Was: Re: 
> 1-BETA3 Panic: __lockmgr_args: downgrade a recursed lockmgr nfs @ 
> /usr/local/share/deploy-tools/RELENG_11/src/sys/fs/unionfs/union_vnops.c:1905]
> 
>  Bez?glich Harry Schmalzbauer's Nachricht vom 07.03.2017 19:44 (localtime):
> >  Bez?glich Harry Schmalzbauer's Nachricht vom 07.03.2017 13:42 (localtime):
> > ?
> >> Something ufs related seems to have tightened the unionfs locking
> >> problem in stable/11.  Now the machine instantaniously panics during
> >> boot after mounting root with Rick's latest patch.
> >>
> >> Unfortunately I don't have SWAP available on that machine (yet), but
> >> maybe shit is a hint for anybody.
> >>
> >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
> >> 0xfe00982220e0
> >> vpanic() at vpanic+0x186/frame 0xfe0098222160
> >> kassert_panic() at kassert_panic+0x126/frame 0xfe00982221d0
> >> witness_assert() at witness_assert+0x35a/frame 0xfe009830
> >> __lockmgr_args() at __lockmgr_args+0x517/frame 0xfe0098d0
> >> vop_stdunlock() at vop_stdunlock+0x3b/frame 0xfe0098f0
> >> VOP_UNLOCK_APV() at VOP_UNLOCK_APV+0xe0/frame 0xfe0098222320
> >> unionfs_unlock() at unionfs_unlock+0x112/frame 0xfe0098222390
> >> VOP_UNLOCK_APV() at VOP_UNLOCK_APV+0xe0/frame 0xfe00982223c0
> >> unionfs_nodeget() at unionfs_nodeget+0x3ef/frame 0xfe0098222470
> >> unionfs_domount() at unionfs_domount+0x518/frame 0xfe00982226b0
> >> vfs_donmount() at vfs_donmount+0xe37/frame 0xfe00982228f0
> >> sys_nmount() at sys_nmount+0x72/frame 0xfe0098222930
> >> amd64_syscall() at amd64_syscall+0x2f9/frame 0xfe0098222ab0
> >> Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfe0098222ab0
> >> --- syscall (378, FreeBSD ELF64, sys_nmount), rip = 0x80086ecea, rsp =
> >> 0x7fffe318, rbp = 0x7fffeca0 ---
> > New discovery:
> > Rick's latest patch casues panic only with KDB. If I compile a kernel
> > without witenss and KDB, the machine boots fine!
> > Also, it's at least not so easy anymore to trigger the deadlock :-) . I
> > need to do more testing but until now Rick's approach seems very
> > promising :-) .
> 
> My unionfs deadlock problem isn't really solved with Rick's latest
> patch, I still can reproduce it: krb5.conf and krb5.keytab are files on
> unionfs referenced by /etc.  libexec/negotiate_kerberos_auth reads these
> and if I have enough helper processes handling requests, the deadlock
> occurs.
> 
> _But_: If I move the files outside the unionfs and create a symlink, I
> cannot reproduce the deadlock anymore, which was similar easily
> reproducable without it or any of the other workarounds.
> So it looks like I have an acceptable solution for now, although it's
> only usable under certain conditions.
> 
> Unfortunately I can't do tests with a debug kernel since the patch
> prevents the system with the debug kernel from starting up.
> But if this was ironed out, I'd happily provide more info.
> 
> 
> Thanks,
> 
> -Harry
> 
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: unionfs bugs, a partial patch and some comments [Was: Re: 1-BETA3 Panic: __lockmgr_args: downgrade a recursed lockmgr nfs @ /usr/local/share/deploy-tools/RELENG_11/src/sys/fs/unionfs/union_vnops.c

2017-03-07 Thread Rick Macklem 
Hmm, this is going to sound dumb, but I don't recall generating any
unionfs patch;-)
I'll go look for it. Maybe it was Kostik's?

rick

From: Harry Schmalzbauer 
Sent: Tuesday, March 7, 2017 2:45:40 PM
To: Rick Macklem
Cc: Konstantin Belousov; FreeBSD Stable; Mark Johnston; k...@freebsd.org
Subject: Re: unionfs bugs, a partial patch and some comments [Was: Re: 1-BETA3 
Panic: __lockmgr_args: downgrade a recursed lockmgr nfs @ 
/usr/local/share/deploy-tools/RELENG_11/src/sys/fs/unionfs/union_vnops.c:1905]

 Bezüglich Harry Schmalzbauer's Nachricht vom 07.03.2017 19:44 (localtime):
>  Bezüglich Harry Schmalzbauer's Nachricht vom 07.03.2017 13:42 (localtime):
> …
>> Something ufs related seems to have tightened the unionfs locking
>> problem in stable/11.  Now the machine instantaniously panics during
>> boot after mounting root with Rick's latest patch.
>>
>> Unfortunately I don't have SWAP available on that machine (yet), but
>> maybe shit is a hint for anybody.
>>
>> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
>> 0xfe00982220e0
>> vpanic() at vpanic+0x186/frame 0xfe0098222160
>> kassert_panic() at kassert_panic+0x126/frame 0xfe00982221d0
>> witness_assert() at witness_assert+0x35a/frame 0xfe009830
>> __lockmgr_args() at __lockmgr_args+0x517/frame 0xfe0098d0
>> vop_stdunlock() at vop_stdunlock+0x3b/frame 0xfe0098f0
>> VOP_UNLOCK_APV() at VOP_UNLOCK_APV+0xe0/frame 0xfe0098222320
>> unionfs_unlock() at unionfs_unlock+0x112/frame 0xfe0098222390
>> VOP_UNLOCK_APV() at VOP_UNLOCK_APV+0xe0/frame 0xfe00982223c0
>> unionfs_nodeget() at unionfs_nodeget+0x3ef/frame 0xfe0098222470
>> unionfs_domount() at unionfs_domount+0x518/frame 0xfe00982226b0
>> vfs_donmount() at vfs_donmount+0xe37/frame 0xfe00982228f0
>> sys_nmount() at sys_nmount+0x72/frame 0xfe0098222930
>> amd64_syscall() at amd64_syscall+0x2f9/frame 0xfe0098222ab0
>> Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfe0098222ab0
>> --- syscall (378, FreeBSD ELF64, sys_nmount), rip = 0x80086ecea, rsp =
>> 0x7fffe318, rbp = 0x7fffeca0 ---
> New discovery:
> Rick's latest patch casues panic only with KDB. If I compile a kernel
> without witenss and KDB, the machine boots fine!
> Also, it's at least not so easy anymore to trigger the deadlock :-) . I
> need to do more testing but until now Rick's approach seems very
> promising :-) .

My unionfs deadlock problem isn't really solved with Rick's latest
patch, I still can reproduce it: krb5.conf and krb5.keytab are files on
unionfs referenced by /etc.  libexec/negotiate_kerberos_auth reads these
and if I have enough helper processes handling requests, the deadlock
occurs.

_But_: If I move the files outside the unionfs and create a symlink, I
cannot reproduce the deadlock anymore, which was similar easily
reproducable without it or any of the other workarounds.
So it looks like I have an acceptable solution for now, although it's
only usable under certain conditions.

Unfortunately I can't do tests with a debug kernel since the patch
prevents the system with the debug kernel from starting up.
But if this was ironed out, I'd happily provide more info.


Thanks,

-Harry

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: unionfs bugs, a partial patch and some comments [Was: Re: 1-BETA3 Panic: __lockmgr_args: downgrade a recursed lockmgr nfs @ /usr/local/share/deploy-tools/RELENG_11/src/sys/fs/unionfs/union_vnops.c

2017-03-07 Thread Harry Schmalzbauer 
 Bezüglich Harry Schmalzbauer's Nachricht vom 07.03.2017 19:44 (localtime):
>  Bezüglich Harry Schmalzbauer's Nachricht vom 07.03.2017 13:42 (localtime):
> …
>> Something ufs related seems to have tightened the unionfs locking
>> problem in stable/11.  Now the machine instantaniously panics during
>> boot after mounting root with Rick's latest patch.
>>
>> Unfortunately I don't have SWAP available on that machine (yet), but
>> maybe shit is a hint for anybody.
>>
>> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
>> 0xfe00982220e0
>> vpanic() at vpanic+0x186/frame 0xfe0098222160
>> kassert_panic() at kassert_panic+0x126/frame 0xfe00982221d0
>> witness_assert() at witness_assert+0x35a/frame 0xfe009830
>> __lockmgr_args() at __lockmgr_args+0x517/frame 0xfe0098d0
>> vop_stdunlock() at vop_stdunlock+0x3b/frame 0xfe0098f0
>> VOP_UNLOCK_APV() at VOP_UNLOCK_APV+0xe0/frame 0xfe0098222320
>> unionfs_unlock() at unionfs_unlock+0x112/frame 0xfe0098222390
>> VOP_UNLOCK_APV() at VOP_UNLOCK_APV+0xe0/frame 0xfe00982223c0
>> unionfs_nodeget() at unionfs_nodeget+0x3ef/frame 0xfe0098222470
>> unionfs_domount() at unionfs_domount+0x518/frame 0xfe00982226b0
>> vfs_donmount() at vfs_donmount+0xe37/frame 0xfe00982228f0
>> sys_nmount() at sys_nmount+0x72/frame 0xfe0098222930
>> amd64_syscall() at amd64_syscall+0x2f9/frame 0xfe0098222ab0
>> Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfe0098222ab0
>> --- syscall (378, FreeBSD ELF64, sys_nmount), rip = 0x80086ecea, rsp =
>> 0x7fffe318, rbp = 0x7fffeca0 ---
> New discovery:
> Rick's latest patch casues panic only with KDB. If I compile a kernel
> without witenss and KDB, the machine boots fine!
> Also, it's at least not so easy anymore to trigger the deadlock :-) . I
> need to do more testing but until now Rick's approach seems very
> promising :-) . 

My unionfs deadlock problem isn't really solved with Rick's latest
patch, I still can reproduce it: krb5.conf and krb5.keytab are files on
unionfs referenced by /etc.  libexec/negotiate_kerberos_auth reads these
and if I have enough helper processes handling requests, the deadlock
occurs.

_But_: If I move the files outside the unionfs and create a symlink, I
cannot reproduce the deadlock anymore, which was similar easily
reproducable without it or any of the other workarounds.
So it looks like I have an acceptable solution for now, although it's
only usable under certain conditions.

Unfortunately I can't do tests with a debug kernel since the patch
prevents the system with the debug kernel from starting up.
But if this was ironed out, I'd happily provide more info.


Thanks,

-Harry

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: unionfs bugs, a partial patch and some comments [Was: Re: 1-BETA3 Panic: __lockmgr_args: downgrade a recursed lockmgr nfs @ /usr/local/share/deploy-tools/RELENG_11/src/sys/fs/unionfs/union_vnops.c

2017-03-07 Thread Harry Schmalzbauer 
 Bezüglich Harry Schmalzbauer's Nachricht vom 07.03.2017 13:42 (localtime):
…
> Something ufs related seems to have tightened the unionfs locking
> problem in stable/11.  Now the machine instantaniously panics during
> boot after mounting root with Rick's latest patch.
>
> Unfortunately I don't have SWAP available on that machine (yet), but
> maybe shit is a hint for anybody.
>
> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
> 0xfe00982220e0
> vpanic() at vpanic+0x186/frame 0xfe0098222160
> kassert_panic() at kassert_panic+0x126/frame 0xfe00982221d0
> witness_assert() at witness_assert+0x35a/frame 0xfe009830
> __lockmgr_args() at __lockmgr_args+0x517/frame 0xfe0098d0
> vop_stdunlock() at vop_stdunlock+0x3b/frame 0xfe0098f0
> VOP_UNLOCK_APV() at VOP_UNLOCK_APV+0xe0/frame 0xfe0098222320
> unionfs_unlock() at unionfs_unlock+0x112/frame 0xfe0098222390
> VOP_UNLOCK_APV() at VOP_UNLOCK_APV+0xe0/frame 0xfe00982223c0
> unionfs_nodeget() at unionfs_nodeget+0x3ef/frame 0xfe0098222470
> unionfs_domount() at unionfs_domount+0x518/frame 0xfe00982226b0
> vfs_donmount() at vfs_donmount+0xe37/frame 0xfe00982228f0
> sys_nmount() at sys_nmount+0x72/frame 0xfe0098222930
> amd64_syscall() at amd64_syscall+0x2f9/frame 0xfe0098222ab0
> Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfe0098222ab0
> --- syscall (378, FreeBSD ELF64, sys_nmount), rip = 0x80086ecea, rsp =
> 0x7fffe318, rbp = 0x7fffeca0 ---

New discovery:
Rick's latest patch casues panic only with KDB. If I compile a kernel
without witenss and KDB, the machine boots fine!
Also, it's at least not so easy anymore to trigger the deadlock :-) . I
need to do more testing but until now Rick's approach seems very
promising :-) . Unfortunately I can't provide a fix or suggestion to why
the KDB kernel panics and the non-KDB doesn't, just the dull imagination
it could be that additional locking checks (KASSERT?), preventing more
damage, are not in place. So I guess I'm in danger waters, but it
defenitly is a highly appreciated improvement for me and my bery best
bet for now (neither eliminating unionfs nor holding off 11 updates were
real options for me, especially because unionfs isn't really well
wokring on 10.3 either, just not leading to deadlocks in more environments)!

I tried the non-debug kernel because I browsed old unionfs discussions
and desperately gave Attilio Rao's patch a try since I couldn't remember
why I haven't kept it locally:
https://people.freebsd.org/~attilio/unionfs_nodeget4.patch (he tried to
solve unionfs problems for RELENG_9 back in 2012:
https://lists.freebsd.org/pipermail/freebsd-stable/2012-November/070358.html)

It's still true that his patch leads to a panic with debugging kernel –
only. Same patch without KDB allows to boot and start squid. But the
result is the same as with plain r314856, the system deadlocks reproducibly.

Also, the trace with his patch looks identical to the plain r314856
unionfs panic.

So I hope Rick or someone else can pick up the latest patch and polish
it to make KDB-kernels happy :-)
I can offer a small donation if that helps!
Of course, I'll also provide KDB info if needed/helpful.

thanks,

-harry
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: unionfs bugs, a partial patch and some comments [Was: Re: 1-BETA3 Panic: __lockmgr_args: downgrade a recursed lockmgr nfs @ /usr/local/share/deploy-tools/RELENG_11/src/sys/fs/unionfs/union_vnops.c

2017-03-07 Thread Harry Schmalzbauer 
Bezüglich Rick Macklem's Nachricht vom 05.09.2016 23:21 (localtime):
> Harry Schmalzbauer  wrote:
>>Bezüglich Rick Macklem's Nachricht vom 18.08.2016 02:03 (localtime):
>>>  Kostik wrote:
>>> [stuff snipped]
 insmnque() performs the cleanup on its own, and that default cleanup
> isnot suitable >for the situation.  I think that insmntque1() would
> betterfit your requirements, your >need to move the common code into a
> helper.It seems that >unionfs_ins_cached_vnode() cleanup could reuse it.
>>> 
>>> I've attached an updated patch (untested like the last one). This one
> creates a
>>> custom version insmntque_stddtr() that first calls unionfs_noderem()
> and then
>>> does the same stuff as insmntque_stddtr(). This looks like it does the
> required
>>> stuff (unionfs_noderem() is what the unionfs VOP_RECLAIM() does).
>>> It switches the node back to using its own v_vnlock that is
> exclusively locked,
>>> among other things.
>>
>>Thanks a lot, today I gave it a try.
>>
>>With this patch, one reproducable panic can still be easily triggered:
>>I have directory A unionfs_mounted under directory B.
>>Then I mount_unionfs the same directory A below another directory C.
>>panic: __lockmgr_args: downgrade a recursed lockmgr nfs @
>>/usr/local/share/deploy-tools/RELENG_11/src/sys/fs/unionfs/union_vnops.c:1905
>>Result is this backtrace, hardly helpful I guess:
>>
>>#1  0x80ae5fd9 in kern_reboot (howto=260) at
>>/usr/local/share/deploy-tools/RELENG_11/src/sys/kern/kern_shutdown.c:366
>>#2  0x80ae658b in vpanic (fmt=, ap=>optimized out>)
>>at
>>/usr/local/share/deploy-tools/RELENG_11/src/sys/kern/kern_shutdown.c:759
>>#3  0x80ae63c3 in panic (fmt=0x0) at
>>/usr/local/share/deploy-tools/RELENG_11/src/sys/kern/kern_shutdown.c:690
>>#4  0x80ab7ab7 in __lockmgr_args (lk=,
>>flags=, ilk=, wmesg=>optimized out>,
>>pri=, timo=, file=>optimized out>, line=)
>>  >   at
> /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/kern_lock.c:992
>>#5  0x80ba510c in vop_stdlock (ap=) at
>>lockmgr.h:98
>>#6  0x8111932d in VOP_LOCK1_APV (vop=,
>>a=) at vnode_if.c:2087
>>#7  0x80a18cfc in unionfs_lock (ap=0xfe007a3ba6a0) at
>>vnode_if.h:859
>>#8  0x8111932d in VOP_LOCK1_APV (vop=,
>>a=) at vnode_if.c:2087
>>#9  0x80bc9b93 in _vn_lock (vp=,
>>flags=66560, file=, line=) at
>>vnode_if.h:859
>>#10 0x80a18460 in unionfs_readdir (ap=) at
>>/usr/local/share/deploy-tools/RELENG_11/src/sys/fs/unionfs/union_vnops.c:1531
>>#11 0x81118ecf in VOP_READDIR_APV (vop=,
>>a=) at vnode_if.c:1822
>>#12 0x80bc6e3b in kern_getdirentries (td=,
>>fd=, buf=0x800c3d000 >bounds>,
>>count=, basep=0xfe007a3ba980, residp=0x0)
>>at vnode_if.h:758
>>#13 0x80bc6bf8 in sys_getdirentries (td=0x0,
>>uap=0xfe007a3baa40) at
>>/usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_syscalls.c:3940
>>#14 0x80fad6b8 in amd64_syscall (td=,
>>traced=0) at subr_syscall.c:135
>>#15 0x80f8feab in Xfast_syscall () at
>>/usr/local/share/deploy-tools/RELENG_11/src/sys/amd64/amd64/exception.S:396
>>#16 0x00452eea in ?? ()
>>Previous frame inner to this frame (corrupt stack?
> Ok, I finally got around to looking at this and the panic() looks like a
> pretty straightforward
> bug in the unionfs code.
> - In unionfs_readdir(), it does a vn_lock(..LK_UPGRADE) and then later
> in the code
>   vn_lock(..LK_DOWNGRADE) if it did the upgrade. (At line#1531 as noted
> in the backtrace.)
>   - In unionfs_lock(), it sets LK_CANRECURSE when it is the rootvp and
> LK_EXCLUSIVE.
>(So it allows recursive acquisition in this case.)
> --> Then it would call vn_lock(..LK_DOWNGRADE), which would panic if it
> has recursed.
> 
> Now, I'll admit unionfs_lock() is too obscure for me to understand, but...
> Is it necessary to vn_lock(..LK_DOWNGRADE) or can unionfs_readdir() just
> return
> with the vnode exclusively locked?
> (It would be easy to change the code to avoid the
> vn_lock(..LK_DOWNGRADE) call
>  when it has done the vn_lock(..LK_EXCLUSIVE) after
> vn_lock(..LK_UPGRADE) fails.)
> 
> rick
> 
>>I ran your previous patch with for some time.
>>Similarly, mounting one directory below a 2nd mountpount crashed the
>>machine (forgot to config dumpdir, so can't compare backtrace with the
>>current patch).
>>Otherwise, at least with the previous patch, I haven't had any other
>>panic for about one week.

Something ufs related seems to have tightened the unionfs locking
problem in stable/11.  Now the machine instantaniously panics during
boot after mounting root with Rick's latest patch.

Unfortunately I don't have SWAP available on that machine (yet), but
maybe shit is a hint for anybody.

db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
0xfe00982220e0
vpanic() at vpanic+0x186/frame 0xfe0098222160
kassert_panic() at kassert_panic+0x126/frame 0xfe00982221d0
witness_assert() at witness_assert+0

Re: unionfs bugs, a partial patch and some comments [Was: Re: 1-BETA3 Panic: __lockmgr_args: downgrade a recursed lockmgr nfs @ /usr/local/share/deploy-tools/RELENG_11/src/sys/fs/unionfs/union_vnops.c

2016-09-05 Thread Rick Macklem 
Harry Schmalzbauer  wrote:
>Bezüglich Rick Macklem's Nachricht vom 18.08.2016 02:03 (localtime):
>>  Kostik wrote:
>> [stuff snipped]
>>> insmnque() performs the cleanup on its own, and that default cleanup isnot 
>>> suitable >for the situation.  I think that insmntque1() would betterfit 
>>> your requirements, your >need to move the common code into a helper.It 
>>> seems that >unionfs_ins_cached_vnode() cleanup could reuse it.
>> 
>> I've attached an updated patch (untested like the last one). This one 
>> creates a
>> custom version insmntque_stddtr() that first calls unionfs_noderem() and then
>> does the same stuff as insmntque_stddtr(). This looks like it does the 
>> required
>> stuff (unionfs_noderem() is what the unionfs VOP_RECLAIM() does).
>> It switches the node back to using its own v_vnlock that is exclusively 
>> locked,
>> among other things.
>
>Thanks a lot, today I gave it a try.
>
>With this patch, one reproducable panic can still be easily triggered:
>I have directory A unionfs_mounted under directory B.
>Then I mount_unionfs the same directory A below another directory C.
>panic: __lockmgr_args: downgrade a recursed lockmgr nfs @
>/usr/local/share/deploy-tools/RELENG_11/src/sys/fs/unionfs/union_vnops.c:1905
>Result is this backtrace, hardly helpful I guess:
>
>#1  0x80ae5fd9 in kern_reboot (howto=260) at
>/usr/local/share/deploy-tools/RELENG_11/src/sys/kern/kern_shutdown.c:366
>#2  0x80ae658b in vpanic (fmt=, ap=optimized out>)
>at
>/usr/local/share/deploy-tools/RELENG_11/src/sys/kern/kern_shutdown.c:759
>#3  0x80ae63c3 in panic (fmt=0x0) at
>/usr/local/share/deploy-tools/RELENG_11/src/sys/kern/kern_shutdown.c:690
>#4  0x80ab7ab7 in __lockmgr_args (lk=,
>flags=, ilk=, wmesg=optimized out>,
>pri=, timo=, file=optimized out>, line=)
>  >   at /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/kern_lock.c:992
>#5  0x80ba510c in vop_stdlock (ap=) at
>lockmgr.h:98
>#6  0x8111932d in VOP_LOCK1_APV (vop=,
>a=) at vnode_if.c:2087
>#7  0x80a18cfc in unionfs_lock (ap=0xfe007a3ba6a0) at
>vnode_if.h:859
>#8  0x8111932d in VOP_LOCK1_APV (vop=,
>a=) at vnode_if.c:2087
>#9  0x80bc9b93 in _vn_lock (vp=,
>flags=66560, file=, line=) at
>vnode_if.h:859
>#10 0x80a18460 in unionfs_readdir (ap=) at
>/usr/local/share/deploy-tools/RELENG_11/src/sys/fs/unionfs/union_vnops.c:1531
>#11 0x81118ecf in VOP_READDIR_APV (vop=,
>a=) at vnode_if.c:1822
>#12 0x80bc6e3b in kern_getdirentries (td=,
>fd=, buf=0x800c3d000 bounds>,
>count=, basep=0xfe007a3ba980, residp=0x0)
>at vnode_if.h:758
>#13 0x80bc6bf8 in sys_getdirentries (td=0x0,
>uap=0xfe007a3baa40) at
>/usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_syscalls.c:3940
>#14 0x80fad6b8 in amd64_syscall (td=,
>traced=0) at subr_syscall.c:135
>#15 0x80f8feab in Xfast_syscall () at
>/usr/local/share/deploy-tools/RELENG_11/src/sys/amd64/amd64/exception.S:396
>#16 0x00452eea in ?? ()
>Previous frame inner to this frame (corrupt stack?
Ok, I finally got around to looking at this and the panic() looks like a pretty 
straightforward
bug in the unionfs code.
- In unionfs_readdir(), it does a vn_lock(..LK_UPGRADE) and then later in the 
code
  vn_lock(..LK_DOWNGRADE) if it did the upgrade. (At line#1531 as noted in the 
backtrace.)
  - In unionfs_lock(), it sets LK_CANRECURSE when it is the rootvp and 
LK_EXCLUSIVE.
   (So it allows recursive acquisition in this case.)
--> Then it would call vn_lock(..LK_DOWNGRADE), which would panic if it has 
recursed.

Now, I'll admit unionfs_lock() is too obscure for me to understand, but...
Is it necessary to vn_lock(..LK_DOWNGRADE) or can unionfs_readdir() just return
with the vnode exclusively locked?
(It would be easy to change the code to avoid the vn_lock(..LK_DOWNGRADE) call
 when it has done the vn_lock(..LK_EXCLUSIVE) after vn_lock(..LK_UPGRADE) 
fails.)

rick

>I ran your previous patch with for some time.
>Similarly, mounting one directory below a 2nd mountpount crashed the
>machine (forgot to config dumpdir, so can't compare backtrace with the
>current patch).
>Otherwise, at least with the previous patch, I haven't had any other
>panic for about one week.
>
>Thanks,
>
>-Harry
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: unionfs bugs, a partial patch and some comments [Was: Re: 1-BETA3 Panic: __lockmgr_args: downgrade a recursed lockmgr nfs @ /usr/local/share/deploy-tools/RELENG_11/src/sys/fs/unionfs/union_vnops.c

2016-08-20 Thread Harry Schmalzbauer 
Bezüglich Rick Macklem's Nachricht vom 18.08.2016 02:03 (localtime):
>  Kostik wrote:
> [stuff snipped]
>> insmnque() performs the cleanup on its own, and that default cleanup isnot 
>> suitable >for the situation.  I think that insmntque1() would betterfit your 
>> requirements, your >need to move the common code into a helper.It seems that 
>> >unionfs_ins_cached_vnode() cleanup could reuse it.
> 
> I've attached an updated patch (untested like the last one). This one creates 
> a
> custom version insmntque_stddtr() that first calls unionfs_noderem() and then
> does the same stuff as insmntque_stddtr(). This looks like it does the 
> required
> stuff (unionfs_noderem() is what the unionfs VOP_RECLAIM() does).
> It switches the node back to using its own v_vnlock that is exclusively 
> locked,
> among other things.

Thanks a lot, today I gave it a try.

With this patch, one reproducable panic can still be easily triggered:
I have directory A unionfs_mounted under directory B.
Then I mount_unionfs the same directory A below another directory C.
panic: __lockmgr_args: downgrade a recursed lockmgr nfs @
/usr/local/share/deploy-tools/RELENG_11/src/sys/fs/unionfs/union_vnops.c:1905
Result is this backtrace, hardly helpful I guess:

#1  0x80ae5fd9 in kern_reboot (howto=260) at
/usr/local/share/deploy-tools/RELENG_11/src/sys/kern/kern_shutdown.c:366
#2  0x80ae658b in vpanic (fmt=, ap=)
at
/usr/local/share/deploy-tools/RELENG_11/src/sys/kern/kern_shutdown.c:759
#3  0x80ae63c3 in panic (fmt=0x0) at
/usr/local/share/deploy-tools/RELENG_11/src/sys/kern/kern_shutdown.c:690
#4  0x80ab7ab7 in __lockmgr_args (lk=,
flags=, ilk=, wmesg=,
pri=, timo=, file=, line=)
at /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/kern_lock.c:992
#5  0x80ba510c in vop_stdlock (ap=) at
lockmgr.h:98
#6  0x8111932d in VOP_LOCK1_APV (vop=,
a=) at vnode_if.c:2087
#7  0x80a18cfc in unionfs_lock (ap=0xfe007a3ba6a0) at
vnode_if.h:859
#8  0x8111932d in VOP_LOCK1_APV (vop=,
a=) at vnode_if.c:2087
#9  0x80bc9b93 in _vn_lock (vp=,
flags=66560, file=, line=) at
vnode_if.h:859
#10 0x80a18460 in unionfs_readdir (ap=) at
/usr/local/share/deploy-tools/RELENG_11/src/sys/fs/unionfs/union_vnops.c:1531
#11 0x81118ecf in VOP_READDIR_APV (vop=,
a=) at vnode_if.c:1822
#12 0x80bc6e3b in kern_getdirentries (td=,
fd=, buf=0x800c3d000 ,
count=, basep=0xfe007a3ba980, residp=0x0)
at vnode_if.h:758
#13 0x80bc6bf8 in sys_getdirentries (td=0x0,
uap=0xfe007a3baa40) at
/usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_syscalls.c:3940
#14 0x80fad6b8 in amd64_syscall (td=,
traced=0) at subr_syscall.c:135
#15 0x80f8feab in Xfast_syscall () at
/usr/local/share/deploy-tools/RELENG_11/src/sys/amd64/amd64/exception.S:396
#16 0x00452eea in ?? ()
Previous frame inner to this frame (corrupt stack?

I ran your previous patch with for some time.
Similarly, mounting one directory below a 2nd mountpount crashed the
machine (forgot to config dumpdir, so can't compare backtrace with the
current patch).
Otherwise, at least with the previous patch, I haven't had any other
panic for about one week.

Thanks,

-Harry
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: unionfs bugs, a partial patch and some comments [Was: Re: 1-BETA3 Panic: __lockmgr_args: downgrade a recursed lockmgr nfs @ /usr/local/share/deploy-tools/RELENG_11/src/sys/fs/unionfs/union_vnops.c

2016-08-18 Thread Konstantin Belousov 
On Thu, Aug 18, 2016 at 12:03:33AM +, Rick Macklem wrote:
>  Kostik wrote:
> [stuff snipped]
> >insmnque() performs the cleanup on its own, and that default cleanup isnot 
> >suitable >for the situation.  I think that insmntque1() would betterfit your 
> >requirements, your >need to move the common code into a helper.It seems that 
> >>unionfs_ins_cached_vnode() cleanup could reuse it.
> 
> I've attached an updated patch (untested like the last one). This one creates 
> a
> custom version insmntque_stddtr() that first calls unionfs_noderem() and then
> does the same stuff as insmntque_stddtr(). This looks like it does the 
> required
> stuff (unionfs_noderem() is what the unionfs VOP_RECLAIM() does).
> It switches the node back to using its own v_vnlock that is exclusively 
> locked,
> among other things.

The patch looks good to me, from the visual shape of it.
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: unionfs bugs, a partial patch and some comments [Was: Re: 1-BETA3 Panic: __lockmgr_args: downgrade a recursed lockmgr nfs @ /usr/local/share/deploy-tools/RELENG_11/src/sys/fs/unionfs/union_vnops.c

2016-08-17 Thread Rick Macklem 
 Kostik wrote:
[stuff snipped]
>insmnque() performs the cleanup on its own, and that default cleanup isnot 
>suitable >for the situation.  I think that insmntque1() would betterfit your 
>requirements, your >need to move the common code into a helper.It seems that 
>>unionfs_ins_cached_vnode() cleanup could reuse it.

I've attached an updated patch (untested like the last one). This one creates a
custom version insmntque_stddtr() that first calls unionfs_noderem() and then
does the same stuff as insmntque_stddtr(). This looks like it does the required
stuff (unionfs_noderem() is what the unionfs VOP_RECLAIM() does).
It switches the node back to using its own v_vnlock that is exclusively locked,
among other things.

rick



unionfs-newvnode.patch
Description: unionfs-newvnode.patch
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: unionfs bugs, a partial patch and some comments [Was: Re: 1-BETA3 Panic: __lockmgr_args: downgrade a recursed lockmgr nfs @ /usr/local/share/deploy-tools/RELENG_11/src/sys/fs/unionfs/union_vnops.c

2016-08-12 Thread Konstantin Belousov 
On Thu, Aug 11, 2016 at 10:53:03PM +, Rick Macklem wrote:
> Harry Schmalzbauer wrote:
> Bez??glich Mark Johnston's Nachricht vom 09.08.2016 08:02 (localtime):
> ???
> >>
> >> Just for anybody else needing unionfs:
> >> https://people.freebsd.org/~attilio/unionfs_missing_insmntque_lock.patch
> >>
> >> This patch still applies and I'm successfully using this (unmodified) up
> >> to FreeBSD-10.3 and never had any panic in all these years.
> >
> > Having spent some time looking at unionfs, I'm a bit skeptical that this
> > patch will address the panic you reported earlier, though I'd be
> > interested to know if it does.
> [stuff snipped for brevity]
> I took a look at this. (I know nothing about unionfs, but a little w.r.t. the 
> VFS).
> I can confirm that this function (unionfs_nodeget()) is weird and appears to
> be broken to me.
> 
> The function calls insmntque() before it initializes the vnode, which seems
> racey, especially if it isn't LK_EXCLUSIVE locked.
> Also, line#s 278-281:
> if (uppervp != NULLVP)
>  vp->v_vnlock = uppervp->v_vnlock;
> else
>  vp->v_vnlock = lowervp->v_vnlock;
> so your patch isn't locking the vnode lock that it actually uses.
> I think the vp argument to insmntque() is required to be LK_EXCLUSIVE
> locked mostly so other threads won't fiddle with the vnode until this
> function is done with it, but I am not sure?
> 
> I think a more correct version of this (not saying it would be correct[],
> would call insmntque() later in the function, after it has been initialized.
> (This means that the cleanup if it fails is more involved, but...)
Yes.

> 
> I've attached a patch (untested) that does this. Maybe you could try it?
> 
> rick
> ps: I've cc'd Kostik, in case he has some insight w.r.t. how this should be 
> handled?
> 
insmnque() performs the cleanup on its own, and that default cleanup is
not suitable for the situation.  I think that insmntque1() would better
fit your requirements, your need to move the common code into a helper.
It seems that unionfs_ins_cached_vnode() cleanup could reuse it.
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: unionfs bugs, a partial patch and some comments [Was: Re: 1-BETA3 Panic: __lockmgr_args: downgrade a recursed lockmgr nfs @ /usr/local/share/deploy-tools/RELENG_11/src/sys/fs/unionfs/union_vnops.c

2016-08-12 Thread Rick Macklem 
Harry Schmalzbauer wrote:
Bezüglich Mark Johnston's Nachricht vom 09.08.2016 08:02 (localtime):
…
>>
>> Just for anybody else needing unionfs:
>> https://people.freebsd.org/~attilio/unionfs_missing_insmntque_lock.patch
>>
>> This patch still applies and I'm successfully using this (unmodified) up
>> to FreeBSD-10.3 and never had any panic in all these years.
>
> Having spent some time looking at unionfs, I'm a bit skeptical that this
> patch will address the panic you reported earlier, though I'd be
> interested to know if it does.
[stuff snipped for brevity]
I took a look at this. (I know nothing about unionfs, but a little w.r.t. the 
VFS).
I can confirm that this function (unionfs_nodeget()) is weird and appears to
be broken to me.

The function calls insmntque() before it initializes the vnode, which seems
racey, especially if it isn't LK_EXCLUSIVE locked.
Also, line#s 278-281:
if (uppervp != NULLVP)
 vp->v_vnlock = uppervp->v_vnlock;
else
 vp->v_vnlock = lowervp->v_vnlock;
so your patch isn't locking the vnode lock that it actually uses.
I think the vp argument to insmntque() is required to be LK_EXCLUSIVE
locked mostly so other threads won't fiddle with the vnode until this
function is done with it, but I am not sure?

I think a more correct version of this (not saying it would be correct[😉],
would call insmntque() later in the function, after it has been initialized.
(This means that the cleanup if it fails is more involved, but...)

I've attached a patch (untested) that does this. Maybe you could try it?

rick
ps: I've cc'd Kostik, in case he has some insight w.r.t. how this should be 
handled?



unionfs-newvnode.patch
Description: unionfs-newvnode.patch
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: unionfs bugs, a partial patch and some comments [Was: Re: 1-BETA3 Panic: __lockmgr_args: downgrade a recursed lockmgr nfs @ /usr/local/share/deploy-tools/RELENG_11/src/sys/fs/unionfs/union_vnops.c

2016-08-09 Thread Harry Schmalzbauer 
Bezüglich Mark Johnston's Nachricht vom 09.08.2016 08:02 (localtime):
…
>>
>> Just for anybody else needing unionfs:
>> https://people.freebsd.org/~attilio/unionfs_missing_insmntque_lock.patch
>>
>> This patch still applies and I'm successfully using this (unmodified) up
>> to FreeBSD-10.3 and never had any panic in all these years.
> 
> Having spent some time looking at unionfs, I'm a bit skeptical that this
> patch will address the panic you reported earlier, though I'd be
> interested to know if it does. 

Thanks for your attention.
I can confirm that it has prevented panics for more than 4 years
(9.0-10.3) and it seems to be still "good enough" to also prevent panics
in 11-BETA4.
I updated my build host (stable/11, this time with the
unionfs_missing_insmntque_lock.patch), where the recent panics happened
and unionfs gets much more utilized than usually in my setups: No panic
with that patch anymore.
Just one message like "prevented resource deadlock" occured.

> Reading the code, I think it will just
> address an INVARIANTS-only assertion in insmntque1().
> 
> Unfortunately, unionfs is quite difficult to fix within the current
> constraints of FreeBSD's VFS. unionfs_readdir() is a particularly good
> demonstration of this fact: some callers of VOP_READDIR expect the
> cookies returned by the FS to be monotonically increasing, but unionfs
> has no straightforward way to make this guarantee.

I'm sorry, I can't provide help here. My skills would require a huge
ammount of lerning-time to get into that matter. I'd love to do that,
but I can't afford :-(

Thanks,

-Harry
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: unionfs bugs, a partial patch and some comments [Was: Re: 1-BETA3 Panic: __lockmgr_args: downgrade a recursed lockmgr nfs @ /usr/local/share/deploy-tools/RELENG_11/src/sys/fs/unionfs/union_vnops.c

2016-08-09 Thread Harry Schmalzbauer 
Bezüglich Kurt Jaeger's Nachricht vom 09.08.2016 07:32 (localtime):
> Hi!
> 
>> Since then I'm draging a minimal patch which prevents at least the
>> kernel panics for me.
>> Unfortunately I don't have the skills to continue Attilio Raos work.
>>
>> Just for anybody else needing unionfs:
>> https://people.freebsd.org/~attilio/unionfs_missing_insmntque_lock.patch
> 
> Is this referenced in any PR ? If not, can you create one ?

Good question, bad answer: No
I had been told not to file a PR at that time because unionfs overhaul
was planned/needed. Partial fixes would have been counterproductive in
that state and not "the right way to go".
Looks like overhaul hasn't really happened during the last 4 years and
since I was happy with the partial patch, I haven't followed unionfs
development.
In the mean time I remember that lots of fs-stress-tests were added to
the FreeBSD test suite, but I'm not familar with a single one, even not
with ATF and the like, and won't find the time to dig into them.
So presently I hesitate picking up that old issue and file a PR, because
developers resources are extremely limited regarding unionfs, and the PR
should be attended by someone who has time and knowledge to make
developers life easier by providing qualified analysis and tests. All I
could do at the moment is to point at a dysfunction, which is documented
in the man page, which isn't really helpful/needed.


>> First thing to do for me, after I won in lottery, was to find someone
>> who can be sponsored fixing unionfs ;-) And bringing MNAMELEN into 21st
>> century state, matching ZFS needs:
>> https://lists.freebsd.org/pipermail/freebsd-hackers/2015-November/048640.html
>> This is another patch I'm carrying for a very long time which solves
>> tremendous limitations for me. Without that, I couldn't use ZFS
>> snapshots in real world, along with a human-friendly dataset naming :-)
> 
> And is there a PR for that ?

Sadly, the same answer with similar reasons is true.
I asked Doug Ambrisko (the author of the mount_bigger_2_1.patch) at
10.2-BETA time (2015/07/10) about progress and future handling of his patch.
He answered that Marshal Kirk McKusik asked him not to continue in that
direction, since »it would make the 64 bit inode work harder«.

So we agreed continuing being happy with the patch in our world, leaving
the rest unhappy ;-) See the link to that discussion above.

I was kind of astonsihed that this patch still applies to 11 and it
seems 11 still has the MNAMELEN limitation besides make_dev_p() ability
to handle extended lengths.

CC'ed Doug, perhaps he joins this thread calrifying things.

The unionfs thing is a edge-case thing, but MNAMELEN is much more
important, imho, since ZFS is one of FreeBSD's most appreciated "new
features" and this MNAMELEN limit needlessly counteracts ZFS' deployment.

Of course I'll keep nagging from time to time ;-) And file PRs if not
told otherwise :-)

-Harry
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: unionfs bugs, a partial patch and some comments [Was: Re: 1-BETA3 Panic: __lockmgr_args: downgrade a recursed lockmgr nfs @ /usr/local/share/deploy-tools/RELENG_11/src/sys/fs/unionfs/union_vnops.c

2016-08-08 Thread Mark Johnston 
On Mon, Aug 08, 2016 at 10:02:00AM +0200, Harry Schmalzbauer wrote:
> Bezüglich Rick Macklem's Nachricht vom 07.08.2016 23:34 (localtime):
> > Harry Schmalzbauer wrote:
> >> Hello,
> >>
> >> I had another crash which I'm quite sure was triggered by mount_unionfs:
> > Just in case you are not already aware, unionfs is always broken. Read
> > the BUGS
> > section at the end of "man mount_unionfs". If it were easy to fix,
> > someone would
> > have done so long ago. Yes, some use it successfully, but if not...
> > 
> > Sorry, but I suspect that is how it will remain, rick
> 
> Thanks for the hint, not happy to hear that, but I was not aware of that
> explicit warning in man 8 mount_unionfs :-(
> 
> This feature is utterly important for me (all my productive machines
> have "/" read-only mounted and "/etc" is an union to a writable, synch
> mounted separate fs), so back in 2012, after a lot of locking redesign
> has been done in 9-current, I got Attilio Raos attention and he gave out
> some test patches for 9.0.
> He was aware of missing locking adjustments, but patches addressing the
> majority of them didn't work.
> Since then I'm draging a minimal patch which prevents at least the
> kernel panics for me.
> Unfortunately I don't have the skills to continue Attilio Raos work.
> 
> Just for anybody else needing unionfs:
> https://people.freebsd.org/~attilio/unionfs_missing_insmntque_lock.patch
> 
> This patch still applies and I'm successfully using this (unmodified) up
> to FreeBSD-10.3 and never had any panic in all these years.

Having spent some time looking at unionfs, I'm a bit skeptical that this
patch will address the panic you reported earlier, though I'd be
interested to know if it does. Reading the code, I think it will just
address an INVARIANTS-only assertion in insmntque1().

Unfortunately, unionfs is quite difficult to fix within the current
constraints of FreeBSD's VFS. unionfs_readdir() is a particularly good
demonstration of this fact: some callers of VOP_READDIR expect the
cookies returned by the FS to be monotonically increasing, but unionfs
has no straightforward way to make this guarantee.

> 
> I will continue using it for FreeBSD-11 and I guess it will also prevent
> my last reported panics.
> But I wanted to take part in the BETA test without local modifications
> at first.
> 
> Another very importend usage scenario of unionfs for me is for my build
> host(s). I'm (nfs4-)sharing a svn-checked out read-only portstree. My
> inofficial "ports/inofficial" directory perfectly shows up by
> unionfs-mounting it below the unaltered portstree :-)
> 
> For me, unionfs is as important as ZFS (and nullfs) is in FreeBSD.
> 
> First thing to do for me, after I won in lottery, was to find someone
> who can be sponsored fixing unionfs ;-) And bringing MNAMELEN into 21st
> century state, matching ZFS needs:
> https://lists.freebsd.org/pipermail/freebsd-hackers/2015-November/048640.html
> This is another patch I'm carrying for a very long time which solves
> tremendous limitations for me. Without that, I couldn't use ZFS
> snapshots in real world, along with a human-friendly dataset naming :-)
> 
> -Harry
> 
> ___
> freebsd-stable@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: unionfs bugs, a partial patch and some comments [Was: Re: 1-BETA3 Panic: __lockmgr_args: downgrade a recursed lockmgr nfs @ /usr/local/share/deploy-tools/RELENG_11/src/sys/fs/unionfs/union_vnops.c

2016-08-08 Thread Kurt Jaeger 
Hi!

> Since then I'm draging a minimal patch which prevents at least the
> kernel panics for me.
> Unfortunately I don't have the skills to continue Attilio Raos work.
> 
> Just for anybody else needing unionfs:
> https://people.freebsd.org/~attilio/unionfs_missing_insmntque_lock.patch

Is this referenced in any PR ? If not, can you create one ?

> First thing to do for me, after I won in lottery, was to find someone
> who can be sponsored fixing unionfs ;-) And bringing MNAMELEN into 21st
> century state, matching ZFS needs:
> https://lists.freebsd.org/pipermail/freebsd-hackers/2015-November/048640.html
> This is another patch I'm carrying for a very long time which solves
> tremendous limitations for me. Without that, I couldn't use ZFS
> snapshots in real world, along with a human-friendly dataset naming :-)

And is there a PR for that ?

-- 
p...@opsec.eu+49 171 3101372 4 years to go !
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

unionfs bugs, a partial patch and some comments [Was: Re: 1-BETA3 Panic: __lockmgr_args: downgrade a recursed lockmgr nfs @ /usr/local/share/deploy-tools/RELENG_11/src/sys/fs/unionfs/union_vnops.c:190

2016-08-08 Thread Harry Schmalzbauer 
Bezüglich Rick Macklem's Nachricht vom 07.08.2016 23:34 (localtime):
> Harry Schmalzbauer wrote:
>> Hello,
>>
>> I had another crash which I'm quite sure was triggered by mount_unionfs:
> Just in case you are not already aware, unionfs is always broken. Read
> the BUGS
> section at the end of "man mount_unionfs". If it were easy to fix,
> someone would
> have done so long ago. Yes, some use it successfully, but if not...
> 
> Sorry, but I suspect that is how it will remain, rick

Thanks for the hint, not happy to hear that, but I was not aware of that
explicit warning in man 8 mount_unionfs :-(

This feature is utterly important for me (all my productive machines
have "/" read-only mounted and "/etc" is an union to a writable, synch
mounted separate fs), so back in 2012, after a lot of locking redesign
has been done in 9-current, I got Attilio Raos attention and he gave out
some test patches for 9.0.
He was aware of missing locking adjustments, but patches addressing the
majority of them didn't work.
Since then I'm draging a minimal patch which prevents at least the
kernel panics for me.
Unfortunately I don't have the skills to continue Attilio Raos work.

Just for anybody else needing unionfs:
https://people.freebsd.org/~attilio/unionfs_missing_insmntque_lock.patch

This patch still applies and I'm successfully using this (unmodified) up
to FreeBSD-10.3 and never had any panic in all these years.

I will continue using it for FreeBSD-11 and I guess it will also prevent
my last reported panics.
But I wanted to take part in the BETA test without local modifications
at first.

Another very importend usage scenario of unionfs for me is for my build
host(s). I'm (nfs4-)sharing a svn-checked out read-only portstree. My
inofficial "ports/inofficial" directory perfectly shows up by
unionfs-mounting it below the unaltered portstree :-)

For me, unionfs is as important as ZFS (and nullfs) is in FreeBSD.

First thing to do for me, after I won in lottery, was to find someone
who can be sponsored fixing unionfs ;-) And bringing MNAMELEN into 21st
century state, matching ZFS needs:
https://lists.freebsd.org/pipermail/freebsd-hackers/2015-November/048640.html
This is another patch I'm carrying for a very long time which solves
tremendous limitations for me. Without that, I couldn't use ZFS
snapshots in real world, along with a human-friendly dataset naming :-)

-Harry

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"