Re: Vnode API change: add global vnode cache

2015-03-09 Thread J. Hannken-Illjes
On 22 May 2014, at 00:48, David Holland dholland-t...@netbsd.org wrote: snip As I discovered while prodding lfs last weekend, this is too optimistic, as there's another family of cases: allocating a new inode and constructing a vnode for it. It is *possible* to do this by allocating an

Re: Vnode API change: add global vnode cache

2014-05-21 Thread David Holland
On Sat, Apr 26, 2014 at 07:22:50PM +, David Holland wrote: - It seems to me that in the long run, all of this baloney should be hidden away inside the vfs layer; filesystems that use the vnode cache should only need to call vcache_get, and the only thing that should ever see a

Re: Vnode API change: add global vnode cache

2014-05-09 Thread Matthew Mondor
On Wed, 30 Apr 2014 17:15:16 +0200 J. Hannken-Illjes hann...@eis.cs.tu-bs.de wrote: vcache_get(mp, key, key_len, vpp) to lookup and possibly load a vnode. vcache_lookup(mp, key, key_len, vpp) to lookup a vnode. vcache_remove(mp, key, key_len) to remove a vnode from the cache.

Re: Vnode API change: add global vnode cache

2014-05-09 Thread Taylor R Campbell
Date: Fri, 9 May 2014 19:01:04 -0400 From: Matthew Mondor mm_li...@pulsar-zone.net Is it expected in vcache_common() for the interlock to remain held even if returning an error? vget unconditionally drops the interlock, so it will never remain held, error or not.

Re: Vnode API change: add global vnode cache

2014-05-09 Thread Matthew Mondor
On Sat, 10 May 2014 01:29:47 + Taylor R Campbell campbell+netbsd-tech-k...@mumble.net wrote: Is it expected in vcache_common() for the interlock to remain held even if returning an error? vget unconditionally drops the interlock, so it will never remain held, error or not. Oh,

Re: Vnode API change: add global vnode cache

2014-05-02 Thread J . Hannken-Illjes
On 01 May 2014, at 20:02, David Holland dholland-t...@netbsd.org wrote: On Thu, May 01, 2014 at 06:49:41PM +0200, J. Hannken-Illjes wrote: ok, in that case I think the only issue with the patch I looked at Sunday (was it that long ago already?) is that it does duplicate the getnewvnode logic

Re: Vnode API change: add global vnode cache

2014-05-01 Thread David Holland
On Mon, Apr 28, 2014 at 07:29:44PM +0400, Ilia Zykov wrote: First, one of the goals here is to have one table for all vnodes. Allow me to disagree, what is advantage has only one table? I see only disadvantage. The reason (the only reason, really) is to bound the total size of the

Re: Vnode API change: add global vnode cache

2014-05-01 Thread David Holland
On Mon, Apr 28, 2014 at 10:56:44AM +0200, J. Hannken-Illjes wrote: Wading into this after the fact, I see the following points: - Duplicating the getnewvnode logic is definitely bad; we have enough cruft without adding new redundant but not quite equivalent code paths. - It

Re: Vnode API change: add global vnode cache

2014-05-01 Thread J. Hannken-Illjes
On 01 May 2014, at 18:06, David Holland dholland-t...@netbsd.org wrote: On Mon, Apr 28, 2014 at 10:56:44AM +0200, J. Hannken-Illjes wrote: Wading into this after the fact, I see the following points: - Duplicating the getnewvnode logic is definitely bad; we have enough cruft without adding

Re: Vnode API change: add global vnode cache

2014-05-01 Thread David Holland
On Thu, May 01, 2014 at 06:49:41PM +0200, J. Hannken-Illjes wrote: ok, in that case I think the only issue with the patch I looked at Sunday (was it that long ago already?) is that it does duplicate the getnewvnode logic -- if you've since fixed that I don't think I have further

Re: Vnode API change: add global vnode cache

2014-05-01 Thread Ilia Zykov
On 01.05.2014 22:02, David Holland wrote: I don't think that's necessary. For vnodes belonging to virtual objects, there's very little to be gained by counting them under maxvnodes or cycling them out of memory. The actual underlying objects can't go away and the vnode itself is

Re: Vnode API change: add global vnode cache

2014-05-01 Thread Ilia Zykov
maxvnodes or cycling them out of memory. The actual underlying objects can't go away and the vnode itself is just a fairly small administrative structure. Maybe it is not to place. procfs' objects can go away. It's not actual objects, sorry.

Re: Vnode API change: add global vnode cache

2014-04-30 Thread Ilia Zykov
Hello. What do you think about DragonFly's VOP_NRESOLVE() API? I think it resolves many problems. It's not replace other changes, but very good supplementation. If I understood it correctly, VFS LOCKS name component, before subsequent vnode's operations. Ilia.

Re: Vnode API change: add global vnode cache

2014-04-30 Thread J. Hannken-Illjes
On 28 Apr 2014, at 10:56, J. Hannken-Illjes hann...@eis.cs.tu-bs.de wrote: snip Could we finally agree on this interface: vcache_get(mp, key, key_len, vpp) to lookup and possibly load a vnode. vcache_lookup(mp, key, key_len, vpp) to lookup a vnode. vcache_remove(mp, key, key_len) to

Re: Vnode API change: add global vnode cache

2014-04-29 Thread Taylor R Campbell
Sorry, I need to focus my time on other things at the moment, so I'll stop getting in the way of progress -- I withdraw my objection. I still think we should not have two different code paths for creating vnodes, and it would be nice to reduce the overhead, but we can sort that out later. By the

Re: Vnode API change: add global vnode cache

2014-04-28 Thread J . Hannken-Illjes
On 26 Apr 2014, at 21:22, David Holland dholland-t...@netbsd.org wrote: On Tue, Apr 15, 2014 at 04:11:58PM +, Taylor R Campbell wrote: New diff at http://www.netbsd.org/~hannken/vnode-pass6-3.diff Plan to commit early wednesday ... I still don't think this approach is right. It

Re: Vnode API change: add global vnode cache

2014-04-28 Thread Ilia Zykov
Sorry to disturb you. But I have small comment about subject. I think it is unnecessary overload involve mount point in lookup for global cache. You save few bytes for hashhead for every mount point, but loose effectively lookup. Every lookup has one more compare for mp, and your hashtable will

Re: Vnode API change: add global vnode cache

2014-04-28 Thread Ilia Zykov
First, one of the goals here is to have one table for all vnodes. Allow me to disagree, what is advantage has only one table? I see only disadvantage. For instance: You mount more than one FSes, and one have hardware problem, bad sector ... Do You have also problems with other or only with

Re: Vnode API change: add global vnode cache

2014-04-28 Thread Ilia Zykov
Having per-mount tables would not change anything here. Ok. It is not fundamental thing. If needed, this can be changed later. Thank you. Ilia.

Re: Vnode API change: add global vnode cache

2014-04-26 Thread David Holland
On Tue, Apr 15, 2014 at 04:11:58PM +, Taylor R Campbell wrote: New diff at http://www.netbsd.org/~hannken/vnode-pass6-3.diff Plan to commit early wednesday ... I still don't think this approach is right. It makes a long-term copy of logic in getnewvnode (because this

Re: Vnode API change: add global vnode cache

2014-04-16 Thread J. Hannken-Illjes
On 15 Apr 2014, at 18:11, Taylor R Campbell campbell+netbsd-tech-k...@mumble.net wrote: Date: Mon, 14 Apr 2014 15:58:28 +0200 From: J. Hannken-Illjes hann...@eis.cs.tu-bs.de Also replaced the cv_wait/cv_broadcast with kpause as insert collisions are very rare and moved the common

Re: Vnode API change: add global vnode cache

2014-04-15 Thread Taylor R Campbell
Date: Mon, 14 Apr 2014 15:58:28 +0200 From: J. Hannken-Illjes hann...@eis.cs.tu-bs.de Also replaced the cv_wait/cv_broadcast with kpause as insert collisions are very rare and moved the common vnode initialisation to vnalloc. New diff at

Re: Vnode API change: add global vnode cache

2014-04-14 Thread J. Hannken-Illjes
On 07 Apr 2014, at 19:28, Chuck Silvers c...@chuq.com wrote: On Sun, Apr 06, 2014 at 12:14:24PM +0200, J. Hannken-Illjes wrote: Currently all file systems have to implement their own cache of vnode / fs node pairs. Most file systems use a copy and pasted version of ufs_ihash. So add a

Re: Vnode API change: add global vnode cache

2014-04-13 Thread Ilia Zykov
Hello! Can you consider possibility integration global vnode cache with one more generic FS function genfs_lookup(). genfs_lookup can be used by other FS, if FS no need something exotic. Interface for genfs_lookup() can be next: /* * Call back function for initiate new created vnode (vnode / fs

Re: Vnode API change: add global vnode cache

2014-04-10 Thread J. Hannken-Illjes
On 09 Apr 2014, at 19:09, Chuck Silvers c...@chuq.com wrote: On Tue, Apr 08, 2014 at 11:23:11AM +0200, J. Hannken-Illjes wrote: On 07 Apr 2014, at 19:28, Chuck Silvers c...@chuq.com wrote: On Sun, Apr 06, 2014 at 12:14:24PM +0200, J. Hannken-Illjes wrote: Currently all file systems have

Re: Vnode API change: add global vnode cache

2014-04-09 Thread J. Hannken-Illjes
On 07 Apr 2014, at 18:02, Taylor R Campbell campbell+netbsd-tech-k...@mumble.net wrote: Date: Mon, 7 Apr 2014 15:51:00 +0200 From: J. Hannken-Illjes hann...@eis.cs.tu-bs.de On 07 Apr 2014, at 03:22, Taylor R Campbell campbell+netbsd-tech-k...@mumble.net wrote: Is the plan to nix

Re: Vnode API change: add global vnode cache

2014-04-09 Thread Taylor R Campbell
Date: Wed, 9 Apr 2014 11:10:37 +0200 From: J. Hannken-Illjes hann...@eis.cs.tu-bs.de There is no need to do this VI_CHANGING etc. here. Primary goal of the vnode cache is to always initialise vnode / fs node pairs before they become visible or appear on any lists. But the

Re: Vnode API change: add global vnode cache

2014-04-09 Thread J. Hannken-Illjes
On 09 Apr 2014, at 15:57, Taylor R Campbell campbell+netbsd-tech-k...@mumble.net wrote: Date: Wed, 9 Apr 2014 11:10:37 +0200 From: J. Hannken-Illjes hann...@eis.cs.tu-bs.de There is no need to do this VI_CHANGING etc. here. Primary goal of the vnode cache is to always initialise

Re: Vnode API change: add global vnode cache

2014-04-09 Thread Chuck Silvers
On Tue, Apr 08, 2014 at 11:23:11AM +0200, J. Hannken-Illjes wrote: On 07 Apr 2014, at 19:28, Chuck Silvers c...@chuq.com wrote: On Sun, Apr 06, 2014 at 12:14:24PM +0200, J. Hannken-Illjes wrote: Currently all file systems have to implement their own cache of vnode / fs node pairs. Most

Re: Vnode API change: add global vnode cache

2014-04-09 Thread J. Hannken-Illjes
On 09 Apr 2014, at 19:09, Chuck Silvers c...@chuq.com wrote: On Tue, Apr 08, 2014 at 11:23:11AM +0200, J. Hannken-Illjes wrote: On 07 Apr 2014, at 19:28, Chuck Silvers c...@chuq.com wrote: On Sun, Apr 06, 2014 at 12:14:24PM +0200, J. Hannken-Illjes wrote: Currently all file systems have

Re: Vnode API change: add global vnode cache

2014-04-08 Thread Alan Barrett
On Tue, 08 Apr 2014, Mindaugas Rasiukevicius wrote: Nothing [in NetBSD] really uses intern. Perhaps not a great naming, but other subsystems usually just use get. Yes, that's a good argument for just using get. --apb (Alan Barrett)

Re: Vnode API change: add global vnode cache

2014-04-08 Thread J. Hannken-Illjes
On 07 Apr 2014, at 19:28, Chuck Silvers c...@chuq.com wrote: On Sun, Apr 06, 2014 at 12:14:24PM +0200, J. Hannken-Illjes wrote: Currently all file systems have to implement their own cache of vnode / fs node pairs. Most file systems use a copy and pasted version of ufs_ihash. So add a

Re: Vnode API change: add global vnode cache

2014-04-07 Thread J. Hannken-Illjes
On 07 Apr 2014, at 03:22, Taylor R Campbell campbell+netbsd-tech-k...@mumble.net wrote: Date: Sun, 6 Apr 2014 12:14:24 +0200 From: J. Hannken-Illjes hann...@eis.cs.tu-bs.de Currently all file systems have to implement their own cache of vnode / fs node pairs. Most file systems use

Re: Vnode API change: add global vnode cache

2014-04-07 Thread Mindaugas Rasiukevicius
J. Hannken-Illjes hann...@eis.cs.tu-bs.de wrote: Currently all file systems have to implement their own cache of vnode / fs node pairs. Most file systems use a copy and pasted version of ufs_ihash. So add a global vnode cache with lookup and remove: ... Diff implementing this for file

Re: Vnode API change: add global vnode cache

2014-04-07 Thread J. Hannken-Illjes
On 07 Apr 2014, at 17:44, Mindaugas Rasiukevicius rm...@netbsd.org wrote: J. Hannken-Illjes hann...@eis.cs.tu-bs.de wrote: Currently all file systems have to implement their own cache of vnode / fs node pairs. Most file systems use a copy and pasted version of ufs_ihash. So add a global

Re: Vnode API change: add global vnode cache

2014-04-07 Thread Taylor R Campbell
Date: Mon, 7 Apr 2014 15:51:00 +0200 From: J. Hannken-Illjes hann...@eis.cs.tu-bs.de On 07 Apr 2014, at 03:22, Taylor R Campbell campbell+netbsd-tech-k...@mumble.net wrote: Is the plan to nix getnewvnode and ungetnewvnode? It would be good to avoid long-term duplication of its

Re: Vnode API change: add global vnode cache

2014-04-07 Thread Taylor R Campbell
Date: Mon, 7 Apr 2014 16:44:47 +0100 From: Mindaugas Rasiukevicius rm...@netbsd.org One concern though: can you benchmark ./build.sh on a machine with 8 or more CPUs? I fear that vcache.lock might be a bit contended and we ought to figure out how much. If the contention is not

Re: Vnode API change: add global vnode cache

2014-04-07 Thread Mindaugas Rasiukevicius
Taylor R Campbell campbell+netbsd-tech-k...@mumble.net wrote: Date: Mon, 7 Apr 2014 16:44:47 +0100 From: Mindaugas Rasiukevicius rm...@netbsd.org One concern though: can you benchmark ./build.sh on a machine with 8 or more CPUs? I fear that vcache.lock might be a bit contended

Re: Vnode API change: add global vnode cache

2014-04-07 Thread Taylor R Campbell
Date: Mon, 7 Apr 2014 17:27:01 +0100 From: Mindaugas Rasiukevicius rm...@netbsd.org Should be fine, but once tmpfs is switched to use vcache, it might become a little bit more visible for some workloads.. Why switch tmpfs to use vcache? That would add memory and time overhead for no

Re: Vnode API change: add global vnode cache

2014-04-07 Thread J. Hannken-Illjes
On 07 Apr 2014, at 18:02, Taylor R Campbell campbell+netbsd-tech-k...@mumble.net wrote: Date: Mon, 7 Apr 2014 15:51:00 +0200 From: J. Hannken-Illjes hann...@eis.cs.tu-bs.de On 07 Apr 2014, at 03:22, Taylor R Campbell campbell+netbsd-tech-k...@mumble.net wrote: Is the plan to nix

Re: Vnode API change: add global vnode cache

2014-04-07 Thread Taylor R Campbell
Date: Mon, 7 Apr 2014 18:32:02 +0200 From: J. Hannken-Illjes hann...@eis.cs.tu-bs.de On 07 Apr 2014, at 18:02, Taylor R Campbell campbell+netbsd-tech-k...@mumble.net wrote: In that case, could you set the VI_CHANGING bit in vp, rather than node-vn_vnode = NULL in the vcache

Re: Vnode API change: add global vnode cache

2014-04-07 Thread J. Hannken-Illjes
On 07 Apr 2014, at 18:38, Taylor R Campbell campbell+netbsd-tech-k...@mumble.net wrote: Date: Mon, 7 Apr 2014 18:32:02 +0200 From: J. Hannken-Illjes hann...@eis.cs.tu-bs.de On 07 Apr 2014, at 18:02, Taylor R Campbell campbell+netbsd-tech-k...@mumble.net wrote: In that case,

Re: Vnode API change: add global vnode cache

2014-04-07 Thread Mindaugas Rasiukevicius
Taylor R Campbell campbell+netbsd-tech-k...@mumble.net wrote: Date: Mon, 7 Apr 2014 17:27:01 +0100 From: Mindaugas Rasiukevicius rm...@netbsd.org Should be fine, but once tmpfs is switched to use vcache, it might become a little bit more visible for some workloads.. Why switch

Re: Vnode API change: add global vnode cache

2014-04-07 Thread Chuck Silvers
On Sun, Apr 06, 2014 at 12:14:24PM +0200, J. Hannken-Illjes wrote: Currently all file systems have to implement their own cache of vnode / fs node pairs. Most file systems use a copy and pasted version of ufs_ihash. So add a global vnode cache with lookup and remove: [...] hi, the

Re: Vnode API change: add global vnode cache

2014-04-07 Thread Greg Troxel
Mindaugas Rasiukevicius rm...@netbsd.org writes: Taylor R Campbell campbell+netbsd-tech-k...@mumble.net wrote: Date: Mon, 7 Apr 2014 16:56:58 +0100 From: Mindaugas Rasiukevicius rm...@netbsd.org What is intern? `Intern' means `lookup, or create and insert if not there'. The

Re: Vnode API change: add global vnode cache

2014-04-07 Thread Alan Barrett
On Mon, 07 Apr 2014, Mindaugas Rasiukevicius wrote: Taylor R Campbell campbell+netbsd-tech-k...@mumble.net wrote: What is intern? `Intern' means `lookup, or create and insert if not there'. The point being is that I do not find it meaningful/intuitive. Many other systems just use get().

Re: Vnode API change: add global vnode cache

2014-04-07 Thread Mindaugas Rasiukevicius
Alan Barrett a...@cequrux.com wrote: On Mon, 07 Apr 2014, Mindaugas Rasiukevicius wrote: Taylor R Campbell campbell+netbsd-tech-k...@mumble.net wrote: What is intern? `Intern' means `lookup, or create and insert if not there'. The point being is that I do not find it

Re: Vnode API change: add global vnode cache

2014-04-07 Thread Thor Lancelot Simon
On Mon, Apr 07, 2014 at 04:04:28PM +, Taylor R Campbell wrote: Date: Mon, 7 Apr 2014 16:44:47 +0100 From: Mindaugas Rasiukevicius rm...@netbsd.org One concern though: can you benchmark ./build.sh on a machine with 8 or more CPUs? I fear that vcache.lock might be a bit

Vnode API change: add global vnode cache

2014-04-06 Thread J. Hannken-Illjes
Currently all file systems have to implement their own cache of vnode / fs node pairs. Most file systems use a copy and pasted version of ufs_ihash. So add a global vnode cache with lookup and remove: /* * Lookup a vnode / fs node pair by key and return it referenced through vpp. */ int

Re: Vnode API change: add global vnode cache

2014-04-06 Thread David Holland
On Sun, Apr 06, 2014 at 12:14:24PM +0200, J. Hannken-Illjes wrote: Currently all file systems have to implement their own cache of vnode / fs node pairs. Most file systems use a copy and pasted version of ufs_ihash. So add a global vnode cache with lookup and remove: Heh, would you

Re: Vnode API change: add global vnode cache

2014-04-06 Thread J. Hannken-Illjes
On 06 Apr 2014, at 21:12, David Holland dholland-t...@netbsd.org wrote: On Sun, Apr 06, 2014 at 12:14:24PM +0200, J. Hannken-Illjes wrote: Currently all file systems have to implement their own cache of vnode / fs node pairs. Most file systems use a copy and pasted version of ufs_ihash.

Re: Vnode API change: add global vnode cache

2014-04-06 Thread Taylor R Campbell
Date: Sun, 6 Apr 2014 12:14:24 +0200 From: J. Hannken-Illjes hann...@eis.cs.tu-bs.de Currently all file systems have to implement their own cache of vnode / fs node pairs. Most file systems use a copy and pasted version of ufs_ihash. So add a global vnode cache with lookup and