Re: [rfc][patch] dynamic resizing dentry hash using RCU

2007-03-04 Thread Nick Piggin
On Mon, Mar 05, 2007 at 05:27:24AM +0100, Nick Piggin wrote: > On Sun, Mar 04, 2007 at 08:11:42PM -0800, David Miller wrote: > > One minor nit: > > > > > +struct dentry_hash { > > > + unsigned int shift; > > > + unsigned long mask; > > > + struct hlist_head *table; > > > +}; > > > > I don't see

Re: [rfc][patch] dynamic resizing dentry hash using RCU

2007-03-04 Thread David Miller
From: Nick Piggin <[EMAIL PROTECTED]> Date: Mon, 5 Mar 2007 05:27:24 +0100 > Sounds great, I would be happy to help review it. If we can create a > bit of common infrastructure, the dcache conversion might become a bit > more palatable and we could look at other things like the inode hash > as

Re: [rfc][patch] dynamic resizing dentry hash using RCU

2007-03-04 Thread Nick Piggin
On Sun, Mar 04, 2007 at 08:11:42PM -0800, David Miller wrote: > From: Nick Piggin <[EMAIL PROTECTED]> > Date: Fri, 23 Feb 2007 16:37:43 +0100 > > > So I introduce a new method for resizing hash tables with RCU, and apply > > that to the dentry hash. > > Thanks for doing this work Nick. I'm

Re: [rfc][patch] dynamic resizing dentry hash using RCU

2007-03-04 Thread David Miller
From: Nick Piggin <[EMAIL PROTECTED]> Date: Fri, 23 Feb 2007 16:37:43 +0100 > So I introduce a new method for resizing hash tables with RCU, and apply > that to the dentry hash. Thanks for doing this work Nick. I'm going to take your ideas and apply them to an ipv4 routing cache dynamic growth

Re: [rfc][patch] dynamic resizing dentry hash using RCU

2007-03-04 Thread David Miller
From: Nick Piggin [EMAIL PROTECTED] Date: Fri, 23 Feb 2007 16:37:43 +0100 So I introduce a new method for resizing hash tables with RCU, and apply that to the dentry hash. Thanks for doing this work Nick. I'm going to take your ideas and apply them to an ipv4 routing cache dynamic growth

Re: [rfc][patch] dynamic resizing dentry hash using RCU

2007-03-04 Thread Nick Piggin
On Sun, Mar 04, 2007 at 08:11:42PM -0800, David Miller wrote: From: Nick Piggin [EMAIL PROTECTED] Date: Fri, 23 Feb 2007 16:37:43 +0100 So I introduce a new method for resizing hash tables with RCU, and apply that to the dentry hash. Thanks for doing this work Nick. I'm going to take

Re: [rfc][patch] dynamic resizing dentry hash using RCU

2007-03-04 Thread David Miller
From: Nick Piggin [EMAIL PROTECTED] Date: Mon, 5 Mar 2007 05:27:24 +0100 Sounds great, I would be happy to help review it. If we can create a bit of common infrastructure, the dcache conversion might become a bit more palatable and we could look at other things like the inode hash as well.

Re: [rfc][patch] dynamic resizing dentry hash using RCU

2007-03-04 Thread Nick Piggin
On Mon, Mar 05, 2007 at 05:27:24AM +0100, Nick Piggin wrote: On Sun, Mar 04, 2007 at 08:11:42PM -0800, David Miller wrote: One minor nit: +struct dentry_hash { + unsigned int shift; + unsigned long mask; + struct hlist_head *table; +}; I don't see any reason to make

Re: [rfc][patch] dynamic resizing dentry hash using RCU

2007-02-24 Thread Paul E. McKenney
On Fri, Feb 23, 2007 at 04:37:43PM +0100, Nick Piggin wrote: > > The dentry hash uses up 8MB for 1 million entries on my 4GB system is one > of the biggest wasters of memory for me. Because I rarely have more than one > or > two hundred thousand dentries. And that's with several kernel trees

Re: [rfc][patch] dynamic resizing dentry hash using RCU

2007-02-24 Thread William Lee Irwin III
> From: William Lee Irwin III <[EMAIL PROTECTED]> > Date: Sat, 24 Feb 2007 14:56:31 -0800 >> just do it on a per-directory basis so you don't intermix children >> of different parents in some boot-time -allocated global trainwreck >> and you're home free. Benchmarking is probably needed to gauge

Re: [rfc][patch] dynamic resizing dentry hash using RCU

2007-02-24 Thread David Miller
From: William Lee Irwin III <[EMAIL PROTECTED]> Date: Sat, 24 Feb 2007 14:56:31 -0800 > just do it on a per-directory basis so you don't intermix children > of different parents in some boot-time -allocated global trainwreck > and you're home free. Benchmarking is probably needed to gauge >

Re: [rfc][patch] dynamic resizing dentry hash using RCU

2007-02-24 Thread William Lee Irwin III
On Fri, Feb 23, 2007 at 08:24:44PM -0800, William Lee Irwin III wrote: >> You would be better served by a data structure different from a hashtable. On Sat, Feb 24, 2007 at 06:09:37AM +0100, Nick Piggin wrote: > Out of curiosity, what better data structure do you have in mind for > the dentry

Re: [rfc][patch] dynamic resizing dentry hash using RCU

2007-02-24 Thread William Lee Irwin III
On Fri, Feb 23, 2007 at 08:24:44PM -0800, William Lee Irwin III wrote: You would be better served by a data structure different from a hashtable. On Sat, Feb 24, 2007 at 06:09:37AM +0100, Nick Piggin wrote: Out of curiosity, what better data structure do you have in mind for the dentry hash?

Re: [rfc][patch] dynamic resizing dentry hash using RCU

2007-02-24 Thread David Miller
From: William Lee Irwin III [EMAIL PROTECTED] Date: Sat, 24 Feb 2007 14:56:31 -0800 just do it on a per-directory basis so you don't intermix children of different parents in some boot-time -allocated global trainwreck and you're home free. Benchmarking is probably needed to gauge which

Re: [rfc][patch] dynamic resizing dentry hash using RCU

2007-02-24 Thread William Lee Irwin III
From: William Lee Irwin III [EMAIL PROTECTED] Date: Sat, 24 Feb 2007 14:56:31 -0800 just do it on a per-directory basis so you don't intermix children of different parents in some boot-time -allocated global trainwreck and you're home free. Benchmarking is probably needed to gauge which

Re: [rfc][patch] dynamic resizing dentry hash using RCU

2007-02-24 Thread Paul E. McKenney
On Fri, Feb 23, 2007 at 04:37:43PM +0100, Nick Piggin wrote: The dentry hash uses up 8MB for 1 million entries on my 4GB system is one of the biggest wasters of memory for me. Because I rarely have more than one or two hundred thousand dentries. And that's with several kernel trees worth of

Re: [rfc][patch] dynamic resizing dentry hash using RCU

2007-02-23 Thread Nick Piggin
On Sat, Feb 24, 2007 at 01:07:23PM +0900, KAMEZAWA Hiroyuki wrote: > On Fri, 23 Feb 2007 16:37:43 +0100 > Nick Piggin <[EMAIL PROTECTED]> wrote: > > > +static void dcache_hash_resize(unsigned int new_shift); > > +static void mod_nr_dentry(int mod) > > +{ > > + unsigned long dentry_size; > > + >

Re: [rfc][patch] dynamic resizing dentry hash using RCU

2007-02-23 Thread Nick Piggin
On Fri, Feb 23, 2007 at 08:24:44PM -0800, William Lee Irwin III wrote: > On Fri, Feb 23, 2007 at 04:37:43PM +0100, Nick Piggin wrote: > > The dentry hash uses up 8MB for 1 million entries on my 4GB system is > > one of the biggest wasters of memory for me. Because I rarely have > > more than one

Re: [rfc][patch] dynamic resizing dentry hash using RCU

2007-02-23 Thread William Lee Irwin III
On Fri, Feb 23, 2007 at 04:37:43PM +0100, Nick Piggin wrote: > The dentry hash uses up 8MB for 1 million entries on my 4GB system is > one of the biggest wasters of memory for me. Because I rarely have > more than one or two hundred thousand dentries. And that's with > several kernel trees worth

Re: [rfc][patch] dynamic resizing dentry hash using RCU

2007-02-23 Thread KAMEZAWA Hiroyuki
On Fri, 23 Feb 2007 16:37:43 +0100 Nick Piggin <[EMAIL PROTECTED]> wrote: > +static void dcache_hash_resize(unsigned int new_shift); > +static void mod_nr_dentry(int mod) > +{ > + unsigned long dentry_size; > + > + dentry_stat.nr_dentry += mod; > + > + dentry_size = 1 <<

Re: [rfc][patch] dynamic resizing dentry hash using RCU

2007-02-23 Thread Nick Piggin
On Sat, Feb 24, 2007 at 02:26:02AM +0100, Nick Piggin wrote: > On Fri, Feb 23, 2007 at 09:25:28AM -0800, Zach Brown wrote: > > > > On Feb 23, 2007, at 7:37 AM, Nick Piggin wrote: > > > > > > > >The dentry hash uses up 8MB for 1 million entries on my 4GB system > > >is one > > >of the biggest

Re: [rfc][patch] dynamic resizing dentry hash using RCU

2007-02-23 Thread Nick Piggin
On Fri, Feb 23, 2007 at 05:31:30PM -0800, Michael K. Edwards wrote: > On 2/23/07, Zach Brown <[EMAIL PROTECTED]> wrote: > >I'd love to see a generic implementation of RCU hashing that > >subsystems can then take advantage of. It's long been on the fun > >side of my todo list. The side I never

Re: [rfc][patch] dynamic resizing dentry hash using RCU

2007-02-23 Thread Michael K. Edwards
On 2/23/07, Zach Brown <[EMAIL PROTECTED]> wrote: I'd love to see a generic implementation of RCU hashing that subsystems can then take advantage of. It's long been on the fun side of my todo list. The side I never get to :/. There's an active thread on netdev about implementing an RCU hash.

Re: [rfc][patch] dynamic resizing dentry hash using RCU

2007-02-23 Thread Nick Piggin
On Fri, Feb 23, 2007 at 09:25:28AM -0800, Zach Brown wrote: > > On Feb 23, 2007, at 7:37 AM, Nick Piggin wrote: > > > > >The dentry hash uses up 8MB for 1 million entries on my 4GB system > >is one > >of the biggest wasters of memory for me. Because I rarely have more > >than one or > >two

Re: [rfc][patch] dynamic resizing dentry hash using RCU

2007-02-23 Thread Nick Piggin
On Fri, Feb 23, 2007 at 05:31:17PM +0100, Eric Dumazet wrote: > On Friday 23 February 2007 16:37, Nick Piggin wrote: > > The dentry hash uses up 8MB for 1 million entries on my 4GB system is one > > of the biggest wasters of memory for me. Because I rarely have more than > > one or two hundred

Re: [rfc][patch] dynamic resizing dentry hash using RCU

2007-02-23 Thread Zach Brown
On Feb 23, 2007, at 7:37 AM, Nick Piggin wrote: The dentry hash uses up 8MB for 1 million entries on my 4GB system is one of the biggest wasters of memory for me. Because I rarely have more than one or two hundred thousand dentries. And that's with several kernel trees worth of entries.

Re: [rfc][patch] dynamic resizing dentry hash using RCU

2007-02-23 Thread Eric Dumazet
On Friday 23 February 2007 16:37, Nick Piggin wrote: > The dentry hash uses up 8MB for 1 million entries on my 4GB system is one > of the biggest wasters of memory for me. Because I rarely have more than > one or two hundred thousand dentries. And that's with several kernel trees > worth of

[rfc][patch] dynamic resizing dentry hash using RCU

2007-02-23 Thread Nick Piggin
The dentry hash uses up 8MB for 1 million entries on my 4GB system is one of the biggest wasters of memory for me. Because I rarely have more than one or two hundred thousand dentries. And that's with several kernel trees worth of entries. Most desktop and probably even many types of servers will

[rfc][patch] dynamic resizing dentry hash using RCU

2007-02-23 Thread Nick Piggin
The dentry hash uses up 8MB for 1 million entries on my 4GB system is one of the biggest wasters of memory for me. Because I rarely have more than one or two hundred thousand dentries. And that's with several kernel trees worth of entries. Most desktop and probably even many types of servers will

Re: [rfc][patch] dynamic resizing dentry hash using RCU

2007-02-23 Thread Eric Dumazet
On Friday 23 February 2007 16:37, Nick Piggin wrote: The dentry hash uses up 8MB for 1 million entries on my 4GB system is one of the biggest wasters of memory for me. Because I rarely have more than one or two hundred thousand dentries. And that's with several kernel trees worth of entries.

Re: [rfc][patch] dynamic resizing dentry hash using RCU

2007-02-23 Thread Zach Brown
On Feb 23, 2007, at 7:37 AM, Nick Piggin wrote: The dentry hash uses up 8MB for 1 million entries on my 4GB system is one of the biggest wasters of memory for me. Because I rarely have more than one or two hundred thousand dentries. And that's with several kernel trees worth of entries.

Re: [rfc][patch] dynamic resizing dentry hash using RCU

2007-02-23 Thread Nick Piggin
On Fri, Feb 23, 2007 at 05:31:17PM +0100, Eric Dumazet wrote: On Friday 23 February 2007 16:37, Nick Piggin wrote: The dentry hash uses up 8MB for 1 million entries on my 4GB system is one of the biggest wasters of memory for me. Because I rarely have more than one or two hundred thousand

Re: [rfc][patch] dynamic resizing dentry hash using RCU

2007-02-23 Thread Nick Piggin
On Fri, Feb 23, 2007 at 09:25:28AM -0800, Zach Brown wrote: On Feb 23, 2007, at 7:37 AM, Nick Piggin wrote: The dentry hash uses up 8MB for 1 million entries on my 4GB system is one of the biggest wasters of memory for me. Because I rarely have more than one or two hundred thousand

Re: [rfc][patch] dynamic resizing dentry hash using RCU

2007-02-23 Thread Michael K. Edwards
On 2/23/07, Zach Brown [EMAIL PROTECTED] wrote: I'd love to see a generic implementation of RCU hashing that subsystems can then take advantage of. It's long been on the fun side of my todo list. The side I never get to :/. There's an active thread on netdev about implementing an RCU hash.

Re: [rfc][patch] dynamic resizing dentry hash using RCU

2007-02-23 Thread Nick Piggin
On Fri, Feb 23, 2007 at 05:31:30PM -0800, Michael K. Edwards wrote: On 2/23/07, Zach Brown [EMAIL PROTECTED] wrote: I'd love to see a generic implementation of RCU hashing that subsystems can then take advantage of. It's long been on the fun side of my todo list. The side I never get to :/.

Re: [rfc][patch] dynamic resizing dentry hash using RCU

2007-02-23 Thread Nick Piggin
On Sat, Feb 24, 2007 at 02:26:02AM +0100, Nick Piggin wrote: On Fri, Feb 23, 2007 at 09:25:28AM -0800, Zach Brown wrote: On Feb 23, 2007, at 7:37 AM, Nick Piggin wrote: The dentry hash uses up 8MB for 1 million entries on my 4GB system is one of the biggest wasters of memory

Re: [rfc][patch] dynamic resizing dentry hash using RCU

2007-02-23 Thread KAMEZAWA Hiroyuki
On Fri, 23 Feb 2007 16:37:43 +0100 Nick Piggin [EMAIL PROTECTED] wrote: +static void dcache_hash_resize(unsigned int new_shift); +static void mod_nr_dentry(int mod) +{ + unsigned long dentry_size; + + dentry_stat.nr_dentry += mod; + + dentry_size = 1 dentry_hash-shift; +

Re: [rfc][patch] dynamic resizing dentry hash using RCU

2007-02-23 Thread William Lee Irwin III
On Fri, Feb 23, 2007 at 04:37:43PM +0100, Nick Piggin wrote: The dentry hash uses up 8MB for 1 million entries on my 4GB system is one of the biggest wasters of memory for me. Because I rarely have more than one or two hundred thousand dentries. And that's with several kernel trees worth of

Re: [rfc][patch] dynamic resizing dentry hash using RCU

2007-02-23 Thread Nick Piggin
On Fri, Feb 23, 2007 at 08:24:44PM -0800, William Lee Irwin III wrote: On Fri, Feb 23, 2007 at 04:37:43PM +0100, Nick Piggin wrote: The dentry hash uses up 8MB for 1 million entries on my 4GB system is one of the biggest wasters of memory for me. Because I rarely have more than one or two

Re: [rfc][patch] dynamic resizing dentry hash using RCU

2007-02-23 Thread Nick Piggin
On Sat, Feb 24, 2007 at 01:07:23PM +0900, KAMEZAWA Hiroyuki wrote: On Fri, 23 Feb 2007 16:37:43 +0100 Nick Piggin [EMAIL PROTECTED] wrote: +static void dcache_hash_resize(unsigned int new_shift); +static void mod_nr_dentry(int mod) +{ + unsigned long dentry_size; + +