Re: [patch] speed up single bio_vec allocation

2006-12-14 Thread Jens Axboe
On Fri, Dec 08 2006, Chen, Kenneth W wrote: > > Chen, Kenneth wrote on Wednesday, December 06, 2006 10:20 AM > > > Jens Axboe wrote on Wednesday, December 06, 2006 2:09 AM > > > This is what I had in mind, in case it wasn't completely clear. Not > > > tested, other than it compiles. Basically it

Re: [patch] speed up single bio_vec allocation

2006-12-14 Thread Jens Axboe
On Fri, Dec 08 2006, Chen, Kenneth W wrote: Chen, Kenneth wrote on Wednesday, December 06, 2006 10:20 AM Jens Axboe wrote on Wednesday, December 06, 2006 2:09 AM This is what I had in mind, in case it wasn't completely clear. Not tested, other than it compiles. Basically it eliminates

RE: [patch] speed up single bio_vec allocation

2006-12-08 Thread Chen, Kenneth W
> Chen, Kenneth wrote on Wednesday, December 06, 2006 10:20 AM > > Jens Axboe wrote on Wednesday, December 06, 2006 2:09 AM > > This is what I had in mind, in case it wasn't completely clear. Not > > tested, other than it compiles. Basically it eliminates the small > > bio_vec pool, and grows the

Re: [patch] speed up single bio_vec allocation

2006-12-08 Thread Jens Axboe
On Thu, Dec 07 2006, Nate Diller wrote: > On 12/7/06, Chen, Kenneth W <[EMAIL PROTECTED]> wrote: > >Nate Diller wrote on Thursday, December 07, 2006 1:46 PM > >> the current code is straightforward and obviously correct. you want > >> to make the alloc/dealloc paths more complex, by

Re: [patch] speed up single bio_vec allocation

2006-12-08 Thread Jens Axboe
On Thu, Dec 07 2006, Nate Diller wrote: On 12/7/06, Chen, Kenneth W [EMAIL PROTECTED] wrote: Nate Diller wrote on Thursday, December 07, 2006 1:46 PM the current code is straightforward and obviously correct. you want to make the alloc/dealloc paths more complex, by special-casing for an

RE: [patch] speed up single bio_vec allocation

2006-12-08 Thread Chen, Kenneth W
Chen, Kenneth wrote on Wednesday, December 06, 2006 10:20 AM Jens Axboe wrote on Wednesday, December 06, 2006 2:09 AM This is what I had in mind, in case it wasn't completely clear. Not tested, other than it compiles. Basically it eliminates the small bio_vec pool, and grows the bio by

Re: [patch] speed up single bio_vec allocation

2006-12-07 Thread Andi Kleen
On Friday 08 December 2006 05:23, Chen, Kenneth W wrote: > Andi Kleen wrote on Thursday, December 07, 2006 6:28 PM > > "Chen, Kenneth W" <[EMAIL PROTECTED]> writes: > > > I tried to use cache_line_size() to find out the alignment of struct bio, > > > but > > > stumbled on that it is a runtime

RE: [patch] speed up single bio_vec allocation

2006-12-07 Thread Chen, Kenneth W
Andi Kleen wrote on Thursday, December 07, 2006 6:28 PM > "Chen, Kenneth W" <[EMAIL PROTECTED]> writes: > > I tried to use cache_line_size() to find out the alignment of struct bio, > > but > > stumbled on that it is a runtime function for x86_64. > > It's a single global variable access: > >

Re: [patch] speed up single bio_vec allocation

2006-12-07 Thread Andi Kleen
"Chen, Kenneth W" <[EMAIL PROTECTED]> writes: > > I tried to use cache_line_size() to find out the alignment of struct bio, but > stumbled on that it is a runtime function for x86_64. It's a single global variable access: #define cache_line_size() (boot_cpu_data.x86_cache_alignment) Or do you

Re: [patch] speed up single bio_vec allocation

2006-12-07 Thread Nate Diller
On 12/7/06, Chen, Kenneth W <[EMAIL PROTECTED]> wrote: Nate Diller wrote on Thursday, December 07, 2006 1:46 PM > the current code is straightforward and obviously correct. you want > to make the alloc/dealloc paths more complex, by special-casing for an > arbitrary limit of "small" I/O,

RE: [patch] speed up single bio_vec allocation

2006-12-07 Thread Chen, Kenneth W
Nate Diller wrote on Thursday, December 07, 2006 1:46 PM > the current code is straightforward and obviously correct. you want > to make the alloc/dealloc paths more complex, by special-casing for an > arbitrary limit of "small" I/O, AFAICT. of *course* you can expect > less overhead when you're

Re: [patch] speed up single bio_vec allocation

2006-12-07 Thread Nate Diller
On 12/7/06, Chen, Kenneth W <[EMAIL PROTECTED]> wrote: Nate Diller wrote on Thursday, December 07, 2006 11:22 AM > > > I still can't help but think we can do better than this, and that this > > > is nothing more than optimizing for a benchmark. For high performance > > > I/O, you will be doing >

RE: [patch] speed up single bio_vec allocation

2006-12-07 Thread Chen, Kenneth W
Nate Diller wrote on Thursday, December 07, 2006 11:22 AM > > > I still can't help but think we can do better than this, and that this > > > is nothing more than optimizing for a benchmark. For high performance > > > I/O, you will be doing > 1 page bio's anyway and this patch wont help > > > you

Re: [patch] speed up single bio_vec allocation

2006-12-07 Thread Nate Diller
On 12/6/06, Chen, Kenneth W <[EMAIL PROTECTED]> wrote: Jens Axboe wrote on Wednesday, December 06, 2006 2:09 AM > > > I will try that too. I'm a bit touchy about sharing a cache line for > > > different bio. But given that there are 200,000 I/O per second we are > > > currently pushing the

Re: [patch] speed up single bio_vec allocation

2006-12-07 Thread Nate Diller
On 12/6/06, Chen, Kenneth W [EMAIL PROTECTED] wrote: Jens Axboe wrote on Wednesday, December 06, 2006 2:09 AM I will try that too. I'm a bit touchy about sharing a cache line for different bio. But given that there are 200,000 I/O per second we are currently pushing the kernel, the

RE: [patch] speed up single bio_vec allocation

2006-12-07 Thread Chen, Kenneth W
Nate Diller wrote on Thursday, December 07, 2006 11:22 AM I still can't help but think we can do better than this, and that this is nothing more than optimizing for a benchmark. For high performance I/O, you will be doing 1 page bio's anyway and this patch wont help you at all.

Re: [patch] speed up single bio_vec allocation

2006-12-07 Thread Nate Diller
On 12/7/06, Chen, Kenneth W [EMAIL PROTECTED] wrote: Nate Diller wrote on Thursday, December 07, 2006 11:22 AM I still can't help but think we can do better than this, and that this is nothing more than optimizing for a benchmark. For high performance I/O, you will be doing 1 page

RE: [patch] speed up single bio_vec allocation

2006-12-07 Thread Chen, Kenneth W
Nate Diller wrote on Thursday, December 07, 2006 1:46 PM the current code is straightforward and obviously correct. you want to make the alloc/dealloc paths more complex, by special-casing for an arbitrary limit of small I/O, AFAICT. of *course* you can expect less overhead when you're doing

Re: [patch] speed up single bio_vec allocation

2006-12-07 Thread Nate Diller
On 12/7/06, Chen, Kenneth W [EMAIL PROTECTED] wrote: Nate Diller wrote on Thursday, December 07, 2006 1:46 PM the current code is straightforward and obviously correct. you want to make the alloc/dealloc paths more complex, by special-casing for an arbitrary limit of small I/O, AFAICT. of

Re: [patch] speed up single bio_vec allocation

2006-12-07 Thread Andi Kleen
Chen, Kenneth W [EMAIL PROTECTED] writes: I tried to use cache_line_size() to find out the alignment of struct bio, but stumbled on that it is a runtime function for x86_64. It's a single global variable access: #define cache_line_size() (boot_cpu_data.x86_cache_alignment) Or do you mean it

RE: [patch] speed up single bio_vec allocation

2006-12-07 Thread Chen, Kenneth W
Andi Kleen wrote on Thursday, December 07, 2006 6:28 PM Chen, Kenneth W [EMAIL PROTECTED] writes: I tried to use cache_line_size() to find out the alignment of struct bio, but stumbled on that it is a runtime function for x86_64. It's a single global variable access: #define

Re: [patch] speed up single bio_vec allocation

2006-12-07 Thread Andi Kleen
On Friday 08 December 2006 05:23, Chen, Kenneth W wrote: Andi Kleen wrote on Thursday, December 07, 2006 6:28 PM Chen, Kenneth W [EMAIL PROTECTED] writes: I tried to use cache_line_size() to find out the alignment of struct bio, but stumbled on that it is a runtime function for

RE: [patch] speed up single bio_vec allocation

2006-12-06 Thread Chen, Kenneth W
Jens Axboe wrote on Wednesday, December 06, 2006 2:09 AM > > > I will try that too. I'm a bit touchy about sharing a cache line for > > > different bio. But given that there are 200,000 I/O per second we are > > > currently pushing the kernel, the chances of two cpu working on two > > > bio that

Re: [patch] speed up single bio_vec allocation

2006-12-06 Thread Jens Axboe
On Wed, Dec 06 2006, Jens Axboe wrote: > I still can't help but think we can do better than this, and that this > is nothing more than optimizing for a benchmark. For high performance > I/O, you will be doing > 1 page bio's anyway and this patch wont help > you at all. Perhaps we can just kill

Re: [patch] speed up single bio_vec allocation

2006-12-06 Thread Jens Axboe
On Mon, Dec 04 2006, Jens Axboe wrote: > On Mon, Dec 04 2006, Chen, Kenneth W wrote: > > > [...] > > > > > > Another idea would be to kill SLAB_HWCACHE_ALIGN (it's pretty pointless, > > > I bet), and always alloc sizeof(*bio) + sizeof(*bvl) in one go when a > > > bio is allocated. It doesn't add

Re: [patch] speed up single bio_vec allocation

2006-12-06 Thread Jens Axboe
On Mon, Dec 04 2006, Jens Axboe wrote: On Mon, Dec 04 2006, Chen, Kenneth W wrote: [...] Another idea would be to kill SLAB_HWCACHE_ALIGN (it's pretty pointless, I bet), and always alloc sizeof(*bio) + sizeof(*bvl) in one go when a bio is allocated. It doesn't add a lot of

Re: [patch] speed up single bio_vec allocation

2006-12-06 Thread Jens Axboe
On Wed, Dec 06 2006, Jens Axboe wrote: I still can't help but think we can do better than this, and that this is nothing more than optimizing for a benchmark. For high performance I/O, you will be doing 1 page bio's anyway and this patch wont help you at all. Perhaps we can just kill bio_vec

RE: [patch] speed up single bio_vec allocation

2006-12-06 Thread Chen, Kenneth W
Jens Axboe wrote on Wednesday, December 06, 2006 2:09 AM I will try that too. I'm a bit touchy about sharing a cache line for different bio. But given that there are 200,000 I/O per second we are currently pushing the kernel, the chances of two cpu working on two bio that sits in the

RE: [patch] speed up single bio_vec allocation

2006-12-04 Thread Chen, Kenneth W
Jens Axboe wrote on Monday, December 04, 2006 12:07 PM > On Mon, Dec 04 2006, Chen, Kenneth W wrote: > > On 64-bit arch like x86_64, struct bio is 104 byte. Since bio slab is > > created with SLAB_HWCACHE_ALIGN flag, there are usually spare memory > > available at the end of bio. I think we can

Re: [patch] speed up single bio_vec allocation

2006-12-04 Thread Jens Axboe
On Mon, Dec 04 2006, Chen, Kenneth W wrote: > > [...] > > > > Another idea would be to kill SLAB_HWCACHE_ALIGN (it's pretty pointless, > > I bet), and always alloc sizeof(*bio) + sizeof(*bvl) in one go when a > > bio is allocated. It doesn't add a lot of overhead even for the case > > where we do

Re: [patch] speed up single bio_vec allocation

2006-12-04 Thread Jens Axboe
On Mon, Dec 04 2006, Chen, Kenneth W wrote: > On 64-bit arch like x86_64, struct bio is 104 byte. Since bio slab is > created with SLAB_HWCACHE_ALIGN flag, there are usually spare memory > available at the end of bio. I think we can utilize that memory for > bio_vec allocation. The purpose is

Re: [patch] speed up single bio_vec allocation

2006-12-04 Thread Jens Axboe
On Mon, Dec 04 2006, Chen, Kenneth W wrote: On 64-bit arch like x86_64, struct bio is 104 byte. Since bio slab is created with SLAB_HWCACHE_ALIGN flag, there are usually spare memory available at the end of bio. I think we can utilize that memory for bio_vec allocation. The purpose is not

Re: [patch] speed up single bio_vec allocation

2006-12-04 Thread Jens Axboe
On Mon, Dec 04 2006, Chen, Kenneth W wrote: [...] Another idea would be to kill SLAB_HWCACHE_ALIGN (it's pretty pointless, I bet), and always alloc sizeof(*bio) + sizeof(*bvl) in one go when a bio is allocated. It doesn't add a lot of overhead even for the case where we do 1 page

RE: [patch] speed up single bio_vec allocation

2006-12-04 Thread Chen, Kenneth W
Jens Axboe wrote on Monday, December 04, 2006 12:07 PM On Mon, Dec 04 2006, Chen, Kenneth W wrote: On 64-bit arch like x86_64, struct bio is 104 byte. Since bio slab is created with SLAB_HWCACHE_ALIGN flag, there are usually spare memory available at the end of bio. I think we can utilize