Re: [HACKERS] Improve lseek scalability v3

2011-09-21 Thread Marco Stornelli
2011/9/19 Matthew Wilcox matt...@wil.cx: On Mon, Sep 19, 2011 at 08:31:00AM -0400, Stephen Frost wrote: * Benjamin LaHaise (b...@kvack.org) wrote: For such tables, can't Postgres track the size of the file internally?  I'm assuming it's keeping file descriptors open on the tables it manages,

Re: [HACKERS] Improve lseek scalability v3

2011-09-19 Thread Benjamin LaHaise
On Fri, Sep 16, 2011 at 07:27:33PM +0200, Andres Freund wrote: many tuples does the table have. Those statistics are only updated every now and then though. So it uses those old stats to check how many tuples are normally stored on a page and then uses that to extrapolate the number of

Re: [HACKERS] Improve lseek scalability v3

2011-09-19 Thread Stephen Frost
* Benjamin LaHaise (b...@kvack.org) wrote: For such tables, can't Postgres track the size of the file internally? I'm assuming it's keeping file descriptors open on the tables it manages, in which case when it writes to a file to extend it, the internally stored size could be updated. Not

Re: [HACKERS] Improve lseek scalability v3

2011-09-19 Thread Robert Haas
On Mon, Sep 19, 2011 at 8:31 AM, Stephen Frost sfr...@snowman.net wrote: * Benjamin LaHaise (b...@kvack.org) wrote: For such tables, can't Postgres track the size of the file internally?  I'm assuming it's keeping file descriptors open on the tables it manages, in which case when it writes to

Re: [HACKERS] Improve lseek scalability v3

2011-09-19 Thread Matthew Wilcox
On Mon, Sep 19, 2011 at 08:31:00AM -0400, Stephen Frost wrote: * Benjamin LaHaise (b...@kvack.org) wrote: For such tables, can't Postgres track the size of the file internally? I'm assuming it's keeping file descriptors open on the tables it manages, in which case when it writes to a

Re: [HACKERS] Improve lseek scalability v3

2011-09-16 Thread Andres Freund
Hi, On Friday 16 Sep 2011 17:36:20 Matthew Wilcox wrote: On Fri, Sep 16, 2011 at 04:16:49PM +0200, Andres Freund wrote: I sent an email containing benchmarks from Robert Haas regarding the Subject. Looking at lkml.org I can't see it right now, Will recheck when I am at home. He

Re: [HACKERS] Improve lseek scalability v3

2011-09-16 Thread Alvaro Herrera
Excerpts from Andres Freund's message of vie sep 16 14:27:33 -0300 2011: Hi, On Friday 16 Sep 2011 17:36:20 Matthew Wilcox wrote: Does the query planner need to know the exact number of bytes in the file, or is it after an order-of-magnitude? Or to-the-nearest-gigabyte? It depends on

Re: [HACKERS] Improve lseek scalability v3

2011-09-16 Thread Andres Freund
On Friday, September 16, 2011 10:08:17 PM Benjamin LaHaise wrote: On Fri, Sep 16, 2011 at 07:27:33PM +0200, Andres Freund wrote: many tuples does the table have. Those statistics are only updated every now and then though. So it uses those old stats to check how many tuples are normally

Re: [HACKERS] Improve lseek scalability v3

2011-09-16 Thread Andres Freund
On Friday, September 16, 2011 11:02:38 PM Andres Freund wrote: Also with fstat() instead of lseek() there was no bottleneck anymore, so I don't think the benefits would warrant that. At least thats what I observed on a 4 x 6 machine without the patch applied (can't reboot it). That shouldn't

Re: [HACKERS] Improve lseek scalability v3

2011-09-16 Thread Greg Stark
On Fri, Sep 16, 2011 at 9:08 PM, Benjamin LaHaise b...@kvack.org wrote: For such tables, can't Postgres track the size of the file internally?  I'm assuming it's keeping file descriptors open on the tables it manages, in which case when it writes to a file to extend it, the internally stored

Re: [HACKERS] Improve lseek scalability v3

2011-09-16 Thread Andi Kleen
One other thing we're interested in is portability. I mean, even if Linux were to introduce a new hypothetical syscall that was able to return the file size at a ridiculously low cost, we probably wouldn't use it because it'd be Linux-specific. So an improvement of lseek() seems to be the

Re: [HACKERS] Improve lseek scalability v3

2011-09-16 Thread Matthew Wilcox
On Fri, Sep 16, 2011 at 04:16:49PM +0200, Andres Freund wrote: I sent an email containing benchmarks from Robert Haas regarding the Subject. Looking at lkml.org I can't see it right now, Will recheck when I am at home. He replaced lseek(SEEK_END) with fstat() and got speedups up to 8.7 times