Re: [HACKERS] Improve lseek scalability v3
On Mon, Sep 19, 2011 at 08:31:00AM -0400, Stephen Frost wrote: > * Benjamin LaHaise (b...@kvack.org) wrote: > > For such tables, can't Postgres track the size of the file internally? I'm > > assuming it's keeping file descriptors open on the tables it manages, in > > which case when it writes to a file to extend it, the internally stored > > size > > could be updated. Not making a syscall at all would scale far better than > > even a modified lseek() will perform. > > We'd have to have it in shared memory and have a lock around it, it > wouldn't be cheap at all. Yep, that makes perfect sense. After all, the kernel does basically the same thing to maintain this information; why should we have userspace duplicating the same infrastructure? I must admit, I'd never heard of this usage of lseek to get the current size of a file before; I'd assumed everybody used fstat. Given this legitimate reason for a high-frequency calling of lseek, I withdraw my earlier objection to the patch series. -- Matthew Wilcox Intel Open Source Technology Centre "Bill, look, we understand that you're interested in selling us this operating system, but compare it to ours. We can't possibly take such a retrograde step." -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Improve lseek scalability v3
On Fri, Sep 16, 2011 at 04:16:49PM +0200, Andres Freund wrote: > I sent an email containing benchmarks from Robert Haas regarding the Subject. > Looking at lkml.org I can't see it right now, Will recheck when I am at home. > > He replaced lseek(SEEK_END) with fstat() and got speedups up to 8.7 times the > lseek performance. > The workload was 64 clients hammering postgres with a simple readonly > workload > (pgbench -S). Yay! Data! > For reference see the thread in the postgres archives which also links to > performance data: http://archives.postgresql.org/message- > id/CA+TgmoawRfpan35wzvgHkSJ0+i-W=vkjpknrxk2ktdr+hsa...@mail.gmail.com So both fstat and lseek do more work than postgres wants. lseek modifies the file pointer while fstat copies all kinds of unnecessary information into userspace. I imagine this is the source of the slowdown seen in the 1-client case. There have been various proposals to make the amount of information returned by fstat limited to the 'cheap' (for various definitions of 'cheap') fields. I'd like to dig into the requirement for knowing the file size a little better. According to the blog entry it's used for "the query planner". Does the query planner need to know the exact number of bytes in the file, or is it after an order-of-magnitude? Or to-the-nearest-gigabyte? -- Matthew Wilcox Intel Open Source Technology Centre "Bill, look, we understand that you're interested in selling us this operating system, but compare it to ours. We can't possibly take such a retrograde step." -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
