[HACKERS] lseek/read/write overhead becomes visible at scale ..

2017-01-24 Thread Tobias Oberstein
Hi guys, pls bare with me, this is my first post here. Pls also excuse the length .. I was trying to do all my homework before posting here;) The overhead of lseek/read/write vs pread/pwrite (or even pvread/pvwrite) was previously discussed here https://www.postgresql.org/message-id/flat/CA

Re: [HACKERS] lseek/read/write overhead becomes visible at scale ..

2017-01-24 Thread Tobias Oberstein
Hi, Switching to sync engine, it drops to 9.1 mio - but the system load then is also much higher! I doubt those have very much to do with postgres - I'd quite strongly In the machine in production, we see 8kB reads in the 300k-650k/s range. In spikes, because, yes, due to the 3TB RAM, we ha

Re: [HACKERS] lseek/read/write overhead becomes visible at scale ..

2017-01-24 Thread Tobias Oberstein
Hi, Am 24.01.2017 um 18:41 schrieb Andres Freund: Hi, On 2017-01-24 18:37:14 +0100, Tobias Oberstein wrote: assume that it'd get more than swamped with doing actualy work, and with buffering the frequently accessed stuff in memory. What I am trying to say is: the syscall overhead of

Re: [HACKERS] lseek/read/write overhead becomes visible at scale ..

2017-01-24 Thread Tobias Oberstein
Hi, pid |syscall| cnt | cnt_per_sec -+---+-+- | syscalls:sys_enter_lseek | 4091584 | 136386 | syscalls:sys_enter_newfstat | 2054988 | 68500 | syscalls

Re: [HACKERS] lseek/read/write overhead becomes visible at scale ..

2017-01-25 Thread Tobias Oberstein
Hi Alvaro, Am 24.01.2017 um 19:36 schrieb Alvaro Herrera: Tobias Oberstein wrote: I am benchmarking IOPS, and while doing so, it becomes apparent that at these scales it does matter _how_ IO is done. The most efficient way is libaio. I get 9.7 million/sec IOPS with low CPU load. Using any

Re: [HACKERS] lseek/read/write overhead becomes visible at scale ..

2017-01-25 Thread Tobias Oberstein
Hi Andres, Using pread instead of lseek+read halfes the syscalls. I really don't understand what you are fighting here .. Sure, there's some overhead. And as I said upthread, I'm much less against this change than Tom. What I'm saying is that your benchmarks haven't shown a benefit in a mean

Re: [HACKERS] lseek/read/write overhead becomes visible at scale ..

2017-01-25 Thread Tobias Oberstein
Hi, Synthetic PG workload or real world production workload? Both might work, production-like has bigger pull, but I'd guess synthetic is good enough. Thanks! The box should get PostgreSQL in the not too distant future. It'll get a backup from prod, but will act as new prod, so it might tak