Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch

2014-08-25 Thread Heikki Linnakangas
On 08/25/2014 12:49 AM, johnlumby wrote: On 08/19/14 18:27, Heikki Linnakangas wrote: Please write the patch without atomic CAS operation. Just use a spinlock. Umm, this is a new criticism I think. Yeah. Be prepared that new issues will crop up as the patch gets slimmer and easier to

Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch

2014-08-24 Thread johnlumby
Thanks for the replies and thoughts. On 08/19/14 18:27, Heikki Linnakangas wrote: On 08/20/2014 12:17 AM, John Lumby wrote: I am attaching a new version of the patch for consideration in the current commit fest. Thanks for working on this! Relative to the one I submitted on 25 June in

Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch

2014-08-19 Thread Heikki Linnakangas
On 08/20/2014 12:17 AM, John Lumby wrote: I am attaching a new version of the patch for consideration in the current commit fest. Thanks for working on this! Relative to the one I submitted on 25 June in bay175-w412ff89303686022a9f16aa3...@phx.gbl the method for handling aio completion

Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch

2014-08-19 Thread Claudio Freire
On Tue, Aug 19, 2014 at 7:27 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: Also, please split prefetching of regular index scans into a separate patch. It's orthogonal to doing async I/O; we could prefetch regular index scans with posix_fadvise too, and AIO should be useful for

Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch

2014-06-25 Thread Claudio Freire
On Tue, Jun 24, 2014 at 12:08 PM, John Lumby johnlu...@hotmail.com wrote: The question is, if you receive the notification of the I/O completion using a signal or a thread, is it safe to release the lwlock from the signal handler or a separate thread? In the forthcoming new version of the

Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch

2014-06-25 Thread Heikki Linnakangas
On 06/25/2014 09:20 PM, Claudio Freire wrote: On Tue, Jun 24, 2014 at 12:08 PM, John Lumby johnlu...@hotmail.com wrote: The question is, if you receive the notification of the I/O completion using a signal or a thread, is it safe to release the lwlock from the signal handler or a separate

Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch

2014-06-25 Thread Andres Freund
On 2014-06-26 00:08:48 +0300, Heikki Linnakangas wrote: LWLocks are implemented with semaphores, so if you can increment a semaphore in the signal handler / callback thread, then in theory you should be able to release a LWLock. I don't think that's a convincing argument even if semop et al

Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch

2014-06-25 Thread John Lumby
My cut'n'pasting failed me at one point corrected below. discussion about what is the difference between a synchronous read versus an asynchronous read as far as non-originator waiting on it is concerned. I thought a bit more about this. There are currently two differences, one of which

Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch

2014-06-24 Thread John Lumby
From: st...@mit.edu Date: Mon, 23 Jun 2014 16:04:50 -0700 Subject: Re: Extended Prefetching using Asynchronous IO - proposal and patch To: johnlu...@hotmail.com CC: klaussfre...@gmail.com; hlinnakan...@vmware.com; pgsql-hackers@postgresql.org On

Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch

2014-06-24 Thread Heikki Linnakangas
On 06/24/2014 04:29 PM, John Lumby wrote: On Mon, Jun 23, 2014 at 2:43 PM, John Lumby johnlu...@hotmail.com wrote: It is when some *other* backend gets there first with the ReadBuffer that things are a bit trickier. The current version of the patch did polling for that case but that drew

Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch

2014-06-24 Thread John Lumby
Thanks Heikki, Date: Tue, 24 Jun 2014 17:02:38 +0300 From: hlinnakan...@vmware.com To: johnlu...@hotmail.com; st...@mit.edu CC: klaussfre...@gmail.com; pgsql-hackers@postgresql.org Subject: Re: Extended Prefetching using Asynchronous IO - proposal and

Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch

2014-06-24 Thread Heikki Linnakangas
On 06/24/2014 06:08 PM, John Lumby wrote: The question is, if you receive the notification of the I/O completion using a signal or a thread, is it safe to release the lwlock from the signal handler or a separate thread? In the forthcoming new version of the patch that uses sigevent, the

Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch

2014-06-23 Thread Jim Nasby
On 6/20/14, 5:12 PM, John Lumby wrote: I also appreciate it is not easy to review the patch. There are really 4 (or maybe 5) parts : . async io (librt functions) . buffer management (allocating, locking and pinning etc) . scanners making prefetch calls .

Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch

2014-06-23 Thread John Lumby
Date: Thu, 19 Jun 2014 15:43:44 -0300 Subject: Re: Extended Prefetching using Asynchronous IO - proposal and patch From: klaussfre...@gmail.com To: st...@mit.edu CC: hlinnakan...@vmware.com; johnlu...@hotmail.com; pgsql-hackers@postgresql.org On

Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch

2014-06-23 Thread Greg Stark
On Mon, Jun 23, 2014 at 2:43 PM, John Lumby johnlu...@hotmail.com wrote: It is when some *other* backend gets there first with the ReadBuffer that things are a bit trickier.The current version of the patch did polling for that case but that drew criticism,and so an imminent new

Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch

2014-06-20 Thread John Lumby
, 20 Jun 2014 04:21:19 +0900 Subject: Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch From: masao.fu...@gmail.com To: johnlu...@hotmail.com CC: pgsql-hackers@postgresql.org; klaussfre...@gmail.com On Mon, Jun 9, 2014 at 11:12 AM, johnlumby johnlu...@hotmail.com

Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch

2014-06-19 Thread Greg Stark
On Wed, May 28, 2014 at 2:19 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: How portable is POSIX aio nowadays? Googling around, it still seems that on Linux, it's implemented using threads. Does the thread-emulation implementation cause problems with the rest of the backend, which

Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch

2014-06-19 Thread Claudio Freire
On Thu, Jun 19, 2014 at 2:49 PM, Greg Stark st...@mit.edu wrote: I don't think the threaded implementation on Linux is the one to use though. I find this *super* confusing but the kernel definitely supports aio syscalls, glibc also has a threaded implementation it uses if run on a kernel that

Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch

2014-06-19 Thread Fujii Masao
On Mon, Jun 9, 2014 at 11:12 AM, johnlumby johnlu...@hotmail.com wrote: updated version of patch compatible with git head of 140608, (adjusted proc oid and a couple of minor fixes) Compilation of patched version on MacOS failed. The error messages were gcc -O0 -Wall -Wmissing-prototypes

Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch

2014-06-09 Thread Claudio Freire
I'm having trouble setting max_async_io_prefetchers in postgresql.conf It says it cannot be changed. Is that fixed at initdb time? (sounds like a bad idea if it is) On Sun, Jun 8, 2014 at 11:12 PM, johnlumby johnlu...@hotmail.com wrote: updated version of patch compatible with git head of

Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch

2014-06-01 Thread Heikki Linnakangas
On 06/01/2014 03:44 AM, johnlumby wrote: If you look at the new patch, you'll see that for the different-pid case, I still call aio_suspend with a timeout. As you or Claudio pointed out earlier,it could just as well sleep for the same timeout, but the small advantage of calling aio_suspend

Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch

2014-05-31 Thread johnlumby
On 05/31/14 20:44, johnlumby wrote: On 05/30/14 09:36, Claudio Freire wrote: Good point. I have included the guts of your little test program (modified to do polling) into the existing autoconf test program that decides on the #define USE_AIO_ATOMIC_BUILTIN_COMP_SWAP. See

Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch

2014-05-31 Thread Claudio Freire
On Sat, May 31, 2014 at 9:44 PM, johnlumby johnlu...@hotmail.com wrote: I'll try to do some measuring of performance with: a) git head b) git head + patch as-is c) git head + patch without aio_suspend in foreign processes (just re-read) d) git head + patch with a lwlock (or whatever works)

Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch

2014-05-30 Thread Heikki Linnakangas
On 05/30/2014 12:53 AM, John Lumby wrote: Date: Thu, 29 May 2014 18:00:28 -0300 Subject: Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch From: klaussfre...@gmail.com To: hlinnakan...@vmware.com CC: johnlu...@hotmail.com; pgsql-hackers@postgresql.org On Thu, May 29

Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch

2014-05-30 Thread Claudio Freire
On Fri, May 30, 2014 at 4:15 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: We don't want polling... And even if we did, calling aio_suspend() in a way that's known to be broken, in a loop, is a pretty crappy way of polling. Well, as mentioned earlier, it is not broken. Whether

Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch

2014-05-29 Thread John Lumby
Thanks for looking at it! Date: Thu, 29 May 2014 00:19:33 +0300 From: hlinnakan...@vmware.com To: johnlu...@hotmail.com; pgsql-hackers@postgresql.org CC: klaussfre...@gmail.com Subject: Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch On 05/28/2014 11:52 PM

Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch

2014-05-29 Thread John Lumby
(and it sounds as though Claudio is on it too). The area of exactly what the best prefetch strategy should be for each particular type of scan and context is a good one to work on. John Date: Wed, 28 May 2014 18:12:23 -0700 Subject: Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal

Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch

2014-05-29 Thread Claudio Freire
El 28/05/2014 22:12, Peter Geoghegan p...@heroku.com escribió: On Wed, May 28, 2014 at 5:59 PM, Claudio Freire klaussfre...@gmail.com wrote: For nestloop, correct me if I'm wrong, but index scan nodes don't have visibility of the next tuple to be searched for. Nested loop joins are

Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch

2014-05-29 Thread Heikki Linnakangas
On 05/29/2014 04:12 PM, John Lumby wrote: Thanks for looking at it! Date: Thu, 29 May 2014 00:19:33 +0300 From: hlinnakan...@vmware.com To: johnlu...@hotmail.com; pgsql-hackers@postgresql.org CC: klaussfre...@gmail.com Subject: Re: [HACKERS] Extended Prefetching using Asynchronous IO

Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch

2014-05-29 Thread Claudio Freire
...@gmail.com Subject: Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch On 05/28/2014 11:52 PM, John Lumby wrote: The patch seems to assume that you can put the aiocb struct in shared memory, initiate an asynchronous I/O request from one process, and wait for its

Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch

2014-05-29 Thread Heikki Linnakangas
On 05/29/2014 11:34 PM, Claudio Freire wrote: On Thu, May 29, 2014 at 5:23 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 05/29/2014 04:12 PM, John Lumby wrote: On 05/28/2014 11:52 PM, John Lumby wrote: The patch seems to assume that you can put the aiocb struct in shared memory,

Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch

2014-05-29 Thread Claudio Freire
On Thu, May 29, 2014 at 5:39 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 05/29/2014 11:34 PM, Claudio Freire wrote: On Thu, May 29, 2014 at 5:23 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 05/29/2014 04:12 PM, John Lumby wrote: On 05/28/2014 11:52 PM, John Lumby

Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch

2014-05-29 Thread Tom Lane
Claudio Freire klaussfre...@gmail.com writes: Didn't fix that, but the attached patch does fix regression tests when scanning over index types other than btree (was invoking elog when the index am didn't have ampeeknexttuple) ampeeknexttuple? That's a bit scary. It would certainly be unsafe

Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch

2014-05-29 Thread Claudio Freire
On Thu, May 29, 2014 at 6:19 PM, Tom Lane t...@sss.pgh.pa.us wrote: Claudio Freire klaussfre...@gmail.com writes: Didn't fix that, but the attached patch does fix regression tests when scanning over index types other than btree (was invoking elog when the index am didn't have ampeeknexttuple)

Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch

2014-05-29 Thread Claudio Freire
On Thu, May 29, 2014 at 6:43 PM, Claudio Freire klaussfre...@gmail.com wrote: On Thu, May 29, 2014 at 6:19 PM, Tom Lane t...@sss.pgh.pa.us wrote: Claudio Freire klaussfre...@gmail.com writes: Didn't fix that, but the attached patch does fix regression tests when scanning over index types other

Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch

2014-05-29 Thread John Lumby
Date: Thu, 29 May 2014 18:00:28 -0300 Subject: Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch From: klaussfre...@gmail.com To: hlinnakan...@vmware.com CC: johnlu...@hotmail.com; pgsql-hackers@postgresql.org On Thu, May 29, 2014 at 5:39 PM, Heikki

Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch

2014-05-29 Thread Tom Lane
Claudio Freire klaussfre...@gmail.com writes: On Thu, May 29, 2014 at 6:43 PM, Claudio Freire klaussfre...@gmail.com wrote: On Thu, May 29, 2014 at 6:19 PM, Tom Lane t...@sss.pgh.pa.us wrote: ampeeknexttuple? That's a bit scary. It would certainly be unsafe for non-MVCC snapshots (read

Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch

2014-05-29 Thread John Lumby
From: t...@sss.pgh.pa.us To: klaussfre...@gmail.com CC: hlinnakan...@vmware.com; johnlu...@hotmail.com; pgsql-hackers@postgresql.org Subject: Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch Date: Thu, 29 May 2014 17:56:57 -0400 Claudio Freire klaussfre

Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch

2014-05-29 Thread Claudio Freire
On Thu, May 29, 2014 at 6:56 PM, Tom Lane t...@sss.pgh.pa.us wrote: Claudio Freire klaussfre...@gmail.com writes: On Thu, May 29, 2014 at 6:43 PM, Claudio Freire klaussfre...@gmail.com wrote: On Thu, May 29, 2014 at 6:19 PM, Tom Lane t...@sss.pgh.pa.us wrote: ampeeknexttuple? That's a bit

Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch

2014-05-29 Thread Claudio Freire
On Thu, May 29, 2014 at 7:11 PM, John Lumby johnlu...@hotmail.com wrote: Nonetheless, getting the next tid out of the index may involve stepping to the next index page, at which point you've lost your interlock I think we are ok as peeknexttuple (yes bad name, sorry, can change it ...

Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch

2014-05-29 Thread Andres Freund
Hi, On 2014-05-29 17:53:51 -0400, John Lumby wrote: to see how we have done it. And I am attaching corrected version of your test program which runs just fine. It's perfectly fine to not be up to the coding style at this point, but trying to adhere to it to some degree will make code review

Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch

2014-05-29 Thread John Lumby
Date: Thu, 29 May 2014 18:00:28 -0300 Subject: Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch From: klaussfre...@gmail.com To: hlinnakan...@vmware.com CC: johnlu...@hotmail.com; pgsql-hackers@postgresql.org Even if it worked on Linux today, it would

Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch

2014-05-29 Thread Claudio Freire
On Thu, May 29, 2014 at 6:53 PM, John Lumby johnlu...@hotmail.com wrote: Well, as mentioned earlier, it is not broken. Whether it is efficient I am not sure. I have looked at the mutex in aio_suspend that you mentioned and I am not quite convinced that, if caller is not the original

Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch

2014-05-29 Thread Claudio Freire
On Thu, May 29, 2014 at 7:26 PM, Claudio Freire klaussfre...@gmail.com wrote: 1) If it's the same process, wait for the full timeout (no looping). If you have to loop (EAGAIN or EINTR - which I just noticed you don't check for), that's ok. Sorry, meant to say just looping on EINTR. About the

Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch

2014-05-28 Thread Heikki Linnakangas
On 05/28/2014 11:52 PM, John Lumby wrote: The patch is attached. It is based on clone of today's 9.4dev source. I have noticed that this source is (not suprisingly) quite a moving target at present, meaning that this patch becomes stale quite quickly. So although this copy is fine for

Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch

2014-05-28 Thread Peter Geoghegan
On Tue, May 27, 2014 at 3:17 PM, John Lumby johnlu...@hotmail.com wrote: Below I am pasting the README we have written for this new functionality which mentions some of the measurements, advantages (and disadvantages) and we welcome all and any comments on this. I think that this is likely to

Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch

2014-05-28 Thread Claudio Freire
On Wed, May 28, 2014 at 6:51 PM, Peter Geoghegan p...@heroku.com wrote: Have you thought about things like specialized prefetching for nested loop joins? Currently, such a thing would need some non-trivial changes to the execution nodes, I believe. For nestloop, correct me if I'm wrong, but

Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch

2014-05-28 Thread Peter Geoghegan
On Wed, May 28, 2014 at 5:59 PM, Claudio Freire klaussfre...@gmail.com wrote: For nestloop, correct me if I'm wrong, but index scan nodes don't have visibility of the next tuple to be searched for. Nested loop joins are considered a particularly compelling case for prefetching, actually. --