On 08/25/2014 12:49 AM, johnlumby wrote:
On 08/19/14 18:27, Heikki Linnakangas wrote:
Please write the patch without atomic CAS operation. Just use a spinlock.
Umm, this is a new criticism I think.
Yeah. Be prepared that new issues will crop up as the patch gets slimmer
and easier to
Thanks for the replies and thoughts.
On 08/19/14 18:27, Heikki Linnakangas wrote:
On 08/20/2014 12:17 AM, John Lumby wrote:
I am attaching a new version of the patch for consideration in the
current commit fest.
Thanks for working on this!
Relative to the one I submitted on 25 June in
On 08/20/2014 12:17 AM, John Lumby wrote:
I am attaching a new version of the patch for consideration in the current
commit fest.
Thanks for working on this!
Relative to the one I submitted on 25 June in
bay175-w412ff89303686022a9f16aa3...@phx.gbl
the method for handling aio completion
On Tue, Aug 19, 2014 at 7:27 PM, Heikki Linnakangas
hlinnakan...@vmware.com wrote:
Also, please split prefetching of regular index scans into a separate patch.
It's orthogonal to doing async I/O; we could prefetch regular index scans
with posix_fadvise too, and AIO should be useful for
On Tue, Jun 24, 2014 at 12:08 PM, John Lumby johnlu...@hotmail.com wrote:
The question is, if you receive the notification of the I/O completion
using a signal or a thread, is it safe to release the lwlock from the
signal handler or a separate thread?
In the forthcoming new version of the
On 06/25/2014 09:20 PM, Claudio Freire wrote:
On Tue, Jun 24, 2014 at 12:08 PM, John Lumby johnlu...@hotmail.com wrote:
The question is, if you receive the notification of the I/O completion
using a signal or a thread, is it safe to release the lwlock from the
signal handler or a separate
On 2014-06-26 00:08:48 +0300, Heikki Linnakangas wrote:
LWLocks are implemented with semaphores, so if you can increment a semaphore
in the signal handler / callback thread, then in theory you should be able
to release a LWLock.
I don't think that's a convincing argument even if semop et al
My cut'n'pasting failed me at one point corrected below.
discussion about what is the difference between a synchronous read
versus an asynchronous read as far as non-originator waiting on it is
concerned.
I thought a bit more about this. There are currently two differences,
one of which
From: st...@mit.edu
Date: Mon, 23 Jun 2014 16:04:50 -0700
Subject: Re: Extended Prefetching using Asynchronous IO - proposal and patch
To: johnlu...@hotmail.com
CC: klaussfre...@gmail.com; hlinnakan...@vmware.com;
pgsql-hackers@postgresql.org
On
On 06/24/2014 04:29 PM, John Lumby wrote:
On Mon, Jun 23, 2014 at 2:43 PM, John Lumby johnlu...@hotmail.com wrote:
It is when some *other* backend gets there first with the ReadBuffer that
things are a bit trickier. The current version of the patch did polling for
that case
but that drew
Thanks Heikki,
Date: Tue, 24 Jun 2014 17:02:38 +0300
From: hlinnakan...@vmware.com
To: johnlu...@hotmail.com; st...@mit.edu
CC: klaussfre...@gmail.com; pgsql-hackers@postgresql.org
Subject: Re: Extended Prefetching using Asynchronous IO - proposal and
On 06/24/2014 06:08 PM, John Lumby wrote:
The question is, if you receive the notification of the I/O completion
using a signal or a thread, is it safe to release the lwlock from the
signal handler or a separate thread?
In the forthcoming new version of the patch that uses sigevent,
the
On 6/20/14, 5:12 PM, John Lumby wrote:
I also appreciate it is not easy to review the patch.
There are really 4 (or maybe 5) parts :
. async io (librt functions)
. buffer management (allocating, locking and pinning etc)
. scanners making prefetch calls
.
Date: Thu, 19 Jun 2014 15:43:44 -0300
Subject: Re: Extended Prefetching using Asynchronous IO - proposal and patch
From: klaussfre...@gmail.com
To: st...@mit.edu
CC: hlinnakan...@vmware.com; johnlu...@hotmail.com;
pgsql-hackers@postgresql.org
On
On Mon, Jun 23, 2014 at 2:43 PM, John Lumby johnlu...@hotmail.com wrote:
It is when some *other* backend gets there first with the ReadBuffer that
things are a bit trickier.The current version of the patch did polling
for that case
but that drew criticism,and so an imminent new
, 20 Jun 2014 04:21:19 +0900
Subject: Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal
and patch
From: masao.fu...@gmail.com
To: johnlu...@hotmail.com
CC: pgsql-hackers@postgresql.org; klaussfre...@gmail.com
On Mon, Jun 9, 2014 at 11:12 AM, johnlumby johnlu...@hotmail.com
On Wed, May 28, 2014 at 2:19 PM, Heikki Linnakangas
hlinnakan...@vmware.com wrote:
How portable is POSIX aio nowadays? Googling around, it still seems that on
Linux, it's implemented using threads. Does the thread-emulation
implementation cause problems with the rest of the backend, which
On Thu, Jun 19, 2014 at 2:49 PM, Greg Stark st...@mit.edu wrote:
I don't think the threaded implementation on Linux is the one to use
though. I find this *super* confusing but the kernel definitely
supports aio syscalls, glibc also has a threaded implementation it
uses if run on a kernel that
On Mon, Jun 9, 2014 at 11:12 AM, johnlumby johnlu...@hotmail.com wrote:
updated version of patch compatible with git head of 140608,
(adjusted proc oid and a couple of minor fixes)
Compilation of patched version on MacOS failed. The error messages were
gcc -O0 -Wall -Wmissing-prototypes
I'm having trouble setting max_async_io_prefetchers in postgresql.conf
It says it cannot be changed.
Is that fixed at initdb time? (sounds like a bad idea if it is)
On Sun, Jun 8, 2014 at 11:12 PM, johnlumby johnlu...@hotmail.com wrote:
updated version of patch compatible with git head of
On 06/01/2014 03:44 AM, johnlumby wrote:
If you look at the new patch, you'll see that for the different-pid case,
I still call aio_suspend with a timeout.
As you or Claudio pointed out earlier,it could just as well sleep
for the same timeout,
but the small advantage of calling aio_suspend
On 05/31/14 20:44, johnlumby wrote:
On 05/30/14 09:36, Claudio Freire wrote:
Good point. I have included the guts of your little test program
(modified to do polling) into the existing autoconf test program
that decides on the
#define USE_AIO_ATOMIC_BUILTIN_COMP_SWAP.
See
On Sat, May 31, 2014 at 9:44 PM, johnlumby johnlu...@hotmail.com wrote:
I'll try to do some measuring of performance with:
a) git head
b) git head + patch as-is
c) git head + patch without aio_suspend in foreign processes (just re-read)
d) git head + patch with a lwlock (or whatever works)
On 05/30/2014 12:53 AM, John Lumby wrote:
Date: Thu, 29 May 2014 18:00:28 -0300
Subject: Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal
and patch
From: klaussfre...@gmail.com
To: hlinnakan...@vmware.com
CC: johnlu...@hotmail.com; pgsql-hackers@postgresql.org
On Thu, May 29
On Fri, May 30, 2014 at 4:15 AM, Heikki Linnakangas
hlinnakan...@vmware.com wrote:
We don't want polling... And even if we did, calling aio_suspend() in a
way
that's known to be broken, in a loop, is a pretty crappy way of polling.
Well, as mentioned earlier, it is not broken. Whether
Thanks for looking at it!
Date: Thu, 29 May 2014 00:19:33 +0300
From: hlinnakan...@vmware.com
To: johnlu...@hotmail.com; pgsql-hackers@postgresql.org
CC: klaussfre...@gmail.com
Subject: Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal
and patch
On 05/28/2014 11:52 PM
(and it sounds as though Claudio is on it too).
The area of exactly what the best prefetch strategy should be for
each particular type of scan and context is a good one to work on.
John
Date: Wed, 28 May 2014 18:12:23 -0700
Subject: Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal
El 28/05/2014 22:12, Peter Geoghegan p...@heroku.com escribió:
On Wed, May 28, 2014 at 5:59 PM, Claudio Freire klaussfre...@gmail.com
wrote:
For nestloop, correct me if I'm wrong, but index scan nodes don't have
visibility of the next tuple to be searched for.
Nested loop joins are
On 05/29/2014 04:12 PM, John Lumby wrote:
Thanks for looking at it!
Date: Thu, 29 May 2014 00:19:33 +0300
From: hlinnakan...@vmware.com
To: johnlu...@hotmail.com; pgsql-hackers@postgresql.org
CC: klaussfre...@gmail.com
Subject: Re: [HACKERS] Extended Prefetching using Asynchronous IO
...@gmail.com
Subject: Re: [HACKERS] Extended Prefetching using Asynchronous IO -
proposal and patch
On 05/28/2014 11:52 PM, John Lumby wrote:
The patch seems to assume that you can put the aiocb struct in shared
memory, initiate an asynchronous I/O request from one process, and wait
for its
On 05/29/2014 11:34 PM, Claudio Freire wrote:
On Thu, May 29, 2014 at 5:23 PM, Heikki Linnakangas
hlinnakan...@vmware.com wrote:
On 05/29/2014 04:12 PM, John Lumby wrote:
On 05/28/2014 11:52 PM, John Lumby wrote:
The patch seems to assume that you can put the aiocb struct in shared
memory,
On Thu, May 29, 2014 at 5:39 PM, Heikki Linnakangas
hlinnakan...@vmware.com wrote:
On 05/29/2014 11:34 PM, Claudio Freire wrote:
On Thu, May 29, 2014 at 5:23 PM, Heikki Linnakangas
hlinnakan...@vmware.com wrote:
On 05/29/2014 04:12 PM, John Lumby wrote:
On 05/28/2014 11:52 PM, John Lumby
Claudio Freire klaussfre...@gmail.com writes:
Didn't fix that, but the attached patch does fix regression tests when
scanning over index types other than btree (was invoking elog when the
index am didn't have ampeeknexttuple)
ampeeknexttuple? That's a bit scary. It would certainly be unsafe
On Thu, May 29, 2014 at 6:19 PM, Tom Lane t...@sss.pgh.pa.us wrote:
Claudio Freire klaussfre...@gmail.com writes:
Didn't fix that, but the attached patch does fix regression tests when
scanning over index types other than btree (was invoking elog when the
index am didn't have ampeeknexttuple)
On Thu, May 29, 2014 at 6:43 PM, Claudio Freire klaussfre...@gmail.com wrote:
On Thu, May 29, 2014 at 6:19 PM, Tom Lane t...@sss.pgh.pa.us wrote:
Claudio Freire klaussfre...@gmail.com writes:
Didn't fix that, but the attached patch does fix regression tests when
scanning over index types other
Date: Thu, 29 May 2014 18:00:28 -0300
Subject: Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal
and patch
From: klaussfre...@gmail.com
To: hlinnakan...@vmware.com
CC: johnlu...@hotmail.com; pgsql-hackers@postgresql.org
On Thu, May 29, 2014 at 5:39 PM, Heikki
Claudio Freire klaussfre...@gmail.com writes:
On Thu, May 29, 2014 at 6:43 PM, Claudio Freire klaussfre...@gmail.com
wrote:
On Thu, May 29, 2014 at 6:19 PM, Tom Lane t...@sss.pgh.pa.us wrote:
ampeeknexttuple? That's a bit scary. It would certainly be unsafe
for non-MVCC snapshots (read
From: t...@sss.pgh.pa.us
To: klaussfre...@gmail.com
CC: hlinnakan...@vmware.com; johnlu...@hotmail.com;
pgsql-hackers@postgresql.org
Subject: Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal
and patch
Date: Thu, 29 May 2014 17:56:57 -0400
Claudio Freire klaussfre
On Thu, May 29, 2014 at 6:56 PM, Tom Lane t...@sss.pgh.pa.us wrote:
Claudio Freire klaussfre...@gmail.com writes:
On Thu, May 29, 2014 at 6:43 PM, Claudio Freire klaussfre...@gmail.com
wrote:
On Thu, May 29, 2014 at 6:19 PM, Tom Lane t...@sss.pgh.pa.us wrote:
ampeeknexttuple? That's a bit
On Thu, May 29, 2014 at 7:11 PM, John Lumby johnlu...@hotmail.com wrote:
Nonetheless, getting the next tid out of the index may involve stepping
to the next index page, at which point you've lost your interlock
I think we are ok as peeknexttuple (yes bad name, sorry, can change it
...
Hi,
On 2014-05-29 17:53:51 -0400, John Lumby wrote:
to see how we have done it. And I am attaching corrected version
of your test program which runs just fine.
It's perfectly fine to not be up to the coding style at this point, but
trying to adhere to it to some degree will make code review
Date: Thu, 29 May 2014 18:00:28 -0300
Subject: Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal
and patch
From: klaussfre...@gmail.com
To: hlinnakan...@vmware.com
CC: johnlu...@hotmail.com; pgsql-hackers@postgresql.org
Even if it worked on Linux today, it would
On Thu, May 29, 2014 at 6:53 PM, John Lumby johnlu...@hotmail.com wrote:
Well, as mentioned earlier, it is not broken. Whether it is efficient
I am not sure.
I have looked at the mutex in aio_suspend that you mentioned and I am not
quite convinced that, if caller is not the original
On Thu, May 29, 2014 at 7:26 PM, Claudio Freire klaussfre...@gmail.com wrote:
1) If it's the same process, wait for the full timeout (no looping).
If you have to loop (EAGAIN or EINTR - which I just noticed you don't
check for), that's ok.
Sorry, meant to say just looping on EINTR.
About the
On 05/28/2014 11:52 PM, John Lumby wrote:
The patch is attached.
It is based on clone of today's 9.4dev source.
I have noticed that this source is
(not suprisingly) quite a moving target at present,
meaning that this patch becomes stale quite quickly.
So although this copy is fine for
On Tue, May 27, 2014 at 3:17 PM, John Lumby johnlu...@hotmail.com wrote:
Below I am pasting the README we have written for this new functionality
which mentions some of the measurements, advantages (and disadvantages)
and we welcome all and any comments on this.
I think that this is likely to
On Wed, May 28, 2014 at 6:51 PM, Peter Geoghegan p...@heroku.com wrote:
Have you thought about things like specialized prefetching for nested
loop joins?
Currently, such a thing would need some non-trivial changes to the
execution nodes, I believe.
For nestloop, correct me if I'm wrong, but
On Wed, May 28, 2014 at 5:59 PM, Claudio Freire klaussfre...@gmail.com wrote:
For nestloop, correct me if I'm wrong, but index scan nodes don't have
visibility of the next tuple to be searched for.
Nested loop joins are considered a particularly compelling case for
prefetching, actually.
--
48 matches
Mail list logo
