Re: [SPAM?] Re: [HACKERS] Asynchronous I/O Support

2006-10-23 Thread Zeugswetter Andreas ADI SD
> > Yup, that would be the scenario where it helps (provided that you have > > a smart disk or a disk array and an intelligent OS aio implementation). > > It would be used to fetch the data pages pointed at from an index > > leaf, or the next level index pages. > > We measured the IO bandwidth d

Re: [SPAM?] Re: [HACKERS] Asynchronous I/O Support

2006-10-23 Thread Zeugswetter Andreas ADI SD
> > So far I've seen no evidence that async I/O would help us, only a lot > > of wishful thinking. > > is this thread moot? while researching this thread I came across this > article: http://kerneltrap.org/node/6642 describing claims of > 30% performance boost when using posix_fadvise to ask t

Re: [SPAM?] Re: [HACKERS] Asynchronous I/O Support

2006-10-23 Thread Zeugswetter Andreas ADI SD
> > >So far I've seen no evidence that async I/O would help us, only a lot > > >of wishful thinking. > > > > is this thread moot? while researching this thread I came across this > > article: http://kerneltrap.org/node/6642 describing claims of 30% > > performance boost when using posix_fadvis

Re: [SPAM?] Re: [HACKERS] Asynchronous I/O Support

2006-10-21 Thread Bruce Momjian
Martijn van Oosterhout wrote: -- Start of PGP signed section. > On Fri, Oct 20, 2006 at 03:04:55PM -0400, Merlin Moncure wrote: > > On 10/20/06, Tom Lane <[EMAIL PROTECTED]> wrote: > > >So far I've seen no evidence that async I/O would help us, only a lot > > >of wishful thinking. > > > > is this

Re: [SPAM?] Re: [HACKERS] Asynchronous I/O Support

2006-10-20 Thread Merlin Moncure
On 10/21/06, Martijn van Oosterhout wrote: On Fri, Oct 20, 2006 at 03:04:55PM -0400, Merlin Moncure wrote: > On 10/20/06, Tom Lane <[EMAIL PROTECTED]> wrote: > >So far I've seen no evidence that async I/O would help us, only a lot > >of wishful thinking. > > is this thread moot? while researchi

Re: [SPAM?] Re: [HACKERS] Asynchronous I/O Support

2006-10-20 Thread Martijn van Oosterhout
On Fri, Oct 20, 2006 at 03:04:55PM -0400, Merlin Moncure wrote: > On 10/20/06, Tom Lane <[EMAIL PROTECTED]> wrote: > >So far I've seen no evidence that async I/O would help us, only a lot > >of wishful thinking. > > is this thread moot? while researching this thread I came across this > article:

Re: [SPAM?] Re: [HACKERS] Asynchronous I/O Support

2006-10-20 Thread Merlin Moncure
On 10/20/06, Tom Lane <[EMAIL PROTECTED]> wrote: So far I've seen no evidence that async I/O would help us, only a lot of wishful thinking. is this thread moot? while researching this thread I came across this article: http://kerneltrap.org/node/6642 describing claims of 30% performance boost

Re: [SPAM?] Re: [HACKERS] Asynchronous I/O Support

2006-10-20 Thread Tom Lane
> On Fri, Oct 20, 2006 at 10:05:01AM -0400, [EMAIL PROTECTED] wrote: >> One would need to consider the PostgreSQL architecture, determine where >> the bottleneck actually is, and understand why it is a bottleneck fully, >> before one could decide how to fix it. So, what is the bottleneck? I think

Re: [SPAM?] Re: [HACKERS] Asynchronous I/O Support

2006-10-20 Thread Martijn van Oosterhout
On Fri, Oct 20, 2006 at 10:05:01AM -0400, [EMAIL PROTECTED] wrote: > Only if it can be shown that async I/O actually results in an improvement. > > Currently, it's speculation, with the one trial implementation showing > little to no improvement. Support is a big word in the face of this > initial

Re: [SPAM?] Re: [HACKERS] Asynchronous I/O Support

2006-10-20 Thread mark
On Fri, Oct 20, 2006 at 05:37:48PM +0200, Zeugswetter Andreas ADI SD wrote: > Yup, that would be the scenario where it helps (provided that you have > a smart disk or a disk array and an intelligent OS aio implementation). > It would be used to fetch the data pages pointed at from an index leaf, >

Re: [SPAM?] Re: [HACKERS] Asynchronous I/O Support

2006-10-20 Thread Zeugswetter Andreas ADI SD
> > >Good idea, but async i/o is generally poorly supported. > Only if it can be shown that async I/O actually results in an > improvement. sure. > fix it. So, what is the bottleneck? Is PostgreSQL unable to > max out the I/O bandwidth? Where? Why? Yup, that would be the scenario where it he

Re: [SPAM?] Re: [HACKERS] Asynchronous I/O Support

2006-10-20 Thread mark
On Fri, Oct 20, 2006 at 11:13:33AM +0530, NikhilS wrote: > >Good idea, but async i/o is generally poorly supported. > Async i/o is stably supported on most *nix (apart from Linux 2.6.*) plus > Windows. > Guess it would be still worth it, since one fine day 2.6.* will start > supporting it properly

Re: [HACKERS] Asynchronous I/O Support

2006-10-20 Thread Alvaro Herrera
Zeugswetter Andreas ADI SD wrote: > > An improvement is going into 2.6.19 to handle asynchronous > > vector reads and writes. This was covered by Linux Weekly > > News a couple of weeks ago: > > http://lwn.net/Articles/201682/ > > That is orthogonal. We don't really need vector io so much, sinc

Re: [HACKERS] Asynchronous I/O Support

2006-10-20 Thread Zeugswetter Andreas ADI SD
> > At least according to [1], kernel AIO on Linux still doesn't work for > > buffered (i.e. non-O_DIRECT) files. There have been patches available > > for quite some time that implement this, but I'm not sure when they > > are likely to get into the mainline kernel. > > > > -Neil > > > > [1]

Re: [HACKERS] Asynchronous I/O Support

2006-10-19 Thread NikhilS
Hi, On 10/18/06, Martijn van Oosterhout wrote: On Wed, Oct 18, 2006 at 08:04:29PM +1300, Mark Kirkwood wrote:> >"bgwriter doing aysncronous I/O for the dirty buffers that it is> >supposed to sync"> >Another decent use-case? Good idea, but async i/o is generally poorly supported.

Re: [HACKERS] Asynchronous I/O Support

2006-10-19 Thread Bruno Wolff III
On Sun, Oct 15, 2006 at 14:26:12 -0400, Neil Conway <[EMAIL PROTECTED]> wrote: > > At least according to [1], kernel AIO on Linux still doesn't work for > buffered (i.e. non-O_DIRECT) files. There have been patches available > for quite some time that implement this, but I'm not sure when they a

Re: [HACKERS] Asynchronous I/O Support

2006-10-18 Thread Martijn van Oosterhout
On Wed, Oct 18, 2006 at 08:04:29PM +1300, Mark Kirkwood wrote: > >"bgwriter doing aysncronous I/O for the dirty buffers that it is > >supposed to sync" > >Another decent use-case? Good idea, but async i/o is generally poorly supported. > Is it worth considering using readv(2) instead? Err, read

Re: [HACKERS] Asynchronous I/O Support

2006-10-18 Thread Mark Kirkwood
NikhilS wrote: Hi, "bgwriter doing aysncronous I/O for the dirty buffers that it is supposed to sync" Another decent use-case? Regards, Nikhils EnterpriseDB http://www.enterprisedb.com On 10/15/06, *Luke Lonergan* <[EMAIL PROTECTED] > wrote: Martijn, O

Re: [HACKERS] Asynchronous I/O Support

2006-10-17 Thread NikhilS
Hi, "bgwriter doing aysncronous I/O for the dirty buffers that it is supposed to sync" Another decent use-case? Regards, Nikhils EnterpriseDB   http://www.enterprisedb.com On 10/15/06, Luke Lonergan <[EMAIL PROTECTED]> wrote: Martijn,On 10/15/06 10:56 AM, "Martijn van Oosterhout"

Re: [HACKERS] Asynchronous I/O Support

2006-10-17 Thread Raja Agrawal
Have a look at this: [2]http://www-128.ibm.com/developerworks/linux/library/l-async/ This gives a good description of AIO. I'm doing some testing. Will notify, if I get any positive results. Please let me know, if you get any ideas after reading [2]. Regards, Raja On 10/17/06, Florian Weimer

Re: [HACKERS] Asynchronous I/O Support

2006-10-17 Thread Florian Weimer
* Neil Conway: > [1] http://lse.sourceforge.net/io/aio.html Last Modified Mon, 07 Jun 2004 12:00:09 GMT But you are right -- it seems that io_submit still blocks without O_DIRECT. *sigh* -- Florian Weimer<[EMAIL PROTECTED]> BFK edv-consulting GmbH http://www.bfk.de/ D

Re: [HACKERS] Asynchronous I/O Support

2006-10-17 Thread Merlin Moncure
On 10/15/06, Luke Lonergan <[EMAIL PROTECTED]> wrote: Martijn, The killer use-case we've identified is for the scattered I/O associated with index + heap scans in Postgres. If we can issue ~5-15 I/Os in advance when the TIDs are widely separated it has the potential to increase the I/O speed by

Re: [HACKERS] Asynchronous I/O Support

2006-10-15 Thread Martijn van Oosterhout
On Sun, Oct 15, 2006 at 02:26:12PM -0400, Neil Conway wrote: > On Sun, 2006-10-15 at 19:56 +0200, Martijn van Oosterhout wrote: > > Sure, I even implemented it once. Didn't get any faster. > > Did you just do something akin to s/read/aio_read/ etc., or something > more ambitious? I think that real

Re: [HACKERS] Asynchronous I/O Support

2006-10-15 Thread Neil Conway
On Sun, 2006-10-15 at 19:56 +0200, Martijn van Oosterhout wrote: > Sure, I even implemented it once. Didn't get any faster. Did you just do something akin to s/read/aio_read/ etc., or something more ambitious? I think that really taking advantage of the ability to have multiple I/O requests outsta

Re: [HACKERS] Asynchronous I/O Support

2006-10-15 Thread Luke Lonergan
Martijn, On 10/15/06 10:56 AM, "Martijn van Oosterhout" wrote: > Have enough systems actually got to the point of actually supporting > async I/O that it's worth implementing? I think there are enough high end applications / systems that need it at this point. The killer use-case we've identif

Re: [HACKERS] Asynchronous I/O Support

2006-10-15 Thread Martijn van Oosterhout
On Sun, Oct 15, 2006 at 04:16:07AM +0530, Raja Agrawal wrote: > Postgre8.1 doesn't seem to support asynchronous I/O. Has its design > been thought off already? Sure, I even implemented it once. Didn't get any faster. At that point I realised that my kernel didn't actually support async I/O, and th

[HACKERS] Asynchronous I/O Support

2006-10-15 Thread Raja Agrawal
Postgre8.1 doesn't seem to support asynchronous I/O. Has its design been thought off already? To tried doing with a simple example: For a Index Nest loop join: Fetch the outer tuples in an array, and then send all the corresponding inner-tuple fetch requests asynchronously. Hence while the IO is