Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-07-05 Thread Jeff Davis
On Mon, 2013-07-01 at 00:52 -0400, Greg Smith wrote: On 6/30/13 9:28 PM, Jon Nelson wrote: The performance of the latter (new) test sometimes seems to perform worse and sometimes seems to perform better (usually worse) than either of the other two. In all cases, posix_fallocate performs

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-07-05 Thread Greg Smith
On 7/5/13 2:50 AM, Jeff Davis wrote: So, my simple conclusion is that glibc emulation should be about the same as what we're doing now, so there's no reason to avoid it. That means, if posix_fallocate() is present, we should use it, because it's either the same (if emulated in glibc) or

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-07-05 Thread Jon Nelson
On Fri, Jul 5, 2013 at 2:23 AM, Greg Smith g...@2ndquadrant.com wrote: On 7/5/13 2:50 AM, Jeff Davis wrote: So, my simple conclusion is that glibc emulation should be about the same as what we're doing now, so there's no reason to avoid it. That means, if posix_fallocate() is present, we

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-07-05 Thread Jeff Davis
On Fri, 2013-07-05 at 08:21 -0500, Jon Nelson wrote: Wonderful. Is removing the GUC something that I should do or should that be done by somebody that knows more about what they are doing? (I am happy to give it a go!) I'll take care of that. Should the small test program that I made also be

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-07-01 Thread Jon Nelson
On Sun, Jun 30, 2013 at 11:52 PM, Greg Smith g...@2ndquadrant.com wrote: On 6/30/13 9:28 PM, Jon Nelson wrote: The performance of the latter (new) test sometimes seems to perform worse and sometimes seems to perform better (usually worse) than either of the other two. In all cases,

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-07-01 Thread Jeff Davis
On Sun, 2013-06-30 at 16:09 -0400, Andrew Dunstan wrote: It was originally generated. Since then it's been maintained by hand. What is the procedure for maintaining it by hand? Why are HAVE_POSIX_SIGNALS and HAVE_SYNC_FILE_RANGE in there (though commented out), but not HAVE_POSIX_FADVISE?

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-07-01 Thread Jeff Davis
On Sun, 2013-06-30 at 18:55 -0400, Greg Smith wrote: This makes platform level testing a lot easier, thanks. Attached is an updated copy of that program with some error checking. If the files it creates already existed, the code didn't notice, and a series of write errors happened. If

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-07-01 Thread Fujii Masao
On Tue, Jul 2, 2013 at 1:55 AM, Jeff Davis pg...@j-davis.com wrote: On Sun, 2013-06-30 at 18:55 -0400, Greg Smith wrote: This makes platform level testing a lot easier, thanks. Attached is an updated copy of that program with some error checking. If the files it creates already existed, the

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-07-01 Thread Jeff Davis
On Tue, 2013-07-02 at 02:13 +0900, Fujii Masao wrote: Even in that case, if a user can easily know which platform posix_fallocate should be used in, we can commit the patch with the configurable GUC parameter. I disagree here. We're not talking about a huge win; this speedup may not even be

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-07-01 Thread Robert Haas
On Mon, Jul 1, 2013 at 2:17 PM, Jeff Davis pg...@j-davis.com wrote: On Tue, 2013-07-02 at 02:13 +0900, Fujii Masao wrote: Even in that case, if a user can easily know which platform posix_fallocate should be used in, we can commit the patch with the configurable GUC parameter. I disagree

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-07-01 Thread Peter Eisentraut
On 7/1/13 12:34 PM, Jeff Davis wrote: On Sun, 2013-06-30 at 16:09 -0400, Andrew Dunstan wrote: It was originally generated. Since then it's been maintained by hand. What is the procedure for maintaining it by hand? Edit away. Why are HAVE_POSIX_SIGNALS and HAVE_SYNC_FILE_RANGE in there

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-07-01 Thread Greg Smith
On 7/1/13 3:44 PM, Robert Haas wrote: Yeah. If the patch isn't going to be a win on RHEL 5, I'd consider that a good reason to scrap it for now and revisit it in 3 years. There are still a LOT of people running RHEL 5, and the win isn't big enough to engineer a more complex solution. I'm

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-06-30 Thread Jeff Davis
On Tue, 2013-05-28 at 22:10 -0400, Greg Smith wrote: I was just thinking of something to run in your test program, not another build time check. Just run the new allocation sequence, and then check the resulting WAL file for a) correct length, and b) 16K of zero bytes. I would like to

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-06-30 Thread Jeff Davis
On Fri, 2013-06-28 at 11:38 -0700, Josh Berkus wrote: Since Greg seems to be busy, what needs to be done to test this? As I understand it, he was mainly asking if posix_fallocate works at all. I tried to address that question with a simple test, which behaves as I expected it to:

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-06-30 Thread Jeff Davis
On Sun, 2013-06-30 at 11:11 -0700, Jeff Davis wrote: Unless something surprising comes up, or someone thinks and objection has been missed, I am going to commit this soon. Quick question to anyone who happens to know: What is the standard procedure for changes to pg_config.h.win32? I looked at

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-06-30 Thread Andrew Dunstan
On 06/30/2013 03:50 PM, Jeff Davis wrote: On Sun, 2013-06-30 at 11:11 -0700, Jeff Davis wrote: Unless something surprising comes up, or someone thinks and objection has been missed, I am going to commit this soon. Quick question to anyone who happens to know: What is the standard procedure

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-06-30 Thread Greg Smith
On 6/30/13 2:01 PM, Jeff Davis wrote: Simple test program attached, which creates two files and fills them: one by 2048 8KB writes; and another by 1 posix_fallocate of 16MB. Then, I just cmp the resulting files (and also ls them, to make sure they are 16MB). This makes platform level testing a

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-06-30 Thread Jon Nelson
On Sun, Jun 30, 2013 at 5:55 PM, Greg Smith g...@2ndquadrant.com wrote: pwrite(4, \0, 1, 16769023)= 1 pwrite(4, \0, 1, 16773119)= 1 pwrite(4, \0, 1, 16777215)= 1 That's glibc helpfully converting your call to posix_fallocate into small writes, because

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-06-30 Thread Greg Smith
On 5/28/13 10:00 PM, Jon Nelson wrote: A note: The attached test program uses *fsync* instead of *fdatasync* after calling fallocate (or writing out 16MB of zeroes), per an earlier suggestion. I tried this out on the RHEL5 platform I'm worried about now. There's something weird about the

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-06-30 Thread Jon Nelson
On Sun, Jun 30, 2013 at 6:49 PM, Greg Smith g...@2ndquadrant.com wrote: On 5/28/13 10:00 PM, Jon Nelson wrote: A note: The attached test program uses *fsync* instead of *fdatasync* after calling fallocate (or writing out 16MB of zeroes), per an earlier suggestion. I tried this out on the

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-06-30 Thread Greg Smith
On 6/30/13 9:28 PM, Jon Nelson wrote: The performance of the latter (new) test sometimes seems to perform worse and sometimes seems to perform better (usually worse) than either of the other two. In all cases, posix_fallocate performs better, but I don't have a sufficiently old kernel to test

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-06-28 Thread Josh Berkus
On 06/20/2013 07:19 PM, Jeff Davis wrote: On Sun, 2013-06-16 at 23:53 -0500, Jon Nelson wrote: Please find attached v5 of the patch, with the above correction in place. The only change from the v4 patch is wrapping the if (wal_use_fallocate) block in an #ifdef USE_POSIX_FALLOCATE. Thanks!

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-06-20 Thread Jeff Davis
On Sun, 2013-06-16 at 23:53 -0500, Jon Nelson wrote: Please find attached v5 of the patch, with the above correction in place. The only change from the v4 patch is wrapping the if (wal_use_fallocate) block in an #ifdef USE_POSIX_FALLOCATE. Thanks! Thank you. Greg, are you still working on

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-06-16 Thread Jon Nelson
On Fri, Jun 14, 2013 at 12:06 PM, Jeff Davis pg...@j-davis.com wrote: On Sat, 2013-05-25 at 13:55 -0500, Jon Nelson wrote: Ack. I've revised the patch to always have the GUC (for now), default to false, and if configure can't find posix_fallocate (or the user disables it by way of

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-06-16 Thread Jon Nelson
On Sun, Jun 16, 2013 at 9:59 PM, Jon Nelson jnelson+pg...@jamponi.net wrote: On Fri, Jun 14, 2013 at 12:06 PM, Jeff Davis pg...@j-davis.com wrote: On Sat, 2013-05-25 at 13:55 -0500, Jon Nelson wrote: .. * You check for the presence of posix_fallocate at configure time, but don't #ifdef the

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-06-14 Thread Jeff Davis
On Sat, 2013-05-25 at 13:55 -0500, Jon Nelson wrote: Ack. I've revised the patch to always have the GUC (for now), default to false, and if configure can't find posix_fallocate (or the user disables it by way of pg_config_manual.h) then it remains a GUC that simply can't be changed. Why have

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-06-14 Thread Greg Smith
On 6/14/13 1:06 PM, Jeff Davis wrote: Why have a GUC here at all? Perhaps this was already discussed, and I missed it? Is it just for testing purposes, or did you intend for it to be in the final version? You have guessed correctly! I suggested it stay in there only to make review

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-06-14 Thread Jeff Davis
On Tue, 2013-06-11 at 12:58 -0400, Stephen Frost wrote: My main question is really- would this be useful for extending *relations*? Apologies if it's already been discussed; I do plan to go back and read the threads about this more fully, but I wanted to voice my support for using

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-06-14 Thread Jeff Davis
On Fri, 2013-06-14 at 13:21 -0400, Greg Smith wrote: I'm planning to duplicate Jon's test program on a few machines here, and then see if that turns into a useful latency improvement for clients. I'm trying to get this pgbench rate limit stuff working first though, because one of the tests

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-06-12 Thread Robert Haas
On Tue, Jun 11, 2013 at 12:58 PM, Stephen Frost sfr...@snowman.net wrote: * Merlin Moncure (mmonc...@gmail.com) wrote: It's understood that posix_fallocate is faster at this -- the question on the table is 'does this matter in context of postgres?'. Personally I think this patch should go in

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-06-11 Thread Jon Nelson
There hasn't been much activity here recently. I'm curious, then, if there are questions that I can answer. It may be useful to summarize some things here: - the purpose of the patch is to use posix_fallocate when creating new WAL files, because it's (usually) much quicker - using

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-06-11 Thread Stefan Drees
On 2013-11.06 17:28, Jon Nelson wrote: There hasn't been much activity here recently. I'm curious, then, if there are questions that I can answer. It may be useful to summarize some things here: - the purpose of the patch is to use posix_fallocate when creating new WAL files, because it's

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-06-11 Thread Merlin Moncure
On Tue, Jun 11, 2013 at 11:08 AM, Stefan Drees ste...@drees.name wrote: On 2013-11.06 17:28, Jon Nelson wrote: There hasn't been much activity here recently. I'm curious, then, if there are questions that I can answer. It may be useful to summarize some things here: - the purpose of the

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-06-11 Thread Stephen Frost
* Merlin Moncure (mmonc...@gmail.com) wrote: It's understood that posix_fallocate is faster at this -- the question on the table is 'does this matter in context of postgres?'. Personally I think this patch should go in regardless -- the concerns made IMNSHO are specious. I've not had a chance

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-06-11 Thread Greg Smith
On 6/11/13 12:22 PM, Merlin Moncure wrote: Personally I think this patch should go in regardless -- the concerns made IMNSHO are specious. That's nice, but we have this process for validating whether features go in or not that relies on review instead of opinions. -- Greg Smith

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-06-11 Thread Stefan Drees
On 2013-06-11 19:45 CEST, Greg Smith wrote: On 6/11/13 12:22 PM, Merlin Moncure wrote: Personally I think this patch should go in regardless -- the concerns made IMNSHO are specious. That's nice, but we have this process for validating whether features go in or not that relies on review

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-06-11 Thread Jon Nelson
On Tue, Jun 11, 2013 at 12:49 PM, Stefan Drees ste...@drees.name wrote: On 2013-06-11 19:45 CEST, Greg Smith wrote: On 6/11/13 12:22 PM, Merlin Moncure wrote: Personally I think this patch should go in regardless -- the concerns made IMNSHO are specious. That's nice, but we have this

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-06-11 Thread Merlin Moncure
On Tue, Jun 11, 2013 at 12:45 PM, Greg Smith g...@2ndquadrant.com wrote: On 6/11/13 12:22 PM, Merlin Moncure wrote: Personally I think this patch should go in regardless -- the concerns made IMNSHO are specious. That's nice, but we have this process for validating whether features go in or

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-31 Thread Noah Misch
On Thu, May 30, 2013 at 02:58:26PM +0200, Andres Freund wrote: * Andres Freund (and...@2ndquadrant.com) wrote: But really, I am not at all concerned about some obscure values being returned, but about a read() not being successful.. After a bit of standard perusing writing a single byte

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-30 Thread Robert Haas
On Wed, May 29, 2013 at 10:42 AM, Andres Freund and...@2ndquadrant.com wrote: FWIW, posix' description about posix_fallocate() doesn't actually say *anything* about reading. The guarantee it makes is: If posix_fallocate() returns successfully, subsequent writes to the specified file data shall

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-30 Thread Greg Smith
On 5/30/13 6:49 AM, Robert Haas wrote: On Wed, May 29, 2013 at 10:42 AM, Andres Freund and...@2ndquadrant.com wrote: So we don't even know whether we can read. I think that means we need to zero the file anyway... Surely this is undue pessimism. There have been many occasions where I've

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-30 Thread Andres Freund
On 2013-05-30 06:49:42 -0400, Robert Haas wrote: On Wed, May 29, 2013 at 10:42 AM, Andres Freund and...@2ndquadrant.com wrote: FWIW, posix' description about posix_fallocate() doesn't actually say *anything* about reading. The guarantee it makes is: If posix_fallocate() returns

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-30 Thread Andres Freund
On 2013-05-30 06:55:16 -0400, Greg Smith wrote: On 5/30/13 6:49 AM, Robert Haas wrote: On Wed, May 29, 2013 at 10:42 AM, Andres Freund and...@2ndquadrant.com wrote: So we don't even know whether we can read. I think that means we need to zero the file anyway... Surely this is undue

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-30 Thread Greg Smith
On 5/30/13 7:17 AM, Andres Freund wrote: That argument in contrast I find not very convincing though. What was the last incidence of such a system call that did not just error out with ENOTSUPP or such? http://linux.die.net/man/2/posix_fadvise talks about POSIX_FADV_NOREUSE and

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-30 Thread Andres Freund
On 2013-05-30 07:48:51 -0400, Greg Smith wrote: On 5/30/13 7:17 AM, Andres Freund wrote: That argument in contrast I find not very convincing though. What was the last incidence of such a system call that did not just error out with ENOTSUPP or such? http://linux.die.net/man/2/posix_fadvise

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-30 Thread Greg Smith
On 5/30/13 7:52 AM, Andres Freund wrote: fadvise is a hint which is pretty different from a fallocate where ignoring would have way much more severe consequences. Yes, it will. That's why I want to see it tested. There is more than enough past examples of bad behavior here to be skeptical

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-30 Thread Robert Haas
On Thu, May 30, 2013 at 7:13 AM, Andres Freund and...@2ndquadrant.com wrote: Surely this is undue pessimism. Why? The spec doesn't specify that case and that very well allows other behaviour. Glibc sure does behave sensibly and zeroes the data (sysdeps/posix/posix_fallocate64.c for the

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-30 Thread Andres Freund
On 2013-05-30 08:02:56 -0400, Robert Haas wrote: On Thu, May 30, 2013 at 7:13 AM, Andres Freund and...@2ndquadrant.com wrote: Surely this is undue pessimism. Why? The spec doesn't specify that case and that very well allows other behaviour. Glibc sure does behave sensibly and zeroes the

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-30 Thread Peter Eisentraut
On 5/30/13 7:13 AM, Andres Freund wrote: Why? The spec doesn't specify that case and that very well allows other behaviour. Glibc sure does behave sensibly and zeroes the data (sysdeps/posix/posix_fallocate64.c for the generic implementation) and so does linux' fallocate() syscall, but that

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-30 Thread Peter Eisentraut
On 5/30/13 8:02 AM, Robert Haas wrote: If there's some OS out there that chooses to fill the pre-extended pages with 0x55 or cat /dev/urandom instead of 0x00, they probably deserve what they get. Even that wouldn't be a problem for our purpose. The only problem would be if you can't read from

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-30 Thread Andres Freund
On 2013-05-30 08:19:17 -0400, Peter Eisentraut wrote: On 5/30/13 8:02 AM, Robert Haas wrote: If there's some OS out there that chooses to fill the pre-extended pages with 0x55 or cat /dev/urandom instead of 0x00, they probably deserve what they get. Even that wouldn't be a problem for

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-30 Thread Andres Freund
On 2013-05-30 08:17:28 -0400, Peter Eisentraut wrote: On 5/30/13 7:13 AM, Andres Freund wrote: Why? The spec doesn't specify that case and that very well allows other behaviour. Glibc sure does behave sensibly and zeroes the data (sysdeps/posix/posix_fallocate64.c for the generic

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-30 Thread Stephen Frost
* Peter Eisentraut (pete...@gmx.net) wrote: On 5/30/13 7:13 AM, Andres Freund wrote: Why? The spec doesn't specify that case and that very well allows other behaviour. Glibc sure does behave sensibly and zeroes the data (sysdeps/posix/posix_fallocate64.c for the generic implementation) and

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-30 Thread Stephen Frost
* Andres Freund (and...@2ndquadrant.com) wrote: On 2013-05-30 08:19:17 -0400, Peter Eisentraut wrote: On 5/30/13 8:02 AM, Robert Haas wrote: If there's some OS out there that chooses to fill the pre-extended pages with 0x55 or cat /dev/urandom instead of 0x00, they probably deserve

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-30 Thread Stephen Frost
* Andres Freund (and...@2ndquadrant.com) wrote: But really, I am not at all concerned about some obscure values being returned, but about a read() not being successful.. Alright, so what do we need to do to test this? We really just need a short C program written up and then a bunch of folks

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-30 Thread Andres Freund
On 2013-05-30 08:53:37 -0400, Stephen Frost wrote: * Andres Freund (and...@2ndquadrant.com) wrote: But really, I am not at all concerned about some obscure values being returned, but about a read() not being successful.. Alright, so what do we need to do to test this? We really just need

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-30 Thread Stephen Frost
* Andres Freund (and...@2ndquadrant.com) wrote: After a bit of standard perusing writing a single byte to the end of the file after the fallocate ought to make at least the reading guaranteed to be defined. If we did seek(last_byte); write(); posix_fallocate() we should even always have

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-30 Thread Greg Smith
On 5/30/13 8:50 AM, Stephen Frost wrote: I don't think this solves the locking issue around the relation extention lock, but it might help some and it may make it easier to tweak that logic to allocate larger chunks in the future. It might make it a bit faster, but it doesn't make it any

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-30 Thread Robert Haas
On Thu, May 30, 2013 at 8:14 AM, Andres Freund and...@2ndquadrant.com wrote: I don't think there's much danger of getting uninitialized data or such. That clearly would be insane. I think somebody might interpret it as read(2) returning an error until the page has been written to which isn't

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-30 Thread Alvaro Herrera
Greg Smith escribió: The messy part of extending relations in larger chunks is how to communicate that back into the buffer manager usefully. The extension path causing trouble is RelationGetBufferForTuple calling ReadBufferBI. All of that is passing a single buffer around. There's no

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-30 Thread Greg Smith
On 5/30/13 11:21 AM, Alvaro Herrera wrote: Greg Smith escribió: The messy part of extending relations in larger chunks is how to communicate that back into the buffer manager usefully. The extension path causing trouble is RelationGetBufferForTuple calling ReadBufferBI. All of that is passing

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-29 Thread Peter Eisentraut
On 5/28/13 11:36 AM, Greg Smith wrote: Outside of the run for performance testing, I think it would be good at this point to validate that there is really a 16MB file full of zeroes resulting from these operations. I am not really concerned that posix_fallocate might be slower in some cases;

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-29 Thread Stephen Frost
* Peter Eisentraut (pete...@gmx.net) wrote: On 5/28/13 11:36 AM, Greg Smith wrote: Outside of the run for performance testing, I think it would be good at this point to validate that there is really a 16MB file full of zeroes resulting from these operations. I am not really concerned that

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-29 Thread Andres Freund
On 2013-05-29 10:36:07 -0400, Stephen Frost wrote: * Peter Eisentraut (pete...@gmx.net) wrote: On 5/28/13 11:36 AM, Greg Smith wrote: Outside of the run for performance testing, I think it would be good at this point to validate that there is really a 16MB file full of zeroes resulting

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-29 Thread Peter Eisentraut
On 5/29/13 10:42 AM, Andres Freund wrote: On 2013-05-29 10:36:07 -0400, Stephen Frost wrote: I *really* hope that the Linux kernel, and other, folks are smart enough to realize that they can't just re-use random blocks from an I/O device without cleaning it first. FWIW, posix' description

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-28 Thread Robert Haas
On Sat, May 25, 2013 at 2:55 PM, Jon Nelson jnelson+pg...@jamponi.net wrote: The biggest thing missing from this submission is information about what performance testing you did. Ideally performance patches are submitted with enough information for a reviewer to duplicate the same test the

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-28 Thread Andres Freund
On 2013-05-28 10:03:58 -0400, Robert Haas wrote: On Sat, May 25, 2013 at 2:55 PM, Jon Nelson jnelson+pg...@jamponi.net wrote: The biggest thing missing from this submission is information about what performance testing you did. Ideally performance patches are submitted with enough

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-28 Thread Robert Haas
On Tue, May 28, 2013 at 10:15 AM, Andres Freund and...@2ndquadrant.com wrote: On 2013-05-28 10:03:58 -0400, Robert Haas wrote: On Sat, May 25, 2013 at 2:55 PM, Jon Nelson jnelson+pg...@jamponi.net wrote: The biggest thing missing from this submission is information about what performance

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-28 Thread Jon Nelson
On Tue, May 28, 2013 at 9:19 AM, Robert Haas robertmh...@gmail.com wrote: On Tue, May 28, 2013 at 10:15 AM, Andres Freund and...@2ndquadrant.com wrote: On 2013-05-28 10:03:58 -0400, Robert Haas wrote: On Sat, May 25, 2013 at 2:55 PM, Jon Nelson jnelson+pg...@jamponi.net wrote: The biggest

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-28 Thread Andres Freund
On 2013-05-28 10:12:05 -0500, Jon Nelson wrote: On Tue, May 28, 2013 at 9:19 AM, Robert Haas robertmh...@gmail.com wrote: On Tue, May 28, 2013 at 10:15 AM, Andres Freund and...@2ndquadrant.com wrote: On 2013-05-28 10:03:58 -0400, Robert Haas wrote: On Sat, May 25, 2013 at 2:55 PM, Jon

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-28 Thread Greg Smith
On 5/28/13 11:12 AM, Jon Nelson wrote: It opens a new file, fallocates 16MB, calls fdatasync. Outside of the run for performance testing, I think it would be good at this point to validate that there is really a 16MB file full of zeroes resulting from these operations. I am not really

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-28 Thread Jon Nelson
On Tue, May 28, 2013 at 10:36 AM, Greg Smith g...@2ndquadrant.com wrote: On 5/28/13 11:12 AM, Jon Nelson wrote: It opens a new file, fallocates 16MB, calls fdatasync. Outside of the run for performance testing, I think it would be good at this point to validate that there is really a 16MB

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-28 Thread Greg Smith
On 5/28/13 10:00 PM, Jon Nelson wrote: On Tue, May 28, 2013 at 10:36 AM, Greg Smith g...@2ndquadrant.com wrote: On 5/28/13 11:12 AM, Jon Nelson wrote: It opens a new file, fallocates 16MB, calls fdatasync. Outside of the run for performance testing, I think it would be good at this point

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-25 Thread Jon Nelson
On Thu, May 16, 2013 at 7:05 PM, Greg Smith g...@2ndquadrant.com wrote: On 5/16/13 9:16 AM, Jon Nelson wrote: Am I doing this the right way? Should I be posting the full patch each time, or incremental patches? There are guidelines for getting your patch in the right format at

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-17 Thread Andres Freund
On 2013-05-15 16:46:33 -0500, Jon Nelson wrote: * Is wal file creation performance actually relevant? Is the performance of a system running on fallocate()d wal files any different? In my limited testing, I noticed a drop of approx. 100ms per WAL file. I do not have a good idea for how

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-17 Thread Merlin Moncure
On Fri, May 17, 2013 at 4:47 AM, Andres Freund and...@2ndquadrant.com wrote: On 2013-05-15 16:46:33 -0500, Jon Nelson wrote: * Is wal file creation performance actually relevant? Is the performance of a system running on fallocate()d wal files any different? In my limited testing, I

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-17 Thread Merlin Moncure
On Fri, May 17, 2013 at 8:29 AM, Merlin Moncure mmonc...@gmail.com wrote: On Fri, May 17, 2013 at 4:47 AM, Andres Freund and...@2ndquadrant.com wrote: On 2013-05-15 16:46:33 -0500, Jon Nelson wrote: * Is wal file creation performance actually relevant? Is the performance of a system

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-17 Thread Andres Freund
On 2013-05-17 15:48:38 -0500, Merlin Moncure wrote: On Fri, May 17, 2013 at 8:29 AM, Merlin Moncure mmonc...@gmail.com wrote: On Fri, May 17, 2013 at 4:47 AM, Andres Freund and...@2ndquadrant.com wrote: On 2013-05-15 16:46:33 -0500, Jon Nelson wrote: * Is wal file creation performance

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-17 Thread Merlin Moncure
On Fri, May 17, 2013 at 4:18 PM, Andres Freund and...@2ndquadrant.com wrote: On 2013-05-17 15:48:38 -0500, Merlin Moncure wrote: On Fri, May 17, 2013 at 8:29 AM, Merlin Moncure mmonc...@gmail.com wrote: On Fri, May 17, 2013 at 4:47 AM, Andres Freund and...@2ndquadrant.com wrote: On

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-16 Thread Jon Nelson
On Wed, May 15, 2013 at 10:36 PM, Jon Nelson jnelson+pg...@jamponi.net wrote: On Wed, May 15, 2013 at 10:17 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Jon Nelson escribió: On Wed, May 15, 2013 at 4:46 PM, Jon Nelson jnelson+pg...@jamponi.net wrote: That's true. I originally wrote

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-16 Thread Alvaro Herrera
Jon Nelson escribió: Am I doing this the right way? Should I be posting the full patch each time, or incremental patches? Full patch each time is okay. Context-format patch is even better. -- Álvaro Herrerahttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support,

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-16 Thread Greg Smith
On 5/16/13 9:16 AM, Jon Nelson wrote: Am I doing this the right way? Should I be posting the full patch each time, or incremental patches? There are guidelines for getting your patch in the right format at https://wiki.postgresql.org/wiki/Working_with_Git#Context_diffs_with_Git that would

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-15 Thread Jon Nelson
On Tue, May 14, 2013 at 9:43 PM, Robert Haas robertmh...@gmail.com wrote: On Mon, May 13, 2013 at 9:54 PM, Jon Nelson jnelson+pg...@jamponi.net wrote: Pertinent to another thread titled [HACKERS] corrupt pages detected by enabling checksums I hope to explore the possibility of using fallocate

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-15 Thread Andres Freund
Hi, On 2013-05-15 16:26:15 -0500, Jon Nelson wrote: I have written up a patch to use posix_fallocate in new WAL file creation, including configuration by way of a GUC variable, but I've not contributed to the PostgreSQL project before. Therefore, I'm fairly certain the patch is not

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-15 Thread Jon Nelson
On Wed, May 15, 2013 at 4:34 PM, Andres Freund and...@2ndquadrant.com wrote: Hi, On 2013-05-15 16:26:15 -0500, Jon Nelson wrote: I have written up a patch to use posix_fallocate in new WAL file creation, including configuration by way of a GUC variable, but I've not contributed to the

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-15 Thread Jon Nelson
On Wed, May 15, 2013 at 4:46 PM, Jon Nelson jnelson+pg...@jamponi.net wrote: On Wed, May 15, 2013 at 4:34 PM, Andres Freund and...@2ndquadrant.com wrote: .. Some where quick comments, without thinking about this: Thank you for the kind feedback. * needs a configure check for posix_fallocate.

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-15 Thread Alvaro Herrera
Jon Nelson escribió: On Wed, May 15, 2013 at 4:46 PM, Jon Nelson jnelson+pg...@jamponi.net wrote: That's true. I originally wrote the patch using fallocate(2). What would be appropriate here? Should I switch on the return value and the six (6) or so relevant error codes? I addressed

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-15 Thread Jon Nelson
On Wed, May 15, 2013 at 10:17 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Jon Nelson escribió: On Wed, May 15, 2013 at 4:46 PM, Jon Nelson jnelson+pg...@jamponi.net wrote: That's true. I originally wrote the patch using fallocate(2). What would be appropriate here? Should I switch

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-14 Thread Robert Haas
On Mon, May 13, 2013 at 9:54 PM, Jon Nelson jnelson+pg...@jamponi.net wrote: Pertinent to another thread titled [HACKERS] corrupt pages detected by enabling checksums I hope to explore the possibility of using fallocate (or posix_fallocate) for new WAL file creation. Most modern Linux

[HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-13 Thread Jon Nelson
Pertinent to another thread titled [HACKERS] corrupt pages detected by enabling checksums I hope to explore the possibility of using fallocate (or posix_fallocate) for new WAL file creation. Most modern Linux filesystems support fast fallocate/posix_fallocate, reducing extent fragmentation (where

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-13 Thread David Fetter
On Mon, May 13, 2013 at 08:54:39PM -0500, Jon Nelson wrote: Pertinent to another thread titled [HACKERS] corrupt pages detected by enabling checksums I hope to explore the possibility of using fallocate (or posix_fallocate) for new WAL file creation. Most modern Linux filesystems support