Re: [HACKERS] POSIX shared memory redux

2011-04-14 Thread Florian Weimer
* Tom Lane: Well, the fundamental point is that ignoring NFS is not the real world. We can't tell people not to put data directories on NFS, and even if we did tell them not to, they'd still do it. And NFS locking is not trustworthy, because the remote lock daemon can crash and restart

Re: [HACKERS] POSIX shared memory redux

2011-04-14 Thread A.M.
On Apr 13, 2011, at 9:30 PM, Robert Haas wrote: On Wed, Apr 13, 2011 at 6:11 PM, A.M. age...@themactionfaction.com wrote: I don't see why we need to get rid of SysV shared memory; needing less of it seems just as good. 1. As long one keeps SysV shared memory around, the postgresql project

Re: [HACKERS] POSIX shared memory redux

2011-04-14 Thread A.M.
On Apr 14, 2011, at 8:22 AM, Florian Weimer wrote: * Tom Lane: Well, the fundamental point is that ignoring NFS is not the real world. We can't tell people not to put data directories on NFS, and even if we did tell them not to, they'd still do it. And NFS locking is not trustworthy,

Re: [HACKERS] POSIX shared memory redux

2011-04-14 Thread A.M.
On Apr 13, 2011, at 11:37 PM, Tom Lane wrote: A.M. age...@themactionfaction.com writes: 1. As long one keeps SysV shared memory around, the postgresql project has to maintain the annoying platform-specific document on how to configure the poorly named kernel parameters. No, if it's just a

Re: [HACKERS] POSIX shared memory redux

2011-04-14 Thread Martijn van Oosterhout
On Thu, Apr 14, 2011 at 10:26:33AM -0400, A.M. wrote: 1) the SysV nattch method's primary purpose is to protect the shmem region. This is no longer necessary in my patch because the shared memory in unlinked immediately after creation, so only the initial postmaster and its children have

Re: [HACKERS] POSIX shared memory redux

2011-04-14 Thread Robert Haas
On Thu, Apr 14, 2011 at 7:26 AM, A.M. age...@themactionfaction.com wrote: From what I understood, the primary purpose of the SysV check was to protect the shared memory from multiple stompers. The interlock was a neat side-effect. Not really - the purpose of the interlock is to protect the

Re: [HACKERS] POSIX shared memory redux

2011-04-13 Thread Tom Lane
A.M. age...@themactionfaction.com writes: On Apr 11, 2011, at 7:13 PM, Tom Lane wrote: Robert Haas robertmh...@gmail.com writes: I mean I'm not convinced that fcntl() locking will be as reliable. I'm not either. Particularly not on NFS. Is there an example of a recent system where fcntl is

Re: [HACKERS] POSIX shared memory redux

2011-04-13 Thread A . M .
On Apr 13, 2011, at 2:06 AM, Tom Lane wrote: A.M. age...@themactionfaction.com writes: On Apr 11, 2011, at 7:13 PM, Tom Lane wrote: Robert Haas robertmh...@gmail.com writes: I mean I'm not convinced that fcntl() locking will be as reliable. I'm not either. Particularly not on NFS. Is

Re: [HACKERS] POSIX shared memory redux

2011-04-13 Thread Robert Haas
On Wed, Apr 13, 2011 at 7:20 AM, A.M. age...@themactionfaction.com wrote: The goal of this patch is to eliminate SysV shared memory, not to implement NFS-capable locking which, as you point out, is virtually impossible. As far as I can tell, in the worst case, my patch does not change how

Re: [HACKERS] POSIX shared memory redux

2011-04-13 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: In answer to your off-list question, one of the principle ways I've seen fcntl() locking fall over and die is when someone removes the lock file. You might think that this could be avoided by picking something important like pg_control as the log

Re: [HACKERS] POSIX shared memory redux

2011-04-13 Thread A.M.
On Apr 13, 2011, at 8:36 PM, Robert Haas wrote: I don't see why we need to get rid of SysV shared memory; needing less of it seems just as good. 1. As long one keeps SysV shared memory around, the postgresql project has to maintain the annoying platform-specific document on how to configure

Re: [HACKERS] POSIX shared memory redux

2011-04-13 Thread Robert Haas
On Wed, Apr 13, 2011 at 6:11 PM, A.M. age...@themactionfaction.com wrote: I don't see why we need to get rid of SysV shared memory; needing less of it seems just as good. 1. As long one keeps SysV shared memory around, the postgresql project has to maintain the annoying platform-specific

Re: [HACKERS] POSIX shared memory redux

2011-04-13 Thread Tom Lane
A.M. age...@themactionfaction.com writes: 1. As long one keeps SysV shared memory around, the postgresql project has to maintain the annoying platform-specific document on how to configure the poorly named kernel parameters. No, if it's just a small area, I don't see that that's an issue.

Re: [HACKERS] POSIX shared memory redux

2011-04-11 Thread Robert Haas
On Sun, Apr 10, 2011 at 5:03 PM, A.M. age...@themactionfaction.com wrote: To ensure that no two postmasters can startup in the same data directory, I use fcntl range locking on the data directory lock file, which also works properly on (properly configured) NFS volumes. Whenever a postmaster

Re: [HACKERS] POSIX shared memory redux

2011-04-11 Thread A.M.
On Apr 11, 2011, at 6:06 PM, Robert Haas wrote: On Sun, Apr 10, 2011 at 5:03 PM, A.M. age...@themactionfaction.com wrote: To ensure that no two postmasters can startup in the same data directory, I use fcntl range locking on the data directory lock file, which also works properly on

Re: [HACKERS] POSIX shared memory redux

2011-04-11 Thread Robert Haas
On Mon, Apr 11, 2011 at 3:11 PM, A.M. age...@themactionfaction.com wrote: On Apr 11, 2011, at 6:06 PM, Robert Haas wrote: On Sun, Apr 10, 2011 at 5:03 PM, A.M. age...@themactionfaction.com wrote: To ensure that no two postmasters can startup in the same data directory, I use fcntl range

Re: [HACKERS] POSIX shared memory redux

2011-04-11 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Mon, Apr 11, 2011 at 3:11 PM, A.M. age...@themactionfaction.com wrote: What do you mean by leakier? The goal here is to extinguish SysV shared memory for portability and convenience benefits. The mini-SysV proposal was implemented and shot down by

Re: [HACKERS] POSIX shared memory redux

2011-04-11 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Sun, Apr 10, 2011 at 5:03 PM, A.M. age...@themactionfaction.com wrote: To ensure that no two postmasters can startup in the same data directory, I use fcntl range locking on the data directory lock file, which also works properly on (properly

Re: [HACKERS] POSIX shared memory redux

2011-04-11 Thread A.M.
On Apr 11, 2011, at 7:25 PM, Tom Lane wrote: Robert Haas robertmh...@gmail.com writes: On Sun, Apr 10, 2011 at 5:03 PM, A.M. age...@themactionfaction.com wrote: To ensure that no two postmasters can startup in the same data directory, I use fcntl range locking on the data directory lock

Re: [HACKERS] POSIX shared memory redux

2011-04-11 Thread A.M.
On Apr 11, 2011, at 7:13 PM, Tom Lane wrote: Robert Haas robertmh...@gmail.com writes: On Mon, Apr 11, 2011 at 3:11 PM, A.M. age...@themactionfaction.com wrote: What do you mean by leakier? The goal here is to extinguish SysV shared memory for portability and convenience benefits. The

Re: [HACKERS] POSIX shared memory redux

2010-11-14 Thread Martijn van Oosterhout
On Sat, Nov 13, 2010 at 08:07:52PM -0500, Tom Lane wrote: A.M. age...@themactionfaction.com writes: The goal of this work is to address all of the shortcomings of previous POSIX shared memory patches as pointed out mostly by Tom Lane. It seems like you've failed to understand the main

Re: [HACKERS] POSIX shared memory redux

2010-11-14 Thread Tom Lane
Martijn van Oosterhout klep...@svana.org writes: The only real solution seems to me to be to keep a small SysV shared memory segment for the locking and allocate the rest of the shared memory some other way. Yeah, that's been discussed. It throws all the portability gains out the window. It

Re: [HACKERS] POSIX shared memory redux

2010-11-14 Thread Robert Haas
On Sun, Nov 14, 2010 at 11:06 AM, Tom Lane t...@sss.pgh.pa.us wrote: Martijn van Oosterhout klep...@svana.org writes: The only real solution seems to me to be to keep a small SysV shared memory segment for the locking and allocate the rest of the shared memory some other way. Yeah, that's

[HACKERS] POSIX shared memory redux

2010-11-13 Thread A . M .
The goal of this work is to address all of the shortcomings of previous POSIX shared memory patches as pointed out mostly by Tom Lane. Branch: http://git.postgresql.org/gitweb?p=users/agentm/postgresql.git;a=shortlog;h=refs/heads/posix_shmem Main file:

Re: [HACKERS] POSIX shared memory redux

2010-11-13 Thread Tom Lane
A.M. age...@themactionfaction.com writes: The goal of this work is to address all of the shortcomings of previous POSIX shared memory patches as pointed out mostly by Tom Lane. It seems like you've failed to understand the main shortcoming of this whole idea, which is the loss of ability to