Re: [HACKERS] Dynamic Shared Memory stuff

2014-04-16 Thread Robert Haas
On Tue, Apr 15, 2014 at 10:46 PM, Amit Kapila amit.kapil...@gmail.com wrote: On Wed, Apr 16, 2014 at 3:01 AM, Robert Haas robertmh...@gmail.com wrote: On Tue, Apr 15, 2014 at 12:33 AM, Amit Kapila amit.kapil...@gmail.com wrote: On Mon, Apr 14, 2014 at 10:03 PM, Robert Haas

Re: [HACKERS] Dynamic Shared Memory stuff

2014-04-15 Thread Robert Haas
On Tue, Apr 15, 2014 at 12:33 AM, Amit Kapila amit.kapil...@gmail.com wrote: On Mon, Apr 14, 2014 at 10:03 PM, Robert Haas robertmh...@gmail.com wrote: On Sat, Apr 12, 2014 at 1:32 AM, Amit Kapila amit.kapil...@gmail.com wrote: I have checked that other place in code also check handle to

Re: [HACKERS] Dynamic Shared Memory stuff

2014-04-15 Thread Amit Kapila
On Wed, Apr 16, 2014 at 3:01 AM, Robert Haas robertmh...@gmail.com wrote: On Tue, Apr 15, 2014 at 12:33 AM, Amit Kapila amit.kapil...@gmail.com wrote: On Mon, Apr 14, 2014 at 10:03 PM, Robert Haas robertmh...@gmail.com wrote: For the create case, I'm wondering if we should put the block that

Re: [HACKERS] Dynamic Shared Memory stuff

2014-04-14 Thread Robert Haas
On Sat, Apr 12, 2014 at 1:32 AM, Amit Kapila amit.kapil...@gmail.com wrote: On Wed, Apr 9, 2014 at 9:20 PM, Robert Haas robertmh...@gmail.com wrote: On Wed, Apr 9, 2014 at 7:41 AM, Amit Kapila amit.kapil...@gmail.com wrote: I am just not sure whether it is okay to rearrange the code and call

Re: [HACKERS] Dynamic Shared Memory stuff

2014-04-14 Thread Amit Kapila
On Mon, Apr 14, 2014 at 10:03 PM, Robert Haas robertmh...@gmail.com wrote: On Sat, Apr 12, 2014 at 1:32 AM, Amit Kapila amit.kapil...@gmail.com wrote: I have checked that other place in code also check handle to decide if API has failed. Refer function PGSharedMemoryIsInUse(). So I think fix

Re: [HACKERS] Dynamic Shared Memory stuff

2014-04-11 Thread Amit Kapila
On Wed, Apr 9, 2014 at 9:20 PM, Robert Haas robertmh...@gmail.com wrote: On Wed, Apr 9, 2014 at 7:41 AM, Amit Kapila amit.kapil...@gmail.com wrote: I am just not sure whether it is okay to rearrange the code and call GetLastError() only if returned handle is Invalid (NULL) or try to look for

Re: [HACKERS] Dynamic Shared Memory stuff

2014-04-09 Thread Amit Kapila
On Tue, Apr 8, 2014 at 9:15 PM, Robert Haas robertmh...@gmail.com wrote: Apparently not. However, I'm fairly sure this is a step toward addressing the complaints previously raised, even if there may be some details people still want changed, so I've gone ahead and committed it. Few

Re: [HACKERS] Dynamic Shared Memory stuff

2014-04-09 Thread Robert Haas
On Wed, Apr 9, 2014 at 7:41 AM, Amit Kapila amit.kapil...@gmail.com wrote: Few Observations: 1. One new warning has been introduced in code. 1src\backend\port\win32_shmem.c(295): warning C4013: 'dsm_set_control_handle' undefined; assuming extern returning int Attached patch fixes this

Re: [HACKERS] Dynamic Shared Memory stuff

2014-04-09 Thread Andres Freund
On 2014-04-09 11:50:33 -0400, Robert Haas wrote: One question: 1. I have seen that initdb still creates pg_dynshmem, is it required after your latest changes? It's only used now if dynamic_shared_memory_type = mmap. I know Andres was never a huge fan of the mmap implementation, so we

Re: [HACKERS] Dynamic Shared Memory stuff

2014-04-08 Thread Robert Haas
On Fri, Apr 4, 2014 at 10:01 AM, Robert Haas robertmh...@gmail.com wrote: On Wed, Jan 22, 2014 at 10:17 AM, Noah Misch n...@leadboat.com wrote: Yeah, abandoning the state file is looking attractive. Here's a draft patch getting rid of the state file. This should address concerns raised by

Re: [HACKERS] Dynamic Shared Memory stuff

2014-04-04 Thread Robert Haas
On Wed, Jan 22, 2014 at 10:17 AM, Noah Misch n...@leadboat.com wrote: Yeah, abandoning the state file is looking attractive. Here's a draft patch getting rid of the state file. This should address concerns raised by Heikki and Fujii Masao and echoed by Tom that dynamic shared memory behaves

Re: [HACKERS] Dynamic Shared Memory stuff

2014-01-22 Thread Robert Haas
On Tue, Jan 21, 2014 at 2:58 PM, Noah Misch n...@leadboat.com wrote: What do people prefer? I recommend performing cleanup on the control segment named in PGShmemHeader just before shmdt() in PGSharedMemoryCreate(). No new ERROR or WARNING sites are necessary. Have dsm_postmaster_startup()

Re: [HACKERS] Dynamic Shared Memory stuff

2014-01-22 Thread Noah Misch
On Wed, Jan 22, 2014 at 09:32:09AM -0500, Robert Haas wrote: On Tue, Jan 21, 2014 at 2:58 PM, Noah Misch n...@leadboat.com wrote: What do people prefer? I recommend performing cleanup on the control segment named in PGShmemHeader just before shmdt() in PGSharedMemoryCreate(). No new

Re: [HACKERS] Dynamic Shared Memory stuff

2014-01-21 Thread Noah Misch
On Wed, Dec 18, 2013 at 12:21:08PM -0500, Robert Haas wrote: On Tue, Dec 10, 2013 at 6:26 PM, Tom Lane t...@sss.pgh.pa.us wrote: The larger point is that such a shutdown process has never in the history of Postgres been successful at removing shared-memory (or semaphore) resources. I do

Re: [HACKERS] Dynamic Shared Memory stuff

2013-12-18 Thread Robert Haas
On Tue, Dec 10, 2013 at 6:26 PM, Tom Lane t...@sss.pgh.pa.us wrote: Noah Misch n...@leadboat.com writes: On Tue, Dec 10, 2013 at 07:50:20PM +0200, Heikki Linnakangas wrote: Let's not add more cases like that, if we can avoid it. Only if we can avoid it for a modicum of effort and feature

Re: [HACKERS] Dynamic Shared Memory stuff

2013-12-10 Thread Noah Misch
On Thu, Dec 05, 2013 at 06:12:48PM +0200, Heikki Linnakangas wrote: On 11/20/2013 09:58 PM, Robert Haas wrote: On Wed, Nov 20, 2013 at 8:32 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: * As discussed in the Something fishy happening on frogmouth thread, I don't like the fact that

Re: [HACKERS] Dynamic Shared Memory stuff

2013-12-10 Thread Heikki Linnakangas
On 12/10/2013 07:27 PM, Noah Misch wrote: On Thu, Dec 05, 2013 at 06:12:48PM +0200, Heikki Linnakangas wrote: On 11/20/2013 09:58 PM, Robert Haas wrote: On Wed, Nov 20, 2013 at 8:32 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: * As discussed in the Something fishy happening on

Re: [HACKERS] Dynamic Shared Memory stuff

2013-12-10 Thread Noah Misch
On Tue, Dec 10, 2013 at 07:50:20PM +0200, Heikki Linnakangas wrote: On 12/10/2013 07:27 PM, Noah Misch wrote: On Thu, Dec 05, 2013 at 06:12:48PM +0200, Heikki Linnakangas wrote: On Wed, Nov 20, 2013 at 8:32 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: * As discussed in the Something

Re: [HACKERS] Dynamic Shared Memory stuff

2013-12-10 Thread Andres Freund
On 2013-12-10 18:12:53 -0500, Noah Misch wrote: On Tue, Dec 10, 2013 at 07:50:20PM +0200, Heikki Linnakangas wrote: On 12/10/2013 07:27 PM, Noah Misch wrote: On Thu, Dec 05, 2013 at 06:12:48PM +0200, Heikki Linnakangas wrote: Let's not add more cases like that, if we can avoid it. Only

Re: [HACKERS] Dynamic Shared Memory stuff

2013-12-10 Thread Tom Lane
Noah Misch n...@leadboat.com writes: On Tue, Dec 10, 2013 at 07:50:20PM +0200, Heikki Linnakangas wrote: Let's not add more cases like that, if we can avoid it. Only if we can avoid it for a modicum of effort and feature compromise. You're asking for PostgreSQL to reshape its use of

Re: [HACKERS] Dynamic Shared Memory stuff

2013-12-06 Thread Robert Haas
On Thu, Dec 5, 2013 at 4:06 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: That's a very interesting idea. I've been thinking that we needed to preserve the property that new workers could attach to the shared memory segment at any time, but that might not be necessary in all case. We

Re: [HACKERS] Dynamic Shared Memory stuff

2013-12-05 Thread Heikki Linnakangas
On 11/20/2013 09:58 PM, Robert Haas wrote: On Wed, Nov 20, 2013 at 8:32 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: How many allocations? What size will they have have typically, minimum and maximum? The facility is intended to be general, so the answer could vary widely by

Re: [HACKERS] Dynamic Shared Memory stuff

2013-12-05 Thread Robert Haas
On Thu, Dec 5, 2013 at 11:12 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: Hmm. Those two use cases are quite different. For message-passing, you want a lot of small queues, but for parallel sort, you want one huge allocation. I wonder if we shouldn't even try a one-size-fits-all

Re: [HACKERS] Dynamic Shared Memory stuff

2013-12-05 Thread Heikki Linnakangas
On 12/05/2013 09:34 PM, Robert Haas wrote: On Thu, Dec 5, 2013 at 11:12 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: One idea is to create the shared memory object with shm_open, and wait until all the worker processes that need it have attached to it. Then, shm_unlink() it, before

Re: [HACKERS] Dynamic Shared Memory stuff

2013-11-23 Thread Jeremy Harris
On 20/11/13 19:58, Robert Haas wrote: Parallel sort, and then parallel other stuff. Eventually general parallel query. I have recently updated https://wiki.postgresql.org/wiki/Parallel_Sort and you may find that interesting/helpful as a statement of intent. I've been playing with an internal

Re: [HACKERS] Dynamic Shared Memory stuff

2013-11-23 Thread Peter Geoghegan
On Sat, Nov 23, 2013 at 4:21 PM, Jeremy Harris j...@wizmail.org wrote: Its performance shines on partially- or reverse-sorted input. Search the archives for the work I did on timsort support a while back. A patch was posted, that had some impressive results provided you just considered the

[HACKERS] Dynamic Shared Memory stuff

2013-11-20 Thread Heikki Linnakangas
I'm trying to catch up on all of this dynamic shared memory stuff. A bunch of random questions and complaints: What kind of usage are we trying to cater with the dynamic shared memory? How many allocations? What size will they have have typically, minimum and maximum? I looked at the message

Re: [HACKERS] Dynamic Shared Memory stuff

2013-11-20 Thread Robert Haas
On Wed, Nov 20, 2013 at 8:32 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: I'm trying to catch up on all of this dynamic shared memory stuff. A bunch of random questions and complaints: What kind of usage are we trying to cater with the dynamic shared memory? Parallel sort, and then