Re: [HACKERS] Dynamic Shared Memory stuff

2014-04-16 Thread Robert Haas
On Tue, Apr 15, 2014 at 10:46 PM, Amit Kapila wrote: > On Wed, Apr 16, 2014 at 3:01 AM, Robert Haas wrote: >> On Tue, Apr 15, 2014 at 12:33 AM, Amit Kapila >> wrote: >>> On Mon, Apr 14, 2014 at 10:03 PM, Robert Haas wrote: For the create case, I'm wondering if we should put the block that

Re: [HACKERS] Dynamic Shared Memory stuff

2014-04-15 Thread Amit Kapila
On Wed, Apr 16, 2014 at 3:01 AM, Robert Haas wrote: > On Tue, Apr 15, 2014 at 12:33 AM, Amit Kapila wrote: >> On Mon, Apr 14, 2014 at 10:03 PM, Robert Haas wrote: >>> For the create case, I'm wondering if we should put the block that >>> tests for !hmap *before* the _dosmaperr() and check for EE

Re: [HACKERS] Dynamic Shared Memory stuff

2014-04-15 Thread Robert Haas
On Tue, Apr 15, 2014 at 12:33 AM, Amit Kapila wrote: > On Mon, Apr 14, 2014 at 10:03 PM, Robert Haas wrote: >> On Sat, Apr 12, 2014 at 1:32 AM, Amit Kapila wrote: >>> I have checked that other place in code also check handle to >>> decide if API has failed. Refer function PGSharedMemoryIsInUse(

Re: [HACKERS] Dynamic Shared Memory stuff

2014-04-14 Thread Amit Kapila
On Mon, Apr 14, 2014 at 10:03 PM, Robert Haas wrote: > On Sat, Apr 12, 2014 at 1:32 AM, Amit Kapila wrote: >> I have checked that other place in code also check handle to >> decide if API has failed. Refer function PGSharedMemoryIsInUse(). >> So I think fix to call GetLastError() after checking

Re: [HACKERS] Dynamic Shared Memory stuff

2014-04-14 Thread Robert Haas
On Sat, Apr 12, 2014 at 1:32 AM, Amit Kapila wrote: > On Wed, Apr 9, 2014 at 9:20 PM, Robert Haas wrote: >> On Wed, Apr 9, 2014 at 7:41 AM, Amit Kapila wrote: >>> I am just not sure whether it is okay to rearrange the code and call >>> GetLastError() only if returned handle is Invalid (NULL) or

Re: [HACKERS] Dynamic Shared Memory stuff

2014-04-11 Thread Amit Kapila
On Wed, Apr 9, 2014 at 9:20 PM, Robert Haas wrote: > On Wed, Apr 9, 2014 at 7:41 AM, Amit Kapila wrote: >> I am just not sure whether it is okay to rearrange the code and call >> GetLastError() only if returned handle is Invalid (NULL) or try to look >> for more info. > > Well, I don't know eithe

Re: [HACKERS] Dynamic Shared Memory stuff

2014-04-09 Thread Andres Freund
On 2014-04-09 11:50:33 -0400, Robert Haas wrote: > > One question: > > 1. I have seen that initdb still creates pg_dynshmem, is it required > > after your latest changes? > > It's only used now if dynamic_shared_memory_type = mmap. I know > Andres was never a huge fan of the mmap implementation,

Re: [HACKERS] Dynamic Shared Memory stuff

2014-04-09 Thread Robert Haas
On Wed, Apr 9, 2014 at 7:41 AM, Amit Kapila wrote: > Few Observations: > > 1. One new warning has been introduced in code. > 1>src\backend\port\win32_shmem.c(295): warning C4013: > 'dsm_set_control_handle' undefined; assuming extern returning int > Attached patch fixes this warning. OK, committed

Re: [HACKERS] Dynamic Shared Memory stuff

2014-04-09 Thread Amit Kapila
On Tue, Apr 8, 2014 at 9:15 PM, Robert Haas wrote: > Apparently not. However, I'm fairly sure this is a step toward > addressing the complaints previously raised, even if there may be some > details people still want changed, so I've gone ahead and committed > it. Few Observations: 1. One new w

Re: [HACKERS] Dynamic Shared Memory stuff

2014-04-08 Thread Robert Haas
On Fri, Apr 4, 2014 at 10:01 AM, Robert Haas wrote: > On Wed, Jan 22, 2014 at 10:17 AM, Noah Misch wrote: >> Yeah, abandoning the state file is looking attractive. > > Here's a draft patch getting rid of the state file. This should > address concerns raised by Heikki and Fujii Masao and echoed b

Re: [HACKERS] Dynamic Shared Memory stuff

2014-04-04 Thread Robert Haas
On Wed, Jan 22, 2014 at 10:17 AM, Noah Misch wrote: > Yeah, abandoning the state file is looking attractive. Here's a draft patch getting rid of the state file. This should address concerns raised by Heikki and Fujii Masao and echoed by Tom that dynamic shared memory behaves differently than the

Re: [HACKERS] Dynamic Shared Memory stuff

2014-01-22 Thread Noah Misch
On Wed, Jan 22, 2014 at 09:32:09AM -0500, Robert Haas wrote: > On Tue, Jan 21, 2014 at 2:58 PM, Noah Misch wrote: > >> What do people prefer? > > > > I recommend performing cleanup on the control segment named in PGShmemHeader > > just before shmdt() in PGSharedMemoryCreate(). No new ERROR or WAR

Re: [HACKERS] Dynamic Shared Memory stuff

2014-01-22 Thread Robert Haas
On Tue, Jan 21, 2014 at 2:58 PM, Noah Misch wrote: >> What do people prefer? > > I recommend performing cleanup on the control segment named in PGShmemHeader > just before shmdt() in PGSharedMemoryCreate(). No new ERROR or WARNING sites > are necessary. Have dsm_postmaster_startup() continue to

Re: [HACKERS] Dynamic Shared Memory stuff

2014-01-21 Thread Noah Misch
On Wed, Dec 18, 2013 at 12:21:08PM -0500, Robert Haas wrote: > On Tue, Dec 10, 2013 at 6:26 PM, Tom Lane wrote: > > The larger point is that such a shutdown process has never in the history > > of Postgres been successful at removing shared-memory (or semaphore) > > resources. I do not feel a nee

Re: [HACKERS] Dynamic Shared Memory stuff

2013-12-18 Thread Robert Haas
On Tue, Dec 10, 2013 at 6:26 PM, Tom Lane wrote: > Noah Misch writes: >> On Tue, Dec 10, 2013 at 07:50:20PM +0200, Heikki Linnakangas wrote: >>> Let's not add more cases like that, if we can avoid it. > >> Only if we can avoid it for a modicum of effort and feature compromise. >> You're asking fo

Re: [HACKERS] Dynamic Shared Memory stuff

2013-12-10 Thread Tom Lane
Noah Misch writes: > On Tue, Dec 10, 2013 at 07:50:20PM +0200, Heikki Linnakangas wrote: >> Let's not add more cases like that, if we can avoid it. > Only if we can avoid it for a modicum of effort and feature compromise. > You're asking for PostgreSQL to reshape its use of persistent resources s

Re: [HACKERS] Dynamic Shared Memory stuff

2013-12-10 Thread Andres Freund
On 2013-12-10 18:12:53 -0500, Noah Misch wrote: > On Tue, Dec 10, 2013 at 07:50:20PM +0200, Heikki Linnakangas wrote: > > On 12/10/2013 07:27 PM, Noah Misch wrote: > > >On Thu, Dec 05, 2013 at 06:12:48PM +0200, Heikki Linnakangas wrote: > > Let's not add more cases like that, if we can avoid it. >

Re: [HACKERS] Dynamic Shared Memory stuff

2013-12-10 Thread Noah Misch
On Tue, Dec 10, 2013 at 07:50:20PM +0200, Heikki Linnakangas wrote: > On 12/10/2013 07:27 PM, Noah Misch wrote: > >On Thu, Dec 05, 2013 at 06:12:48PM +0200, Heikki Linnakangas wrote: > >>>On Wed, Nov 20, 2013 at 8:32 AM, Heikki Linnakangas > >>> wrote: > * As discussed in the "Something fishy

Re: [HACKERS] Dynamic Shared Memory stuff

2013-12-10 Thread Heikki Linnakangas
On 12/10/2013 07:27 PM, Noah Misch wrote: On Thu, Dec 05, 2013 at 06:12:48PM +0200, Heikki Linnakangas wrote: On 11/20/2013 09:58 PM, Robert Haas wrote: On Wed, Nov 20, 2013 at 8:32 AM, Heikki Linnakangas wrote: * As discussed in the "Something fishy happening on frogmouth" thread, I don't l

Re: [HACKERS] Dynamic Shared Memory stuff

2013-12-10 Thread Noah Misch
On Thu, Dec 05, 2013 at 06:12:48PM +0200, Heikki Linnakangas wrote: > On 11/20/2013 09:58 PM, Robert Haas wrote: >> On Wed, Nov 20, 2013 at 8:32 AM, Heikki Linnakangas >> wrote: >>> * As discussed in the "Something fishy happening on frogmouth" thread, I >>> don't like the fact that the dynamic s

Re: [HACKERS] Dynamic Shared Memory stuff

2013-12-06 Thread Robert Haas
On Thu, Dec 5, 2013 at 4:06 PM, Heikki Linnakangas wrote: >> That's a very interesting idea. I've been thinking that we needed to >> preserve the property that new workers could attach to the shared >> memory segment at any time, but that might not be necessary in all >> case. We could introduce

Re: [HACKERS] Dynamic Shared Memory stuff

2013-12-05 Thread Heikki Linnakangas
On 12/05/2013 09:34 PM, Robert Haas wrote: On Thu, Dec 5, 2013 at 11:12 AM, Heikki Linnakangas wrote: One idea is to create the shared memory object with shm_open, and wait until all the worker processes that need it have attached to it. Then, shm_unlink() it, before using it for anything. That

Re: [HACKERS] Dynamic Shared Memory stuff

2013-12-05 Thread Robert Haas
On Thu, Dec 5, 2013 at 11:12 AM, Heikki Linnakangas wrote: > Hmm. Those two use cases are quite different. For message-passing, you want > a lot of small queues, but for parallel sort, you want one huge allocation. > I wonder if we shouldn't even try a one-size-fits-all solution. > > For message-p

Re: [HACKERS] Dynamic Shared Memory stuff

2013-12-05 Thread Heikki Linnakangas
On 11/20/2013 09:58 PM, Robert Haas wrote: On Wed, Nov 20, 2013 at 8:32 AM, Heikki Linnakangas wrote: How many allocations? What size will they have have typically, minimum and maximum? The facility is intended to be general, so the answer could vary widely by application. The testing that I

Re: [HACKERS] Dynamic Shared Memory stuff

2013-11-23 Thread Peter Geoghegan
On Sat, Nov 23, 2013 at 4:21 PM, Jeremy Harris wrote: > Its performance shines on partially- or reverse-sorted input. Search the archives for the work I did on timsort support a while back. A patch was posted, that had some impressive results provided you just considered the number of comparisons

Re: [HACKERS] Dynamic Shared Memory stuff

2013-11-23 Thread Jeremy Harris
On 20/11/13 19:58, Robert Haas wrote: Parallel sort, and then parallel other stuff. Eventually general parallel query. I have recently updated https://wiki.postgresql.org/wiki/Parallel_Sort and you may find that interesting/helpful as a statement of intent. I've been playing with an internal

Re: [HACKERS] Dynamic Shared Memory stuff

2013-11-20 Thread Robert Haas
On Wed, Nov 20, 2013 at 8:32 AM, Heikki Linnakangas wrote: > I'm trying to catch up on all of this dynamic shared memory stuff. A bunch > of random questions and complaints: > > What kind of usage are we trying to cater with the dynamic shared memory? Parallel sort, and then parallel other stuff.