Re: [HACKERS] [WIP] showing index maintenance on EXPLAIN

2014-05-07 Thread Jaime Casanova
On Wed, May 7, 2014 at 10:52 PM, Amit Kapila wrote: > On Thu, May 8, 2014 at 5:30 AM, Jaime Casanova wrote: >> Hi, >> >> This patch implements $subject only when ANALYZE and VERBOSE are on. >> I made it that way because for years nobody seemed interested in this >> info (at least no one did it) s

[HACKERS] Ignore files in src/interfaces/libpq generated by windows builds

2014-05-07 Thread Michael Paquier
Hi all, While doing some builds with mingw and 9.4, I noticed that a couple of files generated by build are not ignored in the source tree. Those two files are system.c and win32setlocale.c in src/interfaces/libpq. Please find attached a patch fixing that. Note that this is recent and is partially

[HACKERS] Compilation errors with mingw build caused by undefined optreset

2014-05-07 Thread Michael Paquier
Hi all, Since commit 60ff2fd introducing the centralizated getopt-related things in a global header file, build on Windows with mingw is failing because of some declarations of HAVE_INT_OPTRESET causing optreset to become undefined: postmaster.c: In function 'PostmasterMain': postmaster.c:853:2: e

[HACKERS] popen and pclose redefinitions causing many warning in Windows build

2014-05-07 Thread Michael Paquier
Hi all, Since commit a692ee5, code compilation on windows is full of warnings caused by the re-definitions of popen and pclose: In file included from ../../../src/include/c.h:1028:0, from ../../../src/include/postgres.h:47, from analyze.c:25: ../../../src/include/

Re: [HACKERS] PGDLLEXPORTing all GUCs?

2014-05-07 Thread Craig Ringer
On 05/08/2014 10:53 AM, Tom Lane wrote: > Craig Ringer writes: >> On 05/08/2014 12:21 AM, Tom Lane wrote: >>> If Craig has a concrete argument why all GUCs should be accessible >>> to external modules, then let's see it > >> As for just GUCs: I suggested GUCs because GUCs are what's been coming >

Re: [HACKERS] [WIP] showing index maintenance on EXPLAIN

2014-05-07 Thread Amit Kapila
On Thu, May 8, 2014 at 5:30 AM, Jaime Casanova wrote: > Hi, > > This patch implements $subject only when ANALYZE and VERBOSE are on. > I made it that way because for years nobody seemed interested in this > info (at least no one did it) so i decided that maybe is to much > information for most peo

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-07 Thread Kouhei Kaigai
> >> > * ForeignScan node that is not associated with a particular > foreign-table. > >> > Once we try to apply ForeignScan node instead of Sort or > >> > Aggregate, > >> existing > >> > FDW implementation needs to be improved. These nodes scan on a > >> materialized > >> > relation (generate

Re: [HACKERS] Unhappy with error handling in psql's handleCopyOut()

2014-05-07 Thread Noah Misch
On Tue, Feb 11, 2014 at 03:43:08PM -0500, Tom Lane wrote: > While looking at the pending patch to make psql report a line count > after COPY, I came across this business in handleCopyOut(): > > * Check command status and return to normal libpq state. After a > * client-side error, the s

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-07 Thread Shigeru Hanada
2014-05-07 18:06 GMT+09:00 Kouhei Kaigai : > Let me list up the things to be clarified / developed randomly. > > * Join replacement by FDW; We still don't have consensus about join > replacement > by FDW. Probably, it will be designed to remote-join implementation > primarily, > however, thin

Re: [HACKERS] PGDLLEXPORTing all GUCs?

2014-05-07 Thread Tom Lane
Craig Ringer writes: > On 05/08/2014 12:21 AM, Tom Lane wrote: >> If Craig has a concrete argument why all GUCs should be accessible >> to external modules, then let's see it > As for just GUCs: I suggested GUCs because GUCs are what's been coming > up repeatedly as an actual practical issue. Me

Re: [HACKERS] Wanted: jsonb on-disk representation documentation

2014-05-07 Thread Tom Lane
Andrew Dunstan writes: > On 05/07/2014 04:13 PM, Tom Lane wrote: >> A README file would be better, >> perhaps, but there's not a specific directory associated with the jsonb >> code; so I think this sort of info belongs either in jsonb.h or in the >> file header comment for jsonb_gin.c. > Is ther

Re: [HACKERS] Minor improvement to fdwhandler.sgml

2014-05-07 Thread Etsuro Fujita
(2014/05/05 23:05), Robert Haas wrote: On Wed, Apr 30, 2014 at 4:10 AM, Etsuro Fujita wrote: (2014/04/28 23:31), Robert Haas wrote: On Thu, Apr 24, 2014 at 7:59 AM, Etsuro Fujita wrote: The patch attached improves docs in fdwhandler.sgml a little bit. When you submit a patch, it's helpfu

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-07 Thread Stephen Frost
Simon, * Simon Riggs (si...@2ndquadrant.com) wrote: > I'm proposing something that is like an index, not like a plan node. > > The reason that proposal is being made is that we need to consider > data structure, data location and processing details. > > * In the case of Mat Views, if there is no

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-07 Thread Stephen Frost
Simon, * Simon Riggs (si...@2ndquadrant.com) wrote: > On 8 May 2014 01:49, Kouhei Kaigai wrote: > >From your description, my understanding is that you would like to > stream data from 2 standard tables to the GPU, then perform a join on > the GPU itself. > > I have been told that is not likely t

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-07 Thread Simon Riggs
On 7 May 2014 18:39, Stephen Frost wrote: > * Simon Riggs (si...@2ndquadrant.com) wrote: >> On 7 May 2014 17:43, Stephen Frost wrote: >> > It's the optimizer's job to figure out which path to pick though, based >> > on which will have the lowest cost. >> >> Of course. I'm not suggesting otherwise

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-07 Thread Simon Riggs
On 8 May 2014 01:49, Kouhei Kaigai wrote: >> > * ForeignScan node that is not associated with a particular foreign-table. >> > Once we try to apply ForeignScan node instead of Sort or Aggregate, >> existing >> > FDW implementation needs to be improved. These nodes scan on a >> materialized >>

Re: [HACKERS] 9.4 release notes

2014-05-07 Thread Tatsuo Ishii
> I have completed the initial version of the 9.4 release notes. You can > view them here: > > http://www.postgresql.org/docs/devel/static/release-9-4.html > > I will be adding additional markup in the next few days. > > Feedback expected and welcomed. I expect to be modifying this until

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-07 Thread Kouhei Kaigai
> > Let me list up the things to be clarified / developed randomly. > > > > * Join replacement by FDW; We still don't have consensus about join > replacement > > by FDW. Probably, it will be designed to remote-join implementation > primarily, > > however, things to do is similar. We may need to

Re: [HACKERS] bgworker crashed or not?

2014-05-07 Thread Craig Ringer
On 05/08/2014 05:45 AM, Robert Haas wrote: > I've pushed your rebased patch now with some further kibitzing, > especially in regards to the documentation. Thanks for tacking that Robert. It's great to have that API sorted out before 9.4 hits and it's locked in. -- Craig Ringer

Re: [HACKERS] PGDLLEXPORTing all GUCs?

2014-05-07 Thread Craig Ringer
On 05/08/2014 12:21 AM, Tom Lane wrote: > If Craig has a concrete argument why all GUCs should be accessible > to external modules, then let's see it Because they already are. The only difference here is that that access works only on !windows. I agree (strongly) that we should have a better def

Re: [HACKERS] proposal: Set effective_cache_size to greater of .conf value, shared_buffers

2014-05-07 Thread Peter Geoghegan
On Wed, May 7, 2014 at 12:06 PM, Josh Berkus wrote: > For that matter, our advice on shared_buffers ... and our design for it > ... is going to need to change radically soon, since Linux is getting an > ARC with a frequency cache as well as a recency cache, and FreeBSD and > OpenSolaris already ha

Re: [HACKERS] [WIP] showing index maintenance on EXPLAIN

2014-05-07 Thread Jaime Casanova
On Wed, May 7, 2014 at 7:03 PM, Josh Berkus wrote: > On 05/07/2014 05:00 PM, Jaime Casanova wrote: >> Hi, >> >> This patch implements $subject only when ANALYZE and VERBOSE are on. >> I made it that way because for years nobody seemed interested in this >> info (at least no one did it) so i decide

Re: [HACKERS] [WIP] showing index maintenance on EXPLAIN

2014-05-07 Thread Josh Berkus
On 05/07/2014 05:00 PM, Jaime Casanova wrote: > Hi, > > This patch implements $subject only when ANALYZE and VERBOSE are on. > I made it that way because for years nobody seemed interested in this > info (at least no one did it) so i decided that maybe is to much > information for most people (act

[HACKERS] [WIP] showing index maintenance on EXPLAIN

2014-05-07 Thread Jaime Casanova
Hi, This patch implements $subject only when ANALYZE and VERBOSE are on. I made it that way because for years nobody seemed interested in this info (at least no one did it) so i decided that maybe is to much information for most people (actually btree indexes are normally very fast). But because

Re: [HACKERS] bgworker crashed or not?

2014-05-07 Thread Petr Jelinek
On 07/05/14 23:45, Robert Haas wrote: It was, but I felt that the different pieces of this should be separated into separate commits, and (regardless of how I committed it) I needed to review each change separately. I wasn't saying it was your fault that it wasn't done; just that I hadn't gotten

Re: [HACKERS] pg_shmem_allocations view

2014-05-07 Thread Andres Freund
On 2014-05-07 17:48:15 -0400, Robert Haas wrote: > On Tue, May 6, 2014 at 6:09 PM, Andres Freund wrote: > >> I guess I'd vote for > >> ditching the allocated column completely and outputting the memory > >> allocated without ShmemIndex using some fixed tag (like "ShmemIndex" > >> or "Bootstrap" or

Re: [HACKERS] proposal: Set effective_cache_size to greater of .conf value, shared_buffers

2014-05-07 Thread Andres Freund
On 2014-05-07 16:24:53 -0500, Merlin Moncure wrote: > On Wed, May 7, 2014 at 4:15 PM, Andres Freund wrote: > > On 2014-05-07 13:51:57 -0700, Jeff Janes wrote: > >> On Wed, May 7, 2014 at 11:38 AM, Andres Freund > >> wrote: > >> > >> > On 2014-05-07 13:32:41 -0500, Merlin Moncure wrote: > >> > > >

Re: [HACKERS] pg_shmem_allocations view

2014-05-07 Thread Robert Haas
On Tue, May 6, 2014 at 6:09 PM, Andres Freund wrote: >> I guess I'd vote for >> ditching the allocated column completely and outputting the memory >> allocated without ShmemIndex using some fixed tag (like "ShmemIndex" >> or "Bootstrap" or "Overhead" or something). > > My way feels slightly cleane

Re: [HACKERS] bgworker crashed or not?

2014-05-07 Thread Robert Haas
On Wed, May 7, 2014 at 4:55 PM, Petr Jelinek wrote: >> This isn't done yet. > > Unless I am missing something this change was included in every patch I sent > - setting rw->rw_terminate = true; in CleanupBackgroundWorker for zero exit > code + comment changes. Or do you have objections to this app

Re: [HACKERS] proposal: Set effective_cache_size to greater of .conf value, shared_buffers

2014-05-07 Thread Peter Geoghegan
On Wed, May 7, 2014 at 2:24 PM, Merlin Moncure wrote: > right. This is, IMNSHO, exactly the sort of language that belongs in the > docs. +1 -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.o

Re: [HACKERS] jsonb existence queries are misimplemented by jsonb_ops

2014-05-07 Thread Peter Geoghegan
On Wed, May 7, 2014 at 1:39 PM, Tom Lane wrote: > The indexscan is incorrectly returning rows where the queried key exists > but isn't at top-level. > > We could fix this either by giving up on no-recheck for existence queries, > or by changing the way that non-top-level keys get indexed. However

Re: [HACKERS] jsonb existence queries are misimplemented by jsonb_ops

2014-05-07 Thread Tom Lane
Peter Geoghegan writes: > On Wed, May 7, 2014 at 1:47 PM, Tom Lane wrote: >> No, wait, containment *doesn't* look into sub-objects: >> >> regression=# select * from j where f1 @> '{"foo": {"bar": "baz"}}'; >> f1 >> - >> {"foo": {"bar": "baz"}} >> (1 row) >> >> regression

Re: [HACKERS] proposal: Set effective_cache_size to greater of .conf value, shared_buffers

2014-05-07 Thread Merlin Moncure
On Wed, May 7, 2014 at 4:15 PM, Andres Freund wrote: > On 2014-05-07 13:51:57 -0700, Jeff Janes wrote: >> On Wed, May 7, 2014 at 11:38 AM, Andres Freund wrote: >> >> > On 2014-05-07 13:32:41 -0500, Merlin Moncure wrote: >> > > >> > > *) raising shared buffers does not 'give more memory to postgres

Re: [HACKERS] Wanted: jsonb on-disk representation documentation

2014-05-07 Thread Andrew Dunstan
On 05/07/2014 04:13 PM, Tom Lane wrote: A README file would be better, perhaps, but there's not a specific directory associated with the jsonb code; so I think this sort of info belongs either in jsonb.h or in the file header comment for jsonb_gin.c. Is there any reason we couldn't have a R

Re: [HACKERS] proposal: Set effective_cache_size to greater of .conf value, shared_buffers

2014-05-07 Thread Andres Freund
On 2014-05-07 13:51:57 -0700, Jeff Janes wrote: > On Wed, May 7, 2014 at 11:38 AM, Andres Freund wrote: > > > On 2014-05-07 13:32:41 -0500, Merlin Moncure wrote: > > > > > > *) raising shared buffers does not 'give more memory to postgres for > > > caching' -- it can only reduce it via double pagi

Re: [HACKERS] jsonb existence queries are misimplemented by jsonb_ops

2014-05-07 Thread Peter Geoghegan
On Wed, May 7, 2014 at 1:51 PM, Peter Geoghegan wrote: > Yes it does. It's just not intended to work like that. You have to > "give a path" to what you're looking for. There is exactly one exception explicitly noted as such in the user-visible docs, which is arrays and the containment of single

Re: [HACKERS] bgworker crashed or not?

2014-05-07 Thread Petr Jelinek
On 07/05/14 22:32, Robert Haas wrote: On Tue, May 6, 2014 at 8:25 PM, Petr Jelinek wrote: I've committed the portion of your patch which does this, with pretty extensive changes. I moved the function which resets the crash times to bgworker.c, renamed it, and made it just reset all of the crash

Re: [HACKERS] jsonb existence queries are misimplemented by jsonb_ops

2014-05-07 Thread Peter Geoghegan
On Wed, May 7, 2014 at 1:47 PM, Tom Lane wrote: > No, wait, containment *doesn't* look into sub-objects: > > regression=# select * from j where f1 @> '{"foo": {"bar": "baz"}}'; >f1 > - > {"foo": {"bar": "baz"}} > (1 row) > > regression=# select * from j where f

Re: [HACKERS] proposal: Set effective_cache_size to greater of .conf value, shared_buffers

2014-05-07 Thread Jeff Janes
On Wed, May 7, 2014 at 11:38 AM, Andres Freund wrote: > On 2014-05-07 13:32:41 -0500, Merlin Moncure wrote: > > > > *) raising shared buffers does not 'give more memory to postgres for > > caching' -- it can only reduce it via double paging > > That's absolutely not a necessary consequence. If pag

Re: [HACKERS] proposal: Set effective_cache_size to greater of .conf value, shared_buffers

2014-05-07 Thread Josh Berkus
On 05/07/2014 01:36 PM, Jeff Janes wrote: > On Wed, May 7, 2014 at 11:04 AM, Josh Berkus wrote: >> Unfortunately nobody has the time/resources to do the kind of testing >> required for a new recommendation for shared_buffers. > I think it is worse than that. I don't think we know what such test

Re: [HACKERS] jsonb existence queries are misimplemented by jsonb_ops

2014-05-07 Thread Tom Lane
I wrote: > Another idea would be to change the definition of the exists operator > so that it *does* look into sub-objects. It seems rather random to me > that containment looks into sub-objects but exists doesn't. However, > possibly there are good reasons for the non-orthogonality. No, wait, c

Re: [HACKERS] 9.4 checksum errors in recovery with gin index

2014-05-07 Thread Heikki Linnakangas
On 05/07/2014 10:35 AM, Jeff Janes wrote: When recovering from a crash (with injection of a partial page write at time of crash) against 7c7b1f4ae5ea3b1b113682d4d I get a checksum verification failure. 16396 is a gin index. If I have it ignore checksum failures, there is no apparent misbehavior

[HACKERS] jsonb existence queries are misimplemented by jsonb_ops

2014-05-07 Thread Tom Lane
I wrote: > The readability of that comment starts to go downhill with its use of > "reset" to refer to what everything else calls a "recheck" flag, and in > any case it's claiming that we *don't* need a recheck for exists (a > statement I suspect to be false, but more later). And, indeed, it's fal

Re: [HACKERS] proposal: Set effective_cache_size to greater of .conf value, shared_buffers

2014-05-07 Thread Jeff Janes
On Wed, May 7, 2014 at 11:04 AM, Josh Berkus wrote: > On 05/06/2014 10:35 PM, Peter Geoghegan wrote: > > +1. In my view, we probably should have set it to a much higher > > absolute default value. The main problem with setting it to any > > multiple of shared_buffers that I can see is that shared

Re: [HACKERS] bgworker crashed or not?

2014-05-07 Thread Robert Haas
On Tue, May 6, 2014 at 8:25 PM, Petr Jelinek wrote: >> What I'm inclined to do is change the logic so that: >> >> (1) After a crash-and-restart sequence, zero rw->rw_crashed_at, so >> that anything which is still registered gets restarted immediately. > > Yes, that's quite obvious change which I m

Re: [HACKERS] Wanted: jsonb on-disk representation documentation

2014-05-07 Thread Tom Lane
Peter Geoghegan writes: > On Wed, May 7, 2014 at 12:27 PM, Tom Lane wrote: >> The jsonb_ops storage format for values is inherently lossy, because it >> cannot distinguish the string values "n", "t", "f" from JSON null or >> boolean true, false respectively; nor does it know the difference betwee

Re: [HACKERS] 9.4 checksum errors in recovery with gin index

2014-05-07 Thread Jeff Janes
On Wed, May 7, 2014 at 10:34 AM, Andres Freund wrote: > Hi, > > On 2014-05-07 10:21:26 -0700, Jeff Janes wrote: > > On Wed, May 7, 2014 at 12:48 AM, Andres Freund >wrote: > > > If you have the WAL a pg_xlogdump grepping for everything referring to > > > that block would be helpful. > > > > > > >

Re: [HACKERS] Wanted: jsonb on-disk representation documentation

2014-05-07 Thread Tom Lane
Peter Geoghegan writes: > On Wed, May 7, 2014 at 12:08 PM, Tom Lane wrote: >> Is this an obsolete speculation about writing jsonb_hash_ops, or is there >> still something worth preserving there? If so, what? > This is not obsolete. It would equally apply to a GiST opclass that > wanted to suppo

Re: [HACKERS] Wanted: jsonb on-disk representation documentation

2014-05-07 Thread Peter Geoghegan
On Wed, May 7, 2014 at 12:27 PM, Tom Lane wrote: > The jsonb_ops storage format for values is inherently lossy, because it > cannot distinguish the string values "n", "t", "f" from JSON null or > boolean true, false respectively; nor does it know the difference between > a number and a string cont

Re: [HACKERS] proposal: Set effective_cache_size to greater of .conf value, shared_buffers

2014-05-07 Thread Robert Haas
On Wed, May 7, 2014 at 2:58 PM, Peter Geoghegan wrote: > On Wed, May 7, 2014 at 11:50 AM, Robert Haas wrote: >> But that does not mean, as the phrase "folk >> wisdom" might be taken to imply, that we don't know anything at all >> about what actually works well in practice. > > Folk wisdom doesn't

Re: [HACKERS] Wanted: jsonb on-disk representation documentation

2014-05-07 Thread Tom Lane
And while I'm looking at it ... The jsonb_ops storage format for values is inherently lossy, because it cannot distinguish the string values "n", "t", "f" from JSON null or boolean true, false respectively; nor does it know the difference between a number and a string containing digits. This appe

Re: [HACKERS] Wanted: jsonb on-disk representation documentation

2014-05-07 Thread Peter Geoghegan
On Wed, May 7, 2014 at 12:08 PM, Tom Lane wrote: > * Jsonb Keys and string array elements are treated equivalently when > * serialized to text index storage. One day we may wish to create an opclass > * that only indexes values, but for now keys and values are stored in GIN > * indexes in a w

Re: [HACKERS] proposal: Set effective_cache_size to greater of .conf value, shared_buffers

2014-05-07 Thread Peter Geoghegan
On Wed, May 7, 2014 at 11:50 AM, Robert Haas wrote: >> Doesn't match my experience. Even with the current buffer manager >> there's usually enough locality to keep important pages in s_b for a >> meaningful time. I *have* seen workloads that should have fit into >> memory not fit because of double

Re: [HACKERS] Wanted: jsonb on-disk representation documentation

2014-05-07 Thread Tom Lane
btw ... in jsonb.h there is this comment: * Jsonb Keys and string array elements are treated equivalently when * serialized to text index storage. One day we may wish to create an opclass * that only indexes values, but for now keys and values are stored in GIN * indexes in a way that doesn't

Re: [HACKERS] proposal: Set effective_cache_size to greater of .conf value, shared_buffers

2014-05-07 Thread Josh Berkus
On 05/07/2014 11:52 AM, Peter Geoghegan wrote: > On Wed, May 7, 2014 at 11:40 AM, Josh Berkus wrote: >> So, as one of several people who put literally hundreds of hours into >> the original benchmarking which established the sizing recommendations >> for shared_buffers (and other settings), I find

Re: [HACKERS] Wanted: jsonb on-disk representation documentation

2014-05-07 Thread Andrew Dunstan
On 05/07/2014 02:52 PM, Andres Freund wrote: On 2014-05-07 14:48:51 -0400, Tom Lane wrote: Andres Freund writes: * The jentry representation should be changed so it's possible to get the type of a entry without checking individual types. That'll make code like findJsonbValueFromSuperHea

Re: [HACKERS] making bgworkers without shmem access actually not have shmem access

2014-05-07 Thread Petr Jelinek
On 07/05/14 20:37, Robert Haas wrote: At a minimum, it's got to be better than the status quo, where shared memory is accessible throughout the entire lifetime of non-shmem-access background workers. Seems reasonable to me, it might need to be revisited to at least try to figure out if we can

Re: [HACKERS] proposal: Set effective_cache_size to greater of .conf value, shared_buffers

2014-05-07 Thread Peter Geoghegan
On Wed, May 7, 2014 at 11:50 AM, Robert Haas wrote: > But that does not mean, as the phrase "folk > wisdom" might be taken to imply, that we don't know anything at all > about what actually works well in practice. Folk wisdom doesn't imply that. It implies that we think this works, and we may wel

Re: [HACKERS] making bgworkers without shmem access actually not have shmem access

2014-05-07 Thread Robert Haas
On Wed, May 7, 2014 at 2:44 PM, Tom Lane wrote: > Robert Haas writes: >> I've complained about this problem a few times before: there's nothing >> to prevent a background worker which doesn't request shared memory >> access from calling InitProcess() and then accessing shared memory >> anyway. T

Re: [HACKERS] proposal: Set effective_cache_size to greater of .conf value, shared_buffers

2014-05-07 Thread Peter Geoghegan
On Wed, May 7, 2014 at 11:40 AM, Josh Berkus wrote: > So, as one of several people who put literally hundreds of hours into > the original benchmarking which established the sizing recommendations > for shared_buffers (and other settings), I find the phrase "folk wisdom" > personally offensive. S

Re: [HACKERS] Wanted: jsonb on-disk representation documentation

2014-05-07 Thread Andres Freund
On 2014-05-07 14:48:51 -0400, Tom Lane wrote: > Andres Freund writes: > > * The jentry representation should be changed so it's possible to get the > > type > > of a entry without checking individual types. That'll make code like > > findJsonbValueFromSuperHeader() (well, whatever you've rena

Re: [HACKERS] proposal: Set effective_cache_size to greater of .conf value, shared_buffers

2014-05-07 Thread Andres Freund
On 2014-05-07 11:45:04 -0700, Peter Geoghegan wrote: > On Wed, May 7, 2014 at 11:38 AM, Andres Freund wrote: > >> *) raising shared buffers does not 'give more memory to postgres for > >> caching' -- it can only reduce it via double paging > > > > That's absolutely not a necessary consequence. If

Re: [HACKERS] proposal: Set effective_cache_size to greater of .conf value, shared_buffers

2014-05-07 Thread Robert Haas
On Wed, May 7, 2014 at 2:49 PM, Andres Freund wrote: > On 2014-05-07 11:45:04 -0700, Peter Geoghegan wrote: >> On Wed, May 7, 2014 at 11:38 AM, Andres Freund >> wrote: >> >> *) raising shared buffers does not 'give more memory to postgres for >> >> caching' -- it can only reduce it via double pa

Re: [HACKERS] proposal: Set effective_cache_size to greater of .conf value, shared_buffers

2014-05-07 Thread Robert Haas
On Wed, May 7, 2014 at 2:40 PM, Josh Berkus wrote: > On 05/07/2014 11:13 AM, Peter Geoghegan wrote: >> We ought to be realistic about the fact that the current >> recommendations around sizing shared_buffers are nothing more than >> folk wisdom. That's the best we have right now, but that seems qu

Re: [HACKERS] Wanted: jsonb on-disk representation documentation

2014-05-07 Thread Tom Lane
Andres Freund writes: > * The jentry representation should be changed so it's possible to get the type > of a entry without checking individual types. That'll make code like > findJsonbValueFromSuperHeader() (well, whatever you've renamed it to) > much easier to read. Preferrably so it an ha

Re: [HACKERS] proposal: Set effective_cache_size to greater of .conf value, shared_buffers

2014-05-07 Thread Peter Geoghegan
On Wed, May 7, 2014 at 11:38 AM, Andres Freund wrote: >> *) raising shared buffers does not 'give more memory to postgres for >> caching' -- it can only reduce it via double paging > > That's absolutely not a necessary consequence. If pages are in s_b for a > while the OS will be perfectly happy t

Re: [HACKERS] making bgworkers without shmem access actually not have shmem access

2014-05-07 Thread Tom Lane
Robert Haas writes: > I've complained about this problem a few times before: there's nothing > to prevent a background worker which doesn't request shared memory > access from calling InitProcess() and then accessing shared memory > anyway. The attached patch is a first crack at fixing it. > Com

Re: [HACKERS] proposal: Set effective_cache_size to greater of .conf value, shared_buffers

2014-05-07 Thread Josh Berkus
On 05/07/2014 11:13 AM, Peter Geoghegan wrote: > We ought to be realistic about the fact that the current > recommendations around sizing shared_buffers are nothing more than > folk wisdom. That's the best we have right now, but that seems quite > unsatisfactory to me. So, as one of several people

Re: [HACKERS] proposal: Set effective_cache_size to greater of .conf value, shared_buffers

2014-05-07 Thread Andres Freund
On 2014-05-07 13:32:41 -0500, Merlin Moncure wrote: > On Wed, May 7, 2014 at 1:13 PM, Peter Geoghegan wrote: > > On Wed, May 7, 2014 at 11:04 AM, Josh Berkus wrote: > >> Unfortunately nobody has the time/resources to do the kind of testing > >> required for a new recommendation for shared_buffers

[HACKERS] making bgworkers without shmem access actually not have shmem access

2014-05-07 Thread Robert Haas
I've complained about this problem a few times before: there's nothing to prevent a background worker which doesn't request shared memory access from calling InitProcess() and then accessing shared memory anyway. The attached patch is a first crack at fixing it. Unfortunately, there's still a wind

Re: [HACKERS] proposal: Set effective_cache_size to greater of .conf value, shared_buffers

2014-05-07 Thread Merlin Moncure
On Wed, May 7, 2014 at 1:13 PM, Peter Geoghegan wrote: > On Wed, May 7, 2014 at 11:04 AM, Josh Berkus wrote: >> Unfortunately nobody has the time/resources to do the kind of testing >> required for a new recommendation for shared_buffers. > > I meant to suggest that the buffer manager could be im

Re: [HACKERS] proposal: Set effective_cache_size to greater of .conf value, shared_buffers

2014-05-07 Thread Jeff Janes
On Tue, May 6, 2014 at 9:55 AM, Andres Freund wrote: > On 2014-05-06 17:43:45 +0100, Simon Riggs wrote: > > > All this changes is the cost of > > IndexScans that would use more than 25% of shared_buffers worth of > > data. Hopefully not many of those in your workload. Changing the cost > > doesn

Re: [HACKERS] Wanted: jsonb on-disk representation documentation

2014-05-07 Thread Andres Freund
On 2014-05-07 10:55:28 -0700, Peter Geoghegan wrote: > On Wed, May 7, 2014 at 4:40 AM, Andres Freund wrote: > > * The jentry representation should be changed so it's possible to get the > > type > > of a entry without checking individual types. That'll make code like > > findJsonbValueFromSup

Re: [HACKERS] proposal: Set effective_cache_size to greater of .conf value, shared_buffers

2014-05-07 Thread Peter Geoghegan
On Wed, May 7, 2014 at 11:04 AM, Josh Berkus wrote: > Unfortunately nobody has the time/resources to do the kind of testing > required for a new recommendation for shared_buffers. I meant to suggest that the buffer manager could be improved to the point that the old advice becomes obsolete. Right

Re: [HACKERS] proposal: Set effective_cache_size to greater of .conf value, shared_buffers

2014-05-07 Thread Josh Berkus
On 05/06/2014 10:35 PM, Peter Geoghegan wrote: > +1. In my view, we probably should have set it to a much higher > absolute default value. The main problem with setting it to any > multiple of shared_buffers that I can see is that shared_buffers is a > very poor proxy for what effective_cache_size

Re: [HACKERS] Wanted: jsonb on-disk representation documentation

2014-05-07 Thread Peter Geoghegan
On Wed, May 7, 2014 at 4:40 AM, Andres Freund wrote: > * Imo we need space in jsonb ondisk values to indicate a format > version. We won't fully get this right. That would be nice. > * The jentry representation should be changed so it's possible to get the type > of a entry without checking

Re: [HACKERS] proposal: Set effective_cache_size to greater of .conf value, shared_buffers

2014-05-07 Thread Josh Berkus
On 05/07/2014 07:31 AM, Andrew Dunstan wrote: > +1. If we ever want to implement an auto-tuning heuristic it seems we're > going to need some hard empirical evidence to support it, and that > doesn't seem likely to appear any time soon. 4GB default it is, then. -- Josh Berkus PostgreSQL Experts

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-07 Thread Stephen Frost
* Simon Riggs (si...@2ndquadrant.com) wrote: > On 7 May 2014 17:43, Stephen Frost wrote: > > It's the optimizer's job to figure out which path to pick though, based > > on which will have the lowest cost. > > Of course. I'm not suggesting otherwise. > > >> If do you want that, you can write an E

Re: [HACKERS] 9.4 checksum errors in recovery with gin index

2014-05-07 Thread Andres Freund
Hi, On 2014-05-07 10:21:26 -0700, Jeff Janes wrote: > On Wed, May 7, 2014 at 12:48 AM, Andres Freund wrote: > > > Hi, > > > > On 2014-05-07 00:35:35 -0700, Jeff Janes wrote: > > > When recovering from a crash (with injection of a partial page write at > > > time of crash) against 7c7b1f4ae5ea3b1b

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-07 Thread Simon Riggs
On 7 May 2014 17:43, Stephen Frost wrote: > * Simon Riggs (si...@2ndquadrant.com) wrote: >> IMHO we would not want to add indexes to every column, on every table, >> nor would we wish to use lookaside for all tables. It is a good thing >> to be able to add optimizations for individual tables. GPUs

Re: [HACKERS] PGDLLEXPORTing all GUCs?

2014-05-07 Thread Robert Haas
On Wed, May 7, 2014 at 1:08 PM, Tom Lane wrote: > Robert Haas writes: >> On Wed, May 7, 2014 at 12:21 PM, Tom Lane wrote: >>> If Craig has a concrete argument why all GUCs should be accessible >>> to external modules, then let's see it (after which we'd better debate >>> exposing the few that ar

Re: [HACKERS] 9.4 checksum errors in recovery with gin index

2014-05-07 Thread Jeff Janes
On Wed, May 7, 2014 at 12:48 AM, Andres Freund wrote: > Hi, > > On 2014-05-07 00:35:35 -0700, Jeff Janes wrote: > > When recovering from a crash (with injection of a partial page write at > > time of crash) against 7c7b1f4ae5ea3b1b113682d4d I get a checksum > > verification failure. > > > > 16396

Re: [HACKERS] PGDLLEXPORTing all GUCs?

2014-05-07 Thread Andres Freund
On 2014-05-07 13:08:52 -0400, Tom Lane wrote: > Robert Haas writes: > > On Wed, May 7, 2014 at 12:21 PM, Tom Lane wrote: > >> If Craig has a concrete argument why all GUCs should be accessible > >> to external modules, then let's see it (after which we'd better debate > >> exposing the few that a

Re: [HACKERS] PGDLLEXPORTing all GUCs?

2014-05-07 Thread Tom Lane
Robert Haas writes: > On Wed, May 7, 2014 at 12:21 PM, Tom Lane wrote: >> If Craig has a concrete argument why all GUCs should be accessible >> to external modules, then let's see it (after which we'd better debate >> exposing the few that are in fact static in guc.c). > I think there's actually

Re: [HACKERS] Schizophrenic coding in gin_extract_jsonb(_hash)

2014-05-07 Thread Tom Lane
Heikki Linnakangas writes: > gin_extract_jsonb recursively extracts all the elements, keys and values > of any sub-object too, but JB_ROOT_COUNT only counts the top-level elements. Got it. So if the top level is empty, we can exit early, but otherwise we use its length * 2 as a guess at how big

Re: [HACKERS] PGDLLEXPORTing all GUCs?

2014-05-07 Thread Robert Haas
On Wed, May 7, 2014 at 12:21 PM, Tom Lane wrote: > Robert Haas writes: >> I don't accept this argument. In EnterpriseDB's Advanced Server fork >> of PostgreSQL, we've marked a bunch of extra things PGDLLEXPORT >> precisely because we have external modules that need access to them. > > Well, that

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-07 Thread Stephen Frost
* Simon Riggs (si...@2ndquadrant.com) wrote: > IMHO we would not want to add indexes to every column, on every table, > nor would we wish to use lookaside for all tables. It is a good thing > to be able to add optimizations for individual tables. GPUs are not > good for everything; it is good to be

Re: [HACKERS] PGDLLEXPORTing all GUCs?

2014-05-07 Thread Andres Freund
On 2014-05-07 12:21:55 -0400, Tom Lane wrote: > Robert Haas writes: > > I don't accept this argument. In EnterpriseDB's Advanced Server fork > > of PostgreSQL, we've marked a bunch of extra things PGDLLEXPORT > > precisely because we have external modules that need access to them. > > Well, that

Re: [HACKERS] Schizophrenic coding in gin_extract_jsonb(_hash)

2014-05-07 Thread Heikki Linnakangas
On 05/07/2014 06:27 PM, Tom Lane wrote: I think you're just proving the point that this code is woefully underdocumented. If there were, somewhere, some comment explaining what the heck JB_ROOT_COUNT actually counts, maybe I wouldn't be asking this question. jsonb.h is certainly not divulging a

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-07 Thread Stephen Frost
* Simon Riggs (si...@2ndquadrant.com) wrote: > Agreed. My proposal is that if the planner allows the lookaside to an > FDW then we pass the query for full execution on the FDW. That means > that the scan, aggregate and join could take place via the FDW. i.e. > "Custom Plan" == lookaside + FDW How

Re: [HACKERS] PGDLLEXPORTing all GUCs?

2014-05-07 Thread Tom Lane
Robert Haas writes: > I don't accept this argument. In EnterpriseDB's Advanced Server fork > of PostgreSQL, we've marked a bunch of extra things PGDLLEXPORT > precisely because we have external modules that need access to them. Well, that's an argument for marking every darn global variable as P

Re: [HACKERS] PGDLLEXPORTing all GUCs?

2014-05-07 Thread Robert Haas
On Wed, May 7, 2014 at 11:19 AM, Tom Lane wrote: > Andres Freund writes: >> On 2014-05-07 10:29:36 -0400, Tom Lane wrote: >>> To my mind, we PGDLLEXPORT some variable only after deciding that yeah, >>> we're okay with having third-party modules touching that. Craig's >>> proposal is to remove hu

Re: [HACKERS] Schizophrenic coding in gin_extract_jsonb(_hash)

2014-05-07 Thread Tom Lane
Peter Geoghegan writes: > On Tue, May 6, 2014 at 8:08 PM, Tom Lane wrote: >> The early-exit code path supposes that JB_ROOT_COUNT is absolutely >> reliable as an indicator that there's nothing in the jsonb value. >> On the other hand, the realloc logic inside the iteration loop implies >> that JB

Re: [HACKERS] PGDLLEXPORTing all GUCs?

2014-05-07 Thread Tom Lane
Andres Freund writes: > On 2014-05-07 10:29:36 -0400, Tom Lane wrote: >> To my mind, we PGDLLEXPORT some variable only after deciding that yeah, >> we're okay with having third-party modules touching that. Craig's >> proposal is to remove human judgement from that process. > It's just levelling

Re: [HACKERS] PGDLLEXPORTing all GUCs?

2014-05-07 Thread Andres Freund
On 2014-05-07 10:29:36 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2014-05-07 09:35:06 -0400, Tom Lane wrote: > >> Craig Ringer writes: > >>> Is there any reason _not_ to PGDLLEXPORT all GUCs, other than cosmetic > >>> concerns? > > >> That seems morally equivalent to "is there a reaso

Re: [HACKERS] PGDLLEXPORTing all GUCs?

2014-05-07 Thread David G Johnston
Tom Lane-2 wrote > Andres Freund < > andres@ > > writes: >> On 2014-05-07 09:35:06 -0400, Tom Lane wrote: >>> Craig Ringer < > craig@ > > writes: Is there any reason _not_ to PGDLLEXPORT all GUCs, other than cosmetic concerns? > >>> That seems morally equivalent to "is there a reason

Re: [HACKERS] proposal: Set effective_cache_size to greater of .conf value, shared_buffers

2014-05-07 Thread Simon Riggs
On 7 May 2014 15:10, Merlin Moncure wrote: > The core issues are: > 1) There is no place to enter total system memory available to the > database in postgresql.conf > 2) Memory settings (except for the above) are given as absolute > amounts, not percentages. Those sound useful starting points.

Re: [HACKERS] proposal: Set effective_cache_size to greater of .conf value, shared_buffers

2014-05-07 Thread Simon Riggs
On 7 May 2014 15:07, Tom Lane wrote: > Simon Riggs writes: >> I think I'm arguing myself towards using a BufferAccessStrategy of >> BAS_BULKREAD for large IndexScans, BitMapIndexScans and >> BitMapHeapScans. > > As soon as you've got some hard evidence to present in favor of such > changes, we ca

Re: [HACKERS] proposal: Set effective_cache_size to greater of .conf value, shared_buffers

2014-05-07 Thread Andrew Dunstan
On 05/07/2014 10:12 AM, Andres Freund wrote: On 2014-05-07 10:07:07 -0400, Tom Lane wrote: In the meantime, it seems like there is an emerging consensus that nobody much likes the existing auto-tuning behavior for effective_cache_size, and that we should revert that in favor of just increasing

Re: [HACKERS] PGDLLEXPORTing all GUCs?

2014-05-07 Thread Tom Lane
Andres Freund writes: > On 2014-05-07 09:35:06 -0400, Tom Lane wrote: >> Craig Ringer writes: >>> Is there any reason _not_ to PGDLLEXPORT all GUCs, other than cosmetic >>> concerns? >> That seems morally equivalent to "is there a reason not to make every >> static variable global?". > I think

  1   2   >