Re: [HACKERS] Use pread and pwrite instead of lseek + write and read

2016-09-14 Thread Oskari Saarenmaa
17.08.2016, 22:11, Tom Lane kirjoitti: Robert Haas writes: I don't understand why you think this would create non-trivial portability issues. The patch as submitted breaks entirely on platforms without pread/pwrite. Yes, we can add a configure test and some shim functions to fix that, but the

Re: [HACKERS] Push down more full joins in postgres_fdw

2016-09-14 Thread Ashutosh Bapat
On Wed, Sep 14, 2016 at 8:52 PM, Robert Haas wrote: > On Tue, Sep 13, 2016 at 11:38 PM, Ashutosh Bapat > wrote: >> On Tue, Sep 13, 2016 at 10:28 PM, Robert Haas wrote: >>> On Tue, Sep 6, 2016 at 9:07 AM, Ashutosh Bapat >>> wrote: That's not true with the alias information. As long as we de

Re: [HACKERS] Printing bitmap objects in the debugger

2016-09-14 Thread Ashutosh Bapat
On Wed, Sep 14, 2016 at 8:45 PM, Alvaro Herrera wrote: > Tom Lane wrote: >> Ashutosh Bapat writes: >> > While working on partition-wise join, I had to examine Relids objects >> > many times. Printing the Bitmapset::words[] in binary format and then >> > inferring the relids takes time and is erro

Re: [HACKERS] Hash Indexes

2016-09-14 Thread Amit Kapila
One other point, I would like to discuss is that currently, we have a concept for tracking active hash scans (hashscan.c) which I think is mainly to protect splits when the backend which is trying to split has some scan open. You can read "Other Notes" section of access/hash/README for further deta

Re: [HACKERS] Printing bitmap objects in the debugger

2016-09-14 Thread Ashutosh Bapat
>> Alvaro Herrera writes: >>> I don't understand. Why don't you just use "call pprint(the bitmapset)" >>> in the debugger? >> >> Bitmapsets aren't Nodes, so pprint doesn't work directly on them. >> I usually find that I can pprint some node containing the value(s) >> I'm interested in, but maybe

Re: [HACKERS] Hash Indexes

2016-09-14 Thread Amit Kapila
On Thu, Sep 15, 2016 at 12:43 AM, Jesper Pedersen wrote: > Hi, > > On 09/14/2016 07:24 AM, Amit Kapila wrote: > >>> UPDATE also sees an improvement. >>> >> >> Can you explain this more? Is it more compare to HEAD or more as >> compare to Btree? Isn't this contradictory to what the test in below

Re: [HACKERS] Printing bitmap objects in the debugger

2016-09-14 Thread Michael Paquier
On Thu, Sep 15, 2016 at 2:58 PM, Ashutosh Bapat wrote: > On Wed, Sep 14, 2016 at 5:31 PM, Pavan Deolasee > wrote: >> The complete API reference is available here >> http://lldb.llvm.org/python_reference/index.html >> >> Looks like an interesting SoC project to write useful lldb/gdb scripts to >>

Re: [HACKERS] Printing bitmap objects in the debugger

2016-09-14 Thread Ashutosh Bapat
On Wed, Sep 14, 2016 at 5:31 PM, Pavan Deolasee wrote: > > On Wed, Sep 14, 2016 at 3:46 PM, Pavan Deolasee > wrote: >> >> >> >> lately I'm using LVM debugger (which probably does not have something >> equivalent), > > > And I was so clueless about lldb's powerful scripting interface. For > examp

Re: [HACKERS] Hash Indexes

2016-09-14 Thread Amit Kapila
On Thu, Sep 15, 2016 at 4:44 AM, Jeff Janes wrote: > On Tue, May 10, 2016 at 5:09 AM, Amit Kapila > wrote: >> >> >> >> Although, I don't think it is a very good idea to take any performance >> data with WIP patch, still I couldn't resist myself from doing so and below >> are the performance numbe

Re: [HACKERS] Hash Indexes

2016-09-14 Thread Amit Kapila
On Thu, Sep 15, 2016 at 4:04 AM, Jeff Janes wrote: > On Tue, Sep 13, 2016 at 9:31 AM, Jeff Janes wrote: >> >> === >> >> +Vacuum acquires cleanup lock on bucket to remove the dead tuples and or >> tuples >> +that are moved due to split. The need for cleanup lock to remove dead >> tuples >> +i

[HACKERS] Re: [HACKERS] Error running custom plugin: “output plugins have to declare the _PG_output_plugin_init symbol”

2016-09-14 Thread Ashutosh Bapat
On Wed, Sep 14, 2016 at 4:03 PM, valeriof wrote: > Hi, I'm kind of new to Postgres and I'm trying to create a custom output > plugin for logical replication (Postgres is 9.5.4 version and I'm building > the project from a Windows 8/64 bit machine - same machine where the db is > installed). > I st

Re: [HACKERS] make async slave to wait for lsn to be replayed

2016-09-14 Thread Thomas Munro
On Thu, Sep 1, 2016 at 2:16 AM, Ivan Kartyshov wrote: > Hi hackers, > > Few days earlier I've finished my work on WAITLSN statement utility, so I’d > like to share it. > [...] > Your feedback is welcome! > > [waitlsn_10dev.patch] Hi Ivan, Thanks for working on this. Here are some general though

Re: [HACKERS] Logical Replication WIP

2016-09-14 Thread Craig Ringer
On 14 September 2016 at 04:56, Petr Jelinek wrote: > Not sure what you mean by negotiation. Why would that be needed? You know > server version when you connect and when you know that you also know what > capabilities that version of Postgres has. If you send unrecognized option > you get corresp

Re: [HACKERS] [PATCH v2] Add overflow checks to money type input function

2016-09-14 Thread Peter Eisentraut
On 9/9/16 3:19 AM, Fabien COELHO wrote: > >> I have updated the patch with additional tests and comments per your >> review. Final(?) version attached. > > Applied on head, make check ok. No more comments on the code which does > what it says. I'm fine with this patch. Pushed, thanks. -- Pet

Re: [HACKERS] pg_basebackup, pg_receivexlog and data durability (was: silent data loss with ext4 / all current versions)

2016-09-14 Thread Michael Paquier
On Thu, Sep 15, 2016 at 9:44 AM, Peter Eisentraut wrote: > On 9/12/16 11:16 PM, Michael Paquier wrote: >>> I don't think tar file output in pg_basebackup needs to be fsynced. >>> > It's just a backup file like what pg_dump produces, and we don't fsync >>> > that either. The point of this change i

Re: [HACKERS] Printing bitmap objects in the debugger

2016-09-14 Thread Amit Langote
On 2016/09/15 0:04, Tom Lane wrote: > Alvaro Herrera writes: >> I don't understand. Why don't you just use "call pprint(the bitmapset)" >> in the debugger? > > Bitmapsets aren't Nodes, so pprint doesn't work directly on them. > I usually find that I can pprint some node containing the value(s) >

Re: [HACKERS] pg_basebackup, pg_receivexlog and data durability (was: silent data loss with ext4 / all current versions)

2016-09-14 Thread Peter Eisentraut
On 9/12/16 11:16 PM, Michael Paquier wrote: >> I don't think tar file output in pg_basebackup needs to be fsynced. >> > It's just a backup file like what pg_dump produces, and we don't fsync >> > that either. The point of this change is to leave a data directory in >> > a synced state equivalent t

Re: [HACKERS] pg_basebackup wish list

2016-09-14 Thread Michael Paquier
On Thu, Sep 15, 2016 at 9:26 AM, Peter Eisentraut wrote: > On 9/13/16 7:24 PM, Michael Paquier wrote: >> PostgresNode.pm had better use the new --noclean option in its calls, >> the new default is not useful for debugging. > > We don't do it for initdb either. Is that a problem? Right. In case o

Re: [HACKERS] pg_basebackup wish list

2016-09-14 Thread Peter Eisentraut
On 9/13/16 7:24 PM, Michael Paquier wrote: > PostgresNode.pm had better use the new --noclean option in its calls, > the new default is not useful for debugging. We don't do it for initdb either. Is that a problem? -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Developm

Re: [HACKERS] OpenSSL 1.1 breaks configure and more

2016-09-14 Thread Andreas Karlsson
On 09/15/2016 02:03 AM, Andreas Karlsson wrote: On 09/12/2016 06:51 PM, Heikki Linnakangas wrote: Changes since last version: * Added more error checks to the my_BIO_s_socket() function. Check for NULL result from malloc(). Check the return code of BIO_meth_set_*() functions; looking at OpenSSL

Re: [HACKERS] palloc() too large on pg_buffercache with large shared_buffers

2016-09-14 Thread Kouhei Kaigai
> On Wed, Sep 14, 2016 at 12:13 AM, Kouhei Kaigai wrote: > > It looks to me pg_buffercache tries to allocate more than 1GB using > > palloc(), when shared_buffers is more than 256GB. > > > > # show shared_buffers ; > > shared_buffers > > > > 280GB > > (1 row) > > > > # SELECT bu

Re: [HACKERS] OpenSSL 1.1 breaks configure and more

2016-09-14 Thread Andreas Karlsson
On 09/12/2016 06:51 PM, Heikki Linnakangas wrote: Changes since last version: * Added more error checks to the my_BIO_s_socket() function. Check for NULL result from malloc(). Check the return code of BIO_meth_set_*() functions; looking at OpenSSL sources, they always succeed, but all the test/e

Re: [HACKERS] Tracking timezone abbreviation removals in the IANA tz database

2016-09-14 Thread Tom Lane
Robert Haas writes: > I suggest that if you (or someone else who understands it, if there is > any such person) were willing to either improve the existing > documentation or maybe even write a whole new chapter on how we do > time zone handling, it would be most welcome. Well, I'll put that on m

Re: [HACKERS] Implement targetlist SRFs using ROWS FROM() (was Changed SRF in targetlist handling)

2016-09-14 Thread Tom Lane
Andres Freund writes: > On 2016-09-12 19:35:22 -0400, Tom Lane wrote: >> Anyway I'll draft a prototype and then we can compare. > Ok, cool. Here's a draft patch that is just meant to investigate what the planner changes might look like if we do it in the pipelined-result way. Accordingly, I didn

Re: [HACKERS] Tracking timezone abbreviation removals in the IANA tz database

2016-09-14 Thread Robert Haas
On Fri, Sep 2, 2016 at 8:55 AM, Tom Lane wrote: > I had thought that this wouldn't really affect us because PG's > interpretations of zone abbreviations are driven by the info in > timezone/tznames/ rather than the IANA files in timezone/data/. > I forgot however that the "dynamic abbreviation" co

Re: [HACKERS] Logical Replication WIP

2016-09-14 Thread Petr Jelinek
On 14/09/16 21:53, Andres Freund wrote: Hi, On 2016-09-14 21:17:42 +0200, Petr Jelinek wrote: +/* + * Gather Relations based o provided by RangeVar list. + * The gathered tables are locked in access share lock mode. + */ Why access share? Shouldn't we make this ShareUpdateExclusive or similar

Re: [HACKERS] Logical Replication WIP

2016-09-14 Thread Petr Jelinek
On 14/09/16 18:21, Andres Freund wrote: (continuing, uh, a bit happier) On 2016-09-09 00:59:26 +0200, Petr Jelinek wrote: +/* + * Relcache invalidation callback for our relation map cache. + */ +static void +logicalreprelmap_invalidate_cb(Datum arg, Oid reloid) +{ + LogicalRepRelMapEntry

Re: [HACKERS] Hash Indexes

2016-09-14 Thread Jeff Janes
On Tue, May 10, 2016 at 5:09 AM, Amit Kapila wrote: > > > Although, I don't think it is a very good idea to take any performance > data with WIP patch, still I couldn't resist myself from doing so and below > are the performance numbers. To get the performance data, I have dropped > the primary

Re: [HACKERS] kqueue

2016-09-14 Thread Thomas Munro
On Thu, Sep 15, 2016 at 10:48 AM, Keith Fiske wrote: > Thomas Munro brought up in #postgresql on freenode needing someone to test a > patch on a larger FreeBSD server. I've got a pretty decent machine (3.1Ghz > Quad Core Xeon E3-1220V3, 16GB ECC RAM, ZFS mirror on WD Red HDD) so offered > to give

Re: [HACKERS] kqueue

2016-09-14 Thread Keith Fiske
On Wed, Sep 14, 2016 at 9:09 AM, Matteo Beccati wrote: > Hi, > > On 14/09/2016 00:06, Tom Lane wrote: > >> I'm inclined to think the kqueue patch is worth applying just on the >> grounds that it makes things better on OS X and doesn't seem to hurt >> on FreeBSD. Whether anyone would ever get to

Re: [HACKERS] PATCH: Avoid use of __attribute__ when building with old Sun compiler versions

2016-09-14 Thread Robert Haas
On Fri, Sep 2, 2016 at 11:55 AM, Andy Grundman wrote: > The use of __attribute__ is currently enabled by checking for > __SUNPRO_C. However, this compiler only added support for > __attribute__ keywords in version 5.10 [1]. This patch adds a version > check to only enable this for supported versio

Re: [HACKERS] Hash Indexes

2016-09-14 Thread Jeff Janes
On Tue, Sep 13, 2016 at 9:31 AM, Jeff Janes wrote: > === > > +Vacuum acquires cleanup lock on bucket to remove the dead tuples and or > tuples > +that are moved due to split. The need for cleanup lock to remove dead > tuples > +is to ensure that scans' returns correct results. Scan that

Re: [HACKERS] [BUGS] BUG #14244: wrong suffix for pg_size_pretty()

2016-09-14 Thread Gavin Flower
On 15/09/16 03:45, Robert Haas wrote: On Wed, Sep 14, 2016 at 5:22 AM, Thomas Berger wrote: Today, i found the time to read all the mails in this thread, and i think i have to explain, why we decided to open a bug for this behavior. Pn Tuesday, 23. August 2016, 13:30:29 Robert Haas wrote: J.

Re: [HACKERS] Choosing parallel_degree

2016-09-14 Thread Robert Haas
On Wed, Sep 14, 2016 at 3:54 PM, Simon Riggs wrote: >> I do think this comment is confusing: >> >> + *This value is not locked by the transaction, so this value may >> + *be changed while a SELECT that has used these values for planning >> + *is still executing. >> >> I don

Re: [HACKERS] What is the posix_memalign() equivalent for the PostgreSQL?

2016-09-14 Thread Robert Haas
On Fri, Sep 2, 2016 at 1:17 PM, Andres Freund wrote: > On 2016-09-02 13:05:37 -0400, Tom Lane wrote: >> Anderson Carniel writes: >> > If not, according to your experience, is there a >> > significance difference between the performance of the O_DIRECT or not? >> >> AFAIK, nobody's really bothered

Re: [HACKERS] pageinspect: Hash index support

2016-09-14 Thread Jeff Janes
On Tue, Aug 30, 2016 at 10:06 AM, Alvaro Herrera wrote: > Jesper Pedersen wrote: > > Hi, > > > > Attached is a patch that adds support for hash indexes in pageinspect. > > > > The functionality (hash_metap, hash_page_stats and hash_page_items) > follows > > the B-tree functions. > > I suggest tha

Re: [HACKERS] New SQL counter statistics view (pg_stat_sql)

2016-09-14 Thread Robert Haas
On Fri, Sep 2, 2016 at 2:33 AM, Haribabu Kommi wrote: > On Wed, Aug 31, 2016 at 3:19 AM, Alvaro Herrera > wrote: >> Haribabu Kommi wrote: >> >>> Apart from the above, here are the following list of command tags that >>> are generated in the code, I took only the first word of the command tag >>>

Re: [HACKERS] Logical Replication WIP

2016-09-14 Thread Andres Freund
Hi, On 2016-09-14 21:17:42 +0200, Petr Jelinek wrote: > > > +/* > > > + * Gather Relations based o provided by RangeVar list. > > > + * The gathered tables are locked in access share lock mode. > > > + */ > > > > Why access share? Shouldn't we make this ShareUpdateExclusive or > > similar, to pre

Re: [HACKERS] Choosing parallel_degree

2016-09-14 Thread Simon Riggs
On 14 September 2016 at 14:48, Robert Haas wrote: > On Thu, Sep 1, 2016 at 9:39 AM, Simon Riggs wrote: > Can I change this to a lower setting? I would have done this before > applying > the patch, but you beat me to it. I don't have a problem with reducing the lock level

Re: [HACKERS] Choosing parallel_degree

2016-09-14 Thread Robert Haas
On Thu, Sep 1, 2016 at 9:39 AM, Simon Riggs wrote: >>> > Can I change this to a lower setting? I would have done this before >>> > applying >>> > the patch, but you beat me to it. >>> >>> I don't have a problem with reducing the lock level there, if we're >>> convinced that it's safe. >> >> >> I'l

Re: [HACKERS] Comment on GatherPath.single_copy

2016-09-14 Thread Robert Haas
On Thu, Sep 1, 2016 at 3:15 AM, Kyotaro HORIGUCHI wrote: > At Wed, 31 Aug 2016 07:26:22 -0400, Tom Lane wrote in > <5934.1472642...@sss.pgh.pa.us> >> Robert Haas writes: >> > On Tue, Aug 30, 2016 at 6:38 PM, Tom Lane wrote: >> >> Robert, could you fix the documentation for that field so it's >

Re: [HACKERS] Logical Replication WIP

2016-09-14 Thread Petr Jelinek
On 14/09/16 20:50, Andres Freund wrote: On 2016-09-14 13:20:02 -0500, Peter Eisentraut wrote: On 9/14/16 11:21 AM, Andres Freund wrote: + ExecInsert(NULL, /* mtstate is only used for onconflict handling which we don't support atm */ + remoteslot, +

Re: [HACKERS] Logical Replication WIP

2016-09-14 Thread Petr Jelinek
On 14/09/16 00:48, Andres Freund wrote: First read through the current version. Hence no real architectural comments. Hi, Thanks for looking! On 2016-09-09 00:59:26 +0200, Petr Jelinek wrote: diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c ne

Re: [HACKERS] Hash Indexes

2016-09-14 Thread Jesper Pedersen
Hi, On 09/14/2016 07:24 AM, Amit Kapila wrote: On Wed, Sep 14, 2016 at 12:29 AM, Jesper Pedersen wrote: On 09/13/2016 07:26 AM, Amit Kapila wrote: Attached, new version of patch which contains the fix for problem reported on write-ahead-log of hash index thread [1]. I have been testing pa

Re: [HACKERS] Logical Replication WIP

2016-09-14 Thread Andres Freund
On 2016-09-14 13:20:02 -0500, Peter Eisentraut wrote: > On 9/14/16 11:21 AM, Andres Freund wrote: > >> + ExecInsert(NULL, /* mtstate is only used for onconflict handling which > >> we don't support atm */ > >> > + remoteslot, > >> > + remoteslot

Re: [HACKERS] WAL consistency check facility

2016-09-14 Thread Robert Haas
On Tue, Sep 13, 2016 at 9:04 PM, Michael Paquier wrote: > It seems to me that you need to think about the way to document things > properly first, with for example: > - Have a first documentation patch that explains what is a resource > manager for WAL, and what are the types available with a nice

Re: [HACKERS] Logical Replication WIP

2016-09-14 Thread Peter Eisentraut
On 9/14/16 11:21 AM, Andres Freund wrote: >> +ExecInsert(NULL, /* mtstate is only used for onconflict handling which >> we don't support atm */ >> > + remoteslot, >> > + remoteslot, >> > + NIL, >> > + ONCONFLICT_NO

Re: [HACKERS] WAL consistency check facility

2016-09-14 Thread Kuntal Ghosh
On Thu, Sep 8, 2016 at 1:20 PM, Michael Paquier wrote: > >> 2. For BRIN/UPDATE+INIT, block numbers (in rm_tid[0]) are different in >> REVMAP page. This happens only for two cases. I'm not sure what the >> reason can be. > > Hm? This smells like a block reference bug. What are the cases you are > r

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2016-09-14 Thread Robert Haas
On Wed, Sep 14, 2016 at 1:23 PM, Alvaro Herrera wrote: > Robert Haas wrote: >> Actually, I think that probably *is* worthwhile, specifically because >> it might let us avoid multiple index scans in cases where we currently >> require them. Right now, our default maintenance_work_mem value is >> 6

Re: [HACKERS] GiST: interpretation of NaN from penalty function

2016-09-14 Thread Tom Lane
Andrew Borodin writes: > Currently GiST treats NaN penalty as zero penalty, in terms of > generalized tree this means "perfect fit". I think that this situation > should be considered "worst fit" instead. On what basis? It seems hard to me to make any principled argument here. Certainly, "NaN me

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2016-09-14 Thread Tom Lane
Pavan Deolasee writes: > On Wed, Sep 14, 2016 at 10:53 PM, Alvaro Herrera > wrote: >> One thing not quite clear to me is how do we create the bitmap >> representation starting from the array representation in midflight >> without using twice as much memory transiently. Are we going to write >> t

Re: [HACKERS] Tuplesort merge pre-reading

2016-09-14 Thread Heikki Linnakangas
Addressed all your comments one way or another, new patch attached. Comments on some specific points below: On 09/12/2016 01:13 AM, Peter Geoghegan wrote: Other things I noticed: * You should probably point out that typically, access to batch memory will still be sequential, despite your block

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2016-09-14 Thread Pavan Deolasee
On Wed, Sep 14, 2016 at 10:53 PM, Alvaro Herrera wrote: > > > One thing not quite clear to me is how do we create the bitmap > representation starting from the array representation in midflight > without using twice as much memory transiently. Are we going to write > the array to a temp file, fr

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2016-09-14 Thread Simon Riggs
On 14 September 2016 at 11:19, Pavan Deolasee wrote: >> In >> theory we could even start with the list of TIDs and switch to the >> bitmap if the TID list becomes larger than the bitmap would have been, >> but I don't know if it's worth the effort. >> > > Yes, that works too. Or may be even bett

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2016-09-14 Thread Alvaro Herrera
Robert Haas wrote: > Actually, I think that probably *is* worthwhile, specifically because > it might let us avoid multiple index scans in cases where we currently > require them. Right now, our default maintenance_work_mem value is > 64MB, which is enough to hold a little over ten million tuples

[HACKERS] GiST: interpretation of NaN from penalty function

2016-09-14 Thread Andrew Borodin
Hi hackers! Currently GiST treats NaN penalty as zero penalty, in terms of generalized tree this means "perfect fit". I think that this situation should be considered "worst fit" instead. Here is a patch to highlight place in the code. I could not construct test to generate bad tree, which would b

Re: [HACKERS] GiST penalty functions [PoC]

2016-09-14 Thread Andrew Borodin
Here is v6 of cube penalty patch. There was a warning about unused edge function under systems without __STDC_IEC_559__ defined, this patch fixes it. Regards, Andrey Borodin. diff --git a/contrib/cube/cube.c b/contrib/cube/cube.c index 3feddef..ad868ac 100644 --- a/contrib/cube/cube.c +++ b/contr

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2016-09-14 Thread Claudio Freire
On Wed, Sep 14, 2016 at 12:17 PM, Robert Haas wrote: > For instance, one idea to grow memory usage incrementally would be to > store dead tuple information separately for each 1GB segment of the > relation. So we have an array of dead-tuple-representation objects, > one for every 1GB of the relat

Re: [HACKERS] Logical Replication WIP

2016-09-14 Thread Andres Freund
(continuing, uh, a bit happier) On 2016-09-09 00:59:26 +0200, Petr Jelinek wrote: > +/* > + * Relcache invalidation callback for our relation map cache. > + */ > +static void > +logicalreprelmap_invalidate_cb(Datum arg, Oid reloid) > +{ > + LogicalRepRelMapEntry *entry; > + > + /* Just t

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2016-09-14 Thread Pavan Deolasee
On Wed, Sep 14, 2016 at 8:47 PM, Robert Haas wrote: > > > I am kind of doubtful about this whole line of investigation because > we're basically trying pretty hard to fix something that I'm not sure > is broken.I do agree that, all other things being equal, the TID > lookups will probably be

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

2016-09-14 Thread Dilip Kumar
On Wed, Sep 14, 2016 at 8:59 PM, Robert Haas wrote: > Sure, but you're testing at *really* high client counts here. Almost > nobody is going to benefit from a 5% improvement at 256 clients. I agree with your point, but here we need to consider one more thing, that on head we are gaining ~30% wit

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2016-09-14 Thread Arthur Silva
On Sep 14, 2016 5:18 PM, "Robert Haas" wrote: > > On Wed, Sep 14, 2016 at 8:16 AM, Pavan Deolasee > wrote: > > Ah, thanks. So MaxHeapTuplesPerPage sets the upper boundary for the per page > > bitmap size. Thats about 36 bytes for 8K page. IOW if on an average there > > are 6 or more dead tuples p

Re: [HACKERS] [BUGS] BUG #14244: wrong suffix for pg_size_pretty()

2016-09-14 Thread Tom Lane
Robert Haas writes: > Interesting. I think that our documentation should only describe the > way we use unit suffixes in one central place, but other places (like > pg_size_pretty) could link to that central place. > I don't believe that there is any general unanimity among users or > developers

Re: [HACKERS] [BUGS] BUG #14244: wrong suffix for pg_size_pretty()

2016-09-14 Thread Robert Haas
On Wed, Sep 14, 2016 at 5:22 AM, Thomas Berger wrote: > Today, i found the time to read all the mails in this thread, and i think i > have to explain, why we decided to open a bug for this behavior. > > Pn Tuesday, 23. August 2016, 13:30:29 Robert Haas wrote: >> J. Random User: I'm having a probl

Re: [HACKERS] palloc() too large on pg_buffercache with large shared_buffers

2016-09-14 Thread Robert Haas
On Wed, Sep 14, 2016 at 12:13 AM, Kouhei Kaigai wrote: > It looks to me pg_buffercache tries to allocate more than 1GB using > palloc(), when shared_buffers is more than 256GB. > > # show shared_buffers ; > shared_buffers > > 280GB > (1 row) > > # SELECT buffers, d.datname, coal

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

2016-09-14 Thread Robert Haas
On Wed, Sep 14, 2016 at 12:55 AM, Dilip Kumar wrote: > 2. Results > ./pgbench -c $threads -j $threads -T 10 -M prepared postgres -f script.sql > scale factor: 300 > Clients head(tps)grouplock(tps) granular(tps) > --- - --

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2016-09-14 Thread Claudio Freire
On Wed, Sep 14, 2016 at 12:17 PM, Robert Haas wrote: > > I am kind of doubtful about this whole line of investigation because > we're basically trying pretty hard to fix something that I'm not sure > is broken.I do agree that, all other things being equal, the TID > lookups will probably be fa

Re: [HACKERS] Push down more full joins in postgres_fdw

2016-09-14 Thread Robert Haas
On Tue, Sep 13, 2016 at 11:38 PM, Ashutosh Bapat wrote: > On Tue, Sep 13, 2016 at 10:28 PM, Robert Haas wrote: >> On Tue, Sep 6, 2016 at 9:07 AM, Ashutosh Bapat >> wrote: >>> That's not true with the alias information. As long as we detect which >>> relations need subqueries, their RTIs are enou

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2016-09-14 Thread Robert Haas
On Wed, Sep 14, 2016 at 8:16 AM, Pavan Deolasee wrote: > Ah, thanks. So MaxHeapTuplesPerPage sets the upper boundary for the per page > bitmap size. Thats about 36 bytes for 8K page. IOW if on an average there > are 6 or more dead tuples per page, bitmap will outperform the current > representatio

Re: [HACKERS] Printing bitmap objects in the debugger

2016-09-14 Thread Alvaro Herrera
Tom Lane wrote: > Ashutosh Bapat writes: > > While working on partition-wise join, I had to examine Relids objects > > many times. Printing the Bitmapset::words[] in binary format and then > > inferring the relids takes time and is error prone. > > FWIW, I generally rely on pprint() to look at pl

Re: [HACKERS] Printing bitmap objects in the debugger

2016-09-14 Thread Tom Lane
Ashutosh Bapat writes: > While working on partition-wise join, I had to examine Relids objects > many times. Printing the Bitmapset::words[] in binary format and then > inferring the relids takes time and is error prone. FWIW, I generally rely on pprint() to look at planner data structures from t

Re: [HACKERS] Printing bitmap objects in the debugger

2016-09-14 Thread Tom Lane
Alvaro Herrera writes: > I don't understand. Why don't you just use "call pprint(the bitmapset)" > in the debugger? Bitmapsets aren't Nodes, so pprint doesn't work directly on them. I usually find that I can pprint some node containing the value(s) I'm interested in, but maybe that isn't working

Re: [HACKERS] WIP: Covering + unique indexes.

2016-09-14 Thread Anastasia Lubennikova
One more update. I added ORDER BY clause to regression tests. It was done as a separate bugfix patch by Tom Lane some time ago, but it definitely should be included into the patch. -- Anastasia Lubennikova Postgres Professional: http://www.postgrespro.com The Russian Postgres Company diff --git

Re: [HACKERS] Printing bitmap objects in the debugger

2016-09-14 Thread Alvaro Herrera
Ashutosh Bapat wrote: > Hi All, > While working on partition-wise join, I had to examine Relids objects > many times. Printing the Bitmapset::words[] in binary format and then > inferring the relids takes time and is error prone. Instead I wrote a > function bms_to_char() which allocates a StringIn

Re: [HACKERS] Surprising behaviour of \set AUTOCOMMIT ON

2016-09-14 Thread Tom Lane
Rahila Syed writes: >> Looking at the other variables hooks, they already emit errors and can >> deny the effect of a change corresponding to a new value, without >> informing the caller. Why would autocommit be different? > These instances where psql_error occurs inside hooks the command is > su

Re: [HACKERS] condition variables

2016-09-14 Thread Robert Haas
On Tue, Sep 13, 2016 at 10:55 PM, Peter Geoghegan wrote: > On Thu, Aug 11, 2016 at 2:47 PM, Robert Haas wrote: >> Another approach to the problem is to use a latch wait loop. That >> almost works. Interrupts can be serviced, and you can recheck shared >> memory to see whether the condition for

Re: [HACKERS] sequences and pg_upgrade

2016-09-14 Thread Anastasia Lubennikova
The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, passed Spec compliant: tested, passed Documentation:tested, failed Thank you for the patch. As I see there are no objections in

Re: [HACKERS] Surprising behaviour of \set AUTOCOMMIT ON

2016-09-14 Thread Rahila Syed
Hello, >Looking at the other variables hooks, they already emit errors and can >deny the effect of a change corresponding to a new value, without >informing the caller. Why would autocommit be different? The only type of psql_error inside hooks is as follows, psql_error("unrecognized value \"%s\

Re: [HACKERS] kqueue

2016-09-14 Thread Matteo Beccati
Hi, On 14/09/2016 00:06, Tom Lane wrote: I'm inclined to think the kqueue patch is worth applying just on the grounds that it makes things better on OS X and doesn't seem to hurt on FreeBSD. Whether anyone would ever get to the point of seeing intra-kernel contention on these platforms is hard

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2016-09-14 Thread Pavan Deolasee
On Wed, Sep 14, 2016 at 5:32 PM, Robert Haas wrote: > On Wed, Sep 14, 2016 at 5:45 AM, Pavan Deolasee > wrote: > > Another interesting bit about these small tables is that the largest used > > offset for these tables never went beyond 291 which is the value of > > MaxHeapTuplesPerPage. I don't k

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2016-09-14 Thread Robert Haas
On Wed, Sep 14, 2016 at 5:45 AM, Pavan Deolasee wrote: > Another interesting bit about these small tables is that the largest used > offset for these tables never went beyond 291 which is the value of > MaxHeapTuplesPerPage. I don't know if there is something that prevents > inserting more than M

Re: [HACKERS] Printing bitmap objects in the debugger

2016-09-14 Thread Pavan Deolasee
On Wed, Sep 14, 2016 at 3:46 PM, Pavan Deolasee wrote: > > > lately I'm using LVM debugger (which probably does not have something > equivalent), > And I was so clueless about lldb's powerful scripting interface. For example, you can write something like this in bms_utils.py: import lldb def

Re: [HACKERS] Write Ahead Logging for Hash Indexes

2016-09-14 Thread Amit Kapila
On Wed, Sep 14, 2016 at 4:36 PM, Ashutosh Sharma wrote: > Hi All, > > Below is the backtrace for the issue reported in my earlier mail [1]. > From the callstack it looks like we are trying to release lock on a > meta page twice in _hash_expandtable(). > Thanks for the call stack. I think below

Re: [HACKERS] Hash Indexes

2016-09-14 Thread Amit Kapila
On Wed, Sep 14, 2016 at 12:29 AM, Jesper Pedersen wrote: > On 09/13/2016 07:26 AM, Amit Kapila wrote: >> >> Attached, new version of patch which contains the fix for problem >> reported on write-ahead-log of hash index thread [1]. >> > > I have been testing patch in various scenarios, and it has a

Re: [HACKERS] Write Ahead Logging for Hash Indexes

2016-09-14 Thread Ashutosh Sharma
Hi All, Below is the backtrace for the issue reported in my earlier mail [1]. >From the callstack it looks like we are trying to release lock on a meta page twice in _hash_expandtable(). (gdb) bt #0 0x007b01cf in LWLockRelease (lock=0x7f55f59d0570) at lwlock.c:1799 #1 0x0078990

Re: [HACKERS] pgbench - allow to store select results into variables

2016-09-14 Thread Amit Langote
Hi Fabien, On 2016/09/13 17:41, Fabien COELHO wrote: > > Hello Amit, > >> [...] >> There still seems to be a change in behavior of the -r option due to the >> patch. Consider the following example: >> >> select a from a where a = 1 \; >> select a+1 from a where a = 1; >> ... >> - statement laten

[HACKERS] Error running custom plugin: “output plugins have to declare the _PG_output_plugin_init symbol”

2016-09-14 Thread valeriof
Hi, I'm kind of new to Postgres and I'm trying to create a custom output plugin for logical replication (Postgres is 9.5.4 version and I'm building the project from a Windows 8/64 bit machine - same machine where the db is installed). I started from the code of the sample test_decoding, and I was t

Re: [HACKERS] Printing bitmap objects in the debugger

2016-09-14 Thread Pavan Deolasee
On Wed, Sep 14, 2016 at 3:43 PM, Ashutosh Bapat < ashutosh.ba...@enterprisedb.com> wrote: > Hi All, > While working on partition-wise join, I had to examine Relids objects > many times. Printing the Bitmapset::words[] in binary format and then > inferring the relids takes time and is error prone.

Re: [HACKERS] [PATCH] SortSupport for macaddr type

2016-09-14 Thread Julien Rouhaud
On 26/08/2016 19:44, Brandur wrote: > Hello, > Hello, > I've attached a patch to add SortSupport for Postgres' macaddr which has the > effect of improving the performance of sorting operations for the type. The > strategy that I employ is very similar to that for UUID, which is to create > abbre

[HACKERS] Printing bitmap objects in the debugger

2016-09-14 Thread Ashutosh Bapat
Hi All, While working on partition-wise join, I had to examine Relids objects many times. Printing the Bitmapset::words[] in binary format and then inferring the relids takes time and is error prone. Instead I wrote a function bms_to_char() which allocates a StringInfo, calls outBitmapset() to deco

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2016-09-14 Thread Pavan Deolasee
On Wed, Sep 14, 2016 at 8:47 AM, Pavan Deolasee wrote: > >> > Sawada-san offered to reimplement the patch based on what I proposed > upthread. In the new scheme of things, we will allocate a fixed size bitmap > of length 2D bits per page where D is average page density of live + dead > tuples. (T

Re: [HACKERS] [BUGS] BUG #14244: wrong suffix for pg_size_pretty()

2016-09-14 Thread Thomas Berger
Today, i found the time to read all the mails in this thread, and i think i have to explain, why we decided to open a bug for this behavior. Pn Tuesday, 23. August 2016, 13:30:29 Robert Haas wrote: > J. Random User: I'm having a problem! > Mailing List: Gee, how big are your tables? > J. Random U

Re: [HACKERS] Write Ahead Logging for Hash Indexes

2016-09-14 Thread Ashutosh Sharma
Hi All, I am getting following error when running the test script shared by Jeff -[1] . The error is observed upon executing the test script for around 3-4 hrs. 57869 INSERT XX000 2016-09-14 07:58:01.211 IST:ERROR: lock buffer_content 1 is not held 57869 INSERT XX000 2016-09-14 07:58:01.211 IST:

Re: [HACKERS] WAL consistency check facility

2016-09-14 Thread Kuntal Ghosh
On Wed, Sep 14, 2016 at 11:31 AM, Michael Paquier wrote: > On Wed, Sep 14, 2016 at 2:56 PM, Kuntal Ghosh > wrote: >> Master >> --- >> - If wal_consistency check is enabled or needs_backup is set in >> XLogRecordAssemble(), we do a fpw. >> - If a fpw is to be done, then fork_flags is s

Re: [HACKERS] Declarative partitioning - another take

2016-09-14 Thread Rajkumar Raghuwanshi
I have Continued with testing declarative partitioning with the latest patch. Got some more observation, given below -- Observation 1 : Getting overlap error with START with EXCLUSIVE in range partition. create table test_range_bound ( a int) partition by range(a); --creating a partition to conta

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

2016-09-14 Thread Dilip Kumar
On Wed, Sep 14, 2016 at 10:25 AM, Dilip Kumar wrote: > I have tested performance with approach 1 and approach 2. > > 1. Transaction (script.sql): I have used below transaction to run my > bench mark, We can argue that this may not be an ideal workload, but I > tested this to put more load on ClogC