Re: [HACKERS] pgbench: Skipping the creating primary keys after initialization

2017-08-07 Thread Fabien COELHO
Attached patch I'll look into it. -- Fabien. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Adding hook in BufferSync for backup purposes

2017-08-07 Thread Tom Lane
Andrey Borodin writes: > 7 авг. 2017 г., в 18:37, Tom Lane написал(а): >> Yeah. Keep in mind that if the extension does anything at all that could >> possibly throw an error, and if that error condition persists across >> multiple tries, you will have broken the database completely: it will >> b

Re: [HACKERS] Adding hook in BufferSync for backup purposes

2017-08-07 Thread Andrey Borodin
Alvaro, Tom, thank you for your valuable comments. > Alvaro: > I remember discussing the topic of differential base-backups with > somebody (probably Marco and Gabriele). The idea we had was to have a > new relation fork which stores an LSN for each group of pages, > indicating the LSN of the ne

Re: [HACKERS] pgbench: Skipping the creating primary keys after initialization

2017-08-07 Thread Masahiko Sawada
On Thu, Aug 3, 2017 at 11:31 PM, Tom Lane wrote: > Masahiko Sawada writes: >> If we want to create other tables and load data to them as we want we >> can do that using psql -f. So an alternative ways is having a flexible >> style option for example --custom-initialize = { [load, create_pkey, >>

Re: [HACKERS] expanding inheritance in partition bound order

2017-08-07 Thread Amit Langote
On 2017/08/08 4:34, Robert Haas wrote: > On Mon, Aug 7, 2017 at 2:54 AM, Amit Langote > wrote: >> As long as find_all_inheritors() is a place only to determine the order in >> which partitions will be locked, it's fine. My concern is about the time >> of actual locking, which in the current plann

Re: [HACKERS] Pluggable storage

2017-08-07 Thread Amit Kapila
On Tue, Jun 13, 2017 at 7:20 AM, Haribabu Kommi wrote: > > > On Fri, Oct 14, 2016 at 7:26 AM, Alvaro Herrera > wrote: >> >> I have sent the partial patch I have to Hari Babu Kommi. We expect that >> he will be able to further this goal some more. > > > Thanks Alvaro for sharing your development

Re: [HACKERS] Notice message of ALTER SUBSCRIPTION ... RERESH PUBLICATIION

2017-08-07 Thread Yugo Nagata
On Mon, 7 Aug 2017 09:46:56 -0400 Peter Eisentraut wrote: > On 7/27/17 20:51, Yugo Nagata wrote: > > When we run ALTER SUBSCRIPTION ... REFRESH PUBLICATION and there is > > an unkown table at local, it says; > > > > NOTICE: added subscription for table public.tbl > > > > I feel this a bit mi

Re: [HACKERS] expanding inheritance in partition bound order

2017-08-07 Thread Amit Langote
On 2017/08/07 14:37, Amit Khandekar wrote: > On 4 August 2017 at 22:55, Robert Haas wrote: >> P.S. While I haven't reviewed 0002 in detail, I think the concept of >> minimizing what needs to be built in RelationGetPartitionDispatchInfo >> is a very good idea. > > True. I also think, RelationGetPa

Re: [HACKERS] [BUGS] Replication to Postgres 10 on Windows is broken

2017-08-07 Thread Noah Misch
On Sun, Aug 06, 2017 at 08:50:37AM -0700, Noah Misch wrote: > On Sun, Aug 06, 2017 at 11:17:57AM -0400, Tom Lane wrote: > > Noah Misch writes: > > > I've added this as an open item. Confirmed in this setup: > > > > > -- Client > > > Windows Server 2016 > > > postgresql-10.0-beta2-windows-x64-bin

Re: [HACKERS] [TRAP: FailedAssertion] causing server to crash

2017-08-07 Thread Noah Misch
On Mon, Aug 07, 2017 at 05:29:34PM +1200, Thomas Munro wrote: > On Thu, Aug 3, 2017 at 3:03 AM, Robert Haas wrote: > > On Fri, Jul 21, 2017 at 1:31 AM, Thomas Munro > > wrote: > >> Thanks Neha. It's be best to post the back trace and if possible > >> print oldestXact and ShmemVariableCache->olde

Re: [HACKERS] ICU collation variant keywords and pg_collation entries (Was: [BUGS] Crash report for some ICU-52 (debian8) COLLATE and work_mem values)

2017-08-07 Thread Peter Geoghegan
On Mon, Aug 7, 2017 at 3:23 PM, Tom Lane wrote: > The thing that I'm particularly thinking about is that if someone wants > an ICU variant collation that we didn't make initdb provide, they'll do > a CREATE COLLATION and go use it. At update time, pg_dump or pg_upgrade > will export/import that v

Re: [HACKERS] Subscription code improvements

2017-08-07 Thread Masahiko Sawada
On Mon, Aug 7, 2017 at 10:22 PM, Peter Eisentraut wrote: > On 8/7/17 00:27, Masahiko Sawada wrote: However, even with this patch there is still an issue that NOTICE messages "removed subscription for table public.t1" can be appeared even if we rollback ALTER SUBSCRIPTION REFRESH com

Re: [HACKERS] More race conditions in logical replication

2017-08-07 Thread Robert Haas
On Mon, Aug 7, 2017 at 8:14 PM, Alvaro Herrera wrote: > BTW, I noticed that the PG_WAIT_LOCK value that we're using for wait > event here (and in the replication slot case) is bogus. We probably > need something new here. Yeah, if you're adding a new wait point, you should add document a new con

Re: [HACKERS] More race conditions in logical replication

2017-08-07 Thread Alvaro Herrera
Alvaro Herrera wrote: > Alvaro Herrera wrote: > > I just noticed that Jacana failed again in the subscription test, and it > > looks like it's related to this. I'll take a look tomorrow if no one > > beats me to it. > > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=jacana&dt=2017-07-26%

Re: [HACKERS] possible encoding issues with libxml2 functions

2017-08-07 Thread Noah Misch
On Wed, Apr 05, 2017 at 10:53:39PM +0200, Pavel Stehule wrote: > 2017-03-17 4:23 GMT+01:00 Noah Misch : > > On Sun, Mar 12, 2017 at 10:26:33PM +0100, Pavel Stehule wrote: > > > 2017-03-12 21:57 GMT+01:00 Noah Misch : > > > > On Sun, Mar 12, 2017 at 08:36:58PM +0100, Pavel Stehule wrote: > > > > > 2

Re: [HACKERS] ICU collation variant keywords and pg_collation entries (Was: [BUGS] Crash report for some ICU-52 (debian8) COLLATE and work_mem values)

2017-08-07 Thread Tom Lane
Peter Eisentraut writes: > On 8/6/17 20:07, Peter Geoghegan wrote: >> I've looked into this. I'll give an example of what keyword variants >> there are for Greek, and then discuss what I think each is. > I'm not sure why we want to get into editorializing this. We query ICU > for the names of di

Re: [HACKERS] ICU collation variant keywords and pg_collation entries (Was: [BUGS] Crash report for some ICU-52 (debian8) COLLATE and work_mem values)

2017-08-07 Thread Peter Geoghegan
On Mon, Aug 7, 2017 at 2:50 PM, Peter Eisentraut wrote: > On 8/6/17 20:07, Peter Geoghegan wrote: >> I've looked into this. I'll give an example of what keyword variants >> there are for Greek, and then discuss what I think each is. > > I'm not sure why we want to get into editorializing this. We

Re: [HACKERS] max_files_per_processes vs others uses of file descriptors

2017-08-07 Thread Tom Lane
Andres Freund writes: > On 2017-08-07 17:30:13 -0400, Tom Lane wrote: >> Meh. The lack of field complaints about this doesn't indicate to me that >> we have a huge problem, and in any case, just increasing NUM_RESERVED_FDS >> would do nothing for the system-wide limits. > Howso? Via count_usable

Re: [HACKERS] ICU collation variant keywords and pg_collation entries (Was: [BUGS] Crash report for some ICU-52 (debian8) COLLATE and work_mem values)

2017-08-07 Thread Peter Eisentraut
On 8/6/17 20:07, Peter Geoghegan wrote: > I've looked into this. I'll give an example of what keyword variants > there are for Greek, and then discuss what I think each is. I'm not sure why we want to get into editorializing this. We query ICU for the names of distinct collations and use that. I

Re: [HACKERS] max_files_per_processes vs others uses of file descriptors

2017-08-07 Thread Andres Freund
Hi, On 2017-08-07 17:30:13 -0400, Tom Lane wrote: > Meh. The lack of field complaints about this doesn't indicate to me that > we have a huge problem, and in any case, just increasing NUM_RESERVED_FDS > would do nothing for the system-wide limits. Howso? Via count_usable_fds() we test for max_fi

Re: [HACKERS] max_files_per_processes vs others uses of file descriptors

2017-08-07 Thread Tom Lane
Andres Freund writes: > On 2017-08-07 17:05:06 -0400, Tom Lane wrote: >> Probably the best we can hope for there is to have fd.c provide a function >> "close an FD please", which postgres_fdw could call if libpq fails because >> of ENFILE/EMFILE, and then retry. > Unless that takes up a slot in f

Re: [HACKERS] max_files_per_processes vs others uses of file descriptors

2017-08-07 Thread Peter Geoghegan
On Mon, Aug 7, 2017 at 1:40 PM, Andres Freund wrote: > Given how close max_files_per_process is to the default linux limit of > 1024 fds, I wonder if we shouldn't increase NUM_RESERVED_FDS by quite a > bit? Personally, any time I've seen a problem with this it was because an extension leaked FDs,

Re: [HACKERS] max_files_per_processes vs others uses of file descriptors

2017-08-07 Thread Andres Freund
On 2017-08-07 17:05:06 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2017-08-07 16:52:42 -0400, Tom Lane wrote: > >> No, I don't think so. If you're depending on the NUM_RESERVED_FDS > >> headroom for anything meaningful, *you're doing it wrong*. You should be > >> getting an FD via fd.c

Re: [HACKERS] max_files_per_processes vs others uses of file descriptors

2017-08-07 Thread Tom Lane
Andres Freund writes: > On 2017-08-07 16:52:42 -0400, Tom Lane wrote: >> No, I don't think so. If you're depending on the NUM_RESERVED_FDS >> headroom for anything meaningful, *you're doing it wrong*. You should be >> getting an FD via fd.c, so that there is an opportunity to free up an FD >> (b

Re: [HACKERS] max_files_per_processes vs others uses of file descriptors

2017-08-07 Thread Andres Freund
On 2017-08-07 16:52:42 -0400, Tom Lane wrote: > Andres Freund writes: > > These days there's a number of other consumers of > > fds. E.g. postgres_fdw, epoll, ... All these aren't accounted for by > > fd.c. > > > Given how close max_files_per_process is to the default linux limit of > > 1024 fds

Re: [HACKERS] max_files_per_processes vs others uses of file descriptors

2017-08-07 Thread Tom Lane
Andres Freund writes: > These days there's a number of other consumers of > fds. E.g. postgres_fdw, epoll, ... All these aren't accounted for by > fd.c. > Given how close max_files_per_process is to the default linux limit of > 1024 fds, I wonder if we shouldn't increase NUM_RESERVED_FDS by quit

[HACKERS] max_files_per_processes vs others uses of file descriptors

2017-08-07 Thread Andres Freund
Hi, currently the default max_files_per_process is 1000. fd.c uses close to that many (- NUM_RESERVED_FDS/10). count_usable_fds() makes sure that at server start there's at most that many fds available, but that doesn't mean that much for runtime. These days there's a number of other consumers of

[HACKERS] Re: [COMMITTERS] pgsql: Record full paths of programs sought by "configure".

2017-08-07 Thread Andrew Dunstan
On 08/07/2017 04:20 PM, Tom Lane wrote: > Andrew Dunstan writes: >> On 08/07/2017 04:07 PM, Tom Lane wrote: >>> Sorry, I was imprecise. What I'm suggesting is that you drop the >>> runtime PATH-foolery and instead put this in configure's environment: >>> >>> PROVE=$perlpathdir/prove >>> >>> Oth

Re: [HACKERS] [COMMITTERS] pgsql: Record full paths of programs sought by "configure".

2017-08-07 Thread Tom Lane
Andrew Dunstan writes: > On 08/07/2017 04:07 PM, Tom Lane wrote: >> Sorry, I was imprecise. What I'm suggesting is that you drop the >> runtime PATH-foolery and instead put this in configure's environment: >> >> PROVE=$perlpathdir/prove >> >> Otherwise you're basically lying to configure about

[HACKERS] Re: [COMMITTERS] pgsql: Record full paths of programs sought by "configure".

2017-08-07 Thread Andrew Dunstan
On 08/07/2017 04:07 PM, Tom Lane wrote: > Andrew Dunstan writes: >> On 08/07/2017 03:36 PM, Tom Lane wrote: >>> My goodness, that's ugly. Is it really better than injecting >>> "PROVE=prove"? (I'd suggest saying that to configure, not make, >>> so that the configure log bears some resemblance

Re: [HACKERS] [COMMITTERS] pgsql: Record full paths of programs sought by "configure".

2017-08-07 Thread Tom Lane
Andrew Dunstan writes: > On 08/07/2017 03:36 PM, Tom Lane wrote: >> My goodness, that's ugly. Is it really better than injecting >> "PROVE=prove"? (I'd suggest saying that to configure, not make, >> so that the configure log bears some resemblance to what you >> want done.) > This is what we ha

Re: [HACKERS] Effect of dropping a partitioned table's column over time

2017-08-07 Thread Robert Haas
On Sun, Aug 6, 2017 at 11:38 PM, Craig Ringer wrote: > Can we instead create the new partitions with the same dropped columns? > > Ensure that every partition, parent and child, has the same column-set? We could, but that has costs of its own. It means that those calls are stored in the tuple as

[HACKERS] Re: [COMMITTERS] pgsql: Record full paths of programs sought by "configure".

2017-08-07 Thread Andrew Dunstan
On 08/07/2017 03:36 PM, Tom Lane wrote: > Andrew Dunstan writes: >> On 08/07/2017 03:21 PM, Tom Lane wrote: >>> I'm confused. AFAIK, that commit did not change which "prove" would >>> be used --- at least not unless you change PATH between configure and >>> make. It only changed how specifical

Re: [HACKERS] [COMMITTERS] pgsql: Record full paths of programs sought by "configure".

2017-08-07 Thread Tom Lane
Andrew Dunstan writes: > On 08/07/2017 03:21 PM, Tom Lane wrote: >> I'm confused. AFAIK, that commit did not change which "prove" would >> be used --- at least not unless you change PATH between configure and >> make. It only changed how specifically that program would be named in >> Makefile.gl

Re: [HACKERS] expanding inheritance in partition bound order

2017-08-07 Thread Robert Haas
On Mon, Aug 7, 2017 at 2:54 AM, Amit Langote wrote: > I think Amit Khandekar mentioned this on the UPDATE partition key thread [1]. Yes, similar discussion. > As long as find_all_inheritors() is a place only to determine the order in > which partitions will be locked, it's fine. My concern is a

[HACKERS] Re: [COMMITTERS] pgsql: Record full paths of programs sought by "configure".

2017-08-07 Thread Andrew Dunstan
On 08/07/2017 03:21 PM, Tom Lane wrote: > Andrew Dunstan writes: >> On 07/31/2017 01:02 PM, Tom Lane wrote: >>> Record full paths of programs sought by "configure". >> The problem with this commit, as jacana is demonstrating, is that on >> Msys it finds the wrong prove. configure needs to run ag

Re: [HACKERS] [COMMITTERS] pgsql: Record full paths of programs sought by "configure".

2017-08-07 Thread Tom Lane
Andrew Dunstan writes: > On 07/31/2017 01:02 PM, Tom Lane wrote: >> Record full paths of programs sought by "configure". > The problem with this commit, as jacana is demonstrating, is that on > Msys it finds the wrong prove. configure needs to run against the perl > we build against, i.e. a nativ

Re: [HACKERS] expanding inheritance in partition bound order

2017-08-07 Thread Robert Haas
On Mon, Aug 7, 2017 at 1:48 AM, Ashutosh Bapat wrote: > with the way schema is designed. May be it's better to use your idea > of using get_rel_relkind() or find a way to check that the flag is in > sync with the relkind, like when building the relcache. That's got the same problem as building a

Re: [HACKERS] xmltable SQL conformance

2017-08-07 Thread Pavel Stehule
2017-08-07 20:36 GMT+02:00 Peter Eisentraut < peter.eisentr...@2ndquadrant.com>: > On 8/3/17 17:07, Pavel Stehule wrote: > > I'm looking to update the SQL conformance list for the release. > I'm > > wondering whether the new xmltable feature fully completes the > > followin

Re: [HACKERS] xmltable SQL conformance

2017-08-07 Thread Peter Eisentraut
On 8/3/17 17:07, Pavel Stehule wrote: > I'm looking to update the SQL conformance list for the release. I'm > wondering whether the new xmltable feature fully completes the > following > items: > > X300XMLTable > X301XMLTable: derived column

[HACKERS] Re: [COMMITTERS] pgsql: Record full paths of programs sought by "configure".

2017-08-07 Thread Andrew Dunstan
On 07/31/2017 01:02 PM, Tom Lane wrote: > Record full paths of programs sought by "configure". > > Previously we had a mix of uses of AC_CHECK_PROG[S] and AC_PATH_PROG[S]. > The only difference between those macros is that the latter emits the > full path to the program it finds, eg "/usr/bin/pro

[HACKERS] Re: [GSOC][weekly report 9] Eliminate O(N^2) scaling from rw-conflict tracking in serializable transactions

2017-08-07 Thread Alvaro Herrera
Mengxing Liu wrote: > In the last week, I focus on: > > > 1) Continuing on optimization of skip list. Let me state once again that I'm certainly not an expert in the predicate locks module and that I hope Kevin will chime in with more useful feedback than mine. Some random thoughts: * I wonder

[HACKERS] [GSOC][weekly report 9] Eliminate O(N^2) scaling from rw-conflict tracking in serializable transactions

2017-08-07 Thread Mengxing Liu
In the last week, I focus on: 1) Continuing on optimization of skip list. Here is the new logic: At first, I use an unordered linked list to do all inserting/searching operations. When the number of conflicts is more than the threshold, transform the linked list into an ordered skip list. Then

Re: [HACKERS] free space % calculation in pgstathashindex

2017-08-07 Thread Ashutosh Sharma
On Mon, Aug 7, 2017 at 7:19 PM, Amit Kapila wrote: > On Mon, Aug 7, 2017 at 6:07 PM, Ashutosh Sharma wrote: >> Hi, >> > .. >> In step #1, assuming '*' as an arithmetic operator, the left operand >> i.e. 'stats.unused_pages' is of type uint32 whereas the right operand >> i.e. 'stats.space_per_page

Re: [HACKERS] Re: Crash report for some ICU-52 (debian8) COLLATE and work_mem values

2017-08-07 Thread Peter Eisentraut
On 8/5/17 18:56, Noah Misch wrote: >> [Action required within three days. This is a generic notification.] I'm awaiting further testing and discussion. Probably nothing happening for beta3. Will report on Thursday. -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Develo

Re: [HACKERS] Patch: Add --no-comments to skip COMMENTs with pg_dump

2017-08-07 Thread Fabrízio de Royes Mello
On Mon, Aug 7, 2017 at 10:43 AM, Robins Tharakan wrote: > > On 20 July 2017 at 05:14, Robins Tharakan wrote: >> >> On 20 July 2017 at 05:08, Michael Paquier wrote: >>> >>> On Wed, Jul 19, 2017 at 8:59 PM, >>> Fabrízio de Royes Mello >>> > You should add the properly sgml docs for this pg_dumpall

Re: [HACKERS] free space % calculation in pgstathashindex

2017-08-07 Thread Amit Kapila
On Mon, Aug 7, 2017 at 6:07 PM, Ashutosh Sharma wrote: > Hi, > .. > In step #1, assuming '*' as an arithmetic operator, the left operand > i.e. 'stats.unused_pages' is of type uint32 whereas the right operand > i.e. 'stats.space_per_page' is of type int32 and arithmetic > conversions of the ANSI C

Re: [HACKERS] Notice message of ALTER SUBSCRIPTION ... RERESH PUBLICATIION

2017-08-07 Thread Peter Eisentraut
On 7/27/17 20:51, Yugo Nagata wrote: > When we run ALTER SUBSCRIPTION ... REFRESH PUBLICATION and there is > an unkown table at local, it says; > > NOTICE: added subscription for table public.tbl > > I feel this a bit misleading because it says a subscription is added > but actually new subscr

Re: [HACKERS] Adding hook in BufferSync for backup purposes

2017-08-07 Thread Tom Lane
Alvaro Herrera writes: > I suppose your hook idea lets you implement the LSN fork in an > extension, rather than having it be part of core. The idea of hooking > onto BufferSync makes me uneasy, though -- like it's not the correct > place to do it. Yeah. Keep in mind that if the extension does

Re: [HACKERS] Subscription code improvements

2017-08-07 Thread Peter Eisentraut
On 8/7/17 00:27, Masahiko Sawada wrote: >>> However, even with this patch there is still an issue that NOTICE >>> messages "removed subscription for table public.t1" can be appeared >>> even if we rollback ALTER SUBSCRIPTION REFRESH command as I mentioned >>> on earlier thread. Since I think this b

Re: [HACKERS] Subscription code improvements

2017-08-07 Thread Peter Eisentraut
On 8/7/17 00:32, Masahiko Sawada wrote: > On Sun, Aug 6, 2017 at 7:44 AM, Noah Misch wrote: >> On Wed, Aug 02, 2017 at 04:09:43PM -0400, Peter Eisentraut wrote: >>> On 8/1/17 00:17, Noah Misch wrote: The above-described topic is currently a PostgreSQL 10 open item. Peter, since you comm

Re: [HACKERS] Pluggable storage

2017-08-07 Thread Amit Kapila
On Tue, Aug 1, 2017 at 1:56 PM, Haribabu Kommi wrote: > > > On Sun, Jul 23, 2017 at 4:10 PM, Amit Kapila > wrote: >> > >> >> > 1. Design an API that returns values/nulls array and convert that into a >> > HeapTuple whenever it is required in the upper layers. All the existing >> > heap form/defor

Re: [HACKERS] Partition-wise join for join between (declaratively) partitioned tables

2017-08-07 Thread Ashutosh Bapat
On Thu, Aug 3, 2017 at 9:38 PM, Ashutosh Bapat wrote: > On Thu, Aug 3, 2017 at 2:10 AM, Robert Haas wrote: >> On Mon, Jul 31, 2017 at 7:59 AM, Ashutosh Bapat >> wrote: >>> Adding AppendRelInfos to root->append_rel_list as and when they are >>> created would keep parent AppendRelInfos before thos

[HACKERS] free space % calculation in pgstathashindex

2017-08-07 Thread Ashutosh Sharma
Hi, While working on - [1] (one of the bug reported for hash index), i noticed that the percentage of free space shown in 'free_percent' column of pgstathashindex() is incorrect for some of the boundary cases. This is how the free space percentage is calculated by pgstathashindex(), 1) /* Count

Re: [HACKERS] Page Scan Mode in Hash Index

2017-08-07 Thread Ashutosh Sharma
On Fri, Aug 4, 2017 at 7:14 PM, Amit Kapila wrote: > On Sun, Jul 30, 2017 at 2:07 PM, Ashutosh Sharma > wrote: >> Hi, >> >> On Wed, May 10, 2017 at 2:28 PM, Ashutosh Sharma >> wrote: >>> While doing the code coverage testing of v7 patch shared with - [1], I >>> found that there are few lines o

Re: [HACKERS] pgsql 10: hash indexes testing

2017-08-07 Thread AP
On Sat, Aug 05, 2017 at 04:41:24PM +0530, Amit Kapila wrote: > > (On another note, I committed these patches.) > > Thanks. Seconded. :) Now uploading data with fillfactor of 90. I'll know in 2-3 days if the new patches are successful (earlier if they did not help). I compiled (as apt.postgresql

Re: [HACKERS] Adding hook in BufferSync for backup purposes

2017-08-07 Thread Alvaro Herrera
Андрей Бородин wrote: > ==What== > I propose to add hook inside BufferSync() function it inform extensions that > we > are going to write pages to disk. Please see patch attached. I pass a > timestamp > of the checkpoint, but it would be good if we could also pass there number of > checkpoint or