Re: [HACKERS] Mention column name in error messages

2016-10-05 Thread Franck Verrot
Thanks Andres for the review. Michael, please find attached a revised patch addressing, amongst some other changes, the testing issue (`make check` passes now) and the visibility of the ` TransformExprState` struct. Thanks, Franck On Tue, Dec 22, 2015 at 1:49 PM, Michael Paquier

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2016-10-05 Thread Pavan Deolasee
On Wed, Oct 5, 2016 at 1:43 PM, Tomas Vondra wrote: > > > I've been looking at the patch over the past few days, running a bunch of > benchmarks etc. Thanks for doing that. > I can confirm the significant speedup, often by more than 75% (depending > on number of

[HACKERS] memory leak in e94568ecc10 (pre-reading in external sort)

2016-10-05 Thread Tomas Vondra
Hi, it seems e94568ecc10 has a pretty bad memory leak. A simple pgbench -i -s 300 allocates ~32GB of memory before it fails vacuum... set primary keys... ERROR: out of memory DETAIL: Failed on request of size 134184960. The relevant bit from the memory context stats:

Re: [HACKERS] Transactions involving multiple postgres foreign servers

2016-10-05 Thread Ashutosh Bapat
>> >> No, the COMMIT returns after the first phase. It can not wait for all >> the foreign servers to complete their second phase > > Hm, it sounds like it's same as normal commit (not 2PC). > What's the difference? > > My understanding is that basically the local server can not return > COMMIT to

Re: [HACKERS] Our "fallback" atomics implementation doesn't actually work

2016-10-05 Thread Tom Lane
Andres Freund writes: > Hm. After a long battle of head vs. wall I think I see what the problem > is. For the fallback atomics implementation I somehow had assumed that > pg_atomic_write_u32() doesn't need to lock, as it's just an unlocked > write. But that's not true,

Re: [HACKERS] Fast AT ADD COLUMN with DEFAULTs

2016-10-05 Thread Serge Rielau
> On Oct 5, 2016, at 5:52 PM, Vitaly Burovoy wrote: > > On 10/5/16, Serge Rielau wrote: >> I want to point out as a minor "extension" that there is no need for the >> default to be immutable. It is merely required that the default is evaluate >> at

Re: [HACKERS] Our "fallback" atomics implementation doesn't actually work

2016-10-05 Thread Andres Freund
On 2016-10-05 15:02:09 -0400, Tom Lane wrote: > Andres Freund writes: > > Without yet having analyzed this deeply, could it actually be that the > > reason is that sem_post/wait aren't proper memory barriers? On a glance > > the symptoms look like values have been modified

Re: [HACKERS] Hash tables in dynamic shared memory

2016-10-05 Thread Andres Freund
On 2016-10-05 08:02:42 +0200, Magnus Hagander wrote: > On Oct 5, 2016 1:23 AM, "Thomas Munro" > wrote: > > Another thought: it could be used to make things like > > pg_stat_statements not have to be in shared_preload_libraries. > That would indeed be a great

Re: [HACKERS] Hash tables in dynamic shared memory

2016-10-05 Thread Thomas Munro
On Wed, Oct 5, 2016 at 7:02 PM, Magnus Hagander wrote: > On Oct 5, 2016 1:23 AM, "Thomas Munro" > wrote: >> >> On Wed, Oct 5, 2016 at 12:11 PM, Thomas Munro >> wrote: >> > On Wed, Oct 5, 2016 at 11:22 AM, Andres

Re: [HACKERS] Fast AT ADD COLUMN with DEFAULTs

2016-10-05 Thread Vitaly Burovoy
On 10/5/16, Serge Rielau wrote: >On Wed, Oct 5, 2016 at 4:19 PM, Vitaly Burovoy >wrote: >> But what I discover for myself is that we have pg_attrdef separately >> from the pg_attribute. Why? >> Is it time to join them? For not presented defaults it

Re: [HACKERS] Is the last 9.1 release planned?

2016-10-05 Thread Tsunakawa, Takayuki
From: pgsql-hackers-ow...@postgresql.org > [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Magnus Hagander > On Oct 5, 2016 5:42 AM, "Tsunakawa, Takayuki" > wrote: > > Thanks for clarification. Then, I understood that the expression "stop > releases in

Re: [HACKERS] [RFC] Should we fix postmaster to avoid slow shutdown?

2016-10-05 Thread Tsunakawa, Takayuki
From: Robert Haas [mailto:robertmh...@gmail.com] > I have no opinion on this patch, because I haven't reviewed it, but note > recent commit 3b90e38c5d592ea8ec8236287dd5c749fc041728, which appears to > be semi-related. Thank you for interesting information. Maybe Tom-san experienced some trouble

Re: [HACKERS] Switch to unnamed POSIX semaphores as our preferred sema code?

2016-10-05 Thread Tsunakawa, Takayuki
From: pgsql-hackers-ow...@postgresql.org > [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Tom Lane > I've gotten a bit tired of seeing "could not create semaphores: No space > left on device" failures in the buildfarm, so I looked into whether we should > consider preferring unnamed

Re: [HACKERS] Fast AT ADD COLUMN with DEFAULTs

2016-10-05 Thread Serge Rielau
via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx=9.0.74=10.11.6=email_footer_2] On Wed, Oct 5, 2016 at 4:19 PM, Vitaly Burovoy wrote: On 10/5/16, Tom Lane wrote: > I wrote: >> Need a better name for the concept, since evidently this name

Re: [HACKERS] pg_rewind hangs if --source-server is used and syncrep is enabled

2016-10-05 Thread Michael Paquier
On Wed, Oct 5, 2016 at 11:53 PM, Michael Banck wrote: > On Wed, Oct 05, 2016 at 04:39:39PM +0200, Michael Banck wrote: >> To the user, the last thing printed is "need to copy MB [...]". If >> the user cancels the pg_rewind command with ^C, the backend keeps >>

Re: [HACKERS] Fast AT ADD COLUMN with DEFAULTs

2016-10-05 Thread Vitaly Burovoy
On 10/5/16, Tom Lane wrote: > I wrote: >> Need a better name for the concept, since evidently this name isn't >> conveying the idea. > > Maybe "creation default" would work better? Point being it's the > default value at the time of column creation. Hmm... Personaly for me

Re: [HACKERS] Fast AT ADD COLUMN with DEFAULTs

2016-10-05 Thread Pantelis Theodosiou
On Thu, Oct 6, 2016 at 12:05 AM, Serge Rielau wrote: > > via Newton Mail > > > On Wed, Oct 5, 2016 at 3:58 PM, Tom Lane wrote: > > Andres Freund writes: > > On

Re: [HACKERS] Fast AT ADD COLUMN with DEFAULTs

2016-10-05 Thread Serge Rielau
via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx=9.0.74=10.11.6=email_footer_2] On Wed, Oct 5, 2016 at 3:58 PM, Tom Lane wrote: Andres Freund writes: > On 2016-10-05 15:44:56 -0700, Jeff Janes wrote: >>> No, "a second “exist default"" was

Re: [HACKERS] Fast AT ADD COLUMN with DEFAULTs

2016-10-05 Thread Tom Lane
I wrote: > Need a better name for the concept, since evidently this name isn't > conveying the idea. Maybe "creation default" would work better? Point being it's the default value at the time of column creation. regards, tom lane -- Sent via pgsql-hackers mailing list

Re: [HACKERS] Fast AT ADD COLUMN with DEFAULTs

2016-10-05 Thread Andres Freund
On 2016-10-05 18:58:47 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2016-10-05 15:44:56 -0700, Jeff Janes wrote: > >>> No, "a second “exist default"" was mentioned, i.e. it is an additional > >>> column in a system table (pg_attribute) as default column values of >

Re: [HACKERS] Fast AT ADD COLUMN with DEFAULTs

2016-10-05 Thread Vitaly Burovoy
On 10/5/16, Andres Freund wrote: > On 2016-10-05 15:44:56 -0700, Jeff Janes wrote: >> On Wed, Oct 5, 2016 at 3:29 PM, Andres Freund wrote: >> > ALTER TABLE foo ALTER COLUMN withdefault DROP DEFAULT; >> > INSERT id = 1; >> > ALTER TABLE foo ALTER COLUMN

Re: [HACKERS] Fast AT ADD COLUMN with DEFAULTs

2016-10-05 Thread Pantelis Theodosiou
On Wed, Oct 5, 2016 at 11:44 PM, Jeff Janes wrote: > On Wed, Oct 5, 2016 at 3:29 PM, Andres Freund wrote: > >> On 2016-10-05 15:23:05 -0700, Vitaly Burovoy wrote: >> > On 10/5/16, Andres Freund wrote: >> > > On 2016-10-05 11:58:33

Re: [HACKERS] Fast AT ADD COLUMN with DEFAULTs

2016-10-05 Thread Tom Lane
Andres Freund writes: > On 2016-10-05 15:44:56 -0700, Jeff Janes wrote: >>> No, "a second “exist default"" was mentioned, i.e. it is an additional >>> column in a system table (pg_attribute) as default column values of >>> the "pre-alter" era. It solves changing of the

Re: [HACKERS] Fast AT ADD COLUMN with DEFAULTs

2016-10-05 Thread Andres Freund
On 2016-10-05 15:44:56 -0700, Jeff Janes wrote: > On Wed, Oct 5, 2016 at 3:29 PM, Andres Freund wrote: > > > On 2016-10-05 15:23:05 -0700, Vitaly Burovoy wrote: > > > On 10/5/16, Andres Freund wrote: > > > > On 2016-10-05 11:58:33 -0700, Serge Rielau

Re: [HACKERS] Fast AT ADD COLUMN with DEFAULTs

2016-10-05 Thread Jeff Janes
On Wed, Oct 5, 2016 at 3:29 PM, Andres Freund wrote: > On 2016-10-05 15:23:05 -0700, Vitaly Burovoy wrote: > > On 10/5/16, Andres Freund wrote: > > > On 2016-10-05 11:58:33 -0700, Serge Rielau wrote: > > >> Dear Hackers, > > >> I’m working on a patch that

Re: [HACKERS] Fast AT ADD COLUMN with DEFAULTs

2016-10-05 Thread Vitaly Burovoy
On 10/5/16, Vitaly Burovoy wrote: > On 10/5/16, Andres Freund wrote: >> On 2016-10-05 15:23:05 -0700, Vitaly Burovoy wrote: >>> On 10/5/16, Andres Freund wrote: >>> > On 2016-10-05 11:58:33 -0700, Serge Rielau wrote: >>> >> Dear

Re: [HACKERS] Fast AT ADD COLUMN with DEFAULTs

2016-10-05 Thread Vitaly Burovoy
On 10/5/16, Andres Freund wrote: > On 2016-10-05 15:23:05 -0700, Vitaly Burovoy wrote: >> On 10/5/16, Andres Freund wrote: >> > On 2016-10-05 11:58:33 -0700, Serge Rielau wrote: >> >> Dear Hackers, >> >> I’m working on a patch that expands PG’s ability to

Re: [HACKERS] Fast AT ADD COLUMN with DEFAULTs

2016-10-05 Thread Serge Rielau
On Wed, Oct 5, 2016 at 3:23 PM, Vitaly Burovoy wrote: On 10/5/16, Andres Freund wrote: > On 2016-10-05 11:58:33 -0700, Serge Rielau wrote: >> Dear Hackers, >> I’m working on a patch that expands PG’s ability to add columns to a table >> without a

Re: [HACKERS] Fast AT ADD COLUMN with DEFAULTs

2016-10-05 Thread Andres Freund
On 2016-10-05 15:23:05 -0700, Vitaly Burovoy wrote: > On 10/5/16, Andres Freund wrote: > > On 2016-10-05 11:58:33 -0700, Serge Rielau wrote: > >> Dear Hackers, > >> I’m working on a patch that expands PG’s ability to add columns to a table > >> without a table rewrite (i.e. at

Re: [HACKERS] Fast AT ADD COLUMN with DEFAULTs

2016-10-05 Thread Vitaly Burovoy
On 10/5/16, Andres Freund wrote: > On 2016-10-05 11:58:33 -0700, Serge Rielau wrote: >> Dear Hackers, >> I’m working on a patch that expands PG’s ability to add columns to a table >> without a table rewrite (i.e. at O(1) cost) from the >> nullable-without-default to a more

Re: [HACKERS] Fast AT ADD COLUMN with DEFAULTs

2016-10-05 Thread Andres Freund
On 2016-10-05 11:58:33 -0700, Serge Rielau wrote: > Dear Hackers, > I’m working on a patch that expands PG’s ability to add columns to a table > without a table rewrite (i.e. at O(1) cost) from the nullable-without-default > to a more general case. E.g. CREATE TABLE T(pk INT NOT NULL PRIMARY

Re: [HACKERS] Fast AT ADD COLUMN with DEFAULTs

2016-10-05 Thread Serge Rielau
On Wed, Oct 5, 2016 at 2:45 PM, Vitaly Burovoy wrote: On 10/5/16, Serge Rielau wrote: > Dear Hackers, > > I’m working on a patch that expands PG’s ability to add columns to a table > without a table rewrite (i.e. at O(1) cost) from the >

[HACKERS] Switch to unnamed POSIX semaphores as our preferred sema code?

2016-10-05 Thread Tom Lane
I've gotten a bit tired of seeing "could not create semaphores: No space left on device" failures in the buildfarm, so I looked into whether we should consider preferring unnamed POSIX semaphores over SysV semaphores. We've had code for named and unnamed POSIX semaphores in our tree for a long

Re: [HACKERS] Fast AT ADD COLUMN with DEFAULTs

2016-10-05 Thread Vitaly Burovoy
On 10/5/16, Serge Rielau wrote: > Dear Hackers, > > I’m working on a patch that expands PG’s ability to add columns to a table > without a table rewrite (i.e. at O(1) cost) from the > nullable-without-default to a more general case. E.g. > > CREATE TABLE T(pk INT NOT NULL

Re: [HACKERS] Stopping logical replication protocol

2016-10-05 Thread Vladimir Gordiychuk
> Vladimir? I'm running out of time to spend on this at least until the next CF. Think you can make these changes? Yes, I can. But I thinks It should be discuss first. > Terminating COPY BOTH with ERRCODE_QUERY_CANCELED seems fine. If it > was expecting the error the client can Sync and do

Re: [HACKERS] Kernel Tainted

2016-10-05 Thread reiner peterke
> On Oct 5, 2016, at 9:43 PM, Tomas Vondra wrote: > > On 10/05/2016 08:41 PM, reiner peterke wrote: >> Hi, >> >> We are helping a client test an application On Power8 using Postgres >> 9.5.4 which has been compiled specifically for the Power. >> >> This is

Re: [HACKERS] Hash tables in dynamic shared memory

2016-10-05 Thread Thomas Munro
On Thu, Oct 6, 2016 at 12:02 AM, Dilip Kumar wrote: > While reviewing , I found that in dht_iterate_begin function, we are > not initializing > iterator->last_item_pointer to InvalidDsaPointer; Fixed, thanks. -- Thomas Munro http://www.enterprisedb.com dht-v2.patch

Re: [HACKERS] Tracking wait event for latches

2016-10-05 Thread Michael Paquier
On Wed, Oct 5, 2016 at 9:25 PM, Robert Haas wrote: > On Tue, Oct 4, 2016 at 5:07 PM, Michael Paquier > wrote: >> More seriously, the Windows animals have been complaining about >> pg_sleep() crashing the system: >> SELECT pg_sleep(0.1); >> !

[HACKERS] Small doc fix

2016-10-05 Thread Vitaly Burovoy
Hello, hackers, I've just noticed an extra word in a sentence in the docs in the "parallel.sgml". It seems the sentence was constructed one way and changed later with the extra word left. Please, find the fix attached. -- Best regards, Vitaly Burovoy pg-docs-fix.patch Description: Binary

Re: [HACKERS] Kernel Tainted

2016-10-05 Thread Tomas Vondra
On 10/05/2016 08:41 PM, reiner peterke wrote: Hi, We are helping a client test an application On Power8 using Postgres 9.5.4 which has been compiled specifically for the Power. This is running on sles12sp1 the current kernel is 3.12.49-11 We are getting these kernel warning associated with

Re: [HACKERS] Our "fallback" atomics implementation doesn't actually work

2016-10-05 Thread Tom Lane
Andres Freund writes: > Without yet having analyzed this deeply, could it actually be that the > reason is that sem_post/wait aren't proper memory barriers? On a glance > the symptoms look like values have been modified without proper locks... Hmm, possible ...

[HACKERS] Fast AT ADD COLUMN with DEFAULTs

2016-10-05 Thread Serge Rielau
Dear Hackers, I’m working on a patch that expands PG’s ability to add columns to a table without a table rewrite (i.e. at O(1) cost) from the nullable-without-default to a more general case. E.g. CREATE TABLE T(pk INT NOT NULL PRIMARY KEY); INSERT INTO T VALEUS (1), (2), (3); ALTER TABLE T ADD

Re: [HACKERS] PostgreSQL - Weak DH group

2016-10-05 Thread Heikki Linnakangas
On 10/05/2016 05:15 PM, Nicolas Guini wrote: We are working with Postgres 9.3.14 and executing nmap we found that it is using “weak DH group” (nmap –script ssl-dh-params). Weak = 1024 bits. Yeah, it seems that we're a bit behind the times on this... This issue is similar

Re: [HACKERS] WIP: Secure Transport support as OpenSSL alternative on macOS

2016-10-05 Thread Magnus Hagander
On Wed, Oct 5, 2016 at 8:42 PM, Tom Lane wrote: > Robert Haas writes: > > On Wed, Oct 5, 2016 at 5:36 AM, Daniel Gustafsson > wrote: > >> The main questions raised here are: is it of interest to support > multiple SSL > >> libraries

Re: [HACKERS] Kernel Tainted

2016-10-05 Thread Andres Freund
Hi, On 2016-10-05 20:41:54 +0200, reiner peterke wrote: > We are getting these kernel warning associated with the postmaster process. > The application is handling around 15000TPS It appears that one of these > messages is generated for each each transaction which fills up the warn.log >

Re: [HACKERS] WIP: Covering + unique indexes.

2016-10-05 Thread Tom Lane
Robert Haas writes: > On Wed, Oct 5, 2016 at 9:04 AM, Amit Kapila wrote: >> Okay, but in that case I think we don't need to store including >> columns in non-leaf pages to get the exact ordering. As mentioned >> upthread, we can use truncated scan

[HACKERS] Kernel Tainted

2016-10-05 Thread reiner peterke
Hi, We are helping a client test an application On Power8 using Postgres 9.5.4 which has been compiled specifically for the Power. This is running on sles12sp1 the current kernel is 3.12.49-11 We are getting these kernel warning associated with the postmaster process. The application is

Re: [HACKERS] WIP: Secure Transport support as OpenSSL alternative on macOS

2016-10-05 Thread Tom Lane
Robert Haas writes: > On Wed, Oct 5, 2016 at 5:36 AM, Daniel Gustafsson wrote: >> The main questions raised here are: is it of interest to support multiple SSL >> libraries given the additional support burden and; is supporting Secure >> Transport of any

Re: [HACKERS] Our "fallback" atomics implementation doesn't actually work

2016-10-05 Thread Andres Freund
Hi, I was able to reproduce it in a read-write workload, instead of the read-only workload you'd proposed. On 2016-10-05 14:01:05 -0400, Tom Lane wrote: > Curiously, I did not see such a hang with regular SysV semaphores. > That may be just a timing thing, or it may have something to do with >

Re: [HACKERS] Our "fallback" atomics implementation doesn't actually work

2016-10-05 Thread Tom Lane
Andres Freund writes: > On 2016-10-05 14:01:05 -0400, Tom Lane wrote: >> configure USE_UNNAMED_POSIX_SEMAPHORES=1 --disable-cassert >> --disable-spinlocks --disable-atomics > Pretty independent from the complaint at hand, but if I just do that I get > undefined reference to

Re: [HACKERS] Our "fallback" atomics implementation doesn't actually work

2016-10-05 Thread Tom Lane
Andres Freund writes: > On 2016-10-05 14:01:05 -0400, Tom Lane wrote: >> I think what is happening is that there are circular assumptions that end >> up trying to implement a spinlock in terms of a spinlock, or otherwise >> somehow recursively use the process's semaphore.

Re: [HACKERS] Our "fallback" atomics implementation doesn't actually work

2016-10-05 Thread Andres Freund
On 2016-10-05 14:01:05 -0400, Tom Lane wrote: > I was trying to measure whether unnamed POSIX semaphores are any faster > or slower than the SysV kind. Plain pgbench is not very helpful for > determining this, because we've optimized the system to the point that > you don't hit semop waits all

Re: [HACKERS] Move allocation size overflow handling to MemoryContextAllocExtended()?

2016-10-05 Thread Andres Freund
On 2016-10-04 21:40:29 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2016-10-05 09:38:15 +0900, Michael Paquier wrote: > >> The existing interface of MemoryContextAlloc do not care much about > >> anything except Size, so I'd just give the responsability to the > >>

Re: [HACKERS] Our "fallback" atomics implementation doesn't actually work

2016-10-05 Thread Andres Freund
Hi, On 2016-10-05 14:01:05 -0400, Tom Lane wrote: > I think what is happening is that there are circular assumptions that end > up trying to implement a spinlock in terms of a spinlock, or otherwise > somehow recursively use the process's semaphore. It's a bit hard to tell > though because the

[HACKERS] Our "fallback" atomics implementation doesn't actually work

2016-10-05 Thread Tom Lane
I was trying to measure whether unnamed POSIX semaphores are any faster or slower than the SysV kind. Plain pgbench is not very helpful for determining this, because we've optimized the system to the point that you don't hit semop waits all that often. So I tried this: configure

Re: [HACKERS] Cardinality estimation for group by

2016-10-05 Thread Robert Haas
On Tue, Oct 4, 2016 at 8:05 AM, Chenxi Li wrote: > How is cardinality estimation for "group by" is done and where is the code > doing that? I would suggest that you start by looking at estimate_num_groups() in src/backend/utils/adt/selfuncs.c. You might also want to look at

Re: [HACKERS] asynchronous execution

2016-10-05 Thread Robert Haas
On Tue, Oct 4, 2016 at 7:53 AM, Amit Khandekar wrote: > Also, parent pointers are not required in the new design. Thinking of > parent pointers, now it seems the event won't get bubbled up the tree > with the new design. But still, , I think it's possible to switch over >

Re: [HACKERS] Incorrect comment/doc for poll_query_until

2016-10-05 Thread Robert Haas
On Tue, Oct 4, 2016 at 7:25 AM, Daniel Gustafsson wrote: > Commit 2a0f89cd717ce6d raised the timeout in poll_query_until() to 180 seconds > but left the documentation and comments at the previous value of 90 sec. > Patch > attached to update documentation and comment to match

Re: [HACKERS] [PATCH] pgpassfile connection option

2016-10-05 Thread Robert Haas
On Tue, Oct 4, 2016 at 7:42 AM, Julian Markwort wrote: > On 09/26/2016 07:51 PM, Robert Haas wrote: >> However, they don't have >> to accept the possibility that arbitrary local files readable by the >> user ID will be used for authentication and/or disclosed;

Re: [HACKERS] WIP: Covering + unique indexes.

2016-10-05 Thread Robert Haas
On Wed, Oct 5, 2016 at 9:04 AM, Amit Kapila wrote: > Okay, but in that case I think we don't need to store including > columns in non-leaf pages to get the exact ordering. As mentioned > upthread, we can use truncated scan key to reach to leaf level and > then use the

Re: [HACKERS] WIP: Secure Transport support as OpenSSL alternative on macOS

2016-10-05 Thread Robert Haas
On Wed, Oct 5, 2016 at 5:36 AM, Daniel Gustafsson wrote: > The main questions raised here are: is it of interest to support multiple SSL > libraries given the additional support burden and; is supporting Secure > Transport of any interest or is it a wasted effort to continue

Re: [HACKERS] Hash Indexes

2016-10-05 Thread Robert Haas
On Tue, Oct 4, 2016 at 12:36 AM, Amit Kapila wrote: > I think one way to avoid the risk of deadlock in above scenario is to > take the cleanup lock conditionally, if we get the cleanup lock then > we will delete the items as we are doing in patch now, else it will > just

Re: [HACKERS] pgbench more operators & functions

2016-10-05 Thread Fabien COELHO
Hello Tom, (2) The benchmark specification requires the client application to get hold of query results, which are currently discarded by pgbench, so pgbench does not really comply. I have submitted a patch to do that, see: I find this completely bogus. The data is delivered to the client,

Re: [HACKERS] WIP: Secure Transport support as OpenSSL alternative on macOS

2016-10-05 Thread Daniel Gustafsson
> On 05 Oct 2016, at 16:45, Stephen Frost wrote: > > Daniel, > > * Daniel Gustafsson (dan...@yesql.se) wrote: >> The main questions raised here are: is it of interest to support multiple SSL >> libraries given the additional support burden and; is supporting Secure >>

Re: [HACKERS] improving on pgrminclude / pgcompinclude ?

2016-10-05 Thread Robert Haas
On Wed, Oct 5, 2016 at 12:28 PM, Tom Lane wrote: > Robert Haas writes: >> I wonder if we can do better. The attached patch is the rest of a >> couple of hours of hacking on a "see whether all of our includes >> compile separately" project. Directions:

Re: [HACKERS] Cache Hash Index meta page.

2016-10-05 Thread Mithun Cy
On Tue, Oct 4, 2016 at 11:55 PM, Jeff Janes wrote: >Can you describe your benchmarking machine? Your benchmarking data went up to 128 clients. But how many cores does the machine have? Are >you testing how well it can use the resources it has, or how well it can deal with

Re: [HACKERS] improving on pgrminclude / pgcompinclude ?

2016-10-05 Thread Tom Lane
Robert Haas writes: > I wonder if we can do better. The attached patch is the rest of a > couple of hours of hacking on a "see whether all of our includes > compile separately" project. Directions: Don't we have more or less that already in

[HACKERS] improving on pgrminclude / pgcompinclude ?

2016-10-05 Thread Robert Haas
Amit's complaint about unnecessary headers got me thinking about src/tools/pginclude. I think the the stuff in this directory is trying to do something useful, because I think extra #include directives that we don't need are a useful thing to eliminate. However, in practice the contents are very

Re: [HACKERS] [PATCH] Generic type subscription

2016-10-05 Thread Artur Zakirov
Hello, On 04.10.2016 11:28, Victor Wagner wrote: Git complains about whitespace in this version of patch: $ git apply ../generic_type_subscription_v2.patch ../generic_type_subscription_v2.patch:218: tab in indent. int first; ../generic_type_subscription_v2.patch:219: tab in

Re: [HACKERS] pgbench more operators & functions

2016-10-05 Thread Tom Lane
Fabien COELHO writes: > [ re TPC-B ] > (1) The required schema is slightly different : currently the type used > for holding balances is not wide enough per the TCP-B standard, this mean > maybe having an option to do "pgbench -i --standard-tpcb" which would > generate

Re: [HACKERS] pgbench more operators & functions

2016-10-05 Thread Fabien COELHO
Hello Stephen, * Fabien COELHO (coe...@cri.ensmp.fr) wrote: I've got no objection to a more-nearly-TPC-B script as an option. Good, because adding a "per-spec" tpc-b as an additional builtin option is one of my intentions, once pgbench is capable of it. I believe it would be really

Re: [HACKERS] Autovacuum launcher process launches worker process at high frequency

2016-10-05 Thread Jeff Janes
On Wed, Oct 5, 2016 at 7:28 AM, Masahiko Sawada wrote: > Hi all, > > I found the kind of strange behaviour of the autovacuum launcher > process when XID anti-wraparound vacuum. > > Suppose that a database (say test_db) whose age of frozenxid is about > to reach

Re: [HACKERS] Question / requests.

2016-10-05 Thread Francisco Olarte
On Tue, Oct 4, 2016 at 7:50 PM, Robert Haas wrote: > On Mon, Oct 3, 2016 at 5:44 PM, Alvaro Herrera > wrote: ... >> I wonder if the real answer isn't just to disallow -f with parallel >> vacuuming. > Seems like we should figure out which catalog

Re: [HACKERS] pg_rewind hangs if --source-server is used and syncrep is enabled

2016-10-05 Thread Michael Banck
On Wed, Oct 05, 2016 at 04:39:39PM +0200, Michael Banck wrote: > if pg_rewind is told to fetch data via a libpq connection > (--source-server), synchronous replication is enabled and there is only > one sync standby (pilot error, but sill); pg_rewinding the old master > hangs at the CREATE

[HACKERS] PostgreSQL - Weak DH group

2016-10-05 Thread Nicolas Guini
Hello everyone, I sent few days ago to the security DL a mail reporting a vulnerability in how Postgres is requesting DH params to be used later for encryption algorithms. So, due to there is no problem sharing with this group, here is what I sent:

[HACKERS] pg_rewind hangs if --source-server is used and syncrep is enabled

2016-10-05 Thread Michael Banck
Hi, if pg_rewind is told to fetch data via a libpq connection (--source-server), synchronous replication is enabled and there is only one sync standby (pilot error, but sill); pg_rewinding the old master hangs at the CREATE TEMPORARY TABLE step (CREATE TABLE waiting for X/). At least

Re: [HACKERS] WIP: Secure Transport support as OpenSSL alternative on macOS

2016-10-05 Thread Stephen Frost
Daniel, * Daniel Gustafsson (dan...@yesql.se) wrote: > The main questions raised here are: is it of interest to support multiple SSL > libraries given the additional support burden and; is supporting Secure > Transport of any interest or is it a wasted effort to continue towards a full >

[HACKERS] Autovacuum launcher process launches worker process at high frequency

2016-10-05 Thread Masahiko Sawada
Hi all, I found the kind of strange behaviour of the autovacuum launcher process when XID anti-wraparound vacuum. Suppose that a database (say test_db) whose age of frozenxid is about to reach max_autovacuum_max_age has three tables T1 and T2. T1 is very large and is frequently updated, so

Re: [HACKERS] Relids in upper relations

2016-10-05 Thread Tom Lane
Ashutosh Bapat writes: > While reviewing aggregate pushdown patch [1] we noticed that > RelOptInfos for upper relations do not have relids set. Indeed, because they don't correspond to any particular scan relation or set of scan relations. I have in mind that in

Re: [HACKERS] WIP: Covering + unique indexes.

2016-10-05 Thread Amit Kapila
On Tue, Oct 4, 2016 at 7:50 PM, Robert Haas wrote: > On Tue, Oct 4, 2016 at 9:20 AM, Amit Kapila wrote: I'd say that the reason for not using included columns in any operations which require comparisons, is that they don't have opclass.

Re: [HACKERS] pgbench more operators & functions

2016-10-05 Thread Stephen Frost
Fabien, * Fabien COELHO (coe...@cri.ensmp.fr) wrote: > >I've got no objection to a more-nearly-TPC-B script as an option. > > Good, because adding a "per-spec" tpc-b as an additional builtin > option is one of my intentions, once pgbench is capable of it. I believe it would be really helpful to

[HACKERS] Relids in upper relations

2016-10-05 Thread Ashutosh Bapat
Hi, While reviewing aggregate pushdown patch [1] we noticed that RelOptInfos for upper relations do not have relids set. create_foreignscan_plan() copies the relids from RelOptInfo into ForeignScan::fs_relids. That field is used to identify the RTIs covered by the ForeignScan. For example,

Re: [HACKERS] Tracking wait event for latches

2016-10-05 Thread Robert Haas
On Tue, Oct 4, 2016 at 5:07 PM, Michael Paquier wrote: > More seriously, the Windows animals have been complaining about > pg_sleep() crashing the system: > SELECT pg_sleep(0.1); > ! server closed the connection unexpectedly > ! This probably means the server

Re: [HACKERS] Tracking wait event for latches

2016-10-05 Thread Robert Haas
On Tue, Oct 4, 2016 at 3:28 PM, Thomas Munro wrote: > Nitpicking: the includes in bgworker.c weren't sorted properly, and > then this patch added "pgstat.h" in the wrong position. See attached > suggestion. Committed. -- Robert Haas EnterpriseDB:

Re: [HACKERS] postgres_fdw : altering foreign table not invalidating prepare statement execution plan.

2016-10-05 Thread Etsuro Fujita
On 2016/10/05 14:09, Ashutosh Bapat wrote: I wrote: I think record_foreignscan_plan_dependencies in your patch would be a bit inefficient because that tracks such dependencies redundantly in the case where the given ForeignScan has an outer subplan, so I optimized that in the attached. +

Re: [HACKERS] Hash tables in dynamic shared memory

2016-10-05 Thread Dilip Kumar
On Wed, Oct 5, 2016 at 3:10 AM, Thomas Munro wrote: > Here is an experimental hash table implementation that uses DSA memory > so that hash tables can be shared by multiple backends and yet grow > dynamically. Development name: "DHT".

Re: [HACKERS] PATCH: two slab-like memory allocators

2016-10-05 Thread John Gorman
On Tue, Oct 4, 2016 at 10:11 PM, Tomas Vondra For GenSlab the situation is less clear, as there probably are ways to make > it work, but I'd vote to keep it simple for now, and simply do elog(ERROR) > in the realloc() methods - both for Slab and GenSlab. The current use case > (reorderbuffer)

Re: [HACKERS] Stopping logical replication protocol

2016-10-05 Thread Craig Ringer
On 5 October 2016 at 05:14, Andres Freund wrote: > Hi, > > On 2016-09-23 13:01:27 +0800, Craig Ringer wrote: >> From f98f2388c57d938ebbe07ccd2dbe02138312858f Mon Sep 17 00:00:00 2001 >> From: Vladimir Gordiychuk >> Date: Wed, 7 Sep 2016 00:39:18 +0300 >>

Re: [HACKERS] Feature suggestion: ON CONFLICT DO NOTHING RETURNING which returns existing row data

2016-10-05 Thread Pantelis Theodosiou
This can be solved by chaining modifying CTEs. Something like this (not tested) that can work with multiple rows inserted: WITH vals (bk1, bk2, other_t1_columns, other_t2_columns) AS ( VALUES (bk1val, bk2val, other_t1_values, other_t2_values), (bk1val, bk2val,

[HACKERS] WIP: Secure Transport support as OpenSSL alternative on macOS

2016-10-05 Thread Daniel Gustafsson
As noted in 77fb2321-9210-4ad4-b7fc-67623a583...@justatheory.com, OpenSSL is being somewhat dismantled on macOS starting with 10.11 El Capitan. To scratch my own itch I decided to take a stab at adding support for the Secure Transport library. The idea is to make it as much of a drop-in for the

Re: [HACKERS] Dynamic shared memory areas

2016-10-05 Thread Dilip Kumar
On Wed, Oct 5, 2016 at 3:00 AM, Thomas Munro wrote: > Here's a new version that does that. While testing this patch I found some issue, + total_size = DSA_INITIAL_SEGMENT_SIZE; + total_pages = total_size / FPM_PAGE_SIZE; + metadata_bytes = +

Re: [HACKERS] pgbench more operators & functions

2016-10-05 Thread Fabien COELHO
I've got no objection to a more-nearly-TPC-B script as an option. Good, because adding a "per-spec" tpc-b as an additional builtin option is one of my intentions, once pgbench is capable of it. But why do you feel the need to pull the default script out into a separate file? Seems to me

Re: [HACKERS] Declarative partitioning - another take

2016-10-05 Thread Amit Langote
Hi, On 2016/10/05 16:57, Rajkumar Raghuwanshi wrote: > I observed, when creating foreign table with range partition, data is not > inserting into specified partition range. below are steps to reproduce. > > [ ... ] > > postgres=# INSERT INTO test_range (a) values (5),(25),(15); > INSERT 0 3 >

Re: [HACKERS] [PATCH] Generic type subscription

2016-10-05 Thread Oleg Bartunov
On Wed, Oct 5, 2016 at 6:48 AM, Dmitry Dolgov <9erthali...@gmail.com> wrote: > On 5 October 2016 at 03:00, Oleg Bartunov wrote: > >> >> have you ever run 'make check' ? >> >> = >> 53 of 168 tests failed. >> = >> >> > Sure, how

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2016-10-05 Thread Tomas Vondra
On 09/05/2016 06:53 AM, Pavan Deolasee wrote: On Thu, Sep 1, 2016 at 9:44 PM, Bruce Momjian > wrote: On Thu, Sep 1, 2016 at 02:37:40PM +0530, Pavan Deolasee wrote: > I like the simplified approach, as long as it doesn't block further

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

2016-10-05 Thread Amit Kapila
On Wed, Oct 5, 2016 at 12:05 PM, Tomas Vondra wrote: > Hi, > > After collecting a lot more results from multiple kernel versions, I can > confirm that I see a significant improvement with 128 and 192 clients, > roughly by 30%: > >64

Re: [HACKERS] Declarative partitioning - another take

2016-10-05 Thread Rajkumar Raghuwanshi
On Tue, Oct 4, 2016 at 1:32 PM, Amit Langote wrote: Attached updated patches. > > Thanks, > Amit > Hi, I observed, when creating foreign table with range partition, data is not inserting into specified partition range. below are steps to reproduce. CREATE

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

2016-10-05 Thread Tomas Vondra
Hi, After collecting a lot more results from multiple kernel versions, I can confirm that I see a significant improvement with 128 and 192 clients, roughly by 30%: 64128192 master

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

2016-10-05 Thread Michael Paquier
(Squashing replies) On Fri, Sep 30, 2016 at 6:13 PM, Michael Paquier wrote: > On Fri, Sep 30, 2016 at 2:51 PM, Michael Paquier > wrote: >> On Fri, Sep 30, 2016 at 2:05 PM, Kyotaro HORIGUCHI >> wrote: >>> At

[HACKERS] strange case of kernel performance regression (4.3.x and newer)

2016-10-05 Thread Tomas Vondra
Hi, Over the past couple of days I've been doing a bit of benchmarking for the "group clog" patch [1], and I've ran into what I suspect might be a fairly serious performance regression on newer kernels (essentially 4.3.0 and newer). I got to a point where I need help with further

  1   2   >