Re: [Patch] pg_rewind: options to use restore_command from recovery.conf or command line

2019-03-19 Thread Andrey Borodin
Hi! > 7 марта 2019 г., в 20:27, Alexey Kondratov > написал(а): > > I'm a bit confused by by console output routines. E.g. in pg_rewind's main() you call pg_fatal()s, and printf(), and pg_log() with various levels. Shouldn't we use all the pg_* functions? But most of this printing usages

Re: Proposal to suppress errors thrown by to_reg*()

2019-03-19 Thread Pavel Stehule
st 20. 3. 2019 v 5:55 odesílatel Takuma Hoshiai napsal: > On Wed, 20 Mar 2019 09:48:59 +0900 (Tokyo Standard Time) > Kyotaro HORIGUCHI wrote: > > > At Wed, 20 Mar 2019 07:13:28 +0900 (JST), Tatsuo Ishii < > is...@sraoss.co.jp> wrote in < >

Re: current_logfiles not following group access and instead follows log_file_mode permissions

2019-03-19 Thread Michael Paquier
On Fri, Mar 15, 2019 at 06:51:37PM +1100, Haribabu Kommi wrote: > IMO, this update is just a recommendation to the user, and sometimes it is > still possible that there may be strict permissions for the log file > even the data directory is allowed for the group access. So I feel > it is still

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-03-19 Thread Paul Guo
On Wed, Mar 20, 2019 at 1:20 PM Michael Paquier wrote: > On Wed, Mar 20, 2019 at 12:48:52PM +0800, Paul Guo wrote: > > This is a good suggestion also. Will do it. > > Please note also that we don't care about recovery.conf since v12 as > recovery parameters are now GUCs. I would suggest

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-03-19 Thread Michael Paquier
On Wed, Mar 20, 2019 at 12:48:52PM +0800, Paul Guo wrote: > This is a good suggestion also. Will do it. Please note also that we don't care about recovery.conf since v12 as recovery parameters are now GUCs. I would suggest appending those extra parameters to postgresql.auto.conf, which is what

Re: Offline enabling/disabling of data checksums

2019-03-19 Thread Michael Paquier
On Wed, Mar 20, 2019 at 08:09:07AM +0900, Michael Paquier wrote: > In short, you keep the main feature with: > - No tweaks with postmaster.pid. > - Rely just on the control file indicating an instance shutdown > cleanly. > - No tweaks with the system ID. > - No renaming of the control file. FWIW,

Re: Proposal to suppress errors thrown by to_reg*()

2019-03-19 Thread Takuma Hoshiai
On Wed, 20 Mar 2019 09:48:59 +0900 (Tokyo Standard Time) Kyotaro HORIGUCHI wrote: > At Wed, 20 Mar 2019 07:13:28 +0900 (JST), Tatsuo Ishii > wrote in <20190320.071328.48576044685486.t-is...@sraoss.co.jp> > > >> I (and Hoshiai-san) concern about following case: > > >> > > >> # revoke usage

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-03-19 Thread Paul Guo
On Tue, Mar 19, 2019 at 2:18 PM Michael Paquier wrote: > On Tue, Mar 19, 2019 at 02:09:03PM +0800, Paul Guo wrote: > > The first patch adds an option to automatically generate recovery conf > > contents in related files, following pg_basebackup. In the patch, > > GenerateRecoveryConf(),

selecting from partitions and constraint exclusion

2019-03-19 Thread Amit Langote
Hi, While looking at a partition pruning bug [1], I noticed something that started to feel like a regression: Setup: create table p (a int) partition by list (a); create table p1 partition of p for values in (1); In PG 10: set constraint_exclusion to on; explain select * from p1 where a = 2;

Re: [HACKERS] Block level parallel vacuum

2019-03-19 Thread Masahiko Sawada
On Tue, Mar 19, 2019 at 7:29 PM Kyotaro HORIGUCHI wrote: > > At Tue, 19 Mar 2019 17:51:32 +0900, Masahiko Sawada > wrote in > > On Tue, Mar 19, 2019 at 10:39 AM Haribabu Kommi > > wrote: > > > The performance results are good. Do we want to add the recommended > > > size in the document for

Re: Special role for subscriptions

2019-03-19 Thread Euler Taveira
Em qui, 14 de mar de 2019 às 00:03, Stephen Frost escreveu: > > I view that as the first step towards building a more granular privilege > system for subscription creation, and that was the second half of what I > was trying to say before- I do think there's value in having something > more

RE: [survey] New "Stable" QueryId based on normalized query text

2019-03-19 Thread Tsunakawa, Takayuki
From: Kyotaro HORIGUCHI [mailto:horiguchi.kyot...@lab.ntt.co.jp] > > > needs.1: stable accross different databases, > > > > Does this mean different database clusters, not different databases in > a single database cluster? > > Does this mean you want different QueryID for the same-looking >

RE: speeding up planning with partitions

2019-03-19 Thread Imai, Yoshikazu
Amit-san, On Wed, Mar 20, 2019 at 3:01 PM, Amit Langote wrote: > > On Wed, Mar 20, 2019 at 2:34 AM, Amit Langote wrote: > >> On 2019/03/20 11:21, Imai, Yoshikazu wrote: > >>> (4) > >>> We expect the performance does not depend on the number of > >>> partitions > >> after applying all patches, if

Re: speeding up planning with partitions

2019-03-19 Thread Amit Langote
On 2019/03/20 11:51, Imai, Yoshikazu wrote: > Amit-san, > > On Wed, Mar 20, 2019 at 2:34 AM, Amit Langote wrote: >> On 2019/03/20 11:21, Imai, Yoshikazu wrote: >>> (4) >>> We expect the performance does not depend on the number of partitions >> after applying all patches, if possible. >>> >>> num

Re: Automated way to find actual COMMIT LSN of subxact LSN

2019-03-19 Thread Kyotaro HORIGUCHI
At Tue, 19 Mar 2019 12:16:34 -0500, Jeremy Finzel wrote in > I want to build automation to recover a database to a specific LSN > *inclusive*, even if that LSN is from a subtransaction. The problem I am > facing is that I know what specific LSN wrote a row on a remote system, but > if I create

Re: [HACKERS] generated columns

2019-03-19 Thread Michael Paquier
On Mon, Mar 18, 2019 at 03:14:09PM +0100, Pavel Stehule wrote: > postgres=# update foo set name = 'bbbxx' where id = 1; -- error > ERROR: no generation expression found for column number 3 of table > "foo" Yes I can see the problem after adding a generated column and dropping it on an INSERT

RE: speeding up planning with partitions

2019-03-19 Thread Imai, Yoshikazu
Amit-san, On Wed, Mar 20, 2019 at 2:34 AM, Amit Langote wrote: > On 2019/03/20 11:21, Imai, Yoshikazu wrote: > > (4) > > We expect the performance does not depend on the number of partitions > after applying all patches, if possible. > > > > num of partTPS > > --- - > > 1024

Re: Sparse bit set data structure

2019-03-19 Thread Andrey Borodin
Hi! Great job! > 20 марта 2019 г., в 9:10, Heikki Linnakangas написал(а): > > Please review, if you have a chance. > > - Heikki > <0001-Add-IntegerSet-to-hold-large-sets-of-64-bit-ints-eff.patch> I'm looking into the code and have few questions: 1. I'm not sure it is the best interface for

Re: speeding up planning with partitions

2019-03-19 Thread Amit Langote
Imai-san, On 2019/03/20 11:21, Imai, Yoshikazu wrote: > (4) > We expect the performance does not depend on the number of partitions after > applying all patches, if possible. > > num of partTPS > --- - > 1024 7,257 (7274, 7246, 7252) > 2048 6,718 (6627, 6780,

RE: Transaction commits VS Transaction commits (with parallel) VS query mean time

2019-03-19 Thread Jamison, Kirk
I tried to confirm the patch with the following configuration: max_parallel_workers_per_gather = 2 autovacuum = off postgres=# BEGIN; BEGIN postgres=# select xact_commit from pg_stat_database where datname = 'postgres'; xact_commit - 118 (1 row) postgres=# explain

RE: speeding up planning with partitions

2019-03-19 Thread Imai, Yoshikazu
Amit-san, On Wed, Mar 20, 2019 at 0:42 AM, Amit Langote wrote: > On 2019/03/19 20:13, Imai, Yoshikazu wrote: > > Thanks for new patches. > > I looked over them and there are little comments. > > > > ... > > > > I have no more comments about codes other than above :) > > I have fixed all.

Re: Willing to fix a PQexec() in libpq module

2019-03-19 Thread Andres Freund
Hi, On 2019-03-20 02:19:54 +, Wu, Fei wrote: > Hi, thanks for all replies. > According to all your discussions, Maybe the problems is that > 1) keep modifications just in client side; > 2) modifications VS client current applications > > Maybe we could create a new function(May called

RE: Willing to fix a PQexec() in libpq module

2019-03-19 Thread Wu, Fei
Hi, thanks for all replies. According to all your discussions, Maybe the problems is that 1) keep modifications just in client side; 2) modifications VS client current applications Maybe we could create a new function(May called PQexecSafe() ) just likes PQexec() but with additional input

Re: Special role for subscriptions

2019-03-19 Thread Andrey Borodin
Hi! > 13 марта 2019 г., в 22:55, Evgeniy Efimkin > написал(а): > > I've prepare patch with new system role, i'm not sure about name, called it > "pg_subscription_users". > In that patch we don't check permissions on target tables, i don't know, > should we check it? Currently, user with

Re: BUG #15668: Server crash in transformPartitionRangeBounds

2019-03-19 Thread Michael Paquier
On Thu, Mar 14, 2019 at 01:23:08PM +0900, Michael Paquier wrote: > I actually think that what you propose here makes more sense than what > HEAD does because the most inner expression gets evaluated first. > This for example generates the same error as on HEAD: > =# create table foo (a int default

Re: [survey] New "Stable" QueryId based on normalized query text

2019-03-19 Thread Kyotaro HORIGUCHI
At Wed, 20 Mar 2019 00:23:30 +, "Tsunakawa, Takayuki" wrote in <0A3221C70F24FB45833433255569204D1FBE20A4@G01JPEXMBYT05> > From: legrand legrand [mailto:legrand_legr...@hotmail.com] > > There are many projects that use alternate QueryId > > distinct from the famous pg_stat_statements

Re: Sparse bit set data structure

2019-03-19 Thread Heikki Linnakangas
On 14/03/2019 17:37, Julien Rouhaud wrote: On Wed, Mar 13, 2019 at 8:18 PM Heikki Linnakangas wrote: I started to consider rewriting the data structure into something more like B-tree. Then I remembered that I wrote a data structure pretty much like that last year already! We discussed that

Re: speeding up planning with partitions

2019-03-19 Thread Amit Langote
On 2019/03/20 9:49, Robert Haas wrote: > On Fri, Mar 8, 2019 at 4:18 AM Amit Langote > wrote: >> Maybe you know that range_table_mutator() spends quite a long time if >> there are many target children, but I realized there's no need for >> range_table_mutator() to copy/mutate child target RTEs.

Re: speeding up planning with partitions

2019-03-19 Thread Robert Haas
On Fri, Mar 8, 2019 at 4:18 AM Amit Langote wrote: > Maybe you know that range_table_mutator() spends quite a long time if > there are many target children, but I realized there's no need for > range_table_mutator() to copy/mutate child target RTEs. First, there's > nothing to translate in their

Re: Proposal to suppress errors thrown by to_reg*()

2019-03-19 Thread Kyotaro HORIGUCHI
At Wed, 20 Mar 2019 07:13:28 +0900 (JST), Tatsuo Ishii wrote in <20190320.071328.48576044685486.t-is...@sraoss.co.jp> > >> I (and Hoshiai-san) concern about following case: > >> > >> # revoke usage on schema s1 from foo; > >> REVOKE > >> : > >> [connect as foo] > >> test=> select

RE: [survey] New "Stable" QueryId based on normalized query text

2019-03-19 Thread Tsunakawa, Takayuki
From: legrand legrand [mailto:legrand_legr...@hotmail.com] > There are many projects that use alternate QueryId > distinct from the famous pg_stat_statements jumbling algorithm. I'd like to welcome the standard QueryID that DBAs and extension developers can depend on. Are you surveying the

Re: Proposal to suppress errors thrown by to_reg*()

2019-03-19 Thread Kyotaro HORIGUCHI
At Tue, 19 Mar 2019 19:09:59 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20190319.190959.25783254.horiguchi.kyot...@lab.ntt.co.jp> > That works in a transaction. It looks right that the actually > revoked schema cannot be accessed. >From another viewpoint, the behavior really

Re: Online verification of checksums

2019-03-19 Thread Michael Paquier
On Tue, Mar 19, 2019 at 02:44:52PM -0700, Andres Freund wrote: > That's *PRECISELY* my point. I think it's a bad idea to do online > checksumming from outside the backend. It needs to be inside the > backend, and if there's any verification failures on a block, it needs > to acquire the IO lock on

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2019-03-19 Thread Julien Rouhaud
On Tue, Mar 19, 2019 at 8:38 PM legrand legrand wrote: > > Would it make sense to add it in auto explain ? > I don't know for explain itself, but maybe ... I'd think that people interested in getting the queryid in the logs would configure the log_line_prefix to display it consistently rather

Re: Fwd: Add tablespace tap test to pg_rewind

2019-03-19 Thread Michael Paquier
On Tue, Mar 19, 2019 at 08:16:21PM +0800, Shaoqi Bai wrote: > Thanks for your advice, sorry for taking so long to give update in the > thread, because I am stuck in modifing Perl script, knowing little about > Perl language. No problem. It is true that using perl for the first time can be a

Re: Compressed TOAST Slicing

2019-03-19 Thread Paul Ramsey
> On Mar 19, 2019, at 4:47 AM, Stephen Frost wrote: > > Greetings, > > * Paul Ramsey (pram...@cleverelephant.ca) wrote: >>> On Mar 18, 2019, at 7:34 AM, Robert Haas wrote: >>> +1. I think Paul had it right originally. >> >> In that spirit, here is a “one pglz_decompress function, new

Re: Making all nbtree entries unique by having heap TIDs participate in comparisons

2019-03-19 Thread Peter Geoghegan
On Mon, Mar 18, 2019 at 10:17 AM Peter Geoghegan wrote: > The big difference is that you make the possible call to > _bt_stepright() conditional on this being a checkingunique index -- > the duplicate code is indented in that branch of _bt_findsplitloc(). > Whereas I break early in the loop when

Re: Offline enabling/disabling of data checksums

2019-03-19 Thread Michael Paquier
On Tue, Mar 19, 2019 at 09:47:17AM -0700, Andres Freund wrote: > I'm not sure it needs to be this patch's responsibility to come up with > a scheme here at all however. pg_rewind, pg_resetwal, pg_upgrade all > don't really have a lockout mechanism, and it hasn't caused a ton of > problems. I think

Re: Proposal to suppress errors thrown by to_reg*()

2019-03-19 Thread Tatsuo Ishii
>> I (and Hoshiai-san) concern about following case: >> >> # revoke usage on schema s1 from foo; >> REVOKE >> : >> [connect as foo] >> test=> select to_regclass('s1.t1')::oid; >> ERROR: permission denied for schema s1 > > That works in a transaction. It looks right that the actually > revoked

Re: Online verification of checksums

2019-03-19 Thread Andres Freund
Hi, On 2019-03-19 22:39:16 +0100, Michael Banck wrote: > Am Dienstag, den 19.03.2019, 13:00 -0700 schrieb Andres Freund: > > a) checks that the page is all zeroes if PageIsNew() (like > >PageIsVerified() does for the backend). That avoids missing cases > >where corruption just zeroed out

Re: Online verification of checksums

2019-03-19 Thread Michael Banck
Hi, Am Dienstag, den 19.03.2019, 13:00 -0700 schrieb Andres Freund: > On 2019-03-20 03:27:55 +0800, Stephen Frost wrote: > > On Tue, Mar 19, 2019 at 23:59 Andres Freund wrote: > > > On 2019-03-19 16:52:08 +0100, Michael Banck wrote: > > > > Am Dienstag, den 19.03.2019, 11:22 -0400 schrieb Robert

Re: Online verification of checksums

2019-03-19 Thread Robert Haas
On Tue, Mar 19, 2019 at 4:49 PM Andres Freund wrote: > To demonstrate that I ran a loop that verified that a) a normal backend > query using the tale detects the corruption b) pg_basebackup doesn't. > > i=0; > while true; do > i=$(($i+1)); > echo attempt $i; > dd if=/dev/urandom

[survey] New "Stable" QueryId based on normalized query text

2019-03-19 Thread legrand legrand
Hello, There are many projects that use alternate QueryId distinct from the famous pg_stat_statements jumbling algorithm. https://github.com/postgrespro/aqo query_hash https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraPostgreSQL.Optimize.ViewPlans.html sql_hash

Re: Online verification of checksums

2019-03-19 Thread Andres Freund
On 2019-03-19 13:00:50 -0700, Andres Freund wrote: > As it stands, the logic seems to give more false confidence than > anything else. To demonstrate that I ran a loop that verified that a) a normal backend query using the tale detects the corruption b) pg_basebackup doesn't. i=0; while true; do

Re: Online verification of checksums

2019-03-19 Thread Andres Freund
Hi, On 2019-03-20 03:27:55 +0800, Stephen Frost wrote: > On Tue, Mar 19, 2019 at 23:59 Andres Freund wrote: > > On 2019-03-19 16:52:08 +0100, Michael Banck wrote: > > > Am Dienstag, den 19.03.2019, 11:22 -0400 schrieb Robert Haas: > > > > It's torn pages that I am concerned about - the server is

Re: Rare SSL failures on eelpout

2019-03-19 Thread Thomas Munro
On Wed, Mar 20, 2019 at 8:31 AM Tom Lane wrote: > So I'm inclined to (1) commit the patch as-proposed in HEAD, and > (2) hack the ssl test cases in v11 as you suggested. If we see field > complaints about this, we can consider reverting (2) in favor of > a back-patch once v12 beta is over. This

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2019-03-19 Thread legrand legrand
Great, thank you Julien ! Would it make sense to add it in auto explain ? I don't know for explain itself, but maybe ... Regards PAscal -- Sent from: http://www.postgresql-archive.org/PostgreSQL-hackers-f1928748.html

Re: Rare SSL failures on eelpout

2019-03-19 Thread Tom Lane
Thomas Munro writes: > On Tue, Mar 19, 2019 at 9:11 AM Tom Lane wrote: >>> One thing that isn't real clear to me is how much timing sensitivity >>> there is in "when no more input data is available". Can we assume that >>> if we've gotten ECONNRESET or an allied error from a write, then any >>>

Re: Online verification of checksums

2019-03-19 Thread Stephen Frost
Greetings, On Tue, Mar 19, 2019 at 23:59 Andres Freund wrote: > Hi, > > On 2019-03-19 16:52:08 +0100, Michael Banck wrote: > > Am Dienstag, den 19.03.2019, 11:22 -0400 schrieb Robert Haas: > > > It's torn pages that I am concerned about - the server is writing and > > > we are reading, and we

Re: Willing to fix a PQexec() in libpq module

2019-03-19 Thread Daniel Verite
Tom Lane wrote: > Unfortunately, if the default behavior doesn't change, then there's little > argument for doing this at all. The security reasoning behind doing > anything in this area would be to provide an extra measure of protection > against SQL-injection attacks on

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2019-03-19 Thread Robert Haas
On Tue, Mar 19, 2019 at 1:24 PM Jim Finnerty wrote: > The queryId depends on oids, so it is not stable enough for some purposes. > For example, to create a SQL identifier that survives across a server > upgrade, or that can be shipped to another database, the queryId isn't > usable. > > The

Re: [HACKERS] CLUSTER command progress monitor

2019-03-19 Thread Robert Haas
On Mon, Mar 18, 2019 at 10:03 PM Tatsuro Yamada wrote: > Attached patch is a rebased document patch. :) Attached is an updated patch. I went through this patch carefully today, in the hopes of committing it, and I think the attached version is pretty closet to being committable, but there's at

Re: Willing to fix a PQexec() in libpq module

2019-03-19 Thread Tom Lane
Andres Freund writes: > On 2019-03-19 13:18:25 -0400, Tom Lane wrote: >> Having said that ... a better argument for a new API is that it >> could be explicitly designed to handle multiple queries, and in >> particular make some provision for returning multiple PGresults. > Oh, I completely

Re: Concurrency bug with vacuum full (cluster) and toast

2019-03-19 Thread Alexander Korotkov
On Tue, Mar 19, 2019 at 6:48 PM Robert Haas wrote: > On Mon, Mar 18, 2019 at 12:53 PM Alexander Korotkov > wrote: > > I've discovered bug, when vacuum full fails with error, because it > > couldn't find toast chunks deleted by itself. That happens because > > cluster_rel() sets OldestXmin, but

Re: extensions are hitting the ceiling

2019-03-19 Thread Eric Hanson
On Mon, Mar 18, 2019 at 11:56 PM Chapman Flack wrote: > On 03/18/19 22:38, Eric Hanson wrote: > > rows are really second class citizens: They aren't tracked with > > pg_catalog.pg_depend, they aren't deleted when the extension is dropped, > > etc. > > This. You have other interests as well, but

Re: Willing to fix a PQexec() in libpq module

2019-03-19 Thread David Fetter
On Tue, Mar 19, 2019 at 01:18:25PM -0400, Tom Lane wrote: > Andres Freund writes: > > On 2019-03-19 13:59:34 -0300, Alvaro Herrera wrote: > >> I suppose it can be argued that for the cases where they want that, it > >> is not entirely ridiculous to have it be done with a different API call, > >>

Re: Willing to fix a PQexec() in libpq module

2019-03-19 Thread Andres Freund
Hi, On 2019-03-19 13:18:25 -0400, Tom Lane wrote: > Having said that ... a better argument for a new API is that it > could be explicitly designed to handle multiple queries, and in > particular make some provision for returning multiple PGresults. Oh, I completely agree, that'd be hugely

Re: Willing to fix a PQexec() in libpq module

2019-03-19 Thread Tom Lane
Andres Freund writes: > On 2019-03-19 13:59:34 -0300, Alvaro Herrera wrote: >> I suppose it can be argued that for the cases where they want that, it >> is not entirely ridiculous to have it be done with a different API call, >> say PQexecMultiple. > Sure, but what'd the gain be? Using

Automated way to find actual COMMIT LSN of subxact LSN

2019-03-19 Thread Jeremy Finzel
I want to build automation to recover a database to a specific LSN *inclusive*, even if that LSN is from a subtransaction. The problem I am facing is that I know what specific LSN wrote a row on a remote system, but if I create a recovery.conf file with: recovery_target_lsn = '95F/BBA36DF8' and

Re: jsonpath

2019-03-19 Thread Alexander Korotkov
On Sun, Mar 17, 2019 at 6:03 PM Tom Lane wrote: > Andrew Dunstan writes: > > Why are we installing the jsonpath_gram.h file? It's not going to be > > used by anything else, is it? TBH, I'm not sure I see why we're > > installing the scanner.h file either. > > As near as I can see,

Re: Willing to fix a PQexec() in libpq module

2019-03-19 Thread Andres Freund
On 2019-03-19 10:02:33 -0700, Andres Freund wrote: > Hi, > > On 2019-03-19 13:59:34 -0300, Alvaro Herrera wrote: > > On 2019-Mar-19, Andres Freund wrote: > > > On 2019-03-19 12:51:39 -0400, Tom Lane wrote: > > > > David Fetter writes: > > > > > I think the answer is "no," and we should deprecate

Re: Willing to fix a PQexec() in libpq module

2019-03-19 Thread Andres Freund
Hi, On 2019-03-19 13:59:34 -0300, Alvaro Herrera wrote: > On 2019-Mar-19, Andres Freund wrote: > > On 2019-03-19 12:51:39 -0400, Tom Lane wrote: > > > David Fetter writes: > > > > I think the answer is "no," and we should deprecate this misfeature. > > > > It's bad enough that we'll be

Re: Willing to fix a PQexec() in libpq module

2019-03-19 Thread David Fetter
On Tue, Mar 19, 2019 at 01:59:34PM -0300, Alvaro Herrera wrote: > On 2019-Mar-19, Andres Freund wrote: > > > Hi, > > > > On 2019-03-19 12:51:39 -0400, Tom Lane wrote: > > > David Fetter writes: > > > > I think the answer is "no," and we should deprecate this misfeature. > > > > It's bad enough

Re: Willing to fix a PQexec() in libpq module

2019-03-19 Thread Alvaro Herrera
On 2019-Mar-19, Andres Freund wrote: > Hi, > > On 2019-03-19 12:51:39 -0400, Tom Lane wrote: > > David Fetter writes: > > > I think the answer is "no," and we should deprecate this misfeature. > > > It's bad enough that we'll be supporting it for five years after > > > deprecating it, but it's

Re: Willing to fix a PQexec() in libpq module

2019-03-19 Thread Andres Freund
Hi, On 2019-03-19 12:51:39 -0400, Tom Lane wrote: > David Fetter writes: > > I think the answer is "no," and we should deprecate this misfeature. > > It's bad enough that we'll be supporting it for five years after > > deprecating it, but it's worse to leave it hanging around our necks > >

Re: Willing to fix a PQexec() in libpq module

2019-03-19 Thread Tom Lane
David Fetter writes: > I think the answer is "no," and we should deprecate this misfeature. > It's bad enough that we'll be supporting it for five years after > deprecating it, but it's worse to leave it hanging around our necks > forever. https://en.wikipedia.org/wiki/Albatross_(metaphor) The

Re: Offline enabling/disabling of data checksums

2019-03-19 Thread Andres Freund
Hi, On 2019-03-19 17:30:16 +0100, Michael Banck wrote: > Am Dienstag, den 19.03.2019, 09:13 -0700 schrieb Andres Freund: > > On 2019-03-19 17:08:17 +0100, Michael Banck wrote: > > > Am Dienstag, den 19.03.2019, 09:00 -0700 schrieb Andres Freund: > > > > On 2019-03-19 16:55:12 +0100, Michael Banck

Re: Libpq support to connect to standby server as priority

2019-03-19 Thread Robert Haas
On Mon, Mar 18, 2019 at 9:33 PM Haribabu Kommi wrote: > While working on implementation of target_server_type new connection option > for the libpq > to specify master, slave and etc, there is no problem when the newly added > target_server_type > option is used separate, but when it is

Re: Willing to fix a PQexec() in libpq module

2019-03-19 Thread David Fetter
On Tue, Mar 19, 2019 at 10:30:45AM -0400, Tom Lane wrote: > Kyotaro HORIGUCHI writes: > > At Tue, 19 Mar 2019 08:18:23 +, "Wu, Fei" > > wrote in > > <52E6E0843B9D774C8C73D6CF64402F05621F0FFC@G08CNEXMBPEKD02.g08.fujitsu.local> > >> I will try to fix it~ > > > I don't oppose that, but as

Re: Offline enabling/disabling of data checksums

2019-03-19 Thread Michael Banck
Hi, Am Dienstag, den 19.03.2019, 09:13 -0700 schrieb Andres Freund: > On 2019-03-19 17:08:17 +0100, Michael Banck wrote: > > Am Dienstag, den 19.03.2019, 09:00 -0700 schrieb Andres Freund: > > > On 2019-03-19 16:55:12 +0100, Michael Banck wrote: > > > > Am Dienstag, den 19.03.2019, 08:36 -0700

Re: Offline enabling/disabling of data checksums

2019-03-19 Thread Andres Freund
Hi, On 2019-03-19 17:08:17 +0100, Michael Banck wrote: > Am Dienstag, den 19.03.2019, 09:00 -0700 schrieb Andres Freund: > > On 2019-03-19 16:55:12 +0100, Michael Banck wrote: > > > Am Dienstag, den 19.03.2019, 08:36 -0700 schrieb Andres Freund: > > > > On 2019-03-18 17:13:01 +0900, Michael

Re: Offline enabling/disabling of data checksums

2019-03-19 Thread Michael Banck
Hi, Am Dienstag, den 19.03.2019, 09:00 -0700 schrieb Andres Freund: > On 2019-03-19 16:55:12 +0100, Michael Banck wrote: > > Am Dienstag, den 19.03.2019, 08:36 -0700 schrieb Andres Freund: > > > On 2019-03-18 17:13:01 +0900, Michael Paquier wrote: > > > > +/* > > > > + * Locations of persistent

Re: Offline enabling/disabling of data checksums

2019-03-19 Thread Andres Freund
On 2019-03-19 16:55:12 +0100, Michael Banck wrote: > Hi, > > Am Dienstag, den 19.03.2019, 08:36 -0700 schrieb Andres Freund: > > On 2019-03-18 17:13:01 +0900, Michael Paquier wrote: > > > +/* > > > + * Locations of persistent and temporary control files. The control > > > + * file gets renamed

Re: Online verification of checksums

2019-03-19 Thread Andres Freund
Hi, On 2019-03-19 16:52:08 +0100, Michael Banck wrote: > Am Dienstag, den 19.03.2019, 11:22 -0400 schrieb Robert Haas: > > It's torn pages that I am concerned about - the server is writing and > > we are reading, and we get a mix of old and new content. We have been > > quite diligent about

Re: Offline enabling/disabling of data checksums

2019-03-19 Thread Michael Banck
Hi, Am Dienstag, den 19.03.2019, 08:36 -0700 schrieb Andres Freund: > On 2019-03-18 17:13:01 +0900, Michael Paquier wrote: > > +/* > > + * Locations of persistent and temporary control files. The control > > + * file gets renamed into a temporary location when enabling checksums > > + * to

Re: Online verification of checksums

2019-03-19 Thread Michael Banck
Hi, Am Dienstag, den 19.03.2019, 11:22 -0400 schrieb Robert Haas: > It's torn pages that I am concerned about - the server is writing and > we are reading, and we get a mix of old and new content. We have been > quite diligent about protecting ourselves from such risks elsewhere, > and checksum

Re: pg_upgrade version checking questions

2019-03-19 Thread Tom Lane
Peter Eisentraut writes: > On 2019-03-19 00:35, Tom Lane wrote: >> 2. check_cluster_versions() insists that the target version be the >> same major version as pg_upgrade itself, but is that really good enough? >> As things stand, it looks like pg_upgrade 11.3 would happily use pg_dump >> 11.1, or

Re: Concurrency bug with vacuum full (cluster) and toast

2019-03-19 Thread Robert Haas
On Mon, Mar 18, 2019 at 12:53 PM Alexander Korotkov wrote: > I've discovered bug, when vacuum full fails with error, because it > couldn't find toast chunks deleted by itself. That happens because > cluster_rel() sets OldestXmin, but toast accesses gets snapshot later > and independently. That

Re: [HACKERS] Custom compression methods

2019-03-19 Thread Chris Travers
On Tue, Mar 19, 2019 at 12:19 PM Tomas Vondra wrote: > > On 3/19/19 10:59 AM, Chris Travers wrote: > > > > > > Not discussing whether any particular committer should pick this up but > > I want to discuss an important use case we have at Adjust for this sort > > of patch. > > > > The PostgreSQL

Re: Speed up transaction completion faster after many relations are accessed in a transaction

2019-03-19 Thread Peter Eisentraut
On 2019-03-19 10:21, Tsunakawa, Takayuki wrote: > From: Tsunakawa, Takayuki [mailto:tsunakawa.ta...@jp.fujitsu.com] >> Fixed. > > Rebased on HEAD. I have committed the first patch that reorganizes the struct. I'll have to spend some time evaluating the performance impact of the second patch,

Re: Offline enabling/disabling of data checksums

2019-03-19 Thread Andres Freund
Hi, On 2019-03-18 17:13:01 +0900, Michael Paquier wrote: > +/* > + * Locations of persistent and temporary control files. The control > + * file gets renamed into a temporary location when enabling checksums > + * to prevent a parallel startup of Postgres. > + */ > +#define CONTROL_FILE_PATH

Re: BUG #15572: Misleading message reported by "Drop function operation" on DB with functions having same name

2019-03-19 Thread Pavel Stehule
The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, passed Spec compliant: not tested Documentation:not tested I read a discussion and I think so currently implemented behave (by

Re: Online verification of checksums

2019-03-19 Thread Robert Haas
On Mon, Mar 18, 2019 at 2:38 AM Stephen Frost wrote: > Sure the backend has those facilities since it needs to, but these > frontend tools *don't* need that to *never* have any false positives, so > why are we complicating things by saying that this frontend tool and the > backend have to

Re: pg_upgrade version checking questions

2019-03-19 Thread Peter Eisentraut
On 2019-03-19 00:35, Tom Lane wrote: > 2. check_cluster_versions() insists that the target version be the > same major version as pg_upgrade itself, but is that really good enough? > As things stand, it looks like pg_upgrade 11.3 would happily use pg_dump > 11.1, or vice versa. I'd hesitate to

Re: Problem with default partition pruning

2019-03-19 Thread Thibaut Madelaine
Le 19/03/2019 à 08:01, Yuzuko Hosoya a écrit : > Hi Amit-san, > > From: Amit Langote [mailto:langote_amit...@lab.ntt.co.jp] > Sent: Monday, March 18, 2019 6:44 PM > >> Hosoya-san, >> >> On 2019/03/15 15:05, Yuzuko Hosoya wrote: >>> Indeed, it's problematic. I also did test and I found that

Re: [HACKERS] Cached plans and statement generalization

2019-03-19 Thread Konstantin Knizhnik
Thank you very much for the review! On 19.03.2019 5:56, Yamaji, Ryo wrote: On Tue, Jan 29, 2019 at 10:46 AM, Konstantin Knizhnik wrote: Rebased version of the patch is attached. I'm sorry for the late review. I confirmed behavior of autoprepare-12.patch. It is summarized below. ・parameter

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2019-03-19 Thread Julien Rouhaud
On Mon, Mar 18, 2019 at 7:33 PM Julien Rouhaud wrote: > > On Mon, Mar 18, 2019 at 6:23 PM Yun Li wrote: > > > > Let's take one step back. Since queryId is stored in core as Julien pointed > > out, can we just add that global to the pg_stat_get_activity and ultimately > > exposed in

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2019-03-19 Thread Julien Rouhaud
On Tue, Mar 19, 2019 at 2:45 PM Maksim Milyutin wrote: > > But I think that enough to integrate into core the query normalization > routine and store generalized query strings (from which the queryId is > produced) in shared memory (for example, hashtable that maps queryId to > the text

Re: Willing to fix a PQexec() in libpq module

2019-03-19 Thread Tom Lane
Kyotaro HORIGUCHI writes: > At Tue, 19 Mar 2019 08:18:23 +, "Wu, Fei" > wrote in > <52E6E0843B9D774C8C73D6CF64402F05621F0FFC@G08CNEXMBPEKD02.g08.fujitsu.local> >> I will try to fix it~ > I don't oppose that, but as the discussion linked from there [1], > psql already has a feature that

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2019-03-19 Thread Maksim Milyutin
On 3/16/19 5:32 PM, Robert Haas wrote: There's only one query ID field available, and you can't use two extensions that care about query ID unless they compute it the same way, and replicating all the code that computes the query ID into each new extension that wants one sucks. I think we

Re: partitioned tables referenced by FKs

2019-03-19 Thread Alvaro Herrera
On 2019-Mar-14, Robert Haas wrote: > On Thu, Mar 14, 2019 at 3:36 PM Alvaro Herrera > wrote: > > In any case, since the RI > > queries are run via SPI, any unnecessary partitions should get pruned by > > partition pruning based on each partition's constraint. So I'm not so > > certain that

Re: Index Skip Scan

2019-03-19 Thread Dmitry Dolgov
> On Sat, Mar 16, 2019 at 5:14 PM Dmitry Dolgov <9erthali...@gmail.com> wrote: > > > On Fri, Mar 15, 2019 at 4:55 AM Kyotaro HORIGUCHI > > wrote: > > I have some comments on the latest v11 patch. > > Thank you! In the meantime here is a new version, rebased after tableam changes.

Re: pg_upgrade version checking questions

2019-03-19 Thread Daniel Gustafsson
On Tuesday, March 19, 2019 7:55 AM, Bruce Momjian wrote: > On Tue, Mar 19, 2019 at 02:43:49AM -0400, Bruce Momjian wrote: > > > > 3. Actually, I'm kind of wondering why pg_upgrade has a --new-bindir > > > option at all, rather than just insisting on finding the new-version > > >

Re: Offline enabling/disabling of data checksums

2019-03-19 Thread Fabien COELHO
Ok, this might not work, because of the following, less likely, race condition: postmaster opens control file RW pg_checksums moves control file, postmater open file handle follows ... So ISTM that we really need some locking to have something clean. We are talking about complicating a

Re: Offline enabling/disabling of data checksums

2019-03-19 Thread Michael Banck
Hi, Am Dienstag, den 19.03.2019, 11:48 +0100 schrieb Fabien COELHO: > Moving the controlfile looks like an effective way to prevent any > concurrent start, as the fs operation is probably atomic and especially if > external tools uses the same trick. However this is not the case yet, eg >

Re: Fwd: Add tablespace tap test to pg_rewind

2019-03-19 Thread Shaoqi Bai
On Tue, Mar 12, 2019 at 10:27 AM Michael Paquier wrote: > On Mon, Mar 11, 2019 at 07:49:11PM +0800, Shaoqi Bai wrote: > > Thanks, will work on it as you suggested > > Add pg_basebackup --T olddir=newdir to support check the consistency of a > > tablespace created before promotion > > Add

Re: Contribution to Perldoc for TestLib module in Postgres

2019-03-19 Thread Michael Paquier
On Tue, Mar 19, 2019 at 09:05:29AM -0300, Alvaro Herrera wrote: > Yes, it is, please do. +1. -- Michael signature.asc Description: PGP signature

Re: Offline enabling/disabling of data checksums

2019-03-19 Thread Michael Paquier
On Tue, Mar 19, 2019 at 11:48:25AM +0100, Fabien COELHO wrote: > Moving the controlfile looks like an effective way to prevent any concurrent > start, as the fs operation is probably atomic and especially if external > tools uses the same trick. However this is not the case yet, eg > "pg_resetwal"

Re: partitioned tables referenced by FKs

2019-03-19 Thread Alvaro Herrera
On 2019-Mar-19, Amit Langote wrote: > On Tue, Mar 19, 2019 at 8:49 PM Alvaro Herrera > wrote: > > On 2019-Mar-19, Amit Langote wrote: > > > > > Will it suffice or be OK if we skipped invoking the pre-drop callback for > > > objects that are being "indirectly" dropped? I came up with the

Re: partitioned tables referenced by FKs

2019-03-19 Thread Amit Langote
On Tue, Mar 19, 2019 at 8:49 PM Alvaro Herrera wrote: > On 2019-Mar-19, Amit Langote wrote: > > > Will it suffice or be OK if we skipped invoking the pre-drop callback for > > objects that are being "indirectly" dropped? I came up with the attached > > patch to resolve this problem, if that idea

  1   2   >