Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-04-14 Thread Dilip Kumar
On Tue, Apr 14, 2020 at 9:14 PM Erik Rijkers wrote: > > On 2020-04-14 12:10, Dilip Kumar wrote: > > > v14-0001-Immediately-WAL-log-assignments.patch + > > v14-0002-Issue-individual-invalidations-with.patch + > >

Re: Race condition in SyncRepGetSyncStandbysPriority

2020-04-14 Thread Masahiko Sawada
On Tue, 14 Apr 2020 at 18:35, Kyotaro Horiguchi wrote: > > At Tue, 14 Apr 2020 13:06:14 +0900, Masahiko Sawada > wrote in > > On Tue, 14 Apr 2020 at 10:34, Tom Lane wrote: > > > > > > Kyotaro Horiguchi writes: > > > > At Sat, 11 Apr 2020 18:30:30 -0400, Tom Lane wrote > > > > in > > > >>

Re: documenting the backup manifest file format

2020-04-14 Thread Fujii Masao
On 2020/04/14 2:40, Robert Haas wrote: On Fri, Mar 27, 2020 at 4:32 PM Andres Freund wrote: I don't like having a file format that's intended to be used by external tools too that's undocumented except for code that assembles it in a piecemeal fashion. Do you mean in a follow-on patch this

Re: Properly mark NULL returns in numeric aggregates

2020-04-14 Thread David Rowley
On Wed, 15 Apr 2020 at 03:41, Tom Lane wrote: > > David Rowley writes: > > For testing, can't we just have an Assert() in > > advance_transition_function that verifies isnull matches the > > nullability of the return value for INTERNAL returning transfns? i.e, > > the attached > > FTR, I do not

Re: Vacuum o/p with (full 1, parallel 0) option throwing an error

2020-04-14 Thread Amit Kapila
On Wed, Apr 15, 2020 at 9:03 AM Justin Pryzby wrote: > > On Wed, Apr 15, 2020 at 08:54:17AM +0530, Amit Kapila wrote: > > On Tue, Apr 14, 2020 at 8:55 AM Amit Kapila wrote: > > > > > > On Tue, Apr 14, 2020 at 7:52 AM Michael Paquier > > > wrote: > > > > > > > > > > > -VACUUM (PARALLEL 1) tmp;

Re: Vacuum o/p with (full 1, parallel 0) option throwing an error

2020-04-14 Thread Justin Pryzby
On Wed, Apr 15, 2020 at 08:54:17AM +0530, Amit Kapila wrote: > On Tue, Apr 14, 2020 at 8:55 AM Amit Kapila wrote: > > > > On Tue, Apr 14, 2020 at 7:52 AM Michael Paquier wrote: > > > > > > > > -VACUUM (PARALLEL 1) tmp; -- disables parallel vacuum option > > > +VACUUM (PARALLEL 1) tmp; --

Re: Vacuum o/p with (full 1, parallel 0) option throwing an error

2020-04-14 Thread Amit Kapila
On Tue, Apr 14, 2020 at 8:55 AM Amit Kapila wrote: > > On Tue, Apr 14, 2020 at 7:52 AM Michael Paquier wrote: > > > > > -VACUUM (PARALLEL 1) tmp; -- disables parallel vacuum option > > +VACUUM (PARALLEL 1) tmp; -- parallel vacuum disabled for temp tables > > WARNING: disabling parallel option

Re: Race condition in SyncRepGetSyncStandbysPriority

2020-04-14 Thread Noah Misch
On Tue, Apr 14, 2020 at 04:32:40PM -0400, Tom Lane wrote: > I wrote: > > It doesn't seem to me to be that hard to implement the desired > > semantics for synchronous_standby_names with inconsistent info. > > In FIRST mode you basically just need to take the N smallest > > priorities you see in the

Re: [PATCH] Keeps tracking the uniqueness with UniqueKey

2020-04-14 Thread David Rowley
On Wed, 15 Apr 2020 at 12:19, Andy Fan wrote: > >> 2. I think 0002 is overly restrictive in its demands that >> parse->hasAggs must be false. We should be able to just use a Group >> Aggregate with unsorted input when the input_rel is unique on the >> GROUP BY clause. This will save on hashing

Re: Race condition in SyncRepGetSyncStandbysPriority

2020-04-14 Thread Kyotaro Horiguchi
At Tue, 14 Apr 2020 16:32:40 -0400, Tom Lane wrote in > I wrote: > > It doesn't seem to me to be that hard to implement the desired > > semantics for synchronous_standby_names with inconsistent info. > > In FIRST mode you basically just need to take the N smallest > > priorities you see in the

Re: snapshot too old issues, first around wraparound and then more.

2020-04-14 Thread Thomas Munro
On Mon, Apr 13, 2020 at 2:58 PM Thomas Munro wrote: > On Fri, Apr 3, 2020 at 2:22 PM Peter Geoghegan wrote: > > I think that it's worth considering whether or not there are a > > significant number of "snapshot too old" users that rarely or never > > rely on old snapshots used by new queries.

Re: backup manifests

2020-04-14 Thread Fujii Masao
On 2020/04/14 0:15, Robert Haas wrote: On Sun, Apr 12, 2020 at 10:09 PM Fujii Masao wrote: I found other minor issues. I think these are all correct fixes. Thanks for the post-commit review, and sorry for this mistakes. Thanks for the review, Michael and Robert. Pushed the patches!

Re: where should I stick that backup?

2020-04-14 Thread Andres Freund
Hi, On 2020-04-14 11:38:03 -0400, Robert Haas wrote: > I'm fairly deeply uncomfortable with what Andres is proposing. I see > that it's very powerful, and can do a lot of things, and that if > you're building something that does sophisticated things with storage, > you probably want an API like

remove_useless_groupby_columns does not need to record constraint dependencies

2020-04-14 Thread David Rowley
Hi, Over in [1], Tom and I had a discussion in response to some confusion about why remove_useless_groupby_columns() goes to the trouble of recording a dependency on the PRIMARY KEY constraint when removing surplus columns from the GROUP BY clause. The outcome was that we don't need to do this

Re: Race condition in SyncRepGetSyncStandbysPriority

2020-04-14 Thread Kyotaro Horiguchi
At Tue, 14 Apr 2020 09:52:42 -0400, Tom Lane wrote in > Kyotaro Horiguchi writes: > > SyncRepGetSyncStandbysPriority() is runing holding SyncRepLock so > > sync_standby_priority of any walsender can be changed while the > > function is scanning welsenders. The issue is we already have > >

Re: Perl modules for testing/viewing/corrupting/repairing your heap files

2020-04-14 Thread Peter Geoghegan
On Wed, Apr 8, 2020 at 3:51 PM Mark Dilger wrote: > Recently, as part of testing something else, I had need of a tool to create > surgically precise corruption within heap pages. I wanted to make the > corruption from within TAP tests, so I wrote the tool as a set of perl > modules. There is

Re: Perl modules for testing/viewing/corrupting/repairing your heap files

2020-04-14 Thread Mark Dilger
Not having received any feedback on this, I've dusted the modules off for submission as-is. v1-0001-Adding-HeapFile-related-perl-modules.patch Description: Binary data — Mark Dilger EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: [PATCH] Keeps tracking the uniqueness with UniqueKey

2020-04-14 Thread Andy Fan
Hi David: Thanks for your time. > 1. Out of date comment in join.sql > > -- join removal is not possible when the GROUP BY contains a column that is > -- not in the join condition. (Note: as of 9.6, we notice that b.id is a > -- primary key and so drop b.c_id from the GROUP BY of the resulting

Re: wrong relkind error messages

2020-04-14 Thread Tom Lane
Alvaro Herrera writes: > On 2020-Apr-13, Robert Haas wrote: >> + ereport(ERROR, >> + (errcode(ERRCODE_WRONG_OBJECT_TYPE), >> + errmsg("action cannot be performed on relation \"%s\"", >> + RelationGetRelationName(rel)), >> >> Super-vague. > Maybe, but note that the patch proposed to replace this

Re: pg_basebackup, manifests and backends older than ~12

2020-04-14 Thread Tom Lane
Michael Paquier writes: > On Mon, Apr 13, 2020 at 07:04:20PM -0400, Alvaro Herrera wrote: >> I agree, I think forcing users to specify --no-manifest when run on old >> servers will cause users to write bad scripts; I vote for silently >> disabling checksums. > Okay, thanks. Are there any other

Re: wrong relkind error messages

2020-04-14 Thread Alvaro Herrera
On 2020-Apr-13, Robert Haas wrote: > + ereport(ERROR, > + (errcode(ERRCODE_WRONG_OBJECT_TYPE), > + errmsg("action cannot be performed on relation \"%s\"", > + RelationGetRelationName(rel)), > > Super-vague. Maybe, but note that the patch proposed to replace this current error message: ERROR:

Re: pg_basebackup, manifests and backends older than ~12

2020-04-14 Thread Michael Paquier
On Mon, Apr 13, 2020 at 07:04:20PM -0400, Alvaro Herrera wrote: > I agree, I think forcing users to specify --no-manifest when run on old > servers will cause users to write bad scripts; I vote for silently > disabling checksums. Okay, thanks. Are there any other opinions? -- Michael

Re: pg_basebackup, manifests and backends older than ~12

2020-04-14 Thread Michael Paquier
On Tue, Apr 14, 2020 at 03:13:39PM -0400, Robert Haas wrote: > Feel free to go ahead. Thanks, let's do it then. If you have any objections about any parts of the patch, of course please feel free. -- Michael signature.asc Description: PGP signature

Re: wrong relkind error messages

2020-04-14 Thread Alvaro Herrera
On 2020-Apr-14, Michael Paquier wrote: > On Mon, Apr 13, 2020 at 11:13:15AM -0400, Tom Lane wrote: > > ERROR: cannot define statistics for relation "ti" > > DETAIL: This operation is not supported for indexes. > > > > which still leaves implicit that "ti" is an index, but probably that's > >

Re: Poll: are people okay with function/operator table redesign?

2020-04-14 Thread David G. Johnston
On Mon, Apr 13, 2020 at 10:13 AM Tom Lane wrote: > As discussed in the thread at [1], I've been working on redesigning > the tables we use to present SQL functions and operators. The > first installment of that is now up; see tables 9.30 and 9.31 at > >

Re: ERROR: could not determine which collation to use for string comparison

2020-04-14 Thread Tom Lane
Andreas Joseph Krogh writes: > Guys; This errors out with: > ERROR: could not determine which collation to use for string comparison > HINT: Use the COLLATE clause to set the collation explicitly. Fixed, thanks for the report. regards, tom lane

Re: WIP/PoC for parallel backup

2020-04-14 Thread Robert Haas
On Tue, Apr 14, 2020 at 10:37 AM Asif Rehman wrote: > I forgot to make a check for no-manifest. Fixed. Attached is the updated > patch. +typedef struct +{ ... +} BackupFile; + +typedef struct +{ ... +} BackupState; These structures need comments. +list_wal_files_opt_list: +

Re: cleaning perl code

2020-04-14 Thread Alvaro Herrera
On 2020-Apr-14, Andrew Dunstan wrote: > One of the things that's a bit sad is that perlcritic doesn't generally > let you apply policies to a given set of files or files matching some > pattern. It would be nice, for instance, to be able to apply some > additional standards to strategic library

Re: documenting the backup manifest file format

2020-04-14 Thread David Steele
On 4/14/20 4:11 PM, Alvaro Herrera wrote: On 2020-Apr-14, David Steele wrote: Happily ISO-8601 is always UTC. Uh, it is not -- https://en.wikipedia.org/wiki/ISO_8601#Time_zone_designators Whoops, you are correct. I've just never seen non-UTC in the wild yet. -- -David da...@pgmasters.net

Re: documenting the backup manifest file format

2020-04-14 Thread Andrew Dunstan
On 4/14/20 4:09 PM, Alvaro Herrera wrote: > On 2020-Apr-14, Andrew Dunstan wrote: > >> OK, but I think if we're putting a timestamp string in ISO-8601 format >> in the manifest it should be in UTC / Zulu time, precisely to avoid >> these issues. If that's too much trouble then yes an epoch time

Re: Race condition in SyncRepGetSyncStandbysPriority

2020-04-14 Thread Tom Lane
I wrote: > It doesn't seem to me to be that hard to implement the desired > semantics for synchronous_standby_names with inconsistent info. > In FIRST mode you basically just need to take the N smallest > priorities you see in the array, but without assuming there are no > duplicates or holes. It

Re: documenting the backup manifest file format

2020-04-14 Thread Alvaro Herrera
On 2020-Apr-14, David Steele wrote: > Happily ISO-8601 is always UTC. Uh, it is not -- https://en.wikipedia.org/wiki/ISO_8601#Time_zone_designators -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: documenting the backup manifest file format

2020-04-14 Thread Alvaro Herrera
On 2020-Apr-14, Andrew Dunstan wrote: > OK, but I think if we're putting a timestamp string in ISO-8601 format > in the manifest it should be in UTC / Zulu time, precisely to avoid > these issues. If that's too much trouble then yes an epoch time will > probably do. The timestamp is always

Re: documenting the backup manifest file format

2020-04-14 Thread David Steele
On 4/14/20 3:55 PM, Andrew Dunstan wrote: OK, but I think if we're putting a timestamp string in ISO-8601 format in the manifest it should be in UTC / Zulu time, precisely to avoid these issues. If that's too much trouble then yes an epoch time will probably do. Happily ISO-8601 is always

Re: documenting the backup manifest file format

2020-04-14 Thread Andrew Dunstan
On 4/14/20 3:19 PM, David Steele wrote: > On 4/14/20 3:03 PM, Andrew Dunstan wrote: >> >> On 4/14/20 1:33 PM, David Steele wrote: >>> On 4/14/20 1:27 PM, Alvaro Herrera wrote: On 2020-Apr-14, David Steele wrote: > On 4/14/20 12:56 PM, Robert Haas wrote: > >> Hmm, did David

Re: Parallel copy

2020-04-14 Thread Kuntal Ghosh
Hello, I was going through some literatures on parsing CSV files in a fully parallelized way and found (from [1]) an interesting approach implemented in the open-source project ParaText[2]. The algorithm follows a two-phase approach: the first pass identifies the adjusted chunks in parallel by

Re: documenting the backup manifest file format

2020-04-14 Thread David Steele
On 4/14/20 3:03 PM, Andrew Dunstan wrote: On 4/14/20 1:33 PM, David Steele wrote: On 4/14/20 1:27 PM, Alvaro Herrera wrote: On 2020-Apr-14, David Steele wrote: On 4/14/20 12:56 PM, Robert Haas wrote: Hmm, did David suggest that before? I don't recall for sure. I think he had some

Re: Do we need to handle orphaned prepared transactions in the server?

2020-04-14 Thread Robert Haas
On Wed, Feb 19, 2020 at 10:05 AM Hamid Akhtar wrote: > Attached is version 1 of POC patch for notifying of orphaned > prepared transactions via warnings emitted to a client > application and/or log file. It applies to PostgreSQL branch > "master" on top of "e2e02191" commit. I think this is a

Re: pg_basebackup, manifests and backends older than ~12

2020-04-14 Thread Robert Haas
On Mon, Apr 13, 2020 at 8:23 PM Michael Paquier wrote: > On Mon, Apr 13, 2020 at 07:55:07PM -0400, Robert Haas wrote: > > Oh, hmm. Maybe I'm getting confused with a previous version of the > > patch that behaved differently. > > No problem. If you prefer keeping this part of the code, that's

Re: Corruption during WAL replay

2020-04-14 Thread Teja Mupparti
Thanks Kyotaro and Masahiko for the feedback. I think there is a consensus on the critical-section around truncate, but I just want to emphasize the need for reversing the order of the dropping the buffers and the truncation. Repro details (when full page write = off) 1) Page on disk

Re: documenting the backup manifest file format

2020-04-14 Thread Andrew Dunstan
On 4/14/20 1:33 PM, David Steele wrote: > On 4/14/20 1:27 PM, Alvaro Herrera wrote: >> On 2020-Apr-14, David Steele wrote: >> >>> On 4/14/20 12:56 PM, Robert Haas wrote: >>> Hmm, did David suggest that before? I don't recall for sure. I think he had some suggestion, but I'm not sure if

Re: Do we need to handle orphaned prepared transactions in the server?

2020-04-14 Thread Hamid Akhtar
Thank you so much Bruce and Sawada. Bruce: I'll update the documentation with more details for max_age_prepared_xacts Sawada: You have raised 4 very valid points. Here are my thoughts. (1) I think your concern is that if we can reduce the need of 2 GUCs to one. The purpose of having 2

Re: index paths and enable_indexscan

2020-04-14 Thread Robert Haas
On Tue, Apr 14, 2020 at 10:21 AM Tom Lane wrote: > Awhile back I'd looked into getting rid of disable_cost altogether > by dint of not generating disabled paths. It's harder than it > sounds. We could perhaps change this particular case, but it's > not clear that there's any real benefit of

Re: documenting the backup manifest file format

2020-04-14 Thread David Steele
On 4/14/20 1:27 PM, Alvaro Herrera wrote: On 2020-Apr-14, David Steele wrote: On 4/14/20 12:56 PM, Robert Haas wrote: Hmm, did David suggest that before? I don't recall for sure. I think he had some suggestion, but I'm not sure if it was the same one. "I'm also partial to using epoch time

Re: documenting the backup manifest file format

2020-04-14 Thread Alvaro Herrera
On 2020-Apr-14, David Steele wrote: > On 4/14/20 12:56 PM, Robert Haas wrote: > > > Hmm, did David suggest that before? I don't recall for sure. I think > > he had some suggestion, but I'm not sure if it was the same one. > > "I'm also partial to using epoch time in the manifest because it is >

Re: sqlsmith crash incremental sort

2020-04-14 Thread James Coleman
On Sun, Apr 12, 2020 at 8:09 PM Tomas Vondra wrote: > > On Sun, Apr 12, 2020 at 12:44:45AM +0200, Tomas Vondra wrote: > >Hi, > > > >I've looked into this a bit, and at first I thought that maybe the > >issue is in how cost_incremental_sort picks the EC members. It simply > >does this: > > > >

Re: documenting the backup manifest file format

2020-04-14 Thread David Steele
On 4/14/20 12:56 PM, Robert Haas wrote: On Mon, Apr 13, 2020 at 5:43 PM Alvaro Herrera wrote: Yeah, I guess I'm just saying that it feels brittle to have a file format that's supposed to be good for data exchange and then make it itself depend on representation details such as the order that

Re: documenting the backup manifest file format

2020-04-14 Thread Robert Haas
On Mon, Apr 13, 2020 at 5:43 PM Alvaro Herrera wrote: > Yeah, I guess I'm just saying that it feels brittle to have a file > format that's supposed to be good for data exchange and then make it > itself depend on representation details such as the order that fields > appear in, the letter case,

ERROR: could not determine which collation to use for string comparison

2020-04-14 Thread Andreas Joseph Krogh
Guys; This errors out with: ERROR: could not determine which collation to use for string comparison HINT: Use the COLLATE clause to set the collation explicitly. The database is init'ed with: initdb -D $PGDATA -E utf8 --locale=nb_NO.UTF-8 13-dev HEAD as of

Re: [BUG] non archived WAL removed during production crash recovery

2020-04-14 Thread Jehan-Guillaume de Rorthais
On Fri, 10 Apr 2020 11:00:31 +0900 (JST) Kyotaro Horiguchi wrote: [...] > > > but the mistake here is it thinks that inRecovery represents whether it is > > > running as a standby or not, but actually it is true on primary during > > > crash recovery. > > > > Indeed. > > > > > On the other

Re: Poll: are people okay with function/operator table redesign?

2020-04-14 Thread Tom Lane
Robert Haas writes: > At any rate, if the price of more clarity and more examples is that > the tables become three times as long and harder to read, I am > somewhat inclined to think that the cure is worse than the disease. I > can readily see how something like table 9.10 (Other String

Re: cleaning perl code

2020-04-14 Thread Andrew Dunstan
On 4/13/20 12:47 PM, Andrew Dunstan wrote: > > OK, I've committed all that stuff. I think that takes care of the > non-controversial part of what I proposed :-) > > OK, it seems there is a majority of people commenting in this thread in favor of not doing more except to reverse the policy of

Re: Poll: are people okay with function/operator table redesign?

2020-04-14 Thread Robert Haas
On Tue, Apr 14, 2020 at 11:26 AM Tom Lane wrote: > Well, no it isn't. The main nit I would pick in that claim is that > it's far from obvious that the three examples of float8 * interval > are all talking about the same operator; in fact, a reader would > be very likely to draw the false

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-04-14 Thread Erik Rijkers
On 2020-04-14 12:10, Dilip Kumar wrote: v14-0001-Immediately-WAL-log-assignments.patch + v14-0002-Issue-individual-invalidations-with.patch + v14-0003-Extend-the-output-plugin-API-with-stream-methods.patch+

PG compilation error with Visual Studio 2015/2017/2019

2020-04-14 Thread Ranier Vilela
>>I m still working on testing this patch. If anyone has Idea please suggest. I still see problems with this patch. 1. Variable loct have redundant initialization, it would be enough to declare so: _locale_t loct; 2. Style white space in variable rc declaration. 3. Style variable cp_index can be

Re: Properly mark NULL returns in numeric aggregates

2020-04-14 Thread Tom Lane
David Rowley writes: > For testing, can't we just have an Assert() in > advance_transition_function that verifies isnull matches the > nullability of the return value for INTERNAL returning transfns? i.e, > the attached FTR, I do not like this Assert one bit. nodeAgg.c has NO business inquiring

Re: where should I stick that backup?

2020-04-14 Thread Robert Haas
On Tue, Apr 14, 2020 at 11:08 AM Stephen Frost wrote: > Wouldn't it make sense to, given that we have some idea of what we want > it to eventually look like, to make progress in that direction though? Well, yes. :-) > That is- I tend to agree with Andres that having this supported > server-side

Re: Properly mark NULL returns in numeric aggregates

2020-04-14 Thread Jesse Zhang
Hi David, On Mon, Apr 13, 2020 at 10:46 PM David Rowley wrote: > > On Tue, 14 Apr 2020 at 06:14, Tom Lane wrote: > > Yeah, they're relying exactly on the assumption that nodeAgg is not > > going to try to copy a value declared "internal", and therefore they > > can be loosey-goosey about

Re: Poll: are people okay with function/operator table redesign?

2020-04-14 Thread Tom Lane
Robert Haas writes: > The v12 version of the "Table 9.30. Date/Time Operators" is not that > wide, and is really quite clear. Well, no it isn't. The main nit I would pick in that claim is that it's far from obvious that the three examples of float8 * interval are all talking about the same

Re: where should I stick that backup?

2020-04-14 Thread Stephen Frost
Greetings, * Robert Haas (robertmh...@gmail.com) wrote: > On Sun, Apr 12, 2020 at 8:27 PM Andres Freund wrote: > > I really think we want the option to eventually do this server-side. And > > I don't quite see it as viable to go for an API that allows to specify > > shell fragments that are

Re: Poll: are people okay with function/operator table redesign?

2020-04-14 Thread Robert Haas
On Mon, Apr 13, 2020 at 4:29 PM Tom Lane wrote: > I wouldn't be averse to dropping the text descriptions for operators > in places where they seem obvious ... but who decides what is obvious? Well, we do. We're smart, right? I don't think it's a good idea to add clutter to table A just because

Re: Poll: are people okay with function/operator table redesign?

2020-04-14 Thread Tom Lane
Andreas Karlsson writes: > That said, I agree with that quite many of our tables right now are > ugly, but I prefer ugly to hard to read. For me the mix of having every > third row split into two fields makes the tables very hard to read. I > have a hard time seeing which rows belong to which

Re: Poll: are people okay with function/operator table redesign?

2020-04-14 Thread Andreas Karlsson
On 4/14/20 4:29 PM, Tom Lane wrote: Andreas Karlsson writes: For the old tables I can at least just make the browser window ridiculously wide ro read them. A large part of the point here is to make the tables usable when you don't have that option, as for example in PDF output. Even with a

Re: index paths and enable_indexscan

2020-04-14 Thread Tom Lane
Amit Langote writes: > I was really thinking of this in terms of planner effort, which for > creating an index path is more than creating sequential path, although > sure the payoff can be great. That is, I want the planner to avoid > creating index paths *to save cycles*, but see no way of

Re: Poll: are people okay with function/operator table redesign?

2020-04-14 Thread Tom Lane
Andreas Karlsson writes: > For the old tables I can at least just make the browser window > ridiculously wide ro read them. A large part of the point here is to make the tables usable when you don't have that option, as for example in PDF output. Even with a wide window, though, some of our

Re: index paths and enable_indexscan

2020-04-14 Thread Tom Lane
Amit Langote writes: > I am saying that instead of building index path with disabled cost, > just don't build it at all. A base rel will always have a sequetial > path, even though with disabled cost if enable_seqscan = off. Awhile back I'd looked into getting rid of disable_cost altogether by

Re: index paths and enable_indexscan

2020-04-14 Thread Amit Langote
On Tue, Apr 14, 2020 at 6:12 PM Andy Fan wrote: > On Tue, Apr 14, 2020 at 4:58 PM Amit Langote wrote: >> I am saying that instead of building index path with disabled cost, >> just don't build it at all. A base rel will always have a sequetial >> path, even though with disabled cost if

Re: where should I stick that backup?

2020-04-14 Thread Robert Haas
On Sun, Apr 12, 2020 at 8:27 PM Andres Freund wrote: > > That's quite appealing. One downside - IMHO significant - is that you > > have to have a separate process to do *anything*. If you want to add a > > filter that just logs everything it's asked to do, for example, you've > > gotta have a

Re: Poll: are people okay with function/operator table redesign?

2020-04-14 Thread Andreas Karlsson
On 4/13/20 7:13 PM, Tom Lane wrote: As discussed in the thread at [1], I've been working on redesigning the tables we use to present SQL functions and operators. The first installment of that is now up; see tables 9.30 and 9.31 at https://www.postgresql.org/docs/devel/functions-datetime.html

Re: Race condition in SyncRepGetSyncStandbysPriority

2020-04-14 Thread Tom Lane
Kyotaro Horiguchi writes: > SyncRepGetSyncStandbysPriority() is runing holding SyncRepLock so > sync_standby_priority of any walsender can be changed while the > function is scanning welsenders. The issue is we already have > inconsistent walsender information before we enter the function. Thus

While restoring -getting error if dump contain sql statements generated from generated.sql file

2020-04-14 Thread tushar
Hi , We have a sql file  called 'generated.sql' under src/test/regress/sql folder . if we run this file on psql , take the dump and try to restore it on another db we are getting error like - psql:/tmp/x:434: ERROR:  column "b" of relation "gtest1_1" is a generated column psql:/tmp/x:441:

Re: WIP/PoC for parallel backup

2020-04-14 Thread Kashif Zeeshan
Hi Asif Getting the following error on Parallel backup when --no-manifest option is used. [edb@localhost bin]$ [edb@localhost bin]$ [edb@localhost bin]$ ./pg_basebackup -v -j 5 -D /home/edb/Desktop/backup/ --no-manifest pg_basebackup: initiating base backup, waiting for checkpoint to complete

Re: Poll: are people okay with function/operator table redesign?

2020-04-14 Thread Andrew Dunstan
On 4/13/20 7:55 PM, Tom Lane wrote: > "Jonathan S. Katz" writes: >> On 4/13/20 7:02 PM, Jonathan S. Katz wrote: >>> Perhaps a counterproposal: We eliminate the content in the leftmost >>> "function column, but leave that there to allow the function name / >>> signature to span the full 3

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-04-14 Thread Dilip Kumar
On Tue, Apr 14, 2020 at 3:57 PM Amit Kapila wrote: > > On Tue, Apr 14, 2020 at 3:41 PM Dilip Kumar wrote: > > > > > > > > > > > @@ -281,6 +281,24 @@ DecodeXactOp(LogicalDecodingContext *ctx, > > > > XLogRecordBuffer *buf) > > > > } > > > > case XLOG_XACT_ASSIGNMENT: > > > > break; > > > >

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-04-14 Thread Amit Kapila
On Tue, Apr 14, 2020 at 3:41 PM Dilip Kumar wrote: > > > > > > > @@ -281,6 +281,24 @@ DecodeXactOp(LogicalDecodingContext *ctx, > > > XLogRecordBuffer *buf) > > > } > > > case XLOG_XACT_ASSIGNMENT: > > > break; > > > + case XLOG_XACT_INVALIDATIONS: > > > + { > > > + TransactionId xid; > > >

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-04-14 Thread Dilip Kumar
On Tue, Apr 14, 2020 at 2:57 PM Amit Kapila wrote: > > On Mon, Apr 13, 2020 at 6:12 PM Kuntal Ghosh > wrote: > > > > On Mon, Apr 13, 2020 at 5:20 PM Dilip Kumar wrote: > > > On Mon, Apr 13, 2020 at 4:14 PM Kuntal Ghosh > > > wrote: > > > > > > > > +#define SizeOfTransactionId

Re: [HACKERS] make async slave to wait for lsn to be replayed

2020-04-14 Thread Anna Akenteva
On 2020-04-11 00:44, Andres Freund wrote: I think there's also some advantages of having it in a single statement for poolers. If a pooler analyzes BEGIN AFTER 'xx/xx' it could e.g. redirect the transaction to a node that's caught up far enough, instead of blocking. But that can't work even

Re: Display of buffers for planning time show nothing for second run

2020-04-14 Thread Pavel Stehule
út 14. 4. 2020 v 11:35 odesílatel Julien Rouhaud napsal: > On Tue, Apr 14, 2020 at 11:25 AM Pavel Stehule > wrote: > > > > út 14. 4. 2020 v 10:40 odesílatel Amit Langote > napsal: > >> > >> On Tue, Apr 14, 2020 at 5:27 PM Julien Rouhaud > wrote: > >> > On Tue, Apr 14, 2020 at 10:18 AM Pavel

Re: Race condition in SyncRepGetSyncStandbysPriority

2020-04-14 Thread Kyotaro Horiguchi
At Tue, 14 Apr 2020 13:06:14 +0900, Masahiko Sawada wrote in > On Tue, 14 Apr 2020 at 10:34, Tom Lane wrote: > > > > Kyotaro Horiguchi writes: > > > At Sat, 11 Apr 2020 18:30:30 -0400, Tom Lane wrote in > > >> What I think we should do about this is, essentially, to get rid of > > >>

Re: Display of buffers for planning time show nothing for second run

2020-04-14 Thread Julien Rouhaud
On Tue, Apr 14, 2020 at 11:25 AM Pavel Stehule wrote: > > út 14. 4. 2020 v 10:40 odesílatel Amit Langote > napsal: >> >> On Tue, Apr 14, 2020 at 5:27 PM Julien Rouhaud wrote: >> > On Tue, Apr 14, 2020 at 10:18 AM Pavel Stehule >> > wrote: >> > > For second run I get >> > > >> > > postgres=#

Re: Display of buffers for planning time show nothing for second run

2020-04-14 Thread Pavel Stehule
út 14. 4. 2020 v 10:49 odesílatel Julien Rouhaud napsal: > On Tue, Apr 14, 2020 at 10:36 AM Pavel Stehule > wrote: > > > > út 14. 4. 2020 v 10:27 odesílatel Julien Rouhaud > napsal: > >> > >> Hi, > >> > >> On Tue, Apr 14, 2020 at 10:18 AM Pavel Stehule > wrote: > >> > > >> > Hi > >> > > >> >

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-04-14 Thread Amit Kapila
On Mon, Apr 13, 2020 at 6:12 PM Kuntal Ghosh wrote: > > On Mon, Apr 13, 2020 at 5:20 PM Dilip Kumar wrote: > > On Mon, Apr 13, 2020 at 4:14 PM Kuntal Ghosh > > wrote: > > > > > > +#define SizeOfTransactionId (sizeof(TransactionId) + sizeof(char)) > > > This looks wrong. We should change the

Re: Display of buffers for planning time show nothing for second run

2020-04-14 Thread Pavel Stehule
út 14. 4. 2020 v 10:40 odesílatel Amit Langote napsal: > On Tue, Apr 14, 2020 at 5:27 PM Julien Rouhaud wrote: > > On Tue, Apr 14, 2020 at 10:18 AM Pavel Stehule > wrote: > > > For second run I get > > > > > > postgres=# EXPLAIN (BUFFERS, ANALYZE) SELECT * FROM obce WHERE > okres_id =

Re: index paths and enable_indexscan

2020-04-14 Thread Andy Fan
On Tue, Apr 14, 2020 at 5:12 PM Andy Fan wrote: > > > On Tue, Apr 14, 2020 at 4:58 PM Amit Langote > wrote: > >> On Tue, Apr 14, 2020 at 5:29 PM Andy Fan >> wrote: >> > On Tue, Apr 14, 2020 at 3:40 PM Amit Langote >> wrote: >> >> On Tue, Apr 14, 2020 at 4:13 PM Richard Guo >> wrote: >> >> >

Re: Index Skip Scan

2020-04-14 Thread David Rowley
On Wed, 8 Apr 2020 at 07:40, Dmitry Dolgov <9erthali...@gmail.com> wrote: > Other than that to summarize current open points for future readers > (this thread somehow became quite big): > > * Making UniqueKeys usage more generic to allow using skip scan for more > use cases (hopefully it was

Re: index paths and enable_indexscan

2020-04-14 Thread Andy Fan
On Tue, Apr 14, 2020 at 4:58 PM Amit Langote wrote: > On Tue, Apr 14, 2020 at 5:29 PM Andy Fan wrote: > > On Tue, Apr 14, 2020 at 3:40 PM Amit Langote > wrote: > >> On Tue, Apr 14, 2020 at 4:13 PM Richard Guo > wrote: > >> > On Tue, Apr 14, 2020 at 2:44 PM Amit Langote > wrote: > >> >> Maybe

Re: [PATCH] Keeps tracking the uniqueness with UniqueKey

2020-04-14 Thread David Rowley
On Fri, 3 Apr 2020 at 15:18, Andy Fan wrote: > All above 4 item Done. Just to explain my view on this going forward for PG14. I do plan to do a more thorough review of this soon. I wasn't so keen on pursuing this for PG13 as the skip scans patch [1] needs to use the same infrastructure this

Re: Lexer issues

2020-04-14 Thread Julien Rouhaud
Hello, On Mon, Apr 13, 2020 at 4:04 PM Patrick REED wrote: > > I am experimenting with postgres and am wondering if there is any tutorial on > how to properly add a new command to postgres. > > I want to add a new constraint on "CREATE ROLE" that requires an integer, it > has an identifier

Re: index paths and enable_indexscan

2020-04-14 Thread Amit Langote
On Tue, Apr 14, 2020 at 5:29 PM Andy Fan wrote: > On Tue, Apr 14, 2020 at 3:40 PM Amit Langote wrote: >> On Tue, Apr 14, 2020 at 4:13 PM Richard Guo wrote: >> > On Tue, Apr 14, 2020 at 2:44 PM Amit Langote >> > wrote: >> >> Maybe I am missing something obvious, but is it intentional that >>

Re: Display of buffers for planning time show nothing for second run

2020-04-14 Thread Julien Rouhaud
On Tue, Apr 14, 2020 at 10:40 AM Amit Langote wrote: > > On Tue, Apr 14, 2020 at 5:27 PM Julien Rouhaud wrote: > > On Tue, Apr 14, 2020 at 10:18 AM Pavel Stehule > > wrote: > > > For second run I get > > > > > > postgres=# EXPLAIN (BUFFERS, ANALYZE) SELECT * FROM obce WHERE okres_id = > > >

Re: Display of buffers for planning time show nothing for second run

2020-04-14 Thread Julien Rouhaud
On Tue, Apr 14, 2020 at 10:36 AM Pavel Stehule wrote: > > út 14. 4. 2020 v 10:27 odesílatel Julien Rouhaud napsal: >> >> Hi, >> >> On Tue, Apr 14, 2020 at 10:18 AM Pavel Stehule >> wrote: >> > >> > Hi >> > >> > I am testing some features from Postgres 13, and I am not sure if I >> >

Re: Display of buffers for planning time show nothing for second run

2020-04-14 Thread Amit Langote
On Tue, Apr 14, 2020 at 5:27 PM Julien Rouhaud wrote: > On Tue, Apr 14, 2020 at 10:18 AM Pavel Stehule > wrote: > > For second run I get > > > > postgres=# EXPLAIN (BUFFERS, ANALYZE) SELECT * FROM obce WHERE okres_id = > > 'CZ0201'; > >

Re: Display of buffers for planning time show nothing for second run

2020-04-14 Thread Pavel Stehule
út 14. 4. 2020 v 10:27 odesílatel Julien Rouhaud napsal: > Hi, > > On Tue, Apr 14, 2020 at 10:18 AM Pavel Stehule > wrote: > > > > Hi > > > > I am testing some features from Postgres 13, and I am not sure if I > understand well to behave of EXPLAIN(ANALYZE, BUFFERS) > > > > When I run following

Re: index paths and enable_indexscan

2020-04-14 Thread Andy Fan
On Tue, Apr 14, 2020 at 3:40 PM Amit Langote wrote: > On Tue, Apr 14, 2020 at 4:13 PM Richard Guo > wrote: > > On Tue, Apr 14, 2020 at 2:44 PM Amit Langote > wrote: > >> Maybe I am missing something obvious, but is it intentional that > >> enable_indexscan is checked by cost_index(), that is,

Re: Display of buffers for planning time show nothing for second run

2020-04-14 Thread Julien Rouhaud
Hi, On Tue, Apr 14, 2020 at 10:18 AM Pavel Stehule wrote: > > Hi > > I am testing some features from Postgres 13, and I am not sure if I > understand well to behave of EXPLAIN(ANALYZE, BUFFERS) > > When I run following statement first time in session I get > > postgres=# EXPLAIN (BUFFERS,

Display of buffers for planning time show nothing for second run

2020-04-14 Thread Pavel Stehule
Hi I am testing some features from Postgres 13, and I am not sure if I understand well to behave of EXPLAIN(ANALYZE, BUFFERS) When I run following statement first time in session I get postgres=# EXPLAIN (BUFFERS, ANALYZE) SELECT * FROM obce WHERE okres_id = 'CZ0201';

Re: index paths and enable_indexscan

2020-04-14 Thread Amit Langote
On Tue, Apr 14, 2020 at 4:13 PM Richard Guo wrote: > On Tue, Apr 14, 2020 at 2:44 PM Amit Langote wrote: >> Maybe I am missing something obvious, but is it intentional that >> enable_indexscan is checked by cost_index(), that is, *after* creating >> an index path? I was expecting that if

Re: PG compilation error with Visual Studio 2015/2017/2019

2020-04-14 Thread davinder singh
On Fri, Apr 10, 2020 at 5:33 PM Amit Kapila wrote: > > It seems the right direction to use GetLocaleInfoEx as we have already > used it to handle a similar problem (lc_codepage is missing in > _locale_t in higher versions of MSVC (cf commit 0fb54de9aa)) in > chklocale.c. However, I see that we

Re: pgsql: Improve handling of parameter differences in physical replicatio

2020-04-14 Thread Peter Eisentraut
On 2020-03-30 20:10, Andres Freund wrote: Also, shouldn't dynahash be adjusted as well? There's e.g. the following HASH_ENTER path: /* report a generic message */ if (hashp->isshared)

Re: index paths and enable_indexscan

2020-04-14 Thread Richard Guo
On Tue, Apr 14, 2020 at 2:44 PM Amit Langote wrote: > Hi, > > Maybe I am missing something obvious, but is it intentional that > enable_indexscan is checked by cost_index(), that is, *after* creating > an index path? I was expecting that if enable_indexscan is off, then > no index paths would

  1   2   >