Re: Control your disk usage in PG: Introduction to Disk Quota Extension

2020-03-27 Thread Haozhou Wang
Thanks David! We rebased this patch with the newest master. Thank you very much! Regards, Haozhou On Wed, Mar 25, 2020 at 12:00 AM David Steele wrote: > On 12/2/19 1:39 AM, Haozhou Wang wrote: > > > > Thank you very much for your email. I rebased the code with the newest > > master and

Re: [Proposal] Global temporary tables

2020-03-27 Thread tushar
On 3/27/20 10:55 AM, 曾文旌 wrote: Hi Wenjing, This patch(gtt_v21_pg13.patch) is not applicable on PG HEAD, I hope you have prepared the patch on top of some previous commit. Could you please rebase the patch which we can apply on HEAD ? Yes, It looks like the built-in functions are in conflict

Re: Berserk Autovacuum (let's save next Mandrill)

2020-03-27 Thread Laurenz Albe
On Fri, 2020-03-27 at 10:18 +1300, David Rowley wrote: > > > I believe there are enough options to disable insert-only vacuuming for > > > an individual table: > > > > > - Set the threshold to 2147483647. True, that will not work for very > > > large tables, but I think that there are few

Re: INSERT ... OVERRIDING USER VALUE vs GENERATED ALWAYS identity columns

2020-03-27 Thread Peter Eisentraut
We appear to have lost track of this. I have re-read everything and expanded your patch a bit with additional documentation and comments in the tests. -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services From

Re: Some problems of recovery conflict wait events

2020-03-27 Thread Fujii Masao
On 2020/03/04 14:31, Masahiko Sawada wrote: On Wed, 4 Mar 2020 at 13:48, Fujii Masao wrote: On 2020/03/04 13:27, Michael Paquier wrote: On Wed, Mar 04, 2020 at 01:13:19PM +0900, Masahiko Sawada wrote: Yeah, so 0001 patch sets existing wait events to recovery conflict resolution. For

Re: error context for vacuum to include block number

2020-03-27 Thread Amit Kapila
On Fri, Mar 27, 2020 at 1:29 PM Masahiko Sawada wrote: > > On Fri, 27 Mar 2020 at 07:17, Justin Pryzby wrote: > > > > On Thu, Mar 26, 2020 at 10:04:57AM -0500, Justin Pryzby wrote: > > > Does that address your comment ? > > > > I hope so. > > Thank you for updating the patch. I'm concerned a bit

Re: [Proposal] Global temporary tables

2020-03-27 Thread tushar
On 3/27/20 10:55 AM, 曾文旌 wrote: Hi Wenjing, This patch(gtt_v21_pg13.patch) is not applicable on PG HEAD, I hope you have prepared the patch on top of some previous commit. Could you please rebase the patch which we can apply on HEAD ? Yes, It looks like the built-in functions are in conflict

Re: pgsql: Provide a TLS init hook

2020-03-27 Thread Andrew Dunstan
[discussion from -committers] On 3/26/20 4:31 PM, Tom Lane wrote: > Andrew Dunstan writes: >> On 3/26/20 11:31 AM, Tom Lane wrote: >>> Andrew Dunstan writes: I don't think this belongs in installcheck, we should add 'NO_INSTALLCHECK = 1' to the Makefile. >>> Why? The other

Re: WAL usage calculation patch

2020-03-27 Thread Julien Rouhaud
On Mon, Mar 23, 2020 at 3:24 PM Fujii Masao wrote: > > On 2020/03/23 21:01, Fujii Masao wrote: > > > > > > On 2020/03/23 7:32, Kirill Bychik wrote: > > I'm attaching a v5 with fp records only for temp tables, so there's no > > risk of > > instability. As I previously said I'm fine

Re: Planning counters in pg_stat_statements (using pgss_store)

2020-03-27 Thread Julien Rouhaud
On Thu, Mar 26, 2020 at 02:22:47PM +0100, Julien Rouhaud wrote: > On Thu, Mar 26, 2020 at 08:08:35PM +0900, Fujii Masao wrote: > > > > Here are other comments. > > > > - if (jstate) > > + if (kind == PGSS_JUMBLE) > > > > Why is PGSS_JUMBLE necessary? ISTM that we can still

Re: Planning counters in pg_stat_statements (using pgss_store)

2020-03-27 Thread Fujii Masao
On 2020/03/27 19:00, Julien Rouhaud wrote: On Thu, Mar 26, 2020 at 02:22:47PM +0100, Julien Rouhaud wrote: On Thu, Mar 26, 2020 at 08:08:35PM +0900, Fujii Masao wrote: Here are other comments. - if (jstate) + if (kind == PGSS_JUMBLE) Why is PGSS_JUMBLE

Re: allow online change primary_conninfo

2020-03-27 Thread Sergei Kornilov
Hello > I realized that the reason the tests broke after Sergei's patch is that > recovery/t/001_stream_rep.pl's get_slot_xmins() is broken for temp > walreceiver slots, since it's using the non-temp name it tries to give > to the slot, rather than the temp name under which it is actually >

Re: error context for vacuum to include block number

2020-03-27 Thread Amit Kapila
On Fri, Mar 27, 2020 at 11:46 AM Justin Pryzby wrote: > > On Thu, Mar 26, 2020 at 11:44:24PM -0500, Justin Pryzby wrote: > > On Fri, Mar 27, 2020 at 09:49:29AM +0530, Amit Kapila wrote: > > > On Fri, Mar 27, 2020 at 3:47 AM Justin Pryzby > > > wrote: > > > > > > > > > Hm, I was just wondering

Re: Unicode normalization SQL functions

2020-03-27 Thread John Naylor
I wrote: > > Regression tests pass, but I haven't measured performance yet. Using a test similar to one upthread: select count(*) from (select md5(i::text) as t from generate_series(1,10) as i) s where t is nfc normalized ; I get (median of three) v4 419ms v5 310ms with binary size v4

Re: Some problems of recovery conflict wait events

2020-03-27 Thread Fujii Masao
On 2020/03/27 15:39, Masahiko Sawada wrote: On Fri, 27 Mar 2020 at 10:32, Fujii Masao wrote: On 2020/03/26 14:33, Masahiko Sawada wrote: On Tue, 24 Mar 2020 at 17:04, Fujii Masao wrote: On 2020/03/05 20:16, Fujii Masao wrote: On 2020/03/05 16:58, Masahiko Sawada wrote: On Wed,

Re: pg_checksums in backend/storage/page/README

2020-03-27 Thread Michael Paquier
On Thu, Mar 26, 2020 at 03:06:34PM +0100, Magnus Hagander wrote: > That seems obvious enough. Pushed, thanks! Thanks for the fix. -- Michael signature.asc Description: PGP signature

Re: potential stuck lock in SaveSlotToPath()

2020-03-27 Thread Michael Paquier
On Thu, Mar 26, 2020 at 02:16:05PM +0100, Peter Eisentraut wrote: > committed and backpatched The patch committed does that in three places: /* rename to permanent file, fsync file and directory */ if (rename(tmppath, path) != 0) { + LWLockRelease(>io_in_progress_lock);

Re: backend type in log_line_prefix?

2020-03-27 Thread Kyotaro Horiguchi
Hello. At Mon, 23 Mar 2020 18:38:53 -0400, Bruce Momjian wrote in > Patch applied to master, thanks. The patch (8e8a0becb3) named archiver process as just "archiver". On the other hand the discussion in the thread [1] was going to name the process as "WAL/wal archiver". As all other

Re: [Patch] pg_rewind: options to use restore_command from recovery.conf or command line

2020-03-27 Thread Michael Paquier
On Fri, Mar 27, 2020 at 12:24:19AM +0300, Alexey Kondratov wrote: > The block of function declarations for xlogarchive.c inside xlog_internal.h > looks a bit dangling already, since all other functions and variables > defined/initialized in xlog.c. That way, it looks good to me to move it >

Re: error context for vacuum to include block number

2020-03-27 Thread Masahiko Sawada
On Fri, 27 Mar 2020 at 07:17, Justin Pryzby wrote: > > On Thu, Mar 26, 2020 at 10:04:57AM -0500, Justin Pryzby wrote: > > Does that address your comment ? > > I hope so. Thank you for updating the patch. I'm concerned a bit about overhead of frequently updating and reverting the callback

Re: Some problems of recovery conflict wait events

2020-03-27 Thread Masahiko Sawada
On Fri, 27 Mar 2020 at 10:32, Fujii Masao wrote: > > > > On 2020/03/26 14:33, Masahiko Sawada wrote: > > On Tue, 24 Mar 2020 at 17:04, Fujii Masao > > wrote: > >> > >> > >> > >> On 2020/03/05 20:16, Fujii Masao wrote: > >>> > >>> > >>> On 2020/03/05 16:58, Masahiko Sawada wrote: > On Wed,

Re: error context for vacuum to include block number

2020-03-27 Thread Justin Pryzby
On Thu, Mar 26, 2020 at 11:44:24PM -0500, Justin Pryzby wrote: > On Fri, Mar 27, 2020 at 09:49:29AM +0530, Amit Kapila wrote: > > On Fri, Mar 27, 2020 at 3:47 AM Justin Pryzby wrote: > > > > > > > Hm, I was just wondering what happens if an error happens *during* > > > >

Re: backup manifests

2020-03-27 Thread Andres Freund
Hi, On 2020-03-23 12:15:54 -0400, Robert Haas wrote: > + > +MANIFEST > + > + > + When this option is specified with a value of > ye' s/ye'/yes/ > + or force-escape, a backup manifest is created > + and sent along with the backup. The

Re: Planning counters in pg_stat_statements (using pgss_store)

2020-03-27 Thread Julien Rouhaud
On Fri, Mar 27, 2020 at 2:01 PM Fujii Masao wrote: > > On 2020/03/27 19:00, Julien Rouhaud wrote: > > On Thu, Mar 26, 2020 at 02:22:47PM +0100, Julien Rouhaud wrote: > >> On Thu, Mar 26, 2020 at 08:08:35PM +0900, Fujii Masao wrote: > >>> > >>> Here are other comments. > >>> > >>> - if

Re: Asymmetric partition-wise JOIN

2020-03-27 Thread David Steele
Hi Thomas, On 12/27/19 2:34 AM, Kohei KaiGai wrote: > This crash was reproduced on our environment also. It looks to me adjust_child_relids_multilevel() didn't expect a case when supplied 'relids' (partially) indicate normal and non-partitioned relation. It tries to build a new 'parent_relids'

Re: pgsql: Provide a TLS init hook

2020-03-27 Thread Tom Lane
Andrew Dunstan writes: > [discussion from -committers] > On 3/26/20 4:31 PM, Tom Lane wrote: >> So? We clearly document that for the TAP tests, "make installcheck" >> means "use the installed executables, but run a new instance" [1]. > I think we were probably a bit shortsighted about that.

Re: backup manifests

2020-03-27 Thread Stephen Frost
Greetings, * Andres Freund (and...@anarazel.de) wrote: > On 2020-03-26 11:37:48 -0400, Robert Haas wrote: > > I'm sorry that you can't see how that's sensible, but it doesn't mean > > that it isn't sensible. It is totally unrealistic to expect that any > > backup verification tool can verify that

Re: Planning counters in pg_stat_statements (using pgss_store)

2020-03-27 Thread Julien Rouhaud
On Fri, Mar 27, 2020 at 12:02 PM Fujii Masao wrote: > > On 2020/03/27 19:00, Julien Rouhaud wrote: > > On Thu, Mar 26, 2020 at 02:22:47PM +0100, Julien Rouhaud wrote: > >> On Thu, Mar 26, 2020 at 08:08:35PM +0900, Fujii Masao wrote: > >>> > >>> Here are other comments. > >>> > >>> - if

Re: Allow cluster owner to bypass authentication

2020-03-27 Thread David Steele
Hi Peter, On 12/27/19 3:22 PM, Stephen Frost wrote: * Tom Lane (t...@sss.pgh.pa.us) wrote: I think it'd be great if this behavior could be implemented within the notation, because we could then just set up a non-empty default pg_ident.conf with useful behavioral examples in the form of prefab

Re: adding partitioned tables to publications

2020-03-27 Thread Amit Langote
On Thu, Mar 26, 2020 at 11:23 PM Amit Langote wrote: > On Wed, Mar 25, 2020 at 9:29 PM Peter Eisentraut > wrote: > > On 2020-03-23 06:02, Amit Langote wrote: > > > Okay, added some tests. > > > > > > Attached updated patches. > > > > I have committed the worker.c refactoring patch. > > > > "Add

Re: Planning counters in pg_stat_statements (using pgss_store)

2020-03-27 Thread Fujii Masao
On 2020/03/27 19:00, Julien Rouhaud wrote: On Thu, Mar 26, 2020 at 02:22:47PM +0100, Julien Rouhaud wrote: On Thu, Mar 26, 2020 at 08:08:35PM +0900, Fujii Masao wrote: Here are other comments. - if (jstate) + if (kind == PGSS_JUMBLE) Why is PGSS_JUMBLE

Re: backup manifests

2020-03-27 Thread Robert Haas
On Fri, Mar 27, 2020 at 11:26 AM Stephen Frost wrote: > > Seems better to (later?) add support for generating manifests for WAL > > files, and then have a tool that can verify all the manifests required > > to restore a base backup. > > I'm not trying to expand on the feature set here or move the

Re: allow online change primary_conninfo

2020-03-27 Thread Alvaro Herrera
On 2020-Mar-27, Alvaro Herrera wrote: > I pushed the wal_receiver_create_temp_slot bugfix, because I realized > after looking for long enough at WalReceiverMain() that the code was > beyond saving. I'll be pushing the rest of this later today. So here's the next one. I still have to go over

Re: backup manifests

2020-03-27 Thread Stephen Frost
Greetings, * Robert Haas (robertmh...@gmail.com) wrote: > On Fri, Mar 27, 2020 at 11:26 AM Stephen Frost wrote: > > > Seems better to (later?) add support for generating manifests for WAL > > > files, and then have a tool that can verify all the manifests required > > > to restore a base backup.

Re: Internal key management system

2020-03-27 Thread Bruce Momjian
On Wed, Mar 25, 2020 at 05:51:08PM +0900, Masahiko Sawada wrote: > On Tue, 24 Mar 2020 at 23:15, Bruce Momjian wrote: > > > > On Tue, Mar 24, 2020 at 02:29:57PM +0900, Masahiko Sawada wrote: > > > That seems to work fine. > > > > > > So we will have pg_cryptokeys within PGDATA and each key is

Re: backup manifests

2020-03-27 Thread Stephen Frost
Greetings, * Andres Freund (and...@anarazel.de) wrote: > On 2020-03-27 15:20:27 -0400, Robert Haas wrote: > > On Fri, Mar 27, 2020 at 2:29 AM Andres Freund wrote: > > > Hm. Should this warn if the directory's permissions are set too openly > > > (world writable?)? > > > > I don't think so, but

Re: backup manifests

2020-03-27 Thread Stephen Frost
Greetings, * Andres Freund (and...@anarazel.de) wrote: > On 2020-03-27 17:44:07 -0400, Stephen Frost wrote: > > * Andres Freund (and...@anarazel.de) wrote: > > > On 2020-03-27 15:20:27 -0400, Robert Haas wrote: > > > > On Fri, Mar 27, 2020 at 2:29 AM Andres Freund > > > > wrote: > > > > > Hm.

Re: backup manifests

2020-03-27 Thread Robert Haas
On Fri, Mar 27, 2020 at 1:06 AM Andres Freund wrote: > > Like, suppose we change the default from CRC-32C to SHA-something. On > > the upside, the error detection rate will increase from 99.999+% > > to something much closer to 100%. > > FWIW, I don't buy the relevancy of 99.999+% at all.

Re: proposal \gcsv

2020-03-27 Thread Pavel Stehule
čt 26. 3. 2020 v 19:41 odesílatel Erik Rijkers napsal: > On 2020-03-26 18:49, Pavel Stehule wrote: > > Hi > > > > [psql-gfmt.patch] > > This seems useful and works well; I haven't found any errors. Well done. > > However, I have a suggestion that is perhaps slightly outside of this > patch but

Re: allow online change primary_conninfo

2020-03-27 Thread Sergei Kornilov
Hello Thank you! > I think I should set aside your new draft for now I agree, this patch definitely needs a bit more time to review. (currently it applies on top of v13 patch cleanly) > but I think we should still get it in pg13 to avoid having the change the > semantics of the > walreceiver

Re: Conflict handling for COPY FROM

2020-03-27 Thread Tom Lane
Surafel Temesgen writes: > [ conflict-handling-copy-from-v16.patch ] I took a quick look at this patch, since it was marked "ready for committer", but I don't see how it can possibly be considered committable. 1. Covering only the errors that are thrown in DoCopy itself doesn't seem to me to

Re: psql FETCH_COUNT feature does not work with combined queries

2020-03-27 Thread Fabien COELHO
Assuming there's no one willing to fix the behavior (and that seems unlikely given we're in the middle of a 2020-01 commitfest) it makes sense to at least document the behavior. That being said, I think the proposed patch may be a tad too brief. I don't think we need to describe the exact

Re: allow online change primary_conninfo

2020-03-27 Thread Sergei Kornilov
Hello In other words, patches in reverse order: 0001 will change primary_conninfo and primary_slot_name to reloadable 0002 will move wal_receiver_create_temp_slot logic to startup process (without changing to PGC_POSTMASTER) 0003 is new draft patch: wal_receiver_create_temp_slot will use the

Re: BUG #16040: PL/PGSQL RETURN QUERY statement never uses a parallel plan

2020-03-27 Thread Hamid Akhtar
The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, passed Spec compliant: tested, passed Documentation:tested, passed All good with this patch. -- Highgo Software

Re: Reinitialize stack base after fork (for the benefit of rr)?

2020-03-27 Thread Andres Freund
Hi, On 2020-03-27 14:34:56 -0400, Tom Lane wrote: > Andres Freund writes: > > I've locally fixed the issue by computing the stack base address anew > > for postmaster children. Currently in InitPostmasterChild(). > > > I'd like to get that change upstream. The rr hackers have fixed a number > >

Re: pgsql: Provide a TLS init hook

2020-03-27 Thread Tom Lane
Andrew Dunstan writes: > On 3/27/20 11:09 AM, Tom Lane wrote: >> It seems like what the buildfarm would like is a way to invoke TAP tests >> and traditional-framework tests separately, so that it could apply special >> tooling to the former. I'd have no objection to making that possible. >

Re: pgbench - refactor init functions with buffers

2020-03-27 Thread Fabien COELHO
Hello David, I'd prefer not to expand the use of pqexpbuffer in more places, and instead rather see this use StringInfo, now that's also available to frontend programs. Franckly, one or the other does not matter much to me. FWIW, I agree with Andres with regard to using StringInfo. Ok.

Possible copy and past error? (\usr\backend\commands\analyze.c)

2020-03-27 Thread Ranier Vilela
Hi, Can someone check if there is a copy and paste error, at file: \usr\backend\commands\analyze.c, at lines 2225 and 2226? int num_mcv = stats->attr->attstattarget; int num_bins = stats->attr->attstattarget; If they really are the same values, it could be changed to: int num_mcv =

Re: fix for BUG #3720: wrong results at using ltree

2020-03-27 Thread Nikita Glukhov
On 24.01.2020 21:29, Tomas Vondra wrote: Hi Nikita, This patch seems inactive / stuck in "waiting on author" since November. It's marked as bugfix, so it'd be good to get it committed instead of just punting it to the next CF. I did a quick review, and I came mostly with the same two

Re: pgbench - rework variable management

2020-03-27 Thread David Steele
On 3/27/20 6:25 PM, Fabien COELHO wrote: Patch v5 is a rebase with some adjustements. This patch is failing on the Windows build: https://ci.appveyor.com/project/postgresql-cfbot/postgresql/build/1.0.85698 I'm not sure if this had been fixed in v3 and this is a new issue or if it has

Re: backup manifests

2020-03-27 Thread Robert Haas
On Thu, Mar 26, 2020 at 4:37 PM David Steele wrote: > I agree with Stephen that this should be done, but I agree with you that > it can wait for a future commit. However, I do think: > > 1) It should be called out rather plainly in the documentation. > 2) If there are files in pg_wal then

Re: Reinitialize stack base after fork (for the benefit of rr)?

2020-03-27 Thread Tom Lane
Andres Freund writes: > I've locally fixed the issue by computing the stack base address anew > for postmaster children. Currently in InitPostmasterChild(). > I'd like to get that change upstream. The rr hackers have fixed a number > of other issues that could be hit with postgres, but they

Re: error context for vacuum to include block number

2020-03-27 Thread Justin Pryzby
On Fri, Mar 27, 2020 at 11:50:30AM +0530, Amit Kapila wrote: > > > The crash scenario I'm trying to avoid would be like statement_timeout or > > > other > > > asynchronous event occurring between two non-atomic operations. > > > > > +if (errinfo->phase==VACUUM_ERRCB_PHASE_VACUUM_INDEX && > >

Re: WAL usage calculation patch

2020-03-27 Thread Julien Rouhaud
On Fri, Mar 27, 2020 at 8:21 PM Kirill Bychik wrote: > > > > > I'm attaching a v5 with fp records only for temp tables, so there's > > > > no risk of > > > > instability. As I previously said I'm fine with your two patches, > > > > so unless > > > > you have objections on

Re: backup manifests

2020-03-27 Thread David Steele
On 3/27/20 3:20 PM, Robert Haas wrote: On Fri, Mar 27, 2020 at 2:29 AM Andres Freund wrote: Hm. Is it a great choice to include the checksum for the manifest inside the manifest itself? With a cryptographic checksum it seems like it could make a ton of sense to store the checksum somewhere

Re: backup manifests

2020-03-27 Thread Stephen Frost
Greetings, * Andres Freund (and...@anarazel.de) wrote: > On 2020-03-27 14:34:19 -0400, Robert Haas wrote: > > I think #2 is an interesting idea and could possibly reduce the danger > > of user confusion on this point considerably - because, let's face it, > > not everyone is going to read the

Re: Berserk Autovacuum (let's save next Mandrill)

2020-03-27 Thread David Rowley
On Fri, 27 Mar 2020 at 22:40, Laurenz Albe wrote: > The new meaning of -2 should be documented, other than that it looks > good to me. But the users don't need to know anything about -2. It's not possible to explicitly set the value to -2. This is just the reset value of the reloption which

Re: pgbench - refactor init functions with buffers

2020-03-27 Thread Fabien COELHO
Ok. I find it strange to mix PQExpBuffer & StringInfo in the same file. Agreed, but we'd rather use StringInfo going forward. However, I don't think that puts you on the hook for updating all the PQExpBuffer references. Unless you want to... I cannot say that I "want" to fix something

Re: allow online change primary_conninfo

2020-03-27 Thread Alvaro Herrera
On 2020-Mar-27, Sergei Kornilov wrote: > Hello > > > I realized that the reason the tests broke after Sergei's patch is that > > recovery/t/001_stream_rep.pl's get_slot_xmins() is broken for temp > > walreceiver slots, since it's using the non-temp name it tries to give > > to the slot, rather

Re: BUG #16040: PL/PGSQL RETURN QUERY statement never uses a parallel plan

2020-03-27 Thread Tom Lane
Hamid Akhtar writes: > The following review has been posted through the commitfest application: > make installcheck-world: tested, passed > Implements feature: tested, passed > Spec compliant: tested, passed > Documentation:tested, passed > All good with this patch.

Re: backup manifests

2020-03-27 Thread Andres Freund
Hi, On 2020-03-27 14:34:19 -0400, Robert Haas wrote: > I think #2 is an interesting idea and could possibly reduce the danger > of user confusion on this point considerably - because, let's face it, > not everyone is going to read the documentation. However, I'm having a > hard time figuring out

Re: Add A Glossary

2020-03-27 Thread Justin Pryzby
On Fri, Mar 20, 2020 at 11:32:25PM +0100, Jürgen Purtz wrote: > > > +Archiver > > Can you change that to archiver process ? > > I prefer the short term without the addition of 'process' - concerning > 'Archiver' as well as the other cases. But I'm not an native English > speaker. I didn't

Re: backup manifests

2020-03-27 Thread David Steele
On 3/27/20 3:55 PM, Stephen Frost wrote: * Robert Haas (robertmh...@gmail.com) wrote: I think that what we have seen so far is that all of the SHA-n algorithms that PostgreSQL supports are about equally slow, so it doesn't really matter which one you pick there from a performance point of view.

Re: Reinitialize stack base after fork (for the benefit of rr)?

2020-03-27 Thread Andres Freund
Hi, On 2020-03-27 14:59:56 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2020-03-27 14:34:56 -0400, Tom Lane wrote: > >> Andres Freund writes: > >>> Tom, while imo not a fix of the right magnitude here: Are you planning / > >>> hoping to work again on your postmaster latch patch? > >

Re: backup manifests

2020-03-27 Thread Andres Freund
Hi, On 2020-03-27 16:57:46 -0400, Stephen Frost wrote: > I really don't know what to say to this. WAL is absolutely critical to > a backup being valid. pgBackRest doesn't have a way to *just* validate > a backup today, unfortunately, but we're planning to support it in the > future and we will

Re: backup manifests

2020-03-27 Thread Stephen Frost
Greetings, * Andres Freund (and...@anarazel.de) wrote: > On 2020-03-27 16:57:46 -0400, Stephen Frost wrote: > > I really don't know what to say to this. WAL is absolutely critical to > > a backup being valid. pgBackRest doesn't have a way to *just* validate > > a backup today, unfortunately,

Re: backup manifests

2020-03-27 Thread Bruce Momjian
On Fri, Mar 27, 2020 at 06:38:33PM -0400, Stephen Frost wrote: > Greetings, > > On Fri, Mar 27, 2020 at 18:36 Bruce Momjian wrote: > > On Thu, Mar 26, 2020 at 12:34:52PM -0400, Stephen Frost wrote: > > * Robert Haas (robertmh...@gmail.com) wrote: > > > This is where I feel like I'm

Re: backup manifests

2020-03-27 Thread Robert Haas
On Thu, Mar 26, 2020 at 4:44 PM Stephen Frost wrote: > Is it actually possible, today, in PG, to have a 4GB WAL record? > Judging this based on the WAL record size doesn't seem quite right. I'm not sure. I mean, most records are quite small, but I think if you set REPLICA IDENTITY FULL on a

Re: A bug when use get_bit() function for a long bytea string

2020-03-27 Thread Tom Lane
"Daniel Verite" writes: > So aside from the integer overflow bug, isn't there the issue that the > "offset" argument of get_bit() and set_bit() should have been an > int8 in the first place? Good point, but a fix for that wouldn't be back-patchable. It does suggest that we should just make all

Re: backup manifests

2020-03-27 Thread Robert Haas
On Fri, Mar 27, 2020 at 2:29 AM Andres Freund wrote: > s/ye'/yes/ Ugh, sorry. Fixed in the version posted earlier. > Are you planning to include a specification of the manifest file format > anywhere? I looked through the patches and didn't find anything. I thought about that. I think it would

Re: backup manifests

2020-03-27 Thread Andres Freund
Hi, On 2020-03-27 14:13:17 -0400, Robert Haas wrote: > On Thu, Mar 26, 2020 at 4:44 PM Stephen Frost wrote: > > > I mean, the property that I care about is the one where it detects > > > better than 999,999,999 errors out of every 1,000,000,000, regardless > > > of input length. > > > > Throwing

Re: backup manifests

2020-03-27 Thread Stephen Frost
Greetings, * Robert Haas (robertmh...@gmail.com) wrote: > On Thu, Mar 26, 2020 at 4:44 PM Stephen Frost wrote: > > Is it actually possible, today, in PG, to have a 4GB WAL record? > > Judging this based on the WAL record size doesn't seem quite right. > > I'm not sure. I mean, most records are

Re: backup manifests

2020-03-27 Thread David Steele
On 3/27/20 3:29 PM, Robert Haas wrote: On Fri, Mar 27, 2020 at 11:26 AM Stephen Frost wrote: Seems better to (later?) add support for generating manifests for WAL files, and then have a tool that can verify all the manifests required to restore a base backup. I'm not trying to expand on the

Re: Allow CLUSTER, VACUUM FULL and REINDEX to change tablespace on the fly

2020-03-27 Thread Justin Pryzby
On Thu, Mar 26, 2020 at 11:01:06PM -0500, Justin Pryzby wrote: > > Another issue is this: > > > +VACUUM ( FULL [, ...] ) [ TABLESPACE > > class="parameter">new_tablespace ] [ > > class="parameter">table_and_columns [, ...] ] > > As you mentioned in your v1 patch, in the other cases, "tablespace

Re: pgbench - rework variable management

2020-03-27 Thread Fabien COELHO
Patch v5 is a rebase with some adjustements. This patch is failing on the Windows build: https://ci.appveyor.com/project/postgresql-cfbot/postgresql/build/1.0.85698 I'm not sure if this had been fixed in v3 and this is a new issue or if it has been failing all along. Either way, it should

Re: pgbench - refactor init functions with buffers

2020-03-27 Thread David Steele
On 3/27/20 6:13 PM, Fabien COELHO wrote: Hello David, I'd prefer not to expand the use of pqexpbuffer in more places, and instead rather see this use StringInfo, now that's also available to frontend programs. Franckly, one or the other does not matter much to me. FWIW, I agree with

Re: Reinitialize stack base after fork (for the benefit of rr)?

2020-03-27 Thread Tom Lane
Andres Freund writes: > On 2020-03-27 14:34:56 -0400, Tom Lane wrote: >> Andres Freund writes: >>> Tom, while imo not a fix of the right magnitude here: Are you planning / >>> hoping to work again on your postmaster latch patch? >> Um ... -ESWAPPEDOUT. What are you thinking of? >

Re: Improve handling of parameter differences in physical replication

2020-03-27 Thread Sergei Kornilov
Hello > I think we can set wait event WAIT_EVENT_RECOVERY_PAUSE here. +1, since we added this in recoveryPausesHere. PS: do we need to add a prototype for the RecoveryRequiredIntParameter function in top of xlog.c? regards, Sergei

Re: backup manifests

2020-03-27 Thread Stephen Frost
Greetings, * Robert Haas (robertmh...@gmail.com) wrote: > On Thu, Mar 26, 2020 at 4:37 PM David Steele wrote: > > I agree with Stephen that this should be done, but I agree with you that > > it can wait for a future commit. However, I do think: > > > > 1) It should be called out rather plainly

Re: backup manifests

2020-03-27 Thread Andres Freund
Hi, On 2020-03-27 15:29:02 -0400, Robert Haas wrote: > On Fri, Mar 27, 2020 at 11:26 AM Stephen Frost wrote: > > > Seems better to (later?) add support for generating manifests for WAL > > > files, and then have a tool that can verify all the manifests required > > > to restore a base backup. >

Re: backup manifests

2020-03-27 Thread Andres Freund
Hi, On 2020-03-27 15:20:27 -0400, Robert Haas wrote: > On Fri, Mar 27, 2020 at 2:29 AM Andres Freund wrote: > > Are you planning to include a specification of the manifest file format > > anywhere? I looked through the patches and didn't find anything. > > I thought about that. I think it would

Reinitialize stack base after fork (for the benefit of rr)?

2020-03-27 Thread Andres Freund
Hi, I've found rr [1] very useful to debug issues in postgres. The ability to hit a bug, and then e.g. identify a pointer with problematic contents, set a watchpoint on its contents, and reverse-continue is extremely powerful. Unfortunately, when running postgres, it currently occasionally

Re: WAL usage calculation patch

2020-03-27 Thread Kirill Bychik
> > > I'm attaching a v5 with fp records only for temp tables, so there's > > > no risk of > > > instability. As I previously said I'm fine with your two patches, so > > > unless > > > you have objections on the fpi test for temp tables or the > > > documentation > >

Re: some AppVeyor files

2020-03-27 Thread Mike Palmiotto
On Tue, Mar 24, 2020 at 10:27 PM Thomas Munro wrote: > Thanks! I added a link to this thread to a Wiki page that tries to > collect information on this topic[1]. Another thing you could be > interested in is the ability to test on several different MSVC > versions (I tried to find some

Re: backup manifests

2020-03-27 Thread Andres Freund
Hi, On 2020-03-27 17:44:07 -0400, Stephen Frost wrote: > * Andres Freund (and...@anarazel.de) wrote: > > On 2020-03-27 15:20:27 -0400, Robert Haas wrote: > > > On Fri, Mar 27, 2020 at 2:29 AM Andres Freund wrote: > > > > Hm. Should this warn if the directory's permissions are set too openly > >

Re: backup manifests

2020-03-27 Thread Alvaro Herrera
On 2020-Mar-27, Stephen Frost wrote: > I don't think we should just compare all permissions or ownership with > some arbitrary idea of what we think they should be, even though if you > use pg_basebackup in 'plain' format, you actually end up with > differences, today, from what the source system

Re: [HACKERS] Re: [COMMITTERS] pgsql: Remove pgbench "progress" test pending solution of its timing is (fwd)

2020-03-27 Thread Fabien COELHO
This patch is registered in 2020-01, but the last message in the thread seems to be from 2019/05/23. The patch itself seems to be OK (it applies fine etc.) What do we need to get it over the line, instead of just moving it to the next one CF over and over? It does not look like the remainder

Re: backup manifests

2020-03-27 Thread David Steele
On 3/27/20 6:07 PM, Andres Freund wrote: Hi, On 2020-03-27 16:57:46 -0400, Stephen Frost wrote: I really don't know what to say to this. WAL is absolutely critical to a backup being valid. pgBackRest doesn't have a way to *just* validate a backup today, unfortunately, but we're planning to

Re: pgsql: Provide a TLS init hook

2020-03-27 Thread Andrew Dunstan
On 3/27/20 11:09 AM, Tom Lane wrote: > >> Right now the explicit TAP test code in the buildfarm knows how to collect >> all the relevant output. The installcheck code doesn't know about that for >> TAP tests. > It seems like what the buildfarm would like is a way to invoke TAP tests > and

Re: backup manifests

2020-03-27 Thread David Steele
On 3/27/20 1:53 PM, Robert Haas wrote: On Thu, Mar 26, 2020 at 4:37 PM David Steele wrote: I know you and Stephen have agreed on a number of doc changes, would it be possible to get a new patch with those included? I finally have time to do a review of this tomorrow. I saw some mistakes in

Re: backup manifests

2020-03-27 Thread Andres Freund
Hi, On 2020-03-27 17:07:42 -0400, Stephen Frost wrote: > I had suggested up-thread, and I'm still fine with, having > pg_validatebackup scan the WAL and check the internal checksums. I'd > prefer an option that uses hashes to check when the user has asked for > hashes with SHA256 or something,

Re: backup manifests

2020-03-27 Thread Stephen Frost
Greetings, On Fri, Mar 27, 2020 at 18:36 Bruce Momjian wrote: > On Thu, Mar 26, 2020 at 12:34:52PM -0400, Stephen Frost wrote: > > * Robert Haas (robertmh...@gmail.com) wrote: > > > This is where I feel like I'm trying to make decisions in a vacuum. If > > > we had a few more people weighing in

Re: backup manifests

2020-03-27 Thread Bruce Momjian
On Thu, Mar 26, 2020 at 12:34:52PM -0400, Stephen Frost wrote: > * Robert Haas (robertmh...@gmail.com) wrote: > > This is where I feel like I'm trying to make decisions in a vacuum. If > > we had a few more people weighing in on the thread on this point, I'd > > be happy to go with whatever the

Re: error context for vacuum to include block number

2020-03-27 Thread Justin Pryzby
On Sat, Mar 28, 2020 at 06:28:38AM +0530, Amit Kapila wrote: > > Hm, but I caused a crash *without* adding CHECK_FOR_INTERRUPTS, just > > kill+sleep. The kill() could come from running pg_cancel_backend(). And > > the > > sleep() just encourages a context switch, which can happen at any time. >

Re: SLRU statistics

2020-03-27 Thread Tomas Vondra
Hi, here is a bit improved version of the patch - I've been annoyed by how the resetting works (per-entry timestamp, but resetting all entries) so I've added a new function pg_stat_reset_slru() that allows resetting either all entries or just one entry (identified by name). So SELECT

Re: error context for vacuum to include block number

2020-03-27 Thread Justin Pryzby
On Sat, Mar 28, 2020 at 06:59:10AM +0530, Amit Kapila wrote: > On Sat, Mar 28, 2020 at 6:46 AM Justin Pryzby wrote: > > > > On Sat, Mar 28, 2020 at 06:28:38AM +0530, Amit Kapila wrote: > > > > Hm, but I caused a crash *without* adding CHECK_FOR_INTERRUPTS, just > > > > kill+sleep. The kill()

Re: Make mesage at end-of-recovery less scary.

2020-03-27 Thread James Coleman
On Thu, Mar 26, 2020 at 12:41 PM Robert Haas wrote: > > On Wed, Mar 25, 2020 at 8:53 AM Peter Eisentraut > wrote: > > HINT: This is to be expected if this is the end of the WAL. Otherwise, > > it could indicate corruption. > > First, I agree that this general issue is a problem, because it's

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-27 Thread Tomas Vondra
On Fri, Mar 27, 2020 at 09:36:55PM -0400, James Coleman wrote: On Fri, Mar 27, 2020 at 9:19 PM Tomas Vondra wrote: On Fri, Mar 27, 2020 at 12:51:34PM -0400, James Coleman wrote: >In a previous email I'd summarized remaining TODOs I'd found. Here's >an updated listed with several resolved. >

Re: [HACKERS] Re: [COMMITTERS] pgsql: Remove pgbench "progress" test pending solution of its timing is (fwd)

2020-03-27 Thread Tom Lane
Fabien COELHO writes: >> It does not look like the remainder of this patch is going to be committed >> and I don't think it makes sense to keep moving the patch indefinitely. >> Unless something changes by the end of this CF I'll mark it Returned With >> Feedback. > I'd be rather unclear about

Re: [PATCH] Btree BackwardScan race condition on Standby during VACUUM

2020-03-27 Thread Peter Geoghegan
On Mon, Mar 16, 2020 at 7:08 AM Michail Nikolaev wrote: > I was sure I have broken something in btree and spent a lot of time > trying to figure what. > And later... I realized what it is bug in btree since a very old times... > Because of much faster scans with LP_DEAD support on a standby it >

  1   2   >