Re: pg_dump partitions can lead to inconsistent state after restore

2019-06-20 Thread Amit Langote
On Thu, Apr 25, 2019 at 4:01 AM Alvaro Herrera wrote: > So, while testing this I noticed that pg_restore fails with deadlocks if > you do a parallel restore if the --load-via-partition-root switch was > given to pg_dump. Is that a known bug? Was investigating --load-via-partition-root with a

Re: clean up docs for v12

2019-06-20 Thread Michael Paquier
On Thu, Jun 20, 2019 at 07:34:10PM -0400, Alvaro Herrera wrote: > This patch was applied as f73293aba4d4. Thanks, Paul and Michael. Thanks for the thread update, Alvaro. I completely forgot to mention the commit on this thread. -- Michael signature.asc Description: PGP signature

Re: unlogged sequences

2019-06-20 Thread Michael Paquier
On Thu, Jun 20, 2019 at 09:30:34AM +0200, Peter Eisentraut wrote: > The discussion in bug #15631 revealed that serial/identity sequences of > temporary tables should really also be temporary (easy), and that > serial/identity sequences of unlogged tables should also be unlogged. > But there is no

Re: clean up docs for v12

2019-06-20 Thread Alvaro Herrera
This patch was applied as f73293aba4d4. Thanks, Paul and Michael. -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: clean up docs for v12

2019-06-20 Thread Alvaro Herrera
On 2019-May-20, Paul A Jungwirth wrote: > On Mon, May 20, 2019 at 9:44 PM Amit Langote > wrote: > > This sounds more like an open item to me [1], not something that have to > > be postponed until the next CF. > > > > [1] https://wiki.postgresql.org/wiki/PostgreSQL_12_Open_Items > > Oh sorry, I

Re: Choosing values for multivariate MCV lists

2019-06-20 Thread Tomas Vondra
On Thu, Jun 20, 2019 at 06:55:41AM +0100, Dean Rasheed wrote: On Tue, 18 Jun 2019 at 21:59, Tomas Vondra wrote: The current implementation of multi-column MCV lists (added in this cycle) uses a fairly simple algorithm to pick combinations to include in the MCV list. We just compute a minimum

Re: Multivariate MCV list vs. statistics target

2019-06-20 Thread Tomas Vondra
On Thu, Jun 20, 2019 at 08:08:44AM +0100, Dean Rasheed wrote: On Tue, 18 Jun 2019 at 22:34, Tomas Vondra wrote: One slightly inconvenient thing I realized while playing with the address data set is that it's somewhat difficult to set the desired size of the multi-column MCV list. At the

Re: UCT (Re: pgsql: Update time zone data files to tzdata release 2019a.)

2019-06-20 Thread Tom Lane
Andrew Gierth writes: > Tom's "fix" of backpatching 23bd3cec6 (which happened on Friday 14th) > addressed only a subset of cases, as far as I know working only on Linux > (the historical convention has always been for /etc/localtime to be a > copy of a zonefile, not a symlink to one). I only

Re: UCT (Re: pgsql: Update time zone data files to tzdata release 2019a.)

2019-06-20 Thread Tom Lane
Andrew Gierth writes: > "Tom" == Tom Lane writes: > Tom> Keep in mind that dealing with whatever tzdb chooses to ship is > Tom> not optional from our standpoint. Even if we'd refused to import > Tom> 2019a, every installation using --with-system-tzdata (which, I > Tom> sincerely hope,

Re: Tweaking DSM and DSA limits

2019-06-20 Thread David Fetter
On Thu, Jun 20, 2019 at 02:20:27PM -0400, Robert Haas wrote: > On Tue, Jun 18, 2019 at 9:08 PM Thomas Munro wrote: > > It's currently set to 4, but I now think that was too cautious. It > > tries to avoid fragmentation by ramping up slowly (that is, memory > > allocated and in some cases

Re: UCT (Re: pgsql: Update time zone data files to tzdata release 2019a.)

2019-06-20 Thread Andrew Gierth
> "Tom" == Tom Lane writes: >> I was referring to the fact that the regression was introduced by a, >> presumably important, tzdb update (2019a, as mentioned above). At >> least, I made the assumption that the commit of the import of 2019a >> had more than just the change that introduced

Re: UCT (Re: pgsql: Update time zone data files to tzdata release 2019a.)

2019-06-20 Thread Tom Lane
Stephen Frost writes: > * Andrew Gierth (and...@tao11.riddles.org.uk) wrote: > "Stephen" == Stephen Frost writes: >> Stephen> Piling on to that, the regression was entwined with other >> Stephen> important changes that we wanted to include in the release. >> >> I'm not sure what you're

Re: UCT (Re: pgsql: Update time zone data files to tzdata release 2019a.)

2019-06-20 Thread Stephen Frost
Greetings, * Andrew Gierth (and...@tao11.riddles.org.uk) wrote: > > "Stephen" == Stephen Frost writes: > > Stephen> In the situation that started this discussion, a change had > Stephen> already been made and it was only later realized that it > Stephen> caused a regression. > > Just to

Re: UCT (Re: pgsql: Update time zone data files to tzdata release 2019a.)

2019-06-20 Thread Andrew Gierth
> "Stephen" == Stephen Frost writes: Stephen> In the situation that started this discussion, a change had Stephen> already been made and it was only later realized that it Stephen> caused a regression. Just to keep the facts straight: The regression was introduced by importing tzdb

Re: commitfest application - create patch doesn't work

2019-06-20 Thread Pavel Stehule
čt 20. 6. 2019 v 20:27 odesílatel Stephen Frost napsal: > Greetings, > > * Pavel Stehule (pavel.steh...@gmail.com) wrote: > > Searching subject for "Specify thread msgid" field doesn't work. It > returns > > empty result set every time. > > Is this still not working? I was chatting with Magnus

Re: Tweaking DSM and DSA limits

2019-06-20 Thread Andres Freund
Hi, On 2019-06-20 14:20:27 -0400, Robert Haas wrote: > On Tue, Jun 18, 2019 at 9:08 PM Thomas Munro wrote: > > Perhaps also the number of slots per backend should be dynamic, so > > that you have the option to increase it from the current hard-coded > > value of 2 if you don't want to increase

Re: psql UPDATE field [tab] expands to DEFAULT?

2019-06-20 Thread Tom Lane
I wrote: > Nosing around in tab-complete.c, I notice a fair number of other > places where we're doing COMPLETE_WITH() with just a single possible > completion. Knowing what we know now, in each one of those places > libreadline will suppose that that completion is the only syntactically > legal

Re: commitfest application - create patch doesn't work

2019-06-20 Thread Stephen Frost
Greetings, * Pavel Stehule (pavel.steh...@gmail.com) wrote: > Searching subject for "Specify thread msgid" field doesn't work. It returns > empty result set every time. Is this still not working? I was chatting with Magnus and it seems possible this was broken and then fixed already. Thanks,

Re: UCT (Re: pgsql: Update time zone data files to tzdata release 2019a.)

2019-06-20 Thread Stephen Frost
Greetings, * Robert Haas (robertmh...@gmail.com) wrote: > On Thu, Jun 20, 2019 at 1:28 PM Alvaro Herrera > wrote: > > I suppose we could have a moratorium on commits starting from (say) EOB > > Wednesday of the week prior to the release; patches can only be > > committed after that if they have

Re: Tweaking DSM and DSA limits

2019-06-20 Thread Robert Haas
On Tue, Jun 18, 2019 at 9:08 PM Thomas Munro wrote: > It's currently set to 4, but I now think that was too cautious. It > tries to avoid fragmentation by ramping up slowly (that is, memory > allocated and in some cases committed by the operating system that we > don't turn out to need), but

Re: UCT (Re: pgsql: Update time zone data files to tzdata release 2019a.)

2019-06-20 Thread Robert Haas
On Thu, Jun 20, 2019 at 1:28 PM Alvaro Herrera wrote: > I suppose we could have a moratorium on commits starting from (say) EOB > Wednesday of the week prior to the release; patches can only be > committed after that if they have ample support (where "ample support" > might be defined as having

Re: UCT (Re: pgsql: Update time zone data files to tzdata release 2019a.)

2019-06-20 Thread Alvaro Herrera
On 2019-Jun-20, Andres Freund wrote: > On 2019-06-20 12:02:30 -0400, Robert Haas wrote: > > I agree that it's a difficult situation. I do kind of wonder whether > > we were altogether overreacting. If we had shipped it as it was, > > what's the worst thing that would have happened? > > I

Re: UCT (Re: pgsql: Update time zone data files to tzdata release 2019a.)

2019-06-20 Thread Stephen Frost
Greetings, * Andres Freund (and...@anarazel.de) wrote: > On 2019-06-20 12:02:30 -0400, Robert Haas wrote: > > On Thu, Jun 20, 2019 at 8:52 AM Stephen Frost wrote: > > > I don't want to come across as implying that I'm saying what was done > > > was 'fine', or that we shouldn't be having this

commitfest application - create patch doesn't work

2019-06-20 Thread Pavel Stehule
Hi Searching subject for "Specify thread msgid" field doesn't work. It returns empty result set every time. Regards Pavel

Do we need to do better for pg_ctl timeouts?

2019-06-20 Thread Andres Freund
Hi, Right now using -w for shutting down clusters with a bit bigger shared buffers will very frequently fail, because the shutdown checkpoint takes much longer than 60s. Obviously that can be addressed by manually setting PGCTLTIMEOUT to something higher, but forcing many users to do that

Re: POC: Cleaning up orphaned files using undo logs

2019-06-20 Thread Robert Haas
On Thu, Jun 20, 2019 at 11:35 AM Amit Kapila wrote: > This delay is for *not* choking the system by constantly performing > undo requests that consume a lot of CPU and I/O as discussed in above > point. For holding off the same error request to be re-tried, we need > next_retry_time type of

Re: UCT (Re: pgsql: Update time zone data files to tzdata release 2019a.)

2019-06-20 Thread Andres Freund
Hi, On 2019-06-20 12:02:30 -0400, Robert Haas wrote: > On Thu, Jun 20, 2019 at 8:52 AM Stephen Frost wrote: > > I don't want to come across as implying that I'm saying what was done > > was 'fine', or that we shouldn't be having this conversation, I'm just > > trying to figure out how we can

Re: benchmarking Flex practices

2019-06-20 Thread Andres Freund
Hi, On 2019-06-20 10:52:54 -0400, Tom Lane wrote: > John Naylor writes: > > It would be nice to have confirmation to make sure I didn't err > > somewhere, and to try a more real-world benchmark. > > I don't see much wrong with using information_schema.sql as a parser/lexer > benchmark case. We

Re: POC: Cleaning up orphaned files using undo logs

2019-06-20 Thread Amit Kapila
On Thu, Jun 20, 2019 at 8:01 PM Robert Haas wrote: > > On Thu, Jun 20, 2019 at 2:42 AM Amit Kapila wrote: > > Okay, one reason that comes to mind is we don't want to choke the > > system as applying undo can consume CPU and generate a lot of I/O. Is > > that you have in mind or something else?

Re: Disconnect from SPI manager on error

2019-06-20 Thread Tom Lane
RekGRpth writes: > A patch fixing this bug > https://www.postgresql.org/message-id/flat/15738-21723084f3009ceb%40postgresql.org I do not think this code change is necessary or appropriate. It is not plpgsql's job to clean up after other backend subsystems during a transaction abort. Maybe if

Re: POC: Cleaning up orphaned files using undo logs

2019-06-20 Thread Robert Haas
On Thu, Jun 20, 2019 at 6:44 AM Amit Kapila wrote: > BTW, while looking at the code of UndoFetchRecord, I see some problem. > There is a coding pattern like > if() > { > } > else > { >LWLockAcquire() > .. > .. > } > > LWLockRelease(). > > I think this is not correct. Independently of

Re: improve transparency of bitmap-only heap scans

2019-06-20 Thread Emre Hasegeli
> Looking at the discussion where the feature was added, I think changing the > EXPLAIN just wasn't considered. I think this is an oversight. It is very useful to have this on EXPLAIN. > The attached patch adds "avoided" to "exact" and "lossy" as a category > under "Heap Blocks". It took me a

Re: benchmarking Flex practices

2019-06-20 Thread Tom Lane
John Naylor writes: > I decided to do some experiments with how we use Flex. The main > takeaway is that backtracking, which we removed in 2005, doesn't seem > to matter anymore for the core scanner. Also, state table size is of > marginal importance. Huh. That's really interesting, because

Re: POC: Cleaning up orphaned files using undo logs

2019-06-20 Thread Robert Haas
On Thu, Jun 20, 2019 at 6:48 AM Amit Kapila wrote: > > for (;;) > > { > > UnpackedUndoRecord *uur = UndoFetchRecord(urp); > > if (i like this one) > > break; > > urp = uur->uur_blkprev; // should be renamed, since zedstore + > > probably others will have tuple chains not block

benchmarking Flex practices

2019-06-20 Thread John Naylor
I decided to do some experiments with how we use Flex. The main takeaway is that backtracking, which we removed in 2005, doesn't seem to matter anymore for the core scanner. Also, state table size is of marginal importance. Using the information_schema Flex+Bison microbenchmark from Tom [1], I

Re: POC: Cleaning up orphaned files using undo logs

2019-06-20 Thread Robert Haas
On Thu, Jun 20, 2019 at 2:42 AM Amit Kapila wrote: > Okay, one reason that comes to mind is we don't want to choke the > system as applying undo can consume CPU and generate a lot of I/O. Is > that you have in mind or something else? Yeah, mainly that, but also things like log spam, and even

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-06-20 Thread Joe Conway
On 6/20/19 8:34 AM, Masahiko Sawada wrote: > I think even if we provide the per table encryption we can have > encryption keys per tablepaces. That is, all tables on the same > tablespace encryption use the same encryption keys but user can > control encrypted objects per tables. > >> Will we add

Re: check_recovery_target_lsn() does a PG_CATCH without a throw

2019-06-20 Thread Peter Eisentraut
On 2019-06-12 13:16, Peter Eisentraut wrote: > I haven't figured out the time zone issue yet, but I guess the solution > might involve moving some of the code from check_recovery_target_time() > to assign_recovery_target_time(). I think that won't work either. What we need to do is postpone the

old_snapshot_threshold vs indexes

2019-06-20 Thread Thomas Munro
Hello, I ran into someone with a system where big queries scanning 8GB+ of all-in-cache data took consistently ~2.5x longer on a primary server than on a replica. Both servers had concurrent activity on them but plenty of spare capacity and similar specs. After some investigation it turned out

Re: Index Skip Scan

2019-06-20 Thread Jesper Pedersen
Hi, On 6/19/19 9:57 AM, Dmitry Dolgov wrote: Here is what I was talking about, POC for an integration with index scan. About using of create_skipscan_unique_path and suggested planner improvements, I hope together with Jesper we can come up with something soon. I made some minor changes, but

Re: UCT (Re: pgsql: Update time zone data files to tzdata release 2019a.)

2019-06-20 Thread Stephen Frost
Greetings, * Tom Lane (t...@sss.pgh.pa.us) wrote: > Also on the topic of process: 48 hours before a wrap deadline is > *particularly* not the time to play fast and loose with this sort of > thing. It'd have been better to wait till after this week's releases, > so there'd at least be time to

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-06-20 Thread Masahiko Sawada
On Tue, Jun 18, 2019 at 5:07 PM shawn wang wrote: > > Masahiko Sawada 于2019年6月17日周一 下午8:30写道: >> >> On Fri, Jun 14, 2019 at 7:41 AM Tomas Vondra >> wrote: >> > I personally find the idea of encrypting tablespaces rather strange. >> > Tablespaces are meant to define hysical location for objects,

Re: POC: Cleaning up orphaned files using undo logs

2019-06-20 Thread Amit Kapila
On Wed, Jun 19, 2019 at 11:35 PM Robert Haas wrote: > > On Tue, Jun 18, 2019 at 2:07 PM Robert Haas wrote: > > On Tue, Jun 18, 2019 at 7:31 AM Amit Kapila wrote: > > > [ new patches ] > > > > I tried writing some code [ to use these patches ]. > > > for (;;) > { > UnpackedUndoRecord *uur =

Re: POC: Cleaning up orphaned files using undo logs

2019-06-20 Thread Amit Kapila
On Thu, Jun 20, 2019 at 2:24 PM Dilip Kumar wrote: > > On Wed, Jun 19, 2019 at 11:35 PM Robert Haas wrote: > > > > On Tue, Jun 18, 2019 at 2:07 PM Robert Haas wrote: > > > On Tue, Jun 18, 2019 at 7:31 AM Amit Kapila > > > wrote: > > > > [ new patches ] > > > > > > I tried writing some code [

Re: Minimal logical decoding on standbys

2019-06-20 Thread Amit Khandekar
I am yet to work on Andres's latest detailed review comments, but I thought before that, I should submit a patch for the below reported issue because I was almost ready with the fix. Now I will start to work on Andres's comments, for which I will reply separately. On Fri, 1 Mar 2019 at 13:33,

Re: POC: Cleaning up orphaned files using undo logs

2019-06-20 Thread Dilip Kumar
On Wed, Jun 19, 2019 at 11:35 PM Robert Haas wrote: > > On Tue, Jun 18, 2019 at 2:07 PM Robert Haas wrote: > > On Tue, Jun 18, 2019 at 7:31 AM Amit Kapila wrote: > > > [ new patches ] > > > > I tried writing some code [ to use these patches ]. > > I spent some more time experimenting with this

unlogged sequences

2019-06-20 Thread Peter Eisentraut
The discussion in bug #15631 revealed that serial/identity sequences of temporary tables should really also be temporary (easy), and that serial/identity sequences of unlogged tables should also be unlogged. But there is no support for unlogged sequences, so I looked into that. If you copy the

Re: New vacuum option to do only freezing

2019-06-20 Thread Michael Paquier
On Wed, Jun 19, 2019 at 12:51:41PM -0700, Peter Geoghegan wrote: > On Tue, Jun 18, 2019 at 10:39 PM Michael Paquier wrote: >> +INSERT INTO no_index_cleanup(i, t) VALUES(1, repeat('1234567890',3)); >> Do we really need a string as long as that? > > Specifying EXTERNAL storage might make

Re: POC: Cleaning up orphaned files using undo logs

2019-06-20 Thread Amit Kapila
On Wed, Jun 19, 2019 at 8:25 PM Robert Haas wrote: > > On Wed, Jun 19, 2019 at 2:45 AM Amit Kapila wrote: > > The reason for the same is that currently, the undo worker keep on > > executing the requests if there are any. I think this is good when > > there are different requests, but getting

Re: Race conditions with TAP test for syncrep

2019-06-20 Thread Michael Paquier
On Wed, Jun 19, 2019 at 04:08:44PM -0400, Alvaro Herrera wrote: > Ho ho .. you know what misled me into thinking that that would work? > Just look at the name of the test that failed, "asterisk comes before > another standby name". That doesn't seem to be what the test is > testing! I agree that