Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-08-18 Thread Amit Kapila
On Wed, Aug 19, 2020 at 10:10 AM Amit Kapila wrote: > > On Mon, Aug 17, 2020 at 6:29 PM Dilip Kumar wrote: > > > > > > In last patch v49-0001, there is one issue, Basically, I have called > > BufFileFlush in all the cases. But, ideally, we can not call this if > > the underlying files are delet

Re: Creating a function for exposing memory usage of backend process

2020-08-18 Thread Fujii Masao
On 2020/08/19 9:43, torikoshia wrote: On 2020-08-18 22:54, Fujii Masao wrote: On 2020/08/18 18:41, torikoshia wrote: On 2020-08-17 21:19, Fujii Masao wrote: On 2020/08/17 21:14, Fujii Masao wrote: On 2020-08-07 16:38, Kasahara Tatsuhito wrote: The following review has been posted through

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-08-18 Thread Ashutosh Sharma
On Wed, Aug 19, 2020 at 9:27 AM Masahiko Sawada wrote: > > On Mon, 17 Aug 2020 at 15:05, Ashutosh Sharma wrote: > > > > > pg_force_freeze() can revival a tuple that is already deleted but not > > > vacuumed yet. Therefore, the user might need to reindex indexes after > > > using that function. Fo

Re: Pulling up direct-correlated ANY_SUBLINK

2020-08-18 Thread Andy Fan
On Tue, Sep 17, 2019 at 4:41 PM Richard Guo wrote: > > On Thu, Sep 12, 2019 at 11:35 PM Antonin Houska wrote: > >> Richard Guo wrote: >> >> > On Wed, Sep 11, 2019 at 3:25 PM Antonin Houska >> > wrote: >> > >> > >> > Nevertheless, I don't know how to overcome the problems that I >> > me

RE: New statistics for tuning WAL buffer size

2020-08-18 Thread Masahiro Ikeda
On 2020-08-19 13:49, tsunakawa.ta...@fujitsu.com wrote: From: Masahiro Ikeda If my understanding is correct, we have to measure the performance impact first. Do you know HariBabu is now trying to solve it? If not, I will try to modify patches to apply HEAD. No, he's not doing it anymore. It'

Re: Creating foreign key on partitioned table is too slow

2020-08-18 Thread Amit Langote
Hi Alvaro, On Thu, Aug 6, 2020 at 4:25 PM kato-...@fujitsu.com wrote: > On Wednesday, August 5, 2020 9:43 AM I wrote: > > I'll report the result before the end of August . > > I test v2-0001-build-partdesc-memcxt.patch at 9a9db08ae4 and it is ok. Is this patch meant for HEAD or back-patching? I

RE: New statistics for tuning WAL buffer size

2020-08-18 Thread tsunakawa.ta...@fujitsu.com
From: Masahiro Ikeda > If my understanding is correct, we have to measure the performance > impact first. > Do you know HariBabu is now trying to solve it? If not, I will try to > modify patches to apply HEAD. No, he's not doing it anymore. It'd be great if you could resume it. However, I reco

RE: New statistics for tuning WAL buffer size

2020-08-18 Thread Masahiro Ikeda
On 2020-08-18 16:35, tsunakawa.ta...@fujitsu.com wrote: From: Masahiro Ikeda It's important to provide the metrics for tuning the size of WAL buffers. For now, it's lack of the statistics how often processes wait to write WAL because WAL buffer is full. If those situation are often occurred,

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-08-18 Thread Amit Kapila
On Mon, Aug 17, 2020 at 6:29 PM Dilip Kumar wrote: > > > In last patch v49-0001, there is one issue, Basically, I have called > BufFileFlush in all the cases. But, ideally, we can not call this if > the underlying files are deleted/truncated because those files/blocks > might not exist now. So

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-08-18 Thread Ashutosh Sharma
On Tue, Aug 18, 2020 at 9:44 PM Alvaro Herrera wrote: > > On 2020-Aug-17, Ashutosh Sharma wrote: > > > > + if (heap_force_opt == HEAP_FORCE_KILL) > > > + ItemIdSetDead(itemid); > > > > > > I think that if the page is an all-visible page, we should clear an > > > all-visible

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2020-08-18 Thread Tom Lane
David Rowley writes: > I don't object to making the change. I just object to making it only > to put it back again later when someone else speaks up that they'd > prefer to keep nodes modular and not overload them in obscure ways. > So other input is welcome. Is it too weird to overload SubPlan

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2020-08-18 Thread Pavel Stehule
st 19. 8. 2020 v 5:48 odesílatel David Rowley napsal: > On Tue, 18 Aug 2020 at 21:42, David Rowley wrote: > > > > On Tue, 11 Aug 2020 at 17:44, Andres Freund wrote: > > > > > > Hi, > > > > > > On 2020-08-11 17:23:42 +1200, David Rowley wrote: > > > > On Tue, 11 Aug 2020 at 12:21, Andres Freund

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-08-18 Thread Masahiko Sawada
On Mon, 17 Aug 2020 at 15:05, Ashutosh Sharma wrote: > > Hello Masahiko-san, > > Thanks for the review. Please check the comments inline below: > > On Fri, Aug 14, 2020 at 10:07 AM Masahiko Sawada > wrote: > > > Thank you for updating the patch! Here are my comments on v5 patch: > > > > --- a/con

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2020-08-18 Thread David Rowley
On Tue, 18 Aug 2020 at 21:42, David Rowley wrote: > > On Tue, 11 Aug 2020 at 17:44, Andres Freund wrote: > > > > Hi, > > > > On 2020-08-11 17:23:42 +1200, David Rowley wrote: > > > On Tue, 11 Aug 2020 at 12:21, Andres Freund wrote: > > > > > > > > On 2020-07-09 10:25:14 +1200, David Rowley wrote

RE: [PATCH]Fix ja.po error

2020-08-18 Thread Lu, Chenyang
Ping: sorry, did Alvaro and Peter forget this email?( Maybe didn't see this email~ ), I found that the patch of ja.po has not been applied to the Translation Repository. -Original Message- From: Michael Paquier Sent: Wednesday, July 29, 2020 5:25 PM To: Lu, Chenyang/陆 晨阳 Cc: pgsql-hac

Re: [PG13] Planning (time + buffers) data structure in explain plan (format text)

2020-08-18 Thread Justin Pryzby
On Fri, Aug 07, 2020 at 02:30:01PM +0200, Pierre Giraud wrote: > Hi all, > > As far as I understand, in the upcoming version 13, information about > buffers used during planning is now available in the explain plan. > > […] > Planning Time: 0.203 ms >Buffers: shared hit=14 > […] > > For a m

Re: prepared transaction isolation tests

2020-08-18 Thread Andres Freund
Hi, On 2020-08-18 22:24:20 -0400, Tom Lane wrote: > Andres Freund writes: > > ISTM we should just add an alternative output for disabled prepared > > xacts, and re-add the test? > > I believe the buildfarm runs the isolation step with "make installcheck", > so if you're hoping to get buildfarm c

Re: prepared transaction isolation tests

2020-08-18 Thread Tom Lane
Andres Freund writes: > ISTM we should just add an alternative output for disabled prepared > xacts, and re-add the test? I believe the buildfarm runs the isolation step with "make installcheck", so if you're hoping to get buildfarm coverage that way, you're mistaken. Having said that, it'd prob

prepared transaction isolation tests

2020-08-18 Thread Andres Freund
Hi, This thread started on committers, at https://www.postgresql.org/message-id/20200818234532.uiafo5br5lo6zhya%40alap3.anarazel.de In it I wanted to add a isolation test around prepared transactions: On 2020-08-18 16:45:32 -0700, Andres Freund wrote: > I think it's worth adding an isolation tes

Re: Implementing Incremental View Maintenance

2020-08-18 Thread Tatsuo Ishii
I have looked into this. > Hi, > > Attached is the rebased patch (v16) to add support for Incremental > Materialized View Maintenance (IVM). It is able to be applied to > current latest master branch. > > This also includes the following small fixes: > > - Add a query check for expressions cont

Re: Creating a function for exposing memory usage of backend process

2020-08-18 Thread torikoshia
On 2020-08-18 22:54, Fujii Masao wrote: On 2020/08/18 18:41, torikoshia wrote: On 2020-08-17 21:19, Fujii Masao wrote: On 2020/08/17 21:14, Fujii Masao wrote: On 2020-08-07 16:38, Kasahara Tatsuhito wrote: The following review has been posted through the commitfest application: make installc

RE: please update ps display for recovery checkpoint

2020-08-18 Thread k.jami...@fujitsu.com
On Wednesday, August 19, 2020 7:53 AM (GMT+9), Justin Pryzby wrote: Hi, All the patches apply, although when applying them the following appears: (Stripping trailing CRs from patch; use --binary to disable.) > During crash recovery, the server writes this to log: > > < 2020-08-16 08:46:08.

please update ps display for recovery checkpoint

2020-08-18 Thread Justin Pryzby
During crash recovery, the server writes this to log: < 2020-08-16 08:46:08.601 -03 >LOG: redo done at 2299C/1EC6BA00 < 2020-08-16 08:46:08.877 -03 >LOG: checkpoint starting: end-of-recovery immediate But runs a checkpoint, which can take a long time, while the "ps" display still says "recov

Re: Print logical WAL message content

2020-08-18 Thread Alvaro Herrera
On 2020-Aug-18, Ashutosh Bapat wrote: > Right now pg_waldump just prints whether the message is transactional > or not and its size. That doesn't help much to understand the message > itself. If it prints the contents of a logical WAL message, it helps > debugging logical replication related probl

Re: BUG #16583: merge join on tables with different DB collation behind postgres_fdw fails

2020-08-18 Thread Tom Lane
I wrote: >> So I think what is happening here is that postgres_fdw's version of >> IMPORT FOREIGN SCHEMA translates "COLLATE default" on the remote >> server to "COLLATE default" on the local one, which of course is >> a big fail if the defaults don't match. That allows the local >> planner to bel

Re: proposal: enhancing plpgsql debug API - returns text value of variable content

2020-08-18 Thread Pavel Stehule
po 17. 8. 2020 v 8:40 odesílatel Pavel Stehule napsal: > Hi > > I am working on tracing support to plpgsql_check > > https://github.com/okbob/plpgsql_check > > I would like to print content of variables - and now, I have to go some > deeper than I would like. I need to separate between scalar, ro

Re: doc review for v13

2020-08-18 Thread Justin Pryzby
I stand by these changes which I proposed handful of times since April, but not yet included by Michael's previous commits. -- Justin >From f029efd79c4ad14ae003ed1a1c692931cdc33f1e Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Mon, 30 Mar 2020 19:43:22 -0500 Subject: [PATCH v6 01/10] doc: b

Re: use pg_get_functiondef() in pg_dump

2020-08-18 Thread Tom Lane
Stephen Frost writes: > So.. the code that's in pg_dump today works to go from "whatever the > connected server's version is" to "whatever the version is of the > pg_dump command itself". If we had the code in libpgcommon, and > functions in the backend to get at it along with psql having that c

Re: Nicer error when connecting to standby with hot_standby=off

2020-08-18 Thread Fujii Masao
On 2020/08/01 5:18, James Coleman wrote: On Wed, Jul 29, 2020 at 11:24 AM Fujii Masao wrote: On 2020/04/03 22:49, James Coleman wrote: On Thu, Apr 2, 2020 at 5:53 PM David Zhang wrote: The following review has been posted through the commitfest application: make installcheck-world: n

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-08-18 Thread Alvaro Herrera
On 2020-Aug-17, Ashutosh Sharma wrote: > > + if (heap_force_opt == HEAP_FORCE_KILL) > > + ItemIdSetDead(itemid); > > > > I think that if the page is an all-visible page, we should clear an > > all-visible bit on the visibility map corresponding to the page and > > PD_ALL_VI

Re: Yet another fast GiST build

2020-08-18 Thread Pavel Borisov
The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, passed Spec compliant: not tested Documentation:not tested I consider this patch almost ready for commit with minor corrections

Re: track_planning causing performance regression

2020-08-18 Thread Hamid Akhtar
On Tue, Aug 18, 2020 at 8:43 PM Fujii Masao wrote: > > > Yes, I pushed the document_overhead_by_track_planning.patch, but this > > CF entry is for pgss_lwlock_v1.patch which replaces spinlocks with > lwlocks > > in pg_stat_statements. The latter patch has not been committed yet. > > Probably atta

Re: track_planning causing performance regression

2020-08-18 Thread Fujii Masao
Yes, I pushed the document_overhead_by_track_planning.patch, but this CF entry is for pgss_lwlock_v1.patch which replaces spinlocks with lwlocks in pg_stat_statements. The latter patch has not been committed yet. Probably attachding the different patches in the same thread would cause this confu

Re: [patch] demote

2020-08-18 Thread Jehan-Guillaume de Rorthais
Hi, Please find in attachment v5 of the patch set rebased on master after various conflicts. Regards, On Wed, 5 Aug 2020 00:04:53 +0200 Jehan-Guillaume de Rorthais wrote: > Demote now keeps backends with no active xid alive. Smart mode keeps all > backends: it waits for them to finish their xa

Re: EDB builds Postgres 13 with an obsolete ICU version

2020-08-18 Thread Bruce Momjian
On Tue, Aug 18, 2020 at 11:38:38AM +0200, Magnus Hagander wrote: > On Tue, Aug 18, 2020 at 11:24 AM Dave Page wrote: > Not necessarily. It's likely that not all indexes use ICU collations, and > you still save time loading what may be large amounts of data. > > I agree though, that it

Re: psql: add \si, \sm, \st and \sr functions to show CREATE commands for indexes, matviews, triggers and tables

2020-08-18 Thread Tom Lane
a.pervush...@postgrespro.ru writes: > [ si_st_sm_sr_v2.patch ] I hadn't particularly noticed this thread before, but I happened to look through this patch, and I've got to say that this proposed feature seems like an absolute disaster from a maintenance standpoint. There will be no value in an \s

Re: Creating a function for exposing memory usage of backend process

2020-08-18 Thread Fujii Masao
On 2020/08/18 18:41, torikoshia wrote: On 2020-08-17 21:19, Fujii Masao wrote: On 2020/08/17 21:14, Fujii Masao wrote: On 2020-08-07 16:38, Kasahara Tatsuhito wrote: The following review has been posted through the commitfest application: make installcheck-world:  tested, passed Implements

Re: INSERT INTO SELECT, Why Parallelism is not selected?

2020-08-18 Thread Amit Kapila
On Tue, Aug 18, 2020 at 1:37 PM Bharath Rupireddy wrote: > > On Tue, Jul 14, 2020 at 1:20 PM Dilip Kumar wrote: > > > > On Mon, Jul 13, 2020 at 4:23 PM Amit Kapila wrote: > > > > > I think we can do more than this by > > > parallelizing the Insert part of this query as well as we have lifted > >

Re: use pg_get_functiondef() in pg_dump

2020-08-18 Thread Stephen Frost
Greetings, * Corey Huinker (corey.huin...@gmail.com) wrote: > > I'm sure there's a lot of folks who'd like to see more of the logic we > > have in pg_dump for building objects from the catalog available to more > > tools through libpgcommon- psql being one of the absolute first > > use-cases for e

Documentation patch for backup manifests in protocol.sgml

2020-08-18 Thread Bernd Helmle
Hi, protocol.sgml describes the protocol messages received by a BASE_BACKUP streaming command, but doesn't tell anything about the additional CopyResponse data message containing the contents of the backup manifest (if requested) after having received the tar files. So i propose the attached to gi

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-08-18 Thread Rajkumar Raghuwanshi
Thanks for suggestion Ashutosh, I have done testing around these suggestion and found no issues. I will continue testing same with updated patch posted on this thread. On Fri, Aug 7, 2020 at 12:45 PM Ashutosh Sharma wrote: > Thanks Rajkumar for testing the patch. > > Here are some of the additio

Re: jsonb, collection & postgres_fdw

2020-08-18 Thread Ashutosh Bapat
On Mon, Aug 17, 2020 at 7:32 PM Bharath Rupireddy wrote: > > On Fri, Aug 14, 2020 at 12:46 PM Konstantin Knizhnik > wrote: > > > > Right now postgres_fdw treat as shippable only builtin functions or > > functions from extensions explicitly specified as shippable extensions > > in parameters of t

Re: Libpq support to connect to standby server as priority

2020-08-18 Thread Greg Nancarrow
Hi Peter, I have updated the patch (attached) based on your comments, with adjustments made for additional changes based on feedback (which I tend to agree with) from Robert Haas and Tsunakawa san, who suggested read-write/read-only should be functionally different to primary/standby, and not just

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-08-18 Thread Chris Travers
On Tue, Jul 14, 2020 at 12:28 AM Peter Geoghegan wrote: > On Mon, Jul 13, 2020 at 2:12 PM Robert Haas wrote: > > 1. There's nothing to identify the tuple that has the problem, and no > > way to know how many more of them there might be. Back-patching > > b61d161c146328ae6ba9ed937862d66e5c8b035a

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-08-18 Thread Ashutosh Sharma
Attached is the new version of patch that addresses the comments from Asim Praveen and Masahiko-san. It also improves the documentation to some extent. On Tue, Aug 18, 2020 at 1:46 PM Ashutosh Sharma wrote: > > Hello Masahiko-san, > > I've spent some more time trying to understand the code in >

Re: [HACKERS] make async slave to wait for lsn to be replayed

2020-08-18 Thread Anna Akenteva
On 2020-07-13 14:21, Daniel Gustafsson wrote: This patch require some rewording of documentation/comments and variable names after the language change introduced by 229f8c219f8f..a9a4a7ad565b, the thread below can be used as reference for how to change: https://www.postgresql.org/message-id/fl

Re: EDB builds Postgres 13 with an obsolete ICU version

2020-08-18 Thread Julien Rouhaud
On Tue, Aug 18, 2020 at 11:39 AM Magnus Hagander wrote: > > On Tue, Aug 18, 2020 at 11:24 AM Dave Page wrote: >> >> On Mon, Aug 17, 2020 at 7:23 PM Bruce Momjian wrote: >>> >>> On Mon, Aug 17, 2020 at 04:55:13PM +0100, Dave Page wrote: >> I wasn't aware that ICU had the concept of collation vers

Re: EDB builds Postgres 13 with an obsolete ICU version

2020-08-18 Thread Thomas Kellerer
Magnus Hagander schrieb am 18.08.2020 um 11:38: > It might be a slightly larger percentage on Windows who use it, but > I'm willing to bet it's still quite low. I have seen increasingly more questions around ICU collations on Windows due to the fact that people that migrate from SQL Server to Pos

Re: Implementing Incremental View Maintenance

2020-08-18 Thread Yugo NAGATA
Hi, Attached is the rebased patch (v16) to add support for Incremental Materialized View Maintenance (IVM). It is able to be applied to current latest master branch. This also includes the following small fixes: - Add a query check for expressions containing aggregates in it - [doc] Add descript

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2020-08-18 Thread David Rowley
On Tue, 11 Aug 2020 at 17:44, Andres Freund wrote: > > Hi, > > On 2020-08-11 17:23:42 +1200, David Rowley wrote: > > On Tue, 11 Aug 2020 at 12:21, Andres Freund wrote: > > > > > > On 2020-07-09 10:25:14 +1200, David Rowley wrote: > > > > On Thu, 9 Jul 2020 at 04:53, Andres Freund wrote: > > > >

Re: Creating a function for exposing memory usage of backend process

2020-08-18 Thread torikoshia
On 2020-08-17 21:19, Fujii Masao wrote: On 2020/08/17 21:14, Fujii Masao wrote: On 2020-08-07 16:38, Kasahara Tatsuhito wrote: The following review has been posted through the commitfest application: make installcheck-world:  tested, passed Implements feature:   tested, passed Spec complia

Re: EDB builds Postgres 13 with an obsolete ICU version

2020-08-18 Thread Magnus Hagander
On Tue, Aug 18, 2020 at 11:24 AM Dave Page wrote: > > > On Mon, Aug 17, 2020 at 7:23 PM Bruce Momjian wrote: > >> On Mon, Aug 17, 2020 at 04:55:13PM +0100, Dave Page wrote: >> > That was more if the installer actually handles the whole chain. It >> clearly >> > doesn't today (since it do

Re: EDB builds Postgres 13 with an obsolete ICU version

2020-08-18 Thread Dave Page
On Mon, Aug 17, 2020 at 7:23 PM Bruce Momjian wrote: > On Mon, Aug 17, 2020 at 04:55:13PM +0100, Dave Page wrote: > > That was more if the installer actually handles the whole chain. It > clearly > > doesn't today (since it doesn't support upgrades), I agree this might > > definitely

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-08-18 Thread Ashutosh Sharma
Hello Masahiko-san, I've spent some more time trying to understand the code in lazy_scan_heap function to know under what all circumstances a VACUUM can fail with "found xmin ... before relfrozenxid ..." error for a tuple whose xmin is behind relfrozenxid. Here are my observations: 1) It can fail

Re: INSERT INTO SELECT, Why Parallelism is not selected?

2020-08-18 Thread Bharath Rupireddy
On Tue, Jul 14, 2020 at 1:20 PM Dilip Kumar wrote: > > On Mon, Jul 13, 2020 at 4:23 PM Amit Kapila wrote: > > > I think we can do more than this by > > parallelizing the Insert part of this query as well as we have lifted > > group locking restrictions related to RelationExtension and Page lock >

Re: Is it worth accepting multiple CRLs?

2020-08-18 Thread Kyotaro Horiguchi
Hello. At Sat, 15 Aug 2020 13:18:22 -0400, Stephen Frost wrote in > > Looking closer I realized that certificates are verified in each > > backend so CRL cache doesn't work at all for the hashed directory > > method. Therefore, all CRL files relevant to a certificate to be > > verfied are loaded

Re: Display individual query in pg_stat_activity

2020-08-18 Thread Pavel Stehule
Hi út 18. 8. 2020 v 8:54 odesílatel Masahiro Ikeda napsal: > Hi, > > > I've attached a patch to display individual query in the > > pg_stat_activity query field when multiple SQL statements are > > currently displayed. > > > > Motivation: > > > > When multiple statements are displayed then we do

RE: New statistics for tuning WAL buffer size

2020-08-18 Thread tsunakawa.ta...@fujitsu.com
From: Masahiro Ikeda > It's important to provide the metrics for tuning the size of WAL buffers. > For now, it's lack of the statistics how often processes wait to write WAL > because WAL buffer is full. > > If those situation are often occurred, WAL buffer is too small for the > workload. > DBA

New statistics for tuning WAL buffer size

2020-08-18 Thread Masahiro Ikeda
Hi, It's important to provide the metrics for tuning the size of WAL buffers. For now, it's lack of the statistics how often processes wait to write WAL because WAL buffer is full. If those situation are often occurred, WAL buffer is too small for the workload. DBAs must to tune the WAL buf