Re: pendingOps table is not cleared with fsync=off

2020-05-13 Thread Heikki Linnakangas
On 09/05/2020 02:53, Thomas Munro wrote: On Sat, May 9, 2020 at 9:21 AM Heikki Linnakangas wrote: I noticed that commit 3eb77eba5a changed the logic in ProcessSyncRequests() (formerly mdsync()) so that if you have fsync=off, the entries are not removed from the pendingOps hash table. I don't

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-05-13 Thread Amit Kapila
On Wed, May 13, 2020 at 9:16 PM Dilip Kumar wrote: > > On Wed, May 13, 2020 at 4:50 PM Amit Kapila wrote: > > > > > > > > 4. > > > > +static void > > > > +stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn) > > > > +{ > > > > + LogicalDecodingContext *ctx = cache->private_data;

Re: PG 13 release notes, first draft

2020-05-13 Thread Fabien COELHO
Hello Bruce, * 34a0a81bfb We already have: Reformat tables containing function information for better clarity (Tom Lane) so it seems it is covered as part of this. AFAICR this one is not by the same author, and although the point was about better clarity, it was not

Re: MultiXact\SLRU buffers configuration

2020-05-13 Thread Andrey M. Borodin
> 14 мая 2020 г., в 06:25, Kyotaro Horiguchi > написал(а): > > At Wed, 13 May 2020 23:08:37 +0500, "Andrey M. Borodin" > wrote in >> >> >>> 11 мая 2020 г., в 16:17, Andrey M. Borodin >>> написал(а): >>> >>> I've went ahead and created 3 patches: >>> 1. Configurable SLRU buffer sizes

Re: pg13: xlogreader API adjust

2020-05-13 Thread Kyotaro Horiguchi
At Thu, 14 May 2020 01:03:48 +, "Bossart, Nathan" wrote in > I think I've discovered a problem with 850196b6. The following steps > can be used to trigger a segfault: > > # wal_level = logical > psql postgres -c "create database testdb;" > psql testdb -c "select

Re: making update/delete of inheritance trees scale better

2020-05-13 Thread Amit Langote
On Thu, May 14, 2020 at 7:55 AM David Rowley wrote: > On Wed, 13 May 2020 at 19:02, Amit Langote wrote: > > > As for which ResultRelInfos to initialize, couldn't we just have the > > > planner generate an OidList of all the ones that we could need. > > > Basically, all the non-pruned partitions.

Re: ldap tls test fails in some environments

2020-05-13 Thread Thomas Munro
On Thu, May 14, 2020 at 4:05 AM Christoph Berg wrote: > Some other problem emerged here in the ldap test: Hi Christoph, > 17:28:59 Data directory: > /<>/build/src/test/ldap/tmp_check/t_001_auth_node_data/pgdata I know nothing about Debian package building so I could be missing something about

Re: Another modest proposal for docs formatting: catalog descriptions

2020-05-13 Thread Tom Lane
"Jonathan S. Katz" writes: > Time slipped a bit (sorry!), but while prepping for the release I > reviewed this. Visually, it looks WAY better. The code checks out too. I > think any tweaks would be primarily around personal preference on the > UI, so it may be better just to commit and get it out

Re: [PATCH] Keeps tracking the uniqueness with UniqueKey

2020-05-13 Thread Andy Fan
On Wed, May 13, 2020 at 11:48 PM Andy Fan wrote: > >> My impression about the one row stuff, is that there is too much >> special casing around it. We should somehow structure the UniqueKey >> data so that one row unique keys come naturally rather than special >> cased. E.g every column in such

Re: tablespace_map code cleanup

2020-05-13 Thread Kyotaro Horiguchi
At Wed, 13 May 2020 15:10:30 -0400, Robert Haas wrote in > On Tue, May 12, 2020 at 10:26 PM Amit Kapila wrote: > > I was trying to say that tablespace listing will happen under > > PROGRESS_BASEBACKUP_PHASE_WAIT_CHECKPOINT phase which could be a > > problem if it is a costly operation but as

Re: PG 13 release notes, first draft

2020-05-13 Thread Bruce Momjian
On Thu, May 14, 2020 at 09:51:41AM +0900, Kyotaro Horiguchi wrote: > At Wed, 13 May 2020 11:15:18 -0400, Bruce Momjian wrote in > > On Wed, May 13, 2020 at 11:56:33AM +0900, Kyotaro Horiguchi wrote: > It is just an more accurate (not an detailed) version of the > previously proposed description.

Re: [PATCH] Keeps tracking the uniqueness with UniqueKey

2020-05-13 Thread Andy Fan
On Thu, May 14, 2020 at 6:20 AM David Rowley wrote: > On Thu, 14 May 2020 at 03:48, Andy Fan wrote: > > On Wed, May 13, 2020 at 8:04 PM Ashutosh Bapat < > ashutosh.bapat@gmail.com> wrote: > >> My impression about the one row stuff, is that there is too much > >> special casing around it. We

Re: Another modest proposal for docs formatting: catalog descriptions

2020-05-13 Thread Jonathan S. Katz
On 5/10/20 2:03 PM, Jonathan S. Katz wrote: > On 5/10/20 12:27 PM, Tom Lane wrote: >> Just FTR, here's a complete patch for this. > > Cool. I'll play around with it tonight once I clear out release work. > Per upthread reference, I believe you've become a CSS maven yourself. Time slipped a bit

Re: SLRU statistics

2020-05-13 Thread Tom Lane
Fujii Masao writes: > On 2020/05/14 2:44, Tom Lane wrote: >> I got through check-world with the assertion shown that we are not >> counting any SLRU operations in the postmaster. Don't know if we >> want to commit that or not --- any thoughts? > +1 to add this assertion because basically it's

Re: refactoring basebackup.c

2020-05-13 Thread Suraj Kharage
Hi, On Wed, May 13, 2020 at 7:49 PM Robert Haas wrote: > > So the patch came out slightly faster at 8kB and slightly slower in the > other tests. That's kinda strange. I wonder if it's just noise. How much do > the results vary run to run? > It is not varying much except for 8kB run. Please see

Re: SLRU statistics

2020-05-13 Thread Fujii Masao
On 2020/05/14 2:44, Tom Lane wrote: I wrote: (IIRC only the Async module is doing that.) Hm, maybe we can fix that. Yeah, it's quite easy to make async.c postpone its first write to the async SLRU. This seems like a win all around, because many installations don't use NOTIFY and so

Re: Strange decreasing value of pg_last_wal_receive_lsn()

2020-05-13 Thread godjan •
-> Why do you kill -9 your standby? Hi, it’s Jepsen test for our HA solution. It checks that we don’t lose data in such situation. So, now we update logic as Michael said. All ha alive standbys now waiting for replaying all WAL that they have and after we use pg_last_replay_lsn() to choose

Re: Our naming of wait events is a disaster.

2020-05-13 Thread Tom Lane
The attached patch doesn't actually change any LWLock names, but it is a useful start on that project. What it does is to get rid of the current scheme of dynamically registering the names of built-in LWLocks in favor of having a constant array of those names. It's completely foolish to expend

Re: MultiXact\SLRU buffers configuration

2020-05-13 Thread Kyotaro Horiguchi
At Wed, 13 May 2020 23:08:37 +0500, "Andrey M. Borodin" wrote in > > > > 11 мая 2020 г., в 16:17, Andrey M. Borodin > > написал(а): > > > > I've went ahead and created 3 patches: > > 1. Configurable SLRU buffer sizes for MultiXacOffsets and MultiXactMembers > > 2. Reduce locking level to

Re: JSON output from psql

2020-05-13 Thread Gurjeet Singh
On Wed, May 13, 2020 at 2:01 PM Chapman Flack wrote: > > Seems like a lot of work just to get json-shaped query results from psql, +1. If we look at the amount of work needed for the hybrid approach you describe, compared to running CSV result through something like csv2json, there's a 100%

Re: pg13: xlogreader API adjust

2020-05-13 Thread Bossart, Nathan
I think I've discovered a problem with 850196b6. The following steps can be used to trigger a segfault: # wal_level = logical psql postgres -c "create database testdb;" psql testdb -c "select pg_create_logical_replication_slot('slot', 'test_decoding');" psql

Re: PG 13 release notes, first draft

2020-05-13 Thread Kyotaro Horiguchi
At Wed, 13 May 2020 11:15:18 -0400, Bruce Momjian wrote in > On Wed, May 13, 2020 at 11:56:33AM +0900, Kyotaro Horiguchi wrote: > > > > > > Allow skipping of WAL for new tables and indexes if wal_level is > > > 'minimal' (Kyotaro Horiguchi) > > > > > > Relations larger than

Re: new heapcheck contrib module

2020-05-13 Thread Peter Geoghegan
On Wed, May 13, 2020 at 5:18 PM Mark Dilger wrote: > I am not removing any assertions. I do not propose to remove any assertions. > When I talk about "hardening against assertions", that is not in any way a > proposal to remove assertions from the code. I'm sorry if I seemed to suggest that

Re: new heapcheck contrib module

2020-05-13 Thread Mark Dilger
> On May 13, 2020, at 3:29 PM, Peter Geoghegan wrote: > > On Wed, May 13, 2020 at 3:10 PM Alvaro Herrera > wrote: >> Hmm. I think we should (try to?) write code that avoids all crashes >> with production builds, but not extend that to assertion failures. > > Assertions are only a problem

Re: new heapcheck contrib module

2020-05-13 Thread Peter Geoghegan
On Wed, May 13, 2020 at 4:32 PM Alvaro Herrera wrote: > I think my initial approach for this would be to use a fuzzing tool that > generates data blocks semi-randomly, then uses them as Postgres data > pages somehow, and see what happens -- examine any resulting crashes and > make individual

Re: pg13: xlogreader API adjust

2020-05-13 Thread Alvaro Herrera
On 2020-May-13, Tom Lane wrote: > Alvaro Herrera writes: > > Pushed. Thanks for the help! > > This seems to have fixed bowerbird. Were you expecting that? Hm, not really. -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA,

Re: new heapcheck contrib module

2020-05-13 Thread Alvaro Herrera
On 2020-May-13, Peter Geoghegan wrote: > On Wed, May 13, 2020 at 3:10 PM Alvaro Herrera > wrote: > > Hmm. I think we should (try to?) write code that avoids all crashes > > with production builds, but not extend that to assertion failures. > > Assertions are only a problem at all because Mark

Re: pg13: xlogreader API adjust

2020-05-13 Thread Tom Lane
Alvaro Herrera writes: > Pushed. Thanks for the help! This seems to have fixed bowerbird. Were you expecting that? regards, tom lane

Re: Concurrency bug in amcheck

2020-05-13 Thread Peter Geoghegan
On Mon, May 11, 2020 at 5:56 AM Alexander Korotkov wrote: > Thank you. I've worked a bit on comments and commit message. I would > appreciate you review. This looks good to me. > > I like your idea of making the primary consistent with the REDO > > routine on the master branch only. I wonder

Re: gcov coverage data not full with immediate stop

2020-05-13 Thread Peter Geoghegan
On Tue, May 12, 2020 at 10:10 AM Tom Lane wrote: > > Can that easily be accommodated? > > There's no real lead time needed AFAICS: when we are ready to branch, > we can just do it. So sure, let's wait till the end of May to decide. > If things look bad then, we could reconsider again mid-June.

Re: making update/delete of inheritance trees scale better

2020-05-13 Thread David Rowley
On Wed, 13 May 2020 at 19:02, Amit Langote wrote: > > As for which ResultRelInfos to initialize, couldn't we just have the > > planner generate an OidList of all the ones that we could need. > > Basically, all the non-pruned partitions. > > Why would replacing list of RT indexes by OIDs be

Re: Index Skip Scan

2020-05-13 Thread Peter Geoghegan
On Mon, Jan 20, 2020 at 5:05 PM Peter Geoghegan wrote: > You can add another assertion that calls a new utility function in > bufmgr.c. That can use the same logic as this existing assertion in > FlushOneBuffer(): > > Assert(LWLockHeldByMe(BufferDescriptorGetContentLock(bufHdr))); > > We haven't

Re: new heapcheck contrib module

2020-05-13 Thread Peter Geoghegan
On Wed, May 13, 2020 at 3:10 PM Alvaro Herrera wrote: > Hmm. I think we should (try to?) write code that avoids all crashes > with production builds, but not extend that to assertion failures. Assertions are only a problem at all because Mark would like to write tests that involve a selection

Re: pgstat_read_statsfiles() and reset timestamp

2020-05-13 Thread Tom Lane
Fujii Masao writes: > pgstat_read_statsfiles() sets each stat_reset_timestamp to > the current timestamp, at the beginning of the function, > just in case we fail to load the existing statsfile. This code is > originally introduced by commit 4c468b37a2. > But commit ad1b5c842b changed

Re: [PATCH] Keeps tracking the uniqueness with UniqueKey

2020-05-13 Thread David Rowley
On Thu, 14 May 2020 at 03:48, Andy Fan wrote: > On Wed, May 13, 2020 at 8:04 PM Ashutosh Bapat > wrote: >> My impression about the one row stuff, is that there is too much >> special casing around it. We should somehow structure the UniqueKey >> data so that one row unique keys come naturally

Re: new heapcheck contrib module

2020-05-13 Thread Alvaro Herrera
On 2020-May-12, Peter Geoghegan wrote: > > The point is that when checking the table for corruption I avoid > > calling anything that might assert (or segfault, or whatever). > > I don't think that you can expect to avoid assertion failures in > general. Hmm. I think we should (try to?) write

Re: new heapcheck contrib module

2020-05-13 Thread Peter Geoghegan
On Wed, May 13, 2020 at 12:22 PM Robert Haas wrote: > I think this is a good summary of the problems in this area. On the > one hand, I think it's hideous that we sanity check user input to > death, but blindly trust the bytes on disk to the point of seg > faulting if they're wrong. The idea that

Re: Our naming of wait events is a disaster.

2020-05-13 Thread Tom Lane
I looked through the naming situation for LWLocks, and SLRUs which turn out to be intimately connected to that, because most of the dubious LWLock names we're currently exposing actually are derived from SLRUs. Here's some ideas (not quite a full proposal yet) for changing that. Note that because

document? Re: Do I understand commit timestamps correctly?

2020-05-13 Thread Chapman Flack
On 03/23/18 11:29, Chapman Flack wrote: > Can somebody confirm or correct what I (think I)'ve gleaned from > the code? > ... > - The extra machinery turned on by track_commit_timestamp maintains > a cache of recent ones so they can be efficiently queried from SQL > in normal operation. I

Re: JSON output from psql

2020-05-13 Thread Chapman Flack
On 05/13/20 16:16, Gurjeet Singh wrote: > On Wed, May 13, 2020 at 1:14 PM Gurjeet Singh wrote: >> >> Arguably, delivering JSON (with its repeating attribute names in every >> element of the array, dquotes and commas) is more network intensive >> than converting the resultset to JSON on [client]

Re: COPY, lock release and MVCC

2020-05-13 Thread Laurenz Albe
On Wed, 2020-05-13 at 19:29 +0530, Amit Kapila wrote: > > > > > On Fri, May 8, 2020 at 4:58 AM Laurenz Albe > > > > > wrote: > > > > > > I happened to notice that COPY TO releases the ACCESS SHARE lock > > > > > > on the table right when the command ends rather than holding it > > > > > > until

Re: JSON output from psql

2020-05-13 Thread Gurjeet Singh
On Wed, May 13, 2020 at 1:14 PM Gurjeet Singh wrote: > > Arguably, delivering JSON (with its repeating attribute names in every > element of the array, dquotes and commas) is more network intensive > than converting the resultset to JSON on network side. s/network side/client side/ Best

Re: JSON output from psql

2020-05-13 Thread Gurjeet Singh
On Wed, May 13, 2020 at 12:50 PM Robert Haas wrote: > > On Mon, May 11, 2020 at 4:42 PM Gurjeet Singh wrote: > > That's a good point! It might still be desirable, perhaps for performance > > trade-off of JSON conversion on the client-side instead of on the > > server-side. > > If there's a

Re: SLRU statistics

2020-05-13 Thread Ranier Vilela
Em qua., 13 de mai. de 2020 às 11:46, Fujii Masao < masao.fu...@oss.nttdata.com> escreveu: > > > On 2020/05/13 23:26, Tom Lane wrote: > > Fujii Masao writes: > >> On 2020/05/13 17:21, Tomas Vondra wrote: > >>> On Wed, May 13, 2020 at 04:10:30PM +0900, Fujii Masao wrote: > Also I found

Re: JSON output from psql

2020-05-13 Thread Robert Haas
On Mon, May 11, 2020 at 4:42 PM Gurjeet Singh wrote: > That's a good point! It might still be desirable, perhaps for performance > trade-off of JSON conversion on the client-side instead of on the server-side. If there's a performance problem with the server's code here, we should probably try

Re: Add "-Wimplicit-fallthrough" to default flags

2020-05-13 Thread Alvaro Herrera
On 2020-May-13, Alvaro Herrera wrote: > On 2020-May-13, Tom Lane wrote: > > Let's just go to level 3 overall (and revert the changes you made for > > level 4 compliance --- they're more likely to cause back-patching > > pain than do anything useful). > > Ok. And done. -- Álvaro Herrera

Re: new heapcheck contrib module

2020-05-13 Thread Robert Haas
On Tue, May 12, 2020 at 11:06 PM Peter Geoghegan wrote: > try/catch seems like the way to do it. Not all amcheck errors come > from amcheck -- some are things that the backend code does, that are > known to appear in amcheck from time to time. I'm thinking in > particular of the >

Re: tablespace_map code cleanup

2020-05-13 Thread Robert Haas
On Tue, May 12, 2020 at 10:26 PM Amit Kapila wrote: > I was trying to say that tablespace listing will happen under > PROGRESS_BASEBACKUP_PHASE_WAIT_CHECKPOINT phase which could be a > problem if it is a costly operation but as you said it is pretty cheap > so I think we don't need to bother

Re: Parallel copy

2020-05-13 Thread Robert Haas
On Tue, May 12, 2020 at 1:01 AM Amit Kapila wrote: > I don't understand why we need to do something special for combo CIDs > if they are not generated during this operation? Hmm. Well I guess if they're not being generated then we don't need to do anything about them, but I still think we should

Re: MultiXact\SLRU buffers configuration

2020-05-13 Thread Andrey M. Borodin
> 11 мая 2020 г., в 16:17, Andrey M. Borodin написал(а): > > I've went ahead and created 3 patches: > 1. Configurable SLRU buffer sizes for MultiXacOffsets and MultiXactMembers > 2. Reduce locking level to shared on read of MultiXactId members > 3. Configurable cache size I'm looking more at

Re: SLRU statistics

2020-05-13 Thread Tom Lane
I wrote: >>> (IIRC only the Async module is doing that.) > Hm, maybe we can fix that. Yeah, it's quite easy to make async.c postpone its first write to the async SLRU. This seems like a win all around, because many installations don't use NOTIFY and so will never need to do that work at all.

Re: Is it useful to record whether plans are generic or custom?

2020-05-13 Thread legrand legrand
Hello, yes this can be usefull, under the condition of differentiating all the counters for a queryid using a generic plan and the one using a custom one. For me one way to do that is adding a generic_plan column to pg_stat_statements key, someting like: - userid, - dbid, - queryid, -

Re: SLRU statistics

2020-05-13 Thread Tom Lane
Fujii Masao writes: > But since the counter that postmaster incremented is propagated to > child processes via fork, it should be zeroed at postmaster or the > beginning of child process? Otherwise that counter always starts > with non-zero in child process. Yes, if the postmaster is

Re: SLRU statistics

2020-05-13 Thread Fujii Masao
On 2020/05/14 1:14, Alvaro Herrera wrote: On 2020-May-14, Fujii Masao wrote: So I tried the similar test again and found that postmaster seems to be able to increment the counters unless I'm missing something. For example, frame #2: 0x00010d93845f

Re: pg13: xlogreader API adjust

2020-05-13 Thread Alvaro Herrera
Pushed. Thanks for the help! -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: SLRU statistics

2020-05-13 Thread Alvaro Herrera
On 2020-May-14, Fujii Masao wrote: > So I tried the similar test again and found that postmaster seems to be > able to increment the counters unless I'm missing something. > For example, > > frame #2: 0x00010d93845f > postgres`pgstat_count_slru_page_zeroed(ctl=0x00010de27320) at >

Re: SLRU statistics

2020-05-13 Thread Fujii Masao
On 2020/05/14 0:38, Tom Lane wrote: Tomas Vondra writes: I think it counts as a variable with "static storage duration" per 6.7.8 (para 10), see [1]. I wasn't aware of this either, but it probably means the memset is unnecessary. Also, it seems a bit strange/confusing to handle this

ldap tls test fails in some environments

2020-05-13 Thread Christoph Berg
> I just added the extra tests to the postgresql-13 package and got > this: Some other problem emerged here in the ldap test: 17:28:58 17:28:58 # Failed test 'StartTLS' 17:28:58 # at t/001_auth.pl line 169. 17:28:58 # got: '2' 17:28:58 # expected: '0' 17:28:58 17:28:58 #

Re: Add "-Wimplicit-fallthrough" to default flags

2020-05-13 Thread Alvaro Herrera
On 2020-May-13, Tom Lane wrote: > Alvaro Herrera writes: > >> I fear that is going to mean that we revert this patch. > > > Or we can filter-out the -Wimplicit-fallthrough, or change to level 3, > > for bison-emitted files. > > Let's just go to level 3 overall (and revert the changes you made

Re: [PATCH] Keeps tracking the uniqueness with UniqueKey

2020-05-13 Thread Andy Fan
Hi Ashutosh: All your suggestions are followed except the UNION ALL one. I replied the reason below. For the suggestions about partitioned table, looks lot of cases to handle, so I summarize/enrich your idea in README and email thread, we can continue to talk about that. >> + >> +

Re: [PATCH] Keeps tracking the uniqueness with UniqueKey

2020-05-13 Thread Andy Fan
On Wed, May 13, 2020 at 8:04 PM Ashutosh Bapat wrote: > On Fri, May 8, 2020 at 7:27 AM Andy Fan wrote: > > >> +else if (inner_is_onerow) > >> +{ > >> +/* Since rows in innerrel can't be duplicated AND if > >> innerrel is onerow, > >> + *

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-05-13 Thread Dilip Kumar
On Wed, May 13, 2020 at 4:50 PM Amit Kapila wrote: > > On Wed, May 13, 2020 at 11:35 AM Dilip Kumar wrote: > > > > On Tue, May 12, 2020 at 4:39 PM Amit Kapila wrote: > > > > > > > > > v20-0003-Extend-the-output-plugin-API-with-stream-methods > > >

Re: SLRU statistics

2020-05-13 Thread Tom Lane
Tomas Vondra writes: > I think it counts as a variable with "static storage duration" per 6.7.8 > (para 10), see [1]. I wasn't aware of this either, but it probably means > the memset is unnecessary. > Also, it seems a bit strange/confusing to handle this differently from > BgWriterStats. And

Re: Add "-Wimplicit-fallthrough" to default flags

2020-05-13 Thread Tom Lane
Alvaro Herrera writes: >> I fear that is going to mean that we revert this patch. > Or we can filter-out the -Wimplicit-fallthrough, or change to level 3, > for bison-emitted files. Let's just go to level 3 overall (and revert the changes you made for level 4 compliance --- they're more likely

Re: Is it possible to find out write_lsn on standby?

2020-05-13 Thread Jehan-Guillaume de Rorthais
Hello, On Fri, 8 May 2020 18:25:02 +0500 godjan • wrote: > On primary I can execute ’SELECT write_lsn FROM pg_stat_replication;’ and get > write_lsn of standby. I didn’t find function like "pg_last_write_lsn()” to > get write_lsn on standby. Is it possible? No, there's no admin function

Re: Do I understand commit timestamps correctly?

2020-05-13 Thread Alvaro Herrera
On 2020-May-13, Chapman Flack wrote: > Getting around to trying it out, simply changing the setting in > postgresql.conf before starting the server does not seem sufficient: > once it comes online, it has track_commit_timestamp on, but has not > populated the cache from transactions it applied

Re: SLRU statistics

2020-05-13 Thread Tomas Vondra
On Wed, May 13, 2020 at 11:46:30PM +0900, Fujii Masao wrote: On 2020/05/13 23:26, Tom Lane wrote: Fujii Masao writes: On 2020/05/13 17:21, Tomas Vondra wrote: On Wed, May 13, 2020 at 04:10:30PM +0900, Fujii Masao wrote: Also I found another minor issue; SLRUStats has not been initialized

Re: SLRU statistics

2020-05-13 Thread Tomas Vondra
On Wed, May 13, 2020 at 11:01:47AM -0400, Tom Lane wrote: Tomas Vondra writes: On Wed, May 13, 2020 at 10:26:39AM -0400, Tom Lane wrote: Why is that necessary? A static variable is defined by C to start off as zeroes. But is it a static variable? It's not declared as 'static' but maybe we

Re: Add "-Wimplicit-fallthrough" to default flags

2020-05-13 Thread Alvaro Herrera
On 2020-May-13, Tom Lane wrote: > Andy Fan writes: > >> FWIW, I got a warning for jsonpath_gram.c. > > Ugh. Confirmed here on Fedora 30 (bison 3.0.5). Ugh. > > I just found this just serval minutes ago. Upgrading your bison to the > > latest version (3.6) is ok. I'd like we have a better

Re: PG 13 release notes, first draft

2020-05-13 Thread Bruce Momjian
On Wed, May 13, 2020 at 07:18:38AM +0200, Fabien COELHO wrote: > > Hello Bruce, > > > > * e1ff780485 > > > > I was told in this email thread to not include that one. > > Ok. > > > > * 34a0a81bfb > > > > We already have: > > > > Reformat tables containing function information for

Re: PG 13 release notes, first draft

2020-05-13 Thread Bruce Momjian
On Wed, May 13, 2020 at 11:56:33AM +0900, Kyotaro Horiguchi wrote: > > > > Allow skipping of WAL for new tables and indexes if wal_level is > > 'minimal' (Kyotaro Horiguchi) > > > > Relations larger than wal_skip_threshold will have their files > > fsync'ed rather than writing

Re: Strange decreasing value of pg_last_wal_receive_lsn()

2020-05-13 Thread Jehan-Guillaume de Rorthais
On Mon, 11 May 2020 15:54:02 +0900 Michael Paquier wrote: [...] > There are several HA solutions floating around in the community, and I > got to wonder as well if some of them don't just scan the local > pg_wal/ of each standby in this case, even if that's more simple to > let the nodes start

Re: SLRU statistics

2020-05-13 Thread Tom Lane
Tomas Vondra writes: > On Wed, May 13, 2020 at 10:26:39AM -0400, Tom Lane wrote: >> Why is that necessary? A static variable is defined by C to start off >> as zeroes. > But is it a static variable? It's not declared as 'static' but maybe we > can assume it inits to zeroes anyway? I see we do

Re: SLRU statistics

2020-05-13 Thread Tomas Vondra
On Wed, May 13, 2020 at 10:26:39AM -0400, Tom Lane wrote: Fujii Masao writes: On 2020/05/13 17:21, Tomas Vondra wrote: On Wed, May 13, 2020 at 04:10:30PM +0900, Fujii Masao wrote: Also I found another minor issue; SLRUStats has not been initialized to 0 and which could update the counters

Re: Strange decreasing value of pg_last_wal_receive_lsn()

2020-05-13 Thread Jehan-Guillaume de Rorthais
(too bad the history has been removed to keep context) On Fri, 8 May 2020 15:02:26 +0500 godjan • wrote: > I got it, thank you. > Can you recommend what to use to determine which quorum standby should be > promoted in such case? We planned to use pg_last_wal_receive_lsn() to > determine which

Re: SLRU statistics

2020-05-13 Thread Fujii Masao
On 2020/05/13 23:26, Tom Lane wrote: Fujii Masao writes: On 2020/05/13 17:21, Tomas Vondra wrote: On Wed, May 13, 2020 at 04:10:30PM +0900, Fujii Masao wrote: Also I found another minor issue; SLRUStats has not been initialized to 0 and which could update the counters unexpectedly.

Re: Inefficiency in SLRU stats collection

2020-05-13 Thread Tom Lane
Kyotaro Horiguchi writes: > AFAICS it is right and the change suggested looks reasonable to me. > One arguable point might be whether it is right that SlruData holds > pgstats internal index from the standpoint of modularity. (It is one > of the reasons I didn't propose a patch for that..)

Ideas about moving live rows to the top of the table

2020-05-13 Thread Антон Пацев
I read about the utility – https://github.com/dataegret/pgcompacttable . It is more careful about resources, because it works on slightly different principles. The main point of pgcompacttable is that it moves all live rows to the beginning of the table with updates in the table. And then it

Re: Add explanations which are influenced by track_io_timing

2020-05-13 Thread Fujii Masao
On 2020/05/13 21:54, Atsushi Torikoshi wrote: Hi, When track_io_timing is on, I/O timing information is displayed in pg_stat_database, in the output of EXPLAIN when the BUFFERS option is used, and in pg_stat_statements as documented in [1]. This is also described in the manual for

Re: SLRU statistics

2020-05-13 Thread Tom Lane
Fujii Masao writes: > On 2020/05/13 17:21, Tomas Vondra wrote: >> On Wed, May 13, 2020 at 04:10:30PM +0900, Fujii Masao wrote: >>> Also I found another minor issue; SLRUStats has not been initialized to 0 >>> and which could update the counters unexpectedly. Attached patch fixes >>> this issue.

Re: refactoring basebackup.c

2020-05-13 Thread Robert Haas
On Wed, May 13, 2020 at 12:01 AM Suraj Kharage < suraj.khar...@enterprisedb.com> wrote: > 8kb 32kb (default value) 128kB 1024kB > Without refactor patch real 10m22.718s > user 1m23.629s > sys 8m51.410s real 8m36.245s > user 1m8.471s > sys 7m21.520s real 6m54.299s > user 0m55.690s > sys 5m46.502s

Re: PG compilation error with Visual Studio 2015/2017/2019

2020-05-13 Thread Tom Lane
Amit Kapila writes: > Now that branches are tagged, I would like to commit and backpatch > this patch tomorrow unless there are any more comments/objections. The "quiet period" is over as soon as the tags appear in git. regards, tom lane

Re: Add "-Wimplicit-fallthrough" to default flags

2020-05-13 Thread Tom Lane
Andy Fan writes: >> FWIW, I got a warning for jsonpath_gram.c. Ugh. Confirmed here on Fedora 30 (bison 3.0.5). > I just found this just serval minutes ago. Upgrading your bison to the > latest version (3.6) is ok. I'd like we have a better way to share this > knowledge through. I spend ~30

Re: COPY, lock release and MVCC

2020-05-13 Thread Amit Kapila
On Wed, May 13, 2020 at 1:20 AM Laurenz Albe wrote: > > On Tue, 2020-05-12 at 11:50 -0400, Tom Lane wrote: > > Laurenz Albe writes: > > > On Mon, 2020-05-11 at 15:43 -0400, Robert Haas wrote: > > > > On Fri, May 8, 2020 at 4:58 AM Laurenz Albe > > > > wrote: > > > > > I happened to notice that

Re: No core file generated after PostgresNode->start

2020-05-13 Thread Tom Lane
Michael Paquier writes: > On Tue, May 12, 2020 at 04:15:26PM -0400, Robert Haas wrote: >> On Mon, May 11, 2020 at 10:48 PM Tom Lane wrote: >>> I have a standing note to check the permissions on /cores after any macOS >>> upgrade, because every so often Apple decides that that directory ought to

Re: PG compilation error with Visual Studio 2015/2017/2019

2020-05-13 Thread Amit Kapila
On Thu, May 7, 2020 at 6:51 PM Tom Lane wrote: > > Amit Kapila writes: > > On Thu, May 7, 2020 at 11:57 AM Tom Lane wrote: > >> Monday is a back-branch release wrap day. > > > How can I get the information about release wrap day? The minor > > release dates are mentioned on the website [1],

Re: SLRU statistics

2020-05-13 Thread Fujii Masao
On 2020/05/13 17:21, Tomas Vondra wrote: On Wed, May 13, 2020 at 04:10:30PM +0900, Fujii Masao wrote: On 2020/05/07 13:47, Fujii Masao wrote: On 2020/05/03 1:59, Tomas Vondra wrote: On Sat, May 02, 2020 at 12:55:00PM +0200, Tomas Vondra wrote: On Sat, May 02, 2020 at 03:56:07PM +0900,

Re: making update/delete of inheritance trees scale better

2020-05-13 Thread Amit Langote
On Tue, May 12, 2020 at 10:57 PM Tom Lane wrote: > Amit Langote writes: > > On Tue, May 12, 2020 at 5:25 AM Robert Haas wrote: > >> Ah, that makes sense. If we can invent dummy columns on the parent > >> rel, then most of what I was worrying about no longer seems very > >> worrying. > > > IIUC,

Add explanations which are influenced by track_io_timing

2020-05-13 Thread Atsushi Torikoshi
Hi, When track_io_timing is on, I/O timing information is displayed in pg_stat_database, in the output of EXPLAIN when the BUFFERS option is used, and in pg_stat_statements as documented in [1]. This is also described in the manual for pg_stat_statements [2], however, manuals for

Re: Inefficiency in SLRU stats collection

2020-05-13 Thread Fujii Masao
On 2020/05/13 11:26, Kyotaro Horiguchi wrote: At Tue, 12 May 2020 15:50:35 -0400, Tom Lane wrote in I happened to come across this code added by 28cac71bd: static PgStat_MsgSLRU * slru_entry(SlruCtl ctl) { intidx = pgstat_slru_index(ctl->shared->lwlock_tranche_name);

pgstat_read_statsfiles() and reset timestamp

2020-05-13 Thread Fujii Masao
Hi, pgstat_read_statsfiles() sets each stat_reset_timestamp to the current timestamp, at the beginning of the function, just in case we fail to load the existing statsfile. This code is originally introduced by commit 4c468b37a2. But commit ad1b5c842b changed pgstat_read_statsfiles() so that

Re: [PATCH] Keeps tracking the uniqueness with UniqueKey

2020-05-13 Thread Ashutosh Bapat
On Fri, May 8, 2020 at 7:27 AM Andy Fan wrote: >> +else if (inner_is_onerow) >> +{ >> +/* Since rows in innerrel can't be duplicated AND if >> innerrel is onerow, >> + * the join result will be onerow also as well. Note: >> onerow implies

Re: refactoring basebackup.c

2020-05-13 Thread Sumanta Mukherjee
Hi Suraj, Two points I wanted to mention. 1. The max rate at which the transfer is happening when the tar size is 128 Kb is at most .48 GB/sec. Is there a possibility to understand what is the buffer size which is being used. That could help us explain some part of the puzzle. 2.

Re: Add "-Wimplicit-fallthrough" to default flags

2020-05-13 Thread Julien Rouhaud
On Wed, May 13, 2020 at 12:16 PM Kyotaro Horiguchi wrote: > > At Wed, 13 May 2020 16:17:50 +0800, Andy Fan wrote > in > > On Wed, May 13, 2020 at 4:13 PM Kyotaro Horiguchi > > wrote: > > > > jsonpath_gram.c:1026:16: warning: this statement may fall through > > > [-Wimplicit-fallthrough=] > > >

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-05-13 Thread Amit Kapila
On Wed, May 13, 2020 at 11:35 AM Dilip Kumar wrote: > > On Tue, May 12, 2020 at 4:39 PM Amit Kapila wrote: > > > > > > v20-0003-Extend-the-output-plugin-API-with-stream-methods > > > > 1. > > +static void >

Re: Add "-Wimplicit-fallthrough" to default flags

2020-05-13 Thread Kyotaro Horiguchi
At Wed, 13 May 2020 16:17:50 +0800, Andy Fan wrote in > On Wed, May 13, 2020 at 4:13 PM Kyotaro Horiguchi > wrote: > > > jsonpath_gram.c:1026:16: warning: this statement may fall through > > [-Wimplicit-fallthrough=] > > > if (*++yyp != '\\') > > > ^ > > >

Re: Index Skip Scan

2020-05-13 Thread Dilip Kumar
On Mon, May 11, 2020 at 4:55 PM Dmitry Dolgov <9erthali...@gmail.com> wrote: > > > On Mon, May 11, 2020 at 04:04:00PM +0530, Dilip Kumar wrote: > > > > > > +static inline bool > > > > +_bt_scankey_within_page(IndexScanDesc scan, BTScanInsert key, > > > > + Buffer buf, ScanDirection dir) > > > > +{

Re: A patch for get origin from commit_ts.

2020-05-13 Thread movead...@highgo.ca
>I have not thought about this matter, but it seems to me that you >should add this patch to the upcoming commit fest for evaluation: >https://commitfest.postgresql.org/28/ Thanks. I think about it more detailed, and find it's better to show the 'roident' other than 'roname'. Because an old

Re: SLRU statistics

2020-05-13 Thread Tomas Vondra
On Wed, May 13, 2020 at 04:10:30PM +0900, Fujii Masao wrote: On 2020/05/07 13:47, Fujii Masao wrote: On 2020/05/03 1:59, Tomas Vondra wrote: On Sat, May 02, 2020 at 12:55:00PM +0200, Tomas Vondra wrote: On Sat, May 02, 2020 at 03:56:07PM +0900, Fujii Masao wrote: ... Another thing I

Re: Add "-Wimplicit-fallthrough" to default flags

2020-05-13 Thread Andy Fan
On Wed, May 13, 2020 at 4:13 PM Kyotaro Horiguchi wrote: > At Tue, 12 May 2020 17:12:51 -0400, Tom Lane wrote in > > Alvaro Herrera writes: > > > Fixed one straggler in contrib, and while testing it I realized why > > > ccache doesn't pay attention to the changes I was doing in the file: > > >

  1   2   >