Why not represent "never vacuumed" accurately wrt pg_class.relpages?

2018-12-10 Thread Andres Freund
Hi, estimate_rel_size() explains: /* * HACK: if the relation has never yet been vacuumed, use a * minimum size estimate of 10 pages. The idea here is to avoid * assuming a newly-created table is

Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock

2018-12-10 Thread Andrey Borodin
Hi! > 11 дек. 2018 г., в 3:43, Alexander Korotkov написал(а): > > Attached patch appears to be incomplete. GinPageSetDeleteXid() is > called only in ginRedoDeletePage(), so only in recovery, while it > should be set during normal work too. deleteXid field of > ginxlogDeletePage is never set.

Should get_rel_data_width take alignment into account?

2018-12-10 Thread Andres Freund
Hi, I'm working on the pluggable storage patch and zheap at the moment. The latter the latter has no alignment padding for byval attributes - since we copy them anyway - and I was looking which places need to be adjusted for that. That lead me to notice that get_rel_data_width() currently

Re: Interesting paper: Contention-Aware Lock Scheduling

2018-12-10 Thread Ibrar Ahmed
On Mon, May 7, 2018 at 10:54 PM Юрий Соколов wrote: > 2018-05-04 23:45 GMT+03:00 AJG : > > > > Another interesting article from Jan 2018 (Tsinghua University and > Microsoft > > Research) > > > > http://madsys.cs.tsinghua.edu.cn/publications/TS2018-liu.pdf > > > > DudeTx: Durable Transactions

Re: Sketch of a fix for that truncation data corruption issue

2018-12-10 Thread Andres Freund
Hi, On 2018-12-11 07:09:34 +0100, Laurenz Albe wrote: > Tom Lane wrote: > > We got another report today [1] that seems to be due to the problem > > we've seen before with failed vacuum truncations leaving corrupt state > > on-disk [2]. Reflecting on that some more, [...] > > This may seem

Re: Sketch of a fix for that truncation data corruption issue

2018-12-10 Thread Laurenz Albe
Tom Lane wrote: > We got another report today [1] that seems to be due to the problem > we've seen before with failed vacuum truncations leaving corrupt state > on-disk [2]. Reflecting on that some more, [...] This may seem heretical, but I'll say it anyway. Why don't we do away with vacuum

Re: Sketch of a fix for that truncation data corruption issue

2018-12-10 Thread Tom Lane
Robert Haas writes: > On Tue, Dec 11, 2018 at 5:39 AM Tom Lane wrote: >> 9. If actual truncation boundary was different from plan, issue another >> WAL record saying "oh, we only managed to truncate to here, not there". > I don't entirely understand how this fix addresses the problems in > this

Re: Sketch of a fix for that truncation data corruption issue

2018-12-10 Thread Robert Haas
On Tue, Dec 11, 2018 at 5:39 AM Tom Lane wrote: > We got another report today [1] that seems to be due to the problem > we've seen before with failed vacuum truncations leaving corrupt state > on-disk [2]. Reflecting on that some more, it seems to me that we're > never going to get to a solution

Re: [HACKERS] Bug when dumping "empty" operator classes

2018-12-10 Thread Michael Paquier
On Mon, Dec 10, 2018 at 01:20:18PM -0500, Stephen Frost wrote: > This has now been pushed. Thanks for taking care of it, Stephen! -- Michael signature.asc Description: PGP signature

Re: allow online change primary_conninfo

2018-12-10 Thread Michael Paquier
On Mon, Nov 26, 2018 at 09:46:36AM -0800, Andres Freund wrote: > I'm not sure I understand what you mean? The way I'd solve this is that > that only walreceiver, at startup, writes out its conninfo/slot_name, > sourcing the values from the GUCs. That way there's no issue with values > being

Re: Thinking about EXPLAIN ALTER TABLE

2018-12-10 Thread David Fetter
On Tue, Dec 11, 2018 at 11:25:12AM +0900, Robert Haas wrote: > On Tue, Dec 11, 2018 at 1:32 AM Tom Lane wrote: > > Simon Riggs writes: > > > I suggest ALTER TABLE should respond to a parameter setting of ddl_dry_run > > > = on, so the whole world doesn't need to rewrite its syntax to support the

Re: Thinking about EXPLAIN ALTER TABLE

2018-12-10 Thread Michael Paquier
On Tue, Dec 11, 2018 at 11:25:12AM +0900, Robert Haas wrote: > Yeah, I like EXPLAIN better. +1. -- Michael signature.asc Description: PGP signature

Re: Should new partitions inherit their tablespace from their parent?

2018-12-10 Thread Michael Paquier
On Mon, Dec 10, 2018 at 10:56:47PM +0900, Michael Paquier wrote: > Cool, thanks for updating the assertion. At quick glance the patch > seems fine to me. Let's keep ATExecSetTableSpaceNoStorage as routine > name as you suggest then. + /* +* For partitions, when no other tablespace

Re: Thinking about EXPLAIN ALTER TABLE

2018-12-10 Thread Robert Haas
On Tue, Dec 11, 2018 at 1:32 AM Tom Lane wrote: > Simon Riggs writes: > > I suggest ALTER TABLE should respond to a parameter setting of ddl_dry_run > > = on, so the whole world doesn't need to rewrite its syntax to support the > > new option. > > We were just busy shooting down a different

Re: Pluggable Storage - Andres's take

2018-12-10 Thread Andres Freund
Hi, On 2018-11-26 17:55:57 -0800, Andres Freund wrote: > FWIW, now that oids are removed, and the tuple table slot abstraction > got in, I'm working on rebasing the pluggable storage patchset ontop of > that. I've pushed a version to that to the git tree, including a rebased version of zheap:

Re: pg_dump emits ALTER TABLE ONLY partitioned_table

2018-12-10 Thread Amit Langote
On Tue, Dec 11, 2018 at 3:20 AM Stephen Frost wrote: > * Stephen Frost (sfr...@snowman.net) wrote: > > * Amit Langote (langote_amit...@lab.ntt.co.jp) wrote: > > > Do you intend to push the other patch to add regression tests for the > > > non-inherited constraints? Here it is attached again for

Re: Pluggable Storage - Andres's take

2018-12-10 Thread Andres Freund
Hi, Thanks for these changes. I've merged a good chunk of them. On 2018-11-16 12:05:26 +1100, Haribabu Kommi wrote: > diff --git a/src/backend/access/heap/heapam_handler.c > b/src/backend/access/heap/heapam_handler.c > index c3960dc91f..3254e30a45 100644 > ---

Re: Add timeline to partial WAL segments

2018-12-10 Thread Michael Paquier
On Mon, Dec 10, 2018 at 10:21:23AM -0500, David Steele wrote: > We recommend that archive commands not overwrite an existing segment. > Some backup tools will compare the contents and succeed if they are > equal, but in this case that will still often fail because recycled WAL > segments will have

Re: Thinking about EXPLAIN ALTER TABLE

2018-12-10 Thread Greg Stark
On Mon, 10 Dec 2018 at 12:27, Simon Riggs wrote: > > An event trigger with a table_rewrite event, allows you to scan a whole > script for objectionable activity on a test server before you put it into > production. > > Perhaps we just need a few extra events. That's not a bad idea at all.

Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock

2018-12-10 Thread Alexander Korotkov
On Sun, Dec 9, 2018 at 10:25 PM Alexander Korotkov wrote: > On Sat, Dec 8, 2018 at 12:48 PM Andrey Borodin wrote: > > > 8 дек. 2018 г., в 6:54, Alexander Korotkov > > > написал(а): > > > > > > Yep, please find attached draft patch. > > > > Patch seems good to me, I'll check it in more detail.

Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock

2018-12-10 Thread Alexander Korotkov
On Mon, Dec 10, 2018 at 4:58 PM Andrey Borodin wrote: > > 10 дек. 2018 г., в 18:56, Andrey Borodin написал(а): > > > > PFA this part. In thread about GiST vacuum Heikki was not very happy with > > reusing PageHeader->pd_prune_xid for this. But we somewhat concluded that > > this might be OK. >

Re: [HACKERS][PATCH] Applying PMDK to WAL operations for persistent memory

2018-12-10 Thread Dmitry Dolgov
> On Thu, Nov 29, 2018 at 6:48 PM Dmitry Dolgov <9erthali...@gmail.com> wrote: > > > On Tue, Oct 2, 2018 at 4:53 AM Michael Paquier wrote: > > > > On Mon, Aug 06, 2018 at 06:00:54PM +0900, Yoshimi Ichiyanagi wrote: > > > The libpmem's pmem_map_file() supported 2M/1G(the size of huge page) > > >

Sketch of a fix for that truncation data corruption issue

2018-12-10 Thread Tom Lane
We got another report today [1] that seems to be due to the problem we've seen before with failed vacuum truncations leaving corrupt state on-disk [2]. Reflecting on that some more, it seems to me that we're never going to get to a solution that everybody finds acceptable without some rather

Re: Dead code in toast_fetch_datum_slice?

2018-12-10 Thread Paul Ramsey
> On Dec 10, 2018, at 10:30 AM, Stephen Frost wrote: > > Greetings, > > * Alvaro Herrera (alvhe...@2ndquadrant.com) wrote: >> On 2018-Dec-10, Paul Ramsey wrote: >>> Your analysis looks correct to me, I'm pretty sure I had the same reaction, >>> first time I read through. It would be nice to

Re: Dead code in toast_fetch_datum_slice?

2018-12-10 Thread Stephen Frost
Greetings, * Alvaro Herrera (alvhe...@2ndquadrant.com) wrote: > On 2018-Dec-10, Paul Ramsey wrote: > > Your analysis looks correct to me, I'm pretty sure I had the same reaction, > > first time I read through. It would be nice to handle partial decompression > > all the way down at this level,

Re: Dead code in toast_fetch_datum_slice?

2018-12-10 Thread Alvaro Herrera
On 2018-Dec-10, Paul Ramsey wrote: > Your analysis looks correct to me, I'm pretty sure I had the same reaction, > first time I read through. It would be nice to handle partial decompression > all the way down at this level, but unfortunately the comment at the > Assert() is right: there's no way

Re: pg_dump emits ALTER TABLE ONLY partitioned_table

2018-12-10 Thread Stephen Frost
Greetings, * Stephen Frost (sfr...@snowman.net) wrote: > Greetings, > > * Amit Langote (langote_amit...@lab.ntt.co.jp) wrote: > > On 2017/05/08 12:42, Stephen Frost wrote: > > > * Amit Langote (langote_amit...@lab.ntt.co.jp) wrote: > > >> Thanks for committing the patch after improving it quite

Re: [HACKERS] Bug when dumping "empty" operator classes

2018-12-10 Thread Stephen Frost
Greetings, * Stephen Frost (sfr...@snowman.net) wrote: > * Michael Paquier (michael.paqu...@gmail.com) wrote: > > On Fri, May 26, 2017 at 8:14 AM, Daniel Gustafsson wrote: > > >> On 26 May 2017, at 17:08, Tom Lane wrote: > > >> I'll commit and back-patch this without a test case. Possibly

Re: Dead code in toast_fetch_datum_slice?

2018-12-10 Thread Stephen Frost
Greetings, * Paul Ramsey (pram...@cleverelephant.ca) wrote: > On Fri, Dec 7, 2018 at 3:25 PM Stephen Frost wrote: > > Perhaps I'm missing something, but in toast_fetch_datum_slice() there's: > > > > Assert(!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)); > > > > Further, the only caller

Re: Dead code in toast_fetch_datum_slice?

2018-12-10 Thread Paul Ramsey
On Fri, Dec 7, 2018 at 3:25 PM Stephen Frost wrote: > > > Perhaps I'm missing something, but in toast_fetch_datum_slice() there's: > > Assert(!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)); > > Further, the only caller of this function today is > heap_tuple_untoast_attr_slice(), which

Re: Thinking about EXPLAIN ALTER TABLE

2018-12-10 Thread Simon Riggs
On Mon, 10 Dec 2018 at 16:52, Tom Lane wrote: > Simon Riggs writes: > > On Mon, 10 Dec 2018 at 16:32, Tom Lane wrote: > >> We were just busy shooting down a different suggestion of > >> behavior-changing GUCs. A GUC that turns all ALTERs into no-ops > >> sure seems like a foot-gun to me. > >

Re: Thinking about EXPLAIN ALTER TABLE

2018-12-10 Thread Tom Lane
Simon Riggs writes: > On Mon, 10 Dec 2018 at 16:32, Tom Lane wrote: >> We were just busy shooting down a different suggestion of >> behavior-changing GUCs. A GUC that turns all ALTERs into no-ops >> sure seems like a foot-gun to me. > How would you test a script? Manually edit each one with

Re: Thinking about EXPLAIN ALTER TABLE

2018-12-10 Thread Simon Riggs
On Mon, 10 Dec 2018 at 16:32, Tom Lane wrote: > Simon Riggs writes: > > I suggest ALTER TABLE should respond to a parameter setting of > ddl_dry_run > > = on, so the whole world doesn't need to rewrite its syntax to support > the > > new option. > > We were just busy shooting down a different

Re: Psql patch to show access methods info

2018-12-10 Thread s . cherkashin
Here are some fixes. But I'm not sure that the renaming of columns for the '\dAp' command is sufficiently laconic and informative. If you have any suggestions on how to improve them, I will be very grateful. Best regards, Sergey Cherkashin.diff --git a/doc/src/sgml/catalogs.sgml

Re: Thinking about EXPLAIN ALTER TABLE

2018-12-10 Thread Tom Lane
Simon Riggs writes: > I suggest ALTER TABLE should respond to a parameter setting of ddl_dry_run > = on, so the whole world doesn't need to rewrite its syntax to support the > new option. We were just busy shooting down a different suggestion of behavior-changing GUCs. A GUC that turns all

Re: Thinking about EXPLAIN ALTER TABLE

2018-12-10 Thread Simon Riggs
On Mon, 10 Dec 2018 at 16:14, Alvaro Herrera wrote: > Hi Greg > > On 2018-Dec-07, Greg Stark wrote: > > > I'm thinking I should try to move all these decisions to phase 1 as > > much as possible but I'm not sure how feasible it will be to get the > > results exactly correct. Of course the cases

Re: Thinking about EXPLAIN ALTER TABLE

2018-12-10 Thread Alvaro Herrera
Hi Greg On 2018-Dec-07, Greg Stark wrote: > I'm thinking I should try to move all these decisions to phase 1 as > much as possible but I'm not sure how feasible it will be to get the > results exactly correct. Of course the cases where it's hardest to > predict are precisely where users would

Re: Pluggable Storage - Andres's take

2018-12-10 Thread Dmitry Dolgov
> On Fri, Nov 16, 2018 at 2:05 AM Haribabu Kommi > wrote: > > I tried running the pgbench performance tests with minimal clients in my > laptop and I didn't > find any performance issues, may be issue is visible only with higher > clients. Even with > perf tool, I am not able to get a clear

Re: docs: outdated reference to recursive expression evaluation

2018-12-10 Thread Tom Lane
John Naylor writes: > On 12/8/18, Tom Lane wrote: >> We could say "expression compilation" and it'd still be valid. Or just >> drop the last four words altogether. I don't think we want to expend the >> verbiage to be more precise here, since it's only a tangential point. > I'm inclined to

Add timeline to partial WAL segments

2018-12-10 Thread David Steele
Hackers, The .partial mechanism was added in de768844 to help avoid conflicts between a newly promoted primary and an old primary that might produce the same WAL segment. This works for a single promotion but can become problematic in HA configurations where there may be several promotions

Re: docs: outdated reference to recursive expression evaluation

2018-12-10 Thread John Naylor
On 12/8/18, Tom Lane wrote: > John Naylor writes: >> In confg.sgml, in the section about max_stack_depth, there's this >> sentence: >> "The safety margin is needed because the stack depth is not checked in >> every routine in the server, but only in key potentially-recursive >> routines such as

Re: automatically assigning catalog toast oids

2018-12-10 Thread John Naylor
On 12/9/18, Tom Lane wrote: > Another thing I seriously dislike is that this allows people to omit OIDs > from .dat entries in catalogs where we traditionally hand-assign OIDs. > That's not a good idea because it would mean those entries don't have > stable OIDs, whereas the whole point of hand

Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock

2018-12-10 Thread Andrey Borodin
> 10 дек. 2018 г., в 18:56, Andrey Borodin написал(а): > > PFA this part. In thread about GiST vacuum Heikki was not very happy with > reusing PageHeader->pd_prune_xid for this. But we somewhat concluded that > this might be OK. > Anyway, there's whole page, we can store those bits anywhere.

Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock

2018-12-10 Thread Andrey Borodin
Hi! > 10 дек. 2018 г., в 0:25, Alexander Korotkov написал(а): >> >> There's a patch above in this thread >> 0001-Use-correct-locking-protocol-during-GIN-posting-tree.patch where I >> propose stamping every deleted page with GinPageSetDeleteXid(page, >> ReadNewTransactionId()); and avoid

Re: Should new partitions inherit their tablespace from their parent?

2018-12-10 Thread Michael Paquier
On Mon, Dec 10, 2018 at 11:37:16PM +1300, David Rowley wrote: > Likely it would be nice if we had a macro to determine if the relkind > should have storage or not, and then just Assert() we get one of those > into the function. The other thread can deal about that matter I guess. > For now, I've

Re: pg_partition_tree crashes for a non-defined relation

2018-12-10 Thread Michael Paquier
On Mon, Dec 10, 2018 at 08:21:43AM -0500, Stephen Frost wrote: > * Michael Paquier (mich...@paquier.xyz) wrote: >> This still looks adapted to me. Or would you reword it because "allow" >> rather implies that an ERROR is returned? Would you prefer changing it >> something like that perhaps: >>

Re: pg_partition_tree crashes for a non-defined relation

2018-12-10 Thread Stephen Frost
Greetings, * Michael Paquier (mich...@paquier.xyz) wrote: > On Sun, Dec 09, 2018 at 12:49:18PM -0500, Stephen Frost wrote: > > Looks alright on a quick glance, but shouldn't you also update the > > comment..? > > The comment on HEAD or with the patch is that: > /* Only allow relation types

Re: pg_partition_tree crashes for a non-defined relation

2018-12-10 Thread Stephen Frost
Greetings, * Michael Paquier (mich...@paquier.xyz) wrote: > On Sun, Dec 09, 2018 at 02:07:29PM -0500, Tom Lane wrote: > > Stephen Frost writes: > >> I don't entirely buy off on the argument that it's code that's 'highly > >> unlikely to break once written' though- we do add new relkinds from

Re: Should new partitions inherit their tablespace from their parent?

2018-12-10 Thread David Rowley
On Mon, 10 Dec 2018 at 15:22, Michael Paquier wrote: > - Assert(rel->rd_rel->relkind == RELKIND_PARTITIONED_INDEX); > + Assert(rel->rd_rel->relkind == RELKIND_VIEW || > + rel->rd_rel->relkind == RELKIND_COMPOSITE_TYPE || > + rel->rd_rel->relkind ==

Re: [PATCH] Log PostgreSQL version number on startup

2018-12-10 Thread Christoph Berg
Re: Michael Paquier 2018-12-10 <20181210063307.gd1...@paquier.xyz> > On Wed, Nov 21, 2018 at 11:32:46AM -0500, Stephen Frost wrote: > > * Christoph Berg (christoph.b...@credativ.de) wrote: > >> it has bugged me for a long time that there's no clear "PostgreSQL is > >> starting" message in the

Re: vacuum and autovacuum - is it good to configure the threshold at TABLE LEVEL?

2018-12-10 Thread amul sul
On Mon, Dec 10, 2018 at 2:51 PM rajan wrote: > > Thanks for the reply Amul. > > Btw, No my table does not has any index. Thanks for the information. Regards, Amul

Re: vacuum and autovacuum - is it good to configure the threshold at TABLE LEVEL?

2018-12-10 Thread rajan
Thanks for the reply Amul. Btw, No my table does not has any index. - -- Thanks, Rajan. -- Sent from: http://www.postgresql-archive.org/PostgreSQL-hackers-f1928748.html

Re: vacuum and autovacuum - is it good to configure the threshold at TABLE LEVEL?

2018-12-10 Thread amul sul
Apologize for delayed response. On Mon, Dec 3, 2018 at 12:30 PM rajan wrote: > > Hello Amul, > > I have a doubt. Please find below the details, > > 1. autovacuum_vacuum_threshold and autovacuum_analyse_threshold are set at > 300 > 2. autovacuum_vacuum_scale_factor and

RE: Too many logs are written on Windows (LOG: could not reserve shared memory region (addr=%p) for child %p:)

2018-12-10 Thread Takahashi, Ryohei
Hi, Thank you for replying. > Like a couple of others on this thread I doubt that lowering the elevel > here would be a good thing, as keeping it noisy has been what allows to > know that something has gone wrong, no? The current behavior is > useful. Currently, I agree with you. I cancel my