Re: [HACKERS] WAL logging problem in 9.4.3?

At Wed, 13 Sep 2017 17:42:39 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20170913.174239.25978735.horiguchi.kyot...@lab.ntt.co.jp> > filterdiff seems to did something wrong.. # to did... The patch is broken by filterdiff so I send a new patch made

Re: [HACKERS] WAL logging problem in 9.4.3?

At Wed, 13 Sep 2017 15:05:31 +1200, Thomas Munro wrote in

Re: [HACKERS] WAL logging problem in 9.4.3?

Kyotaro HORIGUCHI wrote: > The CF status of this patch turned into "Waiting on Author" by > automated CI checking. I object to automated turning of patches to waiting on author by machinery. Sending occasional reminder messages to authors making them know about outdated patches seems acceptable

Re: [HACKERS] WAL logging problem in 9.4.3?

On Wed, Sep 13, 2017 at 1:04 PM, Kyotaro HORIGUCHI wrote: > The CF status of this patch turned into "Waiting on Author" by > automated CI checking. However, I still don't get any error even > on the current master (69835bc) after make distclean. Also I > don't see

Re: [HACKERS] WAL logging problem in 9.4.3?

Hello, (does this seem to be a top post?) The CF status of this patch turned into "Waiting on Author" by automated CI checking. However, I still don't get any error even on the current master (69835bc) after make distclean. Also I don't see any difference between the "problematic" patch and my

Re: [HACKERS] WAL logging problem in 9.4.3?

Hello, At Fri, 08 Sep 2017 16:30:01 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20170908.163001.53230385.horiguchi.kyot...@lab.ntt.co.jp> > > >> 2017-04-13 12:11:27.065 JST [85441] t/102_vacuumdb_stages.pl > > >> STATEMENT: ANALYZE; > > >>

Re: [HACKERS] WAL logging problem in 9.4.3?

Thank you for your notification. At Tue, 5 Sep 2017 12:05:01 +0200, Daniel Gustafsson wrote in > > On 13 Apr 2017, at 11:42, Kyotaro HORIGUCHI > > wrote: > > > > At Thu, 13 Apr 2017 13:52:40

Re: [HACKERS] WAL logging problem in 9.4.3?

> On 13 Apr 2017, at 11:42, Kyotaro HORIGUCHI > wrote: > > At Thu, 13 Apr 2017 13:52:40 +0900, Michael Paquier > wrote in > >> On Tue, Apr 11, 2017 at 5:38 PM,

Re: [HACKERS] WAL logging problem in 9.4.3?

At Thu, 13 Apr 2017 13:52:40 +0900, Michael Paquier wrote in > On Tue, Apr 11, 2017 at 5:38 PM, Kyotaro HORIGUCHI > wrote: > > Sorry, what I have just sent was

Re: [HACKERS] WAL logging problem in 9.4.3?

I'd like to put a supplimentary explanation. At Tue, 11 Apr 2017 17:38:12 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20170411.173812.133964522.horiguchi.kyot...@lab.ntt.co.jp> > Sorry, what I have just sent was broken. > > At Tue, 11 Apr 2017

Re: [HACKERS] WAL logging problem in 9.4.3?

On Tue, Apr 11, 2017 at 5:38 PM, Kyotaro HORIGUCHI wrote: > Sorry, what I have just sent was broken. You can use PROVE_TESTS when running make check to select a subset of tests you want to run. I use that all the time when working on patches dedicated to certain

Re: [HACKERS] WAL logging problem in 9.4.3?

Sorry, what I have just sent was broken. At Tue, 11 Apr 2017 17:33:41 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20170411.173341.257028732.horiguchi.kyot...@lab.ntt.co.jp> > At Tue, 11 Apr 2017 09:56:06 +0900 (Tokyo Standard Time), Kyotaro

Re: [HACKERS] WAL logging problem in 9.4.3?

At Tue, 11 Apr 2017 09:56:06 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20170411.095606.245908357.horiguchi.kyot...@lab.ntt.co.jp> > Hello, thank you for looking this. > > At Fri, 07 Apr 2017 20:38:35 -0400, Tom Lane wrote in

Re: [HACKERS] WAL logging problem in 9.4.3?

Hello, thank you for looking this. At Fri, 07 Apr 2017 20:38:35 -0400, Tom Lane wrote in <27309.1491611...@sss.pgh.pa.us> > Alvaro Herrera writes: > > Interesting. I wonder if it's possible that a relcache invalidation > > would cause these values

Re: [HACKERS] WAL logging problem in 9.4.3?

Alvaro Herrera writes: > Interesting. I wonder if it's possible that a relcache invalidation > would cause these values to get lost for some reason, because that would > be dangerous. > I suppose the rationale is that this shouldn't happen because any > operation that

Re: [HACKERS] WAL logging problem in 9.4.3?

Alvaro Herrera wrote: > I suppose the rationale is that this shouldn't happen because any > operation that does things this way must hold an exclusive lock on the > relation. But that doesn't guarantee that the relcache entry is > completely stable, does it? If we can get proof of that, then

Re: [HACKERS] WAL logging problem in 9.4.3?

I have claimed this patch as committer FWIW. -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription:

Re: [HACKERS] WAL logging problem in 9.4.3?

Kyotaro HORIGUCHI wrote: > The attached patch is quiiiccck-and-dirty-hack of Michael's patch > just as a PoC of my proposal quoted above. This also passes the > 006 test. The major changes are the following. > > - Moved sync_above and truncted_to into RelationData. Interesting. I wonder if

Re: [HACKERS] WAL logging problem in 9.4.3?

On 1/30/17 11:33 PM, Michael Paquier wrote: > On Fri, Dec 2, 2016 at 1:39 PM, Haribabu Kommi > wrote: >> The latest proposed patch still having problems. >> Closed in 2016-11 commitfest with "moved to next CF" status because of a bug >> fix patch. >> Please feel free to

Re: [HACKERS] WAL logging problem in 9.4.3?

On Fri, Dec 2, 2016 at 1:39 PM, Haribabu Kommi wrote: > The latest proposed patch still having problems. > Closed in 2016-11 commitfest with "moved to next CF" status because of a bug > fix patch. > Please feel free to update the status once you submit the updated patch.

Re: [HACKERS] WAL logging problem in 9.4.3?

On Wed, Nov 9, 2016 at 5:55 PM, Michael Paquier wrote: > > > On Wed, Nov 9, 2016 at 9:27 AM, Michael Paquier > wrote: > > On Wed, Nov 9, 2016 at 5:39 AM, Robert Haas > wrote: > >> On Thu, Feb 4, 2016 at 7:24 AM,

Re: [HACKERS] WAL logging problem in 9.4.3?

On Wed, Nov 9, 2016 at 9:27 AM, Michael Paquier wrote: > On Wed, Nov 9, 2016 at 5:39 AM, Robert Haas wrote: >> On Thu, Feb 4, 2016 at 7:24 AM, Heikki Linnakangas wrote: >>> I dropped the ball on this one back in July, so here's

Re: [HACKERS] WAL logging problem in 9.4.3?

On Wed, Nov 9, 2016 at 5:39 AM, Robert Haas wrote: > On Thu, Feb 4, 2016 at 7:24 AM, Heikki Linnakangas wrote: >> I dropped the ball on this one back in July, so here's an attempt to revive >> this thread. >> >> I spent some time fixing the remaining

Re: [HACKERS] WAL logging problem in 9.4.3?

On Thu, Feb 4, 2016 at 7:24 AM, Heikki Linnakangas wrote: > I dropped the ball on this one back in July, so here's an attempt to revive > this thread. > > I spent some time fixing the remaining issues with the prototype patch I > posted earlier, and rebased that on top of current

Re: [HACKERS] WAL logging problem in 9.4.3?

Hi, At Sun, 2 Oct 2016 21:43:46 +0900, Michael Paquier wrote in > On Thu, Sep 29, 2016 at 10:02 PM, Kyotaro HORIGUCHI > wrote: > > Hello, > > > > At Thu, 29 Sep 2016

Re: [HACKERS] WAL logging problem in 9.4.3?

On Thu, Sep 29, 2016 at 10:02 PM, Kyotaro HORIGUCHI wrote: > Hello, > > At Thu, 29 Sep 2016 16:59:55 +0900, Michael Paquier > wrote in > >> On Mon, Sep 26, 2016 at

Re: [HACKERS] WAL logging problem in 9.4.3?

Hello, At Thu, 29 Sep 2016 16:59:55 +0900, Michael Paquier wrote in > On Mon, Sep 26, 2016 at 5:03 PM, Kyotaro HORIGUCHI > wrote: > > Hello, I return to this before

Re: [HACKERS] WAL logging problem in 9.4.3?

On Mon, Sep 26, 2016 at 5:03 PM, Kyotaro HORIGUCHI wrote: > Hello, I return to this before my things:) > > Though I haven't played with the patch yet.. Be sure to run the test cases in the patch or base your tests on them then! > Though I don't know how it

Re: [HACKERS] WAL logging problem in 9.4.3?

Hello, I return to this before my things:) Though I haven't played with the patch yet.. At Fri, 29 Jul 2016 16:54:42 +0900, Michael Paquier wrote in > > Well, not that soon at the end, but I am back

Re: [HACKERS] WAL logging problem in 9.4.3?

On Thu, Jul 28, 2016 at 4:59 PM, Michael Paquier wrote: > On Wed, Apr 6, 2016 at 3:11 PM, Michael Paquier > wrote: >> On Wed, Mar 23, 2016 at 12:45 PM, Michael Paquier >> wrote: >>> On Wed, Mar 23, 2016 at 11:11

Re: [HACKERS] WAL logging problem in 9.4.3?

On Wed, Apr 6, 2016 at 3:11 PM, Michael Paquier wrote: > On Wed, Mar 23, 2016 at 12:45 PM, Michael Paquier > wrote: >> On Wed, Mar 23, 2016 at 11:11 AM, David Steele wrote: >>> I would prefer not to bump it to the next

Re: [HACKERS] WAL logging problem in 9.4.3?

On Wed, Mar 23, 2016 at 12:45 PM, Michael Paquier wrote: > On Wed, Mar 23, 2016 at 11:11 AM, David Steele wrote: >> I would prefer not to bump it to the next CF unless we decide this will >> not get fixed for 9.6. > > It may make sense to add that

Re: [HACKERS] WAL logging problem in 9.4.3?

On Wed, Mar 23, 2016 at 11:11 AM, David Steele wrote: > I would prefer not to bump it to the next CF unless we decide this will > not get fixed for 9.6. It may make sense to add that to the list of open items for 9.6 instead. That's not a feature. -- Michael -- Sent via

Re: [HACKERS] WAL logging problem in 9.4.3?

On 3/22/16 8:54 PM, Michael Paquier wrote: > On Wed, Mar 23, 2016 at 9:52 AM, Michael Paquier > wrote: >> On Wed, Mar 23, 2016 at 1:38 AM, David Steele wrote: >>> On 3/15/16 10:01 PM, Kyotaro HORIGUCHI wrote: >>> Ok, I understand that this is

Re: [HACKERS] WAL logging problem in 9.4.3?

On Wed, Mar 23, 2016 at 9:52 AM, Michael Paquier wrote: > On Wed, Mar 23, 2016 at 1:38 AM, David Steele wrote: >> On 3/15/16 10:01 PM, Kyotaro HORIGUCHI wrote: >> >>> Ok, I understand that this is not an issue in a hurry. I'll go to >>> another

Re: [HACKERS] WAL logging problem in 9.4.3?

On Wed, Mar 23, 2016 at 1:38 AM, David Steele wrote: > On 3/15/16 10:01 PM, Kyotaro HORIGUCHI wrote: > >> Ok, I understand that this is not an issue in a hurry. I'll go to >> another patch that needs review. > > Since we're getting towards the end of the CF is it time to pick

Re: [HACKERS] WAL logging problem in 9.4.3?

On 3/15/16 10:01 PM, Kyotaro HORIGUCHI wrote: > Ok, I understand that this is not an issue in a hurry. I'll go to > another patch that needs review. Since we're getting towards the end of the CF is it time to pick this up again? Thanks, -- -David da...@pgmasters.net -- Sent via

Re: [HACKERS] WAL logging problem in 9.4.3?

Thank you for the comment. I understand that this is not an issue in a hurry so don't bother to reply. At Tue, 15 Mar 2016 18:21:34 +0100, Michael Paquier wrote in

Re: [HACKERS] WAL logging problem in 9.4.3?

On Fri, Mar 11, 2016 at 9:32 AM, Kyotaro HORIGUCHI wrote: > At Fri, 19 Feb 2016 22:27:00 +0900, Michael Paquier > wrote in > >> > Worth noting that this patch does

Re: [HACKERS] WAL logging problem in 9.4.3?

Hello, I considered on the original issue. At Fri, 19 Feb 2016 22:27:00 +0900, Michael Paquier wrote in > > Worth noting that this patch does not address the problem with index > > relations when a

Re: [HACKERS] WAL logging problem in 9.4.3?

On Fri, Feb 19, 2016 at 4:33 PM, Michael Paquier wrote: > On Thu, Feb 18, 2016 at 4:27 PM, Michael Paquier > wrote: >> On Thu, Feb 4, 2016 at 3:24 PM, Heikki Linnakangas wrote: >>> I dropped the ball on this one back in

Re: [HACKERS] WAL logging problem in 9.4.3?

On Thu, Feb 18, 2016 at 4:27 PM, Michael Paquier wrote: > On Thu, Feb 4, 2016 at 3:24 PM, Heikki Linnakangas wrote: >> I dropped the ball on this one back in July, so here's an attempt to revive >> this thread. >> >> I spent some time fixing the

Re: [HACKERS] WAL logging problem in 9.4.3?

On Thu, Feb 4, 2016 at 3:24 PM, Heikki Linnakangas wrote: > I dropped the ball on this one back in July, so here's an attempt to revive > this thread. > > I spent some time fixing the remaining issues with the prototype patch I > posted earlier, and rebased that on top of current

Re: [HACKERS] WAL logging problem in 9.4.3?

On 22/10/15 03:56, Michael Paquier wrote: On Wed, Oct 21, 2015 at 11:53 PM, Alvaro Herrera wrote: Heikki Linnakangas wrote: Thanks. For comparison, I wrote a patch to implement what I had in mind. When a WAL-skipping COPY begins, we add an entry for that relation

Re: [HACKERS] WAL logging problem in 9.4.3?

Heikki Linnakangas wrote: > Thanks. For comparison, I wrote a patch to implement what I had in mind. > > When a WAL-skipping COPY begins, we add an entry for that relation in a > "pending-fsyncs" hash table. Whenever we perform any action on a heap that > would normally be WAL-logged, we check

Re: [HACKERS] WAL logging problem in 9.4.3?

On Wed, Oct 21, 2015 at 11:53 PM, Alvaro Herrera wrote: > Heikki Linnakangas wrote: > >> Thanks. For comparison, I wrote a patch to implement what I had in mind. >> >> When a WAL-skipping COPY begins, we add an entry for that relation in a >> "pending-fsyncs" hash table.

Re: [HACKERS] WAL logging problem in 9.4.3?

On 07/23/2015 09:38 PM, Robert Haas wrote: On Wed, Jul 22, 2015 at 12:21 PM, Heikki Linnakangas hlinn...@iki.fi wrote: This is more invasive than I'd like to backpatch, but I think it's the simplest approach that works, and doesn't disable any of the important optimizations we have. Hmm,

Re: [HACKERS] WAL logging problem in 9.4.3?

On Wed, Jul 22, 2015 at 12:21 PM, Heikki Linnakangas hlinn...@iki.fi wrote: This is more invasive than I'd like to backpatch, but I think it's the simplest approach that works, and doesn't disable any of the important optimizations we have. Hmm, isn't HeapNeedsWAL() a lot more costly than

Re: [HACKERS] WAL logging problem in 9.4.3?

On 22 July 2015 at 17:21, Heikki Linnakangas hlinn...@iki.fi wrote: When a WAL-skipping COPY begins, we add an entry for that relation in a pending-fsyncs hash table. Whenever we perform any action on a heap that would normally be WAL-logged, we check if the relation is in the hash table,

Re: [HACKERS] WAL logging problem in 9.4.3?

On 07/22/2015 11:18 AM, Simon Riggs wrote: On 10 July 2015 at 00:06, Tom Lane t...@sss.pgh.pa.us wrote: Andres Freund and...@anarazel.de writes: On 2015-07-06 11:49:54 -0400, Tom Lane wrote: Rather than reverting cab9a0656c36739f, which would re-introduce a different performance problem,

Re: [HACKERS] WAL logging problem in 9.4.3?

On 2015-07-21 21:37:41 +0200, Martijn van Oosterhout wrote: On Tue, Jul 21, 2015 at 02:24:47PM -0400, Todd A. Cook wrote: Hi, This thread seemed to trail off without a resolution. Was anything done? Not that I can tell. Heikki and I had some in-person conversation about it at a

Re: [HACKERS] WAL logging problem in 9.4.3?

On 10 July 2015 at 00:06, Tom Lane t...@sss.pgh.pa.us wrote: Andres Freund and...@anarazel.de writes: On 2015-07-06 11:49:54 -0400, Tom Lane wrote: Rather than reverting cab9a0656c36739f, which would re-introduce a different performance problem, perhaps we could have COPY create a new

Re: [HACKERS] WAL logging problem in 9.4.3?

Hi, This thread seemed to trail off without a resolution. Was anything done? (See more below.) On 07/09/15 19:06, Tom Lane wrote: Andres Freund and...@anarazel.de writes: On 2015-07-06 11:49:54 -0400, Tom Lane wrote: Rather than reverting cab9a0656c36739f, which would re-introduce a

Re: [HACKERS] WAL logging problem in 9.4.3?

On Tue, Jul 21, 2015 at 02:24:47PM -0400, Todd A. Cook wrote: Hi, This thread seemed to trail off without a resolution. Was anything done? Not that I can tell. I was the original poster of this thread. We've worked around the issue by placing a CHECKPOINT command at the end of the migration

Re: [HACKERS] WAL logging problem in 9.4.3?

On 07/10/2015 02:06 AM, Tom Lane wrote: Andres Freund and...@anarazel.de writes: On 2015-07-06 11:49:54 -0400, Tom Lane wrote: Rather than reverting cab9a0656c36739f, which would re-introduce a different performance problem, perhaps we could have COPY create a new relfilenode when it does

Re: [HACKERS] WAL logging problem in 9.4.3?

On 2015-07-09 19:06:11 -0400, Tom Lane wrote: What evidence have you got to base that value judgement on? cab9a0656c36739f was based on an actual user complaint, so we have good evidence that there are people out there who care about the cost of truncating a table many times in one

Re: [HACKERS] WAL logging problem in 9.4.3?

On 2015-07-10 19:23:28 +0900, Fujii Masao wrote: Maybe I'm missing something. But I start wondering why TRUNCATE and INSERT (or even all the operations on the table created at the current transaction) need to be WAL-logged while COPY can be optimized. If no WAL records are generated on that

Re: [HACKERS] WAL logging problem in 9.4.3?

On 2015-07-10 13:38:50 +0300, Heikki Linnakangas wrote: In the long-term, I'd like to refactor this whole thing so that we never WAL-log any operations on a relation that's created in the same transaction (when wal_level=minimal). Instead, at COMMIT, we'd fsync() the relation, or if it's

Re: [HACKERS] WAL logging problem in 9.4.3?

On Fri, Jul 10, 2015 at 2:27 AM, Tom Lane t...@sss.pgh.pa.us wrote: Fujii Masao masao.fu...@gmail.com writes: On Tue, Jul 7, 2015 at 12:49 AM, Tom Lane t...@sss.pgh.pa.us wrote: One idea I had was to allow the COPY optimization only if the heap file is physically zero-length at the time the

Re: [HACKERS] WAL logging problem in 9.4.3?

Fujii Masao masao.fu...@gmail.com writes: On Tue, Jul 7, 2015 at 12:49 AM, Tom Lane t...@sss.pgh.pa.us wrote: One idea I had was to allow the COPY optimization only if the heap file is physically zero-length at the time the COPY starts. This seems not helpful for the case where TRUNCATE is

Re: [HACKERS] WAL logging problem in 9.4.3?

On 2015-07-06 11:49:54 -0400, Tom Lane wrote: One idea I had was to allow the COPY optimization only if the heap file is physically zero-length at the time the COPY starts. That would still be able to optimize in all the cases we care about making COPY fast for. Rather than reverting

Re: [HACKERS] WAL logging problem in 9.4.3?

On Tue, Jul 7, 2015 at 12:49 AM, Tom Lane t...@sss.pgh.pa.us wrote: Andres Freund and...@anarazel.de writes: On 2015-07-06 11:14:40 -0400, Tom Lane wrote: The COUNT() correctly says 11 rows, but after crash-and-recover, only the row with -1 is there. This is because the INSERT writes out an

Re: [HACKERS] WAL logging problem in 9.4.3?

Andres Freund and...@anarazel.de writes: On 2015-07-06 11:49:54 -0400, Tom Lane wrote: Rather than reverting cab9a0656c36739f, which would re-introduce a different performance problem, perhaps we could have COPY create a new relfilenode when it does this. That should be safe if the table was

Re: [HACKERS] WAL logging problem in 9.4.3?

On Sat, Jul 4, 2015 at 2:26 AM, Andres Freund and...@anarazel.de wrote: On 2015-07-03 19:02:29 +0200, Andres Freund wrote: Maybe I'm just daft right now (35C outside, 32 inside, so ...), but I'm right now missing how the whole skip wal logging if relation has just been truncated optimization

Re: [HACKERS] WAL logging problem in 9.4.3?

Fujii Masao masao.fu...@gmail.com writes: On Sat, Jul 4, 2015 at 2:26 AM, Andres Freund and...@anarazel.de wrote: We actually used to use a different relfilenode, but optimized that away: cab9a0656c36739f59277b34fea8ab9438395869 commit cab9a0656c36739f59277b34fea8ab9438395869 Author: Tom

Re: [HACKERS] WAL logging problem in 9.4.3?

Andres Freund and...@anarazel.de writes: On 2015-07-06 11:14:40 -0400, Tom Lane wrote: The COUNT() correctly says 11 rows, but after crash-and-recover, only the row with -1 is there. This is because the INSERT writes out an INSERT+INIT WAL record, which we happily replay, clobbering the data

Re: [HACKERS] WAL logging problem in 9.4.3?

On 2015-07-06 11:14:40 -0400, Tom Lane wrote: BEGIN; CREATE TABLE test (i int primary key); INSERT INTO test VALUES(-1); \copy test from /tmp/num.csv with csv COMMIT; SELECT COUNT(*) FROM test; The COUNT() correctly says 11 rows, but after crash-and-recover, only the row with -1 is

Re: [HACKERS] WAL logging problem in 9.4.3?

On Fri, Jul 3, 2015 at 3:01 PM, Martijn van Oosterhout klep...@svana.org wrote: On Fri, Jul 03, 2015 at 02:34:44PM +0900, Fujii Masao wrote: Hmm, for me it is 100% reproducable. Are you familiar with Docker? I can probably construct a Dockerfile that reproduces it pretty reliably. I could

Re: [HACKERS] WAL logging problem in 9.4.3?

Fujii Masao masao.fu...@gmail.com writes: The optimization of TRUNCATE opereation that we can use when CREATE TABLE and TRUNCATE are executed in the same transaction block seems to cause the problem. In this case, only index file truncation is logged, and index creation in btbuild() is not

Re: [HACKERS] WAL logging problem in 9.4.3?

On Fri, Jul 03, 2015 at 12:53:56PM -0400, Tom Lane wrote: Fujii Masao masao.fu...@gmail.com writes: Okay, so probably we need to change WAL replay of TRUNCATE so that the index file is truncated to one containing only meta page instead of empty one. That is, the WAL replay of TRUNCATE would

Re: [HACKERS] WAL logging problem in 9.4.3?

Fujii Masao masao.fu...@gmail.com writes: Okay, so probably we need to change WAL replay of TRUNCATE so that the index file is truncated to one containing only meta page instead of empty one. That is, the WAL replay of TRUNCATE would need to call index_build() after smgrtruncate() maybe. That

Re: [HACKERS] WAL logging problem in 9.4.3?

On 2015-07-03 18:49:31 +0200, Andres Freund wrote: But the more interesting question is why that's not hhappening today. RelationTruncateIndexes() does call the index_build() which should end up WAL logging the index creation. So that's because there's an XLogIsNeeded() preventing it. Maybe

Re: [HACKERS] WAL logging problem in 9.4.3?

On 2015-07-03 19:14:26 +0200, Martijn van Oosterhout wrote: Am I missing something. ISTM that if the truncate record was simply not logged at all everything would work fine. The whole point is that the table was created in this transaction and so if it exists the table on disk must be the

Re: [HACKERS] WAL logging problem in 9.4.3?

On 2015-07-03 19:02:29 +0200, Andres Freund wrote: Maybe I'm just daft right now (35C outside, 32 inside, so ...), but I'm right now missing how the whole skip wal logging if relation has just been truncated optimization can ever actually be crashsafe unless we use a new relfilenode (which we

Re: [HACKERS] WAL logging problem in 9.4.3?

On Fri, Jul 3, 2015 at 11:52 PM, Tom Lane t...@sss.pgh.pa.us wrote: Fujii Masao masao.fu...@gmail.com writes: The optimization of TRUNCATE opereation that we can use when CREATE TABLE and TRUNCATE are executed in the same transaction block seems to cause the problem. In this case, only index

Re: [HACKERS] WAL logging problem in 9.4.3?

On 2015-07-04 01:39:42 +0900, Fujii Masao wrote: Okay, so probably we need to change WAL replay of TRUNCATE so that the index file is truncated to one containing only meta page instead of empty one. That is, the WAL replay of TRUNCATE would need to call index_build() after smgrtruncate()

Re: [HACKERS] WAL logging problem in 9.4.3?

On 2015-07-03 18:38:37 -0400, Tom Lane wrote: Why exactly? The first truncation in the (sub)xact would have assigned a new relfilenode, why do we need another one? The file in question will go away on crash/rollback in any case, and no other transaction can see it yet. Consider: BEGIN;

Re: [HACKERS] WAL logging problem in 9.4.3?

Martijn van Oosterhout klep...@svana.org writes: With inserts the WAL records look as follows (relfilenodes changed): ... And amazingly, the database cluster successfuly recovers and there's no error now. So the problem is *only* because there is no data in the table at commit time. Which

Re: [HACKERS] WAL logging problem in 9.4.3?

On 2015-07-03 19:26:05 +0200, Andres Freund wrote: On 2015-07-03 19:02:29 +0200, Andres Freund wrote: Maybe I'm just daft right now (35C outside, 32 inside, so ...), but I'm right now missing how the whole skip wal logging if relation has just been truncated optimization can ever actually

Re: [HACKERS] WAL logging problem in 9.4.3?

Andres Freund and...@anarazel.de writes: On 2015-07-03 19:26:05 +0200, Andres Freund wrote: commit cab9a0656c36739f59277b34fea8ab9438395869 Author: Tom Lane t...@sss.pgh.pa.us Date: Sun Aug 23 19:23:41 2009 + Make TRUNCATE do truncate-in-place when processing a relation that was

Re: [HACKERS] WAL logging problem in 9.4.3?

On Fri, Jul 03, 2015 at 07:21:21PM +0200, Andres Freund wrote: On 2015-07-03 19:14:26 +0200, Martijn van Oosterhout wrote: Am I missing something. ISTM that if the truncate record was simply not logged at all everything would work fine. The whole point is that the table was created in this

Re: [HACKERS] WAL logging problem in 9.4.3?

On Fri, Jul 03, 2015 at 02:34:44PM +0900, Fujii Masao wrote: Hmm, for me it is 100% reproducable. Are you familiar with Docker? I can probably construct a Dockerfile that reproduces it pretty reliably. I could reproduce the problem in the master branch by doing the following steps. Thank

Re: [HACKERS] WAL logging problem in 9.4.3?

On Fri, Jul 3, 2015 at 2:20 PM, Martijn van Oosterhout klep...@svana.org wrote: On Fri, Jul 03, 2015 at 12:21:02AM +0200, Andres Freund wrote: Hi, On 2015-07-03 00:05:24 +0200, Martijn van Oosterhout wrote: === Start with an empty database My guess is you have wal_level = minimal?

Re: [HACKERS] WAL logging problem in 9.4.3?

On Fri, Jul 03, 2015 at 12:21:02AM +0200, Andres Freund wrote: Hi, On 2015-07-03 00:05:24 +0200, Martijn van Oosterhout wrote: === Start with an empty database My guess is you have wal_level = minimal? Default config, was just initdb'd. So yes, the default wal_level = minimal. ===

[HACKERS] WAL logging problem in 9.4.3?

Hoi, I ran into this in our CI setup and I thinks it's an actual bug. The issue appears to be that when a table is created *and truncated* in a single transaction, that the WAL log is logging a truncate record it shouldn't, such that if the database crashes you have a broken index. It would also

Re: [HACKERS] WAL logging problem in 9.4.3?

Hi, On 2015-07-03 00:05:24 +0200, Martijn van Oosterhout wrote: === Start with an empty database My guess is you have wal_level = minimal? ctmp=# begin; BEGIN ctmp=# create table test(id serial primary key); CREATE TABLE ctmp=# truncate table test; TRUNCATE TABLE ctmp=# commit; COMMIT