Re: [HACKERS] Something is broken in logical decoding with CLOBBER_CACHE_ALWAYS
On 2015-01-04 21:54:40 +0100, Andres Freund wrote:
On January 4, 2015 9:51:43 PM CET, Andrew Dunstan and...@dunslane.net wrote:
On 12/15/2014 12:04 PM, Andres Freund wrote:
I think the safest fix would be to defer catchup interrupt
processing
while you're in this mode. You don't really want to be processing
any
remote sinval messages at all, I'd think.
Well, we need to do relmap, smgr and similar things. So I think
that'd
be more complicated than we want.
Where are we on this? Traffic seems to have gone quite but we still
have
a bunch of buildfarm animals red.
I've a simple fix (similar too what I iriginally outkined) which I
plan to post soonish. I've tried a bunch of things roughly in the vein
of Tom's suggestions, but they all are more invasive and still
incomplete.
Attached.
Note that part 1) referenced in the commit message is actually
problematic in all branches. I think we actually should backpatch that
part all the way, because if we ever hit that case the consequences of
the current coding will be rather hard to analyze. If others think so as
well, I'm going to split the commit into two parts, so commit messages
for 9.4 won't reference logical decoding. Since there hasn't been a
report of that error for a long while (~8.1 era), I can also live with
not backpatching further than 9.4.
Greetings,
Andres Freund
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training Services
From e5a6c2e9211561bc5c9985a8bf107d9911aad9f6 Mon Sep 17 00:00:00 2001
From: Andres Freund and...@anarazel.de
Date: Mon, 5 Jan 2015 11:52:05 +0100
Subject: [PATCH] Improve handling of relcache invalidations to currently
invisible relations.
The corner case where a relcache invalidation tried to rebuild the
entry for a referenced relation but couldn't find it in the catalog
wasn't correct.
1) The code tried to RelationCacheDelete/RelationDestroyRelation the
entry. That didn't work when assertions are enabled because the
latter contains an assertion ensuring the refcount is zero. It's
also a bad idea, because by virtue of being referenced somebody
might actually look at the entry, which is possible if the error is
trapped and handled via a subtransaction abort. Instead just error
out, without deleting the entry. As the entry is marked invalid,
the worst that can happen is that we leak some memory.
2) When using a historic snapshot for logical decoding it can validly
happen that a relation that's in the relcache isn't visible to that
historic snapshot. E.g. if the relation is referenced in the query
that uses the SQL interface for logical decoding. While erroring
cleanly, instead of hitting an assertion due to 1) is already an
improvement, it's not good enough. So additionally allow that case
without an error if a historic snapshot is set up - that won't
allow an invalid entry to stay in the cache because it's a) already
marked invalid and will thus be rebuilt during the next access b)
the syscaches will be reset at the end of decoding.
There might be prettier solutions to handle this case, but all that
we could think of so far end up being much more complex than this
simple fix.
This fixes the assertion failures reported by the buildfarm (markhor,
tick, leech) after the introduction of new regression tests in
89fd41b390a4. The failure there weren't actually directly caused by
CLOBBER_CACHE_ALWAYS but via the long runtimes due to it.
Backpatch the whole patch to 9.4, and 1) from above all the way back.
---
src/backend/utils/cache/relcache.c | 21 ++---
1 file changed, 18 insertions(+), 3 deletions(-)
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index c2e574d..2cce2c1 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -2192,9 +2192,24 @@ RelationClearRelation(Relation relation, bool rebuild)
newrel = RelationBuildDesc(save_relid, false);
if (newrel == NULL)
{
- /* Should only get here if relation was deleted */
- RelationCacheDelete(relation);
- RelationDestroyRelation(relation, false);
+ /*
+ * We can validly get here, if we're using a historic snapshot in
+ * which a relation, accessed from outside logical decoding, is
+ * still invisible. In that case it's fine to just mark the
+ * relation as invalid and return - it'll fully get reloaded after
+ * by the cache reset at the end of logical decoding.
+ */
+ if (HistoricSnapshotActive())
+return;
+
+ /*
+ * This situation shouldn't happen as dropping a relation
+ * shouldn't be possible if still referenced
+ * (c.f. CheckTableNotInUse()). But if get here anyway, we can't
+ * just delete the relcache entry as it's possibly still
+ * referenced and the error might get caught and handled via a
+ * subtransaction rollback.
+ */
elog(ERROR, relation %u
Re: [HACKERS] Something is broken in logical decoding with CLOBBER_CACHE_ALWAYS
On 12/15/2014 12:04 PM, Andres Freund wrote: I think the safest fix would be to defer catchup interrupt processing while you're in this mode. You don't really want to be processing any remote sinval messages at all, I'd think. Well, we need to do relmap, smgr and similar things. So I think that'd be more complicated than we want. Where are we on this? Traffic seems to have gone quite but we still have a bunch of buildfarm animals red. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Something is broken in logical decoding with CLOBBER_CACHE_ALWAYS
On January 4, 2015 9:51:43 PM CET, Andrew Dunstan and...@dunslane.net wrote: On 12/15/2014 12:04 PM, Andres Freund wrote: I think the safest fix would be to defer catchup interrupt processing while you're in this mode. You don't really want to be processing any remote sinval messages at all, I'd think. Well, we need to do relmap, smgr and similar things. So I think that'd be more complicated than we want. Where are we on this? Traffic seems to have gone quite but we still have a bunch of buildfarm animals red. I've a simple fix (similar too what I iriginally outkined) which I plan to post soonish. I've tried a bunch of things roughly in the vein of Tom's suggestions, but they all are more invasive and still incomplete. Andres -- Please excuse brevity and formatting - I am writing this on my mobile phone. Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Something is broken in logical decoding with CLOBBER_CACHE_ALWAYS
Hi,
On 2014-12-15 10:15:30 -0500, Tom Lane wrote:
The CLOBBER_CACHE_ALWAYS buildfarm members occasionally fail in
contrib/test_decoding with
TRAP: FailedAssertion(!(((bool)((relation)-rd_refcnt == 0))), File:
relcache.c, Line: 1981)
for example here:
http://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=markhordt=2014-12-14%2005%3A50%3A09
and here:
http://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=markhordt=2014-12-13%2021%3A26%3A05
Yes, I've been trying to debug that. I've finally managed to reproduce
locally. Unfortunately it's not directly reproducible on my laptop...
A fair amount of tinkering later I've found out that it's not actually
CLOBBER_CACHE_ALWAYS itself that triggers the problem but catchup
interrupts. It triggers with CLOBBER_CACHE_ALWAYS because that happens
to make parts of the code slow enough to not reach a ordinary
AcceptInvalidationMessages() in time.
The problem is that in the added regression test there can be a
situation where there's a relcache entry that's *not* currently visible
but still referenced. Consider:
SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot',
'test_decoding');
CREATE TABLE somechange(id serial primary key);
INSERT INTO changeresult
SELECT data FROM pg_logical_slot_get_changes(...);
While get_changes stores it data into a tuplestore there can be a moment
where 'changeresult' has a refcnt 0 due to the INSERT, but is
invisible because we're using a historic snapshot from when changeresult
doesn't exist.
Without catchup invalidations and/or an outside reference to a relation
that's normally not a problem because it won't get reloaded from the
caches at an inappropriate time while invisible. Until a few weeks ago
there was no regression test covering that case which is why these
crashes are only there now.
It triggers via:
RelationCacheInvalidate() -
RelationClearRelation(relation, true) -
/* Build temporary entry, but don't link it into hashtable */
newrel = RelationBuildDesc(save_relid, false);
if (newrel == NULL)
{
/* Should only get here if relation was deleted */
RelationCacheDelete(relation);
RelationDestroyRelation(relation, false);
elog(ERROR, relation %u deleted while still in use, save_relid);
}
That path is only hit while refcnt 0
RelationDestroyRelation() has
Assert(RelationHasReferenceCountZero(relation));
So that path doesn't really work... Without assertions we'd just get a
transient ERROR. I think that path should generally loose the
RelationDestroyRelation() - if it's referenced that's surely not the
right thing to do. I'm not actually convinced logical decoding is the
only way that assert can be reached.
Since logical decoding happens inside a subtransaction (when called via
SQL) and invalidate caches one relatively straightforward way to fix
this would be to add something roughly akin to:
Assert(!relation-rd_isvalid)
/*
* This should only happen when we're doing logical decoding where
* it can happen when [above].
*/
if (HistoricSnapshotActive())
return;
There's no chance that such a entry could survive too long in an invalid
state (minus preexisting bugs independent of decoding) since we hit the
same paths that subtransaction abort hits.
Greetings,
Andres Freund
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Something is broken in logical decoding with CLOBBER_CACHE_ALWAYS
Andres Freund and...@2ndquadrant.com writes: On 2014-12-15 10:15:30 -0500, Tom Lane wrote: The CLOBBER_CACHE_ALWAYS buildfarm members occasionally fail in contrib/test_decoding with TRAP: FailedAssertion(!(((bool)((relation)-rd_refcnt == 0))), File: relcache.c, Line: 1981) Without catchup invalidations and/or an outside reference to a relation that's normally not a problem because it won't get reloaded from the caches at an inappropriate time while invisible. Until a few weeks ago there was no regression test covering that case which is why these crashes are only there now. I've always thought that this whole idea of allowing the caches to contain stale information was a Rube Goldberg plan that was going to bite you on the ass eventually. This case isn't doing anything to increase my faith in it, and the proposed patch seems like just another kluge on top of a rickety stack. I think the safest fix would be to defer catchup interrupt processing while you're in this mode. You don't really want to be processing any remote sinval messages at all, I'd think. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Something is broken in logical decoding with CLOBBER_CACHE_ALWAYS
On 2014-12-15 11:30:40 -0500, Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: On 2014-12-15 10:15:30 -0500, Tom Lane wrote: The CLOBBER_CACHE_ALWAYS buildfarm members occasionally fail in contrib/test_decoding with TRAP: FailedAssertion(!(((bool)((relation)-rd_refcnt == 0))), File: relcache.c, Line: 1981) Without catchup invalidations and/or an outside reference to a relation that's normally not a problem because it won't get reloaded from the caches at an inappropriate time while invisible. Until a few weeks ago there was no regression test covering that case which is why these crashes are only there now. I've always thought that this whole idea of allowing the caches to contain stale information was a Rube Goldberg plan that was going to bite you on the ass eventually. I guess we'll see. I don't think this specific case is that dramatic. The complication here is that there's a reference to a relation from outside logical decoding - that's not something actually likely to be used at least by replication solutions. It's also impossible to hit from the walsender interface. We could just prohibit referencing relations like that... The SQL interface primarily is used for regression tests and discovery of the feature. At least as long there's no way to do streaming from SQL which I can't really see how to do. This case isn't doing anything to increase my faith in it, and the proposed patch seems like just another kluge on top of a rickety stack. Another possibility would be to replace else if (!IsTransactionState()) by else if (!IsTransactionState() || HistoricSnapshotActive()) as that pretty much what we need - invalidations during decoding happen either outside of a valid transaction state or when entries aren't referenced anyway. Btw, if ever hit that code path would Assert() out without logical decoding as well - it's not allowed to Destroy() a relation that's still referenced as that assert shows. It's an especially bad idea if the reference is actually from outside a subxact and the error is caught. I think we should just remove the Destroy() call - the normal refcount and/or resowners will take care of deleting the entry later. I think the safest fix would be to defer catchup interrupt processing while you're in this mode. You don't really want to be processing any remote sinval messages at all, I'd think. Well, we need to do relmap, smgr and similar things. So I think that'd be more complicated than we want. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
