On Sun, 2009-11-15 at 16:07 +0200, Heikki Linnakangas wrote:
- When replaying b-tree deletions, we currently wait out/cancel all
running (read-only) transactions. We take the ultra-conservative
stance because we don't know how recent the tuples being deleted are.
If we could store a better
On Sun, 2009-11-15 at 16:07 +0200, Heikki Linnakangas wrote:
- When switching from standby mode to normal operation, we momentarily
hold all AccessExclusiveLocks held by prepared xacts twice, needing
twice the lock space. You can run out of lock space at that point,
causing failover to fail.
Simon Riggs wrote:
On Sun, 2009-11-15 at 16:07 +0200, Heikki Linnakangas wrote:
- When switching from standby mode to normal operation, we momentarily
hold all AccessExclusiveLocks held by prepared xacts twice, needing
twice the lock space. You can run out of lock space at that point,
On Sun, 2009-11-15 at 16:07 +0200, Heikki Linnakangas wrote:
- The assumption that b-tree vacuum records don't need conflict
resolution because we did that with the additional cleanup-info record
works ATM, but it hinges on the fact that we don't delete any tuples
marked as killed while we do
Simon Riggs wrote:
On Sun, 2009-11-15 at 16:07 +0200, Heikki Linnakangas wrote:
- The assumption that b-tree vacuum records don't need conflict
resolution because we did that with the additional cleanup-info record
works ATM, but it hinges on the fact that we don't delete any tuples
marked
On Thu, 2009-11-19 at 10:13 +0200, Heikki Linnakangas wrote:
At backend start, we normally take
RowExclusiveLock on the database in postinit.c, but you had to modify
to acquire AccessShareLock instead in standby mode.
The consensus from earlier discussion was that allowing users to grab
On Fri, Nov 20, 2009 at 7:31 AM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
Simon Riggs wrote:
On Fri, 2009-11-20 at 06:47 +, Greg Stark wrote:
I missed the original discussion of this problem, do you happen to
remember the subject or url for the details?
December
On Sun, 2009-11-15 at 17:17 -0500, Tom Lane wrote:
Josh Berkus j...@agliodbs.com writes:
So I'm in favor of committing part of the HS code even if there are
known failure conditions, as long as those conditions are well-defined.
If we're thinking of committing something that is known
On Fri, 2009-11-20 at 16:40 +0900, Josh Berkus wrote:
Yes. I'm realizing that because of the highly techincal nature of the
discussion and the language used few people other than you and Heikki
are aware of the major issues which still need work. It would be
helpful if someone could post a
Greg Stark wrote:
From discussions in the bar it sounds like this was actually a false
start however as the RecentGlobalXmin in the backend doing the split
could be less aggressive than the RecentGlobalXmin used by some other
backend to hit the hint bits leading to inconsistent results :(
Simon Riggs wrote:
Recovery does *not* take the same locks as the original statements on
the master took. For example, the WAL record for an INSERT just makes
its changes without acquiring locks. This is OK as long as we only allow
read-only users to acquire AccessShareLocks. If we allowed
Simon Riggs wrote:
Recovery does *not* take the same locks as the original statements on
the master took. For example, the WAL record for an INSERT just makes
its changes without acquiring locks. This is OK as long as we only allow
read-only users to acquire AccessShareLocks. If we
Tatsuo Ishii wrote:
Sorry for confusion. My example is under normal PostgreSQL, not under
HS enabled.
You get the same result in standby:
postgres=# begin;
BEGIN
postgres=# prepare a(int) as insert into foo values($1);
PREPARE
postgres=# SELECT * FROM pg_locks;
locktype │ database │
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes:
Tatsuo Ishii wrote:
Sorry for confusion. My example is under normal PostgreSQL, not under
HS enabled.
You get the same result in standby:
AFAICT Tatsuo's example just shows that we might wish to add a check
for read-only
Tom Lane wrote:
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes:
Tatsuo Ishii wrote:
Sorry for confusion. My example is under normal PostgreSQL, not under
HS enabled.
You get the same result in standby:
AFAICT Tatsuo's example just shows that we might wish to add a check
On 11/15/09 11:07 PM, Heikki Linnakangas wrote:
- When replaying b-tree deletions, we currently wait out/cancel all
running (read-only) transactions. We take the ultra-conservative stance
because we don't know how recent the tuples being deleted are. If we
could store a better estimate for
On Fri, 2009-11-20 at 11:14 +0900, Josh Berkus wrote:
On 11/15/09 11:07 PM, Heikki Linnakangas wrote:
- When replaying b-tree deletions, we currently wait out/cancel all
running (read-only) transactions. We take the ultra-conservative stance
because we don't know how recent the tuples being
Joshua D. Drake wrote:
On Fri, 2009-11-20 at 11:14 +0900, Josh Berkus wrote:
On 11/15/09 11:07 PM, Heikki Linnakangas wrote:
- When replaying b-tree deletions, we currently wait out/cancel all
running (read-only) transactions. We take the ultra-conservative stance
because we don't
On Thu, 2009-11-19 at 10:13 +0200, Heikki Linnakangas wrote:
Simon Riggs wrote:
Recovery does *not* take the same locks as the original statements on
the master took. For example, the WAL record for an INSERT just makes
its changes without acquiring locks. This is OK as long as we only
On Thu, 2009-11-19 at 17:15 +0900, Tatsuo Ishii wrote:
Simon Riggs wrote:
Recovery does *not* take the same locks as the original statements on
the master took. For example, the WAL record for an INSERT just makes
its changes without acquiring locks. This is OK as long as we only allow
Simon Riggs wrote:
On Thu, 2009-11-19 at 17:15 +0900, Tatsuo Ishii wrote:
Simon Riggs wrote:
Recovery does *not* take the same locks as the original statements on
the master took. For example, the WAL record for an INSERT just makes
its changes without acquiring locks. This is OK as long as we
Joshua D. Drake wrote:
On Fri, 2009-11-20 at 11:14 +0900, Josh Berkus wrote:
On 11/15/09 11:07 PM, Heikki Linnakangas wrote:
- When replaying b-tree deletions, we currently wait out/cancel all
running (read-only) transactions. We take the ultra-conservative stance
because we don't know how
On Fri, Nov 20, 2009 at 11:14 AM, Josh Berkus j...@agliodbs.com wrote:
On 11/15/09 11:07 PM, Heikki Linnakangas wrote:
- When replaying b-tree deletions, we currently wait out/cancel all
running (read-only) transactions. We take the ultra-conservative stance
because we don't know how recent
On Fri, Nov 20, 2009 at 2:58 AM, Andrew Dunstan and...@dunslane.net wrote:
Right. The major use I was hoping for from HS was exactly to be able to run
long-running queries. In once case I am thinking of we have moved the
business intelligence uses off the OLTP server onto a londiste replica,
On Fri, 2009-11-20 at 11:14 +0900, Josh Berkus wrote:
On 11/15/09 11:07 PM, Heikki Linnakangas wrote:
- When replaying b-tree deletions, we currently wait out/cancel all
running (read-only) transactions. We take the ultra-conservative stance
because we don't know how recent the tuples being
On Fri, 2009-11-20 at 06:47 +, Greg Stark wrote:
On Fri, Nov 20, 2009 at 2:58 AM, Andrew Dunstan and...@dunslane.net wrote:
Right. The major use I was hoping for from HS was exactly to be able to run
long-running queries. In once case I am thinking of we have moved the
business
Simon Riggs wrote:
On Fri, 2009-11-20 at 06:47 +, Greg Stark wrote:
I missed the original discussion of this problem, do you happen to
remember the subject or url for the details?
December 2008; hackers; you, me and Heikki.
Yep:
Simon,
sigh This post isn't really very helpful. You aren't providing the
second part of the discussion, nor even requesting that this issue be
fixed. I can see such comments being taken up by people with a clear
interest in dissing HS.
OK, I'm requesting that the issue be fixed. I'm not
Tatsuo Ishii wrote:
Please correct me if I'm wrong. Parse will result in obtaining
RowExclusiveLock on the target table if it is parsing
INSERT/UPDATE/DELETE. If so, is this ok in the standby?
Any attempt to take RowExclusiveLock will fail.
Any attempt to execute INSERT/UPDATE/DELETE will
On Wed, 2009-11-18 at 14:51 +0200, Heikki Linnakangas wrote:
Tatsuo Ishii wrote:
Please correct me if I'm wrong. Parse will result in obtaining
RowExclusiveLock on the target table if it is parsing
INSERT/UPDATE/DELETE. If so, is this ok in the standby?
Any attempt to take
Robert Haas robertmh...@gmail.com wrote:
On Mon, Nov 16, 2009 at 11:07 AM, Kevin Grittner
kevin.gritt...@wicourts.gov wrote:
Tom Lane wrote:
I agree with Heikki that it would be better not to commit as long
as any clear showstoppers remain unresolved.
I agree that it would be better not
On Mon, 2009-11-16 at 13:23 +0900, Tatsuo Ishii wrote:
Just a question:
- Does Hot Standby allow to use prepared query (not prepared
transaction) in standby? I mean: Parse message from frontend can be
accepted by standby?
Yes, no problem with any of those kind of facilities
- Can we
- Does Hot Standby allow to use prepared query (not prepared
transaction) in standby? I mean: Parse message from frontend can be
accepted by standby?
Yes, no problem with any of those kind of facilities
Please correct me if I'm wrong. Parse will result in obtaining
RowExclusiveLock
Tom Lane wrote:
I agree with Heikki that it would be better not to commit as long as
any clear showstoppers remain unresolved.
I agree that it would be better not to commit as long as any of the
following are true:
(1) There are any known issues which would break things for clusters
On Mon, Nov 16, 2009 at 11:07 AM, Kevin Grittner
kevin.gritt...@wicourts.gov wrote:
Tom Lane wrote:
I agree with Heikki that it would be better not to commit as long as
any clear showstoppers remain unresolved.
I agree that it would be better not to commit as long as any of the
following
On Mon, 2009-11-16 at 19:06 +0900, Tatsuo Ishii wrote:
- Does Hot Standby allow to use prepared query (not prepared
transaction) in standby? I mean: Parse message from frontend can be
accepted by standby?
Yes, no problem with any of those kind of facilities
Please correct me
Please correct me if I'm wrong. Parse will result in obtaining
RowExclusiveLock on the target table if it is parsing
INSERT/UPDATE/DELETE. If so, is this ok in the standby?
Any attempt to take RowExclusiveLock will fail.
Any attempt to execute INSERT/UPDATE/DELETE will fail.
This
After some time thinking about the best way forward for Hot Standby, I
have some observations and proposals.
First, the project is very large. We have agreed ways to trim the patch,
yet it remains large. Trying to do everything in one lump is almost
always a bad plan, so we need to phase things.
On Sun, Nov 15, 2009 at 09:06, Simon Riggs si...@2ndquadrant.com wrote:
* Issues relating to handling of prepared transactions
There are some delicate issues surrounding what happens at the end of
recovery if there is a prepared transaction still holding an access
exclusive lock. It is
On Sun, 2009-11-15 at 10:00 +0100, Magnus Hagander wrote:
What does the time depend on?
We need to wait for all current transactions to complete. (i.e. any
backend that has (or could) take an xid or an AccessExclusiveLock before
it commits.). Similar-ish to the wait for a CREATE INDEX
On Sunday, November 15, 2009, Simon Riggs si...@2ndquadrant.com wrote:
On Sun, 2009-11-15 at 10:00 +0100, Magnus Hagander wrote:
What does the time depend on?
We need to wait for all current transactions to complete. (i.e. any
backend that has (or could) take an xid or an AccessExclusiveLock
Simon Riggs wrote:
We need to wait for all current transactions to complete. (i.e. any
backend that has (or could) take an xid or an AccessExclusiveLock before
it commits.). Similar-ish to the wait for a CREATE INDEX CONCURRENTLY.
The standby already performs this wait in the case where we
Simon Riggs wrote:
* Issues relating to handling of prepared transactions
There are some delicate issues surrounding what happens at the end of
recovery if there is a prepared transaction still holding an access
exclusive lock.
Can you describe in more detail what problem this is again? We
On Sun, Nov 15, 2009 at 3:06 AM, Simon Riggs si...@2ndquadrant.com wrote:
Please can we agree a way forwards?
I don't have a strong position on the technical issues, but I am very
much in favor of getting something committed, even something with
limitations, as soon as we practically can.
Simon Riggs wrote:
There are two remaining areas of significant thought/effort:
Here's a list of other TODO items I've collected so far. Some of them
are just improvements or nice-to-have stuff, but some are more serious:
- If WAL recovery runs out of lock space while acquiring an
On Sun, 2009-11-15 at 16:07 +0200, Heikki Linnakangas wrote:
Simon Riggs wrote:
There are two remaining areas of significant thought/effort:
Here's a list of other TODO items I've collected so far. Some of them
are just improvements or nice-to-have stuff, but some are more serious:
- If
On Sun, Nov 15, 2009 at 2:32 PM, Simon Riggs si...@2ndquadrant.com wrote:
- The standby delay is measured as current timestamp - timestamp of
last replayed commit record. If there's little activity in the master,
that can lead to surprising results. For example, imagine that
max_standby_delay
Simon Riggs wrote:
On Sun, 2009-11-15 at 16:07 +0200, Heikki Linnakangas wrote:
- If WAL recovery runs out of lock space while acquiring an
AccessExclusiveLock on behalf of a transaction that ran in the master,
it will FATAL and abort recovery, bringing down the standby. Seems like
it should
On Sun, 2009-11-15 at 14:47 +, Greg Stark wrote:
On Sun, Nov 15, 2009 at 2:32 PM, Simon Riggs si...@2ndquadrant.com wrote:
- The standby delay is measured as current timestamp - timestamp of
last replayed commit record. If there's little activity in the master,
that can lead to
On Sun, 2009-11-15 at 16:50 +0200, Heikki Linnakangas wrote:
Simon Riggs wrote:
On Sun, 2009-11-15 at 16:07 +0200, Heikki Linnakangas wrote:
- If WAL recovery runs out of lock space while acquiring an
AccessExclusiveLock on behalf of a transaction that ran in the master,
it will FATAL and
On Sun, Nov 15, 2009 at 9:50 AM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
Simon Riggs wrote:
On Sun, 2009-11-15 at 16:07 +0200, Heikki Linnakangas wrote:
- If WAL recovery runs out of lock space while acquiring an
AccessExclusiveLock on behalf of a transaction that ran in
Robert Haas wrote:
OK, but... there will always be things that will bring down the
stand-by, just as there will always be things that can bring down the
primary. A bucket of ice-water will probably do it, for example. I
mean, it would be great to make it better, but is it so bad that we
On Sun, 2009-11-15 at 16:07 +0200, Heikki Linnakangas wrote:
The assumption that b-tree vacuum records don't need conflict
resolution because we did that with the additional cleanup-info record
works ATM, but it hinges on the fact that we don't delete any tuples
marked as killed while we do
Simon Riggs wrote:
On Sun, 2009-11-15 at 16:07 +0200, Heikki Linnakangas wrote:
The assumption that b-tree vacuum records don't need conflict
resolution because we did that with the additional cleanup-info record
works ATM, but it hinges on the fact that we don't delete any tuples
marked as
On Sun, 2009-11-15 at 19:36 +0200, Heikki Linnakangas wrote:
Simon Riggs wrote:
On Sun, 2009-11-15 at 16:07 +0200, Heikki Linnakangas wrote:
The assumption that b-tree vacuum records don't need conflict
resolution because we did that with the additional cleanup-info record
works ATM,
On Sun, Nov 15, 2009 at 11:49 AM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
Agreed. Believe me, I'd like to have this committed as much as everyone
else. But once I do that, I'm also committing myself to fix all the
remaining issues before the release. The criteria for
Simon Riggs wrote:
On Sun, 2009-11-15 at 19:36 +0200, Heikki Linnakangas wrote:
Simon Riggs wrote:
On Sun, 2009-11-15 at 16:07 +0200, Heikki Linnakangas wrote:
The assumption that b-tree vacuum records don't need conflict
resolution because we did that with the additional cleanup-info record
Robert Haas wrote:
But a
query getting canceled because it touches a lot of tables sounds more
like a limitation than an outright bug,
It's not that the query might get canceled. It will abort WAL recovery,
kill all backends, and bring the whole standby down.
--
Heikki Linnakangas
On Sun, 2009-11-15 at 13:15 -0500, Robert Haas wrote:
I know Simon has said that he feels that the effort
to finish the HS and SR patches for 9/15 was somewhat of an artificial
deadline, but ISTM that with only 3 months remaining until the close
of the final CommitFest for this release, and
On Sun, 2009-11-15 at 20:30 +0200, Heikki Linnakangas wrote:
The LSN doesn't help there, because when an itemid is marked as dead,
the LSN is not updated.
I was thinking we could update the index block LSN without writing WAL
using the LSN of the heap block that leads to the killed tuple.
Simon Riggs wrote:
On Sun, 2009-11-15 at 20:30 +0200, Heikki Linnakangas wrote:
The LSN doesn't help there, because when an itemid is marked as dead,
the LSN is not updated.
I was thinking we could update the index block LSN without writing WAL
using the LSN of the heap block that leads
On Sun, 2009-11-15 at 20:37 +0200, Heikki Linnakangas wrote:
Robert Haas wrote:
But a
query getting canceled because it touches a lot of tables sounds more
like a limitation than an outright bug,
It's not that the query might get canceled. It will abort WAL recovery,
kill all backends,
On Sun, Nov 15, 2009 at 7:29 PM, Simon Riggs si...@2ndquadrant.com wrote:
You agree there should be two phases?
I don't understand this repeated suggestion of phases. Nobody's
every suggested that we would refuse to add new features to HS after
the initial commit or the 8.5 release. Of course
On Sun, 2009-11-15 at 21:20 +0200, Heikki Linnakangas wrote:
Simon Riggs wrote:
On Sun, 2009-11-15 at 20:30 +0200, Heikki Linnakangas wrote:
The LSN doesn't help there, because when an itemid is marked as dead,
the LSN is not updated.
I was thinking we could update the index block
Simon Riggs wrote:
You agree there should be two phases?
I'm hesitant to say 'yes', because then you will harass me with but you
said that you would be OK with fixing X, Y, Z later! Why don't you
commit already!.
Of course there should be several phases! We've *already* punted a lot
of stuff
Simon Riggs wrote:
On Sun, 2009-11-15 at 21:20 +0200, Heikki Linnakangas wrote:
Simon Riggs wrote:
On Sun, 2009-11-15 at 20:30 +0200, Heikki Linnakangas wrote:
The LSN doesn't help there, because when an itemid is marked as dead,
the LSN is not updated.
I was thinking we could update the
On Sun, 2009-11-15 at 21:56 +0200, Heikki Linnakangas wrote:
If you actually want to help, can you please focus on fixing the
must-fix bugs we know about? We can then discuss which of the
remaining known issues we're willing to live with.
I intend to work on all of the issues, so not sure
Simon Riggs wrote:
Right now, I don't know which you
consider to be the must-fix issues, hence the thread.
Ok, could you tackle the b-tree vacuum bug, where we neglect to pin the
index pages after the last b-tree vacuum record? Thanks.
--
Heikki Linnakangas
EnterpriseDB
On Sun, 2009-11-15 at 22:45 +0200, Heikki Linnakangas wrote:
Simon Riggs wrote:
Right now, I don't know which you
consider to be the must-fix issues, hence the thread.
Ok, could you tackle the b-tree vacuum bug, where we neglect to pin the
index pages after the last b-tree vacuum record?
On 11/15/09 12:58 PM, Simon Riggs wrote:
On Sun, 2009-11-15 at 22:45 +0200, Heikki Linnakangas wrote:
Simon Riggs wrote:
Right now, I don't know which you
consider to be the must-fix issues, hence the thread.
Ok, could you tackle the b-tree vacuum bug, where we neglect to pin the
index pages
On Sun, 2009-11-15 at 23:14 +0200, Heikki Linnakangas wrote:
Simon Riggs wrote:
On Sun, 2009-11-15 at 22:45 +0200, Heikki Linnakangas wrote:
Simon Riggs wrote:
Right now, I don't know which you
consider to be the must-fix issues, hence the thread.
Ok, could you tackle the b-tree vacuum
Simon Riggs wrote:
On Sun, 2009-11-15 at 22:45 +0200, Heikki Linnakangas wrote:
Simon Riggs wrote:
Right now, I don't know which you
consider to be the must-fix issues, hence the thread.
Ok, could you tackle the b-tree vacuum bug, where we neglect to pin the
index pages after the last b-tree
Josh Berkus j...@agliodbs.com writes:
So I'm in favor of committing part of the HS code even if there are
known failure conditions, as long as those conditions are well-defined.
If we're thinking of committing something that is known broken, I would
want to have a clearly defined and
On Nov 15, 2009, at 2:17 PM, Tom Lane wrote:
So I'm in favor of committing part of the HS code even if there are
known failure conditions, as long as those conditions are well-defined.
If we're thinking of committing something that is known broken, I would
want to have a clearly defined and
David E. Wheeler da...@kineticode.com writes:
On Nov 15, 2009, at 2:17 PM, Tom Lane wrote:
I agree with Heikki that it would be better not to commit as long as
any clear showstoppers remain unresolved.
If ever there were an argument for topic branches, *this is it*.
How so? They've got a
On Nov 15, 2009, at 4:19 PM, Simon Riggs si...@2ndquadrant.com wrote:
On Sun, 2009-11-15 at 23:14 +0200, Heikki Linnakangas wrote:
Simon Riggs wrote:
On Sun, 2009-11-15 at 22:45 +0200, Heikki Linnakangas wrote:
Simon Riggs wrote:
Right now, I don't know which you
consider to be the must-fix
Just a question:
- Does Hot Standby allow to use prepared query (not prepared
transaction) in standby? I mean: Parse message from frontend can be
accepted by standby?
- Can we create tempory tables in standby?
--
Tatsuo Ishii
SRA OSS, Inc. Japan
After some time thinking about the best way
Tatsuo Ishii wrote:
Just a question:
- Does Hot Standby allow to use prepared query (not prepared
transaction) in standby? I mean: Parse message from frontend can be
accepted by standby?
Yes.
- Can we create tempory tables in standby?
No, because creating a temporary table needs to
Tatsuo Ishii wrote:
Just a question:
- Does Hot Standby allow to use prepared query (not prepared
transaction) in standby? I mean: Parse message from frontend can be
accepted by standby?
Yes.
In my understanding, Parse will aquire locks on the target table. Is
this harmless?
--
Tatsuo Ishii wrote:
In my understanding, Parse will aquire locks on the target table. Is
this harmless?
That's ok, you can take AccessShareLocks in a standby. All queries lock
the target table (in AccessShare mode).
--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com
--
80 matches
Mail list logo
