Re: [HACKERS] Multixid hindsight design

2015-11-12 Thread Robert Haas
On Mon, Nov 9, 2015 at 9:57 PM, Craig Ringer wrote: > On 10 November 2015 at 02:26, Robert Haas wrote: >>> I'd like to see, say, python and the 'unittest' module added, and >>> to see acceptance of tests using psycopg2 to stream and decode WAL >>> from a slot. >> >> I actually kind of hate this s

Re: [HACKERS] Multixid hindsight design

2015-11-09 Thread Craig Ringer
On 10 November 2015 at 02:26, Robert Haas wrote: >> I'd like to see, say, python and the 'unittest' module added, and >> to see acceptance of tests using psycopg2 to stream and decode WAL >> from a slot. > > I actually kind of hate this sort of thing. For what it's worth, I don't actually like it

Re: [HACKERS] Multixid hindsight design

2015-11-09 Thread Tom Lane
Robert Haas writes: > On Mon, Nov 9, 2015 at 2:02 AM, Craig Ringer wrote: >> Something that'd really help with that, IMO, would be weakening the >> requirement that everything be C or be core Perl. Instead require that >> configure detect whether or not facilities for some tests are present, >> a

Re: [HACKERS] Multixid hindsight design

2015-11-09 Thread Robert Haas
On Mon, Nov 9, 2015 at 2:02 AM, Craig Ringer wrote: > On 25 June 2015 at 00:52, Robert Haas wrote: >> I agree that we can do much better at testing than we traditionally >> have done, and I think pretty much everyone in the room for the >> developer unconference session on testing at PGCon was al

Fwd: [HACKERS] Multixid hindsight design

2015-11-08 Thread Craig Ringer
On 25 June 2015 at 00:52, Robert Haas wrote: > I agree that we can do much better at testing than we traditionally > have done, and I think pretty much everyone in the room for the > developer unconference session on testing at PGCon was also in > agreement with that. I really like the idea of t

Re: [HACKERS] Multixid hindsight design

2015-06-24 Thread Simon Riggs
On 24 June 2015 at 16:30, Simon Riggs wrote: > Though TED sounds nice, the way to avoid another round of on-disk bugs is > by using a new kind of testing, not simply by moving the bits around. > > It might be argued that we are increasing the diagnostic/forensic > capabilities by making CIDs mor

Re: [HACKERS] Multixid hindsight design

2015-06-24 Thread Robert Haas
On Wed, Jun 24, 2015 at 11:30 AM, Simon Riggs wrote: > Though TED sounds nice, the way to avoid another round of on-disk bugs is by > using a new kind of testing, not simply by moving the bits around. I agree that we can do much better at testing than we traditionally have done, and I think prett

Re: [HACKERS] Multixid hindsight design

2015-06-24 Thread Simon Riggs
On 24 June 2015 at 14:57, Robert Haas wrote: > On Fri, Jun 5, 2015 at 10:46 AM, Robert Haas > wrote: > > It would be a great deal nicer if we didn't have to keep ANY of the > > transactional data for a tuple around once it's all-visible. Heikki > > defined ephemeral as "only needed when xmin or

Re: [HACKERS] Multixid hindsight design

2015-06-24 Thread Robert Haas
On Fri, Jun 5, 2015 at 10:46 AM, Robert Haas wrote: > It would be a great deal nicer if we didn't have to keep ANY of the > transactional data for a tuple around once it's all-visible. Heikki > defined ephemeral as "only needed when xmin or xmax is in-progress", > but if we extended that definiti

Re: [HACKERS] Multixid hindsight design

2015-06-05 Thread Robert Haas
On Fri, Jun 5, 2015 at 10:27 AM, Tom Lane wrote: >> That way if we need to make Offsets SLRU persistent it won't bloat. >> We then leave the Members SLRU as non-persistent, just as it was <9.3 > > I don't think you can do that, because it supposes that locking XIDs need > not be remembered across

Re: [HACKERS] Multixid hindsight design

2015-06-05 Thread Tom Lane
Simon Riggs writes: > On 11 May 2015 at 22:20, Heikki Linnakangas wrote: >> So the lesson here is that having a permanent pg_multixact is not nice, >> and we should get rid of it. Here's how to do that: > An alternate proposal: > 1. Store only the Locking xids in the Members SLRU > 2. In the Of

Re: [HACKERS] Multixid hindsight design

2015-06-05 Thread Robert Haas
On Fri, Jun 5, 2015 at 6:17 AM, Simon Riggs wrote: > I think we should think back to exactly what we are trying to store, why and > for how long. A clear definition of what we are trying to achieve is > essential to solving the problem. > > In my understanding we need to store > * at most one xid

Re: [HACKERS] Multixid hindsight design

2015-06-05 Thread Simon Riggs
On 5 June 2015 at 11:02, Andres Freund wrote: > On 2015-06-05 10:45:09 +0100, Simon Riggs wrote: > > On 1 June 2015 at 20:53, Thomas Munro > wrote: > > > > > On Tue, May 12, 2015 at 9:20 AM, Heikki Linnakangas > > > wrote: > > > > The beauty of this would be that the TED entries can be zapped a

Re: [HACKERS] Multixid hindsight design

2015-06-05 Thread Simon Riggs
On 11 May 2015 at 22:20, Heikki Linnakangas wrote: > I'd like to discuss how we should've implemented the infamous 9.3 > multixid/row-locking stuff, and perhaps still should in 9.6. Hindsight is > always 20/20 - I'll readily admit that I didn't understand the problems > until well after the relea

Re: [HACKERS] Multixid hindsight design

2015-06-05 Thread Andres Freund
On 2015-06-05 10:45:09 +0100, Simon Riggs wrote: > On 1 June 2015 at 20:53, Thomas Munro wrote: > > > On Tue, May 12, 2015 at 9:20 AM, Heikki Linnakangas > > wrote: > > > The beauty of this would be that the TED entries can be zapped at > > restart, > > > just like pg_subtrans, and pg_multixact

Re: [HACKERS] Multixid hindsight design

2015-06-05 Thread Simon Riggs
On 1 June 2015 at 20:53, Thomas Munro wrote: > On Tue, May 12, 2015 at 9:20 AM, Heikki Linnakangas > wrote: > > The beauty of this would be that the TED entries can be zapped at > restart, > > just like pg_subtrans, and pg_multixact before 9.3. It doesn't need to be > > WAL-logged, and we are fr

Re: [HACKERS] Multixid hindsight design

2015-06-01 Thread Greg Stark
On Mon, Jun 1, 2015 at 8:53 PM, Thomas Munro wrote: > What about prepared transactions? They can lock rows FOR SHARE that > survive server restarts. And they can make update chains that are still uncommitted after a restart. I think we should think separately about what information we want to s

Re: [HACKERS] Multixid hindsight design

2015-06-01 Thread Thomas Munro
On Tue, May 12, 2015 at 9:20 AM, Heikki Linnakangas wrote: > The beauty of this would be that the TED entries can be zapped at restart, > just like pg_subtrans, and pg_multixact before 9.3. It doesn't need to be > WAL-logged, and we are free to change its on-disk layout even in a minor > release.

Re: [HACKERS] Multixid hindsight design

2015-05-12 Thread Robert Haas
On Mon, May 11, 2015 at 5:20 PM, Heikki Linnakangas wrote: > The main problem with the infamous multixid changes was that it made > pg_multixact a permanent, critical, piece of data. Without it, you cannot > decipher whether some rows have been deleted or not. The 9.3 changes > uncovered pre-exist

Re: [HACKERS] Multixid hindsight design

2015-05-12 Thread Heikki Linnakangas
On 05/12/2015 01:51 AM, Tom Lane wrote: Heikki Linnakangas writes: So the lesson here is that having a permanent pg_multixact is not nice, and we should get rid of it. Here's how to do that: That would be cool, but ... Looking at the tuple header, the CID and CTID fields are only needed, wh

Re: [HACKERS] Multixid hindsight design

2015-05-11 Thread Tom Lane
Heikki Linnakangas writes: > So the lesson here is that having a permanent pg_multixact is not nice, > and we should get rid of it. Here's how to do that: That would be cool, but ... > Looking at the tuple header, the CID and CTID fields are only needed, > when either xmin or xmax is running.

[HACKERS] Multixid hindsight design

2015-05-11 Thread Heikki Linnakangas
I'd like to discuss how we should've implemented the infamous 9.3 multixid/row-locking stuff, and perhaps still should in 9.6. Hindsight is always 20/20 - I'll readily admit that I didn't understand the problems until well after the release - so this isn't meant to bash what's been done. Rather