Re: [HACKERS] preserving forensic information when we freeze

2014-01-02 Thread Andres Freund
Hi, On 2014-01-02 05:26:26 +, Greg Stark wrote: 2) refetching a row could conceivably end up retrieving different data than was present when the row was originally read. (In some cases that might actually be the intended behaviour) That's possible with system columns as well. In the

Re: [HACKERS] preserving forensic information when we freeze

2014-01-02 Thread Robert Haas
On Thu, Jan 2, 2014 at 12:26 AM, Greg Stark st...@mit.edu wrote: I fail to see why. What is so ugly about this: select x.* from pgbench_accounts a, pg_tuple_header(a.tableoid, a.ctid) x; Two points: 1) it's a bit weird to go to this effort to eliminate system columns by using a scheme

Re: [HACKERS] preserving forensic information when we freeze

2014-01-02 Thread Andres Freund
On 2014-01-02 07:35:11 -0500, Robert Haas wrote: On Thu, Jan 2, 2014 at 12:26 AM, Greg Stark st...@mit.edu wrote: I fail to see why. What is so ugly about this: select x.* from pgbench_accounts a, pg_tuple_header(a.tableoid, a.ctid) x; Two points: 1) it's a bit weird to go to this

Re: [HACKERS] preserving forensic information when we freeze

2014-01-02 Thread Robert Haas
On Thu, Jan 2, 2014 at 7:40 AM, Andres Freund and...@2ndquadrant.com wrote: On 2014-01-02 07:35:11 -0500, Robert Haas wrote: On Thu, Jan 2, 2014 at 12:26 AM, Greg Stark st...@mit.edu wrote: I fail to see why. What is so ugly about this: select x.* from pgbench_accounts a,

Re: [HACKERS] preserving forensic information when we freeze

2014-01-02 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Thu, Jan 2, 2014 at 12:26 AM, Greg Stark st...@mit.edu wrote: 1) it's a bit weird to go to this effort to eliminate system columns by using a scheme that depends on having a system column -- ctid At any rate, my goal isn't really to get rid of

Re: [HACKERS] preserving forensic information when we freeze

2014-01-02 Thread Andres Freund
On 2014-01-02 09:40:54 -0500, Tom Lane wrote: Robert Haas robertmh...@gmail.com writes: On Thu, Jan 2, 2014 at 12:26 AM, Greg Stark st...@mit.edu wrote: 1) it's a bit weird to go to this effort to eliminate system columns by using a scheme that depends on having a system column -- ctid

Re: [HACKERS] preserving forensic information when we freeze

2014-01-02 Thread Tom Lane
Andres Freund and...@2ndquadrant.com writes: On 2014-01-02 09:40:54 -0500, Tom Lane wrote: Actually, I thought the function approach was a good proposal. You are right that func(tab.*) isn't going to work, because it's going to get a Datum-ified tuple not a pointer to raw on-disk storage.

Re: [HACKERS] preserving forensic information when we freeze

2014-01-02 Thread Robert Haas
On Thu, Jan 2, 2014 at 12:46 PM, Tom Lane t...@sss.pgh.pa.us wrote: Andres Freund and...@2ndquadrant.com writes: On 2014-01-02 09:40:54 -0500, Tom Lane wrote: Actually, I thought the function approach was a good proposal. You are right that func(tab.*) isn't going to work, because it's going

Re: [HACKERS] preserving forensic information when we freeze

2014-01-02 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Thu, Jan 2, 2014 at 12:46 PM, Tom Lane t...@sss.pgh.pa.us wrote: In any case, upon further reflection I'm not convinced that doing this with a SELECT-based query is the right thing, no matter whether the query looks at a function or a system column;

Re: [HACKERS] preserving forensic information when we freeze

2014-01-02 Thread Andres Freund
On 2014-01-02 12:46:34 -0500, Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: On 2014-01-02 09:40:54 -0500, Tom Lane wrote: Actually, I thought the function approach was a good proposal. You are right that func(tab.*) isn't going to work, because it's going to get a

Re: [HACKERS] preserving forensic information when we freeze

2014-01-02 Thread Robert Haas
On Thu, Jan 2, 2014 at 2:03 PM, Tom Lane t...@sss.pgh.pa.us wrote: I both agree and disagree with this. I think that pgstattuple is useful, and I further agree that adding a stat to it about how much of the heap is frozen would be worthwhile. However, an aggregate number isn't always what

Re: [HACKERS] preserving forensic information when we freeze

2014-01-02 Thread Tom Lane
Andres Freund and...@2ndquadrant.com writes: On 2014-01-02 12:46:34 -0500, Tom Lane wrote: For real forensics work, you need to be able to see all tuples, which makes me think that something akin to pgstattuple is the right API; that is return a set of the header info for all tuples on

Re: [HACKERS] preserving forensic information when we freeze

2014-01-02 Thread Robert Haas
On Thu, Jan 2, 2014 at 2:44 PM, Tom Lane t...@sss.pgh.pa.us wrote: As I commented to Robert, the page-at-a-time behavior of pageinspect is not an API detail we'd want to copy for this. I envision something like select hdr.*, foo.* from tuple_header_details('foo'::regclass)

Re: [HACKERS] preserving forensic information when we freeze

2014-01-02 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: We could certainly add a function that returns SETOF record, taking e.g. regclass as an argument, but it doesn't seem a stretch to me to think that you might want to get tuple header information for some but not all tuples in the relation, and I don't

Re: [HACKERS] preserving forensic information when we freeze

2014-01-02 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Thu, Jan 2, 2014 at 2:44 PM, Tom Lane t...@sss.pgh.pa.us wrote: As I commented to Robert, the page-at-a-time behavior of pageinspect is not an API detail we'd want to copy for this. I envision something like select hdr.*, foo.* from

Re: [HACKERS] preserving forensic information when we freeze

2014-01-02 Thread Robert Haas
On Thu, Jan 2, 2014 at 2:56 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Thu, Jan 2, 2014 at 2:44 PM, Tom Lane t...@sss.pgh.pa.us wrote: As I commented to Robert, the page-at-a-time behavior of pageinspect is not an API detail we'd want to copy for this.

Re: [HACKERS] preserving forensic information when we freeze

2014-01-02 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: Well, that's fair enough. I don't mind having two functions. Should the whole-table function also include invisible tuples? Certainly, that's exactly why I was proposing it. You can do a join if you want to suppress them.

Re: [HACKERS] preserving forensic information when we freeze

2014-01-02 Thread Andres Freund
On 2014-01-02 14:44:34 -0500, Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: On 2014-01-02 12:46:34 -0500, Tom Lane wrote: For real forensics work, you need to be able to see all tuples, which makes me think that something akin to pgstattuple is the right API; that is return

Re: [HACKERS] preserving forensic information when we freeze

2014-01-01 Thread Robert Haas
On Fri, Dec 27, 2013 at 9:24 PM, Stephen Frost sfr...@snowman.net wrote: I'm not sure what you mean by doesn't work, because it clearly does work. I've already posted a patch. You may find it ugly, but that's not the same as not working. I meant *practically*, it doesn't work. By which, I

Re: [HACKERS] preserving forensic information when we freeze

2014-01-01 Thread Greg Stark
I fail to see why. What is so ugly about this: select x.* from pgbench_accounts a, pg_tuple_header(a.tableoid, a.ctid) x; Two points: 1) it's a bit weird to go to this effort to eliminate system columns by using a scheme that depends on having a system column -- ctid 2) refetching a row

Re: [HACKERS] preserving forensic information when we freeze

2013-12-27 Thread Andres Freund
On 2013-12-26 15:25:41 -0800, Robert Haas wrote: On Tue, Dec 24, 2013 at 6:22 AM, Andres Freund and...@2ndquadrant.com wrote: If system columns don't have an overhead anymore, I fail to see the advantage that functions have over simply accessing parts of the row in the normal way parts of

Re: [HACKERS] preserving forensic information when we freeze

2013-12-27 Thread Stephen Frost
* Andres Freund (and...@2ndquadrant.com) wrote: On 2013-12-26 15:25:41 -0800, Robert Haas wrote: I dunno, I just have an uneasy feeling about it. I guess if everyone's happy to add pg_infomask and pg_infomask2 as new system columns, we could go that route. For my part, I think that

Re: [HACKERS] preserving forensic information when we freeze

2013-12-27 Thread Robert Haas
On Fri, Dec 27, 2013 at 6:15 AM, Stephen Frost sfr...@snowman.net wrote: * Andres Freund (and...@2ndquadrant.com) wrote: On 2013-12-26 15:25:41 -0800, Robert Haas wrote: I dunno, I just have an uneasy feeling about it. I guess if everyone's happy to add pg_infomask and pg_infomask2 as new

Re: [HACKERS] preserving forensic information when we freeze

2013-12-27 Thread Stephen Frost
Robert, * Robert Haas (robertmh...@gmail.com) wrote: On Fri, Dec 27, 2013 at 6:15 AM, Stephen Frost sfr...@snowman.net wrote: For my 2c, I tend to agree w/ Andres on this one. I like the *idea* of having a function instead, but, practically, it doesn't really work. Perhaps if the

Re: [HACKERS] preserving forensic information when we freeze

2013-12-26 Thread Robert Haas
On Tue, Dec 24, 2013 at 6:22 AM, Andres Freund and...@2ndquadrant.com wrote: On 2013-12-23 14:15:29 -0500, Robert Haas wrote: On Mon, Dec 23, 2013 at 1:57 PM, Andres Freund and...@2ndquadrant.com wrote: Well, all of the fundamental changes (combocids, the initial multixact introduction)

Re: [HACKERS] preserving forensic information when we freeze

2013-12-24 Thread Andres Freund
On 2013-12-23 14:15:29 -0500, Robert Haas wrote: On Mon, Dec 23, 2013 at 1:57 PM, Andres Freund and...@2ndquadrant.com wrote: Well, all of the fundamental changes (combocids, the initial multixact introduction) have been quite some time ago. I think backward compat has a much higher value

Re: [HACKERS] preserving forensic information when we freeze

2013-12-23 Thread Robert Haas
On Fri, Dec 20, 2013 at 9:56 PM, Robert Haas robertmh...@gmail.com wrote: On Fri, Dec 20, 2013 at 7:51 AM, Andres Freund and...@2ndquadrant.com wrote: I wondered that, too, but it's not well-defined for all tuples. What happens if you pass in constructed tuple rather than an on-disk tuple?

Re: [HACKERS] preserving forensic information when we freeze

2013-12-23 Thread Andres Freund
On 2013-12-20 13:22:07 +0100, Andres Freund wrote: On 2013-12-20 07:12:01 -0500, Robert Haas wrote: I think the root of the problem is that nobody's very eager to add more hidden system catalog columns because each one bloats pg_attribute significantly. I think that part is actually

Re: [HACKERS] preserving forensic information when we freeze

2013-12-23 Thread Andres Freund
On 2013-12-20 21:56:42 -0500, Robert Haas wrote: On Fri, Dec 20, 2013 at 7:51 AM, Andres Freund and...@2ndquadrant.com wrote: I wondered that, too, but it's not well-defined for all tuples. What happens if you pass in constructed tuple rather than an on-disk tuple? Those should be

Re: [HACKERS] preserving forensic information when we freeze

2013-12-23 Thread Robert Haas
On Mon, Dec 23, 2013 at 7:45 AM, Andres Freund and...@2ndquadrant.com wrote: On 2013-12-20 13:22:07 +0100, Andres Freund wrote: On 2013-12-20 07:12:01 -0500, Robert Haas wrote: I think the root of the problem is that nobody's very eager to add more hidden system catalog columns because each

Re: [HACKERS] preserving forensic information when we freeze

2013-12-23 Thread Andres Freund
On 2013-12-23 10:26:49 -0500, Robert Haas wrote: On Mon, Dec 23, 2013 at 7:45 AM, Andres Freund and...@2ndquadrant.com wrote: I've spent some time yesterday hacking up a prototype for this. The amount of effort seems reasonable, although the attached patch certainly doesn't do all the

Re: [HACKERS] preserving forensic information when we freeze

2013-12-23 Thread Robert Haas
On Mon, Dec 23, 2013 at 10:42 AM, Andres Freund and...@2ndquadrant.com wrote: But TBH, I'm kind of enamored of the function-based approach at the moment. It just seems a lot more flexible. Adding a function is nearly free and we can add as many as we need doing whatever we want, and they can

Re: [HACKERS] preserving forensic information when we freeze

2013-12-23 Thread Kevin Grittner
Robert Haas robertmh...@gmail.com wrote: Meh.  Who are all of these people who are fetching xmin, xmax, cmin, and cmax in complex queries, and why are they doing that?  If those columns are just for forensic purposes, then whatever performance disadvantages the function-based approach has

Re: [HACKERS] preserving forensic information when we freeze

2013-12-23 Thread Andres Freund
On 2013-12-23 10:56:07 -0500, Robert Haas wrote: Well, it really depends on the usecase. Reading SELECT header.xmin, header.xmax FROM sometable tbl, pg_tuple_header(tbl.tableoid, tbl.ctid) header; is surely harder to understand than SELECT tbl.xmin, tbl.xmax FROM sometable tbl;

Re: [HACKERS] preserving forensic information when we freeze

2013-12-23 Thread Robert Haas
On Mon, Dec 23, 2013 at 1:57 PM, Andres Freund and...@2ndquadrant.com wrote: Well, all of the fundamental changes (combocids, the initial multixact introduction) have been quite some time ago. I think backward compat has a much higher value these days (I also don't see much point in looking at

Re: [HACKERS] preserving forensic information when we freeze

2013-12-22 Thread Robert Haas
On Fri, Dec 20, 2013 at 8:01 AM, Andres Freund and...@2ndquadrant.com wrote: On 2013-12-20 07:58:46 -0500, Robert Haas wrote: I think the immediate problem is to decide whether this patch ought to make the xmin column display the result of GetXmin() or GetRawXmin(). Thoughts on that? I

Re: [HACKERS] preserving forensic information when we freeze

2013-12-20 Thread Robert Haas
On Thu, Dec 19, 2013 at 9:22 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Jim Nasby escribió: One thing users will lose in this patch is the ability to reliably see if a tuple is frozen via SQL. Today you can do that just by selecting xmin from the table. Obviously people don't

Re: [HACKERS] preserving forensic information when we freeze

2013-12-20 Thread Andres Freund
On 2013-12-20 07:12:01 -0500, Robert Haas wrote: I think the root of the problem is that nobody's very eager to add more hidden system catalog columns because each one bloats pg_attribute significantly. I think that part is actually solveable - there's no real need for them to be real columns,

Re: [HACKERS] preserving forensic information when we freeze

2013-12-20 Thread Robert Haas
On Fri, Dec 20, 2013 at 7:22 AM, Andres Freund and...@2ndquadrant.com wrote: Maybe what we should do is add a function something like pg_tuple_header(tableoid, ctid) that returns a record, maybe something like (rawxmin xid, rawxmax xid, rawcid cid, infomask int, infomask2 int, hoff int). Or

Re: [HACKERS] preserving forensic information when we freeze

2013-12-20 Thread Andres Freund
On 2013-12-20 07:47:17 -0500, Robert Haas wrote: On Fri, Dec 20, 2013 at 7:22 AM, Andres Freund and...@2ndquadrant.com wrote: Maybe what we should do is add a function something like pg_tuple_header(tableoid, ctid) that returns a record, maybe something like (rawxmin xid, rawxmax xid,

Re: [HACKERS] preserving forensic information when we freeze

2013-12-20 Thread Robert Haas
On Fri, Dec 20, 2013 at 7:51 AM, Andres Freund and...@2ndquadrant.com wrote: On 2013-12-20 07:47:17 -0500, Robert Haas wrote: On Fri, Dec 20, 2013 at 7:22 AM, Andres Freund and...@2ndquadrant.com wrote: Maybe what we should do is add a function something like pg_tuple_header(tableoid,

Re: [HACKERS] preserving forensic information when we freeze

2013-12-20 Thread Andres Freund
On 2013-12-20 07:58:46 -0500, Robert Haas wrote: I think the immediate problem is to decide whether this patch ought to make the xmin column display the result of GetXmin() or GetRawXmin(). Thoughts on that? I slightly favor GetRawXmin(). Greetings, Andres Freund -- Andres Freund

Re: [HACKERS] preserving forensic information when we freeze

2013-12-20 Thread Alvaro Herrera
Robert Haas escribió: Maybe what we should do is add a function something like pg_tuple_header(tableoid, ctid) that returns a record, maybe something like (rawxmin xid, rawxmax xid, rawcid cid, infomask int, infomask2 int, hoff int). Or perhaps some slightly more cooked version of that

Re: [HACKERS] preserving forensic information when we freeze

2013-12-20 Thread Alvaro Herrera
Robert Haas escribió: On Fri, Dec 20, 2013 at 7:22 AM, Andres Freund and...@2ndquadrant.com wrote: Maybe what we should do is add a function something like pg_tuple_header(tableoid, ctid) that returns a record, maybe something like (rawxmin xid, rawxmax xid, rawcid cid, infomask int,

Re: [HACKERS] preserving forensic information when we freeze

2013-12-20 Thread Robert Haas
On Fri, Dec 20, 2013 at 1:41 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Robert Haas escribió: On Fri, Dec 20, 2013 at 7:22 AM, Andres Freund and...@2ndquadrant.com wrote: Maybe what we should do is add a function something like pg_tuple_header(tableoid, ctid) that returns a record,

Re: [HACKERS] preserving forensic information when we freeze

2013-12-20 Thread Alvaro Herrera
Robert Haas escribió: On Fri, Dec 20, 2013 at 1:41 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: I assume without checking that passing reloid/ctid would allow this to work for tuples in a RETURNING clause; and if we ever have an OLD reference for the RETURNING clause of an UPDATE,

Re: [HACKERS] preserving forensic information when we freeze

2013-12-20 Thread Robert Haas
On Fri, Dec 20, 2013 at 2:17 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Robert Haas escribió: On Fri, Dec 20, 2013 at 1:41 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: I assume without checking that passing reloid/ctid would allow this to work for tuples in a RETURNING

Re: [HACKERS] preserving forensic information when we freeze

2013-12-20 Thread Robert Haas
On Fri, Dec 20, 2013 at 7:51 AM, Andres Freund and...@2ndquadrant.com wrote: I wondered that, too, but it's not well-defined for all tuples. What happens if you pass in constructed tuple rather than an on-disk tuple? Those should be discernible I think, t_self/t_tableOid won't be set for

Re: [HACKERS] preserving forensic information when we freeze

2013-12-19 Thread Andres Freund
On 2013-12-18 21:42:25 -0500, Robert Haas wrote: On Wed, Dec 18, 2013 at 5:54 PM, Andres Freund and...@2ndquadrant.com wrote: if (frz-frzflags XLH_FREEZE_XVAC) + { HeapTupleHeaderSetXvac(tuple, FrozenTransactionId); + /* If we somehow haven't hinted

Re: [HACKERS] preserving forensic information when we freeze

2013-12-19 Thread Robert Haas
On Thu, Dec 19, 2013 at 5:44 AM, Andres Freund and...@2ndquadrant.com wrote: On 2013-12-18 21:42:25 -0500, Robert Haas wrote: On Wed, Dec 18, 2013 at 5:54 PM, Andres Freund and...@2ndquadrant.com wrote: if (frz-frzflags XLH_FREEZE_XVAC) + {

Re: [HACKERS] preserving forensic information when we freeze

2013-12-19 Thread Andres Freund
On 2013-12-19 07:40:40 -0500, Robert Haas wrote: On Thu, Dec 19, 2013 at 5:44 AM, Andres Freund and...@2ndquadrant.com wrote: On 2013-12-18 21:42:25 -0500, Robert Haas wrote: I dislike transporting the infomask in the wal record and then changing it away from that again afterwards.

Re: [HACKERS] preserving forensic information when we freeze

2013-12-19 Thread Robert Haas
On Thu, Dec 19, 2013 at 9:19 AM, Andres Freund and...@2ndquadrant.com wrote: On 2013-12-19 07:40:40 -0500, Robert Haas wrote: On Thu, Dec 19, 2013 at 5:44 AM, Andres Freund and...@2ndquadrant.com wrote: On 2013-12-18 21:42:25 -0500, Robert Haas wrote: I dislike transporting the infomask

Re: [HACKERS] preserving forensic information when we freeze

2013-12-19 Thread Robert Haas
On Thu, Dec 19, 2013 at 9:37 AM, Robert Haas robertmh...@gmail.com wrote: On Thu, Dec 19, 2013 at 9:19 AM, Andres Freund and...@2ndquadrant.com wrote: On 2013-12-19 07:40:40 -0500, Robert Haas wrote: On Thu, Dec 19, 2013 at 5:44 AM, Andres Freund and...@2ndquadrant.com wrote: On 2013-12-18

Re: [HACKERS] preserving forensic information when we freeze

2013-12-19 Thread Alvaro Herrera
Robert Haas escribió: I think it may have been a mistake to divide responsibility between the prepare and execute functions the way we did in this case, because it doesn't appear to be a clean separation of concerns. But it's not this patch's job to kibitz that decision, so this version just

Re: [HACKERS] preserving forensic information when we freeze

2013-12-19 Thread Robert Haas
On Thu, Dec 19, 2013 at 3:36 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Robert Haas escribió: I think it may have been a mistake to divide responsibility between the prepare and execute functions the way we did in this case, because it doesn't appear to be a clean separation of

Re: [HACKERS] preserving forensic information when we freeze

2013-12-19 Thread Jim Nasby
One thing users will lose in this patch is the ability to reliably see if a tuple is frozen via SQL. Today you can do that just by selecting xmin from the table. Obviously people don't generally need to do that... but it's one of those things that when you do need it it's incredibly handy to

Re: [HACKERS] preserving forensic information when we freeze

2013-12-19 Thread Alvaro Herrera
Jim Nasby escribió: One thing users will lose in this patch is the ability to reliably see if a tuple is frozen via SQL. Today you can do that just by selecting xmin from the table. Obviously people don't generally need to do that... but it's one of those things that when you do need it

Re: [HACKERS] preserving forensic information when we freeze

2013-12-18 Thread Andres Freund
Hi, On 2013-12-17 16:00:14 -0500, Robert Haas wrote: @@ -5874,19 +5858,27 @@ heap_prepare_freeze_tuple(HeapTupleHeader tuple, TransactionId cutoff_xid, void heap_execute_freeze_tuple(HeapTupleHeader tuple, xl_heap_freeze_tuple *frz) { + tuple-t_infomask = frz-t_infomask; +

Re: [HACKERS] preserving forensic information when we freeze

2013-12-18 Thread Robert Haas
On Wed, Dec 18, 2013 at 5:54 PM, Andres Freund and...@2ndquadrant.com wrote: if (frz-frzflags XLH_FREEZE_XVAC) + { HeapTupleHeaderSetXvac(tuple, FrozenTransactionId); + /* If we somehow haven't hinted the tuple previously, do it now. */ +

Re: [HACKERS] preserving forensic information when we freeze

2013-12-17 Thread Robert Haas
On Wed, Dec 11, 2013 at 5:25 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Andres Freund escribió: What's your plan to commit this? I'd prefer to wait till Alvaro's freezing changes get in, so his patch will look the same in HEAD and 9.3. But I think he plans to commit soon. Yes, I do.

Re: [HACKERS] preserving forensic information when we freeze

2013-12-11 Thread Robert Haas
On Thu, Nov 21, 2013 at 4:51 PM, Andres Freund and...@2ndquadrant.com wrote: On 2013-11-21 15:59:35 -0500, Robert Haas wrote: * Should HeapTupleHeaderXminFrozen also check for FrozenTransactionId? It seems quite possible that people think they've delt with frozen xmin entirely after

Re: [HACKERS] preserving forensic information when we freeze

2013-12-11 Thread Heikki Linnakangas
On 12/11/2013 09:17 PM, Robert Haas wrote: On Thu, Nov 21, 2013 at 4:51 PM, Andres Freund and...@2ndquadrant.com wrote: On 2013-11-21 15:59:35 -0500, Robert Haas wrote: Separate patch, but yeah, something like that. If we have to mark the page all-visible, we might as well freeze it while

Re: [HACKERS] preserving forensic information when we freeze

2013-12-11 Thread Andres Freund
On 2013-12-11 14:17:27 -0500, Robert Haas wrote: But in which cases would that actually be slower? There'll be no additional code executed if the hint bits for frozen are set, and in case not it will usually safe us an external function call to TransactionIdPrecedes(). Dunno. It's at

Re: [HACKERS] preserving forensic information when we freeze

2013-12-11 Thread Alvaro Herrera
Andres Freund escribió: What's your plan to commit this? I'd prefer to wait till Alvaro's freezing changes get in, so his patch will look the same in HEAD and 9.3. But I think he plans to commit soon. Yes, I do. I'm waiting on feedback on the patch I posted this afternoon, so if there's

Re: [HACKERS] preserving forensic information when we freeze

2013-11-24 Thread Simon Riggs
On 21 November 2013 16:51, Andres Freund and...@2ndquadrant.com wrote: Definitely separate yes. And I agree, it's partially moot if Heikki's patch gets in, but I am not sure it will make it into 9.4. There seems to be quite some work left. I'd prefer to do all 3 patches in one release. Don't

Re: [HACKERS] preserving forensic information when we freeze

2013-11-21 Thread Andres Freund
On 2013-11-21 15:59:35 -0500, Robert Haas wrote: * Should HeapTupleHeaderXminFrozen also check for FrozenTransactionId? It seems quite possible that people think they've delt with frozen xmin entirely after checking, but they still might get FrozenTransactionId back in a pg_upgraded

Re: [HACKERS] preserving forensic information when we freeze

2013-11-21 Thread Andres Freund
Hi, As promised I'm currently updating the patch. Some questions arose during that: * Should HeapTupleHeaderXminFrozen also check for FrozenTransactionId? It seems quite possible that people think they've delt with frozen xmin entirely after checking, but they still might get

Re: [HACKERS] preserving forensic information when we freeze

2013-11-21 Thread Robert Haas
On Thu, Nov 21, 2013 at 9:47 AM, Andres Freund and...@2ndquadrant.com wrote: As promised I'm currently updating the patch. Some questions arose during that: * Should HeapTupleHeaderXminFrozen also check for FrozenTransactionId? It seems quite possible that people think they've delt with

Re: [HACKERS] preserving forensic information when we freeze

2013-07-08 Thread Josh Berkus
On 07/03/2013 11:59 AM, Robert Haas wrote: Yeah. I think the system columns that we have today are pretty much crap. I wonder if we couldn't throw them out and replace them with some kind of functions that you can pass a row to. That would allow us to expose a lot more detail without adding

Re: [HACKERS] preserving forensic information when we freeze

2013-07-08 Thread Robert Haas
On Jul 8, 2013, at 1:34 PM, Josh Berkus j...@agliodbs.com wrote: On 07/03/2013 11:59 AM, Robert Haas wrote: Yeah. I think the system columns that we have today are pretty much crap. I wonder if we couldn't throw them out and replace them with some kind of functions that you can pass a row

Re: [HACKERS] preserving forensic information when we freeze

2013-07-03 Thread Andres Freund
On 2013-07-02 16:29:56 -0400, Robert Haas wrote: On Mon, Jun 24, 2013 at 8:12 AM, Andres Freund and...@2ndquadrant.com wrote: Agreed. And it also improves the status quo when debugging. I wish this would have been the representation since the beginning. Some remarks: * I don't really

Re: [HACKERS] preserving forensic information when we freeze

2013-07-03 Thread Robert Haas
On Wed, Jul 3, 2013 at 1:07 PM, Andres Freund and...@2ndquadrant.com wrote: Well, nothing would prevent using the HeapTupleHeaderGetRawXmin() in those places. Exactly the number of callsites is what makes me think that somebody will get this wrong in the future. Well, I guess I could go rework

Re: [HACKERS] preserving forensic information when we freeze

2013-07-03 Thread Marko Kreen
On Wed, Jul 3, 2013 at 8:07 PM, Andres Freund and...@2ndquadrant.com wrote: On 2013-07-02 16:29:56 -0400, Robert Haas wrote: There's no possibility of getting confused here; if X is still around at all, it's xmax is of the same generation as Y's xmin. Otherwise, we've had an undetected XID

Re: [HACKERS] preserving forensic information when we freeze

2013-07-02 Thread Robert Haas
On Mon, Jun 24, 2013 at 8:12 AM, Andres Freund and...@2ndquadrant.com wrote: Agreed. And it also improves the status quo when debugging. I wish this would have been the representation since the beginning. Some remarks: * I don't really like that HeapTupleHeaderXminFrozen() now is frequently

Re: [HACKERS] preserving forensic information when we freeze

2013-06-24 Thread Andres Freund
On 2013-05-28 09:21:27 -0400, Robert Haas wrote: As a general statement, I view this work as something that is likely needed no matter which one of the remove freezing approaches that have been proposed we choose to adopt. It does not fix anything in and of itself, but it (hopefully) removes

Re: [HACKERS] preserving forensic information when we freeze

2013-05-29 Thread Robert Haas
On Tue, May 28, 2013 at 10:08 PM, Andres Freund and...@2ndquadrant.com wrote: On 2013-05-28 21:26:49 -0400, Robert Haas wrote: On Tue, May 28, 2013 at 8:00 PM, Andres Freund and...@2ndquadrant.com wrote: I only suggested MOVED_IN/OFF because I didn't remember

Re: [HACKERS] preserving forensic information when we freeze

2013-05-29 Thread Peter Eisentraut
On 5/28/13 8:00 PM, Andres Freund wrote: - Various procedural languages use the combination of TID and XMIN to determine whether a function needs to be recompiled. Hm. As previously said, I am less than convinced of those adhoc mechanisms and I think this should get properly integrated into

Re: [HACKERS] preserving forensic information when we freeze

2013-05-29 Thread Alvaro Herrera
Robert Haas escribió: On Tue, May 28, 2013 at 10:08 PM, Andres Freund and...@2ndquadrant.com wrote: On 2013-05-28 21:26:49 -0400, Robert Haas wrote: I am all for adding a comment why this is safe though. We thought about it for some while before we were convinced... I'm fine with

[HACKERS] preserving forensic information when we freeze

2013-05-28 Thread Robert Haas
Various people, including at least Heikki, Andres, and myself, have proposed various schemes for avoiding freezing that amount to doing it sooner, when we're already writing WAL anyway, or at least when the buffer is already dirty anyway, or at least while the buffer is already in shared_buffers

Re: [HACKERS] preserving forensic information when we freeze

2013-05-28 Thread Josh Berkus
On 05/28/2013 06:21 AM, Robert Haas wrote: As a general statement, I view this work as something that is likely needed no matter which one of the remove freezing approaches that have been proposed we choose to adopt. It does not fix anything in and of itself, but it (hopefully) removes an

Re: [HACKERS] preserving forensic information when we freeze

2013-05-28 Thread Andres Freund
On 2013-05-28 09:39:13 -0700, Josh Berkus wrote: On 05/28/2013 06:21 AM, Robert Haas wrote: As a general statement, I view this work as something that is likely needed no matter which one of the remove freezing approaches that have been proposed we choose to adopt. It does not fix anything

Re: [HACKERS] preserving forensic information when we freeze

2013-05-28 Thread Robert Haas
On Tue, May 28, 2013 at 7:27 PM, Andres Freund and...@2ndquadrant.com wrote: On 2013-05-28 09:39:13 -0700, Josh Berkus wrote: On 05/28/2013 06:21 AM, Robert Haas wrote: As a general statement, I view this work as something that is likely needed no matter which one of the remove freezing

Re: [HACKERS] preserving forensic information when we freeze

2013-05-28 Thread Andres Freund
On 2013-05-28 09:21:27 -0400, Robert Haas wrote: I have attempted to implement this. Trouble is, we're out of infomask bits. Using an infomask2 bit might work but we don't have many of them left either, so it's worth casting about a bit for a better solution. Andres proposed using

Re: [HACKERS] preserving forensic information when we freeze

2013-05-28 Thread Robert Haas
On Tue, May 28, 2013 at 8:00 PM, Andres Freund and...@2ndquadrant.com wrote: I only suggested MOVED_IN/OFF because I didn't remember HEAP_XMIN_COMMITTED|HEAP_XMIN_INVALID was still free ;). So, unless that combination could have been produced in the past in a way I couldn't find so far, I am

Re: [HACKERS] preserving forensic information when we freeze

2013-05-28 Thread Andres Freund
On 2013-05-28 21:26:49 -0400, Robert Haas wrote: On Tue, May 28, 2013 at 8:00 PM, Andres Freund and...@2ndquadrant.com wrote: I only suggested MOVED_IN/OFF because I didn't remember HEAP_XMIN_COMMITTED|HEAP_XMIN_INVALID was still free ;). So, unless that combination could have been produced