Re: [HACKERS] tracking commit timestamps

2015-03-09 Thread Alvaro Herrera
Petr Jelinek wrote: On Thu, Dec 4, 2014 at 12:08 PM, Fujii Masao masao.fu...@gmail.com wrote: I got the following assertion failure when I executed pg_xact_commit_timestamp() in the standby server. =# select pg_xact_commit_timestamp('1000'::xid); TRAP:

Re: [HACKERS] tracking commit timestamps

2015-01-22 Thread Petr Jelinek
On 05/01/15 17:50, Petr Jelinek wrote: On 05/01/15 16:17, Petr Jelinek wrote: On 05/01/15 07:28, Fujii Masao wrote: On Thu, Dec 4, 2014 at 12:08 PM, Fujii Masao masao.fu...@gmail.com wrote: On Wed, Dec 3, 2014 at 11:54 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Pushed with some extra

Re: [HACKERS] tracking commit timestamps

2015-01-06 Thread Robert Haas
On Tue, Jan 6, 2015 at 2:58 AM, Michael Paquier michael.paqu...@gmail.com wrote: So, we would need additional information other than the node ID *and* the timestamp to ensure proper transaction commit ordering on Windows. That's not cool and makes this feature very limited on this platform.

Re: [HACKERS] tracking commit timestamps

2015-01-05 Thread Michael Paquier
On Fri, Dec 19, 2014 at 3:53 PM, Noah Misch n...@leadboat.com wrote: localhost template1=# select clock_timestamp(), pg_sleep(.1 * (n % 2)) from generate_series(0,7) t(n); clock_timestamp| pg_sleep ---+-- 2014-12-18 08:34:34.522126+00 |

Re: [HACKERS] tracking commit timestamps

2015-01-05 Thread Petr Jelinek
On 05/01/15 16:17, Petr Jelinek wrote: On 05/01/15 07:28, Fujii Masao wrote: On Thu, Dec 4, 2014 at 12:08 PM, Fujii Masao masao.fu...@gmail.com wrote: On Wed, Dec 3, 2014 at 11:54 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Pushed with some extra cosmetic tweaks. I got the following

Re: [HACKERS] tracking commit timestamps

2015-01-05 Thread Petr Jelinek
On 05/01/15 07:28, Fujii Masao wrote: On Thu, Dec 4, 2014 at 12:08 PM, Fujii Masao masao.fu...@gmail.com wrote: On Wed, Dec 3, 2014 at 11:54 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Pushed with some extra cosmetic tweaks. I got the following assertion failure when I executed

Re: [HACKERS] tracking commit timestamps

2015-01-04 Thread Craig Ringer
On 12/19/2014 02:53 PM, Noah Misch wrote: The test assumed that no two transactions of a given backend will get the same timestamp value from now(). That holds so long as ticks of the system time are small enough. Not so on at least some Windows configurations. Most Windows systems with

Re: [HACKERS] tracking commit timestamps

2015-01-04 Thread Fujii Masao
On Thu, Dec 4, 2014 at 12:08 PM, Fujii Masao masao.fu...@gmail.com wrote: On Wed, Dec 3, 2014 at 11:54 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Pushed with some extra cosmetic tweaks. I got the following assertion failure when I executed pg_xact_commit_timestamp() in the standby

Re: [HACKERS] tracking commit timestamps

2014-12-18 Thread Noah Misch
On Tue, Dec 16, 2014 at 01:05:31AM -0300, Alvaro Herrera wrote: Noah Misch wrote: On Mon, Dec 15, 2014 at 12:12:10AM -0800, Michael Paquier wrote: FWIW, I just tried that with MinGW-32 and I can see the error on Win7. I also checked that changing now() to = now() fixed the problem,

Re: [HACKERS] tracking commit timestamps

2014-12-15 Thread Michael Paquier
On Wed, Dec 10, 2014 at 6:50 PM, Noah Misch n...@leadboat.com wrote: On Mon, Dec 08, 2014 at 02:23:39AM +0100, Petr Jelinek wrote: On 08/12/14 00:56, Noah Misch wrote: The commit_ts test suite gives me the attached diff on a 32-bit MinGW build running on 64-bit Windows Server 2003. I have not

Re: [HACKERS] tracking commit timestamps

2014-12-15 Thread Petr Jelinek
On 15/12/14 09:12, Michael Paquier wrote: On Wed, Dec 10, 2014 at 6:50 PM, Noah Misch n...@leadboat.com wrote: On Mon, Dec 08, 2014 at 02:23:39AM +0100, Petr Jelinek wrote: On 08/12/14 00:56, Noah Misch wrote: The commit_ts test suite gives me the attached diff on a 32-bit MinGW build running

Re: [HACKERS] tracking commit timestamps

2014-12-15 Thread Noah Misch
On Mon, Dec 15, 2014 at 12:12:10AM -0800, Michael Paquier wrote: On Wed, Dec 10, 2014 at 6:50 PM, Noah Misch n...@leadboat.com wrote: On Mon, Dec 08, 2014 at 02:23:39AM +0100, Petr Jelinek wrote: On 08/12/14 00:56, Noah Misch wrote: The commit_ts test suite gives me the attached diff on a

Re: [HACKERS] tracking commit timestamps

2014-12-15 Thread Alvaro Herrera
Noah Misch wrote: On Mon, Dec 15, 2014 at 12:12:10AM -0800, Michael Paquier wrote: FWIW, I just tried that with MinGW-32 and I can see the error on Win7. I also checked that changing now() to = now() fixed the problem, so your assumption was right, Petr. Committed, after fixing the

Re: [HACKERS] tracking commit timestamps

2014-12-10 Thread Noah Misch
On Mon, Dec 08, 2014 at 02:23:39AM +0100, Petr Jelinek wrote: On 08/12/14 00:56, Noah Misch wrote: The commit_ts test suite gives me the attached diff on a 32-bit MinGW build running on 64-bit Windows Server 2003. I have not checked other Windows configurations; the suite does pass on

Re: [HACKERS] tracking commit timestamps

2014-12-07 Thread Noah Misch
On Wed, Dec 03, 2014 at 11:54:38AM -0300, Alvaro Herrera wrote: Pushed with some extra cosmetic tweaks. The commit_ts test suite gives me the attached diff on a 32-bit MinGW build running on 64-bit Windows Server 2003. I have not checked other Windows configurations; the suite does pass on

Re: [HACKERS] tracking commit timestamps

2014-12-07 Thread Petr Jelinek
On 08/12/14 00:56, Noah Misch wrote: On Wed, Dec 03, 2014 at 11:54:38AM -0300, Alvaro Herrera wrote: Pushed with some extra cosmetic tweaks. The commit_ts test suite gives me the attached diff on a 32-bit MinGW build running on 64-bit Windows Server 2003. I have not checked other Windows

Re: [HACKERS] tracking commit timestamps

2014-12-03 Thread Alvaro Herrera
Pushed with some extra cosmetic tweaks. -- Álvaro Herrerahttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription:

Re: [HACKERS] tracking commit timestamps

2014-12-03 Thread Petr Jelinek
On 03/12/14 15:54, Alvaro Herrera wrote: Pushed with some extra cosmetic tweaks. Cool, thanks! -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To

Re: [HACKERS] tracking commit timestamps

2014-12-03 Thread Robert Haas
On Mon, Dec 1, 2014 at 5:34 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: I made two more changes: 1. introduce newestCommitTs. Original code was using lastCommitXact to check that no future transaction is asked for, but this doesn't really work if a long-running transaction is

Re: [HACKERS] tracking commit timestamps

2014-12-03 Thread Alvaro Herrera
Robert Haas wrote: On Mon, Dec 1, 2014 at 5:34 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: I made two more changes: 1. introduce newestCommitTs. Original code was using lastCommitXact to check that no future transaction is asked for, but this doesn't really work if a

Re: [HACKERS] tracking commit timestamps

2014-12-03 Thread Alvaro Herrera
Alvaro Herrera wrote: Robert Haas wrote: I'm kind of disappointed that, in spite of previous review comments, this got committed with extensive use of the CommitTs naming. I think that's confusing, but it's also something that will be awkward if we want to add other data, such as the

Re: [HACKERS] tracking commit timestamps

2014-12-03 Thread Robert Haas
On Wed, Dec 3, 2014 at 2:36 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Alvaro Herrera wrote: Robert Haas wrote: I'm kind of disappointed that, in spite of previous review comments, this got committed with extensive use of the CommitTs naming. I think that's confusing, but it's

Re: [HACKERS] tracking commit timestamps

2014-12-03 Thread Fujii Masao
On Wed, Dec 3, 2014 at 11:54 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Pushed with some extra cosmetic tweaks. I got the following assertion failure when I executed pg_xact_commit_timestamp() in the standby server. =# select pg_xact_commit_timestamp('1000'::xid); TRAP:

Re: [HACKERS] tracking commit timestamps

2014-12-03 Thread Simon Riggs
On 4 December 2014 at 03:08, Fujii Masao masao.fu...@gmail.com wrote: #1. set up and start the master and standby servers with track_commit_timestamp disabled #2. enable track_commit_timestamp in the master and restart the master #3. run some write transactions #4. enable

Re: [HACKERS] tracking commit timestamps

2014-12-03 Thread Fujii Masao
On Thu, Dec 4, 2014 at 12:58 PM, Simon Riggs si...@2ndquadrant.com wrote: On 4 December 2014 at 03:08, Fujii Masao masao.fu...@gmail.com wrote: #1. set up and start the master and standby servers with track_commit_timestamp disabled #2. enable track_commit_timestamp in the master and restart

Re: [HACKERS] tracking commit timestamps

2014-12-01 Thread Alvaro Herrera
Petr Jelinek wrote: On 25/11/14 16:30, Alvaro Herrera wrote: Petr Jelinek wrote: On 25/11/14 16:23, Alvaro Herrera wrote: Robert Haas wrote: Maybe 0 should get translated to a NULL return, instead of a bogus timestamp. That's one idea --- surely no transaction is going to commit at

Re: [HACKERS] tracking commit timestamps

2014-11-25 Thread Fujii Masao
On Tue, Nov 25, 2014 at 7:58 AM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: And here is v10 which fixes conflicts with Heikki's WAL API changes (no changes otherwise). After some slight additional changes, here's v11, which I intend to commit early tomorrow. The main change is moving the

Re: [HACKERS] tracking commit timestamps

2014-11-25 Thread Alvaro Herrera
Fujii Masao wrote: On Tue, Nov 25, 2014 at 7:58 AM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: And here is v10 which fixes conflicts with Heikki's WAL API changes (no changes otherwise). After some slight additional changes, here's v11, which I intend to commit early tomorrow. The

Re: [HACKERS] tracking commit timestamps

2014-11-25 Thread Fujii Masao
On Tue, Nov 25, 2014 at 11:19 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Fujii Masao wrote: On Tue, Nov 25, 2014 at 7:58 AM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: And here is v10 which fixes conflicts with Heikki's WAL API changes (no changes otherwise). After some

Re: [HACKERS] tracking commit timestamps

2014-11-25 Thread Robert Haas
On Tue, Nov 25, 2014 at 9:19 AM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Fujii Masao wrote: On Tue, Nov 25, 2014 at 7:58 AM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: And here is v10 which fixes conflicts with Heikki's WAL API changes (no changes otherwise). After some

Re: [HACKERS] tracking commit timestamps

2014-11-25 Thread Alvaro Herrera
Robert Haas wrote: On Tue, Nov 25, 2014 at 9:19 AM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Fujii Masao wrote: On Tue, Nov 25, 2014 at 7:58 AM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: And here is v10 which fixes conflicts with Heikki's WAL API changes (no changes

Re: [HACKERS] tracking commit timestamps

2014-11-25 Thread Petr Jelinek
On 25/11/14 16:23, Alvaro Herrera wrote: Robert Haas wrote: On Tue, Nov 25, 2014 at 9:19 AM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Fujii Masao wrote: On Tue, Nov 25, 2014 at 7:58 AM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: And here is v10 which fixes conflicts with Heikki's

Re: [HACKERS] tracking commit timestamps

2014-11-25 Thread Alvaro Herrera
Petr Jelinek wrote: On 25/11/14 16:23, Alvaro Herrera wrote: Robert Haas wrote: Maybe 0 should get translated to a NULL return, instead of a bogus timestamp. That's one idea --- surely no transaction is going to commit at 00:00:00 on 2000-01-01 anymore. Yet this is somewhat

Re: [HACKERS] tracking commit timestamps

2014-11-25 Thread Petr Jelinek
On 25/11/14 16:30, Alvaro Herrera wrote: Petr Jelinek wrote: On 25/11/14 16:23, Alvaro Herrera wrote: Robert Haas wrote: Maybe 0 should get translated to a NULL return, instead of a bogus timestamp. That's one idea --- surely no transaction is going to commit at 00:00:00 on 2000-01-01

Re: [HACKERS] tracking commit timestamps

2014-11-25 Thread Simon Riggs
On 25 November 2014 at 13:35, Fujii Masao masao.fu...@gmail.com wrote: Can I check my understanding? Probably we cannot use this feature to calculate the actual replication lag by, for example, comparing the result of pg_last_committed_xact() in the master and that of

Re: [HACKERS] tracking commit timestamps

2014-11-25 Thread Petr Jelinek
On 25/11/14 17:16, Simon Riggs wrote: On 25 November 2014 at 13:35, Fujii Masao masao.fu...@gmail.com wrote: Can I check my understanding? Probably we cannot use this feature to calculate the actual replication lag by, for example, comparing the result of pg_last_committed_xact() in the master

Re: [HACKERS] tracking commit timestamps

2014-11-25 Thread Simon Riggs
On 25 November 2014 at 16:18, Petr Jelinek p...@2ndquadrant.com wrote: Won't the pg_last_committed_xact() on slave + pg_xact_commit_timestamp() on master with the xid returned by slave accomplish the same thing? Surely the pg_last_committed_xact() will return the same value on standby as it

Re: [HACKERS] tracking commit timestamps

2014-11-25 Thread Michael Paquier
On Wed, Nov 26, 2014 at 1:51 AM, Simon Riggs si...@2ndquadrant.com wrote: On 25 November 2014 at 16:18, Petr Jelinek p...@2ndquadrant.com wrote: Won't the pg_last_committed_xact() on slave + pg_xact_commit_timestamp() on master with the xid returned by slave accomplish the same thing? Surely

Re: [HACKERS] tracking commit timestamps

2014-11-24 Thread Alvaro Herrera
And here is v10 which fixes conflicts with Heikki's WAL API changes (no changes otherwise). After some slight additional changes, here's v11, which I intend to commit early tomorrow. The main change is moving the test module from contrib to src/test/modules. -- Álvaro Herrera

Re: [HACKERS] tracking commit timestamps

2014-11-20 Thread Petr Jelinek
On 19/11/14 17:30, Steve Singer wrote: On 11/19/2014 08:22 AM, Alvaro Herrera wrote: I think we're overblowing the pg_upgrade issue. Surely we don't need to preserve commit_ts data when upgrading across major versions; and pg_upgrade is perfectly prepared to remove old data when upgrading

Re: [HACKERS] tracking commit timestamps

2014-11-19 Thread Simon Riggs
On 19 November 2014 02:12, Petr Jelinek p...@2ndquadrant.com wrote: Maybe we need better explanation of the LSN use-case(s) to understand why it should be stored here and why the other solutions are significantly worse. We should apply the same standard that has been applied elsewhere. If

Re: [HACKERS] tracking commit timestamps

2014-11-19 Thread Petr Jelinek
On 19/11/14 12:20, Simon Riggs wrote: On 19 November 2014 02:12, Petr Jelinek p...@2ndquadrant.com wrote: Maybe we need better explanation of the LSN use-case(s) to understand why it should be stored here and why the other solutions are significantly worse. We should apply the same standard

Re: [HACKERS] tracking commit timestamps

2014-11-19 Thread Alvaro Herrera
Petr Jelinek wrote: This is good point, we are not too late in the cycle that LSN couldn't be added later if we find that it is indeed needed (and we don't have to care about pg_upgrade until beta). I think we're overblowing the pg_upgrade issue. Surely we don't need to preserve commit_ts

Re: [HACKERS] tracking commit timestamps

2014-11-19 Thread Robert Haas
On Wed, Nov 19, 2014 at 8:22 AM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Petr Jelinek wrote: This is good point, we are not too late in the cycle that LSN couldn't be added later if we find that it is indeed needed (and we don't have to care about pg_upgrade until beta). I think we're

Re: [HACKERS] tracking commit timestamps

2014-11-19 Thread Steve Singer
On 11/19/2014 08:22 AM, Alvaro Herrera wrote: I think we're overblowing the pg_upgrade issue. Surely we don't need to preserve commit_ts data when upgrading across major versions; and pg_upgrade is perfectly prepared to remove old data when upgrading (actually it just doesn't copy it; consider

Re: [HACKERS] tracking commit timestamps

2014-11-18 Thread Petr Jelinek
On 15/11/14 13:36, Simon Riggs wrote: On 15 November 2014 04:32, Steve Singer st...@ssinger.info wrote: The use cases I'm talking about aren't really replication related. Often I have come across systems that want to do something such as 'select * from orders where X the_last_row_I_saw order

Re: [HACKERS] tracking commit timestamps

2014-11-15 Thread Simon Riggs
On 15 November 2014 04:32, Steve Singer st...@ssinger.info wrote: The use cases I'm talking about aren't really replication related. Often I have come across systems that want to do something such as 'select * from orders where X the_last_row_I_saw order by X' and then do further processing

Re: [HACKERS] tracking commit timestamps

2014-11-14 Thread Robert Haas
On Thu, Nov 13, 2014 at 6:55 PM, Simon Riggs si...@2ndquadrant.com wrote: On 13 November 2014 21:24, Robert Haas robertmh...@gmail.com wrote: On Thu, Nov 13, 2014 at 8:18 AM, Simon Riggs si...@2ndquadrant.com wrote: Ordering transactions in LSN order is very precisly the remit of the existing

Re: [HACKERS] tracking commit timestamps

2014-11-14 Thread Simon Riggs
On 14 November 2014 17:12, Robert Haas robertmh...@gmail.com wrote: We are not talking about loading 20 new requirements on top of this patch; that would be intolerable. We're talking about adding one additional piece of information that has been requested multiple times over the years. The

Re: [HACKERS] tracking commit timestamps

2014-11-14 Thread Steve Singer
On 11/14/2014 08:21 PM, Simon Riggs wrote: The requested information is already available, as discussed. Logical decoding adds commit ordering for *exactly* the purpose of using it for replication, available to all solutions. This often requested feature has now been added and doesn't need to be

Re: [HACKERS] tracking commit timestamps

2014-11-13 Thread Petr Jelinek
On 13/11/14 07:04, Michael Paquier wrote: On Wed, Nov 12, 2014 at 10:06 PM, Petr Jelinek p...@2ndquadrant.com wrote: Brief list of changes: - the commit timestamp record now stores timestamp, lsn and nodeid Now that not only the commit timestamp is stored, calling that commit timestamp,

Re: [HACKERS] tracking commit timestamps

2014-11-13 Thread Simon Riggs
On 9 November 2014 16:57, Steve Singer st...@ssinger.info wrote: On 11/07/2014 07:07 PM, Petr Jelinek wrote: The list of what is useful might be long, but we can't have everything there as there are space constraints, and LSN is another 8 bytes and I still want to have some bytes for storing

Re: [HACKERS] tracking commit timestamps

2014-11-13 Thread Petr Jelinek
On 13/11/14 14:18, Simon Riggs wrote: So in summary... the components are * Commit LSN order is useful for applying changes - available by logical decoding * Commit timestamps and nodeid are useful for conflict resolution - available from this patch Both components have been designed in ways

Re: [HACKERS] tracking commit timestamps

2014-11-13 Thread Robert Haas
On Thu, Nov 13, 2014 at 8:18 AM, Simon Riggs si...@2ndquadrant.com wrote: Ordering transactions in LSN order is very precisly the remit of the existing logical decoding API. Any user that wishes to see a commits in sequence can do so using that API. BDR already does this, as do other users of

Re: [HACKERS] tracking commit timestamps

2014-11-13 Thread Simon Riggs
On 13 November 2014 21:24, Robert Haas robertmh...@gmail.com wrote: On Thu, Nov 13, 2014 at 8:18 AM, Simon Riggs si...@2ndquadrant.com wrote: Ordering transactions in LSN order is very precisly the remit of the existing logical decoding API. Any user that wishes to see a commits in sequence

Re: [HACKERS] tracking commit timestamps

2014-11-12 Thread Petr Jelinek
On 10/11/14 14:53, Robert Haas wrote: On Mon, Nov 10, 2014 at 8:39 AM, Petr Jelinek p...@2ndquadrant.com wrote: I did the calculation above wrong btw, it's actually 20 bytes not 24 bytes per record, I am inclined to just say we can live with that. If you do it as 20 bytes, you'll have to do

Re: [HACKERS] tracking commit timestamps

2014-11-12 Thread Jim Nasby
On 11/12/14, 7:06 AM, Petr Jelinek wrote: - if the xid passed to get interface is out of range -infinity timestamp is returned (I think it's bad idea to throw errors here as the valid range is not static and same ID can start throwing errors between calls theoretically) Wouldn't NULL be

Re: [HACKERS] tracking commit timestamps

2014-11-12 Thread Michael Paquier
On Thu, Nov 13, 2014 at 7:56 AM, Jim Nasby jim.na...@bluetreble.com wrote: On 11/12/14, 7:06 AM, Petr Jelinek wrote: - if the xid passed to get interface is out of range -infinity timestamp is returned (I think it's bad idea to throw errors here as the valid range is not static and same ID

Re: [HACKERS] tracking commit timestamps

2014-11-12 Thread Michael Paquier
On Wed, Nov 12, 2014 at 10:06 PM, Petr Jelinek p...@2ndquadrant.com wrote: Brief list of changes: - the commit timestamp record now stores timestamp, lsn and nodeid Now that not only the commit timestamp is stored, calling that commit timestamp, committs or commit_timestamp is strange, no? If

Re: [HACKERS] tracking commit timestamps

2014-11-11 Thread Simon Riggs
On 4 November 2014 08:23, Andres Freund and...@2ndquadrant.com wrote: 6) Shouldn't any value update of track_commit_timestamp be tracked in XLogReportParameters? That's thinking about making the commit timestamp available on standbys as well.. Yes, it should. Agree committs should be able

Re: [HACKERS] tracking commit timestamps

2014-11-11 Thread Andres Freund
On 2014-11-11 16:10:47 +, Simon Riggs wrote: On 4 November 2014 08:23, Andres Freund and...@2ndquadrant.com wrote: 6) Shouldn't any value update of track_commit_timestamp be tracked in XLogReportParameters? That's thinking about making the commit timestamp available on standbys as

Re: [HACKERS] tracking commit timestamps

2014-11-11 Thread Simon Riggs
On 9 November 2014 16:57, Steve Singer st...@ssinger.info wrote: On 11/07/2014 07:07 PM, Petr Jelinek wrote: The list of what is useful might be long, but we can't have everything there as there are space constraints, and LSN is another 8 bytes and I still want to have some bytes for storing

Re: [HACKERS] tracking commit timestamps

2014-11-11 Thread Simon Riggs
On 11 November 2014 16:19, Andres Freund and...@2ndquadrant.com wrote: On 2014-11-11 16:10:47 +, Simon Riggs wrote: On 4 November 2014 08:23, Andres Freund and...@2ndquadrant.com wrote: 6) Shouldn't any value update of track_commit_timestamp be tracked in XLogReportParameters? That's

Re: [HACKERS] tracking commit timestamps

2014-11-11 Thread Andres Freund
On 2014-11-11 17:09:54 +, Simon Riggs wrote: On 11 November 2014 16:19, Andres Freund and...@2ndquadrant.com wrote: On 2014-11-11 16:10:47 +, Simon Riggs wrote: On 4 November 2014 08:23, Andres Freund and...@2ndquadrant.com wrote: 6) Shouldn't any value update of

Re: [HACKERS] tracking commit timestamps

2014-11-11 Thread Jim Nasby
On 11/10/14, 7:40 AM, Alvaro Herrera wrote: Robert Haas wrote: On Sun, Nov 9, 2014 at 8:41 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Robert Haas wrote: I think the key question here is the time for which the data needs to be retained. 2^32 of anything is a lot, but why keep around

Re: [HACKERS] tracking commit timestamps

2014-11-11 Thread Alvaro Herrera
Jim Nasby wrote: On 11/10/14, 7:40 AM, Alvaro Herrera wrote: Ah, right. So AFAIK we don't need to keep anything older than RecentXmin or something like that -- which is not too old. If I recall correctly Josh Berkus was saying in a thread about pg_multixact that it used about 128kB or so

Re: [HACKERS] tracking commit timestamps

2014-11-11 Thread Jim Nasby
On 11/11/14, 2:03 PM, Alvaro Herrera wrote: Jim Nasby wrote: On 11/10/14, 7:40 AM, Alvaro Herrera wrote: Ah, right. So AFAIK we don't need to keep anything older than RecentXmin or something like that -- which is not too old. If I recall correctly Josh Berkus was saying in a thread about

Re: [HACKERS] tracking commit timestamps

2014-11-11 Thread Alvaro Herrera
Jim Nasby wrote: On 11/11/14, 2:03 PM, Alvaro Herrera wrote: Jim Nasby wrote: On 11/10/14, 7:40 AM, Alvaro Herrera wrote: Ah, right. So AFAIK we don't need to keep anything older than RecentXmin or something like that -- which is not too old. If I recall correctly Josh Berkus was saying

Re: [HACKERS] tracking commit timestamps

2014-11-10 Thread Robert Haas
On Sun, Nov 9, 2014 at 8:41 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Robert Haas wrote: I think the key question here is the time for which the data needs to be retained. 2^32 of anything is a lot, but why keep around that number of records rather than more (after all, we have

Re: [HACKERS] tracking commit timestamps

2014-11-10 Thread Petr Jelinek
On 10/11/14 08:01, Anssi Kääriäinen wrote: On Sun, 2014-11-09 at 11:57 -0500, Steve Singer wrote: The reason why Jim and myself are asking for the LSN and not just the timestamp is that I want to be able to order the transactions. Jim pointed out earlier in the thread that just ordering on

Re: [HACKERS] tracking commit timestamps

2014-11-10 Thread Robert Haas
On Mon, Nov 10, 2014 at 2:01 AM, Anssi Kääriäinen anssi.kaariai...@thl.fi wrote: There is no guarantee that a commit with later LSN has a later timestamp. There are cases where the clock could move significantly backwards. Good point. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] tracking commit timestamps

2014-11-10 Thread Petr Jelinek
On 09/11/14 17:57, Steve Singer wrote: On 11/07/2014 07:07 PM, Petr Jelinek wrote: The list of what is useful might be long, but we can't have everything there as there are space constraints, and LSN is another 8 bytes and I still want to have some bytes for storing the origin or whatever you

Re: [HACKERS] tracking commit timestamps

2014-11-10 Thread Alvaro Herrera
Robert Haas wrote: On Sun, Nov 9, 2014 at 8:41 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Robert Haas wrote: I think the key question here is the time for which the data needs to be retained. 2^32 of anything is a lot, but why keep around that number of records rather than more

Re: [HACKERS] tracking commit timestamps

2014-11-10 Thread Robert Haas
On Mon, Nov 10, 2014 at 8:39 AM, Petr Jelinek p...@2ndquadrant.com wrote: I did the calculation above wrong btw, it's actually 20 bytes not 24 bytes per record, I am inclined to just say we can live with that. If you do it as 20 bytes, you'll have to do some work to squeeze out the alignment

Re: [HACKERS] tracking commit timestamps

2014-11-10 Thread Robert Haas
On Mon, Nov 10, 2014 at 8:40 AM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Ah, right. So AFAIK we don't need to keep anything older than RecentXmin or something like that -- which is not too old. If I recall correctly Josh Berkus was saying in a thread about pg_multixact that it used

Re: [HACKERS] tracking commit timestamps

2014-11-10 Thread Steve Singer
On 11/10/2014 08:39 AM, Petr Jelinek wrote: On 09/11/14 17:57, Steve Singer wrote: On 11/07/2014 07:07 PM, Petr Jelinek wrote: The list of what is useful might be long, but we can't have everything there as there are space constraints, and LSN is another 8 bytes and I still want to have some

Re: [HACKERS] tracking commit timestamps

2014-11-09 Thread Steve Singer
On 11/07/2014 07:07 PM, Petr Jelinek wrote: The list of what is useful might be long, but we can't have everything there as there are space constraints, and LSN is another 8 bytes and I still want to have some bytes for storing the origin or whatever you want to call it there, as that's the

Re: [HACKERS] tracking commit timestamps

2014-11-09 Thread Alvaro Herrera
Robert Haas wrote: I think the key question here is the time for which the data needs to be retained. 2^32 of anything is a lot, but why keep around that number of records rather than more (after all, we have epochs to distinguish one use of a given txid from another) or fewer? The problem

Re: [HACKERS] tracking commit timestamps

2014-11-09 Thread Anssi Kääriäinen
On Sun, 2014-11-09 at 11:57 -0500, Steve Singer wrote: The reason why Jim and myself are asking for the LSN and not just the timestamp is that I want to be able to order the transactions. Jim pointed out earlier in the thread that just ordering on timestamp allows for multiple transactions

Re: [HACKERS] tracking commit timestamps

2014-11-08 Thread Petr Jelinek
On 08/11/14 03:05, Robert Haas wrote: On Fri, Nov 7, 2014 at 7:07 PM, Petr Jelinek p...@2ndquadrant.com wrote: but we can't have everything there as there are space constraints, and LSN is another 8 bytes and I still want to have some bytes for storing the origin or whatever you want to call it

Re: [HACKERS] tracking commit timestamps

2014-11-08 Thread Robert Haas
On Sat, Nov 8, 2014 at 5:35 AM, Petr Jelinek p...@2ndquadrant.com wrote: That's not what I said. I am actually ok with adding the LSN if people see it useful. I was just wondering if we can make the record smaller somehow - 24bytes per txid is around 96GB of data for whole txid range and won't

Re: [HACKERS] tracking commit timestamps

2014-11-07 Thread Petr Jelinek
On 06/11/14 08:50, Andres Freund wrote: On 2014-11-05 19:31:52 -0500, Steve Singer wrote: It also doesn't allow you to load two extensions at once on a system. You wouldn't be able to have both the 'steve_commit_order' extension and BDR installed at the same time. I don't think this patch does

Re: [HACKERS] tracking commit timestamps

2014-11-07 Thread Petr Jelinek
On 06/11/14 01:31, Steve Singer wrote: On 11/05/2014 05:43 PM, Andres Freund wrote: Is this patch supposed to: A) Add commit timestamp tracking but nothing more B) Add infrastructure to store commit timestamps and provide a facility for storing additional bits of data extensions might want

Re: [HACKERS] tracking commit timestamps

2014-11-07 Thread Andres Freund
On 2014-11-07 17:54:32 +0100, Petr Jelinek wrote: On 06/11/14 08:50, Andres Freund wrote: On 2014-11-05 19:31:52 -0500, Steve Singer wrote: It also doesn't allow you to load two extensions at once on a system. You wouldn't be able to have both the 'steve_commit_order' extension and BDR

Re: [HACKERS] tracking commit timestamps

2014-11-07 Thread Robert Haas
On Nov 5, 2014, at 7:31 PM, Steve Singer st...@ssinger.info wrote: On 11/05/2014 05:43 PM, Andres Freund wrote: On 2014-11-05 17:17:05 -0500, Steve Singer wrote: Imo that's essentially a different feature. What you essentially would need here is a 'commit sequence number' - but no timestamps.

Re: [HACKERS] tracking commit timestamps

2014-11-07 Thread Petr Jelinek
On 08/11/14 00:35, Robert Haas wrote: On Nov 5, 2014, at 7:31 PM, Steve Singer st...@ssinger.info wrote: On 11/05/2014 05:43 PM, Andres Freund wrote: On 2014-11-05 17:17:05 -0500, Steve Singer wrote: Imo that's essentially a different feature. What you essentially would need here is a 'commit

Re: [HACKERS] tracking commit timestamps

2014-11-07 Thread Robert Haas
On Fri, Nov 7, 2014 at 7:07 PM, Petr Jelinek p...@2ndquadrant.com wrote: The list of what is useful might be long, That's FUD. It might also be short. but we can't have everything there as there are space constraints, and LSN is another 8 bytes and I still want to have some bytes for

Re: [HACKERS] tracking commit timestamps

2014-11-06 Thread Craig Ringer
On 11/01/2014 09:00 PM, Michael Paquier wrote: 1) It is untested and actually there is no direct use for it in core. 2) Pushing code that we know as dead is no good, that's a feature more or less defined as maybe-useful-but-we-are-not-sure-yet-what-to-do-with-it. 3) If you're going to re-use

Re: [HACKERS] tracking commit timestamps

2014-11-05 Thread Anssi Kääriäinen
On Tue, 2014-11-04 at 23:43 -0600, Jim Nasby wrote: I'm worried about 2 commits in the same microsecond on the same system, not on 2 different systems. Or, put another way, if we're going to expose this I think it should also provide a guaranteed unique commit ordering for a single cluster.

Re: [HACKERS] tracking commit timestamps

2014-11-05 Thread Michael Paquier
On Wed, Nov 5, 2014 at 5:24 PM, Anssi Kääriäinen anssi.kaariai...@thl.fi wrote: On Tue, 2014-11-04 at 23:43 -0600, Jim Nasby wrote: I'm worried about 2 commits in the same microsecond on the same system, not on 2 different systems. Or, put another way, if we're going to expose this I

Re: [HACKERS] tracking commit timestamps

2014-11-05 Thread Jim Nasby
On 11/5/14, 6:10 AM, Michael Paquier wrote: In addition, I wonder if this feature would be misused. Record transaction ids to a table to find out commit order (use case could be storing historical row versions for example). Do a dump and restore on another cluster, and all the

Re: [HACKERS] tracking commit timestamps

2014-11-05 Thread Jim Nasby
On 11/5/14, 10:30 AM, Andres Freund wrote: Except that commit time is not guaranteed unique *even on a single system*. That's my whole point. If we're going to bother with all the commit time machinery it seems really silly to provide a way to uniquely order every commit. Well. I think that's

Re: [HACKERS] tracking commit timestamps

2014-11-05 Thread Andres Freund
On 2014-11-05 10:23:15 -0600, Jim Nasby wrote: On 11/5/14, 6:10 AM, Michael Paquier wrote: In addition, I wonder if this feature would be misused. Record transaction ids to a table to find out commit order (use case could be storing historical row versions for example). Do a

Re: [HACKERS] tracking commit timestamps

2014-11-05 Thread Andres Freund
On 2014-11-05 10:34:40 -0600, Jim Nasby wrote: On 11/5/14, 10:30 AM, Andres Freund wrote: Except that commit time is not guaranteed unique *even on a single system*. That's my whole point. If we're going to bother with all the commit time machinery it seems really silly to provide a way to

Re: [HACKERS] tracking commit timestamps

2014-11-05 Thread Kevin Grittner
Jim Nasby jim.na...@bluetreble.com wrote: for a single system AIUI all we need to do is expose the LSN of each commit record and that will give you the exact and unique order in which transactions committed. This isn't a hypothetical feature either; if we had this, logical replication

Re: [HACKERS] tracking commit timestamps

2014-11-05 Thread Steve Singer
On 11/05/2014 11:23 AM, Jim Nasby wrote: Except that commit time is not guaranteed unique *even on a single system*. That's my whole point. If we're going to bother with all the commit time machinery it seems really silly to provide a way to uniquely order every commit. Clearly trying to

Re: [HACKERS] tracking commit timestamps

2014-11-05 Thread Andres Freund
On 2014-11-05 17:17:05 -0500, Steve Singer wrote: It isn't just 'replication' systems that have a need for getting the commit order of transactions on a single system. I have a application (not slony) where we want to query a table but order the output based on the transaction commit order of

Re: [HACKERS] tracking commit timestamps

2014-11-05 Thread Steve Singer
On 11/05/2014 05:43 PM, Andres Freund wrote: On 2014-11-05 17:17:05 -0500, Steve Singer wrote: Imo that's essentially a different feature. What you essentially would need here is a 'commit sequence number' - but no timestamps. And probably to be useful that number has to be 8 bytes in itself.

Re: [HACKERS] tracking commit timestamps

2014-11-05 Thread Andres Freund
On 2014-11-05 19:31:52 -0500, Steve Singer wrote: On 11/05/2014 05:43 PM, Andres Freund wrote: On 2014-11-05 17:17:05 -0500, Steve Singer wrote: Imo that's essentially a different feature. What you essentially would need here is a 'commit sequence number' - but no timestamps. And probably to

Re: [HACKERS] tracking commit timestamps

2014-11-04 Thread Andres Freund
On 2014-11-02 19:27:25 +0100, Petr Jelinek wrote: Well, Michael has point that the extradata is pretty much useless currently, perhaps it would help to add the interface to set extradata? Only accessible via C and useless aren't the same thing. But sure, add it. Greetings, Andres Freund --

  1   2   >