Re: [HACKERS] tracking commit timestamps

2015-03-09 Thread Alvaro Herrera
Petr Jelinek wrote: > >>On Thu, Dec 4, 2014 at 12:08 PM, Fujii Masao > >>wrote: > >>>I got the following assertion failure when I executed > >>>pg_xact_commit_timestamp() > >>>in the standby server. > >>> > >>>=# select pg_xact_commit_timestamp('1000'::xid); > >>>TRAP: FailedAssertion("!(((oldes

Re: [HACKERS] tracking commit timestamps

2015-01-22 Thread Petr Jelinek
On 05/01/15 17:50, Petr Jelinek wrote: On 05/01/15 16:17, Petr Jelinek wrote: On 05/01/15 07:28, Fujii Masao wrote: On Thu, Dec 4, 2014 at 12:08 PM, Fujii Masao wrote: On Wed, Dec 3, 2014 at 11:54 PM, Alvaro Herrera wrote: Pushed with some extra cosmetic tweaks. I got the following assert

Re: [HACKERS] tracking commit timestamps

2015-01-06 Thread Robert Haas
On Tue, Jan 6, 2015 at 2:58 AM, Michael Paquier wrote: > So, we would need additional information other than the node ID *and* > the timestamp to ensure proper transaction commit ordering on Windows. > That's not cool and makes this feature very limited on this platform. You can't use the timesta

Re: [HACKERS] tracking commit timestamps

2015-01-06 Thread Petr Jelinek
On 06/01/15 08:58, Michael Paquier wrote: On Fri, Dec 19, 2014 at 3:53 PM, Noah Misch wrote: localhost template1=# select clock_timestamp(), pg_sleep(.1 * (n % 2)) from generate_series(0,7) t(n); clock_timestamp| pg_sleep ---+-- 2014-12-18

Re: [HACKERS] tracking commit timestamps

2015-01-05 Thread Michael Paquier
On Fri, Dec 19, 2014 at 3:53 PM, Noah Misch wrote: > localhost template1=# select clock_timestamp(), pg_sleep(.1 * (n % 2)) from > generate_series(0,7) t(n); > clock_timestamp| pg_sleep > ---+-- > 2014-12-18 08:34:34.522126+00 | > 2014-12-18 0

Re: [HACKERS] tracking commit timestamps

2015-01-05 Thread Petr Jelinek
On 05/01/15 16:17, Petr Jelinek wrote: On 05/01/15 07:28, Fujii Masao wrote: On Thu, Dec 4, 2014 at 12:08 PM, Fujii Masao wrote: On Wed, Dec 3, 2014 at 11:54 PM, Alvaro Herrera wrote: Pushed with some extra cosmetic tweaks. I got the following assertion failure when I executed pg_xact_comm

Re: [HACKERS] tracking commit timestamps

2015-01-05 Thread Petr Jelinek
On 05/01/15 07:28, Fujii Masao wrote: On Thu, Dec 4, 2014 at 12:08 PM, Fujii Masao wrote: On Wed, Dec 3, 2014 at 11:54 PM, Alvaro Herrera wrote: Pushed with some extra cosmetic tweaks. I got the following assertion failure when I executed pg_xact_commit_timestamp() in the standby server. =

Re: [HACKERS] tracking commit timestamps

2015-01-04 Thread Craig Ringer
On 12/19/2014 02:53 PM, Noah Misch wrote: > The test assumed that no two transactions of a given backend will get the same > timestamp value from now(). That holds so long as ticks of the system time > are small enough. Not so on at least some Windows configurations. Most Windows systems with no

Re: [HACKERS] tracking commit timestamps

2015-01-04 Thread Fujii Masao
On Thu, Dec 4, 2014 at 12:08 PM, Fujii Masao wrote: > On Wed, Dec 3, 2014 at 11:54 PM, Alvaro Herrera > wrote: >> Pushed with some extra cosmetic tweaks. > > I got the following assertion failure when I executed > pg_xact_commit_timestamp() > in the standby server. > > =# select pg_xact_commit_t

Re: [HACKERS] tracking commit timestamps

2014-12-18 Thread Noah Misch
On Tue, Dec 16, 2014 at 01:05:31AM -0300, Alvaro Herrera wrote: > Noah Misch wrote: > > On Mon, Dec 15, 2014 at 12:12:10AM -0800, Michael Paquier wrote: > > > > FWIW, I just tried that with MinGW-32 and I can see the error on Win7. > > > I also checked that changing "< now()" to "<= now()" fixed t

Re: [HACKERS] tracking commit timestamps

2014-12-15 Thread Alvaro Herrera
Noah Misch wrote: > On Mon, Dec 15, 2014 at 12:12:10AM -0800, Michael Paquier wrote: > > FWIW, I just tried that with MinGW-32 and I can see the error on Win7. > > I also checked that changing "< now()" to "<= now()" fixed the > > problem, so your assumption was right, Petr. > > Committed, after

Re: [HACKERS] tracking commit timestamps

2014-12-15 Thread Noah Misch
On Mon, Dec 15, 2014 at 12:12:10AM -0800, Michael Paquier wrote: > On Wed, Dec 10, 2014 at 6:50 PM, Noah Misch wrote: > > On Mon, Dec 08, 2014 at 02:23:39AM +0100, Petr Jelinek wrote: > >> On 08/12/14 00:56, Noah Misch wrote: > >> >The commit_ts test suite gives me the attached diff on a 32-bit Mi

Re: [HACKERS] tracking commit timestamps

2014-12-15 Thread Petr Jelinek
On 15/12/14 09:12, Michael Paquier wrote: On Wed, Dec 10, 2014 at 6:50 PM, Noah Misch wrote: On Mon, Dec 08, 2014 at 02:23:39AM +0100, Petr Jelinek wrote: On 08/12/14 00:56, Noah Misch wrote: The commit_ts test suite gives me the attached diff on a 32-bit MinGW build running on 64-bit Windows

Re: [HACKERS] tracking commit timestamps

2014-12-15 Thread Michael Paquier
On Wed, Dec 10, 2014 at 6:50 PM, Noah Misch wrote: > On Mon, Dec 08, 2014 at 02:23:39AM +0100, Petr Jelinek wrote: >> On 08/12/14 00:56, Noah Misch wrote: >> >The commit_ts test suite gives me the attached diff on a 32-bit MinGW build >> >running on 64-bit Windows Server 2003. I have not checked

Re: [HACKERS] tracking commit timestamps

2014-12-10 Thread Noah Misch
On Mon, Dec 08, 2014 at 02:23:39AM +0100, Petr Jelinek wrote: > On 08/12/14 00:56, Noah Misch wrote: > >The commit_ts test suite gives me the attached diff on a 32-bit MinGW build > >running on 64-bit Windows Server 2003. I have not checked other Windows > >configurations; the suite does pass on G

Re: [HACKERS] tracking commit timestamps

2014-12-07 Thread Petr Jelinek
On 08/12/14 00:56, Noah Misch wrote: On Wed, Dec 03, 2014 at 11:54:38AM -0300, Alvaro Herrera wrote: Pushed with some extra cosmetic tweaks. The commit_ts test suite gives me the attached diff on a 32-bit MinGW build running on 64-bit Windows Server 2003. I have not checked other Windows conf

Re: [HACKERS] tracking commit timestamps

2014-12-07 Thread Noah Misch
On Wed, Dec 03, 2014 at 11:54:38AM -0300, Alvaro Herrera wrote: > Pushed with some extra cosmetic tweaks. The commit_ts test suite gives me the attached diff on a 32-bit MinGW build running on 64-bit Windows Server 2003. I have not checked other Windows configurations; the suite does pass on GNU/

Re: [HACKERS] tracking commit timestamps

2014-12-03 Thread Fujii Masao
On Thu, Dec 4, 2014 at 12:58 PM, Simon Riggs wrote: > On 4 December 2014 at 03:08, Fujii Masao wrote: > >> #1. set up and start the master and standby servers with >> track_commit_timestamp disabled >> #2. enable track_commit_timestamp in the master and restart the master >> #3. run some write tr

Re: [HACKERS] tracking commit timestamps

2014-12-03 Thread Simon Riggs
On 4 December 2014 at 03:08, Fujii Masao wrote: > #1. set up and start the master and standby servers with > track_commit_timestamp disabled > #2. enable track_commit_timestamp in the master and restart the master > #3. run some write transactions > #4. enable track_commit_timestamp in the standb

Re: [HACKERS] tracking commit timestamps

2014-12-03 Thread Fujii Masao
On Wed, Dec 3, 2014 at 11:54 PM, Alvaro Herrera wrote: > Pushed with some extra cosmetic tweaks. I got the following assertion failure when I executed pg_xact_commit_timestamp() in the standby server. =# select pg_xact_commit_timestamp('1000'::xid); TRAP: FailedAssertion("!(((oldestCommitTs) !=

Re: [HACKERS] tracking commit timestamps

2014-12-03 Thread Robert Haas
On Wed, Dec 3, 2014 at 2:36 PM, Alvaro Herrera wrote: > Alvaro Herrera wrote: >> Robert Haas wrote: >> > I'm kind of disappointed that, in spite of previous review comments, >> > this got committed with extensive use of the CommitTs naming. I think >> > that's confusing, but it's also something t

Re: [HACKERS] tracking commit timestamps

2014-12-03 Thread Alvaro Herrera
Alvaro Herrera wrote: > Robert Haas wrote: > > I'm kind of disappointed that, in spite of previous review comments, > > this got committed with extensive use of the CommitTs naming. I think > > that's confusing, but it's also something that will be awkward if we > > want to add other data, such a

Re: [HACKERS] tracking commit timestamps

2014-12-03 Thread Alvaro Herrera
Robert Haas wrote: > On Mon, Dec 1, 2014 at 5:34 PM, Alvaro Herrera > wrote: > > I made two more changes: > > 1. introduce newestCommitTs. Original code was using lastCommitXact to > > check that no "future" transaction is asked for, but this doesn't really > > work if a long-running transaction

Re: [HACKERS] tracking commit timestamps

2014-12-03 Thread Robert Haas
On Mon, Dec 1, 2014 at 5:34 PM, Alvaro Herrera wrote: > I made two more changes: > 1. introduce newestCommitTs. Original code was using lastCommitXact to > check that no "future" transaction is asked for, but this doesn't really > work if a long-running transaction is committed, because asking fo

Re: [HACKERS] tracking commit timestamps

2014-12-03 Thread Petr Jelinek
On 03/12/14 15:54, Alvaro Herrera wrote: Pushed with some extra cosmetic tweaks. Cool, thanks! -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To ma

Re: [HACKERS] tracking commit timestamps

2014-12-03 Thread Alvaro Herrera
Pushed with some extra cosmetic tweaks. -- Álvaro Herrerahttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/m

Re: [HACKERS] tracking commit timestamps

2014-12-01 Thread Alvaro Herrera
Petr Jelinek wrote: > On 25/11/14 16:30, Alvaro Herrera wrote: > >Petr Jelinek wrote: > >>On 25/11/14 16:23, Alvaro Herrera wrote: > >>>Robert Haas wrote: > > > Maybe 0 should get translated to a NULL return, instead of a bogus > timestamp. > >>> > >>>That's one idea --- surely no transact

Re: [HACKERS] tracking commit timestamps

2014-11-25 Thread Michael Paquier
On Wed, Nov 26, 2014 at 1:51 AM, Simon Riggs wrote: > On 25 November 2014 at 16:18, Petr Jelinek wrote: > >> Won't the pg_last_committed_xact() on slave + pg_xact_commit_timestamp() on >> master with the xid returned by slave accomplish the same thing? > > Surely the pg_last_committed_xact() will

Re: [HACKERS] tracking commit timestamps

2014-11-25 Thread Simon Riggs
On 25 November 2014 at 16:18, Petr Jelinek wrote: > Won't the pg_last_committed_xact() on slave + pg_xact_commit_timestamp() on > master with the xid returned by slave accomplish the same thing? Surely the pg_last_committed_xact() will return the same value on standby as it did on the master? -

Re: [HACKERS] tracking commit timestamps

2014-11-25 Thread Petr Jelinek
On 25/11/14 17:16, Simon Riggs wrote: On 25 November 2014 at 13:35, Fujii Masao wrote: Can I check my understanding? Probably we cannot use this feature to calculate the actual replication lag by, for example, comparing the result of pg_last_committed_xact() in the master and that of pg_last_x

Re: [HACKERS] tracking commit timestamps

2014-11-25 Thread Simon Riggs
On 25 November 2014 at 13:35, Fujii Masao wrote: > Can I check my understanding? Probably we cannot use this feature to calculate > the actual replication lag by, for example, comparing the result of > pg_last_committed_xact() in the master and that of > pg_last_xact_replay_timestamp() > in the s

Re: [HACKERS] tracking commit timestamps

2014-11-25 Thread Petr Jelinek
On 25/11/14 16:30, Alvaro Herrera wrote: Petr Jelinek wrote: On 25/11/14 16:23, Alvaro Herrera wrote: Robert Haas wrote: Maybe 0 should get translated to a NULL return, instead of a bogus timestamp. That's one idea --- surely no transaction is going to commit at 00:00:00 on 2000-01-01 anym

Re: [HACKERS] tracking commit timestamps

2014-11-25 Thread Alvaro Herrera
Petr Jelinek wrote: > On 25/11/14 16:23, Alvaro Herrera wrote: > >Robert Haas wrote: > >>Maybe 0 should get translated to a NULL return, instead of a bogus > >>timestamp. > > > >That's one idea --- surely no transaction is going to commit at 00:00:00 > >on 2000-01-01 anymore. Yet this is somewha

Re: [HACKERS] tracking commit timestamps

2014-11-25 Thread Petr Jelinek
On 25/11/14 16:23, Alvaro Herrera wrote: Robert Haas wrote: On Tue, Nov 25, 2014 at 9:19 AM, Alvaro Herrera wrote: Fujii Masao wrote: On Tue, Nov 25, 2014 at 7:58 AM, Alvaro Herrera wrote: And here is v10 which fixes conflicts with Heikki's WAL API changes (no changes otherwise). After so

Re: [HACKERS] tracking commit timestamps

2014-11-25 Thread Alvaro Herrera
Robert Haas wrote: > On Tue, Nov 25, 2014 at 9:19 AM, Alvaro Herrera > wrote: > > Fujii Masao wrote: > >> On Tue, Nov 25, 2014 at 7:58 AM, Alvaro Herrera > >> wrote: > >> >> And here is v10 which fixes conflicts with Heikki's WAL API changes (no > >> >> changes otherwise). > >> > > >> > After som

Re: [HACKERS] tracking commit timestamps

2014-11-25 Thread Robert Haas
On Tue, Nov 25, 2014 at 9:19 AM, Alvaro Herrera wrote: > Fujii Masao wrote: >> On Tue, Nov 25, 2014 at 7:58 AM, Alvaro Herrera >> wrote: >> >> And here is v10 which fixes conflicts with Heikki's WAL API changes (no >> >> changes otherwise). >> > >> > After some slight additional changes, here's v

Re: [HACKERS] tracking commit timestamps

2014-11-25 Thread Fujii Masao
On Tue, Nov 25, 2014 at 11:19 PM, Alvaro Herrera wrote: > Fujii Masao wrote: >> On Tue, Nov 25, 2014 at 7:58 AM, Alvaro Herrera >> wrote: >> >> And here is v10 which fixes conflicts with Heikki's WAL API changes (no >> >> changes otherwise). >> > >> > After some slight additional changes, here's

Re: [HACKERS] tracking commit timestamps

2014-11-25 Thread Alvaro Herrera
Fujii Masao wrote: > On Tue, Nov 25, 2014 at 7:58 AM, Alvaro Herrera > wrote: > >> And here is v10 which fixes conflicts with Heikki's WAL API changes (no > >> changes otherwise). > > > > After some slight additional changes, here's v11, which I intend to > > commit early tomorrow. The main chang

Re: [HACKERS] tracking commit timestamps

2014-11-25 Thread Fujii Masao
On Tue, Nov 25, 2014 at 7:58 AM, Alvaro Herrera wrote: >> And here is v10 which fixes conflicts with Heikki's WAL API changes (no >> changes otherwise). > > After some slight additional changes, here's v11, which I intend to > commit early tomorrow. The main change is moving the test module from

Re: [HACKERS] tracking commit timestamps

2014-11-24 Thread Alvaro Herrera
> And here is v10 which fixes conflicts with Heikki's WAL API changes (no > changes otherwise). After some slight additional changes, here's v11, which I intend to commit early tomorrow. The main change is moving the test module from contrib to src/test/modules. -- Álvaro Herrera

Re: [HACKERS] tracking commit timestamps

2014-11-21 Thread Petr Jelinek
On 21/11/14 00:17, Petr Jelinek wrote: On 19/11/14 17:30, Steve Singer wrote: On 11/19/2014 08:22 AM, Alvaro Herrera wrote: I think we're overblowing the pg_upgrade issue. Surely we don't need to preserve commit_ts data when upgrading across major versions; and pg_upgrade is perfectly prepare

Re: [HACKERS] tracking commit timestamps

2014-11-20 Thread Petr Jelinek
On 19/11/14 17:30, Steve Singer wrote: On 11/19/2014 08:22 AM, Alvaro Herrera wrote: I think we're overblowing the pg_upgrade issue. Surely we don't need to preserve commit_ts data when upgrading across major versions; and pg_upgrade is perfectly prepared to remove old data when upgrading (act

Re: [HACKERS] tracking commit timestamps

2014-11-19 Thread Steve Singer
On 11/19/2014 08:22 AM, Alvaro Herrera wrote: I think we're overblowing the pg_upgrade issue. Surely we don't need to preserve commit_ts data when upgrading across major versions; and pg_upgrade is perfectly prepared to remove old data when upgrading (actually it just doesn't copy it; consider

Re: [HACKERS] tracking commit timestamps

2014-11-19 Thread Robert Haas
On Wed, Nov 19, 2014 at 8:22 AM, Alvaro Herrera wrote: > Petr Jelinek wrote: >> This is good point, we are not too late in the cycle that LSN couldn't be >> added later if we find that it is indeed needed (and we don't have to care >> about pg_upgrade until beta). > > I think we're overblowing the

Re: [HACKERS] tracking commit timestamps

2014-11-19 Thread Alvaro Herrera
Petr Jelinek wrote: > This is good point, we are not too late in the cycle that LSN couldn't be > added later if we find that it is indeed needed (and we don't have to care > about pg_upgrade until beta). I think we're overblowing the pg_upgrade issue. Surely we don't need to preserve commit_ts

Re: [HACKERS] tracking commit timestamps

2014-11-19 Thread Petr Jelinek
On 19/11/14 12:20, Simon Riggs wrote: On 19 November 2014 02:12, Petr Jelinek wrote: Maybe we need better explanation of the LSN use-case(s) to understand why it should be stored here and why the other solutions are significantly worse. We should apply the same standard that has been applied

Re: [HACKERS] tracking commit timestamps

2014-11-19 Thread Simon Riggs
On 19 November 2014 02:12, Petr Jelinek wrote: > Maybe we need better explanation of the LSN use-case(s) to understand why it > should be stored here and why the other solutions are significantly worse. We should apply the same standard that has been applied elsewhere. If someone can show some s

Re: [HACKERS] tracking commit timestamps

2014-11-18 Thread Petr Jelinek
On 15/11/14 13:36, Simon Riggs wrote: On 15 November 2014 04:32, Steve Singer wrote: The use cases I'm talking about aren't really replication related. Often I have come across systems that want to do something such as 'select * from orders where X > the_last_row_I_saw order by X' and then do

Re: [HACKERS] tracking commit timestamps

2014-11-15 Thread Simon Riggs
On 15 November 2014 04:32, Steve Singer wrote: > The use cases I'm talking about aren't really replication related. Often I > have come across systems that want to do something such as 'select * from > orders where X > the_last_row_I_saw order by X' and then do further > processing on the order.

Re: [HACKERS] tracking commit timestamps

2014-11-14 Thread Steve Singer
On 11/14/2014 08:21 PM, Simon Riggs wrote: The requested information is already available, as discussed. Logical decoding adds commit ordering for *exactly* the purpose of using it for replication, available to all solutions. This often requested feature has now been added and doesn't need to be

Re: [HACKERS] tracking commit timestamps

2014-11-14 Thread Simon Riggs
On 14 November 2014 17:12, Robert Haas wrote: > We are not talking about loading 20 new > requirements on top of this patch; that would be intolerable. We're > talking about adding one additional piece of information that has been > requested multiple times over the years. The requested informa

Re: [HACKERS] tracking commit timestamps

2014-11-14 Thread Robert Haas
On Thu, Nov 13, 2014 at 6:55 PM, Simon Riggs wrote: > On 13 November 2014 21:24, Robert Haas wrote: >> On Thu, Nov 13, 2014 at 8:18 AM, Simon Riggs wrote: >>> Ordering transactions in LSN order is very precisly the remit of the >>> existing logical decoding API. Any user that wishes to see a com

Re: [HACKERS] tracking commit timestamps

2014-11-13 Thread Simon Riggs
On 13 November 2014 21:24, Robert Haas wrote: > On Thu, Nov 13, 2014 at 8:18 AM, Simon Riggs wrote: >> Ordering transactions in LSN order is very precisly the remit of the >> existing logical decoding API. Any user that wishes to see a commits >> in sequence can do so using that API. BDR already

Re: [HACKERS] tracking commit timestamps

2014-11-13 Thread Robert Haas
On Thu, Nov 13, 2014 at 8:18 AM, Simon Riggs wrote: > Ordering transactions in LSN order is very precisly the remit of the > existing logical decoding API. Any user that wishes to see a commits > in sequence can do so using that API. BDR already does this, as do > other users of the decoding API.

Re: [HACKERS] tracking commit timestamps

2014-11-13 Thread Petr Jelinek
On 13/11/14 14:18, Simon Riggs wrote: So in summary... the components are * Commit LSN order is useful for applying changes - available by logical decoding * Commit timestamps and nodeid are useful for conflict resolution - available from this patch Both components have been designed in ways th

Re: [HACKERS] tracking commit timestamps

2014-11-13 Thread Simon Riggs
On 9 November 2014 16:57, Steve Singer wrote: > On 11/07/2014 07:07 PM, Petr Jelinek wrote: > > >> The list of what is useful might be long, but we can't have everything >> there as there are space constraints, and LSN is another 8 bytes and I still >> want to have some bytes for storing the "orig

Re: [HACKERS] tracking commit timestamps

2014-11-13 Thread Petr Jelinek
On 13/11/14 07:04, Michael Paquier wrote: On Wed, Nov 12, 2014 at 10:06 PM, Petr Jelinek wrote: Brief list of changes: - the commit timestamp record now stores timestamp, lsn and nodeid Now that not only the commit timestamp is stored, calling that "commit timestamp", "committs" or "commit_t

Re: [HACKERS] tracking commit timestamps

2014-11-12 Thread Michael Paquier
On Wed, Nov 12, 2014 at 10:06 PM, Petr Jelinek wrote: > Brief list of changes: > - the commit timestamp record now stores timestamp, lsn and nodeid Now that not only the commit timestamp is stored, calling that "commit timestamp", "committs" or "commit_timestamp" is strange, no? If this patch is

Re: [HACKERS] tracking commit timestamps

2014-11-12 Thread Michael Paquier
On Thu, Nov 13, 2014 at 7:56 AM, Jim Nasby wrote: > On 11/12/14, 7:06 AM, Petr Jelinek wrote: >> >> - if the xid passed to get interface is out of range -infinity timestamp >> is returned (I think it's bad idea to throw errors here as the valid range >> is not static and same ID can start throwi

Re: [HACKERS] tracking commit timestamps

2014-11-12 Thread Jim Nasby
On 11/12/14, 7:06 AM, Petr Jelinek wrote: - if the xid passed to get interface is out of range -infinity timestamp is returned (I think it's bad idea to throw errors here as the valid range is not static and same ID can start throwing errors between calls theoretically) Wouldn't NULL be mor

Re: [HACKERS] tracking commit timestamps

2014-11-12 Thread Petr Jelinek
On 10/11/14 14:53, Robert Haas wrote: On Mon, Nov 10, 2014 at 8:39 AM, Petr Jelinek wrote: I did the calculation above wrong btw, it's actually 20 bytes not 24 bytes per record, I am inclined to just say we can live with that. If you do it as 20 bytes, you'll have to do some work to squeeze o

Re: [HACKERS] tracking commit timestamps

2014-11-11 Thread Alvaro Herrera
Jim Nasby wrote: > On 11/11/14, 2:03 PM, Alvaro Herrera wrote: > >Jim Nasby wrote: > >>On 11/10/14, 7:40 AM, Alvaro Herrera wrote: > > > >>>Ah, right. So AFAIK we don't need to keep anything older than > >>>RecentXmin or something like that -- which is not too old. If I recall > >>>correctly Josh

Re: [HACKERS] tracking commit timestamps

2014-11-11 Thread Jim Nasby
On 11/11/14, 2:03 PM, Alvaro Herrera wrote: Jim Nasby wrote: On 11/10/14, 7:40 AM, Alvaro Herrera wrote: Ah, right. So AFAIK we don't need to keep anything older than RecentXmin or something like that -- which is not too old. If I recall correctly Josh Berkus was saying in a thread about pg

Re: [HACKERS] tracking commit timestamps

2014-11-11 Thread Alvaro Herrera
Jim Nasby wrote: > On 11/10/14, 7:40 AM, Alvaro Herrera wrote: > >Ah, right. So AFAIK we don't need to keep anything older than > >RecentXmin or something like that -- which is not too old. If I recall > >correctly Josh Berkus was saying in a thread about pg_multixact that it > >used about 128kB

Re: [HACKERS] tracking commit timestamps

2014-11-11 Thread Jim Nasby
On 11/10/14, 7:40 AM, Alvaro Herrera wrote: Robert Haas wrote: On Sun, Nov 9, 2014 at 8:41 PM, Alvaro Herrera wrote: Robert Haas wrote: I think the key question here is the time for which the data needs to be retained. 2^32 of anything is a lot, but why keep around that number of records rat

Re: [HACKERS] tracking commit timestamps

2014-11-11 Thread Andres Freund
On 2014-11-11 17:09:54 +, Simon Riggs wrote: > On 11 November 2014 16:19, Andres Freund wrote: > > On 2014-11-11 16:10:47 +, Simon Riggs wrote: > >> On 4 November 2014 08:23, Andres Freund wrote: > >> > >> >> 6) Shouldn't any value update of track_commit_timestamp be tracked in > >> >> XL

Re: [HACKERS] tracking commit timestamps

2014-11-11 Thread Simon Riggs
On 11 November 2014 16:19, Andres Freund wrote: > On 2014-11-11 16:10:47 +, Simon Riggs wrote: >> On 4 November 2014 08:23, Andres Freund wrote: >> >> >> 6) Shouldn't any value update of track_commit_timestamp be tracked in >> >> XLogReportParameters? That's thinking about making the commit t

Re: [HACKERS] tracking commit timestamps

2014-11-11 Thread Simon Riggs
On 9 November 2014 16:57, Steve Singer wrote: > On 11/07/2014 07:07 PM, Petr Jelinek wrote: > > >> The list of what is useful might be long, but we can't have everything >> there as there are space constraints, and LSN is another 8 bytes and I still >> want to have some bytes for storing the "orig

Re: [HACKERS] tracking commit timestamps

2014-11-11 Thread Andres Freund
On 2014-11-11 16:10:47 +, Simon Riggs wrote: > On 4 November 2014 08:23, Andres Freund wrote: > > >> 6) Shouldn't any value update of track_commit_timestamp be tracked in > >> XLogReportParameters? That's thinking about making the commit timestamp > >> available on standbys as well.. > > > >

Re: [HACKERS] tracking commit timestamps

2014-11-11 Thread Simon Riggs
On 4 November 2014 08:23, Andres Freund wrote: >> 6) Shouldn't any value update of track_commit_timestamp be tracked in >> XLogReportParameters? That's thinking about making the commit timestamp >> available on standbys as well.. > > Yes, it should. Agree committs should be able to run on standb

Re: [HACKERS] tracking commit timestamps

2014-11-10 Thread Steve Singer
On 11/10/2014 08:39 AM, Petr Jelinek wrote: On 09/11/14 17:57, Steve Singer wrote: On 11/07/2014 07:07 PM, Petr Jelinek wrote: The list of what is useful might be long, but we can't have everything there as there are space constraints, and LSN is another 8 bytes and I still want to have some by

Re: [HACKERS] tracking commit timestamps

2014-11-10 Thread Robert Haas
On Mon, Nov 10, 2014 at 8:40 AM, Alvaro Herrera wrote: > Ah, right. So AFAIK we don't need to keep anything older than > RecentXmin or something like that -- which is not too old. If I recall > correctly Josh Berkus was saying in a thread about pg_multixact that it > used about 128kB or so in <=

Re: [HACKERS] tracking commit timestamps

2014-11-10 Thread Robert Haas
On Mon, Nov 10, 2014 at 8:39 AM, Petr Jelinek wrote: > I did the calculation above wrong btw, it's actually 20 bytes not 24 bytes > per record, I am inclined to just say we can live with that. If you do it as 20 bytes, you'll have to do some work to squeeze out the alignment padding. I'm incline

Re: [HACKERS] tracking commit timestamps

2014-11-10 Thread Alvaro Herrera
Robert Haas wrote: > On Sun, Nov 9, 2014 at 8:41 PM, Alvaro Herrera > wrote: > > Robert Haas wrote: > >> I think the key question here is the time for which the data needs to > >> be retained. 2^32 of anything is a lot, but why keep around that > >> number of records rather than more (after all,

Re: [HACKERS] tracking commit timestamps

2014-11-10 Thread Petr Jelinek
On 09/11/14 17:57, Steve Singer wrote: On 11/07/2014 07:07 PM, Petr Jelinek wrote: The list of what is useful might be long, but we can't have everything there as there are space constraints, and LSN is another 8 bytes and I still want to have some bytes for storing the "origin" or whatever you

Re: [HACKERS] tracking commit timestamps

2014-11-10 Thread Robert Haas
On Mon, Nov 10, 2014 at 2:01 AM, Anssi Kääriäinen wrote: > There is no guarantee that a commit with later LSN has a later > timestamp. There are cases where the clock could move significantly > backwards. Good point. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgr

Re: [HACKERS] tracking commit timestamps

2014-11-10 Thread Petr Jelinek
On 10/11/14 08:01, Anssi Kääriäinen wrote: On Sun, 2014-11-09 at 11:57 -0500, Steve Singer wrote: The reason why Jim and myself are asking for the LSN and not just the timestamp is that I want to be able to order the transactions. Jim pointed out earlier in the thread that just ordering on time

Re: [HACKERS] tracking commit timestamps

2014-11-10 Thread Robert Haas
On Sun, Nov 9, 2014 at 8:41 PM, Alvaro Herrera wrote: > Robert Haas wrote: >> I think the key question here is the time for which the data needs to >> be retained. 2^32 of anything is a lot, but why keep around that >> number of records rather than more (after all, we have epochs to >> distinguis

Re: [HACKERS] tracking commit timestamps

2014-11-09 Thread Anssi Kääriäinen
On Sun, 2014-11-09 at 11:57 -0500, Steve Singer wrote: > The reason why Jim and myself are asking for the LSN and not just the > timestamp is that I want to be able to order the transactions. Jim > pointed out earlier in the thread that just ordering on timestamp allows > for multiple transacti

Re: [HACKERS] tracking commit timestamps

2014-11-09 Thread Alvaro Herrera
Robert Haas wrote: > I think the key question here is the time for which the data needs to > be retained. 2^32 of anything is a lot, but why keep around that > number of records rather than more (after all, we have epochs to > distinguish one use of a given txid from another) or fewer? The probl

Re: [HACKERS] tracking commit timestamps

2014-11-09 Thread Steve Singer
On 11/07/2014 07:07 PM, Petr Jelinek wrote: The list of what is useful might be long, but we can't have everything there as there are space constraints, and LSN is another 8 bytes and I still want to have some bytes for storing the "origin" or whatever you want to call it there, as that's the

Re: [HACKERS] tracking commit timestamps

2014-11-08 Thread Robert Haas
On Sat, Nov 8, 2014 at 5:35 AM, Petr Jelinek wrote: > That's not what I said. I am actually ok with adding the LSN if people see > it useful. > I was just wondering if we can make the record smaller somehow - 24bytes per > txid is around 96GB of data for whole txid range and won't work with pages

Re: [HACKERS] tracking commit timestamps

2014-11-08 Thread Petr Jelinek
On 08/11/14 03:05, Robert Haas wrote: On Fri, Nov 7, 2014 at 7:07 PM, Petr Jelinek wrote: but we can't have everything there as there are space constraints, and LSN is another 8 bytes and I still want to have some bytes for storing the "origin" or whatever you want to call it there, as that's t

Re: [HACKERS] tracking commit timestamps

2014-11-07 Thread Robert Haas
On Fri, Nov 7, 2014 at 7:07 PM, Petr Jelinek wrote: > The list of what is useful might be long, That's FUD. It might also be short. > but we can't have everything there > as there are space constraints, and LSN is another 8 bytes and I still want > to have some bytes for storing the "origin" or

Re: [HACKERS] tracking commit timestamps

2014-11-07 Thread Petr Jelinek
On 08/11/14 00:35, Robert Haas wrote: On Nov 5, 2014, at 7:31 PM, Steve Singer wrote: On 11/05/2014 05:43 PM, Andres Freund wrote: On 2014-11-05 17:17:05 -0500, Steve Singer wrote: Imo that's essentially a different feature. What you essentially would need here is a 'commit sequence number' - b

Re: [HACKERS] tracking commit timestamps

2014-11-07 Thread Robert Haas
On Nov 4, 2014, at 4:20 PM, Peter Eisentraut wrote: > On 11/3/14 5:17 PM, Petr Jelinek wrote: >>> Please don't name anything "committs". That looks like a misspelling of >>> something. >>> >>> There is nothing wrong with >>> >>> pg_get_transaction_commit_timestamp() >>> >>> If you want to redu

Re: [HACKERS] tracking commit timestamps

2014-11-07 Thread Robert Haas
> On Nov 5, 2014, at 7:31 PM, Steve Singer wrote: >> On 11/05/2014 05:43 PM, Andres Freund wrote: >> On 2014-11-05 17:17:05 -0500, Steve Singer wrote: >> Imo that's essentially a different feature. What you essentially would >> need here is a 'commit sequence number' - but no timestamps. And >> pr

Re: [HACKERS] tracking commit timestamps

2014-11-07 Thread Andres Freund
On 2014-11-07 17:54:32 +0100, Petr Jelinek wrote: > On 06/11/14 08:50, Andres Freund wrote: > >On 2014-11-05 19:31:52 -0500, Steve Singer wrote: > >>It also doesn't allow you to load two extensions at once on a system. > >>You wouldn't be able to have both the 'steve_commit_order' extension > >>and

Re: [HACKERS] tracking commit timestamps

2014-11-07 Thread Petr Jelinek
On 06/11/14 01:31, Steve Singer wrote: On 11/05/2014 05:43 PM, Andres Freund wrote: Is this patch supposed to: A) Add commit timestamp tracking but nothing more B) Add infrastructure to store commit timestamps and provide a facility for storing additional bits of data extensions might want to

Re: [HACKERS] tracking commit timestamps

2014-11-07 Thread Petr Jelinek
On 06/11/14 08:50, Andres Freund wrote: On 2014-11-05 19:31:52 -0500, Steve Singer wrote: It also doesn't allow you to load two extensions at once on a system. You wouldn't be able to have both the 'steve_commit_order' extension and BDR installed at the same time. I don't think this patch does

Re: [HACKERS] tracking commit timestamps

2014-11-06 Thread Craig Ringer
On 11/01/2014 09:00 PM, Michael Paquier wrote: > 1) It is untested and actually there is no direct use for it in core. > 2) Pushing code that we know as dead is no good, that's a feature more > or less defined as maybe-useful-but-we-are-not-sure-yet-what-to-do-with-it. > 3) If you're going to re-us

Re: [HACKERS] tracking commit timestamps

2014-11-05 Thread Andres Freund
On 2014-11-05 19:31:52 -0500, Steve Singer wrote: > On 11/05/2014 05:43 PM, Andres Freund wrote: > >On 2014-11-05 17:17:05 -0500, Steve Singer wrote: > >Imo that's essentially a different feature. What you essentially would > >need here is a 'commit sequence number' - but no timestamps. And > >prob

Re: [HACKERS] tracking commit timestamps

2014-11-05 Thread Steve Singer
On 11/05/2014 05:43 PM, Andres Freund wrote: On 2014-11-05 17:17:05 -0500, Steve Singer wrote: Imo that's essentially a different feature. What you essentially would need here is a 'commit sequence number' - but no timestamps. And probably to be useful that number has to be 8 bytes in itself. I

Re: [HACKERS] tracking commit timestamps

2014-11-05 Thread Andres Freund
On 2014-11-05 17:17:05 -0500, Steve Singer wrote: > It isn't just 'replication' systems that have a need for getting the commit > order of transactions on a single system. I have a application (not slony) > where we want to query a table but order the output based on the transaction > commit order

Re: [HACKERS] tracking commit timestamps

2014-11-05 Thread Steve Singer
On 11/05/2014 11:23 AM, Jim Nasby wrote: Except that commit time is not guaranteed unique *even on a single system*. That's my whole point. If we're going to bother with all the commit time machinery it seems really silly to provide a way to uniquely order every commit. Clearly trying to u

Re: [HACKERS] tracking commit timestamps

2014-11-05 Thread Kevin Grittner
Jim Nasby wrote: > for a single system AIUI all we need to do is expose the LSN of > each commit record and that will give you the exact and unique > order in which transactions committed. > > This isn't a hypothetical feature either; if we had this, > logical replication systems wouldn't have to

Re: [HACKERS] tracking commit timestamps

2014-11-05 Thread Andres Freund
On 2014-11-05 10:34:40 -0600, Jim Nasby wrote: > On 11/5/14, 10:30 AM, Andres Freund wrote: > >>Except that commit time is not guaranteed unique *even on a single > >>>system*. That's my whole point. If we're going to bother with all the > >>>commit time machinery it seems really silly to provide a

Re: [HACKERS] tracking commit timestamps

2014-11-05 Thread Andres Freund
On 2014-11-05 10:23:15 -0600, Jim Nasby wrote: > > > On 11/5/14, 6:10 AM, Michael Paquier wrote: > >In addition, I wonder if this feature would be misused. Record > >transaction ids to a table to find out commit order (use case could be > >storing historical row versions for example).

Re: [HACKERS] tracking commit timestamps

2014-11-05 Thread Jim Nasby
On 11/5/14, 10:30 AM, Andres Freund wrote: Except that commit time is not guaranteed unique *even on a single >system*. That's my whole point. If we're going to bother with all the >commit time machinery it seems really silly to provide a way to >uniquely order every commit. Well. I think that's

Re: [HACKERS] tracking commit timestamps

2014-11-05 Thread Jim Nasby
On 11/5/14, 6:10 AM, Michael Paquier wrote: In addition, I wonder if this feature would be misused. Record transaction ids to a table to find out commit order (use case could be storing historical row versions for example). Do a dump and restore on another cluster, and all the t

  1   2   >