Re: [HACKERS] [PATCHES] [PATCH] Provide 8-byte transaction IDs to user level
Marko Kreen [EMAIL PROTECTED] writes: Following patch exports 8 byte txid and snapshot to user level allowing its use in regular SQL. It is based on Slony-I xxid module. It provides special 'snapshot' type for snapshot but uses regular int8 for transaction ID's. Per discussion, I've applied a patch that just implements tracking of XID epoch in checkpoints. This should be sufficient to let xxid be handled as an external module. regards, tom lane ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] [PATCHES] [PATCH] Provide 8-byte transaction IDs to user level
On 7/27/06, Darcy Buskermolen [EMAIL PROTECTED] wrote: In one of those 3am lightbulbs I belive I have a way to make use of the 64-bit XID counter and still maintain the ability to have backwards compatibility. Is there any chance you could break this patch up into the 2 separate componenets that Hannu mentions, and rework the XID stuff into TransactionIdAdvance as per tom's recommendation. And in the meantime I'll pencil out the slony stuff to utilize this. Yes, I can. As I am on vacation right now, my computer-time is rather unstable, hopefully I can do it on weekend. -- marko ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] [PATCHES] [PATCH] Provide 8-byte transaction IDs to user level
On Wednesday 26 July 2006 14:27, Darcy Buskermolen wrote: On Wednesday 26 July 2006 14:03, Tom Lane wrote: Darcy Buskermolen [EMAIL PROTECTED] writes: The question though is if we did that, would Slony actually use it? If it made sence to do it, then yes we would do it. The problem ends up being Slony is designed to work across a multitude of versions of PG, and unless this was backported to at least 7.4, it would take a while (ie when we stopped supporting versions older than it was ported into) before we would make use of it. [ shrug... ] That's not happening; for one thing the change requires a layout change in pg_control and we have no mechanism to do that without initdb. I'll take a bit more of a look through the patch and see if it is a real boot to use it on those platforms that support it, and that we have a suitable way around it on those that don't. But at this point I wouldn't hold my breath on that In one of those 3am lightbulbs I belive I have a way to make use of the 64-bit XID counter and still maintain the ability to have backwards compatibility. Is there any chance you could break this patch up into the 2 separate componenets that Hannu mentions, and rework the XID stuff into TransactionIdAdvance as per tom's recommendation. And in the meantime I'll pencil out the slony stuff to utilize this. regards, tom lane -- Darcy Buskermolen CommandPrompt, Inc. Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240 http://www.commandprompt.com ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] [PATCHES] [PATCH] Provide 8-byte transaction IDs to user level
I am sure you worked hard on this, but I don't see the use case, nor have I heard people in the community requesting such functionality. Perhaps pgfoundry would be a better place for this. --- Marko Kreen wrote: Intro - Following patch exports 8 byte txid and snapshot to user level allowing its use in regular SQL. It is based on Slony-I xxid module. It provides special 'snapshot' type for snapshot but uses regular int8 for transaction ID's. Exported API Type: snapshot Functions: current_txid() returns int8 current_snapshot() returns snapshot snapshot_xmin(snapshot) returns int8 snapshot_xmax(snapshot) returns int8 snapshot_active_list(snapshot) returns setof int8 snapshot_contains(snapshot, int8) returns bool pg_sync_txid(int8) returns int8 Operation - Extension to 8-byte is done by keeping track of wraparound count in pg_control. On every checkpoint, nextxid is compared to one stored in pg_control. If value is smaller wraparound happened and epoch is inreased. When long txid or snapshot is requested, pg_control is locked with LW_SHARED for retrieving epoch value from it. The patch does not affect core functionality in any other way. Backup/restore of txid data --- Currently I made pg_dumpall output following statement: SELECT pg_sync_txid(%d), current_txid() then on target database, pg_sync_txid if it's current (epoch + GetTopTransactionId()) are larger than given argument. If not then it bumps epoch, until they are, thus guaranteeing that new issued txid's are larger then in source database. If restored into same database instance, nothing will happen. Advantages of 8-byte txids -- * Indexes won't break silently. No need for mandatory periodic truncate which may not happen for various reasons. * Allows to keep values from different databases in one table/index. * Ability to bring data into different server and continue there. Advantages in being in core --- * Core code can guarantee that wraparound check happens in 2G transactions. * Core code can update pg_control non-transactionally. Module needs to operate inside user transaction when updating epoch row, which bring various problems (READ COMMITTED vs. SERIALIZABLE, long transactions, locking, etc). * Core code has only one place where it needs to update, module needs to have epoch table in each database. Todo, tothink - * Flesh out the documentation. Probably needs some background. * Better names for some functions? * pg_sync_txid allows use of pg_dump for moveing database, but also adds possibility to shoot in the foot by allowing epoch wraparound to happen. Is Don't do it then enough? * Currently txid keeps its own copy of nextxid in pg_control, this makes clear data dependencies. Its possible to drop it and use -checkPointCopy-nextXid directly, thus saving 4 bytes. * Should the pg_sync_txid() issued by pg_dump instead pg_dumpall? -- marko [ Attachment, skipping... ] ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster -- Bruce Momjian [EMAIL PROTECTED] EnterpriseDBhttp://www.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] [PATCHES] [PATCH] Provide 8-byte transaction IDs to user level
Bruce Momjian [EMAIL PROTECTED] writes: I am sure you worked hard on this, but I don't see the use case, nor have I heard people in the community requesting such functionality. Perhaps pgfoundry would be a better place for this. The part of this that would actually be useful to put in core is maintaining a 64-bit XID counter, ie, keep an additional counter that bumps every time XID wraps around. This cannot be done very well from outside core but it would be nearly trivial, and nearly free, to add inside. Everything else in the patch could be done just as well as an extension datatype. (I wouldn't do it like this though --- TransactionIdAdvance itself is the place to bump the secondary counter.) The question though is if we did that, would Slony actually use it? regards, tom lane ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] [PATCHES] [PATCH] Provide 8-byte transaction IDs to user level
On Wednesday 26 July 2006 13:04, Tom Lane wrote: Bruce Momjian [EMAIL PROTECTED] writes: I am sure you worked hard on this, but I don't see the use case, nor have I heard people in the community requesting such functionality. Perhaps pgfoundry would be a better place for this. The part of this that would actually be useful to put in core is maintaining a 64-bit XID counter, ie, keep an additional counter that bumps every time XID wraps around. This cannot be done very well from outside core but it would be nearly trivial, and nearly free, to add inside. Everything else in the patch could be done just as well as an extension datatype. (I wouldn't do it like this though --- TransactionIdAdvance itself is the place to bump the secondary counter.) The question though is if we did that, would Slony actually use it? If it made sence to do it, then yes we would do it. The problem ends up being Slony is designed to work across a multitude of versions of PG, and unless this was backported to at least 7.4, it would take a while (ie when we stopped supporting versions older than it was ported into) before we would make use of it. regards, tom lane ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster -- Darcy Buskermolen CommandPrompt, Inc. Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240 http://www.commandprompt.com ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] [PATCHES] [PATCH] Provide 8-byte transaction IDs to user level
Darcy Buskermolen [EMAIL PROTECTED] writes: The question though is if we did that, would Slony actually use it? If it made sence to do it, then yes we would do it. The problem ends up being Slony is designed to work across a multitude of versions of PG, and unless this was backported to at least 7.4, it would take a while (ie when we stopped supporting versions older than it was ported into) before we would make use of it. [ shrug... ] That's not happening; for one thing the change requires a layout change in pg_control and we have no mechanism to do that without initdb. regards, tom lane ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] [PATCHES] [PATCH] Provide 8-byte transaction IDs to user level
On Wednesday 26 July 2006 14:03, Tom Lane wrote: Darcy Buskermolen [EMAIL PROTECTED] writes: The question though is if we did that, would Slony actually use it? If it made sence to do it, then yes we would do it. The problem ends up being Slony is designed to work across a multitude of versions of PG, and unless this was backported to at least 7.4, it would take a while (ie when we stopped supporting versions older than it was ported into) before we would make use of it. [ shrug... ] That's not happening; for one thing the change requires a layout change in pg_control and we have no mechanism to do that without initdb. I'll take a bit more of a look through the patch and see if it is a real boot to use it on those platforms that support it, and that we have a suitable way around it on those that don't. But at this point I wouldn't hold my breath on that regards, tom lane -- Darcy Buskermolen CommandPrompt, Inc. Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240 http://www.commandprompt.com ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] [PATCHES] [PATCH] Provide 8-byte transaction IDs to user level
Darcy Buskermolen wrote: On Wednesday 26 July 2006 14:03, Tom Lane wrote: Darcy Buskermolen [EMAIL PROTECTED] writes: The question though is if we did that, would Slony actually use it? If it made sence to do it, then yes we would do it. The problem ends up being Slony is designed to work across a multitude of versions of PG, and unless this was backported to at least 7.4, it would take a while (ie when we stopped supporting versions older than it was ported into) before we would make use of it. [ shrug... ] That's not happening; for one thing the change requires a layout change in pg_control and we have no mechanism to do that without initdb. I'll take a bit more of a look through the patch and see if it is a real boot to use it on those platforms that support it, and that we have a suitable way around it on those that don't. But at this point I wouldn't hold my breath on that The alternative seems to be that the Slony-I team doesn't feel they have a need for it, nobody else pushes hard enough for the feature to be in core, and thus Slony-I and all the rest stays broken forever. -- Alvaro Herrerahttp://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
