Re: [HACKERS] eXtensible Transaction Manager API (v2)

2016-03-19 Thread David Steele
On 3/16/16 7:59 AM, Stas Kelvich wrote: > On 12 Mar 2016, at 13:19, Michael Paquier wrote: >> >> On Fri, Mar 11, 2016 at 9:35 PM, Tom Lane wrote: >>> IMO this is not committable as-is, and I don't think that it's something >>> that will become committable during this 'fest. I think we'd be well

Re: [HACKERS] eXtensible Transaction Manager API (v2)

2016-03-19 Thread Stas Kelvich
On 12 Mar 2016, at 13:19, Michael Paquier wrote: > > On Fri, Mar 11, 2016 at 9:35 PM, Tom Lane wrote: >> IMO this is not committable as-is, and I don't think that it's something >> that will become committable during this 'fest. I think we'd be well >> advised to boot it to the 2016-09 CF and f

Re: [HACKERS] eXtensible Transaction Manager API (v2)

2016-03-14 Thread Kevin Grittner
On Sat, Mar 12, 2016 at 11:21 AM, Robert Haas wrote: > I'd also be interested in hearing Kevin Grittner's thoughts about > serializability in a distributed environment, since he's obviously > thought about the topic of serializability quite a bit. I haven't done a thorough search of the academic

Re: [HACKERS] eXtensible Transaction Manager API (v2)

2016-03-12 Thread Robert Haas
On Sat, Mar 12, 2016 at 11:06 AM, Tom Lane wrote: > Michael Paquier writes: >> On Fri, Mar 11, 2016 at 9:35 PM, Tom Lane wrote: >>> IMO this is not committable as-is, and I don't think that it's something >>> that will become committable during this 'fest. I think we'd be well >>> advised to bo

Re: [HACKERS] eXtensible Transaction Manager API (v2)

2016-03-12 Thread Tom Lane
Michael Paquier writes: > On Fri, Mar 11, 2016 at 9:35 PM, Tom Lane wrote: >> IMO this is not committable as-is, and I don't think that it's something >> that will become committable during this 'fest. I think we'd be well >> advised to boot it to the 2016-09 CF and focus our efforts on other st

Re: [HACKERS] eXtensible Transaction Manager API (v2)

2016-03-12 Thread Michael Paquier
On Fri, Mar 11, 2016 at 9:35 PM, Tom Lane wrote: > IMO this is not committable as-is, and I don't think that it's something > that will become committable during this 'fest. I think we'd be well > advised to boot it to the 2016-09 CF and focus our efforts on other stuff > that has a better chance

Re: [HACKERS] eXtensible Transaction Manager API (v2)

2016-03-11 Thread Konstantin Knizhnik
On 03/11/2016 11:35 PM, Tom Lane wrote: Robert Haas writes: On Fri, Mar 11, 2016 at 1:11 PM, David Steele wrote: Is anyone willing to volunteer a review or make an argument for the importance of this patch? There's been a lot of discussion on another thread about this patch. The subject is "

Re: [HACKERS] eXtensible Transaction Manager API (v2)

2016-03-11 Thread Tom Lane
Robert Haas writes: > On Fri, Mar 11, 2016 at 1:11 PM, David Steele wrote: >> Is anyone willing to volunteer a review or make an argument for the >> importance of this patch? > There's been a lot of discussion on another thread about this patch. > The subject is "The plan for FDW-based sharding"

Re: [HACKERS] eXtensible Transaction Manager API (v2)

2016-03-11 Thread David Steele
On 3/11/16 2:00 PM, Oleg Bartunov wrote: > On Fri, Mar 11, 2016 at 7:11 PM, David Steele I'm concerned about the lack of response or reviewers for this patch. > It may be because everyone believes they had their say on the original > thread, or because it seems like a big change to go

Re: [HACKERS] eXtensible Transaction Manager API (v2)

2016-03-11 Thread Oleg Bartunov
On Fri, Mar 11, 2016 at 7:11 PM, David Steele wrote: > On 2/10/16 12:50 PM, Konstantin Knizhnik wrote: > > > PostgresProffesional cluster teams wants to propose new version of > > eXtensible Transaction Manager API. > > Previous discussion concerning this patch can be found here: > > > > > http:/

Re: [HACKERS] eXtensible Transaction Manager API (v2)

2016-03-11 Thread David Steele
On 3/11/16 1:30 PM, Robert Haas wrote: > There's been a lot of discussion on another thread about this patch. > The subject is "The plan for FDW-based sharding", but the thread kind > of got partially hijacked by this issue. The net-net of that is that > I don't think we have a clear enough idea

Re: [HACKERS] eXtensible Transaction Manager API (v2)

2016-03-11 Thread Robert Haas
On Fri, Mar 11, 2016 at 1:11 PM, David Steele wrote: > On 2/10/16 12:50 PM, Konstantin Knizhnik wrote: >> PostgresProffesional cluster teams wants to propose new version of >> eXtensible Transaction Manager API. >> Previous discussion concerning this patch can be found here: >> >> http://www.postg

Re: [HACKERS] eXtensible Transaction Manager API (v2)

2016-03-11 Thread David Steele
On 2/10/16 12:50 PM, Konstantin Knizhnik wrote: > PostgresProffesional cluster teams wants to propose new version of > eXtensible Transaction Manager API. > Previous discussion concerning this patch can be found here: > > http://www.postgresql.org/message-id/f2766b97-555d-424f-b29f-e0ca0f6d1...@p

[HACKERS] eXtensible Transaction Manager API (v2)

2016-02-10 Thread Konstantin Knizhnik
Hi, PostgresProffesional cluster teams wants to propose new version of eXtensible Transaction Manager API. Previous discussion concerning this patch can be found here: http://www.postgresql.org/message-id/f2766b97-555d-424f-b29f-e0ca0f6d1...@postgrespro.ru The API patch itself is small enough

Re: [HACKERS] eXtensible Transaction Manager API

2015-12-03 Thread Craig Ringer
On 4 December 2015 at 06:41, Robert Haas wrote: > > I think there are ways to reduce the cost of this. Some distributed > systems have solved it by retreating from snapshot isolation and going > back to using locks. This can improve scalability if you've got lots > of transactions but they're v

Re: [HACKERS] eXtensible Transaction Manager API

2015-12-03 Thread Robert Haas
On Tue, Dec 1, 2015 at 9:19 AM, Bruce Momjian wrote: > On Tue, Nov 17, 2015 at 12:48:38PM -0500, Robert Haas wrote: >> > At this time, the number of round trips needed particularly for READ >> > COMMITTED transactions that need a new snapshot for each query was >> > really a performance killer. We

Re: [HACKERS] eXtensible Transaction Manager API

2015-12-01 Thread Bruce Momjian
On Tue, Nov 17, 2015 at 12:48:38PM -0500, Robert Haas wrote: > > At this time, the number of round trips needed particularly for READ > > COMMITTED transactions that need a new snapshot for each query was > > really a performance killer. We used DBT-1 (TPC-W) which is less > > OLTP-like than DBT-2

Re: [HACKERS] eXtensible Transaction Manager API

2015-11-17 Thread Robert Haas
On Fri, Nov 13, 2015 at 8:35 AM, Michael Paquier wrote: > As well as there could be FS, OS, network problems... To come back to > the point, my point is simply that I found surprising the sentence of > Konstantin upthread saying that if commit fails on some of the nodes > we should rollback the pr

Re: [HACKERS] eXtensible Transaction Manager API

2015-11-15 Thread Konstantin Knizhnik
I am sorry, looks like I had incorrectly interpret Michael's comment: Not always. If COMMIT PREPARED fails at some of the nodes but succeeds on others, the transaction is already partially acknowledged as committed in the cluster. Hence it makes more sense for the coordinator to commit transactio

Re: [HACKERS] eXtensible Transaction Manager API

2015-11-15 Thread Kevin Grittner
On Saturday, November 14, 2015 8:41 AM, Craig Ringer wrote: > On 13 November 2015 at 21:35, Michael Paquier > wrote: >> On Tue, Nov 10, 2015 at 3:46 AM, Robert Haas wrote: >>> If the database is corrupted, there's no way to guarantee that >>> anything works as planned. This is like saying th

Re: [HACKERS] eXtensible Transaction Manager API

2015-11-14 Thread Craig Ringer
On 13 November 2015 at 21:35, Michael Paquier wrote: > On Tue, Nov 10, 2015 at 3:46 AM, Robert Haas > wrote: > > On Sun, Nov 8, 2015 at 6:35 PM, Michael Paquier > > wrote: > >> Sure. Now imagine that the pg_twophase entry is corrupted for this > >> transaction on one node. This would trigger a

Re: [HACKERS] eXtensible Transaction Manager API

2015-11-13 Thread Michael Paquier
On Thu, Nov 12, 2015 at 11:23 PM, Robert Haas wrote: > On Mon, Nov 9, 2015 at 2:47 PM, Simon Riggs wrote: >> On 9 November 2015 at 18:46, Robert Haas wrote: >>> One point I'd like to mention is that it's absolutely critical to >>> design this in a way that minimizes network roundtrips without >>

Re: [HACKERS] eXtensible Transaction Manager API

2015-11-13 Thread Michael Paquier
On Tue, Nov 10, 2015 at 3:46 AM, Robert Haas wrote: > On Sun, Nov 8, 2015 at 6:35 PM, Michael Paquier > wrote: >> Sure. Now imagine that the pg_twophase entry is corrupted for this >> transaction on one node. This would trigger a PANIC on it, and >> transaction would not be committed everywhere.

Re: [HACKERS] eXtensible Transaction Manager API

2015-11-12 Thread Robert Haas
On Mon, Nov 9, 2015 at 2:47 PM, Simon Riggs wrote: > On 9 November 2015 at 18:46, Robert Haas wrote: >> > I am aware of the fact >> > that by definition PREPARE TRANSACTION ensures that a transaction will >> > be committed with COMMIT PREPARED, just trying to see any corner cases >> > with the ap

Re: [HACKERS] eXtensible Transaction Manager API

2015-11-09 Thread Amit Kapila
On Mon, Nov 9, 2015 at 2:08 PM, Konstantin Knizhnik < k.knizh...@postgrespro.ru> wrote: > > > On 09.11.2015 07:46, Amit Kapila wrote: > > I think so. Basically DLM should be responsible for maintaining > all the lock information which inturn means that any backend process > that needs to acquire/r

Re: [HACKERS] eXtensible Transaction Manager API

2015-11-09 Thread Simon Riggs
On 9 November 2015 at 18:46, Robert Haas wrote: > > I am aware of the fact > > that by definition PREPARE TRANSACTION ensures that a transaction will > > be committed with COMMIT PREPARED, just trying to see any corner cases > > with the approach proposed. The DTM approach is actually rather clo

Re: [HACKERS] eXtensible Transaction Manager API

2015-11-09 Thread Robert Haas
On Sun, Nov 8, 2015 at 6:35 PM, Michael Paquier wrote: > Sure. Now imagine that the pg_twophase entry is corrupted for this > transaction on one node. This would trigger a PANIC on it, and > transaction would not be committed everywhere. If the database is corrupted, there's no way to guarantee t

Re: [HACKERS] eXtensible Transaction Manager API

2015-11-09 Thread Simon Riggs
On 8 November 2015 at 23:35, Michael Paquier wrote: > > COMMIT PREPARED is a pretty thin layer; the work is all done in the > PREPARE. > > With a DTM, the main commit itself is done once only in the DTM, so all > the > > COMMIT PREPARED would do is release local node resources. > > Sure. Now ima

Re: [HACKERS] eXtensible Transaction Manager API

2015-11-09 Thread Konstantin Knizhnik
On 09.11.2015 07:46, Amit Kapila wrote: On Sat, Nov 7, 2015 at 10:23 PM, Konstantin Knizhnik mailto:k.knizh...@postgrespro.ru>> wrote: Lock manager is one of the tasks we are currently working on. There are still a lot of open questions: 1. Should distributed lock manager (DLM) d

Re: [HACKERS] eXtensible Transaction Manager API

2015-11-08 Thread Amit Kapila
On Sat, Nov 7, 2015 at 10:23 PM, Konstantin Knizhnik < k.knizh...@postgrespro.ru> wrote: > > Lock manager is one of the tasks we are currently working on. > There are still a lot of open questions: > 1. Should distributed lock manager (DLM) do something else except > detection of distributed deadl

Re: [HACKERS] eXtensible Transaction Manager API

2015-11-08 Thread Michael Paquier
On Mon, Nov 9, 2015 at 4:33 AM, Simon Riggs wrote: > On 8 November 2015 at 16:59, Konstantin Knizhnik > wrote: >> >> On 11/08/2015 02:46 PM, Michael Paquier wrote: >>> >>> On Sun, Nov 8, 2015 at 1:53 AM, Konstantin Knizhnik wrote: In tsDTM approach two phase commit is performed by coord

Re: [HACKERS] eXtensible Transaction Manager API

2015-11-08 Thread Simon Riggs
On 8 November 2015 at 16:59, Konstantin Knizhnik wrote: > On 11/08/2015 02:46 PM, Michael Paquier wrote: > >> On Sun, Nov 8, 2015 at 1:53 AM, Konstantin Knizhnik wrote: >> >>> In tsDTM approach two phase commit is performed by coordinator and >>> currently >>> is using standard PostgreSQL two pha

Re: [HACKERS] eXtensible Transaction Manager API

2015-11-08 Thread Konstantin Knizhnik
On 11/08/2015 02:46 PM, Michael Paquier wrote: On Sun, Nov 8, 2015 at 1:53 AM, Konstantin Knizhnik wrote: In tsDTM approach two phase commit is performed by coordinator and currently is using standard PostgreSQL two phase commit: Code in GO performing two phase commit: exec(conn1, "

Re: [HACKERS] eXtensible Transaction Manager API

2015-11-08 Thread Simon Riggs
On 7 November 2015 at 16:53, Konstantin Knizhnik wrote: > On 11/07/2015 05:11 PM, Amit Kapila wrote: > > > Today, while studying your proposal and related material, I noticed > that in both the approaches DTM and tsDTM, you are talking about > committing a transaction and acquiring the snapshot

Re: [HACKERS] eXtensible Transaction Manager API

2015-11-08 Thread Michael Paquier
On Sun, Nov 8, 2015 at 1:53 AM, Konstantin Knizhnik wrote: > In tsDTM approach two phase commit is performed by coordinator and currently > is using standard PostgreSQL two phase commit: > > Code in GO performing two phase commit: > > exec(conn1, "prepare transaction '" + gtid + "'") >

Re: [HACKERS] eXtensible Transaction Manager API

2015-11-07 Thread Konstantin Knizhnik
Hi, Thank you for your feedback. My comments are inside. On 11/07/2015 05:11 PM, Amit Kapila wrote: Today, while studying your proposal and related material, I noticed that in both the approaches DTM and tsDTM, you are talking about committing a transaction and acquiring the snapshot consistent

Re: [HACKERS] eXtensible Transaction Manager API

2015-11-07 Thread Amit Kapila
On Sat, Oct 31, 2015 at 2:52 PM, konstantin knizhnik < k.knizh...@postgrespro.ru> wrote: > Hi, > > PostgresPro cluster team wants to announce proposal for eXtensible > Transaction Manager API and reference implementation of distributed > transaction manager (pg_dtm). > pg_dtm is just a standard Po

Re: [HACKERS] eXtensible Transaction Manager API

2015-11-03 Thread Simon Riggs
On 2 November 2015 at 13:24, Konstantin Knizhnik wrote: > PostgreSQL assumes that top-level xid commit is atomic, along with all of > its subtransactions. So the API having access to only Get/Set at the xid > level would not work. We would need TransactionIdSetTreeStatus() rather > than just SetS

Re: [HACKERS] eXtensible Transaction Manager API

2015-11-02 Thread Konstantin Knizhnik
On 02.11.2015 12:01, Simon Riggs wrote: At first I was concerned about recovery, but that looks to be covered. Yes, we have not considered all possible scenarios of working with PostgreSQL. We have tested work with different isolation levels: repeatable read and read committed, but we hav

Re: [HACKERS] eXtensible Transaction Manager API

2015-11-02 Thread Konstantin Knizhnik
On 02.11.2015 06:17, Craig Ringer wrote: On 31 October 2015 at 17:22, konstantin knizhnik wrote: Waiting for your feedback For anyone wondering about performance impact, there are some graphs on page 23 of the PDF presentation. I didn't see anything else, and the graphs don't seem to cover co

Re: [HACKERS] eXtensible Transaction Manager API

2015-11-02 Thread Simon Riggs
On 31 October 2015 at 10:22, konstantin knizhnik wrote: > Hi, > > PostgresPro cluster team wants to announce proposal for eXtensible > Transaction Manager API and reference implementation of distributed > transaction manager (pg_dtm). > pg_dtm is just a standard PostgreSQL extension which should

Re: [HACKERS] eXtensible Transaction Manager API

2015-11-01 Thread Craig Ringer
On 31 October 2015 at 17:22, konstantin knizhnik wrote: > Waiting for your feedback For anyone wondering about performance impact, there are some graphs on page 23 of the PDF presentation. I didn't see anything else, and the graphs don't seem to cover comparison of Pg with the XTM transaction ma

[HACKERS] eXtensible Transaction Manager API

2015-10-31 Thread konstantin knizhnik
Hi, PostgresPro cluster team wants to announce proposal for eXtensible Transaction Manager API and reference implementation of distributed transaction manager (pg_dtm). pg_dtm is just a standard PostgreSQL extension which should be installed in normal way. Source of pg_dtm and PostgreSQL patc