Re: POC and rebased patch for CSN based snapshots

2020-07-15 Thread Andrey V. Lepikhov
On 7/13/20 11:46 AM, movead...@highgo.ca wrote: I continue to see your patch. Some code improvements see at the attachment. Questions: * csnSnapshotActive is the only member of the CSNshapshotShared struct. * The WriteAssignCSNXlogRec() routine. I din't understand why you add 20 nanosec to

Re: POC and rebased patch for CSN based snapshots

2020-07-13 Thread Fujii Masao
On 2020/07/14 11:02, movead...@highgo.ca wrote: When checking each tuple visibility, we always have to get the CSN corresponding to XMIN or XMAX from CSN SLRU. In the past discussion, there was the suggestion that CSN should be stored in the tuple header or somewhere (like hint bit) to

Re: POC and rebased patch for CSN based snapshots

2020-07-13 Thread movead...@highgo.ca
>When checking each tuple visibility, we always have to get the CSN >corresponding to XMIN or XMAX from CSN SLRU. In the past discussion, >there was the suggestion that CSN should be stored in the tuple header >or somewhere (like hint bit) to avoid the overhead by very frequehntly >lookup for CSN

Re: POC and rebased patch for CSN based snapshots

2020-07-13 Thread Fujii Masao
On 2020/06/19 14:54, Fujii Masao wrote: On 2020/06/19 13:36, movead...@highgo.ca wrote:  >You mean that the last generated CSN needs to be WAL-logged because any smaller CSN than the last one should not be reused after crash recovery. Right? Yes that's it. If right, that WAL-logging

Re: POC and rebased patch for CSN based snapshots

2020-07-13 Thread movead...@highgo.ca
>I have doubts that I fully understood your question, but still. >What real problems do you see here? Transaction t1 doesn't get state of >shard2 until time at node with shard2 won't reach start time of t1. >If transaction, that inserted B wants to know about it position in time >relatively to t1

Re: POC and rebased patch for CSN based snapshots

2020-07-12 Thread Andrey V. Lepikhov
On 7/4/20 7:56 PM, movead...@highgo.ca wrote: As far as I know about Clock-SI, left part of the blue line will setup as a snapshot if master require a snapshot at time t1. But in fact data A should in snapshot but not and data B should out of snapshot but not. If

Re: POC and rebased patch for CSN based snapshots

2020-07-04 Thread movead...@highgo.ca
Hello Andrey >> I have researched your patch which is so great, in the patch only data >> out of 'global_snapshot_defer_time' can be vacuum, and it keep dead >> tuple even if no snapshot import at all,right? >> >> I am thanking about a way if we can start remain dead tuple just before >> we

Re: POC and rebased patch for CSN based snapshots

2020-07-02 Thread Andrey V. Lepikhov
On 7/2/20 7:31 PM, Movead Li wrote: Thanks for the remarks, >Some remarks on your patch: >1. The variable last_max_csn can be an atomic variable. Yes will consider. >2. GenerateCSN() routine: in the case than csn < csnState->last_max_csn >This is the case when someone changed the value of

Re: POC and rebased patch for CSN based snapshots

2020-07-02 Thread Movead Li
Thanks for the remarks, >Some remarks on your patch: >1. The variable last_max_csn can be an atomic variable. Yes will consider. >2. GenerateCSN() routine: in the case than csn < csnState->last_max_csn >This is the case when someone changed the value of the system clock. I >think it

Re: POC and rebased patch for CSN based snapshots

2020-06-29 Thread Andrey V. Lepikhov
On 6/12/20 2:41 PM, movead...@highgo.ca wrote: Hello hackers, Currently, I do some changes based on the last version: 1. Catch up to the current  commit (c2bd1fec32ab54). 2. Add regression and document. 3. Add support to switch from xid-base snapshot to csn-base snapshot, and the same with

Re: POC and rebased patch for CSN based snapshots

2020-06-18 Thread Fujii Masao
On 2020/06/19 13:36, movead...@highgo.ca wrote: >You mean that the last generated CSN needs to be WAL-logged because any smaller CSN than the last one should not be reused after crash recovery. Right? Yes that's it. If right, that WAL-logging seems not necessary because CSN mechanism

Re: POC and rebased patch for CSN based snapshots

2020-06-18 Thread movead...@highgo.ca
>You mean that the last generated CSN needs to be WAL-logged because any smaller >CSN than the last one should not be reused after crash recovery. Right? Yes that's it. >If right, that WAL-logging seems not necessary because CSN mechanism assumes >CSN is increased monotonically. IOW, even

Re: POC and rebased patch for CSN based snapshots

2020-06-18 Thread Fujii Masao
On 2020/06/19 12:12, movead...@highgo.ca wrote: Thanks for reply. >Probably it's not time to do the code review yet, but when I glanced the patch, I came up with one question. 0002 patch changes GenerateCSN() so that it generates CSN-related WAL records (and inserts it into WAL buffers).

Re: POC and rebased patch for CSN based snapshots

2020-06-18 Thread movead...@highgo.ca
Thanks for reply. >AFAIU, this patch is to improve scalability and also will be helpful >for Global Snapshots stuff, is that right? If so, how much >performance/scalability benefit this patch will have after Andres's >recent work on scalability [1]? The patch focus on to be an infrastructure of

Re: POC and rebased patch for CSN based snapshots

2020-06-18 Thread movead...@highgo.ca
Thanks for reply. >Probably it's not time to do the code review yet, but when I glanced the patch, >I came up with one question. >0002 patch changes GenerateCSN() so that it generates CSN-related WAL records >(and inserts it into WAL buffers). Which means that new WAL record is generated

Re: POC and rebased patch for CSN based snapshots

2020-06-18 Thread Fujii Masao
On 2020/06/15 16:48, Fujii Masao wrote: On 2020/06/12 18:41, movead...@highgo.ca wrote: Hello hackers, Currently, I do some changes based on the last version: 1. Catch up to the current  commit (c2bd1fec32ab54). 2. Add regression and document. 3. Add support to switch from xid-base

Re: POC and rebased patch for CSN based snapshots

2020-06-15 Thread Fujii Masao
On 2020/06/12 18:41, movead...@highgo.ca wrote: Hello hackers, Currently, I do some changes based on the last version: 1. Catch up to the current  commit (c2bd1fec32ab54). 2. Add regression and document. 3. Add support to switch from xid-base snapshot to csn-base snapshot, and the same with

Re: POC and rebased patch for CSN based snapshots

2020-06-13 Thread Amit Kapila
On Fri, Jun 12, 2020 at 3:11 PM movead...@highgo.ca wrote: > > Hello hackers, > > Currently, I do some changes based on the last version: > 1. Catch up to the current commit (c2bd1fec32ab54). > 2. Add regression and document. > 3. Add support to switch from xid-base snapshot to csn-base

Re: POC and rebased patch for CSN based snapshots

2020-06-12 Thread movead...@highgo.ca
Hello hackers, Currently, I do some changes based on the last version: 1. Catch up to the current commit (c2bd1fec32ab54). 2. Add regression and document. 3. Add support to switch from xid-base snapshot to csn-base snapshot, and the same with standby side. Regards, Highgo Software

POC and rebased patch for CSN based snapshots

2020-05-08 Thread Movead Li
Hello hackers, I have read the community mail from 'postgrespro' which the link below ①, a summary for the patch, it generals a CSN by timestamp when a transaction is committed and assigns a special value as CSN for abort transaction, and record them in CSN SLRU file. Now we can judge if a