Re: Fastpath while arranging the changes in LSN order in logical decoding

2020-03-06 Thread Dilip Kumar
On Sat, Mar 7, 2020 at 12:30 AM Andres Freund wrote: > > Hi, > > On 2020-01-08 18:06:52 +0530, Dilip Kumar wrote: > > On Wed, 8 Jan 2020 at 5:28 PM, Heikki Linnakangas wrote: > > > > > On 25/11/2019 05:52, Dilip Kumar wrote: > > > > In logical deco

Re: Fastpath while arranging the changes in LSN order in logical decoding

2020-03-06 Thread Dilip Kumar
On Sat, Mar 7, 2020 at 9:59 AM Dilip Kumar wrote: > > On Sat, Mar 7, 2020 at 12:30 AM Andres Freund wrote: > > > > Hi, > > > > On 2020-01-08 18:06:52 +0530, Dilip Kumar wrote: > > > On Wed, 8 Jan 2020 at 5:28 PM, Heikki Linnakangas wrote: > > >

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-06 Thread Dilip Kumar
On Fri, Mar 6, 2020 at 9:47 AM Amit Kapila wrote: > > On Thu, Mar 5, 2020 at 1:54 PM Dilip Kumar wrote: > > > > On Thu, Mar 5, 2020 at 12:15 PM Amit Kapila wrote: > > > > > > > > > 5. I have also tried to think of another way to check if we already

Re: Fastpath while arranging the changes in LSN order in logical decoding

2020-03-08 Thread Dilip Kumar
On Sun, Mar 8, 2020 at 9:24 PM James Coleman wrote: > > On Saturday, March 7, 2020, Dilip Kumar wrote: >> >> On Sat, Mar 7, 2020 at 9:59 AM Dilip Kumar wrote: >> > >> > On Sat, Mar 7, 2020 at 12:30 AM Andres Freund wrote: >> > > >> >

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-07 Thread Dilip Kumar
On Sat, Mar 7, 2020 at 3:26 PM Amit Kapila wrote: > > On Sat, Mar 7, 2020 at 11:14 AM Amit Kapila wrote: >> >> On Sat, Mar 7, 2020 at 9:57 AM Dilip Kumar wrote: >>> >>> On Fri, Mar 6, 2020 at 9:47 AM Amit Kapila wrote: >>> > >>

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-12 Thread Dilip Kumar
On Thu, Mar 12, 2020 at 5:28 PM Amit Kapila wrote: > > On Thu, Mar 12, 2020 at 11:15 AM Dilip Kumar wrote: > > > > I have fixed this in the attached patch set. > > > > I have modified your > v4-0003-Conflict-Extension-Page-lock-in-group-member patch. The &

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-12 Thread Dilip Kumar
On Fri, Mar 13, 2020 at 8:29 AM Amit Kapila wrote: > > On Thu, Mar 12, 2020 at 7:50 PM Kuntal Ghosh > wrote: > > > > On Thu, Mar 12, 2020 at 5:28 PM Amit Kapila wrote: > > > > > > On Thu, Mar 12, 2020 at 11:15 AM Dilip Kumar > > > wrote: &

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-12 Thread Dilip Kumar
On Fri, Mar 13, 2020 at 11:08 AM Amit Kapila wrote: > > On Thu, Mar 12, 2020 at 3:04 PM Dilip Kumar wrote: > > > > On Wed, Mar 11, 2020 at 2:36 PM Amit Kapila wrote: > > > > > > > > > If we have no other choice, then I see a few downsides of ad

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-15 Thread Dilip Kumar
On Sat, Mar 14, 2020 at 7:39 PM Amit Kapila wrote: > > On Fri, Mar 13, 2020 at 7:02 PM Dilip Kumar wrote: > > > > Apart from that, I have also extended the solution for the page lock. > > And, I have also broken down the 3rd patch in two parts for relation > > e

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-16 Thread Dilip Kumar
On Mon, Mar 16, 2020 at 11:56 AM Kuntal Ghosh wrote: > > On Mon, Mar 16, 2020 at 9:43 AM Dilip Kumar wrote: > > On Mon, Mar 16, 2020 at 8:57 AM Masahiko Sawada > > wrote: > > > IsRelationExtensionLockHeld and IsPageLockHeld are used only when > > > asser

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-04-13 Thread Dilip Kumar
On Mon, Apr 13, 2020 at 6:12 PM Kuntal Ghosh wrote: > > On Mon, Apr 13, 2020 at 5:20 PM Dilip Kumar wrote: > > On Mon, Apr 13, 2020 at 4:14 PM Kuntal Ghosh > > wrote: > > > > > > +#define SizeOfTransactionId (sizeof(TransactionId) + sizeof(char)) >

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-04-14 Thread Dilip Kumar
On Tue, Apr 14, 2020 at 9:14 PM Erik Rijkers wrote: > > On 2020-04-14 12:10, Dilip Kumar wrote: > > > v14-0001-Immediately-WAL-log-assignments.patch + > > v14-0002-Issue-individual-invalidations-with.patch + > > v14-0003-Extend-the-ou

Re: fixing old_snapshot_threshold's time->xid mapping

2020-04-18 Thread Dilip Kumar
stamp will become the headtimestamp. So in TransactionIdLimitedForOldSnapshots if (current_ts - old_snapshot_threshold) is still >= head_timestap then we can still do early pruning. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-04-18 Thread Dilip Kumar
(by the way: this build's regression tests 'ddl', 'toast', and > 'spill' fail) > > I seem now able to run all my test programs on these instances without > errors. > > Sorry, I seem to have raised a false alarm (although there was initially > certainly a problem). No problem, Thanks for confirming. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-04-14 Thread Dilip Kumar
On Tue, Apr 14, 2020 at 2:57 PM Amit Kapila wrote: > > On Mon, Apr 13, 2020 at 6:12 PM Kuntal Ghosh > wrote: > > > > On Mon, Apr 13, 2020 at 5:20 PM Dilip Kumar wrote: > > > On Mon, Apr 13, 2020 at 4:14 PM Kuntal Ghosh > > > wrote: > > >

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-04-14 Thread Dilip Kumar
On Tue, Apr 14, 2020 at 3:57 PM Amit Kapila wrote: > > On Tue, Apr 14, 2020 at 3:41 PM Dilip Kumar wrote: > > > > > > > > > > > @@ -281,6 +281,24 @@ DecodeXactOp(LogicalDecodingContext *ctx, > > > > XLogRecordBuffer *buf) > > >

Re: fixing old_snapshot_threshold's time->xid mapping

2020-04-20 Thread Dilip Kumar
On Mon, Apr 20, 2020 at 11:31 PM Robert Haas wrote: > > On Mon, Apr 20, 2020 at 12:10 AM Dilip Kumar wrote: > > I have started reviewing these patches. I think, the fixes looks right to > > me. > > > > + LWLockAcquire(OldSnapshotTimeMapLock, LW_SHARE

Re: fixing old_snapshot_threshold's time->xid mapping

2020-04-21 Thread Dilip Kumar
02:19:00Z'); +is(summarize_mapping(), "20|02:00:00|02:19:00"); But, I think we should try to extend it to test that we have put the new xid only in those slots where we suppose to and not in other slots?. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-04-21 Thread Dilip Kumar
[v14-0007-Track-statistics-for-streaming.patch] > > [v14-0008-Enable-streaming-for-all-subscription-TAP-tests.patch] > > [v14-0009-Add-TAP-test-for-streaming-vs.-DDL.patch] > > [v14-0010-Bugfix-handling-of-incomplete-toast-tuple.patch] > > [bugfix_in_schema_sent.patch] > > (

Re: fixing old_snapshot_threshold's time->xid mapping

2020-04-20 Thread Dilip Kumar
On Mon, Apr 20, 2020 at 12:29 PM Thomas Munro wrote: > > On Mon, Apr 20, 2020 at 6:35 PM Dilip Kumar wrote: > > On Mon, Apr 20, 2020 at 11:24 AM Thomas Munro > > wrote: > > > > > > On Sat, Apr 18, 2020 at 9:27 PM Dilip Kumar wrote: > > > &g

Re: fixing old_snapshot_threshold's time->xid mapping

2020-04-19 Thread Dilip Kumar
hotControl->head_offset; + mapping->head_timestamp = oldSnapshotControl->head_timestamp; + mapping->count_used = oldSnapshotControl->count_used; + for (int i = 0; i < OLD_SNAPSHOT_TIME_MAP_ENTRIES; ++i) + mapping->xid_by_minute[i] = oldSnapshotControl->xid_by_minute[i]; + LWLockRelease(OldSnapshotTimeMapLock); I think memcpy would be a better choice instead of looping it for all the entries, since we are doing this under a lock? -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: fixing old_snapshot_threshold's time->xid mapping

2020-04-20 Thread Dilip Kumar
On Mon, Apr 20, 2020 at 11:24 AM Thomas Munro wrote: > > On Sat, Apr 18, 2020 at 9:27 PM Dilip Kumar wrote: > > On Sat, Apr 18, 2020 at 11:47 AM Thomas Munro > > wrote: > > > I think I found another bug in MaintainOldSnapshotTimeMapping(): if > >

Re: Parallel copy

2020-04-09 Thread Dilip Kumar
o is generating the data while workers are busy inserting the data. So IMHO, if we have a specific leader process then there will always be work available for all the workers. I agree that we need to find the correct point when the leader will work as a worker. One idea could be that when the queue is full and there is no space to push more work to queue then the leader himself processes that work. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Problem with logical replication

2020-04-16 Thread Dilip Kumar
00877b42 in PostmasterMain (argc=3, argv=0x2079120) at postmaster.c:1400 #19 0x0077f256 in main (argc=3, argv=0x2079120) at main.c:210 To reproduce this issue run start1.sh then execute below commands on publishers. insert into pgbench_accounts values(1,2); update pgbench_accounts

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-04-16 Thread Dilip Kumar
On Tue, Apr 14, 2020 at 9:14 PM Erik Rijkers wrote: > > On 2020-04-14 12:10, Dilip Kumar wrote: > > > v14-0001-Immediately-WAL-log-assignments.patch + > > v14-0002-Issue-individual-invalidations-with.patch + > > v14-0003-Extend-the-ou

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-13 Thread Dilip Kumar
f the patch, instead of a counter, we have done with a flag. So I think now we can just keep a single variable and we can just reset the bit in a single instruction. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-13 Thread Dilip Kumar
On Fri, Mar 13, 2020 at 3:39 PM Amit Kapila wrote: > > On Fri, Mar 13, 2020 at 8:37 AM Dilip Kumar wrote: > > > > On Thu, Mar 12, 2020 at 5:28 PM Amit Kapila wrote: > > > > > > On Thu, Mar 12, 2020 at 11:15 AM Dilip Kumar > > > wrote: > >

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-13 Thread Dilip Kumar
On Fri, Mar 13, 2020 at 11:16 AM Dilip Kumar wrote: > > On Fri, Mar 13, 2020 at 11:08 AM Amit Kapila wrote: > > > > On Thu, Mar 12, 2020 at 3:04 PM Dilip Kumar wrote: > > > > > > On Wed, Mar 11, 2020 at 2:36 PM Amit Kapila > > > wrote: >

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-15 Thread Dilip Kumar
On Sun, Mar 15, 2020 at 1:15 PM Dilip Kumar wrote: > > On Sat, Mar 14, 2020 at 7:39 PM Amit Kapila wrote: > > > > On Fri, Mar 13, 2020 at 7:02 PM Dilip Kumar wrote: > > > > > > Apart from that, I have also extended the solution for the page lock. > > &g

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-15 Thread Dilip Kumar
On Sun, Mar 15, 2020 at 6:20 PM Amit Kapila wrote: > > On Sun, Mar 15, 2020 at 4:34 PM Dilip Kumar wrote: > > > > I have modified 0001 and 0002 slightly, Basically, instead of two > > function CheckAndSetLockHeld and CheckAndReSetLockHeld, I have cr

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-15 Thread Dilip Kumar
On Mon, Mar 16, 2020 at 8:57 AM Masahiko Sawada wrote: > > On Mon, 16 Mar 2020 at 00:54, Dilip Kumar wrote: > > > > On Sun, Mar 15, 2020 at 6:20 PM Amit Kapila wrote: > > > > > > On Sun, Mar 15, 2020 at 4:34 PM Dilip Kumar wrote: > > > >

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-15 Thread Dilip Kumar
On Mon, Mar 16, 2020 at 8:15 AM Amit Kapila wrote: > > On Sun, Mar 15, 2020 at 9:17 PM Dilip Kumar wrote: > > > > On Sun, Mar 15, 2020 at 5:58 PM Amit Kapila wrote: > > > > > > > > > 1. Group members wait for page locks. If you test that the leader

Re: WAL usage calculation patch

2020-04-01 Thread Dilip Kumar
ACUUM (PARALLEL 0) test; VACUUM postgres[106479]=# select query , wal_bytes, wal_records, wal_num_fpw from pg_stat_statements where query like 'VACUUM%'; query | wal_bytes | wal_records | wal_num_fpw --+---+-+----- VACUUM (PARALLEL 0)

Re: pg_stat_statements issue with parallel maintenance (Was Re: WAL usage calculation patch)

2020-03-31 Thread Dilip Kumar
On Wed, Apr 1, 2020 at 8:26 AM Masahiko Sawada wrote: > > On Wed, 1 Apr 2020 at 11:46, Amit Kapila wrote: > > > > On Tue, Mar 31, 2020 at 7:32 PM Dilip Kumar wrote: > > > > > > While testing I have found one issue. Basically, during a parallel >

Re: pg_stat_statements issue with parallel maintenance (Was Re: WAL usage calculation patch)

2020-03-31 Thread Dilip Kumar
On Wed, Apr 1, 2020 at 8:16 AM Amit Kapila wrote: > > On Tue, Mar 31, 2020 at 7:32 PM Dilip Kumar wrote: > > > > While testing I have found one issue. Basically, during a parallel > > vacuum, it was showing more number of > > shared_blk_hits+shared_blks_read. Afte

Re: Index Skip Scan

2020-04-05 Thread Dilip Kumar
On Wed, Mar 25, 2020 at 2:19 PM Dmitry Dolgov <9erthali...@gmail.com> wrote: > > > On Wed, Mar 25, 2020 at 11:31:56AM +0530, Dilip Kumar wrote: > > > > Seems like you forgot to add the uniquekey.c file in the > > v33-0001-Unique-key.patch. > > Oh, you're rig

Re: Index Skip Scan

2020-04-05 Thread Dilip Kumar
On Sun, Apr 5, 2020 at 9:39 PM Dmitry Dolgov <9erthali...@gmail.com> wrote: > > > On Sun, Apr 05, 2020 at 04:30:51PM +0530, Dilip Kumar wrote: > > > > I was just wondering how the distinct will work with the "skip scan" > > if we have some filter?

Re: Index Skip Scan

2020-04-06 Thread Dilip Kumar
On Mon, Apr 6, 2020 at 1:14 PM Floris Van Nee wrote: > > > > > On Sun, Apr 05, 2020 at 04:30:51PM +0530, Dilip Kumar wrote: > > > > > > > > I was just wondering how the distinct will work with the "skip scan" > > > > if we have some fil

Re: pg_stat_statements issue with parallel maintenance (Was Re: WAL usage calculation patch)

2020-04-01 Thread Dilip Kumar
On Wed, Apr 1, 2020 at 12:01 PM Amit Kapila wrote: > > On Wed, Apr 1, 2020 at 8:51 AM Dilip Kumar wrote: > > > > > Agreed. I've attached the updated patch. > > > > > > Thank you for testing, Dilip! > > > > Thanks! One hunk is failing on the lat

Re: WAL usage calculation patch

2020-04-02 Thread Dilip Kumar
On Fri, Apr 3, 2020 at 8:14 AM Amit Kapila wrote: > > On Fri, Apr 3, 2020 at 6:37 AM Amit Kapila wrote: > > > > On Thu, Apr 2, 2020 at 8:06 PM Dilip Kumar wrote: > > > > > > On Thu, Apr 2, 2020 at 6:41 PM Amit Kapila > > > wrote: > >

Re: WAL usage calculation patch

2020-04-02 Thread Dilip Kumar
On Thu, Apr 2, 2020 at 9:28 PM Dilip Kumar wrote: > > On Thu, Apr 2, 2020 at 6:41 PM Amit Kapila wrote: > > > > On Thu, Apr 2, 2020 at 6:18 PM Julien Rouhaud wrote: > > > > > > =# select query, calls, wal_bytes, wal_records, wal_num_fpw from > > >

Re: WAL usage calculation patch

2020-04-02 Thread Dilip Kumar
On Fri, Apr 3, 2020 at 9:02 AM Amit Kapila wrote: > > On Fri, Apr 3, 2020 at 8:55 AM Dilip Kumar wrote: > > > > I think now I got the reason. Basically, both of these records are > > storing the FPW, and FPW size can vary based on the hole size on the > >

Re: WAL usage calculation patch

2020-04-02 Thread Dilip Kumar
On Fri, Apr 3, 2020 at 9:17 AM Dilip Kumar wrote: > > On Fri, Apr 3, 2020 at 9:02 AM Amit Kapila wrote: > > > > On Fri, Apr 3, 2020 at 8:55 AM Dilip Kumar wrote: > > > > > > I think now I got the reason. Basically, both of these records are > > >

Re: User Interface for WAL usage data

2020-04-03 Thread Dilip Kumar
t; > I think this is more close to the case of Buffers where all fields are > > directly related to buffers/blocks. Here all the fields we want to > > display are related to WAL, so we should try to make it display > > similar to Buffers. > > > > Dilip, Julien, others, do you have any suggestions here? I think we > need to decide something now. We can change a few things like from > 'two spaces' to 'one space' between fields later as well. I also think it is more close to the BufferUsage so better to keep similar to that. If we think the parsing is the problem we can keep '_' in the multi-word name as shown below. WAL: records=n full_page_writes=n bytes=n And, all three fields are related to WAL so we can use WAL: followed by other fields as we are doing now in the current patch. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-03-28 Thread Dilip Kumar
On Sat, Mar 28, 2020 at 11:56 AM Amit Kapila wrote: > > On Wed, Mar 4, 2020 at 9:14 AM Dilip Kumar wrote: > > > > On Wed, Mar 4, 2020 at 3:16 AM Tomas Vondra > > wrote: > > > > > > > > > The first thing I realized that WAL-logging of assignme

Re: pg_stat_statements issue with parallel maintenance (Was Re: WAL usage calculation patch)

2020-03-31 Thread Dilip Kumar
laring i outside the for scope? > > We can do that but I was not sure if it's good since other codes > around there don't use that. So I'd like to leave it for committers. > It's a trivial change. I have reviewed the patch and the patch looks fine to me. One minor comment /+ /* Points to buffer usage are in DSM */ + BufferUsage *buffer_usage; + /buffer usage are in DSM / buffer usage area in DSM -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: WAL usage calculation patch

2020-03-31 Thread Dilip Kumar
nt? 4. Currently, we are combining all full-page write force/normal/consistency checks in one category. I am not sure whether it will be good information to know how many are force_fpw and how many are normal_fpw? -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: pg_stat_statements issue with parallel maintenance (Was Re: WAL usage calculation patch)

2020-03-31 Thread Dilip Kumar
On Tue, Mar 31, 2020 at 12:20 PM Dilip Kumar wrote: > > On Tue, Mar 31, 2020 at 10:44 AM Masahiko Sawada > wrote: > > > > On Tue, 31 Mar 2020 at 12:58, Amit Kapila wrote: > > > > > > On Mon, Mar 30, 2020 at 12:31 PM Masahiko Sawada > > > wrote

Re: pg_stat_statements issue with parallel maintenance (Was Re: WAL usage calculation patch)

2020-04-01 Thread Dilip Kumar
On Wed, Apr 1, 2020 at 8:51 AM Dilip Kumar wrote: > > On Wed, Apr 1, 2020 at 8:26 AM Masahiko Sawada > wrote: > > > > On Wed, 1 Apr 2020 at 11:46, Amit Kapila wrote: > > > > > > On Tue, Mar 31, 2020 at 7:32 PM Dilip Kumar wrote: > > > > >

Re: pg_stat_statements issue with parallel maintenance (Was Re: WAL usage calculation patch)

2020-04-01 Thread Dilip Kumar
erences, overall. > Can Dilip demonstrate the the "extra" buffer accesses are > proportionate to the number of workers launched in some constant, > predictable way? Okay, I will test this. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: pg_stat_statements issue with parallel maintenance (Was Re: WAL usage calculation patch)

2020-04-01 Thread Dilip Kumar
On Thu, Apr 2, 2020 at 9:13 AM Dilip Kumar wrote: > > On Thu, Apr 2, 2020 at 8:34 AM Peter Geoghegan wrote: > > > > On Wed, Apr 1, 2020 at 7:52 PM Amit Kapila wrote: > > > Peter, Is this behavior expected? > > > > > > Let me summarize the s

Re: WAL usage calculation patch

2020-04-02 Thread Dilip Kumar
> + values[i++] = UInt64GetDatum(tmp.wal_num_fpw); > > > > Why are they different? I think we should use the same *GetDatum API > > (probably Int64GetDatumFast) for these. > > > Oops, that's a mistake from when I was working on the wal_bytes output, fixed. > > > > v

Re: WAL usage calculation patch

2020-04-02 Thread Dilip Kumar
gt; > It is better to keep wal_bytes should be after wal_num_fpw as it is in > the main patch. Also, consider changing at other places in this > patch. I think we should add these new fields after blk_write_time or > at the end after usage. > > 4. > /* # of WAL full page image generated */ > Can we change it to "/* # of WAL full page image records generated */"? IMHO, "# of WAL full-page image records" seems like the number of wal record which contains the full-page image. But, actually, this is the total number of the full-page images, not the number of records that have a full-page image. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: WAL usage calculation patch

2020-04-02 Thread Dilip Kumar
0: rmgr: Heaplen (rec/tot): 54/ 7498, tx:489, lsn: 0/0167B9B0, prev 0/0167B970, desc: INSERT off 30 flags 0x01, blkref #0: rel 1663/13580/1249 t1_idx_parallel_1: rmgr: Heaplen (rec/tot): 54/ 8218, tx:494, lsn: 0/016B84F8, prev 0/016B84B8, desc: INSERT off

Re: Index Skip Scan

2020-03-25 Thread Dilip Kumar
c | 46 + src/include/nodes/nodes.h | 1 + src/include/nodes/pathnodes.h | 19 +++ src/include/nodes/print.h | 1 + src/include/optimizer/pathnode.h | 2 + src/include/optimizer/paths.h | 11 + 15 files changed, 272 insertions(+), 23 del

Re: Fastpath while arranging the changes in LSN order in logical decoding

2020-03-24 Thread Dilip Kumar
On Wed, Mar 25, 2020 at 9:23 AM Amit Kapila wrote: > > On Wed, Mar 25, 2020 at 12:46 AM Andres Freund wrote: > > > > On 2020-03-24 18:36:03 +0530, Dilip Kumar wrote: > > > IMHO, I have tried the best case but did not see any performance gain > > > so I am n

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-04-27 Thread Dilip Kumar
On Mon, Apr 27, 2020 at 4:13 PM Amit Kapila wrote: > > On Mon, Apr 27, 2020 at 4:05 PM Dilip Kumar wrote: > > > > I have also fixed a couple of bugs internally reported by my colleague > > Neha Sharma. > > > > I think it would be good if you can briefly expl

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-04-29 Thread Dilip Kumar
On Wed, Apr 29, 2020 at 12:37 PM Mahendra Singh Thalor wrote: > > On Wed, 29 Apr 2020 at 11:15, Mahendra Singh Thalor > wrote: > > > > On Fri, 24 Apr 2020 at 11:55, Dilip Kumar wrote: > > > > > > On Thu, Apr 23, 2020 at 2:28 PM Erik Rijkers wrote: >

Re: Trying to pull up EXPR SubLinks

2020-04-24 Thread Dilip Kumar
least) and the result > is correct, > then we think about how to cost it. The purpose of my writing is about the > first step > and see what people think about it. Ok > > As for how to cost it, I'm agreed with your suggestion, but we may need more > than that, like. (1, 2, 1) and (1, 1, 2) is same for your suggestion, but > they > are not different in this path. Valid point. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Trying to pull up EXPR SubLinks

2020-04-24 Thread Dilip Kumar
th); Can we just directly add the material path on top of the best path? I mean there are possibilities that we might not get any benefit of the material because there is no duplicate from the outer node but we are paying the cost of materialization right? The correct idea would be that we should select this based on the cost comparison. Basically, we can consider how many duplicates we have from the outer table variable no? -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-04-24 Thread Dilip Kumar
On Thu, Apr 23, 2020 at 2:28 PM Erik Rijkers wrote: > > On 2020-04-23 05:24, Dilip Kumar wrote: > > On Wed, Apr 22, 2020 at 9:31 PM Erik Rijkers wrote: > >> > >> The 'ddl' one is apparently not quite fixed - I get this in (cd > >> contrib; make check)'

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-04-22 Thread Dilip Kumar
On Wed, Apr 22, 2020 at 9:31 PM Erik Rijkers wrote: > > On 2020-04-22 16:49, Dilip Kumar wrote: > > On Tue, Apr 21, 2020 at 5:30 PM Dilip Kumar > > wrote: > >> > >> > > >> > (by the way: this build's regression tests 'ddl', 'toast', and &g

Re: fixing old_snapshot_threshold's time->xid mapping

2020-04-21 Thread Dilip Kumar
On Tue, Apr 21, 2020 at 4:52 PM Dilip Kumar wrote: > > On Tue, Apr 21, 2020 at 3:44 PM Thomas Munro wrote: > > > > On Tue, Apr 21, 2020 at 2:05 PM Thomas Munro wrote: > > > As before, these two apply on top of Robert's patches (or at least his > > >

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-05-04 Thread Dilip Kumar
On Mon, May 4, 2020 at 5:16 PM Amit Kapila wrote: > > On Fri, May 1, 2020 at 8:41 PM Dilip Kumar wrote: > > > 5. Shouldn't we add a check in table_scan_sample_next_block and > table_scan_sample_next_tuple APIs as well? I am not sure that we need to do that, Because generally

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-05-04 Thread Dilip Kumar
On Tue, May 5, 2020 at 10:25 AM Amit Kapila wrote: > > On Tue, May 5, 2020 at 9:27 AM Dilip Kumar wrote: > > > > On Mon, May 4, 2020 at 5:16 PM Amit Kapila wrote: > > > > > > On Fri, May 1, 2020 at 8:41 PM Dilip Kumar wrote: > >

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-04-28 Thread Dilip Kumar
On Tue, Apr 28, 2020 at 3:11 PM Amit Kapila wrote: > > On Mon, Apr 27, 2020 at 4:05 PM Dilip Kumar wrote: > > > [latest patches] > > v16-0004-Gracefully-handle-concurrent-aborts-of-uncommitt > - Any actions leading to transaction ID assignment are prohibit

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-04-29 Thread Dilip Kumar
On Tue, Apr 28, 2020 at 3:55 PM Dilip Kumar wrote: > > On Tue, Apr 28, 2020 at 3:11 PM Amit Kapila wrote: > > > > On Mon, Apr 27, 2020 at 4:05 PM Dilip Kumar wrote: > > > > > [latest patches] > > > > v16-0004-Gracefully-handle-concurrent-abor

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-04-29 Thread Dilip Kumar
On Wed, Apr 29, 2020 at 2:56 PM Dilip Kumar wrote: > > On Tue, Apr 28, 2020 at 3:55 PM Dilip Kumar wrote: > > > > On Tue, Apr 28, 2020 at 3:11 PM Amit Kapila wrote: > > > > > > On Mon, Apr 27, 2020 at 4:05 PM Dilip Kumar wrote: > > > > > &

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-04-13 Thread Dilip Kumar
On Mon, Apr 13, 2020 at 4:14 PM Kuntal Ghosh wrote: > > On Thu, Apr 9, 2020 at 2:40 PM Dilip Kumar wrote: > > > > I have rebased the patch on the latest head. I haven't yet changed > > anything for xid assignment thing because it is not yet concluded. > > &g

Re: refactoring basebackup.c

2020-05-12 Thread Dilip Kumar
ink? The overall idea looks quite nice. I had a look at some of the patches at least 0005 and 0006. At first look, I have one comment. +/* + * Each archive is set as a separate stream of COPY data, and thus begins + * with a CopyOutResponse message. + */ +static void +bbsink_libpq_begin_archive(bbsink *sink, const char *archive_name) +{ + SendCopyOutResponse(); +} Some of the bbsink_libpq_* functions are directly calling pq_* e.g. bbsink_libpq_begin_backup whereas others are calling SendCopy* functions and therein those are calling pq_* functions. I think bbsink_libpq_* function can directly call pq_* functions instead of adding one more level of the function call. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: refactoring basebackup.c

2020-05-12 Thread Dilip Kumar
On Wed, May 13, 2020 at 1:56 AM Robert Haas wrote: > > On Tue, May 12, 2020 at 4:32 AM Dilip Kumar wrote: > > Some of the bbsink_libpq_* functions are directly calling pq_* e.g. > > bbsink_libpq_begin_backup whereas others are calling SendCopy* > > functions and there

Re: Problem with logical replication

2020-05-12 Thread Dilip Kumar
gt;> build_replindex_scan_key needs to be updated. >> >> * This is not generic routine, it expects the idxrel to be replication >> * identity of a rel and meet all limitations associated with that. >> > It is implicit that a primary key can be a replica identity so I think this > comment is fine. I like your idea of modifying the assert instead of completely removing. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-05-13 Thread Dilip Kumar
On Tue, May 12, 2020 at 4:39 PM Amit Kapila wrote: > > On Thu, May 7, 2020 at 6:17 PM Dilip Kumar wrote: > > > > On Tue, May 5, 2020 at 7:13 PM Dilip Kumar wrote: > > > > I have fixed one more issue in 0010 patch. The issue was that once > > the transaction

Re: Index Skip Scan

2020-05-11 Thread Dilip Kumar
On Sun, May 10, 2020 at 11:17 PM Dmitry Dolgov <9erthali...@gmail.com> wrote: > > > On Sat, Apr 11, 2020 at 03:17:25PM +0530, Dilip Kumar wrote: > > > > Some more comments... > > Thanks for reviewing. Since this patch took much longer than I expected, >

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-05-15 Thread Dilip Kumar
On Fri, May 15, 2020 at 4:35 PM Amit Kapila wrote: > > On Fri, May 15, 2020 at 4:20 PM Dilip Kumar wrote: > > > > On Fri, May 15, 2020 at 4:04 PM Amit Kapila wrote: > > > > > > > > > > > > - In the example we can not show a real example,

Re: Parallel copy

2020-05-14 Thread Dilip Kumar
to worry that those triggers could do something on the primary table before we check the constraint. I am not sure if there are any other factors that I am missing. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Index Skip Scan

2020-05-15 Thread Dilip Kumar
On Fri, 15 May 2020 at 6:06 PM, Dmitry Dolgov <9erthali...@gmail.com> wrote: > > On Wed, May 13, 2020 at 02:37:21PM +0530, Dilip Kumar wrote: > > > > + if (_bt_scankey_within_page(scan, so->skipScanKey, so->currPos.buf, > dir)) > > + { > > > &g

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-05-19 Thread Dilip Kumar
ng is successful then > we can truncate the changes only up to completed changes LSN. What do > you think? > > I wonder why you have done this as 0010 in the patch series, it should > be as 0006 after the > 0005-Implement-streaming-mode-in-ReorderBuffer.patch. If we can do > that way then it would be easier for me to review. Is there a reason > for not doing so? No reason, I can do that. Actually, later we can merge the changes to 0005 only, I kept separate for review. Anyway, in the next version, I will make it as 0006. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-05-15 Thread Dilip Kumar
On Wed, May 13, 2020 at 4:50 PM Amit Kapila wrote: > > On Wed, May 13, 2020 at 11:35 AM Dilip Kumar wrote: > > > > On Tue, May 12, 2020 at 4:39 PM Amit Kapila wrote: > > > > > > > > > v20-0003-Ext

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-05-15 Thread Dilip Kumar
On Fri, May 15, 2020 at 4:04 PM Amit Kapila wrote: > > On Fri, May 15, 2020 at 2:47 PM Dilip Kumar wrote: > > > > > 6. I think it will be good if we can provide an example of streaming > > > changes via test_decoding at > > > https://www.postgresql.org/do

Re: Index Skip Scan

2020-05-13 Thread Dilip Kumar
On Mon, May 11, 2020 at 4:55 PM Dmitry Dolgov <9erthali...@gmail.com> wrote: > > > On Mon, May 11, 2020 at 04:04:00PM +0530, Dilip Kumar wrote: > > > > > > +static inline bool > > > > +_bt_scankey_within_page(IndexScanDesc scan, BTScanInsert

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-05-13 Thread Dilip Kumar
On Wed, May 13, 2020 at 4:50 PM Amit Kapila wrote: > > On Wed, May 13, 2020 at 11:35 AM Dilip Kumar wrote: > > > > On Tue, May 12, 2020 at 4:39 PM Amit Kapila wrote: > > > > > > > > > v20-0003-Ext

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-17 Thread Dilip Kumar
On Tue, Mar 17, 2020 at 5:14 PM Amit Kapila wrote: > > On Mon, Mar 16, 2020 at 3:24 PM Dilip Kumar wrote: > > > > + > + /* > + * Indicate that the lock is released for certain types of locks > + */ > +#ifdef USE_ASSERT_CHECKING > + CheckAndSetLock

Re: Fastpath while arranging the changes in LSN order in logical decoding

2020-03-24 Thread Dilip Kumar
On Tue, Mar 24, 2020 at 6:16 PM Amit Kapila wrote: > > On Mon, Mar 9, 2020 at 11:07 PM Andres Freund wrote: > > > > On 2020-03-07 11:15:27 +0530, Dilip Kumar wrote: > > > IMHO, if we conclude that because there is no performance gain so we > > > don't wa

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-07 Thread Dilip Kumar
On Sat, Mar 7, 2020 at 8:53 PM Tom Lane wrote: > > Dilip Kumar writes: > > I think instead of the flag we need to keep the counter because we can > > acquire the same relation extension lock multiple times. > > Uh ... what? How would that not be broken usage on its face

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-05-22 Thread Dilip Kumar
On Tue, May 19, 2020 at 4:33 PM Amit Kapila wrote: > > On Tue, May 19, 2020 at 3:31 PM Dilip Kumar wrote: > > > > On Tue, May 19, 2020 at 2:34 PM Amit Kapila wrote: > > > > > > On Mon, May 18, 2020 at 5:57 PM Amit Kapila &g

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-05-22 Thread Dilip Kumar
On Mon, May 18, 2020 at 4:10 PM Amit Kapila wrote: > > On Sun, May 17, 2020 at 12:41 PM Dilip Kumar wrote: > > > > On Fri, May 15, 2020 at 4:04 PM Amit Kapila wrote: > > > > > > > > > Review comments: > > >

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-05-22 Thread Dilip Kumar
On Tue, May 19, 2020 at 6:01 PM Amit Kapila wrote: > > On Fri, May 15, 2020 at 2:48 PM Dilip Kumar wrote: > > > > On Wed, May 13, 2020 at 4:50 PM Amit Kapila wrote: > > > > > > > > 3. > > > And, during catalog scan we can check the status of th

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-05-22 Thread Dilip Kumar
On Tue, May 19, 2020 at 5:34 PM Amit Kapila wrote: > > On Fri, May 15, 2020 at 2:47 PM Dilip Kumar wrote: > > > > On Tue, May 12, 2020 at 4:39 PM Amit Kapila wrote: > > > > > > > > 4. > > > +static void > > > +stream_

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-09-05 Thread Dilip Kumar
luded that in one of the existing tests in > > 018_stream_subxact_abort.pl. I have added one test for Rollback, > > changed few messages, removed one test case which was not making any > > sense in the patch. See attached and let me know what you think about > > it? I have reviewed the changes and looks fine to me. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-09-02 Thread Dilip Kumar
On Tue, Sep 1, 2020 at 8:33 PM Amit Kapila wrote: > > On Tue, Sep 1, 2020 at 9:28 AM Amit Kapila wrote: > > > > On Mon, Aug 31, 2020 at 7:28 PM Dilip Kumar wrote: > > > > > > On Mon, Aug 31, 2020 at 1:24 PM Amit Kapila > > > wrote

Re: Re: [HACKERS] Custom compression methods

2020-09-02 Thread Dilip Kumar
On Mon, Aug 31, 2020 at 10:45 AM Amit Khandekar wrote: > > On Thu, 13 Aug 2020 at 17:18, Dilip Kumar wrote: > > I have rebased the patch on the latest head and currently, broken into 3 > > parts. > > > > v1-0001: As suggested by Robert, it provides the

Re: [HACKERS] Custom compression methods

2020-09-02 Thread Dilip Kumar
On Wed, Sep 2, 2020 at 4:57 AM Mark Dilger wrote: > > > > > On Aug 13, 2020, at 4:48 AM, Dilip Kumar wrote: > > > > v1-0001: As suggested by Robert, it provides the syntax support for > > setting the compression method for a column while creating a table and

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-09-02 Thread Dilip Kumar
On Wed, Sep 2, 2020 at 7:19 PM Amit Kapila wrote: > On Wed, Sep 2, 2020 at 3:41 PM Dilip Kumar wrote: > > > > On Wed, Sep 2, 2020 at 10:55 AM Amit Kapila > wrote: > > > > > > > > > > > > > We can combine the tests in 015_stream_simpl

Re: Bug in logical decoding of in-progress transactions

2020-09-10 Thread Dilip Kumar
On Thu, Sep 10, 2020 at 2:48 PM Amit Kapila wrote: > On Thu, Sep 10, 2020 at 12:00 PM Dilip Kumar > wrote: > > > > On Thu, Sep 10, 2020 at 11:53 AM Dilip Kumar > wrote: > >> > >>> > > >>> > I have written a test case t

Re: Bug in logical decoding of in-progress transactions

2020-09-10 Thread Dilip Kumar
ction TXN 510 streaming change for TXN 510 closing a streamed block for transaction TXN 510 (5 rows) After patch data -- opening a streamed block for transaction TXN 510 streaming change for TXN 510 closing a streamed block for transaction TXN 510 --

Re: Bug in logical decoding of in-progress transactions

2020-09-10 Thread Dilip Kumar
On Thu, Sep 10, 2020 at 11:47 AM Amit Kapila wrote: > On Thu, Sep 10, 2020 at 11:42 AM Dilip Kumar > wrote: > > > > On Thu, Sep 10, 2020 at 11:29 AM Amit Kapila > wrote: > >> > >> Hi, > >> > >> There is a recent build farm failure [1]

Re: Bug in logical decoding of in-progress transactions

2020-09-10 Thread Dilip Kumar
On Thu, Sep 10, 2020 at 11:53 AM Dilip Kumar wrote: > On Thu, Sep 10, 2020 at 11:47 AM Amit Kapila > wrote: > >> On Thu, Sep 10, 2020 at 11:42 AM Dilip Kumar >> wrote: >> > >> > On Thu, Sep 10, 2020 at 11:29 AM Amit Kapila >> wrote: >> &g

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-09-07 Thread Dilip Kumar
On Mon, Sep 7, 2020 at 12:00 PM Amit Kapila wrote: > On Sat, Sep 5, 2020 at 8:55 PM Dilip Kumar wrote: > > > > > > I have reviewed the changes and looks fine to me. > > > > Thanks, I have pushed the last patch. Let's wait for a day or so to > see the buildf

Re: Allow ERROR from heap_prepare_freeze_tuple to be downgraded to WARNING

2020-09-15 Thread Dilip Kumar
On Tue, Sep 15, 2020 at 11:14 AM Andres Freund wrote: > > On 2020-09-15 10:54:29 +0530, Dilip Kumar wrote: > > What problem do you see if we set xmax to the InvalidTransactionId and > > HEAP_XMAX_INVALID flag in the infomask ? > > 1) It'll make a dead tuple ap

<    1   2   3   4   5   6   7   8   9   10   >