Re: Allow ERROR from heap_prepare_freeze_tuple to be downgraded to WARNING

2020-09-14 Thread Dilip Kumar
can now mark tuple with HEAP_XMAX_INVALID without freezing the xmin but that is anyway a valid status for a tuple. However, if we think it still can cause some issues then I feel that we can skip the whole page as Robert suggested. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-09-14 Thread Dilip Kumar
I am not sure if CCI build > is impacting this in some way. Even I have analyzed this but did not find any reason why the streamed_txns list should be anything other than NULL. The only thing is we are initializing the entry->streamed_txns to NULL and the list free is checking "if (list == NIL)" then return. However IMHO, that should not be an issue becase NIL is defined as (List*) NULL. I am doing further testing and investigation. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-09-14 Thread Dilip Kumar
On Mon, Sep 14, 2020 at 4:50 PM Amit Kapila wrote: > > On Mon, Sep 14, 2020 at 1:23 PM Dilip Kumar wrote: > > > > On Mon, Sep 14, 2020 at 8:48 AM Amit Kapila wrote: > > > > > > > > > Yeah, this is right, and here is some initial analysi

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-09-09 Thread Dilip Kumar
typo in one of the comments > > Patch attached. > Looks good to me. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Custom compression methods

2020-10-06 Thread Dilip Kumar
On Tue, Oct 6, 2020 at 10:21 PM Tomas Vondra wrote: > > On Tue, Oct 06, 2020 at 11:00:55AM +0530, Dilip Kumar wrote: > >On Mon, Oct 5, 2020 at 9:34 PM Tomas Vondra > > wrote: > >> > >> On Mon, Oct 05, 2020 at 07:57:41PM +0530, Dilip Kumar wrote: > >>

Re: Logical replication CPU-bound with TRUNCATE/DROP/CREATE many tables

2020-10-08 Thread Dilip Kumar
On Fri, Oct 9, 2020 at 10:29 AM Amit Kapila wrote: > > On Fri, Oct 9, 2020 at 10:06 AM Dilip Kumar wrote: > > > > On Fri, Oct 9, 2020 at 8:40 AM Amit Kapila wrote: > > > > > > On Thu, Oct 8, 2020 at 2:34 PM Simon Riggs wrote: > > > > > &g

Re: Logical replication CPU-bound with TRUNCATE/DROP/CREATE many tables

2020-10-08 Thread Dilip Kumar
On Fri, Oct 9, 2020 at 8:40 AM Amit Kapila wrote: > > On Thu, Oct 8, 2020 at 2:34 PM Simon Riggs wrote: > > > > On Thu, 8 Oct 2020 at 09:47, Dilip Kumar wrote: > > > > > > This script will wait 10 seconds after INSERT exits > > > > befo

Re: Logical replication CPU-bound with TRUNCATE/DROP/CREATE many tables

2020-10-15 Thread Dilip Kumar
On Thu, Oct 15, 2020 at 12:38 PM Amit Kapila wrote: > > On Wed, Oct 14, 2020 at 4:51 PM Dilip Kumar wrote: > > > > On Wed, Oct 14, 2020 at 4:12 PM Amit Kapila wrote: > > > > > > > > > Thanks for the tests. The latest patch looks mostly good to me.

Re: [HACKERS] Custom compression methods

2020-10-05 Thread Dilip Kumar
On Mon, Oct 5, 2020 at 9:34 PM Tomas Vondra wrote: > > On Mon, Oct 05, 2020 at 07:57:41PM +0530, Dilip Kumar wrote: > >On Mon, Oct 5, 2020 at 5:53 PM Tomas Vondra > > wrote: > >> > >> On Mon, Oct 05, 2020 at 11:17:28AM +0530, Dilip Kumar wrote: > >>

Re: [HACKERS] Custom compression methods

2020-10-04 Thread Dilip Kumar
enough to store for the current compression method only. What's your thoughts? Other comments look fine to me so I will work on them and post the updated patch set. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Logical replication CPU-bound with TRUNCATE/DROP/CREATE many tables

2020-10-12 Thread Dilip Kumar
On Fri, Oct 9, 2020 at 2:34 PM Amit Kapila wrote: > > On Fri, Oct 9, 2020 at 2:19 PM Simon Riggs wrote: > > > > On Fri, 9 Oct 2020 at 04:10, Amit Kapila wrote: > > > > > > On Thu, Oct 8, 2020 at 2:34 PM Simon Riggs wrote: > > > > > &g

Re: [HACKERS] Custom compression methods

2020-10-12 Thread Dilip Kumar
On Mon, Oct 12, 2020 at 7:32 PM Tomas Vondra wrote: > > On Mon, Oct 12, 2020 at 02:28:43PM +0530, Dilip Kumar wrote: > > > >> ... > > > >I have worked on this patch, so as discussed now I am maintaining the > >preserved compression methods using de

Is Recovery actually paused?

2020-10-19 Thread Dilip Kumar
ually paused). Any opinion on this? -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Is Recovery actually paused?

2020-10-20 Thread Dilip Kumar
On Tue, Oct 20, 2020 at 1:22 PM Dilip Kumar wrote: > > On Tue, Oct 20, 2020 at 1:11 PM Simon Riggs wrote: > > > > On Mon, 19 Oct 2020 at 15:11, Dilip Kumar wrote: > > > > > We have an interface to pause the WAL replay (pg_wal_replay_pause) and > >

Re: Is Recovery actually paused?

2020-10-20 Thread Dilip Kumar
On Tue, Oct 20, 2020 at 1:11 PM Simon Riggs wrote: > > On Mon, 19 Oct 2020 at 15:11, Dilip Kumar wrote: > > > We have an interface to pause the WAL replay (pg_wal_replay_pause) and > > to know whether the WAL replay pause is requested > > (pg_is_wal_replay_paused).

Re: Is Recovery actually paused?

2020-10-20 Thread Dilip Kumar
On Tue, Oct 20, 2020 at 3:00 PM Simon Riggs wrote: > > On Tue, 20 Oct 2020 at 09:50, Dilip Kumar wrote: > > > > > Why would we want this? What problem are you trying to solve? > > > > > > The requirement is to know the last replayed WAL on the s

Re: [HACKERS] Custom compression methods

2020-10-09 Thread Dilip Kumar
On Fri, Oct 9, 2020 at 3:24 AM Tomas Vondra wrote: > > On Thu, Oct 08, 2020 at 02:38:27PM +0530, Dilip Kumar wrote: > >On Wed, Oct 7, 2020 at 5:00 PM Dilip Kumar wrote: > >> > >> On Wed, Oct 7, 2020 at 10:26 AM Dilip Kumar wrote: > >> > > >

Re: Parallel INSERT (INTO ... SELECT ...)

2020-10-05 Thread Dilip Kumar
sting approach as the CTAS patch except in > > the case of a specified RETURNING clause. > > > > I have one question which is common to both this patch and parallel > inserts in CTAS[1], do we need to skip creating tuple > queues(ExecParallelSetupTupleQueues) as we don

Re: Parallel INSERT (INTO ... SELECT ...)

2020-10-05 Thread Dilip Kumar
s a Returning clause. I think for the cases where we don't need > to return tuples we might want to skip creating these queues if it is > feasible without too many changes. +1 -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Logical replication CPU-bound with TRUNCATE/DROP/CREATE many tables

2020-10-08 Thread Dilip Kumar
RUNCATE: (no-flags) > 2020-10-02 14:59:42 COMMIT 509 (at 2020-10-02 14:55:50.349219+09) I could not see this issue even without the patch, it is taking less than 1s even without the patch. See below results 2020-10-08 13:00:49 BEGIN 509 2020-10-08 13:00:49 table nsp_001.part_0001: INSERT:... 2020-10-08 13:00:49 COMMIT 509 (at 2020-10-08 13:00:48.741986+05:30) Am I missing something? -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Custom compression methods

2020-10-05 Thread Dilip Kumar
On Mon, Oct 5, 2020 at 5:53 PM Tomas Vondra wrote: > > On Mon, Oct 05, 2020 at 11:17:28AM +0530, Dilip Kumar wrote: > >On Mon, Oct 5, 2020 at 3:37 AM Tomas Vondra > > wrote: > > > >Thanks, Tomas for your feedback. > > > >> 9) attcompression ... >

Re: Logical replication CPU-bound with TRUNCATE/DROP/CREATE many tables

2020-10-08 Thread Dilip Kumar
> > When TRUNCATE completes, > the walsender process should be at 100% CPU. Okay, thanks for the info, I will run again and see this. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Logical replication CPU-bound with TRUNCATE/DROP/CREATE many tables

2020-10-08 Thread Dilip Kumar
On Thu, Oct 8, 2020 at 2:17 PM Dilip Kumar wrote: > > On Thu, Oct 8, 2020 at 2:05 PM Keisuke Kuroda > wrote: > > > > Hi Dilip, > > > > > I could not see this issue even without the patch, it is taking less > > > than 1s even without the patch. See

Re: [HACKERS] Custom compression methods

2020-10-08 Thread Dilip Kumar
On Wed, Oct 7, 2020 at 5:00 PM Dilip Kumar wrote: > > On Wed, Oct 7, 2020 at 10:26 AM Dilip Kumar wrote: > > > > On Tue, Oct 6, 2020 at 10:21 PM Tomas Vondra > > wrote: > > > > > > On Tue, Oct 06, 2020 at 11:00:55AM +0530, Dilip Kumar wrote: > >

Re: Logical replication CPU-bound with TRUNCATE/DROP/CREATE many tables

2020-10-14 Thread Dilip Kumar
On Wed, Oct 14, 2020 at 4:12 PM Amit Kapila wrote: > > On Mon, Oct 12, 2020 at 3:23 PM Dilip Kumar wrote: > > > > On Fri, Oct 9, 2020 at 2:34 PM Amit Kapila wrote: > > > > > > > > > Okay, I think this makes sense. I think we should see the pe

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-08-25 Thread Dilip Kumar
On Tue, Aug 25, 2020 at 6:27 PM Amit Kapila wrote: > > On Tue, Aug 25, 2020 at 10:41 AM Dilip Kumar wrote: > > > > On Tue, Aug 25, 2020 at 9:31 AM Amit Kapila wrote: > > > > > > > > > I think the existing design is superior as it allows the

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-08-24 Thread Dilip Kumar
On Tue, Aug 25, 2020 at 9:31 AM Amit Kapila wrote: > > On Mon, Aug 24, 2020 at 9:41 PM Dilip Kumar wrote: > > > > On Sat, Aug 22, 2020 at 8:38 AM Amit Kapila wrote: > > > > > > On Fri, Aug 21, 2020 at 5:10 PM Dilip Kumar wrote: > > > &

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-08-19 Thread Dilip Kumar
On Wed, Aug 19, 2020 at 12:20 PM Amit Kapila wrote: > > On Wed, Aug 19, 2020 at 10:10 AM Amit Kapila wrote: > > > > On Mon, Aug 17, 2020 at 6:29 PM Dilip Kumar wrote: > > > > > > > > > In last patch v49-0001, there is one issue, Basically, I ha

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-08-19 Thread Dilip Kumar
On Wed, Aug 19, 2020 at 10:11 AM Amit Kapila wrote: > > On Mon, Aug 17, 2020 at 6:29 PM Dilip Kumar wrote: > > > > > > In last patch v49-0001, there is one issue, Basically, I have called > > BufFileFlush in all the cases. But, ideally, we can not call th

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-08-28 Thread Dilip Kumar
PG_USED_FOR_ASSERTS_ONLY = false; > > > > Thanks for the report. Tom Lane has already fixed this [1]. > > [1] - > https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=e942af7b8261cd8070d0eeaf518dbc1a664859fd As discussed, I have added a another test case fo

Re: Allow ERROR from heap_prepare_freeze_tuple to be downgraded to WARNING

2020-08-29 Thread Dilip Kumar
have frozen the second tuple. But, I feel this is easily fixable right? I mean instead of not doing anything to the corrupted tuple we can partially freeze it? I mean we can just leave the corrupted xid alone but mark the other xid as frozen if that is smaller then cutoff_xid. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Allow ERROR from heap_prepare_freeze_tuple to be downgraded to WARNING

2020-08-29 Thread Dilip Kumar
On Fri, Aug 28, 2020 at 9:49 PM Robert Haas wrote: > > On Tue, Jul 21, 2020 at 9:21 AM Dilip Kumar wrote: > > In the previous version, the feature was enabled for cluster/vacuum > > full command as well. in the attached patch I have enabled it only > > if we ar

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-08-21 Thread Dilip Kumar
On Fri, Aug 21, 2020 at 3:13 PM Amit Kapila wrote: > > On Fri, Aug 21, 2020 at 10:33 AM Dilip Kumar wrote: > > > > On Fri, Aug 21, 2020 at 9:14 AM Amit Kapila wrote: > > > > > > 2. > > > + /* > > > + * If the new location is smaller th

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-08-20 Thread Dilip Kumar
On Fri, Aug 21, 2020 at 9:14 AM Amit Kapila wrote: > > On Thu, Aug 20, 2020 at 5:42 PM Dilip Kumar wrote: > > > > On Thu, Aug 20, 2020 at 2:30 PM Amit Kapila wrote: > > > > > > > > > Right, I think this can happen if one has changed those by BufFi

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-08-20 Thread Dilip Kumar
On Fri, Aug 21, 2020 at 10:20 AM Amit Kapila wrote: > > On Fri, Aug 21, 2020 at 9:14 AM Amit Kapila wrote: > > > > On Thu, Aug 20, 2020 at 5:42 PM Dilip Kumar wrote: > > > > > > On Thu, Aug 20, 2020 at 2:30 PM Amit Kapila > > > wrote: > >

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-08-20 Thread Dilip Kumar
On Wed, Aug 19, 2020 at 1:35 PM Dilip Kumar wrote: > > On Wed, Aug 19, 2020 at 12:20 PM Amit Kapila wrote: > > > > On Wed, Aug 19, 2020 at 10:10 AM Amit Kapila > > wrote: > > > > > > On Mon, Aug 17, 2020 at 6:29 PM Dilip Kumar wrote: >

Re: Re: [HACKERS] Custom compression methods

2020-08-24 Thread Dilip Kumar
On Thu, Aug 13, 2020 at 5:18 PM Dilip Kumar wrote: > There was some question which Robert has asked in this mail, please find my answer inline. Also, I have a few questions regarding further splitting up this patch. > On Fri, Jun 19, 2020 at 10:33 PM Robert Haas wrote: > > &

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-08-24 Thread Dilip Kumar
On Sat, Aug 22, 2020 at 8:38 AM Amit Kapila wrote: > > On Fri, Aug 21, 2020 at 5:10 PM Dilip Kumar wrote: > > > > I have reviewed and tested the patch and the changes look fine to me. > > > > Thanks, I will push the next patch early next week (by Tuesday) unless

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-08-15 Thread Dilip Kumar
On Thu, Aug 13, 2020 at 6:47 PM Amit Kapila wrote: > > On Thu, Aug 13, 2020 at 12:08 PM Amit Kapila wrote: > > > > On Fri, Aug 7, 2020 at 2:04 PM Dilip Kumar wrote: > > > > > > On Thu, Aug 6, 2020 at 2:43 PM Amit Kapila > > > wrote: > >

Re: Parallel bitmap index scan

2020-08-17 Thread Dilip Kumar
On Mon, 17 Aug 2020 at 7:42 PM, Bharath Rupireddy < bharath.rupireddyforpostg...@gmail.com> wrote: > On Sun, Jul 26, 2020 at 6:43 PM Dilip Kumar wrote: > > > > > > I would like to propose a patch for enabling the parallelism for the > > > bitmap ind

Re: Logical replication from PG v13 and below to PG v14 (devel version) is not working.

2020-09-24 Thread Dilip Kumar
On Thu, Sep 24, 2020 at 5:31 PM Amit Kapila wrote: > > On Thu, Sep 24, 2020 at 5:11 PM Dilip Kumar wrote: > > > > On Thu, Sep 24, 2020 at 4:45 PM Amit Kapila wrote: > > > > > > > > > Have you checked what will function walrcv_server_version() will

Re: Parallel INSERT (INTO ... SELECT ...)

2020-09-29 Thread Dilip Kumar
hink you still need to work on the costing part, basically if we are parallelizing whole insert then plan is like below -> Gather -> Parallel Insert -> Parallel Seq Scan That means the tuple we are selecting via scan are not sent back to the gather node, so in cost_gather we need to see if it is for the INSERT then there is no row transferred through the parallel queue that mean we need not to pay any parallel tuple cost. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] logical decoding of two-phase transactions

2020-09-29 Thread Dilip Kumar
i < TwoPhaseState->numPrepXacts; i++) + { + GlobalTransaction gxact = TwoPhaseState->prepXacts[i]; I think we should take LW_SHARED lowck here no? -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Parallel INSERT (INTO ... SELECT ...)

2020-09-29 Thread Dilip Kumar
On Tue, Sep 29, 2020 at 8:27 PM Dilip Kumar wrote: > > On Mon, Sep 28, 2020 at 8:45 AM Greg Nancarrow wrote: > > > > On Sat, Sep 26, 2020 at 3:30 PM Bharath Rupireddy > > wrote: > > > > > I further checked on full txn id and command id. Yes, these a

Re: Parallel INSERT (INTO ... SELECT ...)

2020-09-29 Thread Dilip Kumar
s far as cost_gather() method is concerned, for Parallel INSERT, it > can probably use the same costing approach as the CTAS patch except in > the case of a specified RETURNING clause. Yeah right. I did not think about the returning part. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Resetting spilled txn statistics in pg_stat_replication

2020-09-30 Thread Dilip Kumar
isplay physical slots in this view as all the stats are for logical > slots but anyway we can add stats w.r.t physical slots in the future. > I am fine either way (don't show physical slots in this view or show > them but keep stats as 0). Let me know if you have any thoughts on > these points, other than that I am happy with the current state of the > patch. IMHO, It will make more sense to only show the logical replication slots in this view. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Logical replication CPU-bound with TRUNCATE/DROP/CREATE many tables

2020-09-30 Thread Dilip Kumar
idation in form of changes. I have rebased on the current head so that we collect both in form of changes as well as collection of the invalidation. So if we get individiaual invalidation we execute them and on abort we execute all the invalidation together. -- Regards, Dilip Kumar E

Re: [HACKERS] logical decoding of two-phase transactions

2020-09-30 Thread Dilip Kumar
On Wed, Sep 30, 2020 at 2:36 PM Amit Kapila wrote: > > On Tue, Sep 29, 2020 at 8:04 PM Dilip Kumar wrote: > > > > I have started looking into you latest patches, as of now I have a > > few comments. > > > > v6-0001 > > > > @@ -1987,7 +207

Re: [HACKERS] logical decoding of two-phase transactions

2020-09-30 Thread Dilip Kumar
On Wed, Sep 30, 2020 at 3:27 PM Amit Kapila wrote: > > On Wed, Sep 30, 2020 at 3:12 PM Dilip Kumar wrote: > > > > On Wed, Sep 30, 2020 at 3:08 PM Amit Kapila wrote: > > > > > > On Wed, Sep 30, 2020 at 2:46 PM Dilip Kumar wrote: > > > > >

Re: [HACKERS] logical decoding of two-phase transactions

2020-09-30 Thread Dilip Kumar
On Wed, Sep 30, 2020 at 3:08 PM Amit Kapila wrote: > > On Wed, Sep 30, 2020 at 2:46 PM Dilip Kumar wrote: > > > > On Wed, Sep 30, 2020 at 2:36 PM Amit Kapila wrote: > > > > > > On Tue, Sep 29, 2020 at 8:04 PM Dilip Kumar wrote: > > > > > &

Re: Resetting spilled txn statistics in pg_stat_replication

2020-09-30 Thread Dilip Kumar
On Wed, Sep 30, 2020 at 2:40 PM Amit Kapila wrote: > > On Wed, Sep 30, 2020 at 1:12 PM Dilip Kumar wrote: > > > > On Fri, Sep 25, 2020 at 4:33 PM Amit Kapila wrote: > > > > > > On Thu, Sep 24, 2020 at 5:44 PM Amit Kapila > > > wrote: > >

Re: Logical replication from PG v13 and below to PG v14 (devel version) is not working.

2020-09-22 Thread Dilip Kumar
Dilip but I think we can declare them in define number order like > below: > #define LOGICALREP_PROTO_MIN_VERSION_NUM 1 > #define LOGICALREP_PROTO_VERSION_NUM 1 > #define LOGICALREP_PROTO_STREAM_VERSION_NUM 2 > #define LOGICALREP_PROTO_MAX_VERSION_NUM LOGICALREP_PROTO_STREAM_VERSION_NUM Done this way. > Anothe

Re: Logical replication from PG v13 and below to PG v14 (devel version) is not working.

2020-09-24 Thread Dilip Kumar
On Thu, Sep 24, 2020 at 4:45 PM Amit Kapila wrote: > > On Thu, Sep 24, 2020 at 4:31 PM Dilip Kumar wrote: > > > > On Thu, Sep 24, 2020 at 11:55 AM Amit Kapila > > wrote: > > > > > > On Tue, Sep 22, 2020 at 5:15 PM Dilip Kumar wrote: > > > >

Re: Logical replication from PG v13 and below to PG v14 (devel version) is not working.

2020-09-24 Thread Dilip Kumar
On Thu, Sep 24, 2020 at 11:55 AM Amit Kapila wrote: > > On Tue, Sep 22, 2020 at 5:15 PM Dilip Kumar wrote: > > > > On Tue, Sep 22, 2020 at 12:02 PM Amit Kapila > > wrote: > > > > > > > > > I am not sure if this suggestion makes it better than

Re: Logical replication from PG v13 and below to PG v14 (devel version) is not working.

2020-09-25 Thread Dilip Kumar
ogical.streaming && > PQserverVersion(conn->streamConn) >= 14) > appendStringInfo(, ", streaming 'on'"); > .. > } Ok, I have modified as per your suggestion. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com v4-0001-Bugfix-in-logical-protocol-version.patch Description: Binary data

Re: Logical replication CPU-bound with TRUNCATE/DROP/CREATE many tables

2020-09-30 Thread Dilip Kumar
s and then the > rollback happens and we need to process all the invalidations > together. Basically, this is to cover the new code, if such a test > already exists then it is fine. I think we already have such a test case. 019_stream_subxact_ddl_abort.pl is covering this scenario. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Logical replication CPU-bound with TRUNCATE/DROP/CREATE many tables

2020-09-30 Thread Dilip Kumar
On Wed, Sep 30, 2020 at 3:00 PM Amit Kapila wrote: > > On Wed, Sep 30, 2020 at 12:16 PM Dilip Kumar wrote: > > > > On Mon, Sep 28, 2020 at 2:22 PM Amit Kapila wrote: > > > > > > > > > I don't have a patch for this idea but you can refer [1] &

Re: Logical replication CPU-bound with TRUNCATE/DROP/CREATE many tables

2020-10-01 Thread Dilip Kumar
On Thu, Oct 1, 2020 at 4:44 PM Amit Kapila wrote: > > On Thu, Oct 1, 2020 at 10:12 AM Dilip Kumar wrote: > > > > On Wed, Sep 30, 2020 at 3:00 PM Amit Kapila wrote: > > > > > > On Wed, Sep 30, 2020 at 12:16 PM Dilip Kumar > > > wrote: > > &

Re: Logical replication CPU-bound with TRUNCATE/DROP/CREATE many tables

2020-10-02 Thread Dilip Kumar
s for confirming I will test. > Also, the argument of palloc was still MemoryContextAlloc, > which prevented me from applying the patch, so I've only fixed that part. Oh my bad, I think I forgot to amend that part in the patch after compiling. Thanks for fixing this. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-05-28 Thread Dilip Kumar
On Tue, May 26, 2020 at 12:00 PM Amit Kapila wrote: > > On Fri, May 22, 2020 at 6:22 PM Dilip Kumar wrote: > > > > On Mon, May 18, 2020 at 4:10 PM Amit Kapila wrote: > > > > > > On Sun, May 17, 2020 at 12:41 PM Dilip Kumar > > > wrote: > &g

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-05-28 Thread Dilip Kumar
On Tue, May 26, 2020 at 3:04 PM Amit Kapila wrote: > > On Mon, May 25, 2020 at 8:07 PM Dilip Kumar wrote: > > > > On Fri, May 22, 2020 at 11:54 AM Amit Kapila > > wrote: > > > > > > 4. > > > + * XXX Do we need to allocate it

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-05-28 Thread Dilip Kumar
On Thu, May 28, 2020 at 3:15 PM Amit Kapila wrote: > > On Thu, May 28, 2020 at 12:57 PM Dilip Kumar wrote: > > > > On Tue, May 26, 2020 at 3:04 PM Amit Kapila wrote: > > > > > > Why we need memory-related to subxacts till the worker is alive? As > >

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-05-28 Thread Dilip Kumar
On Thu, May 28, 2020 at 2:41 PM Amit Kapila wrote: > > On Thu, May 28, 2020 at 12:46 PM Dilip Kumar wrote: > > > > On Tue, May 26, 2020 at 12:00 PM Amit Kapila > > wrote: > > > > > > Isn't this problem only for subxact file as we anyway create changes

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-05-25 Thread Dilip Kumar
errmsg("could not truncate file \"%s\": %m", path))); > .. > } > > Will truncate works on Windows? I see in the code we ftruncate which > is defined as chsize in win32.h and win32_port.h. I have not tested > this so I am not very sure about this. I got a below warning when I > tried to compile this code on Windows. I think it is better to > ftruncate as it is used at other places in the code as well. > > worker.c(798): warning C4013: 'truncate' undefined; assuming extern > returning int I have changed to the ftruncate. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-05-26 Thread Dilip Kumar
On Tue, May 26, 2020 at 10:27 AM Amit Kapila wrote: > > On Fri, May 22, 2020 at 6:21 PM Dilip Kumar wrote: > > > > On Mon, May 18, 2020 at 5:57 PM Amit Kapila wrote: > > > > > > > > > Few comments on v20-0010-Bugf

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-05-29 Thread Dilip Kumar
On Thu, May 28, 2020 at 2:41 PM Amit Kapila wrote: > > On Thu, May 28, 2020 at 12:46 PM Dilip Kumar wrote: > > > > On Tue, May 26, 2020 at 12:00 PM Amit Kapila > > wrote: > > > > > > Isn't this problem only for subxact file as we anyway create changes

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-05-29 Thread Dilip Kumar
On Tue, May 26, 2020 at 3:04 PM Amit Kapila wrote: > > On Mon, May 25, 2020 at 8:07 PM Dilip Kumar wrote: > > > > On Fri, May 22, 2020 at 11:54 AM Amit Kapila > > wrote: > > > > > > 4. > > > + * XXX Do we need to allocate it

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-05-29 Thread Dilip Kumar
On Tue, May 26, 2020 at 4:46 PM Amit Kapila wrote: > > On Tue, May 26, 2020 at 2:44 PM Dilip Kumar wrote: > > > > On Tue, May 26, 2020 at 10:27 AM Amit Kapila > > wrote: > > > > > > > > > > > 2. There is a bug fix in handl

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-06-02 Thread Dilip Kumar
On Thu, May 28, 2020 at 5:22 PM Amit Kapila wrote: > > On Wed, May 27, 2020 at 8:22 PM Dilip Kumar wrote: > > > > On Tue, May 26, 2020 at 7:45 AM Dilip Kumar wrote: > > > > > > > > > Okay, sending again. > > > > While reviewing/testi

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-06-02 Thread Dilip Kumar
On Tue, Jun 2, 2020 at 4:56 PM Amit Kapila wrote: > > On Tue, Jun 2, 2020 at 3:59 PM Dilip Kumar wrote: > > > > On Thu, May 28, 2020 at 5:22 PM Amit Kapila wrote: > > > > > > > > > Also, what if the changes file size overflows "OS file size lim

Re: [HACKERS] logical decoding of two-phase transactions

2020-09-19 Thread Dilip Kumar
XN(rb, txn, snapshot_now, + command_id, prev_lsn, + specinsert); + } + else + { + elog(LOG, "stopping decoding of %s (%u)", + txn->gid[0] != '\0'? txn->gid:"", txn->xid); + ReorderBufferTruncateTXN(rb, txn, true); + } Why only if (streaming) is not enough? I agree if we are coming here and it is streaming mode then streaming started must be true but we already have an assert for that. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] logical decoding of two-phase transactions

2020-09-21 Thread Dilip Kumar
On Mon, Sep 21, 2020 at 10:20 AM Amit Kapila wrote: > > On Sun, Sep 20, 2020 at 11:01 AM Dilip Kumar wrote: > > > > On Fri, Sep 18, 2020 at 6:02 PM Ajin Cherian wrote: > > > > > > > 3. > > > > + /* > > +

Re: Logical replication from PG v13 and below to PG v14 (devel version) is not working.

2020-09-21 Thread Dilip Kumar
Seems like this commit, should have only set the LOGICALREP_PROTO_STREAM_VERSION_NUM to 2 but the LOGICALREP_PROTO_VERSION_NUM shouln't have been changed. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Parallel bitmap index scan

2020-09-21 Thread Dilip Kumar
On Sun, Jul 26, 2020 at 6:42 PM Dilip Kumar wrote: > > I would like to propose a patch for enabling the parallelism for the > bitmap index scan path. > > Background: > Currently, we support only a parallel bitmap heap scan path. Therein, > the underlying bitmap index sca

Re: Logical replication from PG v13 and below to PG v14 (devel version) is not working.

2020-09-21 Thread Dilip Kumar
On Mon, Sep 21, 2020 at 4:15 PM Dilip Kumar wrote: > > On Mon, Sep 21, 2020 at 3:26 PM Ashutosh Sharma wrote: > > > > Hi All, > > > > Today, while exploring logical replication in PostgreSQL, I noticed > > that logical replication from PG version 13 and belo

Re: Logical replication from PG v13 and below to PG v14 (devel version) is not working.

2020-09-21 Thread Dilip Kumar
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED), > errmsg("client sent proto_version=%d but we only > support protocol %d or lower", > data->protocol_version, LOGICALREP_PROTO_VERSION_NUM))); > > Other than this, I don't have any comments. Thanks for t

Re: [HACKERS] Custom compression methods

2020-10-21 Thread Dilip Kumar
On Thu, Oct 22, 2020 at 2:11 AM Tomas Vondra wrote: > > On Wed, Oct 21, 2020 at 01:59:50PM +0530, Dilip Kumar wrote: > >On Sat, Oct 17, 2020 at 11:34 AM Dilip Kumar wrote: > >> > >> On Tue, Oct 13, 2020 at 10:30 AM Dilip Kumar wrote: > >> > > >

Re: Track statistics for streaming of in-progress transactions

2020-10-22 Thread Dilip Kumar
stead of saying "amount of data streamed to subscriber" it should be " amount of data streamed to the decoding output plugin" -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Custom compression methods

2020-10-22 Thread Dilip Kumar
ng with the pg_am. So I think only for the preserve purpose we will maintain the attribute -> pg_compression dependency so it should be fine. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: parallel distinct union and aggregate support patch

2020-10-22 Thread Dilip Kumar
gregated rows in the shared tuple store based on the hash value and then the worker can pick the batch by batch. By doing this way, we can avoid doing large sorts. And then this approach can also be used with the hash aggregate, I mean the partially aggregated data by the hash aggregate can be put i

Re: Re: parallel distinct union and aggregate support patch

2020-10-27 Thread Dilip Kumar
->batches[i]); + state->batches[i] = NULL; + } + } I did not understand this part, that once each worker has performed their local batch-wise sort why we are clearing the baches? I mean individual workers have their on batches so eventually they supposed to get merged. Can you explain this part and also it will be better if you can add the comments. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Re: parallel distinct union and aggregate support patch

2020-10-27 Thread Dilip Kumar
On Tue, Oct 27, 2020 at 3:27 PM Dilip Kumar wrote: > > On Fri, Oct 23, 2020 at 11:58 AM bu...@sohu.com wrote: > > > > > Interesting idea. So IIUC, whenever a worker is scanning the tuple it > > > will directly put it into the respective batch(shared tuple s

Re: parallel distinct union and aggregate support patch

2020-10-27 Thread Dilip Kumar
On Tue, Oct 27, 2020 at 5:43 PM Robert Haas wrote: > > On Thu, Oct 22, 2020 at 5:08 AM Dilip Kumar wrote: > > Interesting idea. So IIUC, whenever a worker is scanning the tuple it > > will directly put it into the respective batch(shared tuple store), > > based on the

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-08-07 Thread Dilip Kumar
On Thu, Aug 6, 2020 at 2:43 PM Amit Kapila wrote: > > On Wed, Aug 5, 2020 at 7:37 PM Dilip Kumar wrote: > > > > On Wed, Aug 5, 2020 at 6:25 PM Amit Kapila wrote: > > > > > > > > Can we add a test for incomplete changes (probably with toast > > &

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-06-29 Thread Dilip Kumar
On Tue, Jun 30, 2020 at 9:20 AM Amit Kapila wrote: > > On Mon, Jun 29, 2020 at 4:24 PM Dilip Kumar wrote: > > > > On Mon, Jun 22, 2020 at 4:30 PM Amit Kapila wrote: > > > > > > > > > Other than above tests, can we somehow verify that the invalidation

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-07-06 Thread Dilip Kumar
On Mon, Jul 6, 2020 at 11:31 AM Amit Kapila wrote: > > On Sun, Jul 5, 2020 at 4:47 PM Dilip Kumar wrote: > > > > On Sat, Jul 4, 2020 at 11:35 AM Amit Kapila wrote: > > > > > > > > 9. > > > +ReorderBufferHandleConcurrentAb

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-07-09 Thread Dilip Kumar
On Wed, Jul 8, 2020 at 3:32 PM Amit Kapila wrote: > > On Wed, Jul 8, 2020 at 9:36 AM Amit Kapila wrote: > > > > On Sun, Jul 5, 2020 at 8:37 PM Dilip Kumar wrote: > > > > > > I have compared the changes logged at command end vs logged at commit > >

Re: new heapcheck contrib module

2020-07-04 Thread Dilip Kumar
On Sun, Jun 28, 2020 at 11:18 PM Mark Dilger wrote: > > > > > On Jun 28, 2020, at 9:05 AM, Dilip Kumar wrote: > > > > Some more comments on v9_0001. > > 1. > > + LWLockAcquire(XidGenLock, LW_SHARED); > > + nextFullXid = ShmemVariabl

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-07-07 Thread Dilip Kumar
On Mon, Jul 6, 2020 at 3:09 PM Amit Kapila wrote: > > On Mon, Jul 6, 2020 at 11:44 AM Dilip Kumar wrote: > > > > On Mon, Jul 6, 2020 at 11:31 AM Amit Kapila wrote: > > > > > > > > > > 10. I have got the below failure once. I have not investigate

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-07-10 Thread Dilip Kumar
On Fri, Jul 10, 2020 at 11:01 AM Ajin Cherian wrote: > > > > On Fri, Jul 10, 2020 at 3:11 PM Dilip Kumar wrote: >> >> With your changes sometimes due to incomplete toast >> changes, if it can not pick the largest top txn for streaming it will >> hang for

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-07-10 Thread Dilip Kumar
. However, I agree during start replication slot we might decode some of the extra walls of the transaction for which we already got the commit confirmation and we must have a way to avoid that. But I think we don't need to do anything for the CONSISTENT snapshot point. What's your thought on this? @@ -1016,6 +1016,12 @@ CreateReplicationSlot(CreateReplicationSlotCmd *cmd) WalSndPrepareWrite, WalSndWriteData, WalSndUpdateProgress); + /* + * Make sure streaming is disabled here - we may have the methods, + * but we don't have anywhere to send the data yet. + */ + ctx->streaming = false; + -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-07-13 Thread Dilip Kumar
On Mon, Jul 13, 2020 at 2:56 PM Amit Kapila wrote: > > On Mon, Jul 13, 2020 at 2:32 PM Dilip Kumar wrote: > > > > On Mon, Jul 13, 2020 at 11:10 AM Amit Kapila > > wrote: > > > > > > > > > I think you can refer to commit message as well

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-07-13 Thread Dilip Kumar
On Mon, Jul 13, 2020 at 4:00 PM Amit Kapila wrote: > > On Mon, Jul 13, 2020 at 3:04 PM Dilip Kumar wrote: > > > > On Mon, Jul 13, 2020 at 2:56 PM Amit Kapila wrote: > > > > > > On Mon, Jul 13, 2020 at 2:32 PM Dilip Kumar wrote: > > > > >

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-07-13 Thread Dilip Kumar
On Mon, Jul 13, 2020 at 11:10 AM Amit Kapila wrote: > > On Mon, Jul 13, 2020 at 10:40 AM Dilip Kumar wrote: > > > > On Mon, Jul 13, 2020 at 10:14 AM Amit Kapila > > wrote: > > > > > > On Fri, Jul 10, 2020 at 3:37 PM Dilip Kumar wrote: > >

Re: replication_origin and replication_origin_lsn usage on subscriber

2020-07-14 Thread Dilip Kumar
On Tue, Jul 14, 2020 at 11:14 AM Amit Kapila wrote: > > On Tue, Jul 14, 2020 at 11:00 AM Dilip Kumar wrote: > > > > On Thu, Jul 9, 2020 at 6:55 PM Amit Kapila wrote: > > > > > > On Thu, Jul 9, 2020 at 6:14 PM Petr Jelinek wrote: > > > >

Re: INSERT INTO SELECT, Why Parallelism is not selected?

2020-07-14 Thread Dilip Kumar
On Mon, Jul 13, 2020 at 4:23 PM Amit Kapila wrote: > > On Sat, Jul 11, 2020 at 6:07 PM Dilip Kumar wrote: > > > > I have just notice that the parallelism is off even for the select > > part of the query mentioned in the $subject. I see the only reason it > > is n

Re: replication_origin and replication_origin_lsn usage on subscriber

2020-07-13 Thread Dilip Kumar
send origin_lsn earlier than that then we need to record it > > > with other WAL records (other than Commit WAL record). > > > > > > > If we were to support the origin forwarding, then strictly speaking we > > need everything only at commit time from correctn

Re: replication_origin and replication_origin_lsn usage on subscriber

2020-07-14 Thread Dilip Kumar
On Tue, Jul 14, 2020 at 2:47 PM Petr Jelinek wrote: > > Hi, > > On 14/07/2020 10:29, Amit Kapila wrote: > > On Tue, Jul 14, 2020 at 12:05 PM Dilip Kumar wrote: > >> > >> On Tue, Jul 14, 2020 at 11:14 AM Amit Kapila > >> wrote: > >&g

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-07-14 Thread Dilip Kumar
On Mon, Jul 13, 2020 at 4:00 PM Amit Kapila wrote: > > On Mon, Jul 13, 2020 at 3:04 PM Dilip Kumar wrote: > > > > On Mon, Jul 13, 2020 at 2:56 PM Amit Kapila wrote: > > > > > > On Mon, Jul 13, 2020 at 2:32 PM Dilip Kumar wrote: > > > > >

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-07-14 Thread Dilip Kumar
On Mon, Jul 13, 2020 at 10:47 AM Dilip Kumar wrote: > > On Sun, Jul 12, 2020 at 9:56 PM Dilip Kumar wrote: > > > > On Mon, Jul 6, 2020 at 11:43 AM Dilip Kumar wrote: > > > > > > On Mon, Jul 6, 2020 at 11:31 AM Amit Kapila > > > wrote: > > >

<    1   2   3   4   5   6   7   8   9   10   >