Re: "ERROR: latch already owned" on gharial

2024-02-09 Thread Soumyadeep Chakraborty
Hey, Deeply appreciate both your input! On Thu, Feb 8, 2024 at 4:57 AM Heikki Linnakangas wrote: > Hmm, there is a pair of SpinLockAcquire() and SpinLockRelease() in > ProcKill(), before step 3 can happen. Comment in spin.h about > SpinLockAcquire/Release: > > > *Load and store operations

Re: "ERROR: latch already owned" on gharial

2024-02-07 Thread Soumyadeep Chakraborty
s://www.postgresql.org/message-id/flat/20150112154026.GB2092%40awork2.anarazel.de From 9702054c37aa80cb60a16b2d80a475e9b27b087a Mon Sep 17 00:00:00 2001 From: Soumyadeep Chakraborty Date: Wed, 7 Feb 2024 18:04:43 -0800 Subject: [PATCH v1 1/1] Use memory barrier in DisownLatch Failing to do so

Re: brininsert optimization opportunity

2023-12-13 Thread Soumyadeep Chakraborty
On Tue, Dec 12, 2023 at 3:25 AM Tomas Vondra wrote: > The attached 0001 patch fixes this by adding the call to validate_index, which seems like the proper place as it's where the indexInfo is allocated and destroyed. Yes, and by reading validate_index's header comment, there is a clear

Re: brininsert optimization opportunity

2023-11-26 Thread Soumyadeep Chakraborty
On Sun, Nov 26, 2023 at 9:28 PM Richard Guo wrote: > > > On Sun, Nov 26, 2023 at 4:06 AM Tomas Vondra > wrote: >> >> I've done a bit more cleanup on the last version of the patch (renamed >> the fields to start with bis_ as agreed, rephrased the comments / docs / >> commit message a bit) and

Re: brininsert optimization opportunity

2023-11-25 Thread Soumyadeep Chakraborty
Thanks a lot for reviewing and pushing! :) Regards, Soumyadeep (VMware)

Re: brininsert optimization opportunity

2023-11-04 Thread Soumyadeep Chakraborty
On Fri, 3 Nov 2023 at 19:37, Tomas Vondra wrote: > The one thing I'm not entirely sure about is adding new stuff to the > IndexAmRoutine. I don't think we want to end up with too many callbacks > that all AMs have to initialize etc. I can't think of a different/better > way to do this, though.

Re: brininsert optimization opportunity

2023-09-04 Thread Soumyadeep Chakraborty
Rebased against latest HEAD. Regards, Soumyadeep (VMware) From 94a8acd3125aa4a613c238e435ed78dba9f40625 Mon Sep 17 00:00:00 2001 From: Soumyadeep Chakraborty Date: Sat, 29 Jul 2023 09:17:49 -0700 Subject: [PATCH v5 1/1] Reuse revmap and brin desc in brininsert brininsert() used to have code

Re: brininsert optimization opportunity

2023-08-05 Thread Soumyadeep Chakraborty
Created an entry for the Sep CF: https://commitfest.postgresql.org/44/4484/ Regards, Soumyadeep (VMware) On Sat, Jul 29, 2023 at 9:28 AM Soumyadeep Chakraborty wrote: > > Attached v4 of the patch, rebased against latest HEAD. > > Regards, > Soumyadeep (VMware)

Re: brininsert optimization opportunity

2023-07-29 Thread Soumyadeep Chakraborty
Attached v4 of the patch, rebased against latest HEAD. Regards, Soumyadeep (VMware) From b5fb12fbb8b0c1b2963a05a2877b5063bbc75f58 Mon Sep 17 00:00:00 2001 From: Soumyadeep Chakraborty Date: Sat, 29 Jul 2023 09:17:49 -0700 Subject: [PATCH v4 1/1] Reuse revmap and brin desc in brininsert

Re: brininsert optimization opportunity

2023-07-05 Thread Soumyadeep Chakraborty
it's perfectly sensible to > only require the "cleanup callback" when just pfree() is not enough. Yeah no need. Attached v3 of the patch w/ a single index AM callback. Regards, Soumyadeep (VMware) From 563b905da5c2602113044689e37f8385cbfbde64 Mon Sep 17 00:00:00 2001 From: Soumyadee

Re: brininsert optimization opportunity

2023-07-04 Thread Soumyadeep Chakraborty
On Tue, Jul 4, 2023 at 2:54 PM Tomas Vondra wrote: > I'm not sure if memory context callbacks are the right way to rely on > for this purpose. The primary purpose of memory contexts is to track > memory, so using them for this seems a bit weird. Yeah, this just kept getting dirtier and dirtier.

Re: brininsert optimization opportunity

2023-07-04 Thread Soumyadeep Chakraborty
-- 3 runs (branch v2) COPY 2 Time: 135052.752 ms (02:15.053) Time: 135093.131 ms (02:15.093) Time: 138737.048 ms (02:18.737) Regards, Soumyadeep (VMware) From 54ba134f4afe9c4bac19e7d8fde31b9768dc23cd Mon Sep 17 00:00:00 2001 From: Soumyadeep Chakraborty Date: Tue, 4 Jul 2023 11:

brininsert optimization opportunity

2023-07-03 Thread Soumyadeep Chakraborty
hed perf diff between master and branch. We see that we save on a bit of overhead from brinRevmapInitialize(), brinRevmapTerminate() and lock routines. Regards, Soumyadeep (VMware) perf_diff.out Description: Binary data From 5d11219e413fe6806e00dd738b051c3948dffcab Mon Sep 17 00:00:00 2001 Fro

Re: pg_rewind: Skip log directory for file type check like pg_wal

2023-03-06 Thread Soumyadeep Chakraborty
On Mon, Mar 6, 2023 at 11:33 PM Alexander Kukushkin wrote: > > > Lets assume that on the source we have "pg_log" and on the target we have > "my_log" (they are configured using "log_directory" GUC). > When doing rewind in this case we want neither to remove the content of > "my_log" on the

Re: pg_rewind: Skip log directory for file type check like pg_wal

2023-03-06 Thread Soumyadeep Chakraborty
On Mon, Mar 6, 2023 at 12:28 AM Alexander Kukushkin wrote: > > Hello Soumyadeep, > > The problem indeed exists, but IMO the "log" directory case must be handled > differently: > 1. We don't need or I would even say we don't want to sync log files from the > new primary, because it destroys the

pg_rewind: Skip log directory for file type check like pg_wal

2023-03-05 Thread Soumyadeep Chakraborty
137ea9cc93f8576a9792 Mon Sep 17 00:00:00 2001 From: Soumyadeep Chakraborty Date: Sun, 5 Mar 2023 17:57:55 -0800 Subject: [PATCH v1 1/1] Fix pg_rewind when log is a symlink The log directory can often be symlinked in the same way the pg_wal directory is. Today, even if BOTH the source and t

Re: ALTER TABLE SET ACCESS METHOD on partitioned tables

2022-06-09 Thread Soumyadeep Chakraborty
30aa45224fb1e64bd95e0bb64fc00 Mon Sep 17 00:00:00 2001 From: Soumyadeep Chakraborty Date: Thu, 9 Jun 2022 10:57:35 -0700 Subject: [PATCH v3 1/2] Allow ATSETAM on partition roots Setting the access method on partition roots was disallowed. This commit relaxes that restriction. Summary of changes

Re: ALTER TABLE SET ACCESS METHOD on partitioned tables

2022-05-18 Thread Soumyadeep Chakraborty
chy, with one command. Regards, Soumyadeep (VMware) From 3a4a78d46fc74bb3e9b7ac9aefc689d250e1ecf4 Mon Sep 17 00:00:00 2001 From: Soumyadeep Chakraborty Date: Tue, 17 May 2022 15:22:48 -0700 Subject: [PATCH v2 2/2] Make ATSETAM recurse by default ATSETAM now recurses to partition children by default.

ALTER TABLE SET ACCESS METHOD on partitioned tables

2022-05-17 Thread Soumyadeep Chakraborty
that. Thoughts? Regards, Soumyadeep (VMware) [1] https://www.postgresql.org/message-id/20210308010707.GA29832%40telsasoft.com From 440517ff8a5912d4c657a464b2c1de9c8b3a4f70 Mon Sep 17 00:00:00 2001 From: Soumyadeep Chakraborty Date: Tue, 17 May 2022 15:22:48 -0700 Subject: [PATCH v1 2/2] Make ATSETAM

Re: Unnecessary delay in streaming replication due to replay lag

2021-12-15 Thread Soumyadeep Chakraborty
Can we have a better GUC name than wal_receiver_start_condition? > Something like wal_receiver_start_at or wal_receiver_start or some > other? Sure, that makes more sense. Fixed. Regards, Soumyadeep (VMware) [1] https://www.postgresql.org/message-id/CAE-ML%2B-8KnuJqXKHz0mrC7-qFMQJ3ArDC78X3-

Re: Unnecessary delay in streaming replication due to replay lag

2021-11-22 Thread Soumyadeep Chakraborty
Hi Bharath, Yes, that thread has been discussed here. Asim had x-posted the patch to [1]. This thread was more recent when Ashwin and I picked up the patch in Aug 2021, so we continued here. The patch has been significantly updated by us, addressing Michael's long outstanding feedback. Regards,

Re: Unnecessary delay in streaming replication due to replay lag

2021-11-19 Thread Soumyadeep Chakraborty
Hi Daniel, Thanks for checking in on this patch. Attached rebased version. Regards, Soumyadeep (VMware) From 51149e4f877dc2f8bf47a1356fc8b0ec2512ac6d Mon Sep 17 00:00:00 2001 From: Soumyadeep Chakraborty Date: Fri, 19 Nov 2021 00:33:17 -0800 Subject: [PATCH v4 1/1] Introduce feature to start

Re: Unnecessary delay in streaming replication due to replay lag

2021-11-09 Thread Soumyadeep Chakraborty
g are options > wanted by the user. Agreed, it would be much better to depend on the state in pg_wal, namely the files that are available there. Reworking the priority order seems like an appealing fix - if we can say streaming > archiving in terms of priority, then the race that you are refe

Re: Unnecessary delay in streaming replication due to replay lag

2021-10-25 Thread Soumyadeep Chakraborty
Rebased and added a CF entry for Nov CF: https://commitfest.postgresql.org/35/3376/.

Re: Unnecessary delay in streaming replication due to replay lag

2021-08-24 Thread Soumyadeep Chakraborty
d a documentation section describing the GUC. Regards, Ashwin and Soumyadeep (VMware) From b3703fb16a352bd9166ed75de7b68599c735ac63 Mon Sep 17 00:00:00 2001 From: Soumyadeep Chakraborty Date: Fri, 30 Jul 2021 18:21:55 -0700 Subject: [PATCH v1 1/1] Introduce feature to start WAL receiver eagerly This commit introduces a

Re: Changes to recovery_min_apply_delay are ignored while waiting for delay

2021-08-15 Thread Soumyadeep Chakraborty
On Sun, Aug 15, 2021 at 8:16 PM Michael Paquier wrote: > > On Fri, Aug 13, 2021 at 05:59:21PM -0700, Soumyadeep Chakraborty wrote: > > and passes with the code change, as expected. I can't explain why the > > test doesn't freeze up in v3 in wait_for_catchup() at the end. > &

Re: Changes to recovery_min_apply_delay are ignored while waiting for delay

2021-08-13 Thread Soumyadeep Chakraborty
is test, even if that's the default mode, to make clear what the > intention is. Done. Regards, Soumyadeep (VMware) From 824076a977af0e40da1c7eb9e4aeac9a8e81a7ee Mon Sep 17 00:00:00 2001 From: Soumyadeep Chakraborty Date: Mon, 2 Aug 2021 20:50:37 -0700 Subject: [PATCH v4 1/1] Refresh del

Re: Changes to recovery_min_apply_delay are ignored while waiting for delay

2021-08-06 Thread Soumyadeep Chakraborty
startup is in recovery_min_apply_delay"; Attached v3. Regards, Soumyadeep (VMware) From 441716f45d0fbffde1135ba007c3a6c5b6b464de Mon Sep 17 00:00:00 2001 From: Soumyadeep Chakraborty Date: Mon, 2 Aug 2021 20:50:37 -0700 Subject: [PATCH v3 1/1] Refresh delayUntil in recovery delay loop This ensures that the

Re: Changes to recovery_min_apply_delay are ignored while waiting for delay

2021-08-03 Thread Soumyadeep Chakraborty
min_apply_delay and the other for testing recovery pause. So while making this change, I added a header comment for the newly added test case. Please take a look. Regards, Soumyadeep (VMware) From 8b4ea51e1a22548fdd3f6921fe374d3a9d987a77 Mon Sep 17 00:00:00 2001 From: Soumyadeep Chakraborty Date: Mon

Changes to recovery_min_apply_delay are ignored while waiting for delay

2021-08-02 Thread Soumyadeep Chakraborty
ed a test to 005_replay_delay.pl. Regards, Soumyadeep(VMware) [1] https://www.postgresql.org/message-id/flat/CANXE4Tc3FNvZ_xAimempJWv_RH9pCvsZH7Yq93o1VuNLjUT-mQ%40mail.gmail.com From e1b93d4b3874eb0923e18dca9a9eccd453d6cd56 Mon Sep 17 00:00:00 2001 From: Soumyadeep Chakraborty Date: Mon, 2 Aug 2021 20:50

Re: A micro-optimisation for ProcSendSignal()

2021-08-02 Thread Soumyadeep Chakraborty
Hey Thomas, On Mon, Aug 2, 2021 at 6:45 PM Thomas Munro wrote: > > Hi Soumyadeep, > > On Sat, Jul 24, 2021 at 5:26 PM Soumyadeep Chakraborty > wrote: > > On Tue, Jul 20, 2021 at 10:40 PM Thomas Munro > > wrote: > > > I wonder why we need this member

Re: A micro-optimisation for ProcSendSignal()

2021-07-23 Thread Soumyadeep Chakraborty
me special case code for dealing with the startup process. Discussion: https://postgr.es/m/CA%2BhUKGLYRyDaneEwz5Uya_OgFLMx5BgJfkQSD%3Dq9HmwsfRRb-w%40mail.gmail.com Reviewed-by: Soumyadeep Chakraborty Reviewed-by: Ashwin Agrawal --- src/backend/access/transam/clog.c | 2 +- src/backend/access/tr

Re: A micro-optimisation for ProcSendSignal()

2021-07-17 Thread Soumyadeep Chakraborty
Hi Thomas, You might have missed a spot to initialize SERIALIZABLE_XACT->pgprocno in InitPredicateLocks(), so: + PredXact->OldCommittedSxact->pgprocno = INVALID_PGPROCNO; Slightly tangential: we should add a comment to PGPROC.pgprocno, for more immediate understandability: + int pgprocno; /*

Re: pg_stats and range statistics

2021-07-11 Thread Soumyadeep Chakraborty
Hello, This should have been added with [1]. Excerpt from the documentation: "pg_stats is also designed to present the information in a more readable format than the underlying catalog — at the cost that its schema must be extended whenever new slot types are defined for pg_statistic." [2] So,

Re: Unused function parameter in get_qual_from_partbound()

2021-07-10 Thread Soumyadeep Chakraborty
> Marking this as ready for committer. It can be committed when the branch > is cut for 15. I see that REL_14_STABLE is already cut. So this can go in now.

Re: Unused function parameter in get_qual_from_partbound()

2021-07-10 Thread Soumyadeep Chakraborty
Hello, Googling around, I didn't find any extensions that would break from this change. Even if there are any, this change will simplify the relevant callsites. It also aligns the interface nicely with get_qual_for_hash, get_qual_for_list and get_qual_for_range. Marking this as ready for

Re: PITR promote bug: Checkpointer writes to older timeline

2021-03-13 Thread Soumyadeep Chakraborty
Hello, PFA version 2 of the TAP test. I removed the non-deterministic sleep and introduced retries until the WAL segment is archived and promotion is complete. Some additional tidying up too. Regards, Soumyadeep (VMware) diff --git a/src/test/recovery/t/022_pitr_prepared_xact.pl

Re: PITR promote bug: Checkpointer writes to older timeline

2021-03-04 Thread Soumyadeep Chakraborty
Hey all, I took a stab at a quick and dirty TAP test (my first ever). So it can probably be improved a lot. Please take a look. On Thu, Mar 04, 2021 at 10:28:31AM +0900, Kyotaro Horiguchi wrote: > 2. Restore ThisTimeLineID after calling XLogReadRecord() in the > *caller* side. This is what

Re: PITR promote bug: Checkpointer writes to older timeline

2021-03-03 Thread Soumyadeep Chakraborty
On 2021/03/03 17:46, Heikki Linnakangas wrote: > I think it should be reset even earlier, inside XlogReadTwoPhaseData() > probably. With your patch, doesn't the LogStandbySnapshot() call just > above where you're ressetting ThisTimeLineID also write a WAL record > with incorrect timeline?

PITR promote bug: Checkpointer writes to older timeline

2021-03-02 Thread Soumyadeep Chakraborty
003 > /tmp/waldiff.diff cat /tmp/waldiff.diff From dbf5a9f6899bedf28b482fc03a4a2b0571e92e9b Mon Sep 17 00:00:00 2001 From: Soumyadeep Chakraborty Date: Tue, 2 Mar 2021 17:41:20 -0800 Subject: [PATCH 1/1] Prevent checkpointer from writing to older timeline Co-authored-by: Kevin Yeap -

Re: Table AM modifications to accept column projection lists

2020-12-31 Thread Soumyadeep Chakraborty
v2 1/1] tableam: accept column projection list Co-authored-by: Soumyadeep Chakraborty Co-authored-by: Melanie Plageman Co-authored-by: Ashwin Agrawal Co-authored-by: Jacob Champion --- src/backend/access/heap/heapam_handler.c | 5 +- src/backend/access/nbtree/nbtsort.c | 3 +- src

Re: Split copy.

2020-11-17 Thread Soumyadeep Chakraborty
On Tue, Nov 17, 2020 at 2:38 AM Heikki Linnakangas wrote: > > Thanks for feedback, attached is a new patch version. > > On 11/11/2020 21:49, Soumyadeep Chakraborty wrote: > > On Tue, Nov 3, 2020 at 1:30 AM Heikki Linnakangas wrote: > >> I also split/duplicat

Table AM modifications to accept column projection lists

2020-11-13 Thread Soumyadeep Chakraborty
ostgresql.org/message-id/23194.1560618101%40sss.pgh.pa.us From 9a77cbaa839f8b56cd9d93fdf1ddf418fdc8a763 Mon Sep 17 00:00:00 2001 From: Jacob Champion Date: Mon, 9 Nov 2020 16:36:10 -0800 Subject: [PATCH] tableam: accept column projection list Co-authored-by: Soumyadeep Chakraborty Co-authored-by: Melan

Re: Delay of standby shutdown

2020-11-12 Thread Soumyadeep Chakraborty
Thanks! Marking this as ready for committer. Regards, Soumyadeep (VMware)

Re: Split copy.c

2020-11-11 Thread Soumyadeep Chakraborty
Hey Heikki, On Tue, Nov 3, 2020 at 1:30 AM Heikki Linnakangas wrote: Thanks for working on this refactor. LGTM! I had a few minor comments: 1. We should rename the CopyFormatOptions *cstate param in ProcessCopyOptions() to CopyFormatOptions *options and List *options to List *raw_options IMO,

Re: PATCH: Attempt to make dbsize a bit more consistent

2020-11-11 Thread Soumyadeep Chakraborty
Hey Georgios, On Tue, Nov 10, 2020 at 6:20 AM wrote: > > > > > > > ‐‐‐ Original Message ‐‐‐ > On Monday, November 9, 2020 7:50 PM, Soumyadeep Chakraborty > wrote: > > > Hey Georgios, > > > > Thanks for looking for more avenues to invo

Re: PATCH: Attempt to make dbsize a bit more consistent

2020-11-09 Thread Soumyadeep Chakraborty
Hey Georgios, Thanks for looking for more avenues to invoke tableAM APIS! Please find my review below: On Tue, Oct 13, 2020 at 6:28 AM wrote: 1. > /* > - * heap size, including FSM and VM > + * table size, including FSM and VM > */ We should not mention FSM and VM in dbsize.c at all as

Re: Delay of standby shutdown

2020-11-03 Thread Soumyadeep Chakraborty
Hello Fujii, On Thu, Sep 17, 2020 at 6:49 AM Fujii Masao wrote: As far as I understand after debugging, the problem is as follows: 1. After the primary is stopped, and the standby startup process is waiting inside: (void) WaitLatch(>recoveryWakeupLatch, WL_LATCH_SET | WL_TIMEOUT |

Re: Add session statistics to pg_stat_database

2020-10-02 Thread Soumyadeep Chakraborty
On Tue, Sep 29, 2020 at 2:44 AM Laurenz Albe wrote: > > * Are we trying to capture ONLY client initiated disconnects in > > m_aborted (we are not handling other disconnects by not accounting for > > EOF..like if psql was killed)? If yes, why? > > I thought it was interesting to know how many

Re: avoid bitmapOR-ing indexes with scan condition inconsistent with partition constraint

2020-09-30 Thread Soumyadeep Chakraborty
Hi Justin, Attached is a minimal patch that is rebased and removes the dependency on Konstantin's earlier patch[1], making it enough to remove the extraneous index scans as Justin brought up. Is this the direction we want to head in? Tagging Konstantin in the email to hear his input on his old

Re: Refactor pg_rewind code and make it work against a standby

2020-09-24 Thread Soumyadeep Chakraborty
On Thu, Sep 24, 2020 at 10:27 AM Heikki Linnakangas wrote: > >> /* > >> * If this is a relation file, copy the modified blocks. > >> * > >> * This is in addition to any other changes. > >> */ > >> iter = datapagemap_iterate(>target_modified_pages); > >> while (datapagemap_next(iter, )) >

Re: Add session statistics to pg_stat_database

2020-09-24 Thread Soumyadeep Chakraborty
Hello Laurenz, Thanks for submitting this! Please find my feedback below. * Are we trying to capture ONLY client initiated disconnects in m_aborted (we are not handling other disconnects by not accounting for EOF..like if psql was killed)? If yes, why? * pgstat_send_connstats(): How about

Re: Refactor pg_rewind code and make it work against a standby

2020-09-20 Thread Soumyadeep Chakraborty
Hey Heikki, Thanks for refactoring and making the code much easier to read! Before getting into the code review for the patch, I wanted to know why we don't use a Bitmapset for target_modified_pages? Code review: 1. We need to update the comments for process_source_file and

Re: Adding Support for Copy callback functionality on COPY TO api

2020-09-14 Thread Soumyadeep Chakraborty
Hi Bilva, Thank you for registering this patch! I had a few suggestions: 1. Please run pg_indent[1] on your code. Make sure you add copy_data_destination_cb to src/tools/pgindent/typedefs.list. Please run pg_indent on only the files you changed (it will take files as command line args) 2. For

Re: [Patch] ALTER SYSTEM READ ONLY

2020-07-24 Thread Soumyadeep Chakraborty
On Fri, Jul 24, 2020 at 7:34 AM Robert Haas wrote: > > On Thu, Jul 23, 2020 at 12:11 PM Soumyadeep Chakraborty > wrote: > > In the read-only level I was suggesting, I wasn't suggesting that we > > stop WAL flushes, in fact we should flush the WAL before we mark the > >

Re: [Patch] ALTER SYSTEM READ ONLY

2020-07-24 Thread Soumyadeep Chakraborty
On Fri, Jul 24, 2020 at 7:32 AM Robert Haas wrote: > > On Wed, Jul 22, 2020 at 6:03 PM Soumyadeep Chakraborty > wrote: > > So if we are not going to address those cases, we should change the > > syntax and remove the notion of read-only. It could be: > > > > A

Re: [Patch] ALTER SYSTEM READ ONLY

2020-07-24 Thread Soumyadeep Chakraborty
On Thu, Jul 23, 2020 at 10:14 PM Amul Sul wrote: > > On Fri, Jul 24, 2020 at 6:28 AM Soumyadeep Chakraborty > wrote: > > In case it is necessary, the patch set does not wait for the checkpoint to > > complete before marking the system as read-write. Refer: > > > &g

Re: [Patch] ALTER SYSTEM READ ONLY

2020-07-23 Thread Soumyadeep Chakraborty
On Thu, Jun 18, 2020 at 7:54 AM Robert Haas wrote: > I think we'd want the FIRST write operation to be the end-of-recovery > checkpoint, before the system is fully read-write. And then after that > completes you could do other things. I can't see why this is necessary from a correctness or

Re: [Patch] ALTER SYSTEM READ ONLY

2020-07-23 Thread Soumyadeep Chakraborty
On Thu, Jul 23, 2020 at 3:57 AM Amul Sul wrote: > Well, once we've initiated the change to a read-only state, we probably want > to > always either finish that change or go back to read-write, even if the process > that initiated the change is interrupted. Leaving the system in a >

Re: [Patch] ALTER SYSTEM READ ONLY

2020-07-23 Thread Soumyadeep Chakraborty
On Thu, Jul 23, 2020 at 3:42 AM Amul Sul wrote: > The aim of this feature is preventing new WAL records from being generated, > not > preventing them from being flushed to disk, or streamed to standbys, or > anything > else. The rest should happen as normal. > > If you can't flush WAL, then you

Re: [Patch] ALTER SYSTEM READ ONLY

2020-07-22 Thread Soumyadeep Chakraborty
Hi Amul, On Tue, Jun 16, 2020 at 6:56 AM amul sul wrote: > The proposed feature is built atop of super barrier mechanism commit[1] to > coordinate > global state changes to all active backends. Backends which executed > ALTER SYSTEM READ { ONLY | WRITE } command places request to checkpointer >

Re: [Patch] ALTER SYSTEM READ ONLY

2020-07-22 Thread Soumyadeep Chakraborty
Hello, I think we should really term this feature, as it stands, as a means to solely stop WAL writes from happening. The feature doesn't truly make the system read-only (e.g. dirty buffer flushes may succeed the system being put into a read-only state), which does make it confusing to a degree.

Re: Parallel Seq Scan vs kernel read ahead

2020-07-22 Thread Soumyadeep Chakraborty
On Tue, Jul 21, 2020 at 9:33 PM Thomas Munro wrote: > > On Wed, Jul 22, 2020 at 3:57 PM Amit Kapila wrote: > > Yeah, that is true but every time before the test the same amount of > > data should be present in shared buffers (or OS cache) if any which > > will help in getting consistent results.

Re: Parallel Seq Scan vs kernel read ahead

2020-07-22 Thread Soumyadeep Chakraborty
Hi David, Apologies for the delay, I had missed these emails. On Tue, Jul 14, 2020 at 8:52 PM David Rowley wrote: > It would be good to know if the > regression is repeatable or if it was affected by some other process. These are the latest results on the same setup as [1]. (TL;DR: the

Re: Does TupleQueueReaderNext() really need to copy its result?

2020-07-11 Thread Soumyadeep Chakraborty
Hey Thomas, On Fri, Jul 10, 2020 at 7:30 PM Thomas Munro wrote: > > > I could reproduce the speed gain that you saw for a plan with a simple > > parallel sequential scan. However, I got no gain at all for a parallel > > hash join and parallel agg query. > > Right, it's not going to make a

Re: Does TupleQueueReaderNext() really need to copy its result?

2020-07-10 Thread Soumyadeep Chakraborty
Hi Thomas, +1 to the idea! I ran some experiments on both of your patches. I could reproduce the speed gain that you saw for a plan with a simple parallel sequential scan. However, I got no gain at all for a parallel hash join and parallel agg query.

Re: posgres 12 bug (partitioned table)

2020-07-09 Thread Soumyadeep Chakraborty
Hey Amit, On Thu, Jul 9, 2020 at 12:16 AM Amit Langote wrote: > By the way, what happens today if you do INSERT INTO a_zedstore_table > ... RETURNING xmin? Do you get an error "xmin is unrecognized" or > some such in slot_getsysattr() when trying to project the RETURNING > list? > We get

Re: posgres 12 bug (partitioned table)

2020-07-08 Thread Soumyadeep Chakraborty
Hey Amit, On Tue, Jul 7, 2020 at 7:17 PM Amit Langote wrote: > Ah, I see. You might've noticed that ExecInsert() only ever sees leaf > partitions, because tuple routing would've switched the result > relation to a leaf partition by the time we are in ExecInsert(). So, > table_tuple_insert()

Re: posgres 12 bug (partitioned table)

2020-07-07 Thread Soumyadeep Chakraborty
Hi Amit, Thanks for your reply! On Tue, Jul 7, 2020 at 7:18 AM Amit Langote wrote: > > Hi Soumyadeep, > > Thanks for picking this up. > > On Tue, Jul 7, 2020 at 7:46 AM Soumyadeep Chakraborty > wrote: > > Upon investigation, it seems that the problem is

Re: posgres 12 bug (partitioned table)

2020-07-06 Thread Soumyadeep Chakraborty
Hello, ccing pgsql-hack...@postgresql.org Upon investigation, it seems that the problem is caused by the following: The slot passed to the call to ExecProcessReturning() inside ExecInsert() is often a virtual tuple table slot. If there are system columns other than ctid and tableOid referenced

Re: Extracting only the columns needed for a query

2020-06-30 Thread Soumyadeep Chakraborty
Hello, Melanie and me will be posting a separate thread with the scanCols patch listed here, a patch to capture the set of cols in RETURNING and a group of patches to pass down the list of cols to various table AM functions together as a patch set. This will take some time. Thus, we are

Re: Parallel Seq Scan vs kernel read ahead

2020-06-03 Thread Soumyadeep Chakraborty
On Wed, Jun 3, 2020 at 3:18 PM Soumyadeep Chakraborty wrote: > Idk if that is a lesser evil than the workers > being idle..probably not? Apologies, I meant that the extra atomic fetches is probably a lesser evil than the workers being idle. Soumyadeep

Re: Parallel Seq Scan vs kernel read ahead

2020-06-03 Thread Soumyadeep Chakraborty
On Sat, May 23, 2020 at 12:00 AM Robert Haas wrote: > I think there's a significant difference. The idea I remember being > discussed at the time was to divide the relation into equal parts at > the very start and give one part to each worker. I think that carries > a lot of risk of some workers

Re: Parallel Seq Scan vs kernel read ahead

2020-06-03 Thread Soumyadeep Chakraborty
> It doesn't look like it's using table_block_parallelscan_nextpage() as > a block allocator so it's not affected by the patch. It has its own > thing zs_parallelscan_nextrange(), which does > pg_atomic_fetch_add_u64(>pzs_allocatedtids, > ZS_PARALLEL_CHUNK_SIZE), and that macro is 0x10. My

Re: Parallel Seq Scan vs kernel read ahead

2020-05-21 Thread Soumyadeep Chakraborty
Hi Thomas, Some more data points: create table t_heap as select generate_series(1, 1) i; Query: select count(*) from t_heap; shared_buffers=32MB (so that I don't have to clear buffers, OS page cache) OS: FreeBSD 12.1 with UFS on GCP 4 vCPUs, 4GB RAM Intel Skylake 22G Google

Re: WIP: expression evaluation improvements

2020-03-03 Thread Soumyadeep Chakraborty
Hello Andres, Attached is a patch on top of v2-0026-WIP-expression-eval-relative-pointer-suppport.patch that eliminates the const pointer references to fmgrInfo in the generated code. FmgrInfos are now allocated like the FunctionCallInfos are (ExprBuilderAllocFunctionMgrInfo()) and are

Re: WIP: expression evaluation improvements

2020-02-19 Thread Soumyadeep Chakraborty
Hey Andres, > Awesome! +1. Attached 2nd version of patch rebased on master. > (v2-0001-Resolve-PL-handler-names-for-JITed-code-instead-o.patch) > > > > > Did you check whether there's any cases this fails in the tree with your > > patch applied? The way I usually do that is by running the

Re: Zedstore - compressed in-core columnar storage

2020-02-14 Thread Soumyadeep Chakraborty
Hello, We (David and I) recently observed that a Zedstore table can be considerably bloated when we load data into it with concurrent copies. Also, we found that concurrent insert performance was less than desirable. This is a detailed analysis of the extent of the problem, the cause of the

Re: WIP: expression evaluation improvements

2020-02-09 Thread Soumyadeep Chakraborty
Hi Andres, > Could you expand on what you mean here? Are you saying that you got > significantly better optimization results by doing function optimization > early on? That'd be surprising imo? Sorry for the ambiguity, I meant that I had observed differences in the sizes of the bitcode files

Re: WIP: expression evaluation improvements

2020-02-09 Thread Soumyadeep Chakraborty
laces. If we then > redirected all function calls through a common wrapper, for LLVMBuildCall, > that also validated parameter count (and perhaps types), I think it'd be > easier to develop... +1. I was wondering whether such validations should be Asserts instead of ERRORs. Regards, Soumyade

Re: Default JIT setting in V12

2020-01-29 Thread Soumyadeep Chakraborty
Hello, Based on this thread, Alexandra and I decided to investigate if we could borrow some passes from -O1 and add on to the default optimization of -O0 and mem2reg. To determine what passes would make most sense, we ran ICW with jit_above_cost set to 0, dumped all the backends and then analyzed

Re: WIP: expression evaluation improvements

2019-10-29 Thread Soumyadeep Chakraborty
Hi Andres, > I think I'd probably try to apply this to master independent of the > larger patchset, to avoid a large dependency. Awesome! +1. Attached 2nd version of patch rebased on master. (v2-0001-Resolve-PL-handler-names-for-JITed-code-instead-o.patch) > Did you check whether there's any

Re: WIP: expression evaluation improvements

2019-10-28 Thread Soumyadeep Chakraborty
Hi Andres, Apologies, I realize my understanding of symbol resolution and the referenced_functions mechanism wasn't correct. Thank you for your very helpful explanations. > There's also a related edge-case where are unable to figure out a symbol > name in llvm_function_reference(), and then

Re: WIP: expression evaluation improvements

2019-10-24 Thread Soumyadeep Chakraborty
Hey Andres, After looking at v2-0006-jit-Reference-functions-by-name-in-IOCOERCE-steps.patch, I was wondering about other places in the code where we have const pointers to functions outside LLVM's purview: specially EEOP_FUNCEXPR* for any function call expressions, EEOP_DISTINCT and EEOP_NULLIF

JIT: Optimize generated functions earlier to lower memory usage

2019-10-06 Thread Soumyadeep Chakraborty
Hello, This is to introduce a patch to lower the memory footprint of JITed code by optimizing functions at the function level (i.e. with function-level optimization passes) as soon as they are generated. This addresses the code comment inside llvm_optimize_module(): /* * Do function level

Re: Don't codegen deform code for virtual tuples in expr eval for scan fetch

2019-09-30 Thread Soumyadeep Chakraborty
Awesome! Thanks so much for all the review! :) -- Soumyadeep

Re: Don't codegen deform code for virtual tuples in expr eval for scan fetch

2019-09-30 Thread Soumyadeep Chakraborty
Hi Andres, I don't feel very strongly about the changes I proposed. > > I completely agree, that was an important consideration. > > > > I had some purely cosmetic suggestions: > > 1. Rename ExecComputeSlotInfo to eliminate the need for the asserts. > > How does renaming it do so? I feel like

Re: Don't codegen deform code for virtual tuples in expr eval for scan fetch

2019-09-28 Thread Soumyadeep Chakraborty
Hey, I completely agree, that was an important consideration. I had some purely cosmetic suggestions: 1. Rename ExecComputeSlotInfo to eliminate the need for the asserts. 2. Extract return value to a bool variable for slightly better readability. 3. Taking the opportunity to use TTS_IS_VIRTUAL.

Re: Don't codegen deform code for virtual tuples in expr eval for scan fetch

2019-09-25 Thread Soumyadeep Chakraborty
Hi Andres, Thank you for your insight and the link offered just the context I needed! On Wed, Sep 25, 2019 at 1:06 PM Andres Freund wrote: > > I'm doubtful this is worth the complexity - and not that we already have > plenty other places with zero length blocks. Agreed. On Wed, Sep 25, 2019

Re: Don't codegen deform code for virtual tuples in expr eval for scan fetch

2019-09-25 Thread Soumyadeep Chakraborty
Hello Andres, Thank you very much for reviewing my patch! On Wed, Sep 25, 2019 at 1:02 PM Andres Freund wrote: > IOW, wherever ExecComputeSlotInfo() is called, we should only actually > push the expression step, when ExecComputeSlotInfo does not determine > that a) the slot is virtual, b) and

Re: Don't codegen deform code for virtual tuples in expr eval for scan fetch

2019-09-20 Thread Soumyadeep Chakraborty
instructions with undefs constitutes a commonly adopted solution. Otherwise, we can always fall back to making opblocks[i] empty with just the unconditional br, as in my previous patch 0001. -- Soumyadeep On Tue, Sep 17, 2019 at 11:54 PM Soumyadeep Chakraborty < soumyadeep2...@gmail.com> wrote: &

Don't codegen deform code for virtual tuples in expr eval for scan fetch

2019-09-18 Thread Soumyadeep Chakraborty
of conditionals around whether to JIT deform or not. --- Soumyadeep Chakraborty 0001-Don-t-codegen-if-tuple-is-virtual-in-expr-eval-of-sc.patch Description: Binary data 0002-Minor-refactor-JIT-deform-or-not.patch Description: Binary data

Infinite wait for SyncRep while handling USR1

2019-08-22 Thread Soumyadeep Chakraborty
Ashwin Agrawal, Bhuvnesh Chaudhary, Jesse Zhang and Soumyadeep Chakraborty