Re: New standby_slot_names GUC in PG 17

2024-06-25 Thread Masahiko Sawada
On Tue, Jun 25, 2024 at 1:54 PM Amit Kapila wrote: > > On Tue, Jun 25, 2024 at 8:20 AM Masahiko Sawada wrote: > > > > On Tue, Jun 25, 2024 at 11:21 AM Zhijie Hou (Fujitsu) > > wrote: > > > > > > I agree the current name seems too generic and the sug

Re: New standby_slot_names GUC in PG 17

2024-06-24 Thread Masahiko Sawada
n_standbys: it indicates that the standbys that enabled > slot sync should be listed in this GUC. > > logical_replication_wait_slots: it means the logical replication(logical > Walsender process) will wait for these slots to advance the confirm flush > lsn before proceeding. I feel that the

Re: suspicious valgrind reports about radixtree/tidstore on arm64

2024-06-20 Thread Masahiko Sawada
On Thu, Jun 20, 2024 at 4:58 PM John Naylor wrote: > > On Thu, Jun 20, 2024 at 8:12 AM Masahiko Sawada wrote: > > > On Thu, Jun 20, 2024 at 7:54 AM Tom Lane wrote: > > > > > > I don't know if there's any reason why the current order > > > is prefer

Re: suspicious valgrind reports about radixtree/tidstore on arm64

2024-06-19 Thread Masahiko Sawada
adixtree-fix-proposed.patch. Even if we zero more fields in the node it would not add noticeable overheads. > > (The RT_NODE_48 case could be optimized a bit if we cared > to swap the order of its slot_idxs[] and isset[] arrays; > then the initial zeroing could just go up to slot_idxs[]

Re: Revive num_dead_tuples column of pg_stat_progress_vacuum

2024-06-18 Thread Masahiko Sawada
On Sat, Jun 15, 2024 at 8:47 PM Robert Treat wrote: > > On Thu, Jun 13, 2024 at 9:22 PM Masahiko Sawada wrote: > > On Fri, Jun 14, 2024 at 9:57 AM Masahiko Sawada > > wrote: > > > On Fri, Jun 14, 2024 at 9:41 AM Michael Paquier > > > wrote: > > &g

Re: Logical Replication of sequences

2024-06-17 Thread Masahiko Sawada
On Fri, Jun 14, 2024 at 4:04 PM Amit Kapila wrote: > > On Thu, Jun 13, 2024 at 6:14 PM Masahiko Sawada wrote: > > > > On Thu, Jun 13, 2024 at 7:06 PM Amit Kapila wrote: > > > > > > On Thu, Jun 13, 2024 at 1:09 PM Masahiko Sawada > > > wrote: &g

Re: Revive num_dead_tuples column of pg_stat_progress_vacuum

2024-06-13 Thread Masahiko Sawada
On Fri, Jun 14, 2024 at 9:57 AM Masahiko Sawada wrote: > > On Fri, Jun 14, 2024 at 9:41 AM Michael Paquier wrote: > > > > On Thu, Jun 13, 2024 at 08:38:05PM -0400, Tom Lane wrote: > > > Masahiko Sawada writes: > > >> I was about to push th

Re: Revive num_dead_tuples column of pg_stat_progress_vacuum

2024-06-13 Thread Masahiko Sawada
On Fri, Jun 14, 2024 at 9:41 AM Michael Paquier wrote: > > On Thu, Jun 13, 2024 at 08:38:05PM -0400, Tom Lane wrote: > > Masahiko Sawada writes: > >> I was about to push the patch but let me confirm just in case: is it > >> okay to bump the catversion even

Re: Revive num_dead_tuples column of pg_stat_progress_vacuum

2024-06-13 Thread Masahiko Sawada
On Tue, Jun 11, 2024 at 10:38 PM Masahiko Sawada wrote: > > On Fri, Jun 7, 2024 at 10:22 AM Masahiko Sawada wrote: > > > > On Wed, Jun 5, 2024 at 7:19 PM Andrey M. Borodin > > wrote: > > > > > > > > > > > > > On 4 Jun 2024,

Re: Logical Replication of sequences

2024-06-13 Thread Masahiko Sawada
On Thu, Jun 13, 2024 at 7:06 PM Amit Kapila wrote: > > On Thu, Jun 13, 2024 at 1:09 PM Masahiko Sawada wrote: > > > > On Wed, Jun 12, 2024 at 6:59 PM Amit Kapila wrote: > > > > > > > > > Yeah, starting with a single worker sounds good for now. Do you

Re: Logical Replication of sequences

2024-06-13 Thread Masahiko Sawada
On Wed, Jun 12, 2024 at 6:59 PM Amit Kapila wrote: > > On Wed, Jun 12, 2024 at 10:44 AM Masahiko Sawada > wrote: > > > > On Tue, Jun 11, 2024 at 7:36 PM vignesh C wrote: > > > > > > 1) CREATE PUBLICATION syntax enhancement: > > > CREATE PUBLI

Re: Conflict Detection and Resolution

2024-06-13 Thread Masahiko Sawada
tails (table name, column name, conflict type, etc) to somewhere (e.g. a table or server logs) so that the users can resolve them manually. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: Logical Replication of sequences

2024-06-11 Thread Masahiko Sawada
On Tue, Jun 11, 2024 at 7:36 PM vignesh C wrote: > > On Tue, 11 Jun 2024 at 12:38, Masahiko Sawada wrote: > > > > On Tue, Jun 11, 2024 at 12:25 PM vignesh C wrote: > > > > > > On Mon, 10 Jun 2024 at 14:48, Amit Kapila wrote: > > > > > >

Re: Revive num_dead_tuples column of pg_stat_progress_vacuum

2024-06-11 Thread Masahiko Sawada
On Fri, Jun 7, 2024 at 10:22 AM Masahiko Sawada wrote: > > On Wed, Jun 5, 2024 at 7:19 PM Andrey M. Borodin wrote: > > > > > > > > > On 4 Jun 2024, at 00:26, Masahiko Sawada wrote: > > > > Thank you! Vacuum enhancement is a really good step forward

Re: Logical Replication of sequences

2024-06-11 Thread Masahiko Sawada
On Tue, Jun 11, 2024 at 12:25 PM vignesh C wrote: > > On Mon, 10 Jun 2024 at 14:48, Amit Kapila wrote: > > > > On Mon, Jun 10, 2024 at 12:43 PM Masahiko Sawada > > wrote: > > > > > > On Mon, Jun 10, 2024 at 3:14 PM Masahiko Sawada > > > wr

Re: Track the amount of time waiting due to cost_delay

2024-06-11 Thread Masahiko Sawada
ng vacuum delay effects. It doesn't mean to measure the impact of the changes on the vacuum duration, though. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: Logical Replication of sequences

2024-06-10 Thread Masahiko Sawada
On Mon, Jun 10, 2024 at 3:14 PM Masahiko Sawada wrote: > > On Fri, Jun 7, 2024 at 7:30 PM Amit Kapila wrote: > > > > On Fri, Jun 7, 2024 at 7:55 AM Masahiko Sawada > > wrote: > > > > > > On Thu, Jun 6, 2024 at 6:40 PM Amit Kapila > > > wro

Re: Logical Replication of sequences

2024-06-10 Thread Masahiko Sawada
On Fri, Jun 7, 2024 at 7:30 PM Amit Kapila wrote: > > On Fri, Jun 7, 2024 at 7:55 AM Masahiko Sawada wrote: > > > > On Thu, Jun 6, 2024 at 6:40 PM Amit Kapila wrote: > > > > > > On Thu, Jun 6, 2024 at 11:10 AM Masahiko Sawada > > > wrote: > &

Re: Logical Replication of sequences

2024-06-06 Thread Masahiko Sawada
On Thu, Jun 6, 2024 at 6:40 PM Amit Kapila wrote: > > On Thu, Jun 6, 2024 at 11:10 AM Masahiko Sawada wrote: > > > > On Wed, Jun 5, 2024 at 9:30 PM Amit Kapila wrote: > > > > > > > > To achieve this, we can allow sequences to be copied during >

Re: Revive num_dead_tuples column of pg_stat_progress_vacuum

2024-06-06 Thread Masahiko Sawada
On Wed, Jun 5, 2024 at 7:19 PM Andrey M. Borodin wrote: > > > > > On 4 Jun 2024, at 00:26, Masahiko Sawada wrote: > > Thank you! Vacuum enhancement is a really good step forward, and this small > change would help a lot of observability tools. > > > > On 4

Re: Logical Replication of sequences

2024-06-05 Thread Masahiko Sawada
SH PUBLICATION WITH (copy_sequence = true); > > In addition to the above, the command Alter Subscription .. Refresh > Publication will fetch any missing sequences similar to what it does > for tables. On the subscriber side, do we need to track which sequences are created via CREATE/ALTER SUBSCRIPTION? Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: Logical Replication of sequences

2024-06-05 Thread Masahiko Sawada
ublisher, do the last upgrade preparation work (e.g. copying the latest sequence values if requried), and then change the application so that new transactions come to the subscriber. I remember the blog post about Knock doing a similar process to upgrade the clusters with minimal downtime[1]. Regards, [1] https://knock.app/blog/zero-downtime-postgres-upgrades -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Revive num_dead_tuples column of pg_stat_progress_vacuum

2024-06-03 Thread Masahiko Sawada
. Feedback is very welcome. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com 0001-Revive-num_dead_tuples-column-of-pg_stat_progress_va.patch Description: Binary data

Re: First draft of PG 17 release notes

2024-05-21 Thread Masahiko Sawada
count is 188, > which is similar to recent releases: > I found a typo: s/pg_statstatement/pg_stat_statement/ I've attached a patch to fix it. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com fix_pg_stat_statements.patch Description: Binary data

Re: PostgreSQL 17 Beta 1 release announcement draft

2024-05-20 Thread Masahiko Sawada
https://www.postgresql.org/message-id/CANWCAZYqWibTRCWs5mV57mLj1A0nbKX-eV5G%2Bd-KmBOGHTVY-w%40mail.gmail.com -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: Pgoutput not capturing the generated columns

2024-05-20 Thread Masahiko Sawada
ld support it too. Otherwise, we end up filling the target columns data with NULL during the initial tablesync but with replicated data during the streaming changes. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: Lowering the minimum value for maintenance_work_mem

2024-05-19 Thread Masahiko Sawada
gment size of DSA, i.e. DSA_MIN_SEGMENT_SIZE. So we can lower the minimum maintenance_work_mem down to 256kB, from a vacuum perspective. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: pg_trgm comparison bug on cross-architecture replication due to different char implementation

2024-05-14 Thread Masahiko Sawada
tect potential data incompatibility due to changing 'char' to 'unsigned char'? I think these new tests would be useful also for users to check if they really need to reindex indexes due to such changes. Also we fix pg_trgm so that it uses 'unsigned char' in PG18. Users who upgraded to PG18 can run the new amcheck tests on the primary as well as the standby. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: First draft of PG 17 release notes

2024-05-14 Thread Masahiko Sawada
On Thu, May 9, 2024 at 10:48 PM Bruce Momjian wrote: > > On Thu, May 9, 2024 at 02:17:12PM +0900, Masahiko Sawada wrote: > > Hi, > > > > > Also, please consider the following item: > > > > - Improve eviction algorithm in ReorderBuffer using max-heap for man

Re: Fix parallel vacuum buffer usage reporting

2024-05-10 Thread Masahiko Sawada
On Fri, May 10, 2024 at 7:26 PM Nazir Bilal Yavuz wrote: > > Hi, > > Thank you for working on this! > > On Wed, 1 May 2024 at 06:37, Masahiko Sawada wrote: > > > > Thank you for further testing! I've pushed the patch. > > I realized a behaviour change

Re: First draft of PG 17 release notes

2024-05-08 Thread Masahiko Sawada
ound (e255b646a) Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: Fix parallel vacuum buffer usage reporting

2024-05-08 Thread Masahiko Sawada
On Fri, May 3, 2024 at 3:41 PM Anthonin Bonnefoy wrote: > > On Wed, May 1, 2024 at 5:37 AM Masahiko Sawada wrote: >> >> Thank you for further testing! I've pushed the patch. > > Thanks! > > Here is the rebased version for the follow-up patch removing VacuumPage &g

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-05-07 Thread Masahiko Sawada
On Wed, May 1, 2024 at 4:29 PM John Naylor wrote: > > On Thu, Apr 25, 2024 at 8:36 AM Masahiko Sawada wrote: > > > > On Mon, Apr 15, 2024 at 6:12 PM John Naylor wrote: > > > > - RT_KEY_GET_SHIFT is not covered for key=0: > > > > > > https://an

Re: Fix parallel vacuum buffer usage reporting

2024-04-30 Thread Masahiko Sawada
st-patch, we have 16081. The 9 additional block hits come from vacuum > accessing catalog tables like pg_class or pg_class_oid_index. > Thank you for further testing! I've pushed the patch. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: pg_trgm comparison bug on cross-architecture replication due to different char implementation

2024-04-30 Thread Masahiko Sawada
always being signed and write code that will misbehave on arm machines. For example, since logical replication should behave correctly even in cross-arch replication all developers need to be aware of that. I thought of using the -funsigned-char (or -fsigned-char) compiler flag to avoid

Re: pg_trgm comparison bug on cross-architecture replication due to different char implementation

2024-04-30 Thread Masahiko Sawada
On Tue, Apr 30, 2024 at 12:37 PM Tom Lane wrote: > > Masahiko Sawada writes: > > On Tue, Apr 23, 2024 at 11:57 PM Tom Lane wrote: > >> Reject as not a bug. Discourage people from thinking that physical > >> replication will work across architectures. > > >

Re: pg_trgm comparison bug on cross-architecture replication due to different char implementation

2024-04-29 Thread Masahiko Sawada
plication will work across architectures. While cross-arch physical replication is not supported, I think having architecture dependent differences is not good and It's legitimate to fix it. FYI the 'char' data type comparison is done as though char is unsigned. I've attached a small patch to fix i

Re: Fix parallel vacuum buffer usage reporting

2024-04-29 Thread Masahiko Sawada
d the commit message. I realized that parallel vacuum was introduced in pg13 but buffer usage reporting in VACUUM command was implemented in pg15. Therefore, in pg13 and pg14, VACUUM (PARALLEL) is available but VACUUM (PARALLEL, VERBOSE) doesn't show the buffer usage report. Autovacuum does show

Re: New committers: Melanie Plageman, Richard Guo

2024-04-27 Thread Masahiko Sawada
ss and few reverts! > Congratulations to both! Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-04-25 Thread Masahiko Sawada
On Thu, Apr 25, 2024 at 1:38 PM Masahiko Sawada wrote: > > On Thu, Apr 25, 2024 at 12:17 PM John Naylor wrote: > > > > On Thu, Apr 25, 2024 at 9:50 AM Masahiko Sawada > > wrote: > > > > > > > I saw a SIGSEGV there when using tidstore to write a f

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-04-24 Thread Masahiko Sawada
On Thu, Apr 25, 2024 at 12:17 PM John Naylor wrote: > > On Thu, Apr 25, 2024 at 9:50 AM Masahiko Sawada wrote: > > > > > I saw a SIGSEGV there when using tidstore to write a fix for something > > > else. > > > Patch attached. > > &g

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-04-24 Thread Masahiko Sawada
k_offsets(). If these are annoying, we can remove the cases of array[1] and array[1,2]. I've attached a new patch. In addition to the new test case I mentioned, I've added some new comments and removed an unnecessary added line in test_tidstore.sql. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com 0001-radixtree-Fix-SIGSEGV-at-update-of-embeddable-value-.patch Description: Binary data

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-04-24 Thread Masahiko Sawada
9 > https://anarazel.de/postgres/cov/16-vs-HEAD-2024-04-14/src/backend/access/common/tidstore.c.gcov.html#L234 > > Maybe we could experiment with using 1MB for shared, and something > smaller for local. I've confirmed that the local and shared tidstore with small max sizes such as 4kB and 1MB work

Re: Fix parallel vacuum buffer usage reporting

2024-04-24 Thread Masahiko Sawada
ning on a temp table always shows the number of dirtied buffers being 0, which seems to be a bug. The patch (a) will resolve it as well. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: promotion related handling in pg_sync_replication_slots()

2024-04-23 Thread Masahiko Sawada
On Tue, Apr 23, 2024 at 12:37 PM Amit Kapila wrote: > > On Mon, Apr 22, 2024 at 7:04 PM Masahiko Sawada wrote: > > > > On Mon, Apr 22, 2024 at 9:02 PM shveta malik wrote: > > > > > > Thanks for the patch, the changes look good Amit. Please find the mer

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-04-22 Thread Masahiko Sawada
dating the temporary slots even if they are shown as active, which could confuse users. Do we want to somehow deal with it? Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: promotion related handling in pg_sync_replication_slots()

2024-04-22 Thread Masahiko Sawada
ld be a large side effect of this function for users. Also, if users want to have a process periodically calling pg_sync_replication_slots() instead of the slotsync worker, it doesn't support a case where we create a temp not-ready slot and turn it into a persistent slot if it's ready for sync. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)

2024-04-17 Thread Masahiko Sawada
On Wed, Apr 17, 2024 at 4:28 PM torikoshia wrote: > > On 2024-04-16 13:16, Masahiko Sawada wrote: > > On Tue, Apr 2, 2024 at 7:34 PM torikoshia > > wrote: > >> > >> On 2024-04-01 11:31, Masahiko Sawada wrote: > >> > On Fri, M

Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)

2024-04-15 Thread Masahiko Sawada
On Tue, Apr 2, 2024 at 7:34 PM torikoshia wrote: > > On 2024-04-01 11:31, Masahiko Sawada wrote: > > On Fri, Mar 29, 2024 at 11:54 AM torikoshia > > wrote: > >> > >> On 2024-03-28 21:54, Masahiko Sawada wrote: > >> > On Thu,

Re: Improve eviction algorithm in ReorderBuffer

2024-04-11 Thread Masahiko Sawada
ze 0 in the heap is better and I'm not sure that reducing these branches really contributes to the performance improvements.. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: Improve eviction algorithm in ReorderBuffer

2024-04-11 Thread Masahiko Sawada
On Thu, Apr 11, 2024 at 11:52 AM Masahiko Sawada wrote: > > We can see 2% ~ 3% performance regressions compared to the current > HEAD, but it's much smaller than I expected. Given that we can make > the code simple, I think we can go with this direction. Pushed the patch and reverte

Re: Improve eviction algorithm in ReorderBuffer

2024-04-10 Thread Masahiko Sawada
On Thu, Apr 11, 2024 at 10:32 AM Masahiko Sawada wrote: > > Hi, > > Sorry for the late reply, I took two days off. > > On Thu, Apr 11, 2024 at 6:20 AM Heikki Linnakangas wrote: > > > > On 10/04/2024 08:31, Amit Kapila wrote: > > > On Wed, Apr 10, 2024 at 11

Re: Improve eviction algorithm in ReorderBuffer

2024-04-10 Thread Masahiko Sawada
d revert the changes for binaryheap. Regarding the patch, I'm not sure we can remove the MAX_HEAP_TXN_COUNT_THRESHOLD logic because otherwise we need to remove and add the txn node (i.e. O(log n)) for every memory update. I'm concerned it could cause some performance degradation in a case where there are not many transactions being decoded. I'll do performance tests, and share the results soon. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: PostgreSQL 17 Release Management Team & Feature Freeze

2024-04-08 Thread Masahiko Sawada
items. Each item should be reviewed by other than the author and the committer. We may want to set aside a specific period for intensive testing. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: Improve eviction algorithm in ReorderBuffer

2024-04-08 Thread Masahiko Sawada
o it looks better to me. When it comes to performance overhead, I mentioned that there is some regression in the current binaryheap even without indexing. Since function calling contributed to the regression, inlining some functions reduced some overheads. For example, inlining set_node() and repl

Re: Improve tab completion for ALTER DEFAULT PRIVILEGE and ALTER TABLE

2024-04-07 Thread Masahiko Sawada
On Fri, Apr 5, 2024 at 1:18 AM vignesh C wrote: > > On Tue, 2 Apr 2024 at 13:08, Masahiko Sawada wrote: > > > > On Mon, Apr 1, 2024 at 10:41 PM vignesh C wrote: > > > > > > On Thu, 28 Mar 2024 at 13:05, Masahiko Sawada > > > wrote: > > >

Re: Improve eviction algorithm in ReorderBuffer

2024-04-05 Thread Masahiko Sawada
On Fri, Apr 5, 2024 at 2:55 AM Jeff Davis wrote: > > On Thu, 2024-04-04 at 17:28 +0900, Masahiko Sawada wrote: > > > Perhaps it's not worth the effort though, if performance is already > > > good enough? > > > > Yeah, it would be better to measure the overhead

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-04-04 Thread Masahiko Sawada
On Thu, Apr 4, 2024 at 5:36 PM Amit Kapila wrote: > > On Thu, Apr 4, 2024 at 1:32 PM Masahiko Sawada wrote: > > > > On Thu, Apr 4, 2024 at 1:34 PM Bharath Rupireddy > > wrote: > > > > > > On Thu, Apr 4, 2024 at 9:42 AM Masahiko Sawada >

Re: Improve eviction algorithm in ReorderBuffer

2024-04-04 Thread Masahiko Sawada
On Thu, Apr 4, 2024 at 1:54 PM Jeff Davis wrote: > > On Thu, 2024-04-04 at 09:31 +0900, Masahiko Sawada wrote: > > IIUC, with your suggestion, sift_{up|down} needs to update the > > heap_index field as well. Does it mean that the caller needs to pass > > the address of hea

Re: Synchronizing slots from primary to standby

2024-04-04 Thread Masahiko Sawada
d there was no serialized snapshot at restart_lsn, and the slotsync worker called LogicalSlotAdvanceAndCheckSnapState(). However no slot properties were changed even after the function and it set slot_updated = true. So it starts the next slot synchronization after 200ms. It seems to me that w

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-04-04 Thread Masahiko Sawada
On Thu, Apr 4, 2024 at 1:34 PM Bharath Rupireddy wrote: > > On Thu, Apr 4, 2024 at 9:42 AM Masahiko Sawada wrote: > > > > @@ -1368,6 +1416,7 @@ ShutDownSlotSync(void) > > if (SlotSyncCtx->pid == InvalidPid) > > { &

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-04-03 Thread Masahiko Sawada
en releasing the slot? It seems to contradict the fact that inactive_since is updated when releasing or restoring the slot. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: Improve eviction algorithm in ReorderBuffer

2024-04-03 Thread Masahiko Sawada
n go > in 18. IIUC, with your suggestion, sift_{up|down} needs to update the heap_index field as well. Does it mean that the caller needs to pass the address of heap_index down to sift_{up|down}? Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: Improve tab completion for ALTER DEFAULT PRIVILEGE and ALTER TABLE

2024-04-02 Thread Masahiko Sawada
On Mon, Apr 1, 2024 at 10:41 PM vignesh C wrote: > > On Thu, 28 Mar 2024 at 13:05, Masahiko Sawada wrote: > > > > Hi, > > > > Thank you for the patch! > > > > On Mon, Jul 3, 2023 at 12:12 AM vignesh C wrote: > > > > > > Hi, >

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-04-01 Thread Masahiko Sawada
since the last synchronization, by checking the remote slot's inactive_since. This idea seems to handle these cases I mentioned unless I'm missing something, but it requires for the slotsync worker to update inactive_since in a different way than other parameters. Or a simple solution is that the slotsync worker updates inactive_since as it does for non-synced slots, and disables timeout-based slot invalidation for synced slots. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: Add new error_action COPY ON_ERROR "log"

2024-04-01 Thread Masahiko Sawada
On Mon, Apr 1, 2024 at 10:03 AM Masahiko Sawada wrote: > > On Sat, Mar 30, 2024 at 11:05 PM Bharath Rupireddy > wrote: > > > > On Thu, Mar 28, 2024 at 6:31 PM Masahiko Sawada > > wrote: > > > > > > That is, > > > since the LOG_VERBOSITY

Re: Improve eviction algorithm in ReorderBuffer

2024-03-31 Thread Masahiko Sawada
On Mon, Apr 1, 2024 at 11:26 AM Masahiko Sawada wrote: > > On Fri, Mar 29, 2024 at 7:37 PM Amit Kapila wrote: > > > > On Fri, Mar 29, 2024 at 12:13 PM Masahiko Sawada > > wrote: > > > > > > On Fri, Mar 29, 2024 at 2:09 PM vignesh C wrote: > > &g

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-31 Thread Masahiko Sawada
On Fri, Mar 29, 2024 at 4:21 PM John Naylor wrote: > > On Thu, Mar 28, 2024 at 12:55 PM Masahiko Sawada > wrote: > > I think the patch is in good shape. Do you have other comments or > > suggestions, John? > > --- a/doc/src/sgml/config.sgml > +++ b/doc/src/sgml/con

Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)

2024-03-31 Thread Masahiko Sawada
On Fri, Mar 29, 2024 at 11:54 AM torikoshia wrote: > > On 2024-03-28 21:54, Masahiko Sawada wrote: > > On Thu, Mar 28, 2024 at 9:38 PM torikoshia > > wrote: > >> > >> On 2024-03-28 10:20, Masahiko Sawada wrote: > >> > Hi, > >>

Re: Improve eviction algorithm in ReorderBuffer

2024-03-31 Thread Masahiko Sawada
. There are other comments in the same file that are one line and start with lowercase. I've just submitted the updated patches[1] Regards, [1] https://www.postgresql.org/message-id/CAD21AoA6%3D%2BtL%3DbtB_s9N%2BcZK7tKz1W%3DPQyNq72nzjUcdyE%2BwZw%40mail.gmail.com -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: Improve eviction algorithm in ReorderBuffer

2024-03-31 Thread Masahiko Sawada
On Fri, Mar 29, 2024 at 7:37 PM Amit Kapila wrote: > > On Fri, Mar 29, 2024 at 12:13 PM Masahiko Sawada > wrote: > > > > On Fri, Mar 29, 2024 at 2:09 PM vignesh C wrote: > > > > > > On Tue, 26 Mar 2024 at 10:05, Masahiko Sawada > > > wrote:

Re: Add new error_action COPY ON_ERROR "log"

2024-03-31 Thread Masahiko Sawada
On Sat, Mar 30, 2024 at 11:05 PM Bharath Rupireddy wrote: > > On Thu, Mar 28, 2024 at 6:31 PM Masahiko Sawada wrote: > > > > That is, > > since the LOG_VERBOSITY option is an enum parameter, it might make > > more sense to require the value, instead of making the v

Re: Improve eviction algorithm in ReorderBuffer

2024-03-29 Thread Masahiko Sawada
On Fri, Mar 29, 2024 at 2:09 PM vignesh C wrote: > > On Tue, 26 Mar 2024 at 10:05, Masahiko Sawada wrote: > > > > On Thu, Mar 14, 2024 at 12:02 PM Masahiko Sawada > > wrote: > > > > > > > > > I've attached new version patches. > > >

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-29 Thread Masahiko Sawada
On Thu, Mar 28, 2024 at 6:15 PM John Naylor wrote: > > On Thu, Mar 28, 2024 at 12:55 PM Masahiko Sawada > wrote: > > > > Pushed the refactoring patch. > > > > I've attached the rebased vacuum improvement patch for cfbot. I > > mentioned in the commit mes

Re: Add new error_action COPY ON_ERROR "log"

2024-03-28 Thread Masahiko Sawada
but it made me think that the same might be true for the LOG_VERBOSITY option. That is, since the LOG_VERBOSITY option is an enum parameter, it might make more sense to require the value, instead of making the value optional. For example, the following command could not be obvious for users: COPY test FROM stdin (ON_ERROR ignore, LOG_VERBOSITY); Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)

2024-03-28 Thread Masahiko Sawada
On Thu, Mar 28, 2024 at 9:38 PM torikoshia wrote: > > On 2024-03-28 10:20, Masahiko Sawada wrote: > > Hi, > > > > On Thu, Jan 18, 2024 at 5:33 PM Masahiko Sawada > > wrote: > >> > >> On Thu, Jan 18, 2024 at 4:59 PM Alexander Korotkov > >&

Re: Improve tab completion for ALTER DEFAULT PRIVILEGE and ALTER TABLE

2024-03-28 Thread Masahiko Sawada
ault privileges for role masahiko grant all on tables to public [tab] It's already supported in the GRANT statement. > > 4) "DATA TYPE" was missing in "ALTER TABLE table-name ALTER COLUMN > column-name SET" like in: > ALTER TABLE t1 ALTER COLUMN c1 SET DATA TYPE text; > +1. The patch looks good to me, so pushed. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-27 Thread Masahiko Sawada
On Wed, Mar 27, 2024 at 5:43 PM Masahiko Sawada wrote: > > On Wed, Mar 27, 2024 at 9:25 AM John Naylor wrote: > > > > On Mon, Mar 25, 2024 at 8:07 PM Masahiko Sawada > > wrote: > > > > > > On Mon, Mar 25, 2024 at 3:25 PM John Naylor > > > w

Re: Fix parallel vacuum buffer usage reporting

2024-03-27 Thread Masahiko Sawada
(long long) bufferusage.shared_blks_dirtied); appendStringInfo(, _("system usage: %s"), pg_rusage_show()); Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)

2024-03-27 Thread Masahiko Sawada
Hi, On Thu, Jan 18, 2024 at 5:33 PM Masahiko Sawada wrote: > > On Thu, Jan 18, 2024 at 4:59 PM Alexander Korotkov > wrote: > > > > On Thu, Jan 18, 2024 at 4:16 AM torikoshia > > wrote: > > > On 2024-01-18 10:10, jian he wrote: > > > >

Re: Add new error_action COPY ON_ERROR "log"

2024-03-27 Thread Masahiko Sawada
On Thu, Mar 28, 2024 at 2:49 AM Bharath Rupireddy wrote: > > On Wed, Mar 27, 2024 at 7:42 AM Masahiko Sawada wrote: > > > > I think that there are two options to handle it: > > > > 1. change COPY grammar to accept DEFAULT as an option value. > > 2. change t

Re: Skip collecting decoded changes of already-aborted transactions

2024-03-27 Thread Masahiko Sawada
On Wed, Mar 27, 2024 at 8:49 PM Ajin Cherian wrote: > > > > On Mon, Mar 18, 2024 at 7:50 PM Masahiko Sawada wrote: >> >> >> In addition to these changes, I've made some changes to the latest >> patch. Here is the summary: >> >> - Use txn_flag

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-27 Thread Masahiko Sawada
On Wed, Mar 27, 2024 at 9:25 AM John Naylor wrote: > > On Mon, Mar 25, 2024 at 8:07 PM Masahiko Sawada wrote: > > > > On Mon, Mar 25, 2024 at 3:25 PM John Naylor wrote: > > > > > > On Fri, Mar 22, 2024 at 12:20 PM Masahiko Sawada > > > wrote

Re: Add new error_action COPY ON_ERROR "log"

2024-03-26 Thread Masahiko Sawada
On Tue, Mar 26, 2024 at 6:36 PM Bharath Rupireddy wrote: > > On Tue, Mar 26, 2024 at 1:46 PM Masahiko Sawada wrote: > > > > Thank you for updating the patch! Looks good to me. > > > > Please find the attached patch. I've made some changes for the > > docume

Re: Add new error_action COPY ON_ERROR "log"

2024-03-26 Thread Masahiko Sawada
On Tue, Mar 26, 2024 at 3:04 PM Bharath Rupireddy wrote: > > On Tue, Mar 26, 2024 at 9:56 AM Masahiko Sawada wrote: > > > > > > errmsg("data type incompatibility at line %llu for column %s: \"%s\"", > > > > > > I guess it would be be

Re: Improve eviction algorithm in ReorderBuffer

2024-03-25 Thread Masahiko Sawada
On Thu, Mar 14, 2024 at 12:02 PM Masahiko Sawada wrote: > > > I've attached new version patches. Since the previous patch conflicts with the current HEAD, I've attached the rebased patches. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com v10-0001-Make-b

Re: Add new error_action COPY ON_ERROR "log"

2024-03-25 Thread Masahiko Sawada
On Tue, Mar 26, 2024 at 12:23 PM Bharath Rupireddy wrote: > > On Tue, Mar 26, 2024 at 7:16 AM Masahiko Sawada wrote: > > > > > Please see the attached v9 patch set. > > > > Thank you for updating the patch! The patch mostly looks good to me. > > H

Re: Add new error_action COPY ON_ERROR "log"

2024-03-25 Thread Masahiko Sawada
On Mon, Mar 25, 2024 at 8:21 PM Bharath Rupireddy wrote: > > On Mon, Mar 25, 2024 at 10:42 AM Masahiko Sawada > wrote: > > > > The current approach, eliminating the duplicated information in > > CONTEXT, seems good to me. > > Thanks for looking into it. > &

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-25 Thread Masahiko Sawada
On Mon, Mar 25, 2024 at 3:25 PM John Naylor wrote: > > On Fri, Mar 22, 2024 at 12:20 PM Masahiko Sawada > wrote: > > > > On Thu, Mar 21, 2024 at 7:48 PM John Naylor wrote: > > > > v77-0001 > > > > > > - dead_items = (VacDeadItems *)

Re: Add new error_action COPY ON_ERROR "log"

2024-03-24 Thread Masahiko Sawada
ility at line %llu for column %s: null input", + (unsigned long long) cstate->cur_lineno, + cstate->cur_attname)); + How can we reach this path? It seems we don't cover this path by the tests. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-24 Thread Masahiko Sawada
On Mon, Mar 25, 2024 at 10:13 AM Tom Lane wrote: > > Masahiko Sawada writes: > > On Mon, Mar 25, 2024 at 1:53 AM Tom Lane wrote: > >> I think the point here is that if you start with an arbitrary > >> non-negative shift value, the preceding loop may in fact decre

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-24 Thread Masahiko Sawada
o here, why not let the compiler > hard-code that? > > - n4->chunks[0] = RT_GET_KEY_CHUNK(key, shift); > + n4->chunks[0] = RT_GET_KEY_CHUNK(key, 0); Sounds like a good solution. I've attached the patch for that. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com fix_radixtree.patch Description: Binary data

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-21 Thread Masahiko Sawada
On Thu, Mar 21, 2024 at 7:48 PM John Naylor wrote: > > On Thu, Mar 21, 2024 at 4:03 PM Masahiko Sawada wrote: > > > > I've looked into this idea further. Overall, it looks clean and I > > don't see any problem so far in terms of integration with lazy vacuum. > &g

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-21 Thread Masahiko Sawada
On Thu, Mar 21, 2024 at 4:35 PM John Naylor wrote: > > On Thu, Mar 21, 2024 at 1:11 PM Masahiko Sawada wrote: > > > Or we can have a new function for dsa.c to set the initial and max > > segment size (or either one) to the existing DSA area so that > > TidSto

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-21 Thread Masahiko Sawada
On Thu, Mar 21, 2024 at 3:10 PM Masahiko Sawada wrote: > > On Thu, Mar 21, 2024 at 12:40 PM John Naylor wrote: > > > > On Thu, Mar 21, 2024 at 9:37 AM Masahiko Sawada > > wrote: > > > > > > On Wed, Mar 20, 2024 at 11:19 PM John Naylor > > >

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-21 Thread Masahiko Sawada
On Thu, Mar 21, 2024 at 12:40 PM John Naylor wrote: > > On Thu, Mar 21, 2024 at 9:37 AM Masahiko Sawada wrote: > > > > On Wed, Mar 20, 2024 at 11:19 PM John Naylor > > wrote: > > > Are they (the blocks to be precise) really out of order? The VALUES >

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-20 Thread Masahiko Sawada
On Wed, Mar 20, 2024 at 11:19 PM John Naylor wrote: > > On Wed, Mar 20, 2024 at 8:30 PM Masahiko Sawada wrote: > > I forgot to report the results. Yes, I did some tests where I inserted > > many TIDs to make the tidstore use several GB memory. I did two cases: > > &

Re: Have pg_basebackup write "dbname" in "primary_conninfo"?

2024-03-20 Thread Masahiko Sawada
:5431/db5''' > > case 16: > ./pg_basebackup -D test10 -X s -P -R -d > "postgresql://localhost:5431,127.0.0.1:5431/db5" > primary_conninfo will have dbname=db5 > > case 17: > ./pg_basebackup -D test10 -X s -P -R -d > "postgresql:///db6?host=localhost=5431" > primary_conninfo will have dbname=db6 > > case 18: > ./pg_basebackup -D test10 -p 5431 -X s -P -R -d > "postgresql:///db7?host=/home/vignesh/postgres/inst/bin" > primary_conninfo will have dbname=db7 > > case 19: > ./pg_basebackup -D test10 -p 5431 -X s -P -R -d > "postgresql:///db8?host=%2Fhome%2Fvignesh%2Fpostgres%2Finst%2Fbin" > primary_conninfo will have dbname=db8 > > In these cases, the database name specified will be written to the > conf file. The test results look good to me. Thank you for the tests! These results look good to me too. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: Have pg_basebackup write "dbname" in "primary_conninfo"?

2024-03-20 Thread Masahiko Sawada
ults(), > which > is called from conninfo_array_parse(). However, it is done at > fe-connect.c:5956 - the > expand_dbname has already been done at that time. This means there is no > chance > that PGDATABASE is parsed as an expanded style. > Thank you for pointing it out. I tested the use of PGDATABASE with pg_basebackup and somehow missed the fact you explained. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-20 Thread Masahiko Sawada
On Wed, Mar 20, 2024 at 3:48 PM John Naylor wrote: > > On Thu, Mar 14, 2024 at 12:06 PM Masahiko Sawada > wrote: > > > > On Thu, Mar 14, 2024 at 1:29 PM John Naylor wrote: > > > Locally (not CI), we should try big inputs to make sure we can > > > a

  1   2   3   4   5   6   7   8   9   10   >