Re: Fix parallel vacuum buffer usage reporting

2024-05-10 Thread Masahiko Sawada
On Fri, May 10, 2024 at 7:26 PM Nazir Bilal Yavuz wrote: > > Hi, > > Thank you for working on this! > > On Wed, 1 May 2024 at 06:37, Masahiko Sawada wrote: > > > > Thank you for further testing! I've pushed the patch. > > I realized a behaviour change

Re: First draft of PG 17 release notes

2024-05-08 Thread Masahiko Sawada
ound (e255b646a) Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: Fix parallel vacuum buffer usage reporting

2024-05-08 Thread Masahiko Sawada
On Fri, May 3, 2024 at 3:41 PM Anthonin Bonnefoy wrote: > > On Wed, May 1, 2024 at 5:37 AM Masahiko Sawada wrote: >> >> Thank you for further testing! I've pushed the patch. > > Thanks! > > Here is the rebased version for the follow-up patch removing VacuumPage &g

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-05-07 Thread Masahiko Sawada
On Wed, May 1, 2024 at 4:29 PM John Naylor wrote: > > On Thu, Apr 25, 2024 at 8:36 AM Masahiko Sawada wrote: > > > > On Mon, Apr 15, 2024 at 6:12 PM John Naylor wrote: > > > > - RT_KEY_GET_SHIFT is not covered for key=0: > > > > > > https://an

Re: Fix parallel vacuum buffer usage reporting

2024-04-30 Thread Masahiko Sawada
st-patch, we have 16081. The 9 additional block hits come from vacuum > accessing catalog tables like pg_class or pg_class_oid_index. > Thank you for further testing! I've pushed the patch. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: pg_trgm comparison bug on cross-architecture replication due to different char implementation

2024-04-30 Thread Masahiko Sawada
always being signed and write code that will misbehave on arm machines. For example, since logical replication should behave correctly even in cross-arch replication all developers need to be aware of that. I thought of using the -funsigned-char (or -fsigned-char) compiler flag to avoid

Re: pg_trgm comparison bug on cross-architecture replication due to different char implementation

2024-04-30 Thread Masahiko Sawada
On Tue, Apr 30, 2024 at 12:37 PM Tom Lane wrote: > > Masahiko Sawada writes: > > On Tue, Apr 23, 2024 at 11:57 PM Tom Lane wrote: > >> Reject as not a bug. Discourage people from thinking that physical > >> replication will work across architectures. > > >

Re: pg_trgm comparison bug on cross-architecture replication due to different char implementation

2024-04-29 Thread Masahiko Sawada
plication will work across architectures. While cross-arch physical replication is not supported, I think having architecture dependent differences is not good and It's legitimate to fix it. FYI the 'char' data type comparison is done as though char is unsigned. I've attached a small patch to fix i

Re: Fix parallel vacuum buffer usage reporting

2024-04-29 Thread Masahiko Sawada
d the commit message. I realized that parallel vacuum was introduced in pg13 but buffer usage reporting in VACUUM command was implemented in pg15. Therefore, in pg13 and pg14, VACUUM (PARALLEL) is available but VACUUM (PARALLEL, VERBOSE) doesn't show the buffer usage report. Autovacuum does show

Re: New committers: Melanie Plageman, Richard Guo

2024-04-27 Thread Masahiko Sawada
ss and few reverts! > Congratulations to both! Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-04-25 Thread Masahiko Sawada
On Thu, Apr 25, 2024 at 1:38 PM Masahiko Sawada wrote: > > On Thu, Apr 25, 2024 at 12:17 PM John Naylor wrote: > > > > On Thu, Apr 25, 2024 at 9:50 AM Masahiko Sawada > > wrote: > > > > > > > I saw a SIGSEGV there when using tidstore to write a f

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-04-24 Thread Masahiko Sawada
On Thu, Apr 25, 2024 at 12:17 PM John Naylor wrote: > > On Thu, Apr 25, 2024 at 9:50 AM Masahiko Sawada wrote: > > > > > I saw a SIGSEGV there when using tidstore to write a fix for something > > > else. > > > Patch attached. > > &g

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-04-24 Thread Masahiko Sawada
k_offsets(). If these are annoying, we can remove the cases of array[1] and array[1,2]. I've attached a new patch. In addition to the new test case I mentioned, I've added some new comments and removed an unnecessary added line in test_tidstore.sql. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com 0001-radixtree-Fix-SIGSEGV-at-update-of-embeddable-value-.patch Description: Binary data

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-04-24 Thread Masahiko Sawada
9 > https://anarazel.de/postgres/cov/16-vs-HEAD-2024-04-14/src/backend/access/common/tidstore.c.gcov.html#L234 > > Maybe we could experiment with using 1MB for shared, and something > smaller for local. I've confirmed that the local and shared tidstore with small max sizes such as 4kB and 1MB work

Re: Fix parallel vacuum buffer usage reporting

2024-04-24 Thread Masahiko Sawada
ning on a temp table always shows the number of dirtied buffers being 0, which seems to be a bug. The patch (a) will resolve it as well. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: promotion related handling in pg_sync_replication_slots()

2024-04-23 Thread Masahiko Sawada
On Tue, Apr 23, 2024 at 12:37 PM Amit Kapila wrote: > > On Mon, Apr 22, 2024 at 7:04 PM Masahiko Sawada wrote: > > > > On Mon, Apr 22, 2024 at 9:02 PM shveta malik wrote: > > > > > > Thanks for the patch, the changes look good Amit. Please find the mer

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-04-22 Thread Masahiko Sawada
dating the temporary slots even if they are shown as active, which could confuse users. Do we want to somehow deal with it? Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: promotion related handling in pg_sync_replication_slots()

2024-04-22 Thread Masahiko Sawada
ld be a large side effect of this function for users. Also, if users want to have a process periodically calling pg_sync_replication_slots() instead of the slotsync worker, it doesn't support a case where we create a temp not-ready slot and turn it into a persistent slot if it's ready for sync. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)

2024-04-17 Thread Masahiko Sawada
On Wed, Apr 17, 2024 at 4:28 PM torikoshia wrote: > > On 2024-04-16 13:16, Masahiko Sawada wrote: > > On Tue, Apr 2, 2024 at 7:34 PM torikoshia > > wrote: > >> > >> On 2024-04-01 11:31, Masahiko Sawada wrote: > >> > On Fri, M

Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)

2024-04-15 Thread Masahiko Sawada
On Tue, Apr 2, 2024 at 7:34 PM torikoshia wrote: > > On 2024-04-01 11:31, Masahiko Sawada wrote: > > On Fri, Mar 29, 2024 at 11:54 AM torikoshia > > wrote: > >> > >> On 2024-03-28 21:54, Masahiko Sawada wrote: > >> > On Thu,

Re: Improve eviction algorithm in ReorderBuffer

2024-04-11 Thread Masahiko Sawada
ze 0 in the heap is better and I'm not sure that reducing these branches really contributes to the performance improvements.. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: Improve eviction algorithm in ReorderBuffer

2024-04-11 Thread Masahiko Sawada
On Thu, Apr 11, 2024 at 11:52 AM Masahiko Sawada wrote: > > We can see 2% ~ 3% performance regressions compared to the current > HEAD, but it's much smaller than I expected. Given that we can make > the code simple, I think we can go with this direction. Pushed the patch and reverte

Re: Improve eviction algorithm in ReorderBuffer

2024-04-10 Thread Masahiko Sawada
On Thu, Apr 11, 2024 at 10:32 AM Masahiko Sawada wrote: > > Hi, > > Sorry for the late reply, I took two days off. > > On Thu, Apr 11, 2024 at 6:20 AM Heikki Linnakangas wrote: > > > > On 10/04/2024 08:31, Amit Kapila wrote: > > > On Wed, Apr 10, 2024 at 11

Re: Improve eviction algorithm in ReorderBuffer

2024-04-10 Thread Masahiko Sawada
d revert the changes for binaryheap. Regarding the patch, I'm not sure we can remove the MAX_HEAP_TXN_COUNT_THRESHOLD logic because otherwise we need to remove and add the txn node (i.e. O(log n)) for every memory update. I'm concerned it could cause some performance degradation in a case where there are not many transactions being decoded. I'll do performance tests, and share the results soon. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: PostgreSQL 17 Release Management Team & Feature Freeze

2024-04-08 Thread Masahiko Sawada
items. Each item should be reviewed by other than the author and the committer. We may want to set aside a specific period for intensive testing. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: Improve eviction algorithm in ReorderBuffer

2024-04-08 Thread Masahiko Sawada
o it looks better to me. When it comes to performance overhead, I mentioned that there is some regression in the current binaryheap even without indexing. Since function calling contributed to the regression, inlining some functions reduced some overheads. For example, inlining set_node() and repl

Re: Improve tab completion for ALTER DEFAULT PRIVILEGE and ALTER TABLE

2024-04-07 Thread Masahiko Sawada
On Fri, Apr 5, 2024 at 1:18 AM vignesh C wrote: > > On Tue, 2 Apr 2024 at 13:08, Masahiko Sawada wrote: > > > > On Mon, Apr 1, 2024 at 10:41 PM vignesh C wrote: > > > > > > On Thu, 28 Mar 2024 at 13:05, Masahiko Sawada > > > wrote: > > >

Re: Improve eviction algorithm in ReorderBuffer

2024-04-05 Thread Masahiko Sawada
On Fri, Apr 5, 2024 at 2:55 AM Jeff Davis wrote: > > On Thu, 2024-04-04 at 17:28 +0900, Masahiko Sawada wrote: > > > Perhaps it's not worth the effort though, if performance is already > > > good enough? > > > > Yeah, it would be better to measure the overhead

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-04-04 Thread Masahiko Sawada
On Thu, Apr 4, 2024 at 5:36 PM Amit Kapila wrote: > > On Thu, Apr 4, 2024 at 1:32 PM Masahiko Sawada wrote: > > > > On Thu, Apr 4, 2024 at 1:34 PM Bharath Rupireddy > > wrote: > > > > > > On Thu, Apr 4, 2024 at 9:42 AM Masahiko Sawada >

Re: Improve eviction algorithm in ReorderBuffer

2024-04-04 Thread Masahiko Sawada
On Thu, Apr 4, 2024 at 1:54 PM Jeff Davis wrote: > > On Thu, 2024-04-04 at 09:31 +0900, Masahiko Sawada wrote: > > IIUC, with your suggestion, sift_{up|down} needs to update the > > heap_index field as well. Does it mean that the caller needs to pass > > the address of hea

Re: Synchronizing slots from primary to standby

2024-04-04 Thread Masahiko Sawada
d there was no serialized snapshot at restart_lsn, and the slotsync worker called LogicalSlotAdvanceAndCheckSnapState(). However no slot properties were changed even after the function and it set slot_updated = true. So it starts the next slot synchronization after 200ms. It seems to me that w

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-04-04 Thread Masahiko Sawada
On Thu, Apr 4, 2024 at 1:34 PM Bharath Rupireddy wrote: > > On Thu, Apr 4, 2024 at 9:42 AM Masahiko Sawada wrote: > > > > @@ -1368,6 +1416,7 @@ ShutDownSlotSync(void) > > if (SlotSyncCtx->pid == InvalidPid) > > { &

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-04-03 Thread Masahiko Sawada
en releasing the slot? It seems to contradict the fact that inactive_since is updated when releasing or restoring the slot. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: Improve eviction algorithm in ReorderBuffer

2024-04-03 Thread Masahiko Sawada
n go > in 18. IIUC, with your suggestion, sift_{up|down} needs to update the heap_index field as well. Does it mean that the caller needs to pass the address of heap_index down to sift_{up|down}? Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: Improve tab completion for ALTER DEFAULT PRIVILEGE and ALTER TABLE

2024-04-02 Thread Masahiko Sawada
On Mon, Apr 1, 2024 at 10:41 PM vignesh C wrote: > > On Thu, 28 Mar 2024 at 13:05, Masahiko Sawada wrote: > > > > Hi, > > > > Thank you for the patch! > > > > On Mon, Jul 3, 2023 at 12:12 AM vignesh C wrote: > > > > > > Hi, >

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-04-01 Thread Masahiko Sawada
since the last synchronization, by checking the remote slot's inactive_since. This idea seems to handle these cases I mentioned unless I'm missing something, but it requires for the slotsync worker to update inactive_since in a different way than other parameters. Or a simple solution is that the slotsync worker updates inactive_since as it does for non-synced slots, and disables timeout-based slot invalidation for synced slots. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: Add new error_action COPY ON_ERROR "log"

2024-04-01 Thread Masahiko Sawada
On Mon, Apr 1, 2024 at 10:03 AM Masahiko Sawada wrote: > > On Sat, Mar 30, 2024 at 11:05 PM Bharath Rupireddy > wrote: > > > > On Thu, Mar 28, 2024 at 6:31 PM Masahiko Sawada > > wrote: > > > > > > That is, > > > since the LOG_VERBOSITY

Re: Improve eviction algorithm in ReorderBuffer

2024-03-31 Thread Masahiko Sawada
On Mon, Apr 1, 2024 at 11:26 AM Masahiko Sawada wrote: > > On Fri, Mar 29, 2024 at 7:37 PM Amit Kapila wrote: > > > > On Fri, Mar 29, 2024 at 12:13 PM Masahiko Sawada > > wrote: > > > > > > On Fri, Mar 29, 2024 at 2:09 PM vignesh C wrote: > > &g

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-31 Thread Masahiko Sawada
On Fri, Mar 29, 2024 at 4:21 PM John Naylor wrote: > > On Thu, Mar 28, 2024 at 12:55 PM Masahiko Sawada > wrote: > > I think the patch is in good shape. Do you have other comments or > > suggestions, John? > > --- a/doc/src/sgml/config.sgml > +++ b/doc/src/sgml/con

Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)

2024-03-31 Thread Masahiko Sawada
On Fri, Mar 29, 2024 at 11:54 AM torikoshia wrote: > > On 2024-03-28 21:54, Masahiko Sawada wrote: > > On Thu, Mar 28, 2024 at 9:38 PM torikoshia > > wrote: > >> > >> On 2024-03-28 10:20, Masahiko Sawada wrote: > >> > Hi, > >>

Re: Improve eviction algorithm in ReorderBuffer

2024-03-31 Thread Masahiko Sawada
. There are other comments in the same file that are one line and start with lowercase. I've just submitted the updated patches[1] Regards, [1] https://www.postgresql.org/message-id/CAD21AoA6%3D%2BtL%3DbtB_s9N%2BcZK7tKz1W%3DPQyNq72nzjUcdyE%2BwZw%40mail.gmail.com -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: Improve eviction algorithm in ReorderBuffer

2024-03-31 Thread Masahiko Sawada
On Fri, Mar 29, 2024 at 7:37 PM Amit Kapila wrote: > > On Fri, Mar 29, 2024 at 12:13 PM Masahiko Sawada > wrote: > > > > On Fri, Mar 29, 2024 at 2:09 PM vignesh C wrote: > > > > > > On Tue, 26 Mar 2024 at 10:05, Masahiko Sawada > > > wrote:

Re: Add new error_action COPY ON_ERROR "log"

2024-03-31 Thread Masahiko Sawada
On Sat, Mar 30, 2024 at 11:05 PM Bharath Rupireddy wrote: > > On Thu, Mar 28, 2024 at 6:31 PM Masahiko Sawada wrote: > > > > That is, > > since the LOG_VERBOSITY option is an enum parameter, it might make > > more sense to require the value, instead of making the v

Re: Improve eviction algorithm in ReorderBuffer

2024-03-29 Thread Masahiko Sawada
On Fri, Mar 29, 2024 at 2:09 PM vignesh C wrote: > > On Tue, 26 Mar 2024 at 10:05, Masahiko Sawada wrote: > > > > On Thu, Mar 14, 2024 at 12:02 PM Masahiko Sawada > > wrote: > > > > > > > > > I've attached new version patches. > > >

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-29 Thread Masahiko Sawada
On Thu, Mar 28, 2024 at 6:15 PM John Naylor wrote: > > On Thu, Mar 28, 2024 at 12:55 PM Masahiko Sawada > wrote: > > > > Pushed the refactoring patch. > > > > I've attached the rebased vacuum improvement patch for cfbot. I > > mentioned in the commit mes

Re: Add new error_action COPY ON_ERROR "log"

2024-03-28 Thread Masahiko Sawada
but it made me think that the same might be true for the LOG_VERBOSITY option. That is, since the LOG_VERBOSITY option is an enum parameter, it might make more sense to require the value, instead of making the value optional. For example, the following command could not be obvious for users: COPY test FROM stdin (ON_ERROR ignore, LOG_VERBOSITY); Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)

2024-03-28 Thread Masahiko Sawada
On Thu, Mar 28, 2024 at 9:38 PM torikoshia wrote: > > On 2024-03-28 10:20, Masahiko Sawada wrote: > > Hi, > > > > On Thu, Jan 18, 2024 at 5:33 PM Masahiko Sawada > > wrote: > >> > >> On Thu, Jan 18, 2024 at 4:59 PM Alexander Korotkov > >&

Re: Improve tab completion for ALTER DEFAULT PRIVILEGE and ALTER TABLE

2024-03-28 Thread Masahiko Sawada
ault privileges for role masahiko grant all on tables to public [tab] It's already supported in the GRANT statement. > > 4) "DATA TYPE" was missing in "ALTER TABLE table-name ALTER COLUMN > column-name SET" like in: > ALTER TABLE t1 ALTER COLUMN c1 SET DATA TYPE text; > +1. The patch looks good to me, so pushed. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-27 Thread Masahiko Sawada
On Wed, Mar 27, 2024 at 5:43 PM Masahiko Sawada wrote: > > On Wed, Mar 27, 2024 at 9:25 AM John Naylor wrote: > > > > On Mon, Mar 25, 2024 at 8:07 PM Masahiko Sawada > > wrote: > > > > > > On Mon, Mar 25, 2024 at 3:25 PM John Naylor > > > w

Re: Fix parallel vacuum buffer usage reporting

2024-03-27 Thread Masahiko Sawada
(long long) bufferusage.shared_blks_dirtied); appendStringInfo(, _("system usage: %s"), pg_rusage_show()); Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)

2024-03-27 Thread Masahiko Sawada
Hi, On Thu, Jan 18, 2024 at 5:33 PM Masahiko Sawada wrote: > > On Thu, Jan 18, 2024 at 4:59 PM Alexander Korotkov > wrote: > > > > On Thu, Jan 18, 2024 at 4:16 AM torikoshia > > wrote: > > > On 2024-01-18 10:10, jian he wrote: > > > >

Re: Add new error_action COPY ON_ERROR "log"

2024-03-27 Thread Masahiko Sawada
On Thu, Mar 28, 2024 at 2:49 AM Bharath Rupireddy wrote: > > On Wed, Mar 27, 2024 at 7:42 AM Masahiko Sawada wrote: > > > > I think that there are two options to handle it: > > > > 1. change COPY grammar to accept DEFAULT as an option value. > > 2. change t

Re: Skip collecting decoded changes of already-aborted transactions

2024-03-27 Thread Masahiko Sawada
On Wed, Mar 27, 2024 at 8:49 PM Ajin Cherian wrote: > > > > On Mon, Mar 18, 2024 at 7:50 PM Masahiko Sawada wrote: >> >> >> In addition to these changes, I've made some changes to the latest >> patch. Here is the summary: >> >> - Use txn_flag

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-27 Thread Masahiko Sawada
On Wed, Mar 27, 2024 at 9:25 AM John Naylor wrote: > > On Mon, Mar 25, 2024 at 8:07 PM Masahiko Sawada wrote: > > > > On Mon, Mar 25, 2024 at 3:25 PM John Naylor wrote: > > > > > > On Fri, Mar 22, 2024 at 12:20 PM Masahiko Sawada > > > wrote

Re: Add new error_action COPY ON_ERROR "log"

2024-03-26 Thread Masahiko Sawada
On Tue, Mar 26, 2024 at 6:36 PM Bharath Rupireddy wrote: > > On Tue, Mar 26, 2024 at 1:46 PM Masahiko Sawada wrote: > > > > Thank you for updating the patch! Looks good to me. > > > > Please find the attached patch. I've made some changes for the > > docume

Re: Add new error_action COPY ON_ERROR "log"

2024-03-26 Thread Masahiko Sawada
On Tue, Mar 26, 2024 at 3:04 PM Bharath Rupireddy wrote: > > On Tue, Mar 26, 2024 at 9:56 AM Masahiko Sawada wrote: > > > > > > errmsg("data type incompatibility at line %llu for column %s: \"%s\"", > > > > > > I guess it would be be

Re: Improve eviction algorithm in ReorderBuffer

2024-03-25 Thread Masahiko Sawada
On Thu, Mar 14, 2024 at 12:02 PM Masahiko Sawada wrote: > > > I've attached new version patches. Since the previous patch conflicts with the current HEAD, I've attached the rebased patches. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com v10-0001-Make-b

Re: Add new error_action COPY ON_ERROR "log"

2024-03-25 Thread Masahiko Sawada
On Tue, Mar 26, 2024 at 12:23 PM Bharath Rupireddy wrote: > > On Tue, Mar 26, 2024 at 7:16 AM Masahiko Sawada wrote: > > > > > Please see the attached v9 patch set. > > > > Thank you for updating the patch! The patch mostly looks good to me. > > H

Re: Add new error_action COPY ON_ERROR "log"

2024-03-25 Thread Masahiko Sawada
On Mon, Mar 25, 2024 at 8:21 PM Bharath Rupireddy wrote: > > On Mon, Mar 25, 2024 at 10:42 AM Masahiko Sawada > wrote: > > > > The current approach, eliminating the duplicated information in > > CONTEXT, seems good to me. > > Thanks for looking into it. > &

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-25 Thread Masahiko Sawada
On Mon, Mar 25, 2024 at 3:25 PM John Naylor wrote: > > On Fri, Mar 22, 2024 at 12:20 PM Masahiko Sawada > wrote: > > > > On Thu, Mar 21, 2024 at 7:48 PM John Naylor wrote: > > > > v77-0001 > > > > > > - dead_items = (VacDeadItems *)

Re: Add new error_action COPY ON_ERROR "log"

2024-03-24 Thread Masahiko Sawada
ility at line %llu for column %s: null input", + (unsigned long long) cstate->cur_lineno, + cstate->cur_attname)); + How can we reach this path? It seems we don't cover this path by the tests. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-24 Thread Masahiko Sawada
On Mon, Mar 25, 2024 at 10:13 AM Tom Lane wrote: > > Masahiko Sawada writes: > > On Mon, Mar 25, 2024 at 1:53 AM Tom Lane wrote: > >> I think the point here is that if you start with an arbitrary > >> non-negative shift value, the preceding loop may in fact decre

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-24 Thread Masahiko Sawada
o here, why not let the compiler > hard-code that? > > - n4->chunks[0] = RT_GET_KEY_CHUNK(key, shift); > + n4->chunks[0] = RT_GET_KEY_CHUNK(key, 0); Sounds like a good solution. I've attached the patch for that. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com fix_radixtree.patch Description: Binary data

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-21 Thread Masahiko Sawada
On Thu, Mar 21, 2024 at 7:48 PM John Naylor wrote: > > On Thu, Mar 21, 2024 at 4:03 PM Masahiko Sawada wrote: > > > > I've looked into this idea further. Overall, it looks clean and I > > don't see any problem so far in terms of integration with lazy vacuum. > &g

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-21 Thread Masahiko Sawada
On Thu, Mar 21, 2024 at 4:35 PM John Naylor wrote: > > On Thu, Mar 21, 2024 at 1:11 PM Masahiko Sawada wrote: > > > Or we can have a new function for dsa.c to set the initial and max > > segment size (or either one) to the existing DSA area so that > > TidSto

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-21 Thread Masahiko Sawada
On Thu, Mar 21, 2024 at 3:10 PM Masahiko Sawada wrote: > > On Thu, Mar 21, 2024 at 12:40 PM John Naylor wrote: > > > > On Thu, Mar 21, 2024 at 9:37 AM Masahiko Sawada > > wrote: > > > > > > On Wed, Mar 20, 2024 at 11:19 PM John Naylor > > >

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-21 Thread Masahiko Sawada
On Thu, Mar 21, 2024 at 12:40 PM John Naylor wrote: > > On Thu, Mar 21, 2024 at 9:37 AM Masahiko Sawada wrote: > > > > On Wed, Mar 20, 2024 at 11:19 PM John Naylor > > wrote: > > > Are they (the blocks to be precise) really out of order? The VALUES >

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-20 Thread Masahiko Sawada
On Wed, Mar 20, 2024 at 11:19 PM John Naylor wrote: > > On Wed, Mar 20, 2024 at 8:30 PM Masahiko Sawada wrote: > > I forgot to report the results. Yes, I did some tests where I inserted > > many TIDs to make the tidstore use several GB memory. I did two cases: > > &

Re: Have pg_basebackup write "dbname" in "primary_conninfo"?

2024-03-20 Thread Masahiko Sawada
:5431/db5''' > > case 16: > ./pg_basebackup -D test10 -X s -P -R -d > "postgresql://localhost:5431,127.0.0.1:5431/db5" > primary_conninfo will have dbname=db5 > > case 17: > ./pg_basebackup -D test10 -X s -P -R -d > "postgresql:///db6?host=localhost=5431" > primary_conninfo will have dbname=db6 > > case 18: > ./pg_basebackup -D test10 -p 5431 -X s -P -R -d > "postgresql:///db7?host=/home/vignesh/postgres/inst/bin" > primary_conninfo will have dbname=db7 > > case 19: > ./pg_basebackup -D test10 -p 5431 -X s -P -R -d > "postgresql:///db8?host=%2Fhome%2Fvignesh%2Fpostgres%2Finst%2Fbin" > primary_conninfo will have dbname=db8 > > In these cases, the database name specified will be written to the > conf file. The test results look good to me. Thank you for the tests! These results look good to me too. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: Have pg_basebackup write "dbname" in "primary_conninfo"?

2024-03-20 Thread Masahiko Sawada
ults(), > which > is called from conninfo_array_parse(). However, it is done at > fe-connect.c:5956 - the > expand_dbname has already been done at that time. This means there is no > chance > that PGDATABASE is parsed as an expanded style. > Thank you for pointing it out. I tested the use of PGDATABASE with pg_basebackup and somehow missed the fact you explained. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-20 Thread Masahiko Sawada
On Wed, Mar 20, 2024 at 3:48 PM John Naylor wrote: > > On Thu, Mar 14, 2024 at 12:06 PM Masahiko Sawada > wrote: > > > > On Thu, Mar 14, 2024 at 1:29 PM John Naylor wrote: > > > Locally (not CI), we should try big inputs to make sure we can > > > a

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-19 Thread Masahiko Sawada
On Tue, Mar 19, 2024 at 6:40 PM John Naylor wrote: > > On Tue, Mar 19, 2024 at 10:24 AM Masahiko Sawada > wrote: > > > > On Tue, Mar 19, 2024 at 8:35 AM John Naylor wrote: > > > > > > On Mon, Mar 18, 2024 at 11:12 AM Masahiko Sawada > > > wrot

Re: New Table Access Methods for Multi and Single Inserts

2024-03-18 Thread Masahiko Sawada
etDescr(state->rel), +); + And the execution times are: 1588.984 ms 1591.618 ms 1582.519 ms Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-18 Thread Masahiko Sawada
On Tue, Mar 19, 2024 at 8:35 AM John Naylor wrote: > > On Mon, Mar 18, 2024 at 11:12 AM Masahiko Sawada > wrote: > > > > On Sun, Mar 17, 2024 at 11:46 AM John Naylor > > wrote: > > > > Random offsets is what I was thinking of (if made distinct and

Re: Have pg_basebackup write "dbname" in "primary_conninfo"?

2024-03-18 Thread Masahiko Sawada
On Thu, Mar 14, 2024 at 11:46 PM vignesh C wrote: > > On Thu, 14 Mar 2024 at 15:49, Amit Kapila wrote: > > > > On Thu, Mar 14, 2024 at 1:45 PM Masahiko Sawada > > wrote: > > > > > > On Thu, Mar 14, 2024 at 2:27 PM Amit Kapila > > > wro

Re: Skip collecting decoded changes of already-aborted transactions

2024-03-18 Thread Masahiko Sawada
On Fri, Mar 15, 2024 at 1:21 PM Ajin Cherian wrote: > > > > On Fri, Mar 15, 2024 at 3:17 PM Masahiko Sawada wrote: >> >> >> I resumed working on this item. I've attached the new version patch. >> >> I rebased the patch to the current HEAD and updated

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-17 Thread Masahiko Sawada
On Sun, Mar 17, 2024 at 11:46 AM John Naylor wrote: > > On Fri, Mar 15, 2024 at 9:17 PM Masahiko Sawada wrote: > > > > On Fri, Mar 15, 2024 at 4:36 PM John Naylor wrote: > > > > > > On Thu, Mar 14, 2024 at 7:04 PM Masahiko Sawada > > > w

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-15 Thread Masahiko Sawada
On Fri, Mar 15, 2024 at 4:36 PM John Naylor wrote: > > On Thu, Mar 14, 2024 at 7:04 PM Masahiko Sawada wrote: > > > > On Thu, Mar 14, 2024 at 6:55 PM John Naylor wrote: > > > > > > On Thu, Mar 14, 2024 at 12:06 PM Masahiko Sawada > > > wrote: &g

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-14 Thread Masahiko Sawada
On Thu, Mar 14, 2024 at 9:03 PM Masahiko Sawada wrote: > > On Thu, Mar 14, 2024 at 6:55 PM John Naylor wrote: > > > > On Thu, Mar 14, 2024 at 12:06 PM Masahiko Sawada > > wrote: > > > > > > On Thu, Mar 14, 2024 at 1:29 PM John Naylor > > >

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-14 Thread Masahiko Sawada
On Thu, Mar 14, 2024 at 6:55 PM John Naylor wrote: > > On Thu, Mar 14, 2024 at 12:06 PM Masahiko Sawada > wrote: > > > > On Thu, Mar 14, 2024 at 1:29 PM John Naylor wrote: > > > Okay, here's an another idea: Change test_lookup_tids() to be more > > > gen

Re: Have pg_basebackup write "dbname" in "primary_conninfo"?

2024-03-14 Thread Masahiko Sawada
On Thu, Mar 14, 2024 at 2:27 PM Amit Kapila wrote: > > On Thu, Mar 14, 2024 at 5:57 AM Masahiko Sawada wrote: > > > > This fact makes me think that the slotsync worker might be able to > > accept the primary_conninfo value even if there is no

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-13 Thread Masahiko Sawada
On Thu, Mar 14, 2024 at 1:29 PM John Naylor wrote: > > On Thu, Mar 14, 2024 at 8:53 AM Masahiko Sawada wrote: > > > > On Thu, Mar 14, 2024 at 9:59 AM John Naylor wrote: > > > > BTW do we still want to test the tidstore by using a combination of > > > &

Re: Improve eviction algorithm in ReorderBuffer

2024-03-13 Thread Masahiko Sawada
On Wed, Mar 13, 2024 at 11:23 AM Peter Smith wrote: > > On Wed, Mar 13, 2024 at 12:48 PM Masahiko Sawada > wrote: > > > > On Wed, Mar 13, 2024 at 10:15 AM Peter Smith wrote: > > > > > > On Tue, Mar 12, 2024 at 4:23 PM Masahiko Sawada > > > wro

Re: Improve eviction algorithm in ReorderBuffer

2024-03-13 Thread Masahiko Sawada
rderBufferMaybeChangeNoMaxHeap(ReorderBuffer *rb) > +{ > + if (binaryheap_empty(rb->txn_heap)) > + return; > + > ... > + if (binaryheap_size(rb->txn_heap) < REORDER_BUFFER_MEM_TRACK_THRESHOLD * > 0.9) > + binaryheap_reset(rb->txn_heap); > +} Agreed. I removed t

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-13 Thread Masahiko Sawada
On Thu, Mar 14, 2024 at 9:59 AM John Naylor wrote: > > On Wed, Mar 13, 2024 at 9:29 PM Masahiko Sawada wrote: > > > > On Wed, Mar 13, 2024 at 8:05 PM John Naylor wrote: > > > > > > On Wed, Mar 13, 2024 at 8:39 AM Masahiko Sawada > > > wrote: > &

Re: Have pg_basebackup write "dbname" in "primary_conninfo"?

2024-03-13 Thread Masahiko Sawada
-R --primary-coninfo-ext "host=2.2.2.2 dbname=postgres" will produce the connection string that is based on -d option value but is overwritten by --primary-conninfo-ext option value, which will be like: host=2.2.2.2 dbname=postgres port= This option might help not only for users who wan

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-13 Thread Masahiko Sawada
On Wed, Mar 13, 2024 at 8:05 PM John Naylor wrote: > > On Wed, Mar 13, 2024 at 8:39 AM Masahiko Sawada wrote: > > > As I mentioned above, if we implement the test cases in C, we can use > > the debug-build array in the test code. And we won't use it in AND/OR > > ope

Re: Improve eviction algorithm in ReorderBuffer

2024-03-12 Thread Masahiko Sawada
On Wed, Mar 13, 2024 at 10:15 AM Peter Smith wrote: > > On Tue, Mar 12, 2024 at 4:23 PM Masahiko Sawada wrote: > > > > On Fri, Mar 8, 2024 at 12:58 PM Peter Smith wrote: > > > > ... > > > > > 5. > > > > > + * > > >

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-12 Thread Masahiko Sawada
On Tue, Mar 12, 2024 at 7:34 PM John Naylor wrote: > > On Mon, Mar 11, 2024 at 3:13 PM Masahiko Sawada wrote: > > > > On Mon, Mar 11, 2024 at 12:20 PM John Naylor > > wrote: > > > > > > On Thu, Mar 7, 2024 at 10:35 PM Masahiko Sawada > > >

Re: Improve eviction algorithm in ReorderBuffer

2024-03-11 Thread Masahiko Sawada
On Fri, Mar 8, 2024 at 12:58 PM Peter Smith wrote: > > On Thu, Mar 7, 2024 at 2:16 PM Masahiko Sawada wrote: > > > > On Tue, Mar 5, 2024 at 3:28 PM Peter Smith wrote: > > > > > > > 4a. > > > The comment in simplehash.h says > > > *

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-11 Thread Masahiko Sawada
On Mon, Mar 11, 2024 at 5:13 PM Masahiko Sawada wrote: > > In the latest (v69) patch: > > - squashed v68-0005 and v68-0006 patches. > - removed most of the changes in v68-0007 patch. > - addressed above review comments in v69-0002 patch. > - v69-0003, 0004, and 0005 are

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-11 Thread Masahiko Sawada
On Mon, Mar 11, 2024 at 12:20 PM John Naylor wrote: > > On Thu, Mar 7, 2024 at 10:35 PM Masahiko Sawada wrote: > > > > I've attached the remaining patches for CI. I've made some minor > > changes in separate patches and drafted the commit message for > > tidstore

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-07 Thread Masahiko Sawada
On Fri, Mar 8, 2024 at 10:04 AM John Naylor wrote: > > On Thu, Mar 7, 2024 at 11:15 PM Masahiko Sawada wrote: > > > > It looks like it requires a link with pgport_srv but I'm not sure. It > > seems that the recent commit 1f1d73a8b breaks CI, Windows - Server > >

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-07 Thread Masahiko Sawada
On Thu, Mar 7, 2024 at 8:06 PM John Naylor wrote: > > On Thu, Mar 7, 2024 at 4:47 PM Masahiko Sawada wrote: > > > > On Thu, Mar 7, 2024 at 6:37 PM John Naylor wrote: > > > > $ git grep 'link_with: pgport_srv' > > > src/test/modules/test_radixt

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-07 Thread Masahiko Sawada
On Thu, Mar 7, 2024 at 8:06 PM John Naylor wrote: > > On Thu, Mar 7, 2024 at 4:47 PM Masahiko Sawada wrote: > > > > On Thu, Mar 7, 2024 at 6:37 PM John Naylor wrote: > > > > $ git grep 'link_with: pgport_srv' > > > src/test/modules/test_radixt

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-07 Thread Masahiko Sawada
could be the culprit. The test_radixtree/meson.build is the sole extension that explicitly specifies a link with pgport_srv. I think we can get rid of it as I've also confirmed the build still fine even without it. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-06 Thread Masahiko Sawada
On Thu, Mar 7, 2024 at 4:21 PM John Naylor wrote: > > On Thu, Mar 7, 2024 at 2:14 PM Masahiko Sawada wrote: > > > > On Thu, Mar 7, 2024 at 4:01 PM John Naylor wrote: > > > > > > On Thu, Mar 7, 2024 at 1:27 PM Masahiko Sawada > > > wrote: > &

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-06 Thread Masahiko Sawada
On Thu, Mar 7, 2024 at 4:01 PM John Naylor wrote: > > On Thu, Mar 7, 2024 at 1:27 PM Masahiko Sawada wrote: > > > > On Thu, Mar 7, 2024 at 3:20 PM John Naylor wrote: > > > > > > On Thu, Mar 7, 2024 at 12:59 PM John Naylor > > > wrote: > &

Re: Add new error_action COPY ON_ERROR "log"

2024-03-06 Thread Masahiko Sawada
levels for example to get actual soft error contents. One question I have is; do we want to write multiple NOTICE messages for one row if the row has malformed data on some columns? Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-06 Thread Masahiko Sawada
On Thu, Mar 7, 2024 at 3:27 PM Masahiko Sawada wrote: > > On Thu, Mar 7, 2024 at 3:20 PM John Naylor wrote: > > > > > > In addition, olingo and grassquit are showing different kinds of > > "AddressSanitizer: odr-violation" errors, which I'm not sure what t

  1   2   3   4   5   6   7   8   9   10   >