Re: Popcount optimization using AVX512

2024-03-29 Thread Nathan Bossart
On Thu, Mar 28, 2024 at 10:03:04PM +, Amonson, Paul D wrote: >> * I think we need to verify there isn't a huge performance regression for >> smaller arrays. IIUC those will still require an AVX512 instruction or >> two as well as a function call, which might add some noticeable overhead.

RE: Popcount optimization using AVX512

2024-03-29 Thread Amonson, Paul D
> -Original Message- > > Cool. I think we should run the benchmarks again to be safe, though. Ok, sure go ahead. :) > >> I forgot to mention that I also want to understand whether we can > >> actually assume availability of XGETBV when CPUID says we support > >> AVX512: > > > > You

Re: Possibility to disable `ALTER SYSTEM`

2024-03-29 Thread Robert Haas
On Fri, Mar 29, 2024 at 10:48 AM Bruce Momjian wrote: > On Fri, Mar 29, 2024 at 08:46:33AM -0400, Robert Haas wrote: > > On Thu, Mar 28, 2024 at 3:33 PM Bruce Momjian wrote: > > > I am fine with moving ahead. I thought my later emails explaining we > > > have to be careful communicated that. >

Re: Popcount optimization using AVX512

2024-03-29 Thread Nathan Bossart
On Fri, Mar 29, 2024 at 10:59:40AM -0500, Nathan Bossart wrote: > It might be nice if we conditionally built pg_popcount_avx512.o in autoconf > builds, too, but AFAICT we still need to wrap most of that code with > macros, so I'm not sure it's worth the trouble. I'll take another look at >

RE: Popcount optimization using AVX512

2024-03-29 Thread Amonson, Paul D
> On Thu, Mar 28, 2024 at 11:10:33PM +0100, Alvaro Herrera wrote: > > We don't do MSVC via autoconf/Make. We used to have a special build > > framework for MSVC which parsed Makefiles to produce "solution" files, > > but it was removed as soon as Meson was mature enough to build. See > > commit

Re: To what extent should tests rely on VACUUM ANALYZE?

2024-03-29 Thread Tom Lane
I wrote: > I experimented with the attached modified version of the patch, > which probes just after the relevant VACUUMs and reduces the > crankiness of ConditionalLockBufferForCleanup a bit to more nearly > approximate what we're likely to see in the buildfarm. Sigh, forgot to attach the patch,

Re: Allowing DESC for a PRIMARY KEY column

2024-03-29 Thread Mitar
Hi! On Fri, Mar 29, 2024 at 9:41 PM Tom Lane wrote: > You would need a lot stronger case than "I didn't bother checking > whether I really need this". Thanks! I have tested it this way (based on your example): create table t (id int not null, revision int not null); create unique index on t

Re: LLVM 18

2024-03-29 Thread Thomas Munro
On Sat, Mar 30, 2024 at 7:07 AM Christoph Berg wrote: > Ubuntu in their infinite wisdom have switched to LLVM 18 as default > for their upcoming 24.04 "noble" LTS release while Debian is still > defaulting to 16. I'm now seeing LLVM crashes on the 4 architectures > we support on noble. > > Should

Re: Statistics Import and Export

2024-03-29 Thread Corey Huinker
> > > There's still a failure in the pg_upgrade TAP test. One seems to be > ordering, so perhaps we need to ORDER BY the attribute number. Others > seem to be missing relstats and I'm not sure why yet. I suggest doing > some manual pg_upgrade tests and comparing the before/after dumps to > see if

Re: Statistics Import and Export

2024-03-29 Thread Corey Huinker
> > (I've not read the patch yet, but I assume the switch works like > other pg_dump filters in that you can apply it on the restore > side?) > Correct. It follows the existing --no-something pattern.

Re: LLVM 18

2024-03-29 Thread Christoph Berg
Re: Thomas Munro > By the way, while testing on my Debian system with apt.llvm.org > packages, I discovered that we crash with its latest llvm-18 package, > namely: Ubuntu in their infinite wisdom have switched to LLVM 18 as default for their upcoming 24.04 "noble" LTS release while Debian is

Re: Why is parula failing?

2024-03-29 Thread Tom Lane
David Rowley writes: > On Wed, 27 Mar 2024 at 18:28, Tom Lane wrote: >> Let's wait a bit to see if it fails in HEAD ... but if not, would >> it be reasonable to back-patch the additional debugging output? > I think REL_16_STABLE has told us that it's not an auto-vacuum issue. > I'm uncertain

Re: Popcount optimization using AVX512

2024-03-29 Thread Nathan Bossart
On Fri, Mar 29, 2024 at 03:08:28PM -0500, Nathan Bossart wrote: >> +#if defined(HAVE__GET_CPUID) >> +__get_cpuid_count(7, 0, [0], [1], [2], [3]); >> +#elif defined(HAVE__CPUID) >> +__cpuidex(exx, 7, 0); > > Is there any reason we can't use __get_cpuid() and __cpuid() here, given > the

Re: Statistics Import and Export

2024-03-29 Thread Stephen Frost
Greetings, On Fri, Mar 29, 2024 at 11:05 Jeff Davis wrote: > On Fri, 2024-03-29 at 05:32 -0400, Corey Huinker wrote: > > 0002: > > - All relstats and attrstats calls are now their own statement > > instead of a compound statement > > - moved the archive TOC entry from post-data back to

Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index.

2024-03-29 Thread Noah Misch
On Fri, Mar 29, 2024 at 02:17:08PM -0400, Peter Geoghegan wrote: > On Mon, Mar 25, 2024 at 2:24 PM Noah Misch wrote: > > I wasn't thinking about changing the pre-v17 bt_right_page_check_scankey() > > code. I got interested in this area when I saw the interaction of the new > > "first key on the

Re: BitmapHeapScan streaming read user and prelim refactoring

2024-03-29 Thread Thomas Munro
On Sat, Mar 30, 2024 at 12:40 PM Tomas Vondra wrote: > Sorry, I meant the prefetch (readahead) built into ZFS. I may be wrong > but I don't think the regular RA (in linux kernel) works for ZFS, right? Right, it separate page cache ("ARC") and prefetch settings:

Re: Statistics Import and Export

2024-03-29 Thread Stephen Frost
Greetings, On Fri, Mar 29, 2024 at 19:35 Jeff Davis wrote: > On Fri, 2024-03-29 at 18:02 -0400, Stephen Frost wrote: > > I’d certainly think “with stats” would be the preferred default of > > our users. > > I'm concerned there could still be paths that lead to an error. For > pg_restore, or

Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index.

2024-03-29 Thread Peter Geoghegan
On Mon, Mar 25, 2024 at 2:24 PM Noah Misch wrote: > I wasn't thinking about changing the pre-v17 bt_right_page_check_scankey() > code. I got interested in this area when I saw the interaction of the new > "first key on the next page" logic with bt_right_page_check_scankey(). The > patch made

Re: Popcount optimization using AVX512

2024-03-29 Thread Nathan Bossart
Okay, here is a slightly different approach that I've dubbed the "maximum assumption" approach. In short, I wanted to see how much we could simplify the patch by making all possibly-reasonable assumptions about the compiler and CPU. These include: * If the compiler understands AVX512

Re: Security lessons from liblzma

2024-03-29 Thread Thomas Munro
On Sat, Mar 30, 2024 at 11:37 AM Bruce Momjian wrote: > You might have seen reports today about a very complex exploit added to > recent versions of liblzma. Fortunately, it was only enabled two months > ago and has not been pushed to most stable operating systems like Debian > and Ubuntu. The

Re: Security lessons from liblzma

2024-03-29 Thread Daniel Gustafsson
> On 29 Mar 2024, at 23:59, Andres Freund wrote: > On 2024-03-29 18:37:24 -0400, Bruce Momjian wrote: >> Now, we don't take pull requests, and all our committers are known >> individuals, but this might have cautionary lessons for us. > > I am doubtful that every committer would find something

Re: Remove excessive trailing semicolons

2024-03-29 Thread Daniel Gustafsson
> On 29 Mar 2024, at 10:14, Richard Guo wrote: > Noticed some newly introduced excessive trailing semicolons: > > $ git grep -E ";;$" -- *.c *.h > src/include/lib/radixtree.h:int deletepos = > slot - n4->children;; >

Re: BitmapHeapScan streaming read user and prelim refactoring

2024-03-29 Thread Thomas Munro
On Sat, Mar 30, 2024 at 12:34 PM Tomas Vondra wrote: > Hmmm. I admit I didn't think about the "always prefetch" flag too much, > but I did imagine it'd only affect some places (e.g. BHS, but not for > sequential scans). If it could be done by lowering the combine limit, > that could work too - in

Re: Popcount optimization using AVX512

2024-03-29 Thread Nathan Bossart
On Thu, Mar 28, 2024 at 10:29:47PM +, Amonson, Paul D wrote: > I see in the meson.build you added the new file twice? > > @@ -7,6 +7,7 @@ pgport_sources = [ >'noblock.c', >'path.c', >'pg_bitutils.c', > + 'pg_popcount_avx512.c', >'pg_strong_random.c', >'pgcheckdir.c', >

Re: Popcount optimization using AVX512

2024-03-29 Thread Tom Lane
Nathan Bossart writes: >> I see google web references to the xgetbv instruction as far back as 2009 >> for Intel 64 bit HW and 2010 for AMD 64bit HW, maybe you could test for >> _xgetbv() MSVC built-in. How far back do you need to go? > Hm. It seems unlikely that a compiler would understand

Re: Popcount optimization using AVX512

2024-03-29 Thread Nathan Bossart
On Fri, Mar 29, 2024 at 12:30:14PM -0400, Tom Lane wrote: > Nathan Bossart writes: >>> I see google web references to the xgetbv instruction as far back as 2009 >>> for Intel 64 bit HW and 2010 for AMD 64bit HW, maybe you could test for >>> _xgetbv() MSVC built-in. How far back do you need to go?

Re: BitmapHeapScan streaming read user and prelim refactoring

2024-03-29 Thread Thomas Munro
On Sat, Mar 30, 2024 at 10:39 AM Thomas Munro wrote: > On Sat, Mar 30, 2024 at 4:53 AM Tomas Vondra > wrote: > > ... Maybe there should be some flag to force > > issuing fadvise even for sequential patterns, perhaps at the tablespace > > level? ... > > Yeah, I've wondered about trying harder to

Re: [EXTERNAL] Re: Add non-blocking version of PQcancel

2024-03-29 Thread Noah Misch
On Fri, Mar 29, 2024 at 09:17:55AM +0100, Jelte Fennema-Nio wrote: > On Thu, 28 Mar 2024 at 19:03, Tom Lane wrote: > > Could we make this test bulletproof by using an injection point? > > If not, I remain of the opinion that we're better off without it. > > Possibly, and if so, I agree that

Re: Security lessons from liblzma

2024-03-29 Thread Andres Freund
Hi, On 2024-03-29 18:37:24 -0400, Bruce Momjian wrote: > You might have seen reports today about a very complex exploit added to > recent versions of liblzma. Fortunately, it was only enabled two months > ago and has not been pushed to most stable operating systems like Debian > and Ubuntu. The

Re: Statistics Import and Export

2024-03-29 Thread Corey Huinker
On Fri, Mar 29, 2024 at 7:34 PM Jeff Davis wrote: > On Fri, 2024-03-29 at 18:02 -0400, Stephen Frost wrote: > > I’d certainly think “with stats” would be the preferred default of > > our users. > > I'm concerned there could still be paths that lead to an error. For > pg_restore, or when loading

Re: Combine Prune and Freeze records emitted by vacuum

2024-03-29 Thread Heikki Linnakangas
On 29/03/2024 07:04, Melanie Plageman wrote: On Thu, Mar 28, 2024 at 11:07:10AM -0400, Melanie Plageman wrote: These comments could use another pass. I had added some extra (probably redundant) content when I thought I was refactoring it a certain way and then changed my mind. Attached is a

Re: Streaming I/O, vectored I/O (WIP)

2024-03-29 Thread Heikki Linnakangas
On 29/03/2024 09:01, Thomas Munro wrote: On Fri, Mar 29, 2024 at 9:45 AM Heikki Linnakangas wrote: master (213c959a29):8.0 s streaming-api v13: 9.5 s Hmm, that's not great, and I think I know one factor that has confounded my investigation and the conflicting reports

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-29 Thread Euler Taveira
On Fri, Mar 29, 2024, at 9:44 AM, Alexander Korotkov wrote: > This generally makes sense, but I'm not sure about this. The > milliseconds timeout was used initially but received critics in [1]. Alexander, I see why you changed the patch. Peter suggested to use an interval but you proposed

Re: Popcount optimization using AVX512

2024-03-29 Thread Nathan Bossart
On Fri, Mar 29, 2024 at 02:13:12PM -0500, Nathan Bossart wrote: > * If the compiler understands AVX512 intrinsics, we assume that it also > knows about the required CPUID and XGETBV intrinsics, and we assume that > the conditions for TRY_POPCNT_FAST are true. Bleh, cfbot's 32-bit build is

Re: BitmapHeapScan streaming read user and prelim refactoring

2024-03-29 Thread Tomas Vondra
On 3/29/24 23:03, Thomas Munro wrote: > On Sat, Mar 30, 2024 at 10:39 AM Thomas Munro wrote: >> On Sat, Mar 30, 2024 at 4:53 AM Tomas Vondra >> wrote: >>> ... Maybe there should be some flag to force >>> issuing fadvise even for sequential patterns, perhaps at the tablespace >>> level? ... >> >>

Re: Statistics Import and Export

2024-03-29 Thread Jeff Davis
On Fri, 2024-03-29 at 18:02 -0400, Stephen Frost wrote: > I’d certainly think “with stats” would be the preferred default of > our users. I'm concerned there could still be paths that lead to an error. For pg_restore, or when loading a SQL file, a single error isn't fatal (unless -e is

RE: Popcount optimization using AVX512

2024-03-29 Thread Shankaran, Akash
> From: Nathan Bossart > Sent: Friday, March 29, 2024 9:17 AM > To: Amonson, Paul D > On Fri, Mar 29, 2024 at 04:06:17PM +, Amonson, Paul D wrote: >> Yeah, I understand that much, but I want to know how portable the >> XGETBV instruction is. Unless I can assume that all x86_64 systems

Re: To what extent should tests rely on VACUUM ANALYZE?

2024-03-29 Thread Tom Lane
Alexander Lakhin writes: > 29.03.2024 16:51, Tom Lane wrote: >> Ouch. So what's triggering that? The intention of test_setup >> surely is to provide a uniform starting point. > Thanks for your attention to the issue! > Please try the attached... I experimented with the attached modified

Re: Why is parula failing?

2024-03-29 Thread Tom Lane
I wrote: > I'd not looked closely enough at the previous failure, because > now that I have, this is well out in WTFF territory: how can > reltuples be greater than zero when relpages is zero? This can't > be a state that autovacuum would have left behind, unless it's > really seriously broken.

Allowing DESC for a PRIMARY KEY column

2024-03-29 Thread Mitar
Hi! I have the same problem as [1]. I have table something like: CREATE TABLE values ( id int NOT NULL, revision int NOT NULL, data jsonb NOT NULL, PRIMARY KEY (id, revision) ) And I would like to be able to specify PRIMARY KEY (id, revision DESC) because the most common query I am

Re: WIP Incremental JSON Parser

2024-03-29 Thread Jacob Champion
On Fri, Mar 29, 2024 at 9:42 AM Andrew Dunstan wrote: > Here's a new set of patches, with I think everything except the error case > Asserts attended to. There's a change to add semantic processing to the test > suite in patch 4, but I'd fold that into patch 1 when committing. Thanks! 0004 did

Re: BitmapHeapScan streaming read user and prelim refactoring

2024-03-29 Thread Thomas Munro
On Sat, Mar 30, 2024 at 4:53 AM Tomas Vondra wrote: > Two observations: > > * The combine limit seems to have negligible impact. There's no visible > difference between combine_limit=8kB and 128kB. > > * Parallel queries seem to work about the same as master (especially for > optimal cases, but

Re: Table AM Interface Enhancements

2024-03-29 Thread Pavel Borisov
> > I think for better code look this could be removed: > >vlock: > > CHECK_FOR_INTERRUPTS(); > together with CHECK_FOR_INTERRUPTS(); in heapam_tuple_insert_with_arbiter > placed in the beginning of while loop. > To clarify things, this I wrote only about

Re: Combine Prune and Freeze records emitted by vacuum

2024-03-29 Thread Melanie Plageman
On Fri, Mar 29, 2024 at 12:00 PM Heikki Linnakangas wrote: > > On 29/03/2024 07:04, Melanie Plageman wrote: > > On Thu, Mar 28, 2024 at 11:07:10AM -0400, Melanie Plageman wrote: > >> These comments could use another pass. I had added some extra > >> (probably redundant) content when I thought I

RE: Popcount optimization using AVX512

2024-03-29 Thread Amonson, Paul D
> A counterexample is the CRC32C code. AFAICT we assume the presence of > CPUID in that code (and #error otherwise). I imagine its probably safe to > assume the compiler understands CPUID if it understands AVX512 intrinsics, > but that is still mostly a guess. If AVX-512 intrinsics are

Re: psql's FETCH_COUNT (cursor) is not being respected for CTEs

2024-03-29 Thread Laurenz Albe
On Fri, 2024-03-29 at 14:07 +0100, Laurenz Albe wrote: > I had a look at patch 0001 (0002 will follow). Here is the code review for patch number 2: > diff --git a/src/bin/psql/common.c b/src/bin/psql/common.c [...] +static bool +SetupGOutput(PGresult *result, FILE **gfile_fout, bool *is_pipe)

Re: Allowing DESC for a PRIMARY KEY column

2024-03-29 Thread Tom Lane
Mitar writes: > And I would like to be able to specify PRIMARY KEY (id, revision DESC) > because the most common query I am making is: > SELECT data FROM values WHERE id=123 ORDER BY revision DESC LIMIT 1 Did you experiment with whether that actually needs a special index? I get regression=#

Security lessons from liblzma

2024-03-29 Thread Bruce Momjian
You might have seen reports today about a very complex exploit added to recent versions of liblzma. Fortunately, it was only enabled two months ago and has not been pushed to most stable operating systems like Debian and Ubuntu. The original detection report is:

Re: Security lessons from liblzma

2024-03-29 Thread Tom Lane
Daniel Gustafsson writes: > One take-away for me is how important it is to ship recipes for regenerating > any testdata which is included in generated/compiled/binary format. IMO that's a hard, no-exceptions requirement just for maintainability's sake, never mind security risks.

Re: BitmapHeapScan streaming read user and prelim refactoring

2024-03-29 Thread Tomas Vondra
On 3/29/24 22:39, Thomas Munro wrote: > ... > >> I don't recall seeing a system with disabled readahead, but I'm >> sure there are cases where it may not really work - it clearly can't >> work with direct I/O, ... > > Right, for direct I/O everything is slow right now including seq scan. > We

Re: WIP Incremental JSON Parser

2024-03-29 Thread Andrew Dunstan
On 2024-03-26 Tu 17:52, Jacob Champion wrote: On Mon, Mar 25, 2024 at 3:14 PM Jacob Champion wrote: - add Assert calls in impossible error cases [2] To expand on this one, I think these parts of the code (marked with `<- here`) are impossible to reach: switch (top) { case

Re: Comments on Custom RMGRs

2024-03-29 Thread Jeff Davis
On Fri, 2024-03-29 at 18:20 +0700, Danil Anisimow wrote: > > In [rmgr_003.v3.patch] I added a phase argument to RmgrCheckpoint(). > Currently it is only called in two places: before and after > CheckPointBuffers(). I am fine with this. You've moved the discussion forward in two ways: 1.

Re: Building with meson on NixOS/nixpkgs

2024-03-29 Thread walther
Wolfgang Walther: To build on NixOS/nixpkgs I came up with a few small patches to meson.build. All of this works fine with Autoconf/Make already. In v3, I added another small patch for meson, this one about proper handling of -Dlibedit_preferred when used together with -Dreadline=enabled.

Re: Statistics Import and Export

2024-03-29 Thread Tom Lane
Jeff Davis writes: > On Fri, 2024-03-29 at 18:02 -0400, Stephen Frost wrote: >> I’d certainly think “with stats” would be the preferred default of >> our users. > What do you think about starting off with it as non-default, and then > switching it to default in 18? I'm with Stephen: I find it

Re: Popcount optimization using AVX512

2024-03-29 Thread Nathan Bossart
On Thu, Mar 28, 2024 at 11:10:33PM +0100, Alvaro Herrera wrote: > We don't do MSVC via autoconf/Make. We used to have a special build > framework for MSVC which parsed Makefiles to produce "solution" files, > but it was removed as soon as Meson was mature enough to build. See > commit

Re: Popcount optimization using AVX512

2024-03-29 Thread Nathan Bossart
On Fri, Mar 29, 2024 at 04:06:17PM +, Amonson, Paul D wrote: >> Yeah, I understand that much, but I want to know how portable the XGETBV >> instruction is. Unless I can assume that all x86_64 systems and compilers >> support that instruction, we might need an additional configure check and/or

Re: broken JIT support on Fedora 40

2024-03-29 Thread Thomas Munro
On Fri, Mar 22, 2024 at 7:15 AM Dmitry Dolgov <9erthali...@gmail.com> wrote: > > For verification, I've modified the deform.outblock to call LLVMBuildRet > > instead of LLVMBuildRetVoid and this seems to help -- inline and deform > > stages are still performed as before, but nothing crashes. But

Re: Statistics Import and Export

2024-03-29 Thread Jeff Davis
On Fri, 2024-03-29 at 20:54 -0400, Stephen Frost wrote: > What’s different, given the above arguments, in making the change > with 18 instead of now? Acknowledged. You, Tom, and Corey (and perhaps everyone else) seem to be aligned here, so that's consensus enough for me. Default is with stats,

Re: Popcount optimization using AVX512

2024-03-29 Thread Nathan Bossart
Here's a v17 of the patch. This one has configure checks for everything (i.e., CPUID, XGETBV, and the AVX512 intrinsics) as well as the relevant runtime checks (i.e., we call CPUID to check for XGETBV and AVX512 POPCNT availability, and we call XGETBV to ensure the ZMM registers are enabled). I

Re: To what extent should tests rely on VACUUM ANALYZE?

2024-03-29 Thread Alexander Lakhin
29.03.2024 11:59, Alexander Lakhin wrote: This simple change fixes the issue for me: -VACUUM ANALYZE tenk2; +VACUUM (ANALYZE, DISABLE_PAGE_SKIPPING) tenk2; I'm sorry, I wasn't persevering enough when testing that... After more test runs, I see that in fact it doesn't help. Best regards,

Re: remaining sql/json patches

2024-03-29 Thread jian he
On Fri, Mar 29, 2024 at 11:20 AM jian he wrote: > > > + > +JSON_TABLE ( > +context_item, > path_expression AS > json_path_name > PASSING { value AS > varname } , ... > > +COLUMNS ( class="parameter">json_table_column , > ... ) > + { ERROR | EMPTY > } ON ERROR > +) > top level

Re: Add new error_action COPY ON_ERROR "log"

2024-03-29 Thread torikoshia
On 2024-03-28 21:36, torikoshia wrote: On 2024-03-28 17:27, Bharath Rupireddy wrote: On Thu, Mar 28, 2024 at 1:43 PM torikoshia wrote: Attached patch fixes the doc, May I know which patch you are referring to? And, what do you mean by "fixes the doc"? Ugh, I replied to the wrong thread.

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-03-29 Thread Bertrand Drouvot
Hi, On Fri, Mar 29, 2024 at 09:39:31AM +0530, Amit Kapila wrote: > On Wed, Mar 27, 2024 at 9:00 PM Bharath Rupireddy > wrote: > > > > > > Thanks. I'm attaching v29 patches. 0001 managing inactive_since on the > > standby for sync slots. > > > > Commit message states: "why we can't just update

Re: Statistics Import and Export

2024-03-29 Thread Jeff Davis
On Tue, 2024-03-26 at 00:16 +0100, Tomas Vondra wrote: > I did take a closer look at v13 today. I have a bunch of comments and > some minor whitespace fixes in the attached review patches. I also attached a patch implementing a different approach to the pg_dump support. Instead of trying to

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-29 Thread Masahiko Sawada
On Thu, Mar 28, 2024 at 6:15 PM John Naylor wrote: > > On Thu, Mar 28, 2024 at 12:55 PM Masahiko Sawada > wrote: > > > > Pushed the refactoring patch. > > > > I've attached the rebased vacuum improvement patch for cfbot. I > > mentioned in the commit message that this patch eliminates the 1GB >

Re: Psql meta-command conninfo+

2024-03-29 Thread Imseih (AWS), Sami
Thank you for the updated patch. First and foremost, thank you very much for the review. > The initial and central idea was always to keep the metacommand > "\conninfo" in its original state, that is, to preserve the string as it is. > The idea of "\conninfo+" is to expand this to include more

Re: Synchronizing slots from primary to standby

2024-03-29 Thread Amit Kapila
On Fri, Mar 29, 2024 at 6:36 AM Zhijie Hou (Fujitsu) wrote: > > > Attach a new version patch which fixed an un-initialized variable issue and > added some comments. > The other approach to fix this issue could be that the slotsync worker get the serialized snapshot using pg_read_binary_file()

Re: Synchronizing slots from primary to standby

2024-03-29 Thread Bertrand Drouvot
Hi, On Fri, Mar 29, 2024 at 01:06:15AM +, Zhijie Hou (Fujitsu) wrote: > Attach a new version patch which fixed an un-initialized variable issue and > added some comments. Also, temporarily enable DEBUG2 for the 040 tap-test so > that > we can analyze the possible CFbot failures easily. >

Re: Improve eviction algorithm in ReorderBuffer

2024-03-29 Thread Masahiko Sawada
On Fri, Mar 29, 2024 at 2:09 PM vignesh C wrote: > > On Tue, 26 Mar 2024 at 10:05, Masahiko Sawada wrote: > > > > On Thu, Mar 14, 2024 at 12:02 PM Masahiko Sawada > > wrote: > > > > > > > > > I've attached new version patches. > > > > Since the previous patch conflicts with the current HEAD,

Re: To what extent should tests rely on VACUUM ANALYZE?

2024-03-29 Thread Richard Guo
On Fri, Mar 29, 2024 at 1:33 AM Tom Lane wrote: > Tomas Vondra writes: > > Yeah. I think it's good to design the data/queries in such a way that > > the behavior does not flip due to minor noise like in this case. > > +1 Agreed. The query in problem is: -- we can pull up the sublink into

Re: Can't find not null constraint, but \d+ shows that

2024-03-29 Thread jian he
hi. about v4, i think, i understand the changes you made. RemoveConstraintById(Oid conId) will drop a single constraint record. if the constraint is primary key, then primary key associated attnotnull should set to false. but sometimes it shouldn't. for example: drop table if exists t2; CREATE

Re: Synchronizing slots from primary to standby

2024-03-29 Thread Amit Kapila
On Fri, Mar 29, 2024 at 9:34 AM Hayato Kuroda (Fujitsu) wrote: > > Thanks for updating the patch! Here is a comment for it. > > ``` > +/* > + * By advancing the restart_lsn, confirmed_lsn, and xmin using > + * fast-forward logical decoding, we can verify whether a

Remove excessive trailing semicolons

2024-03-29 Thread Richard Guo
Noticed some newly introduced excessive trailing semicolons: $ git grep -E ";;$" -- *.c *.h src/include/lib/radixtree.h:int deletepos = slot - n4->children;; src/test/modules/test_tidstore/test_tidstore.c: BlockNumber prevblkno = 0;; Here is a trivial patch to

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-03-29 Thread Amit Kapila
On Fri, Mar 29, 2024 at 11:49 AM Bertrand Drouvot wrote: > > On Fri, Mar 29, 2024 at 09:39:31AM +0530, Amit Kapila wrote: > > > > Commit message states: "why we can't just update inactive_since for > > synced slots on the standby with the value received from remote slot > > on the primary. This

Re: To what extent should tests rely on VACUUM ANALYZE?

2024-03-29 Thread Alexander Lakhin
28.03.2024 20:33, Tom Lane wrote: But I'm a bit confused - how come the estimates do change at all? The analyze simply fetches 30k rows, and tenk only has 10k of them. So we should have *exact* numbers, and it should be exactly the same for all the analyze runs. So how come it changes like

Re: Synchronizing slots from primary to standby

2024-03-29 Thread Amit Kapila
On Fri, Mar 29, 2024 at 1:08 PM Bertrand Drouvot wrote: > > On Fri, Mar 29, 2024 at 07:23:11AM +, Zhijie Hou (Fujitsu) wrote: > > On Friday, March 29, 2024 2:48 PM Bertrand Drouvot > > wrote: > > > > > > Hi, > > > > > > On Fri, Mar 29, 2024 at 01:06:15AM +, Zhijie Hou (Fujitsu) wrote: >

Logical replication failure modes

2024-03-29 Thread Philip Warner
I am trying to discover the causes of occasional data loss in logical replication; it is VERY rare and happens every few week/months. Our setup is a source DB running in docker on AWS cloud server. The source database is stored in on local disks on the cloud server. The replication target

RE: Improve eviction algorithm in ReorderBuffer

2024-03-29 Thread Hayato Kuroda (Fujitsu)
Dear Sawada-san, > Agreed. > > I think the patch is in good shape. I'll push the patch with the > suggestion next week, barring any objections. Thanks for working on this. Agreed it is committable. Few minor comments: ``` + * Either txn or change must be non-NULL at least. We update the memory

Re: Synchronizing slots from primary to standby

2024-03-29 Thread Bertrand Drouvot
Hi, On Fri, Mar 29, 2024 at 02:35:22PM +0530, Amit Kapila wrote: > On Fri, Mar 29, 2024 at 1:08 PM Bertrand Drouvot > wrote: > > > > On Fri, Mar 29, 2024 at 07:23:11AM +, Zhijie Hou (Fujitsu) wrote: > > > On Friday, March 29, 2024 2:48 PM Bertrand Drouvot > > > wrote: > > > > > > > > Hi, >

Re: Improve eviction algorithm in ReorderBuffer

2024-03-29 Thread Amit Kapila
On Fri, Mar 29, 2024 at 12:13 PM Masahiko Sawada wrote: > > On Fri, Mar 29, 2024 at 2:09 PM vignesh C wrote: > > > > On Tue, 26 Mar 2024 at 10:05, Masahiko Sawada wrote: > > > > > > On Thu, Mar 14, 2024 at 12:02 PM Masahiko Sawada > > > wrote: > > > > > > > > > > > > I've attached new version

Re: BitmapHeapScan streaming read user and prelim refactoring

2024-03-29 Thread Tomas Vondra
On 3/29/24 02:12, Thomas Munro wrote: > On Fri, Mar 29, 2024 at 10:43 AM Tomas Vondra > wrote: >> I think there's some sort of bug, triggering this assert in heapam >> >> Assert(BufferGetBlockNumber(hscan->rs_cbuf) == tbmres->blockno); > > Thanks for the repro. I can't seem to reproduce it

Re: Logical replication failure modes

2024-03-29 Thread Philip Warner
I should have added that the source DB is 16.1 and the target is 16.0

Re: BUG: deadlock between autovacuum worker and client backend during removal of orphan temp tables with sequences

2024-03-29 Thread Akshat Jaimini
The following review has been posted through the commitfest application: make installcheck-world: tested, failed Implements feature: tested, passed Spec compliant: tested, passed Documentation:tested, passed Hii, Thanks for the updated patch. I ran make

Re: BitmapHeapScan streaming read user and prelim refactoring

2024-03-29 Thread Thomas Munro
I spent a bit of time today testing Melanie's v11, except with read_stream.c v13, on Linux, ext4, and 3000 IOPS cloud storage. I think I now know roughly what's going on. Here are some numbers, using your random table from above and a simple SELECT * FROM t WHERE a < 100 OR a = 123456. I'll

Re: Comments on Custom RMGRs

2024-03-29 Thread Danil Anisimow
On Fri, Mar 22, 2024 at 2:02 AM Jeff Davis wrote: > On Thu, 2024-03-21 at 19:47 +0700, Danil Anisimow wrote: > > [pgss_001.v1.patch] adds a custom resource manager to the > > pg_stat_statements extension. > > Did you consider moving the logic for loading the initial contents from > disk from

Re: To what extent should tests rely on VACUUM ANALYZE?

2024-03-29 Thread Alexander Lakhin
29.03.2024 11:59, Alexander Lakhin wrote: But it looks like subselect is not the only test that can fail due to vacuum instability. I see that create_index also suffers from cranky ConditionalLockBufferForCleanup() (+if (rand() % 10 == 0) return false; ), although it placed in parallel_schedule

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-29 Thread Alexander Korotkov
Hi, Euler! On Fri, Mar 29, 2024 at 1:38 AM Euler Taveira wrote: > On Thu, Mar 28, 2024, at 9:39 AM, Alexander Korotkov wrote: > > Fixed along with other issues spotted by Alexander Lakhin. > > > [I didn't read the whole thread. I'm sorry if I missed something ...] > > You renamed the function in

Re: To what extent should tests rely on VACUUM ANALYZE?

2024-03-29 Thread Richard Guo
On Thu, Mar 28, 2024 at 11:00 PM Alexander Lakhin wrote: > When running multiple 027_stream_regress.pl test instances in parallel > (and with aggressive autovacuum) on a rather slow machine, I encountered > test failures due to the subselect test instability just as the following > failures on

Re: Can't find not null constraint, but \d+ shows that

2024-03-29 Thread Tender Wang
jian he 于2024年3月29日周五 14:56写道: > hi. > about v4, i think, i understand the changes you made. > RemoveConstraintById(Oid conId) > will drop a single constraint record. > if the constraint is primary key, then primary key associated > attnotnull should set to false. > but sometimes it shouldn't. >

Re: [EXTERNAL] Re: Add non-blocking version of PQcancel

2024-03-29 Thread Jelte Fennema-Nio
On Thu, 28 Mar 2024 at 19:03, Tom Lane wrote: > > Alvaro Herrera writes: > > It doesn't fail when it's too fast -- it's just that it doesn't cover > > the case we want to cover. > > That's hardly better, because then you think you have test > coverage but maybe you don't. Honestly, that seems

Re: Streaming I/O, vectored I/O (WIP)

2024-03-29 Thread Thomas Munro
On Fri, Mar 29, 2024 at 9:45 AM Heikki Linnakangas wrote: > master (213c959a29):8.0 s > streaming-api v13: 9.5 s Hmm, that's not great, and I think I know one factor that has confounded my investigation and the conflicting reports I have received from a couple of people:

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-29 Thread John Naylor
On Thu, Mar 28, 2024 at 12:55 PM Masahiko Sawada wrote: > I think the patch is in good shape. Do you have other comments or > suggestions, John? --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -1918,11 +1918,6 @@ include_dir 'conf.d' too high. It may be useful to

RE: Synchronizing slots from primary to standby

2024-03-29 Thread Zhijie Hou (Fujitsu)
On Friday, March 29, 2024 2:48 PM Bertrand Drouvot wrote: > > Hi, > > On Fri, Mar 29, 2024 at 01:06:15AM +, Zhijie Hou (Fujitsu) wrote: > > Attach a new version patch which fixed an un-initialized variable > > issue and added some comments. Also, temporarily enable DEBUG2 for the > > 040

Re: Synchronizing slots from primary to standby

2024-03-29 Thread Bertrand Drouvot
Hi, On Fri, Mar 29, 2024 at 07:23:11AM +, Zhijie Hou (Fujitsu) wrote: > On Friday, March 29, 2024 2:48 PM Bertrand Drouvot > wrote: > > > > Hi, > > > > On Fri, Mar 29, 2024 at 01:06:15AM +, Zhijie Hou (Fujitsu) wrote: > > > Attach a new version patch which fixed an un-initialized

Re: Possibility to disable `ALTER SYSTEM`

2024-03-29 Thread Bruce Momjian
On Fri, Mar 29, 2024 at 08:46:33AM -0400, Robert Haas wrote: > On Thu, Mar 28, 2024 at 3:33 PM Bruce Momjian wrote: > > I am fine with moving ahead. I thought my later emails explaining we > > have to be careful communicated that. > > OK. Thanks for clarifying. I've committed the patch with the

Re: Statistics Import and Export

2024-03-29 Thread Jeff Davis
On Fri, 2024-03-29 at 05:32 -0400, Corey Huinker wrote: > That is fairly close to what I came up with per our conversation > (attached below), but I really like the att_stats_arginfo construct > and I definitely want to adopt that and expand it to a third > dimension that flags the fields that

Re: Possibility to disable `ALTER SYSTEM`

2024-03-29 Thread Robert Haas
On Thu, Mar 28, 2024 at 3:33 PM Bruce Momjian wrote: > I am fine with moving ahead. I thought my later emails explaining we > have to be careful communicated that. OK. Thanks for clarifying. I've committed the patch with the two wording changes that you suggested in your subsequent email. --

Re: psql's FETCH_COUNT (cursor) is not being respected for CTEs

2024-03-29 Thread Laurenz Albe
On Tue, 2024-01-30 at 15:29 +0100, Daniel Verite wrote: > PFA a rebased version. I had a look at patch 0001 (0002 will follow). > - > - Retrieving Query Results Row-by-Row > + > + Retrieving Query Results by chunks That should be "in chunks". > + > + > + > +

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-29 Thread Pavel Borisov
Hi, hackers! On Fri, 29 Mar 2024 at 16:45, Alexander Korotkov wrote: > Hi, Euler! > > On Fri, Mar 29, 2024 at 1:38 AM Euler Taveira wrote: > > On Thu, Mar 28, 2024, at 9:39 AM, Alexander Korotkov wrote: > > > > Fixed along with other issues spotted by Alexander Lakhin. > > > > > > [I didn't

Re: To what extent should tests rely on VACUUM ANALYZE?

2024-03-29 Thread Tom Lane
Richard Guo writes: > On Fri, Mar 29, 2024 at 1:33 AM Tom Lane wrote: >> Tomas Vondra writes: >>> Yeah. I think it's good to design the data/queries in such a way that >>> the behavior does not flip due to minor noise like in this case. >> +1 > Agreed. The query in problem is: > -- we can

  1   2   >