On 2024-04-26 14:39:16 -0400, Tom Lane wrote:
> Andres Freund writes:
> > I don't think enabling backtraces without a way to disable them is a good
> > idea
> > - security vulnerablilities in backtrace generation code are far from
> > unheard
> > of and can
at a GUC of this exact shape
> is a design dead end --- and that's what we're seeing now.
I don't think enabling backtraces without a way to disable them is a good idea
- security vulnerablilities in backtrace generation code are far from unheard
of and can make error handling a lot slower...
Greetings,
Andres Freund
On 2024-04-26 06:54:26 -0500, Jonathan S. Katz wrote:
> The Core Team would like to extend our congratulations to Melanie Plageman
> and Richard Guo, who have accepted invitations to become our newest
> PostgreSQL committers.
>
> Please join us in wishing them much success and few reverts!
ort, rather than the AIX (and windows) style. That'd
make it considerably less impactful on the buildsystem level. I don't know
what the performance difference is these days.
Greetings,
Andres Freund
Hi,
On 2024-04-18 10:54:18 +0200, Christoph Berg wrote:
> Re: Andres Freund
> > > This commit broke VPATH builds when the original source directory
> > > contains symlinks.
> > Argh, this is missing spaces around the '=', leading to the branch always
> > being en
eadahead is disabled or just not aggressive enough
after crossing segment boundaries, larger reads reduce the number of times
you're likely to be blocked waiting for read IO.
Which is also why I think that making MAX_SEND_SIZE configurable is a really
poor proxy for improving the situation.
We're imo much better off working on read_stream.[ch] support for reading WAL.
Greetings,
Andres Freund
o only
cast the pointers to the key type, i.e. char *. And incidentally that does
prevent the warning.
The reason it doesn't happen in newer versions of postgres is that we aren't
using guc_var_compare() in the relevant places anymore...
Greetings,
Andres Freund
have, but I think it will be
> tricky to do better, and I don't think I want to spend time on it, at
> least not now.
Oh, yea, that's clearly a much bigger and separate project...
Greetings,
Andres Freund
Hi,
On 2024-04-23 09:15:27 +1200, Thomas Munro wrote:
> I find myself wondering if symlinks should go on the list of "things
> we pretended Windows had out of convenience, that turned out to be
> more inconvenient than we expected, and we'd do better to tackle
> head-on with a more portable
log the current state, so we can in retrospect analyze e.g. whether
there was continual, but slow, replay progress, or whether replay was entirely
stuck. wait_for_catchup() not being debuggable has been a problem in many
different tests, so I think it's high time to fix that.
Greetings,
Andres Freund
On 2024-04-18 23:11:52 -0400, Tom Lane wrote:
> Andres Freund writes:
> > On 2024-04-18 22:18:34 -0400, Tom Lane wrote:
> >> (If our code coverage tools worked on bison/flex stuff,
> >> maybe this'd be less scary ... but they don't.)
>
> > For bison coverage se
so preventing coverage
for the actual scanner definition.
Greetings,
Andres Freund
ostgres is used extensively in our IBM product
> and is being exploited by multiple customers.
To be blunt: Then it'd have been nice to see some investment in that before
now. Both on the code level and the infrastructure level (i.e. access to
machines etc).
Greetings,
Andres Freund
Hi,
On 2024-04-18 09:03:21 -0400, Robert Haas wrote:
> On Wed, Apr 17, 2024 at 5:50 PM Andres Freund wrote:
> > > +If there are tablespace present in the backup, include tablespace_map as
> > > +a keyword parameter whose values is a hash. When tar_program is used,
Hi,
On 2024-04-18 09:07:43 -0300, Ranier Vilela wrote:
> On 18/04/2024 00:39, Andres Freund wrote:
> >There are lots of places that could benefit from adding 'static
> >const'.
>
> I found a few more places.
Good catches.
> Patch 004
>
> The opposite
are also quite useful for exploitation,
as one doesn't need to figure out precise addresses.
Greetings,
Andres Freund
On 2024-04-17 23:10:53 -0400, Tom Lane wrote:
> Jelte Fennema-Nio writes:
> > As I expected this problem was indeed fairly easy to address by still
> > building "backend/parser" before "interfaces". See attached patch.
>
> I think we should hold off on this. I found a simpler way to address
>
Hi,
Uh, huh.
On 2024-04-17 15:42:28 +0200, Christoph Berg wrote:
> Re: Andres Freund
> > https://git.postgresql.org/pg/commitdiff/e6927270cd18d535b77cbe79c55c6584351524be
>
> This commit broke VPATH builds when the original source directory
> contains symlinks.
I.e. a sym
paces instead?
Looks like it'd simplify at least the extended test somewhat?
Greetings,
Andres Freund
that.
Greetings,
Andres Freund
>From d43a10b5ba46b010c8e075b1062f9f30eb013498 Mon Sep 17 00:00:00 2001
From: Andres Freund
Date: Wed, 17 Apr 2024 14:27:03 -0700
Subject: [PATCH v1 1/3] static const: Convert struct option arrays
---
src/bin/initdb/initdb.c | 2 +-
src/
On 2024-04-16 07:32:55 +0900, Michael Paquier wrote:
> On Mon, Apr 15, 2024 at 11:57:20AM -0700, Andres Freund wrote:
> > GlobalVisTestNonRemovableHorizon()/GlobalVisTestNonRemovableFullHorizon()
> > only
> > existed for snapshot_too_old - but that was removed in f691f5b80
within
the backend, according to -Wl,--gc-sections,--print-gc-sections. A lot of that
is entirely expected stuff, like RegisterXactCallback(). But there are also
long-unused things like TransactionIdIsActive().
Greetings,
Andres Freund
t; + * is done.
> + */
> + while (!CommitTransactionCommandInternal());
Personally I'd use
{
}
instead of just ; here. The above scans weirdly for me. But it's also not
important.
Greetings,
Andres Freund
Hi,
On 2024-04-17 12:07:24 +0100, Dagfinn Ilmari Mannsåker wrote:
> Andres Freund writes:
> > I think the manual work for writing signatures in sgml is not insignificant,
> > nor is the volume of sgml for them. Manually maintaining the signatures
> > makes
> > it i
functions.sql, 1016 occurrences of
func_table_entry in funcs.sgml and 3.3k functions in pg_proc. I'm not saying
that differences due to system_functions.sql wouldn't be annoying to deal
with, but it'd also be far from the end of the world.
Greetings,
Andres Freund
Hi,
On 2024-04-16 15:05:32 -0400, Tom Lane wrote:
> Andres Freund writes:
> > I think we should work on generating a lot of func.sgml. Particularly the
> > signature etc should just come from pg_proc.dat, it's pointlessly painful to
> > generate that by hand. And for a lo
Hi,
On 2024-04-15 18:23:21 -0700, Jeff Davis wrote:
> On Mon, 2024-04-15 at 17:05 -0700, Andres Freund wrote:
> > Can't we test this as part of the normal testsuite?
>
> One thing that complicates things a bit is that the test compares the
> results against ICU, so a mismatch
m pg_proc,
because of xrefs etc. Although perhaps we could just strip those out for
pg_proc.
We'd need to add some more metadata to pg_proc, for grouping kinds of
functions together. But that seems doable.
Greetings,
Andres Freund
tially be addressed by having more "block oriented AM"
helpers in core, like we have for table_block_parallelscan*. Doesn't work for
everything, but should for something like analyze.
Greetings,
Andres Freund
On 2024-04-16 13:33:53 +0300, Alexander Korotkov wrote:
> Reverted.
Thanks!
Hi,
On 2024-04-15 10:54:16 -0400, Robert Haas wrote:
> On Fri, Apr 12, 2024 at 3:33 PM Andres Freund wrote:
> > Here's a patch implementing this approach. I confirmed that before we
> > trigger
> > the stuck spinlock logic very quickly and after we don't. Howeve
Hi,
On 2024-04-16 15:45:42 +0300, Alexander Korotkov wrote:
> On Tue, Apr 16, 2024 at 1:48 AM Andres Freund wrote:
> > On 2024-03-06 14:17:23 +0200, Alexander Korotkov wrote:
> > > 0001 Turn tail recursion into iteration in CommitTransactionCommand()
> > > I did
r as you have it.
> > Hm, independent of this, seems a bit odd that we don't include the memory
> > context type in pg_backend_memory_contexts?
>
> That seems like useful information to include. It sure would be
> useful to have in there to verify that I'm testing BumpStats(). I've
> written a patch [2].
Nice!
Greetings,
Andres Freund
Hi,
On 2024-04-15 20:47:16 -0400, Tom Lane wrote:
> Andres Freund writes:
> > On 2024-04-15 17:48:32 -0400, Tom Lane wrote:
> >> But I have no idea about making it work in meson. Any suggestions?
>
> > So you just want to compile define.c twice? The below should
s 1.10, released 2020-01-27.
Greetings,
Andres Freund
Hi,
On 2024-04-15 16:53:48 -0700, Jeff Davis wrote:
> On Sun, 2024-04-14 at 15:33 -0700, Andres Freund wrote:
> > - Coverage for some of the new unicode code is pretty poor:
> >
> > https://anarazel.de/postgres/cov/16-vs-HEAD-2024-04-14/src/common/unicode_category.c.gcov
64d7 has now
> baked long enough that it'd be safe to back-patch.
Looks like a reasonable plan to me.
Greetings,
Andres Freund
ck, because it leads to the build failing
because of main being duplicated. But it'd work the same with another, "non
overlapping", file.
Greetings,
Andres Freund
Hi,
On 2024-04-16 10:26:57 +1200, David Rowley wrote:
> On Mon, 15 Apr 2024 at 10:33, Andres Freund wrote:
> > - The new bump allocator has a fair amount of uncovered functionality:
> >
> > https://anarazel.de/postgres/cov/16-vs-HEAD-2024-04-14/src/backend/utils/mmgr/
n() calls?
That's of course largely unrelated to the coverage aspects. I just got
curious.
Greetings,
Andres Freund
Hi,
On 2024-04-15 15:36:04 -0400, Robert Haas wrote:
> On Sun, Apr 14, 2024 at 6:33 PM Andres Freund wrote:
> > - Some of the new walsummary code could use more tests.
> >
> > https://anarazel.de/postgres/cov/16-vs-HEAD-2024-04-14/src/backend/backup/walsummaryfuncs.c.
Hi,
On 2024-04-15 16:02:00 -0400, Robert Haas wrote:
> On Mon, Apr 15, 2024 at 3:47 PM Andres Freund wrote:
> > That said, I don't like the state after applying
> > https://postgr.es/m/CAPpHfdvuT6DnguzaV-M1UQ2whYGDojaNU%3D-%3DiHc0A7qo9HBEJw%40mail.gmail.com
> > because ther
ations before calling acquirefunc.
This is honestly leaving me somewhat speechless.
Greetings,
Andres Freund
nterface in some form, earlier in the thread.
What are you actually arguing for here?
Greetings,
Andres Freund
Hi,
GlobalVisTestNonRemovableHorizon()/GlobalVisTestNonRemovableFullHorizon() only
existed for snapshot_too_old - but that was removed in f691f5b80a8. I'm
inclined to think we should remove those functions for 17. No new code should
use them.
Greetings,
Andres Freund
age/buffer/bufmgr.c.gcov.html#L5978
...
Greetings,
Andres Freund
#2333
Greetings,
Andres Freund
nfortunately the logs aren't accessible anymore, so I can't check the precise
patch level of the compiler and/or the precise invocation used.
Greetings,
Andres Freund
Thoughts?
On the code level, I thought if it'd be good to have a common prefix for all
the automatically set variables. Right now that's CI_, but I'm not at all
wedded to that.
Greetings,
Andres Freund
>From 5d26ecfedbcbc83b4cb6e41a34c1af9a3ab24cdb Mon Sep 17 00:00:00 2001
From: Andres Freund
he automated image generation
for CI:
https://github.com/anarazel/pg-vm-images/blob/main/scripts/windows_install_mingw64.ps1#L21-L22
I wonder if we should maintain something like that list somewhere in the
postgres repo instead...
Greetings,
Andres Freund
Hi,
On 2024-04-12 11:33:17 -0700, Andres Freund wrote:
> I wonder if, for easy backpatching, the easiest solution is to just reset
> errno before calling pg_usleep(), and only increment status->delays if
> errno != EINTR. Given our pg_usleep() implementations, that'd preven the stuc
Hi,
Given that I found several ways to spuriously trigger the stuck spinlock
logic, I think we need to backpatch a fix.
On 2024-04-11 21:41:39 -0700, Andres Freund wrote:
> Tracking the total amount of time spent sleeping doesn't require any
> additional syscalls, due to the nanosleep()
lots of creaky old code
- libpam - undermaintained, lots of creaky old code
- the rest
Greetings,
Andres Freund
Hi,
On 2024-04-12 09:43:46 -0400, Tom Lane wrote:
> Andres Freund writes:
> > Oh my. There's a workload that completely trivially hits this, without even
> > trying hard. LISTEN/NOTIFY.
>
> Hm. Bug in the NOTIFY logic perhaps?
I don't think it's a bug, async.c:SignalB
Hi,
On 2024-04-11 21:41:39 -0700, Andres Freund wrote:
> FWIW, I just reproduced the scenario with signals. I added tracking of the
> total time actually slept and lost to SpinDelayStatus, and added a function to
> trigger a wait on a spinlock.
>
> To wait less, I set max_standby_
Hi,
On 2024-04-11 20:47:37 -0700, Andres Freund wrote:
> > So there's that. But that's not an argument that we need to be in a
> > hurry to timeout; if the built-in reaction time is less than perhaps
> > 10 minutes you're still miles ahead of the manual solution.
>
e
released on error. Lwlocks also take e.g. care to hold interrupts so code
doesn't just jump out of a section with lwlocks held.
Greetings,
Andres Freund
Hi,
On 2024-04-11 15:07:11 -0700, Andres Freund wrote:
> On 2024-04-11 16:35:58 -0400, Tom Lane wrote:
> > Indeed. I recall reading, not long ago, some Linux kernel docs to the
> > effect that automatic stack growth is triggered by a reference into
> > the page just be
Hi,
On 2024-04-11 16:35:58 -0400, Tom Lane wrote:
> Andres Freund writes:
> > d8f5acbdb9b made me wonder if we should add a compiler option to warn when
> > stack frames are large. gcc compatible compilers have -Wstack-usage=limit,
> > so
> > that's not ha
Hi,
On 2024-04-11 16:46:23 -0400, Robert Haas wrote:
> On Thu, Apr 11, 2024 at 3:52 PM Andres Freund wrote:
> > My suspicion is that most of the false positives are caused by lots of
> > signals
> > interrupting the pg_usleep()s. Because we measure the number of delays, no
Hi,
On 2024-04-11 16:46:10 -0400, Tom Lane wrote:
> Andres Freund writes:
> > On 2024-04-11 16:11:40 -0400, Tom Lane wrote:
> >> We wouldn't need to fix it, if we simply removed the NUM_DELAYS
> >> limit. Whatever kicked us off the sleep doesn't matter, we
right that this post feature-freeze works on
> designing a new API? Yes, 27bc1772fc masked the problem. But it was
> committed on Mar 30.
Note that there were versions of the patch that were targeting the
pre-27bc1772fc interface.
Greetings,
Andres Freund
Hi,
On 2024-04-11 16:11:40 -0400, Tom Lane wrote:
> Andres Freund writes:
> > On 2024-04-11 15:24:28 -0400, Robert Haas wrote:
> >> Or, rip out the whole, whole mechanism and just don't PANIC.
>
> > I continue believe that that'd be a quite bad idea.
>
> I
Hi,
On 2024-04-11 15:16:57 -0400, Andrew Dunstan wrote:
> On 2024-04-11 Th 15:01, Andres Freund wrote:
> > [1345/1940 42 69%] Compiling C object
> > src/bin/pg_verifybackup/pg_verifybackup.p/pg_verifybackup.c.o
> > ../../../../../home/andres/src/postgresql/sr
have come up with a way
to do away with that spinlock.
OTOH, there are plenty other lwlocks where we pay the price of lwlocks being
an rwlock, but just use the exclusive mode. So perhaps we should just add a
exclusive-only lwlock variant.
Greetings,
Andres Freund
r than one that's just
down.
Greetings,
Andres Freund
4400 bytes [-Wstack-usage=]
742 | GetFileBackupMethod(IncrementalBackupInfo *ib, const char *path,
| ^~~
I don't really have an opinion about the concrete warning limit to use.
Greetings,
Andres Freund
.
Could you expand a bit on the concrete scenario you're worried about here?
PG_TRY/CATCH aren't free, so adding something like this to a quite common
path, in the back branches, without a concrete analysis as to why it's needed,
seems a bit scary.
Greetings,
Andres Freund
Hi,
On 2024-04-10 16:50:44 -0400, Melanie Plageman wrote:
> This brings up a question about the prefetching. We never had to have
> this discussion for sequential scan streaming read because it didn't
> (and still doesn't) do prefetching. But, if we push the streaming read
> code down into the
reaming read) to review? Is it
> v1-0001-revert-Generalize-relation-analyze-in-table-AM-in.patch?
I think so. It's at least what I've been looking at.
Greetings,
Andres Freund
Hi,
On 2024-04-10 16:05:21 -0400, Tom Lane wrote:
> Andres Freund writes:
> > I think it could exascerbate the issue. Parag reported ~7k connections on a
> > 128 core machine. The buffer replacement logic in < 16 tries to lock the old
> > and new lock partitions at once
ich is probably not needed. At the same time I can't get this error
> from the verify pass with LLVM-12 or LLVM-15 (I have those at hand by
> accident). To make it even more confusing I've found a few similar
> examples in other projects, where this was really triggered by an issue
> in LLVM [1].
I'm afraid that it actually has changed over time, I'm fairly sure that it was
required at some point.
Greetings,
Andres Freund
o it made sense for 041b96802e
to create the stream in acquire_sample_rows() and have
block_sampling_read_stream_next() be in analyze.c. But eventually that should
be in access/heap/. Compared to 16, the state post the revert does tie
analyze.c a bit closer to the internals of the AM than before, but I'm not
sure the increase matters.
Greetings,
Andres Freund
greatly exacerbated by the issue fixed in a4adc31f690, once the
waitqueue is long, the spinlock will be held for an extended amount of time.
Greetings,
Andres Freund
own, without coordination.
That obviously isn't a reason to not fix the delay logic in lwlock.c.
Looks like the wrong logic was introduced by me in
commit 008608b9d51061b1f598c197477b3dc7be9c4a64
Author: Andres Freund
Date: 2016-04-10 20:12:32 -0700
Avoid the use of a separate spinloc
ock usage pattern that is
> causing this spot to become so heavily contended.)
My suspicion is that it's a4adc31f690 which we only recently backpatched to
< 16.
Greetings,
Andres Freund
h to manager our ratelimiting counters.
What version of PG was this? I think it's much more likely that you're
hitting a bug that caused a lot more contention inside lwlocks. That was fixed
for 16+ in a4adc31f690 on 2022-11-20, but only backpatched to 12-15 on
2024-01-18.
Greetings,
Andres Freund
he commit message it's hard
to evaluate how this actually intended to be used.
Greetings,
Andres Freund
m that RNG unfairness is a real part of the problem here.
Greetings,
Andres Freund
Hi,
On 2024-04-09 20:12:48 -0400, Tom Lane wrote:
> Andres Freund writes:
> > FWIW, just redefining mm_strdup() that way doesn't help much here either,
> > even with an affected compiler. The gain increases substantially after
> > simplifying unreserved_keyword etc to just u
clang-19 -O2
"plain"0m24.354s
mm_strdup redefined0m23.741s
+use default action 0m14.218s
Greetings,
Andres Freund
Hi,
On 2024-04-09 19:00:41 -0400, Tom Lane wrote:
> Andres Freund writes:
> > I think we need to do something about the compile time of this file, even
> > with
> > gcc. Our main grammar already is an issue and stacking all the ecpg stuff on
> > top makes it co
Hi,
On 2024-04-09 15:33:10 -0700, Andres Freund wrote:
> Which leads to a control flow graph with *many* incoming edges to the basic
> block containing the function call to mm_strdup(), triggering a normally
> harmless O(N^2) or such.
With clang-16 -O2 there is a basic block with 3904
ng-16 -O2 from 345.188 to 158.084s
clang-19 -O2 from 26.018s to 15.200s
I suspect what is happening is that clang tries to optimize the number of
calls to mm_strdup(), by separating the argument setup from the function
call. Which leads to a control flow graph with *many* incoming edges to the
basic block containing the function call to mm_strdup(), triggering a normally
harmless O(N^2) or such.
Greetings,
Andres Freund
, I did some
> statistical analysis on the RNG generator that PG uses to create the
> backoff for the spin locks.
ISTM that the fix here is to not use a spinlock for whatever the contention is
on, rather than improve the RNG.
Greetings,
Andres Freund
tiation. Might
make it slightly easier to notice obsoleted uses of simplehash.
Greetings,
Andres Freund
Hi,
Looks like I forgot to update the thread to note that I finally committed the
remaining warning fixes. I had already fixed a bunch of others in upstream
meson.
commit a3da95deee38ee067b0bead639c830eacbe894d5
Author: Andres Freund
Date: 2024-03-13 01:40:53 -0700
meson: macos: Avoid
n without a problem.
Perhaps it's worth picking that up again?
https://github.com/anarazel/postgres/tree/meson-rpath
https://github.com/anarazel/postgres/commit/46f1963fee7525c3cc3837ef8423cbf6cb08d10a
Greetings,
Andres Freund
Hi,
On 2024-04-08 16:01:41 +0530, Amit Kapila wrote:
> Pushed.
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=adder=2024-04-08%2012%3A04%3A27
This unfortunately is a commit after
commit 6f3d8d5e7cc
Author: Amit Kapila
Date: 2024-04-08 13:21:55 +0530
Fix the intermittent
On 2024-04-08 08:37:44 -0700, Andres Freund wrote:
> On 2024-04-08 11:17:51 +0400, Pavel Borisov wrote:
> > On Mon, 8 Apr 2024 at 03:25, Alexander Korotkov
> > > I was under the impression there are not so many out-of-core table
> > > AMs, which have non-dummy analysis
the new API doesn't
look well polished either, so it's not a question of a smoother transition or
something like that.
I don't think redesigning extension APIs at this stage of the release cycle
makes sense.
Greetings,
Andres Freund
Hi,
On 2024-04-08 09:26:09 -0400, Robert Haas wrote:
> On Sun, Apr 7, 2024 at 6:50 PM Michael Paquier wrote:
> And maybe we need to think of a way to further mitigate this crush of
> last minute commits. e.g. In the last week, you can't have more
> feature commits, or more lines of insertions in
Hi,
On 2024-04-08 16:18:21 +1200, David Rowley wrote:
> On Mon, 8 Apr 2024 at 16:08, Andres Freund wrote:
> > I think visible would be ok, the serialization checks are IMO about
> > visibility too. But if you'd prefer I'd also be ok with something like
> > page_collect_tup
Hi,
On 2024-04-08 15:43:12 +1200, David Rowley wrote:
> On Mon, 8 Apr 2024 at 15:13, Andres Freund wrote:
> > Off-list Melanie suggested heap_page_collect_visible_tuples(). Given the
> > separate callsites (making long names annoying) and the fact that it's
> > really
>
Hi,
On 2024-04-08 14:43:21 +1200, David Rowley wrote:
> On Sun, 7 Apr 2024 at 19:30, Andres Freund wrote:
> > Good call. Added and pushed.
>
> I understand you're already aware of the reference in the comment to
> heapgetpage(), which no longer exists as of 44086b097.
Yea, h
On 2024-04-07 21:41:23 -0400, Tom Lane wrote:
> Andres Freund writes:
> > I can't see a way it could hurt in the back branches, so I'm inclined to
> > backpatch the pfree?
>
> +1
Done
t now. If coverity had started to complain
after da41d71070d, I'd understand, but that was ~5 years ago.
I can't see a way it could hurt in the back branches, so I'm inclined to
backpatch the pfree?
Greetings,
Andres Freund
useful to be able to evict arbitrary blocks" that seems to describe
things at least as well as "invalidate"?
Greetings,
Andres Freund
Hi,
On 2024-04-08 02:25:17 +0300, Alexander Korotkov wrote:
> On Mon, Apr 8, 2024 at 12:40 AM Andres Freund wrote:
> > On 2024-03-30 23:33:04 +0200, Alexander Korotkov wrote:
> > > I've pushed 0001, 0002 and 0006.
> >
> > I briefly looked at 27bc1772fc81 an
nteresting for Makefile.global - they're msvc flags - because
you can't use that with msvc.
I'm also doubtful that it's worth supporting warning_level=3/everything, you
end up with a completely flood of warnings that way.
Greetings,
Andres Freund
1 - 100 of 8799 matches
Mail list logo
