On Wed, Aug 23, 2023 at 12:15 AM Peter Eisentraut wrote:
> I suggest to also rename the hook functions (check and assign), like in
> the attached patch. Mainly because utils/guc_hooks.h says to order the
> functions by GUC variable name, which was already wrong under the old
> name, but it would
On Wed, Apr 19, 2023 at 10:43 AM Andres Freund wrote:
> I don't think the "evict on every buffer access" works quite that way - unless
> you have a completely even access pattern, buffer access frequency will
> increase usage count much more frequently on some buffers than others. And if
> you
On 01.05.23 04:47, Thomas Munro wrote:
On Mon, May 1, 2023 at 12:00 PM Tom Lane wrote:
Justin Pryzby writes:
On Sun, Apr 30, 2023 at 06:35:30PM +1200, Thomas Munro wrote:
What about a
warning message about that at startup if it's on?
Such a warning wouldn't be particularly likely to be
On Mon, May 15, 2023 at 9:09 AM Noah Misch wrote:
> This looks reasonable.
Pushed with a small change: a couple of GUC_check_errdetail strings
needed s/io_direct/debug_io_direct/. Thanks.
On Mon, May 01, 2023 at 02:47:57PM +1200, Thomas Munro wrote:
> On Mon, May 1, 2023 at 12:00 PM Tom Lane wrote:
> > Justin Pryzby writes:
> > > Since this is -currently- a developer-only feature, it seems reasonable
> > > to rename the GUC to debug_direct_io, and (at such time as it's
> > >
On Wed, Apr 19, 2023 at 7:35 AM Greg Stark wrote:
> On Mon, 17 Apr 2023 at 17:45, Thomas Munro wrote:
> > (2) without a page cache, you really need to size your shared_buffers
> > adequately and we can't do that automatically.
>
> Well I'm more optimistic... That may not always be
WAL
+files using O_DIRECT (most Unix-like systems),
+F_NOCACHE (macOS) or
+FILE_FLAG_NO_BUFFERING (Windows).
+
+
+May be set to an empty string (the default) to disable use of direct
+ I/O, or a comma-separated list of operations that should use dir
Justin Pryzby writes:
> On Sun, Apr 30, 2023 at 06:35:30PM +1200, Thomas Munro wrote:
>> What about a
>> warning message about that at startup if it's on?
> Such a warning wouldn't be particularly likely to be seen by someone who
> already didn't read/understand the docs for the not-feature that
On Sun, Apr 30, 2023 at 06:35:30PM +1200, Thomas Munro wrote:
> On Sun, Apr 30, 2023 at 4:11 PM Noah Misch wrote:
> > Speaking of the developer-only status, I find the io_direct name more
> > enticing
> > than force_parallel_mode, which PostgreSQL renamed due to overuse from
> > people
> >
feature,
but that documentation might be hard to find. Let's also display a
warning about that in the server log.
Later proposals will provide the infrastructure to use it efficiently,
but by releasing this switch earlier we can learn about direct I/O
quirks in systems in the wild.
Discussion: https://po
On Sun, Apr 30, 2023 at 4:11 PM Noah Misch wrote:
> Speaking of the developer-only status, I find the io_direct name more enticing
> than force_parallel_mode, which PostgreSQL renamed due to overuse from people
> expecting non-developer benefits. Should this have a name starting with
> debug_?
On Mon, Apr 17, 2023 at 12:06:23PM -0400, Tom Lane wrote:
> Robert Haas writes:
> > On Sat, Apr 15, 2023 at 2:19 PM Tom Lane wrote:
> >> I get the impression that we are going to need an actual runtime
> >> test if we want to defend against this. Not entirely convinced
> >> it's worth the
Hi,
On 2023-04-19 13:16:54 -0400, Robert Haas wrote:
> On Wed, Apr 19, 2023 at 12:54 PM Andres Freund wrote:
> > Interestingly, I haven't seen that as much in more recent benchmarks as it
> > used to. Partially I think because common s_b settings have gotten bigger, I
> > guess. But I wonder if
On Wed, Apr 19, 2023 at 12:54 PM Andres Freund wrote:
> Interestingly, I haven't seen that as much in more recent benchmarks as it
> used to. Partially I think because common s_b settings have gotten bigger, I
> guess. But I wonder if we also accidentally improved something else, e.g. by
>
Hi,
On 2023-04-18 15:35:09 -0400, Greg Stark wrote:
> On Mon, 17 Apr 2023 at 17:45, Thomas Munro wrote:
> > It's something you'd
> > opt into for a dedicated database server along with other carefully
> > considered settings. It seems acceptable to me that if you set
> > io_direct to a
Hi,
On 2023-04-18 09:44:10 +1200, Thomas Munro wrote:
> * We have no plans to turn this on by default even when the later
> asynchronous machinery is proposed, and direct I/O starts to make more
> economic sense (think: your stream of small reads and writes will be
> converted to l
Hi,
On 2023-04-19 10:11:32 -0400, Robert Haas wrote:
> On this point specifically, one fairly large problem that we have
> currently is that our buffer replacement algorithm is terrible. In
> workloads I've examined, either almost all buffers end up with a usage
> count of 5 or almost all buffers
On 4/19/23 10:11, Robert Haas wrote:
On Tue, Apr 18, 2023 at 3:35 PM Greg Stark wrote:
Well I'm more optimistic... That may not always be impossible.
We've already added the ability to add more shared memory after
startup. We could implement the ability to add or remove shared buffer
On Tue, Apr 18, 2023 at 3:35 PM Greg Stark wrote:
> Well I'm more optimistic... That may not always be impossible.
> We've already added the ability to add more shared memory after
> startup. We could implement the ability to add or remove shared buffer
> segments after startup. And it
On Mon, 17 Apr 2023 at 17:45, Thomas Munro wrote:
>
> Reasons: (1) There will always be a
> few file systems that refuse O_DIRECT (Linux tmpfs is one such, as we
> learned in this thread; if fails with EINVAL at open() time), and
So why wouldn't we just automatically turn it off (globally or for
rst.
+1
A couple more things I wanted to note:
* We have no plans to turn this on by default even when the later
asynchronous machinery is proposed, and direct I/O starts to make more
economic sense (think: your stream of small reads and writes will be
converted to larger preadv/pwritev or moral
Robert Haas writes:
> On Sat, Apr 15, 2023 at 2:19 PM Tom Lane wrote:
>> I get the impression that we are going to need an actual runtime
>> test if we want to defend against this. Not entirely convinced
>> it's worth the trouble. Who, other than our deliberately rear-guard
>> buildfarm
On Sat, Apr 15, 2023 at 2:19 PM Tom Lane wrote:
> I get the impression that we are going to need an actual runtime
> test if we want to defend against this. Not entirely convinced
> it's worth the trouble. Who, other than our deliberately rear-guard
> buildfarm animals, is going to be building
On 2023-04-16 20:05, Mikael Kjellström wrote:
Oh wait ... this involves a switch from gcc in OpenBSD 5.9 to clang
in OpenBSD 7.3, doesn't it? That isn't something update_personality
will handle; you need a new animal if the compiler product is changing.
Correct.
OK. I registered a new
On Sun, Apr 16, 2023 at 04:51:04PM +0200, Mikael Kjellström wrote:
> That is what I meant with above.
>
> I just use the same animal name and secret and then run
> "update_personality.pl".
>
> That should be enough I think?
Yes, that should be enough as far as I recall. This has been
mentioned
On 2023-04-16 19:59, Andrew Dunstan wrote:
On Apr 16, 2023, at 12:16 PM, Tom Lane wrote:
Andrew Dunstan writes:
On 2023-04-16 Su 10:18, Tom Lane wrote:
Actually, as long as it's still OpenBSD I think you can keep using
the same animal name ... Andrew, what's the policy on that?
> On Apr 16, 2023, at 12:16 PM, Tom Lane wrote:
>
> Andrew Dunstan writes:
>>> On 2023-04-16 Su 10:18, Tom Lane wrote:
>>> Actually, as long as it's still OpenBSD I think you can keep using
>>> the same animal name ... Andrew, what's the policy on that?
>
>> update_personality.pl lets you
Andrew Dunstan writes:
> On 2023-04-16 Su 10:18, Tom Lane wrote:
>> Actually, as long as it's still OpenBSD I think you can keep using
>> the same animal name ... Andrew, what's the policy on that?
> update_personality.pl lets you update the OS version / compiler version
> / owner-name /
On 2023-04-16 Su 10:18, Tom Lane wrote:
=?UTF-8?Q?Mikael_Kjellstr=c3=b6m?= writes:
On 2023-04-16 00:10, Tom Lane wrote:
so curculio should be the only one that's at risk here.
Maybe just upgrading it is the right answer.
Just let me know if I should switch curculio to OpenBSD 7.3.
Yes
On 2023-04-16 16:18, Tom Lane wrote:
=?UTF-8?Q?Mikael_Kjellstr=c3=b6m?= writes:
On 2023-04-16 00:10, Tom Lane wrote:
so curculio should be the only one that's at risk here.
Maybe just upgrading it is the right answer.
Just let me know if I should switch curculio to OpenBSD 7.3.
Yes
=?UTF-8?Q?Mikael_Kjellstr=c3=b6m?= writes:
> On 2023-04-16 00:10, Tom Lane wrote:
>> so curculio should be the only one that's at risk here.
>> Maybe just upgrading it is the right answer.
> Just let me know if I should switch curculio to OpenBSD 7.3.
Yes please.
> I already have a new server
On 2023-04-16 00:10, Tom Lane wrote:
so curculio should be the only one that's at risk here.
Maybe just upgrading it is the right answer.
Just let me know if I should switch curculio to OpenBSD 7.3.
I already have a new server setup so only need to switch the "animal"
and "secret" and
On Sat, Apr 15, 2023 at 02:19:35PM -0400, Tom Lane wrote:
> PS: I don't quite understand how it managed to get through initdb
> when CREATE DATABASE doesn't work. Maybe there is a different
> code path taken in standalone mode?
ad43a413c4f7f5d024a5b2f51e00d280a22f1874
initdb: When running
Thomas Munro writes:
> Full GCC support including stack objects actually began in 4.6, it
> seems.
Hmm. The oldest gcc versions remaining in the buildfarm seem to be
curculio | configure: using compiler=gcc (GCC) 4.2.1 20070719
frogfish | configure: using compiler=gcc (Debian
On Sun, Apr 16, 2023 at 6:19 AM Tom Lane wrote:
> So apparently, the fact that you even get a warning about the
> alignment not being honored is something OpenBSD patched in
> after-the-fact; it's not there in genuine vintage gcc.
Ah, that is an interesting discovery, and indeed kills the
Thomas Munro writes:
> As far as I can tell, the failure to honour large alignment attributes
> even though the compiler passes our configure check that you can do
> that was considered to be approximately a bug[1] or at least a thing
> to be improved in fairly old GCC times but the fix wasn't
On 2023-04-14 21:33, Andres Freund wrote:
Oh! I was misled by the buildfarm label on morepork, which claims
it's running clang 10.0.1. But actually, per its configure report,
it's running
configure: using compiler=gcc (GCC) 4.2.1 20070719
Huh. I wonder if that was an accident in
On 2023-04-15 05:22, Tom Lane wrote:
Thomas Munro writes:
On Sat, Apr 15, 2023 at 7:50 AM Mikael Kjellström
wrote:
want me to switch to clang instead?
I vote +1, that's the system compiler in modern OpenBSD.
Ditto, we need coverage of that.
OK. I switched to clang on morepork now.
Thomas Munro writes:
> On Sat, Apr 15, 2023 at 7:50 AM Mikael Kjellström
> wrote:
>> want me to switch to clang instead?
> I vote +1, that's the system compiler in modern OpenBSD.
Ditto, we need coverage of that.
> As for curculio, I don't understand the motivation for maintaining
> that
On Sat, Apr 15, 2023 at 7:50 AM Mikael Kjellström
wrote:
> want me to switch to clang instead?
I vote +1, that's the system compiler in modern OpenBSD.
https://www.cambus.net/the-state-of-toolchains-in-openbsd/
As for curculio, I don't understand the motivation for maintaining
that machine.
On Sat, Apr 15, 2023 at 7:38 AM Tom Lane wrote:
> Andres Freund writes:
> > On 2023-04-14 15:21:18 -0400, Tom Lane wrote:
> >> +1 for that, though. (Also, the fact that these animals aren't
> >> actually failing suggests that 004_io_direct.pl needs expansion.)
>
> > It's skipped, due to lack of
Andres Freund writes:
> On 2023-04-14 15:21:18 -0400, Tom Lane wrote:
>> +1 for that, though. (Also, the fact that these animals aren't
>> actually failing suggests that 004_io_direct.pl needs expansion.)
> It's skipped, due to lack of O_DIRECT:
> [20:50:22] t/004_io_direct.pl ..
Hi,
On 2023-04-14 15:21:18 -0400, Tom Lane wrote:
> Andres Freund writes:
> > On 2023-04-14 13:21:33 -0400, Tom Lane wrote:
> >> ... I'm not sure why only those two animals
> >> are unhappy, but I think they have a point: typical ABIs don't
> >> guarantee alignment of function stack frames to
Andres Freund writes:
> On 2023-04-14 13:21:33 -0400, Tom Lane wrote:
>> ... I'm not sure why only those two animals
>> are unhappy, but I think they have a point: typical ABIs don't
>> guarantee alignment of function stack frames to better than
>> 16 bytes or so. In principle the compiler
Hi,
On 2023-04-14 13:21:33 -0400, Tom Lane wrote:
> Since the direct I/O commit went in, buildfarm animals
> curculio and morepork have been issuing warnings like
>
> hashpage.c: In function '_hash_expandtable':
> hashpage.c:995: warning: ignoring alignment for stack all
Since the direct I/O commit went in, buildfarm animals
curculio and morepork have been issuing warnings like
hashpage.c: In function '_hash_expandtable':
hashpage.c:995: warning: ignoring alignment for stack allocated 'zerobuf'
in places where there's a local variable of type PGIOAlignedBlock
Re: Thomas Munro
> Linux/tmpfs: 1..0 # SKIP pre-flight test if we can open a file with
> O_DIRECT failed: Invalid argument
I confirm it's working now:
t/004_io_direct.pl .. skipped: pre-flight test if we can open a
file with O_DIRECT failed: Invalid argument
All tests successful.
Thanks both for looking, and thanks for the explanation Ilmari.
Pushed with your improvements. The 4 CI systems run the tests
(Windows and Mac by special always-expected-to-work case, Linux and
FreeBSD by successful pre-flight perl test of O_DIRECT), and I also
tested three unusual systems, two
On 2023-04-12 We 12:38, Dagfinn Ilmari Mannsåker wrote:
Andrew Dunstan writes:
On 2023-04-12 We 10:23, Dagfinn Ilmari Mannsåker wrote:
Andrew Dunstan writes:
On 2023-04-12 We 01:48, Thomas Munro wrote:
On Wed, Apr 12, 2023 at 3:04 PM Thomas Munrowrote:
On Wed, Apr 12, 2023 at 2:56
Andrew Dunstan writes:
> On 2023-04-12 We 10:23, Dagfinn Ilmari Mannsåker wrote:
>> Andrew Dunstan writes:
>>
>>> On 2023-04-12 We 01:48, Thomas Munro wrote:
On Wed, Apr 12, 2023 at 3:04 PM Thomas Munro
wrote:
> On Wed, Apr 12, 2023 at 2:56 PM Christoph Berg wrote:
>> I'm
On 2023-04-12 We 10:23, Dagfinn Ilmari Mannsåker wrote:
Andrew Dunstan writes:
On 2023-04-12 We 01:48, Thomas Munro wrote:
On Wed, Apr 12, 2023 at 3:04 PM Thomas Munro wrote:
On Wed, Apr 12, 2023 at 2:56 PM Christoph Berg wrote:
I'm hitting a panic in t_004_io_direct. The build is
Andrew Dunstan writes:
> On 2023-04-12 We 01:48, Thomas Munro wrote:
>> On Wed, Apr 12, 2023 at 3:04 PM Thomas Munro wrote:
>>> On Wed, Apr 12, 2023 at 2:56 PM Christoph Berg wrote:
I'm hitting a panic in t_004_io_direct. The build is running on
overlayfs on tmpfs/ext4 (upper/lower)
Thomas Munro writes:
> I think I have that working OK. Any Perl hackers want to comment on
> my use of IO::File (copied from examples on the internet that showed
> how to use O_DIRECT)? I am not much of a perl hacker but according to
> my package manager, IO/File.pm came with perl itself.
On 2023-04-12 We 01:48, Thomas Munro wrote:
On Wed, Apr 12, 2023 at 3:04 PM Thomas Munro wrote:
On Wed, Apr 12, 2023 at 2:56 PM Christoph Berg wrote:
I'm hitting a panic in t_004_io_direct. The build is running on
overlayfs on tmpfs/ext4 (upper/lower) which is probably a weird
combination
On Wed, Apr 12, 2023 at 5:48 PM Thomas Munro wrote:
> On Wed, Apr 12, 2023 at 3:04 PM Thomas Munro wrote:
> > On Wed, Apr 12, 2023 at 2:56 PM Christoph Berg wrote:
> > > I'm hitting a panic in t_004_io_direct. The build is running on
> > > overlayfs on tmpfs/ext4 (upper/lower) which is probably
his "developer only" feature long before serious use of it.
From 72e865835bcf1c9dce2090de0da66839908133c6 Mon Sep 17 00:00:00 2001
From: Thomas Munro
Date: Wed, 12 Apr 2023 16:26:54 +1200
Subject: [PATCH] Skip the 004_io_direct.pl test if a pre-flight check fails.
Previously the test had a l
On Wed, Apr 12, 2023 at 2:56 PM Christoph Berg wrote:
> I'm hitting a panic in t_004_io_direct. The build is running on
> overlayfs on tmpfs/ext4 (upper/lower) which is probably a weird
> combination but has worked well for building everything over the last
> decade. On Debian unstable:
>
>
Hi,
I'm hitting a panic in t_004_io_direct. The build is running on
overlayfs on tmpfs/ext4 (upper/lower) which is probably a weird
combination but has worked well for building everything over the last
decade. On Debian unstable:
PANIC: could not open file "pg_wal/00010001":
Hi,
On 2023-04-09 16:40:54 -0700, Noah Misch wrote:
> On Sun, Apr 09, 2023 at 02:45:16PM -0700, Andres Freund wrote:
> > It's not *just* that scenario. With a few concurrent connections you can get
> > into problematic territory even with halfway reasonable shared buffers.
>
> I am not familiar
On Tue, Apr 11, 2023 at 2:31 PM Thomas Munro wrote:
> I tried to find out what POSIX says about this
(But of course whatever it might say is of especially limited value
when O_DIRECT is in the picture, being completely unstandardised.
Really I guess all they meant was "if you *copy* something
On Tue, Apr 11, 2023 at 2:15 PM Andres Freund wrote:
> And the fix has been merged into
> https://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux.git/log/?h=for-next
>
> I think that means it'll have to wait for 6.4 development to open (in a few
> weeks), and then will be merged into the
Hi,
On 2023-04-10 18:55:26 -0700, Andres Freund wrote:
> On 2023-04-10 19:27:27 +1200, Thomas Munro wrote:
> > On Mon, Apr 10, 2023 at 2:57 PM Andres Freund wrote:
> > > Have you tried to write a reproducer for this that doesn't involve
> > > postgres?
> >
> > I tried a bit. I'll try harder
Hi,
On 2023-04-10 19:27:27 +1200, Thomas Munro wrote:
> On Mon, Apr 10, 2023 at 2:57 PM Andres Freund wrote:
> > Have you tried to write a reproducer for this that doesn't involve postgres?
>
> I tried a bit. I'll try harder soon.
>
> > ... What kernel version did you repro
> > this on
On Mon, Apr 10, 2023 at 7:27 PM Thomas Munro wrote:
> Debian's 6.0.10-2 kernel (Debian 12 on a random laptop).
Realising I hadn't updated for a bit, I did so and it still reproduces on:
$ uname -a
Linux x1 6.1.0-7-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.20-1
(2023-03-19) x86_64 GNU/Linux
On Mon, Apr 10, 2023 at 2:57 PM Andres Freund wrote:
> Have you tried to write a reproducer for this that doesn't involve postgres?
I tried a bit. I'll try harder soon.
> ... What kernel version did you repro
> this on Thomas?
Debian's 6.0.10-2 kernel (Debian 12 on a random laptop). Here's
ing about Direct I/O data
corruption. No patch at the moment, they are still discussing how to
address it:
https://lore.kernel.org/linux-btrfs/aa1fb69e-b613-47aa-a99e-a0a2c9ed2...@app.fastmail.com/
Ciao,
Gelma
Hi,
On 2023-04-10 00:17:12 +1200, Thomas Munro wrote:
> I think there are two separate bad phenomena.
>
> 1. A concurrent modification of the user space buffer while writing
> breaks the checksum so you can't read the data back in, as . I can
> reproduce that with a stand-alone program,
On Mon, Apr 10, 2023 at 8:43 AM Tom Lane wrote:
> Boy, it's hard to look at that trace and not call it a filesystem bug.
Agreed.
> Given the apparent dependency on COW, I wonder if this has something
> to do with getting confused about which copy is current?
Yeah, I suppose it would require
On Sun, Apr 09, 2023 at 02:45:16PM -0700, Andres Freund wrote:
> On 2023-04-08 21:29:54 -0700, Noah Misch wrote:
> > On Sat, Apr 08, 2023 at 11:08:16AM -0700, Andres Freund wrote:
> > > On 2023-04-07 23:04:08 -0700, Andres Freund wrote:
> > > > If you look at log_newpage_range(), it's not
Hi,
On 2023-04-08 21:29:54 -0700, Noah Misch wrote:
> On Sat, Apr 08, 2023 at 11:08:16AM -0700, Andres Freund wrote:
> > On 2023-04-07 23:04:08 -0700, Andres Freund wrote:
> > > There were some failures in CI (e.g. [1] (and perhaps also bf, didn't yet
> > > check), about "no unpinned buffers
Thomas Munro writes:
> we have a page at offset 638976, and we can find all system calls that
> touched that offset:
> [pid 26031] 23:26:48.521123 pwritev(50,
> [{iov_base="\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"...,
> iov_len=8192}], 1, 638976) = 8192
> [pid 26040]
On 2023-04-09 Su 09:14, Andrew Dunstan wrote:
On 2023-04-09 Su 08:39, Thomas Munro wrote:
On Sun, Apr 9, 2023 at 11:25 PM Andrew Dunstan wrote:
Didn't seem to make any difference.
Thanks for testing. I think it's COW (and I think that implies also
checksums?) that needs to be turned off,
On 2023-04-09 Su 08:39, Thomas Munro wrote:
On Sun, Apr 9, 2023 at 11:25 PM Andrew Dunstan wrote:
Didn't seem to make any difference.
Thanks for testing. I think it's COW (and I think that implies also
checksums?) that needs to be turned off, at least based on experiments
here.
Googling
On Sun, Apr 9, 2023 at 11:25 PM Andrew Dunstan wrote:
> Didn't seem to make any difference.
Thanks for testing. I think it's COW (and I think that implies also
checksums?) that needs to be turned off, at least based on experiments
here.
On Sun, Apr 9, 2023 at 4:52 PM Thomas Munro wrote:
> Here, btrfs seems to be taking a different path that I can't quite
> make out... I see no warning/error about a checksum failure like [1],
> and we apparently managed to read something other than a mix of the
> old and new page contents
On 2023-04-08 Sa 18:50, Thomas Munro wrote:
On Sun, Apr 9, 2023 at 10:17 AM Andrew Dunstan wrote:
I can run the test in isolation, and it's get an error reliably.
Random idea: it looks like you have compression enabled. What if you
turn it off in the directory where the test runs?
Indeed, I can't reproduce this with (our) checksums on. I also can't
reproduce it with O_DIRECT off. I also can't reproduce it if I use
"mkdir pgdata && chattr +C pgdata && initdb -D pgdata" to have a
pgdata directory with copy-on-write and (their) checksums disabled.
But it reproduces quite
On Sat, Apr 08, 2023 at 11:08:16AM -0700, Andres Freund wrote:
> On 2023-04-07 23:04:08 -0700, Andres Freund wrote:
> > There were some failures in CI (e.g. [1] (and perhaps also bf, didn't yet
> > check), about "no unpinned buffers available". I was worried for a moment
> > that this could
On Sun, Apr 9, 2023 at 2:18 PM Andres Freund wrote:
> On 2023-04-09 13:55:33 +1200, Thomas Munro wrote:
> > I think that particular thing might relate to modifications of the
> > user buffer while a write is in progress (breaking btrfs's internal
> > checksums). I don't think we should ever do
Hi,
On 2023-04-09 13:55:33 +1200, Thomas Munro wrote:
> I think that particular thing might relate to modifications of the
> user buffer while a write is in progress (breaking btrfs's internal
> checksums). I don't think we should ever do that ourselves (not least
> because it'd break our own
Thomas Munro writes:
> It's odd, though, if it is their bug and not ours: I'd expect our
> friends in other databases to have hit all that sort of thing years
> ago, since many comparable systems have a direct I/O knob*.
Yeah, it seems moderately likely that it's our own bug ... but th
rive through).
https://lore.kernel.org/linux-btrfs/20230315195231.gw10...@twin.jikos.cz/T/
It's odd, though, if it is their bug and not ours: I'd expect our
friends in other databases to have hit all that sort of thing years
ago, since many comparable systems have a direct I/O knob*. What are
Thomas Munro writes:
> On Sun, Apr 9, 2023 at 10:08 AM Andrew Dunstan wrote:
>> btrfs
> Aha!
Googling finds a lot of suggestions that O_DIRECT doesn't play nice
with btrfs, for example
https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg92824.html
It's not clear to me how much of
On Sun, Apr 9, 2023 at 10:17 AM Andrew Dunstan wrote:
> I can run the test in isolation, and it's get an error reliably.
Random idea: it looks like you have compression enabled. What if you
turn it off in the directory where the test runs? Something like
btrfs property set compression ...
On 2023-04-08 Sa 17:23, Andres Freund wrote:
Hi,
On 2023-04-08 17:10:19 -0400, Tom Lane wrote:
Thomas Munro writes:
Now crake is doing this:
2023-04-08 16:50:03.177 EDT [2023-04-08 16:50:03 EDT 3257645:3]
004_io_direct.pl LOG: statement: select count(*) from t1
2023-04-08 16:50:03.316 EDT
On Sun, Apr 9, 2023 at 10:08 AM Andrew Dunstan wrote:
> btrfs
Aha!
On 2023-04-08 Sa 17:42, Andres Freund wrote:
Hi,
On 2023-04-08 17:31:02 -0400, Tom Lane wrote:
Andres Freund writes:
On 2023-04-08 17:10:19 -0400, Tom Lane wrote:
It's also odd that it's just crake having the issue. It's just a linux host,
afaics.
Indeed. I'm guessing from the compiler
Hi,
On 2023-04-08 17:31:02 -0400, Tom Lane wrote:
> Andres Freund writes:
> > On 2023-04-08 17:10:19 -0400, Tom Lane wrote:
> > It's also odd that it's just crake having the issue. It's just a linux host,
> > afaics.
>
> Indeed. I'm guessing from the compiler version that it's Fedora 37 now
Andres Freund writes:
> On 2023-04-08 17:10:19 -0400, Tom Lane wrote:
>> (BTW, why are the log lines doubly timestamped?)
> It's odd.
Oh, I guess that's intentional, because crake has
'log_line_prefix = \'%m [%s %p:%l] %q%a \'',
> It's also odd that it's just crake having the
Hi,
On 2023-04-08 17:10:19 -0400, Tom Lane wrote:
> Thomas Munro writes:
> Now crake is doing this:
>
> 2023-04-08 16:50:03.177 EDT [2023-04-08 16:50:03 EDT 3257645:3]
> 004_io_direct.pl LOG: statement: select count(*) from t1
> 2023-04-08 16:50:03.316 EDT [2023-04-08 16:50:03 EDT 3257646:1]
On Sun, Apr 9, 2023 at 9:10 AM Tom Lane wrote:
> 2023-04-08 16:50:03.177 EDT [2023-04-08 16:50:03 EDT 3257645:3]
> 004_io_direct.pl LOG: statement: select count(*) from t1
> 2023-04-08 16:50:03.316 EDT [2023-04-08 16:50:03 EDT 3257646:1] ERROR:
> invalid page in block 56 of relation
Thomas Munro writes:
> On Sun, Apr 9, 2023 at 6:55 AM Andres Freund wrote:
>> Given the frequency of failures on this in the buildfarm, I propose using the
>> temporary workaround of using wal_level=replica. That avoids the use of the
>> over-eager log_newpage_range().
> Will do.
Now crake is
On Sun, Apr 9, 2023 at 6:55 AM Andres Freund wrote:
> Given the frequency of failures on this in the buildfarm, I propose using the
> temporary workaround of using wal_level=replica. That avoids the use of the
> over-eager log_newpage_range().
Will do.
Hi,
Given the frequency of failures on this in the buildfarm, I propose using the
temporary workaround of using wal_level=replica. That avoids the use of the
over-eager log_newpage_range().
Greetings,
Andres Freund
Hi,
On 2023-04-07 23:04:08 -0700, Andres Freund wrote:
> There were some failures in CI (e.g. [1] (and perhaps also bf, didn't yet
> check), about "no unpinned buffers available". I was worried for a moment
> that this could actually be relation to the bulk extension patch.
>
> But it looks
On Sat, Apr 8, 2023 at 4:59 PM Thomas Munro wrote:
> On Sat, Apr 8, 2023 at 4:47 PM Thomas Munro wrote:
> > After a bit more copy-editing on docs and comments and a round of
> > automated indenting, I have now pushed this. I will now watch the
> > build farm. I tested on quite a few OSes that
Hi,
On 2023-04-08 16:59:20 +1200, Thomas Munro wrote:
> On Sat, Apr 8, 2023 at 4:47 PM Thomas Munro wrote:
> > After a bit more copy-editing on docs and comments and a round of
> > automated indenting, I have now pushed this. I will now watch the
> > build farm. I tested on quite a few OSes
Thomas Munro writes:
> I did some testing with non-default block sizes, and found a few minor
> things that needed adjustment.
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=crake=2023-04-08%2004%3A42%3A04
This seems like another thing that should not have been pushed mere
hours before
On Sat, Apr 8, 2023 at 4:47 PM Thomas Munro wrote:
> After a bit more copy-editing on docs and comments and a round of
> automated indenting, I have now pushed this. I will now watch the
> build farm. I tested on quite a few OSes that I have access to, but
> this is obviously a very
on direct I/O for
the relevant operations, because such undersized direct I/Os will fail
on common systems.
FATAL: invalid value for parameter "io_direct": "wal"
DETAIL: io_direct is not supported for WAL because XLOG_BLCKSZ is too small
FATAL: invalid value for parameter "
1 - 100 of 113 matches
Mail list logo
