Re: [sqlsmith] Failed assertion in create_gather_path

Hi, At some places, I have observed that we are adding a partial path even when rel's consider_parallel is false. Due to this, the partial path added has parallel_safe set to false and then later in create_gather_path() assertion fails. Attached patch to fix that. On Sun, Apr 8, 2018 at 12:26 A

Re: csv format for psql

2018-04-07 14:23 GMT+02:00 Daniel Verite : > Peter Eisentraut wrote: > > > Another thought: Isn't CSV just the same as unaligned output plus some > > quoting? Could we add a quote character setting and then define --csv > > to be quote-character = " and fieldsep = , ? > > Plus footer set

Re: Checkpoint not retrying failed fsync?

On Sun, Apr 8, 2018 at 5:36 PM, Amit Kapila wrote: > Won't in the success case, you need to delete each member (by > something like bms_del_member) rather than just using bms_free? Thanks for looking at this. Yeah, if requests for segment numbers 0 and 1 were in "requests", and 0 succeeded but t

Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS

On 04/08/2018 05:27 AM, Craig Ringer wrote:> More below, but here's an idea #5: decide InnoDB has the right idea, and go to using a single massive blob file, or a few giant blobs. FYI: MySQL has by default one file per table these days. The old approach with one massive file was a maintenance

Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS

On 8 April 2018 at 11:46, Christophe Pettus wrote: > > > On Apr 7, 2018, at 20:27, Craig Ringer wrote: > > > > Right now I think we're at option (4): If you see anything that smells > like a write error in your kernel logs, hard-kill postgres with -m > immediate (do NOT let it do a shutdown chec

Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS

On 8 April 2018 at 17:41, Andreas Karlsson wrote: > On 04/08/2018 05:27 AM, Craig Ringer wrote:> More below, but here's an > idea #5: decide InnoDB has the right idea, and > >> go to using a single massive blob file, or a few giant blobs. >> > > FYI: MySQL has by default one file per table these

Re: WIP: a way forward on bootstrap data

I wrote: > I'll check back in 24 hours to see if everything still applies. There were a couple more catalog changes that broke patch context, so attached is version 16. -John Naylor v16-bootstrap-data-conversion.tar.gz Description: GNU Zip compressed data

Re: WIP: Covering + unique indexes.

On Sat, Apr 7, 2018 at 4:02 PM, Teodor Sigaev wrote: > Thanks to everyone, pushed. > > Indeed thanks, this will be a nice feature. It is giving me a compiler warning on non-cassert builds using gcc (Ubuntu 5.4.0-6ubuntu1~16.04.9) 5.4.0 20160609: indextuple.c: In function 'index_truncate_tuple':

Re: PATCH: Configurable file mode mask

Hi Michael, On 4/6/18 10:20 AM, Michael Paquier wrote: > On Fri, Apr 06, 2018 at 09:15:15AM -0400, Stephen Frost wrote: >> I'll reply to David's last email (where the latest set of patches were >> included) with my comments/suggestions and I expect we'll be able to get >> those addressed today and

Re: WIP: Covering + unique indexes.

Thank you, fixed Jeff Janes wrote: On Sat, Apr 7, 2018 at 4:02 PM, Teodor Sigaev > wrote: Thanks to everyone, pushed. Indeed thanks, this will be a nice feature. It is giving me a compiler warning on non-cassert builds using gcc (Ubuntu 5.4.0-6ubuntu1~16.04.9) 5

Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS

> On Apr 8, 2018, at 03:30, Craig Ringer wrote: > > These are way more likely than bit flips or other storage level corruption, > and things that we previously expected to detect and fail gracefully for. This is definitely bad, and it explains a few otherwise-inexplicable corruption issues we

Re: WIP: a way forward on bootstrap data

John Naylor writes: > There were a couple more catalog changes that broke patch context, so > attached is version 16. Pushed with a few more adjustments (mostly, another round of copy-editing on bki.sgml). Now we wait to see what the buildfarm thinks, particularly about the MSVC build ... Congr

Verbosity of genbki.pl

(This was discussed in the "way forward on bootstrap data" thread, but we didn't do anything about it.) Traditionally genbki.pl has printed "Writing foo" for every file it writes out. As of 372728b0d that is a heck of a lot more files than it once was, so you get this: $ make -j8 -s Writing fmgr

Re: Verbosity of genbki.pl

1. Print nothing at all. That's more in keeping with our modern build practices, but maybe it's too big a change? 2. Print just one message like "Generating postgres.bki and related files", and I guess a second one for fmgroids.h and related files. I don't have a strong preference. Opinions?

Re: WIP: a way forward on bootstrap data

On 4/9/18, Tom Lane wrote: > Congratulations, and many thanks, to John for seeing this through! > I know it's been a huge amount of work, but we've needed this for years. Many thanks for your review, advice, and additional hacking. Thanks also to Álvaro for review and initial commits, and to all

Re: WIP: Covering + unique indexes.

Thanks to everyone, pushed. Indeed thanks, this will be a nice feature. It is giving me a compiler warning on non-cassert builds using gcc (Ubuntu 5.4.0-6ubuntu1~16.04.9) 5.4.0 20160609: indextuple.c: In function 'index_truncate_tuple': indextuple.c:462:6: warning: unused variable 'indna

Re: WIP: Covering + unique indexes.

On Sun, Apr 8, 2018 at 11:18 AM, Teodor Sigaev wrote: > Thank you, fixed I suggest that we remove some unneeded amcheck tests, as in the attached patch. They don't seem to add anything. -- Peter Geoghegan From 0dbbee5bfff8816cddf86961bf4959192f62f1ff Mon Sep 17 00:00:00 2001 From: Peter Geogheg

Re: WIP: a way forward on bootstrap data

On April 8, 2018 10:19:59 AM PDT, Tom Lane wrote: >John Naylor writes: >> There were a couple more catalog changes that broke patch context, so >> attached is version 16. > >Pushed with a few more adjustments (mostly, another round of >copy-editing >on bki.sgml). Now we wait to see what the bu

Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS

On 8 April 2018 at 04:27, Craig Ringer wrote: > On 8 April 2018 at 10:16, Thomas Munro > wrote: > > If the kernel does writeback in the middle, how on earth is it supposed to > know we expect to reopen the file and check back later? > > Should it just remember "this file had an error" forever, an

Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS

> On Apr 8, 2018, at 14:23, Greg Stark wrote: > > They consider dirty filesystem buffers when there's > hardware failure preventing them from being written "a memory leak". That's not an irrational position. File system buffers are *not* dedicated memory for file system caching; they're being

Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS

On Sun, Apr 08, 2018 at 10:23:21PM +0100, Greg Stark wrote: > On 8 April 2018 at 04:27, Craig Ringer wrote: > > On 8 April 2018 at 10:16, Thomas Munro > > wrote: > > > > If the kernel does writeback in the middle, how on earth is it supposed to > > know we expect to reopen the file and check back

Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS

On Sun, Apr 8, 2018 at 09:38:03AM -0700, Christophe Pettus wrote: > > > On Apr 8, 2018, at 03:30, Craig Ringer > > wrote: > > > > These are way more likely than bit flips or other storage level > > corruption, and things that we previously expected to detect and > > fail gracefully for. > > This i

Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS

> On Apr 8, 2018, at 15:29, Bruce Momjian wrote: > I think the big problem is that we don't have any way of stopping > Postgres at the time the kernel reports the errors to the kernel log, so > we are then returning potentially incorrect results and committing > transactions that might be wrong o

Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS

On 2018-04-08 18:29:16 -0400, Bruce Momjian wrote: > On Sun, Apr 8, 2018 at 09:38:03AM -0700, Christophe Pettus wrote: > > > > > On Apr 8, 2018, at 03:30, Craig Ringer > > > wrote: > > > > > > These are way more likely than bit flips or other storage level > > > corruption, and things that we prev

Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS

> On Apr 8, 2018, at 16:16, Andres Freund wrote: > We don't panic that way when getting IO > errors during reads either, and they're more likely to be persistent > than errors during writes (because remapping on storage layer can fix > issues, but not during reads). There is a distinction to be

Re: Rewriting the test of pg_upgrade as a TAP test - take two

On Tue, Apr 03, 2018 at 03:04:10PM +0900, Michael Paquier wrote: > At the end, it seems to me that what I am proposing is themost readable > approach, and with proper documentation things should be handled > finely... Or there is an approach you have in mind I do not foresee? With feature freeze

Re: [HACKERS] Optional message to user when terminating/cancelling backend

On Fri, Apr 06, 2018 at 11:18:34AM +0200, Daniel Gustafsson wrote: > Yep, I completely agree. Attached are patches with the quotes removed and > rebased since Oids were taken etc. I still find this idea interesting for plugin authors. However, as feature freeze for v11 is in effect, I am marking

Re: pgsql: Merge catalog/pg_foo_fn.h headers back into pg_foo.h headers.

On Sun, Apr 08, 2018 at 06:35:39PM +, Tom Lane wrote: > Merge catalog/pg_foo_fn.h headers back into pg_foo.h headers. > > Traditionally, include/catalog/pg_foo.h contains extern declarations > for functions in backend/catalog/pg_foo.c, in addition to its function > as the authoritative definit

Warnings and uninitialized variables in TAP tests

Hi all, While looking at the output of the TAP tests, I have seen warnings like the following: Use of uninitialized value $target_lsn in concatenation (.) or string at /home/foo/git/postgres/src/bin/pg_rewind/../../../src/test/perl/PostgresNode.pm line 1540. [...] ./src/bin/pg_basebackup/tmp_check

Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS

On 9 April 2018 at 05:28, Christophe Pettus wrote: > > > On Apr 8, 2018, at 14:23, Greg Stark wrote: > > > > They consider dirty filesystem buffers when there's > > hardware failure preventing them from being written "a memory leak". > > That's not an irrational position. File system buffers ar

Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS

On 9 April 2018 at 06:29, Bruce Momjian wrote: > > I think the big problem is that we don't have any way of stopping > Postgres at the time the kernel reports the errors to the kernel log, so > we are then returning potentially incorrect results and committing > transactions that might be wrong

Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS

Hi, On 2018-04-08 16:27:57 -0700, Christophe Pettus wrote: > > On Apr 8, 2018, at 16:16, Andres Freund wrote: > > We don't panic that way when getting IO > > errors during reads either, and they're more likely to be persistent > > than errors during writes (because remapping on storage layer can

Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS

On 9 April 2018 at 07:16, Andres Freund wrote: > > I think the danger presented here is far smaller than some of the > statements in this thread might make one think. Clearly it's not happening a huge amount or we'd have a lot of noise about Pg eating people's data, people shouting about how u

Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS

On 2018-04-09 10:00:41 +0800, Craig Ringer wrote: > I suspect we've written off a fair few issues in the past as "it'd bad > hardware" when actually, the hardware fault was the trigger for a Pg/kernel > interaction bug. And blamed containers for things that weren't really the > container's fault. B

Re: pgsql: Merge catalog/pg_foo_fn.h headers back into pg_foo.h headers.

Michael Paquier writes: > On Sun, Apr 08, 2018 at 06:35:39PM +, Tom Lane wrote: >> Merge catalog/pg_foo_fn.h headers back into pg_foo.h headers. > This patch or one of its relatives has visibly broken parallel builds > for me. "make -j 4 install" directly called after a configure complains:

Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS

On 9 April 2018 at 10:06, Andres Freund wrote: > > > And in many failure modes there's no reason to expect any data loss at > all, > > like: > > > > * Local disk fills up (seems to be safe already due to space reservation > at > > write() time) > > That definitely should be treated separately. >

Re: [HACKERS] path toward faster partition pruning

While looking at the docs in [1], I saw that we still mention: 4. Ensure that the constraint_exclusion configuration parameter is not disabled in postgresql.conf. If it is, queries will not be optimized as desired. This is no longer true. The attached patch removed it. [1] https://www.postgresql

Re: [HACKERS] [BUGS] Bug in Physical Replication Slots (at least 9.5)?

Hello, I added this as Older Bugs in Open items. (I believe it's legit.) The latest patch still applies on the HEAD with some offsets. At Tue, 23 Jan 2018 18:50:00 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20180123.185000.232069302.horiguchi.kyot...@lab.ntt.co.jp> > At Fri, 19 J

Re: [HACKERS] [BUGS] Bug in Physical Replication Slots (at least 9.5)?

On Mon, Apr 09, 2018 at 01:26:54PM +0900, Kyotaro HORIGUCHI wrote: > Hello, I added this as Older Bugs in Open items. (I believe it's > legit.) Yes, I think that's adapted. I am hesitating to do the same thing with all the other patches marked as bug fixes. -- Michael signature.asc Description:

Fix pg_rewind which can be run as root user

Hi all, I was just going through pg_rewind's code, and noticed the following pearl: /* * Don't allow pg_rewind to be run as root, to avoid overwriting the * ownership of files in the data directory. We need only check for root * -- any other user won't have sufficient permission

Re: [HACKERS] [BUGS] Bug in Physical Replication Slots (at least 9.5)?

At Mon, 9 Apr 2018 13:59:45 +0900, Michael Paquier wrote in <20180409045945.gb1...@paquier.xyz> > On Mon, Apr 09, 2018 at 01:26:54PM +0900, Kyotaro HORIGUCHI wrote: > > Hello, I added this as Older Bugs in Open items. (I believe it's > > legit.) > > Yes, I think that's adapted. I am hesitating

Re: [HACKERS] logical decoding of two-phase transactions

> I object. And I'm negatively surprised that this is even considered. > I am also a bit surprised.. > This is a complicated patch that has been heavily reworked in the last > few days to, among other things, address objections that have first been > made months ago ([1]). There we nontrivial bu

Re: pgsql: Merge catalog/pg_foo_fn.h headers back into pg_foo.h headers.

On Sun, Apr 08, 2018 at 11:05:09PM -0400, Tom Lane wrote: > Hm. I'd tested "make -j all", but not going directly to "install". > Does it help if you add > > $(SUBDIRS:%=install-%-recurse): | submake-generated-headers > > to src/Makefile? > > (That seems like a bit of a brute-force solution, thou

Re: WIP: Covering + unique indexes.

Thank you, pushed. Peter Geoghegan wrote: On Sun, Apr 8, 2018 at 11:18 AM, Teodor Sigaev wrote: Thank you, fixed I suggest that we remove some unneeded amcheck tests, as in the attached patch. They don't seem to add anything. -- Teodor Sigaev E-mail: teo...@sigaev.ru

Re: [HACKERS] Restrict concurrent update/delete with UPDATE of partition key

On Sun, Apr 8, 2018 at 2:04 AM, Andres Freund wrote: > On 2018-04-07 20:13:36 +0530, amul sul wrote: >> Attached is the patch does the renaming of this tests -- need to apply >> to the top of v10 patch[1]. > > These indeed are a bit too long, so I went with the numbers. I've > pushed the patch

Re: [HACKERS] GSoC 2017: weekly progress reports (week 6)

On 16/03/18 00:26, Alexander Korotkov wrote: On Tue, Mar 13, 2018 at 3:26 PM, Andrey Borodin wrote: On 13/03/18 14:02, Alexander Korotkov wrote: And what happen if somebody concurrently set (fastupdate = on)? Can we miss conflicts because of that? No, AccessExclusiveLock will prevent this ki