Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

On 2016-05-05 09:32, Fabien COELHO wrote: I note that C99 specifically mentions this as something a compiler might warn about: [...] Indeed. Neither gcc nor clang emit such warnings... but they might some day, which would be a blow for my suggestion! For what it's worth, newer versions of

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

Hello Tom, I understood the point and I do not see real disadvantages. The C standard really says that an enum is an int, and compilers just do that. No, it doesn't say that, and compilers don't just do that. A compiler is specifically allowed to store an enum in char or short if the enum's

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

* Tom Lane (t...@sss.pgh.pa.us) wrote: > Stephen Frost writes: > > * Tom Lane (t...@sss.pgh.pa.us) wrote: > >> * debugger ability to print variables symbolically > > > I might be misunderstanding what you're getting at here, but if you want > > to be able to use #define'd

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

Stephen Frost writes: > * Tom Lane (t...@sss.pgh.pa.us) wrote: >> * debugger ability to print variables symbolically > I might be misunderstanding what you're getting at here, but if you want > to be able to use #define'd values using their name, you can get that by >

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

Tom, * Tom Lane (t...@sss.pgh.pa.us) wrote: > * debugger ability to print variables symbolically I might be misunderstanding what you're getting at here, but if you want to be able to use #define'd values using their name, you can get that by compiling with -g3. With -g3 and gdb, you can do

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

Fabien COELHO writes: > I understood the point and I do not see real disadvantages. The C standard > really says that an enum is an int, and compilers just do that. No, it doesn't say that, and compilers don't just do that. A compiler is specifically allowed to store an

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

Hello Tom, There's not really a point in using an enum if you use neither the type (which you shouldn't because if you or the bitmask value you have types outside the range of the enum), nor the autogenerated numbers. I do not think that there is such a constraint in C, you can use the enum

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

Fabien COELHO writes: >> There's not really a point in using an enum if you use neither the type >> (which you shouldn't because if you or the bitmask value you have types >> outside the range of the enum), nor the autogenerated numbers. > I do not think that there is such

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

On 4 May 2016 at 09:59, Andres Freund wrote: > On 2016-04-28 17:41:29 +0100, Thom Brown wrote: > > I've noticed another breakage, which I can reproduce consistently. > > Thanks for the testing! I pushed a fix for this. This wasn't actually > an issue in the original patch,

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

On 2016-05-04 07:46:42 +0200, Fabien COELHO wrote: > >Well, TRUNCATED doesn't entirely work, there's I think some cases where > >this currently also applies to deleted relations. I kind of like the > >unintuitive sounding name, because it's really a dangerous options (any > >further mdnblocks will

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

On 2016-04-28 17:41:29 +0100, Thom Brown wrote: > I've noticed another breakage, which I can reproduce consistently. Thanks for the testing! I pushed a fix for this. This wasn't actually an issue in the original patch, but a too strict test added in my fix. Greetings, Andres Freund -- Sent

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

Hello Andres, An enum doesn't have a benefit for a bitmask imo - you can't "legally" use it as a type for functions accepting the bitmask. I do not understand. I suggested to use enum to enumerate the bitmask constants, ISTM that it does not preclude to use it as a bitmask as you do, it is

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

Hi, On 2016-05-03 07:24:46 +0200, Fabien COELHO wrote: > >>I'm unsure about switching enum to #define, could be an enum still with > >>explicit values set, something like: > > > >An enum doesn't have a benefit for a bitmask imo - you can't "legally" > >use it as a type for functions accepting the

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

Hello, I'm unsure about switching enum to #define, could be an enum still with explicit values set, something like: An enum doesn't have a benefit for a bitmask imo - you can't "legally" use it as a type for functions accepting the bitmask. I do not understand. I suggested to use enum to

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

Hi, On 2016-05-03 00:05:35 +0200, Fabien COELHO wrote: > Maybe consider checking for the exclusivity explicitely? I thought about it, and decided it's not worth it. Requiring one of those to be specified seems stringent enough. > I'm unsure about switching enum to #define, could be an enum

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

Hello Andres, I'm not sure this is the best way to go about this. I can see valid arguments for *always* using _mdfd_openseg() in mdsync(); and I'm wondering whether we shouldn't make EXTENSION_* into a bitmask (extend,extend_recovery,return_null,open_deleted). I thought about that when I

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

On 2016-05-02 22:00:14 +0200, Fabien COELHO wrote: > I'm wondering that if _mdfd_openseg may return NULL, then ISTM that > "opened_directly" should logically be false, because it was not opened? > > If so, maybe consider something like: > > opened_directy = (seg != NULL); Hm, don't care

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

Hello Andres, Sure, attached. For what it is worth, it works for me on head. I'm wondering that if _mdfd_openseg may return NULL, then ISTM that "opened_directly" should logically be false, because it was not opened? If so, maybe consider something like: opened_directy = (seg !=

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

On 2016-05-02 12:44:53 -0400, Robert Haas wrote: > On Mon, May 2, 2016 at 12:41 PM, Andres Freund wrote: > > On 2016-05-02 12:29:45 -0400, Robert Haas wrote: > >> On Fri, Apr 29, 2016 at 7:58 PM, Andres Freund wrote: > >> > Basically the reason for the

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

On Mon, May 2, 2016 at 12:41 PM, Andres Freund wrote: > On 2016-05-02 12:29:45 -0400, Robert Haas wrote: >> On Fri, Apr 29, 2016 at 7:58 PM, Andres Freund wrote: >> > Basically the reason for the problem is that mdsync() needs to access >> > "formally

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

On 2016-05-02 12:29:45 -0400, Robert Haas wrote: > On Fri, Apr 29, 2016 at 7:58 PM, Andres Freund wrote: > > Basically the reason for the problem is that mdsync() needs to access > > "formally non-existant segments" (as in ones where previous segments are > > < RELSEG_SIZE),

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

On Fri, Apr 29, 2016 at 7:58 PM, Andres Freund wrote: > On 2016-04-28 17:41:29 +0100, Thom Brown wrote: >> I've noticed another breakage, which I can reproduce consistently. > >> 2016-04-28 17:36:08 BST [18108]: [47-1] user=,db=,client= DEBUG: could not >> fsync file

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

On 2016-04-28 17:41:29 +0100, Thom Brown wrote: > I've noticed another breakage, which I can reproduce consistently. > 2016-04-28 17:36:08 BST [18108]: [47-1] user=,db=,client= DEBUG: could not > fsync file "base/24581/24594.1" but retrying: No such file or directory > 2016-04-28 17:36:08 BST

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

An interesting complement about the previous failing test: Although I disabled the "flushing" feature... sh> grep flush_after xxx/postgresql.conf bgwriter_flush_after = 0# 0 disables, backend_flush_after = 0 # 0 disables, wal_writer_flush_after = 0

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

createdb: database creation failed: ERROR: checkpoint request failed HINT: Consult recent messages in the server log for details. Attached is a simpler/faster version, based on the previous script, I just added a CHECKPOINT after the EXPLAIN ANALYSE. It fails on head with: HINT:

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

On 2016-04-28 17:41:29 +0100, Thom Brown wrote: > I've noticed another breakage, which I can reproduce consistently. > > createdb pgbench > pgbench -i -s 100 --unlogged-tables pgbench > psql -f pgbench_partitions.sql pgbench > vacuumdb -z pgbench > createdb test > > Which produces: > >

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

Hi, On 2016-04-25 20:53:26 +0200, Fabien COELHO wrote: > I have just a small naming point: > > /* ereport if segment not present, create in recovery */ > EXTENSION_FAIL, > /* return NULL if not present, create in recovery */ > EXTENSION_RETURN_NULL, > /* return NULL if not present */ >

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

On 2016-04-26 17:25:18 -0400, Robert Haas wrote: > On Mon, Apr 25, 2016 at 4:57 PM, Andres Freund wrote: > > On 2016-04-25 16:29:36 -0400, Robert Haas wrote: > >> On Mon, Apr 25, 2016 at 2:05 PM, Andres Freund wrote: > >> > Well, I posted a patch. I'd have

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

On Mon, Apr 25, 2016 at 4:57 PM, Andres Freund wrote: > On 2016-04-25 16:29:36 -0400, Robert Haas wrote: >> On Mon, Apr 25, 2016 at 2:05 PM, Andres Freund wrote: >> > Well, I posted a patch. I'd have applied it too (after addressing your >> > comments

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

On 2016-04-25 16:29:36 -0400, Robert Haas wrote: > On Mon, Apr 25, 2016 at 2:05 PM, Andres Freund wrote: > > Well, I posted a patch. I'd have applied it too (after addressing your > > comments obviously), except that there's some interdependencies with the > > nsmg > 0 thread

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

On Mon, Apr 25, 2016 at 2:05 PM, Andres Freund wrote: > Well, I posted a patch. I'd have applied it too (after addressing your > comments obviously), except that there's some interdependencies with the > nsmg > 0 thread (some of my tests fail spuriously without that > fixed).

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

Hello Andres, Fixes it for me too. Yep, for me too. I'm not at ease with the part of the API the code is dealing with, so I do not feel like a competent reviewer. I agree with the "more comments" suggested by Robert. I have just a small naming point: /* ereport if segment not

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

Hi, On 2016-04-25 12:38:36 -0400, Robert Haas wrote: > I think that the point of my message is exactly what I said in my > message. This isn't really about the last couple of days. The issue > was reported on March 20th. On March 31st, Noah asked you for a plan > to get it fixed by April 7th.

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

On Mon, Apr 25, 2016 at 11:56 AM, Andres Freund wrote: > On 2016-04-25 08:55:54 -0400, Robert Haas wrote: >> Andres, this issue has now been open for more than a month, which is >> frankly kind of ridiculous given the schedule we're trying to hit for >> beta. Do you think

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

On 2016-04-25 08:55:54 -0400, Robert Haas wrote: > Andres, this issue has now been open for more than a month, which is > frankly kind of ridiculous given the schedule we're trying to hit for > beta. Do you think it's possible to commit something RSN without > compromising the quality of

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

On 22 April 2016 at 18:07, Andres Freund wrote: > On 2016-04-14 11:09:29 +0530, Abhijit Menon-Sen wrote: >> At 2016-04-12 09:00:57 -0400, robertmh...@gmail.com wrote: >> > >> > On Mon, Apr 11, 2016 at 1:17 PM, Andres Freund wrote: >> > > >> > > 3) Actually

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

On Fri, Apr 22, 2016 at 5:26 PM, Robert Haas wrote: > On Fri, Apr 22, 2016 at 1:07 PM, Andres Freund wrote: >> The attached patch basically adds the segment size checks to >> _mdfd_getseg(), and doesn't perform extension, even in recovery, if >>

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

On Fri, Apr 22, 2016 at 1:07 PM, Andres Freund wrote: > The attached patch basically adds the segment size checks to > _mdfd_getseg(), and doesn't perform extension, even in recovery, if > EXTENSION_REALLY_RETURN_NULL is passed. > > This fixes the issue for me, both in the

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

On 2016-04-14 11:09:29 +0530, Abhijit Menon-Sen wrote: > At 2016-04-12 09:00:57 -0400, robertmh...@gmail.com wrote: > > > > On Mon, Apr 11, 2016 at 1:17 PM, Andres Freund wrote: > > > > > > 3) Actually handle the case of the last open segment not being > > >RELSEG_SIZE

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

On Thu, Apr 21, 2016 at 9:45 AM, Andres Freund wrote: >> My attempts to test things were also singularly unrewarding. Even >> after messing with the filesystem in various ways, I couldn't figure >> out exactly how this makes a difference. > > What do you mean by this? You

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

On 2016-04-15 13:42:34 -0400, Robert Haas wrote: > On Thu, Apr 14, 2016 at 1:39 AM, Abhijit Menon-Sen > wrote: > > At 2016-04-12 09:00:57 -0400, robertmh...@gmail.com wrote: > >> > >> On Mon, Apr 11, 2016 at 1:17 PM, Andres Freund wrote: > >> > > >> >

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

On Sat, Apr 16, 2016 at 03:28:23PM -0700, Andres Freund wrote: > On 2016-04-16 18:23:01 -0400, Noah Misch wrote: > > On Tue, Apr 12, 2016 at 09:00:57AM -0400, Robert Haas wrote: > > > On Mon, Apr 11, 2016 at 1:17 PM, Andres Freund wrote: > > > > I can think of a number of

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

On 2016-04-16 18:23:01 -0400, Noah Misch wrote: > On Tue, Apr 12, 2016 at 09:00:57AM -0400, Robert Haas wrote: > > On Mon, Apr 11, 2016 at 1:17 PM, Andres Freund wrote: > > > I can think of a number of relatively easy ways to address this: > > > 1) Just zap (or issue?) all

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

On Tue, Apr 12, 2016 at 09:00:57AM -0400, Robert Haas wrote: > On Mon, Apr 11, 2016 at 1:17 PM, Andres Freund wrote: > > I can think of a number of relatively easy ways to address this: > > 1) Just zap (or issue?) all pending flush requests when getting an > >

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

On Thu, Apr 14, 2016 at 1:39 AM, Abhijit Menon-Sen wrote: > At 2016-04-12 09:00:57 -0400, robertmh...@gmail.com wrote: >> >> On Mon, Apr 11, 2016 at 1:17 PM, Andres Freund wrote: >> > >> > 3) Actually handle the case of the last open segment not being >>

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

At 2016-04-12 09:00:57 -0400, robertmh...@gmail.com wrote: > > On Mon, Apr 11, 2016 at 1:17 PM, Andres Freund wrote: > > > > 3) Actually handle the case of the last open segment not being > >RELSEG_SIZE properly in _mdfd_getseg() - mdnblocks() does so. > > #3 seems like

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

On Mon, Apr 11, 2016 at 1:17 PM, Andres Freund wrote: > I can think of a number of relatively easy ways to address this: > 1) Just zap (or issue?) all pending flush requests when getting an >smgrinval/smgrclosenode > 2) Do 1), but filter for the closed relnode > 3)

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

Hello Andres, I can think of a number of relatively easy ways to address this: 1) Just zap (or issue?) all pending flush requests when getting an smgrinval/smgrclosenode 2) Do 1), but filter for the closed relnode 3) Actually handle the case of the last open segment not being RELSEG_SIZE

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

On 2016-04-11 13:04:48 -0400, Robert Haas wrote: > You're right, but I think that's more because I didn't say it > correctly than because you haven't done something novel. Could be. > DROP and > relation truncation know about shared buffers, and they go clear > blocks that that might be

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

On Mon, Apr 11, 2016 at 12:50 PM, Andres Freund wrote: > On 2016-04-04 12:45:25 -0400, Robert Haas wrote: >> Well, I agree that it's pretty strange that _mdfd_getseg() makes no >> such check, but I still don't think I understand what's going on here. >> Backends shouldn't be

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

On 2016-04-04 12:45:25 -0400, Robert Haas wrote: > Well, I agree that it's pretty strange that _mdfd_getseg() makes no > such check, but I still don't think I understand what's going on here. > Backends shouldn't be requesting nonexistent blocks from a relation - > higher-level safeguards, like

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

On Mon, Apr 04, 2016 at 12:45:25PM -0400, Robert Haas wrote: > Backends shouldn't be requesting nonexistent blocks from a relation - > higher-level safeguards, like holding AccessExclusiveLock before > trying to complete a DROP or TRUNCATE - are supposed to prevent that. > If this patch is causing

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

On Sat, Mar 26, 2016 at 8:39 AM, Andres Freund wrote: > On 2016-03-25 12:02:05 -0400, Robert Haas wrote: >> Gosh, that's surprising. I wonder if that just revealed an underlying >> issue rather than creating it. > > I think that's the case; it's just somewhat unlikely to hit