Re: [HACKERS] Releasing in September

2016-01-20 Thread Andres Freund
> I admit, I may have grabbed your comment out of an unrelated portion > of the thread. Ceterum censeo Carthaginem esse delendam. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Releasing in September

2016-01-20 Thread Andres Freund
On 2016-01-20 09:42:28 -0800, Joshua D. Drake wrote: > Don't review patches? Don't submit patches. Absolutely. While I think encouraging reviewers is a good thing, it seems independent of requiring contributors to do quid-pro-quo reviews. There'll always be people too busy or uninterested in revie

Re: [HACKERS] Releasing in September

2016-01-20 Thread Andres Freund
On 2016-01-20 09:48:24 -0800, David E. Wheeler wrote: > There will always be patches desirable-enough that they will be reviewed > whether or not the submitter reviewed other patches. I think that's actually getting less and less true. By now the really desirable-enough features imply so much wor

Re: [HACKERS] Releasing in September

2016-01-20 Thread Andres Freund
On 2016-01-20 19:00:05 +0100, Magnus Hagander wrote: > And you want the actual patches, and not just a count? I think the actual patches makes sense, because reviewing one 200KB patch obviously is something different than reviewing 3 onelinesrs. -- Sent via pgsql-hackers mailing list (pgsql-hac

Re: [HACKERS] Releasing in September

2016-01-20 Thread Andres Freund
On 2016-01-20 13:03:05 -0500, Tom Lane wrote: > I am not sure what exactly ought to be different about them [small > patches], but probably something should. I think for small patches, > we are using the CF app mostly to be sure things don't fall through > the cracks, but maybe we don't need the w

Re: [HACKERS] Releasing in September

2016-01-20 Thread Andres Freund
On 2016-01-20 13:13:29 -0500, Robert Haas wrote: > (3) Andres's multixact reworks landed quite late and IMHO were not > safe enough to back-patch, yet they needed to be in the release. In > my view, the last major fix for At least in that case I was, for a long while, basically hoping for more in

Re: [HACKERS] Releasing in September

2016-01-20 Thread Andres Freund
On 2016-01-20 18:53:54 +, Simon Riggs wrote: > What is the point in having a special mailing list to discuss the release > schedule plus a face-to-face dev meeting to discuss this if we are going to > discuss it here? That list exists to discuss concrete releases, and is separate so we e.g. ca

Re: [HACKERS] Releasing in September

2016-01-20 Thread Andres Freund
On 2016-01-20 18:02:20 -0500, Bruce Momjian wrote: > The two sobering blockers I can remember were multi-xact (9.3) and JSONB > compression (9.4). I don't see how those compare. The multixact stuff was all discovered a fair bit after the release, the compression stuff pre release. -- Sent via p

Re: [HACKERS] checkpointer continuous flushing

2016-01-21 Thread Andres Freund
On 2016-01-21 11:33:15 +0530, Amit Kapila wrote: > On Wed, Jan 20, 2016 at 9:07 PM, Andres Freund wrote: > > I don't think it's strongly related - the contention here is on read > > access to the clog, not on write access. > > Aren't reads on clog contended

[HACKERS] log_checkpoint's "0 transaction log file(s) added" is extremely misleading

2016-01-21 Thread Andres Freund
her remove that part of the log output, or make it display the number of segments added since the beginning of the checkpoint. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] log_checkpoint's "0 transaction log file(s) added" is extremely misleading

2016-01-21 Thread Andres Freund
On January 22, 2016 3:29:44 AM GMT+01:00, Simon Riggs wrote: >On 22 January 2016 at 01:12, Andres Freund wrote: > >> Hi, >> >> While in theory correct, I think $subject is basically meaningless >> because other backends may have added thousands of new se

Re: [HACKERS] silent data loss with ext4 / all current versions

2016-01-22 Thread Andres Freund
On 2016-01-22 21:32:29 +0900, Michael Paquier wrote: > Group shot with 3), 4) and 5). Well, there is no data loss here, > putting me in the direction of considering this addition of an fsync > as an optimization and not a bug. I think this is an extremely weak argument. The reasoning when exactly

Re: [HACKERS] WIP: Failover Slots

2016-01-22 Thread Andres Freund
On 2016-01-22 11:40:24 -0500, Robert Haas wrote: > It occurred to me to wonder if it might be better to > propagate logical slots partially or entirely outside the WAL stream, > because with this design you will end up with the logical slots on > every replica, including cascaded replicas, and I'm

Re: [HACKERS] Releasing in September

2016-01-22 Thread Andres Freund
On 2016-01-22 08:40:28 -0600, Jim Nasby wrote: > Ideally reviewers shouldn't be doing any testing, because the tests > that are part of the patch should answer every question they would > have, but I don't see that happening until we have a separate > automation-only target that we don't care how l

Re: [HACKERS] Releasing in September

2016-01-22 Thread Andres Freund
On 2016-01-22 08:18:45 -0600, Jim Nasby wrote: > Personally, I don't see why we have our scarcest resource doing what is > essentially a project management task, especially when at least one > commercial company has offered to donate paid staff time. Because so far all CFs that weren't managed by

Re: [HACKERS] Releasing in September

2016-01-22 Thread Andres Freund
On 2016-01-22 08:50:15 -0600, Jim Nasby wrote: > I think that's a great way to ensure we shrink the pool of reviewers when > someone works on a patch and then it goes nowhere. True, it really sucks. But what's your counter proposal? Commitfests dragging on forever, and people burning out on contin

Re: [HACKERS] Re: pglogical_output - a general purpose logical decoding output plugin

2016-01-24 Thread Andres Freund
On 2016-01-18 21:47:27 +, Tomasz Rybak wrote: > We might also think about changing name of plugin to something resembling > "logical_streaming_decoder" or even "logical_streamer" FWIW, I find those proposals unconvincing. Not that pglogical_output is grand, but "streaming decoder" or "logical

Re: [HACKERS] Set search_path + server-prepared statements = cached plan must not change result type

2016-01-25 Thread Andres Freund
On 2016-01-25 12:39:29 -0500, Robert Haas wrote: > What is the ideal behavior, in your view? FWIW, I think that for a lot of practical cases the previous behaviour, where a prepared statement was defined in the context of the search path set during the PREPARE, made a lot more sense. The current

Re: [HACKERS] 2016-01 Commitfest

2016-01-25 Thread Andres Freund
On 2016-01-25 13:36:04 -0300, Alvaro Herrera wrote: > We still have 41 patches that haven't gotten enough review though. The > bad part about it is that there's a number of patches that have been > bouncing for many commitfests now. Here's a list of the patches with > the most such actions (both

Re: [HACKERS] 2016-01 Commitfest

2016-01-25 Thread Andres Freund
On 2016-01-25 15:02:02 -0300, Alvaro Herrera wrote: > Andres Freund wrote: > > On 2016-01-25 13:36:04 -0300, Alvaro Herrera wrote: > > > * https://commitfest.postgresql.org/8/260/ > > > checkpoint continuous flushing > > > > FWIW, I've been worki

Re: [HACKERS] Releasing in September

2016-01-25 Thread Andres Freund
Hi, On 2016-01-26 00:26:18 -0600, Joshua Berkus wrote: > Let's do quarterly development releases and supported production > releases every 18 months. > A 3-month release cycle would let people see their code go into the wild a > lot faster; basically we'd do a CF then a development release. The

Re: [HACKERS] pg_lsn cast to/from int8

2016-01-26 Thread Andres Freund
On 2016-01-26 14:56:21 +0100, Magnus Hagander wrote: > Is there a reason we don't have casts between int8 and pg_lsn? AFAICT it > works fine if I create the cast manually... Is it because of > signed/unsigned if people have really really many transactions? What for do you want that cast? Yes, the

Re: [HACKERS] Proposal:Use PGDLLEXPORT for libpq

2016-01-27 Thread Andres Freund
On 2016-01-27 21:05:06 +0800, Craig Ringer wrote: > On 27 January 2016 at 20:54, Alvaro Herrera > wrote: > > > If so many problems with MSVC can discard his support of Postgres? > I strongly disagree. MSVC is a high quality compiler and the primary tool > for the platform. I think it's pretty

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-01-29 Thread Andres Freund
XLOG_CHECKPOINT_ONLINE); so the above condition doesn't really something we want to rely on. Am I missing what you're trying to do? Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Template for commit messages

2016-01-29 Thread Andres Freund
On 2016-01-29 04:42:08 -0500, Bruce Momjian wrote: > On Thu, Jan 28, 2016 at 07:30:58PM -0500, Andrew Dunstan wrote: > > I have no prejudice in this area, other than one in favor of any > > rules being fairly precise. As for nuances, I guess they can be > > specified in the commit message. The one

Re: [HACKERS] pglogical_output - a general purpose logical decoding output plugin

2016-01-29 Thread Andres Freund
that aren't in the table is > + * entirely normal, since there's no way to unregister for an > + * invalidation event. So we don't care if it's found or not. > + */ > + (void) hash_search(RelMetaCache, &relid, HASH_REMOVE, NULL); > + } So, I don't buy this, like at all. The cache entry is passed to functions, while we call output functions and such. Which in turn can cause cache invalidations to be processed. > +struct PGLRelMetaCacheEntry > +{ > + Oid relid; > + /* Does the client have this relation cached? */ > + bool is_cached; > + /* Field for API plugin use, must be alloc'd in decoding context */ > + void *api_private; > +}; I don't see how api_private can safely be used. At the very least it needs a lot more documentation about memory lifetime rules and such. Afaics we'd just forever leak memory atm. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Template for commit messages

2016-02-01 Thread Andres Freund
On 2016-01-28 06:03:02 -0500, Bruce Momjian wrote: > Reported-by: > Bug: > Author: > Reviewed-by: > Tested-by: > Backpatch-through: I personally, and I realize that I'm likely alone on that, would really like to see references to relevant threads. A year after a

Re: [HACKERS] Template for commit messages

2016-02-01 Thread Andres Freund
On 2016-02-01 20:53:56 +0900, Michael Paquier wrote: > Last time this was suggested though there were concerns regarding the > 72-80 character limit per line in a commit message. That seems like a misdirect. The limit is there to keep things readable, not because there's a hard technical limit. So

Re: [HACKERS] Template for commit messages

2016-02-01 Thread Andres Freund
On 2016-02-01 12:56:21 +0100, Magnus Hagander wrote: > Let's just assume that we can fix that part. As in, we can expose either an > internal db id or a short hash or something, from the archives server. We > could then have that visible/linkable directly on the archives page, and > one could copy/

Re: [HACKERS] silent data loss with ext4 / all current versions

2016-02-01 Thread Andres Freund
archiveReady, > archiveDone))); > > + /* > + * Make sure the rename is permanent by fsyncing the parent > + * directory. > + */ > + fsync_fname(XLOGDIR "/archive_status", true); > return; > } Afaics the AllocateFile() call below is not protected at all, no? How about introducing a 'safe_rename()' that does something roughly akin to: fsync(oldname); fsync(fname) || true; rename(oldfname, fname); fsync(fname); fsync(basename(fname)); I'd rather have that kind of logic somewhere once, instead of repeated a dozen times... Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] silent data loss with ext4 / all current versions

2016-02-01 Thread Andres Freund
On 2016-02-01 16:49:46 +0100, Alvaro Herrera wrote: > Yeah. On 9.4 there are already some conflicts, and I'm sure there will > be more in almost each branch. Does anyone want to volunteer for > producing per-branch versions? > The next minor release is to be tagged next week and it'd be good to

Re: [HACKERS] Move PinBuffer and UnpinBuffer to atomics

2016-02-01 Thread Andres Freund
; Attached patch is rebased and have better comments. > Also, there is one comment which survive since original version by Andres. > > /* Add exponential backoff? Should seldomly be contended tho. */ > > > Andres, did you mean we should twice the delay with each unsuccessful tr

Re: [HACKERS] Move PinBuffer and UnpinBuffer to atomics

2016-02-01 Thread Andres Freund
On 2016-02-01 22:35:06 +0100, Alvaro Herrera wrote: > I know Andres is already pretty busy with the checkpoint flush patch and > I very much doubt he will be able to give this patch a lot of attention > in the short term. Moving to next CF. Yea, there's no realistic chance I'll be able to take ca

[HACKERS] "using previous checkpoint record at" maybe not the greatest idea?

2016-02-01 Thread Andres Freund
dly wrong. Now that obviously doesn't have a very large significance, because in the situations where it "just worked" are unlikely to be reported... Am I missing a reason for doing this by default? Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pg

[HACKERS] Raising the checkpoint_timeout limit

2016-02-01 Thread Andres Freund
as really shrunk rather drastically due to faster cpus and architectural improvements in postgres (bgwriter, separate checkpointer/bgwriter, restartpoints, ...). Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription:

[HACKERS] checkpoints after database start/immediate checkpoints

2016-02-01 Thread Andres Freund
instead be that we start checkpoint_timeout * _target before the next timeout? Afaics that'd work more graceful in the face of restarts and forced checkpoints. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http

Re: [HACKERS] "using previous checkpoint record at" maybe not the greatest idea?

2016-02-01 Thread Andres Freund
plug during checkpoint) and one > that supposedly completed without interruption but then was somehow > corrupted (solar flares). The former seem legitimate for auto-skip while > the later do not. I don't think such a distinction is really possible (or necessary). If pg_control is corrupt

Re: [HACKERS] silent data loss with ext4 / all current versions

2016-02-01 Thread Andres Freund
On 2016-02-02 09:56:40 +0900, Michael Paquier wrote: > And there is no actual risk of data loss Huh? - Andres -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Raising the checkpoint_timeout limit

2016-02-02 Thread Andres Freund
On 2016-02-01 23:16:16 -0500, Noah Misch wrote: > On Tue, Feb 02, 2016 at 01:13:20AM +0100, Andres Freund wrote: > > I'm not sure what'd actually be a good upper limit. I'd be inclined to > > even go to as high as a week or so. A lot of our settings have > > u

Re: [HACKERS] Raising the checkpoint_timeout limit

2016-02-02 Thread Andres Freund
On 2016-02-02 11:37:15 +0100, Simon Riggs wrote: > If people wish to turn off crash recovery, they can already set fsync=off. > I don't wish to see us support a setting that causes problems for people > that don't understand what checkpoints are and why everybody needs them. I don't think fsync=of

Re: [HACKERS] Relation extension scalability

2016-02-02 Thread Andres Freund
On 2016-02-02 10:12:38 -0500, Robert Haas wrote: > Here's a sketch of another approach to this problem. Get rid of the > relation extension lock. Instead, have an array of, say, 256 lwlocks. > Each one protects the extension of relations where hash(relfilenode) % > 256 maps to that lock. To exte

Re: [HACKERS] Relation extension scalability

2016-02-02 Thread Andres Freund
On 2016-01-28 16:53:08 +0530, Dilip Kumar wrote: > test_script: > > ./psql -d postgres -c "truncate table data" > ./psql -d postgres -c "checkpoint" > ./pgbench -f copy_script -T 120 -c$ -j$ postgres > > Shared Buffer48GB > Table:Unlogged Table > ench -c$ -j$ -f -M Prepared po

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-02-03 Thread Andres Freund
pdated on > each XLOGInsert apart from the XLOGInsert for standby snapshots > and use that in a patch somewhat close to what you have in > hs-checkpoints-v1. That'll require holding locks longer... Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] 2016-01 Commitfest

2016-02-04 Thread Andres Freund
On 2016-02-03 12:32:47 -0500, Tom Lane wrote: > Heikki and/or Andres have their names on three of the remaining RFC > patches; it's unlikely any other committer will touch those patches > unless they take their names off. If you want to take over the timestamp patch, please feel free to do so. Oth

Re: [HACKERS] Raising the checkpoint_timeout limit

2016-02-04 Thread Andres Freund
On 2016-02-03 15:07:12 -0600, Jim Nasby wrote: > On 2/2/16 10:10 PM, Robert Haas wrote: > >Now, you could also set such configuration settings in > >a situation where it will not work out well. But that is true of most > >configuration settings. By that argument we should probably raise the lower

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-02-04 Thread Andres Freund
On 2016-02-04 18:21:41 +0530, Amit Kapila wrote: > I think generally it is good idea, but one thing worth a thought is that > by doing so, we need to acquire all WAL Insertion locks every > LOG_SNAPSHOT_INTERVAL_MS to check the last_insert_pos for > every slot, do you think it is matter of concern

Re: [HACKERS] checkpointer continuous flushing - V16

2016-02-04 Thread Andres Freund
Hi, Fabien asked me to post a new version of the checkpoint flushing patch series. While this isn't entirely ready for commit, I think we're getting closer. I don't want to post a full series right now, but my working state is available on http://git.postgresql.org/gitweb/?p=users/andresfreund/po

Re: [HACKERS] Idle In Transaction Session Timeout, revived

2016-02-04 Thread Andres Freund
On 2016-02-04 22:24:50 +0900, Fujii Masao wrote: > But, IIRC, one of the problems that prevent the adoption of this feature is > the addition of gettimeofday() call after every SQL command receipt. > Have you already resolved that problem? Or we don't need to care about > it because it's almost har

Re: [HACKERS] "using previous checkpoint record at" maybe not the greatest idea?

2016-02-04 Thread Andres Freund
On 2016-02-03 09:28:24 -0500, Robert Haas wrote: > Would we still have some way of forcing the older checkpoint record to > be used if somebody wants to try to do that? I think currently the best way to force an arbitrary checkpoint to be used is creating a "custom" backup label. Not that nice. N

Re: [HACKERS] checkpoints after database start/immediate checkpoints

2016-02-04 Thread Andres Freund
On 2016-02-03 09:57:00 -0500, Robert Haas wrote: > On Mon, Feb 1, 2016 at 7:43 PM, Andres Freund wrote: > > I wonder if this essentially point at checkpoint_timeout being wrongly > > defined: Currently it means we'll try to finish a checkpoint > > (1-checkpoint_completio

Re: [HACKERS] "using previous checkpoint record at" maybe not the greatest idea?

2016-02-04 Thread Andres Freund
On February 5, 2016 2:52:20 AM GMT+03:00, Jim Nasby wrote: >On 2/4/16 3:37 PM, Andres Freund wrote: >> On 2016-02-03 09:28:24 -0500, Robert Haas wrote: >>> Would we still have some way of forcing the older checkpoint record >to >>> be used if somebody wants to

Re: [HACKERS] silent data loss with ext4 / all current versions

2016-02-06 Thread Andres Freund
On 2016-02-06 17:43:48 +0100, Tomas Vondra wrote: > >Still the data is here... But well. I won't insist. > > Huh? This thread started by an example how to cause loss of committed > transactions. That fits my definition of "data loss" quite well. Agreed, that view doesn't seem to make much sense.

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-02-06 Thread Andres Freund
any WAL insert lock. This > + * is normally an operation that does not take long, but leaving this > + * lookup out of the section taken an exclusive lock saves a couple > + * of instructions. > + */ > + progress_lsn = GetProgressRecPtr(); too long for my taste.

Re: [HACKERS] Explanation for bug #13908: hash joins are badly broken

2016-02-06 Thread Andres Freund
nabled, to be able to detect usage of these functions and assert out. I didn't concentrate on improving memory usage, but IIRC it was even noticeable for some simpler things. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-02-08 Thread Andres Freund
On 2016-02-08 15:58:49 +0900, Michael Paquier wrote: > On Sun, Feb 7, 2016 at 2:49 AM, Andres Freund wrote: > > On 2016-02-06 22:03:15 +0900, Michael Paquier wrote: > >> The patch attached will apply on master, on 9.5 there is one minor > >> conflict. For older ve

Re: [HACKERS] backpatch for REL9_4_STABLE of commit 40482e606733675eb9e5b2f7221186cf81352da1

2016-02-08 Thread Andres Freund
On 2016-02-08 20:59:25 +1100, Haribabu Kommi wrote: > On Mon, Feb 8, 2016 at 8:20 PM, Huong Dangminh > wrote: > > Hi, > > > > I think this fixed is also required for REL9_4_STABLE. > > Please confirm the attached patch. > > Yes, this fix was missed for 9.4 stable branch during back patch > and it

Re: [HACKERS] [ADMIN] 9.5 new setting "cluster name" and logging

2016-02-08 Thread Andres Freund
. I've argued[1][2] for this when cluster_name was introduced, but back then I seemed to have been the only one arguing for it. Josh later jumped that train. Given that we now had a number of people wishing for this, can we maybe reconsider? Greetings, Andres Freund [1] http://archives.p

[HACKERS] process type escape for log_line_prefix

2016-02-08 Thread Andres Freund
#x27;m thinking it might make sense to give normal connections "" as the name, they're usually already discernible. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] WAL Re-Writes

2016-02-08 Thread Andres Freund
mplications. You still need to do fdatasyncs and such. Besides, with the new CRC implications, that doesn't really seem like such a large win anyway. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] checkpointer continuous flushing - V16

2016-02-08 Thread Andres Freund
Hi Fabien, On 2016-02-04 16:54:58 +0100, Andres Freund wrote: > I don't want to post a full series right now, but my working state is > available on > http://git.postgresql.org/gitweb/?p=users/andresfreund/postgres.git;a=shortlog;h=refs/heads/checkpoint-flush > git://git.postgre

Re: [HACKERS] a raft of parallelism-related bug fixes

2016-02-08 Thread Andres Freund
ything wrong or if there's any actually better approach. But pushing an unreviewed, complex patch, that originated in a thread orginally about different relatively small/mundane items, for a contentious issue, a few days after the initial post. Hm. Not sure how you'd react if you weren'

Re: [HACKERS] checkpointer continuous flushing - V16

2016-02-08 Thread Andres Freund
On 2016-02-08 19:52:30 +0100, Fabien COELHO wrote: > I think I would appreciate comments to understand why/how the ringbuffer is > used, and more comments in general, so it is fine if you improve this part. I'd suggest to leave out the ringbuffer/new bgwriter parts. I think they'd be committed sep

Re: [HACKERS] enable parallel query by default?

2016-02-08 Thread Andres Freund
Hi, On 2016-02-08 16:07:05 -0500, Robert Haas wrote: > One of the questions I have about parallel query is whether it should > be enabled by default. That is, should we make the default value of > max_parallel_degree to a value higher than 0? Perhaps 1, say? > > There are some good reasons why

Re: [HACKERS] a raft of parallelism-related bug fixes

2016-02-08 Thread Andres Freund
you're intending to commit something potentially needing debate/review, etc. > Oh: another thing that I would like to do is commit the isolation > tests I wrote for the deadlock detector a while back, which nobody has > reviewed either, though Tom and Alvaro seemed reasonably pos

Re: [HACKERS] a raft of parallelism-related bug fixes

2016-02-08 Thread Andres Freund
me. Especially because one definitely feels that nobody is reading those. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] a raft of parallelism-related bug fixes

2016-02-08 Thread Andres Freund
Hi! Thanks for the answer. Sounds good. On 2016-02-08 18:47:18 -0500, Robert Haas wrote: > and if I'd gone out of my way to say "hey, everybody, here's a patch > that you might want to object to" I'm sure I could have found some > volunteers to do just that. But, you know, that's not really what

Re: [HACKERS] checkpointer continuous flushing - V16

2016-02-09 Thread Andres Freund
On February 9, 2016 10:46:34 AM GMT+01:00, Fabien COELHO wrote: > >>> I think I would appreciate comments to understand why/how the >>> ringbuffer is used, and more comments in general, so it is fine if >you >>> improve this part. >> >> I'd suggest to leave out the ringbuffer/new bgwriter parts

Re: [HACKERS] why can the isolation tester handle only one waiting process?

2016-02-09 Thread Andres Freund
On February 9, 2016 7:12:23 PM GMT+01:00, Robert Haas wrote: >On Tue, Sep 8, 2015 at 5:11 PM, Robert Haas >wrote: >> Here's an updated patch series with some more improvements to the >> isolationtester code, and some better test cases. > >OK, here's a final set of patches that I intend to commit

Re: [HACKERS] Relation extension scalability

2016-02-10 Thread Andres Freund
s has on latency? I think you unfortunately have to look at the per-transaction logs, and then see whether the worst case latencies get better or worse. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Updated backup APIs for non-exclusive backups

2016-02-10 Thread Andres Freund
Hi, On 2016-02-10 13:46:05 +0100, Magnus Hagander wrote: > Per discussionat the developer meeting in Brussels, here's a patch that > makes some updates to the backup APIs, to support non-exclusive backups > without using pg_basebackup. Thanks for following through with this! > * If the client di

Re: [HACKERS] Updated backup APIs for non-exclusive backups

2016-02-10 Thread Andres Freund
On 2016-02-10 16:50:26 +0100, Magnus Hagander wrote: > > I would be happy to see the time-stamp returned from the > > pg_start_backup() function as well. It's a bigger change, but once > > pg_start_backup() returns multiple columns it will be easier to add more > > columns in the future. > > > > I

Re: [HACKERS] Tracing down buildfarm "postmaster does not shut down" failures

2016-02-10 Thread Andres Freund
On 2016-02-09 22:27:07 -0500, Tom Lane wrote: > The idea I was toying with is that previous filesystem activity (making > the temp install, the server's never-fsync'd writes, etc) has built up a > bunch of dirty kernel buffers, and at some point the kernel goes nuts > writing all that data. So the

Re: [HACKERS] Updated backup APIs for non-exclusive backups

2016-02-10 Thread Andres Freund
On 2016-02-10 11:06:01 -0500, David Steele wrote: > That makes sense. The backup_label "as is" could be output at the > beginning but if we want to add the minimum recovery point it would need > to be output at the end. > > It seems like tablespace_map could still be output at the beginning, thou

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-02-11 Thread Andres Freund
On 2016-02-11 09:25:30 +0530, Amit Kapila wrote: > On Wed, Feb 10, 2016 at 1:42 PM, Michael Paquier > wrote: > > > > On Wed, Feb 10, 2016 at 5:00 PM, Amit Kapila > wrote: > > > Okay, but isn't it better that we remove the snapshot taken > > > at checkpoint time in the main branch or till where th

Re: [HACKERS] [patch] Proposal for \crosstabview in psql

2016-02-11 Thread Andres Freund
forced to parse psql output... On the other hand, a proper server side solution won't be easy; so maybe this is a okay enough stopgap. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] GinPageIs* don't actually return a boolean

2016-02-11 Thread Andres Freund
On 2016-02-11 08:50:41 -0500, Robert Haas wrote: > Are we thinking to back-patch this? I would be disinclined to > back-patch widespread changes like this. If there's a specific > instance related to Gin where this is causing a tangible problem, we > could back-patch just that part, with a clear

Re: [HACKERS] GinPageIs* don't actually return a boolean

2016-02-11 Thread Andres Freund
On 2016-02-11 09:51:26 -0500, Robert Haas wrote: > I have never been quite clear on what you think is going to cause > stdbool.h inclusions to become more common. Primarily because it's finally starting to be supported across the board, thanks to MS finally catching up. Then there's uglyness like

Re: [HACKERS] Performance degradation in commit ac1d794

2016-02-11 Thread Andres Freund
On 2016-02-11 12:50:58 -0500, Robert Haas wrote: > On Thu, Feb 11, 2016 at 12:48 PM, Tom Lane wrote: > > Robert Haas writes: > >> I think we should either get this fixed RSN or revert the problematic > >> commit until we get it fixed. I'd be rather disappointed about the > >> latter because I th

Re: [HACKERS] GinPageIs* don't actually return a boolean

2016-02-11 Thread Andres Freund
On 2016-02-11 13:06:14 -0500, Tom Lane wrote: > But I'm unconvinced that we need to make our .c files prepared for > stdbool.h to be #included in them. The fixes, besides two stylistic edits around !! use, are all in headers. The issue is that we return things meant to be used in a boolean context

Re: [HACKERS] Performance degradation in commit ac1d794

2016-02-11 Thread Andres Freund
On 2016-02-11 13:09:27 -0500, Robert Haas wrote: > On Thu, Feb 11, 2016 at 12:53 PM, Andres Freund wrote: > >> One problem is that it makes for misleading results if you try to > >> benchmark 9.5 against 9.6. > > > > You need a really beefy box to show the

Re: [HACKERS] GinPageIs* don't actually return a boolean

2016-02-11 Thread Andres Freund
On 2016-02-11 13:37:17 -0500, Tom Lane wrote: > Andres Freund writes: > > And anyway, these macros are a potential issue even without stdbool.h > > style booleans. > > Absolutely; they don't work safely for testing bits that aren't in the > rightmost byte of

Re: [HACKERS] checkpointer continuous flushing - V16

2016-02-11 Thread Andres Freund
On 2016-02-04 16:54:58 +0100, Andres Freund wrote: > Fabien asked me to post a new version of the checkpoint flushing patch > series. While this isn't entirely ready for commit, I think we're > getting closer. > > I don't want to post a full series right now, but m

Re: [HACKERS] GinPageIs* don't actually return a boolean

2016-02-12 Thread Andres Freund
On 2016-02-12 07:59:06 -0500, Robert Haas wrote: > OK, that seems reasonable from here. What I'm still fuzzy about is > why including stdbool.h causes a failure. Is it because it defines a > type called "bool" that clashes with ours? That seems like the > obvious explanation, but then how does th

Re: [HACKERS] GinPageIs* don't actually return a boolean

2016-02-12 Thread Andres Freund
On 2016-02-12 09:39:20 -0500, Tom Lane wrote: > Robert Haas writes: > > On Fri, Feb 12, 2016 at 8:48 AM, Andres Freund wrote: > >> E.g. if you include stdbool.h [ ginStepRight breaks ] > > > Ah-ha. OK, now I get it. So then I agree we should back-patch this >

Re: CustomScan in a larger structure (RE: [HACKERS] CustomScan support on readfuncs.c)

2016-02-12 Thread Andres Freund
itmapset(), and nothing else? Afaics a lot of the other read routines are also pretty necessary from the outside? Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] GinPageIs* don't actually return a boolean

2016-02-12 Thread Andres Freund
On 2016-02-12 09:47:47 -0500, Tom Lane wrote: > Robert Haas writes: > > On Fri, Feb 12, 2016 at 9:39 AM, Tom Lane wrote: > >> Um, no, that does not follow. The unanswered question here is why, > >> when we *have not* included stdbool.h and *have* typedef'd bool as > >> just plain "char", we woul

Re: [HACKERS] GinPageIs* don't actually return a boolean

2016-02-12 Thread Andres Freund
On February 12, 2016 5:15:59 PM GMT+01:00, Teodor Sigaev wrote: >One more option for patch: > >#define GinPageIsLeaf(page)((bool)(GinPageGetOpaque(page)->flags & >GIN_LEAF)) > >Seems it will work on any platform with built-in bool. But I don't know >will it >work with 'typedef char bool' if

Re: [HACKERS] GinPageIs* don't actually return a boolean

2016-02-12 Thread Andres Freund
On February 12, 2016 5:29:44 PM GMT+01:00, Tom Lane wrote: > We should standardize on the "((var & FLAG) != 0)" >pattern, which works reliably in all cases. That's what the second version of my patch, and I presume Michael's updated one as well, does. I think the only open question is how far t

Re: [HACKERS] GinPageIs* don't actually return a boolean

2016-02-12 Thread Andres Freund
On February 12, 2016 5:40:29 PM GMT+01:00, Yury Zhuravlev wrote: >Andres Freund wrote: >> Unless I am missing something major, that doesn't seem to >> achieve all that much. A cast to a char based bool wouldn't >> normalize this to 0 or 1. So you're still

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-02-12 Thread Andres Freund
On 2016-02-12 12:37:35 -0500, Robert Haas wrote: > On Mon, Feb 8, 2016 at 4:18 AM, Andres Freund wrote: > > I'm not really a fan. I'd rather change existing callers to add a > > 'flags' bitmask argument. Right now there can't really be XLogInserts() >

Re: [HACKERS] Re: [COMMITTERS] pgsql: Add some isolation tests for deadlock detection and resolution.

2016-02-12 Thread Andres Freund
On 2016-02-12 13:16:54 -0500, Tom Lane wrote: > Investigation showed that there are a couple of reasons. One, > isolationtester's is-it-waiting query takes an insane amount of > time under CLOBBER_CACHE_ALWAYS --- over half a second on my > reasonably new server. I wonder if we shouldn't just exp

Re: [HACKERS] Defaults for replication/backup

2016-02-13 Thread Andres Freund
On 2016-02-13 22:37:33 +0900, Michael Paquier wrote: > On Sat, Feb 13, 2016 at 10:15 PM, Magnus Hagander wrote: > > So, I suggest the following changes to the defaults: > > wal_level=hot_standby > > max_wal_senders=10 > > max_replication_slots=10 +1. I'm inclined to set slots a bit higher than sen

Re: [HACKERS] Defaults for replication/backup

2016-02-13 Thread Andres Freund
Hi, On 2016-02-13 07:13:55 -0800, Joshua D. Drake wrote: > I would like to add the idea of having archiving on by default. Not everyone > uses streaming replication, some people use PITR. The one that I see is > archive_command and I am not sure how to deal with that. Since that requires addition

Re: [HACKERS] pg_basebackup vs WAL fetching

2016-02-13 Thread Andres Freund
On 2016-02-13 23:25:06 +0800, Craig Ringer wrote: > Internally replication slots have an ephemeral form, where the slot is > dropped automatically on crash recovery. And upon release (including a http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Defaults for replication/backup

2016-02-13 Thread Andres Freund
On 2016-02-13 11:10:58 -0500, Tom Lane wrote: > Magnus Hagander writes: > > The big thing is, IIRC, that we turn off the optimizations in > > min_wal_level. *most* people will see no impact of their regular > > application runtime from that, but it might definitely have an effect on > > data loads

[HACKERS] Remove or weaken hints about "effective resolution of sleep delays is 10 ms"?

2016-02-15 Thread Andres Freund
of 10. Afaik that's not the case on any recent operating system/hardware. So perhaps we should just remove all of those blurbs, or just replace them with something like "on some older systems the effective resolution of sleep delays is limited to multiples of 10 milliseconds"? Greetings

Re: [HACKERS] xlc atomics

2016-02-15 Thread Andres Freund
On 2016-02-15 12:11:29 -0500, Noah Misch wrote: > These suggested OidGenLock wasn't doing its job. I've seen similar symptoms > around WALInsertLocks with "IBM XL C/C++ for Linux, V13.1.2 (5725-C73, > 5765-J08)" for ppc64le. The problem is generic-xlc.h > pg_atomic_compare_exchange_u32_impl() iss

Re: [HACKERS] [PATCH] Code refactoring related to -fsanitize=use-after-scope

2016-02-15 Thread Andres Freund
nto the current inner tuple. >*/ > - SpGistInnerTuple innerTuple; > - spgChooseIn in; > - spgChooseOut out; But I'm not immediately seing why this is necessary? Is this about battling a false pos

Re: [HACKERS] Remove or weaken hints about "effective resolution of sleep delays is 10 ms"?

2016-02-16 Thread Andres Freund
On February 16, 2016 3:09:16 AM GMT+01:00, Merlin Moncure >I guess we should probably explain what is actually happening, namely >that the precise sleep duration is delegated to the operating system >scheduler which may cause the process to sleep longer than requested. In not really seeing why: T

Re: [HACKERS] Remove or weaken hints about "effective resolution of sleep delays is 10 ms"?

2016-02-16 Thread Andres Freund
On February 16, 2016 9:06:57 AM GMT+01:00, Robert Haas wrote: >On Wed, Feb 10, 2016 at 5:15 PM, Andres Freund >wrote: >> Hi, >> >> Several places in our docs have blurbs like >>> Note that on many systems, the effective resolution of sleep delays

<    1   2   3   4   5   6   7   8   9   10   >