[HACKERS] Bogus WAL segments archived after promotion

2014-12-19 Thread Heikki Linnakangas
When streaming replication was introduced in 9.0, we started to recycle old WAL segments in archive recovery, like we do during normal operation. The WAL segments are recycled on the current timeline. There is no guarantee that they are useful, if the current timeline changes, because we step

Re: [HACKERS] Parallel Seq Scan

2014-12-19 Thread Heikki Linnakangas
On 12/19/2014 04:39 PM, Stephen Frost wrote: * Marko Tiikkaja (ma...@joh.to) wrote: On 12/19/14 3:27 PM, Stephen Frost wrote: We'd have to coach our users to constantly be tweaking the enable_parallel_query (or whatever) option for the queries where it helps and turning it off for others. I'm

Re: [HACKERS] Bogus WAL segments archived after promotion

2014-12-19 Thread Heikki Linnakangas
On 12/19/2014 02:55 PM, Heikki Linnakangas wrote: I'm thinking that we should add a step to promotion, where we scan pg_xlog for any segments higher than the timeline switch point, and remove them, or mark them with .done so that they are not archived. There might be some real WAL

Re: [HACKERS] Streaming replication and WAL archive interactions

2014-12-19 Thread Heikki Linnakangas
On 12/18/2014 12:32 PM, Fujii Masao wrote: On Wed, Dec 17, 2014 at 4:11 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 12/16/2014 10:24 AM, Borodin Vladimir wrote: 12 дек. 2014 г., в 16:46, Heikki Linnakangas hlinnakan...@vmware.com написал(а): There have been a few threads

Re: [HACKERS] INSERT ... ON CONFLICT {UPDATE | IGNORE}

2014-12-18 Thread Heikki Linnakangas
On 12/18/2014 01:02 AM, Peter Geoghegan wrote: On Wed, Dec 17, 2014 at 1:12 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: Now, let's imagine a table like this: CREATE TABLE persons ( username text unique, real_name text unique, data text ); Is there any way to specify both

Re: [HACKERS] Proposal: Log inability to lock pages during vacuum

2014-12-18 Thread Heikki Linnakangas
On 12/18/2014 09:41 PM, Alvaro Herrera wrote: Jim Nasby wrote: We have to decide on a tradeoff here. Either we end up with two different log messages (depending on scan_all) that require two different translations, or we end up with a generic message that isn't as clear. The best option I can

Re: [HACKERS] POLA violation with \c service=

2014-12-17 Thread Heikki Linnakangas
On 12/17/2014 10:03 AM, Albe Laurenz wrote: David Fetter wrote: I've noticed that psql's \c function handles service= requests in a way that I can only characterize as broken. This came up in the context of connecting to a cloud hosting service named after warriors or a river or something,

[HACKERS] exitArchiveRecovery woes

2014-12-17 Thread Heikki Linnakangas
At the end of archive recovery, we copy the last segment from the old timeline, to initialize the first segment on the new timeline. For example, if the timeline switch happens in the middle of WAL segment 00010005, the whole 00010005 segment is copied to become

Re: [HACKERS] GiST kNN search queue (Re: KNN-GiST with recheck)

2014-12-17 Thread Heikki Linnakangas
On 12/15/2014 03:14 PM, Andres Freund wrote: If we add another heap implementation we probably should at least hint at the different advantages somewhere. How about adding a src/backend/lib/README for that, per attached? - Heikki diff --git a/src/backend/lib/Makefile

Re: [HACKERS] [PATCH] explain sortorder

2014-12-17 Thread Heikki Linnakangas
On 12/15/2014 06:49 PM, Mike Blackwell wrote: QUERY PLAN Sort Output: n1, n2, ((n1)::character(1)) Sort Key: sortordertest.n1, sortordertest.n2 Sort Order: ASC NULLS LAST, ASC NULLS LAST - Seq Scan on

Re: [HACKERS] Proposal: Log inability to lock pages during vacuum

2014-12-17 Thread Heikki Linnakangas
On 12/01/2014 08:55 PM, Jim Nasby wrote: On 12/1/14, 11:57 AM, Andres Freund wrote: On 2014-11-30 20:46:51 -0600, Jim Nasby wrote: On 11/10/14, 7:52 PM, Tom Lane wrote: On the whole, I'm +1 for just logging the events and seeing what we learn that way. That seems like an appropriate amount

Re: [HACKERS] DROP PRIVILEGES OWNED BY

2014-12-17 Thread Heikki Linnakangas
On 12/15/2014 02:43 AM, Marko Tiikkaja wrote: This week I had a problem where I wanted to drop only the privileges a certain role had in the system, while keeping all the objects. I couldn't figure out a reasonable way to do that, so I've attached a patch for this to this email. Please

Re: [HACKERS] Proposal: Log inability to lock pages during vacuum

2014-12-17 Thread Heikki Linnakangas
On 12/17/2014 08:02 PM, Jim Nasby wrote: BTW, what is it about a dynamic message that makes it untranslatable? Doesn't the translation happen down-stream, so that at most we'd just need two translation messages? I'm not sure what you mean by down-stream. There is some explanation on this in

Re: [HACKERS] Compiling C++ extensions on MSVC using scripts in src/tools

2014-12-17 Thread Heikki Linnakangas
On 11/27/2014 06:39 AM, Michael Paquier wrote: On Thu, Nov 27, 2014 at 1:40 AM, Tom Lane t...@sss.pgh.pa.us wrote: Andrew Dunstan and...@dunslane.net writes: This doesn't seem to me to be terribly well expressed (I know it's not your fault, quite possibly it's mine.) Perhaps we should replace

Re: [HACKERS] SSL information view

2014-12-17 Thread Heikki Linnakangas
On 11/19/2014 02:36 PM, Magnus Hagander wrote: + /* Create or attach to the shared SSL status buffers */ + size = mul_size(NAMEDATALEN, MaxBackends); + BackendSslVersionBuffer = (char *) + ShmemInitStruct(Backend SSL Version Buffer, size, found); + + if

Re: [HACKERS] INSERT ... ON CONFLICT {UPDATE | IGNORE}

2014-12-17 Thread Heikki Linnakangas
It looks like we are close to reaching consensus on the syntax. Phew! Thanks for maintaining the wiki pages and the documentation. All of the below is based on those, I haven't looked at the patch itself yet. The one thing that I still feel uneasy about is the Unique Index Inference thing.

Re: [HACKERS] NUMERIC private methods?

2014-12-16 Thread Heikki Linnakangas
On 12/16/2014 08:34 AM, David Fetter wrote: Folks, While noodling with some weighted statistics https://github.com/davidfetter/weighted_stats, I noticed I was having to jump through a lot of hoops because of all the private methods in numeric.c, especially NumericVar. Would there be some major

Re: [HACKERS] GiST kNN search queue (Re: KNN-GiST with recheck)

2014-12-16 Thread Heikki Linnakangas
On 12/15/2014 11:59 PM, Jeff Janes wrote: On Mon, Dec 15, 2014 at 5:08 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: Here's a new version of the patch. It now uses the same pairing heap code that I posted in the other thread (advance local xmin more aggressivley, http

Re: [HACKERS] pg_rewind in contrib

2014-12-16 Thread Heikki Linnakangas
On 12/16/2014 11:23 AM, Satoshi Nagayasu wrote: Hi, On 2014/12/12 23:13, Heikki Linnakangas wrote: Hi, I'd like to include pg_rewind in contrib. I originally wrote it as an external project so that I could quickly get it working with the existing versions, and because I didn't

Re: [HACKERS] KNN-GiST with recheck

2014-12-16 Thread Heikki Linnakangas
On 10/06/2014 12:36 PM, Emre Hasegeli wrote: Thanks. The main question now is design of this patch. Currently, it does all the work inside access method. We already have some discussion of pro and cons of this method. I would like to clarify alternatives now. I can see following way: 1.

Re: [HACKERS] Possibly a comment typo in xlogrecord.h

2014-12-16 Thread Heikki Linnakangas
On 12/16/2014 12:44 PM, Rahila Syed wrote: Hello, The comment before declaration of XLogRecordBlockHeader says * 'data_length' is the length of the payload data associated with this, * and includes the possible full-page image, and rmgr-specific data. It IIUC, data_length does not include

Re: [HACKERS] WALWriter active during recovery

2014-12-16 Thread Heikki Linnakangas
On 12/15/2014 08:51 PM, Simon Riggs wrote: Currently, WALReceiver writes and fsyncs data it receives. Clearly, while we are waiting for an fsync we aren't doing any other useful work. Following patch starts WALWriter during recovery and makes it responsible for fsyncing data, allowing

Re: [HACKERS] Logical Decoding follows timelines

2014-12-16 Thread Heikki Linnakangas
On 12/15/2014 08:54 PM, Simon Riggs wrote: Currently, it doesn't. This patch is a WIP version of doing that, but only currently attempts to do this in the WALSender. Objective is to allow cascaded logical replication. Very WIP, but here for comments. With the patch, XLogSendLogical uses the

Re: [HACKERS] Comment typo in typedef struct BrinTuple

2014-12-16 Thread Heikki Linnakangas
On 12/15/2014 09:04 AM, Amit Langote wrote: Hi, Find attached that does: -* mt_info is laid out in the following fashion: +* bt_info is laid out in the following fashion: snip-comment uint8 bt_info; } BrinTuple; Thanks. Fixed along with a bunch of other misc

Re: [HACKERS] pg_basebackup vs. Windows and tablespaces

2014-12-16 Thread Heikki Linnakangas
On 12/16/2014 06:30 PM, Andrew Dunstan wrote: I'm not clear why human readability is the major criterion here. As for that, it will be quite difficult for a human to distinguish a name with a space at the end from one without. I really think a simple encoding scheme would be much the best. For

Re: [HACKERS] Streaming replication and WAL archive interactions

2014-12-16 Thread Heikki Linnakangas
On 12/16/2014 10:24 AM, Borodin Vladimir wrote: 12 дек. 2014 г., в 16:46, Heikki Linnakangas hlinnakan...@vmware.com написал(а): There have been a few threads on the behavior of WAL archiving, after a standby server is promoted [1] [2]. In short, it doesn't work as you might expect

Re: [HACKERS] GiST kNN search queue (Re: KNN-GiST with recheck)

2014-12-15 Thread Heikki Linnakangas
On 12/15/2014 03:49 AM, Michael Paquier wrote: On Thu, Dec 11, 2014 at 12:50 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 01/28/2014 04:12 PM, Alexander Korotkov wrote: 3. A binary heap would be a better data structure to buffer the rechecked values. A Red-Black tree allows

Re: [HACKERS] Commit fest 2014-12, let's begin!

2014-12-15 Thread Heikki Linnakangas
On 12/15/2014 08:37 AM, Michael Paquier wrote: On Mon, Dec 15, 2014 at 3:09 PM, Tom Lane t...@sss.pgh.pa.us wrote: Michael Paquier michael.paqu...@gmail.com writes: - Point to polygon distance operator I looked at that briefly during the last fest, but was unsure whether it was too entangled

Re: [HACKERS] speedup tidbitmap patch: hash BlockNumber

2014-12-15 Thread Heikki Linnakangas
On 10/22/2014 04:14 PM, Teodor Sigaev wrote: Just replace tag_hash in tidbitmap which uses hash_any to direct call of hash_uint32, it saves ~5% of execution time. An example: # create extension btree_gin; # select (v / 10)::int4 as i into t from generate_series(1, 500) as v; # create index

Re: [HACKERS] Fractions in GUC variables

2014-12-15 Thread Heikki Linnakangas
On 12/07/2014 09:48 PM, John Gorman wrote: This patch implements the first wiki/Todo Configuration Files item Consider normalizing fractions in postgresql.conf, perhaps using '%'. The Fractions in GUC variables discussion is here.

Re: [HACKERS] Commit fest 2014-12, let's begin!

2014-12-15 Thread Heikki Linnakangas
On 12/15/2014 03:45 PM, Alexander Korotkov wrote: On Mon, Dec 15, 2014 at 4:12 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 12/15/2014 08:37 AM, Michael Paquier wrote: On Mon, Dec 15, 2014 at 3:09 PM, Tom Lane t...@sss.pgh.pa.us wrote: Michael Paquier michael.paqu...@gmail.com

Re: [HACKERS] Custom timestamp format in logs

2014-12-15 Thread Heikki Linnakangas
On 12/14/2014 06:36 PM, Magnus Hagander wrote: A separate GUC seems kind of weird. Wouldn't it be better with something like %(format)t or such in the log_line_prefix itself in that case? That could also be expanded to other parameters, should we need them? %t isn't the only thing that prints

Re: [HACKERS] KNN-GiST with recheck

2014-12-15 Thread Heikki Linnakangas
On 08/03/2014 04:48 PM, Emre Hasegeli wrote: 1. This patch introduces a new polygon - point operator. That seems useful on its own, with or without this patch. Yeah, but exact-knn cant come with no one implementation. But it would better come in a separate patch. I tried to split them.

Re: [HACKERS] Commit fest 2014-12, let's begin!

2014-12-15 Thread Heikki Linnakangas
On 12/15/2014 05:22 PM, Alexander Korotkov wrote: On Mon, Dec 15, 2014 at 6:20 PM, Tom Lane t...@sss.pgh.pa.us wrote: Alexander Korotkov aekorot...@gmail.com writes: On Mon, Dec 15, 2014 at 4:12 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: Right. I also looked at it briefly, but I

Re: [HACKERS] duplicate #define

2014-12-13 Thread Heikki Linnakangas
On 12/13/2014 04:45 PM, Mark Dilger wrote: In commit 2c03216d831160bedd72d45f712601b6f7d03f1c, the following define occurs twice in src/include/access/xlogrecord.h: #define SizeOfXLogRecordDataHeaderLong (sizeof(uint8) + sizeof(uint32)) It is no big deal, as the definitions don't contradict

Re: [HACKERS] Review of Refactoring code for sync node detection

2014-12-12 Thread Heikki Linnakangas
On 12/12/2014 04:29 AM, Michael Paquier wrote: On Thu, Dec 11, 2014 at 10:07 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: I propose the attached (I admit I haven't tested it). Actually if you do it this way I think that it would be worth adding the small optimization Fujii-san

Re: [HACKERS] moving from contrib to bin

2014-12-12 Thread Heikki Linnakangas
On 12/12/2014 03:07 PM, Peter Eisentraut wrote: On 12/9/14 4:10 PM, Alvaro Herrera wrote: Maybe it makes sense to have a distinction between client programs and server programs. Can we have src/sbin/ and move stuff that involves the server side in there? I think that'd be pg_xlogdump,

Re: [HACKERS] moving from contrib to bin

2014-12-12 Thread Heikki Linnakangas
On 12/12/2014 03:11 PM, Heikki Linnakangas wrote: On 12/12/2014 03:07 PM, Peter Eisentraut wrote: On 12/9/14 4:10 PM, Alvaro Herrera wrote: Maybe it makes sense to have a distinction between client programs and server programs. Can we have src/sbin/ and move stuff that involves the server

[HACKERS] Streaming replication and WAL archive interactions

2014-12-12 Thread Heikki Linnakangas
There have been a few threads on the behavior of WAL archiving, after a standby server is promoted [1] [2]. In short, it doesn't work as you might expect. The standby will start archiving after it's promoted, but it will not archive files that were replicated from the old master via streaming

Re: [HACKERS] BUG: *FF WALs under 9.2 (WAS: .ready files appearing on slaves)

2014-12-12 Thread Heikki Linnakangas
for some reasons, these old WALs were forgotten by the restartpoint mechanism when they should have been recylced/deleted. Am 08.10.2014 um 11:54 schrieb Heikki Linnakangas hlinnakan...@vmware.com: 1. Where do the FF files come from? In 9.2, FF-segments are not supposed to created, ever. […] 2. Why

[HACKERS] pg_rewind in contrib

2014-12-12 Thread Heikki Linnakangas
. - Heikki commit 2300e28b0d07328c7b37a92f7150e75edf24b10c Author: Heikki Linnakangas heikki.linnakan...@iki.fi Date: Fri Dec 12 16:08:14 2014 +0200 Add pg_rewind to contrib. diff --git a/contrib/Makefile b/contrib/Makefile index 195d447..2fe861f 100644 --- a/contrib/Makefile +++ b/contrib

Re: [HACKERS] pg_rewind in contrib

2014-12-12 Thread Heikki Linnakangas
On 12/12/2014 04:20 PM, Andres Freund wrote: Not sure if the copyright notices in the current form are actually ok? Hmm. We do have such copyright notices in the source tree, but I know that we're trying to avoid it in new code. They had to be there when the code lived as a separate project,

Re: [HACKERS] WRITE_UINT_FIELD used where WRITE_OID_FIELD likely intended

2014-12-11 Thread Heikki Linnakangas
On 12/11/2014 05:57 AM, Michael Paquier wrote: On Thu, Dec 11, 2014 at 7:44 AM, Mark Dilger m...@port25.com wrote: At line 1787 of outfuncs.c, the line: WRITE_UINT_FIELD(reltablespace) should probably say WRITE_OID_FIELD(reltablespace) since that variable is of type Oid,

Re: [HACKERS] Too strict check when starting from a basebackup taken off a standby

2014-12-11 Thread Heikki Linnakangas
On 12/11/2014 05:45 AM, Andres Freund wrote: A customer recently reported getting backup_label contains data inconsistent with control file after taking a basebackup from a standby and starting it with a typo in primary_conninfo. When starting postgres from a basebackup StartupXLOG() has the

Re: [HACKERS] Directory/File Access Permissions for COPY and Generic File Access Functions

2014-12-11 Thread Heikki Linnakangas
I'm marking this as Rejected in the commitfest. It's quite clear that this isn't going to fly in its current form. For the COPY FROM use case, I'd suggest just doing COPY FROM STDIN. Yes, it's slower, but not much. And you probably could optimize it further - there's some gratuitous

Re: [HACKERS] Review of Refactoring code for sync node detection

2014-12-11 Thread Heikki Linnakangas
On 11/18/2014 11:23 PM, Michael Paquier wrote: On Tue, Nov 18, 2014 at 6:33 PM, Simon Riggs si...@2ndquadrant.com wrote: Can we just wait on this patch until we have the whole feature? Well, this may take some time to even define, and even if goals are clearly defined this may take even more

Re: [HACKERS] WIP: multivariate statistics / proof of concept

2014-12-11 Thread Heikki Linnakangas
On 10/13/2014 01:00 AM, Tomas Vondra wrote: Hi, attached is a WIP patch implementing multivariate statistics. Great! Really glad to see you working on this. +* FIXME This sample sizing is mostly OK when computing stats for +* individual columns, but when computing

Re: [HACKERS] 9.5 release scheduling (was Re: logical column ordering)

2014-12-11 Thread Heikki Linnakangas
On 12/11/2014 06:59 PM, Robert Haas wrote: On Thu, Dec 11, 2014 at 11:03 AM, Tom Lane t...@sss.pgh.pa.us wrote: I think 9.4 dragged almost entirely because of one issue: the compressibility of JSONB. Meh. While we certainly weren't very speedy about resolving that, I don't think that issue

Re: [HACKERS] 9.5 release scheduling (was Re: logical column ordering)

2014-12-11 Thread Heikki Linnakangas
On 12/11/2014 08:51 PM, Josh Berkus wrote: On 12/11/2014 09:22 AM, Heikki Linnakangas wrote: I imagine that it's the same for everyone else. Many of the patches that sit in the commitfest for weeks are patches that no-one really cares much about. I'm not sure what to do about that. It would

Re: [HACKERS] Small TRUNCATE glitch

2014-12-10 Thread Heikki Linnakangas
On 12/10/2014 03:04 AM, Alvaro Herrera wrote: Alex Shulgin wrote: The 2PC part requires extending bool flag to fit the trunc flag, is this approach sane? Given that 2PC transaction should survive server restart, it's reasonable to expect it to also survive the upgrade, so I see no clean way

Re: [HACKERS] advance local xmin more aggressively

2014-12-10 Thread Heikki Linnakangas
On 12/09/2014 10:35 PM, Robert Haas wrote: On Mon, Dec 8, 2014 at 9:31 AM, Robert Haas robertmh...@gmail.com wrote: On Mon, Dec 8, 2014 at 4:56 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: I don't immediately see the problem either, but I have to say that grovelling through all

[HACKERS] GiST kNN search queue (Re: KNN-GiST with recheck)

2014-12-10 Thread Heikki Linnakangas
On 01/28/2014 04:12 PM, Alexander Korotkov wrote: 3. A binary heap would be a better data structure to buffer the rechecked values. A Red-Black tree allows random insertions and deletions, but in this case you need to insert arbitrary values but only remove the minimum item. That's exactly what

Re: [HACKERS] advance local xmin more aggressively

2014-12-10 Thread Heikki Linnakangas
On 12/10/2014 06:56 PM, Robert Haas wrote: On Wed, Dec 10, 2014 at 9:49 AM, Robert Haas robertmh...@gmail.com wrote: I guess this bears some further thought. I certainly don't like the fact that this makes the whole system crap out at a lower number of subtransactions than presently. The

Re: [HACKERS] advance local xmin more aggressively

2014-12-10 Thread Heikki Linnakangas
On 12/10/2014 08:35 PM, Robert Haas wrote: On Wed, Dec 10, 2014 at 12:57 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: Clever. Could we use that method in ResourceOwnerReleaseInternal and ResourceOwnerDelete, too? Might be best to have a ResourceOwnerWalk(resowner, callback) function

Re: [HACKERS] GiST kNN search queue (Re: KNN-GiST with recheck)

2014-12-10 Thread Heikki Linnakangas
On 12/10/2014 10:59 PM, Arthur Silva wrote: It may be better to replace the lib/binaryheap altogether as it offers comparable/better performance. It's not always better. A binary heap is more memory-efficient, for starters. There are only two uses of lib/binaryheap: reorderbuffer.c and

Re: [HACKERS] advance local xmin more aggressively

2014-12-08 Thread Heikki Linnakangas
On 12/05/2014 05:05 PM, Robert Haas wrote: [ reviving a very old thread ] On Tue, Feb 10, 2009 at 4:11 PM, Tom Lane t...@sss.pgh.pa.us wrote: Alvaro Herrera alvhe...@commandprompt.com writes: For example, maybe we could keep track of counts of snapshots removed since the last xmin

Re: [HACKERS] Compression of full-page-writes

2014-12-08 Thread Heikki Linnakangas
On 12/08/2014 09:21 PM, Andres Freund wrote: I still think that just compressing the whole record if it's above a certain size is going to be better than compressing individual parts. Michael argued thta that'd be complicated because of the varying size of the required 'scratch space'. I don't

Re: [HACKERS] Misunderstanding on the FSM README file

2014-12-07 Thread Heikki Linnakangas
On 12/07/2014 02:03 PM, Guillaume Lelarge wrote: Hi, I've been reading the FSM README file lately (src/backend/storage/freespace/README), and I'm puzzled by one of the graph (the binary tree structure of an FSM file). Here it is: 4 4 2 3 4 0 2- This level represents heap pages

Re: [HACKERS] New wal format distorts pg_xlogdump --stats

2014-12-05 Thread Heikki Linnakangas
On 12/05/2014 02:31 AM, Andres Freund wrote: On 2014-12-05 08:58:33 +0900, Michael Paquier wrote: On Fri, Dec 5, 2014 at 8:09 AM, Andres Freund and...@2ndquadrant.com wrote: On 2014-12-04 16:26:02 +0200, Heikki Linnakangas wrote: Yeah, that's broken. I propose the attached. Or does anyone

Re: [HACKERS] XLOG_PARAMETER_CHANGE (WAL record) missing two params in its desc routine

2014-12-05 Thread Heikki Linnakangas
On 12/05/2014 04:54 AM, Michael Paquier wrote: Hi all, While reading the code in this area this morning, I noticed that wal_log_hints and track_commit_timestamp are not mentioned in the desc routine of XLOG_CHANGE_PARAMETER. Also, it is not mentioned in postgresql.conf.sample that a value

Re: [HACKERS] check-world failure: dummy_seclabel

2014-12-05 Thread Heikki Linnakangas
On 12/05/2014 07:29 AM, Adam Brightwell wrote: All, I've noticed that 'check-world' fails for dummy_seclabel after a 'clean'. I believe that in commit da34731, the EXTRA_CLEAN statement should have been removed from 'src/test/modules/dummy_seclabel/Makefile' as well. Ah, that's why those

Re: [HACKERS] libpq pipelining

2014-12-05 Thread Heikki Linnakangas
On 12/05/2014 02:30 AM, Matt Newell wrote: The explanation of PQgetFirstQuery makes it sound pretty hard to match up the result with the query. You have to pay attention to PQisBusy. PQgetFirstQuery should also be valid after calling PQgetResult and then you don't have to worry about

Re: [HACKERS] initdb: Improve error recovery.

2014-12-05 Thread Heikki Linnakangas
On 11/27/2014 02:28 PM, Mats Erik Andersson wrote: Hello there, I would like to improve error recovery of initdb when the password file is empty. The present code declares Error 0 as the cause of failure. It suffices to use ferrer() since fgets() returns NULL also at a premature EOF. Thanks,

Re: [HACKERS] compress method for spgist - 2

2014-12-05 Thread Heikki Linnakangas
On 12/01/2014 02:44 PM, Teodor Sigaev wrote: Initial message: http://www.postgresql.org/message-id/5447b3ff.2080...@sigaev.ru Second version fixes a forgotten changes in pg_am. + /* Get the information we need about each relevant datatypes */ + + if

Re: [HACKERS] libpq pipelining

2014-12-04 Thread Heikki Linnakangas
On 12/04/2014 03:11 AM, Matt Newell wrote: The recent discussion about pipelining in the jodbc driver prompted me to look at what it would take for libpq. Great! I have a proof of concept patch working. The results are even more promising than I expected. While it's true that many

Re: [HACKERS] [COMMITTERS] pgsql: Keep track of transaction commit timestamps

2014-12-04 Thread Heikki Linnakangas
On 12/03/2014 04:54 PM, Alvaro Herrera wrote: ir commit timestamp directly as they commit, or an external transaction c Sorry, I'm late to the party, but here's some random comments on this after a quick review: * The whole concept of a node ID seems to be undocumented, and unused. No-one

Re: [HACKERS] [COMMITTERS] pgsql: Keep track of transaction commit timestamps

2014-12-04 Thread Heikki Linnakangas
On 12/04/2014 01:16 PM, Petr Jelinek wrote: On 04/12/14 10:42, Heikki Linnakangas wrote: On 12/03/2014 04:54 PM, Alvaro Herrera wrote: ir commit timestamp directly as they commit, or an external transaction c Sorry, I'm late to the party, but here's some random comments on this after a quick

Re: [HACKERS] [COMMITTERS] pgsql: Keep track of transaction commit timestamps

2014-12-04 Thread Heikki Linnakangas
On 12/04/2014 01:47 PM, Petr Jelinek wrote: On 04/12/14 12:26, Heikki Linnakangas wrote: On 12/04/2014 01:16 PM, Petr Jelinek wrote: On 04/12/14 10:42, Heikki Linnakangas wrote: On 12/03/2014 04:54 PM, Alvaro Herrera wrote: ir commit timestamp directly as they commit, or an external

Re: [HACKERS] SSL regression test suite

2014-12-04 Thread Heikki Linnakangas
On 10/06/2014 04:21 PM, Heikki Linnakangas wrote: Here's a new version of the SSL regression suite I wrote earlier. It now specifies both host and hostaddr in the connection string as Andres suggested, so it no longer requires changes to network configuration. I added a bunch of tests

Re: [HACKERS] New wal format distorts pg_xlogdump --stats

2014-12-04 Thread Heikki Linnakangas
On 11/25/2014 05:36 AM, Andres Freund wrote: Hi, The new WAL format calculates FPI vs plain record data like: rec_len = XLogRecGetDataLen(record) + SizeOfXLogRecord; fpi_len = record-decoded_record-xl_tot_len - rec_len; Due to the amount of data now handled outside the main

Re: [HACKERS] make check-world regress failed

2014-12-04 Thread Heikki Linnakangas
On 11/23/2014 08:37 PM, Vladimir Koković wrote: PostgreSQL check-world regress failed with current GIT HEAD on my Kubuntu 14.10. uname -a Linux vlD-kuci 3.16.0-24-generic #32-Ubuntu SMP Tue Oct 28 13:13:18 UTC 2014 i686 athlon i686 GNU/Linux gdb -d

Re: [HACKERS] libpq pipelining

2014-12-04 Thread Heikki Linnakangas
On 12/04/2014 09:11 PM, Matt Newell wrote: With the API i am proposing, only 2 new functions (PQgetFirstQuery, PQgetLastQuery) are required to be able to match each result to the query that caused it. Another function, PQgetNextQuery allows iterating through the pending queries, and

Re: [HACKERS] Removing INNER JOINs

2014-12-03 Thread Heikki Linnakangas
On 12/03/2014 07:41 PM, Robert Haas wrote: On Wed, Dec 3, 2014 at 12:33 PM, Tom Lane t...@sss.pgh.pa.us wrote: Stephen Frost sfr...@snowman.net writes: * Tom Lane (t...@sss.pgh.pa.us) wrote: However, even granting that that is a concern, so what? You *have* to do the planning twice, or

Re: [HACKERS] Allocation in critical section after node exits archive recovery

2014-11-27 Thread Heikki Linnakangas
On 11/28/2014 08:35 AM, Michael Paquier wrote: Hi all, While performing some archive recovery tests I found the following assertion failure once standby gets out of archive recovery (HEAD at a5eb85e): (lldb) * thread #1: tid = 0x, 0x7fff8faaf282 libsystem_kernel.dylib`__pthread_kill +

Re: [HACKERS] Using pg_rewind for differential backup

2014-11-27 Thread Heikki Linnakangas
On 11/28/2014 09:30 AM, Michael Paquier wrote: On Thu, Nov 27, 2014 at 9:39 PM, Sameer Kumar sameer.ku...@ashnik.com wrote: Can we tweak pg_rewind to take differential backups in PostgreSQL? I was wondering can we hack the pg_rewind code to print the details of the file which have been

Re: [HACKERS] PITR failing to stop before DROP DATABASE

2014-11-26 Thread Heikki Linnakangas
On 11/26/2014 11:19 AM, Christoph Berg wrote: Re: Heikki Linnakangas 2014-11-25 5474b848.3060...@vmware.com db1 is registered in pg_database, but the directory is missing on disk. Yeah, DROP DATABASE cheats. It deletes all the files first, and commits the transaction only after that. There's

Missing OOM checks in libpq (was Re: [HACKERS] Replication connection URI?)

2014-11-25 Thread Heikki Linnakangas
On 11/24/2014 06:05 PM, Alex Shulgin wrote: The first patch is not on topic, I just spotted this missing check. *** a/src/interfaces/libpq/fe-connect.c --- b/src/interfaces/libpq/fe-connect.c *** conninfo_array_parse(const char *const * *** 4402,4407 --- 4402,4415

Re: Missing OOM checks in libpq (was Re: [HACKERS] Replication connection URI?)

2014-11-25 Thread Heikki Linnakangas
On 11/25/2014 01:37 PM, Alex Shulgin wrote: Heikki Linnakangas hlinnakan...@vmware.com writes: On 11/24/2014 06:05 PM, Alex Shulgin wrote: The first patch is not on topic, I just spotted this missing check. *** a/src/interfaces/libpq/fe-connect.c --- b/src/interfaces/libpq/fe-connect.c

Re: [HACKERS] Replication connection URI?

2014-11-25 Thread Heikki Linnakangas
On 11/24/2014 06:05 PM, Alex Shulgin wrote: Heikki Linnakangas hlinnakan...@vmware.com writes: It appears that replication connection doesn't support URI but only the traditional conninfo string. src/backend/replication/libpqwalreceiver/libpqwalreceiver.c:99: in libpqrcv_connect

Re: [HACKERS] Replication connection URI?

2014-11-25 Thread Heikki Linnakangas
On 11/25/2014 05:11 PM, Heikki Linnakangas wrote: On 11/24/2014 06:05 PM, Alex Shulgin wrote: Heikki Linnakangas hlinnakan...@vmware.com writes: It appears that replication connection doesn't support URI but only the traditional conninfo string. src/backend/replication/libpqwalreceiver

Re: [HACKERS] PITR failing to stop before DROP DATABASE

2014-11-25 Thread Heikki Linnakangas
On 11/25/2014 06:06 PM, Christoph Berg wrote: In 9.3.5, if I set up archiving, create a database, pull a base backup, look at the clock, drop database, stop the server, rm -rf datadir, put back the backup, edit recovery.conf: cd /tmp; initdb foo edit postgresql.conf with archive_mode = on,

Re: [HACKERS] Change in HEAP_NEWPAGE logging makes diagnosis harder

2014-11-24 Thread Heikki Linnakangas
On 11/14/2014 10:17 AM, Heikki Linnakangas wrote: On 10/30/2014 04:12 PM, Andres Freund wrote: Hi, I've just once more looked at the WAL stream ans was briefly confused about all the XLOG_FPI records. Since 54685338e3 log_newpage/log_newpage_buffer and XLogSaveBufferForHint() use the same WAL

Re: [HACKERS] WIP: Access method extendability

2014-11-24 Thread Heikki Linnakangas
On 11/10/2014 10:30 PM, Alexander Korotkov wrote: Don't allowing CREATE ACCESS METHOD command seems problematic for me. How could it work with pg_upgrade? pg_dump wouldn't dump extra pg_am records. So, pg_upgrade would break at creating operator classes on new cluster. So, I agree with dropping

Re: [HACKERS] Sequence Access Method WIP

2014-11-24 Thread Heikki Linnakangas
On 11/08/2014 04:21 PM, Simon Riggs wrote: On 5 November 2014 17:32, Heikki Linnakangas hlinnakan...@vmware.com wrote: Why does sequence_alloc need the current value? If it's a remote seqam, the current value is kept in the remote server, and the last value that was given to this PostgreSQL

Re: [HACKERS] Stating the significance of Lehman Yao in the nbtree README

2014-11-24 Thread Heikki Linnakangas
On 09/27/2014 09:36 AM, Peter Geoghegan wrote: On Fri, Sep 26, 2014 at 11:34 PM, Amit Kapila amit.kapil...@gmail.com wrote: I have observed that this patch is in 'Needs Review' state for next CF. Do you expect any further review from myside? I think we can use text recommended by Heikki and

Re: [HACKERS] add modulo (%) operator to pgbench

2014-11-24 Thread Heikki Linnakangas
On 09/25/2014 05:10 AM, Robert Haas wrote: On Wed, Sep 24, 2014 at 2:34 PM, Fabien COELHO coe...@cri.ensmp.fr wrote: Sigh. How to transform a trivial 10 lines patch into a probably 500+ lines project involving flex bison some non trivial data structures, and which may get rejected on any

Re: [HACKERS] Replication connection URI?

2014-11-24 Thread Heikki Linnakangas
On 11/24/2014 02:41 PM, Alex Shulgin wrote: Hackers, It appears that replication connection doesn't support URI but only the traditional conninfo string. src/backend/replication/libpqwalreceiver/libpqwalreceiver.c:99: in libpqrcv_connect(): snprintf(conninfo_repl,

Re: [HACKERS] GIN pageinspect functions

2014-11-21 Thread Heikki Linnakangas
On 11/20/2014 05:52 AM, Michael Paquier wrote: On Wed, Nov 19, 2014 at 7:01 AM, Peter Geoghegan p...@heroku.com wrote: On Tue, Nov 4, 2014 at 7:26 AM, Amit Kapila amit.kapil...@gmail.com wrote: 1. Documentation seems to be missing, other API's exposed via pageinspect are documented at:

Re: [HACKERS] WAL format and API changes (9.5)

2014-11-21 Thread Heikki Linnakangas
On 11/21/2014 09:19 AM, Michael Paquier wrote: On Fri, Nov 21, 2014 at 2:06 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: As you may have noticed, I committed this (after some more cleanup). Of course, feel free to still review it, and please point out any issues you may find

Re: [HACKERS] What exactly is our CRC algorithm?

2014-11-21 Thread Heikki Linnakangas
On 11/21/2014 12:11 PM, Abhijit Menon-Sen wrote: At 2014-11-20 13:47:00 +0530, a...@2ndquadrant.com wrote: Suggestions for how to address (b) are welcome. With help from Andres, I set up a workload where XLogInsert* was at the top of my profiles: server with fsync and synchronous_commit

Re: [HACKERS] What exactly is our CRC algorithm?

2014-11-21 Thread Heikki Linnakangas
On 11/21/2014 01:06 PM, Andres Freund wrote: On 2014-11-21 13:01:50 +0200, Heikki Linnakangas wrote: On 11/21/2014 12:11 PM, Abhijit Menon-Sen wrote: At 2014-11-20 13:47:00 +0530, a...@2ndquadrant.com wrote: Suggestions for how to address (b) are welcome. With help from Andres, I set up

Re: [HACKERS] Comment header for src/test/regress/regress.c

2014-11-21 Thread Heikki Linnakangas
On 11/21/2014 06:23 AM, Ian Barwick wrote: I thought it might be useful to add a few words at the top of 'src/test/regress/regress.c' to explain what it does and to help differentiate it from 'pg_regress.c' and 'pg_regress_main.c'. Makes sense, committed. I remember being a bit confused on

Re: [HACKERS] WAL format and API changes (9.5)

2014-11-21 Thread Heikki Linnakangas
On 11/21/2014 05:58 AM, Amit Kapila wrote: On Thu, Nov 20, 2014 at 10:36 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: As you may have noticed, I committed this (after some more cleanup). Of course, feel free to still review it, and please point out any issues you may find. Few

Re: [HACKERS] How to use brin indexes?

2014-11-21 Thread Heikki Linnakangas
On 11/21/2014 09:59 PM, hubert depesz lubaczewski wrote: I tried to add BRIN index on very simple table, to test it for Waiting for blogpost, btu I can't get it to work. I got newest git head, and ran: $ create table t (id int8); CREATE TABLE (depesz@[local]:5930) 20:56:22 [depesz] $ insert

Re: [HACKERS] WAL format and API changes (9.5)

2014-11-20 Thread Heikki Linnakangas
As you may have noticed, I committed this (after some more cleanup). Of course, feel free to still review it, and please point out any issues you may find. - Heikki -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription:

[HACKERS] Increasing test coverage of WAL redo functions

2014-11-19 Thread Heikki Linnakangas
To test WAL replay, I often set up a master-standby system with streaming replication and run make installcheck on the master. However, the regression suite doesn't generate all WAL record types. I spent some time looking at the lcov report (make coverage-html), and crafted new tests to test

Re: [HACKERS] What exactly is our CRC algorithm?

2014-11-19 Thread Heikki Linnakangas
On 11/19/2014 05:58 PM, Abhijit Menon-Sen wrote: At 2014-11-11 16:56:00 +0530, a...@2ndquadrant.com wrote: I'm working on this (first speeding up the default calculation using slice-by-N, then adding support for the SSE4.2 CRC instruction on top). I've done the first part in the attached

Re: [HACKERS] Increasing test coverage of WAL redo functions

2014-11-19 Thread Heikki Linnakangas
On 11/19/2014 04:54 PM, Alvaro Herrera wrote: Also I'm surprised that BRIN did not turn up here. At least the page evacuation protocol to obtain a new revmap page is not exercised by the current tests. I suppose it's because all WAL records are covered by other activity, and page evacuation

Re: [HACKERS] What exactly is our CRC algorithm?

2014-11-19 Thread Heikki Linnakangas
On 11/19/2014 06:49 PM, Robert Haas wrote: On Wed, Nov 19, 2014 at 11:44 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: That's an interesting choice of workload. That sure is heavy on the CRC calculation, but the speed of pg_xlogdump hardly matters in real life. But isn't a workload

Re: [HACKERS] Increasing test coverage of WAL redo functions

2014-11-19 Thread Heikki Linnakangas
On 11/19/2014 05:01 PM, Andres Freund wrote: On 2014-11-19 11:54:47 -0300, Alvaro Herrera wrote: Heikki Linnakangas wrote: Schema | Name | Type | Owner | Size | Description +--+---++-+- public | btree_tall_tbl

<    5   6   7   8   9   10   11   12   13   14   >