Re: [HACKERS] Securing make check (CVE-2014-0067)

2014-12-25 Thread David Rowley
On 25 December 2014 at 18:27, Noah Misch n...@leadboat.com wrote: On Thu, Dec 25, 2014 at 03:55:02PM +1300, David Rowley wrote: f6dc6dd seems to have broken vcregress check for me: FATAL: no pg_hba.conf entry for host ::1, user David, database postgres ... FATAL: no pg_hba.conf

Re: [HACKERS] pgaudit - an auditing extension for PostgreSQL

2014-12-25 Thread Abhijit Menon-Sen
At 2014-12-22 08:05:57 -0500, robertmh...@gmail.com wrote: On Tue, Dec 16, 2014 at 1:28 PM, Stephen Frost sfr...@snowman.net wrote: … ok, does the audit role have any permissions here? and, if the result is yes, then the command is audited. … This is a little weird because you're

Re: [HACKERS] Moving RestoreBlockImage from xlogreader.c to xlogutils.c

2014-12-25 Thread Andres Freund
On 2014-12-25 08:52:05 +0900, Michael Paquier wrote: On Wed, Dec 24, 2014 at 10:51 PM, Fujii Masao masao.fu...@gmail.com wrote: Fair enough. Anyway I wait for applying the patch which moves pg_lzcompress.c until we will have reached any consensus about this. Just to be clear (after

Re: hash_create API changes (was Re: [HACKERS] speedup tidbitmap patch: hash BlockNumber)

2014-12-25 Thread Andres Freund
On 2014-12-24 13:58:41 -0600, Jim Nasby wrote: This surprises me given that SMHasher shows XXH to be 50% faster than Spooky for 20 byte keys. Note that there's quite some differences in how smhasher and postgres use the hash functions. The former nearly exclusively exercises the hash function

Re: [HACKERS] Moving RestoreBlockImage from xlogreader.c to xlogutils.c

2014-12-25 Thread Michael Paquier
On Thu, Dec 25, 2014 at 7:48 PM, Andres Freund and...@2ndquadrant.com wrote: I think it's a bad idea to move it away - the current placement provides a API that allows to get at the image data without having to deal with the low level details. E.g. the in_use, has_image and hole logic.

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-12-25 Thread Michael Paquier
On Wed, Dec 24, 2014 at 9:03 PM, Michael Paquier michael.paqu...@gmail.com wrote: Returning only a boolean is fine for me (that's what my first patch did), especially if we add at some point hooks for compression and decompression calls. Here is a patch rebased on current HEAD (60838df) for the

Re: [HACKERS] Moving RestoreBlockImage from xlogreader.c to xlogutils.c

2014-12-25 Thread Andres Freund
On 2014-12-25 21:12:54 +0900, Michael Paquier wrote: On Thu, Dec 25, 2014 at 7:48 PM, Andres Freund and...@2ndquadrant.com wrote: I think it's a bad idea to move it away - the current placement provides a API that allows to get at the image data without having to deal with the low level

Re: [HACKERS] pgaudit - an auditing extension for PostgreSQL

2014-12-25 Thread Jaime Casanova
On Thu, Dec 25, 2014 at 5:42 AM, Abhijit Menon-Sen a...@2ndquadrant.com wrote: To summarise for people who haven't followed the thread in detail, the idea is that you would do: grant select on foo to audit; ...and the server would audit-log any select ... from foo ... queries (by any

[HACKERS] Better way of dealing with pgstat wait timeout during buildfarm runs?

2014-12-25 Thread Andres Freund
Hi, We quite regularly have buildfarm failures that are caused by 'WARNING: pgstat wait timeout' at random points during the build. Especially on some of the ARM buildfarm animals those are really frequent, to the point that it's hard to know the actual state of the buildfarm without looking at

Re: [HACKERS] Better way of dealing with pgstat wait timeout during buildfarm runs?

2014-12-25 Thread Tom Lane
Andres Freund and...@2ndquadrant.com writes: So I think a better way to deal with that warning would be a good idea. Besides somehow making the mechanism there are two ways to attack this that I can think of, neither of them awe inspiring: 1) Make that WARNING a LOG message instead. Since

Re: [HACKERS] Better way of dealing with pgstat wait timeout during buildfarm runs?

2014-12-25 Thread Andres Freund
On 2014-12-25 14:36:42 -0500, Tom Lane wrote: I wonder whether when multiple processes are demanding statsfile updates, there's some misbehavior that causes them to suck CPU away from the stats collector and/or convince it that it doesn't need to write anything. There are odd things in the

[HACKERS] Some other odd buildfarm failures

2014-12-25 Thread Tom Lane
These two recent failures look suspiciously similar: http://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=jaguarundidt=2014-12-24%2021%3A03%3A05 http://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=koupreydt=2014-12-25%2018%3A43%3A17 to wit: ***

Re: [HACKERS] Some other odd buildfarm failures

2014-12-25 Thread Tom Lane
I wrote: These two recent failures look suspiciously similar: http://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=jaguarundidt=2014-12-24%2021%3A03%3A05 http://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=koupreydt=2014-12-25%2018%3A43%3A17 And I'd barely finished posting that before

Re: [HACKERS] Better way of dealing with pgstat wait timeout during buildfarm runs?

2014-12-25 Thread Tomas Vondra
On 25.12.2014 20:36, Tom Lane wrote: Yeah, I've been getting more annoyed by that too lately. I keep wondering though whether there's an actual bug underneath that behavior that we're failing to see. PGSTAT_MAX_WAIT_TIME is already 10 seconds; it's hard to credit that increasing it still

Re: [HACKERS] Better way of dealing with pgstat wait timeout during buildfarm runs?

2014-12-25 Thread Tom Lane
Tomas Vondra t...@fuzzy.cz writes: On 25.12.2014 20:36, Tom Lane wrote: BTW, I notice that in the current state of pgstat.c, all the logic for keeping track of request arrival times is dead code, because nothing is actually looking at DBWriteRequest.request_time. Really? Which part of the

[HACKERS] POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)

2014-12-25 Thread Alex Shulgin
Trent Shipley trent_ship...@qwest.net writes: On Friday 2007-12-14 16:22, Tom Lane wrote: Neil Conway ne...@samurai.com writes: By modifying COPY: COPY IGNORE ERRORS or some such would instruct COPY to drop (and log) rows that contain malformed data. That is, rows with too many or too few

Re: [HACKERS] Proposal: two new role attributes and/or capabilities?

2014-12-25 Thread Robert Haas
On Tue, Dec 23, 2014 at 11:20 AM, José Luis Tallón jltal...@adv-solutions.net wrote: I've found myself needing two role capabilities? as of lately, when thinking about restricting some roles to the barely minimum allowed permissions needed to perform their duties ... as opposed to having a

Re: [HACKERS] Better way of dealing with pgstat wait timeout during buildfarm runs?

2014-12-25 Thread Tomas Vondra
On 25.12.2014 21:14, Andres Freund wrote: On 2014-12-25 14:36:42 -0500, Tom Lane wrote: My guess is that a checkpoint happened at that time. Maybe it'd be a good idea to make pg_regress start postgres with log_checkpoints enabled? My guess is that we'd find horrendous 'sync' times.

Re: [HACKERS] Better way of dealing with pgstat wait timeout during buildfarm runs?

2014-12-25 Thread Tom Lane
Tomas Vondra t...@fuzzy.cz writes: The strange thing is that the split happened ~2 years ago, which is inconsistent with the sudden increase of this kind of issues. So maybe something changed on that particular animal (a failing SD card causing I/O stalls, perhaps)? I think that hamster has

Re: [HACKERS] Better way of dealing with pgstat wait timeout during buildfarm runs?

2014-12-25 Thread Tomas Vondra
On 25.12.2014 22:16, Tom Lane wrote: Tomas Vondra t...@fuzzy.cz writes: On 25.12.2014 20:36, Tom Lane wrote: BTW, I notice that in the current state of pgstat.c, all the logic for keeping track of request arrival times is dead code, because nothing is actually looking at

Re: [HACKERS] Better way of dealing with pgstat wait timeout during buildfarm runs?

2014-12-25 Thread Tomas Vondra
On 25.12.2014 22:40, Tom Lane wrote: Tomas Vondra t...@fuzzy.cz writes: The strange thing is that the split happened ~2 years ago, which is inconsistent with the sudden increase of this kind of issues. So maybe something changed on that particular animal (a failing SD card causing I/O stalls,

Re: [HACKERS] Securing make check (CVE-2014-0067)

2014-12-25 Thread Noah Misch
On Thu, Dec 25, 2014 at 11:35:31PM +1300, David Rowley wrote: On 25 December 2014 at 18:27, Noah Misch n...@leadboat.com wrote: This needs to be conditional on whether the platform supports IPv6, like we do in setup_config(). The attached patch works on these configurations: 64-bit

Re: [HACKERS] Some other odd buildfarm failures

2014-12-25 Thread Andres Freund
Hi, On 2014-12-25 16:01:47 -0500, Tom Lane wrote: In any case, it now seems dead certain that this is a recently introduced bug. Andres is fortunate that the first instance occurred before his recent batch of commits, or I'd be on him to revert them. Yes, Phew. These look rather odd. As

Re: [HACKERS] Some other odd buildfarm failures

2014-12-25 Thread Tom Lane
Andres Freund and...@2ndquadrant.com writes: I wonder if it'd not be a good idea if the event trigger code installed a error context callback? Since they can be called in situations we don't routinely expect that'd make diagnosis in many cases easier. +1 ... even if that's not related to the

Re: [HACKERS] Better way of dealing with pgstat wait timeout during buildfarm runs?

2014-12-25 Thread Tom Lane
Tomas Vondra t...@fuzzy.cz writes: On 25.12.2014 22:40, Tom Lane wrote: I think that hamster has basically got a tin can and string for an I/O subsystem. It's not real clear to me whether there's actually been an increase in wait timeout failures recently; somebody would have to go through

Re: [HACKERS] Better way of dealing with pgstat wait timeout during buildfarm runs?

2014-12-25 Thread Tomas Vondra
On 26.12.2014 02:59, Tom Lane wrote: Tomas Vondra t...@fuzzy.cz writes: On 25.12.2014 22:40, Tom Lane wrote: I think that hamster has basically got a tin can and string for an I/O subsystem. It's not real clear to me whether there's actually been an increase in wait timeout failures

Re: [HACKERS] Better way of dealing with pgstat wait timeout during buildfarm runs?

2014-12-25 Thread Michael Paquier
On Fri, Dec 26, 2014 at 6:28 AM, Tomas Vondra t...@fuzzy.cz wrote: On 25.12.2014 21:14, Andres Freund wrote: On 2014-12-25 14:36:42 -0500, Tom Lane wrote: My guess is that a checkpoint happened at that time. Maybe it'd be a good idea to make pg_regress start postgres with log_checkpoints

Re: [HACKERS] Some other odd buildfarm failures

2014-12-25 Thread Tom Lane
Andres Freund and...@2ndquadrant.com writes: My guess is that it's related to d7ee82e50f. It seems realistic that the event trigger added by it to the object_address test can cause errors at varying times. [ squint... ] Event triggers are global across the whole database, aren't they? Isn't

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-12-25 Thread Michael Paquier
On Thu, Dec 25, 2014 at 10:10 PM, Michael Paquier michael.paqu...@gmail.com wrote: On Wed, Dec 24, 2014 at 9:03 PM, Michael Paquier michael.paqu...@gmail.com wrote: Returning only a boolean is fine for me (that's what my first patch did), especially if we add at some point hooks for

Re: [HACKERS] Some other odd buildfarm failures

2014-12-25 Thread Alvaro Herrera
Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: My guess is that it's related to d7ee82e50f. It seems realistic that the event trigger added by it to the object_address test can cause errors at varying times. [ squint... ] Event triggers are global across the whole

Re: [HACKERS] Better way of dealing with pgstat wait timeout during buildfarm runs?

2014-12-25 Thread Alvaro Herrera
Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: So I think a better way to deal with that warning would be a good idea. Besides somehow making the mechanism there are two ways to attack this that I can think of, neither of them awe inspiring: 1) Make that WARNING a LOG

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-12-25 Thread Noah Misch
On Thu, Nov 20, 2014 at 09:15:51PM +0100, Andres Freund wrote: On 2014-11-20 13:57:25 -0500, Robert Haas wrote: On Thu, Nov 20, 2014 at 12:19 PM, Andres Freund and...@2ndquadrant.com wrote: If the deadlock detector would kill the processes anyway, it doesn't matter, because

Re: [HACKERS] Join push-down support for foreign tables

2014-12-25 Thread Shigeru Hanada
2014-12-16 0:45 GMT+09:00 Tom Lane t...@sss.pgh.pa.us: Shigeru Hanada shigeru.han...@gmail.com writes: I'm working on $SUBJECT and would like to get comments about the design. Attached patch is for the design below. Note that the patch requires Kaigai-san's custom foriegn join patch[1] For

Re: [HACKERS] Join push-down support for foreign tables

2014-12-25 Thread Shigeru Hanada
2014-12-16 1:22 GMT+09:00 Robert Haas robertmh...@gmail.com: On Mon, Dec 15, 2014 at 3:40 AM, Shigeru Hanada shigeru.han...@gmail.com wrote: I'm working on $SUBJECT and would like to get comments about the design. Attached patch is for the design below. I'm glad you are working on this.

Re: [HACKERS] inherit support for foreign tables

2014-12-25 Thread Etsuro Fujita
On 2014/12/23 0:36, Tom Lane wrote: Yeah, we need to do something about the PlanRowMark data structure. Aside from the pre-existing issue in postgres_fdw, we need to fix it to support inheritance trees in which more than one rowmark method is being used. That rte.hasForeignChildren thing is a

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-12-25 Thread Fujii Masao
On Fri, Dec 26, 2014 at 12:31 PM, Michael Paquier michael.paqu...@gmail.com wrote: On Thu, Dec 25, 2014 at 10:10 PM, Michael Paquier michael.paqu...@gmail.com wrote: On Wed, Dec 24, 2014 at 9:03 PM, Michael Paquier michael.paqu...@gmail.com wrote: Returning only a boolean is fine for me

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-12-25 Thread Michael Paquier
On Fri, Dec 26, 2014 at 3:24 PM, Fujii Masao masao.fu...@gmail.com wrote: pglz_compress() and pglz_decompress() still use PGLZ_Header, so the frontend which uses those functions needs to handle PGLZ_Header. But it basically should be handled via the varlena macros. That is, the frontend still

[HACKERS] The return value of allocate_recordbuf()

2014-12-25 Thread Fujii Masao
Hi, While reviewing FPW compression patch, I found that allocate_recordbuf() always returns TRUE though its source code comment says that FALSE is returned if out of memory. Its return value is checked in two places, but which is clearly useless. allocate_recordbuf() was introduced by 7fcbf6a,

Re: [HACKERS] a fast bloat measurement tool (was Re: Measuring relation free space)

2014-12-25 Thread Abhijit Menon-Sen
At 2014-09-25 15:40:11 +0530, a...@2ndquadrant.com wrote: All right, then I'll post a version that addresses Amit's other points, adds a new file/function to pgstattuple, acquires content locks, and uses HeapTupleSatisfiesVacuum, hint-bit setting and all. Sorry, I forgot to post this patch.