Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2014-01-22 Thread KONDO Mitsumasa
(2014/01/22 9:34), Simon Riggs wrote: AFAICS, all that has happened is that people have given their opinions and we've got almost the same identical patch, with a rush-rush comment to commit even though we've waited months. If you submit a patch, then you need to listen to feedback and be clear

Re: [HACKERS] Trigger information for auto_explain.

2014-01-22 Thread Kyotaro HORIGUCHI
Hello, I came back with doc patch and revised 0002 patch. I think documentation is the only thing that stops this patch to be commitable... can you add it? Agreed. I have pushed patch 0001 for now. Thank you, I'll put it sooner. I found the default setting for log_triggers was

Re: [HACKERS] GIN improvements part 1: additional information

2014-01-22 Thread Heikki Linnakangas
On 01/22/2014 09:25 AM, Alexander Korotkov wrote: On Wed, Jan 22, 2014 at 1:21 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 01/21/2014 11:35 AM, Heikki Linnakangas wrote: Oh, I see what's going on. I had assumed that there cannot be duplicate insertions into the posting tree,

Re: [HACKERS] Proposal: variant of regclass

2014-01-22 Thread Marti Raudsepp
On Tue, Jan 14, 2014 at 9:28 AM, Yugo Nagata nag...@sraoss.co.jp wrote: Here is the patch to implement to_regclass, to_regproc, to_regoper, and to_regtype. + static Datum regclass_guts(char *class_name_or_oid, bool raiseError); Minor bikeshedding, a lot of code currently uses an argument named

Re: [HACKERS] Storing pg_stat_statements query texts externally, pg_stat_statements in core

2014-01-22 Thread Peter Geoghegan
On Tue, Jan 21, 2014 at 8:01 PM, Peter Geoghegan p...@heroku.com wrote: Actually, I think the whole thing is rather badly designed, as warm cache or no you're still proposing to do thousands of kernel calls while holding a potentially contended LWLock. I suggest what we do is (1) read in the

Re: [HACKERS] Storing pg_stat_statements query texts externally, pg_stat_statements in core

2014-01-22 Thread Peter Geoghegan
On Tue, Jan 21, 2014 at 8:01 PM, Peter Geoghegan p...@heroku.com wrote: BTW shouldn't there be an fflush() in qtext_store? I don't think so, no. Whenever qtext_store() callers don't fflush() before releasing their exclusive lock, they FreeFile() before doing so instead. In the case of

Re: [HACKERS] Proposal: variant of regclass

2014-01-22 Thread Tatsuo Ishii
On Tue, Jan 14, 2014 at 9:28 AM, Yugo Nagata nag...@sraoss.co.jp wrote: Here is the patch to implement to_regclass, to_regproc, to_regoper, and to_regtype. + static Datum regclass_guts(char *class_name_or_oid, bool raiseError); Minor bikeshedding, a lot of code currently uses an argument

Re: [HACKERS] Patch: Show process IDs of processes holding a lock; show relation and tuple infos of a lock to acquire

2014-01-22 Thread Simon Riggs
On 22 January 2014 04:42, Rajeev rastogi rajeev.rast...@huawei.com wrote: On 31st December 2013, Christian Kruse Wrote: Hi there, I created two patches improving the log messages generated by log_lock_waits. The first patch shows the process IDs holding a lock we try to acquire

Re: [HACKERS] Proposal: variant of regclass

2014-01-22 Thread Yugo Nagata
On Wed, 22 Jan 2014 20:04:12 +0900 (JST) Tatsuo Ishii is...@postgresql.org wrote: On Tue, Jan 14, 2014 at 9:28 AM, Yugo Nagata nag...@sraoss.co.jp wrote: Here is the patch to implement to_regclass, to_regproc, to_regoper, and to_regtype. + static Datum regclass_guts(char

Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)

2014-01-22 Thread Simon Riggs
On 22 January 2014 01:23, Tom Lane t...@sss.pgh.pa.us wrote: Andres Freund and...@2ndquadrant.com writes: On 2014-01-21 18:59:13 -0500, Tom Lane wrote: Another thing to think about is whether we couldn't put a hard limit on WAL record size somehow. Multi-megabyte WAL records are an abuse of

Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)

2014-01-22 Thread Simon Riggs
On 22 January 2014 01:30, Tom Lane t...@sss.pgh.pa.us wrote: Andres Freund and...@2ndquadrant.com writes: How are we supposed to wait while e.g. ProcArrayLock? Aborting transactions doesn't work either, that writes abort records which can get signficantly large. Yeah, that's an interesting

GIN pending list pages not recycled promptly (was Re: [HACKERS] GIN improvements part 1: additional information)

2014-01-22 Thread Heikki Linnakangas
On 01/22/2014 03:39 AM, Tomas Vondra wrote: What annoys me a bit is the huge size difference between the index updated incrementally (by a sequence of INSERT commands), and the index rebuilt from scratch using VACUUM FULL. It's a bit better with the patch (2288 vs. 2035 MB), but is there a

Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)

2014-01-22 Thread Heikki Linnakangas
On 01/22/2014 02:10 PM, Simon Riggs wrote: As Jeff points out, the blocks being modified would be locked until space is freed up. Which could make other users wait. The code required to avoid that wait would be complex and not worth any overhead. Checkpoint also acquires the content lock of

Re: [HACKERS] GIN improvements part 1: additional information

2014-01-22 Thread Alexander Korotkov
On Wed, Jan 22, 2014 at 12:30 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 01/22/2014 09:25 AM, Alexander Korotkov wrote: On Wed, Jan 22, 2014 at 1:21 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 01/21/2014 11:35 AM, Heikki Linnakangas wrote: Oh, I see

Re: [HACKERS] improve the help message about psql -F

2014-01-22 Thread Jov
OK,I will do it.Thanks. Jov blog: http:amutu.com/blog http://amutu.com/blog 2014/1/21 Marti Raudsepp ma...@juffo.org On Mon, Jan 20, 2014 at 2:04 PM, Jov am...@amutu.com wrote: reasonable,I removed the set,patch attached. Hi Jov, A new commitfest was just opened, due on 2014-06. Please

Re: [HACKERS] [bug fix] pg_ctl always uses the same event source

2014-01-22 Thread Robert Haas
On Tue, Jan 21, 2014 at 11:20 PM, Amit Kapila amit.kapil...@gmail.com wrote: On Wed, Jan 22, 2014 at 9:19 AM, Tom Lane t...@sss.pgh.pa.us wrote: Amit Kapila amit.kapil...@gmail.com writes: On Tue, Jan 21, 2014 at 6:57 PM, MauMau maumau...@gmail.com wrote: To follow this, we have the line as:

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2014-01-22 Thread Robert Haas
On Wed, Jan 22, 2014 at 3:32 AM, KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp wrote: OK, Kondo, please demonstrate benchmarks that show we have 1% impact from this change. Otherwise we may need a config parameter to allow the calculation. OK, testing DBT-2 now. However, error range of

Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)

2014-01-22 Thread Simon Riggs
On 22 January 2014 13:14, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 01/22/2014 02:10 PM, Simon Riggs wrote: As Jeff points out, the blocks being modified would be locked until space is freed up. Which could make other users wait. The code required to avoid that wait would be

Re: GIN pending list pages not recycled promptly (was Re: [HACKERS] GIN improvements part 1: additional information)

2014-01-22 Thread Alvaro Herrera
Heikki Linnakangas wrote: I wrote a little utility that scans all pages in a gin index, and prints out the flags indicating what kind of a page it is. The distribution looks like this: 19 DATA 7420 DATA LEAF 24701 DELETED 1 LEAF 1 META Hah. I think we need to

Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)

2014-01-22 Thread Kevin Grittner
Tom Lane t...@sss.pgh.pa.us wrote: Well, PANIC is certainly bad, but what I'm suggesting is that we just focus on getting that down to ERROR and not worry about trying to get out of the disk-shortage situation automatically. Nor do I believe that it's such a good idea to have the database

Re: [HACKERS] ALTER SYSTEM SET typos and fix for temporary file name management

2014-01-22 Thread Robert Haas
On Tue, Jan 21, 2014 at 7:02 PM, Michael Paquier michael.paqu...@gmail.com wrote: On Wed, Jan 22, 2014 at 5:29 AM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: I agree with Michael that having pg_basebackup be aware of the .temp suffix is ugly; for instance if we were to fix it to .tmp in

Re: [HACKERS] Dynamic Shared Memory stuff

2014-01-22 Thread Robert Haas
On Tue, Jan 21, 2014 at 2:58 PM, Noah Misch n...@leadboat.com wrote: What do people prefer? I recommend performing cleanup on the control segment named in PGShmemHeader just before shmdt() in PGSharedMemoryCreate(). No new ERROR or WARNING sites are necessary. Have dsm_postmaster_startup()

Re: [HACKERS] [bug fix] pg_ctl always uses the same event source

2014-01-22 Thread Amit Kapila
On Wed, Jan 22, 2014 at 6:54 PM, Robert Haas robertmh...@gmail.com wrote: On Tue, Jan 21, 2014 at 11:20 PM, Amit Kapila amit.kapil...@gmail.com wrote: On Wed, Jan 22, 2014 at 9:19 AM, Tom Lane t...@sss.pgh.pa.us wrote: Amit Kapila amit.kapil...@gmail.com writes: On Tue, Jan 21, 2014 at 6:57

[HACKERS] The problems of PQhost()

2014-01-22 Thread Fujii Masao
Hi, I reported in other thread that PQhost() has three problems. http://www.postgresql.org/message-id/cahgqgwe77akyabywde5+8bjldopthp7cnswao_syedjogyv...@mail.gmail.com (1) PQhost() can return Unix-domain socket directory path even in the platform that doesn't

Re: [HACKERS] Changeset Extraction v7.0 (was logical changeset generation)

2014-01-22 Thread Andres Freund
On 2014-01-18 08:35:47 -0500, Robert Haas wrote: I am not sure I understand that point. We can either update the in-memory bit before performing the on-disk operations or afterwards. Either way, there's a way to be inconsistent if the disk operation fails somewhere inbetween (it might fail

Re: [HACKERS] Re: [BUGS] BUG #7873: pg_restore --clean tries to drop tables that don't exist

2014-01-22 Thread Pavel Stehule
Hello 2014/1/22 Dean Rasheed dean.a.rash...@gmail.com On 21 January 2014 22:28, Alvaro Herrera alvhe...@2ndquadrant.com wrote: I have been mulling over this patch, and I can't seem to come to terms with it. I first started making it look nicer here and there, thinking it was all mostly

Re: [HACKERS] Changeset Extraction v7.0 (was logical changeset generation)

2014-01-22 Thread Robert Haas
On Wed, Jan 22, 2014 at 9:48 AM, Andres Freund and...@2ndquadrant.com wrote: On 2014-01-18 08:35:47 -0500, Robert Haas wrote: I am not sure I understand that point. We can either update the in-memory bit before performing the on-disk operations or afterwards. Either way, there's a way to be

Re: [HACKERS] Dynamic Shared Memory stuff

2014-01-22 Thread Noah Misch
On Wed, Jan 22, 2014 at 09:32:09AM -0500, Robert Haas wrote: On Tue, Jan 21, 2014 at 2:58 PM, Noah Misch n...@leadboat.com wrote: What do people prefer? I recommend performing cleanup on the control segment named in PGShmemHeader just before shmdt() in PGSharedMemoryCreate(). No new

Re: [HACKERS] Why conf.d should be default, and auto.conf and recovery.conf should be in it

2014-01-22 Thread Tom Lane
Amit Kapila amit.kapil...@gmail.com writes: On Tue, Jan 21, 2014 at 8:25 PM, Robert Haas robertmh...@gmail.com wrote: While reviewing, I noted that the skipping missing configuration file message in ParseConfigFile() uses an elevel of LOG, while the other messages in the same file use elevel.

Re: [HACKERS] patch: option --if-exists for pg_dump

2014-01-22 Thread Pavel Stehule
Tomorrow I'll send updated version Regards Pavel 2014/1/21 Jeevan Chalke jeevan.cha...@enterprisedb.com Hi Pavel, Consider following test scenario: mydb=# \d emp Table public.emp Column | Type | Modifiers +-+--- empno | integer | not null deptno

Re: [HACKERS] [bug fix] pg_ctl always uses the same event source

2014-01-22 Thread Tom Lane
Amit Kapila amit.kapil...@gmail.com writes: On Wed, Jan 22, 2014 at 6:54 PM, Robert Haas robertmh...@gmail.com wrote: I wonder if the port number wouldn't be a better choice. And that could even be done without adding a parameter. We need this for register of event source which might be done

Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)

2014-01-22 Thread Andres Freund
On 2014-01-21 21:42:19 -0500, Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: On 2014-01-21 19:45:19 -0500, Tom Lane wrote: I don't think that's a comparable case. Incomplete actions are actions to be taken immediately, and which the replayer then has to complete somehow if

Re: [HACKERS] Funny representation in pg_stat_statements.query.

2014-01-22 Thread Alvaro Herrera
Kyotaro HORIGUCHI wrote: Hello, Since this becomes more than a simple bug fix, I think it is too late for 9.4 now. I'll work on this in the longer term. OK. I will get around to it someday, but if you do it first, that's fine. Nevertheless of my words, the drive is getting

Re: [HACKERS] Changeset Extraction v7.0 (was logical changeset generation)

2014-01-22 Thread Andres Freund
On 2014-01-22 10:14:27 -0500, Robert Haas wrote: On Wed, Jan 22, 2014 at 9:48 AM, Andres Freund and...@2ndquadrant.com wrote: On 2014-01-18 08:35:47 -0500, Robert Haas wrote: I am not sure I understand that point. We can either update the in-memory bit before performing the on-disk

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-22 Thread Robert Haas
On Tue, Jan 21, 2014 at 3:20 PM, Jan Kara j...@suse.cz wrote: But that still doesn't work out very well, because now the guy who does the write() has to wait for it to finish before he can do anything else. That's not always what we want, because WAL gets written out from our internal buffers

Re: [HACKERS] Patch: Show process IDs of processes holding a lock; show relation and tuple infos of a lock to acquire

2014-01-22 Thread Christian Kruse
Hi, On 22/01/14 12:40, Simon Riggs wrote: 1. I find a issue in following scenario: session 1 with process id X: BEGIN; LOCK TABLE foo IN SHARE MODE; session 2 with process id Y: BEGIN; LOCK TABLE foo IN EXCLUSIVE MODE; session 3 with process id Z: BEGIN; LOCK

Re: [HACKERS] Patch: Show process IDs of processes holding a lock; show relation and tuple infos of a lock to acquire

2014-01-22 Thread Alvaro Herrera
Christian Kruse wrote: I think this could use some more comments -- for instance at the top of the while loop, explain what's its purpose. if (deadlock_state == DS_SOFT_DEADLOCK) ereport(LOG,

[HACKERS] WAL replay should fdatasync() segments?

2014-01-22 Thread Andres Freund
Hi, Currently, XLogInsert(), XLogFlush() or XLogBackgroundFlush() will write() data before fdatasync()ing them (duh, kinda obvious). But I think given the current recovery code that leaves a window where we can get into strange inconsistencies. Consider what happens if postgres (not the OS!)

Re: [HACKERS] Incorrectly reporting config errors

2014-01-22 Thread Kevin Grittner
Andres Freund and...@2ndquadrant.com wrote: A rather common and sensible configuration is to have a common configuration file used across servers, which then is overwritten by a per-server or per-cluster config file containing values specific to a server/cluster. Agreed. My preference would

Re: [HACKERS] dynamic shared memory and locks

2014-01-22 Thread Andres Freund
On 2014-01-10 13:11:32 -0500, Robert Haas wrote: OK, I've implemented this: here's what I believe to be a complete patch, based on the previous lwlock-pointers.patch but now handling LOCK_DEBUG and TRACE_LWLOCKS and dtrace and a bunch of other loose ends. I think this should be adequate for

Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)

2014-01-22 Thread Tom Lane
Andres Freund and...@2ndquadrant.com writes: On 2014-01-21 21:42:19 -0500, Tom Lane wrote: Uh, what? The behavior I'm talking about is *exactly the same* as what happens now. The only change is that the data sent to the WAL file is laid out a bit differently, and the replay logic has to

Re: [HACKERS] WAL replay should fdatasync() segments?

2014-01-22 Thread Fujii Masao
On Thu, Jan 23, 2014 at 1:21 AM, Andres Freund and...@2ndquadrant.com wrote: Hi, Currently, XLogInsert(), XLogFlush() or XLogBackgroundFlush() will write() data before fdatasync()ing them (duh, kinda obvious). But I think given the current recovery code that leaves a window where we can get

Re: [HACKERS] Patch: Show process IDs of processes holding a lock; show relation and tuple infos of a lock to acquire

2014-01-22 Thread Tom Lane
Alvaro Herrera alvhe...@2ndquadrant.com writes: This ngettext() call is repeated four times in the new code, which is a bit annoying because it's not trivial. I think you could assign the ngettext() to a char * at the bottom of the loop, and then in the ereport() calls use it: Would that not

Re: [HACKERS] WAL replay should fdatasync() segments?

2014-01-22 Thread Andres Freund
On 2014-01-23 02:05:48 +0900, Fujii Masao wrote: On Thu, Jan 23, 2014 at 1:21 AM, Andres Freund and...@2ndquadrant.com wrote: Hi, Currently, XLogInsert(), XLogFlush() or XLogBackgroundFlush() will write() data before fdatasync()ing them (duh, kinda obvious). But I think given the

Re: [HACKERS] dynamic shared memory and locks

2014-01-22 Thread Tom Lane
Andres Freund and...@2ndquadrant.com writes: Shouldn't we introduce a typedef LWLock* LWLockid; or something to avoid breaking external code using lwlocks? +1, in fact there's probably no reason to touch most *internal* code using that type name either. regards, tom

Re: [HACKERS] Incorrectly reporting config errors

2014-01-22 Thread Andres Freund
On 2014-01-22 12:10:30 -0500, Tom Lane wrote: Kevin Grittner kgri...@ymail.com writes: My preference would be to not generate noise for interim states; just report net changes. Yeah. Is it worth explicitly detecting and dropping redundant assignments to the same variable? A naive check

Re: [HACKERS] Incorrectly reporting config errors

2014-01-22 Thread Tom Lane
Kevin Grittner kgri...@ymail.com writes: My preference would be to not generate noise for interim states; just report net changes. Yeah. Is it worth explicitly detecting and dropping redundant assignments to the same variable? A naive check for that would be O(N^2) in the number of entries in

Re: [HACKERS] GIN improvements part 1: additional information

2014-01-22 Thread Heikki Linnakangas
On 01/22/2014 02:17 PM, Alexander Korotkov wrote: We already spent a lot of time with compression. Now we need to figure out the result we want see. I spent quite long time debugging varbyte encoding without segments. Finally, it passed very many tests without any problems. Now, it is just piece

Re: [HACKERS] dynamic shared memory and locks

2014-01-22 Thread Robert Haas
On Wed, Jan 22, 2014 at 12:11 PM, Tom Lane t...@sss.pgh.pa.us wrote: Andres Freund and...@2ndquadrant.com writes: Shouldn't we introduce a typedef LWLock* LWLockid; or something to avoid breaking external code using lwlocks? +1, in fact there's probably no reason to touch most *internal* code

Re: [HACKERS] dynamic shared memory and locks

2014-01-22 Thread Andres Freund
On 2014-01-22 12:40:34 -0500, Robert Haas wrote: On Wed, Jan 22, 2014 at 12:11 PM, Tom Lane t...@sss.pgh.pa.us wrote: Andres Freund and...@2ndquadrant.com writes: Shouldn't we introduce a typedef LWLock* LWLockid; or something to avoid breaking external code using lwlocks? +1, in fact

Re: [HACKERS] proposal: hide application_name from other users

2014-01-22 Thread Josh Berkus
On 01/21/2014 05:21 PM, Andres Freund wrote: I think the only realistic thing is a monitoring capability, like we have replication. GRANT/REVOKE doesn't even come close to being able to generically allow to grant permissions of even the moderate complexity pg_stat_get_activity() has. That

Re: [HACKERS] new json funcs

2014-01-22 Thread Andrew Dunstan
On 01/21/2014 06:21 PM, Marko Tiikkaja wrote: Hi Andrew, On 1/18/14, 10:05 PM, I wrote: But I'll continue with my review now that this has been sorted out. Sorry about the delay. I think the API for the new functions looks good. They are all welcome additions to the JSON family. The

Re: [HACKERS] WAL replay should fdatasync() segments?

2014-01-22 Thread Fujii Masao
On Thu, Jan 23, 2014 at 2:08 AM, Andres Freund and...@2ndquadrant.com wrote: On 2014-01-23 02:05:48 +0900, Fujii Masao wrote: On Thu, Jan 23, 2014 at 1:21 AM, Andres Freund and...@2ndquadrant.com wrote: Hi, Currently, XLogInsert(), XLogFlush() or XLogBackgroundFlush() will write() data

Re: [HACKERS] Changeset Extraction v7.0 (was logical changeset generation)

2014-01-22 Thread Robert Haas
On Wed, Jan 22, 2014 at 10:48 AM, Andres Freund and...@2ndquadrant.com wrote: Yes, individual operations should be, but you cannot be sure whether a rename()/unlink() will survive a crash until the directory is fsync()ed. So, what is one going to do if the unlink suceeded, but the fsync

Re: [HACKERS] proposal: hide application_name from other users

2014-01-22 Thread Josh Berkus
On 01/21/2014 05:22 PM, Mark Kirkwood wrote: If said malicious attacker can log into postgres and issue its own queries, and connect to other database then you are in serious trouble already. I also wonder that if such an attacker knows the application name, that would suggest that they

Re: [HACKERS] Storing pg_stat_statements query texts externally, pg_stat_statements in core

2014-01-22 Thread Tom Lane
Peter Geoghegan p...@heroku.com writes: On Tue, Jan 21, 2014 at 8:01 PM, Peter Geoghegan p...@heroku.com wrote: Actually, I think the whole thing is rather badly designed, as warm cache or no you're still proposing to do thousands of kernel calls while holding a potentially contended LWLock.

Re: [HACKERS] Incorrectly reporting config errors

2014-01-22 Thread Kevin Grittner
Tom Lane t...@sss.pgh.pa.us wrote: Kevin Grittner kgri...@ymail.com writes: My preference would be to not generate noise for interim states; just report net changes. Yeah.  Is it worth explicitly detecting and dropping redundant assignments to the same variable?  A naive check for that would

Re: [HACKERS] proposal: hide application_name from other users

2014-01-22 Thread Kevin Grittner
Stephen Frost sfr...@snowman.net wrote: Harold Giménez (har...@heroku.com) wrote: This is just an example of how running monitoring as superuser is not necessarily the worst thing, and there are other reasons to do it already. It's a horrible thing and that isn't a good reason- if my

Re: [HACKERS] Patch: Show process IDs of processes holding a lock; show relation and tuple infos of a lock to acquire

2014-01-22 Thread Christian Kruse
Hi, attached you will find a new version of the patch containing more comments. On 22/01/14 12:00, Tom Lane wrote: Alvaro Herrera alvhe...@2ndquadrant.com writes: This ngettext() call is repeated four times in the new code, which is a bit annoying because it's not trivial. I think you

Re: [HACKERS] plpgsql.consistent_into

2014-01-22 Thread Jim Nasby
On 1/15/14, 12:35 AM, Tom Lane wrote: Jim Nasby j...@nasby.net writes: Do we actually support = right now? We already support v_field := field FROM table ... ; and I think it's a bad idea to have different meaning for = and :=. That ship sailed a *very* long time ago. See other thread about

Re: [HACKERS] Patch: Show process IDs of processes holding a lock; show relation and tuple infos of a lock to acquire

2014-01-22 Thread Tom Lane
Christian Kruse christ...@2ndquadrant.com writes: However, the real problem here is that you shouldn't be calling ngettext manually in an ereport context in the first place. There is infrastructure in place for that, and this isn't using it. Fixed in attached patch. I changed it from calling

Re: [HACKERS] proposal: hide application_name from other users

2014-01-22 Thread Greg Stark
On Tue, Jan 21, 2014 at 7:38 PM, Stephen Frost sfr...@snowman.net wrote: * Harold Giménez (har...@heroku.com) wrote: Definitely agree with you. This is just an example of how running monitoring as superuser is not necessarily the worst thing, and there are other reasons to do it already.

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-22 Thread Dave Chinner
On Tue, Jan 21, 2014 at 09:20:52PM +0100, Jan Kara wrote: On Fri 17-01-14 08:57:25, Robert Haas wrote: On Fri, Jan 17, 2014 at 7:34 AM, Jeff Layton jlay...@redhat.com wrote: So this says to me that the WAL is a place where DIO should really be reconsidered. It's mostly sequential writes

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-22 Thread Jan Kara
On Wed 22-01-14 09:07:19, Dave Chinner wrote: On Tue, Jan 21, 2014 at 09:20:52PM +0100, Jan Kara wrote: If we're forcing the WAL out to disk because of transaction commit or because we need to write the buffer protected by a certain WAL record only after the WAL hits the platter, then

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-22 Thread Jan Kara
On Fri 17-01-14 08:57:25, Robert Haas wrote: On Fri, Jan 17, 2014 at 7:34 AM, Jeff Layton jlay...@redhat.com wrote: So this says to me that the WAL is a place where DIO should really be reconsidered. It's mostly sequential writes that need to hit the disk ASAP, and you need to know that

Re: [HACKERS] PoC: Duplicate Tuple Elidation during External Sort for DISTINCT

2014-01-22 Thread Jeremy Harris
On 22/01/14 03:16, Jon Nelson wrote: Greetings -hackers: I have worked up a patch to PostgreSQL which elides tuples during an external sort. The primary use case is when sorted input is being used to feed a DISTINCT operation. The idea is to throw out tuples that compare as identical whenever

Re: [HACKERS] PoC: Duplicate Tuple Elidation during External Sort for DISTINCT

2014-01-22 Thread Jeremy Harris
On 22/01/14 03:53, Tom Lane wrote: Jon Nelson jnelson+pg...@jamponi.net writes: A rough summary of the patch follows: - a GUC variable enables or disables this capability - in nodeAgg.c, eliding duplicate tuples is enabled if the number of distinct columns is equal to the number of sort

Re: [HACKERS] Storing pg_stat_statements query texts externally, pg_stat_statements in core

2014-01-22 Thread Tom Lane
Peter Geoghegan p...@heroku.com writes: I ran an strace on a pg_stat_statements backend with a full ~5,000 entries. BTW, have you got any sort of test scenario you could share for this purpose? I'm sure I could build something, but if you've already done it ...

Re: [HACKERS] PoC: Duplicate Tuple Elidation during External Sort for DISTINCT

2014-01-22 Thread Tom Lane
Jeremy Harris j...@wizmail.org writes: On 22/01/14 03:53, Tom Lane wrote: Jon Nelson jnelson+pg...@jamponi.net writes: - in createplan.c, eliding duplicate tuples is enabled if we are creating a unique plan which involves sorting first [ raised eyebrow ... ] And what happens if the planner

Re: [HACKERS] Storing pg_stat_statements query texts externally, pg_stat_statements in core

2014-01-22 Thread Peter Geoghegan
On Wed, Jan 22, 2014 at 1:22 PM, Tom Lane t...@sss.pgh.pa.us wrote: BTW, have you got any sort of test scenario you could share for this purpose? I'm sure I could build something, but if you've already done it ... I simply ran the standard regression tests, and then straced a backend as it

Re: [HACKERS] Storing pg_stat_statements query texts externally, pg_stat_statements in core

2014-01-22 Thread Tom Lane
Peter Geoghegan p...@heroku.com writes: On Wed, Jan 22, 2014 at 1:22 PM, Tom Lane t...@sss.pgh.pa.us wrote: BTW, have you got any sort of test scenario you could share for this purpose? I'm sure I could build something, but if you've already done it ... I simply ran the standard regression

Re: [HACKERS] PoC: Duplicate Tuple Elidation during External Sort for DISTINCT

2014-01-22 Thread Jon Nelson
On Wed, Jan 22, 2014 at 3:26 PM, Tom Lane t...@sss.pgh.pa.us wrote: Jeremy Harris j...@wizmail.org writes: On 22/01/14 03:53, Tom Lane wrote: Jon Nelson jnelson+pg...@jamponi.net writes: - in createplan.c, eliding duplicate tuples is enabled if we are creating a unique plan which involves

Re: [HACKERS] Storing pg_stat_statements query texts externally, pg_stat_statements in core

2014-01-22 Thread Peter Geoghegan
On Wed, Jan 22, 2014 at 1:38 PM, Tom Lane t...@sss.pgh.pa.us wrote: Hm, are there 5K distinct queries in the regression tests? I'd have thought they weren't enough to fill a large pgss hash. Well, yes. They're a really bad case for pg_stat_statements, in that almost all distinct queries are

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-22 Thread Bruce Momjian
On Tue, Jan 21, 2014 at 09:20:52PM +0100, Jan Kara wrote: If we're forcing the WAL out to disk because of transaction commit or because we need to write the buffer protected by a certain WAL record only after the WAL hits the platter, then it's fine. But sometimes we're writing WAL just

Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)

2014-01-22 Thread Simon Riggs
On 22 January 2014 14:25, Simon Riggs si...@2ndquadrant.com wrote: On 22 January 2014 13:14, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 01/22/2014 02:10 PM, Simon Riggs wrote: As Jeff points out, the blocks being modified would be locked until space is freed up. Which could make

Re: [HACKERS] [PATCH] Negative Transition Aggregate Functions (WIP)

2014-01-22 Thread David Rowley
On Tue, Jan 21, 2014 at 3:20 AM, Florian Pflug f...@phlo.org wrote: On Jan20, 2014, at 08:42 , David Rowley dgrowle...@gmail.com wrote: On Mon, Jan 20, 2014 at 2:45 PM, Florian Pflug f...@phlo.org wrote: * I've also renamed INVFUNC to INVSFUNC. That's a pretty invasive change, and it's

Re: [HACKERS] [PATCH] Negative Transition Aggregate Functions (WIP)

2014-01-22 Thread David Rowley
On Wed, Jan 22, 2014 at 12:46 AM, Florian Pflug f...@phlo.org wrote: On Jan21, 2014, at 10:53 , David Rowley dgrowle...@gmail.com wrote: It came to me that it might be possible to implement inverse transitions for floating point aggregates by just detecting if precision has been lost

Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)

2014-01-22 Thread Jim Nasby
On 1/21/14, 6:46 PM, Andres Freund wrote: On 2014-01-21 16:34:45 -0800, Peter Geoghegan wrote: On Tue, Jan 21, 2014 at 3:43 PM, Andres Freundand...@2ndquadrant.com wrote: I personally think this isn't worth complicating the code for. You're probably right. However, I don't see why the bar

Re: [HACKERS] [PATCH] Negative Transition Aggregate Functions (WIP)

2014-01-22 Thread Florian Pflug
On Jan23, 2014, at 01:17 , David Rowley dgrowle...@gmail.com wrote: On Wed, Jan 22, 2014 at 12:46 AM, Florian Pflug f...@phlo.org wrote: If you want to play with this, I think the first step has to be to find a set of guarantees that SUM(float) is supposed to meet. Currently, SUM(float)

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-22 Thread Jim Nasby
On 1/17/14, 7:57 AM, Robert Haas wrote: - WAL files are written (and sometimes read) sequentially and fsync'd very frequently and it's always good to write the data out to disk as soon as possible - Temp files are written and read sequentially and never fsync'd. They should only be written to

Re: [HACKERS] proposal: hide application_name from other users

2014-01-22 Thread Greg Stark
On Wed, Jan 22, 2014 at 1:03 PM, Josh Berkus j...@agliodbs.com wrote: Probably Heroku has some more specific exploit case to be concerned about here; if so, might I suggest taking it up with the -security list? I don't think there's a specific vulnerability that needs to be kept secret here.

Re: [HACKERS] [PATCH] Relocation of tablespaces in pg_basebackup

2014-01-22 Thread Peter Eisentraut
My review: Clearly, everyone likes this feature. I'm tempted to think it should be mandatory to specify this option in plain mode when tablespaces are present. Otherwise, creating a base backup is liable to create random files all over the place. Obviously, there would be backward

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-22 Thread Jim Nasby
On 1/17/14, 2:24 PM, Gregory Smith wrote: I am skeptical that the database will take over very much of this work and perform better than the Linux kernel does. My take is that our most useful role would be providing test cases kernel developers can add to a performance regression suite. Ugly

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-22 Thread Jim Nasby
On 1/19/14, 5:51 PM, Dave Chinner wrote: Postgres is far from being the only application that wants this; many people resort to tmpfs because of this: https://lwn.net/Articles/499410/ Yes, we covered the possibility of using tmpfs much earlier in the thread, and came to the conclusion that temp

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2014-01-22 Thread KONDO Mitsumasa
(2014/01/22 22:26), Robert Haas wrote: On Wed, Jan 22, 2014 at 3:32 AM, KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp wrote: OK, Kondo, please demonstrate benchmarks that show we have 1% impact from this change. Otherwise we may need a config parameter to allow the calculation. OK, testing

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2014-01-22 Thread Peter Geoghegan
On Wed, Jan 22, 2014 at 5:28 PM, KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp wrote: Oh, thanks to inform me. I think essential problem of my patch has bottle neck in sqrt() function and other division caluculation. Well, that's a pretty easy theory to test. Just stop calling them (and do

Re: [HACKERS] Proposal: variant of regclass

2014-01-22 Thread Marti Raudsepp
On Wed, Jan 22, 2014 at 1:44 PM, Yugo Nagata nag...@sraoss.co.jp wrote: On Wed, 22 Jan 2014 20:04:12 +0900 (JST) Tatsuo Ishii is...@postgresql.org wrote: parseTypeString() is called by some other functions and I avoided influences of modifying the definition on them, since this should raise

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-22 Thread Claudio Freire
On Wed, Jan 22, 2014 at 10:08 PM, Jim Nasby j...@nasby.net wrote: Probably more useful is the case of index scans; if we pre-read more data from the index we could hand the kernel a list of base relation blocks that we know we'll need. Actually, I've already tried this. The most important

[HACKERS] Confusing documentation of ordered-set aggregates?

2014-01-22 Thread Florian Pflug
Hi After reading through the relevant parts of sytnax.sgml, create_aggregate.smgl and xaggr.sgml, I think I understand how these work - they work exactly like regular aggregates, except that some arguments are evaluated only once and passed to the final function instead of the transition

Re: [HACKERS] [PATCH] Negative Transition Aggregate Functions (WIP)

2014-01-22 Thread Florian Pflug
On Jan23, 2014, at 01:07 , David Rowley dgrowle...@gmail.com wrote: On Tue, Jan 21, 2014 at 3:20 AM, Florian Pflug f...@phlo.org wrote: On Jan20, 2014, at 08:42 , David Rowley dgrowle...@gmail.com wrote: On Mon, Jan 20, 2014 at 2:45 PM, Florian Pflug f...@phlo.org wrote: * I've also renamed

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2014-01-22 Thread Andrew Dunstan
On 01/22/2014 08:28 PM, KONDO Mitsumasa wrote: (2014/01/22 22:26), Robert Haas wrote: On Wed, Jan 22, 2014 at 3:32 AM, KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp wrote: OK, Kondo, please demonstrate benchmarks that show we have 1% impact from this change. Otherwise we may need a config

Re: [HACKERS] [bug fix] pg_ctl always uses the same event source

2014-01-22 Thread Amit Kapila
On Wed, Jan 22, 2014 at 9:03 PM, Tom Lane t...@sss.pgh.pa.us wrote: Amit Kapila amit.kapil...@gmail.com writes: On Wed, Jan 22, 2014 at 6:54 PM, Robert Haas robertmh...@gmail.com wrote: I wonder if the port number wouldn't be a better choice. And that could even be done without adding a

Re: [HACKERS] Confusing documentation of ordered-set aggregates?

2014-01-22 Thread Tom Lane
Florian Pflug f...@phlo.org writes: After reading through the relevant parts of sytnax.sgml, create_aggregate.smgl and xaggr.sgml, I think I understand how these work - they work exactly like regular aggregates, except that some arguments are evaluated only once and passed to the final

Re: [HACKERS] Confusing documentation of ordered-set aggregates?

2014-01-22 Thread Tom Lane
I wrote: Florian Pflug f...@phlo.org writes: array_agg(), for example, also has a result type that is determined by the argument type, yet it's final function doesn't take an argument of type anyelement, even though it returns anyarray. Yeah. So it's a complete leap of faith on the type

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2014-01-22 Thread KONDO Mitsumasa
(2014/01/23 12:00), Andrew Dunstan wrote: On 01/22/2014 08:28 PM, KONDO Mitsumasa wrote: (2014/01/22 22:26), Robert Haas wrote: On Wed, Jan 22, 2014 at 3:32 AM, KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp wrote: OK, Kondo, please demonstrate benchmarks that show we have 1% impact from this

Re: [HACKERS] [bug fix] pg_ctl always uses the same event source

2014-01-22 Thread Tom Lane
Amit Kapila amit.kapil...@gmail.com writes: On Wed, Jan 22, 2014 at 9:03 PM, Tom Lane t...@sss.pgh.pa.us wrote: So? Anything which can know the value of a GUC parameter can certainly know the selected port number. 1. In case of registration of event source, either user has to pass the name

Re: [HACKERS] GIN improvements part 1: additional information

2014-01-22 Thread Alexander Korotkov
On Wed, Jan 22, 2014 at 9:28 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 01/22/2014 02:17 PM, Alexander Korotkov wrote: We already spent a lot of time with compression. Now we need to figure out the result we want see. I spent quite long time debugging varbyte encoding without

Re: [HACKERS] WIP patch (v2) for updatable security barrier views

2014-01-22 Thread KaiGai Kohei
(2014/01/21 18:18), Dean Rasheed wrote: On 21 January 2014 01:09, KaiGai Kohei kai...@ak.jp.nec.com wrote: (2014/01/13 22:53), Craig Ringer wrote: On 01/09/2014 11:19 PM, Tom Lane wrote: Dean Rasheed dean.a.rash...@gmail.com writes: My first thought was that it should just preprocess any

Re: [HACKERS] [PATCH] Support for pg_stat_archiver view

2014-01-22 Thread Michael Paquier
On Thu, Jan 9, 2014 at 6:36 AM, Gabriele Bartolini gabriele.bartol...@2ndquadrant.it wrote: Il 08/01/14 18:42, Simon Riggs ha scritto: Not sure I see why it needs to be an SRF. It only returns one row. Attached is version 4, which removes management of SRF stages. I have been looking at v4 a

Re: [HACKERS] Patch: Show process IDs of processes holding a lock; show relation and tuple infos of a lock to acquire

2014-01-22 Thread Christian Kruse
Hi, On 22/01/14 14:45, Tom Lane wrote: Well, is it context or detail? Those fields have reasonably well defined meanings IMO. I find the distinction somewhat blurry and think both would be appropriate. But since I wasn't sure I changed to detail. If we need errcontext_plural, let's add it,

  1   2   >