Re: [HACKERS] Partitioning performance: cache stringToNode() of pg_constraint.ccbin

2013-06-07 Thread Amit Kapila
On Friday, June 07, 2013 2:10 AM Noah Misch wrote: On Thu, Jun 06, 2013 at 07:02:27PM +0530, Amit Kapila wrote: On Tuesday, June 04, 2013 12:37 AM Noah Misch wrote: This patch can give good performance gain in the scenario described by you. Infact I had taken the readings with patch, it

Re: [HACKERS] RFC: ExecNodeExtender

2013-06-07 Thread Kohei KaiGai
2013/6/6 Robert Haas robertmh...@gmail.com: On Tue, Jun 4, 2013 at 2:50 PM, Kohei KaiGai kai...@kaigai.gr.jp wrote: Also, I don't think ExecNodeExtender is not a good naming, because it is a bit long and abbreviation (ENE?) is hard to imagine the feature. Please give this feature a cool and

[HACKERS] system catalog pg_rewrite column ev_attr document description problem

2013-06-07 Thread Hari Babu
Hi, system catalog pg_rewrite column ev_attr document description as shown below ev_attr - The column this rule is for (currently, always zero to indicate the whole table) But In the code the column value is always set as -1. can we change the column description as below is fine?

Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)

2013-06-07 Thread Heikki Linnakangas
On 07.06.2013 00:38, Andres Freund wrote: On 2013-06-06 23:28:19 +0200, Christian Ullrich wrote: * Heikki Linnakangas wrote: The current situation is that if you run out of disk space while writing WAL, you get a PANIC, and the server shuts down. That's awful. We can So we need to somehow

Re: [HACKERS] Proposal for CSN based snapshots

2013-06-07 Thread Markus Wanner
Ants, On 06/07/2013 12:42 AM, Ants Aasma wrote: Given the recent ideas being thrown about changing how freezing and clog is handled and MVCC catalog access I thought I would write out the ideas that I have had about speeding up snapshots in case there is an interesting tie in with the current

Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)

2013-06-07 Thread Bernd Helmle
--On 6. Juni 2013 16:25:29 -0700 Josh Berkus j...@agliodbs.com wrote: Archiving - In some ways, this is the simplest case. Really, we just need a way to know when the available WAL space has become 90% full, and abort archiving at that stage. Once we stop attempting to archive, we

Re: [HACKERS] Proposal for CSN based snapshots

2013-06-07 Thread Greg Stark
On Thu, Jun 6, 2013 at 11:42 PM, Ants Aasma a...@cybertec.at wrote: To refresh your memory the basic idea is to change visibility determination to be based on a commit sequence number (CSN for short) - a 8 byte number incremented on every commit representing the total ordering of commits I

Re: [HACKERS] Proposal for CSN based snapshots

2013-06-07 Thread Ants Aasma
On Fri, Jun 7, 2013 at 2:59 PM, Markus Wanner mar...@bluegap.ch wrote: To refresh your memory the basic idea is to change visibility determination to be based on a commit sequence number (CSN for short) - a 8 byte number incremented on every commit representing the total ordering of commits.

Re: [HACKERS] Proposal for CSN based snapshots

2013-06-07 Thread Ants Aasma
On Fri, Jun 7, 2013 at 3:47 PM, Greg Stark st...@mit.edu wrote: On Thu, Jun 6, 2013 at 11:42 PM, Ants Aasma a...@cybertec.at wrote: To refresh your memory the basic idea is to change visibility determination to be based on a commit sequence number (CSN for short) - a 8 byte number incremented

Re: [HACKERS] system catalog pg_rewrite column ev_attr document description problem

2013-06-07 Thread Kevin Grittner
Hari Babu haribabu.ko...@huawei.com wrote: system catalog pg_rewrite column ev_attr document description as shown below ev_attr  - The column this rule is for (currently, always zero to indicate the whole table) But In the code the column value is always set as -1. can we change the column

Re: [HACKERS] system catalog pg_rewrite column ev_attr document description problem

2013-06-07 Thread Tom Lane
Kevin Grittner kgri...@ymail.com writes: Hari Babu haribabu.ko...@huawei.com wrote: system catalog pg_rewrite column ev_attr document description as shown below ev_attr  - The column this rule is for (currently, always zero to indicate the whole table) But In the code the column value is

Re: [HACKERS] extensible external toast tuple support snappy prototype

2013-06-07 Thread Robert Haas
On Wed, Jun 5, 2013 at 11:01 AM, Andres Freund and...@2ndquadrant.com wrote: On 2013-05-31 23:42:51 -0400, Robert Haas wrote: This should allow for fairly easy development of a new compression scheme for out-of-line toast tuples. It will *not* work for compressed inline tuples (i.e.

Re: [HACKERS] Cost limited statements RFC

2013-06-07 Thread Robert Haas
On Thu, Jun 6, 2013 at 7:36 PM, Greg Smith g...@2ndquadrant.com wrote: I have also subjected some busy sites to a field test here since the original discussion, to try and nail down if this is really necessary. So far I haven't gotten any objections, and I've seen one serious improvement,

Re: [HACKERS] Regarding GIN Fast Update Technique

2013-06-07 Thread Robert Haas
On Wed, Jun 5, 2013 at 10:06 PM, Amit Langote amitlangot...@gmail.com wrote: At what point do the entries in the pending list are moved to the main GIN data structure? From documentation, I read that overflowing work_mem and vacuum are two such causes; what about when the concerned backend is

Re: [HACKERS] Regarding GIN Fast Update Technique

2013-06-07 Thread Amit Langote
On Fri, Jun 7, 2013 at 11:15 PM, Robert Haas robertmh...@gmail.com wrote: On Wed, Jun 5, 2013 at 10:06 PM, Amit Langote amitlangot...@gmail.com wrote: At what point do the entries in the pending list are moved to the main GIN data structure? From documentation, I read that overflowing work_mem

Re: [HACKERS] extensible external toast tuple support snappy prototype

2013-06-07 Thread Andres Freund
On 2013-06-07 10:04:15 -0400, Robert Haas wrote: On Wed, Jun 5, 2013 at 11:01 AM, Andres Freund and...@2ndquadrant.com wrote: On 2013-05-31 23:42:51 -0400, Robert Haas wrote: This should allow for fairly easy development of a new compression scheme for out-of-line toast tuples. It will

Re: [HACKERS] Regarding GIN Fast Update Technique

2013-06-07 Thread Andres Freund
On 2013-06-07 23:28:56 +0900, Amit Langote wrote: On Fri, Jun 7, 2013 at 11:15 PM, Robert Haas robertmh...@gmail.com wrote: On Wed, Jun 5, 2013 at 10:06 PM, Amit Langote amitlangot...@gmail.com wrote: At what point do the entries in the pending list are moved to the main GIN data

Re: [HACKERS] system catalog pg_rewrite column ev_attr document description problem

2013-06-07 Thread Kevin Grittner
Tom Lane t...@sss.pgh.pa.us wrote: Actually, I think this is a bug and the right thing is to make the code match the documentation not vice versa.  ev_attr isn't being used for much at the moment, but if it were being used as an AttrNumber, -1 would not mean whole row.  It would be a

Re: [HACKERS] Regarding GIN Fast Update Technique

2013-06-07 Thread Tom Lane
Amit Langote amitlangot...@gmail.com writes: Okay, aside from that case, what else would move those to the main structure? They (the entries in the unsorted pending list) are in the local memory (work_mem?) of the backend, right? No. If they were, it wouldn't be crash-safe.

Re: [HACKERS] extensible external toast tuple support snappy prototype

2013-06-07 Thread Robert Haas
On Fri, Jun 7, 2013 at 10:30 AM, Andres Freund and...@2ndquadrant.com wrote: Turns out the benefits are imo big enough to make it worth pursuing further. Yeah, those were nifty numbers. The problem is that to discern from pglz on little endian the byte with the two high bits unset is

Re: [HACKERS] create a git symbolic-ref for REL9_3_STABLE

2013-06-07 Thread Robert Haas
On Mon, Jun 3, 2013 at 9:30 PM, Peter Eisentraut pete...@gmx.net wrote: I suppose we'll be branching off 9.3 in a few weeks. That event always creates a service gap in the build farm and similar services, and a race in the NLS service to get everything adjusted to the new branch. It seems to

Re: [HACKERS] MVCC catalog access

2013-06-07 Thread Robert Haas
On Thu, Jun 6, 2013 at 2:48 PM, Jim Nasby j...@nasby.net wrote: On 6/5/13 3:49 PM, Robert Haas wrote: Now, I did find a couple that I thought should probably stick with SnapshotNow, specifically pgrowlocks and pgstattuple. FWIW, I've often wished for a way to make all stat access

Re: [HACKERS] Regarding GIN Fast Update Technique

2013-06-07 Thread Amit Langote
On Fri, Jun 7, 2013 at 11:36 PM, Andres Freund and...@2ndquadrant.com wrote: On 2013-06-07 23:28:56 +0900, Amit Langote wrote: On Fri, Jun 7, 2013 at 11:15 PM, Robert Haas robertmh...@gmail.com wrote: On Wed, Jun 5, 2013 at 10:06 PM, Amit Langote amitlangot...@gmail.com wrote: At what

Re: [HACKERS] extensible external toast tuple support snappy prototype

2013-06-07 Thread Andres Freund
On 2013-06-07 10:44:24 -0400, Robert Haas wrote: On Fri, Jun 7, 2013 at 10:30 AM, Andres Freund and...@2ndquadrant.com wrote: Turns out the benefits are imo big enough to make it worth pursuing further. Yeah, those were nifty numbers. The problem is that to discern from pglz on little

Re: [HACKERS] system catalog pg_rewrite column ev_attr document description problem

2013-06-07 Thread Tom Lane
Kevin Grittner kgri...@ymail.com writes: Tom Lane t...@sss.pgh.pa.us wrote: Actually, I think this is a bug and the right thing is to make the code match the documentation not vice versa. I assume that this should be a 9.3 code fix, and a doc fix prior to that, since it would require

Re: [HACKERS] pg_ugprade use of --check and --link

2013-06-07 Thread Bruce Momjian
On Thu, Jun 6, 2013 at 10:24:30AM -0400, Alvaro Herrera wrote: Bruce Momjian wrote: In a private bug report, I have realized that if you are eventually going to be using link mode with pg_upgrade, and you run --check mode, you should use --link with --check to check that both clusters are

Re: [HACKERS] Proposal for CSN based snapshots

2013-06-07 Thread Kevin Grittner
Ants Aasma a...@cybertec.at wrote: Serializable transactions - I won't pretend to be familiar with SSI code, but as far as I can tell serializable transactions don't need any modifications to work with the CSN based snapshot scheme. There actually already is a

Re: [HACKERS] extensible external toast tuple support snappy prototype

2013-06-07 Thread Hannu Krosing
On 06/07/2013 04:54 PM, Andres Freund wrote: I mean, we don't necessarily need to make it configurable if we just add one canonical new better compression format. I am not sure that's sufficient since I can see usecases for 'very fast but not too well compressed' and 'very well compressed but

Re: [HACKERS] Cost limited statements RFC

2013-06-07 Thread Greg Smith
On 6/7/13 10:14 AM, Robert Haas wrote: If the page hit limit goes away, the user with a single core server who used to having autovacuum only pillage shared_buffers at 78MB/s might complain that if it became unbounded. Except that it shouldn't become unbounded, because of the ring-buffer

Re: [HACKERS] extensible external toast tuple support snappy prototype

2013-06-07 Thread Andres Freund
On 2013-06-07 17:27:28 +0200, Hannu Krosing wrote: On 06/07/2013 04:54 PM, Andres Freund wrote: I mean, we don't necessarily need to make it configurable if we just add one canonical new better compression format. I am not sure that's sufficient since I can see usecases for 'very fast but

Re: [HACKERS] extensible external toast tuple support snappy prototype

2013-06-07 Thread Tom Lane
Andres Freund and...@2ndquadrant.com writes: I mean, we don't necessarily need to make it configurable if we just add one canonical new better compression format. I am not sure that's sufficient since I can see usecases for 'very fast but not too well compressed' and 'very well compressed but

Re: [HACKERS] system catalog pg_rewrite column ev_attr document description problem

2013-06-07 Thread Kevin Grittner
Tom Lane t...@sss.pgh.pa.us wrote: Kevin Grittner kgri...@ymail.com writes: Tom Lane t...@sss.pgh.pa.us wrote: Actually, I think this is a bug and the right thing is to make the code match the documentation not vice versa. I assume that this should be a 9.3 code fix, and a doc fix prior to

Re: [HACKERS] extensible external toast tuple support snappy prototype

2013-06-07 Thread Andres Freund
On 2013-06-07 11:46:45 -0400, Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: I mean, we don't necessarily need to make it configurable if we just add one canonical new better compression format. I am not sure that's sufficient since I can see usecases for 'very fast but not

Re: [HACKERS] extensible external toast tuple support snappy prototype

2013-06-07 Thread Alvaro Herrera
Andres Freund escribió: 2) Combined patch that adds indirect tuple and snappy compression. Those coul be separated, but this is an experiment so far... Can we have a separate header for toast definitions? (i.e. split them out of postgres.h) -- Álvaro Herrera

Re: [HACKERS] extensible external toast tuple support snappy prototype

2013-06-07 Thread Tom Lane
Andres Freund and...@2ndquadrant.com writes: On 2013-06-07 11:46:45 -0400, Tom Lane wrote: IME, once we've changed it once, the odds that we'll want to change it again go up drastically. I think if we're going to make a change here we should leave room for future revisions. The above bit

Re: [HACKERS] extensible external toast tuple support snappy prototype

2013-06-07 Thread Hannu Krosing
On 06/07/2013 05:38 PM, Andres Freund wrote: On 2013-06-07 17:27:28 +0200, Hannu Krosing wrote: On 06/07/2013 04:54 PM, Andres Freund wrote: I mean, we don't necessarily need to make it configurable if we just add one canonical new better compression format. I am not sure that's sufficient

Re: [HACKERS] extensible external toast tuple support snappy prototype

2013-06-07 Thread Stephen Frost
* Andres Freund (and...@2ndquadrant.com) wrote: Currently on a little endian system the pglz header contains the length in the first four bytes as: [][][][xxdd] Where dd are valid length bits for pglz and xx are the two bits which are always zero since we only ever

Re: [HACKERS] create a git symbolic-ref for REL9_3_STABLE

2013-06-07 Thread Stephen Frost
* Robert Haas (robertmh...@gmail.com) wrote: I think we should just go ahead and really create the branch. The CF is supposed to start in a week. +1 Thanks, Stephen signature.asc Description: Digital signature

Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)

2013-06-07 Thread Heikki Linnakangas
On 06.06.2013 17:00, Heikki Linnakangas wrote: A more workable idea is to sprinkle checks in higher-level code, before you hold any critical locks, to check that there is enough preallocated WAL. Like, at the beginning of heap_insert, heap_update, etc., and all similar indexam entry points.

Re: [HACKERS] ALTER DEFAULT PRIVILEGES FOR ROLE is broken

2013-06-07 Thread Tom Lane
I wrote: Noah Misch n...@leadboat.com writes: The particular restriction at hand, namely that a role have CREATE rights on a schema before assigning role-specific default privileges, seems like needless paternalism. It would be akin to forbidding ALTER ROLE ... PASSWORD on a NOLOGIN role.

Re: [HACKERS] extensible external toast tuple support snappy prototype

2013-06-07 Thread Andres Freund
On 2013-06-07 12:16:48 -0400, Stephen Frost wrote: * Andres Freund (and...@2ndquadrant.com) wrote: Currently on a little endian system the pglz header contains the length in the first four bytes as: [][][][xxdd] Where dd are valid length bits for pglz and xx

Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)

2013-06-07 Thread Tom Lane
Heikki Linnakangas hlinnakan...@vmware.com writes: On 06.06.2013 17:00, Heikki Linnakangas wrote: A more workable idea is to sprinkle checks in higher-level code, before you hold any critical locks, to check that there is enough preallocated WAL. Like, at the beginning of heap_insert,

Re: [HACKERS] Cost limited statements RFC

2013-06-07 Thread Robert Haas
On Fri, Jun 7, 2013 at 11:35 AM, Greg Smith g...@2ndquadrant.com wrote: I wasn't talking about disruption of the data that's in the buffer cache. The only time the scenario I was describing plays out is when the data is already in shared_buffers. The concern is damage done to the CPU's data

Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)

2013-06-07 Thread Heikki Linnakangas
On 07.06.2013 19:33, Tom Lane wrote: Heikki Linnakangashlinnakan...@vmware.com writes: On 06.06.2013 17:00, Heikki Linnakangas wrote: A more workable idea is to sprinkle checks in higher-level code, before you hold any critical locks, to check that there is enough preallocated WAL. Like, at

Re: [HACKERS] ALTER DEFAULT PRIVILEGES FOR ROLE is broken

2013-06-07 Thread Stephen Frost
* Tom Lane (t...@sss.pgh.pa.us) wrote: Essentially the argument for allowing this without a permissions check is I'm not really doing anything to the schema, just preconfiguring the rights that will be attached to a new object if I later (successfully) create one in this schema. Makes sense

Re: [HACKERS] Cost limited statements RFC

2013-06-07 Thread Greg Smith
On 6/7/13 12:42 PM, Robert Haas wrote: GUCs in terms of units that are meaningful to the user. One could have something like io_rate_limit (measured in MB/s), io_read_multiplier = 1.0, io_dirty_multiplier = 1.0, and I think that would be reasonably clear. There's one other way to frame this:

Re: [HACKERS] ALTER DEFAULT PRIVILEGES FOR ROLE is broken

2013-06-07 Thread Tom Lane
Stephen Frost sfr...@snowman.net writes: * Tom Lane (t...@sss.pgh.pa.us) wrote: Essentially the argument for allowing this without a permissions check is I'm not really doing anything to the schema, just preconfiguring the rights that will be attached to a new object if I later (successfully)

Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)

2013-06-07 Thread Tom Lane
Heikki Linnakangas hlinnakan...@vmware.com writes: On 07.06.2013 19:33, Tom Lane wrote: Not only is that a horrible layering/modularity violation, but surely LockBuffer can have no idea how much WAL space will be needed. It can be just a conservative guess, like, 32KB. That should be enough

Re: [HACKERS] ALTER DEFAULT PRIVILEGES FOR ROLE is broken

2013-06-07 Thread Stephen Frost
* Tom Lane (t...@sss.pgh.pa.us) wrote: Hm. Throwing a NOTICE saying btw, this won't be of any value until the user has CREATE rights in that schema might be a reasonable compromise. Yeah, a NOTICE makes more sense than a WARNING, so +1 from me. Thanks, Stephen

Re: [HACKERS] ALTER DEFAULT PRIVILEGES FOR ROLE is broken

2013-06-07 Thread Robert Haas
On Fri, Jun 7, 2013 at 12:44 PM, Stephen Frost sfr...@snowman.net wrote: * Tom Lane (t...@sss.pgh.pa.us) wrote: Essentially the argument for allowing this without a permissions check is I'm not really doing anything to the schema, just preconfiguring the rights that will be attached to a new

Re: [HACKERS] SPGist triple parity concept doesn't work

2013-06-07 Thread Alvaro Herrera
Tom Lane wrote: I don't immediately see any good way to fix this. I think the triple parity rule as it stands is hopelessly broken, but I don't know what to replace it with, even granting that we don't need to maintain on-disk compatibility. (We'd have to tell people to reindex SPGist

[HACKERS] Parallell Optimizer

2013-06-07 Thread FredDaniPandoraAquiles
I asked a while ago in this group about the possibility to implement a parallel planner in a multithread way, and the replies were that the proposed approach couldn't be implemented, because the postgres is not thread-safe. With the new feature Background Worker Processes, such implementation

Re: [HACKERS] ALTER DEFAULT PRIVILEGES FOR ROLE is broken

2013-06-07 Thread Noah Misch
On Fri, Jun 07, 2013 at 12:26:59PM -0400, Tom Lane wrote: I wrote: Noah Misch n...@leadboat.com writes: The particular restriction at hand, namely that a role have CREATE rights on a schema before assigning role-specific default privileges, seems like needless paternalism. It would

Re: [HACKERS] Regarding GIN Fast Update Technique

2013-06-07 Thread Jesper Krogh
On 07/06/13 16:39, Tom Lane wrote: Amit Langote amitlangot...@gmail.com writes: Okay, aside from that case, what else would move those to the main structure? They (the entries in the unsorted pending list) are in the local memory (work_mem?) of the backend, right? No. If they were, it

Re: [HACKERS] background processes vs. hot standby

2013-06-07 Thread Robert Haas
On Fri, May 24, 2013 at 12:35 PM, Robert Haas robertmh...@gmail.com wrote: But I don't know what to do about the problem of needing to know how many backends there are. I agree with Andres that it's not very friendly to enforce a restriction that all the same worker processes must be present

Re: [HACKERS] Cost limited statements RFC

2013-06-07 Thread Robert Haas
On Fri, Jun 7, 2013 at 12:55 PM, Greg Smith g...@2ndquadrant.com wrote: On 6/7/13 12:42 PM, Robert Haas wrote: GUCs in terms of units that are meaningful to the user. One could have something like io_rate_limit (measured in MB/s), io_read_multiplier = 1.0, io_dirty_multiplier = 1.0, and I

Re: [HACKERS] Freezing without write I/O

2013-06-07 Thread Robert Haas
On Thu, Jun 6, 2013 at 6:28 PM, Greg Stark st...@mit.edu wrote: On Thu, Jun 6, 2013 at 1:39 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: That will keep OldestXmin from advancing. Which will keep vacuum from advancing relfrozenxid/datfrozenxid. Which will first trigger the warnings

Re: [HACKERS] SPGist triple parity concept doesn't work

2013-06-07 Thread Tom Lane
Alvaro Herrera alvhe...@2ndquadrant.com writes: Is this the chance to add a metapage? SPGist indexes do have a metapage already --- you're confusing them with Gist. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes

Re: [HACKERS] Freezing without write I/O

2013-06-07 Thread Heikki Linnakangas
On 07.06.2013 20:54, Robert Haas wrote: On Thu, Jun 6, 2013 at 6:28 PM, Greg Starkst...@mit.edu wrote: On Thu, Jun 6, 2013 at 1:39 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: That will keep OldestXmin from advancing. Which will keep vacuum from advancing relfrozenxid/datfrozenxid.

Re: [HACKERS] Parallell Optimizer

2013-06-07 Thread Tom Lane
FredDaniPandoraAquiles f...@nti.ufop.br writes: I asked a while ago in this group about the possibility to implement a parallel planner in a multithread way, and the replies were that the proposed approach couldn't be implemented, because the postgres is not thread-safe. With the new feature

Re: [HACKERS] Freezing without write I/O

2013-06-07 Thread Simon Riggs
On 7 June 2013 19:08, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 07.06.2013 20:54, Robert Haas wrote: On Thu, Jun 6, 2013 at 6:28 PM, Greg Starkst...@mit.edu wrote: On Thu, Jun 6, 2013 at 1:39 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: That will keep OldestXmin from

Re: [HACKERS] Redesigning checkpoint_segments

2013-06-07 Thread Robert Haas
On Thu, Jun 6, 2013 at 10:43 PM, Greg Smith g...@2ndquadrant.com wrote: The general complaint the last time I suggested a change in this area, to make checkpoint_segments larger for the average user, was that some people had seen workloads where that was counterproductive. Pretty sure Kevin

[HACKERS] Bad error message on valuntil

2013-06-07 Thread Joshua D. Drake
Hello, I had a customer pulling their hair out today because they couldn't login to their system. The error was consistently: 2013-06-07 08:42:44 MST postgres 10.1.11.67 27440 FATAL: password authentication failed for user user However the problem had nothing to do with password

Re: [HACKERS] ALTER DEFAULT PRIVILEGES FOR ROLE is broken

2013-06-07 Thread Tom Lane
Noah Misch n...@leadboat.com writes: On Fri, Jun 07, 2013 at 12:26:59PM -0400, Tom Lane wrote: Essentially the argument for allowing this without a permissions check is I'm not really doing anything to the schema, just preconfiguring the rights that will be attached to a new object if I later

Re: [HACKERS] Freezing without write I/O

2013-06-07 Thread Heikki Linnakangas
On 07.06.2013 21:33, Simon Riggs wrote: Now that I consider Greg's line of thought, the idea we focused on here was about avoiding freezing. But Greg makes me think that we may also wish to look at allowing queries to run longer than one epoch as well, if the epoch wrap time is likely to come

Re: [HACKERS] Bad error message on valuntil

2013-06-07 Thread Tom Lane
Joshua D. Drake j...@commandprompt.com writes: I had a customer pulling their hair out today because they couldn't login to their system. The error was consistently: 2013-06-07 08:42:44 MST postgres 10.1.11.67 27440 FATAL: password authentication failed for user user However the problem

Re: [HACKERS] Parallell Optimizer

2013-06-07 Thread Robert Haas
On Fri, Jun 7, 2013 at 2:27 PM, Tom Lane t...@sss.pgh.pa.us wrote: FredDaniPandoraAquiles f...@nti.ufop.br writes: I asked a while ago in this group about the possibility to implement a parallel planner in a multithread way, and the replies were that the proposed approach couldn't be

Re: [HACKERS] Freezing without write I/O

2013-06-07 Thread Simon Riggs
On 7 June 2013 19:56, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 07.06.2013 21:33, Simon Riggs wrote: Now that I consider Greg's line of thought, the idea we focused on here was about avoiding freezing. But Greg makes me think that we may also wish to look at allowing queries to run

Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)

2013-06-07 Thread Josh Berkus
I would oppose that as the solution, either an unconditional one, or configurable with is it as the default. Those segments are not unneeded. I need them. That is why I set up archiving in the first place. If you need to shut down the database rather than violate my established retention

Re: [HACKERS] Redesigning checkpoint_segments

2013-06-07 Thread Kevin Grittner
Robert Haas robertmh...@gmail.com wrote: (As to why smaller checkpoint_segments can help, here's my guess: if checkpoint_segments is relatively small, then when we recycle a segment we're likely to find its data already in cache.  That's a lot better than reading it back in from disk just to

Re: [HACKERS] Freezing without write I/O

2013-06-07 Thread Robert Haas
On Fri, Jun 7, 2013 at 3:10 PM, Simon Riggs si...@2ndquadrant.com wrote: The long running query problem hasn't ever been looked at, it seems, until here and now. For what it's worth (and that may not be much), I think most people will die a horrible death due to bloat after holding a

Re: [HACKERS] Freezing without write I/O

2013-06-07 Thread Andres Freund
On 2013-06-07 20:10:55 +0100, Simon Riggs wrote: On 7 June 2013 19:56, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 07.06.2013 21:33, Simon Riggs wrote: Now that I consider Greg's line of thought, the idea we focused on here was about avoiding freezing. But Greg makes me think

Re: [HACKERS] Redesigning checkpoint_segments

2013-06-07 Thread Robert Haas
On Fri, Jun 7, 2013 at 3:14 PM, Kevin Grittner kgri...@ymail.com wrote: Some findings were unsurprising, like that a direct connection between the servers using a cross-wired network patch cable was faster than plugging both machines into the same switch. But we tested all of our assumptions,

Re: [HACKERS] Bad error message on valuntil

2013-06-07 Thread Joshua D. Drake
On 06/07/2013 11:57 AM, Tom Lane wrote: Joshua D. Drake j...@commandprompt.com writes: I had a customer pulling their hair out today because they couldn't login to their system. The error was consistently: 2013-06-07 08:42:44 MST postgres 10.1.11.67 27440 FATAL: password authentication

Re: [HACKERS] Parallell Optimizer

2013-06-07 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Fri, Jun 7, 2013 at 2:27 PM, Tom Lane t...@sss.pgh.pa.us wrote: I don't think that bgworkers as currently implemented make this any more practical than it was before. The communication overhead with a separate process would swamp any benefit in

Re: [HACKERS] Bad error message on valuntil

2013-06-07 Thread David Johnston
Tom Lane-2 wrote Joshua D. Drake lt; jd@ gt; writes: I had a customer pulling their hair out today because they couldn't login to their system. The error was consistently: 2013-06-07 08:42:44 MST postgres 10.1.11.67 27440 FATAL: password authentication failed for user user However

Re: [HACKERS] Freezing without write I/O

2013-06-07 Thread Simon Riggs
On 7 June 2013 20:16, Andres Freund and...@2ndquadrant.com wrote: On 2013-06-07 20:10:55 +0100, Simon Riggs wrote: On 7 June 2013 19:56, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 07.06.2013 21:33, Simon Riggs wrote: Now that I consider Greg's line of thought, the idea we focused

Re: [HACKERS] Bad error message on valuntil

2013-06-07 Thread Tom Lane
Joshua D. Drake j...@commandprompt.com writes: On 06/07/2013 11:57 AM, Tom Lane wrote: I think it's intentional that we don't tell the *client* that level of detail. Why? That seems rather silly. The general policy on authentication failure reports is that we don't tell the client anything

Avoiding bloat in the presence of a long-running transaction (Re: [HACKERS] Freezing without write I/O)

2013-06-07 Thread Heikki Linnakangas
On 07.06.2013 22:15, Robert Haas wrote: On Fri, Jun 7, 2013 at 3:10 PM, Simon Riggssi...@2ndquadrant.com wrote: The long running query problem hasn't ever been looked at, it seems, until here and now. For what it's worth (and that may not be much), I think most people will die a horrible

Re: [HACKERS] Redesigning checkpoint_segments

2013-06-07 Thread Kevin Grittner
Robert Haas robertmh...@gmail.com wrote: Kevin Grittner kgri...@ymail.com wrote: One such surprise was that the conversion ran faster, even on a largish database of around 200GB, with 3 checkpoint_segments than with larger settings. ! I can't account for that finding, because my

Re: [HACKERS] Bad error message on valuntil

2013-06-07 Thread Tom Lane
David Johnston pol...@yahoo.com writes: I presume that password in this context refers to the method by which identity is checked; some alternatives being trust and ident? Right. Using the same logic of why you would not expose the fact that the user is expired versus the user has provided

Re: [HACKERS] Parallell Optimizer

2013-06-07 Thread Robert Haas
On Fri, Jun 7, 2013 at 3:23 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Fri, Jun 7, 2013 at 2:27 PM, Tom Lane t...@sss.pgh.pa.us wrote: I don't think that bgworkers as currently implemented make this any more practical than it was before. The

Re: [HACKERS] Parallell Optimizer

2013-06-07 Thread Simon Riggs
On 7 June 2013 20:23, Tom Lane t...@sss.pgh.pa.us wrote: As for other databases, I suspect that ones that have parallel execution are probably doing it with a thread model not a process model. Separate processes are more common because it covers the general case where query execution is spread

Re: [HACKERS] Bad error message on valuntil

2013-06-07 Thread Joshua D. Drake
On 06/07/2013 12:31 PM, Tom Lane wrote: Joshua D. Drake j...@commandprompt.com writes: On 06/07/2013 11:57 AM, Tom Lane wrote: I think it's intentional that we don't tell the *client* that level of detail. Why? That seems rather silly. The general policy on authentication failure reports

Re: [HACKERS] Bad error message on valuntil

2013-06-07 Thread Rodrigo Gonzalez
On Fri, 07 Jun 2013 13:07:21 -0700 Joshua D. Drake j...@commandprompt.com wrote: On 06/07/2013 12:31 PM, Tom Lane wrote: Joshua D. Drake j...@commandprompt.com writes: On 06/07/2013 11:57 AM, Tom Lane wrote: I think it's intentional that we don't tell the *client* that level of detail.

Re: [HACKERS] Freezing without write I/O

2013-06-07 Thread Hannu Krosing
On 06/07/2013 08:56 PM, Heikki Linnakangas wrote: On 07.06.2013 21:33, Simon Riggs wrote: Now that I consider Greg's line of thought, the idea we focused on here was about avoiding freezing. But Greg makes me think that we may also wish to look at allowing queries to run longer than one epoch

Re: [HACKERS] About large objects asynchronous and non-blocking support

2013-06-07 Thread Giovanni Mascellani
Hi. Il 05/06/2013 22:52, Dmitriy Igrishin ha scritto: At the moment libpq doesn't seem to support asynchronous and non-blocking support for large objects, in the style of PQsendQuery/PQgetResult. This makes large objects hardly suited for single-threaded programs based on some variant of

Re: [HACKERS] Freezing without write I/O

2013-06-07 Thread Hannu Krosing
On 06/07/2013 09:16 PM, Andres Freund wrote: On 2013-06-07 20:10:55 +0100, Simon Riggs wrote: On 7 June 2013 19:56, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 07.06.2013 21:33, Simon Riggs wrote: Now that I consider Greg's line of thought, the idea we focused on here was about

Re: [HACKERS] Bad error message on valuntil

2013-06-07 Thread David Johnston
Joshua D. Drake wrote On 06/07/2013 12:31 PM, Tom Lane wrote: Joshua D. Drake lt; jd@ gt; writes: On 06/07/2013 11:57 AM, Tom Lane wrote: I think it's intentional that we don't tell the *client* that level of detail. Why? That seems rather silly. The general policy on authentication

Re: [HACKERS] About large objects asynchronous and non-blocking support

2013-06-07 Thread Dmitriy Igrishin
2013/6/8 Giovanni Mascellani g.mascell...@gmail.com Hi. Il 05/06/2013 22:52, Dmitriy Igrishin ha scritto: At the moment libpq doesn't seem to support asynchronous and non-blocking support for large objects, in the style of PQsendQuery/PQgetResult. This makes large objects hardly suited

Re: [HACKERS] Bad error message on valuntil

2013-06-07 Thread Joshua D. Drake
On 06/07/2013 01:41 PM, David Johnston wrote: Please check server log for specifics is not a good message for something sent to a client that in many normal situation would have no access to said logs. I don't agree. The user doesn't need access to the logs. If they get that error they

Re: [HACKERS] ALTER DEFAULT PRIVILEGES FOR ROLE is broken

2013-06-07 Thread Noah Misch
On Fri, Jun 07, 2013 at 02:49:37PM -0400, Tom Lane wrote: Noah Misch n...@leadboat.com writes: On Fri, Jun 07, 2013 at 12:26:59PM -0400, Tom Lane wrote: Essentially the argument for allowing this without a permissions check is I'm not really doing anything to the schema, just preconfiguring

Re: [HACKERS] About large objects asynchronous and non-blocking support

2013-06-07 Thread Tatsuo Ishii
2013/6/6 Tatsuo Ishii is...@postgresql.org Hi. At the moment libpq doesn't seem to support asynchronous and non-blocking support for large objects, in the style of PQsendQuery/PQgetResult. This makes large objects hardly suited for single-threaded programs based on some variant of

[HACKERS] Last log line for log_temp_files is disassociated with query

2013-06-07 Thread Josh Berkus
Folks, I've been trying to write some tools which (for example) give how much total temp file space was used by specific individual slow queries. And this turns out to be inordinately hard to do because of how log_temp_files works. In many cases, this is what I see (9.2.4):

Re: [HACKERS] Last log line for log_temp_files is disassociated with query

2013-06-07 Thread Josh Berkus
On 06/07/2013 04:52 PM, Josh Berkus wrote: In many cases, this is what I see (9.2.4): session_line_num|message|query 423|temporary file: path ...procid.0 file size: 525122|SELECT ... 424|temporary file: path ...procid.1 file size: 622044|SELECT ... 425|duration: 17078.635 ms execute

Re: [HACKERS] Partitioning performance: cache stringToNode() of pg_constraint.ccbin

2013-06-07 Thread Noah Misch
On Fri, Jun 07, 2013 at 11:58:14AM +0530, Amit Kapila wrote: So the memory increase number's would like: Example for 64-bit m/c In database, there are 100 tables, each having 2 constraints and 30 live connections Size increase = no. of tables * size of (Node*) * number of constraints *

Re: [HACKERS] Last log line for log_temp_files is disassociated with query

2013-06-07 Thread Tom Lane
Josh Berkus j...@agliodbs.com writes: ... that is, for some reason, the last log line reported by log_temp_files becomes disassociated with the query which was using temp space in the first place. Often, that last file is the biggest one, as well. Is this a fixable bug? Without a concrete

Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)

2013-06-07 Thread Daniel Farina
On Fri, Jun 7, 2013 at 12:14 PM, Josh Berkus j...@agliodbs.com wrote: Right now, what we're telling users is You can have continuous backup with Postgres, but you'd better hire and expensive consultant to set it up for you, or use this external tool of dubious provenance which there's no