Re: [HACKERS] [rfc] overhauling pgstat.stat

2013-09-04 Thread Atri Sharma
Sent from my iPad On 04-Sep-2013, at 10:54, Satoshi Nagayasu sn...@uptime.jp wrote: Hi, (2013/09/04 12:52), Atri Sharma wrote: On Wed, Sep 4, 2013 at 6:40 AM, Satoshi Nagayasu sn...@uptime.jp wrote: Hi, I'm considering overhauling pgstat.stat, and would like to know how many people

Re: [HACKERS] [9.3 doc fix] clarification of Solaris versions

2013-09-04 Thread MauMau
From: Robert Haas robertmh...@gmail.com OK, patch committed and back-patched to 9.3. The patch file turned out to be sorta garbled. I'm not sure if a broken version of diff was used to generate this or whether MauMau hand-edited it after the fact, but the number of lines that were indicated in

Re: [HACKERS] [rfc] overhauling pgstat.stat

2013-09-04 Thread Satoshi Nagayasu
(2013/09/04 15:23), Atri Sharma wrote: Sent from my iPad On 04-Sep-2013, at 10:54, Satoshi Nagayasu sn...@uptime.jp wrote: Hi, (2013/09/04 12:52), Atri Sharma wrote: On Wed, Sep 4, 2013 at 6:40 AM, Satoshi Nagayasu sn...@uptime.jp wrote: Hi, I'm considering overhauling pgstat.stat, and

[HACKERS] Re: Is it necessary to rewrite table while increasing the scale of datatype numeric???

2013-09-04 Thread Noah Misch
On Wed, Sep 04, 2013 at 12:08:48PM +0800, wangs...@highgo.com.cn wrote: I find that it takes a long time when I increase the scale of a numeric datatype. By checking the code, I found that's because it needs to rewrite that table's file. After checking that table's data file, I found only

Re: [HACKERS] [rfc] overhauling pgstat.stat

2013-09-04 Thread Stephen Frost
Satoshi, * Satoshi Nagayasu (sn...@uptime.jp) wrote: (2013/09/04 13:07), Alvaro Herrera wrote: We already changed it: commit 187492b6c2e8cafc5b39063ca3b67846e8155d24 Author: Alvaro Herrera alvhe...@alvh.no-ip.org Date: Mon Feb 18 17:56:08 2013 -0300 Split pgstat file in

Re: [HACKERS] [rfc] overhauling pgstat.stat

2013-09-04 Thread Atri Sharma
Sent from my iPad On 04-Sep-2013, at 15:39, Satoshi Nagayasu sn...@uptime.jp wrote: (2013/09/04 15:23), Atri Sharma wrote: Sent from my iPad On 04-Sep-2013, at 10:54, Satoshi Nagayasu sn...@uptime.jp wrote: Hi, (2013/09/04 12:52), Atri Sharma wrote: On Wed, Sep 4, 2013 at 6:40

Re: [HACKERS] [rfc] overhauling pgstat.stat

2013-09-04 Thread Pavel Stehule
2013/9/4 Atri Sharma atri.j...@gmail.com Sent from my iPad On 04-Sep-2013, at 15:39, Satoshi Nagayasu sn...@uptime.jp wrote: (2013/09/04 15:23), Atri Sharma wrote: Sent from my iPad On 04-Sep-2013, at 10:54, Satoshi Nagayasu sn...@uptime.jp wrote: Hi, (2013/09/04 12:52),

Re: [HACKERS] [rfc] overhauling pgstat.stat

2013-09-04 Thread Pavel Stehule
2013/9/4 Atri Sharma atri.j...@gmail.com Sent from my iPad On 04-Sep-2013, at 15:39, Satoshi Nagayasu sn...@uptime.jp wrote: (2013/09/04 15:23), Atri Sharma wrote: Sent from my iPad On 04-Sep-2013, at 10:54, Satoshi Nagayasu sn...@uptime.jp wrote: Hi, (2013/09/04 12:52),

Re: [HACKERS] logical changeset generation v5

2013-09-04 Thread Andres Freund
On 2013-09-04 10:02:05 -0400, Robert Haas wrote: On Tue, Sep 3, 2013 at 7:10 PM, Andres Freund and...@2ndquadrant.com wrote: I don't think it particularly needs to be configurable, but I wonder if we can't be a bit smarter about when we do it. For example, suppose we logged it every 15 s

Re: [HACKERS] The PostgreSQL License requires LICENSE file?

2013-09-04 Thread Josh Berkus
On 09/03/2013 11:20 PM, Tomonari Katsumata wrote: I understands that the PostgreSQL license does not require any specified file name and missing LICENSE is not problem. I'm Sorry for the stupid question. No, thank *you* for pointing it out! None of us had actually looked at that page. --

Re: [HACKERS] [9.4] Make full_page_writes only settable on server start?

2013-09-04 Thread Jeff Davis
On Tue, 2013-09-03 at 15:42 -0400, Robert Haas wrote: Although this is true, the administrator's estimate of whether that guarantee is or is not provided might not be as consistent as the hardware behavior itself. I am generally of the feeling that having to restart the server to change

Re: [HACKERS] [tiny doc fix] statistics are not retained across immediate shutdown

2013-09-04 Thread Andres Freund
On 2013-09-04 10:53:19 -0400, Tom Lane wrote: Tsunakawa, Takayuki tsunakawa.ta...@jp.fujitsu.com writes: I propose a tiny fix to clarify this. Please find the attached patch. That's not an accurate description of what happens, though. AFAIR, we do not throw away pg_stats files as a result

Re: [HACKERS] logical changeset generation v5

2013-09-04 Thread Robert Haas
On Tue, Sep 3, 2013 at 7:10 PM, Andres Freund and...@2ndquadrant.com wrote: I don't think it particularly needs to be configurable, but I wonder if we can't be a bit smarter about when we do it. For example, suppose we logged it every 15 s but only until we log a non-overflowed snapshot.

Re: [HACKERS] [v9.4] row level security

2013-09-04 Thread Robert Haas
On Wed, Sep 4, 2013 at 10:50 AM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Wed, Sep 4, 2013 at 10:45 AM, Tom Lane t...@sss.pgh.pa.us wrote: Well, the security-barrier view stuff did not present itself as a 100% solution. But perhaps more to the point, it

Re: [HACKERS] 9.4 regression

2013-09-04 Thread Andres Freund
On 2013-09-04 08:01:30 -0700, Jeff Davis wrote: On Thu, 2013-08-08 at 17:42 -0400, Tom Lane wrote: Jon Nelson jnelson+pg...@jamponi.net writes: At this point I'm convinced that the issue is a pathological case in ext4. The performance impact disappears as soon as the unwritten

Re: [HACKERS] [v9.4] row level security

2013-09-04 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Fri, Aug 30, 2013 at 3:43 PM, Tom Lane t...@sss.pgh.pa.us wrote: I think it's entirely sensible to question whether we should reject (not hold up) RLS if it has major covert-channel problems. We've already had this argument before, about the

Re: [HACKERS] [9.4] Make full_page_writes only settable on server start?

2013-09-04 Thread Andres Freund
On 2013-09-04 07:57:15 -0700, Jeff Davis wrote: XLogSaveBufferForHint() calls XLogCheckBuffer() but doesn't also look at the full page writes setting (like in XLogInsert()). That means, if checksums are enabled and full_page_writes is off, we'll still write some full page images for checksums.

Re: [HACKERS] [v9.4] row level security

2013-09-04 Thread Robert Haas
On Sun, Sep 1, 2013 at 11:47 PM, Greg Smith g...@2ndquadrant.com wrote: And if someone can INSERT values that they can't actually see once they're committed, that's a similarly bad we should describe. This is desirable in some cases but not others. If the goal is compartmentalization, then

Re: [HACKERS] [v9.4] row level security

2013-09-04 Thread Kohei KaiGai
2013/9/3 Bruce Momjian br...@momjian.us: On Sun, Sep 1, 2013 at 11:05:58AM -0700, Josh Berkus wrote: Security community also concludes it is not avoidable nature as long as human can observe system behavior and estimate something, thus, security evaluation criteria does not require

Re: [HACKERS] 9.4 regression

2013-09-04 Thread Jeff Davis
On Thu, 2013-08-08 at 17:42 -0400, Tom Lane wrote: Jon Nelson jnelson+pg...@jamponi.net writes: At this point I'm convinced that the issue is a pathological case in ext4. The performance impact disappears as soon as the unwritten extent(s) are written to with real data. Thus, even though

Re: [HACKERS] [v9.4] row level security

2013-09-04 Thread Robert Haas
On Fri, Aug 30, 2013 at 3:43 PM, Tom Lane t...@sss.pgh.pa.us wrote: I think it's entirely sensible to question whether we should reject (not hold up) RLS if it has major covert-channel problems. We've already had this argument before, about the security_barrier view stuff, and that code got

Re: [HACKERS] 9.4 regression

2013-09-04 Thread Stephen Frost
* Andres Freund (and...@2ndquadrant.com) wrote: I'd vote for adding zeroing *after* the fallocate() first. That's what's suggested by kernel hackers and what several other large applications do. As it looks like it's what we would have to do if we ever get to use fallocate for relation

Re: [HACKERS] [v9.4] row level security

2013-09-04 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Wed, Sep 4, 2013 at 10:50 AM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: Right. IMHO, this new feature should be similarly simple: when an unprivileged user references a table, treat that as a reference to a

Re: [HACKERS] 9.4 regression

2013-09-04 Thread Andres Freund
Hi, On 2013-09-04 11:15:37 -0400, Stephen Frost wrote: * Andres Freund (and...@2ndquadrant.com) wrote: I'd vote for adding zeroing *after* the fallocate() first. That's what's suggested by kernel hackers and what several other large applications do. As it looks like it's what we would have

Re: [HACKERS] getting rid of maintainer-check

2013-09-04 Thread Andres Freund
On 2013-09-03 22:41:17 -0400, Peter Eisentraut wrote: The maintainer-check target never really caught on, I think. Most people don't run it, and that in turn annoys those who do. Also, it doesn't provide much functionality. I propose that we get rid of it and roll the functionality into

Re: [HACKERS] getting rid of maintainer-check

2013-09-04 Thread Robert Haas
On Tue, Sep 3, 2013 at 10:41 PM, Peter Eisentraut pete...@gmx.net wrote: The maintainer-check target never really caught on, I think. Most people don't run it, and that in turn annoys those who do. Also, it doesn't provide much functionality. I propose that we get rid of it and roll the

Re: [HACKERS] [v9.4] row level security

2013-09-04 Thread Stephen Frost
* Robert Haas (robertmh...@gmail.com) wrote: On Sun, Sep 1, 2013 at 11:47 PM, Greg Smith g...@2ndquadrant.com wrote: And if someone can INSERT values that they can't actually see once they're committed, that's a similarly bad we should describe. This is desirable in some cases but not

Re: [HACKERS] [v9.4] row level security

2013-09-04 Thread Kevin Grittner
Robert Haas robertmh...@gmail.com wrote: Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: IMHO, this new feature should be similarly simple: when an unprivileged user references a table, treat that as a reference to a leakproof view over the table, with the RLS

Re: [HACKERS] [tiny doc fix] statistics are not retained across immediate shutdown

2013-09-04 Thread Tom Lane
Tsunakawa, Takayuki tsunakawa.ta...@jp.fujitsu.com writes: I propose a tiny fix to clarify this. Please find the attached patch. That's not an accurate description of what happens, though. AFAIR, we do not throw away pg_stats files as a result of recovery. regards, tom

Re: [HACKERS] [v9.4] row level security

2013-09-04 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Wed, Sep 4, 2013 at 10:45 AM, Tom Lane t...@sss.pgh.pa.us wrote: Well, the security-barrier view stuff did not present itself as a 100% solution. But perhaps more to the point, it was conceptually simple to implement, ie don't flatten views if they

Re: [HACKERS] 9.4 regression

2013-09-04 Thread Tom Lane
Stephen Frost sfr...@snowman.net writes: * Andres Freund (and...@2ndquadrant.com) wrote: I'd vote for adding zeroing *after* the fallocate() first. That's what's suggested by kernel hackers and what several other large applications do. As it looks like it's what we would have to do if we ever

Re: [HACKERS] [9.4] Make full_page_writes only settable on server start?

2013-09-04 Thread Tom Lane
Jeff Davis pg...@j-davis.com writes: On Tue, 2013-09-03 at 15:42 -0400, Robert Haas wrote: Although this is true, the administrator's estimate of whether that guarantee is or is not provided might not be as consistent as the hardware behavior itself. I am generally of the feeling that having

Re: [HACKERS] [v9.4] row level security

2013-09-04 Thread Kohei KaiGai
2013/9/4 Tom Lane t...@sss.pgh.pa.us: Robert Haas robertmh...@gmail.com writes: On Wed, Sep 4, 2013 at 10:45 AM, Tom Lane t...@sss.pgh.pa.us wrote: Well, the security-barrier view stuff did not present itself as a 100% solution. But perhaps more to the point, it was conceptually simple to

Re: [HACKERS] [v9.4] row level security

2013-09-04 Thread Kohei KaiGai
2013/9/1 Greg Stark st...@mit.edu: On Sun, Sep 1, 2013 at 8:31 PM, Kohei KaiGai kai...@kaigai.gr.jp wrote: Or, any other criteria even though? My (current) preference is plan (c: we will be able to fix up *this* cover-channel with reasonable efforts on explain code. probably, we can close it

Re: [HACKERS] UTF8 national character data type support WIP patch and list of open issues.

2013-09-04 Thread Tom Lane
Boguk, Maksym maks...@fast.au.fujitsu.com writes: Hi, my task is implementing ANSI NATIONAL character string types as part of PostgreSQL core. No, that's not a given. You have a problem to solve, ie store some UTF8 strings in a database that's mostly just 1-byte data. It is not clear that

Re: [HACKERS] [v9.4] row level security

2013-09-04 Thread Robert Haas
On Wed, Sep 4, 2013 at 10:45 AM, Tom Lane t...@sss.pgh.pa.us wrote: Well, the security-barrier view stuff did not present itself as a 100% solution. But perhaps more to the point, it was conceptually simple to implement, ie don't flatten views if they have this bit set, and don't push down

[HACKERS] Re: [HACKERS] Is it necessary to rewrite table while increasing the scale of datatype numeric?

2013-09-04 Thread Jeff Janes
On Tue, Sep 3, 2013 at 9:08 PM, wangs...@highgo.com.cn wrote: Hi, Hackers! I find that it takes a long time when I increase the scale of a numeric datatype. By checking the code, I found that's because it needs to rewrite that table's file. After checking that table's data file, I found

Re: [HACKERS] Re: Proposal/design feedback needed: WITHIN GROUP (sql standard ordered set aggregate functions)

2013-09-04 Thread Peter Eisentraut
On 7/19/13 11:59 AM, Atri Sharma wrote: Hi all, This is our current work-in-progress patch for WITHIN GROUP. This patch needs to be rebased. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription:

Re: [HACKERS] [v9.4] row level security

2013-09-04 Thread Kohei KaiGai
2013/9/4 Kevin Grittner kgri...@ymail.com: Robert Haas robertmh...@gmail.com wrote: Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: IMHO, this new feature should be similarly simple: when an unprivileged user references a table, treat that as a reference to a

Re: [HACKERS] Eliminating pg_catalog.pg_rewrite.ev_attr

2013-09-04 Thread Alvaro Herrera
Is this transformation correct? If I read this correctly, you're missing the rangeTableEntry_used() condition, no? *** a/src/backend/rewrite/rewriteHandler.c --- b/src/backend/rewrite/rewriteHandler.c *** *** 1273,1287 matchLocks(CmdType event, }

Re: [HACKERS] [9.4] Make full_page_writes only settable on server start?

2013-09-04 Thread Jeff Davis
On Wed, 2013-09-04 at 11:32 -0400, Tom Lane wrote: Jeff Davis pg...@j-davis.com writes: I think code complexity matters quite a lot. If we can eliminate some complex code in a complex area, and all we give up is a feature with essentially no use case, that sounds like we're moving in the

Re: [HACKERS] Re: Proposal/design feedback needed: WITHIN GROUP (sql standard ordered set aggregate functions)

2013-09-04 Thread Atri Sharma
Sent from my iPad On 04-Sep-2013, at 21:38, Peter Eisentraut pete...@gmx.net wrote: On 7/19/13 11:59 AM, Atri Sharma wrote: Hi all, This is our current work-in-progress patch for WITHIN GROUP. This patch needs to be rebased. This version of patch is quite old.We will be sending an

[HACKERS] Eliminating pg_catalog.pg_rewrite.ev_attr

2013-09-04 Thread Kevin Grittner
This was previously discussed here: http://www.postgresql.org/message-id/flat/24836.1370713...@sss.pgh.pa.us#24836.1370713...@sss.pgh.pa.us The attached patch implements what I think we agreed on. To recap, ev_attr was present in pg_rewrite at the point that Postgres95 version 1.01 source code

Re: [HACKERS] [v9.4] row level security

2013-09-04 Thread Kohei KaiGai
2013/9/4 Tom Lane t...@sss.pgh.pa.us: Robert Haas robertmh...@gmail.com writes: On Wed, Sep 4, 2013 at 10:50 AM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: Right. IMHO, this new feature should be similarly simple: when an unprivileged user references a

Re: [HACKERS] [9.4] Make full_page_writes only settable on server start?

2013-09-04 Thread Jeff Davis
On Wed, 2013-09-04 at 17:00 +0200, Andres Freund wrote: On 2013-09-04 07:57:15 -0700, Jeff Davis wrote: XLogSaveBufferForHint() calls XLogCheckBuffer() but doesn't also look at the full page writes setting (like in XLogInsert()). That means, if checksums are enabled and full_page_writes is

Re: [HACKERS] Eliminating pg_catalog.pg_rewrite.ev_attr

2013-09-04 Thread Tom Lane
Kevin Grittner kgri...@ymail.com writes: Unless someone has an objection or thinks this needs to go through the CF process, I will commit it tomorrow, with a catversion bump. Shouldn't attribute_used() be removed from rewriteManip.h? I was a bit surprised by your removal of the

Re: [HACKERS] [v9.4] row level security

2013-09-04 Thread Robert Haas
On Wed, Sep 4, 2013 at 11:22 AM, Tom Lane t...@sss.pgh.pa.us wrote: The same mechanism will prevent UPDATE and DELETE from seeing any rows the user shouldn't be able to touch. No, it won't, because we don't support direct update/delete on views (and if you look, you'll notice the

Re: [HACKERS] ENABLE/DISABLE CONSTRAINT NAME

2013-09-04 Thread Robert Haas
On Tue, Sep 3, 2013 at 3:13 AM, wangs...@highgo.com.cn wrote: 于 2013-09-03 08:15, David Johnston 回复: Jeff Davis-8 wrote Is there any semantic difference between marking a constraint as DISABLED and simply dropping it? Or does it just make it easier to re-add it later? David Johnston

Re: [HACKERS] Improving avg performance for numeric

2013-09-04 Thread Peter Eisentraut
On 7/8/13 10:05 AM, Pavel Stehule wrote: I am testing your code, and It increase speed of sum about 24% faster then original implementation. This patch needs to be rebased (and/or the later version registered in the commit fest). -- Sent via pgsql-hackers mailing list

Re: [HACKERS] Eliminating pg_catalog.pg_rewrite.ev_attr

2013-09-04 Thread Kevin Grittner
Tom Lane t...@sss.pgh.pa.us wrote: Shouldn't attribute_used() be removed from rewriteManip.h? Yeah, I don't know how I missed that.  Thanks. I was a bit surprised by your removal of the rangeTableEntry_used() test in the hunk at rewriteHandler.c:1273ff.  That's probably all right, but it

Re: [HACKERS] Improving avg performance for numeric

2013-09-04 Thread Pavel Stehule
2013/9/4 Peter Eisentraut pete...@gmx.net On 7/8/13 10:05 AM, Pavel Stehule wrote: I am testing your code, and It increase speed of sum about 24% faster then original implementation. This patch needs to be rebased (and/or the later version registered in the commit fest). I updated a

Re: [HACKERS] [9.4] Make full_page_writes only settable on server start?

2013-09-04 Thread Andres Freund
On 2013-09-04 09:23:20 -0700, Jeff Davis wrote: On Wed, 2013-09-04 at 11:32 -0400, Tom Lane wrote: Jeff Davis pg...@j-davis.com writes: I think code complexity matters quite a lot. If we can eliminate some complex code in a complex area, and all we give up is a feature with essentially

Re: [HACKERS] [rfc] overhauling pgstat.stat

2013-09-04 Thread Tomas Vondra
On 4.9.2013 07:24, Satoshi Nagayasu wrote: Hi, (2013/09/04 12:52), Atri Sharma wrote: On Wed, Sep 4, 2013 at 6:40 AM, Satoshi Nagayasu sn...@uptime.jp wrote: Hi, I'm considering overhauling pgstat.stat, and would like to know how many people are interested in this topic. As you may

Re: [HACKERS] [v9.4] row level security

2013-09-04 Thread Tom Lane
Kohei KaiGai kai...@kaigai.gr.jp writes: 2013/9/4 Tom Lane t...@sss.pgh.pa.us: And for insert/update/delete, we do what exactly? Regarding to update/delete, this patch also enhanced to allow update or delete mechanism allows to take a sub-query on top of the table scan plan. So, its explain

Re: [HACKERS] [rfc] overhauling pgstat.stat

2013-09-04 Thread Tomas Vondra
On 4.9.2013 14:43, Pavel Stehule wrote: 2013/9/4 Atri Sharma atri.j...@gmail.com mailto:atri.j...@gmail.com Sent from my iPad On 04-Sep-2013, at 15:39, Satoshi Nagayasu sn...@uptime.jp mailto:sn...@uptime.jp wrote: (2013/09/04 15:23), Atri Sharma wrote:

Re: [HACKERS] [rfc] overhauling pgstat.stat

2013-09-04 Thread Pavel Stehule
we very successfully use a tmpfs volume for pgstat files (use a backport of multiple statfiles from 9.3 to 9.1 It works quite well as long as you have the objects (tables, indexes, functions) spread across multiple databases. Once you have one database with very large number of objects,

Re: [HACKERS] [rfc] overhauling pgstat.stat

2013-09-04 Thread Alvaro Herrera
Tomas Vondra wrote: My idea was to keep the per-database stats, but allow some sort of random access - updating / deleting the records in place, adding records etc. The simplest way I could think of was adding a simple index - a mapping of OID to position in the stat file. I.e. a simple

Re: [HACKERS] Further XLogInsert scaling tweaking

2013-09-04 Thread Heikki Linnakangas
On 03.09.2013 16:22, Merlin Moncure wrote: On Mon, Sep 2, 2013 at 10:32 PM, Bruce Momjianbr...@momjian.us wrote: On Mon, Sep 2, 2013 at 10:14:03AM +0300, Heikki Linnakangas wrote: diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 39c58d0..28e62ea

Re: [HACKERS] INSERT...ON DUPLICATE KEY IGNORE

2013-09-04 Thread Robert Haas
On Sat, Aug 31, 2013 at 2:34 PM, Andres Freund and...@2ndquadrant.com wrote: After some thinking I don't think any solution primarily based on holding page level locks across other index operations is going to scale ok. I'd like to chime in with a large +1 for this sentiment and pretty much

Re: [HACKERS] Improving avg performance for numeric

2013-09-04 Thread Peter Eisentraut
On 9/4/13 2:26 PM, Pavel Stehule wrote: 2013/9/4 Peter Eisentraut pete...@gmx.net mailto:pete...@gmx.net On 7/8/13 10:05 AM, Pavel Stehule wrote: I am testing your code, and It increase speed of sum about 24% faster then original implementation. This patch needs to

Re: [HACKERS] lcr v5 - introduction of InvalidCommandId

2013-09-04 Thread Andres Freund
On 2013-09-03 11:40:57 -0400, Robert Haas wrote: 0002 wal_decoding: Introduce InvalidCommandId and declare that to be the new maximum for CommandCounterIncrement I'm still unconvinced we want this. Ok, so the reason for the existance of this patch is that currently there is no way to

[HACKERS] De-normalization optimizer research project

2013-09-04 Thread arthernan
I am working on a research project that looks at an entire database access layer written against a normalized database and suggests de-normalizations to be done to the database structure. In effect looking at de-normalizations as a kind of cache. I am looking for any code I can leverage to do this

Re: [HACKERS] [rfc] overhauling pgstat.stat

2013-09-04 Thread Tomas Vondra
On 4.9.2013 20:59, Alvaro Herrera wrote: Tomas Vondra wrote: My idea was to keep the per-database stats, but allow some sort of random access - updating / deleting the records in place, adding records etc. The simplest way I could think of was adding a simple index - a mapping of OID to

Re: [HACKERS] [rfc] overhauling pgstat.stat

2013-09-04 Thread Tomas Vondra
On 4.9.2013 14:13, Stephen Frost wrote: * Satoshi Nagayasu (sn...@uptime.jp) wrote: Yes, we have already split single pgstat.stat file into several pieces. However, we still need to read/write large amount of statistics data when we have a large number of tables in single database or

Re: [HACKERS] [PERFORM] encouraging index-only scans

2013-09-04 Thread Bruce Momjian
On Thu, Dec 13, 2012 at 03:31:06PM +, Peter Geoghegan wrote: On 13 December 2012 03:51, Tom Lane t...@sss.pgh.pa.us wrote: ANALYZE does not set that value, and is not going to start doing so, because it doesn't scan enough of the table to derive a trustworthy value. I'm slightly

Re: [HACKERS] INSERT...ON DUPLICATE KEY IGNORE

2013-09-04 Thread Peter Geoghegan
On Wed, Sep 4, 2013 at 1:26 PM, Robert Haas robertmh...@gmail.com wrote: Concurrent readers will block in a non-interruptible wait if they try to access a buffer, and that's a situation that will be intolerable if, for example, it can persist across a disk I/O. And I don't see any way to

[HACKERS] Analysis on backend-private memory usage (and a patch)

2013-09-04 Thread Heikki Linnakangas
I received a complaint that each backend consumes a lot of backend-private memory, even if it's completely idle. a lot is of course very subjective and how much memory is actually used depends heavily on the application. In this case, the database is fairly small, but they have 250

Re: [HACKERS] dynamic shared memory

2013-09-04 Thread Jim Nasby
On 8/31/13 7:17 AM, Robert Haas wrote: On Thu, Aug 29, 2013 at 8:12 PM, Jim Nasby j...@nasby.net wrote: On 8/13/13 8:09 PM, Robert Haas wrote: is removed, the segment automatically goes away (we could allow for server-lifespan segments as well with only trivial changes, but I'm not sure

Re: [HACKERS] Analysis on backend-private memory usage (and a patch)

2013-09-04 Thread Tom Lane
Heikki Linnakangas hlinnakan...@vmware.com writes: One fairly simple thing we could do is to teach catcache.c to resize the caches. Then we could make the initial size of all the syscaches much smaller. I think this is attractive for the *other* reason you mention, namely preserving

Re: [HACKERS] 9.4 regression

2013-09-04 Thread Robert Haas
On Wed, Sep 4, 2013 at 11:26 AM, Tom Lane t...@sss.pgh.pa.us wrote: Stephen Frost sfr...@snowman.net writes: * Andres Freund (and...@2ndquadrant.com) wrote: I'd vote for adding zeroing *after* the fallocate() first. That's what's suggested by kernel hackers and what several other large

Re: [HACKERS] INSERT...ON DUPLICATE KEY IGNORE

2013-09-04 Thread Andres Freund
On 2013-09-04 15:01:57 -0700, Peter Geoghegan wrote: On Wed, Sep 4, 2013 at 1:26 PM, Robert Haas robertmh...@gmail.com wrote: Concurrent readers will block in a non-interruptible wait if they try to access a buffer, and that's a situation that will be intolerable if, for example, it can

Re: [HACKERS] INSERT...ON DUPLICATE KEY IGNORE

2013-09-04 Thread Peter Geoghegan
On Wed, Sep 4, 2013 at 3:39 PM, Andres Freund and...@2ndquadrant.com wrote: Sorry to be harsh here, but I don't think I need to do that. I've explained most of the reasons I see that that approach won't work out and so far I don't see those refuted. And to me those issues seem to be fatal for

Re: [HACKERS] INSERT...ON DUPLICATE KEY IGNORE

2013-09-04 Thread Andres Freund
Hi, We seem to be miscommunication a bit. You've proposed an framework and algorithm for something I'd really, really like to see. I don't think that it can work explicitly as you proposed, so I roughly sketched out a solution I can see. I don't want my idea to win, I want a idea to win. I

Re: [HACKERS] INSERT...ON DUPLICATE KEY IGNORE

2013-09-04 Thread Peter Geoghegan
On Wed, Sep 4, 2013 at 5:08 PM, Andres Freund and...@2ndquadrant.com wrote: I don't want my idea to win, I want a idea to win. I know. I want the same thing. You're the patch author here whose plans are laid open to be scrutinized ;). If you think my idea has merit, use and adapt it to

Re: [HACKERS] INSERT...ON DUPLICATE KEY IGNORE

2013-09-04 Thread Gavin Flower
On 05/09/13 08:26, Robert Haas wrote: On Sat, Aug 31, 2013 at 2:34 PM, Andres Freund and...@2ndquadrant.com wrote: After some thinking I don't think any solution primarily based on holding page level locks across other index operations is going to scale ok. I'd like to chime in with a large +1

Re: [HACKERS] proposal: Set effective_cache_size to greater of .conf value, shared_buffers

2013-09-04 Thread Bruce Momjian
On Tue, Jan 8, 2013 at 08:40:44PM -0500, Andrew Dunstan wrote: On 01/08/2013 08:08 PM, Tom Lane wrote: Robert Haas robertmh...@gmail.com writes: On Tue, Jan 8, 2013 at 7:17 PM, Tom Lane t...@sss.pgh.pa.us wrote: ... And I don't especially like the idea of trying to make it depend directly

Re: [HACKERS] Frontend/backend protocol improvements proposal (request).

2013-09-04 Thread Noah Misch
On Fri, Jun 21, 2013 at 12:37:32PM +0400, Dmitriy Igrishin wrote: 2013/6/21 Albe Laurenz laurenz.a...@wien.gv.at Dmitriy Igrishin wrote: While developing a C++ client library for Postgres I felt lack of extra information in command tags in the CommandComplete (B) message for the

Re: [HACKERS] strange IS NULL behaviour

2013-09-04 Thread Bruce Momjian
On Tue, Sep 3, 2013 at 09:32:44PM -0400, Bruce Momjian wrote: In this test, SELECT NULL (which internally would produce SELECT ROW(NULL)), returns TRUE, while SELECT ROW(NULL) and further nesting returns false. This has made me adjust my goal and change it so SELECT ROW(NULL) IS NULL

Re: [HACKERS] improve Chinese locale performance

2013-09-04 Thread Quan Zongliang
On 07/23/2013 09:42 PM, Craig Ringer wrote: (Replying on phone, please forgive bad quoting) Isn't this pretty much what adopting ICU is supposed to give us? OS-independent collations? I'd be interested in seeing the rest data for this performance report, partly as I'd like to see how ICU

Re: [HACKERS] [tiny doc fix] statistics are not retained across immediate shutdown

2013-09-04 Thread Fujii Masao
On Wed, Sep 4, 2013 at 11:56 PM, Andres Freund and...@2ndquadrant.com wrote: On 2013-09-04 10:53:19 -0400, Tom Lane wrote: Tsunakawa, Takayuki tsunakawa.ta...@jp.fujitsu.com writes: I propose a tiny fix to clarify this. Please find the attached patch. That's not an accurate description of

[HACKERS] Re: Is it necessary to rewrite table while increasing the scale of datatype numeric???

2013-09-04 Thread wangshuo
于 2013-09-04 19:30, Noah Misch 回复: On Wed, Sep 04, 2013 at 12:08:48PM +0800, wangs...@highgo.com.cn wrote: I find that it takes a long time when I increase the scale of a numeric datatype. By checking the code, I found that's because it needs to rewrite that table's file. After checking that

Re: [HACKERS] [rfc] overhauling pgstat.stat

2013-09-04 Thread Satoshi Nagayasu
(2013/09/05 3:50), Pavel Stehule wrote: we very successfully use a tmpfs volume for pgstat files (use a backport of multiple statfiles from 9.3 to 9.1 It works quite well as long as you have the objects (tables, indexes, functions) spread across multiple databases. Once

Re: [HACKERS] ENABLE/DISABLE CONSTRAINT NAME

2013-09-04 Thread wangshuo
于 2013-09-05 01:56, Robert Haas 回复: On Tue, Sep 3, 2013 at 3:13 AM, wangs...@highgo.com.cn wrote: 于 2013-09-03 08:15, David Johnston 回复: Jeff Davis-8 wrote Is there any semantic difference between marking a constraint as DISABLED and simply dropping it? Or does it just make it easier to

Re: [HACKERS] [rfc] overhauling pgstat.stat

2013-09-04 Thread Satoshi Nagayasu
(2013/09/05 3:59), Alvaro Herrera wrote: Tomas Vondra wrote: My idea was to keep the per-database stats, but allow some sort of random access - updating / deleting the records in place, adding records etc. The simplest way I could think of was adding a simple index - a mapping of OID to

Re: [HACKERS] [rfc] overhauling pgstat.stat

2013-09-04 Thread Atri Sharma
Sent from my iPad On 05-Sep-2013, at 8:58, Satoshi Nagayasu sn...@uptime.jp wrote: (2013/09/05 3:59), Alvaro Herrera wrote: Tomas Vondra wrote: My idea was to keep the per-database stats, but allow some sort of random access - updating / deleting the records in place, adding records

Re: [HACKERS] [9.4] Make full_page_writes only settable on server start?

2013-09-04 Thread Fujii Masao
On Thu, Sep 5, 2013 at 1:55 AM, Andres Freund and...@2ndquadrant.com wrote: On 2013-09-04 09:23:20 -0700, Jeff Davis wrote: On Wed, 2013-09-04 at 11:32 -0400, Tom Lane wrote: Jeff Davis pg...@j-davis.com writes: I think code complexity matters quite a lot. If we can eliminate some

Re: [HACKERS] Is it necessary to rewrite table while increasing the scale of datatype numeric?

2013-09-04 Thread wangshuo
于 2013-09-04 23:41, Jeff Janes 回复: On Tue, Sep 3, 2013 at 9:08 PM, wangs...@highgo.com.cn wrote: Hi, Hackers! I find that it takes a long time when I increase the scale of a numeric datatype. By checking the code, I found that's because it needs to rewrite that table's file. After

Re: [HACKERS] [rfc] overhauling pgstat.stat

2013-09-04 Thread Alvaro Herrera
Satoshi Nagayasu wrote: But, for now, I think we should have a real index for the statistics data because we already have several index storages, and it will allow us to minimize read/write operations. BTW, what kind of index would be preferred for this purpose? btree or hash? I find it