Re: [HACKERS] Enabling Checksums

2013-03-08 Thread Heikki Linnakangas
On 08.03.2013 05:31, Bruce Momjian wrote: Also, don't all modern storage drives have built-in checksums, and report problems to the system administrator? Does smartctl help report storage corruption? Let me take a guess at answering this --- we have several layers in a database server:

Re: [HACKERS] Enabling Checksums

2013-03-08 Thread Heikki Linnakangas
On 07.03.2013 23:45, Jeff Davis wrote: By the way, I can not find any trace of XLogCheckBufferNeedsBackup(), was that a typo? Ah, sorry, that was a new function introduced by another patch I was reviewing at the same time, and I conflated the two. - Heikki -- Sent via pgsql-hackers

[HACKERS] Re: [BUGS] BUG #7873: pg_restore --clean tries to drop tables that don't exist

2013-03-08 Thread Pavel Stehule
2013/3/8 Josh Kupershmidt schmi...@gmail.com: [Moving to -hackers] On Sat, Feb 23, 2013 at 2:51 PM, Pavel Stehule pavel.steh...@gmail.com wrote: so * --conditional-drops replaced by --if-exists Thanks for the fixes, I played around with the patch a bit. I was sort of expecting this

Re: [HACKERS] REFRESH MATERIALIZED VIEW locklevel

2013-03-08 Thread Nicolas Barbier
2013/3/8 Andres Freund and...@anarazel.de: On 2013-03-07 15:21:35 -0800, Josh Berkus wrote: This limitation is in no way crippling for this feature, or even a major detraction. I still intend to promote the heck out of this feature. Thats scaring me. Because the current state of the

Re: [HACKERS] [v9.3] OAT_POST_ALTER object access hooks

2013-03-08 Thread Kohei KaiGai
Thanks for your reviewing. 2013/3/7 Robert Haas robertmh...@gmail.com: On Sun, Jan 27, 2013 at 1:55 AM, Kohei KaiGai kai...@kaigai.gr.jp wrote: The part-2 patch adds new OAT_POST_ALTER event type, and its relevant permission checks on contrib/sepgsql. This documentation hunk is unclear: +

Re: [HACKERS] SP-GiST for ranges based on 2d-mapping and quad-tree

2013-03-08 Thread Heikki Linnakangas
On 03.03.2013 19:42, Alexander Korotkov wrote: This patch only adds one more operator to already committed new opclass. Tests already cover this case. Without patch corresponding test leads to sequential scan instead of index scan. Thanks, committed with some trivial cleanup. However, I

Re: [HACKERS] sql_drop Event Triggerg

2013-03-08 Thread Robert Haas
On Tue, Mar 5, 2013 at 12:45 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Hmm, maybe I should be considering a pair of macros instead -- UTILITY_START_DROP and UTILITY_END_DROP. I'll give this a try. Other ideas are welcome. That seems like a possibly promising idea. I do wonder how

Re: [HACKERS] Index Unqiueness

2013-03-08 Thread Robert Haas
On Tue, Mar 5, 2013 at 10:41 PM, abhinav batra abba...@gmail.com wrote: Hey I want to work towards the follwing feature in TODO list: Prevent index uniqueness checks when UPDATE does not modify the columnUniqueness (index) checks are done when updating a column even if the column is not

Re: [HACKERS] Index Unqiueness

2013-03-08 Thread Andres Freund
On 2013-03-06 09:11:38 +0530, abhinav batra wrote: Hey I want to work towards the follwing feature in TODO list: Prevent index uniqueness checks when UPDATE does not modify the columnUniqueness (index) checks are done when updating a column even if the column is not modified by the UPDATE.

Re: [HACKERS] Materialized views WIP patch

2013-03-08 Thread Bruce Momjian
On Wed, Mar 6, 2013 at 09:16:59AM -0500, Tom Lane wrote: Simon Riggs si...@2ndquadrant.com writes: On 5 March 2013 22:02, Tom Lane t...@sss.pgh.pa.us wrote: FWIW, my opinion is that doing anything like this in the planner is going to be enormously expensive. As we already said: no MVs

Re: [HACKERS] sql_drop Event Triggerg

2013-03-08 Thread Alvaro Herrera
Robert Haas escribió: On Tue, Mar 5, 2013 at 12:45 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Hmm, maybe I should be considering a pair of macros instead -- UTILITY_START_DROP and UTILITY_END_DROP. I'll give this a try. Other ideas are welcome. That seems like a possibly

Re: [HACKERS] Parameterized paths vs index clauses extracted from OR clauses

2013-03-08 Thread Robert Haas
On Tue, Mar 5, 2013 at 11:53 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Tue, Mar 5, 2013 at 3:44 PM, Tom Lane t...@sss.pgh.pa.us wrote: If foo OR bar is useful as an indexqual condition in the inner scan, that's one thing. But if it isn't, the cycles

Re: [HACKERS] Materialized views WIP patch

2013-03-08 Thread Bruce Momjian
On Tue, Mar 5, 2013 at 08:50:39AM +, Simon Riggs wrote: Its not a different feature, its what most people expect a feature called MV to deliver. That's not a matter of opinion, its simply how every other database works currently - Oracle, Teradata, SQLServer at least. The fact that we

Re: [HACKERS] Index Unqiueness

2013-03-08 Thread Robert Haas
On Fri, Mar 8, 2013 at 8:58 AM, Andres Freund and...@2ndquadrant.com wrote: I'd be interested in something slightly related that is doing HOT on a per-index basis. Currently we don't do hot if any index is updated even though quite possibly most of the indexes don't change. I think that might

Re: [HACKERS] Materialized views WIP patch

2013-03-08 Thread Robert Haas
On Thu, Mar 7, 2013 at 12:14 PM, David E. Wheeler da...@justatheory.com wrote: On Mar 7, 2013, at 7:55 AM, Kevin Grittner kgri...@ymail.com wrote: If the answer to both those questions is “yes,” I think the term should remain “table,” with a few mentions that the term includes materialized

Re: [HACKERS] sql_drop Event Triggerg

2013-03-08 Thread Robert Haas
On Fri, Mar 8, 2013 at 9:18 AM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Robert Haas escribió: On Tue, Mar 5, 2013 at 12:45 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Hmm, maybe I should be considering a pair of macros instead -- UTILITY_START_DROP and UTILITY_END_DROP. I'll

[HACKERS] Re: [BUGS] BUG #7873: pg_restore --clean tries to drop tables that don't exist

2013-03-08 Thread Josh Kupershmidt
On Fri, Mar 8, 2013 at 2:23 AM, Pavel Stehule pavel.steh...@gmail.com wrote: 2013/3/8 Josh Kupershmidt schmi...@gmail.com: Cool. I think it would also be useful to check that --clean may only be used with --format=p to avoid any confusion there. (This issue could be addressed in a separate

Re: [HACKERS] sql_drop Event Triggerg

2013-03-08 Thread Alvaro Herrera
Robert Haas escribió: On Fri, Mar 8, 2013 at 9:18 AM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Robert Haas escribió: On Tue, Mar 5, 2013 at 12:45 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Hmm, maybe I should be considering a pair of macros instead --

Re: [HACKERS] Support for REINDEX CONCURRENTLY

2013-03-08 Thread Fujii Masao
On Fri, Mar 8, 2013 at 10:00 PM, Michael Paquier michael.paqu...@gmail.com wrote: On Fri, Mar 8, 2013 at 1:41 AM, Fujii Masao masao.fu...@gmail.com wrote: On Thu, Mar 7, 2013 at 7:19 AM, Andres Freund and...@2ndquadrant.com wrote: The strange think about hoge_pkey_cct_cct is that it seems

[HACKERS] Re: [BUGS] BUG #7873: pg_restore --clean tries to drop tables that don't exist

2013-03-08 Thread Pavel Stehule
2013/3/8 Josh Kupershmidt schmi...@gmail.com: On Fri, Mar 8, 2013 at 2:23 AM, Pavel Stehule pavel.steh...@gmail.com wrote: 2013/3/8 Josh Kupershmidt schmi...@gmail.com: Cool. I think it would also be useful to check that --clean may only be used with --format=p to avoid any confusion there.

Re: [HACKERS] [DOCS] Contrib module xml2 status

2013-03-08 Thread Bruce Momjian
On Fri, Feb 22, 2013 at 08:25:31AM +0900, Ian Lawrence Barwick wrote: The point is we can remove the module when someone fixes and replaces the functionality that's left in there that some people rely on. So far nobody has stepped up to the plate, although now that we have lateral a sane

Re: [HACKERS] Enabling Checksums

2013-03-08 Thread Josh Berkus
I also see the checksum patch is taking a beating. I wanted to step back and ask what pertntage of known corruptions cases will this checksum patch detect? I'm pretty sure that early on Jeff posted some statstics which indicated that the current approach would detect 99% of corruption

Re: [HACKERS] Identity projection

2013-03-08 Thread Heikki Linnakangas
On 12.02.2013 11:03, Amit Kapila wrote: + /* + * equivalent_tlists + * returns whether two traget lists are equivalent + * + * We consider two target lists equivalent if both have + * only Var entries and resjunk of each target entry is same. + * + * This function is used to decide

Re: [HACKERS] REFRESH MATERIALIZED VIEW locklevel

2013-03-08 Thread Josh Berkus
Andres, Crossing this over to pgsql-advocacy, because this is really an Advocacy discussion. The point is that a) refreshing is the only way to update materialized views. There's no incremental support. b) refreshing will take a long time (otherwise you wouldn't have create a

Re: [HACKERS] REFRESH MATERIALIZED VIEW locklevel

2013-03-08 Thread Merlin Moncure
On Fri, Mar 8, 2013 at 11:59 AM, Josh Berkus j...@agliodbs.com wrote: Andres, Crossing this over to pgsql-advocacy, because this is really an Advocacy discussion. The point is that a) refreshing is the only way to update materialized views. There's no incremental support. b) refreshing

Re: [HACKERS] REFRESH MATERIALIZED VIEW locklevel

2013-03-08 Thread Joshua D. Drake
On 03/08/2013 10:09 AM, Merlin Moncure wrote: On Fri, Mar 8, 2013 at 11:59 AM, Josh Berkus j...@agliodbs.com wrote: Andres, Further, we get pretty much one and only one chance to promote a new major feature, which is when that feature is first introduced. Improving the feature in the next

Re: [HACKERS] Why do we still perform a check for pre-sorted input within qsort variants?

2013-03-08 Thread Bruce Momjian
On Mon, Feb 25, 2013 at 02:31:21PM +, Peter Geoghegan wrote: On 25 February 2013 11:49, Robert Haas robertmh...@gmail.com wrote: I did attempt to do some tinkering with this while I was playing with it, but I didn't come up with anything really compelling. You can reduce the number of

Re: [HACKERS] Why do we still perform a check for pre-sorted input within qsort variants?

2013-03-08 Thread Dann Corbit
-Original Message- From: pgsql-hackers-ow...@postgresql.org [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Bruce Momjian Sent: Friday, March 08, 2013 11:22 AM To: Peter Geoghegan Cc: Robert Haas; Tom Lane; PG Hackers Subject: Re: [HACKERS] Why do we still perform a check for

Re: [HACKERS] Why do we still perform a check for pre-sorted input within qsort variants?

2013-03-08 Thread 'Bruce Momjian'
On Fri, Mar 8, 2013 at 07:43:10PM +, Dann Corbit wrote: I seem to recall that a year or two back some study was done on quicksort methodology as used in PostgreSQL. As I recall, the algorithm used in PostgreSQL fared well in the tests. Well, that's good to hear. -- Bruce Momjian

Re: [HACKERS] Why do we still perform a check for pre-sorted input within qsort variants?

2013-03-08 Thread Peter Geoghegan
On 8 March 2013 11:48, Bruce Momjian br...@momjian.us wrote: On Fri, Mar 8, 2013 at 07:43:10PM +, Dann Corbit wrote: I seem to recall that a year or two back some study was done on quicksort methodology as used in PostgreSQL. As I recall, the algorithm used in PostgreSQL fared well in the

Re: [HACKERS] Why do we still perform a check for pre-sorted input within qsort variants?

2013-03-08 Thread Dann Corbit
-Original Message- From: Peter Geoghegan [mailto:peter.geoghega...@gmail.com] Sent: Friday, March 08, 2013 12:00 PM To: Bruce Momjian Cc: Dann Corbit; Robert Haas; Tom Lane; PG Hackers Subject: Re: [HACKERS] Why do we still perform a check for pre-sorted input within qsort variants? On

Re: [HACKERS] Duplicate JSON Object Keys

2013-03-08 Thread Robert Haas
On Thu, Mar 7, 2013 at 2:48 PM, David E. Wheeler da...@justatheory.com wrote: In the spirit of being liberal about what we accept but strict about what we store, it seems to me that JSON object key uniqueness should be enforced either by throwing an error on duplicate keys, or by flattening

Re: [HACKERS] Duplicate JSON Object Keys

2013-03-08 Thread Hannu Krosing
On 03/08/2013 09:39 PM, Robert Haas wrote: On Thu, Mar 7, 2013 at 2:48 PM, David E. Wheeler da...@justatheory.com wrote: In the spirit of being liberal about what we accept but strict about what we store, it seems to me that JSON object key uniqueness should be enforced either by throwing an

Re: [HACKERS] Duplicate JSON Object Keys

2013-03-08 Thread Alvaro Herrera
Hannu Krosing escribió: On 03/08/2013 09:39 PM, Robert Haas wrote: On Thu, Mar 7, 2013 at 2:48 PM, David E. Wheeler da...@justatheory.com wrote: In the spirit of being liberal about what we accept but strict about what we store, it seems to me that JSON object key uniqueness should be

Re: [HACKERS] Duplicate JSON Object Keys

2013-03-08 Thread David E. Wheeler
On Mar 8, 2013, at 1:01 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: If it does not meet these semantic constraints, then it is not really JSON - it is merely JSON-like. this sounds very much like MySQLs decision to support timestamp -00-00 00:00 - syntactically correct, but

Re: [HACKERS] Duplicate JSON Object Keys

2013-03-08 Thread Gavin Flower
Well I would much prefer to find out sooner rather than later that there is a problem, so I would much prefer know I've created a duplicate as soon as the system can detect it. In general, Postgresql appears much better at this than MySQL On 09/03/13 10:01, Alvaro Herrera wrote: Hannu

Re: [HACKERS] Duplicate JSON Object Keys

2013-03-08 Thread Andrew Dunstan
On 03/08/2013 04:01 PM, Alvaro Herrera wrote: Hannu Krosing escribió: On 03/08/2013 09:39 PM, Robert Haas wrote: On Thu, Mar 7, 2013 at 2:48 PM, David E. Wheeler da...@justatheory.com wrote: In the spirit of being liberal about what we accept but strict about what we store, it seems to me

Re: [HACKERS] Duplicate JSON Object Keys

2013-03-08 Thread David E. Wheeler
On Mar 8, 2013, at 1:21 PM, Andrew Dunstan and...@dunslane.net wrote: Here's what rfc2119 says about that wording: 4. SHOULD NOT This phrase, or the phrase NOT RECOMMENDED mean that there may exist valid reasons in particular circumstances when the particular behavior is acceptable or

Re: [HACKERS] Duplicate JSON Object Keys

2013-03-08 Thread Hannu Krosing
On 03/08/2013 10:01 PM, Alvaro Herrera wrote: Hannu Krosing escribió: On 03/08/2013 09:39 PM, Robert Haas wrote: On Thu, Mar 7, 2013 at 2:48 PM, David E. Wheeler da...@justatheory.com wrote: In the spirit of being liberal about what we accept but strict about what we store, it seems to me

Re: [HACKERS] Duplicate JSON Object Keys

2013-03-08 Thread Andrew Dunstan
On 03/08/2013 04:28 PM, David E. Wheeler wrote: On Mar 8, 2013, at 1:21 PM, Andrew Dunstan and...@dunslane.net wrote: Here's what rfc2119 says about that wording: 4. SHOULD NOT This phrase, or the phrase NOT RECOMMENDED mean that there may exist valid reasons in particular

Re: [HACKERS] Duplicate JSON Object Keys

2013-03-08 Thread Hannu Krosing
On 03/08/2013 10:42 PM, Andrew Dunstan wrote: On 03/08/2013 04:28 PM, David E. Wheeler wrote: On Mar 8, 2013, at 1:21 PM, Andrew Dunstan and...@dunslane.net wrote: Here's what rfc2119 says about that wording: 4. SHOULD NOT This phrase, or the phrase NOT RECOMMENDED mean that there may

Re: [HACKERS] Duplicate JSON Object Keys

2013-03-08 Thread Andrew Dunstan
On 03/08/2013 04:42 PM, Andrew Dunstan wrote: So my order of preference for the options would be: 1. Have the JSON type collapse objects so the last instance of a key wins and is actually stored 2. Throw an error when a JSON type has duplicate keys 3. Have the accessors find the last

[HACKERS] pg_dump selectively ignores extension configuration tables

2013-03-08 Thread Joe Conway
(reposting apparently I used a verboten word the first time around sigh. Sorry for any duplicates) The -t and -n options of pg_dump do not dump anything from an extension configuration table, whereas normal pg_dump will dump the user data. To see what I mean, in psql do (tested on pg9.2):

Re: [HACKERS] Duplicate JSON Object Keys

2013-03-08 Thread Josh Berkus
Actually, now I think more about it 3 is the best answer. Here's why: even the JSON generators can produce JSON with non-unique field names: +1 Also, I think we should add a json_normalize() function to the TODO list. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent

Re: [HACKERS] scanner/parser minimization

2013-03-08 Thread Bruce Momjian
On Thu, Feb 28, 2013 at 04:09:11PM -0500, Tom Lane wrote: Robert Haas robertmh...@gmail.com writes: A whole lot of those state transitions are attributable to states which have separate transitions for each of many keywords. Yeah, that's no surprise. The idea that's been in the back of

Re: [HACKERS] Duplicate JSON Object Keys

2013-03-08 Thread Hannu Krosing
On 03/08/2013 11:03 PM, Andrew Dunstan wrote: On 03/08/2013 04:42 PM, Andrew Dunstan wrote: So my order of preference for the options would be: 1. Have the JSON type collapse objects so the last instance of a key wins and is actually stored 2. Throw an error when a JSON type has

Re: [HACKERS] Duplicate JSON Object Keys

2013-03-08 Thread Andrew Dunstan
On 03/08/2013 06:37 PM, Hannu Krosing wrote: I suspect that 99.98% of the time we will get valid and unique JS Object serializations or equivalent as input to json_in() If we want the getter functions to handle the loose JSON to Object conversion side assuming our stored JSON can

Re: [HACKERS] Index Unqiueness

2013-03-08 Thread Bruce Momjian
On Fri, Mar 8, 2013 at 10:26:21AM -0500, Robert Haas wrote: overhead seems badly overpriced for insert-only tables. These are not fundamental truths of the universe, or even of PostgreSQL; they are specific consequences of the representation we've chosen for heaps. Many of them are things

Re: [HACKERS] Enabling Checksums

2013-03-08 Thread Greg Stark
On Fri, Mar 8, 2013 at 5:46 PM, Josh Berkus j...@agliodbs.com wrote: After some examination of the systems involved, we conculded that the issue was the FreeBSD drivers for the new storage, which were unstable and had custom source patches. However, without PostgreSQL checksums, we couldn't

Re: [HACKERS] Why do we still perform a check for pre-sorted input within qsort variants?

2013-03-08 Thread Greg Stark
On Fri, Mar 8, 2013 at 7:43 PM, Dann Corbit dcor...@connx.com wrote: Checking for pre-sorted input will not make the routine faster on average. However, it prevents a common perverse case where the runtime goes quadratic, so sorting 10^6 elements will take K*10^12th operations when the bad

Re: [HACKERS] [pgsql-advocacy] Call for Google Summer of Code mentors, admins

2013-03-08 Thread Josh Berkus
Thom. I don't mind being an admin again. Can you gather together all of the projects suggested on this thread and use them to create updated text for the GSOC page? If you don't have web repo access, I can create a patch, but if you can do the text, that would be a big help. -- Josh Berkus

Re: [HACKERS] Enabling Checksums

2013-03-08 Thread Greg Smith
On 3/8/13 3:38 AM, Heikki Linnakangas wrote: See https://www.kernel.org/doc/Documentation/block/data-integrity.txt That includes an interesting comment that's along the lines of the MySQL checksum tests already mentioned: The 16-bit CRC checksum mandated by both the SCSI and SATA specs is

Re: [HACKERS] Duplicate JSON Object Keys

2013-03-08 Thread Noah Misch
On Fri, Mar 08, 2013 at 10:34:20PM +0100, Hannu Krosing wrote: On 03/08/2013 10:01 PM, Alvaro Herrera wrote: Hannu Krosing escribi?: If it does not meet these semantic constraints, then it is not really JSON - it is merely JSON-like. Is it wrong? The standard cited says SHOULD, not MUST.

Re: [HACKERS] Request for vote to move forward with recovery.conf overhaul

2013-03-08 Thread Michael Paquier
On Wed, Mar 6, 2013 at 2:08 PM, Michael Paquier michael.paqu...@gmail.comwrote: Thanks for taking time in typing a complete summary of the situation. That really helps. On Mon, Mar 4, 2013 at 11:25 AM, Greg Smith g...@2ndquadrant.com wrote: On 1/23/13 6:36 AM, Michael Paquier wrote: The

Re: [HACKERS] Request for vote to move forward with recovery.conf overhaul

2013-03-08 Thread Alvaro Herrera
Michael Paquier escribió: On Wed, Mar 6, 2013 at 2:08 PM, Michael Paquier michael.paqu...@gmail.comwrote: Looks good to me too. Based on the patch I already sent before, there are a couple of things missing: - There are no pg_ctl standby/recover commands implemented yet (no that

Re: [HACKERS] Request for vote to move forward with recovery.conf overhaul

2013-03-08 Thread Greg Smith
On 3/8/13 10:34 PM, Michael Paquier wrote: This patch is still in the current commit fest. Any objections in marking it as returned with feedback and put it in the next commit fest? There are currently 20 Needs Review and 14 Waiting on Author things left in the queue, so it's not quite that

Re: [HACKERS] Request for vote to move forward with recovery.conf overhaul

2013-03-08 Thread Michael Paquier
On Sat, Mar 9, 2013 at 1:20 PM, Greg Smith g...@2ndquadrant.com wrote: There are currently 20 Needs Review and 14 Waiting on Author things left in the queue, so it's not quite that there's no time left. There really isn't very much left to do on this. The rough consensus idea from before

Re: [HACKERS] Fix pgstattuple/pgstatindex to use regclass-type as the argument

2013-03-08 Thread Satoshi Nagayasu
(2013/03/05 22:46), Robert Haas wrote: On Sun, Mar 3, 2013 at 5:32 PM, Tom Lane t...@sss.pgh.pa.us wrote: Maybe this is acceptable collateral damage. I don't know. But we definitely stand a chance of breaking applications if we change pgstatindex like this. It might be better to invent a