Re: [HACKERS] Platform-dependent(?) failure in timeout handling

2013-11-27 Thread Martijn van Oosterhout
On Tue, Nov 26, 2013 at 06:50:28PM -0500, Tom Lane wrote: I believe the reason for this is the mechanism that I speculated about in that previous thread. The platform is blocking SIGALRM while it executes handle_sig_alarm(), and that calls LockTimeoutHandler() which does kill(MyProcPid,

Re: [HACKERS] Incomplete freezing when truncating a relation during vacuum

2013-11-27 Thread Heikki Linnakangas
On 11/27/13 01:21, Andres Freund wrote: On 2013-11-26 13:32:44 +0100, Andres Freund wrote: This seems to be the case since b4b6923e03f4d29636a94f6f4cc2f5cf6298b8c8. I suggest we go back to using scan_all to determine whether we can set new_frozen_xid. That's a slight pessimization when we scan

Re: [HACKERS] Platform-dependent(?) failure in timeout handling

2013-11-27 Thread Andres Freund
Hi, On 2013-11-26 18:50:28 -0500, Tom Lane wrote: I don't know how many platforms block signals during handlers in this way, but I'm seeing it on Linux (RHEL6.5 to be exact) and we know that at least OpenBSD acts likewise, so that's a pretty darn large chunk of the world. Just as a datapoint,

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2013-11-27 Thread Peter Geoghegan
On Tue, Nov 26, 2013 at 8:19 PM, Peter Geoghegan p...@heroku.com wrote: There are some visibility-related race conditions even still I also see this, sandwiched between the very many deadlock detected errors recorded over 6 or so hours (this is in chronological order, with no ERRORs omitted

Re: [HACKERS] [GENERAL] pg_upgrade ?deficiency

2013-11-27 Thread Karsten Hilbert
On Tue, Nov 26, 2013 at 03:25:44PM -0800, Kevin Grittner wrote: doc patch? Instead of the fix you mean, or with it?  I don't see what we would change in the docs for the fix; the alternative might be to document that pg_dumpall output will fail to restore if any database (or the restoring

Re: [HACKERS] Incomplete freezing when truncating a relation during vacuum

2013-11-27 Thread Andres Freund
On 2013-11-27 11:01:55 +0200, Heikki Linnakangas wrote: On 11/27/13 01:21, Andres Freund wrote: On 2013-11-26 13:32:44 +0100, Andres Freund wrote: This seems to be the case since b4b6923e03f4d29636a94f6f4cc2f5cf6298b8c8. I suggest we go back to using scan_all to determine whether we can set

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2013-11-27 Thread Andres Freund
On 2013-11-27 01:09:49 -0800, Peter Geoghegan wrote: On Tue, Nov 26, 2013 at 8:19 PM, Peter Geoghegan p...@heroku.com wrote: There are some visibility-related race conditions even still I also see this, sandwiched between the very many deadlock detected errors recorded over 6 or so hours

Re: [HACKERS] PostgreSQL Service on Windows does not start. ~ is not a valid Win32 application

2013-11-27 Thread Naoya Anzai
Hi, Rajeev I tested the latest patch. My observation is: If we give relative data directory path while registering the service, then service start fails. But same works if the data directory is absolute path. Looks like an existing issue. May be we need to internally

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2013-11-27 Thread Peter Geoghegan
On Wed, Nov 27, 2013 at 1:20 AM, Andres Freund and...@2ndquadrant.com wrote: Even the deadlock detected errors might be a fkey-locking issue. Bug #8434, but that's really hard to know without more details. Thanks, I was aware of that but didn't make the connection. I've written a test-case

[HACKERS] Name type in postgres

2013-11-27 Thread mohsen soodkhah mohammadi
hello. I want do understand that did can we have name without null-terminator? in src/backend/util/adt/name.c in nameout() function is: Names = PG_GETARG_NAME(0); PG_RETURN_CSTRING(pstrdup(NameStr(*s))); what do the pstrdup() function? do this function create string from name with

Re: [HACKERS] Assertions in PL/PgSQL

2013-11-27 Thread Pavel Stehule
2013/11/27 Peter Eisentraut pete...@gmx.net On Tue, 2013-11-19 at 10:40 -0500, Robert Haas wrote: I think the goal was to get to RAISE ASSERT WHEN ...; then, if assertions are off, you do nothing; if they're on, you error. IF condition THEN RAISE... isn't a suitable surrogate in that

Re: [HACKERS] Status of FDW pushdowns

2013-11-27 Thread Shigeru Hanada
Hi Merlin, 2013/11/22 Merlin Moncure mmonc...@gmail.com: On Thu, Nov 21, 2013 at 6:43 PM, Shigeru Hanada shigeru.han...@gmail.com wrote: 2013/11/22 Tom Lane t...@sss.pgh.pa.us: Merlin Moncure mmonc...@gmail.com writes: On Thu, Nov 21, 2013 at 9:05 AM, Bruce Momjian br...@momjian.us wrote: I

Re: [HACKERS] Get more from indices.

2013-11-27 Thread Etsuro Fujita
Kyotaro HORIGUCHI wrote: * In get_relation_info(), the patch determines the branch condition keyattno != ObjectIdAttributeNumber. I fail to understand the meaning of this branch condition. Could you explain about it? Literally answering, it means oid cannot be NULL (if it exists).

Re: [HACKERS] PostgreSQL Service on Windows does not start. ~ is not a valid Win32 application

2013-11-27 Thread Rajeev rastogi
On 27 November 2013, Naoya Anzai wrote: Hi, Rajeev I tested the latest patch. My observation is: If we give relative data directory path while registering the service, then service start fails. But same works if the data directory is absolute path. Looks like an existing

Re: [HACKERS] [PATCH] Add transforms feature

2013-11-27 Thread Hannu Krosing
On 11/15/2013 05:04 PM, Dimitri Fontaine wrote: Hi, Peter Eisentraut pete...@gmx.net writes: Rebased patch. No changes except that merge conflicts were resolved, and I had to add some Data::Dumper tweaks to the regression tests so that the results came out in consistent order on different

Re: [HACKERS] Incomplete freezing when truncating a relation during vacuum

2013-11-27 Thread Heikki Linnakangas
On 11/27/13 11:15, Andres Freund wrote: On 2013-11-27 11:01:55 +0200, Heikki Linnakangas wrote: On 11/27/13 01:21, Andres Freund wrote: On 2013-11-26 13:32:44 +0100, Andres Freund wrote: This seems to be the case since b4b6923e03f4d29636a94f6f4cc2f5cf6298b8c8. I suggest we go back to using

Re: [HACKERS] PL/Python: domain over array support

2013-11-27 Thread Marko Kreen
On Tue, Nov 26, 2013 at 07:12:00PM -0200, Rodolfo Campero wrote: 2013/11/26 Heikki Linnakangas hlinnakan...@vmware.com Oops, sorry about that. Fixed. Maybe be you forgot to modify plpython_types_3.out Yes. Heikki, please fix plpython_types_3.out too. See attached patch. -- marko diff

Re: [HACKERS] Incomplete freezing when truncating a relation during vacuum

2013-11-27 Thread Andres Freund
On 2013-11-27 13:56:58 +0200, Heikki Linnakangas wrote: Ok, committed and backpatched that. Thanks. I wonder if we need to integrate any mitigating logic? Currently the corruption may only become apparent long after it occurred, that's pretty bad. And instructing people run a vacuum after

Re: [HACKERS] PL/Python: domain over array support

2013-11-27 Thread Heikki Linnakangas
On 11/27/13 14:15, Marko Kreen wrote: On Tue, Nov 26, 2013 at 07:12:00PM -0200, Rodolfo Campero wrote: 2013/11/26 Heikki Linnakangas hlinnakan...@vmware.com Oops, sorry about that. Fixed. Maybe be you forgot to modify plpython_types_3.out Yes. Heikki, please fix plpython_types_3.out too.

Re: [HACKERS] Re: [BUGS] BUG #7873: pg_restore --clean tries to drop tables that don't exist

2013-11-27 Thread Pavel Stehule
I'll prepare patch 2013/11/27 Tom Lane t...@sss.pgh.pa.us Dean Rasheed dean.a.rash...@gmail.com writes: Actually the IF EXISTS in DROP TABLE now applies to the schema as well. Unfortunately there is currently no consistency across the various DROP commands --- some tolerate a

Re: [HACKERS] Incomplete freezing when truncating a relation during vacuum

2013-11-27 Thread Heikki Linnakangas
On 11/27/13 14:11, Andres Freund wrote: On 2013-11-27 13:56:58 +0200, Heikki Linnakangas wrote: Afaics the likelihood of the issue occuring on non-all-visible pages is pretty low, since they'd need to be skipped due to lock contention repeatedly. Hmm. If a page has its visibility-map flag

Re: [HACKERS] Incomplete freezing when truncating a relation during vacuum

2013-11-27 Thread Andres Freund
On 2013-11-27 14:45:25 +0200, Heikki Linnakangas wrote: On 11/27/13 14:11, Andres Freund wrote: I don't think this warning is likely to be hit as the code stands - heap_page_prune() et. al. will have removed all dead tuples already, right and so has_dead_tuples won't be set. It might be a

Re: [HACKERS] Status of FDW pushdowns

2013-11-27 Thread Alvaro Herrera
Shigeru Hanada escribió: SQL/MED standard doesn't say much about PASS THROUGH mode, especially about interaction between client. Besides it, I think it would be nice to allow arbitrary FDW as backend of dblink interface like this: postgres= SELECT dblink_connect('con1', 'server name of an

Re: [HACKERS] Traffic jams in fn_extra

2013-11-27 Thread Greg Stark
On Sun, Nov 24, 2013 at 4:21 PM, Simon Riggs si...@2ndquadrant.com wrote: Why do you need to do this dance with fn_extra? It's possible to allocate a hash table in a Transaction-lifetime memory context on first call into a function then cache things there. fn_extra gives a handle per

Re: [HACKERS] Handling GIN incomplete splits

2013-11-27 Thread Heikki Linnakangas
On 11/22/13 15:04, Michael Paquier wrote: On Thu, Nov 21, 2013 at 9:44 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: Here's a new version. To ease the review, I split the remaining patch again into two, where the first patch is just yet more refactoring. I also fixed some bugs in WAL

Re: [HACKERS] Status of FDW pushdowns

2013-11-27 Thread Dimitri Fontaine
Alvaro Herrera alvhe...@2ndquadrant.com writes: Seems to me that if you want to read remote tables without creating a foreign table, you could define them locally using something like the WITH syntax and then use them normally in the rest of the query. I guess what's needed here is a kind of

Re: [HACKERS] [GENERAL] pg_upgrade ?deficiency

2013-11-27 Thread Kevin Grittner
Bruce Momjian br...@momjian.us wrote: On Tue, Nov 26, 2013 at 03:25:44PM -0800, Kevin Grittner wrote: Bruce Momjian br...@momjian.us wrote: How are we handling breakage of pg_dump, not pg_dumpall? That was discussed.  Do you have something to add? I am confused what we are patching.  Are

Re: [HACKERS] Performance Improvement by reducing WAL for Update Operation

2013-11-27 Thread Robert Haas
On Wed, Nov 27, 2013 at 12:56 AM, Amit Kapila amit.kapil...@gmail.com wrote: - There is a comment TODO: It would be nice to behave like the history and the source strings were concatenated, so that you could compress using the new data, too. If we're not already doing that, then how are we

Re: [HACKERS] Shave a few instructions from child-process startup sequence

2013-11-27 Thread Robert Haas
On Tue, Nov 26, 2013 at 10:12 PM, Gurjeet Singh gurj...@singh.im wrote: On Tue, Nov 26, 2013 at 9:42 PM, Peter Eisentraut pete...@gmx.net wrote: src/backend/postmaster/postmaster.c:2255: indent with spaces. +else src/backend/postmaster/postmaster.c:2267: indent with spaces. +

Re: [HACKERS] new unicode table border styles for psql

2013-11-27 Thread Pavel Stehule
2013/11/26 Pavel Stehule pavel.steh...@gmail.com 2013/11/26 Peter Eisentraut pete...@gmx.net On 11/22/13, 3:26 AM, Pavel Stehule wrote: website is related to patch for 9.3 (I add note there) patch for 9.4 is fixed - and now with small doc I think it would help if we considered the

Re: [HACKERS] Extension Templates S03E11

2013-11-27 Thread Vik Fearing
On 11/26/2013 10:07 PM, Dimitri Fontaine wrote: What I now think we should do is only grant superusers the privileges to install an extension from a template they own or is owned by another superuser. Say what? Superusers bypass all privileges by definition. -- Vik -- Sent via

Re: [HACKERS] doPickSplit stack buffer overflow in XLogInsert?

2013-11-27 Thread Kevin Grittner
Andres Freund and...@2ndquadrant.com wrote: On 2013-11-26 14:14:38 -0800, Kevin Grittner wrote: I happened to build in a shell that was still set up for the clang address sanitizer, and got the attached report.  On a rerun it was repeatable.  XLogInsert() seems to read past the end of a

Re: [HACKERS] Platform-dependent(?) failure in timeout handling

2013-11-27 Thread Kevin Grittner
Tom Lane t...@sss.pgh.pa.us wrote: 3. Establish a coding rule that if you catch an error with PG_TRY() and don't re-throw it, you have to unblock signals in your PG_CATCH block. Could that be done in the PG_END_TRY macro? -- Kevin Grittner EDB: http://www.enterprisedb.com The Enterprise

Re: [HACKERS] Performance Improvement by reducing WAL for Update Operation

2013-11-27 Thread Amit Kapila
On Wed, Nov 27, 2013 at 7:35 PM, Robert Haas robertmh...@gmail.com wrote: On Wed, Nov 27, 2013 at 12:56 AM, Amit Kapila amit.kapil...@gmail.com wrote: The basic idea is that you use a rolling hash function to divide up the history data into chunks of a given average size. So we scan the

[HACKERS] [HITB-Announce] #HITB2014AMS Call for Papers Now Open

2013-11-27 Thread Hafez Kamal
Hi everyone - The Call for Papers for the 5th annual HITB Security Conference in Amsterdam is now open. #HITB2014AMS takes place at the Beurs van Berlage from the 27th - 30th of May 2014. The official conference hotel for the event is the Hilton DoubleTree. As always we start with 2-days of

Re: [HACKERS] Name type in postgres

2013-11-27 Thread Tom Lane
mohsen soodkhah mohammadi mohsensoodk...@gmail.com writes: I want do understand that did can we have name without null-terminator? No. Possibly that was the original intention, but for many years the system has enforced that Name contains at least one trailing null byte, ie the maximum usable

Re: [HACKERS] Status of FDW pushdowns

2013-11-27 Thread Merlin Moncure
On Wed, Nov 27, 2013 at 4:20 AM, Shigeru Hanada shigeru.han...@gmail.com wrote: Hi Merlin, 2013/11/22 Merlin Moncure mmonc...@gmail.com: On Thu, Nov 21, 2013 at 6:43 PM, Shigeru Hanada shigeru.han...@gmail.com wrote: 2013/11/22 Tom Lane t...@sss.pgh.pa.us: Merlin Moncure mmonc...@gmail.com

Re: [HACKERS] Status of FDW pushdowns

2013-11-27 Thread Shigeru Hanada
2013/11/27 Dimitri Fontaine dimi...@2ndquadrant.fr: Alvaro Herrera alvhe...@2ndquadrant.com writes: Seems to me that if you want to read remote tables without creating a foreign table, you could define them locally using something like the WITH syntax and then use them normally in the rest of

Re: [HACKERS] Modify the DECLARE CURSOR command tag depending on the scrollable flag

2013-11-27 Thread Boszormenyi Zoltan
2013-11-23 22:01 keltezéssel, Tom Lane írta: Boszormenyi Zoltan z...@cybertec.at writes: Attached is the patch that modified the command tag returned by the DECLARE CURSOR command. It returns DECLARE SCROLL CURSOR or DECLARE NO SCROLL CURSOR depending on the cursor's scrollable flag that can be

Re: [HACKERS] doPickSplit stack buffer overflow in XLogInsert?

2013-11-27 Thread Noah Misch
On Wed, Nov 27, 2013 at 06:23:38AM -0800, Kevin Grittner wrote: Andres Freund and...@2ndquadrant.com wrote: On 2013-11-26 14:14:38 -0800, Kevin Grittner wrote: I happened to build in a shell that was still set up for the clang address sanitizer, and got the attached report.  On a rerun it

Re: [HACKERS] Errors on missing pg_subtrans/ files with 9.3

2013-11-27 Thread Alvaro Herrera
Alvaro Herrera escribió: Andres Freund escribió: This seems simple to handle by adding the check you propose to the loop. Basically if the xmax doesn't match the xmin, we reached the end, there's nothing more to lock and we can return success without any further work: As mentioned in the

Re: [HACKERS] Status of FDW pushdowns

2013-11-27 Thread Dimitri Fontaine
Shigeru Hanada shigeru.han...@gmail.com writes: I'm sorry but I don't see the point here. Do you mean that user executes CREATE FOREIGN VIEW in advance and uses the view in a Yes that's what I mean. I think it's nice to support executing ad-hoc remote query written in the syntax which is

Re: [HACKERS] lock on object is already held

2013-11-27 Thread Tom Lane
Daniel Wood dw...@salesforce.com writes: Does the original version of my stress test not repro the problem on 9.2? [ tries it ... ] No, it doesn't, or at least the MTBF is a couple orders of magnitude better than on 9.3. Another odd thing (seen with my short version as well as your original)

Re: [HACKERS] Status of FDW pushdowns

2013-11-27 Thread Atri Sharma
I guess the view query would have to be validated by the FDW, which would just receive a text. +1 This is not exactly in context, but I and David Fetter discussed recently how we could do similar thing. This would work,but how can we do it for FDWs which do not parse SQL? Am I missing

Re: [HACKERS] Status of FDW pushdowns

2013-11-27 Thread Dimitri Fontaine
Atri Sharma atri.j...@gmail.com writes: This would work,but how can we do it for FDWs which do not parse SQL? Am I missing something here? Worst case: CREATE FOREIGN VIEW foo AS $$ whatever syntax is accepted on the other side $$; Regards, -- Dimitri Fontaine http://2ndQuadrant.fr

Re: [HACKERS] Status of FDW pushdowns

2013-11-27 Thread Atri Sharma
On Wed, Nov 27, 2013 at 11:08 PM, Dimitri Fontaine dimi...@2ndquadrant.fr wrote: Atri Sharma atri.j...@gmail.com writes: This would work,but how can we do it for FDWs which do not parse SQL? Am I missing something here? Worst case: CREATE FOREIGN VIEW foo AS $$ whatever syntax is

Re: [HACKERS] Handling GIN incomplete splits

2013-11-27 Thread Jeff Janes
On Wed, Nov 13, 2013 at 8:49 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: Here's another part of my crusade against xlog cleanup routines. This series of patches gets rid of the gin_cleanup() function, which is currently used to finish splits of GIN b-tree pages, if the system

Re: [HACKERS] Handling GIN incomplete splits

2013-11-27 Thread Heikki Linnakangas
On 11/22/13 15:04, Michael Paquier wrote: 2) post recovery cleanup: - OK, so roughly the soul of this patch is to change the update mechanism for a left child gin page so as the parent split is always done first before any new data is inserted in this child. And this ensures that we can remove

Re: [HACKERS] Platform-dependent(?) failure in timeout handling

2013-11-27 Thread Tom Lane
Kevin Grittner kgri...@ymail.com writes: Tom Lane t...@sss.pgh.pa.us wrote: 3. Establish a coding rule that if you catch an error with PG_TRY() and don't re-throw it, you have to unblock signals in your PG_CATCH block. Could that be done in the PG_END_TRY macro? Interesting idea ... [

Re: [HACKERS] Modify the DECLARE CURSOR command tag depending on the scrollable flag

2013-11-27 Thread Tom Lane
Boszormenyi Zoltan z...@cybertec.at writes: 2013-11-23 22:01 keltezéssel, Tom Lane írta: Boszormenyi Zoltan z...@cybertec.at writes: Attached is the patch that modified the command tag returned by the DECLARE CURSOR command. It returns DECLARE SCROLL CURSOR or DECLARE NO SCROLL CURSOR

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2013-11-27 Thread Peter Geoghegan
On Wed, Nov 27, 2013 at 1:09 AM, Peter Geoghegan p...@heroku.com wrote: Since it took me a relatively long time to recreate this, it may not be trivial to do so. Unless you don't think it's useful to do so, I'm going to give this test a full 24 hours, just in case it shows up anything else

Re: [HACKERS] Status of FDW pushdowns

2013-11-27 Thread David Fetter
On Wed, Nov 27, 2013 at 10:29:34AM -0300, Alvaro Herrera wrote: Shigeru Hanada escribió: SQL/MED standard doesn't say much about PASS THROUGH mode, especially about interaction between client. Besides it, I think it would be nice to allow arbitrary FDW as backend of dblink interface like

[HACKERS] Should we improve documentation on isolation levels?

2013-11-27 Thread AK
I am not sure if i am posting in the right place - correct me if I am wrong. The following is not precise: 13.2.1. Read Committed Isolation Level Also note that two successive SELECT commands can see different data, even though they are within a single transaction, if other transactions commit

Re: [HACKERS] doPickSplit stack buffer overflow in XLogInsert?

2013-11-27 Thread Kevin Grittner
Noah Misch n...@leadboat.com wrote: (Kevin, I saw no attachment.) Apologies.  Trying again. The threat is that rounding the read size up to the next MAXALIGN would cross into an unreadable memory page, resulting in a SIGSEGV.  Every palloc chunk has MAXALIGN'd size under the hood, so the

Re: [HACKERS] logical changeset generation v6.7

2013-11-27 Thread Fabrízio de Royes Mello
On Thu, Nov 14, 2013 at 11:46 AM, Andres Freund and...@2ndquadrant.com wrote: On 2013-11-12 18:50:33 +0100, Andres Freund wrote: You've actually changed the meaning of this section (and not in a good way): be set at server start. varnamewal_level/ must be set -to

Re: [HACKERS] Modify the DECLARE CURSOR command tag depending on the scrollable flag

2013-11-27 Thread Boszormenyi Zoltan
2013-11-27 19:16 keltezéssel, Tom Lane írta: Boszormenyi Zoltan z...@cybertec.at writes: 2013-11-23 22:01 keltezéssel, Tom Lane írta: Boszormenyi Zoltan z...@cybertec.at writes: Attached is the patch that modified the command tag returned by the DECLARE CURSOR command. It returns DECLARE

Re: [HACKERS] Should we improve documentation on isolation levels?

2013-11-27 Thread Kevin Grittner
AK alk...@gmail.com wrote: The following is not precise: 13.2.1. Read Committed Isolation Level Also note that two successive SELECT commands can see different data, even though they are within a single transaction, if other transactions commit changes during execution of the first SELECT.

Re: [HACKERS] Modify the DECLARE CURSOR command tag depending on the scrollable flag

2013-11-27 Thread Alvaro Herrera
Boszormenyi Zoltan escribió: If you consider all these: - certain combinations of query and DECLARE stmt flags fail; - adding NO SCROLL is breaking backward compatibility; - the readahead code has to really know whether the cursor is scrollable so it can behave just like the server;

Re: [HACKERS] Incomplete freezing when truncating a relation during vacuum

2013-11-27 Thread Noah Misch
How would you characterize the chances of this happening with default *vacuum_freeze_*_age settings? Offhand, it seems you would need to encounter this bug during each of ~10 generations of autovacuum_freeze_max_age before the old rows actually become invisible. On Wed, Nov 27, 2013 at

Re: [HACKERS] Should we improve documentation on isolation levels?

2013-11-27 Thread AK
I concur - the documentation is not incorrect, it is just incomplete. -- View this message in context: http://postgresql.1045698.n5.nabble.com/Should-we-improve-documentation-on-isolation-levels-tp5780629p5780636.html Sent from the PostgreSQL - hackers mailing list archive at Nabble.com. --

Re: [HACKERS] Status of FDW pushdowns

2013-11-27 Thread Atri Sharma
On Thu, Nov 28, 2013 at 12:54 AM, David Fetter da...@fetter.org wrote: On Wed, Nov 27, 2013 at 10:29:34AM -0300, Alvaro Herrera wrote: Shigeru Hanada escribió: SQL/MED standard doesn't say much about PASS THROUGH mode, especially about interaction between client. Besides it, I think it

Re: [HACKERS] doPickSplit stack buffer overflow in XLogInsert?

2013-11-27 Thread Noah Misch
On Wed, Nov 27, 2013 at 11:38:23AM -0800, Kevin Grittner wrote: Noah Misch n...@leadboat.com wrote: The threat is that rounding the read size up to the next MAXALIGN would cross into an unreadable memory page, resulting in a SIGSEGV.  Every palloc chunk has MAXALIGN'd size under the hood,

Re: [HACKERS] Should we improve documentation on isolation levels?

2013-11-27 Thread Kevin Grittner
AK alk...@gmail.com wrote: the documentation is not incorrect, it is just incomplete. Yeah, that's exactly how I saw it.  :-) Docs changed on master only.  Thanks for the report! -- Kevin Grittner EDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via

Re: [HACKERS] Modify the DECLARE CURSOR command tag depending on the scrollable flag

2013-11-27 Thread Tom Lane
Boszormenyi Zoltan z...@cybertec.at writes: If you consider all these: - certain combinations of query and DECLARE stmt flags fail; - adding NO SCROLL is breaking backward compatibility; - the readahead code has to really know whether the cursor is scrollable so it can behave just like

[HACKERS] bytea_ouput = escape vs encode(byte, 'escape')

2013-11-27 Thread Jim Nasby
I'm wondering why bytes_output = escape produces different output than encode(byte, 'escape') does. Is this intentional? If so, why? cnuapp_prod@postgres=# select e'\r'::bytea AS cr, e'\n'::bytea AS lf; cr | lf --+-- \x0d | \x0a (1 row) cnuapp_prod@postgres=# set bytea_output =

Re: [HACKERS] Suggestion: Issue warning when calling SET TRANSACTION outside transaction block

2013-11-27 Thread Robert Haas
On Tue, Nov 26, 2013 at 5:02 PM, Bruce Momjian br...@momjian.us wrote: On Tue, Nov 26, 2013 at 01:58:04PM -0300, Alvaro Herrera wrote: Bruce Momjian escribió: On Tue, Nov 26, 2013 at 11:22:39AM -0500, Tom Lane wrote: Uh, I ended up mentioning no effect to highlight it does nothing,

Re: [HACKERS] Status of FDW pushdowns

2013-11-27 Thread David Fetter
On Thu, Nov 28, 2013 at 01:29:46AM +0530, Atri Sharma wrote: On Thu, Nov 28, 2013 at 12:54 AM, David Fetter da...@fetter.org wrote: On Wed, Nov 27, 2013 at 10:29:34AM -0300, Alvaro Herrera wrote: Shigeru Hanada escribió: SQL/MED standard doesn't say much about PASS THROUGH mode,

Re: [HACKERS] bytea_ouput = escape vs encode(byte, 'escape')

2013-11-27 Thread David Johnston
Jim Nasby-2 wrote I'm wondering why bytes_output = escape produces different output than encode(byte, 'escape') does. Is this intentional? If so, why? cnuapp_prod@postgres=# select e'\r'::bytea AS cr, e'\n'::bytea AS lf; cr | lf --+-- \x0d | \x0a (1 row)

Re: [HACKERS] Modify the DECLARE CURSOR command tag depending on the scrollable flag

2013-11-27 Thread Peter Eisentraut
On 11/27/13, 2:49 PM, Alvaro Herrera wrote: Would it work to have a function of some sort to which you give a cursor name and it returns whether it is scrollable or not? That might make sense. I think this case is similar to the question whether a view is updatable. You wouldn't put that

Re: [HACKERS] Modify the DECLARE CURSOR command tag depending on the scrollable flag

2013-11-27 Thread Peter Eisentraut
On 11/27/13, 3:47 PM, Tom Lane wrote: Given these considerations, I think it'd be better to allow explicit application control over whether read-ahead happens for a particular query. And I have no problem whatsoever with requiring that the cursor be explicitly marked SCROLL or NO SCROLL

Re: [HACKERS] 9.2.1 index-only scans : abnormal heap fetches after VACUUM FULL

2013-11-27 Thread Bruce Momjian
On Sat, Jan 12, 2013 at 02:14:03PM -0500, Kevin Grittner wrote: Amit Kapila wrote: On Thursday, January 10, 2013 6:09 AM Josh Berkus wrote: Surely VACUUM FULL should rebuild the visibility map, and make tuples in the new relation all-visible, no? Certainly it seems odd to me that

Re: [HACKERS] [GENERAL] pg_upgrade ?deficiency

2013-11-27 Thread Bruce Momjian
On Wed, Nov 27, 2013 at 06:05:13AM -0800, Kevin Grittner wrote: Bruce Momjian br...@momjian.us wrote: On Tue, Nov 26, 2013 at 03:25:44PM -0800, Kevin Grittner wrote: Bruce Momjian br...@momjian.us wrote: How are we handling breakage of pg_dump, not pg_dumpall? That was discussed.  Do

Re: [HACKERS] Incomplete freezing when truncating a relation during vacuum

2013-11-27 Thread Andres Freund
On 2013-11-27 14:53:27 -0500, Noah Misch wrote: How would you characterize the chances of this happening with default *vacuum_freeze_*_age settings? Offhand, it seems you would need to encounter this bug during each of ~10 generations of autovacuum_freeze_max_age before the old rows actually

Re: [HACKERS] doPickSplit stack buffer overflow in XLogInsert?

2013-11-27 Thread Andres Freund
On 2013-11-27 15:29:24 -0500, Noah Misch wrote: If you are confident that neither of these is a real risk, I'll relax about this. If there is a real risk, I'm not seeing it. Me neither. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/

Re: [HACKERS] Suggestion: Issue warning when calling SET TRANSACTION outside transaction block

2013-11-27 Thread Bruce Momjian
On Wed, Nov 27, 2013 at 03:59:31PM -0500, Robert Haas wrote: On Tue, Nov 26, 2013 at 5:02 PM, Bruce Momjian br...@momjian.us wrote: On Tue, Nov 26, 2013 at 01:58:04PM -0300, Alvaro Herrera wrote: Bruce Momjian escribió: On Tue, Nov 26, 2013 at 11:22:39AM -0500, Tom Lane wrote: Uh, I

Re: [HACKERS] MultiXact pessmization in 9.3

2013-11-27 Thread Alvaro Herrera
Alvaro Herrera wrote: Correct. The only difficulty here is that we would need to pass down the fact that we know for certain that this is only a locking Multixact. There are some callers that go to it indirectly via MultiXactIdWait or MultiXactIdExpand, but now that I look I think it's fine

[HACKERS] Another bug introduced by fastpath patch

2013-11-27 Thread Tom Lane
In LockReleaseAll, we have this coding: for (partition = 0; partition NUM_LOCK_PARTITIONS; partition++) { LWLockIdpartitionLock = FirstLockMgrLock + partition; SHM_QUEUE *procLocks = (MyProc-myProcLocks[partition]); proclock = (PROCLOCK *)

Re: [HACKERS] Suggestion: Issue warning when calling SET TRANSACTION outside transaction block

2013-11-27 Thread Bruce Momjian
On Wed, Nov 27, 2013 at 04:44:02PM -0500, Bruce Momjian wrote: I could live with this: Issuing commandROLLBACK/ outside of a transaction block has no effect except emitting a warning. Proposed doc patch attached. -- Bruce Momjian br...@momjian.ushttp://momjian.us

Re: [HACKERS] [RFC] overflow checks optimized away

2013-11-27 Thread Bruce Momjian
On Mon, Sep 9, 2013 at 12:21:56PM -0400, Robert Haas wrote: On Sat, Sep 7, 2013 at 6:55 PM, Greg Stark st...@mit.edu wrote: Should these patches be applied? I have a copy of the program and was going to take care of this. When? 2.5 months later, status report? -- Bruce Momjian

Re: [HACKERS] Another bug introduced by fastpath patch

2013-11-27 Thread Andres Freund
On 2013-11-27 17:25:44 -0500, Tom Lane wrote: Or we could add a restriction to EligibleForRelationFastPath that restricts the fast-path mechanism to non-session locks, in which case we'd not need to make the zeroing contingent on allLocks either. I don't think we take any fast-path-eligible

Re: [HACKERS] MultiXact pessmization in 9.3

2013-11-27 Thread Andres Freund
On 2013-11-27 19:24:35 -0300, Alvaro Herrera wrote: One other thought is that MultiXactIdIsRunning and GetMultiXactIdMembers are public functions, so this patch would represent an API change in 9.3. I doubt any external modules would be relying on these functions, but there's so many care and

Re: [HACKERS] Errors on missing pg_subtrans/ files with 9.3

2013-11-27 Thread Andres Freund
On 2013-11-27 13:57:52 -0300, Alvaro Herrera wrote: Per bug report by J Smith in cadfupgc5bmtv-yg9znxv-vcfkb+jprqs7m2oesqxam_4z1j...@mail.gmail.com diagnosed by Andres Freund. Alvaro, do you see a way this could actually have caused J.'s problems? I thought about a few, but each

Re: [HACKERS] MultiXact bugs

2013-11-27 Thread Kevin Grittner
Andres Freund and...@2ndquadrant.com wrote: HeapTupleHeaderGetUpdateXid() ignores aborted updaters and returns InvalidTransactionId in that case, but HeapTupleSatisfiesVacuum() returns HEAPTUPLE_DELETE_IN_PROGRESS... That sure *sounds* like it should cause a problem for this code in

Re: [HACKERS] Modify the DECLARE CURSOR command tag depending on the scrollable flag

2013-11-27 Thread Tom Lane
Peter Eisentraut pete...@gmx.net writes: On 11/27/13, 3:47 PM, Tom Lane wrote: Given these considerations, I think it'd be better to allow explicit application control over whether read-ahead happens for a particular query. And I have no problem whatsoever with requiring that the cursor be

Re: [HACKERS] Incomplete freezing when truncating a relation during vacuum

2013-11-27 Thread Noah Misch
On Wed, Nov 27, 2013 at 10:43:05PM +0100, Andres Freund wrote: On 2013-11-27 14:53:27 -0500, Noah Misch wrote: How would you characterize the chances of this happening with default *vacuum_freeze_*_age settings? Offhand, it seems you would need to encounter this bug during each of ~10

Re: [HACKERS] MultiXact bugs

2013-11-27 Thread Andres Freund
On 2013-11-27 15:14:11 -0800, Kevin Grittner wrote: Andres Freund and...@2ndquadrant.com wrote: HeapTupleHeaderGetUpdateXid() ignores aborted updaters and returns InvalidTransactionId in that case, but HeapTupleSatisfiesVacuum() returns HEAPTUPLE_DELETE_IN_PROGRESS... That sure

Re: [HACKERS] MultiXact pessmization in 9.3

2013-11-27 Thread Tom Lane
Andres Freund and...@2ndquadrant.com writes: On 2013-11-27 19:24:35 -0300, Alvaro Herrera wrote: The other idea is to just not backpatch this. I think backpatching is a good idea, I have seen GetMultiXactIdMembers() + slru code take up 80% cpu in strange workloads. But it possibly might be a

Re: [HACKERS] [GENERAL] pg_upgrade ?deficiency

2013-11-27 Thread Kevin Grittner
Bruce Momjian br...@momjian.us wrote: On Wed, Nov 27, 2013 at 06:05:13AM -0800, Kevin Grittner wrote: Bruce Momjian br...@momjian.us wrote: I am confused what we are patching.  Are we patching pg_dump, pg_dumpall, or both? Just pg_dumpall.c. OK, there was a pg_dump patch earlier which we

Re: [HACKERS] MultiXact bugs

2013-11-27 Thread Kevin Grittner
Andres Freund and...@2ndquadrant.com wrote: On 2013-11-27 15:14:11 -0800, Kevin Grittner wrote: ... however, I have not been able to trigger that Assert even with gdb breakpoints at what I think are the right spots.  Any suggestions?  How far back is it true that the above

Re: [HACKERS] MultiXact bugs

2013-11-27 Thread Andres Freund
On 2013-11-27 15:42:11 -0800, Kevin Grittner wrote: Andres Freund and...@2ndquadrant.com wrote: What do you mean with how far back? What back-patching will be needed for a fix?  It sounds like 9.3? Yep. Greetings, Andres Freund -- Andres Freund

Re: [HACKERS] Status of FDW pushdowns

2013-11-27 Thread Dimitri Fontaine
Atri Sharma atri.j...@gmail.com writes: Can we add a function to the FDW API to define a SQL to foreign server side conversion? I think that's not tenable. Even if you limit the discussion to the postgres_fdw, some queries against past version will stop working against new version (keywords

Re: [HACKERS] Another bug introduced by fastpath patch

2013-11-27 Thread Tom Lane
Andres Freund and...@2ndquadrant.com writes: On 2013-11-27 17:25:44 -0500, Tom Lane wrote: Or we could add a restriction to EligibleForRelationFastPath that restricts the fast-path mechanism to non-session locks, in which case we'd not need to make the zeroing contingent on allLocks either.

Re: [HACKERS] Status of FDW pushdowns

2013-11-27 Thread Tom Lane
Dimitri Fontaine dimi...@2ndquadrant.fr writes: Atri Sharma atri.j...@gmail.com writes: Can we add a function to the FDW API to define a SQL to foreign server side conversion? I think that's not tenable. Even if you limit the discussion to the postgres_fdw, some queries against past version

Re: [HACKERS] Incomplete freezing when truncating a relation during vacuum

2013-11-27 Thread Andres Freund
On 2013-11-27 18:18:02 -0500, Noah Misch wrote: I think the likelihood of the problem affecting !all-visible pages is close to zero. Each vacuum will try to clean those, so they surely will get vacuumed at some point. I think the only way that could happen is if the

Re: [HACKERS] doPickSplit stack buffer overflow in XLogInsert?

2013-11-27 Thread Peter Eisentraut
On 11/26/13, 5:14 PM, Kevin Grittner wrote: I happened to build in a shell that was still set up for the clang address sanitizer, and got the attached report. On a rerun it was repeatable. XLogInsert() seems to read past the end of a variable allocated on the stack in doPickSplit(). I

[HACKERS] Proposed feature: Selective Foreign Keys

2013-11-27 Thread Tom Dunstan
Hi all! The Problem - One case that traditional SQL doesn't handle very well is when you have a child entity which can be attached to a number of different parent entities. Examples might be comments, tags or file attachments - we might have 20 different entities in the system

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2013-11-27 Thread Peter Geoghegan
On Wed, Nov 27, 2013 at 1:09 AM, Peter Geoghegan p...@heroku.com wrote: This, along with the already-discussed attempted to update invisible tuple forms a full account of unexpected ERRORs seen during the extended run of the test case, so far. Actually, it was slightly misleading of me to say

Re: [HACKERS] Another bug introduced by fastpath patch

2013-11-27 Thread Tom Lane
I wrote: We could still do this if we were willing to actually reject requests for session level locks on fast-path-eligible lock types. At the moment that costs us nothing really. If anyone ever did have a use case, we could consider adding the extra logic to support it. Nope, that *still*

Re: [HACKERS] [GENERAL] pg_upgrade ?deficiency

2013-11-27 Thread Bruce Momjian
On Wed, Nov 27, 2013 at 03:36:12PM -0800, Kevin Grittner wrote: Of the people who posted on this thread supporting that, I think Tom said it best: Tom Lane t...@sss.pgh.pa.us wrote: I'm inclined to agree with Kevin that this behavior is wrong and should be fixed (and back-patched), so

Re: [HACKERS] [GENERAL] pg_upgrade ?deficiency

2013-11-27 Thread Bruce Momjian
On Wed, Nov 27, 2013 at 03:36:12PM -0800, Kevin Grittner wrote: Tom Lane t...@sss.pgh.pa.us wrote: I'm inclined to agree with Kevin that this behavior is wrong and should be fixed (and back-patched), so far as pg_dumpall is concerned. pg_dumpall's charter is to be able to recreate a

  1   2   >