Re: [HACKERS] [DESIGN] ParallelAppend

2015-07-31 Thread Amit Kapila
On Tue, Jul 28, 2015 at 6:08 PM, Kouhei Kaigai kai...@ak.jp.nec.com wrote: On Tue, Jul 28, 2015 at 7:59 AM, Kouhei Kaigai kai...@ak.jp.nec.com wrote: -Original Message- From: pgsql-hackers-ow...@postgresql.org [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of

Re: [HACKERS] 64-bit XIDs again

2015-07-31 Thread Tom Lane
Josh Berkus j...@agliodbs.com writes: On 07/31/2015 02:46 PM, Heikki Linnakangas wrote: Well, sure, if you don't want the clog to grow arbitrarily large, then you need to freeze. And most people would want to freeze regularly, to keep the clog size in check. The point is that you wouldn't

Re: [HACKERS] [sqlsmith] Failed assertion in joinrels.c

2015-07-31 Thread Tom Lane
Andreas Seltenreich seltenre...@gmx.de writes: sqlsmith triggered the following assertion in master (c188204). TRAP: FailedAssertion(!(!bms_overlap(joinrelids, sjinfo-min_lefthand)), File: joinrels.c, Line: 500) Cool, I'll take a look. As usual, the query is against the regression

Re: [HACKERS] patch: prevent user from setting wal_buffers over 2GB bytes

2015-07-31 Thread Josh Berkus
On 07/31/2015 10:43 AM, Robert Haas wrote: On Thu, Jul 30, 2015 at 9:17 PM, Josh Berkus j...@agliodbs.com wrote: In guc.c, the maximum for wal_buffers is INT_MAX. However, wal_buffers is actually measured in 8KB buffers, not in bytes. This means that users are able to set wal_buffers 2GB.

[HACKERS] [sqlsmith] Failed assertion in joinrels.c

2015-07-31 Thread Andreas Seltenreich
Hi, sqlsmith triggered the following assertion in master (c188204). TRAP: FailedAssertion(!(!bms_overlap(joinrelids, sjinfo-min_lefthand)), File: joinrels.c, Line: 500) As usual, the query is against the regression database. It is rather unwieldy… I wonder if I should stop working on new

Re: [HACKERS] Proposal: backend niceness / session_priority

2015-07-31 Thread Amit Kapila
On Fri, Jul 31, 2015 at 3:48 AM, Jim Nasby jim.na...@bluetreble.com wrote: On 7/30/15 10:54 AM, Tom Lane wrote: =?ISO-8859-15?Q?Jos=E9_Luis_Tall=F3n?= jltal...@adv-solutions.net writes: Since PostgreSQL lacks the resource management capabilities of the Big Ones ( Resource Groups -

Re: [HACKERS] brin index vacuum versus transaction snapshots

2015-07-31 Thread Simon Riggs
On 30 July 2015 at 22:24, Tom Lane t...@sss.pgh.pa.us wrote: Alvaro Herrera alvhe...@2ndquadrant.com writes: Kevin Grittner wrote: If you run `make installcheck` against a cluster with default_transaction_isolation = 'repeatable read' you get one failure: I am tempted to say that we

Re: [HACKERS] brin index vacuum versus transaction snapshots

2015-07-31 Thread Kevin Grittner
Alvaro Herrera alvhe...@2ndquadrant.com wrote: Kevin Grittner wrote: If you run `make installcheck` against a cluster with default_transaction_isolation = 'repeatable read' you get one failure: I am tempted to say that we should just disallow to run vacuum on a table containing a brin

Re: [HACKERS] brin index vacuum versus transaction snapshots

2015-07-31 Thread Kevin Grittner
Simon Riggs si...@2ndquadrant.com wrote: On 30 July 2015 at 22:24, Tom Lane t...@sss.pgh.pa.us wrote: Alvaro Herrera alvhe...@2ndquadrant.com writes: Kevin Grittner wrote: If you run `make installcheck` against a cluster with default_transaction_isolation = 'repeatable read' you get one

Re: [HACKERS] creating extension including dependencies

2015-07-31 Thread Petr Jelinek
On 2015-07-31 03:03, Michael Paquier wrote: On Thu, Jul 30, 2015 at 10:58 PM, Petr Jelinek wrote: On 2015-07-27 15:18, Michael Paquier wrote: Something also has not been discussed yet: what to do with new_version and old_version (the options of CreateExtensionStmt)? As of now if those options

Re: [HACKERS] Encoding of early PG messages

2015-07-31 Thread Shay Rojansky
Thanks for the suggestions Tom. As I'm developing a general-purpose driver I can't do anything in PostgreSQL config, but it's a good workaround suggestion for users who encounter this error. Sending lc_messages in the startup packet could work, but if I understand correctly that setting combines

Re: [HACKERS] Doubt about AccessExclusiveLock in ALTER TABLE .. SET ( .. );

2015-07-31 Thread Michael Paquier
On Fri, Jul 31, 2015 at 10:05 PM, Simon Riggs si...@2ndquadrant.com wrote: On 31 July 2015 at 02:46, Michael Paquier michael.paqu...@gmail.com wrote: Added. I really don't know if my isolation tests are completely correct, is my first time writing this kind of tests. This patch size has

Re: [HACKERS] 64-bit XIDs again

2015-07-31 Thread Alexander Korotkov
On Fri, Jul 31, 2015 at 12:23 AM, Tom Lane t...@sss.pgh.pa.us wrote: But the elephant in the room is on-disk compatibility. There is absolutely no way that we can just change xmin/xmax to 64 bits without a disk format break. That seems problematic. But I'm not yet convinced that there is

Re: [HACKERS] Doubt about AccessExclusiveLock in ALTER TABLE .. SET ( .. );

2015-07-31 Thread Michael Paquier
On Fri, Jul 31, 2015 at 11:41 AM, Fabrízio de Royes Mello wrote: We usually don't compare lock values that way, i.e. there's not guaranteed to be a strict monotonicity between lock levels. I don't really agree with that policy, but it's nonetheless there. And how is the better way to compare

[HACKERS] Encoding of early PG messages

2015-07-31 Thread Shay Rojansky
Hi hackers. Developing Npgsql I've encountered the problem described in http://www.postgresql.org/message-id/20081223212414.gd3...@merkur.hilbert.loc: a German installation of PostgreSQL seems to respond to an incorrect password with a non-UTF8 encoding of the error messages, even if the startup

Re: [HACKERS] deparsing utility commands

2015-07-31 Thread Shulgin, Oleksandr
On Wed, Jul 29, 2015 at 12:44 PM, Shulgin, Oleksandr oleksandr.shul...@zalando.de wrote: On Tue, May 12, 2015 at 12:25 AM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: David Steele wrote: I have reviewed and tested this patch and everything looks good to me. It also looks like you

Re: [HACKERS] Encoding of early PG messages

2015-07-31 Thread Tom Lane
Shay Rojansky r...@roji.org writes: Developing Npgsql I've encountered the problem described in http://www.postgresql.org/message-id/20081223212414.gd3...@merkur.hilbert.loc: a German installation of PostgreSQL seems to respond to an incorrect password with a non-UTF8 encoding of the error

Re: [HACKERS] brin index vacuum versus transaction snapshots

2015-07-31 Thread Simon Riggs
On 31 July 2015 at 15:48, Kevin Grittner kgri...@ymail.com wrote: Reread the thread. I stand corrected. The error is being hit on a VACUUM command (which is not, of course in any explicit transaction). Seems like VACUUM only ever needs a read-committed snapshot. Whether it takes a

Re: [HACKERS] LWLock deadlock and gdb advice

2015-07-31 Thread Jeff Janes
On Thu, Jul 30, 2015 at 8:22 PM, Andres Freund and...@anarazel.de wrote: On 2015-07-30 09:03:01 -0700, Jeff Janes wrote: On Wed, Jul 29, 2015 at 6:10 AM, Andres Freund and...@anarazel.de wrote: What do you think about something roughly like the attached? I've not evaluated the code,

[HACKERS] RequestAddinLWLocks(int n)

2015-07-31 Thread Jesper Pedersen
Hi, Currently Max(lock_addin_request, NUM_USER_DEFINED_LWLOCKS); LWLock's are added during startup for extensions. However, this presents a problem if an extension doesn't specify the correct number of LWLock's needed, if the total number is = 4. The attached patch requires extensions to

Re: [HACKERS] Doubt about AccessExclusiveLock in ALTER TABLE .. SET ( .. );

2015-07-31 Thread Michael Paquier
On Fri, Jul 31, 2015 at 12:21 PM, Fabrízio de Royes Mello fabriziome...@gmail.com wrote: On Thu, Jul 30, 2015 at 10:46 PM, Michael Paquier wrote: On further notice, I would recommend not to use the same string name for the session and the query identifiers. And I think that inserting only

Re: [HACKERS] Doubt about AccessExclusiveLock in ALTER TABLE .. SET ( .. );

2015-07-31 Thread Simon Riggs
On 31 July 2015 at 02:46, Michael Paquier michael.paqu...@gmail.com wrote: Added. I really don't know if my isolation tests are completely correct, is my first time writing this kind of tests. This patch size has increased from 16k to 157k because of the output of the isolation tests you

Re: [HACKERS] 64-bit XIDs again

2015-07-31 Thread Simon Riggs
On 31 July 2015 at 11:00, Alexander Korotkov a.korot...@postgrespro.ru wrote: If use upgrade database cluster with pg_upgrade, he would stop old postmaster, pg_upgrade, start new postmaster. That means we start from the point when there is no running transactions. Thus, between tuples of old

Re: [HACKERS] Re: Using quicksort and a merge step to significantly improve on tuplesort's single run external sort

2015-07-31 Thread Robert Haas
On Fri, Jul 31, 2015 at 7:21 AM, Jeremy Harris j...@wizmail.org wrote: Heapification is O(n) already, whether siftup (existing) or down. That's not my impression, or what Wikipedia says. Source? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company --

Re: [HACKERS] RequestAddinLWLocks(int n)

2015-07-31 Thread Jesper Pedersen
On 07/31/2015 01:35 PM, Robert Haas wrote: On Fri, Jul 31, 2015 at 11:56 AM, Jesper Pedersen jesper.peder...@redhat.com wrote: Currently Max(lock_addin_request, NUM_USER_DEFINED_LWLOCKS); LWLock's are added during startup for extensions. However, this presents a problem if an extension

Re: [HACKERS] [PROPOSAL] VACUUM Progress Checker.

2015-07-31 Thread Joshua D. Drake
On 07/31/2015 11:21 AM, Alvaro Herrera wrote: This looks pretty complicated to understand from the user POV, but anything other than this seems to me too simplistic to be of any use. I would agree and I don't think it is all that complicated. This is an RDBMS not a web browser downloading

Re: [HACKERS] [PROPOSAL] VACUUM Progress Checker.

2015-07-31 Thread Alvaro Herrera
I think the only way to produce usable estimates is to report more than one number. And in the particular case of lazy vacuuming, ISTM the way to do it is to consider heap scanning as one phase, index cleanup as another phase; these two phases can be interleaved. And there's a final heap scan

Re: [HACKERS] Transactions involving multiple postgres foreign servers

2015-07-31 Thread Robert Haas
On Fri, Jul 31, 2015 at 6:33 AM, Ashutosh Bapat ashutosh.ba...@enterprisedb.com wrote: I'm not hung up on the table-level attribute, but I think having a server-level attribute rather than a global GUC is a good idea. However, I welcome other thoughts on that. The patch supports server level

Re: [HACKERS] [PROPOSAL] VACUUM Progress Checker.

2015-07-31 Thread Robert Haas
On Fri, Jul 31, 2015 at 2:21 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: I think the only way to produce usable estimates is to report more than one number. And in the particular case of lazy vacuuming, ISTM the way to do it is to consider heap scanning as one phase, index cleanup as

Re: [HACKERS] RequestAddinLWLocks(int n)

2015-07-31 Thread Robert Haas
On Fri, Jul 31, 2015 at 11:56 AM, Jesper Pedersen jesper.peder...@redhat.com wrote: Currently Max(lock_addin_request, NUM_USER_DEFINED_LWLOCKS); LWLock's are added during startup for extensions. However, this presents a problem if an extension doesn't specify the correct number of

Re: [HACKERS] Autonomous Transaction is back

2015-07-31 Thread Robert Haas
On Tue, Jul 28, 2015 at 6:01 AM, Craig Ringer cr...@2ndquadrant.com wrote: That should be practical to special-case by maintaining a list of parent transaction (virtual?) transaction IDs. Attempts to wait on a lock held by any of those should fail immediately. There's no point waiting for the

Re: [HACKERS] patch: prevent user from setting wal_buffers over 2GB bytes

2015-07-31 Thread Robert Haas
On Thu, Jul 30, 2015 at 9:17 PM, Josh Berkus j...@agliodbs.com wrote: In guc.c, the maximum for wal_buffers is INT_MAX. However, wal_buffers is actually measured in 8KB buffers, not in bytes. This means that users are able to set wal_buffers 2GB. When the database is started, this can

[HACKERS] Use of PRId64 with PostgreSQL functions

2015-07-31 Thread Paragon Corporation
I've been currently suffering an issue in mingw64 with changes made in PostGIS topology code base. Sadly it only happens to me (compiling in mingw64) Issue is when code like this was introduced appendStringInfo(str, %s% PRId64, sep, edge-start_node); Which uses a PostgreSQL function

Re: [HACKERS] [PROPOSAL] VACUUM Progress Checker.

2015-07-31 Thread Alvaro Herrera
Robert Haas wrote: On Fri, Jul 31, 2015 at 2:21 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: I think the only way to produce usable estimates is to report more than one number. And in the particular case of lazy vacuuming, ISTM the way to do it is to consider heap scanning as one

Re: [HACKERS] brin index vacuum versus transaction snapshots

2015-07-31 Thread Kevin Grittner
Alvaro Herrera alvhe...@2ndquadrant.com wrote: I think the real solution to this problem is to avoid use of GetTransactionSnapshot(), and instead use GetLatestSnapshot(). As far as I can see, that should completely close the hole. This requires patching IndexBuildHeapRangeScan() to allow

Re: [HACKERS] brin index vacuum versus transaction snapshots

2015-07-31 Thread Alvaro Herrera
Kevin Grittner wrote: Alvaro Herrera alvhe...@2ndquadrant.com wrote: I think the real solution to this problem is to avoid use of GetTransactionSnapshot(), and instead use GetLatestSnapshot(). As far as I can see, that should completely close the hole. This requires patching

Re: [HACKERS] Planner debug views

2015-07-31 Thread Qingqing Zhou
On Thu, Jul 30, 2015 at 2:42 PM, Jim Nasby jim.na...@bluetreble.com wrote: I think a better option would be shoving it into a backend tuplestore and just leaving it there (maybe with a command to clear it for the paranoid). That gives a relation you can query against, insert into another

Re: [HACKERS] 64-bit XIDs again

2015-07-31 Thread Robert Haas
On Thu, Jul 30, 2015 at 5:23 PM, Arthur Silva arthur...@gmail.com wrote: In the other hand PG tuple overhead is already the largest among the alternatives. Even if storage keeps getting faster and cheaper stuff you can't ignore the overhead of adding yet another 8bytes to each tuple. +1, very

Re: [HACKERS] Doubt about AccessExclusiveLock in ALTER TABLE .. SET ( .. );

2015-07-31 Thread Fabrízio de Royes Mello
On Fri, Jul 31, 2015 at 10:01 AM, Michael Paquier michael.paqu...@gmail.com wrote: On Fri, Jul 31, 2015 at 11:41 AM, Fabrízio de Royes Mello wrote: We usually don't compare lock values that way, i.e. there's not guaranteed to be a strict monotonicity between lock levels. I don't really

Re: [HACKERS] brin index vacuum versus transaction snapshots

2015-07-31 Thread Alvaro Herrera
I think the real solution to this problem is to avoid use of GetTransactionSnapshot(), and instead use GetLatestSnapshot(). As far as I can see, that should completely close the hole. This requires patching IndexBuildHeapRangeScan() to allow for that. Actually I think there's another

Re: [HACKERS] Doubt about AccessExclusiveLock in ALTER TABLE .. SET ( .. );

2015-07-31 Thread Alvaro Herrera
Fabrízio de Royes Mello wrote: In this patch I didn't change all lockmode comparison places previous pointed by you, but I can change it maybe adding other method called LockModeIsValid(lockmode) to do the comparison lockmode = NoLock lockmode MAX_LOCKMODES used in many places. I don't

Re: [HACKERS] Use of PRId64 with PostgreSQL functions

2015-07-31 Thread Alvaro Herrera
Paragon Corporation wrote: I've been currently suffering an issue in mingw64 with changes made in PostGIS topology code base. Sadly it only happens to me (compiling in mingw64) Issue is when code like this was introduced appendStringInfo(str, %s% PRId64, sep, edge-start_node); I

Re: [HACKERS] Use of PRId64 with PostgreSQL functions

2015-07-31 Thread Paragon Corporation
Alvaro and Tom, A big thank you. That seems to have done the trick. My compiler warnings went away and no more syntax errors in the regress tests. I still have some regress failures I got to hunt down, but those could be caused by places in the code I did not replace PRId64 calls or something

Re: [HACKERS] brin index vacuum versus transaction snapshots

2015-07-31 Thread Alvaro Herrera
Let me explain a bit more what's the mechanism at play. When scanning a block range to summarize it, we use an MVCC snapshot. Any tuples that are inserted by transactions that appear as in-progress to that snapshot will not be seen by that range scan; therefore we need another mechanism to

Re: [HACKERS] 64-bit XIDs again

2015-07-31 Thread Gurjeet Singh
On Jul 30, 2015 2:23 PM, Tom Lane t...@sss.pgh.pa.us wrote: Gavin Flower gavinflo...@archidevsys.co.nz writes: On 31/07/15 02:24, Heikki Linnakangas wrote: There is a big downside to expanding xmin/xmax to 64 bits: it takes space. More space means more memory needed for caching, more

Re: [HACKERS] Using quicksort and a merge step to significantly improve on tuplesort's single run external sort

2015-07-31 Thread Peter Geoghegan
On Thu, Jul 30, 2015 at 4:01 PM, Peter Geoghegan p...@heroku.com wrote: On Thu, Jul 30, 2015 at 12:00 AM, Heikki Linnakangas hlinn...@iki.fi wrote: Hmm. You don't really need to merge the in-memory array into the tape, as you know that all the tuples in the in-memory must come after the tuples

Re: [HACKERS] Using quicksort and a merge step to significantly improve on tuplesort's single run external sort

2015-07-31 Thread Heikki Linnakangas
On 07/31/2015 02:01 AM, Peter Geoghegan wrote: What prevents the tuple at the top of the in-memory heap at the point of tuplesort_performsort() (say, one of the ones added to the heap as our glut of memory was*partially* consumed) being less than the last/greatest tuple on tape? If the answer

Re: [HACKERS] 64-bit XIDs again

2015-07-31 Thread Heikki Linnakangas
On 07/31/2015 09:22 AM, Gurjeet Singh wrote: On Jul 30, 2015 2:23 PM, Tom Lane t...@sss.pgh.pa.us wrote: But the elephant in the room is on-disk compatibility. There is absolutely no way that we can just change xmin/xmax to 64 bits without a disk format break. However, if we do something like

Re: [HACKERS] Updatable view?

2015-07-31 Thread Atri Sharma
On 31 Jul 2015 11:59, Tatsuo Ishii is...@postgresql.org wrote: On 31 Jul 2015 10:15, Tatsuo Ishii is...@postgresql.org wrote: I think it would be nice to have... but not to the point of working on it myself. Might be worth an email to -general to see how many people have

Re: [HACKERS] Updatable view?

2015-07-31 Thread Tatsuo Ishii
On 31 Jul 2015 10:15, Tatsuo Ishii is...@postgresql.org wrote: I think it would be nice to have... but not to the point of working on it myself. Might be worth an email to -general to see how many people have immediate use for it. What I am thinking about is, 1) Implement certain

Re: [HACKERS] Solaris testers wanted for strxfrm() behavior

2015-07-31 Thread Josh Berkus
Noah, All: Where are we with this? Do we feel confident that this bug is only on old versions of Solaris we don't care about? Or does it remain to be resolved? -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)

Re: [HACKERS] Use of PRId64 with PostgreSQL functions

2015-07-31 Thread Tom Lane
Paragon Corporation l...@pcorp.us writes: Huh. Apparently, whichever Windows compiler you're using defines PRId64 as d, which surely seems pretty broken. I think it was only failing on the postgresql function calls. And this: gcc -Wall -E testint.c | grep '\(printf.*x)\| int64_t;\)'

Re: [HACKERS] Use of PRId64 with PostgreSQL functions

2015-07-31 Thread Tom Lane
Paragon Corporation l...@pcorp.us writes: I've been currently suffering an issue in mingw64 with changes made in PostGIS topology code base. Sadly it only happens to me (compiling in mingw64) Issue is when code like this was introduced appendStringInfo(str, %s% PRId64, sep,

Re: [HACKERS] 64-bit XIDs again

2015-07-31 Thread Heikki Linnakangas
On 07/31/2015 12:29 AM, Josh Berkus wrote: On 07/30/2015 07:24 AM, Heikki Linnakangas wrote: I think we should move to 64-bit XIDs in in-memory structs snapshots, proc array etc. And expand clog to handle 64-bit XIDs. But keep the xmin/xmax fields on heap pages at 32-bits, and add an

Re: [HACKERS] 64-bit XIDs again

2015-07-31 Thread Alexander Korotkov
On Fri, Jul 31, 2015 at 1:27 AM, Petr Jelinek p...@2ndquadrant.com wrote: On 2015-07-30 23:23, Tom Lane wrote: But the elephant in the room is on-disk compatibility. There is absolutely no way that we can just change xmin/xmax to 64 bits without a disk format break. However, if we do

[HACKERS] Re: Using quicksort and a merge step to significantly improve on tuplesort's single run external sort

2015-07-31 Thread Jeremy Harris
On 30/07/15 02:05, Peter Geoghegan wrote: Since heapification is now a big fraction of the total cost of a sort sometimes, even where the heap invariant need not be maintained for any length of time afterwards, it might be worth revisiting the patch to make that an O(n) rather than a O(log n)

Re: [HACKERS] LWLock deadlock and gdb advice

2015-07-31 Thread Heikki Linnakangas
On 07/30/2015 09:14 PM, Andres Freund wrote: On 2015-07-30 17:36:52 +0300, Heikki Linnakangas wrote: In 9.4, LWLockAcquire holds the spinlock when it marks the lock as held, until it has updated the variable. And LWLockWaitForVar() holds the spinlock when it checks that the lock is held and

Re: [HACKERS] Transactions involving multiple postgres foreign servers

2015-07-31 Thread Ashutosh Bapat
On Thu, Jul 30, 2015 at 1:52 AM, Robert Haas robertmh...@gmail.com wrote: On Wed, Jul 29, 2015 at 6:58 AM, Ashutosh Bapat ashutosh.ba...@enterprisedb.com wrote: A user may set atomic_foreign_transaction to ON to guarantee atomicity, IOW it throws error when atomicity can not be guaranteed.