Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

2016-10-20 Thread Amit Kapila
On Fri, Oct 21, 2016 at 6:31 AM, Robert Haas wrote: > On Thu, Oct 20, 2016 at 4:04 PM, Tomas Vondra > wrote: >>> I then started a run at 96 clients which I accidentally killed shortly >>> before it was scheduled to finish, but the results are not much >>> different; there is no hint of the runawa

Re: [HACKERS] Transactions involving multiple postgres foreign servers

2016-10-20 Thread Ashutosh Bapat
On Wed, Oct 19, 2016 at 9:17 PM, Robert Haas wrote: > On Thu, Oct 13, 2016 at 7:27 AM, Amit Langote > wrote: >> However, when I briefly read the description in "Transaction Management in >> the R* Distributed Database Management System (C. Mohan et al)" [2], it >> seems that what Ashutosh is sayi

Re: [HACKERS] Fun fact about autovacuum and orphan temp tables

2016-10-20 Thread Michael Paquier
On Fri, Oct 21, 2016 at 2:29 PM, Michael Paquier wrote: > On Thu, Oct 20, 2016 at 9:30 PM, Constantin S. Pan wrote: >> I tried to fix the problem with a new backend not being >> able to reuse a temporary namespace when it contains >> thousands of temporary tables. I disabled locking of objects >>

Re: [HACKERS] Fun fact about autovacuum and orphan temp tables

2016-10-20 Thread Michael Paquier
On Thu, Oct 20, 2016 at 9:30 PM, Constantin S. Pan wrote: > I tried to fix the problem with a new backend not being > able to reuse a temporary namespace when it contains > thousands of temporary tables. I disabled locking of objects > during namespace clearing process. See the patch attached. > P

Re: [HACKERS] Fun fact about autovacuum and orphan temp tables

2016-10-20 Thread Michael Paquier
On Thu, Sep 8, 2016 at 12:38 AM, Robert Haas wrote: > On Mon, Sep 5, 2016 at 1:14 PM, Bruce Momjian wrote: >> I don't think we look at those temp tables frequently enough to justify >> keeping them around for all users. > > +1. I think it would be much better to nuke them more aggressively. +1

Re: [HACKERS] Remove autovacuum GUC?

2016-10-20 Thread Craig Ringer
On 21 Oct. 2016 12:57 am, "Joshua D. Drake" wrote: > > Hello, > > What about a simpler solution to all of this. Let's just remove it from postgresql.conf. Out of sight. If someone needs to test they can but a uneducated user won't immediately know what to do about that "autovacuum process" and whe

Re: [HACKERS] FSM corruption leading to errors

2016-10-20 Thread Michael Paquier
On Thu, Oct 20, 2016 at 3:37 PM, Pavan Deolasee wrote: > Just to clarify, I meant if we truncate the entire FSM then we'll need API > to truncate VM as well so that VACUUM rebuilds everything completely. OTOH > if we provide a function just to truncate FSM to match the size of the > table, then we

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

2016-10-20 Thread Dilip Kumar
On Thu, Oct 20, 2016 at 9:15 PM, Robert Haas wrote: > So here's my theory. The whole reason why Tomas is having difficulty > seeing any big effect from these patches is because he's testing on > x86. When Dilip tests on x86, he doesn't see a big effect either, > regardless of workload. But when

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

2016-10-20 Thread Dilip Kumar
On Thu, Oct 20, 2016 at 9:03 PM, Tomas Vondra wrote: > In the results you've posted on 10/12, you've mentioned a regression with 32 > clients, where you got 52k tps on master but only 48k tps with the patch (so > ~10% difference). I have no idea what scale was used for those tests, That test was

Re: [HACKERS] Renaming of pg_xlog and pg_clog

2016-10-20 Thread David G. Johnston
On Thu, Oct 20, 2016 at 6:03 PM, Robert Haas wrote: > On Thu, Oct 20, 2016 at 3:46 PM, Tom Lane wrote: > > I'm mostly with Stephen on this. As the names stand, they encourage > > people to go look at the documentation, > > https://www.postgresql.org/docs/devel/static/storage-file-layout.html >

Re: [HACKERS] Renaming of pg_xlog and pg_clog

2016-10-20 Thread Michael Paquier
On Fri, Oct 21, 2016 at 10:03 AM, Robert Haas wrote: > On Thu, Oct 20, 2016 at 3:46 PM, Tom Lane wrote: >> I'm mostly with Stephen on this. As the names stand, they encourage >> people to go look at the documentation, >> https://www.postgresql.org/docs/devel/static/storage-file-layout.html >> wh

[HACKERS] Typo in pgstat.c

2016-10-20 Thread vinayak
Hi, Attached patch fixes a typo in pgstat.c s/addtions/additions/g Regards, Vinayak Pokale NTT Open Source Software Center typo-pgstat-c.patch Description: application/download -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: htt

Re: [HACKERS] Renaming of pg_xlog and pg_clog

2016-10-20 Thread Robert Haas
On Thu, Oct 20, 2016 at 3:46 PM, Tom Lane wrote: > I'm mostly with Stephen on this. As the names stand, they encourage > people to go look at the documentation, > https://www.postgresql.org/docs/devel/static/storage-file-layout.html > which will provide more information than you'd ever get out of

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

2016-10-20 Thread Robert Haas
On Thu, Oct 20, 2016 at 4:04 PM, Tomas Vondra wrote: >> I then started a run at 96 clients which I accidentally killed shortly >> before it was scheduled to finish, but the results are not much >> different; there is no hint of the runaway CLogControlLock contention >> that Dilip sees on power2. >

[HACKERS] Typo in pgstat.h

2016-10-20 Thread vinayak
Hi, - * st_progress_command_target, and st_progress_command[]. + * st_progress_command_target, and st_progress_param[]. Attached patch fixed typo. Regards, Vinayak Pokale NTT Open Source Software Center typo-pgstat-h.patch Description: application/download -- Sent via pgsql-

Re: [HACKERS] Renaming of pg_xlog and pg_clog

2016-10-20 Thread Michael Paquier
On Fri, Oct 21, 2016 at 12:35 AM, Robert Haas wrote: > On Wed, Oct 12, 2016 at 10:22 PM, Michael Paquier > wrote: >> OK. I can live with that as well. Attached are three patches. The >> pg_xlog -> pg_wal move, the pg_clog -> pg_transaction move, and the >> pg_clog -> pg_xact move. Only one surviv

Re: [HACKERS] Indirect indexes

2016-10-20 Thread Claudio Freire
On Thu, Oct 20, 2016 at 1:08 PM, Pavan Deolasee wrote: > On Thu, Oct 20, 2016 at 9:20 PM, Claudio Freire > wrote: >> >> >> >> With indirect indexes, since you don't need to insert a tid, you can >> just "insert on conflict do nothing" on the index. > > > Would that work with non-unique indexes? A

Re: [HACKERS] Improve output of BitmapAnd EXPLAIN ANALYZE

2016-10-20 Thread Tom Lane
Stephen Frost writes: > * Tom Lane (t...@sss.pgh.pa.us) wrote: >> That would break code that tries to parse that stuff, eg depesz.com. > I don't believe Jim was suggesting that we back-patch such a change. I don't either. > Changing it in a new major release seems entirely reasonable. It's sti

Re: [HACKERS] emergency outage requiring database restart

2016-10-20 Thread Merlin Moncure
On Thu, Oct 20, 2016 at 3:16 PM, Alvaro Herrera wrote: > Merlin Moncure wrote: > >> single user mode dumps core :( >> >> bash-4.1$ postgres --single -D /var/lib/pgsql/9.5/data castaging >> LOG: 0: could not change directory to "/root": Permission denied >> LOCATION: resolve_symlinks, exec.c:

Re: [HACKERS] WIP: Fix invalid XML explain plans for track_io_timing

2016-10-20 Thread Tom Lane
I wrote: > Markus Winand writes: >> The XML output of explain potentially outputs the XML tag names >> "I/O-Write-Time" >> and "I/O-Read-Time", which are invalid due to the slash. > Ooops. After further thought I decided we should go with the whitelist solution. The extra time needed to produce

Re: [HACKERS] Improve output of BitmapAnd EXPLAIN ANALYZE

2016-10-20 Thread Stephen Frost
Tom, * Tom Lane (t...@sss.pgh.pa.us) wrote: > Jim Nasby writes: > > A customer just pinged me wondering how it was that a BitmapAnd node was > > reporting 0 tuples when the Bitmap Heap Scan above it showed it had in > > fact generated tuples. > > > While this is mentioned in the docs, I think

Re: [HACKERS] Improve output of BitmapAnd EXPLAIN ANALYZE

2016-10-20 Thread Tom Lane
Jim Nasby writes: > A customer just pinged me wondering how it was that a BitmapAnd node was > reporting 0 tuples when the Bitmap Heap Scan above it showed it had in > fact generated tuples. > While this is mentioned in the docs, I think it would be very helpful to > have ANALYZE spit out "N/A

[HACKERS] Improve output of BitmapAnd EXPLAIN ANALYZE

2016-10-20 Thread Jim Nasby
A customer just pinged me wondering how it was that a BitmapAnd node was reporting 0 tuples when the Bitmap Heap Scan above it showed it had in fact generated tuples. While this is mentioned in the docs, I think it would be very helpful to have ANALYZE spit out "N/A" instead of 0 for these nod

Re: [HACKERS] emergency outage requiring database restart

2016-10-20 Thread Alvaro Herrera
Merlin Moncure wrote: > single user mode dumps core :( > > bash-4.1$ postgres --single -D /var/lib/pgsql/9.5/data castaging > LOG: 0: could not change directory to "/root": Permission denied > LOCATION: resolve_symlinks, exec.c:293 > Segmentation fault (core dumped) > > Core was generated

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

2016-10-20 Thread Tomas Vondra
On 10/20/2016 07:59 PM, Robert Haas wrote: On Thu, Oct 20, 2016 at 11:45 AM, Robert Haas wrote: On Thu, Oct 20, 2016 at 3:36 AM, Dilip Kumar wrote: On Thu, Oct 13, 2016 at 12:25 AM, Robert Haas wrote: >> ... So here's my theory. The whole reason why Tomas is having difficulty seeing any

Re: [HACKERS] Renaming of pg_xlog and pg_clog

2016-10-20 Thread Tom Lane
Stephen Frost writes: > * David Fetter (da...@fetter.org) wrote: >> On Thu, Oct 20, 2016 at 02:23:32PM -0400, Tom Lane wrote: >>> I don't see one single one of those subdirectory names that I'd call >>> self-documenting. >> That's a problem we should do something about, even if we can't do it >>

Re: [HACKERS] Renaming of pg_xlog and pg_clog

2016-10-20 Thread Stephen Frost
* David Fetter (da...@fetter.org) wrote: > On Thu, Oct 20, 2016 at 02:23:32PM -0400, Tom Lane wrote: > > Robert Haas writes: > > > On Thu, Oct 20, 2016 at 2:09 PM, Tom Lane wrote: > > >> We have the two precedents "pg_subtrans" and "pg_multixact", so > > >> unless we want to get into renaming tho

Re: [HACKERS] emergency outage requiring database restart

2016-10-20 Thread Merlin Moncure
On Thu, Oct 20, 2016 at 2:07 PM, Tom Lane wrote: > Merlin Moncure writes: >> single user mode dumps core :( > > You've got a mess there :-( > >> Missing separate debuginfos, use: debuginfo-install >> postgresql95-server-9.5.2-1PGDG.rhel6.x86_64 > > This backtrace would likely be much more informa

Re: [HACKERS] emergency outage requiring database restart

2016-10-20 Thread Tom Lane
Merlin Moncure writes: > single user mode dumps core :( You've got a mess there :-( > Missing separate debuginfos, use: debuginfo-install > postgresql95-server-9.5.2-1PGDG.rhel6.x86_64 This backtrace would likely be much more informative if you did the above. regards, t

Re: [HACKERS] emergency outage requiring database restart

2016-10-20 Thread Merlin Moncure
On Thu, Oct 20, 2016 at 1:52 PM, Merlin Moncure wrote: > On Wed, Oct 19, 2016 at 2:39 PM, Merlin Moncure wrote: >> On Wed, Oct 19, 2016 at 9:56 AM, Bruce Momjian wrote: >>> On Wed, Oct 19, 2016 at 08:54:48AM -0500, Merlin Moncure wrote: > Yeah. Believe me -- I know the drill. Most or all

Re: [HACKERS] Renaming of pg_xlog and pg_clog

2016-10-20 Thread David Fetter
On Thu, Oct 20, 2016 at 02:23:32PM -0400, Tom Lane wrote: > Robert Haas writes: > > On Thu, Oct 20, 2016 at 2:09 PM, Tom Lane wrote: > >> We have the two precedents "pg_subtrans" and "pg_multixact", so > >> unless we want to get into renaming those too, I think "pg_trans" > >> and "pg_xact" are r

Re: [HACKERS] emergency outage requiring database restart

2016-10-20 Thread Merlin Moncure
On Wed, Oct 19, 2016 at 2:39 PM, Merlin Moncure wrote: > On Wed, Oct 19, 2016 at 9:56 AM, Bruce Momjian wrote: >> On Wed, Oct 19, 2016 at 08:54:48AM -0500, Merlin Moncure wrote: >>> > Yeah. Believe me -- I know the drill. Most or all the damage seemed >>> > to be to the system catalogs with at

Re: [HACKERS] Renaming of pg_xlog and pg_clog

2016-10-20 Thread Robert Haas
On Thu, Oct 20, 2016 at 2:23 PM, Tom Lane wrote: > Robert Haas writes: >> On Thu, Oct 20, 2016 at 2:09 PM, Tom Lane wrote: >>> We have the two precedents "pg_subtrans" and "pg_multixact", so >>> unless we want to get into renaming those too, I think "pg_trans" >>> and "pg_xact" are really the on

Re: [HACKERS] Renaming of pg_xlog and pg_clog

2016-10-20 Thread Bruce Momjian
On Thu, Oct 20, 2016 at 02:02:27PM -0400, Robert Haas wrote: > On Thu, Oct 20, 2016 at 1:39 PM, Bruce Momjian wrote: > > On Thu, Oct 20, 2016 at 12:29:47PM -0400, Robert Haas wrote: > >> > When it comes to the name, I tend to think of 'pg_xact' as saying "this > >> > is where we persist info we ne

Re: [HACKERS] Renaming of pg_xlog and pg_clog

2016-10-20 Thread Tom Lane
Robert Haas writes: > On Thu, Oct 20, 2016 at 2:09 PM, Tom Lane wrote: >> We have the two precedents "pg_subtrans" and "pg_multixact", so >> unless we want to get into renaming those too, I think "pg_trans" >> and "pg_xact" are really the only options worth considering. >> >> Personally I'd go f

Re: [HACKERS] Renaming of pg_xlog and pg_clog

2016-10-20 Thread Robert Haas
On Thu, Oct 20, 2016 at 2:09 PM, Tom Lane wrote: > Robert Haas writes: >> Is pg_xact actually better than pg_clog? > > Yes, because it doesn't contain the three letters "log". I figured somebody was going to say that. > We have the two precedents "pg_subtrans" and "pg_multixact", so > unless we

Re: [HACKERS] Renaming of pg_xlog and pg_clog

2016-10-20 Thread Tom Lane
Robert Haas writes: > Is pg_xact actually better than pg_clog? Yes, because it doesn't contain the three letters "log". We have the two precedents "pg_subtrans" and "pg_multixact", so unless we want to get into renaming those too, I think "pg_trans" and "pg_xact" are really the only options wort

Re: [HACKERS] Renaming of pg_xlog and pg_clog

2016-10-20 Thread Robert Haas
On Thu, Oct 20, 2016 at 1:39 PM, Bruce Momjian wrote: > On Thu, Oct 20, 2016 at 12:29:47PM -0400, Robert Haas wrote: >> > When it comes to the name, I tend to think of 'pg_xact' as saying "this >> > is where we persist info we need to keep about transactions." Today >> > that's just the commit st

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

2016-10-20 Thread Robert Haas
On Thu, Oct 20, 2016 at 11:45 AM, Robert Haas wrote: > On Thu, Oct 20, 2016 at 3:36 AM, Dilip Kumar wrote: >> On Thu, Oct 13, 2016 at 12:25 AM, Robert Haas wrote: >>> I agree with these conclusions. I had a chance to talk with Andres >>> this morning at Postgres Vision and based on that convers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

2016-10-20 Thread Bruce Momjian
On Thu, Oct 20, 2016 at 12:29:47PM -0400, Robert Haas wrote: > > When it comes to the name, I tend to think of 'pg_xact' as saying "this > > is where we persist info we need to keep about transactions." Today > > that's just the commit status info, but I could imagine that there > > might, someday

Re: [HACKERS] LLVM Address Sanitizer (ASAN) and valgrind support

2016-10-20 Thread Greg Stark
On Oct 20, 2016 5:27 PM, "Noah Misch" wrote: > > On Wed, Oct 19, 2016 at 11:08:39AM +0100, Greg Stark wrote: > > > The MEMPOOL_FREE doesn't take any size argument and mcxt.c doesn't > > have convenient access to a size argument. It could call the > > GetChunkSpace method but that will include the

Re: [HACKERS] Remove autovacuum GUC?

2016-10-20 Thread Robert Haas
On Thu, Oct 20, 2016 at 12:32 PM, Joshua D. Drake wrote: > That argument suggests we shouldn't have autovacuum :P It certainly does not. That, too, would be removing a useful option. In fact, it would be removing the most useful option that is the right choice for most users in 99% of cases. >

Re: [HACKERS] Parallel Index Scans

2016-10-20 Thread Robert Haas
On Wed, Oct 19, 2016 at 11:07 PM, Amit Kapila wrote: >> Ideally, the parallel_workers storage parameter will rarely be >> necessary because the optimizer will generally do the right thing in >> all case. > > Yeah, we can choose not to provide any parameter for parallel index > scans, but some user

Re: [HACKERS] Remove autovacuum GUC?

2016-10-20 Thread Joshua D. Drake
Hello, What about a simpler solution to all of this. Let's just remove it from postgresql.conf. Out of sight. If someone needs to test they can but a uneducated user won't immediately know what to do about that "autovacuum process" and when they look it up the documentation is exceedingly blun

Re: [HACKERS] Renaming of pg_xlog and pg_clog

2016-10-20 Thread Robert Haas
On Thu, Oct 20, 2016 at 12:12 PM, Stephen Frost wrote: > That said, I'd also like to see a --force or similar option or mechanism > put in place to reduce the risk of users trashing their system because > they think pg_resetwal is "safe." ("It's just gonna reset things to make > the database start

Re: [HACKERS] Renaming of pg_xlog and pg_clog

2016-10-20 Thread Robert Haas
On Thu, Oct 20, 2016 at 12:05 PM, Stephen Frost wrote: >> To be honest, I don't really like either pg_transaction or pg_xact. > >> Neither name captures the fact that what we're really tracking here is >> the transaction *status*. pg_xact is slightly worse because it's a >> poor abbreviation for

Re: [HACKERS] Remove autovacuum GUC?

2016-10-20 Thread Joshua D. Drake
On 10/20/2016 09:24 AM, Robert Haas wrote: On Thu, Oct 20, 2016 at 11:53 AM, Joshua D. Drake wrote: The right answer isn't the answer founded in the reality for many if not most of our users. I think that's high-handed nonsense. Sure, there are some unsophisticated users who do incredibly s

Re: [HACKERS] Remove autovacuum GUC?

2016-10-20 Thread Peter Geoghegan
On Thu, Oct 20, 2016 at 9:24 AM, Robert Haas wrote: > On Thu, Oct 20, 2016 at 11:53 AM, Joshua D. Drake > wrote: >> The right answer isn't the answer founded in the reality for many if not >> most of our users. > > I think that's high-handed nonsense. Sure, there are some > unsophisticated user

Re: [HACKERS] Remove autovacuum GUC?

2016-10-20 Thread Robert Haas
On Thu, Oct 20, 2016 at 11:53 AM, Joshua D. Drake wrote: > The right answer isn't the answer founded in the reality for many if not > most of our users. I think that's high-handed nonsense. Sure, there are some unsophisticated users who do incredibly stupid things and pay the price for it, but

Re: [HACKERS] Renaming of pg_xlog and pg_clog

2016-10-20 Thread Joshua D. Drake
On 10/20/2016 09:12 AM, Stephen Frost wrote: * Tom Lane (t...@sss.pgh.pa.us) wrote: Robert Haas writes: That said, I'd also like to see a --force or similar option or mechanism put in place to reduce the risk of users trashing their system because they think pg_resetwal is "safe." ("It's jus

Re: [HACKERS] Renaming of pg_xlog and pg_clog

2016-10-20 Thread Stephen Frost
* Tom Lane (t...@sss.pgh.pa.us) wrote: > Robert Haas writes: > > One idea would be to rename pg_resetxlog to pg_resetwal. I think > > that's actually an improvement. > > This would fit in as part of a general plan to s/xlog/wal/g throughout > our user-visible names and documentation. Which seem

Re: [HACKERS] Indirect indexes

2016-10-20 Thread Pavan Deolasee
On Thu, Oct 20, 2016 at 9:20 PM, Claudio Freire wrote: > > > With indirect indexes, since you don't need to insert a tid, you can > just "insert on conflict do nothing" on the index. > Would that work with non-unique indexes? Anyways, the point I was trying to make is that there are a similar te

Re: [HACKERS] Renaming of pg_xlog and pg_clog

2016-10-20 Thread Stephen Frost
* Robert Haas (robertmh...@gmail.com) wrote: > On Wed, Oct 12, 2016 at 10:22 PM, Michael Paquier > wrote: > > OK. I can live with that as well. Attached are three patches. The > > pg_xlog -> pg_wal move, the pg_clog -> pg_transaction move, and the > > pg_clog -> pg_xact move. Only one survivor to

Re: [HACKERS] Disable autovacuum guc?

2016-10-20 Thread Joshua D. Drake
On 10/20/2016 08:54 AM, Josh Berkus wrote: On 10/20/2016 06:34 AM, Joshua D. Drake wrote: On 10/19/2016 07:22 PM, Josh Berkus wrote: On 10/19/2016 06:27 PM, Joshua D. Drake wrote: Hrm, true although that is by far a minority of our users. What if we made it so we disabled the autovacuum guc

Re: [HACKERS] Disable autovacuum guc?

2016-10-20 Thread Josh Berkus
On 10/20/2016 06:34 AM, Joshua D. Drake wrote: > On 10/19/2016 07:22 PM, Josh Berkus wrote: >> On 10/19/2016 06:27 PM, Joshua D. Drake wrote: >>> Hello, >>> >>> After all these years, we are still regularly running into people who >>> say, "performance was bad so we disabled autovacuum". I am not t

Re: [HACKERS] Remove autovacuum GUC?

2016-10-20 Thread Joshua D. Drake
On 10/20/2016 07:12 AM, Robert Haas wrote: On Wed, Oct 19, 2016 at 9:24 PM, Joshua D. Drake wrote: Setting autovacuum=off is at least useful for testing purposes and I've used it that way. On the other hand, I haven't seen a customer disable this unintentionally in years. Generally, the custo

Re: [HACKERS] Indirect indexes

2016-10-20 Thread Claudio Freire
On Thu, Oct 20, 2016 at 12:30 PM, Pavan Deolasee wrote: > > > On Thu, Oct 20, 2016 at 8:44 PM, Petr Jelinek wrote: >> >> >> >> WARM can do WARM update 50% of time, indirect index can do HOT update >> 100% of time (provided the column is not changed), I don't see why we >> could not have both solu

Re: [HACKERS] Renaming of pg_xlog and pg_clog

2016-10-20 Thread Tom Lane
Robert Haas writes: > One idea would be to rename pg_resetxlog to pg_resetwal. I think > that's actually an improvement. This would fit in as part of a general plan to s/xlog/wal/g throughout our user-visible names and documentation. Which seems like a good idea to me; there's no need to expose

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

2016-10-20 Thread Robert Haas
On Thu, Oct 20, 2016 at 3:36 AM, Dilip Kumar wrote: > On Thu, Oct 13, 2016 at 12:25 AM, Robert Haas wrote: >> I agree with these conclusions. I had a chance to talk with Andres >> this morning at Postgres Vision and based on that conversation I'd >> like to suggest a couple of additional tests:

Re: [HACKERS] Renaming of pg_xlog and pg_clog

2016-10-20 Thread Robert Haas
On Wed, Oct 12, 2016 at 10:22 PM, Michael Paquier wrote: > OK. I can live with that as well. Attached are three patches. The > pg_xlog -> pg_wal move, the pg_clog -> pg_transaction move, and the > pg_clog -> pg_xact move. Only one survivor to be chosen among the last > two ones. Committed 0001.

Re: [HACKERS] Indirect indexes

2016-10-20 Thread Petr Jelinek
On 20/10/16 17:24, Bruce Momjian wrote: > On Thu, Oct 20, 2016 at 05:14:51PM +0200, Petr Jelinek wrote: >>> Also, it seems indirect indexes would be useful for indexing columns >>> that are not updated frequently on tables that are updated frequently, >>> and whose primary key is not updated freque

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

2016-10-20 Thread Tomas Vondra
On 10/20/2016 09:36 AM, Dilip Kumar wrote: On Thu, Oct 13, 2016 at 12:25 AM, Robert Haas wrote: I agree with these conclusions. I had a chance to talk with Andres this morning at Postgres Vision and based on that conversation I'd like to suggest a couple of additional tests: 1. Repeat this te

Re: [HACKERS] Indirect indexes

2016-10-20 Thread Pavan Deolasee
On Thu, Oct 20, 2016 at 8:44 PM, Petr Jelinek wrote: > > > WARM can do WARM update 50% of time, indirect index can do HOT update > 100% of time (provided the column is not changed), I don't see why we > could not have both solutions. > > I think the reason why I restricted WARM to one update per

Re: [HACKERS] Indirect indexes

2016-10-20 Thread Bruce Momjian
On Thu, Oct 20, 2016 at 05:14:51PM +0200, Petr Jelinek wrote: > > Also, it seems indirect indexes would be useful for indexing columns > > that are not updated frequently on tables that are updated frequently, > > and whose primary key is not updated frequently. That's quite a logic > > problem fo

Re: [HACKERS] Indirect indexes

2016-10-20 Thread Claudio Freire
On Thu, Oct 20, 2016 at 12:14 PM, Petr Jelinek wrote: > WARM can do WARM update 50% of time, indirect index can do HOT update > 100% of time (provided the column is not changed), I don't see why we > could not have both solutions. > > That all being said, it would be interesting to hear Álvaro's t

Re: [HACKERS] WIP: Fix invalid XML explain plans for track_io_timing

2016-10-20 Thread Tom Lane
Markus Winand writes: > The XML output of explain potentially outputs the XML tag names > "I/O-Write-Time" > and "I/O-Read-Time", which are invalid due to the slash. Ooops. > Although the patch fixes the problem for the moment, it is incomplete in that > sense that it continues to check against

Re: [HACKERS] Indirect indexes

2016-10-20 Thread Petr Jelinek
On 20/10/16 14:29, Bruce Momjian wrote: > On Wed, Oct 19, 2016 at 01:04:16PM -0400, Bruce Momjian wrote: >> On Wed, Oct 19, 2016 at 06:58:05PM +0200, Simon Riggs wrote: > I agree. Also, I think the recheck mechanism will have to be something > like > what I wrote for WARM i.e. only che

Re: [HACKERS] File content logging during execution of COPY queries

2016-10-20 Thread Stephen Frost
Grigory, * Grigory Smolkin (g.smol...@postgrespro.ru) wrote: > On 10/20/2016 12:36 PM, Aleksander Alekseev wrote: > According to my colleagues it would be very nice to have this feature. > For instance, if you are trying to optimize PostgreSQL for application > that uses COPY and you d

Re: [HACKERS] Renaming of pg_xlog and pg_clog

2016-10-20 Thread Robert Haas
On Wed, Oct 19, 2016 at 7:05 AM, Christoph Berg wrote: > (tl;dr: rename pg_xlog yes, rename pg_resetxlog only if we have a good > alternative.) I'm amused by the idea of a TL;DR in parentheses at the very bottom of the email, but maybe I'm just easily amused. One idea would be to rename pg_reset

[HACKERS] WIP: Fix invalid XML explain plans for track_io_timing

2016-10-20 Thread Markus Winand
Hi! The XML output of explain potentially outputs the XML tag names "I/O-Write-Time" and "I/O-Read-Time", which are invalid due to the slash. This can easily be seen with this: set track_io_timing = on; explain (analyze true, buffers true, format xml) select 1; [...] 0.000 0.000

Re: [HACKERS] File content logging during execution of COPY queries

2016-10-20 Thread Grigory Smolkin
On 10/20/2016 12:36 PM, Aleksander Alekseev wrote: According to my colleagues it would be very nice to have this feature. For instance, if you are trying to optimize PostgreSQL for application that uses COPY and you don't have access to or something like this. It could also be useful in some ot

Re: [HACKERS] Avoiding pin scan during btree vacuum

2016-10-20 Thread Robert Haas
On Wed, Oct 19, 2016 at 6:30 PM, Alvaro Herrera wrote: > Robert Haas wrote: >> On Mon, Jan 4, 2016 at 10:30 AM, Tom Lane wrote: >> >> This seems like a might subtle thing to backpatch. If we really want to >> >> go there, ISTM that the relevant code should stew in an unreleased >> >> branch for a

Re: [HACKERS] PATCH: two slab-like memory allocators

2016-10-20 Thread Robert Haas
On Tue, Oct 18, 2016 at 6:27 PM, Petr Jelinek wrote: > I agree though that the usability beyond the ReoderBuffer is limited > because everything that will want to use it for part of allocations will > get much more complicated by the fact that it will have to use two > different allocators. > > I

Re: [HACKERS] incorrect libpq comment

2016-10-20 Thread Robert Haas
On Wed, Oct 19, 2016 at 1:35 PM, Bruce Momjian wrote: > On Wed, Oct 19, 2016 at 01:16:28PM -0400, Robert Haas wrote: >> Bruce's commit 5d305d86bd917723f09ab4f15c075d90586a210a back in April >> of 2014 includes this change: >> >> /* See PQconnectPoll() for how we use 'int' and not 'pgsocket'.

Re: [HACKERS] LLVM Address Sanitizer (ASAN) and valgrind support

2016-10-20 Thread Noah Misch
On Wed, Oct 19, 2016 at 11:08:39AM +0100, Greg Stark wrote: > On Sat, Feb 6, 2016 at 4:52 AM, Noah Misch wrote: > > aset.c relies on the fact that VALGRIND_MEMPOOL_ALLOC() has an implicit > > VALGRIND_MAKE_MEM_UNDEFINED() and VALGRIND_MEMPOOL_FREE() has an implicit > > VALGRIND_MAKE_MEM_NOACCESS()

Re: [HACKERS] Hash Indexes

2016-10-20 Thread Robert Haas
On Tue, Oct 18, 2016 at 8:27 PM, Amit Kapila wrote: > By this problem, I mean to say deadlocks for suspended scans, that can > happen in btree for non-Mvcc or other type of scans where we don't > release pin during scan. In my mind, we have below options: > > a. problem of deadlocks for suspended

Re: [HACKERS] Indirect indexes

2016-10-20 Thread Alvaro Herrera
Joshua D. Drake wrote: > That said would it be possible to make this index an extension (like rum?). > Assuming of course we can get any required infrastructure changes done in a > general way. Well, the patch I currently have creates a separate index AM called "ibtree" which is an indirect btree

Re: [HACKERS] Remove vacuum_defer_cleanup_age

2016-10-20 Thread Robert Haas
On Wed, Oct 19, 2016 at 9:09 PM, Bruce Momjian wrote: > On Wed, Oct 19, 2016 at 08:17:46PM -0400, Robert Haas wrote: >> On Wed, Oct 19, 2016 at 12:59 PM, Bruce Momjian wrote: >> > Uh, vacuum_defer_cleanup_age sets an upper limit on how long, in terms >> > of xids, that a standby query can run bef

Re: [HACKERS] Remove autovacuum GUC?

2016-10-20 Thread Robert Haas
On Wed, Oct 19, 2016 at 9:24 PM, Joshua D. Drake wrote: > After all these years, we are still regularly running into people who say, > "performance was bad so we disabled autovacuum". I am not talking about once > in a while, it is often. I would like us to consider removing the autovacuum > optio

Re: [HACKERS] Indirect indexes

2016-10-20 Thread Joshua D. Drake
On 10/20/2016 06:39 AM, Alvaro Herrera wrote: Bruce Momjian wrote: Also, it seems indirect indexes would be useful for indexing columns that are not updated frequently on tables that are updated frequently, and whose primary key is not updated frequently. That's quite a logic problem for user

Re: [HACKERS] Indirect indexes

2016-10-20 Thread Bruce Momjian
On Thu, Oct 20, 2016 at 10:39:23AM -0300, Alvaro Herrera wrote: > Bruce Momjian wrote: > > > Just to clarify, if a feature improves performance by 1%, but is enabled > > by default, that is 10x more useful across our entire user base as the > > feature numbers listed above, 1% vs 0.1%. > > Great.

Re: [HACKERS] Indirect indexes

2016-10-20 Thread Alvaro Herrera
Bruce Momjian wrote: > Just to clarify, if a feature improves performance by 1%, but is enabled > by default, that is 10x more useful across our entire user base as the > feature numbers listed above, 1% vs 0.1%. Great. But not all users are alike. We have big profile users that write blog post

Re: [HACKERS] PATCH: two slab-like memory allocators

2016-10-20 Thread Tomas Vondra
On 10/19/2016 02:51 PM, Tomas Vondra wrote: ... > Yeah. There are three contexts in reorder buffers: - changes (fixed size) - txns (fixed size) - tuples (variable size) The first two work perfectly fine with Slab. The last one (tuples) is used to allocate variable-sized bits, so I've tried t

Re: [HACKERS] Disable autovacuum guc?

2016-10-20 Thread Joshua D. Drake
On 10/19/2016 07:22 PM, Josh Berkus wrote: On 10/19/2016 06:27 PM, Joshua D. Drake wrote: Hello, After all these years, we are still regularly running into people who say, "performance was bad so we disabled autovacuum". I am not talking about once in a while, it is often. I would like us to co

[HACKERS] Remove autovacuum GUC?

2016-10-20 Thread Joshua D. Drake
Hello, After all these years, we are still regularly running into people who say, "performance was bad so we disabled autovacuum". I am not talking about once in a while, it is often. I would like us to consider removing the autovacuum option. Here are a few reasons: 1. It does not hurt anyo

[HACKERS] Re: File content logging during execution of COPY queries (was: Better logging of COPY queries if log_statement='all')

2016-10-20 Thread Stephen Frost
Aleksander, * Aleksander Alekseev (a.aleks...@postgrespro.ru) wrote: > > The idea is to record application workload in real environment and write > > a benchmark based on this record. Then using this benchmark we could try > > different OS/DBMS configuration (or maybe hardware), find an extremum,

Re: [HACKERS] Fun fact about autovacuum and orphan temp tables

2016-10-20 Thread Constantin S. Pan
On Mon, 5 Sep 2016 14:54:05 +0300 Grigory Smolkin wrote: > Hello, hackers! > > We were testing how well some application works with PostgreSQL and > stumbled upon an autovacuum behavior which I fail to understand. > Application in question have a habit to heavily use temporary tables > in funny

Re: [HACKERS] Indirect indexes

2016-10-20 Thread Bruce Momjian
On Wed, Oct 19, 2016 at 01:04:16PM -0400, Bruce Momjian wrote: > On Wed, Oct 19, 2016 at 06:58:05PM +0200, Simon Riggs wrote: > > >> I agree. Also, I think the recheck mechanism will have to be something > > >> like > > >> what I wrote for WARM i.e. only checking for index quals won't be enough >

Re: [HACKERS] Danger of automatic connection reset in psql

2016-10-20 Thread Oleksandr Shulgin
On Thu, Oct 20, 2016 at 12:28 PM, Oleksandr Shulgin < oleksandr.shul...@zalando.de> wrote: > > Since this is already an improvement, I'm attaching a patch. > > If on the other hand, someone is pasting into psql's terminal a block of > commands enclosed in BEGIN/COMMIT, the same bug is triggered: B

Re: [HACKERS] Danger of automatic connection reset in psql

2016-10-20 Thread Oleksandr Shulgin
On Thu, Oct 20, 2016 at 12:28 PM, Oleksandr Shulgin < oleksandr.shul...@zalando.de> wrote: > > > I'm not the first one to discover that, a search in archives gives at least 3 results: > > https://www.postgresql.org/message-id/flat/1097174870.9273.8.camel%40ipso.snappymail.ca#1097174870.9273.8.ca...

Re: [HACKERS] Gather Merge

2016-10-20 Thread Rushabh Lathia
On Thu, Oct 20, 2016 at 12:22 AM, Peter Geoghegan wrote: > On Tue, Oct 4, 2016 at 11:05 PM, Rushabh Lathia > wrote: > > Query 4: With GM 7901.480 -> Without GM 9064.776 > > Query 5: With GM 53452.126 -> Without GM 55059.511 > > Query 9: With GM 52613.132 -> Without GM 98206.793 > > Query 15:

[HACKERS] Danger of automatic connection reset in psql

2016-10-20 Thread Oleksandr Shulgin
Hi Hackers! When using psql interactively one might be tempted to guard potentially destructive commands such as "UPDATE / DELETE / DROP " by starting the input line with an explicit "BEGIN; ...". This has the added benefit that then you invoke the command by reverse-searching the command history

[HACKERS] File content logging during execution of COPY queries (was: Better logging of COPY queries if log_statement='all')

2016-10-20 Thread Aleksander Alekseev
> > > According to my colleagues it would be very nice to have this feature. > > > For instance, if you are trying to optimize PostgreSQL for application > > > that uses COPY and you don't have access to or something like this. > > > It could also be useful in some other cases. > > > > This use-c

Re: [HACKERS] Aggregate Push Down - Performing aggregation on foreign server

2016-10-20 Thread Jeevan Chalke
On Thu, Oct 20, 2016 at 12:49 PM, Ashutosh Bapat < ashutosh.ba...@enterprisedb.com> wrote: > The patch compiles and make check-world doesn't show any failures. > > >> > > > > > > I have tried it. Attached separate patch for it. > > However I have noticed that istoplevel is always false (at-least f

Re: [HACKERS] postgres_fdw : altering foreign table not invalidating prepare statement execution plan.

2016-10-20 Thread Etsuro Fujita
On 2016/10/20 16:27, Ashutosh Bapat wrote: I wrote: Besides that, I modified add_rte_to_flat_rtable so that the plan's dependencies are tracked, but that would lead to tracking the dependencies of unreferenced foreign tables in dead subqueries or the dependencies of foreign tables excluded from

Re: [HACKERS] Gather Merge

2016-10-20 Thread Amit Kapila
On Tue, Oct 18, 2016 at 5:29 PM, Rushabh Lathia wrote: > On Mon, Oct 17, 2016 at 2:26 PM, Amit Kapila > wrote: >> >> There is lot of common code between ExecGatherMerge and ExecGather. >> Do you think it makes sense to have a common function to avoid the >> duplicity? >> >> I see there are small

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

2016-10-20 Thread Dilip Kumar
On Thu, Oct 13, 2016 at 12:25 AM, Robert Haas wrote: > I agree with these conclusions. I had a chance to talk with Andres > this morning at Postgres Vision and based on that conversation I'd > like to suggest a couple of additional tests: > > 1. Repeat this test on x86. In particular, I think yo

Re: [HACKERS] postgres_fdw : altering foreign table not invalidating prepare statement execution plan.

2016-10-20 Thread Ashutosh Bapat
> > >> In fact, I am not in >> favour of tracking the query dependencies for UPDATE/DELETE since we >> don't have any concrete example as to when that would be needed. > > > Right, but as I said before, some FDW might consult FDW options stored in > those objects during AddForeignUpdateTargets, so