Re: [HACKERS] pg_dump -Fd and compression level

2015-07-24 Thread Marc Mamin
>Andrew Dunstan writes: >> Hmm. Yeah. It looks like commit a7ad5cf0cfcfab8418000d652fa4f0c6ad6c8911 >> changed from using the default compression for libz to using the >> compression set in pg_dump options, which defaults to 0. This actually >> seems like the right thing to do, but it certainly

Re: [HACKERS] [PROPOSAL] VACUUM Progress Checker.

2015-07-24 Thread Pavel Stehule
2015-07-25 0:41 GMT+02:00 Josh Berkus : > On 07/24/2015 11:06 AM, Jim Nasby wrote: > > On 7/23/15 5:18 AM, Thakur, Sameer wrote: > >> Hello, > >>> >logged > 25 times > >> Sorry, it is much lower at 7 times. Does not change overall point though > > > > I think it's related to the problem of figurin

Re: [HACKERS] Supporting TAP tests with MSVC and Windows

2015-07-24 Thread Michael Paquier
On Sat, Jul 25, 2015 at 3:27 AM, Andrew Dunstan wrote: > > On 07/24/2015 01:27 PM, Heikki Linnakangas wrote: >> >> On 06/25/2015 07:40 AM, Michael Paquier wrote: >>> >>> On Tue, May 26, 2015 at 3:39 PM, Michael Paquier wrote: Here is v6, a rebased version on HEAD (79f2b5d). There were so

Re: [HACKERS] Reduce ProcArrayLock contention

2015-07-24 Thread Amit Kapila
On Fri, Jul 24, 2015 at 4:26 PM, Pavan Deolasee wrote: > > > > On Mon, Jun 29, 2015 at 8:57 PM, Amit Kapila wrote: >> >> >> >> pgbench setup >> >> scale factor - 300 >> Data is on magnetic disk and WAL on ssd. >> pgbench -M prepared tpc-b >> >> Head : commit 51d0fe5d >> P

Re: [HACKERS] MultiXact member wraparound protections are now enabled

2015-07-24 Thread Robert Haas
On Fri, Jul 24, 2015 at 9:14 PM, Peter Eisentraut wrote: > On 7/22/15 4:45 PM, Robert Haas wrote: >> But it seemed to me that this could be rather confusing. I thought it >> would be better to be explicit about whether the protections are >> enabled in all cases. That way, (1) if you see the mes

Re: [HACKERS] pg_dump -Fd and compression level

2015-07-24 Thread Tom Lane
Andrew Dunstan writes: > Hmm. Yeah. It looks like commit a7ad5cf0cfcfab8418000d652fa4f0c6ad6c8911 > changed from using the default compression for libz to using the > compression set in pg_dump options, which defaults to 0. This actually > seems like the right thing to do, but it certainly shou

Re: [HACKERS] Proposing COPY .. WITH PERMISSIVE

2015-07-24 Thread dinesh kumar
On Fri, Jul 24, 2015 at 10:22 AM, Robert Haas wrote: > On Thu, Jul 23, 2015 at 8:15 PM, dinesh kumar > wrote: > > On Thu, Jul 23, 2015 at 9:21 AM, Robert Haas > wrote: > >> > >> On Thu, Jul 23, 2015 at 12:19 PM, dinesh kumar > > >> wrote: > >> > Sorry for my unclear description about the prop

Re: [HACKERS] pg_dump -Fd and compression level

2015-07-24 Thread Andrew Dunstan
On 07/24/2015 05:22 PM, Marc Mamin wrote: After our last upgrade, we've noticed a 10-20% size increase of our dump size. This comes from our backup scripts were pg_dump was called without setting -Z So it seems, that this fix did modify the default compression to use: http://michael.otacoo.com/

Re: [HACKERS] TABLESAMPLE patch is really in pretty sad shape

2015-07-24 Thread Tom Lane
Petr Jelinek writes: > I was wondering if we should perhaps cache the output of GetTsmRoutine > as we call it up to 4 times in the planner now but it's relatively cheap > call (basically just makeNode) so it's probably not worth it. Yeah, I was wondering about that too. The way to do it would

Re: [HACKERS] fdw_scan_tlist for foreign table scans breaks EPQ testing, doesn't it?

2015-07-24 Thread Kouhei Kaigai
> -Original Message- > From: Robert Haas [mailto:robertmh...@gmail.com] > Sent: Saturday, July 25, 2015 2:59 AM > To: Kaigai Kouhei(海外 浩平) > Cc: Etsuro Fujita; pgsql-hackers@postgresql.org > Subject: ##freemail## Re: [HACKERS] fdw_scan_tlist for foreign table scans > breaks > EPQ testi

Re: [HACKERS] MultiXact member wraparound protections are now enabled

2015-07-24 Thread Peter Eisentraut
On 7/22/15 4:45 PM, Robert Haas wrote: > But it seemed to me that this could be rather confusing. I thought it > would be better to be explicit about whether the protections are > enabled in all cases. That way, (1) if you see the message saying > they are enabled, they are enabled; (2) if you se

Re: [HACKERS] TABLESAMPLE patch is really in pretty sad shape

2015-07-24 Thread Petr Jelinek
On 2015-07-25 00:36, Tom Lane wrote: I wrote: Petr Jelinek writes: The only major difference that I see so far and I'd like you to incorporate that into your patch is that I renamed the SampleScanCost to SampleScanGetRelSize because that reflects much better the use of it, it isn't really used

Re: [HACKERS] [PROPOSAL] VACUUM Progress Checker.

2015-07-24 Thread Mike Blackwell
​Something like that would be helpful. I just had to stop one after an hour and have no idea how much longer it would have taken.​ __ *Mike Blackwell | Technical Analyst, Distribution Services/Rollout Management | RR

Re: [HACKERS] [PROPOSAL] VACUUM Progress Checker.

2015-07-24 Thread Josh Berkus
On 07/24/2015 11:06 AM, Jim Nasby wrote: > On 7/23/15 5:18 AM, Thakur, Sameer wrote: >> Hello, >>> >logged > 25 times >> Sorry, it is much lower at 7 times. Does not change overall point though > > I think it's related to the problem of figuring out how many dead tuples > you expect to find in the

Re: [HACKERS] pg_dump -Fd and compression level

2015-07-24 Thread Marc Mamin
>> After our last upgrade, we've noticed a 10-20% size increase of our dump >> size. >> This comes from our backup scripts were pg_dump was called without setting -Z >> >> So it seems, that this fix did modify the default compression to use: >> http://michael.otacoo.com/postgresql-2/pg_dump-direc

Re: [HACKERS] RLS fails to work with UPDATE ... WHERE CURRENT OF

2015-07-24 Thread Joe Conway
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 07/14/2015 12:40 PM, Dean Rasheed wrote: > On 14 July 2015 at 13:59, Robert Haas > wrote: >> On Thu, Jul 9, 2015 at 5:47 PM, Joe Conway >> wrote: >>> On 06/08/2015 02:08 AM, Dean Rasheed wrote: Actually I think it is fixable just by allowing

Re: [HACKERS] Autonomous Transaction is back

2015-07-24 Thread Merlin Moncure
On Thu, Jul 23, 2015 at 1:49 PM, Josh Berkus wrote: > Batch Jobs: large data-manipulation tasks which need to be broken up > into segments, with each segment committing separately. Example: > updating 1 million records in batches of 1000. Autonomous transactions are not a good fit for this case;

Re: [HACKERS] RFC: replace pg_stat_activity.waiting with something more descriptive

2015-07-24 Thread Robert Haas
On Thu, Jul 23, 2015 at 3:01 PM, Ildus Kurbangaliev wrote: > Hello. > I’ve changed the previous patch. `group` field in LWLock is removed, so the > size of LWLock will not increase. > Instead of the `group` field I've created new tranches for LWLocks from > MainLWLocksArray. This allowed to remo

Re: [HACKERS] Free indexed_tlist memory explicitly within set_plan_refs()

2015-07-24 Thread Peter Geoghegan
On Fri, Jul 24, 2015 at 3:08 AM, Andres Freund wrote: >> + else >> + { >> + Node *nattExpr = list_nth(idxExprs, (natt - 1) - >> nplain); >> + >> + /* >> + * Note that unlike routines like >> match_index_to

Re: [HACKERS] Proposal for CSN based snapshots

2015-07-24 Thread Andres Freund
On 2015-07-24 14:21:39 -0400, Robert Haas wrote: > - Mixing synchronous_commit=on and synchronous_commit=off won't work > as well, because if the LSN ordering of commit records matches the > order in which transactions become visible, then an async-commit > transaction can't become visible before a

Re: [HACKERS] pg_dump -Fd and compression level

2015-07-24 Thread Andrew Dunstan
On 07/24/2015 02:52 AM, Marc Mamin wrote: Hello, After our last upgrade, we've noticed a 10-20% size increase of our dump size. This comes from our backup scripts were pg_dump was called without setting -Z So it seems, that this fix did modify the default compression to use: http://michael.ota

Re: [HACKERS] Supporting TAP tests with MSVC and Windows

2015-07-24 Thread Andrew Dunstan
On 07/24/2015 01:27 PM, Heikki Linnakangas wrote: On 06/25/2015 07:40 AM, Michael Paquier wrote: On Tue, May 26, 2015 at 3:39 PM, Michael Paquier wrote: Here is v6, a rebased version on HEAD (79f2b5d). There were some conflicts with the indentation and some other patches related to pg_rewind a

Re: [HACKERS] Proposal for CSN based snapshots

2015-07-24 Thread Robert Haas
On Fri, Jul 24, 2015 at 1:00 PM, Simon Riggs wrote: > It depends on the exact design we use to get that. Certainly we do not want > them if they cause a significant performance regression. Yeah. I think the performance worries expressed so far are: - Currently, if you see an XID that is between

Re: [HACKERS] "make check" changes have caused buildfarm deterioration.

2015-07-24 Thread Andrew Dunstan
On 07/24/2015 01:21 PM, Jeff Janes wrote: On Tue, Jul 21, 2015 at 6:31 PM, Michael Paquier mailto:michael.paqu...@gmail.com>> wrote: On Wed, Jul 22, 2015 at 10:04 AM, Peter Eisentraut mailto:pete...@gmx.net>> wrote: > On 7/21/15 10:00 AM, Tom Lane wrote: >> I agree; this change

Re: [HACKERS] optimizing vacuum truncation scans

2015-07-24 Thread Jim Nasby
On 7/22/15 11:11 AM, Jeff Janes wrote: I guess add an "Assert(InRecovery || running_a_vacuum);" to the visibilitymap_set with a comment there, except that I don't know how to implement running_a_vacuum so that it covers manual vacs as well as autovac. static bool RunningVacuum; ? It only needs

Re: [HACKERS] [PROPOSAL] VACUUM Progress Checker.

2015-07-24 Thread Jim Nasby
On 7/23/15 5:18 AM, Thakur, Sameer wrote: Hello, >logged > 25 times Sorry, it is much lower at 7 times. Does not change overall point though I think it's related to the problem of figuring out how many dead tuples you expect to find in the overall heap, which you need to do to have any hope

Re: [HACKERS] [PROPOSAL] VACUUM Progress Checker.

2015-07-24 Thread Robert Haas
On Fri, Jul 24, 2015 at 2:00 PM, Jim Nasby wrote: >> You need to restrict the amount of info, because you've got to >> preallocate enough shared memory to store all the data that somebody >> might report. > > I was thinking your DSM stuff would come into play here. We wouldn't want to > be realloc

Re: [HACKERS] [PROPOSAL] VACUUM Progress Checker.

2015-07-24 Thread Jim Nasby
On 7/23/15 2:43 PM, Robert Haas wrote: On Wed, Jul 22, 2015 at 11:28 AM, Jim Nasby wrote: If we want to expose that level of detail, I think either JSON or arrays would make more sense, so we're not stuck with a limited amount of info. Perhaps DDL would be OK with the numbers you suggested, but

Re: [HACKERS] fdw_scan_tlist for foreign table scans breaks EPQ testing, doesn't it?

2015-07-24 Thread Robert Haas
On Thu, Jul 23, 2015 at 8:27 PM, Kouhei Kaigai wrote: > A dark side is, as discussed in this thread, complexity of EvalPlanQual. > RefetchForeignRow() returns a tuple based on foreign table definition, > on the other hands, whole-row var points a tuple based on fdw_scan_tlist > if exists. > An alt

Re: [HACKERS] Free indexed_tlist memory explicitly within set_plan_refs()

2015-07-24 Thread Andres Freund
On July 24, 2015 7:57:43 PM GMT+02:00, Peter Geoghegan wrote: >On Fri, Jul 24, 2015 at 2:58 AM, Andres Freund >wrote: >> I've pushed a version of your patch. I just opted to remove >p_is_update >> instead of allowing both to be set at the same time. To me that >seemed >> simpler. > >I would be he

Re: [HACKERS] Improving replay of XLOG_BTREE_VACUUM records

2015-07-24 Thread Jim Nasby
On 7/24/15 1:53 AM, Heikki Linnakangas wrote: On 05/02/2015 02:10 AM, Jim Nasby wrote: This looks like a good way to address this until the more significant work can be done. I'm not a fan of "RBM_ZERO_NO_BM_VALID"; how about RBM_ZERO_BM_INVALID? or BM_NOT_VALID? Or maybe I'm just trying to imp

Re: [HACKERS] Free indexed_tlist memory explicitly within set_plan_refs()

2015-07-24 Thread Peter Geoghegan
On Fri, Jul 24, 2015 at 2:58 AM, Andres Freund wrote: > I've pushed a version of your patch. I just opted to remove p_is_update > instead of allowing both to be set at the same time. To me that seemed > simpler. I would be hesitant to remove a struct field from a struct that appears as a hook arg

Re: [HACKERS] Supporting TAP tests with MSVC and Windows

2015-07-24 Thread Heikki Linnakangas
On 06/25/2015 07:40 AM, Michael Paquier wrote: On Tue, May 26, 2015 at 3:39 PM, Michael Paquier wrote: Here is v6, a rebased version on HEAD (79f2b5d). There were some conflicts with the indentation and some other patches related to pg_rewind and initdb's tests. Attached is v7, rebased on 0b15

Re: [HACKERS] Proposing COPY .. WITH PERMISSIVE

2015-07-24 Thread Robert Haas
On Thu, Jul 23, 2015 at 8:15 PM, dinesh kumar wrote: > On Thu, Jul 23, 2015 at 9:21 AM, Robert Haas wrote: >> >> On Thu, Jul 23, 2015 at 12:19 PM, dinesh kumar >> wrote: >> > Sorry for my unclear description about the proposal. >> > >> > "WITH PERMISSIVE" is equal to our existing behavior. That

Re: [HACKERS] "make check" changes have caused buildfarm deterioration.

2015-07-24 Thread Jeff Janes
On Tue, Jul 21, 2015 at 6:31 PM, Michael Paquier wrote: > On Wed, Jul 22, 2015 at 10:04 AM, Peter Eisentraut > wrote: > > On 7/21/15 10:00 AM, Tom Lane wrote: > >> I agree; this change may have seemed like a good idea at the time, but > >> it was not. Failures during "make check"'s install step

Re: [HACKERS] REVOKE [ADMIN OPTION FOR] ROLE

2015-07-24 Thread Robert Haas
On Thu, Jul 23, 2015 at 8:30 AM, Egor Rogov wrote: > So, the question: is it a documentation bug (as it seems to me), code bug, > or I missed something? Your analysis looks right to me, but I don't know whether the code or the documentation should be changed. This claim was added by Tom Lane in

Re: [HACKERS] [PATCH] postgres_fdw extension support

2015-07-24 Thread Paul Ramsey
On Thu, Jul 23, 2015 at 7:48 AM, Paul Ramsey wrote: > Here's an updated patch that clears the cache on changes to foreign > wrappers and servers. Any chance one of you folks could by my official commitfest reviewer? Appreciate all the feedback so far! https://commitfest.postgresql.org/6/304/ Th

Re: [HACKERS] Proposal for CSN based snapshots

2015-07-24 Thread Simon Riggs
On 24 July 2015 at 14:43, Alexander Korotkov wrote: > What is current state of this patch? Does community want CSN based > snapshots? > CSN snapshots could give us 1. More scalable GetSnapshotData() calls 2. Faster GetSnapshotData() calls 3. Smaller snapshots which could be passed around more

Re: [HACKERS] BRIN index and aborted transaction

2015-07-24 Thread Simon Riggs
On 23 July 2015 at 19:59, Josh Berkus wrote: > On 07/23/2015 11:18 AM, Robert Haas wrote: > > Cool. I'm not sure exactly what the right solution is either, but it > > seems like the current situation could very well lead to degrading > > index performance over time, with no way to put that right

Re: [HACKERS] creating extension including dependencies

2015-07-24 Thread Petr Jelinek
On 2015-07-22 07:12, Michael Paquier wrote: On Tue, Jul 21, 2015 at 11:20 PM, Tom Lane wrote: Petr Jelinek writes: ... My main question is if we are ok with SCHEMA having different behavior with CASCADE vs without CASCADE. I went originally with "no" and added the DEFAULT flag to SCHEMA. If t

Re: [HACKERS] POC: Cache data in GetSnapshotData()

2015-07-24 Thread Pavan Deolasee
On Mon, Feb 2, 2015 at 8:57 PM, Andres Freund wrote: > Hi, > > I've, for a while, pondered whether we couldn't find a easier way than > CSN to make snapshots cheaper as GetSnapshotData() very frequently is > one of the top profile entries. Especially on bigger servers, where the > pretty much gua

Re: [HACKERS] We need to support ForeignRecheck for late row locking, don't we?

2015-07-24 Thread Kouhei Kaigai
Fujita-san, > On 2015/07/22 19:10, Etsuro Fujita wrote: > > While working on the issue "Foreign join pushdown vs EvalPlanQual", I > > happened to notice odd behaviors of late row locking in FDWs. > > > I think the reason for that is because we don't check pushed-down quals > > inside an EPQ testi

Re: [HACKERS] TABLESAMPLE patch is really in pretty sad shape

2015-07-24 Thread Tom Lane
Petr Jelinek writes: > Ok, attached are couple of cosmetic changes - what I wrote above, plus > comment about why we do float8 + hashing for REPEATABLE (it's not > obvious IMHO) and one additional test query. OK, thanks. > Do you want to do the contrib changes yourself as well or should I take

Re: [HACKERS] TABLESAMPLE patch is really in pretty sad shape

2015-07-24 Thread Tom Lane
Petr Jelinek writes: > The only major difference that I see so far and I'd like you to > incorporate that into your patch is that I renamed the SampleScanCost to > SampleScanGetRelSize because that reflects much better the use of it, it > isn't really used for costing, but for getting the pages

Re: [HACKERS] Proposal for CSN based snapshots

2015-07-24 Thread Alexander Korotkov
On Wed, Aug 27, 2014 at 10:42 AM, Heikki Linnakangas < hlinnakan...@vmware.com> wrote: > On 08/27/2014 09:40 AM, Heikki Linnakangas wrote: > >> On 08/27/2014 08:23 AM, Jeff Davis wrote: >> >>> On Tue, 2014-08-26 at 13:45 +0300, Heikki Linnakangas wrote: >>> Yeah. This patch in the current sta

Re: [HACKERS] GSets: Fix bug involving GROUPING and HAVING together

2015-07-24 Thread Andrew Gierth
> "Jeevan" == Jeevan Chalke writes: Jeevan> Hi, Jeevan> This will fail too. This is in addition to your patch, not instead of it. -- Andrew (irc:RhodiumToad) -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgres

Re: [HACKERS] GSets: Fix bug involving GROUPING and HAVING together

2015-07-24 Thread Jeevan Chalke
Hi, This will fail too. Note that, when we have only one element in GROUPING SETS, we add that in group by list and set parse->groupingSets to NULL. And hence it will have same issue. However tests added in my patch failing too. Thanks -- Jeevan B Chalke Principal Software Engineer, Product D

Re: [HACKERS] Reduce ProcArrayLock contention

2015-07-24 Thread Pavan Deolasee
On Mon, Jun 29, 2015 at 8:57 PM, Amit Kapila wrote: > > > pgbench setup > > scale factor - 300 > Data is on magnetic disk and WAL on ssd. > pgbench -M prepared tpc-b > > Head : commit 51d0fe5d > Patch -1 : group_xid_clearing_at_trans_end_rel_v1 > > > Client Count/TPS18163

Re: [HACKERS] GSets: Fix bug involving GROUPING and HAVING together

2015-07-24 Thread Andrew Gierth
> "Andrew" == Andrew Gierth writes: Andrew> The other is that in subquery_planner, the optimization of Andrew> converting HAVING clauses to WHERE clauses is suppressed if Andrew> parse->groupingSets isn't empty. (It is empty if there's either Andrew> no group by clause at all, or if there

Re: [HACKERS] Free indexed_tlist memory explicitly within set_plan_refs()

2015-07-24 Thread Andres Freund
On 2015-05-28 18:31:56 -0700, Peter Geoghegan wrote: > Subject: [PATCH 3/3] Fix bug in unique index inference > > ON CONFLICT unique index inference had a thinko that could affect cases > where the user-supplied inference clause required that an attribute > match a particular (user named) collatio

Re: [HACKERS] Free indexed_tlist memory explicitly within set_plan_refs()

2015-07-24 Thread Andres Freund
On 2015-07-12 22:45:18 +0200, Andres Freund wrote: > I'm right now not really coming up with a better idea how to fix > this. So I guess I'll apply something close to this tomorrow. Took a bit longer than that :( I've pushed a version of your patch. I just opted to remove p_is_update instead of a

Re: [HACKERS] Eliminating CREATE INDEX comparator TID tie-breaker overhead

2015-07-24 Thread Nicolas Barbier
2015-07-24 Nicolas Barbier : > Especially useful would be to know whether interleaving a small number > of TID ordered streams (as would probably be generated by parallel > scans/processing) would result in an ordering that performs > significantly worse or not. I assume (but cannot prove) that in

Re: [HACKERS] Eliminating CREATE INDEX comparator TID tie-breaker overhead

2015-07-24 Thread Nicolas Barbier
2015-07-23 Robert Haas : > I think what you should do is go find out whether the second rationale > is valid or not. Knowing how much impact on performance such “non TID ordered” entries have, would of course be very useful for future patch authors to know. Especially useful would be to know whet

Re: [HACKERS] We need to support ForeignRecheck for late row locking, don't we?

2015-07-24 Thread Etsuro Fujita
On 2015/07/22 19:10, Etsuro Fujita wrote: > While working on the issue "Foreign join pushdown vs EvalPlanQual", I > happened to notice odd behaviors of late row locking in FDWs. > I think the reason for that is because we don't check pushed-down quals > inside an EPQ testing even if what was fetch

Re: [HACKERS] pgbench - allow backslash-continuations in custom scripts

2015-07-24 Thread Kyotaro HORIGUCHI
Hi, all. Attatched is the revised version of this patch. The first patch is not changed from before. The second is fixed a kind of bug. Ths third is the new one to allow backslash continuation for backslash commands. hoge.sql is the test custom script. == At Fri, 24 Jul 2015 07:39:16 +020

Re: [HACKERS] RFC: replace pg_stat_activity.waiting with something more descriptive

2015-07-24 Thread Ildus Kurbangaliev
> On Jul 24, 2015, at 7:26 AM, Amit Kapila wrote: > > On Fri, Jul 24, 2015 at 12:31 AM, Ildus Kurbangaliev > wrote: > Hello. > I’ve changed the previous patch. `group` field in LWLock is removed, so the > size of LWLock will not increase. > Instead of the `group` field I've created new tranch

Re: [HACKERS] Asynchronous execution on FDW

2015-07-24 Thread Kouhei Kaigai
Hello Horiguchi-san, > > > As for ForeignScan, it is merely an API for FDW and does nothing > > > substantial so it would have nothing special to do. As for > > > postgres_fdw, current patch restricts one execution per one > > > foreign server at once by itself. We would have to provide > > > anot

Re: [HACKERS] Doubt about AccessExclusiveLock in ALTER TABLE .. SET ( .. );

2015-07-24 Thread Michael Paquier
On Fri, Jul 24, 2015 at 7:11 AM, Fabrízio de Royes Mello wrote: > On Thu, Jul 2, 2015 at 2:03 PM, Simon Riggs wrote: >> Looks functionally complete >> >> Need a test to show that ALTER TABLE works on views, as discussed on this >> thread. And confirmation that pg_dump is not broken by this. >> >>