Re: [HACKERS] Possible duplicate release of buffer lock.

2016-08-04 Thread Kyotaro HORIGUCHI
Hello, At Thu, 4 Aug 2016 16:09:17 -0700, Peter Geoghegan wrote in > On Wed, Aug 3, 2016 at 1:31 AM, Kyotaro HORIGUCHI > wrote: > > The first line is emitted for simultaneous

Re: [HACKERS] Possible duplicate release of buffer lock.

2016-08-04 Thread Kyotaro HORIGUCHI
At Wed, 03 Aug 2016 12:00:33 -0400, Tom Lane wrote in <26373.1470240...@sss.pgh.pa.us> > Kyotaro HORIGUCHI writes: > > My point here is that if concurrent deletion can't be perfomed by > > the current implement, this while loop could be

Re: [HACKERS] Heap WARM Tuples - Design Draft

2016-08-04 Thread Pavan Deolasee
On Fri, Aug 5, 2016 at 8:23 AM, Claudio Freire wrote: > On Thu, Aug 4, 2016 at 11:15 PM, Bruce Momjian wrote: > > > > > OK, that's a lot of text, and I am confused. Please tell me the > > advantage of having an index point to a string of HOT chains,

Re: [HACKERS] multivariate statistics (v19)

2016-08-04 Thread Michael Paquier
On Wed, Aug 3, 2016 at 10:58 AM, Tomas Vondra wrote: > Attached is v19 of the "multivariate stats" patch series - essentially v18 > rebased on top of current master. Aside from a few bug fixes, the main > improvement is addition of SGML docs demonstrating the

Re: [HACKERS] pg_ctl promote wait

2016-08-04 Thread Michael Paquier
On Wed, Aug 3, 2016 at 9:35 PM, Peter Eisentraut wrote: > Updated patch set for pg_ctl promote -w, incorporating AFAICT all of the > previous reviews, in particular Michael Paquier's changes to the > StartupXLOG() ordering, and rebasing on top of >

Re: [HACKERS] Detecting skipped data from logical slots (data silently skipped)

2016-08-04 Thread Robert Haas
On Wed, Aug 3, 2016 at 9:39 AM, Craig Ringer wrote: > I think we have a bit of a problem with the behaviour specified for logical > slots, one that makes it hard to prevent a outdated snapshot or backup of a > logical-slot-using downstream from knowing it's missing a chunk

Re: [HACKERS] Heap WARM Tuples - Design Draft

2016-08-04 Thread Claudio Freire
On Fri, Aug 5, 2016 at 12:44 AM, Bruce Momjian wrote: >> UPDATE t SET dat = dat + 1, id = 3, someid = 3 where id = 2; > > This is ends the WARM chain, and creates new index entries because all > indexes are changed. > >> UPDATE t SET dat = dat + 1, id = 1, someid = 2 where id =

[HACKERS] Re: [COMMITTERS] pgsql: Prevent "snapshot too old" from trying to return pruned TOAST tu

2016-08-04 Thread Robert Haas
On Thu, Aug 4, 2016 at 10:58 PM, Robert Haas wrote: > On Thu, Aug 4, 2016 at 7:23 PM, Tom Lane wrote: >> Robert Haas writes: >>> Prevent "snapshot too old" from trying to return pruned TOAST tuples. >> >> Looks like this patch

Re: [HACKERS] PostgreSQL 10 Roadmaps

2016-08-04 Thread Etsuro Fujita
On 2016/08/02 23:54, Simon Riggs wrote: https://wiki.postgresql.org/wiki/PostgreSQL10_Roadmap Thank you for creating the wiki page! I added a link to the NTT roadmap page on the links page. We hope that we can collaborate on the projects with people working at other companies or with

Re: [HACKERS] Heap WARM Tuples - Design Draft

2016-08-04 Thread Bruce Momjian
On Thu, Aug 4, 2016 at 11:53:05PM -0300, Claudio Freire wrote: > The point is avoiding duplicate rows in the output of index scans. > > I don't think you can avoid it simply by applying index condition > rechecks as the original proposal implies, in this case: > > CREATE TABLE t (id integer not

Re: [HACKERS] [sqlsmith] FailedAssertion("!(XLogCtl->Insert.exclusiveBackup)", File: "xlog.c", Line: 10200)

2016-08-04 Thread Michael Paquier
On Thu, Aug 4, 2016 at 4:38 PM, Michael Paquier wrote: > Andreas, with the patch attached is the assertion still triggered? > > Thoughts from others are welcome as well. Self review: * of streaming base backups currently in progress. forcePageWrites is set

Re: [HACKERS] Heap WARM Tuples - Design Draft

2016-08-04 Thread Claudio Freire
On Thu, Aug 4, 2016 at 11:15 PM, Bruce Momjian wrote: > On Thu, Aug 4, 2016 at 09:53:31PM -0300, Claudio Freire wrote: >> It's the other way around. A WARM chain has to span whole HOT chains. >> The WARM chain always begins in the head of a HOT chain (or non-HOT >> tuple of

Re: [HACKERS] Heap WARM Tuples - Design Draft

2016-08-04 Thread Bruce Momjian
On Thu, Aug 4, 2016 at 09:53:31PM -0300, Claudio Freire wrote: > It's the other way around. A WARM chain has to span whole HOT chains. > The WARM chain always begins in the head of a HOT chain (or non-HOT > tuple of course), and cannot end before the HOT chain ends. > > That is, all within a

Re: [HACKERS] max_parallel_degree > 0 for 9.6 beta

2016-08-04 Thread Michael Paquier
On Thu, Aug 4, 2016 at 10:52 PM, David G. Johnston wrote: > My initial reaction was +1 but now I'm leaning toward enabled by default. > > Those who would upgrade to 9.6 within a year of its release are most likely, > process and personality wise, to be those for whom

Re: [HACKERS] Heap WARM Tuples - Design Draft

2016-08-04 Thread Claudio Freire
On Thu, Aug 4, 2016 at 7:59 AM, Pavan Deolasee wrote: > So marking the index would require us to mark both old and new index tuples > as requiring re-check. That requires an additional index scan to locate the > old row and then an additional write to force it to

Re: [HACKERS] Heap WARM Tuples - Design Draft

2016-08-04 Thread Bruce Momjian
>On Thu, Aug 4, 2016 at 06:21:48PM -0400, Bruce Momjian wrote: > Another approach would be to keep updates on the same page in a single > WARM chain, even if all indexes have changed columns. We will already > have code to walk the HOT chain looking for matches, and at most one > tuple in the

Re: [HACKERS] Heap WARM Tuples - Design Draft

2016-08-04 Thread Claudio Freire
On Thu, Aug 4, 2016 at 7:21 PM, Bruce Momjian wrote: > On Thu, Aug 4, 2016 at 03:23:10PM -0300, Claudio Freire wrote: >> INSERT INTO TABLE t (id,val,dat) VALUES (32,'a','blabla'); >> UPDATE t SET dat='blabla2' where id = 32; >> UPDATE t SET dat='blabla3', id=33 where id = 32;

Re: [HACKERS] New version numbering practices

2016-08-04 Thread Craig Ringer
On 5 August 2016 at 04:31, Tom Lane wrote: > In short: if we'd done it that way all along, it would've been nice. But encouraging people to change horses now isn't really going to > make things better. What is far more likely to happen, if we were > to do that, is that

Re: [HACKERS] Fwd: [BUGS] BUG #14247: COMMENT is restored on wrong database

2016-08-04 Thread Craig Ringer
On 5 August 2016 at 05:03, David G. Johnston wrote: > ​ > I'm all for an elegant solution here though at some point having a working > solution now beats waiting for someone to willingly dive more deeply into > pg_dump. I too seem to recall previous proposals for

Re: [HACKERS] Heap WARM Tuples - Design Draft

2016-08-04 Thread Bruce Momjian
On Thu, Aug 4, 2016 at 10:58:33PM +0530, Pavan Deolasee wrote: > 2. I also think that most often the tuple that will be updated will have its > t_ctid pointing to itself. This sounds more invasive, but can we somehow Agreed! I think it has to. Great idea. > utilise the t_ctid field to store

Re: [HACKERS] Possible duplicate release of buffer lock.

2016-08-04 Thread Peter Geoghegan
On Wed, Aug 3, 2016 at 1:31 AM, Kyotaro HORIGUCHI wrote: > The first line is emitted for simultaneous deletion of a index > page, which is impossible by design in a consistent state so the > complained situation should be the result of an index corruption > before

Re: [HACKERS] Bogus ANALYZE results for an otherwise-unique column with many nulls

2016-08-04 Thread Tom Lane
I wrote: > Looking around, there are a couple of places outside commands/analyze.c > that are making the same mistake, so this patch isn't complete, but it > illustrates what needs to be done. Here's a more complete patch. regards, tom lane diff --git

Re: [HACKERS] Heap WARM Tuples - Design Draft

2016-08-04 Thread Bruce Momjian
On Thu, Aug 4, 2016 at 03:33:16PM -0300, Claudio Freire wrote: > On Thu, Aug 4, 2016 at 3:23 PM, Claudio Freire wrote: > >> That is a new HOT chain given our current code, but > >> would the new tuple addition realize it needs to create a new index > >> tuple? The new

[HACKERS] Bogus ANALYZE results for an otherwise-unique column with many nulls

2016-08-04 Thread Tom Lane
I looked into the problem described at https://www.postgresql.org/message-id/flat/VisenaEmail.26.df42f82acae38a58.156463942b8%40tc7-visena and I believe I've reproduced it: the requirement is that the inner join column for the antijoin must contain a lot of NULL values, and what isn't NULL must be

Re: [HACKERS] New version numbering practices

2016-08-04 Thread Andrew Dunstan
On 08/01/2016 04:25 PM, Tom Lane wrote: Andrew Dunstan writes: Somewhat related is how we name the git branches. It would help me from a buildfarm POV if we kept lexically them sortable, which could be done at least for the next 90 major releases :-) by adding an

Re: [HACKERS] Heap WARM Tuples - Design Draft

2016-08-04 Thread Bruce Momjian
On Thu, Aug 4, 2016 at 03:23:10PM -0300, Claudio Freire wrote: > INSERT INTO TABLE t (id,val,dat) VALUES (32,'a','blabla'); > UPDATE t SET dat='blabla2' where id = 32; > UPDATE t SET dat='blabla3', id=33 where id = 32; > UPDATE t SET val='b' where id = 33; > UPDATE t SET dat='blabla4' where id =

Re: [HACKERS] Oddity with NOT IN

2016-08-04 Thread Marko Tiikkaja
On 2016-08-04 11:23 PM, Jim Nasby wrote: I've got a customer that discovered something odd... SELECT f1 FROM v1 WHERE f2 not in (SELECT bad FROM v2 WHERE f3 = 1); does not error, even though bad doesn't exist, but I'm guessing there's a v1.bad? This is a common mistake, and also why I

[HACKERS] Oddity with NOT IN

2016-08-04 Thread Jim Nasby
I've got a customer that discovered something odd... SELECT f1 FROM v1 WHERE f2 not in (SELECT bad FROM v2 WHERE f3 = 1); does not error, even though bad doesn't exist, but SELECT bad FROM v2 WHERE f3 = 1; gives ERROR: column "bad" does not exist Is that expected? This is on 9.4.8, and

Re: [HACKERS] Pgbench performance tuning?

2016-08-04 Thread Jeff Janes
On Thu, Aug 4, 2016 at 10:48 AM, Greg Stark wrote: > I'm trying to run pgbench on a moderately beefy machine (4-core 3.4GHz > with 32G of ram and md mirrored spinning rust drives) at scale 300 > with 32 clients with duration of 15min. I'm getting TPS numbers > between 60-150 which

Re: [HACKERS] Fwd: [BUGS] BUG #14247: COMMENT is restored on wrong database

2016-08-04 Thread David G. Johnston
On Thu, Aug 4, 2016 at 4:50 PM, Tom Lane wrote: > Robert Haas writes: > > On Tue, Aug 2, 2016 at 5:42 PM, David G. Johnston > > wrote: > > The fact that pg_dump is emitting COMMENT ON DATABASE at all is > > fundamentally

Re: [HACKERS] Fwd: [BUGS] BUG #14247: COMMENT is restored on wrong database

2016-08-04 Thread Tom Lane
Robert Haas writes: > On Tue, Aug 2, 2016 at 5:42 PM, David G. Johnston > wrote: > The fact that pg_dump is emitting COMMENT ON DATABASE at all is > fundamentally wrong given the existing division-of-labor decisions, > namely that pg_dump is

Re: [HACKERS] [Patch] RBTree iteration interface improvement

2016-08-04 Thread Robert Haas
On Thu, Jul 28, 2016 at 1:57 PM, Tom Lane wrote: > Aleksander Alekseev writes: >>> Can you explain use case where you need it? > >> ... Or maybe you have different objects, e.g. IndexScanDesc's, that should >> iterate over some tree's independently

Re: [HACKERS] New version numbering practices

2016-08-04 Thread Tom Lane
Peter Eisentraut writes: > On 8/3/16 1:20 PM, Tom Lane wrote: >> I was just wondering whether it would be worth starting to use "git tag -a". > One should always use annotated tags for public releases. That way, the > tag is its own Git object that cannot be

Re: [HACKERS] Fwd: [BUGS] BUG #14247: COMMENT is restored on wrong database

2016-08-04 Thread Robert Haas
On Tue, Aug 2, 2016 at 5:42 PM, David G. Johnston wrote: > Moving to -hackers since this is getting into details > > Bug Report # 14247 > > On Tue, Aug 2, 2016 at 4:58 PM, Tom Lane wrote: >> >> "David G. Johnston"

Re: [HACKERS] New version numbering practices

2016-08-04 Thread Tom Lane
Robert Haas writes: > On Thu, Aug 4, 2016 at 9:57 AM, Tom Lane wrote: >> [ shrug... ] What do you claim is not machine-readable about >> server_version? > Surely you can't have missed the connection between the issue at hand > and what Craig is

Re: [HACKERS] Bug in WaitForBackgroundWorkerShutdown() [REL9_5_STABLE]

2016-08-04 Thread Tom Lane
Yury Zhuravlev writes: > Dmitry Ivanov wrote: >>> Recently I've encountered a strange crash while calling elog(ERROR, "...") >>> after the WaitForBackgroundWorkerShutdown() function. It turns out that >>> _returns_ inside the PG_TRY()..PG_CATCH() block are *highly*

Re: [HACKERS] Keeping CURRENT_DATE and similar constructs in original format

2016-08-04 Thread Peter Eisentraut
On 5/12/16 6:14 PM, Tom Lane wrote: > So what I've wanted to do for some time is invent a new expression node > type that represents any one of these functions and can be reverse-listed > in the same format that the input had. The attached proposed patch does > that. I was experimenting with

Re: [HACKERS] New version numbering practices

2016-08-04 Thread Robert Haas
On Thu, Aug 4, 2016 at 9:57 AM, Tom Lane wrote: > Vladimir Sitnikov writes: >>> Sorry, but I don't buy that. I think sending both server_version and >>> server_version_num would be silly, and we're certainly not going to stop >>> sending

Re: [HACKERS] New version numbering practices

2016-08-04 Thread Peter Eisentraut
On 8/3/16 1:20 PM, Tom Lane wrote: > I was just wondering whether it would be worth starting to use "git tag -a". One should always use annotated tags for public releases. That way, the tag is its own Git object that cannot be later moved around or easily faked (modulo GPG signatures -- a whole

Re: [HACKERS] Pgbench performance tuning?

2016-08-04 Thread Greg Stark
On Thu, Aug 4, 2016 at 6:53 PM, Andres Freund wrote: > What's the config? Version? What activity does pidstat -d -l indicate? > How much WAL was generated? I know the specifics matter but I was also trying to avoid dumping too much into the email. The shared buffers is set

Re: [HACKERS] Heap WARM Tuples - Design Draft

2016-08-04 Thread Claudio Freire
On Thu, Aug 4, 2016 at 3:23 PM, Claudio Freire wrote: >> That is a new HOT chain given our current code, but >> would the new tuple addition realize it needs to create a new index >> tuple? The new tuple code needs to check the index's root HOT tid for a >> non-match. > >

Re: [HACKERS] handling unconvertible error messages

2016-08-04 Thread Tom Lane
Victor Wagner writes: > On Thu, 04 Aug 2016 09:42:10 -0400 > Tom Lane wrote: >> Yeah. I'm inclined to think that we should reset the message locale >> to C as soon as we've forked away from the postmaster, and leave it >> that way until we've absorbed

Re: [HACKERS] Heap WARM Tuples - Design Draft

2016-08-04 Thread Claudio Freire
On Thu, Aug 4, 2016 at 3:07 PM, Bruce Momjian wrote: > On Thu, Aug 4, 2016 at 02:35:32PM -0300, Claudio Freire wrote: >> > Vacuum will need to be taught not to break the invariant, but I >> > believe this is viable and efficient. >> >> >> And I didn't mention the invariant. >>

Re: [HACKERS] improved DefElem list processing

2016-08-04 Thread Tom Lane
I wrote: > Peter Eisentraut writes: >> The other adds a location field to the DefElem node. > +1 for sure, lots of places where that would be a good thing > (the duplicate check itself, for starters). Forgot to mention: seems like you should have added a

Re: [HACKERS] Pgbench performance tuning?

2016-08-04 Thread Andres Freund
On 2016-08-04 19:15:43 +0100, Greg Stark wrote: > On Thu, Aug 4, 2016 at 6:53 PM, Andres Freund wrote: > > What's the config? Version? What activity does pidstat -d -l indicate? > > How much WAL was generated? > > I know the specifics matter but I was also trying to avoid

Re: [HACKERS] Lossy Index Tuple Enhancement (LITE)

2016-08-04 Thread Simon Riggs
On 4 August 2016 at 18:27, Bruce Momjian wrote: >> > Also, why not use this bitmap for all indexes, not just update chains? >> >> I don't understand where you get this update chains thing from. >> >> The bitmap can apply to multiple tuples on one page, which is described. > > I

Re: [HACKERS] Double invocation of InitPostmasterChild in bgworker with -DEXEC_BACKEND

2016-08-04 Thread Robert Haas
On Tue, Aug 2, 2016 at 6:40 PM, Tom Lane wrote: > Thomas Munro writes: >> I discovered that if you build with -DEXEC_BACKEND on a Unix system >> and then try to start a background worker, it dies in >> InitializeLatchSupport: > >> TRAP:

Re: [HACKERS] improved DefElem list processing

2016-08-04 Thread Tom Lane
Peter Eisentraut writes: > Here are two WIP patches to improve the DefElem list processing that is > used by many utility commands. > One factors out the duplicate checks, which are currently taking up a > lot of space with duplicate code. I haven't applied

Re: [HACKERS] [RFC] Change the default of update_process_title to off

2016-08-04 Thread Robert Haas
On Thu, Aug 4, 2016 at 3:52 AM, Andres Freund wrote: > On 2016-08-04 16:48:11 +0900, Michael Paquier wrote: >> Here is a different proposal: documenting instead that disabling that >> parameter on Windows can improve performance, at the cost of losing >> information verbosity

Re: [HACKERS] Heap WARM Tuples - Design Draft

2016-08-04 Thread Bruce Momjian
On Thu, Aug 4, 2016 at 02:35:32PM -0300, Claudio Freire wrote: > > Vacuum will need to be taught not to break the invariant, but I > > believe this is viable and efficient. > > > And I didn't mention the invariant. > > The invariant would be that all WARM chains: > > * contain entire HOT

Re: [HACKERS] Heap WARM Tuples - Design Draft

2016-08-04 Thread Bruce Momjian
On Thu, Aug 4, 2016 at 01:05:53PM -0400, Bruce Momjian wrote: > > Approach 2 seems more reasonable and simple.  > > > > There are only 2 bits for lp_flags and all combinations are already used. > > But > > for LP_REDIRECT root line pointer, we could use the lp_len field to store > > this > >

Re: [HACKERS] Pgbench performance tuning?

2016-08-04 Thread Andres Freund
On 2016-08-04 18:48:31 +0100, Greg Stark wrote: > I'm trying to run pgbench on a moderately beefy machine (4-core 3.4GHz > with 32G of ram and md mirrored spinning rust drives) at scale 300 > with 32 clients with duration of 15min. I'm getting TPS numbers > between 60-150 which seems surprisingly

[HACKERS] Pgbench performance tuning?

2016-08-04 Thread Greg Stark
I'm trying to run pgbench on a moderately beefy machine (4-core 3.4GHz with 32G of ram and md mirrored spinning rust drives) at scale 300 with 32 clients with duration of 15min. I'm getting TPS numbers between 60-150 which seems surprisingly low to me and also to several people on IRC. Now

Re: [HACKERS] Heap WARM Tuples - Design Draft

2016-08-04 Thread Bruce Momjian
On Thu, Aug 4, 2016 at 11:04:49PM +0530, Pavan Deolasee wrote: > If the root tuple still exists, we store the WARM flag (or > HEAP_RECHECK_REQUIRED as used in the original post) in the tuple header > itself. > When the root tuple becomes dead and HOT prune decides to replace it with a >

[HACKERS] psql: Missing option to print current buffer to current output channel (i.e., \qprint)

2016-08-04 Thread David G. Johnston
"\echo" has "\qecho" - I'm basically looking for a "\qprint" "\write" doesn't apply The following doesn't work either... #bash (stock Ubuntu 14.04) psql --set=query_in_var='SELECT version();' > /dev/null <<'SQL' \o /tmp/psql-test.txt \set ECHO queries --still ends up on stdout, hence the

Re: [HACKERS] Heap WARM Tuples - Design Draft

2016-08-04 Thread Pavan Deolasee
On Thu, Aug 4, 2016 at 10:49 PM, Bruce Momjian wrote: > On Thu, Aug 4, 2016 at 06:16:02PM +0100, Simon Riggs wrote: > > On 4 August 2016 at 18:05, Bruce Momjian wrote: > > > > >> Approach 2 seems more reasonable and simple. > > >> > > >> There are only 2

Re: [HACKERS] Heap WARM Tuples - Design Draft

2016-08-04 Thread Claudio Freire
On Thu, Aug 4, 2016 at 2:14 PM, Claudio Freire wrote: > To fix this, what I was pondering, and I believe it would be viable, > is to teach heap_hot_search_buffer about the WARM invariant. It will > make correctness heavily depend on the invariant being true, which > means

Re: [HACKERS] Heap WARM Tuples - Design Draft

2016-08-04 Thread Pavan Deolasee
On Thu, Aug 4, 2016 at 10:41 PM, Simon Riggs wrote: > On 4 August 2016 at 17:31, Andres Freund wrote: > > Hi, > > > > On 2016-08-04 16:29:09 +0530, Pavan Deolasee wrote: > >> Indexes whose values do not change do not require new index pointers. > Only

Re: [HACKERS] Heap WARM Tuples - Design Draft

2016-08-04 Thread Bruce Momjian
On Thu, Aug 4, 2016 at 06:18:47PM +0100, Simon Riggs wrote: > > Have you considered what it'd take to allow index pointers to allow to > > point to "intermediate" root tuples? I.e. have some indexes point into > > the middle of an update-chain, whereas others still point to the > > beginning?

Re: [HACKERS] Lossy Index Tuple Enhancement (LITE)

2016-08-04 Thread Claudio Freire
On Thu, Aug 4, 2016 at 1:58 PM, Simon Riggs wrote: > On 3 August 2016 at 20:37, Claudio Freire wrote: >> On Wed, Aug 3, 2016 at 4:20 AM, Simon Riggs wrote: >>> == IndexScan == >>> >>> Note that the executor code for IndexScan

Re: [HACKERS] Lossy Index Tuple Enhancement (LITE)

2016-08-04 Thread Bruce Momjian
On Thu, Aug 4, 2016 at 06:03:34PM +0100, Simon Riggs wrote: > On 4 August 2016 at 02:13, Bruce Momjian wrote: > > > How do plan to clear the bitmask so it, over time, doesn't end up being > > all-set? > > I don't have a plan, though thoughts welcome. > > Similar situation

Re: [HACKERS] Heap WARM Tuples - Design Draft

2016-08-04 Thread Andres Freund
On 2016-08-04 18:18:47 +0100, Simon Riggs wrote: > On 4 August 2016 at 18:17, Andres Freund wrote: > > On 2016-08-04 18:11:17 +0100, Simon Riggs wrote: > > I'm less afraid of the fiddlyness of finding the root tuple, than the > > cost. It's not cheap to walk through,

Re: [HACKERS] Heap WARM Tuples - Design Draft

2016-08-04 Thread Bruce Momjian
On Thu, Aug 4, 2016 at 06:16:02PM +0100, Simon Riggs wrote: > On 4 August 2016 at 18:05, Bruce Momjian wrote: > > >> Approach 2 seems more reasonable and simple. > >> > >> There are only 2 bits for lp_flags and all combinations are already used. > >> But > >> for LP_REDIRECT

Re: [HACKERS] Heap WARM Tuples - Design Draft

2016-08-04 Thread Simon Riggs
On 4 August 2016 at 18:17, Andres Freund wrote: > On 2016-08-04 18:11:17 +0100, Simon Riggs wrote: >> On 4 August 2016 at 17:31, Andres Freund wrote: >> > Hi, >> > >> > On 2016-08-04 16:29:09 +0530, Pavan Deolasee wrote: >> >> Indexes whose values do not

Re: [HACKERS] Heap WARM Tuples - Design Draft

2016-08-04 Thread Andres Freund
On 2016-08-04 18:11:17 +0100, Simon Riggs wrote: > On 4 August 2016 at 17:31, Andres Freund wrote: > > Hi, > > > > On 2016-08-04 16:29:09 +0530, Pavan Deolasee wrote: > >> Indexes whose values do not change do not require new index pointers. Only > >> the index whose key is

Re: [HACKERS] Heap WARM Tuples - Design Draft

2016-08-04 Thread Bruce Momjian
On Thu, Aug 4, 2016 at 09:58:29AM -0700, Andres Freund wrote: > On 2016-08-04 12:55:25 -0400, Bruce Momjian wrote: > > On Thu, Aug 4, 2016 at 09:31:18AM -0700, Andres Freund wrote: > > > Hi, > > > > > > On 2016-08-04 16:29:09 +0530, Pavan Deolasee wrote: > > > > Indexes whose values do not

Re: [HACKERS] Heap WARM Tuples - Design Draft

2016-08-04 Thread Simon Riggs
On 4 August 2016 at 18:05, Bruce Momjian wrote: >> Approach 2 seems more reasonable and simple. >> >> There are only 2 bits for lp_flags and all combinations are already used. But >> for LP_REDIRECT root line pointer, we could use the lp_len field to store >> this >> special

Re: [HACKERS] Heap WARM Tuples - Design Draft

2016-08-04 Thread Claudio Freire
On Thu, Aug 4, 2016 at 7:59 AM, Pavan Deolasee wrote: > We don’t want to force a recheck for every index fetch because that will > slow everything down. So we would like a simple and efficient way of knowing > about the existence of a WARM tuple in a chain and do a

Re: [HACKERS] Heap WARM Tuples - Design Draft

2016-08-04 Thread Bruce Momjian
On Thu, Aug 4, 2016 at 09:58:29AM -0700, Andres Freund wrote: > On 2016-08-04 12:55:25 -0400, Bruce Momjian wrote: > > On Thu, Aug 4, 2016 at 09:31:18AM -0700, Andres Freund wrote: > > > Hi, > > > > > > On 2016-08-04 16:29:09 +0530, Pavan Deolasee wrote: > > > > Indexes whose values do not

Re: [HACKERS] Heap WARM Tuples - Design Draft

2016-08-04 Thread Simon Riggs
On 4 August 2016 at 17:31, Andres Freund wrote: > Hi, > > On 2016-08-04 16:29:09 +0530, Pavan Deolasee wrote: >> Indexes whose values do not change do not require new index pointers. Only >> the index whose key is being changed will need a new index entry. The new >> index

Re: [HACKERS] Heap WARM Tuples - Design Draft

2016-08-04 Thread Bruce Momjian
On Thu, Aug 4, 2016 at 04:29:09PM +0530, Pavan Deolasee wrote: > Write Amplification Reduction Method (WARM) > > > A few years back, we developed HOT to address the problem associated with MVCC > and frequent updates and it has served us very well. But in the

Re: [HACKERS] Lossy Index Tuple Enhancement (LITE)

2016-08-04 Thread Simon Riggs
On 4 August 2016 at 02:13, Bruce Momjian wrote: > How do plan to clear the bitmask so it, over time, doesn't end up being > all-set? I don't have a plan, though thoughts welcome. Similar situation that our current indexes don't recover from bloat, a situation made worse by

Re: [HACKERS] Heap WARM Tuples - Design Draft

2016-08-04 Thread Andres Freund
On 2016-08-04 12:55:25 -0400, Bruce Momjian wrote: > On Thu, Aug 4, 2016 at 09:31:18AM -0700, Andres Freund wrote: > > Hi, > > > > On 2016-08-04 16:29:09 +0530, Pavan Deolasee wrote: > > > Indexes whose values do not change do not require new index pointers. Only > > > the index whose key is

Re: [HACKERS] Lossy Index Tuple Enhancement (LITE)

2016-08-04 Thread Simon Riggs
On 3 August 2016 at 20:37, Claudio Freire wrote: > On Wed, Aug 3, 2016 at 4:20 AM, Simon Riggs wrote: >> == IndexScan == >> >> Note that the executor code for IndexScan appears identical between >> the two optimizations. The difference between

Re: [HACKERS] Heap WARM Tuples - Design Draft

2016-08-04 Thread Bruce Momjian
On Thu, Aug 4, 2016 at 09:31:18AM -0700, Andres Freund wrote: > Hi, > > On 2016-08-04 16:29:09 +0530, Pavan Deolasee wrote: > > Indexes whose values do not change do not require new index pointers. Only > > the index whose key is being changed will need a new index entry. The new > > index entry

Re: [HACKERS] Why we lost Uber as a user

2016-08-04 Thread Alfred Perlstein
On 8/4/16 2:00 AM, Torsten Zuehlsdorff wrote: On 03.08.2016 21:05, Robert Haas wrote: On Wed, Aug 3, 2016 at 2:23 PM, Tom Lane wrote: Robert Haas writes: I don't think they are saying that logical replication is more reliable than physical

Re: [HACKERS] Heap WARM Tuples - Design Draft

2016-08-04 Thread Andres Freund
Hi, On 2016-08-04 16:29:09 +0530, Pavan Deolasee wrote: > Indexes whose values do not change do not require new index pointers. Only > the index whose key is being changed will need a new index entry. The new > index entry will be set to the CTID of the root line pointer. That seems to require

Re: [HACKERS] Design for In-Core Logical Replication

2016-08-04 Thread Simon Riggs
On 29 July 2016 at 16:53, Robert Haas wrote: > I think that to really understand exactly what you and Petr have in > mind, we'd need a description of where publication and subscription > data is stored within the server, and exactly what gets stored. > Perhaps that will

[HACKERS] improved DefElem list processing

2016-08-04 Thread Peter Eisentraut
Here are two WIP patches to improve the DefElem list processing that is used by many utility commands. One factors out the duplicate checks, which are currently taking up a lot of space with duplicate code. I haven't applied this everywhere yet, but the patch shows how much boring code can be

Re: [HACKERS] regression test for extended query protocol

2016-08-04 Thread Alvaro Herrera
Michael Paquier wrote: > On Thu, Aug 4, 2016 at 12:14 AM, Dave Cramer wrote: > > We currently run tests every time a PR is created on github, but I don't > > think there are any animals running the JDBC test suite > > > > We can add tests, what exactly do we want to test. Then

Re: [HACKERS] handling unconvertible error messages

2016-08-04 Thread Victor Wagner
On Thu, 04 Aug 2016 09:42:10 -0400 Tom Lane wrote: > Victor Wagner writes: > > If error occurs during processing of StartMessage protocol message, > > i.e. client request connection to unexisting database, > > ErrorResponse would contain message in the

Re: [HACKERS] New version numbering practices

2016-08-04 Thread Vladimir Sitnikov
Tom Lane : > [ shrug... ] What do you claim is not machine-readable about > server_version? > 0) server_version needs a dance to parse. For instance, recent "Stamp version 10devel" patch did touch "server_version" parsing in fe-exec.c:

Re: [HACKERS] Hash Indexes

2016-08-04 Thread Mithun Cy
I did some basic testing of same. In that I found one issue with cursor. +BEGIN; +SET enable_seqscan = OFF; +SET enable_bitmapscan = OFF; +CREATE FUNCTION declares_cursor(int) + RETURNS void + AS 'DECLARE c CURSOR FOR SELECT * from con_hash_index_table WHERE keycol = $1;' +LANGUAGE SQL; +

Re: [HACKERS] Surprising behaviour of \set AUTOCOMMIT ON

2016-08-04 Thread Pavel Stehule
2016-08-04 15:37 GMT+02:00 Amit Kapila : > On Wed, Aug 3, 2016 at 5:09 PM, Pavel Stehule > wrote: > > > > > > 2016-08-03 12:16 GMT+02:00 Rahila Syed : > >> > >> Should changing the value from OFF to ON automatically either

Re: [HACKERS] max_parallel_degree > 0 for 9.6 beta

2016-08-04 Thread David G. Johnston
On Thu, Aug 4, 2016 at 9:25 AM, Robert Haas wrote: > On Thu, Aug 4, 2016 at 12:56 AM, Tom Lane wrote: > > Noah Misch writes: > >> On Wed, Apr 20, 2016 at 02:28:15PM -0400, Tom Lane wrote: > >>> +1, but let's put an entry on the 9.6

Re: [HACKERS] New version numbering practices

2016-08-04 Thread Tom Lane
Vladimir Sitnikov writes: >> Sorry, but I don't buy that. I think sending both server_version and >> server_version_num would be silly, and we're certainly not going to stop >> sending server_version. > What is wrong with sending machine-readable value? [ shrug...

[HACKERS] "Some tests to cover hash_index"

2016-08-04 Thread Mithun Cy
I am attaching the patch to improve some coverage of hash index code [1]. I have added some basic tests, which mainly covers overflow pages. It took 2 sec extra time in my machine in parallel schedule. Hit Total Coverage old tests Line Coverage 780 1478 52.7 Function Coverage 63 85 74.1

Re: [HACKERS] New version numbering practices

2016-08-04 Thread Vladimir Sitnikov
> > Sorry, but I don't buy that. I think sending both server_version and > server_version_num would be silly, and we're certainly not going to stop > sending server_version. > What is wrong with sending machine-readable value? Vladimir

Re: [HACKERS] [RFC] Change the default of update_process_title to off

2016-08-04 Thread Tom Lane
Andres Freund writes: > On 2016-08-04 16:48:11 +0900, Michael Paquier wrote: >> Here is a different proposal: documenting instead that disabling that >> parameter on Windows can improve performance, at the cost of losing >> information verbosity for processes. > The benefit

Re: [HACKERS] New version numbering practices

2016-08-04 Thread Tom Lane
Craig Ringer writes: > On 4 August 2016 at 12:45, Tom Lane wrote: >> Craig Ringer writes: >>> Well, this seems like a good time to make server_version_num GUC_REPORT >>> as well... >> To what end? Existing versions of libpq

Re: [HACKERS] handling unconvertible error messages

2016-08-04 Thread Tom Lane
Victor Wagner writes: > If error occurs during processing of StartMessage protocol message, > i.e. client request connection to unexisting database, > ErrorResponse would contain message in the server default locale, > despite of client encoding being specified in the

Re: [HACKERS] Surprising behaviour of \set AUTOCOMMIT ON

2016-08-04 Thread Amit Kapila
On Wed, Aug 3, 2016 at 5:09 PM, Pavel Stehule wrote: > > > 2016-08-03 12:16 GMT+02:00 Rahila Syed : >> >> Should changing the value from OFF to ON automatically either commit or >> rollback transaction in progress? >> >> >> FWIW, running set

Re: [HACKERS] max_parallel_degree > 0 for 9.6 beta

2016-08-04 Thread Robert Haas
On Thu, Aug 4, 2016 at 12:56 AM, Tom Lane wrote: > Noah Misch writes: >> On Wed, Apr 20, 2016 at 02:28:15PM -0400, Tom Lane wrote: >>> +1, but let's put an entry on the 9.6 open-items page to remind us to >>> make that decision at the right time. > >> It's

Re: [HACKERS] [Patch] Temporary tables that do not bloat pg_catalog (a.k.a fast temp tables)

2016-08-04 Thread Geoff Winkless
On 30 July 2016 at 13:42, Tomas Vondra wrote: > I'd argue that if you mess with catalogs directly, you're on your own. Interesting. What would you suggest people use instead? Geoff -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make

Re: [HACKERS] [Patch] Temporary tables that do not bloat pg_catalog (a.k.a fast temp tables)

2016-08-04 Thread Aleksander Alekseev
Thanks everyone for your remarks and comments! Here is an improved version of a patch. Main changes: * Patch passes `make installcheck` * Code is fully commented, also no more TODO's I wish I sent this version of a patch last time. Now I realize it was really hard to read and understand. Hope I

Re: [HACKERS] Bug in WaitForBackgroundWorkerShutdown() [REL9_5_STABLE]

2016-08-04 Thread Yury Zhuravlev
Dmitry Ivanov wrote: Recently I've encountered a strange crash while calling elog(ERROR, "...") after the WaitForBackgroundWorkerShutdown() function. It turns out that _returns_ inside the PG_TRY()..PG_CATCH() block are *highly* undesirable, since they leave PG_exception_stack pointing to a

Re: [HACKERS] Bug in WaitForBackgroundWorkerShutdown() [REL9_5_STABLE]

2016-08-04 Thread Dmitry Ivanov
> Recently I've encountered a strange crash while calling elog(ERROR, "...") > after the WaitForBackgroundWorkerShutdown() function. It turns out that > _returns_ inside the PG_TRY()..PG_CATCH() block are *highly* undesirable, > since they leave PG_exception_stack pointing to a local struct in a

[HACKERS] Bug in WaitForBackgroundWorkerShutdown() [REL9_5_STABLE]

2016-08-04 Thread Dmitry Ivanov
Recently I've encountered a strange crash while calling elog(ERROR, "...") after the WaitForBackgroundWorkerShutdown() function. It turns out that _returns_ inside the PG_TRY()..PG_CATCH() block are *highly* undesirable, since they leave PG_exception_stack pointing to a local struct in a dead

[HACKERS] Unused argument in PinBuffer function

2016-08-04 Thread Ildar Musin
Hi all, I was looking through the buffer manager's code and have noticed that function PinBuffer has an unused 'strategy' argument. It's seems that after refactoring made by Alexander Korotkov and Andres Freund (48354581a49c30f5757c203415aa8412d85b0f70) it became useless. The attached patch

[HACKERS] Heap WARM Tuples - Design Draft

2016-08-04 Thread Pavan Deolasee
Write Amplification Reduction Method (WARM) A few years back, we developed HOT to address the problem associated with MVCC and frequent updates and it has served us very well. But in the light of Uber's recent technical blog highlighting some of the problems

  1   2   >