[HACKERS] amcheck (B-Tree integrity checking tool)

2016-02-29 Thread Peter Geoghegan
ss time than having to REINDEX the index, and doesn't need too much memory. I think that in practice most problems that can be detected by the B-Tree checker functions will be detected with the lighter variant. -- Peter Geoghegan From 573810d8d3c994ce1a16ecffb2f5d208c0ff93e3 Mon Sep 17 00:00:00

Re: Commitfest Bug (was: [HACKERS] Re: Reusing abbreviated keys during second pass of ordered [set] aggregates)

2016-03-01 Thread Peter Geoghegan
been on both sides of this, as a patch author and patch reviewer. If the patch was left as "Waiting on Author", as my review of Alexander's patch was, then it ought to not change to "Needs Review" silently. That makes absolutely no sense. -- Peter Geoghegan -- Sent vi

Re: Commitfest Bug (was: [HACKERS] Re: Reusing abbreviated keys during second pass of ordered [set] aggregates)

2016-03-02 Thread Peter Geoghegan
On Wed, Mar 2, 2016 at 5:41 AM, Magnus Hagander wrote: > Ok, I've pushed a code that does that. Thank you. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Freeze avoidance of very large table.

2016-03-02 Thread Peter Geoghegan
do is restrict the functions to throw an error when called by non-superusers, out of caution. It's a problem that modules like pg_stat_statements and pg_freespacemap are currently lumped together in the documentation, but we all know that. -- Peter Geoghegan -- Sent via pgsql-hackers mailin

Re: [HACKERS] The plan for FDW-based sharding

2016-03-06 Thread Peter Geoghegan
o get agreement about something abstract, like the general idea of a distributed transaction manager. I fear that any particular detailed interpretation of what that phrase means will be very hard to get accepted into PostgreSQL. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] WIP: Upper planner pathification

2016-03-06 Thread Peter Geoghegan
vrkkasy1xbshgzxkd6-hnxx3gq7x-p-dz0zt...@mail.gmail.com In summary, I think it's surprising that a max_parallel_degree of 1 doesn't disable parallel workers entirely. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] ExecGather() + nworkers

2016-03-07 Thread Peter Geoghegan
w useful is it in the general case? I'm not suggesting that it isn't, but I'm not sure. How common is it for the leader process to do anything other than coordinate and consume from worker processes? -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgre

[HACKERS] Minor bug affecting ON CONFLICT lock wait log messages

2016-03-07 Thread Peter Geoghegan
. -- Peter Geoghegan From bc481af77994057cb1ffe4a0e471b38bb00dc228 Mon Sep 17 00:00:00 2001 From: Peter Geoghegan Date: Mon, 7 Mar 2016 13:16:24 -0800 Subject: [PATCH] Avoid incorrectly indicating exclusion constraint wait INSERT ... ON CONFLICT's precheck may have to wait on the outcome of an

Re: [HACKERS] Freeze avoidance of very large table.

2016-03-07 Thread Peter Geoghegan
ls apart for some reason, I'll update this. I > plan to push this RSN if nobody objects. Thanks for making the effort to make the tool generally available. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription:

Re: [HACKERS] Using quicksort for every external sort run

2016-03-07 Thread Peter Geoghegan
hin ~5% of an *internal* sort with the patch series applied, even though ~80% of time is spent copying and sorting SortTuples overall in the internal sort case (the internal case cannot overlap sorting and aggregate processing, since it has no final merge step). This is a nice demonstration of ho

Re: [HACKERS] Using quicksort for every external sort run

2016-03-10 Thread Peter Geoghegan
update coming. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Using quicksort for every external sort run

2016-03-10 Thread Peter Geoghegan
sn't really a concern anymore. I think we may be able to comprehensively address Robert's concerns about regressions with very little work_mem and lots of data by fixing a problem with polyphase merge. More to come soon. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-

Re: [HACKERS] WIP: Upper planner pathification

2016-03-10 Thread Peter Geoghegan
y-paste coding. > > +1. Frustrating Citus's attempt to open-source their stuff is not in > the project's interest. I agree. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Fix for OpenSSL error queue bug

2016-03-10 Thread Peter Geoghegan
urned with Feedback in the CF and move on, which seems a bit of a > shame since this appears to be a bona fide bug, but if nobody's > willing to work on it, it ain't gettin' fixed. Getting to it very soon. Just really busy right this moment. -- Peter Geoghegan -- Sent vi

Re: [HACKERS] Fix for OpenSSL error queue bug

2016-03-10 Thread Peter Geoghegan
On Thu, Mar 10, 2016 at 3:09 PM, Peter Geoghegan wrote: > Getting to it very soon. Just really busy right this moment. That said, I agree with Peter's remarks about doing this frontend and backend. So, while I'm not sure, I think we're in agreement on all issues. I would ha

Re: [HACKERS] Fix for OpenSSL error queue bug

2016-03-10 Thread Peter Geoghegan
for the benefit of others. Is this informed by a concern about some specific downside to taking that extra precaution? Thanks -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Using quicksort for every external sort run

2016-03-10 Thread Peter Geoghegan
On Sun, Feb 14, 2016 at 8:01 PM, Peter Geoghegan wrote: > The query I'm testing is: "reindex index pgbench_accounts_pkey;" > > Now, with a maintenance_work_mem of 5MB, the most recent revision of > my patch takes about 54.2 seconds to complete this, as compared to

Re: [HACKERS] Fix for OpenSSL error queue bug

2016-03-10 Thread Peter Geoghegan
ue *after* an error happens, in SSLerrmessage(). """ The problem with this, as Heikki goes on to say, it that we might not get to that point in SSLerrmessage(); we may not be able to pop the queue/call ERR_get_error(), more or less by accident (e.g. I noticed an OOM could do

Re: [HACKERS] checkpointer continuous flushing - V18

2016-03-10 Thread Peter Geoghegan
e documentation. I don't know what can be done about that. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-03-11 Thread Peter Geoghegan
return compare; There are many options when you want to produce a corrupt B-Tree index! -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-03-11 Thread Peter Geoghegan
On Fri, Mar 11, 2016 at 1:09 PM, Peter Geoghegan wrote: > Or, you could add code like this to comparetup_index_btree(), to > simulate a broken opclass: > > diff --git a/src/backend/utils/sort/tuplesort.c > b/src/backend/utils/sort/tuplesort.c > index 67d86ed..23712ff 100644 &

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-03-11 Thread Peter Geoghegan
th the left-right keyspace checks, which don't have relation ExclusiveLock protection (IOW, I think that that might be buggy). I need to think about that some more, but current thinking is that it would hardly matter if we used the highkey from right page rather than the first data item,

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-03-11 Thread Peter Geoghegan
On Fri, Mar 11, 2016 at 1:31 PM, Peter Geoghegan wrote: > You could have a race, where > there was a concurrent page deletion of the left sibling of the child > page, then a concurrent insertion into the newly expanded keyspace of > the parent. Therefore, the downlink in the parent (

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-03-11 Thread Peter Geoghegan
utes/columns past the first distinguishing/distinct attribute on each item on internal pages). That's a case that amcheck then needs to care about, just like it currently cares about the existing concept of minus infinity items. That's how it goes for amcheck. -- Peter Geoghegan

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-03-11 Thread Peter Geoghegan
ndex relation. Not sure about the cost delay thing. Delays are disabled by default for manually issued VACUUM, so have doubts that that's useful. If you want the tool to limp on when it finds an error, that can be done by changing the constant for the CORRUPTION macro in amcheck.c. But hav

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-03-11 Thread Peter Geoghegan
On Fri, Mar 11, 2016 at 4:17 PM, Peter Geoghegan wrote: > If you want the tool to limp on when it finds an error, that can be > done by changing the constant for the CORRUPTION macro in amcheck.c. > But having that be dynamically configurable is not really compatible > with the go

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-03-11 Thread Peter Geoghegan
I'll add that if people like the interface you propose. (Overloading the VACUUM cost delay functions to cause a delay for amcheck functions, too). Note that the functions already use an appropriate buffer access strategy (it avoids blowing shared_buffers, much like VACUUM itself). -- Peter

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-03-12 Thread Peter Geoghegan
diagnosing a problematic opclass, you have to be an expert, and tweaking amcheck for your requirements (i.e. rebuilding from source) becomes reasonable. Part of the reason that the code is so heavily commented is to make it hackable, because I do not feel optimistic that I can get an expert-orientat

Re: [HACKERS] Refactoring speculative insertion with unique indexes a little

2016-03-12 Thread Peter Geoghegan
tch is basically uncontroversial, and is built to make the AM interface clearer, and the design of speculative insertion easier to understand. It's clear we should have it. I'll get around to revising it before too long. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hacker

Re: [HACKERS] Refactoring speculative insertion with unique indexes a little

2016-03-12 Thread Peter Geoghegan
gorically stated was that he disliked one narrow aspect of the style of one thing in one function. I've already said I'm happy to do that. As things stand, the documentation for amcanunique methods, and the way they are described internally, is fairly misleading. -- Peter Geoghega

Re: [HACKERS] Refactoring speculative insertion with unique indexes a little

2016-03-12 Thread Peter Geoghegan
On Sat, Mar 12, 2016 at 2:53 PM, Peter Geoghegan wrote: > I said "basically uncontroversial", not "uncontroversial". That is a > perfectly accurate characterization of the patch, and if you disagree > than I suggest you re-read the thread. In particular, note that Al

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-03-13 Thread Peter Geoghegan
open source a > version of it later this week that way you can use it for testing. That could be interesting. The earlier prototypes of this tool are known to have detected glibc collation incompatibilities in real production systems. -- Peter Geoghegan -- Sent via pgsql-hackers mailing li

Re: [HACKERS] checkpointer continuous flushing - V18

2016-03-13 Thread Peter Geoghegan
or performance tips will probably only work well if there are still high standards that are actively enforced. There still needs to be tight editorial control. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http:/

Re: [HACKERS] Fix for OpenSSL error queue bug

2016-03-14 Thread Peter Geoghegan
On Thu, Mar 10, 2016 at 8:16 PM, Peter Geoghegan wrote: > On Thu, Mar 10, 2016 at 7:22 PM, Peter Eisentraut wrote: >> Arguably, if everyone followed "my" approach, this should be very easy >> to fix everywhere. > > I don't think that there is any clear indica

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-03-14 Thread Peter Geoghegan
pecially compelling feature. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Fix for OpenSSL error queue bug

2016-03-14 Thread Peter Geoghegan
backend-side code as well. I agree that the backend-side code should be covered. I quickly changed my mind about that, and am happy to produce a revision along those lines. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Fix for OpenSSL error queue bug

2016-03-14 Thread Peter Geoghegan
hat will calling ERR_get_error() more will add useless cycles. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Refactoring speculative insertion with unique indexes a little

2016-03-14 Thread Peter Geoghegan
x 9.6 patches to review, most of which are in need of support. I'm very busy. I said that I'd get to this patch soon. I might be kicking the can down the road a little with this patch; if so, I'm sorry. I suggest we leave it at that, until the CF is almost over or until I produce

Re: [HACKERS] Refactoring speculative insertion with unique indexes a little

2016-03-14 Thread Peter Geoghegan
On Mon, Mar 14, 2016 at 5:21 PM, Andres Freund wrote: > So? You're not the only one. I don't see why we shouldn't move this to > 'returned with feedback' until there's a new version. I don't see any point in that; I intend to get a revision in to the o

Re: [HACKERS] Fix for OpenSSL error queue bug

2016-03-14 Thread Peter Geoghegan
On Mon, Mar 14, 2016 at 4:11 PM, Peter Geoghegan wrote: > Yes, with one small difference: I wouldn't be calling ERR_get_error() > in the common case where SSL_get_error() returns SSL_ERROR_NONE, on > the theory that skipping that case represents no risk. I'm making a > c

Re: [HACKERS] Minor bug affecting ON CONFLICT lock wait log messages

2016-03-14 Thread Peter Geoghegan
On Mon, Mar 7, 2016 at 1:46 PM, Peter Geoghegan wrote: > Attached patch fixes a bug reported privately by Stephen this morning. Bump. I would like to see this in the next point release. It shouldn't be hard to review. Thanks -- Peter Geoghegan -- Sent via pgsql-hackers mailing lis

Re: [HACKERS][PROPOSAL] Covering + unique indexes.

2016-03-14 Thread Peter Geoghegan
"/truncated attributes are involved. This seems necessary. No? It's necessary because you aren't storing any attributes, so it's not acceptable to even attempt a comparison -- I think that will segfault (doesn't matter that the index scan wouldn't have returne

Re: [HACKERS][PROPOSAL] Covering + unique indexes.

2016-03-14 Thread Peter Geoghegan
On Mon, Mar 14, 2016 at 8:43 PM, Peter Geoghegan wrote: > > Does this work with amcheck? Maybe it works with bt_index_check(), but > not bt_index_parent_check()? I think that you need to make sure that > _bt_compare() knows about this, too. That's because it isn't good >

Re: [HACKERS][PROPOSAL] Covering + unique indexes.

2016-03-14 Thread Peter Geoghegan
On Mon, Mar 14, 2016 at 8:43 PM, Peter Geoghegan wrote: > * I think the comparison logic may have a bug. > > Does this work with amcheck? Maybe it works with bt_index_check(), but > not bt_index_parent_check()? I think that you need to make sure that > _bt_compare() knows about th

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-03-14 Thread Peter Geoghegan
ually). I haven't actually tested this theory of mine myself just yet, but let me know what you think of it on the thread for your patch. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] jsonb array-style subscription

2016-03-14 Thread Peter Geoghegan
On Thu, Mar 3, 2016 at 2:31 AM, Dmitry Dolgov <9erthali...@gmail.com> wrote: > Well, actually, I agree with that. I can try to rework the patch to achieve > this goal. Good idea. I wonder, having taken a quick look at the patch, how this works?: +select * from test_jsonb_subscript where test_jso

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-03-15 Thread Peter Geoghegan
nternal pages are checked against actual child pages (every item in the child page, in fact). It's the parent/child relationship that is verified in addition to the standard checks of every page on and across (not between) every level. -- Peter Geoghegan -- Sent via pgsql-hackers mailin

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-03-15 Thread Peter Geoghegan
s a non-native speaker of English language, who > ends up looking up quite a few words regularly) - > > Could we use "conform" or "correspond" instead of "comport" in the > following error message: > > "left link/right link pair in index \"%s\&q

Re: [HACKERS] [NOVICE] WHERE clause not used when index is used

2016-03-15 Thread Peter Geoghegan
. This test suite would indirectly test external sorting, B-Tree page deletion, edge-cases with very large IndexTuples, etc. Ideas for good areas of the B-Tree code to add tests for are welcome. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make chan

Re: [HACKERS] Minor bug affecting ON CONFLICT lock wait log messages

2016-03-15 Thread Peter Geoghegan
On Tue, Mar 15, 2016 at 6:18 AM, Stephen Frost wrote: > Agreed. I'm going to play with it a bit more but barring objections, > I'll commit and back-patch Peter's patch. Thanks for taking care of this, Stephen. -- Peter Geoghegan -- Sent via pgsql-hackers mailin

Re: [HACKERS][PROPOSAL] Covering + unique indexes.

2016-03-15 Thread Peter Geoghegan
On Mon, Mar 14, 2016 at 8:43 PM, Peter Geoghegan wrote: > Does this work with amcheck? Maybe it works with bt_index_check(), but > not bt_index_parent_check()? I think that you need to make sure that > _bt_compare() knows about this, too. That's because it isn't good > eno

Re: [HACKERS] Minor bug affecting ON CONFLICT lock wait log messages

2016-03-16 Thread Peter Geoghegan
came from the fact that historically, when check_exclusion_or_unique_constraint() was called check_exclusion_constraint(), it (almost) was our own tuple that was waited on. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [WIP] speeding up GIN build with parallel workers

2016-03-16 Thread Peter Geoghegan
his is why I wanted an easy way of atomically guaranteeing some number of workers (typically 2), or not using parallelism at all. I think the parallel worker API should offer a simple way to do that in cases like this, where having only 1 worker is never going to win. -- Peter Geoghegan -- Sent via p

Re: [HACKERS] Using quicksort for every external sort run

2016-03-18 Thread Peter Geoghegan
what wasn't so obvious to you when you review my patches. It's probably impossible to stare at something like tuplesort.c for as long as I have and get that balance just right. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to

Re: [HACKERS] WIP: Covering + unique indexes.

2016-03-18 Thread Peter Geoghegan
33/ Marked "Waiting on Author". -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Make primnodes.h gender neutral

2016-03-18 Thread Peter Geoghegan
losing anything important seems like a relatively straightforward task. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Using quicksort for every external sort run

2016-03-19 Thread Peter Geoghegan
On Wed, Mar 16, 2016 at 6:42 PM, Peter Geoghegan wrote: >> - I think that batchmemtuples() is somewhat weird. Normally, >> grow_memtuples() doubles the size of the array each time it's called. >> So if you somehow called this function when you still had lots of >> me

Re: [HACKERS] Using quicksort for every external sort run

2016-03-19 Thread Peter Geoghegan
quot;Can SortTuple.tuple ever be set?", rather than allowing it to remain a specific property of a datum tuplesort. state->datumTypByVal often isn't initialized in master, and so cannot be checked as things stand (unless the code is in a datum-case-specific routine). This new flag controls batch memory in slightly higher-level way than would otherwise be possible. It also controls the memory prefetching added by patch/commit 0003-*, FWIW. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [COMMITTERS] pgsql: Improve memory management for external sorts.

2016-03-19 Thread Peter Geoghegan
mat specifiers. However, I had a bad feeling about integer overflow of state->currentRun, and think I'll need to address that. After all, if runs are now no longer 2x work_mem on average, it's not completely ridiculous to imagine that being an issue on a misconfigured system. -- Pet

Re: [HACKERS] [COMMITTERS] pgsql: Improve memory management for external sorts.

2016-03-19 Thread Peter Geoghegan
On Fri, Mar 18, 2016 at 11:43 AM, Andres Freund wrote: > Yes, that removes the warning, and looks correct. Thanks. We should be careful to not repeat this mistake when the quicksort patch goes in. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)

Re: [HACKERS] fd.c doesn't remove files on a crash-restart

2016-03-19 Thread Peter Geoghegan
a bad idea FWIW, I've seen this get out of hand several times myself. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Make primnodes.h gender neutral

2016-03-19 Thread Peter Geoghegan
ndered pronouns has negative value, though, and it really isn't that hard to fix. So we should just fix it. (In case it matters, I'm in favor of this proposal on its merits). -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To m

Re: [HACKERS] Refactoring speculative insertion with unique indexes a little

2016-03-19 Thread Peter Geoghegan
asy to miss some of the subtleties. I do not pursue something like this without good reason. I'm optimistic that the patch will be accepted if it is carefully considered. -- Peter Geoghegan From 2b2a4c40a5e60ac1f28a75f11204ce88eb48cc73 Mon Sep 17 00:00:00 2001 From: Peter Geoghegan Date: Tue, 2

Re: [HACKERS] snapshot too old, configured by time

2016-03-20 Thread Peter Geoghegan
you consider making amcheck [1] a part of your testing strategy. I think that this patch is a good idea, and I'd be happy to take feedback from you on how to make amcheck more effective for testing this patch in particular. [1] https://commitfest.postgresql.org/9/561/ -- Peter Geoghegan

Re: [HACKERS] snapshot too old, configured by time

2016-03-20 Thread Peter Geoghegan
On Sun, Mar 20, 2016 at 4:25 PM, Peter Geoghegan wrote: > I worry that something weird could happen there. For example, perhaps > the page LSN on what is actually a newly recycled page could be set > such that the backend following a stale right spuriously raises a > "snapshot t

Re: [HACKERS] Using quicksort for every external sort run

2016-03-20 Thread Peter Geoghegan
ocations occurs after batch memory is used -- we need to be careful about how that could impact the "bucketing by size class"). Thanks -- Peter Geoghegan From b921e285ed3f22c9cab9c78c7c610fbdfee5839b Mon Sep 17 00:00:00 2001 From: Peter Geoghegan Date: Thu, 10 Mar 2016 14:52:18 -08

Re: [HACKERS] Minor bug affecting ON CONFLICT lock wait log messages

2016-03-21 Thread Peter Geoghegan
inions, if any are to be had. Sorry that I changed my mind, but it's a subtle issue, I'm sure you'll agree. I'm not going to push on this, but I want to be sure that we're happy with this. To reiterate, I think it boils down to: Is it okay that this new XLTW_InsertIndex case reports someone else's TID, while the only other XLTW_InsertIndex case has always reported our own TID? Discussing these sorts of "ontological" questions reminds me just how painful UPSERT was as a project. :-) -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-03-22 Thread Peter Geoghegan
could be a very low probability race condition when checking the key space across sibling pages, and will work to address that. If I'm right about that then it's not a lot of work to fix; I'm probably just going to use the right page's high key rather than its first data item. --

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-03-22 Thread Peter Geoghegan
On Tue, Mar 22, 2016 at 10:57 AM, Peter Geoghegan wrote: > That's right - I have a small number of feedback items to work > through. I also determined myself that there could be a very low > probability race condition when checking the key space across sibling > pages, and wil

Re: [HACKERS] Using quicksort for every external sort run

2016-03-22 Thread Peter Geoghegan
t think I see any here, nor any high memory tests where RS flops. Now, I think you focused on regressions because that was what was interesting, which is good. I just want to put that in context. Thanks -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] PoC: Partial sort

2016-03-23 Thread Peter Geoghegan
ions (I had no feedback on "partial-sort-basic-6.patch", which didn't use the new upper planner pathification stuff, unlike this latest version). > On Wed, Nov 4, 2015 at 4:44 AM, Peter Geoghegan wrote: >> >> Explain output >> --- >> I th

Re: [HACKERS] Using quicksort for every external sort run

2016-03-23 Thread Peter Geoghegan
te I/O bound with the patch sometimes, where it's usually CPU/memory bound with the heap, especially with lots of work_mem. More importantly, it would be more informative if the temp_tablespace was not affected by I/O from Postgres' heap. I also like seeing a sample of "trace_sort

Re: [HACKERS] Fix for OpenSSL error queue bug

2016-03-23 Thread Peter Geoghegan
On Mon, Mar 14, 2016 at 6:44 PM, Peter Geoghegan wrote: > I can produce a back-patchable variant of this if you and Peter E. > think this approach is okay. Where are we on this? -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes t

Re: [HACKERS] Using quicksort for every external sort run

2016-03-23 Thread Peter Geoghegan
me the slowdown is related to the batching (as it's only happening for > low work_mem values), so perhaps there's an internal heuristics that we > could tune? Can you show trace_sort output for these cases? Both master, and patched? Thanks -- Peter Geoghegan -- Sent via pgsql-h

Re: [HACKERS] Using quicksort for every external sort run

2016-03-23 Thread Peter Geoghegan
iation did, how memory was used, and even if the sort was I/O bound at various stages (it dumps some getrusage stats to the log, too). You can usually tell exactly what happened for external sorts, which is very interesting for those one or two cases that you found to be noticeably worse off with the patch. Thanks for testing! -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [WIP] Effective storage of duplicates in B-tree index.

2016-03-24 Thread Peter Geoghegan
ing with amcheck. I need help with formalizing a "testing first" approach for improving the B-Tree code, because I think it's the only way that we can move forward with projects like this. It's *incredibly* hard to push forward patches like this given our current, limited t

Re: [HACKERS] Using quicksort for every external sort run

2016-03-24 Thread Peter Geoghegan
On Sun, Mar 20, 2016 at 11:01 PM, Peter Geoghegan wrote: > Allowing 0 tuple runs in rare cases seems like the simplest solution. > After all, mergeprereadone() is expressly prepared for 0 tuple runs. > It says "ensure that we have at least one tuple, if any are to be > had".

Re: [HACKERS] Fix for OpenSSL error queue bug

2016-03-26 Thread Peter Geoghegan
ne with that. Will this make it into the next point release? I was rather hoping it would. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Draft release notes for next week's releases

2016-03-27 Thread Peter Geoghegan
take at least that long before affected users even start to think about this. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Draft release notes for next week's releases

2016-03-28 Thread Peter Geoghegan
ICU might also do better in your test case for that reason. [1] https://www.sqlite.org/src/artifact?ci=trunk&filename=ext/icu/README.txt -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Draft release notes for next week's releases

2016-03-28 Thread Peter Geoghegan
all using UTF-16 only. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Dealing with collation and strcoll/strxfrm/etc

2016-03-28 Thread Peter Geoghegan
basis for case-insensitive collations, which would let us kill citext, a module that I consider to be a total kludge. And, we might also be able to lock down WAL compatibility, which would be generally useful. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Dealing with collation and strcoll/strxfrm/etc

2016-03-28 Thread Peter Geoghegan
the CLDR [2]. I don't think we'd want to have Postgres ship "supported collations" in each major version, in roughly the style of the IANA timezone stuff, but it's far too early to rule that out. It would have upsides. [1] https://en.wikipedia.org/wiki/Unicode_collation_algorithm [2] http://cldr.unicode.org/ -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-03-28 Thread Peter Geoghegan
On Tue, Mar 22, 2016 at 10:57 AM, Peter Geoghegan wrote: > That's right - I have a small number of feedback items to work > through. I also determined myself that there could be a very low > probability race condition when checking the key space across sibling > pages, and wil

Re: [HACKERS] Using quicksort for every external sort run

2016-03-28 Thread Peter Geoghegan
On Thu, Mar 10, 2016 at 6:54 PM, Peter Geoghegan wrote: > I've used amcheck [2] to test this latest revision -- the tool ought > to not see any problems with any index created with the patch applied. > Reviewers might find it helpful to use amcheck, too. As 9.6 is > stabilized,

Re: [HACKERS] Using quicksort for every external sort run

2016-03-29 Thread Peter Geoghegan
x27;t done any analysis of how much better that made the problem *across the board* yet, but for int4 cases I could make 1MB work_mem queries faster with gigabytes of data on my laptop. I believe I tested various datum sort cases there, like "select count(distinct(foo)) from bar";

Re: [HACKERS] Using quicksort for every external sort run

2016-03-29 Thread Peter Geoghegan
On Tue, Mar 29, 2016 at 12:43 PM, Peter Geoghegan wrote: > A complete do-over from Tomas would be best, here. He has already > acknowledged that the i5 CREATE INDEX results were completely invalid. The following analysis is all based on Xeon numbers, which as I've said we should focus

Re: [HACKERS] Please correct/improve wiki page about abbreviated keys bug

2016-03-29 Thread Peter Geoghegan
ome things. I noticed it said something about incorrect unique violations on affected systems. Is that really possible? I can't imagine how it could be, although that is left in place largely unchanged for now. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postg

Re: [HACKERS] Please correct/improve wiki page about abbreviated keys bug

2016-03-29 Thread Peter Geoghegan
Since that's the only attribute that could use abbreviated keys, it ought to be safe to not require users to REINDEX indexes that happen to have a second-or-subsequent text/varchar(n) attribute that doesn't use the C locale. Maybe it's not worth worrying about. -- Peter Geoghegan --

Re: [HACKERS] Using quicksort for every external sort run

2016-03-29 Thread Peter Geoghegan
o something or not do something based on this information is subjective. OTOH, whether and to what extent these tests are representative of real user workloads seems much less subjective. This is not a criticism of the test cases you came up with, which rightly emphasized possibly regressed cases. I

Re: [HACKERS] Using quicksort for every external sort run

2016-03-29 Thread Peter Geoghegan
practical setup that anyone would want to use? [1] http://www.postgresql.org/message-id/CAM-w4HOwt0C7ZndowHUuraw+xi+BhY5a6J008XoSq=r9z7h...@mail.gmail.com [2] http://www.postgresql.org/message-id/CAM-w4HM4XW3u5kVEuUrr+L+KX3WZ=5JKk0A=djjzypkb-hy...@mail.gmail.com -- Peter Geoghegan -- Sent v

Re: [HACKERS] Using quicksort for every external sort run

2016-03-30 Thread Peter Geoghegan
ion of sorts, which could effectively decrease memory use on the system. Increasing the memory available to sorts could decrease the overall use of memory. Being really frugal with memory is expensive, maybe even if your primary concern is the expense of memory usage, which it probably i

Re: [HACKERS] snapshot too old, configured by time

2016-03-30 Thread Peter Geoghegan
me hope of catching this kind of race condition (the stale downlink that you followed to get to the spuriously-recycled-early page doesn't stay stale for long). Or, maybe it would happen to catch some other random problem. Difficult to say. Again, this is based on a speculation that might

Re: [HACKERS] snapshot too old, configured by time

2016-03-30 Thread Peter Geoghegan
On Wed, Mar 30, 2016 at 12:21 PM, Peter Geoghegan wrote: > Well, amcheck is a tool that in essence makes sure that B-Trees look > structurally sound, and respect invariants like having every item on > each page in logical order. That could catch a bug of the kind I just > described,

Re: [HACKERS] Please correct/improve wiki page about abbreviated keys bug

2016-03-30 Thread Peter Geoghegan
Okay. I'll look at it later today. -- Peter Geoghegan

Re: [HACKERS] pgindent-polluted commits

2016-01-13 Thread Peter Geoghegan
s approach seems to work nicely, without any downside that I've noticed. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Proposal: speeding up GIN build with parallel workers

2016-01-15 Thread Peter Geoghegan
aving a clear path forward to performing the build in parallel. I understand that a long term ambition for the gin am is to merge it with nbtree, to almost automatically benefit from enhancements, and to reduce the maintenance burden of each. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list

Re: [HACKERS] Proposal: speeding up GIN build with parallel workers

2016-01-17 Thread Peter Geoghegan
and nbtree at some point, though. I think that there are some useful general consequences for performance that come from consolidation. For example, my ongoing work on external sorting makes it use much of the same infrastructure as internal sorting. Now, external sorts automatically benefit from optimi

Re: [HACKERS] Refactoring speculative insertion with unique indexes a little

2016-01-18 Thread Peter Geoghegan
other committer wants to engage with me on this, then I will of course work with them. But that will not be predicated on my first revising the patch in a way that this other committer does not understand. That would be profoundly unfair. -- Peter Geoghegan -- Sent via pgsql-hackers mailin

Re: [HACKERS] Refactoring speculative insertion with unique indexes a little

2016-01-18 Thread Peter Geoghegan
; I don't -- I just think you could have sent a patch that addressed all > the other points, leave this one as initially submitted, and note that > the new submission left it unaddressed because you disagreed. I'll try to do that soon. I've got a very busy schedule over the

  1   2   3   4   5   6   7   8   9   10   >