Re: [HACKERS] Parallel tuplesort (for parallel B-Tree index creation)

2016-11-09 Thread Peter Geoghegan
in too far or we'll create slow polyphase > merges in case that are reasonably likely to occur in real life. I completely agree with your analysis. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Pinning a buffer in TupleTableSlot is unnecessary

2016-11-14 Thread Peter Geoghegan
On Mon, Nov 14, 2016 at 9:22 AM, Heikki Linnakangas wrote: > I think that difference in the API is exactly what caught Peter by surprise > and led to bug #14344. And I didn't see it either, until you two debugged > it. That is accurate, of course. -- Peter Geoghegan --

Re: [HACKERS] Pinning a buffer in TupleTableSlot is unnecessary

2016-11-14 Thread Peter Geoghegan
that there is reason to tie it to adding the PinBuffer() stuff, which we've been talking about for years now. It just caught my eye. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-11-17 Thread Peter Geoghegan
htmost child". This is covered in the nbtree README: """ To preserve consistency on the parent level, we cannot merge the key space of a page into its right sibling unless the right sibling is a child of the same parent --- otherwise, the parent's key space assignment changes too, meaning we'd have to make bounding-key updates in its parent, and perhaps all the way up the tree. Since we can't possibly do that atomically, we forbid this case. That means that the rightmost child of a parent node can't be deleted unless it's the only remaining child, in which case we will delete the parent too (see below). ''"" > I like this. Some of the more complex pieces towards the end of the > field need some attention, there's a fair amount of word-smithing > needed, and I do think we want to make the structural changes outlined > above, but besides these, imo fairly simple adaptions, I do think this > is useful and not that far from being committable. Cool. I'll try to get a revision out soon. I'm happy to do that much. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-11-17 Thread Peter Geoghegan
index scan for 'b' (relevant to T2). I would actually like to change things to make the invariant the classic L&Y "Ki < v <= Ki+1", to avoid bloat in the internal pages and to make suffix truncation in internal pages work. So, we don't have the cousin problem, but si

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-11-18 Thread Peter Geoghegan
On Thu, Nov 17, 2016 at 12:04 PM, Peter Geoghegan wrote: >> Hm, if we want that - and it doesn't seem like a bad idea - I think we >> should be make it available without recompiling. > > I suppose, provided it doesn't let CORRUPTION elevel be < ERROR. That >

Re: [HACKERS] [WIP] [B-Tree] Keep indexes sorted by heap physical location

2016-11-18 Thread Peter Geoghegan
On Thu, Aug 18, 2016 at 2:15 PM, Peter Geoghegan wrote: > I think that this is a bad idea. We need to implement suffix > truncation of internal page index tuples at some point, to make them > contain less information from the original leaf page index tuple. > That's an impor

Re: [HACKERS] [WIP] [B-Tree] Keep indexes sorted by heap physical location

2016-11-18 Thread Peter Geoghegan
bits can be unset. We might balance the avoidance of I/O during the scan against how much we might expect to save in a subsequent bitmap heap scan, and so on. This might be based on a selectivity estimate. That's all fairly hand-wavy, certainly, but I see significant potential along th

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-11-19 Thread Peter Geoghegan
is != ERROR (the thing I mention about elevel < ERROR is already documented in code comments). If that breaks, they get to keep both halves. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [WIP] [B-Tree] Keep indexes sorted by heap physical location

2016-11-21 Thread Peter Geoghegan
r TID in all cases, since the current assumption is that a high key's TID is just filler -- maybe we can lose that at some point. You should use amcheck to specifically verify that that happens reliably in all cases. Presumably, its use of an insertion scankey would automatically see the use of TID as a tie-breaker with patched Postgres amcheck verification, and so amcheck will work for this purpose unmodified. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Partitioning vs ON CONFLICT

2017-02-16 Thread Peter Geoghegan
ON CONFLICT DO NOTHING permits, then it should be possible for it to just work today -- infer_arbiter_indexes() will return immediately. This should be just like the old approach involving inheritance, in that that should be possible. No? -- Peter Geoghegan -- Sent via pgsql-hackers mailing l

Re: [HACKERS] Partitioning vs ON CONFLICT

2017-02-16 Thread Peter Geoghegan
cing this directly. IIRC, that's what happens with inheritance-based partitioning. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Index corruption with CREATE INDEX CONCURRENTLY

2017-02-17 Thread Peter Geoghegan
would need to visit the heap, which tends to be much larger than any one index, or even all indexes. That would probably need to be random I/O, too. It might be possible to mostly not visit the heap, though -- I'm not sure offhand. I'd have to study the problem in detail, which I have no t

Re: [HACKERS] Index corruption with CREATE INDEX CONCURRENTLY

2017-02-17 Thread Peter Geoghegan
s that it will have checks for a large variety of invariants that involve the heap, and related SLRU structures such as MultiXacts. Though, that would probably necessitate code written by other people that are subject matter experts in areas that I am not. -- Peter Geoghegan -- Sent via pgsql-h

Re: [HACKERS] GUC for cleanup indexes threshold.

2017-02-20 Thread Peter Geoghegan
entGlobalXmin respected by VACUUM, that prevents this sort of recycling. I suspect that the restrictions on page deletion as opposed to page recycling is vastly more likely to cause pain to users, and that's not made any worse by this. -- Peter Geoghegan -- Sent via pgsql-hackers maili

Re: [HACKERS] ICU integration

2017-02-20 Thread Peter Geoghegan
problem, and users will never learn to deal with issues like this well when it is by definition something that should never happen. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] ICU integration

2017-02-20 Thread Peter Geoghegan
xists. [1] http://site.icu-project.org/#TOC-Who-Uses-ICU- -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] ICU integration

2017-02-20 Thread Peter Geoghegan
versions many different things, in fact. Importantly, it explicitly decouples behavioral issues (user visible sort order -- UCA version) from technical issues (collator implementation details). So, my original point is that that could change, and if that happens we ought to have a plan. But, it won'

Re: [HACKERS] Documentation improvements for partitioning

2017-02-22 Thread Peter Geoghegan
t surprised at the limitations that this feature has, even if Bruce and Simon are. The documentation needs work, and perhaps the feature itself needs a small tweak here or there. Just not to a particularly notable degree, given the point we are in in the release cycle. -- Peter Geoghegan -- Sent

Re: [HACKERS] Documentation improvements for partitioning

2017-02-23 Thread Peter Geoghegan
h I disagree with. There is nothing disappointing to me about this feature, and, as I said, I am unsurprised that it doesn't support certain things. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Documentation improvements for partitioning

2017-02-23 Thread Peter Geoghegan
feature set, yet. Obviously that general principle is not under discussion. My point, of course, was that it seems pretty clear to me that this is on the right side of that fence. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Poor memory context performance in large hash joins

2017-02-23 Thread Peter Geoghegan
method for memory contexts, it looks like you just reset the parent instead. > But I don't think that would work here. Are you aware of the fact that tuplesort.c got a second memory context for 9.6, entirely on performance grounds? -- Peter Geoghegan -- Sent via pgsql-hackers ma

Re: [HACKERS] GUC for cleanup indexes threshold.

2017-02-24 Thread Peter Geoghegan
putting *fully* dead B-Tree pages in the FSM for recycling. The interlock with RecentGlobalXmin is what makes it impossible for VACUUM to generally fully delete pages, *as well as* mark them as recyclable (put them in the FSM) all at once. Maybe you get this already, since, as I said, the terminolog

[HACKERS] Should logtape.c blocks be of type long?

2017-02-25 Thread Peter Geoghegan
al one should assume that it is no wider than "int". This calls into question why any code that uses "long" didn't just use "int", at least in my mind. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Should logtape.c blocks be of type long?

2017-02-26 Thread Peter Geoghegan
s made more likely by the fact that we've made tuplesort faster in the past few releases (gains which the MAX_KILOBYTES restriction won't impinge on too much, particularly in Postgres 10). I find that unacceptable, at least for Postgres 10. -- Peter Geoghegan -- Sent via pgsql-hacke

Re: [HACKERS] Should logtape.c blocks be of type long?

2017-02-26 Thread Peter Geoghegan
istent about a restriction like this, as Robert said. Given that fixing this issue will not affect the machine code generated by compilers for the majority of platforms we support, doing so seems entirely worthwhile to me. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-

Re: [HACKERS] GUC for cleanup indexes threshold.

2017-02-27 Thread Peter Geoghegan
.0, but there might be non-linear increases in "the serious type of index bloat" as the proposed new setting was scaled up. I'd be much more worried about that. [1] https://archive.org/stream/symmetricconcurr00lani#page/6/mode/2up -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Polyphase merge is obsolete

2017-02-27 Thread Peter Geoghegan
On Mon, Jan 16, 2017 at 5:56 PM, Peter Geoghegan wrote: > On Wed, Oct 12, 2016 at 10:16 AM, Heikki Linnakangas wrote: >> The number of *input* tapes we can use in each merge pass is still limited, >> by the memory needed for the tape buffers and the merge heap, but only one &g

[HACKERS] Cost model for parallel CREATE INDEX

2017-02-28 Thread Peter Geoghegan
l.org/wiki/Parallel_External_Sort#bt_estimated_nblocks.28.29_function_in_pageinspect [3] https://www.postgresql.org/message-id/CAMkU=1y_qp+QUPGk=JBJSTtcYQpW2k=v2lmytzko_8ftuuy...@mail.gmail.com [4] https://www.postgresql.org/message-id/cam3swzr6c+1cwghc40g9z5thfe3u2xbv55w5-tertfeooaz...@mail.gmail.com -- Peter Geo

Re: [HACKERS] Cost model for parallel CREATE INDEX

2017-03-02 Thread Peter Geoghegan
On Thu, Mar 2, 2017 at 5:50 AM, Robert Haas wrote: > On Wed, Mar 1, 2017 at 12:58 AM, Peter Geoghegan wrote: >> * This scales based on output size (projected index size), not input >> size (heap scan input). Apparently, that's what we always do right >> now. > >

Re: [HACKERS] Cleanup: avoid direct use of ip_posid/ip_blkid

2017-03-02 Thread Peter Geoghegan
at is invalid in some sense, even if it isn't actually set to InvalidOffsetNumber. So, seems pretty risky to me. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Statement-level rollback

2017-03-03 Thread Peter Geoghegan
On Fri, Mar 3, 2017 at 9:01 AM, Andres Freund wrote: > On 2017-03-03 11:54:06 -0500, David Steele wrote: >> Given that this landed on March 28 with no discussion beforehand, I >> recommend that we immediately move this patch to the 2017-07 CF. > > Seconded. +1 -- Peter

Re: [HACKERS] GUC for cleanup indexes threshold.

2017-03-03 Thread Peter Geoghegan
a range of values for each index, a little like a BRIN index build. This range is what you go on to use to do a cheap index-scan-based B-Tree VACUUM. This could have far far less I/O, though has obvious risks that we need to worry about. That's work for another release, of course. -- Pete

Re: [HACKERS] GUC for cleanup indexes threshold.

2017-03-03 Thread Peter Geoghegan
On Fri, Mar 3, 2017 at 11:37 AM, Peter Geoghegan wrote: > Please verify my understanding of your thought process: We don't have > to freeze indexes at all, ever, so if we see index bloat as a separate > problem, we also see that there is no need to *link* index needs to > the

Re: [HACKERS] GUC for cleanup indexes threshold.

2017-03-03 Thread Peter Geoghegan
On Fri, Mar 3, 2017 at 11:49 AM, Peter Geoghegan wrote: >> Please verify my understanding of your thought process: We don't have >> to freeze indexes at all, ever, so if we see index bloat as a separate >> problem, we also see that there is no need to *link* index n

Re: [HACKERS] Skip all-visible pages during second HeapScan of CIC

2017-03-03 Thread Peter Geoghegan
rrectness of CIC - a relatively infrequent operation - on the > assumption that no VM bits can be set concurrenty due to the SUE lock. I agree. FWIW, the extra time that CIC takes over a plain CI is much reduced these days. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-ha

Re: [HACKERS] Skip all-visible pages during second HeapScan of CIC

2017-03-03 Thread Peter Geoghegan
lock on the heap relation (i.e. vacuuming it) after the first CIC transaction ends, but before the second CIC transaction begins? -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] GUC for cleanup indexes threshold.

2017-03-03 Thread Peter Geoghegan
On Fri, Mar 3, 2017 at 2:41 PM, Peter Geoghegan wrote: > In other words, the number of B-Tree pages that the last VACUUM > deleted, and thus made eligible to recycle by the next VACUUM has no > relationship with the number of pages the next VACUUM will itself end > up deleting, in gen

Re: [HACKERS] GUC for cleanup indexes threshold.

2017-03-03 Thread Peter Geoghegan
class.relfrozenxid/pg_database.datfrozenxid are advanced past opaque->btpo.xact. While I can't see this explained anywhere, I'm pretty sure that that's supposed to be impossible, which this patch changes. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hacke

Re: [HACKERS] 2017-03 Commitfest In Progress

2017-03-03 Thread Peter Geoghegan
> reviewing the aspects of it that touch on whether parallelism is being > done right, but I would like to have some help on the sorting end of > things. Your covering those aspects seems like something that would make this an easier sell to another reviewer. Thanks! -- Peter Geoghe

Re: [HACKERS] Cost model for parallel CREATE INDEX

2017-03-04 Thread Peter Geoghegan
dex size than current heap size. I agree with everything else you've said, I think. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Cost model for parallel CREATE INDEX

2017-03-04 Thread Peter Geoghegan
stimate. I don't really know what minimum amount of memory to insist workers have, which is why I provisionally chose one of those GUCs as the threshold. Any better ideas? -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subsc

Re: [HACKERS] Cost model for parallel CREATE INDEX

2017-03-04 Thread Peter Geoghegan
index size, not table size. I can change it to be table size, based on what you said. But the workMem related cap, which probably won't end up being applied all that often in practice, *should* still do something with projected index size, since that really is what we're sorting, whic

Re: [HACKERS] [PATCH] Use $ parameters as replacement characters for pg_stat_statements

2017-03-04 Thread Peter Geoghegan
o maximize the chances of that happening, but it's still generally quite possible (e.g. pg_stat_statements never swaps constants in a query like "SELECT 5, pg_stat_statements_reset()"). This means that we cannot really say that this buys us a machine-readable query text format, at least no

Re: [HACKERS] Cost model for parallel CREATE INDEX

2017-03-04 Thread Peter Geoghegan
inal index [1], to let the cost model cap the initial determination when maintenance_work_mem is just too low. (This cap will rarely be applied in practice, as I said.) [1] https://wiki.postgresql.org/wiki/Parallel_External_Sort#bt_estimated_nblocks.28.29_function_in_pageinspect -- Peter Geoghegan

Re: [HACKERS] REINDEX CONCURRENTLY 2.0

2017-03-05 Thread Peter Geoghegan
ls are arcane such that it might as well be that simple most of the time. Even if you have time to listen to me explain it all, which you clearly don't, you're still probably not going to be able to apply what you've learned in a way that helps you. -- Peter Geoghegan -- Sent via

Re: [HACKERS] Cost model for parallel CREATE INDEX

2017-03-05 Thread Peter Geoghegan
On Sat, Mar 4, 2017 at 2:15 PM, Peter Geoghegan wrote: > So, I agree with Robert that we should actually use heap size for the > main, initial determination of # of workers to use, but we still need > to estimate the size of the final index [1], to let the cost model cap >

[HACKERS] on_dsm_detach() callback and parallel tuplesort BufFile resource management

2017-03-06 Thread Peter Geoghegan
will end up being specialized to the problem he is trying to solve. I'm also slightly tempted to hard-code BufFiles as a new type of resource that a resource manager can take ownership of, but that also seems unappealing. What I've come up with may be as robust as anything will be for p

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2017-03-06 Thread Peter Geoghegan
most an alternative interface to the same functionality today. There can be another one in the future, if it serves a purpose, and the locking requirements are roughly the same for all the checks. I'd be fine with that. Let's just get the basic feature in for now, though. -- Peter Geogheg

Re: [HACKERS] GUC for cleanup indexes threshold.

2017-03-07 Thread Peter Geoghegan
in the course of anti-wraparound VACUUM, even if VACUUM has no garbage tuples to kill (even if we only do lazy_cleanup_index() instead of lazy_vacuum_index()). This is the case that this patch proposes to have us skip touching indexes for. -- Peter Geoghegan -- Sent via pgsql-hackers maili

Re: [HACKERS] Cost model for parallel CREATE INDEX

2017-03-08 Thread Peter Geoghegan
testing is in. There are a number of options, none of which are difficult to write code for. The hard part is determining what makes most sense for users on balance. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] on_dsm_detach() callback and parallel tuplesort BufFile resource management

2017-03-09 Thread Peter Geoghegan
ly advantage I immediately see with the approach 0007-hj-shared-buf-file-v6.patch takes (over what I've provisionally written as my V9) is that by putting everything in shared memory all along, there is no weirdness with tying local memory clean-up to a shared memory on_dsm_detach() callback. As

Re: [HACKERS] on_dsm_detach() callback and parallel tuplesort BufFile resource management

2017-03-09 Thread Peter Geoghegan
On Thu, Mar 9, 2017 at 11:19 AM, Peter Geoghegan wrote: > That patch seems to be solving the problem by completely taking over > management of temp files from fd.c. That is, these temp files are not > marked as temp files in the way ordinary temp BufFiles are, with > explicit

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

2017-03-09 Thread Peter Geoghegan
ent" thread. They still apply to this latest version of the patch series. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Cost model for parallel CREATE INDEX

2017-03-09 Thread Peter Geoghegan
't until some contrary evidence emerges. > > I mean, sometimes it is clear that you are going to need special > handling someplace, and then you have to do it. But I don't see that > this is one of those cases, necessarily. That's what I'll do, then. -- Peter Geog

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2017-03-09 Thread Peter Geoghegan
mas Munro, Anastasia Lubennikova, Robert Haas, Amit Langote" Thanks for your help with this! -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] on_dsm_detach() callback and parallel tuplesort BufFile resource management

2017-03-09 Thread Peter Geoghegan
On Thu, Mar 9, 2017 at 4:29 PM, Thomas Munro wrote: > On Fri, Mar 10, 2017 at 8:19 AM, Peter Geoghegan wrote: >> by having state for each segment, it ends up actually *relying on* >> ENOENT-on-unlink() as a condition that terminates execution: > > Yeah, this seems to fall o

Re: [HACKERS] on_dsm_detach() callback and parallel tuplesort BufFile resource management

2017-03-09 Thread Peter Geoghegan
owner clean-up of BufFiles. Otherwise, somebody might get in big trouble if they called BufFileClose() or something in an error path. Arguably, the reliance on ordering already exists today. I'm not saying that that's a good plan, or even an acceptable trade-off. Pick your poison. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2017-03-09 Thread Peter Geoghegan
ildenv\HEAD\pgsql.build\amcheck.vcxproj] Rather than marking RecentGlobalXmin as PGDLLIMPORT, I'd rather just remove the documenting assertion and leave that comment as-is. I'll work on a patch for this soon. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2017-03-09 Thread Peter Geoghegan
to export it. Well, the assertion is completely useless as anything but documentation... -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2017-03-09 Thread Peter Geoghegan
On Thu, Mar 9, 2017 at 7:12 PM, Peter Geoghegan wrote: >> Hm - I think it's fair to export RecentGlobalXmin, given that we >> obviously use it across modules in core code. I see very little reason >> not to export it. > > Well, the assertion is completely useless

Re: [HACKERS] [PATCH] Use $ parameters as replacement characters for pg_stat_statements

2017-03-09 Thread Peter Geoghegan
ay to expose the hash value in pg_stat_activity like that is above my pay grade, as Tom would say. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Parallel tuplesort (for parallel B-Tree index creation)

2017-03-12 Thread Peter Geoghegan
On Thu, Feb 16, 2017 at 8:45 AM, Peter Geoghegan wrote: >> I do not think there should be any reason why we can't get the >> resource accounting exactly correct here. If a single backend manages >> to remove every temporary file that it creates exactly once (and >>

Re: [HACKERS] Parallel tuplesort (for parallel B-Tree index creation)

2017-03-12 Thread Peter Geoghegan
On Sun, Mar 12, 2017 at 3:05 PM, Peter Geoghegan wrote: > There is still an open item here, though: The leader-as-worker > Tuplesortstate, a special case, can still leak files. I phrased this badly. What I mean is that there can be instances where temp files are left on disk following a f

Re: [HACKERS] tuplesort_gettuple_common() and *should_free argument

2017-03-13 Thread Peter Geoghegan
explanation > is is posted by 2017-03-16 AoE I will mark this submission > "Returned with Feedback". Apologies for the delay on this. I intend to get back to it before that time. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To mak

Re: [HACKERS] tuplesort_gettuple_common() and *should_free argument

2017-03-13 Thread Peter Geoghegan
t, because ltsReadBlock() could be involved instead. I don't remember the exact details offhand, so I will have to look into it again. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] tuplesort_gettuple_common() and *should_free argument

2017-03-13 Thread Peter Geoghegan
On Wed, Jan 25, 2017 at 3:11 PM, Peter Geoghegan wrote: > On Wed, Jan 25, 2017 at 3:11 PM, Tom Lane wrote: >> Please. You might want to hit the existing ones with a separate patch, >> but it doesn't much matter; I'd be just as happy with a patch that did >> both t

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-03-14 Thread Peter Geoghegan
without changing the user-visible interface. It seems pretty complementary to what is already there. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] GUC for cleanup indexes threshold.

2017-03-14 Thread Peter Geoghegan
autovauum recycles that > page while index vacuuming(lazy_index_vacuum). Sorry for the delay in getting back to you. I think that that's safe, but it is a little disappointing that it does not allow us to skip work in the case that you really had in mind when writing the patch. Bette

Re: [HACKERS] GUC for cleanup indexes threshold.

2017-03-14 Thread Peter Geoghegan
On Tue, Mar 14, 2017 at 2:48 PM, Peter Geoghegan wrote: > I think that that's safe, but it is a little disappointing that it > does not allow us to skip work in the case that you really had in mind > when writing the patch. Better than nothing, though, and perhaps still > a go

Re: [HACKERS] GUC for cleanup indexes threshold.

2017-03-14 Thread Peter Geoghegan
On Tue, Mar 14, 2017 at 3:10 PM, Peter Geoghegan wrote: > We already have BTPageOpaqueData.btpo, a union whose contained type > varies based on the page being dead. We could just do the same with > some other field in that struct, and then store epoch there. Clearly > nobody really

Re: [HACKERS] Review: GIN non-intrusive vacuum of posting tree

2017-03-16 Thread Peter Geoghegan
rd to make merging of pages that are not completely empty work, while also using the L&Y algorithm. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Parallel Append implementation

2017-03-17 Thread Peter Geoghegan
ed list sort, but it seems we don't > have. Will do the qsort now since it would be faster. relcache.c does an insertion sort with a list of OIDs. See insert_ordered_oid(). -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to you

Re: [HACKERS] [PATCH] SortSupport for macaddr type

2017-03-18 Thread Peter Geoghegan
ation, or if it aborted, then the index_getattr() can happen once per SortTuple, up-front. Nitpick: the patch should probably not refer to 32-bit or 64-bit systems. Rather, it should refer to Datum size only. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [PATCH] SortSupport for macaddr type

2017-03-18 Thread Peter Geoghegan
On Sat, Mar 18, 2017 at 2:54 PM, Peter Geoghegan wrote: > This seems fine to me, especially > because it lets us compare macaddr using simple 3-way unsigned int > comparisons, which isn't otherwise the case. Out of idle curiosity, I decided to generate disassembly of both macadd

Re: [HACKERS] [PATCH] SortSupport for macaddr type

2017-03-19 Thread Peter Geoghegan
er varlenas, because tuplesort has detected that that happens to be generally safe. I doubt that I'll ever get around to posting a patch to do that, since the cost savings are probably still marginal. I could probably find something better to work on. -- Peter Geoghegan -- Sent via

Re: [HACKERS] Parallel tuplesort (for parallel B-Tree index creation)

2017-03-19 Thread Peter Geoghegan
On Sun, Mar 12, 2017 at 3:05 PM, Peter Geoghegan wrote: > I attach my V9 of the patch. I came up some stuff for the design of > resource management that I think meets every design goal that we have > for shared/unified BufFiles: Commit 2609e91fc broke the parallel CREATE INDEX cost

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-03-20 Thread Peter Geoghegan
ow that the heap TIDs that are WARM root pointers are not going to be recycled in the lifetime of the amcheck query such that you get a false positive. A WARM check seems like a neat adjunct to what amcheck does already. It seems like a really good idea for WARM to buy into this kind of verificati

Re: [HACKERS] B-Tree support function number 3 (strxfrm() optimization)

2014-06-12 Thread Peter Geoghegan
trxfrm() in the standard. You're always going to want to have each strcmp() find differences as early as possible. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Doing better at HINTing an appropriate column within errorMissingColumn()

2014-06-16 Thread Peter Geoghegan
sibilities, and pick one that is very probably what was intended. Even if Levenshtein distance works badly with Kanji (which is not obviously the case, at least to me), it might not matter here. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make ch

[HACKERS] btreecheck extension

2014-06-16 Thread Peter Geoghegan
tion to the effectiveness of the tool generally. I envisage tests running with a high amount of concurrency on big databases over days or even longer. Thoughts? -- Peter Geoghegan *** a/contrib/btreecheck/Makefile --- b/contrib/btreecheck/Makefile *** *** 0 --- 1,18 +

Re: [HACKERS] Doing better at HINTing an appropriate column within errorMissingColumn()

2014-06-16 Thread Peter Geoghegan
shtein distance is used as opposed to any other scheme that might be better sometimes. I think that the hint given is a generally useful piece of information in the event of an ERRCODE_UNDEFINED_COLUMN error. Obviously I think the patch is worthwhile, but fundamentally the HINT given is just a

Re: [HACKERS] Doing better at HINTing an appropriate column within errorMissingColumn()

2014-06-16 Thread Peter Geoghegan
bout the deleted file contrib/fuzzystrmatch/levenshtein.c. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Doing better at HINTing an appropriate column within errorMissingColumn()

2014-06-16 Thread Peter Geoghegan
to end up in Maine rather than Oregon, but I think in general you can only go so far in worrying about those cases. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] btreecheck extension

2014-06-17 Thread Peter Geoghegan
n. It seems natural to do this, as the patch I've posted arguably adds a big modularity violation. Besides, it seems worthwhile to pepper the regular regression tests with calls like these, at least in some places, and putting something in core is the most convenient way to do that. -- Pet

Re: [HACKERS] Doing better at HINTing an appropriate column within errorMissingColumn()

2014-06-17 Thread Peter Geoghegan
; is the *best* match is correct at least on its own terms (terms that are self evident). This does pretty effectively communicate to the user that they should totally rethink not just the column name, but perhaps the entire query. On the other hand, showing nothing communicates nothing. -- Peter G

Re: [HACKERS] Doing better at HINTing an appropriate column within errorMissingColumn()

2014-06-17 Thread Peter Geoghegan
patch just looks for the match with the lowest distance, passing the lowest observed distance so far as a "max" to the distance calculation function. That could have some value in certain cases. People have already raised general concerns about added cycles and/or clutter. -- Peter

Re: [HACKERS] btreecheck extension

2014-06-17 Thread Peter Geoghegan
same trick that you came up with for EXPLAIN, to avoid grammar bloat and let the am figure out for itself what to name the various check types, with a generic default check. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Doing better at HINTing an appropriate column within errorMissingColumn()

2014-06-17 Thread Peter Geoghegan
practice, that ambiguity will frequently be something that our users will not care about, and not really see as an ambiguity, as in my "o.orderid or ol.orderid?" example. However, if there are 3 equally distant Vars, and not just 2, that's very probably because none are useful, and so we

Re: [HACKERS] btreecheck extension

2014-06-18 Thread Peter Geoghegan
ething like this work are mostly around the cost/benefit ratio of each of the checks I've outlined. Certainly, for that use case minimizing disruption on a live system becomes even more important. I'll probably look at a buffer access strategy, just to give an example of that off the top of m

Re: [HACKERS] Possible index issue on 9.5 slave

2014-06-18 Thread Peter Geoghegan
on all indexes on both systems. It shouldn't take too long. Thanks -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Possible index issue on 9.5 slave

2014-06-18 Thread Peter Geoghegan
ted, but it's certainly the case that bttextcmp()/varstr_cmp()/strcoll() return values must be immutable between the two systems. Still, it should be possible to determine if that's the problem using btreecheck. Do you get perfectly consistent answers between the two when you ORD

Re: [HACKERS] Possible index issue on 9.5 slave

2014-06-18 Thread Peter Geoghegan
ave more or less invented my own weird index scans. I assume you're referring to the field-verification of indexes use case, which is not an immediate goal of btreecheck, even though it's an important goal that there has already been some discussion of. -- Peter Geoghegan -- Sent via

[HACKERS] Fresh initdb contains a few deleted B-Tree pages

2014-06-30 Thread Peter Geoghegan
ssue. I am not suggesting that it's a real problem that requires immediate fixing, but it is suboptimal. We may wish to fix this someday. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Fresh initdb contains a few deleted B-Tree pages

2014-07-01 Thread Peter Geoghegan
y > more. It's not surprising that some initdb updates create garbage, but the extent to which they do did slightly surprise me. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] 9.5 CF1

2014-07-01 Thread Peter Geoghegan
g. Expect a revised patch soon. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Doing better at HINTing an appropriate column within errorMissingColumn()

2014-07-02 Thread Peter Geoghegan
at popped up in the regression tests that seemed kind of questionable no longer appear. These new measures make the coding somewhat more complex than that of the initial version, although overall the parser code added by this patch is almost entirely confined to code paths concerned only with pr

Re: [HACKERS] Ancient bug in formatting.c/to_char()

2014-07-02 Thread Peter Geoghegan
Where are we on this? -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Stating the significance of Lehman & Yao in the nbtree README

2014-07-02 Thread Peter Geoghegan
On Mon, May 26, 2014 at 1:22 PM, Peter Geoghegan wrote: > I think that this isn't enough. Attached patch proposes to add a small > paragraph at the top of the nbtree README, to clarify the advantages > of L&Y from a high level. I've added this to the ongoing commitfes

Re: [HACKERS] Stating the significance of Lehman & Yao in the nbtree README

2014-07-02 Thread Peter Geoghegan
of thing that generally doesn't need to be handled through a commitfest submission. The only reason I added this to any commitfest was to avoid having it be forgotten about entirely, which there is a real danger of for something like this when it isn't handled quickly. I almost forgot about

<    1   2   3   4   5   6   7   8   9   10   >