Re: [HACKERS] snapshot too old, configured by time

2016-04-07 Thread Peter Geoghegan
o at my insistence at the time; pruning turned out to be very important for many common workloads, and Andres got that right). I can't really imagine a way that what you have here could be any less effective than what Andres did for logical decoding. This is reassuring, since that mechanis

Re: [HACKERS] Using quicksort for every external sort run

2016-04-07 Thread Peter Geoghegan
ement selection begins; it governs where it ends. How do you feel about replacement_sort_tuples? We already use the word "tuple" in the names of GUCs. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Using quicksort for every external sort run

2016-04-07 Thread Peter Geoghegan
mmitted without those > changes. I agree to all this. Now that you've indicated where you stand on replacement_sort_mem, I have all the information I need to produce a new revision. I'll go do that. Thanks -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@p

Re: [HACKERS] Using quicksort for every external sort run

2016-04-07 Thread Peter Geoghegan
uples? Tuples are basically equivalent, but make it way less obvious what the relationship with CPU cache might be. If I revise the patch along these lines, I should also reduce the default replacement_sort_mem to produce roughly equivalent behavior for non-padded cases. -- Peter Geoghegan

Re: [HACKERS] WIP: Covering + unique indexes.

2016-04-06 Thread Peter Geoghegan
On Wed, Apr 6, 2016 at 1:50 PM, Peter Geoghegan wrote: > Personally, I like documenting assertions, and will sometimes write > assertions that the compiler could easily optimize away. Maybe going > *that* far is more a matter of personal style, but I think an > assertion about the new

Re: [HACKERS] WIP: Covering + unique indexes.

2016-04-06 Thread Peter Geoghegan
w index tuple size being <= the old one is just a good idea. It's not about a problem in your code at all. > I do not mind to rename this function, but what name would be better? > index_truncate_tuple()? That seems better, yes. -- Peter Geoghegan -- Sent via pgsql-hacke

Re: [HACKERS] WIP: Covering + unique indexes.

2016-04-05 Thread Peter Geoghegan
On Tue, Apr 5, 2016 at 1:31 PM, Peter Geoghegan wrote: > My new understanding: The extra "included" columns are stored in the > index, but do not affect its sort order at all. They are no more part > of the key than, say, the heap TID that the key points to. They are > just

Re: [HACKERS] WIP: Covering + unique indexes.

2016-04-05 Thread Peter Geoghegan
runcation is 100% guaranteed to produce a tuple smaller than the inserted tuple, otherwise the user could get a non-recoverable "1/3 of page size exceeded" when they were not the one to insert the big IndexTuple. I should try to see if this could be possible due to some index_reform_tuple() edge-case. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Relation extension scalability

2016-04-05 Thread Peter Geoghegan
ct 4 to take 120 seconds, but in really takes > 105 unpatched or 80 patched. So we're not actually I/O constrained on > this test, I think, perhaps because this machine has an SSD. It's not unusual for COPY to not be I/O constrained, I believe. -- Peter Geoghegan -- Sent via p

Re: [HACKERS] Timeline following for logical slots

2016-04-04 Thread Peter Geoghegan
ople to work on the harder problems. It's surprising to me that we don't seem to be much closer to that then we were when 9.4 was released. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] snapshot too old, configured by time

2016-04-04 Thread Peter Geoghegan
s? If so, that > seems quite surprising. Or perhaps I am missing something else here. What proportion of the statements in your simulated workload were updates? Per my last mail to this thread, I'm interested in knowing if this was a delete heavy workload. -- Peter Geoghegan -- Sent via pgs

Re: [HACKERS] snapshot too old, configured by time

2016-04-04 Thread Peter Geoghegan
On Wed, Mar 30, 2016 at 12:46 PM, Kevin Grittner wrote: > On Wed, Mar 30, 2016 at 2:29 PM, Peter Geoghegan wrote: > >> [Does the patch allow dangling page pointers?] > >> Again, I don't want to prejudice anyone against your patch, which I >> haven't read.

Re: [HACKERS] WIP: Covering + unique indexes.

2016-04-04 Thread Peter Geoghegan
to what we have today. > I'll add a flag to distinguish regular and truncated tuples, but it will not > be used in this patch. Please, comment, if I've missed something. > As you've already mentioned, neither high keys, nor tuples on internal pages > are using "itup-&

Re: [HACKERS] WIP: Covering + unique indexes.

2016-04-04 Thread Peter Geoghegan
On Mon, Apr 4, 2016 at 7:14 AM, Teodor Sigaev wrote: > Are there any objectins on it? I'm planning to look closely today or > tommorrow and commit it. I object to committing the patch in that time frame. I'm looking at it again. -- Peter Geoghegan -- Sent via pgsql-hack

Re: [HACKERS] Using quicksort for every external sort run

2016-04-03 Thread Peter Geoghegan
y useful goal to simply making sorting faster. We don't have to be afraid of crossing an arbitrary, fuzzy threshold. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Using quicksort for every external sort run

2016-04-03 Thread Peter Geoghegan
only case I can see working for replacement selection: where indexes can be created with very little memory quickly, by optimistically starting to write out the start of the final index representation almost immediately, before most of the underlying table has even been read in. -- Peter Geoghegan

Re: [HACKERS] Using quicksort for every external sort run

2016-04-03 Thread Peter Geoghegan
some of the regressions? Maybe, but that seems hard. That information isn't conveniently available to the executor/tuplesort, and as we've seen with CREATE INDEX int4 cases, it's far from clear that we'll win even when there definitely is presorted input. Replacement selection ne

Re: [HACKERS] Add schema-qualified relnames in constraint error messages.

2016-04-02 Thread Peter Geoghegan
>> regression=# > > Wow, that's a fabulous idea. I see Oleksandr has tried to implement > it, although I haven't looked at the patch. But I think this would be > REALLY helpful. +1 -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Using quicksort for every external sort run

2016-04-02 Thread Peter Geoghegan
On Sat, Apr 2, 2016 at 3:22 PM, Peter Geoghegan wrote: > On Sat, Apr 2, 2016 at 3:20 PM, Greg Stark wrote: >> There are also some weird cases in this list where there's a >> significant regression at 32MB but not at 8MB. I would like to see >> 16MB and perhaps 12MB

Re: [HACKERS] Using quicksort for every external sort run

2016-04-02 Thread Peter Geoghegan
f this only affects the pathological > cases, we may ignore that. That's fair. I probably shouldn't have mentioned TOAST at all -- what's actually important to keep in mind about padding cases, as already mentioned, is that they can make the 32MB cases behave like the 8MB cases. The memtuples heap is left relatively small for the 32MB case too, and so can remain cache resident. Replacement selection therefore almost accidentally gets fewer heap cache misses for a little longer, but it's still the same pattern. Cache misses come to dominate a bit later. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Using quicksort for every external sort run

2016-04-02 Thread Peter Geoghegan
e the memtuples heap relatively small in each case. So with work_mem=32MB, you get a memtuples heap structure similar to work_mem=8MB. The padding pushes things out a bit further, which favors master. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To

Re: [HACKERS] Using quicksort for every external sort run

2016-04-02 Thread Peter Geoghegan
re's a consistent pattern. I'll need to drill down to trace_sort output to see what happened there. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Using quicksort for every external sort run

2016-04-01 Thread Peter Geoghegan
On Thu, Feb 4, 2016 at 3:14 AM, Peter Geoghegan wrote: > Nyberg et al may have said it best in 1994, in the Alphasort Paper [1]: This paper is available from http://www.vldb.org/journal/VLDBJ4/P603.pdf (previously link is now dead) > The paper also has very good analysis of the econom

Re: [HACKERS] pg_upgrade 9.6->9.6: column "amtype" does not exist

2016-04-01 Thread Peter Geoghegan
On Fri, Apr 1, 2016 at 2:31 PM, Christoph Berg wrote: > I'm not sure this is a bug, but before it bites back too late, I'm > reporting it now. This must be a regression from the changes made to the pg_am interface by commit 65c5fcd353a859da9e61bfb2b92a99f12937de3b. --

Re: [HACKERS] Draft release notes for next week's releases

2016-03-31 Thread Peter Geoghegan
t covered, I suppose, because everyone will look to the Wiki page for guidance. I also noted quite a few non-obvious safe cases on the Wiki page, as pointed out already over on the other thread. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To

Re: [HACKERS] Draft release notes for next week's releases

2016-03-31 Thread Peter Geoghegan
har(n) as affected. That's not actually true, because char(n) SortSupport only came in 9.6. The Wiki page now shows this, which may be the most important place, but ideally we'd fix this in the release notes. I guess it's too late. -- Peter Geoghegan -- Sent via pgsql-hackers mai

Re: [HACKERS] Please correct/improve wiki page about abbreviated keys bug

2016-03-31 Thread Peter Geoghegan
ing column indexes are not affected (although that's just mentioned in passing when discussing the SQL query, as it's a bit esoteric). -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Please correct/improve wiki page about abbreviated keys bug

2016-03-30 Thread Peter Geoghegan
Okay. I'll look at it later today. -- Peter Geoghegan

Re: [HACKERS] snapshot too old, configured by time

2016-03-30 Thread Peter Geoghegan
On Wed, Mar 30, 2016 at 12:21 PM, Peter Geoghegan wrote: > Well, amcheck is a tool that in essence makes sure that B-Trees look > structurally sound, and respect invariants like having every item on > each page in logical order. That could catch a bug of the kind I just > described,

Re: [HACKERS] snapshot too old, configured by time

2016-03-30 Thread Peter Geoghegan
me hope of catching this kind of race condition (the stale downlink that you followed to get to the spuriously-recycled-early page doesn't stay stale for long). Or, maybe it would happen to catch some other random problem. Difficult to say. Again, this is based on a speculation that might

Re: [HACKERS] Using quicksort for every external sort run

2016-03-30 Thread Peter Geoghegan
ion of sorts, which could effectively decrease memory use on the system. Increasing the memory available to sorts could decrease the overall use of memory. Being really frugal with memory is expensive, maybe even if your primary concern is the expense of memory usage, which it probably i

Re: [HACKERS] Using quicksort for every external sort run

2016-03-29 Thread Peter Geoghegan
practical setup that anyone would want to use? [1] http://www.postgresql.org/message-id/CAM-w4HOwt0C7ZndowHUuraw+xi+BhY5a6J008XoSq=r9z7h...@mail.gmail.com [2] http://www.postgresql.org/message-id/CAM-w4HM4XW3u5kVEuUrr+L+KX3WZ=5JKk0A=djjzypkb-hy...@mail.gmail.com -- Peter Geoghegan -- Sent v

Re: [HACKERS] Using quicksort for every external sort run

2016-03-29 Thread Peter Geoghegan
o something or not do something based on this information is subjective. OTOH, whether and to what extent these tests are representative of real user workloads seems much less subjective. This is not a criticism of the test cases you came up with, which rightly emphasized possibly regressed cases. I

Re: [HACKERS] Please correct/improve wiki page about abbreviated keys bug

2016-03-29 Thread Peter Geoghegan
Since that's the only attribute that could use abbreviated keys, it ought to be safe to not require users to REINDEX indexes that happen to have a second-or-subsequent text/varchar(n) attribute that doesn't use the C locale. Maybe it's not worth worrying about. -- Peter Geoghegan --

Re: [HACKERS] Please correct/improve wiki page about abbreviated keys bug

2016-03-29 Thread Peter Geoghegan
ome things. I noticed it said something about incorrect unique violations on affected systems. Is that really possible? I can't imagine how it could be, although that is left in place largely unchanged for now. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postg

Re: [HACKERS] Using quicksort for every external sort run

2016-03-29 Thread Peter Geoghegan
On Tue, Mar 29, 2016 at 12:43 PM, Peter Geoghegan wrote: > A complete do-over from Tomas would be best, here. He has already > acknowledged that the i5 CREATE INDEX results were completely invalid. The following analysis is all based on Xeon numbers, which as I've said we should focus

Re: [HACKERS] Using quicksort for every external sort run

2016-03-29 Thread Peter Geoghegan
x27;t done any analysis of how much better that made the problem *across the board* yet, but for int4 cases I could make 1MB work_mem queries faster with gigabytes of data on my laptop. I believe I tested various datum sort cases there, like "select count(distinct(foo)) from bar";

Re: [HACKERS] Using quicksort for every external sort run

2016-03-28 Thread Peter Geoghegan
On Thu, Mar 10, 2016 at 6:54 PM, Peter Geoghegan wrote: > I've used amcheck [2] to test this latest revision -- the tool ought > to not see any problems with any index created with the patch applied. > Reviewers might find it helpful to use amcheck, too. As 9.6 is > stabilized,

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-03-28 Thread Peter Geoghegan
On Tue, Mar 22, 2016 at 10:57 AM, Peter Geoghegan wrote: > That's right - I have a small number of feedback items to work > through. I also determined myself that there could be a very low > probability race condition when checking the key space across sibling > pages, and wil

Re: [HACKERS] Dealing with collation and strcoll/strxfrm/etc

2016-03-28 Thread Peter Geoghegan
the CLDR [2]. I don't think we'd want to have Postgres ship "supported collations" in each major version, in roughly the style of the IANA timezone stuff, but it's far too early to rule that out. It would have upsides. [1] https://en.wikipedia.org/wiki/Unicode_collation_algorithm [2] http://cldr.unicode.org/ -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Dealing with collation and strcoll/strxfrm/etc

2016-03-28 Thread Peter Geoghegan
basis for case-insensitive collations, which would let us kill citext, a module that I consider to be a total kludge. And, we might also be able to lock down WAL compatibility, which would be generally useful. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Draft release notes for next week's releases

2016-03-28 Thread Peter Geoghegan
all using UTF-16 only. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Draft release notes for next week's releases

2016-03-28 Thread Peter Geoghegan
ICU might also do better in your test case for that reason. [1] https://www.sqlite.org/src/artifact?ci=trunk&filename=ext/icu/README.txt -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Draft release notes for next week's releases

2016-03-27 Thread Peter Geoghegan
take at least that long before affected users even start to think about this. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Fix for OpenSSL error queue bug

2016-03-26 Thread Peter Geoghegan
ne with that. Will this make it into the next point release? I was rather hoping it would. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Using quicksort for every external sort run

2016-03-24 Thread Peter Geoghegan
On Sun, Mar 20, 2016 at 11:01 PM, Peter Geoghegan wrote: > Allowing 0 tuple runs in rare cases seems like the simplest solution. > After all, mergeprereadone() is expressly prepared for 0 tuple runs. > It says "ensure that we have at least one tuple, if any are to be > had".

Re: [HACKERS] [WIP] Effective storage of duplicates in B-tree index.

2016-03-24 Thread Peter Geoghegan
ing with amcheck. I need help with formalizing a "testing first" approach for improving the B-Tree code, because I think it's the only way that we can move forward with projects like this. It's *incredibly* hard to push forward patches like this given our current, limited t

Re: [HACKERS] Using quicksort for every external sort run

2016-03-23 Thread Peter Geoghegan
iation did, how memory was used, and even if the sort was I/O bound at various stages (it dumps some getrusage stats to the log, too). You can usually tell exactly what happened for external sorts, which is very interesting for those one or two cases that you found to be noticeably worse off with the patch. Thanks for testing! -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Using quicksort for every external sort run

2016-03-23 Thread Peter Geoghegan
me the slowdown is related to the batching (as it's only happening for > low work_mem values), so perhaps there's an internal heuristics that we > could tune? Can you show trace_sort output for these cases? Both master, and patched? Thanks -- Peter Geoghegan -- Sent via pgsql-h

Re: [HACKERS] Fix for OpenSSL error queue bug

2016-03-23 Thread Peter Geoghegan
On Mon, Mar 14, 2016 at 6:44 PM, Peter Geoghegan wrote: > I can produce a back-patchable variant of this if you and Peter E. > think this approach is okay. Where are we on this? -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes t

Re: [HACKERS] Using quicksort for every external sort run

2016-03-23 Thread Peter Geoghegan
te I/O bound with the patch sometimes, where it's usually CPU/memory bound with the heap, especially with lots of work_mem. More importantly, it would be more informative if the temp_tablespace was not affected by I/O from Postgres' heap. I also like seeing a sample of "trace_sort

Re: [HACKERS] PoC: Partial sort

2016-03-23 Thread Peter Geoghegan
ions (I had no feedback on "partial-sort-basic-6.patch", which didn't use the new upper planner pathification stuff, unlike this latest version). > On Wed, Nov 4, 2015 at 4:44 AM, Peter Geoghegan wrote: >> >> Explain output >> --- >> I th

Re: [HACKERS] Using quicksort for every external sort run

2016-03-22 Thread Peter Geoghegan
t think I see any here, nor any high memory tests where RS flops. Now, I think you focused on regressions because that was what was interesting, which is good. I just want to put that in context. Thanks -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-03-22 Thread Peter Geoghegan
On Tue, Mar 22, 2016 at 10:57 AM, Peter Geoghegan wrote: > That's right - I have a small number of feedback items to work > through. I also determined myself that there could be a very low > probability race condition when checking the key space across sibling > pages, and wil

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-03-22 Thread Peter Geoghegan
could be a very low probability race condition when checking the key space across sibling pages, and will work to address that. If I'm right about that then it's not a lot of work to fix; I'm probably just going to use the right page's high key rather than its first data item. --

Re: [HACKERS] Minor bug affecting ON CONFLICT lock wait log messages

2016-03-21 Thread Peter Geoghegan
inions, if any are to be had. Sorry that I changed my mind, but it's a subtle issue, I'm sure you'll agree. I'm not going to push on this, but I want to be sure that we're happy with this. To reiterate, I think it boils down to: Is it okay that this new XLTW_InsertIndex case reports someone else's TID, while the only other XLTW_InsertIndex case has always reported our own TID? Discussing these sorts of "ontological" questions reminds me just how painful UPSERT was as a project. :-) -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Using quicksort for every external sort run

2016-03-20 Thread Peter Geoghegan
ocations occurs after batch memory is used -- we need to be careful about how that could impact the "bucketing by size class"). Thanks -- Peter Geoghegan From b921e285ed3f22c9cab9c78c7c610fbdfee5839b Mon Sep 17 00:00:00 2001 From: Peter Geoghegan Date: Thu, 10 Mar 2016 14:52:18 -08

Re: [HACKERS] snapshot too old, configured by time

2016-03-20 Thread Peter Geoghegan
On Sun, Mar 20, 2016 at 4:25 PM, Peter Geoghegan wrote: > I worry that something weird could happen there. For example, perhaps > the page LSN on what is actually a newly recycled page could be set > such that the backend following a stale right spuriously raises a > "snapshot t

Re: [HACKERS] snapshot too old, configured by time

2016-03-20 Thread Peter Geoghegan
you consider making amcheck [1] a part of your testing strategy. I think that this patch is a good idea, and I'd be happy to take feedback from you on how to make amcheck more effective for testing this patch in particular. [1] https://commitfest.postgresql.org/9/561/ -- Peter Geoghegan

Re: [HACKERS] Refactoring speculative insertion with unique indexes a little

2016-03-19 Thread Peter Geoghegan
asy to miss some of the subtleties. I do not pursue something like this without good reason. I'm optimistic that the patch will be accepted if it is carefully considered. -- Peter Geoghegan From 2b2a4c40a5e60ac1f28a75f11204ce88eb48cc73 Mon Sep 17 00:00:00 2001 From: Peter Geoghegan Date: Tue, 2

Re: [HACKERS] Make primnodes.h gender neutral

2016-03-19 Thread Peter Geoghegan
ndered pronouns has negative value, though, and it really isn't that hard to fix. So we should just fix it. (In case it matters, I'm in favor of this proposal on its merits). -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To m

Re: [HACKERS] fd.c doesn't remove files on a crash-restart

2016-03-19 Thread Peter Geoghegan
a bad idea FWIW, I've seen this get out of hand several times myself. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [COMMITTERS] pgsql: Improve memory management for external sorts.

2016-03-19 Thread Peter Geoghegan
On Fri, Mar 18, 2016 at 11:43 AM, Andres Freund wrote: > Yes, that removes the warning, and looks correct. Thanks. We should be careful to not repeat this mistake when the quicksort patch goes in. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)

Re: [HACKERS] [COMMITTERS] pgsql: Improve memory management for external sorts.

2016-03-19 Thread Peter Geoghegan
mat specifiers. However, I had a bad feeling about integer overflow of state->currentRun, and think I'll need to address that. After all, if runs are now no longer 2x work_mem on average, it's not completely ridiculous to imagine that being an issue on a misconfigured system. -- Pet

Re: [HACKERS] Using quicksort for every external sort run

2016-03-19 Thread Peter Geoghegan
quot;Can SortTuple.tuple ever be set?", rather than allowing it to remain a specific property of a datum tuplesort. state->datumTypByVal often isn't initialized in master, and so cannot be checked as things stand (unless the code is in a datum-case-specific routine). This new flag controls batch memory in slightly higher-level way than would otherwise be possible. It also controls the memory prefetching added by patch/commit 0003-*, FWIW. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Using quicksort for every external sort run

2016-03-19 Thread Peter Geoghegan
On Wed, Mar 16, 2016 at 6:42 PM, Peter Geoghegan wrote: >> - I think that batchmemtuples() is somewhat weird. Normally, >> grow_memtuples() doubles the size of the array each time it's called. >> So if you somehow called this function when you still had lots of >> me

Re: [HACKERS] Make primnodes.h gender neutral

2016-03-18 Thread Peter Geoghegan
losing anything important seems like a relatively straightforward task. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] WIP: Covering + unique indexes.

2016-03-18 Thread Peter Geoghegan
33/ Marked "Waiting on Author". -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Using quicksort for every external sort run

2016-03-18 Thread Peter Geoghegan
what wasn't so obvious to you when you review my patches. It's probably impossible to stare at something like tuplesort.c for as long as I have and get that balance just right. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to

Re: [HACKERS] [WIP] speeding up GIN build with parallel workers

2016-03-16 Thread Peter Geoghegan
his is why I wanted an easy way of atomically guaranteeing some number of workers (typically 2), or not using parallelism at all. I think the parallel worker API should offer a simple way to do that in cases like this, where having only 1 worker is never going to win. -- Peter Geoghegan -- Sent via p

Re: [HACKERS] Minor bug affecting ON CONFLICT lock wait log messages

2016-03-16 Thread Peter Geoghegan
came from the fact that historically, when check_exclusion_or_unique_constraint() was called check_exclusion_constraint(), it (almost) was our own tuple that was waited on. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS][PROPOSAL] Covering + unique indexes.

2016-03-15 Thread Peter Geoghegan
On Mon, Mar 14, 2016 at 8:43 PM, Peter Geoghegan wrote: > Does this work with amcheck? Maybe it works with bt_index_check(), but > not bt_index_parent_check()? I think that you need to make sure that > _bt_compare() knows about this, too. That's because it isn't good > eno

Re: [HACKERS] Minor bug affecting ON CONFLICT lock wait log messages

2016-03-15 Thread Peter Geoghegan
On Tue, Mar 15, 2016 at 6:18 AM, Stephen Frost wrote: > Agreed. I'm going to play with it a bit more but barring objections, > I'll commit and back-patch Peter's patch. Thanks for taking care of this, Stephen. -- Peter Geoghegan -- Sent via pgsql-hackers mailin

Re: [HACKERS] [NOVICE] WHERE clause not used when index is used

2016-03-15 Thread Peter Geoghegan
. This test suite would indirectly test external sorting, B-Tree page deletion, edge-cases with very large IndexTuples, etc. Ideas for good areas of the B-Tree code to add tests for are welcome. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make chan

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-03-15 Thread Peter Geoghegan
s a non-native speaker of English language, who > ends up looking up quite a few words regularly) - > > Could we use "conform" or "correspond" instead of "comport" in the > following error message: > > "left link/right link pair in index \"%s\&q

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-03-15 Thread Peter Geoghegan
nternal pages are checked against actual child pages (every item in the child page, in fact). It's the parent/child relationship that is verified in addition to the standard checks of every page on and across (not between) every level. -- Peter Geoghegan -- Sent via pgsql-hackers mailin

Re: [HACKERS] jsonb array-style subscription

2016-03-14 Thread Peter Geoghegan
On Thu, Mar 3, 2016 at 2:31 AM, Dmitry Dolgov <9erthali...@gmail.com> wrote: > Well, actually, I agree with that. I can try to rework the patch to achieve > this goal. Good idea. I wonder, having taken a quick look at the patch, how this works?: +select * from test_jsonb_subscript where test_jso

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-03-14 Thread Peter Geoghegan
ually). I haven't actually tested this theory of mine myself just yet, but let me know what you think of it on the thread for your patch. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS][PROPOSAL] Covering + unique indexes.

2016-03-14 Thread Peter Geoghegan
On Mon, Mar 14, 2016 at 8:43 PM, Peter Geoghegan wrote: > * I think the comparison logic may have a bug. > > Does this work with amcheck? Maybe it works with bt_index_check(), but > not bt_index_parent_check()? I think that you need to make sure that > _bt_compare() knows about th

Re: [HACKERS][PROPOSAL] Covering + unique indexes.

2016-03-14 Thread Peter Geoghegan
On Mon, Mar 14, 2016 at 8:43 PM, Peter Geoghegan wrote: > > Does this work with amcheck? Maybe it works with bt_index_check(), but > not bt_index_parent_check()? I think that you need to make sure that > _bt_compare() knows about this, too. That's because it isn't good >

Re: [HACKERS][PROPOSAL] Covering + unique indexes.

2016-03-14 Thread Peter Geoghegan
"/truncated attributes are involved. This seems necessary. No? It's necessary because you aren't storing any attributes, so it's not acceptable to even attempt a comparison -- I think that will segfault (doesn't matter that the index scan wouldn't have returne

Re: [HACKERS] Minor bug affecting ON CONFLICT lock wait log messages

2016-03-14 Thread Peter Geoghegan
On Mon, Mar 7, 2016 at 1:46 PM, Peter Geoghegan wrote: > Attached patch fixes a bug reported privately by Stephen this morning. Bump. I would like to see this in the next point release. It shouldn't be hard to review. Thanks -- Peter Geoghegan -- Sent via pgsql-hackers mailing lis

Re: [HACKERS] Fix for OpenSSL error queue bug

2016-03-14 Thread Peter Geoghegan
On Mon, Mar 14, 2016 at 4:11 PM, Peter Geoghegan wrote: > Yes, with one small difference: I wouldn't be calling ERR_get_error() > in the common case where SSL_get_error() returns SSL_ERROR_NONE, on > the theory that skipping that case represents no risk. I'm making a > c

Re: [HACKERS] Refactoring speculative insertion with unique indexes a little

2016-03-14 Thread Peter Geoghegan
On Mon, Mar 14, 2016 at 5:21 PM, Andres Freund wrote: > So? You're not the only one. I don't see why we shouldn't move this to > 'returned with feedback' until there's a new version. I don't see any point in that; I intend to get a revision in to the o

Re: [HACKERS] Refactoring speculative insertion with unique indexes a little

2016-03-14 Thread Peter Geoghegan
x 9.6 patches to review, most of which are in need of support. I'm very busy. I said that I'd get to this patch soon. I might be kicking the can down the road a little with this patch; if so, I'm sorry. I suggest we leave it at that, until the CF is almost over or until I produce

Re: [HACKERS] Fix for OpenSSL error queue bug

2016-03-14 Thread Peter Geoghegan
hat will calling ERR_get_error() more will add useless cycles. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Fix for OpenSSL error queue bug

2016-03-14 Thread Peter Geoghegan
backend-side code as well. I agree that the backend-side code should be covered. I quickly changed my mind about that, and am happy to produce a revision along those lines. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-03-14 Thread Peter Geoghegan
pecially compelling feature. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Fix for OpenSSL error queue bug

2016-03-14 Thread Peter Geoghegan
On Thu, Mar 10, 2016 at 8:16 PM, Peter Geoghegan wrote: > On Thu, Mar 10, 2016 at 7:22 PM, Peter Eisentraut wrote: >> Arguably, if everyone followed "my" approach, this should be very easy >> to fix everywhere. > > I don't think that there is any clear indica

Re: [HACKERS] checkpointer continuous flushing - V18

2016-03-13 Thread Peter Geoghegan
or performance tips will probably only work well if there are still high standards that are actively enforced. There still needs to be tight editorial control. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http:/

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-03-13 Thread Peter Geoghegan
open source a > version of it later this week that way you can use it for testing. That could be interesting. The earlier prototypes of this tool are known to have detected glibc collation incompatibilities in real production systems. -- Peter Geoghegan -- Sent via pgsql-hackers mailing li

Re: [HACKERS] Refactoring speculative insertion with unique indexes a little

2016-03-12 Thread Peter Geoghegan
On Sat, Mar 12, 2016 at 2:53 PM, Peter Geoghegan wrote: > I said "basically uncontroversial", not "uncontroversial". That is a > perfectly accurate characterization of the patch, and if you disagree > than I suggest you re-read the thread. In particular, note that Al

Re: [HACKERS] Refactoring speculative insertion with unique indexes a little

2016-03-12 Thread Peter Geoghegan
gorically stated was that he disliked one narrow aspect of the style of one thing in one function. I've already said I'm happy to do that. As things stand, the documentation for amcanunique methods, and the way they are described internally, is fairly misleading. -- Peter Geoghega

Re: [HACKERS] Refactoring speculative insertion with unique indexes a little

2016-03-12 Thread Peter Geoghegan
tch is basically uncontroversial, and is built to make the AM interface clearer, and the design of speculative insertion easier to understand. It's clear we should have it. I'll get around to revising it before too long. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hacker

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-03-12 Thread Peter Geoghegan
diagnosing a problematic opclass, you have to be an expert, and tweaking amcheck for your requirements (i.e. rebuilding from source) becomes reasonable. Part of the reason that the code is so heavily commented is to make it hackable, because I do not feel optimistic that I can get an expert-orientat

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-03-11 Thread Peter Geoghegan
I'll add that if people like the interface you propose. (Overloading the VACUUM cost delay functions to cause a delay for amcheck functions, too). Note that the functions already use an appropriate buffer access strategy (it avoids blowing shared_buffers, much like VACUUM itself). -- Peter

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-03-11 Thread Peter Geoghegan
On Fri, Mar 11, 2016 at 4:17 PM, Peter Geoghegan wrote: > If you want the tool to limp on when it finds an error, that can be > done by changing the constant for the CORRUPTION macro in amcheck.c. > But having that be dynamically configurable is not really compatible > with the go

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-03-11 Thread Peter Geoghegan
ndex relation. Not sure about the cost delay thing. Delays are disabled by default for manually issued VACUUM, so have doubts that that's useful. If you want the tool to limp on when it finds an error, that can be done by changing the constant for the CORRUPTION macro in amcheck.c. But hav

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-03-11 Thread Peter Geoghegan
utes/columns past the first distinguishing/distinct attribute on each item on internal pages). That's a case that amcheck then needs to care about, just like it currently cares about the existing concept of minus infinity items. That's how it goes for amcheck. -- Peter Geoghegan

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-03-11 Thread Peter Geoghegan
On Fri, Mar 11, 2016 at 1:31 PM, Peter Geoghegan wrote: > You could have a race, where > there was a concurrent page deletion of the left sibling of the child > page, then a concurrent insertion into the newly expanded keyspace of > the parent. Therefore, the downlink in the parent (

<    4   5   6   7   8   9   10   11   12   13   >