Re: [HACKERS] Interesting read on SCM upending software and hardware architecture

2016-01-18 Thread Peter Geoghegan
15/ I think it's much more true that main memory scaling (in particular, main memory capacity) has had a huge impact, but that trend appears to now be stalling. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription

Re: [HACKERS] [PATCH] Improve spinlock inline assembly for x86.

2016-01-18 Thread Peter Geoghegan
emoving superfluous code; doing so sometimes clarifies intent and understanding. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [PATCH] Improve spinlock inline assembly for x86.

2016-01-18 Thread Peter Geoghegan
e, let's let it drop. If we don't want to apply this, then I think that a sensible compromise would be to add a code comment that says that we don't believe the LOCK prefix matters. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To

Re: [HACKERS] PATCH: Extending the HyperLogLog API a bit

2016-01-19 Thread Peter Geoghegan
esentation. The distinction is explained in the README for postgresql-hll: https://github.com/aggregateknowledge/postgresql-hll postgresql-hll has no support for merging HLLs that are sparse: https://github.com/aggregateknowledge/postgresql-hll/blob/master/hll.c#L1888 Can't we just tear

Re: [HACKERS] PATCH: Extending the HyperLogLog API a bit

2016-01-19 Thread Peter Geoghegan
t complete implementations seem to switch from sparse to full at a certain point). So, ripping out mergeHyperLogLog() doesn't really make that effort any more difficult. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] PATCH: Extending the HyperLogLog API a bit

2016-01-19 Thread Peter Geoghegan
On Tue, Jan 19, 2016 at 2:22 PM, Peter Geoghegan wrote: > I don't have time right now to do a full analysis of whether or not it's > possible to just fix the bug without doing all that, but I think it > might not be. IOW: I think that Hideaki's bug fix might itself be wr

Re: [HACKERS] Releasing in September

2016-01-20 Thread Peter Geoghegan
developer meeting. I am not on the release list. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Releasing in September

2016-01-20 Thread Peter Geoghegan
.3. I don't think that one of those lessons was "take more time to release". There is reason to doubt that that would have changed matters one bit with 9.3. It might be a good idea to formally state what those lessons are. -- Peter Geoghegan -- Sent via pgsql-hackers maili

Re: [HACKERS] Releasing in September

2016-01-20 Thread Peter Geoghegan
se, to the surprise of many, the number of bugs in each point release (or their severity) did not really taper off. There was a particular qualitative nature to the bugs. This seems to be because the design was excessively complicated. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list

Re: [HACKERS] Releasing in September

2016-01-20 Thread Peter Geoghegan
cking WAL replay by comparing master and slave disk contents. We > should make an effort to get that into a state where anyone can use it. I think that that was initially Jeff Janes. Jeff also provided great testing infrastructure for UPSERT, something that I was very grateful for. -- Peter Geoghe

Re: [HACKERS] Releasing in September

2016-01-20 Thread Peter Geoghegan
o you really think that people need to be able to commit something before seriously working on it? Rules that are expressly coercive are a *bad* idea. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postg

Re: [HACKERS] Releasing in September

2016-01-20 Thread Peter Geoghegan
that you mentioned. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Releasing in September

2016-01-20 Thread Peter Geoghegan
reason why introducing it should be attached to sqlsmith's inclusion in core. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] PoC: Partial sort

2016-01-23 Thread Peter Geoghegan
nterested in seeing Alexander get back to this because it's an important project on its own. We should really have this. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Using quicksort for every external sort run

2016-01-27 Thread Peter Geoghegan
icksort with spillover" cost model. Hopefully this can be cleared up soon. As I've said, I am very receptive to other people's suggestions about how that should work. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] New committer

2016-01-28 Thread Peter Geoghegan
On Thu, Jan 28, 2016 at 6:37 AM, Magnus Hagander wrote: > The PostgreSQL core team would like to welcome Dean Rasheed as a new > committer for the PostgreSQL project. Congratulations, Dean. Well deserved. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-h

Re: [HACKERS] [WIP] Effective storage of duplicates in B-tree index.

2016-01-28 Thread Peter Geoghegan
e all the > values are the same, but perhaps that's due to my lack of understanding > regarding the way they're being stored. I think that you'd need an I/O bound workload to see significant benefits. That seems unsurprising. I believe that random I/O from index writes is a bi

Re: [HACKERS] Using quicksort for every external sort run

2016-01-29 Thread Peter Geoghegan
ult to interpret. > I think for long keys both old (RS) and new (Qsort) sort method has its own > characteristics > based on data distribution. I think work_mem is the key If properly set new > method(Qsort) will > be able to fit most of the cases. If work_mem is not tune

Re: [HACKERS] Using quicksort for every external sort run

2016-01-29 Thread Peter Geoghegan
ed at 50% (bad for "quicksort with spillover"), 25% (better), and 5% (good). An alternative approach that might be acceptable is to add a generic, conservative 90% threshold (so 10% of tuples sorted by heap). -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgr

Re: [HACKERS] Using quicksort for every external sort run

2016-01-29 Thread Peter Geoghegan
hase merges. I'm not sure that that's practical. Currently, tuplesort decides on a number of tapes ahead of time. When we're constrained on those, the stretch multiplier would apply, but I think that that could be invasive because the number of tapes ("merge order" + 1) was a function of non-stretched work_mem. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Generalizing SortSupport for text to work with char(n), bytea, and alternative opclasses

2016-02-04 Thread Peter Geoghegan
On Sun, Jan 31, 2016 at 7:59 PM, Andreas Karlsson wrote: > Nice work, I like your sorting patches. Thanks. I like your reviews of my sorting patches. :-) -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: h

Re: [HACKERS] Using quicksort for every external sort run

2016-02-04 Thread Peter Geoghegan
t adds no reliance on any estimate; it's fully self-tuning. Fine, but the point of "quicksort with spillover" is that it avoids I/O entirely. I'm not promoting it as useful for any of the reasons that replacement selection was traditionally useful (on 1970s hardware). So, we aren't much closer to working out a better cost model for "quicksort with spillover" (I guess you weren't really talking about that, though), an annoying sticking point (as already mentioned). -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Using quicksort for every external sort run

2016-02-04 Thread Peter Geoghegan
On Thu, Feb 4, 2016 at 1:46 AM, Peter Geoghegan wrote: > It wasn't my original insight that replacement selection has become > all but obsolete. It took me a while to come around to that point of > view. Nyberg et al may have said it best in 1994, in the Alphasort Paper [1]:

Re: [HACKERS] Raising the checkpoint_timeout limit

2016-02-04 Thread Peter Geoghegan
de a bad decision, there will usually be a chain of faulty but not outrageous reasoning behind it. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] WIP: Detecting SSI conflicts before reporting constraint violations

2016-02-04 Thread Peter Geoghegan
hat wasn't very important. But that's a problem that I'm closer to, and I won't express an opinion on this patch. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [WIP] Effective storage of duplicates in B-tree index.

2016-02-04 Thread Peter Geoghegan
992 │0.000 │ 99.643 % (3 rows) But this looks healthy -- I see the same with master. And since the accounts table is listed as 1281 MB, this looks like a plausible ratio in the size of the table to its primary index (which I would not say is true of an 87MB primary key index). Are you sure

Re: [HACKERS] [WIP] Effective storage of duplicates in B-tree index.

2016-02-04 Thread Peter Geoghegan
nch_accounts and reviewing the size of that. Right. Because, apart from everything else, unique indexes are not currently supported. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [WIP] Effective storage of duplicates in B-tree index.

2016-02-04 Thread Peter Geoghegan
y, this is really cool work; I have often thought that we don't have nearly enough people thinking about how to optimize B-Tree indexing. It is hard, but so is anything worthwhile. That's all I have for now. Just a quick review focused on code and correctness (and not on the benefits).

[HACKERS] Fix for OpenSSL error queue bug

2016-02-05 Thread Peter Geoghegan
.@wrigleys.postgresql.org#20150224030956.2529.83...@wrigleys.postgresql.org -- Peter Geoghegan From d5433bc562f792afd75ef8664729db9e6a60ee62 Mon Sep 17 00:00:00 2001 From: Peter Geoghegan Date: Tue, 26 Jan 2016 15:11:15 -0800 Subject: [PATCH] Distrust external OpenSSL clients; clear err queue OpenSSL has an unfor

Re: [HACKERS] Generalizing SortSupport for text to work with char(n), bytea, and alternative opclasses

2016-02-05 Thread Peter Geoghegan
On Wed, Feb 3, 2016 at 11:31 AM, Robert Haas wrote: > Thanks for the review. I fixed the OID conflict, tweaked a few > comments, and committed this. Thanks. I noticed a tiny, preexisting typo in the string abbreviated key code. The attached patch fixes it. -- Peter Geoghegan diff --git

Re: [HACKERS] Recently added typedef "string" is a horrid idea

2016-02-07 Thread Peter Geoghegan
able types generally. So, a rename to "VarString" probably makes sense, independent of your pgindent concern. If this sounds like a good idea, I'll produce a patch soon. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes

Re: [HACKERS] Using quicksort for every external sort run

2016-02-07 Thread Peter Geoghegan
sn't matter. "Juggling capacity" has nothing to do with modern hardware characteristics, except that modern machines are where the cost of excessive "juggling capacity" really hurts, so this is simple. It is simple *especially* because we can throw out the idea of a cost model that

Re: [HACKERS] Using quicksort for every external sort run

2016-02-07 Thread Peter Geoghegan
ee > tapes when outputing the final results. There will be the same amount > of I/O happening and a 2-way or 3-way merge for most data types should > be basically free. I basically agree with you, but it seems possible to fix the regression (generally misguided though those regressed cases ar

Re: [HACKERS] Using quicksort for every external sort run

2016-02-07 Thread Peter Geoghegan
On Sun, Feb 7, 2016 at 4:50 PM, Peter Geoghegan wrote: >> I'm not even sure this is necessary. The idea of missing out on >> producing a single sorted run sounds bad but in practice since we >> normally do the final merge on the fly there doesn't seem like there's

Re: [HACKERS] a raft of parallelism-related bug fixes

2016-02-08 Thread Peter Geoghegan
atch patch series for complicated stuff these days, but it's important to make it clear how everything fits together. That's actually what I was thinking about when I said we need to be clear on how things fit together from the CF app patch page, because there doesn't seem to be a cult

Re: [HACKERS] Recently added typedef "string" is a horrid idea

2016-02-08 Thread Peter Geoghegan
On Sun, Feb 7, 2016 at 7:47 AM, Tom Lane wrote: > Works for me. Attached patch is what I came up with. It required only minimal additional changes for consistency. -- Peter Geoghegan From 01d8cb278cecb995ecc30cda0125d10c98f4d05c Mon Sep 17 00:00:00 2001 From: Peter Geoghegan Date: Sun, 7

Re: [HACKERS] a raft of parallelism-related bug fixes

2016-02-08 Thread Peter Geoghegan
shout at me. If I persisted, you wouldn't commit my patch, and for me that probably means it's DOA. I don't think I'm asking for much here. [1] http://www.postgresql.org/message-id/CAM3SWZSefE4uQk3r_3gwpfDWWtT3P51SceVsL4=g8v_me2a...@mail.gmail.com [2] http://www.postgresql.org/

Re: [HACKERS] enable parallel query by default?

2016-02-08 Thread Peter Geoghegan
(it was eventually replaced with a #define before release, though). -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] a raft of parallelism-related bug fixes

2016-02-08 Thread Peter Geoghegan
y try to do that. That's all. Thank you (and Amit) for working really hard on parallelism. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] a raft of parallelism-related bug fixes

2016-02-08 Thread Peter Geoghegan
ssues you noticed'. I would really like for there to be a way to do that more often. It would be a significant time saver, because it removes problems with minor bitrot. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subsc

Re: [HACKERS] Using quicksort for every external sort run

2016-02-14 Thread Peter Geoghegan
On Sun, Feb 7, 2016 at 4:50 PM, Peter Geoghegan wrote: >> I'm not even sure this is necessary. The idea of missing out on >> producing a single sorted run sounds bad but in practice since we >> normally do the final merge on the fly there doesn't seem like there's

Re: [HACKERS] Re: Reusing abbreviated keys during second pass of ordered [set] aggregates

2016-02-15 Thread Peter Geoghegan
On Tue, Dec 22, 2015 at 10:49 PM, Peter Geoghegan wrote: > I attach a rebased patch for 9.6 only. I marked the patch -- my own patch -- "Ready for Committer". I'm the third person to have marked the patch "Ready for Committer", even though no committer bounced the

Re: [HACKERS] Remove or weaken hints about "effective resolution of sleep delays is 10 ms"?

2016-02-15 Thread Peter Geoghegan
ould certainly matter, because optimal settings are rarely greater than 10 milliseconds. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Remove or weaken hints about "effective resolution of sleep delays is 10 ms"?

2016-02-15 Thread Peter Geoghegan
On Mon, Feb 15, 2016 at 11:23 AM, Peter Geoghegan wrote: > commit_delay doesn't have any guidance like this, where it could > certainly matter, because optimal settings are rarely greater than 10 > milliseconds. Actually, it does, but it's in "29.4. WAL Configu

Re: [HACKERS] PoC: Partial sort

2016-02-15 Thread Peter Geoghegan
fixed soon, so moving a patch to the next commitfest no longer clobbers its state. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Re: Reusing abbreviated keys during second pass of ordered [set] aggregates

2016-02-15 Thread Peter Geoghegan
st time in a long time that the CF wasn't closed because the next CF began. :-) -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Bug with int2

2016-02-16 Thread Peter Geoghegan
,768 (1 row) -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Re: Reusing abbreviated keys during second pass of ordered [set] aggregates

2016-02-17 Thread Peter Geoghegan
On Wed, Feb 17, 2016 at 2:12 AM, Robert Haas wrote: > Committed, except I left out one comment hunk that I wasn't convinced about. Thank you. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscripti

Re: [HACKERS] [WIP] speeding up GIN build with parallel workers

2016-02-17 Thread Peter Geoghegan
. Obviously that's the really interesting baseline. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [DOCS] The number of bytes is stored in index_size of pgstatindex() ?

2016-02-18 Thread Peter Geoghegan
attuple and pageinspect going into this kind of detail in their documentation generally. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [DOCS] The number of bytes is stored in index_size of pgstatindex() ?

2016-02-18 Thread Peter Geoghegan
─┼┼┼─── 206 │81 │ 3 │160 │42 (1 row) BTW, I am actively working on the amcheck B-Tree checker tool, and hope to post it soon. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsq

Re: [HACKERS] [DOCS] The number of bytes is stored in index_size of pgstatindex() ?

2016-02-18 Thread Peter Geoghegan
t; possible to see zero or multiple roots because the root's location > changes. That's already mentioned in the documentation, if somewhat > obliquely. Ah, yes. Another consequence of going in physical order. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-h

Re: [HACKERS] [DOCS] The number of bytes is stored in index_size of pgstatindex() ?

2016-02-18 Thread Peter Geoghegan
I vote back patch. Subtle differences between the branches should be avoided. -- Peter Geoghegan

Re: [HACKERS] [DOCS] The number of bytes is stored in index_size of pgstatindex() ?

2016-02-18 Thread Peter Geoghegan
r example by updating the descriptions of the fields avg_leaf_density and leaf_fragmentation in the docs. Just a thought. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Remove or weaken hints about "effective resolution of sleep delays is 10 ms"?

2016-02-18 Thread Peter Geoghegan
dance for commit_delay's effective resolution from "29.4. WAL Configuration" to where commit_delay is introduced, "18.5. Write Ahead Log". -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [DOCS] The number of bytes is stored in index_size of pgstatindex() ?

2016-02-18 Thread Peter Geoghegan
ngs like this because I care about making sure these tools are useful for teaching novice hackers about Postgres internals. I've made my point and will leave it at that, though. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [DOCS] The number of bytes is stored in index_size of pgstatindex() ?

2016-02-18 Thread Peter Geoghegan
t-link/left-link chains, and there are usage patterns where half-dead pages might accumulate. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [DOCS] The number of bytes is stored in index_size of pgstatindex() ?

2016-02-19 Thread Peter Geoghegan
uld happen. I meant where pages could accumulate without being recycled. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [DOCS] The number of bytes is stored in index_size of pgstatindex() ?

2016-02-19 Thread Peter Geoghegan
new > index pages, but I don't think that's an interesting case.) Sure. I'm talking about a narrow issue around how things are represented in pgstatindex() only. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to yo

[HACKERS] CLUSTER, reform_and_rewrite_tuple(), and parallelism

2016-08-17 Thread Peter Geoghegan
aightforward to have reform_and_rewrite_tuple() work occur in parallel workers instead, which buys us a lot. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] CLUSTER, reform_and_rewrite_tuple(), and parallelism

2016-08-17 Thread Peter Geoghegan
tuples, the hash lookups in > rewrite_heap_tuple(), ...? Perhaps the attached svg "flamegraph" will give you some idea. This is based on the "perf cache-misses" event type. -- Peter Geoghegan cache-misses-cluster-presorted-flamegraph.svg.gz Description: GNU Zip compressed da

Re: [HACKERS] CLUSTER, reform_and_rewrite_tuple(), and parallelism

2016-08-17 Thread Peter Geoghegan
ated that the tuplesort CLUSTER takes just under 3 minutes (this includes writing out the new heap, of course). -- Peter Geoghegan cycles-cluster-presorted-flamegraph.svg.gz Description: GNU Zip compressed data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make ch

Re: [HACKERS] CLUSTER, reform_and_rewrite_tuple(), and parallelism

2016-08-17 Thread Peter Geoghegan
tuff in the frame pointer at that point. You either need to use > dwarf or lbr to get accurate ones. Is it worth doing that here, and redoing the test, so that the glibc attributions are correct? -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-08-18 Thread Peter Geoghegan
On Sat, Mar 12, 2016 at 12:38 PM, Peter Geoghegan wrote: > Only insofar as it helps diagnose the underlying issue, when it is a > more subtle issue. Actually fixing the index is almost certainly a > REINDEX. Once you're into the messy business of diagnosing a > problematic opclas

Re: [HACKERS] [WIP] [B-Tree] Keep indexes sorted by heap physical location

2016-08-18 Thread Peter Geoghegan
address this problem is with a duplicate list and/or prefix compression in leaf pages. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [WIP] [B-Tree] Keep indexes sorted by heap physical location

2016-08-18 Thread Peter Geoghegan
s necessary to be very ambitious in order to solve a problem. The understandable and usually well-reasoned approach of making progress as incremental as possible occasionally works against contributors. It's worth considering if this is such a case. -- Peter Geoghegan -- Sent via pgs

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-08-18 Thread Peter Geoghegan
y, I would like to make amcheck verifying the heap through a B-Tree index as a next step. There is also a good case for the tool directly verifying heap relations, without involving any index, but that can come later. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgres

Re: [HACKERS] _mdfd_getseg can be expensive

2016-08-18 Thread Peter Geoghegan
On Thu, Aug 18, 2016 at 5:26 PM, Andres Freund wrote: > Rebased version attached. A review would be welcome. Plan to push this > forward otherwise in the not too far away future. I can review this next week. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-h

Re: [HACKERS] _mdfd_getseg can be expensive

2016-08-18 Thread Peter Geoghegan
t that would be a waste of time. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] _mdfd_getseg can be expensive

2016-08-18 Thread Peter Geoghegan
On Thu, Aug 18, 2016 at 5:42 PM, Andres Freund wrote: > How large was the index & table in question? I mean this really only > comes into effect at 100+ segments. Not that big, but I see no reason to take the chance, I suppose. -- Peter Geoghegan -- Sent via pgsql-hackers ma

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-08-18 Thread Peter Geoghegan
;s much more likely that they'd be "false positive" bugs. In any case, I haven't seen any issue with the tool itself yet, having now run the tool on thousands of servers. I think I'll have a lot more information in about a week, when I've had time to work through more data

[HACKERS] Bug in abbreviated keys abort handling (found with amcheck)

2016-08-19 Thread Peter Geoghegan
fraction of all databases tested, so I don't think it's very common in the wild. I'd be surprised if amcheck does not bring more bugs like this to my attention before too long. We should work on improving it, so that we have greater visibility into problems that occur in the f

Re: [HACKERS] Bug in abbreviated keys abort handling (found with amcheck)

2016-08-22 Thread Peter Geoghegan
yet, even though any instances of corruption of text indexes I've seen originated before the point release in which strxfrm() became distrusted. I guess that not that many Heroku users use the "C" locale, which would still be affected with the latest point release. -- Peter Geogheg

Re: [HACKERS] Parallel tuplesort (for parallel B-Tree index creation)

2016-08-22 Thread Peter Geoghegan
On Mon, Aug 1, 2016 at 3:18 PM, Peter Geoghegan wrote: > Attached WIP patch series: This has bitrot, since commit da1c9163 changed the interface for checking parallel safety. I'll have to fix that, and will probably take the opportunity to change how workers have maintenance_work_mem app

[HACKERS] Re: [BUGS] Re: Missing rows with index scan when collation is not "C" (PostgreSQL 9.5)

2016-08-22 Thread Peter Geoghegan
thousands of servers. For some reason, both cases involved strings with code points from the Arabic alphabet, even though each case was from a totally unrelated customer database. I'll go update the Wiki page for this [1] now. [1] https://wiki.postgresql.org/wiki/Abbreviated_keys_glibc_issue

Re: [HACKERS] Forbid use of LF and CR characters in database and role names

2016-08-22 Thread Peter Geoghegan
I haven't looked at the patch, but offhand I wonder if it's worth considering control characters added by unicode, if you haven't already. -- Peter Geoghegan

Re: [HACKERS] Forbid use of LF and CR characters in database and role names

2016-08-22 Thread Peter Geoghegan
is an interesting discussion of the matter here: http://www.unicode.org/reports/tr36/#Bidirectional_Text_Spoofing -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [RFC] Change the default of update_process_title to off

2016-08-23 Thread Peter Geoghegan
concern here: https://www.postgresql.org/message-id/30619.1428157...@sss.pgh.pa.us ISTM that we don't even care about Windows performance to a minimal degree. Hopefully, the ICU stuff Peter Eisentraut is working on will level the playing field here a little bit, if only as an accidental side

Re: [HACKERS] UPSERT strange behavior

2016-08-25 Thread Peter Geoghegan
27;t upsert while using more than one index as an arbiter index. This is true unless they're more or less equivalent, in which case multiple arbiter indexes can be inferred, but that clearly doesn't apply here. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hacke

Re: [HACKERS] UPSERT strange behavior

2016-08-25 Thread Peter Geoghegan
ct is initially found, and so no guarantees here.) Anyway, I don't have a lot of sympathy for this point of view, because the scenario is completely contrived. You have to draw the line somewhere. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postg

Re: [HACKERS] UPSERT strange behavior

2016-08-25 Thread Peter Geoghegan
lly also wrote a patch to prefer insertion into the primary key first, which also went nowhere (I gave up on that one, to be fair). -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] UPSERT strange behavior

2016-08-25 Thread Peter Geoghegan
On Thu, Aug 25, 2016 at 12:59 PM, Peter Geoghegan wrote: > Maybe we should change the ordering of those IndexInfo structs to > something more suitable, but it must be immutable (it cannot hinge > upon the details of one particular DML statement). I meant that it must be stable (not

Re: [HACKERS] increasing the default WAL segment size

2016-08-25 Thread Peter Geoghegan
k, because WAL segments consist of zeroes at the end when archive_timeout is applied (at least from 9.4 on). We compress the WAL segments, and many zeroes compress very well. I admit that I haven't looked at it in much detail, but that is my current understanding. -- Peter Geoghegan

Re: [HACKERS] WAL consistency check facility

2016-08-27 Thread Peter Geoghegan
s special case for speculative insertion (to WAL-log the virtually useless speculative insertion token value)? I'm certain that the answer must be "no": This tool ought to deal with speculative insertion as a special case, and not vice-versa. -- Peter Geoghegan -- Sent via pgsql-h

Re: [HACKERS] WAL consistency check facility

2016-08-27 Thread Peter Geoghegan
st, because the other flags have clear-cut roles in various atomic operations that we WAL-log. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] WAL consistency check facility

2016-08-27 Thread Peter Geoghegan
ery or on standby after WAL replay. Right you are -- while BTP_INCOMPLETE_SPLIT is set during recovery, BTP_SPLIT_END is not. Still, most of the btpo_flags flags that are masked in the patch shouldn't be. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresq

Re: [HACKERS] ICU integration

2016-08-31 Thread Peter Geoghegan
dual OS packagers of ICU deciding what exact CLDR data to use, which may or may not be of any significant consequence in practice. [1] http://unicode.org/reports/tr10 [2] http://site.icu-project.org/design/size/collation [3] http://userguide.icu-project.org/icudata -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] ICU integration

2016-08-31 Thread Peter Geoghegan
iguration > option perhaps in initdb to change the default so that, say, "fr_FR" > uses ICU and "fr_FR%posix" uses the old stuff. I suspect that we'd be better off adding a mechanism for adding a new collation after initdb runs, on a live production instance. Maybe t

Re: [HACKERS] _mdfd_getseg can be expensive

2016-08-31 Thread Peter Geoghegan
ly doesn't matter, given what that array tracks. I'm just pointing out that that aspect did give me pause. The struct MdfdVec is small enough that that *might* be worthwhile. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your

Re: [HACKERS] _mdfd_getseg can be expensive

2016-08-31 Thread Peter Geoghegan
On Wed, Aug 31, 2016 at 2:09 PM, Peter Geoghegan wrote: > The only thing that stuck out to any degree is that we don't grow the > "reln->md_seg_fds[forknum]" array within the new _fdvec_resize() > function geometrically. That new function looks like this: >

Re: [HACKERS] [PATCH] Reload SSL certificates on SIGHUP

2016-08-31 Thread Peter Geoghegan
On Sun, Nov 22, 2015 at 7:29 PM, Andreas Karlsson wrote: > Sorry for dropping this patch, but now I have started looking at it again. Any chance of picking this up again soon, Andreas? I think it's an important project. I would like to review it. -- Peter Geoghegan -- Sent v

Re: [HACKERS] _mdfd_getseg can be expensive

2016-08-31 Thread Peter Geoghegan
), contrary to my reading of the 0002-* patch comments. I'm mostly talking thinking about x86_64 here, of course. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] _mdfd_getseg can be expensive

2016-08-31 Thread Peter Geoghegan
On Wed, Aug 31, 2016 at 3:08 PM, Andres Freund wrote: > On August 31, 2016 3:06:23 PM PDT, Peter Geoghegan wrote: > >>In other painfully pedantic news, I should point out that >>sizeof(size_t) isn't necessarily word size (the most generic >>definition of word size fo

Re: [HACKERS] WAL consistency check facility

2016-08-31 Thread Peter Geoghegan
rmation about a problem. And, ideally, we'd also have some indication of how big a difference that would make, it terms of measurable performance impact. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] WAL consistency check facility

2016-09-01 Thread Peter Geoghegan
o use it on all production databases. It wouldn't have mattered that the verification was no less effective, since the bugs it found would simply never have been observed in practice. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] less expensive pg_buffercache on big shmem

2016-09-01 Thread Peter Geoghegan
d on the off chance that it mattered, without actually being justified. I would like to be able to run pg_buffercache in production from time to time. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-09-02 Thread Peter Geoghegan
On Thu, Aug 18, 2016 at 5:14 PM, Peter Geoghegan wrote: > I'd certainly welcome that. There are Debian packages available from > the Github version of amcheck, which is otherwise practically > identical to the most recent version of the patch posted here: > > https://githu

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-09-02 Thread Peter Geoghegan
g at.) There are only tiny differences, which in any case you can see in the commit log on Github. There is no reason why code review needs to block on this V3, IMV. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-09-02 Thread Peter Geoghegan
On Fri, Sep 2, 2016 at 11:16 AM, Peter Geoghegan wrote: > There are only tiny differences, which in any case you can see in the > commit log on Github. There is no reason why code review needs to > block on this V3, IMV. Also, I don't think that we need to include V2's test

Re: [HACKERS] System load consideration before spawning parallel workers

2016-09-02 Thread Peter Geoghegan
, priority, and possibly other considerations. I see the 9.6 work on external sort as a building piece for that, as it removed the one thing that was sensitive to work_mem in a surprising, unpredictable way. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

<    1   2   3   4   5   6   7   8   9   10   >