[HACKERS] Re: Patch to address concerns about ICU collcollate stability in v10 (Was: CREATE COLLATION does not sanitize ICU's BCP 47 language tags. Should it?)

2017-09-26 Thread Peter Geoghegan
On Sun, Sep 24, 2017 at 9:24 PM, Peter Geoghegan <p...@bowt.ie> wrote: > * Documents the aforementioned keyword collation attribute restriction > on ICU versions before ICU 54. This was needed anyway. We only claim > for Postgres collations what the ICU docs claim for ICU collators

Re: [HACKERS] CREATE COLLATION does not sanitize ICU's BCP 47 language tags. Should it?

2017-09-25 Thread Peter Geoghegan
On Mon, Sep 25, 2017 at 12:52 PM, Peter Geoghegan <p...@bowt.ie> wrote: > That must have been the real reason why you canonicalized > pg_collation.collname (I doubt it had anything to do with how keyword > variants used to be created during initdb, as you suggested). As Tom > po

Re: [HACKERS] CREATE COLLATION does not sanitize ICU's BCP 47 language tags. Should it?

2017-09-25 Thread Peter Geoghegan
On Mon, Sep 25, 2017 at 12:14 PM, Peter Geoghegan <p...@bowt.ie> wrote: > But then our users categorically have to know about both formats, > without any practical benefit to make up for it. You will also get > people that don't realize that only one format is supported on some &

Re: [HACKERS] Patch to address concerns about ICU collcollate stability in v10 (Was: CREATE COLLATION does not sanitize ICU's BCP 47 language tags. Should it?)

2017-09-25 Thread Peter Geoghegan
On Mon, Sep 25, 2017 at 11:42 AM, Peter Eisentraut <peter.eisentr...@2ndquadrant.com> wrote: > On 9/25/17 00:24, Peter Geoghegan wrote: >> * Creates root collation as root-x-icu (collcollate "root"), not >> und-x-icu. "und" means undefined language

Re: [HACKERS] CREATE COLLATION does not sanitize ICU's BCP 47 language tags. Should it?

2017-09-25 Thread Peter Geoghegan
On Mon, Sep 25, 2017 at 11:40 AM, Peter Eisentraut <peter.eisentr...@2ndquadrant.com> wrote: > On 9/22/17 16:46, Peter Geoghegan wrote: >> But you are *already* canonicalizing ICU collation names as BCP 47. My >> point here is: Why not finish the job off, and *also* canon

Re: [HACKERS] CREATE COLLATION does not sanitize ICU's BCP 47 language tags. Should it?

2017-09-25 Thread Peter Geoghegan
even supported in some ICU versions > we support, so how's that going to work? And if it's been changed in > the recent past, why not again? Peter Geoghegan said that he doesn't > know of any plans to eliminate BCP 47 support, but that doesn't seem > like it's much proof of anything. It's des

[HACKERS] Patch to address concerns about ICU collcollate stability in v10 (Was: CREATE COLLATION does not sanitize ICU's BCP 47 language tags. Should it?)

2017-09-24 Thread Peter Geoghegan
llowing naming scheme and its API functions are deprecated. Use ucol_open() with language tag collation keywords instead (see Collation API Details)" [1] [1] http://userguide.icu-project.org/collation/concepts#TOC-Collator-naming-scheme -- Peter Geoghegan From 989bc2f877aa01af4ac140a43a5ce

Re: [HACKERS] ICU locales and text/char(n) SortSupport on Windows

2017-09-24 Thread Peter Geoghegan
On Sun, Sep 24, 2017 at 5:04 AM, Peter Eisentraut <peter.eisentr...@2ndquadrant.com> wrote: > On 9/22/17 12:25, Peter Geoghegan wrote: >> On Fri, Sep 22, 2017 at 7:25 AM, Peter Eisentraut >> <peter.eisentr...@2ndquadrant.com> wrote: >>> I agree. The attached

Re: [HACKERS] CREATE COLLATION does not sanitize ICU's BCP 47 language tags. Should it?

2017-09-22 Thread Peter Geoghegan
? langtag : name" thing). That looks very much like the tail wagging the dog to me. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] CREATE COLLATION does not sanitize ICU's BCP 47 language tags. Should it?

2017-09-22 Thread Peter Geoghegan
On Fri, Sep 22, 2017 at 5:58 PM, Robert Haas <robertmh...@gmail.com> wrote: > On Fri, Sep 22, 2017 at 4:46 PM, Peter Geoghegan <p...@bowt.ie> wrote: >> But you are *already* canonicalizing ICU collation names as BCP 47. My >> point here is: Why not finish the job o

Re: [HACKERS] CREATE COLLATION does not sanitize ICU's BCP 47 language tags. Should it?

2017-09-22 Thread Peter Geoghegan
s that we'll only really get one chance at here. Admittedly, that's not true of everything I raise here; it's hard to strictly limit discussion to issues that have that quality. [1] postgr.es/m/cah2-wznopmj+3xh6bvea_yuyd4zdgiwg9yce31q09ou3xxw...@mail.gmail.com -- Peter Geoghegan -- Sent via pgsq

Re: [HACKERS] ICU locales and text/char(n) SortSupport on Windows

2017-09-22 Thread Peter Geoghegan
m comparator (that goes through bttextcmp(), in the case of text) will be installed within FinishSortSupportFunction(). Other than that, looks good to me. Thanks -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscr

Re: [HACKERS] CREATE COLLATION does not sanitize ICU's BCP 47 language tags. Should it?

2017-09-22 Thread Peter Geoghegan
On Fri, Sep 22, 2017 at 1:50 AM, Andreas Karlsson <andr...@proxel.se> wrote: > On 09/21/2017 10:55 PM, Peter Geoghegan wrote: >> >> I agree, but the bigger issue is that we're *half way* between >> supporting only one format, and supporting two formats. AFAICT, there &g

Re: [HACKERS] CREATE COLLATION does not sanitize ICU's BCP 47 language tags. Should it?

2017-09-21 Thread Peter Geoghegan
On Tue, Sep 19, 2017 at 7:01 PM, Peter Geoghegan <p...@bowt.ie> wrote: > I really think we need to add some kind of debug mode that makes ICU > optionally spit out a locale display name at key points. We need this > to gain confidence that the behavior that ICU provides actually

Re: [HACKERS] CREATE COLLATION does not sanitize ICU's BCP 47 language tags. Should it?

2017-09-21 Thread Peter Geoghegan
here is no reason that we can't simply support one format on all ICU versions, and keep what ends up within pg_collation at initdb time essentially the same across ICU versions (except for those that are due to cultural/political developments). -- Peter Geoghegan -- Sent via pgsql-hackers maili

Re: [HACKERS] ICU locales and text/char(n) SortSupport on Windows

2017-09-21 Thread Peter Geoghegan
item. I'm still hopeful that it can be worked through for v10 at Peter's discretion. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] ICU locales and text/char(n) SortSupport on Windows

2017-09-21 Thread Peter Geoghegan
ibc) anyway. This would also be a win for clarity, IMV. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] ICU locales and text/char(n) SortSupport on Windows

2017-09-21 Thread Peter Geoghegan
than 3x faster). This is mostly because ICU brings back abbreviated keys. 3x - 5x faster is more of a qualitative difference than a quantitative one, IMHO. That having been said, I'm certainly not going to insist on a v10 fix for this issue. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list

Re: [HACKERS] CREATE COLLATION does not sanitize ICU's BCP 47 language tags. Should it?

2017-09-20 Thread Peter Geoghegan
On Wed, Sep 20, 2017 at 4:08 PM, Peter Geoghegan <p...@bowt.ie> wrote: >> pg_import_system_collations() takes care to use the non-BCP-47 style for >> such versions, so I think this is working correctly. > > But CREATE COLLATION doesn't use pg_import_system_collati

Re: [HACKERS] CREATE COLLATION does not sanitize ICU's BCP 47 language tags. Should it?

2017-09-20 Thread Peter Geoghegan
doesn't accept BCP >> 47 syntax locale strings until ICU 54 [1]. > > pg_import_system_collations() takes care to use the non-BCP-47 style for > such versions, so I think this is working correctly. But CREATE COLLATION doesn't use pg_import_system_collations(). -- Peter Geoghegan

Re: [HACKERS] Boom filters for hash joins (was: A design for amcheck heapam verification)

2017-09-20 Thread Peter Geoghegan
On Tue, Sep 19, 2017 at 1:25 PM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: > On 09/19/2017 06:03 PM, Peter Geoghegan wrote: >> I believe that parallelism makes the use of Bloom filter a lot more >> compelling, too. Obviously this is something that wasn't taken

Re: [HACKERS] CREATE COLLATION does not sanitize ICU's BCP 47 language tags. Should it?

2017-09-20 Thread Peter Geoghegan
On Tue, Sep 19, 2017 at 7:01 PM, Peter Geoghegan <p...@bowt.ie> wrote: > I didn't post the patch that generates the errors in my opening e-mail > because I'm not confident it's correct just yet. And, I think that I > see a bigger problem: we pass a string that is almost certainly a

Re: [HACKERS] Parallel Hash take II

2017-09-20 Thread Peter Geoghegan
artitioning. You may be able to get some ideas from there. And even if you don't, his handling of the topic is very deliberate and high level, which suggests that ours should be, too. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes t

Re: [HACKERS] Varying results when using merge joins over postgres_fdw vs hash joins

2017-09-20 Thread Peter Geoghegan
() this whole time. At least, for ICU versions prior to ICU 54. This will need to be addressed very soon. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Varying results when using merge joins over postgres_fdw vs hash joins

2017-09-20 Thread Peter Geoghegan
rning about server collation in the postgres_fdw docs, or a user-visible NOTICE when incompatible server collations are observed by postgres_fdw's IMPORT? -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Varying results when using merge joins over postgres_fdw vs hash joins

2017-09-20 Thread Peter Geoghegan
I think that Corey describes a user hostile behavior. I feel that we should try to do better here. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] CREATE COLLATION does not sanitize ICU's BCP 47 language tags. Should it?

2017-09-19 Thread Peter Geoghegan
On Tue, Sep 19, 2017 at 5:52 PM, Peter Eisentraut <peter.eisentr...@2ndquadrant.com> wrote: > On 9/18/17 18:46, Peter Geoghegan wrote: >> As I pointed out a couple of times already [1], we don't currently >> sanitize ICU's BCP 47 language tags within CREATE COLLATION. > &

Re: [HACKERS] GUC for cleanup indexes threshold.

2017-09-19 Thread Peter Geoghegan
or was like this since before their xact began/snapshot was acquired. For your reference, this RecentGlobalXmin interlock stuff is what Lanin & Shasha call "The Drain Technique" within "2.5 Freeing Empty Nodes". Seems pretty hard to do it any other way. -- Peter Geoghegan

Re: [HACKERS] Parallel tuplesort (for parallel B-Tree index creation)

2017-09-19 Thread Peter Geoghegan
57:10.054) > > With patch (max_parallel_workers_maintenance = 8): > > Time: 1163445.271 ms (19:23.445) This looks very similar to my v10. While I will need to follow up on this, to make sure, it seems likely that this patch has exactly the same performance characteristics as v10. Thank

Re: [HACKERS] CREATE COLLATION does not sanitize ICU's BCP 47 language tags. Should it?

2017-09-19 Thread Peter Geoghegan
syntax, though. PostgreSQL's CREATE COLLATION is not one of the places where this kind of leniency makes sense. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] CREATE COLLATION does not sanitize ICU's BCP 47 language tags. Should it?

2017-09-19 Thread Peter Geoghegan
nd of thing that we ought to go out of our way to get right in the *first* version. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Boom filters for hash joins (was: A design for amcheck heapam verification)

2017-09-19 Thread Peter Geoghegan
here only small number of > rows from the outer relation has a match in the hash table). I believe that parallelism makes the use of Bloom filter a lot more compelling, too. Obviously this is something that wasn't taken into consideration in 2015. -- Peter Geoghegan -- Sent via pgsql-hackers m

Re: [HACKERS] GUC for cleanup indexes threshold.

2017-09-18 Thread Peter Geoghegan
; > I think this patch clearly still is in the design stage, and has > received plenty feedback this CF. I'll therefore move this to the next > commitfest. Does anyone have ideas on a way forward here? I don't, but then I haven't thought about it in detail in several months. -- Peter Geoghe

[HACKERS] CREATE COLLATION does not sanitize ICU's BCP 47 language tags. Should it?

2017-09-18 Thread Peter Geoghegan
l.icu-project.org/apiref/icu4c/uloc_8h.html#a1d50c91925ca3853fce6f28cf7390c3c -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Boom filters for hash joins (was: A design for amcheck heapam verification)

2017-09-18 Thread Peter Geoghegan
ing a bloom filter may not matter. Obviously this is all pretty speculative. I suspect that this could be true, and it seems worth investigating that framing of the problem first. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscr

Re: [HACKERS] Boom filters for hash joins (was: A design for amcheck heapam verification)

2017-09-18 Thread Peter Geoghegan
the additional overhead of having a Bloom filter is fixed and reasonably low cost, then maybe it makes sense to apply bloom filters as an opportunistic or optimistic optimization. Perhaps they can be applied when there is little to lose but much to gain. [1] http://www.ccs.neu.edu/home/pete/pu

[HACKERS] ICU locales and text/char(n) SortSupport on Windows

2017-09-16 Thread Peter Geoghegan
is all that we need here (ICU builds must also be USE_WIDE_UPPER_LOWER builds). -- Peter Geoghegan From fca07aca51fb7979f19180712707233ff0e6f4b4 Mon Sep 17 00:00:00 2001 From: Peter Geoghegan <p...@bowt.ie> Date: Sat, 16 Sep 2017 13:36:25 -0700 Subject: [PATCH] Allow ICU to use SortSupport on Win

Re: [HACKERS] valgrind vs. shared typmod registry

2017-09-16 Thread Peter Geoghegan
t has indeed been failing within select_parallel only following the commit that you mentioned: https://buildfarm.postgresql.org/cgi-bin/show_history.pl?nm=skink=HEAD -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscr

Re: [HACKERS] A design for amcheck heapam verification

2017-09-16 Thread Peter Geoghegan
On Wed, Sep 6, 2017 at 7:26 PM, Peter Geoghegan <p...@bowt.ie> wrote: > On Wed, Aug 30, 2017 at 9:29 AM, Peter Geoghegan <p...@bowt.ie> wrote: >> On Wed, Aug 30, 2017 at 5:02 AM, Alvaro Herrera >> <alvhe...@2ndquadrant.com> wrote: >>> Eh, if you want to o

Re: [HACKERS] The case for removing replacement selection sort

2017-09-15 Thread Peter Geoghegan
/sort-bench-e5-5450 > > At this point the 10M row tests are running, but I don't expect anything > particularly surprising from those results. That is, it's not something > that should block getting this patch committed, if the agreement is to > commit otherwise. > > regards > > -- >

Re: [HACKERS] The case for removing replacement selection sort

2017-09-15 Thread Peter Geoghegan
sts, not reduced I/O costs. This is the opposite benefit to the one you'd expect from reading Knuth. Thanks for benchmarking! I hope that this removes the doubt that replacement selection previously benefited from; it now clearly deserves to be removed from tuplesort.c. -- Peter Geoghegan --

Re: [HACKERS] Trouble with amcheck

2017-09-14 Thread Peter Geoghegan
ng in the machine set up, but I'm stumped as to > what. I think you need to build and install contrib, so that it is available to the server that you're running an installcheck against. amcheck is alphabetically first among contrib modules that have tests, IIRC. -- Peter Geoghegan -- Sent

Re: [HACKERS] Automatic testing of patches in commit fest

2017-09-12 Thread Peter Geoghegan
t;screen scraping" techniques, which are obviously fairly brittle. Similarly, Thomas' patch testing web application should itself have a web API, potentially usable by the CF app. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to y

Re: [HACKERS] The case for removing replacement selection sort

2017-09-11 Thread Peter Geoghegan
On Wed, Sep 6, 2017 at 2:55 PM, Peter Geoghegan <p...@bowt.ie> wrote: > On Wed, Sep 6, 2017 at 2:47 PM, Thomas Munro > <thomas.mu...@enterprisedb.com> wrote: >>> I attach a patch to remove replacement selection, which I'll submit to CF 1. >> >> This breaks

Re: [HACKERS] CLUSTER command progress monitor

2017-09-11 Thread Peter Geoghegan
esort.c, but the actual sorting component isn't where the real costs are. Profiling shows that writing out the new heap (including moderately complicated bookkeeping) is the bottleneck, IIRC. That's why parallel CLUSTER didn't look attractive, even though it would be a fairly straightforward matter

Re: [HACKERS] The case for removing replacement selection sort

2017-09-11 Thread Peter Geoghegan
lity to exploit presortedness itself within commit 24598337c, and memory utilization for merging got better, too. Very roughly speaking, merging attained the same advantage that replacement selection had all along, and replacement selection lost all ability to compete. -- Peter Geoghegan -- Se

Re: [HACKERS] The case for removing replacement selection sort

2017-09-11 Thread Peter Geoghegan
stgres 10, it is much more likely to hurt than to help, because of the enhancements to merging that went into Postgres 10 reduced the downside of not using replacement selection. And so, for Postgres 11 replacement_sort_tuples deserves to be removed. -- Peter Geoghegan -- Sent via pgsql-hackers mai

Re: [HACKERS] The case for removing replacement selection sort

2017-09-11 Thread Peter Geoghegan
On Mon, Sep 11, 2017 at 8:32 AM, Robert Haas <robertmh...@gmail.com> wrote: > On Sun, Sep 10, 2017 at 9:39 PM, Peter Geoghegan <p...@bowt.ie> wrote: >> To be clear, you'll still need to set replacement_sort_tuples high >> when testing RS, to make sure that we

Re: [HACKERS] The case for removing replacement selection sort

2017-09-11 Thread Peter Geoghegan
rt.c, and the overall performance improvement is just a bonus IMV. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] The case for removing replacement selection sort

2017-09-10 Thread Peter Geoghegan
ot be due to the number of distinct tuples. But, if the extra time it takes doesn't matter to you, then it doesn't matter to me either. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] The case for removing replacement selection sort

2017-09-10 Thread Peter Geoghegan
th 10M > rows will take much more time (it takes 1-2hours for a single work_mem > value, and we're testing 6 of them). I myself don't see that much value in a 10M row test. Thanks for volunteering to test! -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hac

Re: [HACKERS] The case for removing replacement selection sort

2017-09-10 Thread Peter Geoghegan
oiding regressions when upgrading from versions prior to 9.6 (9.6 is the version where we began to generally prefer quicksort). > Also, people often sort on > keys of more than one column.... Very true. I should test this. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hac

Re: [HACKERS] The case for removing replacement selection sort

2017-09-08 Thread Peter Geoghegan
efault replacement_sort_tuples just isn't that many tuples (150,000). * The upside of my patch is not inconsiderable: We can remove a lot of code, which will enable further improvements in the future, which are far more compelling (cleaner memory management, the use of memory batches during run ge

Re: [HACKERS] A design for amcheck heapam verification

2017-09-06 Thread Peter Geoghegan
On Wed, Aug 30, 2017 at 9:29 AM, Peter Geoghegan <p...@bowt.ie> wrote: > On Wed, Aug 30, 2017 at 5:02 AM, Alvaro Herrera > <alvhe...@2ndquadrant.com> wrote: >> Eh, if you want to optimize it for the case where debug output is not >> enabled, make sure to use

Re: [HACKERS] The case for removing replacement selection sort

2017-09-06 Thread Peter Geoghegan
uc-replacement-sort-tuples"> but you removed that id. Thanks for looking into it. I suppose that the solution is to change the 9.6 release notes to no longer have a replacement_sort_tuples link. Anyone else have an opinion on that? -- Peter Geoghegan -- Sent via pgsql-hackers mailing

Re: [HACKERS] INSERT .. ON CONFLICT DO SELECT [FOR ..]

2017-09-04 Thread Peter Geoghegan
ntics that are, shall we say, questionable. That's the cost of having it not lock conflicting rows during big ETL operations. That's a huge practical benefit for ETL use-cases. Whereas here, with ON CONFLICT DO SELECT, I see a somewhat greater risk, and a much, much smaller benefit. A benefit that might actually

Re: [HACKERS] INSERT .. ON CONFLICT DO SELECT [FOR ..]

2017-09-03 Thread Peter Geoghegan
On Tue, Aug 15, 2017 at 12:17 AM, Marko Tiikkaja <ma...@joh.to> wrote: > On Tue, Aug 15, 2017 at 7:43 AM, Peter Geoghegan <p...@bowt.ie> wrote: >> >> On Mon, Aug 14, 2017 at 6:23 PM, Marko Tiikkaja <ma...@joh.to> wrote: >> > Attached is a patch for $SUBJ

Re: [HACKERS] The case for removing replacement selection sort

2017-08-31 Thread Peter Geoghegan
On Wed, Aug 30, 2017 at 4:59 PM, Peter Geoghegan <p...@bowt.ie> wrote: > I may submit the simple patch to remove replacement selection, if > other contributors are receptive. Apart from everything else, the > "incrementalism" of replacement selection works agains

Re: [HACKERS] The case for removing replacement selection sort

2017-08-30 Thread Peter Geoghegan
days. The picture changed. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] The case for removing replacement selection sort

2017-08-30 Thread Peter Geoghegan
On Wed, Aug 30, 2017 at 3:14 PM, Peter Geoghegan <p...@bowt.ie> wrote: > This is significantly faster, in a way that's clearly reproducible and > consistent, despite the fact that we need about 10 merge passes > without replacement selection, and only have enough memory for 7 &

Re: [HACKERS] The case for removing replacement selection sort

2017-08-30 Thread Peter Geoghegan
espite the fact that we need about 10 merge passes without replacement selection, and only have enough memory for 7 tapes. I think that I could find a case that makes replacement selection look much worse, if I tried. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgres

Re: [HACKERS] Polyphase merge is obsolete

2017-08-30 Thread Peter Geoghegan
one a number of things in this area, of which this is only the latest. I'm saying "hey, have you thought about RS too?". Whether or not I'm "hijacking" this thread is, at best, ambiguous. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To m

Re: [HACKERS] The case for removing replacement selection sort

2017-08-30 Thread Peter Geoghegan
On Wed, Aug 30, 2017 at 12:51 PM, Robert Haas <robertmh...@gmail.com> wrote: > On Fri, Jul 14, 2017 at 6:20 PM, Peter Geoghegan <p...@bowt.ie> wrote: >> With the additional enhancements made to Postgres 10, I doubt that >> there are any remaining cases where it wins

Re: [HACKERS] Polyphase merge is obsolete

2017-08-30 Thread Peter Geoghegan
On Mon, Feb 27, 2017 at 2:45 PM, Peter Geoghegan <p...@bowt.ie> wrote: > Since we have an awful lot of stuff in the last CF, and this patch > doesn't seem particularly strategic, I've marked it "Returned with > Feedback". I noticed that this is in the upcoming CF

Re: [HACKERS] A design for amcheck heapam verification

2017-08-30 Thread Peter Geoghegan
t. I should do that, but it's still not really noticeable. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] A design for amcheck heapam verification

2017-08-29 Thread Peter Geoghegan
lacks element. > > I think where "Bloom filter" appears in prose it should have a capital > letter (person's name). Agreed. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] A design for amcheck heapam verification

2017-08-29 Thread Peter Geoghegan
t; +intbitset_bytes = NBITS(filter) / BITS_PER_BYTE; > +int64bits_set = 0; > +inti; > + > +for (i = 0; i < bitset_bytes; i++) > +{ > +unsigned char byte = filter->bitset[i]; > + > +while (byte) > +{ >

Re: [HACKERS] A design for amcheck heapam verification

2017-08-29 Thread Peter Geoghegan
On Thu, May 11, 2017 at 4:30 PM, Peter Geoghegan <p...@bowt.ie> wrote: > I spent only a few hours writing a rough prototype, and came up with > something that does an IndexBuildHeapScan() scan following the > existing index verification steps. Its amcheck callback does an > index

Re: [HACKERS] pgbench: faster version of tpcb-like transaction

2017-08-26 Thread Peter Geoghegan
s > come and go. I must admit that I had a similar unpleasant surprise at one point -- "-M prepared" seems to matter *a lot* these days. That's the default that I'd change, if any. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)

Re: [HACKERS] pgbench: faster version of tpcb-like transaction

2017-08-26 Thread Peter Geoghegan
cb-like What about with "-M prepared"? I think that most of us use that setting already, especially with CPU-bound workloads. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Re: ICU collation variant keywords and pg_collation entries (Was: [BUGS] Crash report for some ICU-52 (debian8) COLLATE and work_mem values)

2017-08-25 Thread Peter Geoghegan
t Serbian as Serbian (sr-*), regardless of what country code may also appear, even if the country code is not just obsolete, but entirely bogus. Events like the dissolution of countries are rare enough that that extra assurance is just a nice-to-have, though. [1] https://en.wikipedia.org/wiki

Re: [HACKERS] Re: ICU collation variant keywords and pg_collation entries (Was: [BUGS] Crash report for some ICU-52 (debian8) COLLATE and work_mem values)

2017-08-21 Thread Peter Geoghegan
On Mon, Aug 21, 2017 at 4:48 PM, Peter Eisentraut <peter.eisentr...@2ndquadrant.com> wrote: > On 8/21/17 12:33, Peter Geoghegan wrote: >> On Mon, Aug 21, 2017 at 8:23 AM, Peter Eisentraut >> <peter.eisentr...@2ndquadrant.com> wrote: >>> Here are my patches to

Re: [HACKERS] Re: ICU collation variant keywords and pg_collation entries (Was: [BUGS] Crash report for some ICU-52 (debian8) COLLATE and work_mem values)

2017-08-21 Thread Peter Geoghegan
On Mon, Aug 21, 2017 at 9:33 AM, Peter Geoghegan <p...@bowt.ie> wrote: > On Mon, Aug 21, 2017 at 8:23 AM, Peter Eisentraut > <peter.eisentr...@2ndquadrant.com> wrote: >> Here are my patches to address this. > > These look good. Also, I don't know why en-u-kr-others-d

Re: [HACKERS] Re: ICU collation variant keywords and pg_collation entries (Was: [BUGS] Crash report for some ICU-52 (debian8) COLLATE and work_mem values)

2017-08-21 Thread Peter Geoghegan
ot;. Apparently, the behavior it implements is sometimes called natural sorting. See https://en.wikipedia.org/wiki/Natural_sort_order. Thanks -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Re: ICU collation variant keywords and pg_collation entries (Was: [BUGS] Crash report for some ICU-52 (debian8) COLLATE and work_mem values)

2017-08-19 Thread Peter Geoghegan
erable, and to explain the capabilities of custom ICU collations more generally. [1] https://postgr.es/m/f67f36d7-ceb6-cfbd-28d4-413c6d22f...@2ndquadrant.com [2] https://postgr.es/m/3862d484-f0a5-9eef-c54e-3f6808338...@2ndquadrant.com -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-h

Re: [HACKERS] Atomics for heap_parallelscan_nextpage()

2017-08-16 Thread Peter Geoghegan
) from a_star; >> >> Not sure if this is your bug or if it's exposing a pre-existing >> deficiency in the atomics code, viz, failure to ensure that >> pg_atomic_uint64 is actually a 64-bit-aligned type. Andres? > > I suspect it's the former. Suspect that the share

Re: [HACKERS] What users can do with custom ICU collations in Postgres 10

2017-08-15 Thread Peter Geoghegan
On Tue, Aug 15, 2017 at 11:33 AM, Peter Eisentraut <peter.eisentr...@2ndquadrant.com> wrote: > On 8/9/17 18:49, Peter Geoghegan wrote: >> I'd like to give a demo on what is already possible, but not currently >> documented. I didn't see anyone else comment on this, includin

Re: [HACKERS] What users can do with custom ICU collations in Postgres 10

2017-08-15 Thread Peter Geoghegan
uot;co" options to choose from, and I think that for the most part it's reasonably obvious which one is desirable. For example, Chinese people are probably well aware of what Pinyin is, and what stroke is. Things like EOR and search are much more esoteric, but also much less useful. So, I woul

Re: [HACKERS] INSERT .. ON CONFLICT DO SELECT [FOR ..]

2017-08-14 Thread Peter Geoghegan
FLICT DO SELECT to raise a cardinality violation error? Why or why not? -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] What users can do with custom ICU collations in Postgres 10

2017-08-14 Thread Peter Geoghegan
ariants. Users can get the "co" variants there. Should be for the most part obvious which one is interesting to which locale, since there is not that many "co" variants to choose from, and users will probably know what to look for if they look at all. -- Peter Geoghegan -- Sent

Re: [HACKERS] [BUGS] Replication to Postgres 10 on Windows is broken

2017-08-13 Thread Peter Geoghegan
On Sun, Aug 13, 2017 at 2:22 PM, Andres Freund <and...@anarazel.de> wrote: > On 2017-08-13 16:55:33 -0400, Tom Lane wrote: >> Peter Geoghegan <p...@bowt.ie> writes: >> > I think that it's useful for these things to be handled in an >> > adversarial

Re: [HACKERS] [BUGS] Replication to Postgres 10 on Windows is broken

2017-08-13 Thread Peter Geoghegan
wrong to disable the use of strxfrm() for abbreviated keys? I think that it's useful for these things to be handled in an adversarial manner, in the same way that litigation is adversarial in a common law court. I doubt that Noah actually set out to demoralize anyone. He is just doing the job he

Re: [HACKERS] Thoughts on unit testing?

2017-08-13 Thread Peter Geoghegan
e, this requires buy-in from patch authors to work. To put it another way, a patch author whose patch touches storage implicitly asserts that their patch is correct. It would be useful if they provided a precise falsifiable statement about its correctness up front, in the form of verification code. --

Re: [HACKERS] Getting server crash on Windows when using ICU collation

2017-08-10 Thread Peter Geoghegan
On Thu, Aug 10, 2017 at 3:57 PM, Peter Geoghegan <p...@bowt.ie> wrote: > On Thu, Aug 10, 2017 at 11:02 AM, Robert Haas <robertmh...@gmail.com> wrote: >> On Fri, Jun 23, 2017 at 1:14 AM, Ashutosh Sharma <ashu.coe...@gmail.com> >> wrote: >>> Okay,

Re: [HACKERS] Getting server crash on Windows when using ICU collation

2017-08-10 Thread Peter Geoghegan
prior to 53. He posted this just yesterday. This removes the existing configure test, replacing it with something that will work just the same on Windows, presuming Peter also reverts the commit that had ICU never use ucol_strcollUTF8() on Windows. -- Peter Geoghegan -- Sent via pgsql-hackers mailing l

Re: [HACKERS] How can I find a specific collation in pg_collation when using ICU?

2017-08-09 Thread Peter Geoghegan
variant options (e.g., traditional Spanish sort order, alternative Japanese sort order, pictographic emoji sorting), with some further generic options for varying how case it handled, how numbers are handled, and other things like that. -- Peter Geoghegan -- Sent via pgsql-hackers mailin

[HACKERS] What users can do with custom ICU collations in Postgres 10

2017-08-09 Thread Peter Geoghegan
rs that are used in certain countries [3], with fixed number/letter fields. These options are very powerful. [1] http://unicode.org/reports/tr35/tr35-collation.html#Setting_Options [2] http://unicode.org/reports/tr35/tr35-collation.html#Common_Settings [3] https://en.wikipedia.org/wiki/Vehi

Re: [HACKERS] How can I find a specific collation in pg_collation when using ICU?

2017-08-09 Thread Peter Geoghegan
JIS X 4061 for Japanese, unlike Glibc. This will give more useful results when sorting Japanese. The best explanation of the difference that I can understand is here, under "Why do CJK strings sort incorrectly in Unicode?": https://dev.mysql.com/doc/refman/5.5/en/faqs-cjk.html -- Pe

Re: [HACKERS] Possible issue with expanded object infrastructure on Postgres 9.6.1

2017-08-08 Thread Peter Geoghegan
On Thu, Jan 19, 2017 at 5:45 PM, Peter Geoghegan <p...@heroku.com> wrote: > A customer is on 9.6.1, and complains of a segfault observed at least > 3 times. > I can use GDB to get details of the instruction pointer that appeared > in the kernel trap error, which shows a function

Re: [HACKERS] ICU collation variant keywords and pg_collation entries (Was: [BUGS] Crash report for some ICU-52 (debian8) COLLATE and work_mem values)

2017-08-07 Thread Peter Geoghegan
not throw an error during initdb import when passed strings like these (that are generated mechanically), why should we not do the same with CREATE COLLATION? While the choice to preserve BCP 47's tolerance of missing collations is debatable, not doing at least this much up-front is a bug IMV. --

Re: [HACKERS] ICU collation variant keywords and pg_collation entries (Was: [BUGS] Crash report for some ICU-52 (debian8) COLLATE and work_mem values)

2017-08-07 Thread Peter Geoghegan
On Mon, Aug 7, 2017 at 2:50 PM, Peter Eisentraut <peter.eisentr...@2ndquadrant.com> wrote: > On 8/6/17 20:07, Peter Geoghegan wrote: >> I've looked into this. I'll give an example of what keyword variants >> there are for Greek, and then discuss what I think each is. > &g

Re: [HACKERS] max_files_per_processes vs others uses of file descriptors

2017-08-07 Thread Peter Geoghegan
of slop for FDs. Is this specifically about postgres_fdw, or is there some other specific problem you have in mind, that this would help solve? -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org

[HACKERS] ICU collation variant keywords and pg_collation entries (Was: [BUGS] Crash report for some ICU-52 (debian8) COLLATE and work_mem values)

2017-08-06 Thread Peter Geoghegan
On Sun, Aug 6, 2017 at 1:06 PM, Peter Geoghegan <p...@bowt.ie> wrote: > On Sat, Aug 5, 2017 at 8:26 PM, Tom Lane <t...@sss.pgh.pa.us> wrote: >> I'm quite disturbed though that the set of installed collations on these >> two test cases seem to be entirely different bo

Re: [HACKERS] modeling parallel contention (was: Parallel Append implementation)

2017-08-05 Thread Peter Geoghegan
ge into a worker at once within the parallel heap scan that feeds workers. The leader, which merges worker runs, may ultimately have to perform fewer comparisons as a result of this, which is where most of the benefit would be. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsq

Re: LP_DEAD hinting and not holding on to a buffer pin on leaf page (Was: [HACKERS] [WIP] Zipfian distribution in pgbench)

2017-08-05 Thread Peter Geoghegan
On Fri, Aug 4, 2017 at 3:30 PM, Peter Geoghegan <p...@bowt.ie> wrote: > Yura Sokolov of Postgres Pro performed this benchmark at my request. > He took the 9.5 commit immediately proceeding 2ed5b87f9 as a baseline. I attach a simple patch that comments out the release of the buffer pi

Re: [HACKERS] [WIP] Zipfian distribution in pgbench

2017-08-04 Thread Peter Geoghegan
previous years. Just another idea, that doesn't have to hold this patch up. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: LP_DEAD hinting and not holding on to a buffer pin on leaf page (Was: [HACKERS] [WIP] Zipfian distribution in pgbench)

2017-08-04 Thread Peter Geoghegan
On Mon, Jul 31, 2017 at 10:54 AM, Peter Geoghegan <p...@bowt.ie> wrote: > Let's wait to see what difference it makes if Alik's zipfian > distribution pgbench test case uses unlogged tables. That may gives us a > good sense of the problem for cases with contention/concurrency

Re: [HACKERS] Small code improvement for btree

2017-08-04 Thread Peter Geoghegan
t especially concerned about which, as long as it's one of those two. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: LP_DEAD hinting and not holding on to a buffer pin on leaf page (Was: [HACKERS] [WIP] Zipfian distribution in pgbench)

2017-07-31 Thread Peter Geoghegan
Robert Haas <robertmh...@gmail.com> wrote: On Mon, Jul 31, 2017 at 1:54 PM, Peter Geoghegan <p...@bowt.ie> wrote: That is hard to justify. I don't think that failing to set LP_DEAD hints is the cost that must be paid to realize a benefit elsewhere, though. I don't see much problem

Re: LP_DEAD hinting and not holding on to a buffer pin on leaf page (Was: [HACKERS] [WIP] Zipfian distribution in pgbench)

2017-07-31 Thread Peter Geoghegan
Robert Haas <robertmh...@gmail.com> wrote: On Thu, Jul 27, 2017 at 10:05 PM, Peter Geoghegan <p...@bowt.ie> wrote: I really don't know if that would be worthwhile. It would certainly fix the regression shown in my test case, but that might not go far enough. I stro

<    1   2   3   4   5   6   7   8   9   10   >