Re: [HACKERS] Draft release notes for next week's releases are up for review
Tobias Bussmannwrites: > another typo taken over from commit log: > s/Tobias Bussman/Tobias Bussmann/ Will fix, thanks! regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Draft release notes for next week's releases are up for review
Am 04.02.2017 um 18:57 schrieb Tom Lane: > Right now the question is whether individual items are > correctly/adequately documented. > Allow statements prepared with PREPARE to be given parallel plans (Amit > Kapila, Tobias Bussman) another typo taken over from commit log: s/Tobias Bussman/Tobias Bussmann/ thanks Tobias -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Draft release notes for next week's releases are up for review
Amit Kapilawrites: > On Sat, Feb 4, 2017 at 11:27 PM, Tom Lane wrote: >> Fix possible miss of socket read events while waiting on Windows (Amit >> Kapial) > Typo > Amit Kapial/Amit Kapila Wups. Copied-and-pasted that from the commit log without stopping to question it. Will fix, thanks for noticing. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Draft release notes for next week's releases are up for review
On Sat, Feb 4, 2017 at 11:27 PM, Tom Lanewrote: > First-draft release notes are available at > https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=9863017b87f3592ff663d03fc663a4f1f8fdb8b2 > They should appear in a more readable form at > https://www.postgresql.org/docs/devel/static/release-9-6-2.html > after guaibasaurus' next buildfarm run, due a couple hours from now. > > As usual, I've filled in the frontmost branch's section with all the > material, even though a couple of items don't apply because they only > went into further-back branches. I'll sort that out when I split > things up. Right now the question is whether individual items are > correctly/adequately documented. > > Fix possible miss of socket read events while waiting on Windows (Amit Kapial) Typo Amit Kapial/Amit Kapila -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Draft release notes for next week's releases
On Fri, Oct 21, 2016 at 4:51 PM, Tom Lanewrote: > Please review ... Is somebody going to look at the bugfix for the issue where ON CONFLICT DO NOTHING is used at higher isolation levels [1]? I think that it's still possible to get it in. [1] https://www.postgresql.org/message-id/cam3swzr6an++h24e6y2nwetjmtcjxdbfgeifoss2jpfpemq...@mail.gmail.com -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Draft release notes for next week's releases
On Thu, Apr 14, 2016 at 4:42 PM, Peter Geogheganwrote: > * We should look into using the ucol_nextSortKeyPart() API: > > http://userguide.icu-project.org/collation/architecture#TOC-Partial-sort-keys Another more rich API we could immediately put to good use is the ICU strcoll() variant that does not require NUL-terminated strings: https://ssl.icu-project.org/apiref/icu4c/ucol_8h.html#a3abc6779e6452106415918199308fab4 We do not use a NUL byte for terminating text data, and so must copy its contents into a temp buffer, or array on the stack, all rather inefficiently. Robert has expressed an interest in an API like this strcoll() variant in the past [1], to avoid this unnecessary overhead. [1] http://rhaas.blogspot.com/2012/03/perils-of-collation-aware-comparisons.html -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Draft release notes for next week's releases
On Tue, Mar 29, 2016 at 5:18 AM, Teodor Sigaevwrote: > It's based on https://people.freebsd.org/~girgen/postgresql-icu/readme.html > work, and it was migrated to 9.5 with abbrevation keys support. > Patch in current state is not ready to commit, of course. Cool. Some quick observations on this: * We need to have a strxfrm_l_icu(), not just a strxfrm_icu(). That seems easy. * We should look into using the ucol_nextSortKeyPart() API: http://userguide.icu-project.org/collation/architecture#TOC-Partial-sort-keys I think that this could be a lot faster, because we only need a part of the collation tables in CPU cache during the generation of abbreviated keys. There is an optimization described at a low level here: https://github.com/icu-project/icu4c/blob/bbd17a792336de5873550794f8304a4b548b0663/source/i18n/collationkeys.cpp#L337 I think this could make our special strxfrm() (which only actually needs 8 bytes for abbreviated keys) a lot faster. I'd be interested to see how your Russian text example does with this extra optimization. We should not be surprised that this kind of support exists within ICU, because abbreviated keys are actually quite an old idea. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Draft release notes for next week's releases
On Thu, Mar 31, 2016 at 2:59 PM, Tom Lanewrote: > Well, too late for 9.5.2 anyway. It still makes sense to correct that > text for future releases. I'm inclined to wait a little bit though and > see what other improvements become apparent. For instance, I think the > point about non-first index columns not being affected is of greater > weight than you seem to place on it. The SQL query on the Wiki page does the right thing there now, so users will have the benefit of not unnecessarily reindexing when text was not the leading/first pg_index attribute. We have that covered, I suppose, because everyone will look to the Wiki page for guidance. I also noted quite a few non-obvious safe cases on the Wiki page, as pointed out already over on the other thread. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Draft release notes for next week's releases
Peter Geogheganwrites: > I just noticed that the release notes mention char(n) as affected. > That's not actually true, because char(n) SortSupport only came in > 9.6. The Wiki page now shows this, which may be the most important > place, but ideally we'd fix this in the release notes. I guess it's > too late. Well, too late for 9.5.2 anyway. It still makes sense to correct that text for future releases. I'm inclined to wait a little bit though and see what other improvements become apparent. For instance, I think the point about non-first index columns not being affected is of greater weight than you seem to place on it. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Draft release notes for next week's releases
On Sat, Mar 26, 2016 at 4:34 PM, Tom Lanewrote: > Probably the most discussion-worthy item is whether we can say > anything more about the strxfrm mess. Should we make a wiki > page about that and have the release note item link to it? I just noticed that the release notes mention char(n) as affected. That's not actually true, because char(n) SortSupport only came in 9.6. The Wiki page now shows this, which may be the most important place, but ideally we'd fix this in the release notes. I guess it's too late. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Draft release notes for next week's releases
Does include ICU mean that collation handling is identical across platforms? E.g. a query on Linux involving string comparison would yield the same result on MacOS and Windows? Yes, it does and that's the most important issue for us. Yes, exactly. Attached patch adds support for libicu with configure flag --with-icu. Patch rebased to current HEAD, hope, it works. It's based on https://people.freebsd.org/~girgen/postgresql-icu/readme.html work, and it was migrated to 9.5 with abbrevation keys support. Patch in current state is not ready to commit, of course. -- Teodor Sigaev E-mail: teo...@sigaev.ru WWW: http://www.sigaev.ru/ libicu-8.patch.gz Description: GNU Zip compressed data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Draft release notes for next week's releases
On Mon, Mar 28, 2016 at 6:08 PM, Robert Haaswrote: > On Mon, Mar 28, 2016 at 10:24 AM, Tom Lane wrote: > > I'm also not exactly convinced by your implicit assumption that ICU is > > bug-free. > > Noah spent some time looking at ICU back when he was EnterpriseDB, and > his conclusion was that ICU collations weren't stable across releases, > which is pretty much the same problem we're running into with glibc > collations. Now it might still be true that they have the equivalent > of strxfrm() and strcoll() and that those things behave consistently > with each other, and that would be very good. Everybody seems to > agree it's faster, and that's good, too. But I wonder what we do > about the fact that, as with glibc, an ICU upgrade involves a REINDEX > of every potentially affected index. It seems like ICU has some > facilities built into it that might be useful for detecting and > handling such situations, but I don't understand them well enough to > know whether they'd solve our versioning problems or how effectively > they would do so, and I think there are packaging issues that tie into > it, too. http://userguide.icu-project.org/design mentions building > with specific configure flags if you need to link with multiple server > versions, and I don't know what operating system packagers typically > do about that stuff. > > In any case, I agree that we'd be very unwise to think that ICU is > necessarily going to be bug-free without testing it carefully. > agree. In other thread I wrote: "Ideally, we should benchmarking all locales on all platforms for all kind indexes. But that's big project." > > -- > Robert Haas > EnterpriseDB: http://www.enterprisedb.com > The Enterprise PostgreSQL Company >
Re: [HACKERS] Draft release notes for next week's releases
On Mon, Mar 28, 2016 at 1:36 PM, Thomas Kellererwrote: > Oleg Bartunov-2 wrote > > But still, icu provides us abbreviated keys and collation stability, > > Does include ICU mean that collation handling is identical across > platforms? > E.g. a query on Linux involving string comparison would yield the same > result on MacOS and Windows? > Yes, it does and that's the most important issue for us. > > If that is the case I'm all for it. > > Currently the different behaviour in handling collation aware string > comparisons is a bug in my eyes from a user's perspective. I do understand > and can accept the technical reasons for that, but it still feels odd that > a > query yields different results (with identical data) just because it runs > on > a different platform. > > > > > -- > View this message in context: > http://postgresql.nabble.com/Draft-release-notes-for-next-week-s-releases-tp5895357p5895484.html > Sent from the PostgreSQL - hackers mailing list archive at Nabble.com. > > > -- > Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) > To make changes to your subscription: > http://www.postgresql.org/mailpref/pgsql-hackers >
Re: [HACKERS] Draft release notes for next week's releases
On Mon, Mar 28, 2016 at 10:24 AM, Tom Lanewrote: > I'm also not exactly convinced by your implicit assumption that ICU is > bug-free. Noah spent some time looking at ICU back when he was EnterpriseDB, and his conclusion was that ICU collations weren't stable across releases, which is pretty much the same problem we're running into with glibc collations. Now it might still be true that they have the equivalent of strxfrm() and strcoll() and that those things behave consistently with each other, and that would be very good. Everybody seems to agree it's faster, and that's good, too. But I wonder what we do about the fact that, as with glibc, an ICU upgrade involves a REINDEX of every potentially affected index. It seems like ICU has some facilities built into it that might be useful for detecting and handling such situations, but I don't understand them well enough to know whether they'd solve our versioning problems or how effectively they would do so, and I think there are packaging issues that tie into it, too. http://userguide.icu-project.org/design mentions building with specific configure flags if you need to link with multiple server versions, and I don't know what operating system packagers typically do about that stuff. In any case, I agree that we'd be very unwise to think that ICU is necessarily going to be bug-free without testing it carefully. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Draft release notes for next week's releases
Oleg Bartunovwrites: > Should we start thinking about ICU ? Isn't it still true that ICU fails to meet our minimum requirements? That would include (a) working with the full Unicode character range (not only UTF16) and (b) working with non-Unicode encodings. No doubt we could deal with (b) by inserting a conversion, but that would take a lot of shine off the performance numbers you mention. I'm also not exactly convinced by your implicit assumption that ICU is bug-free. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Draft release notes for next week's releases
Oleg Bartunov-2 wrote > But still, icu provides us abbreviated keys and collation stability, Does include ICU mean that collation handling is identical across platforms? E.g. a query on Linux involving string comparison would yield the same result on MacOS and Windows? If that is the case I'm all for it. Currently the different behaviour in handling collation aware string comparisons is a bug in my eyes from a user's perspective. I do understand and can accept the technical reasons for that, but it still feels odd that a query yields different results (with identical data) just because it runs on a different platform. -- View this message in context: http://postgresql.nabble.com/Draft-release-notes-for-next-week-s-releases-tp5895357p5895484.html Sent from the PostgreSQL - hackers mailing list archive at Nabble.com. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Draft release notes for next week's releases
On Mon, Mar 28, 2016 at 2:06 PM, Peter Geogheganwrote: > On Mon, Mar 28, 2016 at 12:55 AM, Oleg Bartunov > wrote: > > We'll post the patch. > > Cool. > > > Teodor made something to get abbreviated keys work as > > I remember. I should say, that 27x improvement I got on my macbook. I > will > > check on linux. > > I think that Linux will be much faster. The stxfrm() blob produced by > Mac OSX will have a horribly low concentration of entropy. For an 8 > byte Datum, you get only 2 distinguishing bytes. It's really, really > bad. Mac OSX probably makes very little use of strxfrm() in practice; > there are proprietary APIs that do something similar, but all using > UTF-16 only. > Yes, Linux is much-much faster, I see no difference in performance using latest icu 57_1. I tested on Ubuntu 14.4.04. But still, icu provides us abbreviated keys and collation stability, so let's add --with-icu. > > -- > Peter Geoghegan >
Re: [HACKERS] Draft release notes for next week's releases
On Mon, Mar 28, 2016 at 12:55 AM, Oleg Bartunovwrote: > We'll post the patch. Cool. > Teodor made something to get abbreviated keys work as > I remember. I should say, that 27x improvement I got on my macbook. I will > check on linux. I think that Linux will be much faster. The stxfrm() blob produced by Mac OSX will have a horribly low concentration of entropy. For an 8 byte Datum, you get only 2 distinguishing bytes. It's really, really bad. Mac OSX probably makes very little use of strxfrm() in practice; there are proprietary APIs that do something similar, but all using UTF-16 only. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Draft release notes for next week's releases
On Mon, Mar 28, 2016 at 1:21 PM, Peter Geogheganwrote: > On Mon, Mar 28, 2016 at 12:08 AM, Oleg Bartunov > wrote: > > Should we start thinking about ICU ? I compare Postgres with ICU and > without > > and found 27x improvement in btree index creation for russian strings. > This > > includes effect of abbreviated keys and ICU itself. Also, we'll get > system > > independent locale. > > I think we should. I want to develop a detailed proposal before > talking about it more, though, because the idea is controversial. > > Did you use the FreeBSD ports patch? Do you have your own patch that > you could share? > We'll post the patch. Teodor made something to get abbreviated keys work as I remember. I should say, that 27x improvement I got on my macbook. I will check on linux. > > I'm not surprised that ICU is so much faster, especially now that > UTF-8 is not a second class citizen (it's been possible to build ICU > to specialize all its routines to handle UTF-8 for years now). As you > may know, ICU supports partial sort keys, and sort key compression, > which may have also helped: > http://userguide.icu-project.org/collation/architecture > > > That page also describes how binary sort keys are versioned, which > allows them to be stored on disk. It says "A common example is the use > of keys to build indexes in databases". We'd be crazy to trust Glibc > strxfrm() to be stable *on disk*, but ICU already cares deeply about > the things we need to care about, because it's used by other database > systems like DB2, Firebird, and in some configurations SQLite [1]. > > Glibc strxfrm() is not great with codepoints from the Cyrillic > alphabet -- it seems to store 2 bytes per code-point in the primary > weight level. So ICU might also do better in your test case for that > reason. > Yes, I see on this page, that ICU is ~3 times faster for russian text. http://site.icu-project.org/charts/collation-icu4c48-glibc > > [1] > https://www.sqlite.org/src/artifact?ci=trunk=ext/icu/README.txt > -- > Peter Geoghegan >
Re: [HACKERS] Draft release notes for next week's releases
On Mon, Mar 28, 2016 at 12:08 AM, Oleg Bartunovwrote: > Should we start thinking about ICU ? I compare Postgres with ICU and without > and found 27x improvement in btree index creation for russian strings. This > includes effect of abbreviated keys and ICU itself. Also, we'll get system > independent locale. I think we should. I want to develop a detailed proposal before talking about it more, though, because the idea is controversial. Did you use the FreeBSD ports patch? Do you have your own patch that you could share? I'm not surprised that ICU is so much faster, especially now that UTF-8 is not a second class citizen (it's been possible to build ICU to specialize all its routines to handle UTF-8 for years now). As you may know, ICU supports partial sort keys, and sort key compression, which may have also helped: http://userguide.icu-project.org/collation/architecture That page also describes how binary sort keys are versioned, which allows them to be stored on disk. It says "A common example is the use of keys to build indexes in databases". We'd be crazy to trust Glibc strxfrm() to be stable *on disk*, but ICU already cares deeply about the things we need to care about, because it's used by other database systems like DB2, Firebird, and in some configurations SQLite [1]. Glibc strxfrm() is not great with codepoints from the Cyrillic alphabet -- it seems to store 2 bytes per code-point in the primary weight level. So ICU might also do better in your test case for that reason. [1] https://www.sqlite.org/src/artifact?ci=trunk=ext/icu/README.txt -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Draft release notes for next week's releases
On Mar 28, 2016 09:44, "Peter Geoghegan"wrote: > > On Sat, Mar 26, 2016 at 4:34 PM, Tom Lane wrote: > > Probably the most discussion-worthy item is whether we can say > > anything more about the strxfrm mess. Should we make a wiki > > page about that and have the release note item link to it? > > I think that there is an argument against doing so, which is that > right now, all we have to offer on that are weasel words. However, I'm > still in favor of a Wiki page, because I would not be at all surprised > if our understanding of this problem evolved, and we were able to > offer better answers in several weeks. Realistically, it will probably > take at least that long before affected users even start to think > about this. Should we start thinking about ICU ? I compare Postgres with ICU and without and found 27x improvement in btree index creation for russian strings. This includes effect of abbreviated keys and ICU itself. Also, we'll get system independent locale. > > > -- > Peter Geoghegan > > > -- > Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) > To make changes to your subscription: > http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Draft release notes for next week's releases
On Sun, Mar 27, 2016 at 8:43 PM, Peter Geogheganwrote: > On Sat, Mar 26, 2016 at 4:34 PM, Tom Lane wrote: > > Probably the most discussion-worthy item is whether we can say > > anything more about the strxfrm mess. Should we make a wiki > > page about that and have the release note item link to it? > > I think that there is an argument against doing so, which is that > right now, all we have to offer on that are weasel words. However, I'm > still in favor of a Wiki page, because I would not be at all surprised > if our understanding of this problem evolved, and we were able to > offer better answers in several weeks. Realistically, it will probably > take at least that long before affected users even start to think > about this. > One question to debate is whether placing a list of "known" (collated from the program runs lots of people performed) would do more harm than good. Personally I'd rather see a list of known failures and evaluate my situation objectively (i.e., large index but no reported problem on my combination of locale and platform). I understand that a lack of evidence is not proof that I am unaffected at this stage in the game. Having something I can execute on my server to try and verify behavior - irrespective of the correctness of the indexes themselves - would be welcomed. David J.
Re: [HACKERS] Draft release notes for next week's releases
On Sat, Mar 26, 2016 at 4:34 PM, Tom Lanewrote: > Probably the most discussion-worthy item is whether we can say > anything more about the strxfrm mess. Should we make a wiki > page about that and have the release note item link to it? I think that there is an argument against doing so, which is that right now, all we have to offer on that are weasel words. However, I'm still in favor of a Wiki page, because I would not be at all surprised if our understanding of this problem evolved, and we were able to offer better answers in several weeks. Realistically, it will probably take at least that long before affected users even start to think about this. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Draft release notes for next week's releases
Jeff Janeswrites: > On Sat, Mar 26, 2016 at 4:34 PM, Tom Lane wrote: > + Correctly handle wraparound cases in the pg_subtrans > + startup logic for hot standby (Jeff Janes) > This applies to all recovery scenarios, whether they are hot standby > or just plain-old automatic crash recovery. (However, it does only > matter when prepared transactions are in use.) Thanks for the clarification, will fix! regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Draft release notes for next week's releases
On Sat, Mar 26, 2016 at 4:34 PM, Tom Lanewrote: > I've prepared a first cut at next week's release notes: > > http://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=29b6123ecb4113e366325245cec5a5c221dae691 > > (As usual, I will make the notes for older branches by extracting > relevant items from this list, after it's been reviewed.) Please > review. If you prefer to read it on the web, it should be up at > > http://www.postgresql.org/docs/devel/static/release-9-5-2.html > > in an hour or so, after guaibasaurus's next buildfarm run. > > Probably the most discussion-worthy item is whether we can say > anything more about the strxfrm mess. Should we make a wiki > page about that and have the release note item link to it? > > regards, tom lane Sorry for speaking up late, but: + + + Correctly handle wraparound cases in the pg_subtrans + startup logic for hot standby (Jeff Janes) + + This applies to all recovery scenarios, whether they are hot standby or just plain-old automatic crash recovery. (However, it does only matter when prepared transactions are in use.) Cheers, Jeff -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers