Re: First draft of PG 17 release notes
On Thu, May 23, 2024 at 11:11:10PM -0400, Tom Lane wrote: > Bruce Momjian writes: > > On Thu, May 23, 2024 at 02:27:07PM +1200, David Rowley wrote: > >> I also don't agree these should be left to "Source code" section. I > >> feel that section is best suited for extension authors who might care > >> about some internal API change. I'm talking of stuff that makes some > >> user queries possibly N times (!) faster. Surely "Source Code" isn't > >> the place to talk about that? > > > I said a new section "after our 'Source code' section," not in the > > source code section. > > Surely, if we make this a separate section, it would come before > 'Source code'? > > I am not sure Bruce that you realize that your disregard for > performance improvements is shared by nobody. Arguably, > performance is 90% of what we do these days, and it's also > 90% of what users care about. Please stop saying I don't document performance. I have already explained enough which performance items I choose. Please address my criteria or suggest new criteria. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Thu, May 23, 2024 at 02:27:07PM +1200, David Rowley wrote: > On Thu, 23 May 2024 at 14:01, Bruce Momjian wrote: > > > > On Thu, May 23, 2024 at 01:34:10PM +1200, David Rowley wrote: > > > What is the best way to communicate this stuff so it's easily > > > identifiable when you parse the commit messages? > > > > This is why I think we need an "Internal Performance" section where we > > can be clear about simple scaling improvements that might have no > > user-visible explanation. I would suggest putting it after our "Source > > code" section. > > hmm, but that does not really answer my question. There's still a > communication barrier if you're parsing the commit messages are > committers don't clearly state some performance improvement numbers > for the reason I stated. For a case where O(N^2) become O(N), we might not even know the performance change since it is a micro-optimization. That is why I suggested we call it "Internal Performance". > I also don't agree these should be left to "Source code" section. I > feel that section is best suited for extension authors who might care > about some internal API change. I'm talking of stuff that makes some > user queries possibly N times (!) faster. Surely "Source Code" isn't > the place to talk about that? I said a new section "after our 'Source code' section," not in the source code section. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Thu, May 23, 2024 at 01:34:10PM +1200, David Rowley wrote: > On Thu, 23 May 2024 at 10:04, Bruce Momjian wrote: > > You might have seen in this thread, I do record commits that speed up > > workloads that are user-visible, or specifically make new workloads > > possible. I assume that covers the items above, though I have to > > determine this from the commit message. > > It sometimes is hard to write something specific in the commit message > about the actual performance increase. > > For example, if a commit removes an O(N log2 N) algorithm and replaces > it with an O(1), you can't say there's an X% increase in performance > as the performance increase depends on the value of N. > > Jelte did call me out for not mentioning enough detail about the > performance in c4ab7da60, but if I claimed any % of an increase, it > would have been specific to some workload. > > What is the best way to communicate this stuff so it's easily > identifiable when you parse the commit messages? This is why I think we need an "Internal Performance" section where we can be clear about simple scaling improvements that might have no user-visible explanation. I would suggest putting it after our "Source code" section. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Wed, May 22, 2024 at 09:25:41PM +0900, torikoshia wrote:
> Thanks for working on this as always.
>
>
> Allow pg_stat_reset_shared("slru") to clear SLRU statistics (Atsushi
> Torikoshi)
>
>
> Considering someone may copy and paste this, 'slru' is better than "slru",
> isn't it?
> I also found an older release note[1] used single quotes for this like:
>
> Add pg_stat_reset_shared('bgwriter') to reset the cluster-wide shared
> statistics for the background writer (Greg Smith)
Agreed, patch applied, thanks.
--
Bruce Momjian https://momjian.us
EDB https://enterprisedb.com
Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Wed, May 22, 2024 at 11:29:06AM +0900, Masahiko Sawada wrote: > I found a typo: > > s/pg_statstatement/pg_stat_statement/ > > I've attached a patch to fix it. Agreed, applied, thanks. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Tue, May 21, 2024 at 11:20:02AM +0900, Amit Langote wrote: > Thanks Bruce for working on this as always. > > Failed to notice when I read the notes before: > > > > Add SQL/JSON constructor functions JSON(), JSON_SCALAR(), and > JSON_SERIALIZE() (Amit Langote) > > > > Should be: > > > > Add SQL/JSON constructor functions JSON(), JSON_SCALAR(), and > JSON_SERIALIZE() (Nikita Glukhov, Teodor Sigaev, Oleg Bartunov, > Alexander Korotkov, Andrew Dunstan, Amit Langote) > > Thanks, applied. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Mon, May 20, 2024 at 11:48:09AM -0700, Jeff Davis wrote: > On Sat, 2024-05-18 at 17:51 -0400, Bruce Momjian wrote: > > Okay, I went with the attached applied patch. Adjustments? > > I think it should have more emphasis on the actual new feature: a > platform-independent builtin collation provider and the C.UTF-8 locale. > > The user can look at the documentation for comparison with libc. Okay, changed with the attached applied patch. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you. diff --git a/doc/src/sgml/release-17.sgml b/doc/src/sgml/release-17.sgml index 3025f31f8a0..e381dbe5d96 100644 --- a/doc/src/sgml/release-17.sgml +++ b/doc/src/sgml/release-17.sgml @@ -885,11 +885,11 @@ Author: Jeff Davis -Create "builtin" collation provider similar to libc's C locale (Jeff Davis) +Add a builtin platform-independent collation provider (Jeff Davis) -While its C locale is similar but independent of libc, its C.UTF-8 locale sorts by Unicode code points and has Unicode-based case folding. +This supports "C" and "C.UTF-8" collations.
Re: First draft of PG 17 release notes
On Tue, May 21, 2024 at 09:40:28AM -0700, Andres Freund wrote:
> Hi,
>
> On 2024-05-18 11:13:54 -0400, Bruce Momjian wrote:
> > Please see the email I just posted. There are three goals we have to
> > adjust for:
> >
> > 1. short release notes so they are readable
> > 2. giving people credit for performance improvements
> > 3. showing people Postgres cares about performance
> >
> > I would like to achieve 2 & 3 without harming #1. My experience is if I
> > am reading a long document, and I get to a section where I start to
> > wonder, "Why should I care about this?", I start to skim the rest of
> > the document.
>
> I agree keeping things reasonably short is important. But I don't think you're
> evenly applying it as a goal.
>
> Just skimming the notes from the end, I see
> - an 8 entries long pg_stat_statements section
What item did you want to remove? Those are all user-visible changes.
> - multiple entries about "Create custom wait events for ..."
Well, those are all in different sections, so how can they be merged,
unless I create a "wait event section", I guess.
> - three entries about adding --all to {reindexdb,vacuumdb,clusterdb}.
The problem with merging these is that the "Specifically, --all can now
be used with" is different for all three of them.
> - an entry about adding long options to pg_archivecleanup
Well, that is a user-visible change. Should it not be listed?
> - two entries about grantable maintenance rights, once via pg_maintain, once
> per-table
Well, one is a GRANT and another is a role, so merging them seemed like
it would be too confusing.
> - separate entries about pg_stat_reset_slru(), pg_stat_reset_shared("slru"),
They are different functions with different detail text.
--
Bruce Momjian https://momjian.us
EDB https://enterprisedb.com
Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Tue, May 21, 2024 at 02:26:15PM -0400, Melanie Plageman wrote: > In Postgres development, we break larger projects into smaller ones > and then those smaller projects into multiple individual commits. Each > commit needs to stand alone and each subproject needs to have a > defensible benefit. One thing that is harder with performance-related > work than non-performance feature work is that there isn't always a > final "turn it on" commit. For example, let's say you are adding a new > view that tracks new stats of some kind. You do a bunch of refactoring > and small subprojects to make it possible to add the view. Then the > final commit that actually creates the view has obvious user value to > whoever is reading the log. For performance features, it doesn't > always work like this. > > For the vacuum WAL volume reduction, there were a bunch of smaller > projects throughout the last development year that I worked on that > were committed by different people and with different individual > benefits. Some changes caused vacuum to do less visibility checks (so > less CPU usage), some changed WAL format in a way that saves some > space, and some, like the commit you mention, make vacuum emit less > WAL. That commit by itself doesn't contain all of the user benefits of > the whole project. I couldn't think of a good place to list all of the > commits together that were part of the same project. Perhaps you could > argue that they were not in fact part of the same project and instead > were just small individual changes -- none of which are individually > worth including in the release notes. I try and group them, but I am sure imperfectly. It is very true that infrastucture changes that enable later commits are often missed. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Tue, May 21, 2024 at 01:50:58PM -0400, Robert Haas wrote:
> On Tue, May 21, 2024 at 12:27 PM Andres Freund wrote:
> > To me that's the "General Performance" section. If somebody reading the
> > release notes doesn't care about performance, they can just skip that
> > section
> > ([1]). I don't see why we wouldn't want to include the same level of detail
> > as for other changes.
>
> I'm relatively sure that we've had this argument in previous years and
> essentially everyone but Bruce has agreed with the idea that
> performance changes ought to be treated the same as any other kind of
> improvement. The difficulty is that Bruce is the one doing the release
> notes. I think it might help if someone were willing to prepare a
> patch showing what they think specifically should be changed. Or maybe
> Bruce would be willing to provide a list of all of the performance
> improvements he doesn't think are worth release-noting or isn't sure
> how to release-note, and someone else can then have a go at them.
Well, developers do ask why their individual commits were not listed,
and I repeat the same thing, as I have done in this thread multiple
times. You can probably look over this thread to find them, or in
previous years.
To be clear, it is performance improvements that don't have user-visible
changes or enable new workloads that I skip listing. I will also note
that don't remember any user ever finding a performance boost, and then
coming to use and asking why we didn't list it --- this release note
review process seems to add all of those.
Maybe adding a section called "Internal Performance". Here is our
General Performance contents:
E.1.3.1.3. General Performance
Allow vacuum to more efficiently remove and freeze tuples (Melanie
Plageman)
WAL traffic caused by vacuum is also more compact.
Allow vacuum to more efficiently store tuple references (Masahiko
Sawada, John Naylor)
Additionally, vacuum is no longer silently limited to one gigabyte
of memory when maintenance_work_mem or autovacuum_work_mem are higher.
Optimize vacuuming of relations with no indexes (Melanie Plageman)
Increase default vacuum_buffer_usage_limit to 2MB (Thomas Munro)
Improve performance when checking roles with many memberships
(Nathan Bossart)
Improve performance of heavily-contended WAL writes (Bharath
Rupireddy)
Improve performance when transferring large blocks of data to a
client (Melih Mutlu)
Allow the grouping of file system reads with the new system variable
io_combine_limit (Thomas Munro, Andres Freund, Melanie Plageman, Nazir
Bilal Yavuz)
Do we think more user-invisible changes should be in there?
--
Bruce Momjian https://momjian.us
EDB https://enterprisedb.com
Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Tue, May 21, 2024 at 09:51:09AM -0700, Andres Freund wrote: > Hi, > > On 2024-05-21 09:27:20 -0700, Andres Freund wrote: > > Also, the release notes are also not just important to users. I often go > > back > > and look in the release notes to see when some some important change was > > made, > > because sometimes it's harder to find it in the git log, due to sheer > > volume. And even just keeping up with all the changes between two releases > > is > > hard, it's useful to be able to read the release notes and see what > > happened. > > > > [...] > > > > [1] I've wondered if we should have one more level of TOC on the release > > note > > page, so it's easier to jump to specific sections. > > Which reminds me: Eventually I'd like to add links to the most important > commits related to release note entries. We already do much of the work of > building that list of commits for each entry. That'd allow a reader to find > more details if interested. Yes, it would be cool if they could mouse-over a graphic next to each release note item to get a popup to the commits. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Tue, May 21, 2024 at 09:27:20AM -0700, Andres Freund wrote: > On 2024-05-18 10:59:47 -0400, Bruce Momjian wrote: > > I agree the impact of performance improvements are often greater than > > the average release note item. However, if people expect Postgres to be > > faster, is it important for them to know _why_ it is faster? > > Yes, it very often is. Performance improvements typically aren't "make > everything 3% faster", they're more "make this special thing 20% > faster". Without know what got faster, users don't know if > a) the upgrade will improve their production situation > b) they need to change something to take advantage of the improvement You might have seen in this thread, I do record commits that speed up workloads that are user-visible, or specifically make new workloads possible. I assume that covers the items above, though I have to determine this from the commit message. > > On the flip side, a performance improvement that makes everything 10% > > faster has little behavioral change for users, and in fact I think we > > get ~6% faster in every major release. > > I cannot recall many "make everything faster" improvements, if any. > > And even if it's "make everything faster" - that's useful for users to know, > it might solve their production problem! It's also good for PR. Again, it is down to having three goals for the release notes, and #1 is having it readable/short, and 2 & 3 are for PR and acknowledging authors. > Also, the release notes are also not just important to users. I often go back > and look in the release notes to see when some some important change was made, > because sometimes it's harder to find it in the git log, due to sheer > volume. And even just keeping up with all the changes between two releases is > hard, it's useful to be able to read the release notes and see what happened. Well, I would say we need some _other_ way to record and perhaps advertise such changes. > > > For another, it's also very frustrating for developers that focus on > > > performance. The reticence to note their work, while noting other, far > > > smaller, things in the release notes, pretty much tells us that our work > > > isn't > > > valued. > > > > Yes, but are we willing to add text that every user will have to read > > just for this purpose? > > Of course it's a tradeoff. We shouldn't verbosely note down every small > changes just because of the recognition, that'd make the release notes > unreadable. And it'd just duplicate the commit log. But that's not the same as > defaulting to not noting performance improvements, even if the performance > improvement is more impactful than many other features that are noted. Again, see above, I do mention performance improvements, but they have to be user-visible or enable new workloads. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Mon, May 20, 2024 at 02:47:28PM -0400, Melanie Plageman wrote: > On Mon, May 20, 2024 at 9:37 AM Bruce Momjian wrote: > > > > On Mon, May 20, 2024 at 01:23:02PM +0700, John Naylor wrote: > > > Hi Bruce, thanks for doing this again! > > > > > > I'm a bit late to this discussion -- there's been a bit of churn in > > > the vacuum items, and some streams got crossed along the way. I've > > > attached an attempt to rectify this. > > > > Agreed, patch applied, thanks. > > If "Allow vacuum to more efficiently remove and freeze tuples" stays > in the release notes, I would add Heikki's name. He wasn't listed as a > co-author on all of the commits that were part of this feature, but he > made a substantial investment in it and should be listed, IMO. Patch > attached. Thanks, patch applied. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Mon, May 20, 2024 at 02:35:37PM -0400, Melanie Plageman wrote: > On Sat, May 18, 2024 at 11:13 AM Bruce Momjian wrote: > > Please see the email I just posted. There are three goals we have to > > adjust for: > > > > 1. short release notes so they are readable > > 2. giving people credit for performance improvements > > 3. showing people Postgres cares about performance > > I agree with all three of these goals. I would even add to 3 "show > users Postgres is addressing their performance complaints". That in > particular makes me less excited about having a "generic performance > release note item saying performance has been improved in the > following areas" (from your other email). I think that describing the > specific performance improvements is required to 1) allow users to > change expectations and configurations to take advantage of the > performance enhancements 2) ensure users know that their performance > concerns are being addressed. Well, as you can see, doing #2 & #3 works against accomplishing #1. > > I would like to achieve 2 & 3 without harming #1. My experience is if I > > am reading a long document, and I get to a section where I start to > > wonder, "Why should I care about this?", I start to skim the rest of > > the document. I am particularly critical if I start to wonder, "Why > > does the author _think_ I should care about this?" becasue it feels like > > the author is writing for him/herself and not the audience. > > I see what you are saying. We don't want to just end up with the whole > git log in the release notes. However, we know that not all users will > care about the same features. As someone said somewhere else in this > thread, presumably hackers spend time on features because some users > want them. I think we need as a separate section about performance improvements that don't affect specific workloads. Peter Eisentraut created an Acknowledgements section at the bottom of the release notes, similar to #2 above, so maybe someone else can add a performance internals section too. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Mon, May 20, 2024 at 01:23:02PM +0700, John Naylor wrote: > Hi Bruce, thanks for doing this again! > > I'm a bit late to this discussion -- there's been a bit of churn in > the vacuum items, and some streams got crossed along the way. I've > attached an attempt to rectify this. Agreed, patch applied, thanks. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: commitfest.postgresql.org is no longer fit for purpose
On Sun, May 19, 2024 at 03:18:11PM -0400, Tom Lane wrote: > * Another reason for things sitting a long time is that they're too > big to review without an unreasonable amount of effort. We should > encourage authors to break large patches into smaller stepwise > refinements. It may seem like that will result in taking forever > to reach the end goal, but dropping a huge patchset on the community > isn't going to give speedy results either. I think it is sometimes hard to incrementally apply patches if the long-term goal isn't agreed or know to be achievable. > * Before starting this thread, Robert did a lot of very valuable > review of some individual patches. I think what prompted him to > start the thread was the realization that he'd only made a small > dent in the problem. Maybe we could divide and conquer: get a > dozen-or-so senior contributors to split up the list of pending > patches and then look at each one with an eye to what needs to > happen to move it along (*not* to commit it right away, although > in some cases maybe that's the thing to do). It'd be great if > that could happen just before each commitfest, but that's probably > not practical with the current patch volume. What I'm thinking > for the moment is to try to make that happen once a year or so. For me, if someone already knows what the blocker is, it saves me a lot of time if they can state that somewhere. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Sun, May 19, 2024 at 03:53:38PM +1200, David Rowley wrote: > On Sun, 19 May 2024 at 02:40, Bruce Momjian wrote: > > > > On Thu, May 16, 2024 at 03:35:17PM +1200, David Rowley wrote: > > > "Additionally, vacuum no longer silently imposes a 1GB tuple reference > > > limit even when maintenance_work_mem or autovacuum_work_mem are set to > > > higher values" > > > Slightly adjusted wording patch attached and applied. > > Thanks for adjusting. > > It's a minor detail, but I'll mention it because you went to the > effort to adjust it away from what I'd written... True. > I didn't make a random choice to use "or" between the two GUCs. > Changing it to "and", IMO, isn't an improvement. Using "and" implies > that the silent limited was only imposed when both of these GUCs were > set >= 1GB. That's not true. For the case we're talking about here, if > autovacuum_work_mem is set to anything apart from -1 then the value of > maintenance_work_mem does not matter. Okay, changed to "or". -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: commitfest.postgresql.org is no longer fit for purpose
On Fri, May 17, 2024 at 11:44:40AM -0400, Melanie Plageman wrote: > So, anyway, I'd argue that we need a parking lot for patches which we > all agree are important and have a path forward but need someone to do > the last 20-80% of the work. To avoid this being a dumping ground, > patches should _only_ be allowed in the parking lot if they have a > clear path forward. Patches which haven't gotten any interest don't go > there. Patches in which the author has clearly not addressed feedback > that is reasonable for them to address don't go there. These are > effectively community TODOs which we agree need to be done -- if only > someone had the time. When I am looking to commit something, I have to consider: * do we want the change * are there concerns * are the docs good * does it need tests * is it only a proof-of-concept When people review commit fest entries, they have to figure out what is holding the patch back from being complete, so they have to read the thread from the beginning. Should there be a clearer way in the commit fest app to specify what is missing? -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Fri, May 17, 2024 at 01:30:03PM -0700, Jeff Davis wrote: > On Thu, 2024-05-09 at 00:03 -0400, Bruce Momjian wrote: > > I have committed the first draft of the PG 17 release notes; you can > > see the results here: > > > > https://momjian.us/pgsql_docs/release-17.html > > For this item: > > Create a "builtin" collation provider similar to libc's C > locale (Jeff Davis) > > It uses a "C" locale which is identical but independent of > libc, but it allows the use of non-"C" collations like "en_US" > and "C.UTF-8" with the "C" locale, which libc does not. MORE? > > I suggest something more like: > > New, platform-independent "builtin" collation > provider. (Jeff Davis) > > Currently, it offers the "C" and "C.UTF-8" locales. The > "C.UTF-8" locale combines stable and fast code point order > collation with Unicode character semantics. Okay, I went with the attached applied patch. Adjustments? -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you. diff --git a/doc/src/sgml/release-17.sgml b/doc/src/sgml/release-17.sgml index 0bc1c9a14ad..5d2fcd7d7de 100644 --- a/doc/src/sgml/release-17.sgml +++ b/doc/src/sgml/release-17.sgml @@ -886,11 +886,11 @@ Author: Jeff Davis -Create a "builtin" collation provider similar to libc's C locale (Jeff Davis) +Create "builtin" collation provider similar to libc's C locale (Jeff Davis) -It uses a "C" locale which is identical but independent of libc, but it allows the use of non-"C" collations like "en_US" and "C.UTF-8" with the "C" locale, which libc does not. MORE? +While its C locale is similar but independent of libc, its C.UTF-8 locale sorts by Unicode code points and has Unicode-based case folding.
Re: First draft of PG 17 release notes
On Fri, May 17, 2024 at 09:22:59PM +0800, jian he wrote: > On Thu, May 9, 2024 at 12:04 PM Bruce Momjian wrote: > > > > I have committed the first draft of the PG 17 release notes; you can > > see the results here: > > > > https://momjian.us/pgsql_docs/release-17.html > > > > It will be improved until the final release. The item count is 188, > > which is similar to recent releases: > > > > This thread mentioned performance. > actually this[1] refactored some interval aggregation related functions, > which will make these two aggregate function: avg(interval), sum(interval) > run faster, especially avg(interval). > see [2]. > well, I guess, this is a kind of niche performance improvement to be > mentioned separately. > > > these 3 items need to be removed, because of > https://git.postgresql.org/cgit/postgresql.git/commit/?id=8aee330af55d8a759b2b73f5a771d9d34a7b887f > > >> Add stratnum GiST support function (Paul A. Jungwirth) > > >> Allow foreign keys to reference WITHOUT OVERLAPS primary keys (Paul A. > >> Jungwirth) > >> The keyword PERIOD is used for this purpose. > > >> Allow PRIMARY KEY and UNIQUE constraints to use WITHOUT OVERLAPS for > >> non-overlapping exclusion constraints (Paul A. Jungwirth) > > > [1] > https://git.postgresql.org/cgit/postgresql.git/commit/?id=519fc1bd9e9d7b408903e44f55f83f6db30742b7 > [2] > https://www.postgresql.org/message-id/CAEZATCUJ0xjyQUL7SHKxJ5a%2BDm5pjoq-WO3NtkDLi6c76rh58w%40mail.gmail.com Agreed, I have applied the attached patch to make the release notes current. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you. diff --git a/doc/src/sgml/release-17.sgml b/doc/src/sgml/release-17.sgml index 9c511848943..0bc1c9a14ad 100644 --- a/doc/src/sgml/release-17.sgml +++ b/doc/src/sgml/release-17.sgml @@ -6,7 +6,7 @@ Release date: - 2024-??-??, AS OF 2024-05-14 + 2024-??-??, AS OF 2024-05-18 @@ -480,17 +480,6 @@ Author: Tomas Vondra Allow BRIN indexes to be created using parallel workers (Tomas Vondra, Matthias van de Meent) - - - - - - -Add stratnum GiST support function (Paul A. Jungwirth) - @@ -1467,34 +1456,6 @@ Add DEFAULT setting for ALTER TABLE .. SET ACCESS METHOD (Michael Paquier) - - - - -Allow foreign keys to reference WITHOUT OVERLAPS primary keys (Paul A. Jungwirth) - - - -The keyword PERIOD is used for this purpose. - - - - - - - -Allow PRIMARY KEY and UNIQUE constraints to use WITHOUT OVERLAPS for non-overlapping exclusion constraints (Paul A. Jungwirth) - - -
Re: First draft of PG 17 release notes
On Thu, May 16, 2024 at 11:50:20AM -0400, Andrew Dunstan wrote: > > Maybe "Introduce an incremental JSON parser" would have been a better > > > headline. > > Well, this gets into a level of detail that is beyond the average > > reader. I think at that level people will need to read the git logs or > > review the code. Do we use it for anything yet? > > > Yes, certainly, it's used in handling backup manifests. Without it we can't > handle huge manifests. See commits ea7b4e9a2a and 222e11a10a. > > Other uses are in the works. Okay, added in the attached applied patch. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you. diff --git a/doc/src/sgml/release-17.sgml b/doc/src/sgml/release-17.sgml index 428cb5c5a2e..9c511848943 100644 --- a/doc/src/sgml/release-17.sgml +++ b/doc/src/sgml/release-17.sgml @@ -2451,6 +2451,17 @@ User-defined data type receive functions will no longer receive their data null- + + + + +Add incremental JSON parser for use with huge JSON documents (Andrew Dunstan) + + +
Re: First draft of PG 17 release notes
On Thu, May 16, 2024 at 04:29:38PM +0800, jian he wrote:
> On Thu, May 9, 2024 at 12:04 PM Bruce Momjian wrote:
> >
> > I have committed the first draft of the PG 17 release notes; you can
> > see the results here:
> >
> > https://momjian.us/pgsql_docs/release-17.html
> >
>
> >> Add jsonpath methods to convert JSON values to different data types
> >> (Jeevan Chalke)
> >> The jsonpath methods are .bigint(), .boolean(), .date(),
> >> .decimal([precision [, scale]]), .integer(), .number(), .string(),
> >> .time(), .time_tz(), .timestamp(), and .timestamp_tz().
>
> I think it's slightly incorrect.
>
> for example:
> select jsonb_path_query('"2023-08-15"', '$.date()');
> I think it does is trying to validate json value "2023-08-15" can be a
> date value, if so, print json string out, else error out.
>
>
> "convert JSON values to different data types"
> meaning that we are converting json values to another data type, date?
I see your point. I have reworded it to be:
Add jsonpath methods to convert JSON values to other JSON data
types (Jeevan Chalke)
Does that help? I think your example is causing confusion because once
JSON values enter the SQL data type space, they are strings.
--
Bruce Momjian https://momjian.us
EDB https://enterprisedb.com
Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Thu, May 16, 2024 at 09:09:11AM -0400, Melanie Plageman wrote: > On Wed, May 15, 2024 at 11:48 PM Andres Freund wrote: > > > > On 2024-05-15 10:38:20 +0200, Alvaro Herrera wrote: > > > I disagree with this. IMO the impact of the Sawada/Naylor change is > > > likely to be enormous for people with large tables and large numbers of > > > tuples to clean up (I know we've had a number of customers in this > > > situation, I can't imagine any Postgres service provider that doesn't). > > > The fact that maintenance_work_mem is no longer capped at 1GB is very > > > important and I think we should mention that explicitly in the release > > > notes, as setting it higher could make a big difference in vacuum run > > > times. > > > > +many. > > > > We're having this debate every release. I think the ongoing reticence to > > note > > performance improvements in the release notes is hurting Postgres. > > > > For one, performance improvements are one of the prime reason users > > upgrade. Without them being noted anywhere more dense than the commit log, > > it's very hard to figure out what improved for users. A halfway widely > > applicable performance improvement is far more impactful than many of the > > feature changes we do list in the release notes. > > The practical reason this matters to users is that they want to change > their configuration or expectations in response to performance > improvements. > > And also, as Jelte mentions upthread, describing performance > improvements made each release in Postgres makes it clear that we are > consistently improving it. > > > For another, it's also very frustrating for developers that focus on > > performance. The reticence to note their work, while noting other, far > > smaller, things in the release notes, pretty much tells us that our work > > isn't > > valued. > > +many Please see the email I just posted. There are three goals we have to adjust for: 1. short release notes so they are readable 2. giving people credit for performance improvements 3. showing people Postgres cares about performance I would like to achieve 2 & 3 without harming #1. My experience is if I am reading a long document, and I get to a section where I start to wonder, "Why should I care about this?", I start to skim the rest of the document. I am particularly critical if I start to wonder, "Why does the author _think_ I should care about this?" becasue it feels like the author is writing for him/herself and not the audience. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Wed, May 15, 2024 at 08:48:02PM -0700, Andres Freund wrote: > Hi, > > On 2024-05-15 10:38:20 +0200, Alvaro Herrera wrote: > > I disagree with this. IMO the impact of the Sawada/Naylor change is > > likely to be enormous for people with large tables and large numbers of > > tuples to clean up (I know we've had a number of customers in this > > situation, I can't imagine any Postgres service provider that doesn't). > > The fact that maintenance_work_mem is no longer capped at 1GB is very > > important and I think we should mention that explicitly in the release > > notes, as setting it higher could make a big difference in vacuum run > > times. > > +many. > > We're having this debate every release. I think the ongoing reticence to note > performance improvements in the release notes is hurting Postgres. > > For one, performance improvements are one of the prime reason users > upgrade. Without them being noted anywhere more dense than the commit log, > it's very hard to figure out what improved for users. A halfway widely > applicable performance improvement is far more impactful than many of the > feature changes we do list in the release notes. I agree the impact of performance improvements are often greater than the average release note item. However, if people expect Postgres to be faster, is it important for them to know _why_ it is faster? If we add a new flag to a command, people will want to know about it so they can make use of it, or if there is a performance improvement that allows new workloads, they will want to know about that too so they can consider those workloads. On the flip side, a performance improvement that makes everything 10% faster has little behavioral change for users, and in fact I think we get ~6% faster in every major release. > For another, it's also very frustrating for developers that focus on > performance. The reticence to note their work, while noting other, far > smaller, things in the release notes, pretty much tells us that our work isn't > valued. Yes, but are we willing to add text that every user will have to read just for this purpose? One think we _could_ do is to create a generic performance release note item saying performance has been improved in the following areas, with no more details, but we can list the authors on the item. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Thu, May 16, 2024 at 03:35:17PM +1200, David Rowley wrote: > On Thu, 16 May 2024 at 14:48, Bruce Momjian wrote: > > > > On Wed, May 15, 2024 at 09:13:14AM -0400, Melanie Plageman wrote: > > > Also +1 on the Sawada/Naylor change being on the highlight section of > > > the release (as David suggested upthread). > > > > Agreed, I went with the attached applied patch. > > +Allow vacuum to more efficiently store tuple references and remove > its memory limit (Masahiko Sawada, John Naylor) > + > > I don't want it to seem like I'm splitting hairs, but I'd drop the " > and remove its memory limit" > > + > +Specifically, maintenance_work_mem and autovacuum_work_mem can now be > configured to use more than one gigabyte of memory. WAL traffic > caused by vacuum is also more compact. > > I'd say the first sentence above should be written as: > > "Additionally, vacuum no longer silently imposes a 1GB tuple reference > limit even when maintenance_work_mem or autovacuum_work_mem are set to > higher values" > > It's not "Specifically" as the "more efficiently store tuple > references" isn't the same thing as removing the 1GB cap. Also, there > was never a restriction in configuring maintenance_work_mem or > autovacuum_work_mem to values higher than 1GB. The restriction was > that vacuum was unable to utilize anything more than that. Slightly adjusted wording patch attached and applied. My deep apologies for the delay in addressing this. I should have done it sooner. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you. diff --git a/doc/src/sgml/release-17.sgml b/doc/src/sgml/release-17.sgml index ed1af17bb0a..d7fd87f3d5c 100644 --- a/doc/src/sgml/release-17.sgml +++ b/doc/src/sgml/release-17.sgml @@ -530,11 +530,12 @@ Author: Heikki Linnakangas -Allow vacuum to more efficiently store tuple references and remove its memory limit (Masahiko Sawada, John Naylor) +Allow vacuum to more efficiently store tuple references (Masahiko Sawada, John Naylor) -Specifically, maintenance_work_mem and autovacuum_work_mem can now be configured to use more than one gigabyte of memory. WAL traffic caused by vacuum is also more compact. +Additionally, vacuum is no longer silently limited to one gigabyte of memory when maintenance_work_mem and autovacuum_work_mem are higher. WAL traffic caused by +vacuum is also more compact.
Re: Fix log_line_prefix to display the transaction id (%x) for statements not in a transaction block
On Thu, May 16, 2024 at 08:37:43AM -0400, Robert Haas wrote: > On Thu, May 16, 2024 at 6:01 AM Quan Zongliang wrote: > > I thought about writing statement log when xid assigned. But it's seemed > > too complicated. > > I'm inclined to keep it for a while. Until we find a good way or give > > up. It's a reasonable request, after all. > > I don't think it's reasonable at all. We can't log the XID before it's > assigned, and we can't log the statement after the XID is assigned > without completely changing how the parameter works. I have removed the TODO item. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: More links on release-17.html
On Wed, May 15, 2024 at 04:50:47PM -0300, Marcos Pegoraro wrote: > While seeing changes and new features of > https://www.postgresql.org/docs/devel/release-17.html > I see that there are too little links to other DOC pages, which would be > useful. > > There are links to > "logical-replication", "sql-merge", "plpgsql", "libpq" and "pgstatstatements" > > But no one link is available to > COPY "ON_ERROR ignore", pg_dump, JSON_TABLE(), xmltext(), pg_basetype() , and > a > lot of other important features. So, wouldn't it be good to have their own > links, so the reader doesn't need to manually search for that feature ? Yes, it would be nice to have them. I will be looking for them in the coming weeks. I usually choose the closest link. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: cataloguing NOT NULL constraints
On Wed, May 15, 2024 at 09:50:36AM +0200, Álvaro Herrera wrote: > On 2024-May-14, Bruce Momjian wrote: > > > Turns out these commits generated a single release note item, which I > > have now removed with the attached committed patch. > > Hmm, but the commits about not-null constraints for domains were not > reverted, only the ones for constraints on relations. I think the > release notes don't properly address the ones on domains. I think it's > at least these two commits: > > > -Author: Peter Eisentraut > > -2024-03-20 [e5da0fe3c] Catalog domain not-null constraints > > -Author: Peter Eisentraut > > -2024-04-15 [9895b35cb] Fix ALTER DOMAIN NOT NULL syntax > > It may still be a good idea to make a note about those, at least to > point out that information_schema now lists them. For example, pg11 > release notes had this item Let me explain what I did to adjust the release notes. I took your commit hashes, which were longer than mine, and got the commit subject text from them. I then searched the release notes to see which commit subjects existed in the document. Only the first three did, and the release note item has five commits. The then tested if the last two patches could be reverted, and 'patch' thought they could be, so that confirmed they were not reverted. However, there was no text in the release note item that corresponded to the commits, so I just removed the entire item. What I now think happened is that the last two commits were considered part of the larger NOT NULL change, and not worth mentioning separately, but now that the NOT NULL part is reverted, we might need to mention them. I rarely handle such complex cases so I don't think I was totally correct in my handling. Let's get a reply to Peter Eisentraut's question and we can figure out what to do. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Thu, May 16, 2024 at 10:39:18AM +0800, jian he wrote: > On Thu, May 9, 2024 at 12:04 PM Bruce Momjian wrote: > > > > I have committed the first draft of the PG 17 release notes; you can > > see the results here: > > > > https://momjian.us/pgsql_docs/release-17.html > > > > >> Add local I/O block read/write timing statistics columns of > >> pg_stat_statement (Nazir Bilal Yavuz) > >> The new columns are "local_blk_read_time" and "local_blk_write_time". > here, "pg_stat_statement" should be "pg_stat_statements"? Agreed. > >> Add optional fourth parameter to pg_stat_statements_reset() to allow for > >> the resetting of only min/max statistics (Andrei Zubkov) > >> This parameter defaults to "false". > here, "parameter", should be "argument"? > > maybe > >> Add optional fourth boolean argument (minmax_only) to > >> pg_stat_statements_reset() to allow for the resetting of only min/max > >> statistics (Andrei Zubkov) > >> This argument defaults to "false". Sure. > > in section: E.1.2. Migration to Version 17 > > >> Rename I/O block read/write timing statistics columns of pg_stat_statement > >> (Nazir Bilal Yavuz) > >> This renames "blk_read_time" to "shared_blk_read_time", and > >> "blk_write_time" to "shared_blk_write_time". > > "pg_stat_statement" should be "pg_stat_statements"? Yes, fixed. > also, we only mentioned, pg_stat_statements some columns name changed > in "E.1.2. Migration to Version 17" > but if you look at the release note pg_stat_statements section, > we added a bunch of columns, which are far more incompatibile than > just colunm name changes. > > not sure we need add these in section "E.1.2. Migration to Version 17" Well, new columns don't cause breakage like renamed columns, which is why I only put renames/removed columns in the migration section. Also, thanks everyone for the release notes feedback. In some cases I made a mistake, and in some cases I misjudged the item. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Wed, May 15, 2024 at 09:13:14AM -0400, Melanie Plageman wrote: > I think this wording and organization makes sense. I hadn't thought of > using "traffic" to describe this, but I like it. > > Also +1 on the Sawada/Naylor change being on the highlight section of > the release (as David suggested upthread). Agreed, I went with the attached applied patch. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you. diff --git a/doc/src/sgml/release-17.sgml b/doc/src/sgml/release-17.sgml index e68c499e0db..7f36a35954d 100644 --- a/doc/src/sgml/release-17.sgml +++ b/doc/src/sgml/release-17.sgml @@ -519,6 +519,25 @@ Allow vacuum to more efficiently remove and freeze tuples (Masahiko Sawada, John + + + + +Allow vacuum to more efficiently store tuple references and remove its memory limit (Masahiko Sawada, John Naylor) + + + +Specifically, maintenance_work_mem and autovacuum_work_mem can now be configured to use more than one gigabyte of memory. WAL traffic caused by vacuum is also more compact. + + +
Re: First draft of PG 17 release notes
On Wed, May 15, 2024 at 10:10:28AM +0900, Masahiko Sawada wrote: > > I looked at that item and I don't have a generic "make logical > > replication apply faster" item to merge it into, and many > > subtransactions seemed like enough of an edge-case that I didn't think > > mentioning it make sense. Can you see a good place to add it? > > I think that since many subtransactions cases are no longer becoming > edge-cases these days, we needed to improve that and it might be > helpful for users to mention it. How about the following item for > example? > > Improve logical decoding performance in cases where there are many > subtransactions. Okay, item added in the attached applied patch. > > > Finally, should we mention the following commit in the release note? > > > It's not a user-visible change but added a new regression test module. > > > > > > - Add tests for XID wraparound (e255b646a) > > > > I don't normally add testing infrastructure changes unless they are > > major. > > I've seen we had such item, for example in PG14 release note: > > Add a test module for the regular expression package (Tom Lane) > > But if our policy has already changed, I'm okay with not mentioning > the xid_wraparound test in the PG17 release note. Uh, that PG 14 test suite was huge and flushed out a lot of bugs, not only in our regex code but I think in the TCL/Henry Spencer regex library we inherited. We add 10-40 tests every year, and how many do I mention in the release notes? You had to go back to PG 14 to find one. We have not changed our release note "test item" criteria --- I only mention tests that are significant to our userbase. I think that test suite was significant to anyone using the TCL/Henry Spencer regex library. If you want your test mentioned, you have to explain why it is useful for users to know about it, or the value it brings them. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you. diff --git a/doc/src/sgml/release-17.sgml b/doc/src/sgml/release-17.sgml index 510810b53dc..e68c499e0db 100644 --- a/doc/src/sgml/release-17.sgml +++ b/doc/src/sgml/release-17.sgml @@ -1150,6 +1150,17 @@ Previously only btree indexes could be used for this purpose. + + + + +Improve logical decoding performance in cases where there are many subtransactions (Masahiko Sawada) + + +
Re: First draft of PG 17 release notes
On Wed, May 15, 2024 at 02:03:32PM +1200, David Rowley wrote: > On Wed, 15 May 2024 at 13:00, Bruce Momjian wrote: > > > > On Tue, May 14, 2024 at 03:39:26PM -0400, Melanie Plageman wrote: > > > "Reduce system calls by automatically merging reads up to > > > io_combine_limit" > > > > Uh, as I understand it, the reduced number of system calls is not the > > value of the feature, but rather the ability to request a larger block > > from the I/O subsystem. Without it, you have to make a request and wait > > for each request to finish. I am open to new wording, but I am not sure > > your new wording is accurate. > > I think you have the cause and effect backwards. There's no advantage > to reading 128KB if you only need 8KB. It's the fact that doing > *larger* reads allows *fewer* reads that allows it to be more > efficient. There are also the efficiency gains from fadvise > POSIX_FADV_WILLNEED. I'm unsure how to jam that into a short sentence. > Maybe; "Optimize reading of tables by allowing pages to be prefetched > and read in chunks up to io_combine_limit", or a bit more buzzy; > "Optimize reading of tables by allowing pages to be prefetched and > performing vectored reads in chunks up to io_combine_limit". Yes, my point is that it is not the number of system calls or system call overhead that is the advantage of this patch, but the ability to request more of the I/O system in one call, which is not the same as system calls. I can use your wording, but how much prefetching to we enable now? -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Tue, May 14, 2024 at 10:22:35AM +0800, Tender Wang wrote: > > > jian he 于2024年5月9日周四 18:00写道： > > On Thu, May 9, 2024 at 12:04 PM Bruce Momjian wrote: > > > > I have committed the first draft of the PG 17 release notes; you can > > see the results here: > > > > https://momjian.us/pgsql_docs/release-17.html > > > > another potential incompatibilities issue: > ALTER TABLE DROP PRIMARY KEY > > see: > https://www.postgresql.org/message-id/ > 202404181849.6frtmajobe27%40alvherre.pgsql > > > > Since Alvaro has reverted all changes to not-null constraints, so will not > have > potential incompatibilities issue. Agreed. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: cataloguing NOT NULL constraints
On Mon, May 13, 2024 at 09:00:28AM -0400, Robert Haas wrote: > > Specifically, the problem is that I mentioned that we could restrict the > > NOT NULL NO INHERIT addition in pg_dump for primary keys to occur only > > in pg_upgrade; but it turns this is not correct. In normal > > dump/restore, there's an additional table scan to check for nulls when > > the constraints is not there, so the PK creation would become measurably > > slower. (In a table with a million single-int rows, PK creation goes > > from 2000ms to 2300ms due to the second scan to check for nulls). > > I have a feeling that any theory of the form "X only needs to happen > during pg_upgrade" is likely to be wrong. pg_upgrade isn't really > doing anything especially unusual: just creating some objects and > loading data. Those things can also be done at other times, so > whatever is needed during pg_upgrade is also likely to be needed at > other times. Maybe that's not sound reasoning for some reason or > other, but that's my intuition. I assume Alvaro is saying that pg_upgrade has only a single session, which is unique and might make things easier for him. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: cataloguing NOT NULL constraints
On Sun, May 12, 2024 at 04:56:09PM +0200, Álvaro Herrera wrote: > On 2024-May-11, Alvaro Herrera wrote: > > > I have found two more problems that [] require some more work to fix, > > so I've decided to cut my losses now and revert the whole. > > Here's the revert patch, which I intend to push early tomorrow. > > Commits reverted are: > 21ac38f498b33f0231842238b83847ec63dfe07b > d45597f72fe53a53f6271d5ba4e7acf8fc9308a1 > 13daa33fa5a6d340f9be280db14e7b07ed11f92e > 0cd711271d42b0888d36f8eda50e1092c2fed4b3 > d72d32f52d26c9588256de90b9bc54fe312cee60 > d9f686a72ee91f6773e5d2bc52994db8d7157a8e > c3709100be73ad5af7ff536476d4d713bca41b1a > 3af7217942722369a6eb7629e0fb1cbbef889a9b > b0f7dd915bca6243f3daf52a81b8d0682a38ee3b > ac22a9545ca906e70a819b54e76de38817c93aaf > d0ec2ddbe088f6da35444fad688a62eae4fbd840 > 9b581c53418666205938311ef86047aa3c6b741f > b0e96f311985bceba79825214f8e43f65afa653a > > with some significant conflict fixes (mostly in the last one). Turns out these commits generated a single release note item, which I have now removed with the attached committed patch. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you. diff --git a/doc/src/sgml/release-17.sgml b/doc/src/sgml/release-17.sgml index 9c7c0a0337f..143ee17716d 100644 --- a/doc/src/sgml/release-17.sgml +++ b/doc/src/sgml/release-17.sgml @@ -1486,29 +1486,6 @@ Add event trigger support for REINDEX (Garrett Thornburg, Jian He) - - - - -Allow NOT NULL columns to be specified as column constraints and domains (Alvaro Herrera, Bernd Helmle, Kyotaro Horiguchi, Peter Eisentraut) - - - -Previously NOT NULL could only be specified as a column attribute. - - -
Re: First draft of PG 17 release notes
On Tue, May 14, 2024 at 03:39:26PM -0400, Melanie Plageman wrote: > On Thu, May 9, 2024 at 12:04 AM Bruce Momjian wrote: > > > > I have committed the first draft of the PG 17 release notes; you can > > see the results here: > > > > https://momjian.us/pgsql_docs/release-17.html > > I had two comments: > > > I think the read stream item: > > "Allow the grouping of file system reads with the new system variable > io_combine_limit" > > Might be better if it mentions the effect, like: > > "Reduce system calls by automatically merging reads up to io_combine_limit" Uh, as I understand it, the reduced number of system calls is not the value of the feature, but rather the ability to request a larger block from the I/O subsystem. Without it, you have to make a request and wait for each request to finish. I am open to new wording, but I am not sure your new wording is accurate. > --- > For the vacuum feature: > > "Allow vacuum to more efficiently remove and freeze tuples" > > I think that we need to more clearly point out the implications of the > feature added by Sawada-san (and reviewed by John) in 667e65aac35497. > Vacuum no longer uses a fixed amount of memory for dead tuple TID > storage and it is not preallocated. This affects users as they may > want to change their configuration (and expectations). > > If you make that item more specific to their work, you should also > remove my name, as the work I did on vacuum this release was unrelated > to their work on dead tuple TID storage. > > The work Heikki and I did which culminated in 6dbb490261 mainly has > the impact of improving vacuum's performance (vacuum emits less WAL > and is more efficient). So you could argue for removing it from the > release notes if you are using the requirement that performance > improvements don't go in the release notes. > > However, one of the preliminary commits for this f83d70976 does change > WAL format. There are three WAL records which no longer exist as > separate records. Do users care about this? I don't think users really care about these details, just that it is faster so they will not be surprised if there is a change. I was purposely vague to group multiple commits into the single item. By grouping them together, I got enough impact to warrant listing it. If you split it apart, it is harder to justify mentioning them. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Tue, May 14, 2024 at 02:58:41PM +0100, Pantelis Theodosiou wrote: > On Thu, May 9, 2024 at 5:03 AM Bruce Momjian wrote > > > > > > I welcome feedback. For some reason it was an easier job than usual. > > This looks better if "more case" -> "more cases" : > > Allow query nodes to be run in parallel in more case (Tom Lane) Yes, you are correct, fixed. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Tue, May 14, 2024 at 02:20:24PM +0200, Jelte Fennema-Nio wrote: > On Tue, 14 May 2024 at 02:56, Bruce Momjian wrote: > > > > On Sat, May 11, 2024 at 10:24:39AM -0400, Joe Conway wrote: > > > On 5/11/24 09:57, Jelte Fennema-Nio wrote: > > > > The buffering change improved performance up to ~40% in some of the > > > > benchmarks. The case it improves mostly is COPY of large rows and > > > > streaming a base backup. That sounds user-visible enough to me to > > > > warrant an entry imho. > > > > > > +1 > > > > Attached patch applied. > > I think we shouldn't list this under the libpq changes and shouldn't > mention libpq in the description, since this patch changes > src/backend/libpq files instead of src/interfaces/libpq files. I think > it should be in the "General performance" section and describe the > change as something like the below: > > Improve performance when transferring large blocks of data to a client > > PS. I completely understand that this was not clear from the commit message. Okay, I went with your wording. Attached patch applied. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you. diff --git a/doc/src/sgml/release-17.sgml b/doc/src/sgml/release-17.sgml index a3b4a8fb3b9..448f5653bc5 100644 --- a/doc/src/sgml/release-17.sgml +++ b/doc/src/sgml/release-17.sgml @@ -563,6 +563,17 @@ Improve performance of heavily-contended WAL writes (Bharath Rupireddy) + + + + +Improve performance when transferring large blocks of data to a client (Melih Mutlu) + + + - - - -Allow libpq to more efficiently transfer large blocks of data (Melih Mutlu) - - -
Re: First draft of PG 17 release notes
On Tue, May 14, 2024 at 01:34:56PM +0300, Elena Indrupskaya wrote: > Being a technical writer, I attached a small patch that fixes minor language > stuff. You are absolutely correct. Patch applied, thanks. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Sat, May 11, 2024 at 03:32:55PM -0400, Andrew Dunstan wrote: > > On 2024-05-09 Th 00:03, Bruce Momjian wrote: > > I have committed the first draft of the PG 17 release notes; you can > > see the results here: > > > > https://momjian.us/pgsql_docs/release-17.html > > > > It will be improved until the final release. The item count is 188, > > which is similar to recent releases: > > > > release-10: 189 > > release-11: 170 > > release-12: 180 > > release-13: 178 > > release-14: 220 > > release-15: 184 > > release-16: 206 > > release-17: 188 > > > > I welcome feedback. For some reason it was an easier job than usual. > > > I don't like blowing my own horn but I feel commit 3311ea86ed "Introduce a > non-recursive JSON parser" should be in the release notes. This isn't > something that's purely internal, but it could be used by an extension or a > client program to parse JSON documents that are too large to handle with the > existing API. > > Maybe "Introduce an incremental JSON parser" would have been a better > headline. Well, this gets into a level of detail that is beyond the average reader. I think at that level people will need to read the git logs or review the code. Do we use it for anything yet? It could be put in the source code section but I try to only have user-visible stuff there. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Tue, May 14, 2024 at 10:32:14AM +0800, Andy Fan wrote: > Bruce Momjian writes: > > It was unclear from the commit message exactly what user-visible > > optimization this allowed. Do you have details? > > Yes, It allows the query like "SELECT * FROM t1 WHERE t1.a in (SELECT a > FROM t2 WHERE t2.b = t1.b)" be pulled up a semi join, hence more join > methods / join orders are possible. > > > Yes, for example: (subquery-1) UNION ALL (subquery-2) LIMIT n; > > When planning the subquery-1 or subquery-2, limit N should be > considered. As a consequence, maybe hash join should be replaced with > Nested Loop. Before this commits, it is ignored if it is flatten into > appendrel, and the "flatten" happens very often. > > David provided a summary for the both commits in [1]. Okay, attached patch applied. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you. diff --git a/doc/src/sgml/release-17.sgml b/doc/src/sgml/release-17.sgml index 38c14970822..fa0a703629c 100644 --- a/doc/src/sgml/release-17.sgml +++ b/doc/src/sgml/release-17.sgml @@ -356,6 +356,28 @@ Improve optimization of range values when using containment operators @ and + + + + +Allow correlated IN subqueries to be transformed into joins (Andy Fan, Tom Lane) + + + + + + + +Improve optimization of the LIMIT clause on partitioned tables, inheritance parents, and UNION ALL queries (Andy Fan, David Rowley) + + +
Re: First draft of PG 17 release notes
On Sat, May 11, 2024 at 01:27:25PM +0800, Andy Fan wrote: > > Hello Bruce, > > > I have committed the first draft of the PG 17 release notes; you can > > see the results here: > > > > https://momjian.us/pgsql_docs/release-17.html > > Thank you for working on this! > > > I welcome feedback. For some reason it was an easier job than usual. > > Do you think we need to add the following 2 items? > > - 9f133763961e280d8ba692bcad0b061b861e9138 this is an optimizer > transform improvement. It was unclear from the commit message exactly what user-visible optimization this allowed. Do you have details? > - a8a968a8212ee3ef7f22795c834b33d871fac262 this is an optimizer costing > improvement. Does this allow faster UNION ALL with LIMIT, perhaps? -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Sat, May 11, 2024 at 10:24:39AM -0400, Joe Conway wrote: > On 5/11/24 09:57, Jelte Fennema-Nio wrote: > > On Fri, 10 May 2024 at 23:31, Tom Lane wrote: > > > > > > Bruce Momjian writes: > > > > I looked at both of these. In both cases I didn't see why the user > > > > would need to know these changes were made: > > > > > > I agree that the buffering change is not likely interesting, but > > > the fact that you can now control-C out of a psql "\c" command > > > is user-visible. People might have internalized the fact that > > > it didn't work, or created complicated workarounds. > > > > The buffering change improved performance up to ~40% in some of the > > benchmarks. The case it improves mostly is COPY of large rows and > > streaming a base backup. That sounds user-visible enough to me to > > warrant an entry imho. > > +1 Attached patch applied. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you. commit e87e7324555 Author: Bruce Momjian Date: Mon May 13 20:55:13 2024 -0400 doc PG 17 relnotes: add item about libpq large data transfers Reported-by: Jelte Fennema-Nio Discussion: https://postgr.es/m/cageczqtz5auqlel6dald2hu2fxs_losh4kedndj1fwthsb_...@mail.gmail.com Reviewed-by: Joe Conway Backpatch-through: master diff --git a/doc/src/sgml/release-17.sgml b/doc/src/sgml/release-17.sgml index 9dd3954f3c2..38c14970822 100644 --- a/doc/src/sgml/release-17.sgml +++ b/doc/src/sgml/release-17.sgml @@ -1901,6 +1901,17 @@ Add libpq function PQsetChunkedRowsMode to allow retrieval of results in chunks + + + + +Allow libpq to more efficiently transfer large blocks of data (Melih Mutlu) + + +
Re: First draft of PG 17 release notes
On Fri, May 10, 2024 at 05:31:33PM -0400, Tom Lane wrote: > Bruce Momjian writes: > > On Fri, May 10, 2024 at 06:50:54PM +0200, Jelte Fennema-Nio wrote: > >> There are two commits that I think would benefit from being listed > >> (but maybe they are already listed and I somehow missed them, or they > >> are left out on purpose for some reason): > > > I looked at both of these. In both cases I didn't see why the user > > would need to know these changes were made: > > I agree that the buffering change is not likely interesting, but > the fact that you can now control-C out of a psql "\c" command > is user-visible. People might have internalized the fact that > it didn't work, or created complicated workarounds. Agreed, attached patch applied. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you. diff --git a/doc/src/sgml/release-17.sgml b/doc/src/sgml/release-17.sgml index 5a741ff16f1..9dd3954f3c2 100644 --- a/doc/src/sgml/release-17.sgml +++ b/doc/src/sgml/release-17.sgml @@ -1980,6 +1980,17 @@ The parameter is "min_rows". + + + + +Allow psql connections to be canceled with control-C (Tristan Partin) + + +
Re: First draft of PG 17 release notes
On Fri, May 10, 2024 at 05:31:33PM -0400, Tom Lane wrote: > Bruce Momjian writes: > > On Fri, May 10, 2024 at 06:50:54PM +0200, Jelte Fennema-Nio wrote: > >> There are two commits that I think would benefit from being listed > >> (but maybe they are already listed and I somehow missed them, or they > >> are left out on purpose for some reason): > > > I looked at both of these. In both cases I didn't see why the user > > would need to know these changes were made: > > I agree that the buffering change is not likely interesting, but > the fact that you can now control-C out of a psql "\c" command > is user-visible. People might have internalized the fact that > it didn't work, or created complicated workarounds. It was not clear to me what the user-visible behavior was with the SIGINT control. Yes, based on your details, it should be mentioned. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Fri, May 10, 2024 at 06:50:54PM +0200, Jelte Fennema-Nio wrote: > On Thu, 9 May 2024 at 06:04, Bruce Momjian wrote: > > > > I have committed the first draft of the PG 17 release notes; you can > > see the results here: > > > > https://momjian.us/pgsql_docs/release-17.html > > Great work! > > There are two commits that I think would benefit from being listed > (but maybe they are already listed and I somehow missed them, or they > are left out on purpose for some reason): I looked at both of these. In both cases I didn't see why the user would need to know these changes were made: --- > - c4ab7da60617f020e8d75b1584d0754005d71830 commit c4ab7da6061 Author: David Rowley Date: Sun Apr 7 21:20:18 2024 +1200 Avoid needless large memcpys in libpq socket writing Until now, when calling pq_putmessage to write new data to a libpq socket, all writes are copied into a buffer and that buffer gets flushed when full to avoid having to perform small writes to the socket. There are cases where we must write large amounts of data to the socket, sometimes larger than the size of the buffer. In this case, it's wasteful to memcpy this data into the buffer and flush it out, instead, we can send it directly from the memory location that the data is already stored in. Here we adjust internal_putbytes() so that after having just flushed the buffer to the socket, if the remaining bytes to send is as big or bigger than the buffer size, we just send directly rather than needlessly copying into the PqSendBuffer buffer first. Examples of operations that write large amounts of data in one message are; outputting large tuples with SELECT or COPY TO STDOUT and pg_basebackup. Author: Melih Mutlu Reviewed-by: Heikki Linnakangas Reviewed-by: Jelte Fennema-Nio Reviewed-by: David Rowley Reviewed-by: Ranier Vilela Reviewed-by: Andres Freund Discussion: https://postgr.es/m/cagpvpcr15nosj0f6xe-c2h477zfr88q12e6wjeoezc8zykt...@mail.gmail.com > - cafe1056558fe07cdc52b95205588fcd80870362 commit cafe1056558 Author: Robert Haas Date: Tue Apr 2 10:26:10 2024 -0400 Allow SIGINT to cancel psql database reconnections. After installing the SIGINT handler in psql, SIGINT can no longer cancel database reconnections. For instance, if the user starts a reconnection and then needs to do some form of interaction (ie psql is polling), there is no way to cancel the reconnection process currently. Use PQconnectStartParams() in order to insert a cancel_pressed check into the polling loop. Tristan Partin, reviewed by Gurjeet Singh, Heikki Linnakangas, Jelte Fennema-Nio, and me. Discussion: http://postgr.es/m/d08wwcpvhkhn.3qelikzj2d...@neon.tech -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: Augmenting the deadlock message with application_name
On Fri, May 10, 2024 at 08:10:58PM +, Karoline Pauls wrote: > On Friday, 10 May 2024 at 20:17, Bruce Momjian > wrote: > > > > log_line_prefix supports application name --- why would you not use > > that? > > > > log_line_prefix is effective in the server log. This change is mostly > about improving the message sent back to the client. While the server > log is also changed to reflect the client message, it doesn't need to > be. I was hoping client_min_messages would show the application name, but it doesn't but your bigger point is below. > Additionally, with `%a` added to log_line_prefix, the server log > would only contain the application name of the client affected by the > deadlock, not the application names of all other clients involved in > it. Yeah, getting the application names of the pids reported in the log requires looking backward in the logs to find out what the most recent log line was for the pids involved. Frankly, I think it would be more useful to show the session_id in the deadlock so you could then use that to look back to any details you want in the logs, not only the application name. > Anyway, the server log is not the important part here. The crucial > UX feature is the client getting application names back, so browsing > through server logs can be avoided. Well, we don't want to report too much information because it gets confusing. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Fri, May 10, 2024 at 06:29:11PM +0200, Daniel Verite wrote: > Bruce Momjian wrote: > > > have committed the first draft of the PG 17 release notes; you can > > see the results here: > > > > https://momjian.us/pgsql_docs/release-17.html > > In the psql items, I'd suggest mentioning > > https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=90f5178 > > For the short description, maybe something like that: > > - Improve FETCH_COUNT to work with all queries (Daniel Vérité) > Previously, results would be fetched in chunks only for queries > that start with the SELECT keyword. Agreed, patch attached and applied. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you. diff --git a/doc/src/sgml/release-17.sgml b/doc/src/sgml/release-17.sgml index e4b34d827d1..08238be9cb7 100644 --- a/doc/src/sgml/release-17.sgml +++ b/doc/src/sgml/release-17.sgml @@ -1893,8 +1893,6 @@ This is similar to PQpipelineSync() but it does not flush to the server unless t @@ -1937,6 +1935,17 @@ This is enabled with the client-side option sslnegotation=direct, requires ALPN, + + + + +Allow FETCH_COUNT to work with non-SELECT queries (Daniel Vérité) + + +
Re: Augmenting the deadlock message with application_name
On Thu, May 9, 2024 at 11:44:03PM +, Karoline Pauls wrote: > As we know, the deadlock error message isn't the most friendly one. All the > client gets back is process PIDs, transaction IDs, and lock types. You have to > check the server log to retrieve lock details. This is tedious. > > In one of my apps I even added a deadlock exception handler on the app side to > query pg_stat_activity for processes involved in the deadlock and include > their > application names and queries in the exception message. It is a little racy > but > works well enough. > > Ideally I'd like to see that data coming from Postgres upon detecting the > deadlock. That's why I made this small change. > > The change makes the deadlock error look as follows - the new element is the > application name or "" in its place if the activity > user doesn't match the current user (and the current use isn't a superuser): > > postgres=*> SELECT * FROM q WHERE id = 2 FOR UPDATE; > ERROR: deadlock detected > DETAIL: Process 194520 (application_name: ) waits for > ShareLock on transaction 776; blocked by process 194521. > Process 194521 (application_name: woof) waits for ShareLock on transaction > 775; > blocked by process 194520. > HINT: See server log for query details. > CONTEXT: while locking tuple (0,2) in relation "q" log_line_prefix supports application name --- why would you not use that? -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Fri, May 10, 2024 at 01:54:30PM +0530, Bharath Rupireddy wrote: > On Thu, May 9, 2024 at 9:34 AM Bruce Momjian wrote: > > > > I have committed the first draft of the PG 17 release notes; you can > > see the results here: > > > > https://momjian.us/pgsql_docs/release-17.html > > > > It will be improved until the final release. The item count is 188, > > which is similar to recent releases: > > > > release-10: 189 > > release-11: 170 > > release-12: 180 > > release-13: 178 > > release-14: 220 > > release-15: 184 > > release-16: 206 > > release-17: 188 > > > > I welcome feedback. For some reason it was an easier job than usual. > > Thanks a lot for this work Bruce! It looks like commit > https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=91f2cae7a4e664e9c0472b364c7db29d755ab151 > is missing from daft release notes. Just curious to know if it's > intentional or a miss out. I did not mention it because the commit didn't mention any performance benefit and it seemed more like an internal change than something people needed to know about. I could reword and merge it into this item, if you think I should: Improve performance of heavily-contended WAL writes (Bharath Rupireddy) -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Thu, May 9, 2024 at 08:40:00PM +0200, Álvaro Herrera wrote: > On 2024-May-09, Bruce Momjian wrote: > > > However, I don't see it mentioned as a release note item in the commit > > message or mentioned in our docs. I suppose the release note text would > > be: > > > > Removing a PRIMARY KEY will remove the NOT NULL column specification > > > > Previously the NOT NULL specification would be retained. > > > > Do we have agreement that we want this release note item? > > Yes. Maybe we want some others too (especially regarding inheritance, > but also regarding the way we handle the constraints internally), and > maybe in this one we want different wording. How about something like > this: > > Removing a primary key constraint may change the nullability > characteristic of the columns that the primary key covered. > > If explicit not-null constraints exist on the same column, then they > continue to be /known not nullable/; otherwise they become /possibly > nullable/. > > This is largely based on the SQL standard's language of a column > descriptor having a "nullability characteristic", which for columns with > not-null or primary key constraints is "known not null". I don't think > we use those terms anywhere. I hope this isn't too confusing. Yes, it was confusing, partly because it is using wording we don't use, and partly because it is talking about what can go into the column, rather than the visible column restriction NOT NULL. I also think "may" is too imprecise. How about: Removing a primary key will remove a column's NOT NULL constraint if the constraint was added by the primary key Previously such NOT NULL constraints would remain after a primary key was removed. A column-level NOT NULL constraint would not be emoved. Here is the PG 16 output: CREATE TABLE test ( x INT CONSTRAINT test_pkey PRIMARY KEY ); Table "public.test" Column | Type | Collation | Nullable | Default +-+---+--+- x | integer | | not null | Indexes: "test_pkey" PRIMARY KEY, btree (x) CREATE TABLE test_with_not_null (x INT NOT NULL CONSTRAINT test_pkey_with_not_null PRIMARY KEY); Table "public.test_with_not_null" Column | Type | Collation | Nullable | Default +-+---+--+- x | integer | | not null | Indexes: "test_pkey_with_not_null" PRIMARY KEY, btree (x) ALTER TABLE test DROP CONSTRAINT test_pkey; Table "public.test" Column | Type | Collation | Nullable | Default +-+---+--+- --> x | integer | | not null | ALTER TABLE test_with_not_null DROP CONSTRAINT test_pkey_with_not_null; Table "public.test_with_not_null" Column | Type | Collation | Nullable | Default +-+---+--+- --> x | integer | | not null | Here is the output in PG 17: CREATE TABLE test ( x INT CONSTRAINT test_pkey PRIMARY KEY ); Table "public.test" Column | Type | Collation | Nullable | Default +-+---+--+- x | integer | | not null | Indexes: "test_pkey" PRIMARY KEY, btree (x) CREATE TABLE test_with_not_null (x INT NOT NULL CONSTRAINT test_pkey_with_not_null PRIMARY KEY); Table "public.test_with_not_null" Column | Type | Collation | Nullable | Default +-+---+--+- x | integer | | not null | Indexes: "test_pkey_with_not_null" PRIMARY KEY, btree (x) ALTER TABLE test DROP CONSTRAINT test_pkey; Table "public.test" Column | Type | Collation | Nullable | Default +-+---+--+- --> x | integer | | | ALTER TABLE test_with_not_null DROP CONSTRAINT test_pkey_with_not_null; Table "public.test_with_not_null" Column | Type | Collation | Nullable | Default +-+---+--+- --> x | integer | | not null | Notice that the table without a _column_ NOT NULL removes the NOT NULL designation after removing the primary key only in PG 17. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Fri, May 10, 2024 at 08:05:43AM +1200, Thomas Munro wrote: > On Thu, May 9, 2024 at 4:04 PM Bruce Momjian wrote: > > I welcome feedback. For some reason it was an easier job than usual. > > > 2024-01-25 [820b5af73] jit: Require at least LLVM 10. > > > Require LLVM version 10 or later (Peter Eisentraut) > > Peter reviewed, I authored, and I think you intend to list authors in > parentheses. Yes, my mistake, fixed. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: PERIOD foreign key feature
On Wed, May 8, 2024 at 08:47:45PM -0700, Paul Jungwirth wrote: > On 5/8/24 07:44, Bruce Momjian wrote: > > On Wed, May 8, 2024 at 02:29:34PM +0200, Peter Eisentraut wrote: > > > > Yes, David is correct here on all points. I like his suggestion to > > > > clarify the language here also. If you need a patch from me let me know, > > > > but I assume it's something a committer can just make happen? > > > > > > In principle yes, but it's also very helpful if someone produces an actual > > > patch file, with complete commit message, credits, mailing list link, etc. > > > > I am ready to do the work, but waited a day for Peter to reply, since he > > was the author of the text. > > Here is a patch for this. Thanks, patch applied. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Thu, May 9, 2024 at 12:10:11PM -0400, Andrew Dunstan wrote:
>
> On 2024-05-09 Th 00:03, Bruce Momjian wrote:
>
> I have committed the first draft of the PG 17 release notes; you can
> see the results here:
>
> https://momjian.us/pgsql_docs/release-17.html
>
> It will be improved until the final release. The item count is 188,
> which is similar to recent releases:
>
> release-10: 189
> release-11: 170
> release-12: 180
> release-13: 178
> release-14: 220
> release-15: 184
> release-16: 206
> release-17: 188
>
> I welcome feedback. For some reason it was an easier job than usual.
>
>
> • Remove the ability to build Postgres with Visual Studio (Michael Paquier)
>
> Meson is now the only available Windows build method.
>
>
> This is a category mistake. What was removed was the special code we had for
> building with VS, but not the ability to build with VS. You can build with VS
> using meson (see for example drongo on the buildfarm)
Wow, okay, I am not surprised I was confused. New text is:
Remove the Microsoft Visual Studio Studio-specific Postgres build
option (Michael Paquier)
Meson is now the only method for Visual Studio builds.
Remove the Microsoft Visual Studio Studio-specific Postgres build
option (Michael Paquier)
Meson is now the only method for Visual Studio builds.
--
Bruce Momjian https://momjian.us
EDB https://enterprisedb.com
Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Thu, May 9, 2024 at 11:26:44PM +0800, jian he wrote: > On Thu, May 9, 2024 at 11:12 PM Bruce Momjian wrote: > > > > On Thu, May 9, 2024 at 07:49:55PM +0800, jian he wrote: > > > On Thu, May 9, 2024 at 6:53 PM jian he > > > wrote: > > > > > > > > On Thu, May 9, 2024 at 12:04 PM Bruce Momjian wrote: > > > > > I have committed the first draft of the PG 17 release notes; you can > > > > > see the results here: > > > > > > > > > > https://momjian.us/pgsql_docs/release-17.html > > > > > E.1.3.1.5. Privileges > Add per-table GRANT permission MAINTAIN to control maintenance > operations (Nathan Bossart) > > The operations are VACUUM, ANALYZE, REINDEX, REFRESH MATERIALIZE VIEW, > CLUSTER, and LOCK TABLE. > > Add user-grantable role pg_maintain to control maintenance operations > (Nathan Bossart) > > The operations are VACUUM, ANALYZE, REINDEX, REFRESH MATERIALIZE VIEW, > CLUSTER, and LOCK TABLE. > > Allow roles with pg_monitor privileges to execute pg_current_logfile() > (Pavlo Golub, Nathan Bossart) > --- > should be "REFRESH MATERIALIZED VIEW"? Yes, fixed. > also > "Allow roles with pg_monitor privileges to execute > pg_current_logfile() (Pavlo Golub, Nathan Bossart)" > "pg_monitor" is a predefined role, so technically, "with pg_monitor > privileges" is not correct? Good point, new text: Allow roles with pg_monitor membership to execute pg_current_logfile() (Pavlo Golub, Nathan Bossart) > -- > Add function XMLText() to convert text to a single XML text node (Jim Jones) > > XMLText() > should be > xmltext() Right, fixed. > -- > Add function to_regtypemod() to return the typemod of a string (David > Wheeler, Erik Wienhold) > I think this description does not mean the same thing as the doc[1] Yes, I see your point. I changed the text to: Add function to_regtypemod() to return the type modifier of a type specification (David Wheeler, Erik Wienhold) > [1] > https://www.postgresql.org/docs/devel/functions-info.html#FUNCTIONS-INFO-CATALOG > -- > > Allow GROUP BY columns to be internally ordered to match ORDER BY > (Andrei Lepikhov, Teodor Sigaev) > This can be disabled using server variable enable_group_by_reordering. > > Probably > `This can be disabled by setting the server variable > enable_group_by_reordering to false`. Uh, I usually don't go into that detail. There will be a link to the variable in about a month so users can look up its behavior. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Thu, May 9, 2024 at 07:49:55PM +0800, jian he wrote: > On Thu, May 9, 2024 at 6:53 PM jian he wrote: > > > > On Thu, May 9, 2024 at 12:04 PM Bruce Momjian wrote: > > > I have committed the first draft of the PG 17 release notes; you can > > > see the results here: > > > > > > https://momjian.us/pgsql_docs/release-17.html > > < Add columns to pg_stats to report range histogram information (Egor > Rogov, Soumyadeep Chakraborty) > I think this applies to range type and multi range type, "range > histogram information" seems not very clear to me. > So maybe: > < Add columns to pg_stats to report range-type histogram information > (Egor Rogov, Soumyadeep Chakraborty) Yes, good point, done. > Display length and bounds histograms in pg_stats Uh, isn't that assumed? Is this a detail worth mentioning? > < Add new COPY option "ON_ERROR ignore" to discard error rows (Damir > Belyalov, Atsushi Torikoshi, Alex Shulgin, Jian He, Jian He, Yugo > Nagata) > duplicate name. Fixed. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Thu, May 9, 2024 at 06:57:01PM +0800, jian he wrote: > > << > > Allow ALTER OPERATOR to set more optimization attributes (Tommy Pavlicek) > > This is useful for extensions. > > << > > sorry, I mean > << > Allow the creation of hash indexes on ltree columns (Tommy Pavlicek) > This also enables hash join and hash aggregation on ltree columns. > << > > better description would be: > << > Add hash support functions and hash opclass for contrib/ltree (Tommy Pavlicek) > This also enables hash join and hash aggregation on ltree columns. > << Yes, please see my previous email where I am asking why being more specific is worse. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Thu, May 9, 2024 at 06:53:30PM +0800, jian he wrote: > On Thu, May 9, 2024 at 12:04 PM Bruce Momjian wrote: > > I have committed the first draft of the PG 17 release notes; you can > > see the results here: > > > > https://momjian.us/pgsql_docs/release-17.html > > > > * Add function pg_buffercache_evict() to allow shared buffer eviction > (Palak Chaturvedi, Thomas Munro) > * This is useful for testing. > > this should put it on the section > < E.1.3.11. Additional Modules > ? Oh, it is in the pg_buffercache module --- I should have realized that from the name, fixed. > Then I found out official release notes don't have attributes, > so it doesn't matter? Uh, what are sections? Did previous release notes have it? > I think this commit title "Add hash support functions and hash opclass > for contrib/ltree." > from [1] is more descriptive. Uh, I don't think people know what hash support functions are, but they know what hash indexes are, and maybe hash joins and hash aggregates. Why do you consider the commit text better? > i am not 100% sure of the meaning of "This is useful for extensions." The commit says: commit 2b5154beab7 Author: Tom Lane Date: Fri Oct 20 12:28:38 2023 -0400 Extend ALTER OPERATOR to allow setting more optimization attributes. Allow the COMMUTATOR, NEGATOR, MERGES, and HASHES attributes to be set by ALTER OPERATOR. However, we don't allow COMMUTATOR/NEGATOR to be changed once set, nor allow the MERGES/HASHES flags to be unset once set. Changes like that might invalidate plans already made, and dealing with the consequences seems like more trouble than it's worth. --> The main use-case we foresee for this is to allow addition of missed --> properties in extension update scripts, such as extending an existing --> operator to support hashing. So only transitions from not-set to set states seem very useful. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Thu, May 9, 2024 at 11:31:17AM +0100, Dagfinn Ilmari Mannsåker wrote: > Dagfinn Ilmari Mannsåker writes: > > > Bruce Momjian writes: > > > >> I have committed the first draft of the PG 17 release notes; you can > >> see the results here: > >> > >>https://momjian.us/pgsql_docs/release-17.html > > > > My name is listed twice in the "Improve psql tab completion" item. > > You can move one of them to "Track DEALLOCATE in pg_stat_statements", > which Michael and I co-authored. Yep, also my mistake, fixed. My apologies. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Thu, May 9, 2024 at 11:22:06AM +0100, Dagfinn Ilmari Mannsåker wrote: > Bruce Momjian writes: > > > I have committed the first draft of the PG 17 release notes; you can > > see the results here: > > > > https://momjian.us/pgsql_docs/release-17.html > > My name is listed twice in the "Improve psql tab completion" item. You did such a great job I wanted to list you twice. :-) Actually, the author list was so long I just didn't notice, fixed. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Thu, May 9, 2024 at 06:00:24PM +0800, jian he wrote: > On Thu, May 9, 2024 at 12:04 PM Bruce Momjian wrote: > > > > I have committed the first draft of the PG 17 release notes; you can > > see the results here: > > > > https://momjian.us/pgsql_docs/release-17.html > > > > another potential incompatibilities issue: > ALTER TABLE DROP PRIMARY KEY > > see: > https://www.postgresql.org/message-id/202404181849.6frtmajobe27%40alvherre.pgsql I see it now, and I see Alvaro Herrera saying: https://www.postgresql.org/message-id/202404181849.6frtmajobe27%40alvherre.pgsql > I wonder is there any incompatibility issue, or do we need to say something > about the new behavior when dropping a key column? --> Umm, yeah, maybe we should document it in ALTER TABLE DROP PRIMARY KEY --> and in the release notes to note the different behavior. However, I don't see it mentioned as a release note item in the commit message or mentioned in our docs. I suppose the release note text would be: Removing a PRIMARY KEY will remove the NOT NULL column specification Previously the NOT NULL specification would be retained. Do we have agreement that we want this release note item? -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Thu, May 9, 2024 at 12:18:44PM +0300, Aleksander Alekseev wrote: > Hi, > > > I have committed the first draft of the PG 17 release notes; you can > > see the results here: > > > > https://momjian.us/pgsql_docs/release-17.html > > > > It will be improved until the final release. The item count is 188, > > which is similar to recent releases: > > Thanks for working on this. > > I believe the part of the 64-bit XIDs patchset that was delivered in > PG17 is worth highlighting in "E.1.3.10. Source Code" section: > > 4ed8f0913bfd > 2cdf131c46e6 > 5a1dfde8334b > a60b8a58f435 > > All this can probably be summarized as one bullet "Index SLRUs by > 64-bit integers rather than by 32-bit ones" where the authors are: > Maxim Orlov, Aleksander Alekseev, Alexander Korotkov, Teodor Sigaev, > Nikita Glukhov, Pavel Borisov, Yura Sokolov. Wow, I try to only list source code items that have some user-facing impact, and I don't think these do. I do realize how important they are though. This gets into the balance of mentioning items _users_ need to know about, vs. important improvements that _we_ know about. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Thu, May 9, 2024 at 02:37:57PM +0800, Richard Guo wrote: > > On Thu, May 9, 2024 at 12:04 PM Bruce Momjian wrote: > > I have committed the first draft of the PG 17 release notes; you can > see the results here: > > https://momjian.us/pgsql_docs/release-17.html > > > Thanks for working on that. > > For this item: > > > Allow the optimizer to improve CTE plans by using the sort order of > columns referenced in earlier CTE clauses (Jian Guo) > > > I think you mean a65724dfa. The author should be 'Richard Guo'. Wow the CTE item above it was done by Jian Guo. I probably copied the text from the line above it, modified the description, but thought the author's name was the same, but it was not. Fixed. > And I'm wondering if it is more accurate to state it as "Allow the > optimizer to improve plans for the outer query by leveraging the sort > order of a CTE's output." > > I think maybe a similar revision can be applied to the item just above > this one. Okay, I went with this text: Allow the optimizer to improve CTE plans by considering the statistics of columns referenced in earlier row output clauses (Jian Guo, Tom Lane) Allow the optimizer to improve CTE plans by considering the sort order of columns referenced in earlier row output clauses (Richard Guo) I did not use "leveraging" because I am concerned non-native English speakers might find the term confusing. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Thu, May 9, 2024 at 02:17:12PM +0900, Masahiko Sawada wrote: > Hi, > > On Thu, May 9, 2024 at 1:03 PM Bruce Momjian wrote: > > > > I have committed the first draft of the PG 17 release notes; you can > > see the results here: > > > > https://momjian.us/pgsql_docs/release-17.html > > Thank you for working on that! > > I'd like to mention some of my works. I think we can add the vacuum > performance improvements by the following commits: > > - Add template for adaptive radix tree (ee1b30f1) > - Add TIDStore, to store sets of TIDs (ItemPointerData) efficiently > (30e144287) > - Use TidStore for dead tuple TIDs storage during lazy vacuum (667e65aac) Okay, I reworded the item, added authors, and added the commits: Allow vacuum to more efficiently remove and freeze tuples (John Naylor, Masahiko Sawada, Melanie Plageman) > Also, please consider the following item: > > - Improve eviction algorithm in ReorderBuffer using max-heap for many > subtransactions (5bec1d6bc) I looked at that item and I don't have a generic "make logical replication apply faster" item to merge it into, and many subtransactions seemed like enough of an edge-case that I didn't think mentioning it make sense. Can you see a good place to add it? > Finally, should we mention the following commit in the release note? > It's not a user-visible change but added a new regression test module. > > - Add tests for XID wraparound (e255b646a) I don't normally add testing infrastructure changes unless they are major. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Thu, May 9, 2024 at 04:53:38AM +, Bertrand Drouvot wrote: > Hi, > > On Thu, May 09, 2024 at 12:03:50AM -0400, Bruce Momjian wrote: > > I have committed the first draft of the PG 17 release notes; you can > > see the results here: > > > > https://momjian.us/pgsql_docs/release-17.html > > Thanks for working on that! > > > I welcome feedback. > > > Add system view pg_wait_events that reports wait event types (Michael > > Paquier) > > Michael is the committer for 1e68e43d3f, the author is me. Wow, thank you for finding that. The commit message is very clear so I don't know how I made that mistake. Fixed. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Thu, May 9, 2024 at 04:52:14PM +1200, David Rowley wrote: > On Thu, 9 May 2024 at 16:47, Muhammad Ikram wrote: > > A minor formatting issue in the start below. Bullet is not required here. > > This is a placeholder for the highlight features of v17 will go. > Bruce tends not to decide what those are all by himself. Yes, I already have so much of my opinion in the release notes that I prefer others to make that list, and to make the Acknowledgments list at the bottom. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Thu, May 9, 2024 at 09:47:34AM +0500, Muhammad Ikram wrote:
> Hi Bruce,
>
> A minor formatting issue in the start below. Bullet is not required here.
>
>
> E.1.1. Overview
>
> PostgreSQL 17 contains many new features and enhancements, including:
>
> •
>
> The above items and other new features of PostgreSQL 17 are explained in more
> detail in the sections below.
That is just a place-holder. I changed the bullet text to be:
TO BE COMPLETED LATER
--
Bruce Momjian https://momjian.us
EDB https://enterprisedb.com
Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Thu, May 9, 2024 at 04:44:47PM +1200, David Rowley wrote:
> On Thu, 9 May 2024 at 16:04, Bruce Momjian wrote:
> > I welcome feedback. For some reason it was an easier job than usual.
>
> Thanks for working on that.
>
> > +2023-11-02 [cac169d68] Increase DEFAULT_FDW_TUPLE_COST from 0.01 to 0.2
>
> > +Double the default foreign data wrapper tuple cost (David Rowley, Umair
> > Shahid)
>
> That's 20x rather than 2x.
Oops, changed to:
Increase the default foreign data wrapper tuple cost (David
Rowley, Umair Shahid)
--
Bruce Momjian https://momjian.us
EDB https://enterprisedb.com
Only you can decide what is important to you.
First draft of PG 17 release notes
I have committed the first draft of the PG 17 release notes; you can see the results here: https://momjian.us/pgsql_docs/release-17.html It will be improved until the final release. The item count is 188, which is similar to recent releases: release-10: 189 release-11: 170 release-12: 180 release-13: 178 release-14: 220 release-15: 184 release-16: 206 release-17: 188 I welcome feedback. For some reason it was an easier job than usual. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: PERIOD foreign key feature
On Wed, May 8, 2024 at 02:29:34PM +0200, Peter Eisentraut wrote: > > > Finally, if the foreign key has a PERIOD column_name specification > > > the corresponding refcolumn, if present, must also be marked > > > PERIOD. If the refcolumn clause is omitted, and thus the reftable's > > > primary key constraint chosen, the primary key must have its final > > > column marked WITHOUT OVERLAPS. > > > > Yes, David is correct here on all points. I like his suggestion to > > clarify the language here also. If you need a patch from me let me know, > > but I assume it's something a committer can just make happen? > > In principle yes, but it's also very helpful if someone produces an actual > patch file, with complete commit message, credits, mailing list link, etc. I am ready to do the work, but waited a day for Peter to reply, since he was the author of the text. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: AIX support
On Wed, May 8, 2024 at 03:44:12PM +0200, Peter Eisentraut wrote: > On 08.05.24 13:39, Sriram RK wrote: > > We would like to understand your inputs/plans on reverting the changes > > for AIX. > > I think the ship has sailed for PG17. The way forward would be that you > submit a patch for new, modernized AIX support for PG18. Yes, I think we were clear that someone needs to review the reverted patch and figure out which parts are still needed, and why. We have no "plans" to restore support. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
PERIOD foreign key feature
In this commit:
commit 34768ee3616
Author: Peter Eisentraut
Date: Sun Mar 24 07:37:13 2024 +0100
Add temporal FOREIGN KEY contraints
Add PERIOD clause to foreign key constraint definitions. This is
supported for range and multirange types. Temporal foreign keys
check
for range containment instead of equality.
This feature matches the behavior of the SQL standard temporal
foreign
keys, but it works on PostgreSQL's native ranges instead of SQL's
"periods", which don't exist in PostgreSQL (yet).
Reference actions ON {UPDATE,DELETE} {CASCADE,SET NULL,SET DEFAULT}
are not supported yet.
Author: Paul A. Jungwirth
Reviewed-by: Peter Eisentraut
Reviewed-by: jian he
Discussion:
https://www.postgresql.org/message-id/flat/CA+renyUApHgSZF9-nd-a0+OPGharLQLO=mdhcy4_qq0+noc...@mail.gmail.com
this text was added to create_table.sgml:
In addition, the referenced table must have a primary
key or unique constraint declared with WITHOUT
--> OVERLAPS. Finally, if one side of the foreign key
--> uses PERIOD, the other side must too. If the
refcolumn list is
omitted, the WITHOUT OVERLAPS part of the
primary key is treated as if marked with PERIOD.
In the two marked lines, it says "if one side of the foreign key uses
PERIOD, the other side must too." However, looking at the example
queries, it seems like if the foreign side has PERIOD, the primary side
must have WITHOUT OVERLAPS, not PERIOD.
Does this doc text need correcting?
--
Bruce Momjian https://momjian.us
EDB https://enterprisedb.com
Only you can decide what is important to you.
Re: Removing unneeded self joins
On Mon, May 6, 2024 at 12:24:41PM -0400, Robert Haas wrote: > Now that being said, I do also agree that the planner code is quite > hard to understand, for various reasons. I don't think the structure > of that code and the assumptions underlying it are as well-documented > as they could be, and neither do I think that all of them are optimal. > It has taken me a long time to learn as much as I know, and there is > still quite a lot that I don't know. And I also agree that the planner > does an unfortunate amount of in-place modification of existing > structures without a lot of clarity about how it all works, and an > unfortunate amount of data copying in some places, and even that the > partition-wise join code isn't all that it could be. But I do not > think that adds up to a conclusion that we should just be less > ambitious with planner changes. Indeed, I would like to see us do > more. There is certainly a lot of useful work that could be done. The > trick is figuring out how to do it without breaking too many things, > and that is not easy. I agree with Robert. While writting the Postgres 17 release notes, I am excited to see the many optimizer improvements, and removing self-joins from that list will be unfortunate. I did write a blog entry in 2021 that suggested we could have optimizer aggressiveness control to allow for more expensive optimizations: https://momjian.us/main/blogs/pgblog/2021.html#May_14_2021 -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: Removing unneeded self joins
On Mon, May 6, 2024 at 10:44:33AM -0400, Robert Haas wrote: > I want to go on record right now as disagreeing with the plan proposed > in the commit message for the revert commit, namely, committing this > again early in the v18 cycle. I don't think Tom would have proposed > reverting this feature unless he believed that it had more serious > problems than could be easily fixed in a short period of time. I think > that concern is well-founded, given the number of fixes that were > committed. It seems likely that the patch needs significant rework and > stabilization before it gets committed again, and I think it shouldn't > be committed again without explicit agreement from Tom or one of the > other committers who have significant experience with the query > planner. That is not to say that I don't approve generally of the idea > of committing things earlier in the release cycle: I certainly do. It > gives us more time to shake out problems with patches before we ship. > But it only makes sense if we collectively believe that the patch is > mostly correct, and only needs fine-tuning, and I think there are good > reasons to believe that we shouldn't have that level of confidence in > this case. I think what Robert is saying is that it is an unacceptable plan to just dump the code into PG 18 and clean it up in the following months --- it needs more research before it is re-added to git. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: Help update PostgreSQL 13.12 to 13.14
iption Management repositories. > > This system is registered with an entitlement > server, but is not receiving updates. You can use > subscription-manager to assign subscriptions. > > Last metadata expiration check: 0:07:02 ago on Fri > 26 Apr 2024 10:01:36 AM CST. > Dependencies resolved. > Nothing to do. > Complete! > > > It seemed yum is not able to get the latest package > update, try clearing the cache and rebuilding it > > yum clean all > > yum makecache > > > > > El jue, 25 abr 2024 a las 23:16, Kashif Zeeshan (< > kashi.zees...@gmail.com>) escribió: > > > > On Thu, Apr 25, 2024 at 11:55 PM •Isaac Rv < > isaacrodr...@gmail.com> wrote: > > Entiendo si, me han dicho que es sencillo, > pero no entiendo si solo descargo los > binarios y en cual carpeta reemplazo? no > hay una guía cómo tal de cómo realizarlo, > me podrías ayudar? > > > Follow the below steps > 1. Backup your data > 2. Review the release notes of the update > release > 3. Stop the PG Server > 4. Upgrade postgres to newer version, e.g. on > CentOS use the command 'sudo yum update > postgresql' > 5. Restart PG Server > > Thanks > Kashif Zeeshan > Bitnine Global > > > El jue, 25 abr 2024 a las 11:20, Kashif > Zeeshan () > escribió: > > Hi Isaac > > You are doing the minor version > upgrade > so it's not a big effort as compared > to > major version upgrade, following is > the > process to do it. > > Minor releases never change the > internal storage format and are always > compatible with earlier and later > minor > releases of the same major version > number. For example, version 10.1 is > compatible with version 10.0 and > version 10.6. Similarly, for example, > 9.5.3 is compatible with 9.5.0, 9.5.1, > and 9.5.6. To update between > compatible > versions, you simply replace the > executables while the server is down > and restart the server. The data > directory remains unchanged — minor > upgrades are that simple. > > > Please follow the links below for more > information. > > https://www.postgresql.org/docs/13/ > upgrading.html > https://www.postgresql.org/support/ > versioning/ > > > Thanks > Kashif Zeeshan > Bitnine Global > > On Thu, Apr 25, 2024 at 9:37 PM •Isaac > Rv wrote: > > Hello everyone, I hope you're doing > well. Does anyone have a guide or know > how to perform an upgrade from > PostgreSQL 13.12 to 13.14 on Linux? > I've searched in various places but > haven't found any solid guides, and > truth be told, I'm a bit of a novice > with PostgreSQL. Any help would be > appreciated. > -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: AIX support
On Thu, Apr 25, 2024 at 01:06:24PM +0900, Michael Paquier wrote: > Anyway, getting an access to such compilers to be able to debug issues > on hosts that take less than 12h to just compile the code would > certainly help its adoption. So seeing commitment in the form of > patches and access to environments would help a lot. Overall, > studying that afresh with v18 looks like a good idea, assuming that > anybody who commits such patches has access to hosts to evaluate them, > with buildfarm members running on top, of course. Agreed. They can't even have buildfarm member for PG 17 since it doesn't compile anymore, so someone has to go over the reverted patch, figure out which ones are still valid, and report back. Trying to add a port, with possible breakage, during beta seems too risky compared to the value. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: AIX support
On Thu, Apr 25, 2024 at 10:16:34AM +0200, Álvaro Herrera wrote: > On 2024-Apr-24, Bruce Momjian wrote: > > > I agree that targeting PG 18 for a new-er AIX port is the reasonable > > approach. If there is huge demand, someone can create an AIX fork for > > PG 17 using the reverted patches --- yeah, lots of pain there, but we > > have carried the AIX pain for too long with too little support. > > I'm not sure how large the demand would be for an AIX port specifically > of 17, though. I mean, people using older versions can continue to use > 16 until 18 is released. Upgrading past one major version is hardly > unheard of. Agreed. They seem to have packages for 11/12, and only 15 recently. I don't see how PG 17 would be missed, unless there are many people compiling from source. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: AIX support
On Sat, Apr 20, 2024 at 12:25:47PM -0400, Tom Lane wrote: > > I can see several ways going forward: > > 1. We revert the removal of AIX support and carry on with the status quo > > ante. (The removal of AIX is a regression; it is timely and in scope > > now to revert the change.) > > 2. Like (1), but we consider that notice has been given, and we will > > remove it early in PG18 (like August) unless the situation improves. > > 3. We leave it out of PG17 and consider a new AIX port for PG18 on its > > own merits. > > Andres has ably summarized the reasons why the status quo ante was > getting untenable. The direct-I/O problem could have been tolerable > on its own, but in reality it was the straw that broke the camel's > back so far as our willingness to maintain AIX support went. There > were just too many hacks and workarounds for too many problems, > with too few people interested in looking for better answers. > > So I'm totally not in favor of #1, at least not without some hard > commitments and follow-through on really cleaning up the mess > (which maybe looks more like your #2). What's needed here, as > you said, is for someone with a decent amount of expertise in > modern AIX to review all the issues. Maybe framing that as a > "new port" per #3 would be a good way to think about it. But > I don't want to just revert the AIX-ectomy and continue drifting. > > On the whole, it wouldn't be the worst thing in the world if PG 17 > lacks AIX support but that comes back in PG 18. That approach would > solve the schedule-crunch aspect and give time for considered review > of how many of the hacks removed in 0b16bb877 really need to be put > back, versus being obsolete or amenable to a nicer solution in > late-model AIX. If we take a "new port" mindset then it would be > totally reasonable to say that it only supports very recent AIX > releases, so I'd hope at least some of the cruft could be removed. I agree that targeting PG 18 for a new-er AIX port is the reasonable approach. If there is huge demand, someone can create an AIX fork for PG 17 using the reverted patches --- yeah, lots of pain there, but we have carried the AIX pain for too long with too little support. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: Statistics Import and Export
On Tue, Apr 23, 2024 at 06:33:48PM +0200, Matthias van de Meent wrote: > I've heard of use cases where dumping stats without data would help > with production database planner debugging on a non-prod system. > > Sure, some planner inputs would have to be taken into account too, but > having an exact copy of production stats is at least a start and can > help build models and alerts for what'll happen when the tables grow > larger with the current stats. > > As for other planner inputs: table size is relatively easy to shim > with sparse files; cumulative statistics can be copied from a donor > replica if needed, and btree indexes only really really need to > contain their highest and lowest values (and need their height set > correctly). Is it possible to prevent stats from being updated by autovacuum and other methods? -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: documentation structure
On Tue, Apr 16, 2024 at 03:05:32PM -0400, Tom Lane wrote: > Andres Freund writes: > > I think we should work on generating a lot of func.sgml. Particularly the > > signature etc should just come from pg_proc.dat, it's pointlessly painful to > > generate that by hand. And for a lot of the functions we should probably > > move > > the existing func.sgml comments to the description in pg_proc.dat. > > Where are you going to get the examples and text descriptions from? > (And no, I don't agree that the pg_description string should match > what's in the docs. The description string has to be a short > one-liner in just about every case.) > > This sounds to me like it would be a painful exercise with not a > lot of benefit in the end. Maybe we could _verify_ the contents of func.sgml against pg_proc. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: Reports on obsolete Postgres versions
On Thu, Apr 4, 2024 at 04:51:50PM -0400, Bruce Momjian wrote: > On Thu, Apr 4, 2024 at 04:38:10PM -0400, Greg Sabino Mullane wrote: > > On Thu, Apr 4, 2024 at 2:23 PM Bruce Momjian wrote: > > +Such upgrades might require manual changes to complete so always read > > +the release notes first. > > > > Proposal: > > "Such upgrades might require additional steps, so always read the release > > notes > > first." > > Yes, I modified that sentence. > > > I went with frequently-encountered and low risk bugs". > > > > But neither of those classifications are really true. Especially the "low > > risk" > > part - I could see various ways a reader could wrongly interpret that. > > I see your point. Updated patch attached. I am ready to apply this patch to the website. How do I do this? Do I just commit this to the pgweb git tree? Does that push to the website? -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: Typos in the code and README
On Fri, Apr 12, 2024 at 04:55:16PM -0400, Bruce Momjian wrote: > On Thu, Apr 11, 2024 at 03:37:00PM +0200, Daniel Gustafsson wrote: > > On 11 Apr 2024, at 15:29, David Rowley wrote: > > > > On Fri, 12 Apr 2024, 1:05 am Daniel Gustafsson, wrote: > > > > Now that the tree has settled down a bit post-freeze I ran some > > tooling > > to > > check spelling. I was primarily interested in docs and README* > > which > > were > > mostly free from simply typos, while the code had some in various > > comments and > > one in code. The attached fixes all that I came across (not > > cross-referenced > > against ongoing reverts or any other fixup threads but will be > > before > > pushing > > of course). > > > > > > I see you've corrected "iif" to "if". It should be "iff". > > > > > > Gotcha, will fix. I opted for "if" due to recent threads where the use of > > "iff" was discouraged due to not being commonly known, but that was in doc/ > > and > > not code comments. > > I actually agree "iff" is just not clear enough. "Iff" stands for "if > and only if" and maybe should be spelled out that way. Just to clarify, I think "if and only if" means "if A then B" and B can only happen if A happens, meaning there are not other cases where B can happen. This latter part is what disinguishes "iff" from "if". -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: Typos in the code and README
On Thu, Apr 11, 2024 at 03:37:00PM +0200, Daniel Gustafsson wrote: > On 11 Apr 2024, at 15:29, David Rowley wrote: > > On Fri, 12 Apr 2024, 1:05 am Daniel Gustafsson, wrote: > > Now that the tree has settled down a bit post-freeze I ran some > tooling > to > check spelling. I was primarily interested in docs and README* which > were > mostly free from simply typos, while the code had some in various > comments and > one in code. The attached fixes all that I came across (not > cross-referenced > against ongoing reverts or any other fixup threads but will be before > pushing > of course). > > > I see you've corrected "iif" to "if". It should be "iff". > > > Gotcha, will fix. I opted for "if" due to recent threads where the use of > "iff" was discouraged due to not being commonly known, but that was in doc/ > and > not code comments. I actually agree "iff" is just not clear enough. "Iff" stands for "if and only if" and maybe should be spelled out that way. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: Table AM Interface Enhancements
On Wed, Apr 10, 2024 at 05:42:51PM +0400, Pavel Borisov wrote: > Hi, Alexander! > In my view, the actual list of what has raised discussion is: > dd1f6b0c17 Provide a way block-level table AMs could re-use > acquire_sample_rows > () > 27bc1772fc Generalize relation analyze in table AM interface > > Proposals to revert the other patches in a wholesale way look to me like an > ill-performed continuation of a discussion [1]. I can't believe that "Let's For reference this disussion was: I don't dispute that we could do better, and this is just a simplistic look based on "number of commits per day", but the attached does put it in perspective to some extent. > select which commits close to FF looks worse than the others" based on > whereabouts, not patch contents is a good and productive way for the community > to use. I don't know how you can say these patches are being questioned just because they are near the feature freeze (FF). There are clear concerns, and post-feature freeze is not the time to be evaluating which patches which were pushed in near feature freeze need help. What is the huge rush for these patches, and if they were so important, why was this not done earlier? This can all wait until PG 18. If Supabase or someone else needs these patches for PG 17, they will need to create a patched verison of PG 17 with these patches. > At the same time if Andres, who is the most experienced person in the scope of > access methods is willing to give his post-commit re-review of any of the > committed patches and will recommend some of them reverted, it would be a good > sensible input to act accordingly. > patch So the patches were rushed, have problems, and now we are requiring Andres to stop what he is doing to give immediate feedback --- that is not fair to him. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: PostgreSQL 17 Release Management Team & Feature Freeze
On Mon, Apr 8, 2024 at 10:41:17AM -0400, Robert Treat wrote: > Unfortunately many humans are hardwired towards procrastination and > last minute heroics (it's one reason why deadline driven development > works even though in the long run it tends to be bad practice), and > historically was one of the driving factors in why we started doing > commitfests in the first place (you should have seen the mad dash of > commits before we had that process), so ISTM that it's not completely > avoidable... > > That said, are you suggesting that the feature freeze deadline be > random, and also held in secret by the RMT, only to be announced after > the freeze time has passed? This feels weird, but might apply enough > deadline related pressure while avoiding last minute shenanigans. Committing code is a hard job, no question. However, committers have to give up the idea that they should wait for brilliant ideas before finalizing patches. If you come up with a better idea later, great, but don't wait to finalize patches. I used to write college papers much too late because I expected some brilliant idea to come to me, and it rarely happened. I learned to write the paper with the ideas I had, and if I come up with a better idea later, I can add it to the end. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: PostgreSQL 17 Release Management Team & Feature Freeze
On Mon, Apr 8, 2024 at 09:32:14PM +0200, Jelte Fennema-Nio wrote: > I'll sketch a situation: There's a big patch that some non-committer > submitted that has been sitting on the mailinglist for 6 months or > more, only being reviewed by other non-committers, which the submitter > quickly addresses. Then in the final commit fest it is finally > reviewed by a committer, and they say it requires significant changes. > Right now, the submitter can decide to quickly address those changes, > and hope to keep the attention of this committer, to hopefully get it > merged before the deadline after probably a few more back and forths. > But this new rule would mean that those significant changes would be a > reason not to put it in the upcoming release. Which I expect would > first of all really feel like a slap in the face to the submitter, > because it's not their fault that those changes were only requested in > the last commit fest. This would then likely result in the submitter > not following up quickly (why make time right at this moment, if > you're suddenly going to have another full year), which would then > cause the committer to lose context of the patch and thus interest in > the patch. And then finally getting into the exact same situation next > year in the final commit fest, when some other committer didn't agree > with the redesign of the first one and request a new one pushing it > another year. Yes, I can see this happening. :-( -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: Security lessons from liblzma
On Sat, Mar 30, 2024 at 04:50:26PM -0400, Robert Haas wrote: > An awful lot of what we do operates on the principle that we know the > people who are involved and trust them, and I'm glad we do trust them, > but the world is full of people who trusted somebody too much and > regretted it afterwards. The fact that we have many committers rather > than a single maintainer probably reduces risk at least as far as the > source repository is concerned, because there are more people paying > attention to potentially notice something that isn't as it should be. One unwritten requirement for committers is that we are able to communicate with them securely. If we cannot do that, they potentially could be forced by others, e.g., governments, to add code to our repositories. Unfortunately, there is on good way for them to communicate with us securely once they are unable to communicate with us securely. I suppose some special word could be used to communicate that status --- that is how it was done in non-electronic communication in the past. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: Fixing backslash dot for COPY FROM...CSV
On Sun, Apr 7, 2024 at 12:00:25AM +0200, Daniel Verite wrote: > Tom Lane wrote: > > > This is sufficiently weird that I'm starting to come around to > > Daniel's original proposal that we just drop the server's recognition > > of \. altogether (which would allow removal of some dozens of lines of > > complicated and now known-buggy code) > > FWIW my plan was to not change anything in the TEXT mode, > but I wasn't aware it had this issue that you found when > \. is not in a line by itself. > > > Alternatively, we could fix it so that \. at the end of a line draws > > "end-of-copy marker corrupt" > > which would at least make things consistent, but I'm not sure that has > > any great advantage. I surely don't want to document the current > > behavioral details as being the right thing that we're going to keep > > doing. > > Agreed we don't want to document that, but also why doesn't \. in the > contents represent just a dot (as opposed to being an error), > just like \a is a? I looked into this and started to realize that \. is the only copy backslash command where we define the behavior only alone at the beginning of a line, and not in other circumstances. The \a example above suggests \. should be period in all other cases, as suggested, but I don't know the ramifications if that. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: Security lessons from liblzma
On Thu, Apr 4, 2024 at 10:56:01PM +0200, Daniel Gustafsson wrote: > > On 4 Apr 2024, at 22:47, Tom Lane wrote: > > > > Robert Haas writes: > >> On Thu, Apr 4, 2024 at 4:25 PM Daniel Gustafsson wrote: > >>> I don't disagree, like I said that very email: it's non-trivial and I > >>> wish we > >>> could make it better somehow, but I don't hav an abundance of good ideas. > > > >> Is the basic issue that we can't rely on the necessary toolchain to be > >> present on every machine where someone might try to build PostgreSQL? > > > > IIUC, it's not really that, but that regenerating these files is > > expensive; multiple seconds even on fast machines. Putting that > > into tests that are run many times a day is unappetizing. > > That's one aspect of it. We could cache the results of course to amortize the > cost over multiple test-runs but at the end of the day it will add time to > test-runs regardless of what we do. > > One thing to consider would be to try and rearrange/refactor the tests to > require a smaller set of keys and certificates. I haven't looked into what > sort of savings that could yield (if any) but if we go the route of > regeneration at test-time we shouldn't leave potential savings on the table. Rather then everyone testing it on every build, couldn't we have an automated test every night that checked binary files. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: Reports on obsolete Postgres versions
On Thu, Apr 4, 2024 at 04:38:10PM -0400, Greg Sabino Mullane wrote: > On Thu, Apr 4, 2024 at 2:23 PM Bruce Momjian wrote: > +Such upgrades might require manual changes to complete so always read > +the release notes first. > > Proposal: > "Such upgrades might require additional steps, so always read the release > notes > first." Yes, I modified that sentence. > I went with frequently-encountered and low risk bugs". > > But neither of those classifications are really true. Especially the "low > risk" > part - I could see various ways a reader could wrongly interpret that. I see your point. Updated patch attached. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you. diff --git a/templates/support/versioning.html b/templates/support/versioning.html index d48e11e0..70e368a6 100644 --- a/templates/support/versioning.html +++ b/templates/support/versioning.html @@ -21,9 +21,8 @@ a release available outside of the minor release roadmap. The PostgreSQL Global Development Group supports a major version for 5 years -after its initial release. After its five year anniversary, a major version will -have one last minor release containing any fixes and will be considered -end-of-life (EOL) and no longer supported. +after its initial release. After this, a final minor version will be released +and the software will then be unsupported (end-of-life). Version Numbering @@ -45,17 +44,10 @@ number, e.g. 9.5.3 to 9.5.4. Upgrading - -We always recommend that all users run the latest available minor -release for whatever major version is in use. - - - - -Major versions usually change the internal format of system tables and data -files. These changes are often complex, so we do not maintain backward -compatibility of all stored data. A dump/reload of the database or use of the -pg_upgrade module is required +Major versions make complex changes, so the contents of the data directory +cannot be maintained in a backward compatible way. A dump/reload of the +database or use of the +pg_upgrade application is required for major upgrades. We also recommend reading the upgrading section of the major version you are planning to upgrade to. You can upgrade from one major version @@ -65,18 +57,24 @@ versions prior to doing so. -Upgrading to a minor release does not normally require a dump and restore; you -can stop the database server, install the updated binaries, and restart the -server. For some releases, manual changes may be required to complete the -upgrade, so always read the release notes before upgrading. +Minor release upgrades do not require a dump and restore; you simply stop +the database server, install the updated binaries, and restart the server. +Such upgrades might require additional steps so always read +the release notes first. -While upgrading will always contain some level of risk, PostgreSQL minor releases -fix only frequently-encountered bugs, security -issues, and data corruption problems to reduce the risk associated with -upgrading. For minor releases, the community considers not upgrading to be -riskier than upgrading. +Minor releases only contain fixes for frequently-encountered bugs, +low-risk fixes, security issues, and +data corruption problems. The community considers performing minor +upgrades to be less risky than continuing to run an old minor version. + + + + +We recommend that users always run the current minor release associated +with their major version. + Releases
Re: Reports on obsolete Postgres versions
On Thu, Apr 4, 2024 at 12:27:32PM -0700, David G. Johnston wrote: > How about this: > """ > Major versions make complex changes, so the contents of the data directory > cannot be maintained in a backward compatible way. A dump and restore of the > databases is required, either done manually or as part of running the href=" > /docs/current/pgupgrade.html">pg_upgrade server application. > """ > > My main change here is to mirror "dump and restore" in both paragraphs and > make > it clear that this action is required and that the unnamed pg_dump/pg_restore > tools or pg_upgrade are used in order to perform this task. Since minor > version upgrades do not require "dump and restore" they need not use these > tools. pg_upgrade only dumps/restores the database schema, which is not something most people would consider dump/restore; see: https://momjian.us/main/writings/pgsql/pg_upgrade.pdf > Also, calling pg_upgrade a module doesn't seem correct. It is found under > server applications in our docs and consists solely of that program (and a > bunch of manual steps) from the user's perspective. Yes, you are correct. It used to be under "modules" and we didn't update this text, partly because this it not in our source git tree; updated patch attached. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you. diff --git a/templates/support/versioning.html b/templates/support/versioning.html index d48e11e0..ad975700 100644 --- a/templates/support/versioning.html +++ b/templates/support/versioning.html @@ -21,9 +21,8 @@ a release available outside of the minor release roadmap. The PostgreSQL Global Development Group supports a major version for 5 years -after its initial release. After its five year anniversary, a major version will -have one last minor release containing any fixes and will be considered -end-of-life (EOL) and no longer supported. +after its initial release. After this, a final minor version will be released +and the software will then be unsupported (end-of-life). Version Numbering @@ -45,17 +44,10 @@ number, e.g. 9.5.3 to 9.5.4. Upgrading - -We always recommend that all users run the latest available minor -release for whatever major version is in use. - - - - -Major versions usually change the internal format of system tables and data -files. These changes are often complex, so we do not maintain backward -compatibility of all stored data. A dump/reload of the database or use of the -pg_upgrade module is required +Major versions make complex changes, so the contents of the data directory +cannot be maintained in a backward compatible way. A dump/reload of the +database or use of the +pg_upgrade application is required for major upgrades. We also recommend reading the upgrading section of the major version you are planning to upgrade to. You can upgrade from one major version @@ -65,18 +57,25 @@ versions prior to doing so. -Upgrading to a minor release does not normally require a dump and restore; you -can stop the database server, install the updated binaries, and restart the -server. For some releases, manual changes may be required to complete the -upgrade, so always read the release notes before upgrading. +Minor release upgrades do not require a dump and restore; you simply stop +the database server, install the updated binaries, and restart the server. +Such upgrades might require manual changes to complete so always read +the release notes first. -While upgrading will always contain some level of risk, PostgreSQL minor releases -fix only frequently-encountered bugs, security -issues, and data corruption problems to reduce the risk associated with -upgrading. For minor releases, the community considers not upgrading to be -riskier than upgrading. +Minor releases only fix frequently-encountered and low-risk bugs, security issues, and data corruption +problems, so such upgrades are very low risk. The community +considers performing minor upgrades to be less risky than continuing to +run an old minor version. + + + + +We recommend that users always run the current minor release associated +with their major version. + Releases
Re: Reports on obsolete Postgres versions
f a good adjective > to put there nor does one seem necessary even if I agree it reads a bit odd > otherwise. I think that has more to do with this being just the wrong > structure to get our point across. Can we pick out some statistics from our > long history of publishing minor releases to present an objective reality to > the reader to convince them to trust our track-record in this matter? I went with frequently-encountered and low risk bugs". Patch attached. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you. diff --git a/templates/support/versioning.html b/templates/support/versioning.html index d48e11e0..aef331a6 100644 --- a/templates/support/versioning.html +++ b/templates/support/versioning.html @@ -21,9 +21,8 @@ a release available outside of the minor release roadmap. The PostgreSQL Global Development Group supports a major version for 5 years -after its initial release. After its five year anniversary, a major version will -have one last minor release containing any fixes and will be considered -end-of-life (EOL) and no longer supported. +after its initial release. After this, a final minor version will be released +and the software will then be unsupported (end-of-life). Version Numbering @@ -45,16 +44,9 @@ number, e.g. 9.5.3 to 9.5.4. Upgrading - -We always recommend that all users run the latest available minor -release for whatever major version is in use. - - - - -Major versions usually change the internal format of system tables and data -files. These changes are often complex, so we do not maintain backward -compatibility of all stored data. A dump/reload of the database or use of the +Major versions make complex changes, so the contents of the data directory +cannot be maintained in a backward compatible way. A dump/reload of the +database or use of the pg_upgrade module is required for major upgrades. We also recommend reading the upgrading section of the major @@ -65,18 +57,25 @@ versions prior to doing so. -Upgrading to a minor release does not normally require a dump and restore; you -can stop the database server, install the updated binaries, and restart the -server. For some releases, manual changes may be required to complete the -upgrade, so always read the release notes before upgrading. +Minor release upgrades do not require a dump and restore; you simply stop +the database server, install the updated binaries, and restart the server. +Such upgrades might require manual changes to complete so always read +the release notes first. -While upgrading will always contain some level of risk, PostgreSQL minor releases -fix only frequently-encountered bugs, security -issues, and data corruption problems to reduce the risk associated with -upgrading. For minor releases, the community considers not upgrading to be -riskier than upgrading. +Minor releases only fix frequently-encountered and low-risk bugs, security issues, and data corruption +problems, so such upgrades are very low risk. The community +considers performing minor upgrades to be less risky than continuing to +run an old minor version. + + + + +We recommend that users always run the current minor release associated +with their major version. + Releases
Re: [PATCH] Modify pg_ctl to detect presence of geek user
On Wed, Apr 3, 2024 at 09:25:02AM -0400, Bruce Momjian wrote: > On Wed, Apr 3, 2024 at 04:17:21PM +0300, Panda Developpeur wrote: > > Dear PostgreSQL Hackers, > > > > I am submitting a patch to modify pg_ctl to detect the presence of a geek > > user > > on the system and adapt its behavior accordingly. This patch introduces the > > following changes: > > > > 1. Detection of geek user: The modified pg_ctl now checks user created on > > the > > computer. > > > > 2. No documentation or tests: Please note that I have not included new > > documentation or tests in this patch submission. However, I am open to > > adding them based on the community's feedback. > > > > 3. Performance impact: The performance impact of these changes is minimal, > > with an expected delay of 500ms in specific scenarios only. > > > > > > Please review the patch and provide your feedback. I am open to making any > > necessary improvements based on the community's suggestions. > > > > Thank you for considering my contribution. > > Aside from an extra newline in the patch, I think this is ready to go! Also, it feels like the deadline for this patch was two days ago. ;-) -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: [PATCH] Modify pg_ctl to detect presence of geek user
On Wed, Apr 3, 2024 at 04:17:21PM +0300, Panda Developpeur wrote: > Dear PostgreSQL Hackers, > > I am submitting a patch to modify pg_ctl to detect the presence of a geek user > on the system and adapt its behavior accordingly. This patch introduces the > following changes: > > 1. Detection of geek user: The modified pg_ctl now checks user created on the > computer. > > 2. No documentation or tests: Please note that I have not included new > documentation or tests in this patch submission. However, I am open to > adding them based on the community's feedback. > > 3. Performance impact: The performance impact of these changes is minimal, > with an expected delay of 500ms in specific scenarios only. > > > Please review the patch and provide your feedback. I am open to making any > necessary improvements based on the community's suggestions. > > Thank you for considering my contribution. Aside from an extra newline in the patch, I think this is ready to go! -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: Reports on obsolete Postgres versions
On Tue, Apr 2, 2024 at 11:34:46AM +0200, Magnus Hagander wrote: > On Tue, Apr 2, 2024 at 9:24 AM Daniel Gustafsson wrote: > A few small comments: > > +considers performing minor upgrades to be less risky than continuing to > +run superseded minor versions. > > I think "superseded minor versions" could be unnecessarily complicated for > non-native speakers, I consider myself fairly used to reading english but > still > had to spend a few extra (brain)cycles parsing the meaning of it in this > context. > > + We recommend that users always run the latest minor release associated > > Or perhaps "current minor release" which is the term we use in the table > below > on the same page? > > I do like the term "current" better. It conveys (at least a bit) that we > really consider all the older ones to be, well, obsolete. The difference > "current vs obsolete" is stronger than "latest vs not quite latest". Okay, I changed "superseded" to "old", and changed "latest" to "current", patch attached. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you. diff --git a/templates/support/versioning.html b/templates/support/versioning.html index d48e11e0..0ed79f4f 100644 --- a/templates/support/versioning.html +++ b/templates/support/versioning.html @@ -45,15 +45,8 @@ number, e.g. 9.5.3 to 9.5.4. Upgrading - -We always recommend that all users run the latest available minor -release for whatever major version is in use. - - - - -Major versions usually change the internal format of system tables and data -files. These changes are often complex, so we do not maintain backward +Major versions usually change the internal format of the system tables. +These changes are often complex, so we do not maintain backward compatibility of all stored data. A dump/reload of the database or use of the pg_upgrade module is required for major upgrades. We also recommend reading the @@ -65,18 +58,25 @@ versions prior to doing so. -Upgrading to a minor release does not normally require a dump and restore; you -can stop the database server, install the updated binaries, and restart the -server. For some releases, manual changes may be required to complete the -upgrade, so always read the release notes before upgrading. +Minor release upgrades do not require a dump and restore; you simply stop +the database server, install the updated binaries, and restart the server. +Such upgrades might require manual changes to complete so always read +the release notes first. -While upgrading will always contain some level of risk, PostgreSQL minor releases -fix only frequently-encountered bugs, security -issues, and data corruption problems to reduce the risk associated with -upgrading. For minor releases, the community considers not upgrading to be -riskier than upgrading. +Minor releases only fix frequently-encountered bugs, security issues, and data corruption +problems, so such upgrades are very low risk. The community +considers performing minor upgrades to be less risky than continuing to +run an old minor version. + + + + +We recommend that users always run the current minor release associated +with their major version. + Releases
Re: Reports on obsolete Postgres versions
On Wed, Mar 13, 2024 at 02:04:16PM -0400, Robert Treat wrote: > I tend to agree with Bruce, and major/minor seems to be the more > common usage within the industry; iirc, debian, ubuntu, gnome, suse, > and mariadb all use that nomenclature; and ISTR some distro's who > release packaged versions of postgres with custom patches applied (ie > 12.4-2 for postgres 12.4 patchlevel 2). > > BTW, as a reminder, we do have this statement, in bold, in the > "upgrading" section of the versioning page: > "We always recommend that all users run the latest available minor > release for whatever major version is in use." There is actually > several other phrases and wording on that page that could probably be > propagated as replacement language in some of these other areas. I ended up writing the attached doc patch. I found that some or our text was overly-wordy, causing the impact of what we were trying to say to be lessened. We might want to go farther than this patch, but I think it is an improvement. I also moved the text to the bottom of the section --- previously, our wording referenced minor releases, then we talked about major releases, and then minor releases. This gives a more natural flow. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you. diff --git a/templates/support/versioning.html b/templates/support/versioning.html index d48e11e0..cee06954 100644 --- a/templates/support/versioning.html +++ b/templates/support/versioning.html @@ -45,15 +45,8 @@ number, e.g. 9.5.3 to 9.5.4. Upgrading - -We always recommend that all users run the latest available minor -release for whatever major version is in use. - - - - -Major versions usually change the internal format of system tables and data -files. These changes are often complex, so we do not maintain backward +Major versions usually change the internal format of the system tables. +These changes are often complex, so we do not maintain backward compatibility of all stored data. A dump/reload of the database or use of the pg_upgrade module is required for major upgrades. We also recommend reading the @@ -65,18 +58,25 @@ versions prior to doing so. -Upgrading to a minor release does not normally require a dump and restore; you -can stop the database server, install the updated binaries, and restart the -server. For some releases, manual changes may be required to complete the -upgrade, so always read the release notes before upgrading. +Minor release upgrades do not require a dump and restore; you simply stop +the database server, install the updated binaries, and restart the server. +Such upgrades might require manual changes to complete so always read +the release notes first. -While upgrading will always contain some level of risk, PostgreSQL minor releases -fix only frequently-encountered bugs, security -issues, and data corruption problems to reduce the risk associated with -upgrading. For minor releases, the community considers not upgrading to be -riskier than upgrading. +Minor releases only fix frequently-encountered bugs, security issues, and data corruption +problems, so such upgrades are very low risk. The community +considers performing minor upgrades to be less risky than continuing to +run superseded minor versions. + + + + +We recommend that users always run the latest minor release associated +with their major version. + Releases
Re: Security lessons from liblzma
On Mon, Apr 1, 2024 at 03:17:55PM -0400, Tom Lane wrote: > Bruce Momjian writes: > > I was more asking if users have access to patches so they could recreate > > the build by using the Postgres git tree and supplied OS-specific > > patches. > > AFAIK, every open-source distro makes all the pieces needed to > rebuild their packages available to users. It wouldn't be much > of an open-source situation otherwise. You do have to learn > their package build process. I wasn't clear if all the projects provide a source tree that can be verified against the project's source tree, and then independent patches, or if the patches were integrated and therefore harder to verify against the project source tree. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: Security lessons from liblzma
On Fri, Mar 29, 2024 at 06:37:24PM -0400, Bruce Momjian wrote: > You might have seen reports today about a very complex exploit added to > recent versions of liblzma. Fortunately, it was only enabled two months > ago and has not been pushed to most stable operating systems like Debian > and Ubuntu. The original detection report is: > > https://www.openwall.com/lists/oss-security/2024/03/29/4 I was watching this video about the exploit: https://www.youtube.com/watch?v=bS9em7Bg0iU and at 2:29, they mention "hero software developer", our own Andres Freund as the person who discovered the exploit. I noticed the author's name at the openwall email link above, but I assumed it was someone else with the same name. They mentioned it was found while researching Postgres performance, and then I noticed the email address matched! I thought the analogy he uses at the end of the video is very clear. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: Statistics Import and Export
On Sun, Mar 31, 2024 at 07:04:47PM -0400, Tom Lane wrote: > Corey Huinker writes: > >> I can't quibble with that view of what has priority. I'm just > >> suggesting that redesigning what pg_upgrade does in this area > >> should come later than doing something about extended stats. > > > I mostly agree, with the caveat that pg_upgrade's existing message saying > > that optimizer stats were not carried over wouldn't be 100% true anymore. > > I think we can tweak the message wording. I just don't want to be > doing major redesign of the behavior, nor adding fundamentally new > monitoring capabilities. I think pg_upgrade could check for the existence of extended statistics in any database and adjust the analyze recommdnation wording accordingly. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: Security lessons from liblzma
On Sun, Mar 31, 2024 at 02:12:57PM -0700, Andres Freund wrote: > Hi, > > On 2024-03-31 12:18:29 +0200, Michael Banck wrote: > > If you ask where they are maintained, the answer is here: > > > > https://salsa.debian.org/postgresql/postgresql/-/tree/17/debian/patches?ref_type=heads > > > > the other major versions have their own branch. > > Luckily these are all quite small, leaving little space to hide stuff. I'd > still like to get rid of at least some of them. > > I've previously proposed a patch to make pkglibdir configurable, I think we > should just go for that. > > For the various defines, ISTM we should just make them #ifndef guarded, then > they could be overridden by defining them at configure time. Some of them, > like DEFAULT_PGSOCKET_DIR seem to be overridden by just about every > distro. And others would be nice to easily override anyway, I e.g. dislike the > default DEFAULT_PAGER value. I realize we can move some changes into our code, but packagers are still going to need a way to do immediate adjustments to match their OS in time frames that don't match the Postgres release schedule. I was more asking if users have access to patches so they could recreate the build by using the Postgres git tree and supplied OS-specific patches. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
