Re: Add important info about ANALYZE after create Functional Index

2020-11-16 Thread Bruce Momjian
On Mon, Nov 16, 2020 at 11:59:03AM -0300, Álvaro Herrera wrote: > On 2020-Nov-12, Bruce Momjian wrote: > > > For new expression indexes, it is necessary to run > linkend="sql-analyze">ANALYZE or wait for > > the autovacuum daemon to analyze > > - the table to generate statistics

Re: Add important info about ANALYZE after create Functional Index

2020-11-16 Thread Alvaro Herrera
On 2020-Nov-16, Justin Pryzby wrote: > I see Alvaro already patched the first issue at bcbd77133. > > The problematic language was recently introduced, and I'd reported at: > https://www.postgresql.org/message-id/20201112211143.GL30691%40telsasoft.com > And Erik at: >

Re: Add important info about ANALYZE after create Functional Index

2020-11-16 Thread Alvaro Herrera
On 2020-Nov-12, Bruce Momjian wrote: > For new expression indexes, it is necessary to run linkend="sql-analyze">ANALYZE or wait for > the autovacuum daemon to analyze > - the table to generate statistics about new expression indexes. > + the table to generate statistics for these

Re: Add important info about ANALYZE after create Functional Index

2020-11-16 Thread Justin Pryzby
On Thu, Nov 12, 2020 at 06:01:02PM -0500, Bruce Momjian wrote: > On Thu, Nov 12, 2020 at 03:11:43PM -0600, Justin Pryzby wrote: > > I guess it should say "The system regularly ..." > > > > Also, the last sentence begins "For new expression indexes" and ends with > > "about new expression

Re: Add important info about ANALYZE after create Functional Index

2020-11-12 Thread Bruce Momjian
On Thu, Nov 12, 2020 at 03:11:43PM -0600, Justin Pryzby wrote: > I guess it should say "The system regularly ..." > > Also, the last sentence begins "For new expression indexes" and ends with > "about new expression indexes", which I guess could instead say "about the > expressions". How is this

Re: Add important info about ANALYZE after create Functional Index

2020-11-12 Thread Justin Pryzby
On Mon, Nov 09, 2020 at 06:27:20PM -0500, Bruce Momjian wrote: > On Tue, Oct 27, 2020 at 12:12:00AM -0700, Nikolay Samokhvalov wrote: > > On Mon, Oct 26, 2020 at 3:08 PM Fabrízio de Royes Mello > > wrote: > > > > Would be nice if add some information about it into our docs but not > > sure

Re: Add important info about ANALYZE after create Functional Index

2020-11-12 Thread Bruce Momjian
On Mon, Nov 9, 2020 at 08:35:46PM -0300, Fabrízio de Royes Mello wrote: > > > On Mon, 9 Nov 2020 at 20:27 Bruce Momjian wrote: > > > I see REINDEX CONCURRENTLY was fixed in head, but the docs didn't get > updated to mention the need to run ANALYZE or wait for autovacuum before >

Re: Add important info about ANALYZE after create Functional Index

2020-11-09 Thread Fabrízio de Royes Mello
On Mon, 9 Nov 2020 at 20:27 Bruce Momjian wrote: > > I see REINDEX CONCURRENTLY was fixed in head, but the docs didn't get > updated to mention the need to run ANALYZE or wait for autovacuum before > expression indexes can be fully used by the optimizer. Instead of > putting this mention in the

Re: Add important info about ANALYZE after create Functional Index

2020-11-09 Thread Bruce Momjian
On Tue, Oct 27, 2020 at 12:12:00AM -0700, Nikolay Samokhvalov wrote: > On Mon, Oct 26, 2020 at 3:08 PM Fabrízio de Royes Mello < > fabriziome...@gmail.com> wrote: > > Would be nice if add some information about it into our docs but not sure > where. I'm thinking about: > -

Re: Add important info about ANALYZE after create Functional Index

2020-11-01 Thread Fabrízio de Royes Mello
On Sun, 1 Nov 2020 at 09:29 Michael Paquier wrote: > On Sun, Nov 01, 2020 at 09:23:44AM +0900, Michael Paquier wrote: > > By doing so, there is no need to include pg_statistic.h in index.c. > > Except that, the logic looks fine at quick glance. In the long-term, > > I also think that it would

Re: Add important info about ANALYZE after create Functional Index

2020-11-01 Thread Michael Paquier
On Sun, Nov 01, 2020 at 09:23:44AM +0900, Michael Paquier wrote: > By doing so, there is no need to include pg_statistic.h in index.c. > Except that, the logic looks fine at quick glance. In the long-term, > I also think that it would make sense to move both routnes out of > heap.c into a

Re: Add important info about ANALYZE after create Functional Index

2020-10-31 Thread Justin Pryzby
On Sun, Nov 01, 2020 at 10:11:06AM +0900, Michael Paquier wrote: > On Fri, Oct 30, 2020 at 10:30:13PM -0500, Justin Pryzby wrote: > > (I'm quoting from the commit message of the patch I wrote, which is same as > > your patch). > > (I may have missed something, but you did not send a patch,

Re: Add important info about ANALYZE after create Functional Index

2020-10-31 Thread Michael Paquier
On Fri, Oct 30, 2020 at 10:30:13PM -0500, Justin Pryzby wrote: > (I'm quoting from the commit message of the patch I wrote, which is same as > your patch). (I may have missed something, but you did not send a patch, right?) -- Michael signature.asc Description: PGP signature

Re: Add important info about ANALYZE after create Functional Index

2020-10-31 Thread Michael Paquier
On Sat, Oct 31, 2020 at 07:56:33PM -0300, Fabrízio de Royes Mello wrote: > Even if we won't use it now, IMHO it is more legible to separate this > responsibility into its own CopyStatistics function as attached. By doing so, there is no need to include pg_statistic.h in index.c. Except that, the

Re: Add important info about ANALYZE after create Functional Index

2020-10-31 Thread Fabrízio de Royes Mello
On Fri, Oct 30, 2020 at 3:22 AM Michael Paquier wrote: > > And in spirit, it is possible to address this issue with the patch > attached which copies the set of stats from the old to the new index. Did some tests and everything went ok... some comments below! > For a non-concurrent REINDEX,

Re: Add important info about ANALYZE after create Functional Index

2020-10-30 Thread Justin Pryzby
On Fri, Oct 30, 2020 at 03:22:52PM +0900, Michael Paquier wrote: > On Thu, Oct 29, 2020 at 10:59:52AM +0900, Michael Paquier wrote: > > REINDEX CONCURRENTLY is by design wanted to provide an experience > > transparent to the user similar to what a plain REINDEX would do, at > > least that's the

Re: Add important info about ANALYZE after create Functional Index

2020-10-30 Thread Michael Paquier
On Thu, Oct 29, 2020 at 10:59:52AM +0900, Michael Paquier wrote: > REINDEX CONCURRENTLY is by design wanted to provide an experience > transparent to the user similar to what a plain REINDEX would do, at > least that's the idea behind it, so.. This qualifies as a bug to me, > in spirit. And in

Re: Add important info about ANALYZE after create Functional Index

2020-10-28 Thread Michael Paquier
On Thu, Oct 29, 2020 at 12:02:11AM +0100, Tomas Vondra wrote: > On Wed, Oct 28, 2020 at 05:43:08PM -0300, Fabrízio de Royes Mello wrote: >> 2) REINDEX CONCURRENTLY does not keep statistics (pg_statistc) like a >> regular REINDEX for indexes using expressions and to me it's a bug. Michael >>

Re: Add important info about ANALYZE after create Functional Index

2020-10-28 Thread Tomas Vondra
On Wed, Oct 28, 2020 at 05:43:08PM -0300, Fabrízio de Royes Mello wrote: On Wed, Oct 28, 2020 at 4:35 PM Tomas Vondra wrote: I don't think anyone proposed to do this through autovacuum. There was a reference to auto-analyze but I think that was meant as 'run analyze automatically.' Which

Re: Add important info about ANALYZE after create Functional Index

2020-10-28 Thread Fabrízio de Royes Mello
On Wed, Oct 28, 2020 at 4:35 PM Tomas Vondra wrote: > > I don't think anyone proposed to do this through autovacuum. There was a > reference to auto-analyze but I think that was meant as 'run analyze > automatically.' Which would work in transactions just fine, I think. > Maybe I was not very

Re: Add important info about ANALYZE after create Functional Index

2020-10-28 Thread Tomas Vondra
On Wed, Oct 28, 2020 at 03:18:52PM -0400, Tom Lane wrote: Tomas Vondra writes: On Wed, Oct 28, 2020 at 12:00:54PM -0700, David G. Johnston wrote: Given how simple the manual workaround is not having it be manual seems like it would be safe and straight-forward to implement. Maybe, but I

Re: Add important info about ANALYZE after create Functional Index

2020-10-28 Thread Tom Lane
Tomas Vondra writes: > On Wed, Oct 28, 2020 at 12:00:54PM -0700, David G. Johnston wrote: >> Given how simple the manual workaround is not having it be manual seems >> like it would be safe and straight-forward to implement. > Maybe, but I wouldn't be surprised if it was actually a bit trickier

Re: Add important info about ANALYZE after create Functional Index

2020-10-28 Thread David G. Johnston
On Wed, Oct 28, 2020 at 12:05 PM Tom Lane wrote: > This doesn't seem clearly different from any other situation where > auto-analyze doesn't react fast enough to suit you. > I would not > call it a bug, at least not without a wholesale redefinition of > how auto-analyze is supposed to work.

Re: Add important info about ANALYZE after create Functional Index

2020-10-28 Thread Tomas Vondra
On Wed, Oct 28, 2020 at 03:05:39PM -0400, Tom Lane wrote: Tomas Vondra writes: On Mon, Oct 26, 2020 at 03:46:10PM -0700, David G. Johnston wrote: It would seem preferable to call the lack of auto-analyzing after these operations a bug and back-patch a fix that injects an analyze side-effect

Re: Add important info about ANALYZE after create Functional Index

2020-10-28 Thread Tomas Vondra
On Wed, Oct 28, 2020 at 12:00:54PM -0700, David G. Johnston wrote: On Wed, Oct 28, 2020 at 11:55 AM Tomas Vondra wrote: I agree the lack of stats may be quite annoying and cause issues, but my guess is the chances of backpatching such change are about 0.01%. We have a usable 'workaround'

Re: Add important info about ANALYZE after create Functional Index

2020-10-28 Thread David G. Johnston
On Mon, Oct 26, 2020 at 9:44 PM Nikolay Samokhvalov wrote: > On Mon, Oct 26, 2020 at 7:03 PM David G. Johnston < > david.g.johns...@gmail.com> wrote: > >> On Monday, October 26, 2020, Nikolay Samokhvalov >> wrote: >>> >>> Although, this triggers a question – should ANALYZE be automated in, >>>

Re: Add important info about ANALYZE after create Functional Index

2020-10-28 Thread Tom Lane
Tomas Vondra writes: > On Mon, Oct 26, 2020 at 03:46:10PM -0700, David G. Johnston wrote: >> It would seem preferable to call the lack of auto-analyzing after these >> operations a bug and back-patch a fix that injects an analyze side-effect >> just before their completion. It doesn't have to be

Re: Add important info about ANALYZE after create Functional Index

2020-10-28 Thread David G. Johnston
On Wed, Oct 28, 2020 at 11:55 AM Tomas Vondra wrote: > I agree the lack of stats may be quite annoying and cause issues, but my > guess is the chances of backpatching such change are about 0.01%. We > have a usable 'workaround' for this - manual analyze. > My guess is that it wouldn't be

Re: Add important info about ANALYZE after create Functional Index

2020-10-28 Thread Tomas Vondra
On Mon, Oct 26, 2020 at 03:46:10PM -0700, David G. Johnston wrote: On Mon, Oct 26, 2020 at 3:08 PM Fabrízio de Royes Mello < fabriziome...@gmail.com> wrote: Hi all, As you all already know Postgres supports functions in index expressions (marked as immutable ofc) and for this special index

Re: Add important info about ANALYZE after create Functional Index

2020-10-28 Thread Tomas Vondra
On Tue, Oct 27, 2020 at 11:06:22AM -0300, Fabrízio de Royes Mello wrote: On Mon, Oct 26, 2020 at 7:46 PM David G. Johnston < david.g.johns...@gmail.com> wrote: It would seem preferable to call the lack of auto-analyzing after these operations a bug and back-patch a fix that injects an analyze

Re: Add important info about ANALYZE after create Functional Index

2020-10-28 Thread Fabrízio de Royes Mello
On Wed, Oct 28, 2020 at 2:15 AM Michael Paquier wrote: > > On Tue, Oct 27, 2020 at 11:06:22AM -0300, Fabrízio de Royes Mello wrote: > > When we create a new table or index they will not have statistics until an > > ANALYZE happens. This is the default behaviour and I think is not a big > >

Re: Add important info about ANALYZE after create Functional Index

2020-10-27 Thread Michael Paquier
On Tue, Oct 27, 2020 at 11:06:22AM -0300, Fabrízio de Royes Mello wrote: > When we create a new table or index they will not have statistics until an > ANALYZE happens. This is the default behaviour and I think is not a big > problem here, but we need to add some note on docs about the need of >

Re: Add important info about ANALYZE after create Functional Index

2020-10-27 Thread Fabrízio de Royes Mello
On Tue, Oct 27, 2020 at 4:12 AM Nikolay Samokhvalov wrote: > > On Mon, Oct 26, 2020 at 3:08 PM Fabrízio de Royes Mello < fabriziome...@gmail.com> wrote: >> >> Would be nice if add some information about it into our docs but not sure where. I'm thinking about: >> -

Re: Add important info about ANALYZE after create Functional Index

2020-10-27 Thread Fabrízio de Royes Mello
On Mon, Oct 26, 2020 at 7:46 PM David G. Johnston < david.g.johns...@gmail.com> wrote: > > It would seem preferable to call the lack of auto-analyzing after these operations a bug and back-patch a fix that injects an analyze side-effect just before their completion. It doesn't have to be smart

Re: Add important info about ANALYZE after create Functional Index

2020-10-27 Thread Nikolay Samokhvalov
On Mon, Oct 26, 2020 at 3:08 PM Fabrízio de Royes Mello < fabriziome...@gmail.com> wrote: > Would be nice if add some information about it into our docs but not sure > where. I'm thinking about: > - doc/src/sgml/ref/create_index.sgml > - doc/src/sgml/maintenance.sgml (routine-reindex) >

Re: Add important info about ANALYZE after create Functional Index

2020-10-26 Thread Nikolay Samokhvalov
On Mon, Oct 26, 2020 at 7:03 PM David G. Johnston < david.g.johns...@gmail.com> wrote: > On Monday, October 26, 2020, Nikolay Samokhvalov > wrote: >> >> Although, this triggers a question – should ANALYZE be automated in, say, >> pg_restore as well? >> > > Independent concern. > It's the same

Re: Add important info about ANALYZE after create Functional Index

2020-10-26 Thread David G. Johnston
On Monday, October 26, 2020, Nikolay Samokhvalov wrote: > > > Although, this triggers a question – should ANALYZE be automated in, say, > pg_restore as well? > Independent concern. > > And another question: how ANALYZE needs to be run? If it's under the > user's control, there is an option to

Re: Add important info about ANALYZE after create Functional Index

2020-10-26 Thread Nikolay Samokhvalov
On Mon, Oct 26, 2020 at 3:46 PM David G. Johnston < david.g.johns...@gmail.com> wrote: > It would seem preferable to call the lack of auto-analyzing after these > operations a bug and back-patch a fix that injects an analyze side-effect > just before their completion. It doesn't have to be smart

Re: Add important info about ANALYZE after create Functional Index

2020-10-26 Thread David G. Johnston
On Mon, Oct 26, 2020 at 3:08 PM Fabrízio de Royes Mello < fabriziome...@gmail.com> wrote: > Hi all, > > As you all already know Postgres supports functions in index expressions > (marked as immutable ofc) and for this special index the ANALYZE command > creates some statistics (new pg_statistic

Add important info about ANALYZE after create Functional Index

2020-10-26 Thread Fabrízio de Royes Mello
Hi all, As you all already know Postgres supports functions in index expressions (marked as immutable ofc) and for this special index the ANALYZE command creates some statistics (new pg_statistic entry) about it. The problem is just after creating a new index or rebuilding concurrently (using