Re: pg_attribute, pg_class, pg_depend grow huge in count and size with multiple tenants.

2020-05-07 Thread Laurenz Albe
On Fri, 2020-05-08 at 03:47 -0300, Avinash Kumar wrote: > > Just set "autovacuum_max_workers" higher. > > No, that wouldn't help. If you just increase autovacuum_max_workers, the > total cost limit of > autovacuum_vacuum_cost_limit (or vacuum_cost_limit) is shared by so many > workers and it > f

Re: pg_attribute, pg_class, pg_depend grow huge in count and size with multiple tenants.

2020-05-07 Thread Avinash Kumar
Hi, On Fri, May 8, 2020 at 3:31 AM Laurenz Albe wrote: > On Thu, 2020-05-07 at 18:17 -0300, Avinash Kumar wrote: > > > The nice thing about separate databases is that it is easy to scale > > > horizontally. > > > > Agreed. But, how about autovacuum ? Workers shift from DB to DB and 500 > cluster

Re: pg_attribute, pg_class, pg_depend grow huge in count and size with multiple tenants.

2020-05-07 Thread Laurenz Albe
On Thu, 2020-05-07 at 18:17 -0300, Avinash Kumar wrote: > > The nice thing about separate databases is that it is easy to scale > > horizontally. > > Agreed. But, how about autovacuum ? Workers shift from DB to DB and 500 > clusters > means you may have to have a lot of manual vacuuming in place

Re: AutoVacuum and growing transaction XID's

2020-05-07 Thread github kran
On Thu, May 7, 2020 at 11:04 PM David Rowley wrote: > On Fri, 8 May 2020 at 13:51, github kran wrote: > > I can't either DROP or ALTER any other tables ( REMOVE Inheritance > for any of old tables where the WRITES are not getting written to). Any of > the ALTER TABLE OR DROP TABLE DDL's a

Re: AutoVacuum and growing transaction XID's

2020-05-07 Thread github kran
Thanks David for your replies. On Thu, May 7, 2020 at 11:01 PM David Rowley wrote: > On Fri, 8 May 2020 at 09:18, github kran wrote: > > 1) We haven't changed anything related to autovacuum except a work_mem > parameter which was increased to 4 GB which I believe is not related to > autovacuum

Re: AutoVacuum and growing transaction XID's

2020-05-07 Thread David Rowley
On Fri, 8 May 2020 at 13:51, github kran wrote: > I can't either DROP or ALTER any other tables ( REMOVE Inheritance for > any of old tables where the WRITES are not getting written to). Any of the > ALTER TABLE OR DROP TABLE DDL's arer not getting exeucted even I WAITED FOR > SEVERAL MI

Re: AutoVacuum and growing transaction XID's

2020-05-07 Thread David Rowley
On Fri, 8 May 2020 at 09:18, github kran wrote: > 1) We haven't changed anything related to autovacuum except a work_mem > parameter which was increased to 4 GB which I believe is not related to > autovacuum It might want to look into increasing vacuum_cost_limit to something well above 200 or

Re: AutoVacuum and growing transaction XID's

2020-05-07 Thread github kran
On Thu, May 7, 2020 at 4:18 PM github kran wrote: > > > On Thu, May 7, 2020 at 1:33 PM Michael Lewis wrote: > >> It is trying to do a vacuum freeze. Do you have autovacuum turned off? >> Any settings changed from default related to autovacuum? >> >> https://www.postgresql.org/docs/9.6/routine-va

Re: Explain plan changes - IN CLAUSE ( Passing direct values Vs INNER Query )

2020-05-07 Thread David Rowley
On Fri, 8 May 2020 at 10:00, David G. Johnston wrote: > > On Thu, May 7, 2020 at 11:07 AM Amarendra Konda wrote: >> >> EXPLAIN (ANALYZE, COSTS, VERBOSE, BUFFERS) SELECT pa.process_activity_id AS >> pa_process_activity_id FROM process_activity pa WHERE pa.app_id = >> '126502930200650' AND pa.c

Re: Explain plan changes - IN CLAUSE ( Passing direct values Vs INNER Query )

2020-05-07 Thread David G. Johnston
On Thu, May 7, 2020 at 11:07 AM Amarendra Konda wrote: > EXPLAIN (ANALYZE, COSTS, VERBOSE, BUFFERS) SELECT pa.process_activity_id > AS pa_process_activity_id FROM process_activity pa WHERE pa.app_id = > '126502930200650' AND pa.created > '1970-01-01 00:00:00' AND EXISTS ( > SELECT 1 FROM proce

Re: Explain plan changes - IN CLAUSE ( Passing direct values Vs INNER Query )

2020-05-07 Thread Tom Lane
"David G. Johnston" writes: > On Thu, May 7, 2020 at 10:49 AM Amarendra Konda > wrote: >> Can you please explain, why it is getting more columns in output, even >> though we have asked for only one column ? >> * Output: pa.process_activity_id, pa.process_activity_type, pa.voice_url, >> pa.process

Re: Explain plan changes - IN CLAUSE ( Passing direct values Vs INNER Query )

2020-05-07 Thread David G. Johnston
On Thu, May 7, 2020 at 10:49 AM Amarendra Konda wrote: > Can you please explain, why it is getting more columns in output, even > though we have asked for only one column ? > > > > * Output: pa.process_activity_id, pa.process_activity_type, pa.voice_url, > pa.process_activity_user_id, pa.app_id,

Re: AutoVacuum and growing transaction XID's

2020-05-07 Thread github kran
On Thu, May 7, 2020 at 1:33 PM Michael Lewis wrote: > It is trying to do a vacuum freeze. Do you have autovacuum turned off? Any > settings changed from default related to autovacuum? > > https://www.postgresql.org/docs/9.6/routine-vacuuming.html > Read 24.1.5. Preventing Transaction ID Wraparoun

Re: pg_attribute, pg_class, pg_depend grow huge in count and size with multiple tenants.

2020-05-07 Thread Avinash Kumar
Hi, On Thu, May 7, 2020 at 6:08 PM Rory Campbell-Lange wrote: > On 07/05/20, Avinash Kumar (avinash.vallar...@gmail.com) wrote: > > >> Our application serves multiple tenants. Each tenant has the schema > > >> with a few hundreds of tables and few functions. > > >> We have 2000 clients so we hav

Re: pg_attribute, pg_class, pg_depend grow huge in count and size with multiple tenants.

2020-05-07 Thread Rory Campbell-Lange
On 07/05/20, Avinash Kumar (avinash.vallar...@gmail.com) wrote: > >> Our application serves multiple tenants. Each tenant has the schema > >> with a few hundreds of tables and few functions. > >> We have 2000 clients so we have to create 2000 schemas in a single > >> database. > > That is one opti

Re: pg_attribute, pg_class, pg_depend grow huge in count and size with multiple tenants.

2020-05-07 Thread Avinash Kumar
Hi, On Thu, May 7, 2020 at 5:18 PM David G. Johnston wrote: > On Thu, May 7, 2020 at 1:05 PM samhitha g > wrote: > >> Our application serves multiple tenants. Each tenant has the schema >> with a few hundreds of tables and few functions. >> We have 2000 clients so we have to create 2000 schemas

Re: pg_attribute, pg_class, pg_depend grow huge in count and size with multiple tenants.

2020-05-07 Thread David G. Johnston
On Thu, May 7, 2020 at 1:05 PM samhitha g wrote: > Our application serves multiple tenants. Each tenant has the schema with a > few hundreds of tables and few functions. > We have 2000 clients so we have to create 2000 schemas in a single > database. > That is one option but I wouldn't say you m

pg_attribute, pg_class, pg_depend grow huge in count and size with multiple tenants.

2020-05-07 Thread samhitha g
Hi experts, Our application serves multiple tenants. Each tenant has the schema with a few hundreds of tables and few functions. We have 2000 clients so we have to create 2000 schemas in a single database. While doing this, i observed that the catalog tables pg_attribute, pg_class, pg_depend grow

Re: Explain plan changes - IN CLAUSE ( Passing direct values Vs INNER Query )

2020-05-07 Thread Virendra Kumar
Here is my thought on why row is not limiting when joined vs why it is limiting when not joined. When not joined and where clause is having IN, it is using index process_activity_process_instance_id_app_id_created_idx which has columns process_instance_id, created which is in order by and hence

Re: Explain plan changes - IN CLAUSE ( Passing direct values Vs INNER Query )

2020-05-07 Thread Adrian Klaver
On 5/7/20 10:49 AM, Amarendra Konda wrote: Hi David, Thanks for the reply.This has optimized number of rows. Yeah, but your execution time has increased an order of magnitude. Not sure if that is what you want. Can you please explain, why it is getting more columns in output, even though

Re: Explain plan changes - IN CLAUSE ( Passing direct values Vs INNER Query )

2020-05-07 Thread Amarendra Konda
Hi Virendra, Thanks for your time. Here is the table and index structure * process_activity* Table "public.process_activity" Column |Type | Modifiers +-+-

Re: AutoVacuum and growing transaction XID's

2020-05-07 Thread Michael Lewis
It is trying to do a vacuum freeze. Do you have autovacuum turned off? Any settings changed from default related to autovacuum? https://www.postgresql.org/docs/9.6/routine-vacuuming.html Read 24.1.5. Preventing Transaction ID Wraparound Failures These may also be of help- https://info.crunchydata

AutoVacuum and growing transaction XID's

2020-05-07 Thread github kran
Hello Team, We are using a PostgreSQL version -9.6.12 version and from last 4 weeks our Transaction ID's (XID's) have increased by 195 million to 341 million transactions. I see the below from pg_stat_activity from the postGreSQL DB. 1) Viewing the pg_stat-activity I noticed that the vacuum qu

Re: Explain plan changes - IN CLAUSE ( Passing direct values Vs INNER Query )

2020-05-07 Thread Amarendra Konda
Hi David, In earlier reply, Over looked another condition, hence please ignore that one Here is the correct one with all the needed conditions. According to the latest one, exists also not limiting rows from the process_activity table. EXPLAIN (ANALYZE, COSTS, VERBOSE, BUFFERS) SELECT pa.proce

Re: Explain plan changes - IN CLAUSE ( Passing direct values Vs INNER Query )

2020-05-07 Thread Amarendra Konda
Hi David, Thanks for the reply.This has optimized number of rows. Can you please explain, why it is getting more columns in output, even though we have asked for only one column ? EXPLAIN (ANALYZE, COSTS, VERBOSE, BUFFERS) SELECT pa.process_activity_id AS pa_process_activity_id FROM process_

Re: Explain plan changes - IN CLAUSE ( Passing direct values Vs INNER Query )

2020-05-07 Thread Amarendra Konda
Hi Adrian, Thanks for the reply. And i have kept latest execution plans, for various SQL statements ( inner join, sub queries and placing values instead of sub query) . As suggested, tried with INNER JOIN, however result was similar to subquery. Is there any way we can tell the optimiser to proc

Re: Explain plan changes - IN CLAUSE ( Passing direct values Vs INNER Query )

2020-05-07 Thread David G. Johnston
On Thu, May 7, 2020 at 7:40 AM Adrian Klaver wrote: > On 5/7/20 4:19 AM, Amarendra Konda wrote: > > Hi, > > > > PostgreSQL version : PostgreSQL 9.6.2 on x86_64-pc-linux-gnu, compiled > > by gcc (GCC) 4.8.3 20140911 (Red Hat 4.8.3-9), 64-bit > > > > We have noticed huge difference interms of execu

Re: Explain plan changes - IN CLAUSE ( Passing direct values Vs INNER Query )

2020-05-07 Thread Adrian Klaver
On 5/7/20 4:19 AM, Amarendra Konda wrote: Hi, PostgreSQL version : PostgreSQL 9.6.2 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.8.3 20140911 (Red Hat 4.8.3-9), 64-bit We have noticed huge difference interms of execution plan ( response time) , When we pass the direct values  Vs  inner qu

Explain plan changes - IN CLAUSE ( Passing direct values Vs INNER Query )

2020-05-07 Thread Amarendra Konda
Hi, PostgreSQL version : PostgreSQL 9.6.2 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.8.3 20140911 (Red Hat 4.8.3-9), 64-bit We have noticed huge difference interms of execution plan ( response time) , When we pass the direct values Vs inner query to IN clause. High level details of the us