Re: Transaction Id Space, Freezing and Wraparound

2018-11-20 Thread Martín Fernández
Martín On Tue, Nov 20th, 2018 at 6:0 PM, Tom Lane wrote: > > > > =?UTF-8?q?Mart=C3=ADn_Fern=C3=A1ndez?= < fmarti...@gmail.com > writes: > > First thing that generated a lot of noise in my head was the following, > if pg assigns contiguous numeric values for the txid, how does pg deal > with

Transaction Id Space, Freezing and Wraparound

2018-11-20 Thread Martín Fernández
Hello everyone, second time writing to this awesome mailing list. I’m helping manage a postgresql 9.2.24 high volume transaction database and yesterday we were literally one hour away of having to deal with transaction id wraparound. We were really lucky about identifying the issue one hour

Re: Behaviour when autovacuum is canceled

2018-09-14 Thread Martín Fernández
Alvaro, Thanks for the insight, was really helpful! Best, Martín On Fri, Sep 14th, 2018 at 12:41 PM, Alvaro Herrera wrote: > > > > On 2018-Sep-13, Martín Fernández wrote: > > > By performing this changes we are going to start relying more heavily >

Re: Vacuum not deleting tuples when lockless

2018-09-15 Thread Martín Fernández
Tom & Jerry, Thanks a lot for information! On Monday (weekends don't have the same load patterns compared to business days) I will take a look at ` pg_prepared_xacts` that seems to expose Jerry's suggestion on xacts. Replication slots don't apply to 9.2.X from what I could investigate so I

Vacuum not deleting tuples when lockless

2018-09-14 Thread Martín Fernández
Hello, We are experiencing some `vacuum` issues with a given table (potentially more). When a manual vacuum runs on the given table it seems that the `vacuum` process is not doing the expected cleanup. ``` DETAIL:  113257 dead row versions cannot be removed yet. ``` I've been investigating

Re: Behaviour when autovacuum is canceled

2018-09-13 Thread Martín Fernández
David, Thanks a lot for the quick reply.  I clearly misunderstood the references in the code.  Best, Martín On Thu, Sep 13th, 2018 at 7:55 PM, "David G. Johnston" wrote: > > > On Thu, Sep 13, 2018 at 3:45 PM, Martín Fernández < fmarti...@gmail.com > > wrote

Behaviour when autovacuum is canceled

2018-09-13 Thread Martín Fernández
Hello, I'm working on a high volume transaction database and we are starting to tune our autovacuum setting to improve our vacuuming performance. Once thing that we know about autovacuum is that is can be automatically canceled if a dependent transaction is blocked by the autovacuum

Re: Behaviour when autovacuum is canceled

2018-09-13 Thread Martín Fernández
was performed again.  Thanks! Martín On Thu, Sep 13th, 2018 at 8:0 PM, "Martín Fernández" wrote: > > David, > > > Thanks a lot for the quick reply.  > > > I clearly misunderstood the references in the code.  > > > Best, > > Martín &g

Re: Behaviour when autovacuum is canceled

2018-09-13 Thread Martín Fernández
Tom, Thanks for the detailed explanation. I can start mapping your explanation with the source code I've been reading :) We are in the process of tuning our autovacuum settings (on some tables) and stop relying on crontabs that are performing manual vacuums.  By performing this changes we are

VACUUM FREEZE and replication lag

2019-03-04 Thread Martín Fernández
Hello everyone, We have a very big table in our pg92 database that requires a manual vacuum freeze in order to keep sane number of transaction ids available. Yesterday we did a vacuum freeze on this table that took roughly 9 hours. After performing the operation we got back roughly 0.5 billion

Re: PG Upgrade with hardlinks, when to start/stop master and replicas

2019-02-22 Thread Martín Fernández
On Fri, Feb 22, 2019 at 2:03 AM Tom Lane wrote: > Bruce Momjian writes: > > On Thu, Feb 21, 2019 at 09:31:32PM -0500, Stephen Frost wrote: > >> * Bruce Momjian (br...@momjian.us) wrote: > >>> There was too much concern that users would accidentally start the old > >>> server at some later

Upgrade standby after starting cluster using rsync

2019-03-11 Thread Martín Fernández
Hello, I've wrote a couple of questions around pg_upgrade and updating standbys using rsync last week. We were able to successfully upgrade half of our cluster (the other half was kept for failover) from pg92 with postgis 1.5.8 to pg10 with postgis 2.4. It was a really interesting challenge

Promote replica before being able to accept connections

2019-02-10 Thread Martín Fernández
Hello Everyone! We have a 9.2 pg cluster and we are in the process of rebuilding a master database in our staging environment. In order to achieve the latter goal, we are restoring our staging database using pg_basebackup against one of our production replicas. pg_basebackup has completed and

Re: PG Upgrade with hardlinks, when to start/stop master and replicas

2019-02-19 Thread Martín Fernández
Stephen, Thanks for information! I've refactor our migration scripts to follow the suggestions.  One extra question that popped up. As long as we don't start the standby (after running rsync), we can always `rm -f $PGDATA_10` and promote the standby if necessary for failover right ? We also

Re: PG Upgrade with hardlinks, when to start/stop master and replicas

2019-02-19 Thread Martín Fernández
Stephen, @bilby91 On Tue, Feb 19, 2019 at 1:37 PM Stephen Frost wrote: > Greetings, > > * Martín Fernández (fmarti...@gmail.com) wrote: > > Thanks for information! I've refactor our migration scripts to follow > the suggestions. > > Please don't top-post on these

PG Upgrade with hardlinks, when to start/stop master and replicas

2019-02-18 Thread Martín Fernández
Hello everyone! We are about to upgrade a 6 instance cluster from pg92 to pg10 using pg_upgrade with hardlinks and rsync. Our preliminary tests are working really good so far but on question has popped up that we feel is really critical because it has an important impact on our failover plan.

Re: PG Upgrade with hardlinks, when to start/stop master and replicas

2019-02-18 Thread Martín Fernández
ou can do something similar to what you describe, the only thing that you need to make sure is to no start the standby if you are going to use it as a source of rsync to another standby. Would that make sense ? > > El lun., 18 de feb. de 2019 a la(s) 15:21, Laurenz Albe ( > laurenz.a..

Re: Promote replica before being able to accept connections

2019-02-11 Thread Martín Fernández
Laurenz, That makes sense!  Thanks for the quick reply Best, Martín On Mon, Feb 11th, 2019 at 7:55 AM, Laurenz Albe wrote: > > > > Martín Fernández wrote: > > We have a 9.2 pg cluster and we are in the process of rebuilding a > master database in our staging envi

Logical Replication and table bloat

2020-06-05 Thread Martín Fernández
Hello, Yesterday we stumbled upon a performance issue that we were not expecting. We are replicating our database using AWS DMS which uses logical replication to capture changes. We have some hot tables that get updated very regularly and with the DMS turned on we started noticing that in

Re: Logical Replication, CPU load and Locking contention

2021-03-10 Thread Martín Fernández
> On 10 Mar 2021, at 11:25, Ron wrote: > > On 3/10/21 2:10 AM, Radoslav Nedyalkov wrote: >> >> On Wed, Mar 10, 2021 at 3:56 AM Martín Fernández > <mailto:fmarti...@gmail.com>> wrote: >> Hello, >> >> I’m troubleshooting a problem at my

Logical Replication, CPU load and Locking contention

2021-03-09 Thread Martín Fernández
Hello, I’m troubleshooting a problem at my company with a pg 12 cluster that we run. We are using Amazon DMS to replicate data from our database into S3 buckets. DMS replicates data by using logical replication slots. After introducing DMS in our environment, we have seen an increase in CPU

Reindex "locked" standby database

2021-12-14 Thread Martín Fernández
Hello pg hackers! Today we had to run a `REINDEX table CONCURRENTLY my_table;` in our production database due to considerable index bloat. We used to deal with this problem in the past by using pg_repack but we stopped using it because our data replication tool doesn’t support “re creating”

Re: Reindex "locked" standby database

2021-12-14 Thread Martín Fernández
Micheal, Thanks for much for the quick response. > On 15 Dec 2021, at 00:37, Michael Paquier wrote: > > On Wed, Dec 15, 2021 at 12:15:27AM -0300, Martín Fernández wrote: >> The reindex went fine in the primary database and in one of our >> standby. The other standby

Re: Reindex "locked" standby database

2021-12-14 Thread Martín Fernández
> On 15 Dec 2021, at 00:52, Mladen Gogala wrote: > > On 12/14/21 22:37, Michael Paquier wrote: >> You are referring to the startup process that replays WAL, right? >> Without having an idea about the type of workload your primary and/or >> standbys are facing, as well as an idea of the