Re: Cdc on HA postgresql

2024-11-26 Thread Gunnar Morling
I don't think you necessarily need two sources. You could put a proxy in front of Postgres (e.g. pgbouncer) which has one fixed address, encapsulating the fact that a failover happens. That's at least what I am exploring/demoing in this blog post. Hit me up off-list if you want to take a preview at

Re: Cdc on HA postgresql

2024-11-26 Thread Lasse Nedergaard
Thanks for all the feedback. As I see it Flink cdc only support one host or. Source. This means we have to create at least two sources in Flink. But if one node is down Flink is failing and we don’t have a HA solution. We have concluded it isn’t possible with the current state of Flink cdc to suppo

Re: Cdc on HA postgresql

2024-11-26 Thread Leonard Xu
Thanks Gunnar for sharing the information, looking forward to your blogpost. Best, Leonard > On Nov 26, 2024, at 3:13 PM, Gunnar Morling > wrote: > > Hey all, > > Postgres 17 supports failover slots, i.e. a replication slot can be created > on the primary instance and its state will be sync

Re: Cdc on HA postgresql

2024-11-25 Thread Gunnar Morling
Hey all, Postgres 17 supports failover slots, i.e. a replication slot can be created on the primary instance and its state will be synched to corresponding slots on a stand-by server. This allows consumers to continue to read from that slot after promoting a stand-by to primary, without missing a

Re: Cdc on HA postgresql

2024-11-25 Thread Hongshun Wang
Hi, Steffen Currently, PostgreSQL database not support this feather. Although slave PostgreSQL can also provide logical replication since PG 16+, the replication slot is not existed when checkout database. If you want to use postgres cdc with HA setup, you have to create slots at all the instanc

Re: Cdc on HA postgresql

2024-11-25 Thread Shengkai Fang
Do you use the latest version? Its doc says it supports incremental reading right now[1]. > As we see it the current version of Flink CDC can’t handle the failover If you configure the checkpoint, I think Flink CDC is able to recover from the checkpoint when something goes wrong. > only support