Re: Data loss on logical replication, 12.12 to 14.5, ALTER SUBSCRIPTION

2023-01-04 Thread Amit Kapila
On Tue, Jan 3, 2023 at 8:50 PM Michail Nikolaev wrote: > > > Does that by any chance mean you are using a non-community version of > > Postgres which has some other changes? > > It is a managed Postgres service in the general cloud. Usually, such > providers apply some custom minor patches. > The

Re: Data loss on logical replication, 12.12 to 14.5, ALTER SUBSCRIPTION

2023-01-03 Thread Michail Nikolaev
> Does that by any chance mean you are using a non-community version of > Postgres which has some other changes? It is a managed Postgres service in the general cloud. Usually, such providers apply some custom minor patches. The only one I know about - about forbidding of canceling queries while

Re: Data loss on logical replication, 12.12 to 14.5, ALTER SUBSCRIPTION

2023-01-03 Thread Amit Kapila
On Tue, Jan 3, 2023 at 2:14 PM Michail Nikolaev wrote: > > > The point which is not completely clear from your description is the > > timing of missing records. In one of your previous emails, you seem to > > have indicated that the data missed from Table B is from the time when > > the initial

Re: Data loss on logical replication, 12.12 to 14.5, ALTER SUBSCRIPTION

2023-01-03 Thread Michail Nikolaev
Hello, Amid. > The point which is not completely clear from your description is the > timing of missing records. In one of your previous emails, you seem to > have indicated that the data missed from Table B is from the time when > the initial sync for Table B was in-progress, right? Also, from

Re: Data loss on logical replication, 12.12 to 14.5, ALTER SUBSCRIPTION

2023-01-02 Thread Amit Kapila
On Wed, Dec 28, 2022 at 4:52 PM Michail Nikolaev wrote: > > Hello. > > > None of these entries are from the point mentioned by you [1] > > yesterday where you didn't find the corresponding data in the > > subscriber. How did you identify that the entries corresponding to > > that timing were

Re: Data loss on logical replication, 12.12 to 14.5, ALTER SUBSCRIPTION

2022-12-28 Thread Michail Nikolaev
Hello. > None of these entries are from the point mentioned by you [1] > yesterday where you didn't find the corresponding data in the > subscriber. How did you identify that the entries corresponding to > that timing were missing? Some of the before the interval, some after... But the source

Re: Data loss on logical replication, 12.12 to 14.5, ALTER SUBSCRIPTION

2022-12-27 Thread Amit Kapila
On Tue, Dec 27, 2022 at 5:49 PM Michail Nikolaev wrote: > > > Probably a small part of WAL was somehow skipped by logical worker in > all that mess. > None of these entries are from the point mentioned by you [1] yesterday where you didn't find the corresponding data in the subscriber. How did

Re: Data loss on logical replication, 12.12 to 14.5, ALTER SUBSCRIPTION

2022-12-27 Thread Michail Nikolaev
Hello, Amit! > IUC, this is the time when only table B's initial sync was > in-progress. Table A's initial sync was finished by that time and for > Table C, it is yet not started. Yes, it is correct. C was started too, but unsuccessfully (restarted after, see below). > During the time of > the

Re: Data loss on logical replication, 12.12 to 14.5, ALTER SUBSCRIPTION

2022-12-26 Thread Amit Kapila
On Mon, Dec 26, 2022 at 8:50 PM Michail Nikolaev wrote: > > Hello again. > > Just small a fix for: > > > 2022-12-14 09:21:25.705 to > > 2022-12-14 09:49:20.664 (after synchronization start, but before finish). > > Correct values are: > > 2022-12-14 09:49:31.340 > 2022-12-14 09:49:41.683 > > So,

Re: Data loss on logical replication, 12.12 to 14.5, ALTER SUBSCRIPTION

2022-12-26 Thread Michail Nikolaev
Hello again. Just small a fix for: > 2022-12-14 09:21:25.705 to > 2022-12-14 09:49:20.664 (after synchronization start, but before finish). Correct values are: 2022-12-14 09:49:31.340 2022-12-14 09:49:41.683 So, it looks like we lost about 10s of one of the tables WAL.

Data loss on logical replication, 12.12 to 14.5, ALTER SUBSCRIPTION

2022-12-26 Thread Michail Nikolaev
Hello. Just a small story about small data-loss on logical replication. We were logically replicating a 4 TB database from > PostgreSQL 12.12 (Ubuntu 12.12-201-yandex.49163.d86383ed5b) on > x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, > 64-bit to > PostgreSQL