On Tue, Jul 4, 2017 at 12:21 PM, Peter Eisentraut
wrote:
> On 6/25/17 06:35, Petr Jelinek wrote:
>> - Do LockSharedObject in ALTER SUBSCRIPTION, DROP SUBSCRIPTION (this one
>> is preexisting but mentioning it for context), SetSubscriptionRelState,
>> AddSubscriptionRelState, and in the logicalrep_
On 6/25/17 06:35, Petr Jelinek wrote:
> - Do LockSharedObject in ALTER SUBSCRIPTION, DROP SUBSCRIPTION (this one
> is preexisting but mentioning it for context), SetSubscriptionRelState,
> AddSubscriptionRelState, and in the logicalrep_worker_launch. This means
> we use granular per object locks to
On Wed, Jun 28, 2017 at 2:13 PM, Masahiko Sawada wrote:
> On Wed, Jun 28, 2017 at 1:47 AM, Petr Jelinek
> wrote:
>>
>> On 27/06/17 10:51, Masahiko Sawada wrote:
>>> On Mon, Jun 26, 2017 at 12:12 PM, Masahiko Sawada
>>> wrote:
>>>
>>> I've reviewed this patch briefly.
>>
>> Thanks!
>>
>>>
>>> @@
On 6/29/17 23:39, Noah Misch wrote:
> IMMEDIATE ATTENTION REQUIRED. This PostgreSQL 10 open item is long past due
> for your status update. Please reacquaint yourself with the policy on open
> item ownership[1] and then reply immediately. If I do not hear from you by
> 2017-07-01 04:00 UTC, I wi
On Wed, Jun 28, 2017 at 03:22:18AM +, Noah Misch wrote:
> On Fri, Jun 23, 2017 at 09:42:10PM -0400, Peter Eisentraut wrote:
> > On 6/21/17 22:47, Peter Eisentraut wrote:
> > > On 6/20/17 22:44, Noah Misch wrote:
> > >>> A patch has been posted, and it's being reviewed. Next update Monday.
> >
On Wed, Jun 28, 2017 at 1:47 AM, Petr Jelinek
wrote:
>
> On 27/06/17 10:51, Masahiko Sawada wrote:
>> On Mon, Jun 26, 2017 at 12:12 PM, Masahiko Sawada
>> wrote:
>>
>> I've reviewed this patch briefly.
>
> Thanks!
>
>>
>> @@ -515,6 +533,31 @@ logicalrep_worker_stop(Oid subid, Oid relid)
>> }
>>
On Fri, Jun 23, 2017 at 09:42:10PM -0400, Peter Eisentraut wrote:
> On 6/21/17 22:47, Peter Eisentraut wrote:
> > On 6/20/17 22:44, Noah Misch wrote:
> >>> A patch has been posted, and it's being reviewed. Next update Monday.
> >>
> >> This PostgreSQL 10 open item is past due for your status updat
On 27/06/17 10:51, Masahiko Sawada wrote:
> On Mon, Jun 26, 2017 at 12:12 PM, Masahiko Sawada
> wrote:
>
> I've reviewed this patch briefly.
Thanks!
>
> @@ -515,6 +533,31 @@ logicalrep_worker_stop(Oid subid, Oid relid)
> }
>
> /*
> + * Request worker to be stopped on commit.
> + */
> +voi
On Mon, Jun 26, 2017 at 12:12 PM, Masahiko Sawada wrote:
> On Sun, Jun 25, 2017 at 7:35 PM, Petr Jelinek
> wrote:
>> (was away for a while, got some time now for this again)
>>
>> On 22/06/17 04:43, Peter Eisentraut wrote:
>>> The alternative is that we use the LockSharedObject() approach that wa
On Sun, Jun 25, 2017 at 7:35 PM, Petr Jelinek
wrote:
> (was away for a while, got some time now for this again)
>
> On 22/06/17 04:43, Peter Eisentraut wrote:
>> The alternative is that we use the LockSharedObject() approach that was
>> already alluded to, like in the attached patch. Both approac
(was away for a while, got some time now for this again)
On 22/06/17 04:43, Peter Eisentraut wrote:
> The alternative is that we use the LockSharedObject() approach that was
> already alluded to, like in the attached patch. Both approaches would
> work equally fine AFAICT.
I agree, but I think w
On 6/21/17 22:47, Peter Eisentraut wrote:
> On 6/20/17 22:44, Noah Misch wrote:
>>> A patch has been posted, and it's being reviewed. Next update Monday.
>>
>> This PostgreSQL 10 open item is past due for your status update. Kindly send
>> a status update within 24 hours, and include a date for y
On 6/22/17 03:26, Masahiko Sawada wrote:
> Thank you for the patch. Some comment and question.
> DropSubscriptionRelState requests ExclusiveLock but why is it not
> ShareRowExclusiveLock?
fixed
> I test DROP SUBSCIPTION case but even with this patch, "tuple
> concurrently updated" is still occurr
On Wed, Jun 21, 2017 at 8:10 AM, Peter Eisentraut
wrote:
> On 6/19/17 22:54, Masahiko Sawada wrote:
>>> It seems to me we could just take a stronger lock around
>>> RemoveSubscriptionRel(), so that workers can't write in there concurrently.
>>
>> Since we reduced the lock level of updating pg_subs
Hello,
At Wed, 21 Jun 2017 22:43:32 -0400, Peter Eisentraut
wrote in
<501f75c9-c5d6-d023-add0-3b670ac86...@2ndquadrant.com>
> On 6/20/17 19:10, Peter Eisentraut wrote:
> > On 6/19/17 22:54, Masahiko Sawada wrote:
> >>> It seems to me we could just take a stronger lock around
> >>> RemoveSubscri
On 6/20/17 22:44, Noah Misch wrote:
>> A patch has been posted, and it's being reviewed. Next update Monday.
>
> This PostgreSQL 10 open item is past due for your status update. Kindly send
> a status update within 24 hours, and include a date for your subsequent status
> update. Refer to the p
On 6/19/17 22:41, Masahiko Sawada wrote:
> On Tue, Jun 20, 2017 at 10:47 AM, Peter Eisentraut
> wrote:
>> On 6/16/17 04:16, Masahiko Sawada wrote:
>>> A subscription relation state may have been removed already when we
>>> try to update it. SetSubscriptionRelState didn't emit an error in such
>>>
On 6/20/17 19:10, Peter Eisentraut wrote:
> On 6/19/17 22:54, Masahiko Sawada wrote:
>>> It seems to me we could just take a stronger lock around
>>> RemoveSubscriptionRel(), so that workers can't write in there concurrently.
>>
>> Since we reduced the lock level of updating pg_subscription_rel by
On Thu, Jun 15, 2017 at 11:40:52PM -0400, Peter Eisentraut wrote:
> On 6/13/17 15:49, Peter Eisentraut wrote:
> > On 6/13/17 02:33, Noah Misch wrote:
> >>> Steps to reproduce -
> >>> X cluster -> create 100 tables , publish all tables (create publication
> >>> pub
> >>> for all tables);
> >>> Y C
On 6/19/17 22:54, Masahiko Sawada wrote:
>> It seems to me we could just take a stronger lock around
>> RemoveSubscriptionRel(), so that workers can't write in there concurrently.
>
> Since we reduced the lock level of updating pg_subscription_rel by
> commit 521fd4795e3e the same deadlock issue w
On Tue, Jun 20, 2017 at 10:55 AM, Peter Eisentraut
wrote:
> On 6/1/17 13:37, Petr Jelinek wrote:
>> On 01/06/17 17:32, Masahiko Sawada wrote:
>>> On Thu, May 25, 2017 at 5:29 PM, tushar
>>> wrote:
After applying all your patches, drop subscription no more hangs while
dropping subscri
On Tue, Jun 20, 2017 at 10:47 AM, Peter Eisentraut
wrote:
> On 6/16/17 04:16, Masahiko Sawada wrote:
>> A subscription relation state may have been removed already when we
>> try to update it. SetSubscriptionRelState didn't emit an error in such
>> case but with this patch we end up with an error.
On 6/1/17 13:37, Petr Jelinek wrote:
> On 01/06/17 17:32, Masahiko Sawada wrote:
>> On Thu, May 25, 2017 at 5:29 PM, tushar
>> wrote:
>>> After applying all your patches, drop subscription no more hangs while
>>> dropping subscription but there is an error "ERROR: tuple concurrently
>>> upda
On 6/16/17 04:16, Masahiko Sawada wrote:
> A subscription relation state may have been removed already when we
> try to update it. SetSubscriptionRelState didn't emit an error in such
> case but with this patch we end up with an error. Since we shouldn't
> ignore such error in UpdateSubscriptionRel
On Thu, Jun 15, 2017 at 10:22 AM, Masahiko Sawada wrote:
> On Thu, Jun 15, 2017 at 7:35 AM, Petr Jelinek
> wrote:
>> On 13/06/17 21:49, Peter Eisentraut wrote:
>>> On 6/13/17 02:33, Noah Misch wrote:
> Steps to reproduce -
> X cluster -> create 100 tables , publish all tables (create pub
On 6/13/17 15:49, Peter Eisentraut wrote:
> On 6/13/17 02:33, Noah Misch wrote:
>>> Steps to reproduce -
>>> X cluster -> create 100 tables , publish all tables (create publication pub
>>> for all tables);
>>> Y Cluster -> create 100 tables ,create subscription(create subscription sub
>>> connecti
On 6/15/17 12:22, Petr Jelinek wrote:
> On 15/06/17 17:53, Peter Eisentraut wrote:
>> On 6/14/17 18:35, Petr Jelinek wrote:
>>> Attached fixes it (it was mostly about order of calls).
>> So do I understand this right that the actual fix is just moving up the
>> logicalrep_worker_stop() call in Drop
On 15/06/17 18:36, Peter Eisentraut wrote:
> On 6/15/17 12:22, Petr Jelinek wrote:
>> On 15/06/17 17:53, Peter Eisentraut wrote:
>>> On 6/14/17 18:35, Petr Jelinek wrote:
Attached fixes it (it was mostly about order of calls).
>>>
>>> So do I understand this right that the actual fix is just m
On 6/15/17 12:22, Petr Jelinek wrote:
> On 15/06/17 17:53, Peter Eisentraut wrote:
>> On 6/14/17 18:35, Petr Jelinek wrote:
>>> Attached fixes it (it was mostly about order of calls).
>>
>> So do I understand this right that the actual fix is just moving up the
>> logicalrep_worker_stop() call in D
On 15/06/17 17:53, Peter Eisentraut wrote:
> On 6/14/17 18:35, Petr Jelinek wrote:
>> Attached fixes it (it was mostly about order of calls).
>
> So do I understand this right that the actual fix is just moving up the
> logicalrep_worker_stop() call in DropSubscription().
>
No the fix is heap_ope
On 6/14/17 18:35, Petr Jelinek wrote:
> Attached fixes it (it was mostly about order of calls).
So do I understand this right that the actual fix is just moving up the
logicalrep_worker_stop() call in DropSubscription().
> I also split the
> SetSubscriptionRelState into 2 separate interface while
On Thu, Jun 15, 2017 at 7:35 AM, Petr Jelinek
wrote:
> On 13/06/17 21:49, Peter Eisentraut wrote:
>> On 6/13/17 02:33, Noah Misch wrote:
Steps to reproduce -
X cluster -> create 100 tables , publish all tables (create publication
pub
for all tables);
Y Cluster -> create
On 13/06/17 21:49, Peter Eisentraut wrote:
> On 6/13/17 02:33, Noah Misch wrote:
>>> Steps to reproduce -
>>> X cluster -> create 100 tables , publish all tables (create publication pub
>>> for all tables);
>>> Y Cluster -> create 100 tables ,create subscription(create subscription sub
>>> connect
On 6/13/17 02:33, Noah Misch wrote:
>> Steps to reproduce -
>> X cluster -> create 100 tables , publish all tables (create publication pub
>> for all tables);
>> Y Cluster -> create 100 tables ,create subscription(create subscription sub
>> connection 'user=centos host=localhost' publication pub;
On Thu, May 25, 2017 at 01:59:51PM +0530, tushar wrote:
> After applying all your patches, drop subscription no more hangs while
> dropping subscription but there is an error "ERROR: tuple concurrently
> updated" in the log file of
> standby.
>
> ---
> logical replication synchronization work
On 2017-06-07 11:51:12 +0200, Petr Jelinek wrote:
> On 07/06/17 03:00, Andres Freund wrote:
> > On 2017-06-06 19:36:13 +0200, Petr Jelinek wrote:
> >
> >> As a side note, we are starting to have several IsSomeTypeOfProcess
> >> functions for these kind of things. I wonder if bgworker infrastructur
On 07/06/17 03:00, Andres Freund wrote:
> On 2017-06-06 19:36:13 +0200, Petr Jelinek wrote:
>
>> As a side note, we are starting to have several IsSomeTypeOfProcess
>> functions for these kind of things. I wonder if bgworker infrastructure
>> should somehow provide this type of stuff (the proposed
On 2017-06-06 19:36:13 +0200, Petr Jelinek wrote:
> So the fact that we moved workers to standard interrupt handling broke
> launcher in subtle ways because it still uses it's own SIGTERM handling
> but some function it calls are using CHECK_FOR_INTERRUPTS (they are used
> by worker as well). I thi
On 03/06/17 18:53, Peter Eisentraut wrote:
> On 6/2/17 14:52, Peter Eisentraut wrote:
>> On 5/24/17 15:14, Petr Jelinek wrote:
>>> All the locking works just fine the way it is in master. The issue with
>>> deadlock with apply comes from the wrong handling of the SIGTERM in the
>>> apply (we didn't
On Sun, Jun 4, 2017 at 1:53 AM, Peter Eisentraut
wrote:
> On 6/2/17 14:52, Peter Eisentraut wrote:
>> On 5/24/17 15:14, Petr Jelinek wrote:
>>> All the locking works just fine the way it is in master. The issue with
>>> deadlock with apply comes from the wrong handling of the SIGTERM in the
>>> ap
On 6/2/17 14:52, Peter Eisentraut wrote:
> On 5/24/17 15:14, Petr Jelinek wrote:
>> All the locking works just fine the way it is in master. The issue with
>> deadlock with apply comes from the wrong handling of the SIGTERM in the
>> apply (we didn't set InterruptPending). I changed the SIGTERM han
On 5/24/17 15:14, Petr Jelinek wrote:
> All the locking works just fine the way it is in master. The issue with
> deadlock with apply comes from the wrong handling of the SIGTERM in the
> apply (we didn't set InterruptPending). I changed the SIGTERM handler in
> patch 0001 just to die which is actu
On Thu, Jun 01, 2017 at 12:00:33AM -0400, Peter Eisentraut wrote:
> On 5/31/17 02:51, Noah Misch wrote:
> > On Tue, May 30, 2017 at 01:30:35AM +, Noah Misch wrote:
> >> On Thu, May 18, 2017 at 10:27:51PM -0400, Peter Eisentraut wrote:
> >>> On 5/18/17 11:11, Noah Misch wrote:
> IMMEDIATE A
On 01/06/17 17:32, Masahiko Sawada wrote:
> On Thu, May 25, 2017 at 5:29 PM, tushar wrote:
>> On 05/25/2017 12:44 AM, Petr Jelinek wrote:
>>>
>>> There is still outstanding issue that sync worker will keep running
>>> inside the long COPY because the invalidation messages are also not
>>> processe
On Thu, May 25, 2017 at 5:29 PM, tushar wrote:
> On 05/25/2017 12:44 AM, Petr Jelinek wrote:
>>
>> There is still outstanding issue that sync worker will keep running
>> inside the long COPY because the invalidation messages are also not
>> processed until it finishes but all the original issues r
On 5/31/17 02:51, Noah Misch wrote:
> On Tue, May 30, 2017 at 01:30:35AM +, Noah Misch wrote:
>> On Thu, May 18, 2017 at 10:27:51PM -0400, Peter Eisentraut wrote:
>>> On 5/18/17 11:11, Noah Misch wrote:
IMMEDIATE ATTENTION REQUIRED. This PostgreSQL 10 open item is past due for
your s
On Thu, May 25, 2017 at 4:14 AM, Petr Jelinek
wrote:
> Hi,
>
> I finally had time to properly analyze this, and turns out we've been
> all just trying to fix symptoms and the actual problems.
>
> All the locking works just fine the way it is in master. The issue with
> deadlock with apply comes fr
On Tue, May 30, 2017 at 01:30:35AM +, Noah Misch wrote:
> On Thu, May 18, 2017 at 10:27:51PM -0400, Peter Eisentraut wrote:
> > On 5/18/17 11:11, Noah Misch wrote:
> > > IMMEDIATE ATTENTION REQUIRED. This PostgreSQL 10 open item is past due
> > > for
> > > your status update. Please reacquai
On Thu, May 18, 2017 at 10:27:51PM -0400, Peter Eisentraut wrote:
> On 5/18/17 11:11, Noah Misch wrote:
> > IMMEDIATE ATTENTION REQUIRED. This PostgreSQL 10 open item is past due for
> > your status update. Please reacquaint yourself with the policy on open item
> > ownership[1] and then reply im
On 05/25/2017 12:44 AM, Petr Jelinek wrote:
There is still outstanding issue that sync worker will keep running
inside the long COPY because the invalidation messages are also not
processed until it finishes but all the original issues reported here
disappear for me with the attached patches appl
On Wed, May 24, 2017 at 3:14 PM, Petr Jelinek
wrote:
> Hi,
>
> I finally had time to properly analyze this, and turns out we've been
> all just trying to fix symptoms and the actual problems.
Thank you for the patches!
> All the locking works just fine the way it is in master.
> The issue with
>
Hi,
I finally had time to properly analyze this, and turns out we've been
all just trying to fix symptoms and the actual problems.
All the locking works just fine the way it is in master. The issue with
deadlock with apply comes from the wrong handling of the SIGTERM in the
apply (we didn't set I
On 18/05/17 16:16, Robert Haas wrote:
> On Wed, May 17, 2017 at 6:58 AM, Masahiko Sawada
> wrote:
>> I think the above changes can solve this issue but It seems to me that
>> holding AccessExclusiveLock on pg_subscription by DROP SUBSCRIPTION
>> until commit could lead another deadlock problem in
Hello,
At Thu, 18 May 2017 10:16:35 -0400, Robert Haas wrote
in
> On Wed, May 17, 2017 at 6:58 AM, Masahiko Sawada
> wrote:
> > I think the above changes can solve this issue but It seems to me that
> > holding AccessExclusiveLock on pg_subscription by DROP SUBSCRIPTION
> > until commit could
On 5/18/17 11:11, Noah Misch wrote:
> IMMEDIATE ATTENTION REQUIRED. This PostgreSQL 10 open item is past due for
> your status update. Please reacquaint yourself with the policy on open item
> ownership[1] and then reply immediately. If I do not hear from you by
> 2017-05-19 16:00 UTC, I will tr
On Mon, May 15, 2017 at 03:28:14AM +, Noah Misch wrote:
> On Mon, May 08, 2017 at 06:27:30PM +0900, Masahiko Sawada wrote:
> > I encountered a situation where DROP SUBSCRIPTION got stuck when
> > initial table sync is in progress. In my environment, I created
> > several tables with some data o
On Wed, May 17, 2017 at 6:58 AM, Masahiko Sawada wrote:
> I think the above changes can solve this issue but It seems to me that
> holding AccessExclusiveLock on pg_subscription by DROP SUBSCRIPTION
> until commit could lead another deadlock problem in the future. So I'd
> to contrive ways to redu
On Mon, May 15, 2017 at 8:02 PM, Kyotaro HORIGUCHI
wrote:
> Hello,
>
> At Fri, 12 May 2017 17:24:07 +0900, Masahiko Sawada
> wrote in
>> On Fri, May 12, 2017 at 11:24 AM, Masahiko Sawada
>> wrote:
>> > On Thu, May 11, 2017 at 6:16 PM, Petr Jelinek
>> > wrote:
>> >> On 11/05/17 10:10, Masahik
Hello,
At Fri, 12 May 2017 17:24:07 +0900, Masahiko Sawada
wrote in
> On Fri, May 12, 2017 at 11:24 AM, Masahiko Sawada
> wrote:
> > On Thu, May 11, 2017 at 6:16 PM, Petr Jelinek
> > wrote:
> >> On 11/05/17 10:10, Masahiko Sawada wrote:
> >>> On Thu, May 11, 2017 at 4:06 PM, Michael Paquier
On Mon, May 08, 2017 at 06:27:30PM +0900, Masahiko Sawada wrote:
> I encountered a situation where DROP SUBSCRIPTION got stuck when
> initial table sync is in progress. In my environment, I created
> several tables with some data on publisher. I created subscription on
> subscriber and drop subscri
On Fri, May 12, 2017 at 11:24 AM, Masahiko Sawada wrote:
> On Thu, May 11, 2017 at 6:16 PM, Petr Jelinek
> wrote:
>> On 11/05/17 10:10, Masahiko Sawada wrote:
>>> On Thu, May 11, 2017 at 4:06 PM, Michael Paquier
>>> wrote:
On Wed, May 10, 2017 at 11:57 AM, Masahiko Sawada
wrote:
On Thu, May 11, 2017 at 6:16 PM, Petr Jelinek
wrote:
> On 11/05/17 10:10, Masahiko Sawada wrote:
>> On Thu, May 11, 2017 at 4:06 PM, Michael Paquier
>> wrote:
>>> On Wed, May 10, 2017 at 11:57 AM, Masahiko Sawada
>>> wrote:
Barring any objections, I'll add these two issues to open item.
>>
On 11/05/17 10:10, Masahiko Sawada wrote:
> On Thu, May 11, 2017 at 4:06 PM, Michael Paquier
> wrote:
>> On Wed, May 10, 2017 at 11:57 AM, Masahiko Sawada
>> wrote:
>>> Barring any objections, I'll add these two issues to open item.
>>
>> It seems to me that those open items have not been added
On Thu, May 11, 2017 at 4:06 PM, Michael Paquier
wrote:
> On Wed, May 10, 2017 at 11:57 AM, Masahiko Sawada
> wrote:
>> Barring any objections, I'll add these two issues to open item.
>
> It seems to me that those open items have not been added yet to the
> list. If I am following correctly, the
On Wed, May 10, 2017 at 11:57 AM, Masahiko Sawada wrote:
> Barring any objections, I'll add these two issues to open item.
It seems to me that those open items have not been added yet to the
list. If I am following correctly, they could be defined as follows:
- Dropping subscription may stuck if
On Wed, May 10, 2017 at 2:46 AM, Masahiko Sawada wrote:
> On Mon, May 8, 2017 at 8:42 PM, Petr Jelinek
> wrote:
>> On 08/05/17 11:27, Masahiko Sawada wrote:
>>> Hi,
>>>
>>> I encountered a situation where DROP SUBSCRIPTION got stuck when
>>> initial table sync is in progress. In my environment, I
On Mon, May 8, 2017 at 8:42 PM, Petr Jelinek
wrote:
> On 08/05/17 11:27, Masahiko Sawada wrote:
>> Hi,
>>
>> I encountered a situation where DROP SUBSCRIPTION got stuck when
>> initial table sync is in progress. In my environment, I created
>> several tables with some data on publisher. I created
On Mon, May 8, 2017 at 8:53 PM, Erik Rijkers wrote:
> On 2017-05-08 13:13, Masahiko Sawada wrote:
>>
>> On Mon, May 8, 2017 at 7:14 PM, Erik Rijkers wrote:
>>>
>>> On 2017-05-08 11:27, Masahiko Sawada wrote:
>
>>>
>>> FWIW, running
>>>
0001-WIP-Fix-off-by-one-around-GetLastImportan
On 2017-05-08 13:13, Masahiko Sawada wrote:
On Mon, May 8, 2017 at 7:14 PM, Erik Rijkers wrote:
On 2017-05-08 11:27, Masahiko Sawada wrote:
FWIW, running
0001-WIP-Fix-off-by-one-around-GetLastImportantRecPtr.patch+
0002-WIP-Possibly-more-robust-snapbuild-approach.patch +
fix-statist
On 08/05/17 11:27, Masahiko Sawada wrote:
> Hi,
>
> I encountered a situation where DROP SUBSCRIPTION got stuck when
> initial table sync is in progress. In my environment, I created
> several tables with some data on publisher. I created subscription on
> subscriber and drop subscription immediat
On Mon, May 8, 2017 at 7:14 PM, Erik Rijkers wrote:
> On 2017-05-08 11:27, Masahiko Sawada wrote:
>>
>> Hi,
>>
>> I encountered a situation where DROP SUBSCRIPTION got stuck when
>> initial table sync is in progress. In my environment, I created
>> several tables with some data on publisher. I cre
On 2017-05-08 11:27, Masahiko Sawada wrote:
Hi,
I encountered a situation where DROP SUBSCRIPTION got stuck when
initial table sync is in progress. In my environment, I created
several tables with some data on publisher. I created subscription on
subscriber and drop subscription immediately afte
72 matches
Mail list logo