Re: Salesforce Platform Events Subscription Retries
I'll give it a try, just give me a bit of time and I'll let you know.
cheers
andy
On Fri, Dec 4, 2020 at 9:07 AM Zoran Regvart wrote:
>
> Hi Andy,
> that helps a lot, thanks! I *think* a `restartClient()` invocation
> might help if added here:
>
> https://github.com/apache/camel/blob/b140279d8235ae1aa7cf5010c0bc4945a529fe38/components/camel-salesforce/camel-salesforce-component/src/main/java/org/apache/camel/component/salesforce/internal/streaming/SubscriptionHelper.java#L174
>
> i.e. when connection was not successful, that's when that "Connect
> failure" message is logged.
>
> Would you be able to build and test that change on your end?
>
> zoran
>
> On Thu, Dec 3, 2020 at 11:01 PM Andy Stebbing
> wrote:
> >
> > The only way I could replicate the issue or at least see that a
> > connection is established but the subscription doesnt resubscribe. I
> > did the following:
> >
> > I used UFW to control the network connection so I can block it. For
> > initially starting the route I have the following UFW rule to allow
> > all connections to Salesforce IPs:
> >
> > 13.210.0.0/16 ALLOW OUT Anywhere
> >
> > I then start the camel routes and using "lsof -i tcp" I can see the
> > established connection to Salesforce and in the logs I can see the
> > subscription to our platform events:
> >
> > lsof:
> > 192.168.1.211:55510->13.210.6.47:https (ESTABLISHED)
> >
> > Camel log:
> > 2020-12-01 16:33:45.843 INFO 78386 --- [ent@11180750-31]
> > o.a.c.c.s.i.s.SubscriptionHelper : Subscribed to channel
> > /event/LMSUser__e
> >
> > I then disrrupt the connection using ss:
> > sudo ss -K dst 13.210.6.47
> >
> > The following occurs in the log:
> > 2020-12-01 16:46:08.942 WARN 78386 --- [ent@11180750-26]
> > o.a.c.c.s.i.s.SubscriptionHelper : Connect failure:
> > {failure={exception=java.net.ConnectException: Connection refused,
> > message={clientId=2p21tj4qverluys8vnb9ww96hcue, advice={timeout=0},
> > channel=/meta/connect, id=14, connectionType=long-polling},
> > connectionType=long-polling}, channel=/meta/connect, id=14,
> > successful=false}
> >
> > and this successfully reconnects to Salesforce using a different IP
> > address and the subscription is still working fine.
> >
> > Now if I block all IP addresses to Salesforce using this UFW rule:
> >
> > ufw reject out to 13.210.0.0/16
> >
> > and then I issue a control bus command to restart to the Salesforce
> > route in Camel (as to not restart the java process). The following
> > happens in the log:
> >
> > 2020-12-01 16:49:58.026 INFO 78386 --- [ - ShutdownTask]
> > o.a.c.c.s.i.s.SubscriptionHelper : Unsubscribing from channel
> > /event/LMSUser__e...
> >
> > 2020-12-01 16:51:56.340 ERROR 81511 --- [ent@11180750-30]
> > a.e.a.i.m.r.SalesforceContactEventRoutes : error
> > org.apache.camel.component.salesforce.api.SalesforceException: Error
> > subscribing to event/LMSUser__e: Connection refused..
> >
> > and now if I change the UFW rule to allow connections to Salesforce using:
> >
> > ufw allow out to 13.210.0.0/16
> >
> > I can see in the lsof connections that a connection is established and
> > I can even disrrupt it using the ss command as above and in the log it
> > reports:
> >
> > 2020-12-01 16:53:45.634 WARN 81511 --- [ent@11180750-30]
> > o.a.c.c.s.i.s.SubscriptionHelper : Connect failure:
> > {failure={exception=java.io.EOFException:
> > HttpConnectionOverHTTP@78f09ebd::DecryptedEndPoint@38ed3138{l=/192.168.1.211:52890,r=aib--test.my.salesforce.com/13.210.4.109:443,OPEN,fill=-,flush=-,to=47747/34},
> > message={clientId=2y01uo3aroo464hdwshfmuasq2gx, channel=/meta/connect,
> > id=9, connectionType=long-polling}, connectionType=long-polling},
> > channel=/meta/connect, id=9, successful=false}
> >
> > and it successfully reconnects using a different ip address. but the
> > event subscription never subscibes again and no events are handled
> > eventhough there is an active connection established.
> >
> > Does this expose the issue correctly? Its a bit hard to truly
> > replicate it when Salesforce goes down as its normally not just a
> > simple connection failure situation.
> >
> > thanks
> > Andy
> >
> >
> > On Tue, Dec 1, 2020 at 4:10 PM Andy Stebbing
> > wrote:
> > >
> > > Hi,
> > > I did try it out with 3.7.0-SNAPSHOT but the outcome was the same.
> > > I'll put in a JIRA ticket and I'll provide details on how I was able
> > > to replicate the issue. I'll also post it on here.
> > >
> > > cheers
> > > andy
> > >
> > > On Mon, Nov 30, 2020 at 7:59 PM Zoran Regvart wrote:
> > > >
> > > > Hi Andy,
> > > > sounds like an issue we would not encounter when developing/running
> > > > integration tests. We have a number of issues (some of them have been
> > > > fixed) relating to streaming/push APIs with Salesforce.
> > > >
> > > > Perhaps if you find that the recent versions did not fix your issue
> > > > you could contribute the logic of your custom error handler. Sounds
> > > > like you've also
Re: Salesforce Platform Events Subscription Retries
Hi Andy,
that helps a lot, thanks! I *think* a `restartClient()` invocation
might help if added here:
https://github.com/apache/camel/blob/b140279d8235ae1aa7cf5010c0bc4945a529fe38/components/camel-salesforce/camel-salesforce-component/src/main/java/org/apache/camel/component/salesforce/internal/streaming/SubscriptionHelper.java#L174
i.e. when connection was not successful, that's when that "Connect
failure" message is logged.
Would you be able to build and test that change on your end?
zoran
On Thu, Dec 3, 2020 at 11:01 PM Andy Stebbing wrote:
>
> The only way I could replicate the issue or at least see that a
> connection is established but the subscription doesnt resubscribe. I
> did the following:
>
> I used UFW to control the network connection so I can block it. For
> initially starting the route I have the following UFW rule to allow
> all connections to Salesforce IPs:
>
> 13.210.0.0/16 ALLOW OUT Anywhere
>
> I then start the camel routes and using "lsof -i tcp" I can see the
> established connection to Salesforce and in the logs I can see the
> subscription to our platform events:
>
> lsof:
> 192.168.1.211:55510->13.210.6.47:https (ESTABLISHED)
>
> Camel log:
> 2020-12-01 16:33:45.843 INFO 78386 --- [ent@11180750-31]
> o.a.c.c.s.i.s.SubscriptionHelper : Subscribed to channel
> /event/LMSUser__e
>
> I then disrrupt the connection using ss:
> sudo ss -K dst 13.210.6.47
>
> The following occurs in the log:
> 2020-12-01 16:46:08.942 WARN 78386 --- [ent@11180750-26]
> o.a.c.c.s.i.s.SubscriptionHelper : Connect failure:
> {failure={exception=java.net.ConnectException: Connection refused,
> message={clientId=2p21tj4qverluys8vnb9ww96hcue, advice={timeout=0},
> channel=/meta/connect, id=14, connectionType=long-polling},
> connectionType=long-polling}, channel=/meta/connect, id=14,
> successful=false}
>
> and this successfully reconnects to Salesforce using a different IP
> address and the subscription is still working fine.
>
> Now if I block all IP addresses to Salesforce using this UFW rule:
>
> ufw reject out to 13.210.0.0/16
>
> and then I issue a control bus command to restart to the Salesforce
> route in Camel (as to not restart the java process). The following
> happens in the log:
>
> 2020-12-01 16:49:58.026 INFO 78386 --- [ - ShutdownTask]
> o.a.c.c.s.i.s.SubscriptionHelper : Unsubscribing from channel
> /event/LMSUser__e...
>
> 2020-12-01 16:51:56.340 ERROR 81511 --- [ent@11180750-30]
> a.e.a.i.m.r.SalesforceContactEventRoutes : error
> org.apache.camel.component.salesforce.api.SalesforceException: Error
> subscribing to event/LMSUser__e: Connection refused..
>
> and now if I change the UFW rule to allow connections to Salesforce using:
>
> ufw allow out to 13.210.0.0/16
>
> I can see in the lsof connections that a connection is established and
> I can even disrrupt it using the ss command as above and in the log it
> reports:
>
> 2020-12-01 16:53:45.634 WARN 81511 --- [ent@11180750-30]
> o.a.c.c.s.i.s.SubscriptionHelper : Connect failure:
> {failure={exception=java.io.EOFException:
> HttpConnectionOverHTTP@78f09ebd::DecryptedEndPoint@38ed3138{l=/192.168.1.211:52890,r=aib--test.my.salesforce.com/13.210.4.109:443,OPEN,fill=-,flush=-,to=47747/34},
> message={clientId=2y01uo3aroo464hdwshfmuasq2gx, channel=/meta/connect,
> id=9, connectionType=long-polling}, connectionType=long-polling},
> channel=/meta/connect, id=9, successful=false}
>
> and it successfully reconnects using a different ip address. but the
> event subscription never subscibes again and no events are handled
> eventhough there is an active connection established.
>
> Does this expose the issue correctly? Its a bit hard to truly
> replicate it when Salesforce goes down as its normally not just a
> simple connection failure situation.
>
> thanks
> Andy
>
>
> On Tue, Dec 1, 2020 at 4:10 PM Andy Stebbing wrote:
> >
> > Hi,
> > I did try it out with 3.7.0-SNAPSHOT but the outcome was the same.
> > I'll put in a JIRA ticket and I'll provide details on how I was able
> > to replicate the issue. I'll also post it on here.
> >
> > cheers
> > andy
> >
> > On Mon, Nov 30, 2020 at 7:59 PM Zoran Regvart wrote:
> > >
> > > Hi Andy,
> > > sounds like an issue we would not encounter when developing/running
> > > integration tests. We have a number of issues (some of them have been
> > > fixed) relating to streaming/push APIs with Salesforce.
> > >
> > > Perhaps if you find that the recent versions did not fix your issue
> > > you could contribute the logic of your custom error handler. Sounds
> > > like you've also arrived at a setup where you can reproduce and test
> > > the fix.
> > >
> > > I'm guessing it could fit within the SubscriptionHelper class.
> > >
> > > zoran
> > >
> > > On Fri, Nov 27, 2020 at 4:14 AM Andy Stebbing
> > > wrote:
> > > >
> > > > Hi There,
> > > > I'm using the salesforce component for receiving platform events and
> > > > have been using it for a few
Re: Salesforce Platform Events Subscription Retries
The only way I could replicate the issue or at least see that a
connection is established but the subscription doesnt resubscribe. I
did the following:
I used UFW to control the network connection so I can block it. For
initially starting the route I have the following UFW rule to allow
all connections to Salesforce IPs:
13.210.0.0/16 ALLOW OUT Anywhere
I then start the camel routes and using "lsof -i tcp" I can see the
established connection to Salesforce and in the logs I can see the
subscription to our platform events:
lsof:
192.168.1.211:55510->13.210.6.47:https (ESTABLISHED)
Camel log:
2020-12-01 16:33:45.843 INFO 78386 --- [ent@11180750-31]
o.a.c.c.s.i.s.SubscriptionHelper : Subscribed to channel
/event/LMSUser__e
I then disrrupt the connection using ss:
sudo ss -K dst 13.210.6.47
The following occurs in the log:
2020-12-01 16:46:08.942 WARN 78386 --- [ent@11180750-26]
o.a.c.c.s.i.s.SubscriptionHelper : Connect failure:
{failure={exception=java.net.ConnectException: Connection refused,
message={clientId=2p21tj4qverluys8vnb9ww96hcue, advice={timeout=0},
channel=/meta/connect, id=14, connectionType=long-polling},
connectionType=long-polling}, channel=/meta/connect, id=14,
successful=false}
and this successfully reconnects to Salesforce using a different IP
address and the subscription is still working fine.
Now if I block all IP addresses to Salesforce using this UFW rule:
ufw reject out to 13.210.0.0/16
and then I issue a control bus command to restart to the Salesforce
route in Camel (as to not restart the java process). The following
happens in the log:
2020-12-01 16:49:58.026 INFO 78386 --- [ - ShutdownTask]
o.a.c.c.s.i.s.SubscriptionHelper : Unsubscribing from channel
/event/LMSUser__e...
2020-12-01 16:51:56.340 ERROR 81511 --- [ent@11180750-30]
a.e.a.i.m.r.SalesforceContactEventRoutes : error
org.apache.camel.component.salesforce.api.SalesforceException: Error
subscribing to event/LMSUser__e: Connection refused..
and now if I change the UFW rule to allow connections to Salesforce using:
ufw allow out to 13.210.0.0/16
I can see in the lsof connections that a connection is established and
I can even disrrupt it using the ss command as above and in the log it
reports:
2020-12-01 16:53:45.634 WARN 81511 --- [ent@11180750-30]
o.a.c.c.s.i.s.SubscriptionHelper : Connect failure:
{failure={exception=java.io.EOFException:
HttpConnectionOverHTTP@78f09ebd::DecryptedEndPoint@38ed3138{l=/192.168.1.211:52890,r=aib--test.my.salesforce.com/13.210.4.109:443,OPEN,fill=-,flush=-,to=47747/34},
message={clientId=2y01uo3aroo464hdwshfmuasq2gx, channel=/meta/connect,
id=9, connectionType=long-polling}, connectionType=long-polling},
channel=/meta/connect, id=9, successful=false}
and it successfully reconnects using a different ip address. but the
event subscription never subscibes again and no events are handled
eventhough there is an active connection established.
Does this expose the issue correctly? Its a bit hard to truly
replicate it when Salesforce goes down as its normally not just a
simple connection failure situation.
thanks
Andy
On Tue, Dec 1, 2020 at 4:10 PM Andy Stebbing wrote:
>
> Hi,
> I did try it out with 3.7.0-SNAPSHOT but the outcome was the same.
> I'll put in a JIRA ticket and I'll provide details on how I was able
> to replicate the issue. I'll also post it on here.
>
> cheers
> andy
>
> On Mon, Nov 30, 2020 at 7:59 PM Zoran Regvart wrote:
> >
> > Hi Andy,
> > sounds like an issue we would not encounter when developing/running
> > integration tests. We have a number of issues (some of them have been
> > fixed) relating to streaming/push APIs with Salesforce.
> >
> > Perhaps if you find that the recent versions did not fix your issue
> > you could contribute the logic of your custom error handler. Sounds
> > like you've also arrived at a setup where you can reproduce and test
> > the fix.
> >
> > I'm guessing it could fit within the SubscriptionHelper class.
> >
> > zoran
> >
> > On Fri, Nov 27, 2020 at 4:14 AM Andy Stebbing
> > wrote:
> > >
> > > Hi There,
> > > I'm using the salesforce component for receiving platform events and
> > > have been using it for a few months and is generally quite stable.
> > >
> > > My question is around the durability of the subscription, for normal
> > > network dropouts or timeouts etc, it handles it very well, but
> > > recently Salesforce have had some big outages in my org and it causes
> > > the subscription to stop working without re-subscribing again.
> > >
> > > I can replicate the issue locally, running my camel route so it
> > > initially subscribes to the events then I firewall block the
> > > Salesforce destination then i restart the route. It fails and then
> > > unsubscribes from the events and when i open the firewall it connects
> > > but never re-subscribes to the events again. Any new events from
> > > Salesforce are not consumed and the service remains offline.
> >
