The only way I could replicate the issue or at least see that a
connection is established but the subscription doesnt resubscribe. I
did the following:
I used UFW to control the network connection so I can block it. For
initially starting the route I have the following UFW rule to allow
all connections to Salesforce IPs:
13.210.0.0/16 ALLOW OUT Anywhere
I then start the camel routes and using "lsof -i tcp" I can see the
established connection to Salesforce and in the logs I can see the
subscription to our platform events:
lsof:
192.168.1.211:55510->13.210.6.47:https (ESTABLISHED)
Camel log:
2020-12-01 16:33:45.843 INFO 78386 --- [ent@11180750-31]
o.a.c.c.s.i.s.SubscriptionHelper : Subscribed to channel
/event/LMSUser__e
I then disrrupt the connection using ss:
sudo ss -K dst 13.210.6.47
The following occurs in the log:
2020-12-01 16:46:08.942 WARN 78386 --- [ent@11180750-26]
o.a.c.c.s.i.s.SubscriptionHelper : Connect failure:
{failure={exception=java.net.ConnectException: Connection refused,
message={clientId=2p21tj4qverluys8vnb9ww96hcue, advice={timeout=0},
channel=/meta/connect, id=14, connectionType=long-polling},
connectionType=long-polling}, channel=/meta/connect, id=14,
successful=false}
and this successfully reconnects to Salesforce using a different IP
address and the subscription is still working fine.
Now if I block all IP addresses to Salesforce using this UFW rule:
ufw reject out to 13.210.0.0/16
and then I issue a control bus command to restart to the Salesforce
route in Camel (as to not restart the java process). The following
happens in the log:
2020-12-01 16:49:58.026 INFO 78386 --- [ - ShutdownTask]
o.a.c.c.s.i.s.SubscriptionHelper : Unsubscribing from channel
/event/LMSUser__e...
2020-12-01 16:51:56.340 ERROR 81511 --- [ent@11180750-30]
a.e.a.i.m.r.SalesforceContactEventRoutes : error
org.apache.camel.component.salesforce.api.SalesforceException: Error
subscribing to event/LMSUser__e: Connection refused..
and now if I change the UFW rule to allow connections to Salesforce using:
ufw allow out to 13.210.0.0/16
I can see in the lsof connections that a connection is established and
I can even disrrupt it using the ss command as above and in the log it
reports:
2020-12-01 16:53:45.634 WARN 81511 --- [ent@11180750-30]
o.a.c.c.s.i.s.SubscriptionHelper : Connect failure:
{failure={exception=java.io.EOFException:
HttpConnectionOverHTTP@78f09ebd::DecryptedEndPoint@38ed3138{l=/192.168.1.211:52890,r=aib--test.my.salesforce.com/13.210.4.109:443,OPEN,fill=-,flush=-,to=47747/340000},
message={clientId=2y01uo3aroo464hdwshfmuasq2gx, channel=/meta/connect,
id=9, connectionType=long-polling}, connectionType=long-polling},
channel=/meta/connect, id=9, successful=false}
and it successfully reconnects using a different ip address. but the
event subscription never subscibes again and no events are handled
eventhough there is an active connection established.
Does this expose the issue correctly? Its a bit hard to truly
replicate it when Salesforce goes down as its normally not just a
simple connection failure situation.
thanks
Andy
On Tue, Dec 1, 2020 at 4:10 PM Andy Stebbing <stebbingtow...@gmail.com> wrote:
>
> Hi,
> I did try it out with 3.7.0-SNAPSHOT but the outcome was the same.
> I'll put in a JIRA ticket and I'll provide details on how I was able
> to replicate the issue. I'll also post it on here.
>
> cheers
> andy
>
> On Mon, Nov 30, 2020 at 7:59 PM Zoran Regvart <zo...@regvart.com> wrote:
> >
> > Hi Andy,
> > sounds like an issue we would not encounter when developing/running
> > integration tests. We have a number of issues (some of them have been
> > fixed) relating to streaming/push APIs with Salesforce.
> >
> > Perhaps if you find that the recent versions did not fix your issue
> > you could contribute the logic of your custom error handler. Sounds
> > like you've also arrived at a setup where you can reproduce and test
> > the fix.
> >
> > I'm guessing it could fit within the SubscriptionHelper class.
> >
> > zoran
> >
> > On Fri, Nov 27, 2020 at 4:14 AM Andy Stebbing <stebbingtow...@gmail.com>
> > wrote:
> > >
> > > Hi There,
> > > I'm using the salesforce component for receiving platform events and
> > > have been using it for a few months and is generally quite stable.
> > >
> > > My question is around the durability of the subscription, for normal
> > > network dropouts or timeouts etc, it handles it very well, but
> > > recently Salesforce have had some big outages in my org and it causes
> > > the subscription to stop working without re-subscribing again.
> > >
> > > I can replicate the issue locally, running my camel route so it
> > > initially subscribes to the events then I firewall block the
> > > Salesforce destination then i restart the route. It fails and then
> > > unsubscribes from the events and when i open the firewall it connects
> > > but never re-subscribes to the events again. Any new events from
> > > Salesforce are not consumed and the service remains offline.
> > >
> > > I've bridged the error handler to one I've defined within the same
> > > camel context so that I could detect the salesforce exception when it
> > > happens and then to use the control bus to restart the route which
> > > then does the subscription. But seems like an overkill, especially if
> > > this is a bug that might end up being fixed.
> > >
> > > Has anyone else experienced this? Are there any solutions to get around
> > > this?
> > >
> > > Many thanks!
> >
> >
> >
> > --
> > Zoran Regvart
