Hi Andrés, could storing the age of the last seen id along with the value also work?
Then if the id is older than 24h instead of that last seen value set it to -2 or -1 as the use case dictates. I kinda think that the management of replayId needs to be in the client application not in Camel, it's going to be difficult to satisfy different use cases folk have; meaning some might be interested only in latest some might want to replay all events, and there could be some that wish to crash and let the operator choose. zoran On Tue, May 5, 2020 at 1:32 PM Andres Q <tulsi...@gmail.com> wrote: > > My case is the same as > > https://issues.apache.org/jira/browse/CAMEL-13170 > > And if my understanding is correct, it's simple (the problem, not the > solution): > > Camel reconnects to a topic automatically using always the same replayId > > At least that's what happens in my case when I set the replayId in the > endpoint like: > > from("salesforce:data/ChangeEvents?replayId=" + replayIdForSalesforce) > > Scenario is I start the routes, replayId=X. If a disconnect happens > (and they happen about every 3 hours, I think) then Camel would try to > reconnect using replayId=X. That works fine, but 24 hours later, that > replayId is invalid because Salesforce only stores 24 hours of events. > Hence the reconnection to the topic would throw the error: > > org.apache.camel.component.salesforce.api.SalesforceException - Error > subscribing to data/ChangeEvents: 400::The replayId {13344} you > provided was invalid. Please provide a valid ID, -2 to replay all > events, or -1 to replay only new events. > > And it won't retry reconnecting, hence I would lose any further events. > > On the EMP-Connector project by salesforce they worked around this by > completely removing the replayId after the connection starts, so all > subsequent reconnections would use -2. This can of course result in > repeated events, so I guess it's up to the client code to handle > those. See: > https://github.com/forcedotcom/EMP-Connector/pull/42/commits/19766eca02970658691a7372af4851d3ef10667a > > > El mar., 5 de may. de 2020 a la(s) 05:16, Zoran Regvart > (zo...@regvart.com) escribió: > > > > Hi Andrés, > > folk have found this problem and several similar problems before: > > > > https://issues.apache.org/jira/browse/CAMEL-13170 > > https://issues.apache.org/jira/browse/CAMEL-12812 > > https://issues.apache.org/jira/browse/CAMEL-12871 > > https://issues.apache.org/jira/browse/CAMEL-13577 > > > > I'm afraid that we don't have the full understanding of the issue or a > > reliable way to reproduce it. Any help would be appreciated. > > > > zoran > > > > On Tue, May 5, 2020 at 1:29 AM Andres Q <tulsi...@gmail.com> wrote: > > > > > > Hi > > > > > > I'm subscribing to Salesforce Change Data Capture events. I had the > > > problem that after 24 hours the replayId is stale and if Camel tries > > > to reconnect to it it throws an error (I asked about this here: > > > https://mail-archives.apache.org/mod_mbox/camel-users/202002.mbox/%3CCAJrxdruca8yqT7fs4snObY6QoCD8cHPUWDkqF3XdBXbGBd4Spg%40mail.gmail.com%3E) > > > > > > As suggested by Zoran Regvart I'm storing the replayId and the > > > timestamp, so on app startup if I know the replayId is old enough, I > > > pass ?replayId=-2 to replay all events. > > > > > > This works fine, but the problem I'm facing now is that if the app > > > reconnects after some time I get this event: > > > > > > Connect failure: {advice={reconnect=handshake, interval=0}, > > > channel=/meta/connect, id=3320, error=403::Unknown client, > > > successful=false} > > > > > > And then the client will try to reconnect automatically using the > > > latest replayId. Since this logic is handled by SubscriptionHelper, if > > > the replayId is old enough it will fail on reconnect and then the app > > > will stop receiving events forever. > > > > > > The only thing I can think of is to always use replayId=-2 to fetch > > > all events and ignore the ones I already processed, but it seems > > > suboptimal to say the least. > > > > > > How should I handle this scenario so that the implementation is robust? > > > > > > Thanks, > > > > > > Andrés > > > > > > > > -- > > Zoran Regvart -- Zoran Regvart
