sharan-malyala commented on issue #39780: URL: https://github.com/apache/beam/issues/39780#issuecomment-5313933042
Triage : Ref - https://github.com/apache/beam/blob/master/sdks/java/io/cassandra/src/main/java/org/apache/beam/sdk/io/cassandra/ConnectionManager.java - The cluster object is created during the first attempt by the worker and cached in a Map within ConnectionManager.(line 63) - When `getSession` fails on an attempt (due to any transient issue on Cassandra), the broken Cluster object remains in the cache. - If any other work item runs on the same worker, it fetches this cached cluster object. The Datastax driver checks it via `checkNotClosed()` and immediately fails it with an `IllegalStateException` because of the previous initialization failure. - Because the broken connection is never evicted from the cache, a single transient failure permanently poisons the worker. Proposed solution : - Check if the cached Cluster object is closed/broken before attempting to use it. - If it is broken, remove it from the map and compute a fresh one. I will send a PR soon with the change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
