dmvk commented on pull request #18553: URL: https://github.com/apache/flink/pull/18553#issuecomment-1035369952
So there is actually not a problem with reusing clients, it just makes the root cause more visible for some reason. The actual problem is that if the underlying netty event loop group is not configured, the `SharedSdkEventLoopGroup` is used. This means that clients that are used by the test suite and clients that are used by the actual pipeline are using the very same thread pool for handling HTTP communication. By default the thread pool uses quite high number of threads (I think something like 2x number of CPUs, but would have to dive into Netty to confirm this). Now the problem is that the Netty thread inherits a context classloader from the thread that has created it. This could either be the `main` thread (if thread has been created in the test suite) or `FlinkUserClassLoader` (if the thread has been created by the pipeline - in taskmanager). There is a slight chance, that once the pipeline finishes (and user classloader is closed), we'll initiate a call on thread that has been created with this classloader attached. That has couple of consequences: - It will simply fail. - If two jobs are running on the same TM and both are using AWS related clients, they will also eventually fail. The fix if fairly simple, we need to set separate event loops for tests and for each sink executed by the TM (we can eventually try to find a way to reuse ELG within the TM -> some kind of reference counting scoped per job, if that causes too much overhead). The proper place for fix would be `AWSGeneralUtil#createAsyncHttpClient(software.amazon.awssdk.utils.AttributeMap, software.amazon.awssdk.http.nio.netty.NettyNioAsyncHttpClient.Builder)`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
