dannycranmer edited a comment on pull request #18733: URL: https://github.com/apache/flink/pull/18733#issuecomment-1042989104
> If there are two jobs running on the same TM and they're both using the shared ELG, how comes they can't interfere with each other? When 2 jobs are running on the same TM, they are using separate ELG already, because they are created in static namespace of Flink User classloader. > In general I'm inclined to use the shared reference counted ELG, but we'd probably have to scope it per jobId. Based on above, we already have a shared reference counting ELG scoped per job. So for a single job, multiple sources/sinks using AWS async clients will share the same thread pool > I think I see your point, as long as the SDK is loaded from the user jar, there are actually two instances of this SharedELG singleton, because they class have been loaded by a different classloader (this can be easily verified by running jmap -histo <pid>). Correct yes. I confirmed this by observing the new Netty threads created for each ELG, as explained in my previous comment. > How does this change for the case when the SDK is not shipped with the user jar and is loaded from the system classpath (parent) instead? In this case we would have a single shared ELG for all jobs on the TM. However, since the AWS SDK jar is loaded by parent classloader I assume the Netty threads would be spawned via AWS SDK threads also loaded from parent classloader, thus attaching parent context classloader. But this is just a guess. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
