mosche opened a new issue, #26321:
URL: https://github.com/apache/beam/issues/26321
### What happened?
The underlying issue here is the repeated initialization of filesystems
triggered in `SerializablePipelineOptions` (#18430).
Every call to `FileSystems.setDefaultPipelineOptions()` will silently
replace any existing previous file systems without cleaning up resources.
In the case of `S3FileSystem` this can leak threads if the file system was
already in use.
The simple test below shows the every increasing number of threads in the
system.
```Java
public class S3FileSystemTest {
@Test
public void testSetDefaultPipelineOptions() throws Exception {
S3Options s3Options = s3Options();
s3Options.setAwsCredentialsProvider(StaticCredentialsProvider.create(AwsBasicCredentials.create("key",
"secret")));
for (int i = 0; i < 1000; i++) {
FileSystems.setDefaultPipelineOptions(s3Options);
FileSystems.match(ImmutableList.of("s3://bucket/key*"));
System.out.println("Number of threads " + Thread.activeCount());
}
}
}
```
This is particularly bad with some runners, e.g. see #17645:
> A normal Go Flink ValidatesRunner execution would contain the logging line
from this class nearly 80,000 times, making it much more difficult to read the
logs.
### Issue Priority
Priority: 3 (minor)
### Issue Components
- [ ] Component: Python SDK
- [X] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
