Kontinuation commented on issue #8601:
URL: https://github.com/apache/iceberg/issues/8601#issuecomment-1732469299
I injected some code into
`software.amazon.awssdk.thirdparty.org.apache.http.impl.conn.PoolingHttpClientConnectionManager`
in AWS SDK v2 to print the stacktrace when the connection pool was shut down
and ran several hours of testing. I found that it was shut down by the
finalizer of `S3FileIO`:
```
shutdown() was called
java.base/java.lang.Thread.getStackTrace(Unknown Source)
software.amazon.awssdk.thirdparty.org.apache.http.impl.conn.PoolingHttpClientConnectionManager.shutdown(PoolingHttpClientConnectionManager.java:411)
software.amazon.awssdk.http.apache.ApacheHttpClient.close(ApacheHttpClient.java:247)
software.amazon.awssdk.utils.IoUtils.closeQuietly(IoUtils.java:70)
software.amazon.awssdk.utils.IoUtils.closeIfCloseable(IoUtils.java:87)
software.amazon.awssdk.utils.AttributeMap.lambda$close$0(AttributeMap.java:87)
java.base/java.util.HashMap$Values.forEach(Unknown Source)
software.amazon.awssdk.utils.AttributeMap.close(AttributeMap.java:87)
software.amazon.awssdk.core.client.config.SdkClientConfiguration.close(SdkClientConfiguration.java:79)
software.amazon.awssdk.core.internal.http.HttpClientDependencies.close(HttpClientDependencies.java:80)
software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient.close(AmazonSyncHttpClient.java:73)
software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.close(BaseSyncClientHandler.java:88)
software.amazon.awssdk.services.sts.DefaultStsClient.close(DefaultStsClient.java:1344)
software.amazon.awssdk.utils.IoUtils.closeQuietly(IoUtils.java:70)
software.amazon.awssdk.services.sts.internal.StsWebIdentityCredentialsProviderFactory$StsWebIdentityCredentialsProvider.close(StsWebIdentityCredentialsProviderFactory.java:99)
software.amazon.awssdk.utils.IoUtils.closeQuietly(IoUtils.java:70)
software.amazon.awssdk.utils.IoUtils.closeIfCloseable(IoUtils.java:87)
software.amazon.awssdk.auth.credentials.WebIdentityTokenFileCredentialsProvider.close(WebIdentityTokenFileCredentialsProvider.java:132)
software.amazon.awssdk.utils.IoUtils.closeQuietly(IoUtils.java:70)
software.amazon.awssdk.utils.IoUtils.closeIfCloseable(IoUtils.java:87)
software.amazon.awssdk.auth.credentials.AwsCredentialsProviderChain.lambda$close$2(AwsCredentialsProviderChain.java:122)
java.base/java.util.ArrayList.forEach(Unknown Source)
java.base/java.util.Collections$UnmodifiableCollection.forEach(Unknown
Source)
software.amazon.awssdk.auth.credentials.AwsCredentialsProviderChain.close(AwsCredentialsProviderChain.java:122)
software.amazon.awssdk.utils.IoUtils.closeQuietly(IoUtils.java:70)
software.amazon.awssdk.utils.IoUtils.closeIfCloseable(IoUtils.java:87)
software.amazon.awssdk.utils.Lazy.close(Lazy.java:77)
software.amazon.awssdk.utils.IoUtils.closeQuietly(IoUtils.java:70)
software.amazon.awssdk.utils.IoUtils.closeIfCloseable(IoUtils.java:87)
software.amazon.awssdk.auth.credentials.internal.LazyAwsCredentialsProvider.close(LazyAwsCredentialsProvider.java:50)
software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider.close(DefaultCredentialsProvider.java:131)
software.amazon.awssdk.utils.IoUtils.closeQuietly(IoUtils.java:70)
software.amazon.awssdk.utils.IoUtils.closeIfCloseable(IoUtils.java:87)
software.amazon.awssdk.utils.AttributeMap.lambda$close$0(AttributeMap.java:87)
java.base/java.util.HashMap$Values.forEach(Unknown Source)
software.amazon.awssdk.utils.AttributeMap.close(AttributeMap.java:87)
software.amazon.awssdk.core.client.config.SdkClientConfiguration.close(SdkClientConfiguration.java:79)
software.amazon.awssdk.core.internal.http.HttpClientDependencies.close(HttpClientDependencies.java:80)
software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient.close(AmazonSyncHttpClient.java:73)
software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.close(BaseSyncClientHandler.java:88)
software.amazon.awssdk.services.s3.DefaultS3Client.close(DefaultS3Client.java:11088)
org.apache.iceberg.aws.s3.S3FileIO.close(S3FileIO.java:405)
org.apache.iceberg.aws.s3.S3FileIO.finalize(S3FileIO.java:415)
java.base/java.lang.System$2.invokeFinalize(Unknown Source)
java.base/java.lang.ref.Finalizer.runFinalizer(Unknown Source)
java.base/java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)
```
I think switching to another HTTP client implementation is not a sane
solution to this problem since whatever HTTP client we are using, there is
still the possibility of the client being shut down while in use.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
