[
https://issues.apache.org/jira/browse/HADOOP-19205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17855102#comment-17855102
]
Steve Loughran commented on HADOOP-19205:
-----------------------------------------
h3. client init.
{code}
[53] 3.22% 202 self: 0.00% 0 org/apache/hadoop/fs/FileSystem.access$300
[54] 3.22% 202 self: 0.00% 0 org/apache/hadoop/fs/FileSystem.createFileSystem
[55] 3.22% 202 self: 0.00% 0 org/apache/hadoop/fs/s3a/S3AFileSystem.initialize
[56] 1.74% 109 self: 0.00% 0
org/apache/hadoop/fs/s3a/S3AFileSystem.bindAWSClient
[57] 0.91% 57 self: 0.00% 0
org/apache/hadoop/fs/s3a/DefaultS3ClientFactory.createS3Client
[58] 0.88% 55 self: 0.00% 0
software/amazon/awssdk/core/client/builder/SdkDefaultClientBuilder.build
[59] 0.88% 55 self: 0.00% 0
software/amazon/awssdk/services/s3/DefaultS3ClientBuilder.buildClient
[60] 0.88% 55 self: 0.00% 0
software/amazon/awssdk/services/s3/DefaultS3ClientBuilder.buildClient
[61] 0.88% 55 self: 0.00% 0
software/amazon/awssdk/core/client/builder/SdkDefaultClientBuilder.syncClientConfiguration
[62] 0.46% 29 self: 0.00% 0
software/amazon/awssdk/awscore/client/builder/AwsDefaultClientBuilder.finalizeChildConfiguration
[63] 0.41% 26 self: 0.00% 0
software/amazon/awssdk/services/s3/DefaultS3BaseClientBuilder.finalizeServiceConfiguration
[63] 0.05% 3 self: 0.00% 0
software/amazon/awssdk/awscore/client/builder/AwsDefaultClientBuilder.finalizeAwsConfiguration
[62] 0.37% 23 self: 0.00% 0
software/amazon/awssdk/core/client/builder/SdkDefaultClientBuilder.finalizeConfiguration
[63] 0.30% 19 self: 0.00% 0
software/amazon/awssdk/core/client/builder/SdkDefaultClientBuilder.resolveExecutionInterceptors
[63] 0.05% 3 self: 0.00% 0
software/amazon/awssdk/core/client/config/SdkClientConfiguration$Builder.build
[63] 0.02% 1 self: 0.00% 0
software/amazon/awssdk/core/client/config/SdkClientConfiguration$Builder.lazyOption
[62] 0.02% 1 self: 0.00% 0
software/amazon/awssdk/core/client/builder/SdkDefaultClientBuilder.finalizeSyncConfiguration
[62] 0.02% 1 self: 0.00% 0
software/amazon/awssdk/core/client/builder/SdkDefaultClientBuilder.mergeGlobalDefaults
[62] 0.02% 1 self: 0.00% 0
software/amazon/awssdk/services/s3/DefaultS3BaseClientBuilder.invokePlugins
[58] 0.03% 2 self: 0.00% 0
org/apache/hadoop/fs/s3a/impl/AWSClientConfig.createHttpClientBuilder
[57] 0.78% 49 self: 0.00% 0
org/apache/hadoop/fs/s3a/S3AFileSystem.createS3AsyncClient
[58] 0.78% 49 self: 0.00% 0
org/apache/hadoop/fs/s3a/DefaultS3ClientFactory.createS3AsyncClient
[59] 0.76% 48 self: 0.00% 0
software/amazon/awssdk/core/client/builder/SdkDefaultClientBuilder.build
[60] 0.76% 48 self: 0.00% 0
software/amazon/awssdk/services/s3/DefaultS3AsyncClientBuilder.buildClient
[61] 0.76% 48 self: 0.00% 0
software/amazon/awssdk/services/s3/DefaultS3AsyncClientBuilder.buildClient
[62] 0.72% 45 self: 0.00% 0
software/amazon/awssdk/core/client/builder/SdkDefaultClientBuilder.asyncClientConfiguration
[63] 0.29% 18 self: 0.00% 0
software/amazon/awssdk/core/client/builder/SdkDefaultClientBuilder.finalizeConfiguration
[64] 0.29% 18 self: 0.00% 0
software/amazon/awssdk/core/client/builder/SdkDefaultClientBuilder.resolveExecutionInterceptors
[65] 0.29% 18 self: 0.00% 0
software/amazon/awssdk/core/interceptor/ClasspathInterceptorChainFactory.getGlobalInterceptors
[66] 0.29% 18 self: 0.00% 0
software/amazon/awssdk/core/interceptor/ClasspathInterceptorChainFactory.createExecutionInterceptorsFromClasspath
[67] 0.29% 18 self: 0.00% 0
software/amazon/awssdk/core/interceptor/ClasspathInterceptorChainFactory.createExecutionInterceptorsFromResources
[63] 0.27% 17 self: 0.00% 0
software/amazon/awssdk/awscore/client/builder/AwsDefaultClientBuilder.finalizeChildConfiguration
[63] 0.10% 6 self: 0.00% 0
software/amazon/awssdk/core/client/builder/SdkDefaultClientBuilder.finalizeAsyncConfiguration
[63] 0.03% 2 self: 0.00% 0
software/amazon/awssdk/core/client/builder/SdkDefaultClientBuilder.mergeGlobalDefaults
[63] 0.02% 1 self: 0.00% 0
software/amazon/awssdk/core/client/builder/SdkDefaultClientBuilder.setOverrides
[63] 0.02% 1 self: 0.02% 1
software/amazon/awssdk/services/s3/DefaultS3BaseClientBuilder.invokePlugins
[62] 0.05% 3 self: 0.00% 0
software/amazon/awssdk/services/s3/internal/client/S3AsyncClientDecorator.decorate
[59] 0.02% 1 self: 0.00% 0
org/apache/hadoop/fs/s3a/impl/AWSClientConfig.createAsyncHttpClientBuilder
{code}
h3. looking up encryption secrets. problem is even the algorithm option uses
Configuration.getPassword().
{code}
[56] 0.78% 49 self: 0.00% 0
org/apache/hadoop/fs/s3a/S3AUtils.buildEncryptionSecrets
[57] 0.37% 23 self: 0.00% 0 org/apache/hadoop/fs/s3a/S3AUtils.getS3EncryptionKey
[58] 0.29% 18 self: 0.00% 0 org/apache/hadoop/fs/s3a/S3AUtils.lookupBucketSecret
[59] 0.29% 18 self: 0.00% 0 org/apache/hadoop/fs/s3a/S3AUtils.getPassword
[60] 0.29% 18 self: 0.00% 0 org/apache/hadoop/fs/s3a/S3AUtils.lookupPassword
[61] 0.29% 18 self: 0.00% 0 org/apache/hadoop/conf/Configuration.getPassword
[62] 0.29% 18 self: 0.00% 0
org/apache/hadoop/conf/Configuration.getPasswordFromCredentialProviders
[63] 0.29% 18 self: 0.00% 0
org/apache/hadoop/security/alias/CredentialProviderFactory.getProviders
[64] 0.29% 18 self: 0.00% 0
org/apache/hadoop/security/alias/LocalJavaKeyStoreProvider$Factory.createProvider
[65] 0.29% 18 self: 0.00% 0
org/apache/hadoop/security/alias/LocalJavaKeyStoreProvider.<init>
[66] 0.29% 18 self: 0.00% 0
org/apache/hadoop/security/alias/LocalJavaKeyStoreProvider.<init>
[67] 0.29% 18 self: 0.00% 0
org/apache/hadoop/security/alias/LocalKeyStoreProvider.<init>
[68] 0.29% 18 self: 0.00% 0
org/apache/hadoop/security/alias/AbstractJavaKeyStoreProvider.<init>
[69] 0.27% 17 self: 0.00% 0
org/apache/hadoop/security/alias/AbstractJavaKeyStoreProvider.locateKeystore
[70] 0.19% 12 self: 0.00% 0 java/security/KeyStore.load
[70] 0.05% 3 self: 0.00% 0
org/apache/hadoop/security/alias/LocalKeyStoreProvider.keystoreExists
[70] 0.02% 1 self: 0.00% 0
org/apache/hadoop/security/alias/LocalKeyStoreProvider.getInputStreamForFile
[70] 0.02% 1 self: 0.00% 0 sun/nio/ch/ChannelInputStream.close
{code}
> S3A initialization/close slower than with v1 SDK
> ------------------------------------------------
>
> Key: HADOOP-19205
> URL: https://issues.apache.org/jira/browse/HADOOP-19205
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.4.0
> Reporter: Steve Loughran
> Priority: Major
> Attachments: Screenshot 2024-06-14 at 17.12.59.png, Screenshot
> 2024-06-14 at 17.14.33.png
>
>
> Hive QE have observed slowdown in LLAP queries due to time to create and
> close s3a filesystems instances. A key aspect of that is they keep closing
> the fs instances (HIVE-27884), but looking at the profiles, the reason things
> seem to have regressed is
> * two s3 clients are being created (sync and async)
> * these seem to take a lot of time scanning the classpath for "global
> interceptors", which is at least an O(jars) operation; #of index entries in
> the zip files may factor too.
> Proposed:
> * create async client on demand when the transfer manager is invoked
> * look at why passwords are being scanned for if
> InstanceProfileCredentialsProvider is in use...that seems slow too
> SDK wishes
> * SDK maybe allow us to turn off that scan for interceptors?
> attaching screenshots of the profile. storediag snippet:
> {code}
> [001] fs.s3a.access.key = (unset)
> [002] fs.s3a.secret.key = (unset)
> [003] fs.s3a.session.token = (unset)
> [004] fs.s3a.server-side-encryption-algorithm = (unset)
> [005] fs.s3a.server-side-encryption.key = (unset)
> [006] fs.s3a.encryption.algorithm = (unset)
> [007] fs.s3a.encryption.key = (unset)
> [008] fs.s3a.aws.credentials.provider =
> "com.amazonaws.auth.InstanceProfileCredentialsProvider" [core-site.xml]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
