zhangchiyue opened a new issue, #16874: URL: https://github.com/apache/dolphinscheduler/issues/16874
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues. ### What happened I encountered an issue while deploying DolphinScheduler using Helm. The deployment fails with an error related to S3 configuration. Below are the details of the error and my setup. ### What you expected to happen Error log: `org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'storageOperate' defined in org.apache.dolphinscheduler.plugin.storage.api.StorageConfiguration: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.apache.dolphinscheduler.plugin.storage.api.StorageOperator]: Factory method 'storageOperate' threw exception; nested exception is com.amazonaws.SdkClientException: Unable to execute HTTP request: s3 at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:658) at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:486) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1352) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1195) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:955) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583) at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:147) at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:734) at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:408) at org.springframework.boot.SpringApplication.run(SpringApplication.java:308) at org.springframework.boot.SpringApplication.run(SpringApplication.java:1306) at org.springframework.boot.SpringApplication.run(SpringApplication.java:1295) at org.apache.dolphinscheduler.server.master.MasterServer.main(MasterServer.java:102) Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.apache.dolphinscheduler.plugin.storage.api.StorageOperator]: Factory method 'storageOperate' threw exception; nested exception is com.amazonaws.SdkClientException: Unable to execute HTTP request: s3 at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185) at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:653) ... 19 common frames omitted Caused by: com.amazonaws.SdkClientException: Unable to execute HTTP request: s3 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1219) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1165) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403) at com.amazonaws.services.s3.AmazonS3Client.getAcl(AmazonS3Client.java:4062) at com.amazonaws.services.s3.AmazonS3Client.getBucketAcl(AmazonS3Client.java:1278) at com.amazonaws.services.s3.AmazonS3Client.getBucketAcl(AmazonS3Client.java:1268) at com.amazonaws.services.s3.AmazonS3Client.doesBucketExistV2(AmazonS3Client.java:1406) at org.apache.dolphinscheduler.plugin.storage.s3.S3StorageOperator.exceptionWhenBucketNameNotExists(S3StorageOperator.java:209) at org.apache.dolphinscheduler.plugin.storage.s3.S3StorageOperator.<init>(S3StorageOperator.java:71) at org.apache.dolphinscheduler.plugin.storage.s3.S3StorageOperatorFactory.createStorageOperate(S3StorageOperatorFactory.java:34) at org.apache.dolphinscheduler.plugin.storage.api.StorageConfiguration.lambda$storageOperate$0(StorageConfiguration.java:43) at java.util.Optional.map(Optional.java:215) at org.apache.dolphinscheduler.plugin.storage.api.StorageConfiguration.storageOperate(StorageConfiguration.java:38) at org.apache.dolphinscheduler.plugin.storage.api.StorageConfiguration$$EnhancerBySpringCGLIB$$33e8a190.CGLIB$storageOperate$0(<generated>) at org.apache.dolphinscheduler.plugin.storage.api.StorageConfiguration$$EnhancerBySpringCGLIB$$33e8a190$$FastClassBySpringCGLIB$$d3a2ba71.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244) at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:331) at org.apache.dolphinscheduler.plugin.storage.api.StorageConfiguration$$EnhancerBySpringCGLIB$$33e8a190.storageOperate(<generated>) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154) ... 20 common frames omitted Caused by: java.net.UnknownHostException: s3 at java.net.InetAddress$CachedAddresses.get(InetAddress.java:764) at java.net.InetAddress.getAllByName0(InetAddress.java:1291) at java.net.InetAddress.getAllByName(InetAddress.java:1144) at java.net.InetAddress.getAllByName(InetAddress.java:1065) at com.amazonaws.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:27) at com.amazonaws.http.DelegatingDnsResolver.resolve(DelegatingDnsResolver.java:38) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:112) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76) at com.amazonaws.http.conn.$Proxy191.connect(Unknown Source) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1346) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157) ... 49 common frames omitted [WI-0][TI-0] - 2024-12-04 00:02:55.317 ERROR [Master-Server] o.a.d.c.t.DefaultUncaughtExceptionHandler:[44] - Caught an exception in Thread[Master-Server,5,main]. org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'storageOperate' defined in org.apache.dolphinscheduler.plugin.storage.api.StorageConfiguration: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.apache.dolphinscheduler.plugin.storage.api.StorageOperator]: Factory method 'storageOperate' threw exception; nested exception is com.amazonaws.SdkClientException: Unable to execute HTTP request: s3 at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:658) at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:486) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1352) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1195) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:955) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583) at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:147) at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:734) at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:408) at org.springframework.boot.SpringApplication.run(SpringApplication.java:308) at org.springframework.boot.SpringApplication.run(SpringApplication.java:1306) at org.springframework.boot.SpringApplication.run(SpringApplication.java:1295) at org.apache.dolphinscheduler.server.master.MasterServer.main(MasterServer.java:102) Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.apache.dolphinscheduler.plugin.storage.api.StorageOperator]: Factory method 'storageOperate' threw exception; nested exception is com.amazonaws.SdkClientException: Unable to execute HTTP request: s3 at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185) at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:653) ... 19 common frames omitted Caused by: com.amazonaws.SdkClientException: Unable to execute HTTP request: s3 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1219) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1165) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403) at com.amazonaws.services.s3.AmazonS3Client.getAcl(AmazonS3Client.java:4062) at com.amazonaws.services.s3.AmazonS3Client.getBucketAcl(AmazonS3Client.java:1278) at com.amazonaws.services.s3.AmazonS3Client.getBucketAcl(AmazonS3Client.java:1268) at com.amazonaws.services.s3.AmazonS3Client.doesBucketExistV2(AmazonS3Client.java:1406) at org.apache.dolphinscheduler.plugin.storage.s3.S3StorageOperator.exceptionWhenBucketNameNotExists(S3StorageOperator.java:209) at org.apache.dolphinscheduler.plugin.storage.s3.S3StorageOperator.<init>(S3StorageOperator.java:71) at org.apache.dolphinscheduler.plugin.storage.s3.S3StorageOperatorFactory.createStorageOperate(S3StorageOperatorFactory.java:34) at org.apache.dolphinscheduler.plugin.storage.api.StorageConfiguration.lambda$storageOperate$0(StorageConfiguration.java:43) at java.util.Optional.map(Optional.java:215) at org.apache.dolphinscheduler.plugin.storage.api.StorageConfiguration.storageOperate(StorageConfiguration.java:38) at org.apache.dolphinscheduler.plugin.storage.api.StorageConfiguration$$EnhancerBySpringCGLIB$$33e8a190.CGLIB$storageOperate$0(<generated>) at org.apache.dolphinscheduler.plugin.storage.api.StorageConfiguration$$EnhancerBySpringCGLIB$$33e8a190$$FastClassBySpringCGLIB$$d3a2ba71.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244) at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:331) at org.apache.dolphinscheduler.plugin.storage.api.StorageConfiguration$$EnhancerBySpringCGLIB$$33e8a190.storageOperate(<generated>) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154) ... 20 common frames omitted Caused by: java.net.UnknownHostException: s3 at java.net.InetAddress$CachedAddresses.get(InetAddress.java:764) at java.net.InetAddress.getAllByName0(InetAddress.java:1291) at java.net.InetAddress.getAllByName(InetAddress.java:1144) at java.net.InetAddress.getAllByName(InetAddress.java:1065) at com.amazonaws.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:27) at com.amazonaws.http.DelegatingDnsResolver.resolve(DelegatingDnsResolver.java:38) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:112) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76) at com.amazonaws.http.conn.$Proxy191.connect(Unknown Source) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1346) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157) ... 49 common frames omitted ` Helm Values Configuration: ` conf: auto: false common: data.basedir.path: /tmp/dolphinscheduler resource.storage.type: S3 resource.storage.upload.base.path: /dolphinscheduler resource.aws.access.key.id: id resource.aws.secret.access.key: key resource.aws.region: us-east-1 resource.aws.s3.bucket.name: emr-cdcicd-test resource.aws.s3.endpoint: https://S3.us-east-1.amazonaws.com resource.hdfs.fs.defaultFS: s3a://dolphinscheduler ` I make sure s3 key is avaiable. And This is my step to install dolphinscheduler. export VERSION=3.2.2 helm pull oci://registry-1.docker.io/apache/dolphinscheduler-helm --version ${VERSION} tar -xvf dolphinscheduler-helm-${VERSION}.tgz cd dolphinscheduler-helm helm repo add bitnami https://charts.bitnami.com/bitnami helm dependency update . helm install dolphinscheduler . -n dolphin ### How to reproduce export VERSION=3.2.2 helm pull oci://registry-1.docker.io/apache/dolphinscheduler-helm --version ${VERSION} tar -xvf dolphinscheduler-helm-${VERSION}.tgz cd dolphinscheduler-helm helm repo add bitnami https://charts.bitnami.com/bitnami helm dependency update . helm install dolphinscheduler . -n dolphin ### Anything else _No response_ ### Version 3.2.x ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
