raheen1 commented on issue #5899:
URL: https://github.com/apache/seatunnel/issues/5899#issuecomment-1831317036
And when I do run with the supported bucket format, i.e.
`s3a://<bucket_name>`
Config file:
```
# Defining the runtime environment
env {
# You can set flink configuration here
execution.parallelism = 1
job.mode = "BATCH"
}
source {
S3File {
path = "/sample.json"
fs.s3a.endpoint="s3.us-south.cloud-object-storage.appdomain.cloud"
fs.s3a.aws.credentials.provider =
"org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider"
access_key = "#####"
secret_key = "######"
bucket = "s3a://apache-seatunnel-test-connector"
file_format_type = "json"
}
}
transform {
# If you would like to get more information about how to configure
seatunnel and see full list of transform plugins,
# please go to https://seatunnel.apache.org/docs/category/transform-v2
}
sink {
Console {}
}
```
I get the following error:
```
Caused by: java.nio.file.AccessDeniedException:
s3a://apache-seatunnel-test-connector/sample.json: getFileStatus on
s3a://apache-seatunnel-test-connector/sample.json:
com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon
S3; Status Code: 403; Error Code: 403 Forbidden; Request ID:
e2aca778-23bc-417e-ad97-e2001c176d9a; S3 Extended Request ID: null), S3
Extended Request ID: null:403 Forbidden
```
a more detailed error:
```
2023-11-28 22:26:57,273 INFO
org.apache.seatunnel.core.starter.utils.ConfigBuilder - Parsed config file: {
"env" : {
"execution.parallelism" : 1,
"job.mode" : "BATCH"
},
"source" : [
{
"bucket" : "s3a://apache-seatunnel-test-connector",
"path" : "/sample.json",
"secret_key" : "#######",
"file_format_type" : "json",
"access_key" : "#######",
"fs.s3a.aws.credentials.provider" :
"org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider",
"plugin_name" : "S3File",
"fs.s3a.endpoint" :
"s3.us-south.cloud-object-storage.appdomain.cloud"
}
],
"transform" : [],
"sink" : [
{
"plugin_name" : "Console"
}
]
}
2023-11-28 22:26:57,296 INFO
org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback
configuration key 'plugin_name' instead of key 'factory'
2023-11-28 22:26:57,297 INFO
org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback
configuration key 'plugin_name' instead of key 'factory'
2023-11-28 22:26:57,301 INFO
org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load
SeaTunnelSink Plugin from /root/apache-seatunnel-2.3.3/connectors/seatunnel
2023-11-28 22:26:57,307 INFO
org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Discovery
plugin jar: S3File at:
file:/root/apache-seatunnel-2.3.3/connectors/seatunnel/connector-file-s3-2.3.3.jar
2023-11-28 22:26:57,307 INFO
org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Discovery
plugin jar: Console at:
file:/root/apache-seatunnel-2.3.3/connectors/seatunnel/connector-console-2.3.3.jar
2023-11-28 22:26:57,311 INFO
org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser - start
generating all sources.
2023-11-28 22:26:57,311 INFO
org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback
configuration key 'plugin_name' instead of key 'factory'
2023-11-28 22:26:57,325 INFO
org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load
SeaTunnelSource Plugin from /root/apache-seatunnel-2.3.3/connectors/seatunnel
2023-11-28 22:26:57,331 INFO
org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Discovery
plugin jar: S3File at:
file:/root/apache-seatunnel-2.3.3/connectors/seatunnel/connector-file-s3-2.3.3.jar
2023-11-28 22:26:57,333 INFO
org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load plugin:
PluginIdentifier{engineType='seatunnel', pluginType='source',
pluginName='S3File'} from classpath
2023-11-28 22:26:57,408 WARN
org.apache.seatunnel.connectors.seatunnel.file.sink.util.FileSystemUtils -
Principal [null] or keytabPath [null] is empty, it will skip kerberos
authentication
2023-11-28 22:26:57,486 WARN org.apache.hadoop.util.NativeCodeLoader -
Unable to load native-hadoop library for your platform... using builtin-java
classes where applicable
2023-11-28 22:26:57,661 WARN org.apache.hadoop.metrics2.impl.MetricsConfig
- Cannot locate configuration: tried
hadoop-metrics2-s3a-file-system.properties,hadoop-metrics2.properties
2023-11-28 22:26:57,672 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl - Scheduled Metric snapshot
period at 10 second(s).
2023-11-28 22:26:57,673 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl - s3a-file-system metrics
system started
2023-11-28 22:26:57,715 INFO
org.apache.seatunnel.engine.server.CoordinatorService - [localhost]:5801
[seatunnel-204936] [5.1] This node become a new active master node, begin init
coordinator service
2023-11-28 22:26:57,768 INFO
com.hazelcast.internal.partition.impl.PartitionStateManager - [localhost]:5801
[seatunnel-204936] [5.1] Initializing cluster partition table arrangement...
2023-11-28 22:26:58,824 INFO
org.apache.hadoop.conf.Configuration.deprecation -
fs.s3a.server-side-encryption-key is deprecated. Instead, use
fs.s3a.server-side-encryption.key
2023-11-28 22:26:58,943 INFO com.hazelcast.core.LifecycleService -
hz.client_1 [seatunnel-204936] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7)
is SHUTTING_DOWN
2023-11-28 22:26:58,946 INFO
com.hazelcast.internal.server.tcp.TcpServerConnection - [localhost]:5801
[seatunnel-204936] [5.1] Connection[id=1, /127.0.0.1:5801->/127.0.0.1:43331,
qualifier=null, endpoint=[127.0.0.1]:43331,
remoteUuid=3958491c-5940-4ba8-b5f0-3327765b9e67, alive=false,
connectionType=JVM, planeIndex=-1] closed. Reason: Connection closed by the
other side
2023-11-28 22:26:58,947 INFO
com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1
[seatunnel-204936] [5.1] Removed connection to endpoint:
[localhost]:5801:95ad1a34-75a5-4f47-b9c3-26c65f1d91e8, connection:
ClientConnection{alive=false, connectionId=1,
channel=NioChannel{/127.0.0.1:43331->localhost/127.0.0.1:5801},
remoteAddress=[localhost]:5801, lastReadTime=2023-11-28 22:26:57.885,
lastWriteTime=2023-11-28 22:26:57.178, closedTime=2023-11-28 22:26:58.945,
connected server version=5.1}
2023-11-28 22:26:58,947 INFO com.hazelcast.core.LifecycleService -
hz.client_1 [seatunnel-204936] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7)
is CLIENT_DISCONNECTED
2023-11-28 22:26:58,949 INFO
com.hazelcast.client.impl.ClientEndpointManager - [localhost]:5801
[seatunnel-204936] [5.1] Destroying ClientEndpoint{connection=Connection[id=1,
/127.0.0.1:5801->/127.0.0.1:43331, qualifier=null, endpoint=[127.0.0.1]:43331,
remoteUuid=3958491c-5940-4ba8-b5f0-3327765b9e67, alive=false,
connectionType=JVM, planeIndex=-1],
clientUuid=3958491c-5940-4ba8-b5f0-3327765b9e67, clientName=hz.client_1,
authenticated=true, clientVersion=5.1, creationTime=1701239216992, latest
clientAttributes=lastStatisticsCollectionTime=1701239217026,enterprise=false,clientType=JVM,clientVersion=5.1,clusterConnectionTimestamp=1701239216983,clientAddress=127.0.0.1,clientName=hz.client_1,credentials.principal=null,os.committedVirtualMemorySize=6959763456,os.freePhysicalMemorySize=251396096,os.freeSwapSpaceSize=17174609920,os.maxFileDescriptorCount=262144,os.openFileDescriptorCount=64,os.processCpuTime=6900000000,os.systemLoadAverage=0.05,os.totalPhysicalMemorySize=16525627392,os.
totalSwapSpaceSize=17175670784,runtime.availableProcessors=8,runtime.freeMemory=969642960,runtime.maxMemory=1029177344,runtime.totalMemory=1029177344,runtime.uptime=3055,runtime.usedMemory=59534384,
labels=[]}
2023-11-28 22:26:58,950 INFO com.hazelcast.core.LifecycleService -
hz.client_1 [seatunnel-204936] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7)
is SHUTDOWN
2023-11-28 22:26:58,950 INFO
org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand -
Closed SeaTunnel client......
2023-11-28 22:26:58,950 INFO com.hazelcast.core.LifecycleService -
[localhost]:5801 [seatunnel-204936] [5.1] [localhost]:5801 is SHUTTING_DOWN
2023-11-28 22:26:58,953 INFO
com.hazelcast.internal.partition.impl.MigrationManager - [localhost]:5801
[seatunnel-204936] [5.1] Shutdown request of Member [localhost]:5801 -
95ad1a34-75a5-4f47-b9c3-26c65f1d91e8 this is handled
2023-11-28 22:26:58,958 INFO com.hazelcast.instance.impl.Node -
[localhost]:5801 [seatunnel-204936] [5.1] Shutting down connection manager...
2023-11-28 22:26:58,960 INFO com.hazelcast.instance.impl.Node -
[localhost]:5801 [seatunnel-204936] [5.1] Shutting down node engine...
2023-11-28 22:27:01,078 INFO com.hazelcast.instance.impl.NodeExtension -
[localhost]:5801 [seatunnel-204936] [5.1] Destroying node NodeExtension.
2023-11-28 22:27:01,079 INFO com.hazelcast.instance.impl.Node -
[localhost]:5801 [seatunnel-204936] [5.1] Hazelcast Shutdown is completed in
2127 ms.
2023-11-28 22:27:01,079 INFO com.hazelcast.core.LifecycleService -
[localhost]:5801 [seatunnel-204936] [5.1] [localhost]:5801 is SHUTDOWN
2023-11-28 22:27:01,079 INFO
org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand -
Closed HazelcastInstance ......
2023-11-28 22:27:01,079 ERROR org.apache.seatunnel.core.starter.SeaTunnel -
===============================================================================
2023-11-28 22:27:01,079 ERROR org.apache.seatunnel.core.starter.SeaTunnel -
Fatal Error,
2023-11-28 22:27:01,079 ERROR org.apache.seatunnel.core.starter.SeaTunnel -
Please submit bug report in https://github.com/apache/seatunnel/issues
2023-11-28 22:27:01,080 ERROR org.apache.seatunnel.core.starter.SeaTunnel -
Reason:SeaTunnel job executed failed
2023-11-28 22:27:01,081 ERROR org.apache.seatunnel.core.starter.SeaTunnel -
Exception
StackTrace:org.apache.seatunnel.core.starter.exception.CommandExecuteException:
SeaTunnel job executed failed
at
org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:191)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at
org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by:
org.apache.seatunnel.connectors.seatunnel.file.exception.FileConnectorException:
ErrorCode:[FILE-03], ErrorDescription:[Get file list failed] - Get file list
from this path [/sample.json] failed
at
org.apache.seatunnel.connectors.seatunnel.file.s3.source.S3FileSource.prepare(S3FileSource.java:75)
at
org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSource(JobConfigParser.java:85)
at
org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSource(MultipleTableJobConfigParser.java:317)
at
org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:179)
at
org.apache.seatunnel.engine.core.job.AbstractJobEnvironment.getLogicalDag(AbstractJobEnvironment.java:109)
at
org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:73)
at
org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:143)
... 2 more
Caused by: java.nio.file.AccessDeniedException:
s3a://apache-seatunnel-test-connector/sample.json: getFileStatus on
s3a://apache-seatunnel-test-connector/sample.json:
com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon
S3; Status Code: 403; Error Code: 403 Forbidden; Request ID:
e2aca778-23bc-417e-ad97-e2001c176d9a; S3 Extended Request ID: null), S3
Extended Request ID: null:403 Forbidden
at
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:218)
at
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:145)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:2184)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2149)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2088)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.innerListStatus(S3AFileSystem.java:1889)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$listStatus$9(S3AFileSystem.java:1868)
at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:109)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.listStatus(S3AFileSystem.java:1868)
at
org.apache.seatunnel.connectors.seatunnel.file.source.reader.AbstractReadStrategy.getFileNamesByPath(AbstractReadStrategy.java:130)
at
org.apache.seatunnel.connectors.seatunnel.file.s3.source.S3FileSource.prepare(S3FileSource.java:72)
... 8 more
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden
(Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID:
e2aca778-23bc-417e-ad97-e2001c176d9a; S3 Extended Request ID: null), S3
Extended Request ID: null
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1639)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1304)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1056)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
at
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
at
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4325)
at
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4272)
at
com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1264)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getObjectMetadata$4(S3AFileSystem.java:1235)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:322)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:285)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:1232)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:2169)
... 16 more
2023-11-28 22:27:01,082 ERROR org.apache.seatunnel.core.starter.SeaTunnel -
===============================================================================
Exception in thread "main"
org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel
job executed failed
at
org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:191)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at
org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by:
org.apache.seatunnel.connectors.seatunnel.file.exception.FileConnectorException:
ErrorCode:[FILE-03], ErrorDescription:[Get file list failed] - Get file list
from this path [/sample.json] failed
at
org.apache.seatunnel.connectors.seatunnel.file.s3.source.S3FileSource.prepare(S3FileSource.java:75)
at
org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSource(JobConfigParser.java:85)
at
org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSource(MultipleTableJobConfigParser.java:317)
at
org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:179)
at
org.apache.seatunnel.engine.core.job.AbstractJobEnvironment.getLogicalDag(AbstractJobEnvironment.java:109)
at
org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:73)
at
org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:143)
... 2 more
Caused by: java.nio.file.AccessDeniedException:
s3a://apache-seatunnel-test-connector/sample.json: getFileStatus on
s3a://apache-seatunnel-test-connector/sample.json:
com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon
S3; Status Code: 403; Error Code: 403 Forbidden; Request ID:
e2aca778-23bc-417e-ad97-e2001c176d9a; S3 Extended Request ID: null), S3
Extended Request ID: null:403 Forbidden
at
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:218)
at
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:145)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:2184)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2149)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2088)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.innerListStatus(S3AFileSystem.java:1889)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$listStatus$9(S3AFileSystem.java:1868)
at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:109)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.listStatus(S3AFileSystem.java:1868)
at
org.apache.seatunnel.connectors.seatunnel.file.source.reader.AbstractReadStrategy.getFileNamesByPath(AbstractReadStrategy.java:130)
at
org.apache.seatunnel.connectors.seatunnel.file.s3.source.S3FileSource.prepare(S3FileSource.java:72)
... 8 more
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden
(Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID:
e2aca778-23bc-417e-ad97-e2001c176d9a; S3 Extended Request ID: null), S3
Extended Request ID: null
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1639)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1304)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1056)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
at
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
at
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4325)
at
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4272)
at
com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1264)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getObjectMetadata$4(S3AFileSystem.java:1235)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:322)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:285)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:1232)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:2169)
... 16 more
2023-11-28 22:27:01,091 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl - Stopping s3a-file-system
metrics system...
2023-11-28 22:27:01,092 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl - s3a-file-system metrics
system stopped.
2023-11-28 22:27:01,092 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl - s3a-file-system metrics
system shutdown complete.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
