raheen1 commented on issue #5899:
URL: https://github.com/apache/seatunnel/issues/5899#issuecomment-1831292012
I am using the following config file for IBM COS to Console:
```
# Defining the runtime environment
env {
# You can set flink configuration here
execution.parallelism = 1
job.mode = "BATCH"
}
source {
S3File {
path = "/sample.json"
fs.s3a.endpoint="s3.us-south.cloud-object-storage.appdomain.cloud"
fs.s3a.aws.credentials.provider =
"org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider"
access_key = "bf136a1d5c354fb3a54a464e3127f81a"
secret_key = "ffa78af30da3890268f09a60f240e290f48083b2e7cc0405"
bucket = "cos://apache-seatunnel-test-connector"
file_format_type = "json"
}
}
transform {
# If you would like to get more information about how to configure
seatunnel and see full list of transform plugins,
# please go to https://seatunnel.apache.org/docs/category/transform-v2
}
sink {
Console {}
}
```
I get the following error when running the above connector:
`No FileSystem for scheme "cos"`
A more detailed error:
```
2023-11-28 22:19:14,640 INFO
org.apache.seatunnel.core.starter.utils.ConfigBuilder - Parsed config file: {
"env" : {
"execution.parallelism" : 1,
"job.mode" : "BATCH"
},
"source" : [
{
"bucket" : "cos://apache-seatunnel-test-connector",
"path" : "/sample.json",
"secret_key" :
"ffa78af30da3890268f09a60f240e290f48083b2e7cc0405",
"file_format_type" : "json",
"access_key" : "bf136a1d5c354fb3a54a464e3127f81a",
"fs.s3a.aws.credentials.provider" :
"org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider",
"plugin_name" : "S3File",
"fs.s3a.endpoint" :
"s3.us-south.cloud-object-storage.appdomain.cloud"
}
],
"transform" : [],
"sink" : [
{
"plugin_name" : "Console"
}
]
}
2023-11-28 22:19:14,664 INFO
org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback
configuration key 'plugin_name' instead of key 'factory'
2023-11-28 22:19:14,665 INFO
org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback
configuration key 'plugin_name' instead of key 'factory'
2023-11-28 22:19:14,669 INFO
org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load
SeaTunnelSink Plugin from /root/apache-seatunnel-2.3.3/connectors/seatunnel
2023-11-28 22:19:14,674 INFO
org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Discovery
plugin jar: S3File at:
file:/root/apache-seatunnel-2.3.3/connectors/seatunnel/connector-file-s3-2.3.3.jar
2023-11-28 22:19:14,675 INFO
org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Discovery
plugin jar: Console at:
file:/root/apache-seatunnel-2.3.3/connectors/seatunnel/connector-console-2.3.3.jar
2023-11-28 22:19:14,678 INFO
org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser - start
generating all sources.
2023-11-28 22:19:14,679 INFO
org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback
configuration key 'plugin_name' instead of key 'factory'
2023-11-28 22:19:14,692 INFO
org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load
SeaTunnelSource Plugin from /root/apache-seatunnel-2.3.3/connectors/seatunnel
2023-11-28 22:19:14,697 INFO
org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Discovery
plugin jar: S3File at:
file:/root/apache-seatunnel-2.3.3/connectors/seatunnel/connector-file-s3-2.3.3.jar
2023-11-28 22:19:14,699 INFO
org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load plugin:
PluginIdentifier{engineType='seatunnel', pluginType='source',
pluginName='S3File'} from classpath
2023-11-28 22:19:14,789 WARN
org.apache.seatunnel.connectors.seatunnel.file.sink.util.FileSystemUtils -
Principal [null] or keytabPath [null] is empty, it will skip kerberos
authentication
2023-11-28 22:19:14,874 WARN org.apache.hadoop.util.NativeCodeLoader -
Unable to load native-hadoop library for your platform... using builtin-java
classes where applicable
2023-11-28 22:19:14,992 INFO com.hazelcast.core.LifecycleService -
hz.client_1 [seatunnel-109037] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7)
is SHUTTING_DOWN
2023-11-28 22:19:14,995 INFO
com.hazelcast.internal.server.tcp.TcpServerConnection - [localhost]:5801
[seatunnel-109037] [5.1] Connection[id=1, /127.0.0.1:5801->/127.0.0.1:34765,
qualifier=null, endpoint=[127.0.0.1]:34765,
remoteUuid=2750a798-9d7b-4b78-a6f2-d1e248cc7828, alive=false,
connectionType=JVM, planeIndex=-1] closed. Reason: Connection closed by the
other side
2023-11-28 22:19:14,996 INFO
com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1
[seatunnel-109037] [5.1] Removed connection to endpoint:
[localhost]:5801:eb6e6e4f-9090-4c8a-a35a-8644809abaa2, connection:
ClientConnection{alive=false, connectionId=1,
channel=NioChannel{/127.0.0.1:34765->localhost/127.0.0.1:5801},
remoteAddress=[localhost]:5801, lastReadTime=2023-11-28 22:19:14.546,
lastWriteTime=2023-11-28 22:19:14.543, closedTime=2023-11-28 22:19:14.994,
connected server version=5.1}
2023-11-28 22:19:14,996 INFO com.hazelcast.core.LifecycleService -
hz.client_1 [seatunnel-109037] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7)
is CLIENT_DISCONNECTED
2023-11-28 22:19:14,999 INFO
com.hazelcast.client.impl.ClientEndpointManager - [localhost]:5801
[seatunnel-109037] [5.1] Destroying ClientEndpoint{connection=Connection[id=1,
/127.0.0.1:5801->/127.0.0.1:34765, qualifier=null, endpoint=[127.0.0.1]:34765,
remoteUuid=2750a798-9d7b-4b78-a6f2-d1e248cc7828, alive=false,
connectionType=JVM, planeIndex=-1],
clientUuid=2750a798-9d7b-4b78-a6f2-d1e248cc7828, clientName=hz.client_1,
authenticated=true, clientVersion=5.1, creationTime=1701238754351, latest
clientAttributes=lastStatisticsCollectionTime=1701238754382,enterprise=false,clientType=JVM,clientVersion=5.1,clusterConnectionTimestamp=1701238754342,clientAddress=127.0.0.1,clientName=hz.client_1,credentials.principal=null,os.committedVirtualMemorySize=6959763456,os.freePhysicalMemorySize=250556416,os.freeSwapSpaceSize=17174609920,os.maxFileDescriptorCount=262144,os.openFileDescriptorCount=64,os.processCpuTime=7400000000,os.systemLoadAverage=0.09,os.totalPhysicalMemorySize=16525627392,os.
totalSwapSpaceSize=17175670784,runtime.availableProcessors=8,runtime.freeMemory=966882104,runtime.maxMemory=1029177344,runtime.totalMemory=1029177344,runtime.uptime=3223,runtime.usedMemory=62295240,
labels=[]}
2023-11-28 22:19:15,000 INFO com.hazelcast.core.LifecycleService -
hz.client_1 [seatunnel-109037] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7)
is SHUTDOWN
2023-11-28 22:19:15,000 INFO
org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand -
Closed SeaTunnel client......
2023-11-28 22:19:15,000 INFO com.hazelcast.core.LifecycleService -
[localhost]:5801 [seatunnel-109037] [5.1] [localhost]:5801 is SHUTTING_DOWN
2023-11-28 22:19:15,002 INFO com.hazelcast.instance.impl.Node -
[localhost]:5801 [seatunnel-109037] [5.1] Shutting down connection manager...
2023-11-28 22:19:15,004 INFO com.hazelcast.instance.impl.Node -
[localhost]:5801 [seatunnel-109037] [5.1] Shutting down node engine...
2023-11-28 22:19:15,009 INFO
org.apache.seatunnel.engine.server.SeaTunnelServer - master node check
interrupted
2023-11-28 22:19:18,027 INFO com.hazelcast.instance.impl.NodeExtension -
[localhost]:5801 [seatunnel-109037] [5.1] Destroying node NodeExtension.
2023-11-28 22:19:18,027 INFO com.hazelcast.instance.impl.Node -
[localhost]:5801 [seatunnel-109037] [5.1] Hazelcast Shutdown is completed in
3025 ms.
2023-11-28 22:19:18,028 INFO com.hazelcast.core.LifecycleService -
[localhost]:5801 [seatunnel-109037] [5.1] [localhost]:5801 is SHUTDOWN
2023-11-28 22:19:18,028 INFO
org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand -
Closed HazelcastInstance ......
2023-11-28 22:19:18,028 ERROR org.apache.seatunnel.core.starter.SeaTunnel -
===============================================================================
2023-11-28 22:19:18,028 ERROR org.apache.seatunnel.core.starter.SeaTunnel -
Fatal Error,
2023-11-28 22:19:18,028 ERROR org.apache.seatunnel.core.starter.SeaTunnel -
Please submit bug report in https://github.com/apache/seatunnel/issues
2023-11-28 22:19:18,028 ERROR org.apache.seatunnel.core.starter.SeaTunnel -
Reason:SeaTunnel job executed failed
2023-11-28 22:19:18,029 ERROR org.apache.seatunnel.core.starter.SeaTunnel -
Exception
StackTrace:org.apache.seatunnel.core.starter.exception.CommandExecuteException:
SeaTunnel job executed failed
at
org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:191)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at
org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by:
org.apache.seatunnel.connectors.seatunnel.file.exception.FileConnectorException:
ErrorCode:[FILE-03], ErrorDescription:[Get file list failed] - Get file list
from this path [/sample.json] failed
at
org.apache.seatunnel.connectors.seatunnel.file.s3.source.S3FileSource.prepare(S3FileSource.java:75)
at
org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSource(JobConfigParser.java:85)
at
org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSource(MultipleTableJobConfigParser.java:317)
at
org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:179)
at
org.apache.seatunnel.engine.core.job.AbstractJobEnvironment.getLogicalDag(AbstractJobEnvironment.java:109)
at
org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:73)
at
org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:143)
... 2 more
Caused by: org.apache.hadoop.fs.UnsupportedFileSystemException: No
FileSystem for scheme "cos"
at
org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3281)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3301)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3352)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3320)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:227)
at
org.apache.seatunnel.connectors.seatunnel.file.source.reader.AbstractReadStrategy.getFileNamesByPath(AbstractReadStrategy.java:127)
at
org.apache.seatunnel.connectors.seatunnel.file.s3.source.S3FileSource.prepare(S3FileSource.java:72)
... 8 more
2023-11-28 22:19:18,030 ERROR org.apache.seatunnel.core.starter.SeaTunnel -
===============================================================================
Exception in thread "main"
org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel
job executed failed
at
org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:191)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at
org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by:
org.apache.seatunnel.connectors.seatunnel.file.exception.FileConnectorException:
ErrorCode:[FILE-03], ErrorDescription:[Get file list failed] - Get file list
from this path [/sample.json] failed
at
org.apache.seatunnel.connectors.seatunnel.file.s3.source.S3FileSource.prepare(S3FileSource.java:75)
at
org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSource(JobConfigParser.java:85)
at
org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSource(MultipleTableJobConfigParser.java:317)
at
org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:179)
at
org.apache.seatunnel.engine.core.job.AbstractJobEnvironment.getLogicalDag(AbstractJobEnvironment.java:109)
at
org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:73)
at
org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:143)
... 2 more
Caused by: org.apache.hadoop.fs.UnsupportedFileSystemException: No
FileSystem for scheme "cos"
at
org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3281)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3301)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3352)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3320)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:227)
at
org.apache.seatunnel.connectors.seatunnel.file.source.reader.AbstractReadStrategy.getFileNamesByPath(AbstractReadStrategy.java:127)
at
org.apache.seatunnel.connectors.seatunnel.file.s3.source.S3FileSource.prepare(S3FileSource.java:72)
... 8 more
```
Looks like when using S3File, IBM COS buckets, which are of the form,
`cos://<bucket_name>` are not supported by the backend code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
