danhuawang opened a new issue, #5502:
URL: https://github.com/apache/gravitino/issues/5502
### Version
main branch
### Describe what's wrong
**When the storage is remote S3 storage**
**Set the environment variables as following:**
```
# env
GRAVITINO_S3_SECRET_KEY=*****
GRAVITINO_S3_REGION=ap-northeast-1
HOSTNAME=bc0d4eef54fc
JAVA_HOME=/usr/local/openjdk-17
PWD=/root/gravitino-iceberg-rest-server
GRAVITINO_URI=jdbc:mysql://mysql:3306/db
HOME=/root
GRAVITINO_S3_ACCESS_KEY=*****
LANG=C.UTF-8
TERM=xterm
SHLVL=1
GRAVITINO_WAREHOUSE=s3://trino-s3-test/rest/user/iceberg/warehouse
GRAVITINO_IO_IMPL=org.apache.iceberg.aws.s3.S3FileIO
GRAVITINO_CREDENTIAL_PROVIDER_TYPE=s3-secret-key
PATH=/usr/local/openjdk-17/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
JAVA_VERSION=17.0.2
_=/usr/bin/env
```
**Set other necessary variables in gravitino-iceberg-rest-server.conf (
except these that set up as environment variables ).**
```
gravitino.iceberg-rest.requestHeaderSize = 131072
gravitino.iceberg-rest.jdbc-password = mysql
gravitino.iceberg-rest.catalog-backend = jdbc
gravitino.iceberg-rest.jdbc-initialize = true
gravitino.iceberg-rest.minThreads = 24
gravitino.iceberg-rest.jdbc.schema-version = V1
gravitino.iceberg-rest.idleTimeout = 30000
gravitino.iceberg-rest.jdbc-user = mysql
gravitino.iceberg-rest.httpPort = 9001
gravitino.iceberg-rest.host = 0.0.0.0
gravitino.iceberg-rest.jdbc-driver = com.mysql.cj.jdbc.Driver
gravitino.iceberg-rest.stopTimeout = 30000
gravitino.iceberg-rest.warehouse =
s3://trino-s3-test/rest/user/iceberg/warehouse/
gravitino.iceberg-rest.responseHeaderSize = 131072
gravitino.iceberg-rest.threadPoolWorkQueueSize = 100
gravitino.iceberg-rest.shutdown.timeout = 3000
gravitino.iceberg-rest.maxThreads = 200
gravitino.iceberg-rest.uri = jdbc:mysql://mysql:3306/db
gravitino.iceberg-rest.credential-provider-type = s3-secret-key
```
Error log in spark-sql
```
Server error: RuntimeException: java.lang.IllegalArgumentException: null in
s3-access-key-id is invalid. The value can't be blank
org.apache.iceberg.exceptions.ServiceFailureException: Server error:
RuntimeException: java.lang.IllegalArgumentException: null in s3-access-key-id
is invalid. The value can't be blank
at
org.apache.iceberg.rest.ErrorHandlers$DefaultErrorHandler.accept(ErrorHandlers.java:217)
at
org.apache.iceberg.rest.ErrorHandlers$NamespaceErrorHandler.accept(ErrorHandlers.java:180)
at
org.apache.iceberg.rest.ErrorHandlers$NamespaceErrorHandler.accept(ErrorHandlers.java:166)
at org.apache.iceberg.rest.HTTPClient.throwFailure(HTTPClient.java:201)
at org.apache.iceberg.rest.HTTPClient.execute(HTTPClient.java:313)
at org.apache.iceberg.rest.HTTPClient.execute(HTTPClient.java:252)
at org.apache.iceberg.rest.HTTPClient.get(HTTPClient.java:348)
at org.apache.iceberg.rest.RESTClient.get(RESTClient.java:96)
at org.apache.iceberg.rest.RESTClient.get(RESTClient.java:79)
at
org.apache.iceberg.rest.RESTSessionCatalog.loadNamespaceMetadata(RESTSessionCatalog.java:517)
at
org.apache.iceberg.catalog.BaseSessionCatalog$AsCatalog.loadNamespaceMetadata(BaseSessionCatalog.java:138)
at
org.apache.iceberg.rest.RESTCatalog.loadNamespaceMetadata(RESTCatalog.java:233)
at
org.apache.iceberg.spark.SparkCatalog.loadNamespaceMetadata(SparkCatalog.java:462)
at
org.apache.spark.sql.connector.catalog.SupportsNamespaces.namespaceExists(SupportsNamespaces.java:98)
at
org.apache.spark.sql.execution.datasources.v2.CreateNamespaceExec.run(CreateNamespaceExec.scala:43)
at
org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result$lzycompute(V2CommandExec.scala:43)
at
org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result(V2CommandExec.scala:43)
at
org.apache.spark.sql.execution.datasources.v2.V2CommandExec.executeCollect(V2CommandExec.scala:49)
at
org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:98)
at
org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:118)
at
org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:195)
at
org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:103)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:827)
at
org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:65)
at
org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:98)
at
org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:94)
at
org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:512)
at
org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:104)
at
org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:512)
at
org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:31)
at
org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
at
org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
```
**After add AKSK to gravitino-iceberg-rest-server.conf ,rerun, There's
Error log in gravitino-iceberg-rest-server.log**
```
Caused by: org.apache.hadoop.fs.UnsupportedFileSystemException: No
FileSystem for scheme "s3"
at
org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3239)
~[hadoop-common-2.10.2.jar:?]
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3259)
~[hadoop-common-2.10.2.jar:?]
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:121)
~[hadoop-common-2.10.2.jar:?]
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3310)
~[hadoop-common-2.10.2.jar:?]
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3278)
~[hadoop-common-2.10.2.jar:?]
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:475)
~[hadoop-common-2.10.2.jar:?]
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:356)
~[hadoop-common-2.10.2.jar:?]
at org.apache.iceberg.hadoop.Util.getFs(Util.java:56)
~[iceberg-core-1.5.2.jar:?]
```
**After add "gravitino.iceberg-rest.io-impl =
org.apache.iceberg.aws.s3.S3FileIO" to gravitino-iceberg-rest-server.conf ,run
successfully.**
### Error message and/or stacktrace
see above
### How to reproduce
see above
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
