TheerachotTle opened a new issue, #214:
URL: https://github.com/apache/polaris/issues/214
### Is your feature request related to a problem? Please describe.
I have set the allowed location of the created catalog to S3 storage type
using `s3://` prefix. When I run `remove_orphan_files` procedure in Spark, it
results in an error message: `No FileSystem for scheme "s3"`. To solve this
problem, I attempted to create the catalog with the `s3a://` prefix, but I
received a 400 Bad Request error with the message: `Location prefix not
allowed`.
Here's my spark configuration
```
spark = SparkSession.builder \
.config("spark.jars.packages","org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.5.2,org.apache.iceberg:iceberg-aws-bundle:1.5.2,org.apache.hadoop:hadoop-aws:3.4.0,org.apache.hadoop:hadoop-common:3.4.0")
\
.config("spark.sql.extensions",
"org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions") \
.config('spark.sql.catalog.polaris.header.X-Iceberg-Access-Delegation', 'true')
\
.config("spark.sql.catalog.polaris.uri", POLARIS_URI) \
.config("spark.sql.catalog.polaris.type", "rest") \
.config("spark.sql.catalog.polaris",
"org.apache.iceberg.spark.SparkCatalog") \
.config("spark.sql.catalog.polaris.warehouse",
POLARIS_CATALOG_NAME) \
.config("spark.sql.catalog.polaris.io-impl",
"org.apache.iceberg.aws.s3.S3FileIO") \
.config("spark.hadoop.fs.s3a.impl",
"org.apache.hadoop.fs.s3a.S3AFileSystem") \
.config('spark.sql.catalog.polaris.credential',
POLARIS_CREDENTIALS) \
.config('spark.sql.catalog.polaris.scope', POLARIS_SCOPE) \
.config('spark.sql.catalog.polaris.token-refresh-enabled',
'true') \
.getOrCreate()
```
### Describe the solution you'd like
Probably add the s3a:// prefix as an alternative for the S3 storage type.
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
