whatsthecraic opened a new issue, #13592:
URL: https://github.com/apache/iceberg/issues/13592
### Query engine
(py)spark 3.5.2
iceberg 1.9.1
### Question
I wish to write an iceberg table into a s3 bucket, using pyspark.
My current configuration is :
```
packages = ",".join([
"org.apache.hadoop:hadoop-aws:3.3.4",
"org.apache.iceberg:iceberg-spark-runtime-3.5_2.13:1.9.1", # I don't
think this is necessary?
"org.apache.iceberg:iceberg-aws-bundle:1.9.1",
"software.amazon.awssdk:bundle:2.20.160",
])
spark = SparkSession.builder.appName("Iceberg-Datagen5") \
.config("spark.jars.packages", packages) \
.config("spark.hadoop.fs.s3a.aws.credentials.provider",
"com.amazonaws.auth.DefaultAWSCredentialsProviderChain") \
.config("spark.hadoop.fs.s3a.impl",
"org.apache.hadoop.fs.s3a.S3AFileSystem") \
.config("spark.hadoop.fs.s3a.access.key", S3_ACCESS_KEY_ID) \
.config("spark.hadoop.fs.s3a.secret.key", S3_SECRET_ACCESS_KEY) \
.config("spark.hadoop.fs.s3a.endpoint", "s3.amazonaws.com") \
.config("spark.hadoop.fs.s3a.endpoint.region", S3_REGION) \
.config("spark.sql.catalog.s3", "org.apache.iceberg.spark.SparkCatalog")
\
.config("spark.sql.catalog.s3.type", "hadoop") \
.config("spark.sql.catalog.s3.warehouse", "s3a://my-bucket/iceberg/") \
.config("spark.sql.catalog.s3.io-impl",
"org.apache.iceberg.aws.s3.S3FileIO") \
.config('spark.sql.extensions',
'org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions') \
.getOrCreate()
```
When attempting to create a table:
```
spark.sql("CREATE TABLE s3.test1 (A string, B bigint) USING iceberg");
```
I'm getting the error:
```
IllegalArgumentException: Cannot initialize FileIO implementation
org.apache.iceberg.aws.s3.S3FileIO: Cannot find constructor for interface
org.apache.iceberg.io.FileIO
Missing org.apache.iceberg.aws.s3.S3FileIO
[java.lang.NoClassDefFoundError:
software/amazon/awssdk/services/s3/model/S3Exception]
```
I don't know what's the correct configuration?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
