matago opened a new issue, #6523:
URL: https://github.com/apache/iceberg/issues/6523
### Apache Iceberg version
0.14.1
### Query engine
EMR
### Please describe the bug 🐞
- Spark (3.3.0-amzn-1) on EMR (emr-6.9.0)
**Configuration**
```json
{
"applicationConfiguration": [
{
"classification": "spark-defaults",
"properties": {
"spark.jars":
"/usr/share/aws/iceberg/lib/iceberg-spark3-runtime.jar",
"spark.sql.extensions":
"org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions",
"spark.sql.catalog.glue_catalog":
"org.apache.iceberg.spark.SparkCatalog",
"spark.sql.catalog.glue_catalog.catalog-impl":
"org.apache.iceberg.aws.glue.GlueCatalog",
"spark.sql.catalog.glue_catalog.warehouse":
"s3://'$SCRIPT_BUCKET'/iceberg/",
"spark.sql.catalog.glue_catalog.io-impl":
"org.apache.iceberg.aws.s3.S3FileIO",
"spark.sql.catalog.glue_catalog.client.factory":
"org.apache.iceberg.aws.lakeformation.LakeFormationAwsClientFactory",
"spark.sql.catalog.glue_catalog.client.assume-role.arn":
"$ASSUME_ROLE_ARN",
"spark.sql.catalog.glue_catalog.glue.lakeformation-enabled": "true",
"spark.sql.catalog.glue_catalog.client.assume-role.tags.LakeFormationAuthorizedCaller":
"$ASSUME_ROLE_TAG",
"spark.sql.catalog.glue_catalog.client.assume-role.region":
"us-east-1"
}
}
],
"monitoringConfiguration": {
"s3MonitoringConfiguration": {
"logUri": "s3://'$SCRIPT_BUCKET'/elasticmapreduce/emr-containers"
}
}
}
```
Table creation fails as the job attempts to make a GetTable glue sdk call
and does not handle the table not existing in the Glue Catalog
```
Job failed, please check complete logs in configured logging destination.
ExitCode: 1. Last few exceptions: :
software.amazon.awssdk.services.glue.model.EntityNotFoundException: Table taxis
not found. (Service: Glue, Status Code: 400, Request ID: <>)
```
```python
spark.sql("""
CREATE TABLE IF NOT EXISTS glue_catalog.iceberg_lake_demo.taxis (
vendor_id bigint,
trip_id bigint,
trip_distance float,
fare_amount double,
store_and_fwd_flag string
)
USING iceberg
PARTITIONED BY (vendor_id)
TBLPROPERTIES ('format'='parquet', 'glue.skip-name-validation'=true)
;""")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
