jessiedanwang opened a new issue, #5424:
URL: https://github.com/apache/iceberg/issues/5424
### Query engine
Spark 3.1.2 on EMR 6.5
### Question
We are running Spark 3.1.2 application on EMR 6.5 in account BBB while
writing data to S3 and Glue catalog in account AAA. I would appreciate it any
ideas what could be wrong here
Here is what we did,
spark = SparkSession
.builder()
.config("spark.sql.legacy.parquet.datetimeRebaseModeInWrite",
"CORRECTED")
.config("spark.sql.extensions",
"org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions") // Iceberg
related configs
.config("spark.sql.autoBroadcastJoinThreshold", "-1")
.config("spark.sql.adaptive.autoBroadcastJoinThreshold", "-1")
.config(s"spark.sql.catalog.iceberg_catalog",
"org.apache.iceberg.spark.SparkCatalog")
.config(s"spark.sql.catalog.iceberg_catalog.warehouse",
warehousePath)
.config(s"spark.sql.catalog.iceberg_catalog.catalog-impl",
"org.apache.iceberg.aws.glue.GlueCatalog")
.config(s"spark.sql.catalog.iceberg_catalog.io-impl",
"org.apache.iceberg.aws.s3.S3FileIO")
.config(s"spark.sql.catalog.iceberg_catalog.client.factory",
"org.apache.iceberg.aws.AssumeRoleAwsClientFactory")
.config(s"spark.sql.catalog.iceberg_catalog.client.assume-role.arn",
s"arn:aws:iam::AAA:role/RoleToAssume")
.config(s"spark.sql.catalog.iceberg_catalog.client.assume-role.region",
"us-east-2")
.config("spark.hadoop.hive.metastore.glue.catalogid", AAA)
.enableHiveSupport()
.getOrCreate()
df.writeTo("ns_name.table_name")
.tableProperty("table_type", "iceberg")
.tableProperty("format-version", "2")
//.tableProperty("engine.hive.enabled", "true")
.tableProperty("write.distribution-mode", "hash")
.tableProperty("write.spark.fanout.enabled", "true")
.tableProperty("write.parquet.compression-codec", "snappy")
.tableProperty("write.avro.compression-codec", "snappy")
.tableProperty("write.metadata.delete-after-commit.enabled", "true")
.tableProperty("write.metadata.previous-versions-max", "3")
.tableProperty("write.target-file-size-bytes", s"$GIGABYTE")
.create()
df2.createOrReplaceGlobalTempView("temp_view")
val query: String = s"""MERGE INTO ns_name.table_name t
|USING (SELECT * FROM global_temp.temp_view) s
|s.id = t.id
|WHEN MATCHED THEN UPDATE SET *
|WHEN NOT MATCHED THEN INSERT *
|""".stripMargin
spark.sql(query)
Got the following error when trying to do MERGE INTO
java.lang.UnsupportedOperationException: MERGE INTO TABLE is not supported
temporarily.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
