jingli430 opened a new issue, #5174:
URL: https://github.com/apache/iceberg/issues/5174
Hi, I've seen same issues reported many times here. And I know I should
first check three things.
**[1] is spark version, iceberg version and scala version correct?**
yes, I am using Spark3.2.0, Scala2.12, Iceberg0.13.1. And it is running well
when I execute some other statements like insert, create, etc.
**[2] did I include spark.sql.extensions?**
yes, below is my spark-sql config
`spark-sql --master yarn --deploy-mode client --name
sparksql_iceberg_session \
--driver-cores 1 --driver-memory 1g --executor-cores 2 --executor-memory 16g
--num-executors 2 \
--conf spark.memory.fraction=0.8 --conf spark.storage.memoryFraction=0.3 \
--packages org.apache.iceberg:iceberg-spark-runtime-3.2_2.12:0.13.1 \
--conf
spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
\
--conf spark.sql.catalog.hive_prod=org.apache.iceberg.spark.SparkCatalog \
--conf spark.sql.catalog.hive_prod.type=hive \
--conf spark.sql.catalog.hive_prod.uri=thrift://emr-header-1.cluster:9083 \
--conf spark.sql.catalog.hive_prod.warehouse=oss://xp-warehouse-iceberg/ \
--conf spark.sql.catalog.hive_prod.access.key.id=xxx \
--conf spark.sql.catalog.hive_prod.access.key.secret=yyy \
--conf
spark.sql.catalog.hive_prod.oss.endpoint=oss-cn-shanghai-internal.aliyuncs.com \
--conf
spark.sql.catalog.session_prod=org.apache.iceberg.spark.SparkSessionCatalog \
--conf spark.sql.catalog.session_prod.type=hive --conf
spark.sql.defaultCatalog=hive_prod`
**[3] is target iceberg table?**
yes, But I still saw this error when doing merge into command below at my
spark-sql shell.
`CREATE TABLE xiceberg_dev.ib_mergeinto_target_upsert (
order_id int,
order_ts timestamp,
order_date date,
price int,
order_status boolean,
PRIMARY KEY (order_id) NOT ENFORCED
)
WITH (
'format-version'= '2',
'write.upsert.enable'='true',
'write.distribution-mode'='hash',
'write.metadata.delete-after-commit.enabled'='true',
'write.metadata.previous-versions-max'='9'
);
MERGE INTO hive_prod.xiceberg_dev.ib_mergeinto_target_upsert as target
USING hive_prod.xiceberg_dev.ib_mergeinto_source_insert_pt as source
ON target.order_id = source.order_id
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *
;`
Could someone help me here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
