zhangxia1030 opened a new issue #3705:
URL: https://github.com/apache/iceberg/issues/3705
1. build sparksession
```
val spark = SparkSession
.builder()
.config("spark.sql.extensions",
"org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions")
.config("spark.sql.catalog.spark_catalog",
"org.apache.iceberg.spark.SparkSessionCatalog")
.config("spark.sql.catalog.spark_catalog.type", "hive")
.config(SQLConf.PARTITION_OVERWRITE_MODE.key, "dynamic")
.config("spark.hadoop.hive.metastore.uris", "thrift://ip:port")
.config("hive.exec.dynamic.partition", "true")
.config("hive.exec.dynamic.partition.mode", "nonstrict")
.appName("test-iceberg")
.master("local[*]")
.enableHiveSupport()
.getOrCreate()
```
2. use hive catalog
```
val v2catlog =
spark.sessionState.catalogManager.v2SessionCatalog.asInstanceOf[SparkSessionCatalog[SparkCatalog]]
v2catlog.createTable(id, schema, Spark3Util.toTransforms(spec.build()),
immutableMap)
```
error log:
```
Caused by: org.apache.hadoop.ipc.RemoteException: File does not exist:
/path/metadata/version-hint.text
at
org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:71)
at
org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61)
```
**Spark 3: HadoopTables create table, that not sync hive metastore?**
3. use hadoop catalog
```
val tables = new HadoopTables(spark.sparkContext.hadoopConfiguration)
tables.create(schema, spec, sort, immutableMap, path)
```
**spark struct streaming works normally, but I need to synchronize the meta
information to hive metastroe, Is there any way?**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
