wangsheng has posted comments on this change. (
http://gerrit.cloudera.org:8080/16599 )
Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
......................................................................
Patch Set 7:
> (1 comment)
We need to use relative path when generate data files by Spark, here is a
simple demo, hope it helps:
val conf = new Configuration()
val warehouseLoc =
"/test-warehouse/iceberg_test/hadoop_catalog/hadoop_catalog_test"
val tblLoc =
"/test-warehouse/iceberg_test/hadoop_catalog/hadoop_catalog_test/functional_parquet/hadoop_catalog_test"
val catalog = new HadoopCatalog(conf, warehouseLoc);
val sparkSchema = StructType(List(StructField("id", IntegerType,true),
StructField("user", StringType,false),StructField("action", StringType,false),
StructField("event_time", TimestampType,false)))
val icebergSchema = SparkSchemaUtil.convert(sparkSchema)
val spec =
PartitionSpec.builderFor(icebergSchema).hour("event_time").identity("action").build
val table = catalog.createTable(TableIdentifier.of("functional_parquet",
"hadoop_catalog_test") ,icebergSchema, spec, null)
val action_data = Seq(
Row(1,"Alex","view",Timestamp.valueOf("2020-01-01 08:00:00"))
)
val df = spark.createDataFrame(sc.makeRDD(action_data), sparkSchema)
df.write.format("iceberg").option("write-format",
"parquet").mode("append").save(tblLoc)
spark.read.format("iceberg").load(tblLoc).show
--
To view, visit http://gerrit.cloudera.org:8080/16599
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5
Gerrit-Change-Number: 16599
Gerrit-PatchSet: 7
Gerrit-Owner: Gabor Kaszab <[email protected]>
Gerrit-Reviewer: Gabor Kaszab <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Qifan Chen <[email protected]>
Gerrit-Reviewer: Tim Armstrong <[email protected]>
Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
Gerrit-Reviewer: wangsheng <[email protected]>
Gerrit-Comment-Date: Wed, 18 Nov 2020 12:28:27 +0000
Gerrit-HasComments: No
