Hi:
I have code that write to hudi in append mode:
val hudiOptions = Map[String,String](
HoodieWriteConfig.TABLE_NAME -> tableName,
DataSourceWriteOptions.RECORDKEY_FIELD_OPT_KEY -> pk,
DataSourceWriteOptions.PRECOMBINE_FIELD_OPT_KEY -> timestampKey,
DataSourceWriteOptions.HIVE_SYNC_ENABLED_OPT_KEY -> (env !=
Options.LOCAL).toString,
DataSourceWriteOptions.HIVE_TABLE_OPT_KEY -> tableName,
DataSourceWriteOptions.HIVE_DATABASE_OPT_KEY -> db,
DataSourceWriteOptions.HIVE_URL_OPT_KEY ->
s"jdbc:hive2://${emsMaster(env)}:10000/;ssl=true"
)
df.write
.format("org.apache.hudi")
.options(hudiOptions)
.option(DataSourceWriteOptions.PARTITIONPATH_FIELD_OPT_KEY, partitionKey)
.option(DataSourceWriteOptions.HIVE_PARTITION_FIELDS_OPT_KEY, partitionKey)
.mode(SaveMode.Append)
.save(path)
The partition key is like 2021/05/03, the id and timestamps are unique.
The same code works for one table, but not for another
1. It only write a few records on the last day, all data in previous day got
wipped out.
2. It has nested spark structure
3. it also failed to sync to hive (but no error reported). Eeven if I ran
hive-sync tool, it will not work (no exception)
Is there anything else I need to take a look?
Thanks
Andrew
----------------------------------------------------------------------
This message and any attachments are intended only for the use of the addressee
and may contain information that is privileged and confidential. If the reader
of the message is not the intended recipient or an authorized representative of
the intended recipient, you are hereby notified that any dissemination of this
communication is strictly prohibited. If you have received this communication
in error, notify the sender immediately by return email and delete the message
and any attachments from your system.
