Hi, Short answer, by default any parameter you pass in using option(k,v) or options() beginning with "_" would be saved to the commit metadata. You can change "_" prefix to something else by using the DataSourceWriteOptions.COMMIT_METADATA_KEYPREFIX_OPT_KEY(). Reason you are not seeing the checkpointstr inside the commit metadata is because its just supposed to be a prefix for all such commit metadata.
val metaMap = parameters.filter(kv => kv._1.startsWith(parameters(COMMIT_METADATA_KEYPREFIX_OPT_KEY))) On Thu, May 30, 2019 at 2:56 AM Netsanet Gebretsadkan <[email protected]> wrote: > I am trying to use the HoodieSparkSQLWriter to upsert data from any > dataframe into a hoodie modeled table. Its creating everything correctly > but , i also want to save the checkpoint but i couldn't even though am > passing it as an argument. > > inputDF.write() > .format("com.uber.hoodie") > .option(DataSourceWriteOptions.RECORDKEY_FIELD_OPT_KEY(), "_row_key") > .option(DataSourceWriteOptions.PARTITIONPATH_FIELD_OPT_KEY(), "partition") > .option(DataSourceWriteOptions.PRECOMBINE_FIELD_OPT_KEY(), "timestamp") > .option(HoodieWriteConfig.TABLE_NAME, tableName) > .option(DataSourceWriteOptions.COMMIT_METADATA_KEYPREFIX_OPT_KEY(), > checkpointstr) > .mode(SaveMode.Append) > .save(basePath); > > am using the COMMIT_METADATA_KEYPREFIX_OPT_KEY() for inserting the > checkpoint while using the dataframe writer but i couldn't add the > checkpoint meta data in to the .hoodie meta data. Is there a way i can add > the checkpoint meta data while using the dataframe writer API? >
