nsivabalan opened a new pull request, #8697: URL: https://github.com/apache/hudi/pull/8697
### Change Logs We are adding [auto record key gen](https://github.com/apache/hudi/pull/8107) mainly to cater to append only use-cases. So, as part of the work stream, this is a follow up patch. This patch makes bulk_insert as the default operation type for append only use-cases. For regular pipelines (immutable), "upsert" will be the default operation. This is stacked on top of #8107. commit to review: https://github.com/apache/hudi/pull/8389/commits/cf7b54b92491c7fe516b19097d58a7f04392497a With this patch, users can now do the following w/ hudi. ``` a. non-partitioned: df.write.format("hudi").option("hoodie.table.name","hudi_tbl").save(path) -> will use bulk_insert. b. partitioned dataset: df.write.partitionBy(colA).format("hudi").option("hoodie.table.name","hudi_tbl").save(path) -> will use bulk_insert. c. multi-field partitioning: df.write.partitionBy(colA, colB).format("hudi").option("hoodie.table.name","hudi_tbl").save(path) -> will use bulk_insert. ``` If any of the config is set among (partition path, record key or partition path), default operation is chosen to be "upsert" which has been the behavior so far. ``` df.write.partitionBy(colA).format("hudi"). option("hoodie.table.name","hudi_tbl"). option("hoodie.datasource.write.precombine.field","updated_at"). save(path) -> will use upsert. ``` We do have a limitation wrt MOR table still, since precombine is a mandatory field for MOR table. ### Impact Greatly improves the performance of hudi for append only use-cases. ### Risk level (write none, low medium or high below) low. ### Documentation Update Need to update our hudi website w/ sufficient docs. ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Change Logs and Impact were stated clearly - [ ] Adequate tests were added if applicable - [ ] CI passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
