voonhous commented on code in PR #19791:
URL: https://github.com/apache/hudi/pull/19791#discussion_r3886677396
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableSink.java:
##########
@@ -79,6 +79,10 @@ public HoodieTableSink(Configuration conf, ResolvedSchema
schema, boolean overwr
public SinkRuntimeProvider getSinkRuntimeProvider(Context context) {
return (DataStreamSinkProviderAdapter) dataStream -> {
+ // validate the finalized write operation (after #applyOverwrite /
#applyStaticPartition)
+ // before any table initialization takes place.
+ OptionsResolver.checkNonBlockingConcurrencyControl(conf);
Review Comment:
Verified at 7c36e9ba: single call at the top of
`Pipelines.hoodieStreamWrite`, sink and V2 copies removed. Resolving.
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala:
##########
@@ -617,8 +618,29 @@ class HoodieSparkSqlWriterInternal {
}
/**
- * Resolve wildcards in partitions
+ * Reject insert overwrite combined with non-blocking concurrency control.
*
+ * Insert overwrite reuses the deterministic bucket file id under
non-blocking concurrency
+ * control, but the replace commit records that same file id as replaced.
The file system view
+ * hides a replaced file group by file id (ignoring the replace instant), so
the freshly
+ * overwritten data would become invisible. Reject the combination to avoid
data loss.
+ */
+ private def validateNonBlockingConcurrencyControl(hoodieConfig:
HoodieConfig, operation: WriteOperationType): Unit = {
Review Comment:
Verified at 7c36e9ba: `preWrite` now rejects NB-CC + `isOverwrite` before
`setOperationType`; covers `StreamSync` and, via
`BaseDatasetBulkInsertCommitActionExecutor.preExecute`, the row writer too.
Resolving.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
