Joy-2000 opened a new pull request, #19791:
URL: https://github.com/apache/hudi/pull/19791
### Describe the issue this Pull Request addresses
Insert overwrite must not be combined with Non-Blocking Concurrency Control
(NB-CC), otherwise it silently loses data.
Under NB-CC the write reuses the deterministic bucket file id. The replace
commit produced by insert overwrite records that *same* file id as replaced.
Because the file system view hides a replaced file group by file id (ignoring
the replace instant), the freshly overwritten data becomes invisible after the
commit. Rather than silently dropping data, we reject the combination up front
with a clear error.
### Summary and Changelog
Reject `insert overwrite` / `insert overwrite table` when the write
concurrency mode is `NON_BLOCKING_CONCURRENCY_CONTROL`, across both the Flink
and Spark write paths, with the message:
> `Insert overwrite is not supported with non-blocking concurrency control`
Changes:
- **Flink**
- `OptionsResolver#checkNonBlockingConcurrencyControl` — new validation
that fails fast when NB-CC is combined with insert overwrite.
- `HoodieTableSink#getSinkRuntimeProvider` — invokes the check on the
finalized write operation (after `applyOverwrite` / `applyStaticPartition`),
before any table initialization.
- `PipelinesV2#composePipeline` — invokes the same check at the head of
the Sink V2 pipeline.
- **Spark**
- `HoodieSparkSqlWriterInternal#validateNonBlockingConcurrencyControl` —
new validation invoked right after the write operation is deduced. Covers
`INSERT_OVERWRITE`, `INSERT_OVERWRITE_TABLE`, and the `bulk_insert`-backed
overwrite path (`BULKINSERT_OVERWRITE_OPERATION_TYPE`).
- `ProvidesHoodieConfig` — use `equalsIgnoreCase` when matching the write
operation so the overwrite mapping is resolved consistently regardless of case.
- **Tests**
- `TestOptionsResolver`, `ITTestDataStreamV2Write`,
`ITTestHoodieDataSource` (Flink) covering the resolver, Sink V2 pipeline, and
SQL (whole-table / static / dynamic partition) overwrite paths.
- `TestInsertTable5` (Spark) covering `INSERT_OVERWRITE`,
`INSERT_OVERWRITE_TABLE`, and the `bulk_insert` overwrite path.
### Impact
User-facing behavior change: an `insert overwrite` against a table
configured with non-blocking concurrency control now fails with a clear error
instead of silently losing the overwritten data. No impact on any other
operation or concurrency mode.
### Risk Level
low
The change only adds an up-front guard on an unsupported (and previously
data-losing) combination; all other write paths are unaffected. Covered by new
unit and integration tests on both engines.
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
