nsivabalan opened a new pull request, #20415: URL: https://github.com/apache/hudi/pull/20415
### Describe the issue this Pull Request addresses Stacked on #20412 and #20413. **Review only the last commit**; the first two commits are those PRs and will drop out once they merge. Related: #20400. `HoodieStreamingSink` reuses one write client across micro-batches, so the client's write config keeps the schema of the micro-batch that created it. (On master the sink already hands the batch-0 client back on every later batch; #20412 makes the reuse explicit.) When the table schema evolves under a running query, e.g. another writer adds a column, later micro-batches write with the stale schema: - With default configs, the upsert fails `HoodieTable.validateSchema` (`Failed upsert schema compatibility check`, read as a column drop). The batch retries, fails again, and the sink exits the JVM. - With `hoodie.datasource.write.schema.allow.auto.evolution.column.drop=true`, the check is skipped and the stale schema is committed, silently dropping the new column from the table schema. `HoodieStreamer` does not have this problem: `StreamSync` re-creates its write client when it sees a new schema (`reInitWriteClient`). ### Summary and Changelog - `HoodieSparkSqlWriter`: when the caller owns the write client (streaming sink), reuse it only if its config's schema and internal schema match this write's writer schema; otherwise re-create it through `StreamingWriteParams.writeClientFactory`. - `HoodieStreamingSink`: when the client is replaced, hand the new client to the async compaction/clustering services and close the old one. The sink-owned timeline server keeps running across the swap (same as `StreamSync.reInitWriteClient` + `HoodieStreamer.onInitializingWriteClient`). - New test `testStructuredStreamingWithTableSchemaEvolvedMidQuery` (COW and MOR): a batch writer adds a column between two micro-batches; the next micro-batch must commit and the table schema must keep the new column. Without the fix it fails as described above. ### Impact Streaming queries keep running and write the evolved schema when the table schema changes underneath them. The client is re-created only on an actual schema change. ### Risk Level low. Only affects caller-owned (streaming sink) clients, and only when the writer schema changes. Verified with `TestStructuredStreaming` (21/21 pass): the rebuild triggered exactly twice across the suite (once each in the COW and MOR schema-change tests) and never in the async compaction/clustering tests. As with `HoodieStreamer`, the old client is closed immediately on a schema change, so an async compaction or clustering still running on it at that moment shares the same exposure as in `HoodieStreamer`. ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
