danny0405 opened a new issue, #20407: URL: https://github.com/apache/hudi/issues/20407
### Task Description Support upstream Hudi ST → downstream ST through catalog readStream.table / SQL FROM STREAM and Hudi's existing incremental source/V1 fallback. Explicitly project business columns and preserve supported reader-option propagation through WITH (...) or Python readStream.option(...). Define initial offsets, restart behavior, latest-state incremental output, and expired-history errors. Detect logical truncate/overwrite after a saved offset and physical table recreation before returning misleading increments; fail with a rebuild-required error. Distinguish clustering replace commits from logical reset. Document stopping consumers and rebuilding affected downstream targets/checkpoints in dependency order. Basic latest-state reads do not promise hard-delete propagation or every intermediate change. Complete CDC propagation and bounded admission control are separate work; document the current AvailableNow backlog-size limitation. ### Acceptance criteria - [ ] Two-stage pipeline consumes new upstream commits and persists the expected downstream business records. - [ ] Existing checkpoints resume without replaying prior increments; reader options and conflicts are tested. - [ ] Reserved Hudi system fields do not collide with target schema; _hoodie_is_deleted remains an explicitly supported input marker. - [ ] Expired history, upstream reset, and table recreation fail clearly before producing an unsafe batch; ordinary clustering remains readable. - [ ] An explicit coordinated rebuild produces a correct downstream snapshot. - [ ] COW/MOR supported modes and source limitations are covered in tests/documentation. ### Task Type Code improvement/refactoring ### Related Issues Parent feature issue: #20401 RFC: #19997 (under review; implementation must follow the agreed contract). Dependencies for end-to-end acceptance: #20406. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
