nsivabalan opened a new issue, #20674:
URL: https://github.com/apache/hudi/issues/20674

   ### Task Description
   
   Add a cap on how many commits one micro-batch of Hudi's Spark 
structured-streaming source consumes. This applies to plain `readStream` and to 
SDP streaming-table chaining (#20407). Target: this release. Follow-up from the 
#20516 review 
([review](https://github.com/apache/hudi/pull/20516#pullrequestreview-5930942678)).
   
   **Today.** `HoodieStreamSourceV1` / `HoodieStreamSourceV2` implement only 
Spark's `Source`, and `getOffset` always returns the latest completed instant. 
Each micro-batch therefore reads everything from the checkpoint to the newest 
commit:
   - A consumer that was stopped for a while catches up in one unbounded batch.
   - With #20516's `FAIL` default, a consumer that can't keep up fails once 
archival or cleaning overtakes its checkpoint.
   
   **Precedent.**
   - Hudi's other incremental readers already cap batches: HoodieIncrSource's 
`hoodie.streamer.source.hoodieincr.num_instants` (default 5) and Flink's 
`read.commits.limit`.
   - Delta Lake has `maxFilesPerTrigger` (default 1000) and 
`maxBytesPerTrigger`.
   - Iceberg has `streaming-max-files-per-micro-batch`.
   
   **Proposed shape**
   - **Config:** e.g. 
`hoodie.datasource.read.streaming.max.commits.per.trigger`. No default, so the 
source stays unbounded as today. It counts completed write commits after the 
start offset.
   - **Admission control:** implement `SupportsAdmissionControl`. 
`latestOffset(start, limit)` returns the offset of the Nth completed commit 
after `start`, and `getDefaultReadLimit` comes from the config.
   - **`Trigger.AvailableNow`:** implement `SupportsTriggerAvailableNow`, so 
AvailableNow (SDP triggered updates) pins its end offset once and drains it in 
capped batches. Without it, Spark wraps the source and ignores the limit.
   - **Offset versions:** count only write commits, not cleans or rollbacks, 
which V1 offsets can land on. In V2, count by completion time.
   - **Tests:** COW and MOR, both offset versions. A backlog of N commits with 
a limit of k should give ceil(N/k) batches with exact rows. Add an AvailableNow 
case that stops at the pinned end.
   
   **Out of scope:** chunking `READ_UPTO_LATEST_COMMIT` snapshot recovery 
(#20516). It stays a single batch for now, but this admission-control hook is 
where chunked recovery would plug in later.
   
   ### Task Type
   
   Code improvement/refactoring
   
   ### Related Issues
   
   **Parent feature issue:** #20401
   **Related:** #20516, #20407
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to