pkuzmickas opened a new issue, #39873: URL: https://github.com/apache/beam/issues/39873
## Problem The Flink DataStream runner assigns bounded-source splits lazily: readers request another split after they finish their current one. This balances sources whose splits perform the expensive work, such as reading variable-size files. The same strategy can skew sources that emit inexpensive descriptors for expensive downstream work. Fast-starting readers claim most descriptors, and a pointwise downstream edge preserves that imbalance. Moving production workloads from the Flink 1 DataSet runner to the Flink 2 DataStream runner exposed large performance regressions in this pattern. In two anonymized production descriptor workloads, static round-robin assignment reduced end-to-end runtime by about 45% and 28%. However, forcing static assignment on a direct file-reading workload made it slower because readers could no longer pull work dynamically. ## Proposed change Add an opt-in Flink pipeline option that selects bounded-source split assignment from the estimated input size per source reader: - `0` (default) preserves the current lazy assignment. - A positive threshold uses static round-robin assignment below the threshold and lazy assignment at or above it. - A negative value forces static assignment. - Unknown estimates use lazy assignment. Estimate and split the source on the enumerator's asynchronous worker. Persist the selected mode and pending splits in checkpoint state so restores keep the original strategy, preserve split affinity at unchanged parallelism, and redistribute static splits predictably after rescaling. ## Compatibility Existing pipelines keep lazy bounded-source assignment because the option defaults to `0`. Existing version 0 checkpoints restore bounded sources lazily and unbounded sources statically. The feature does not change Beam source splitting or unbounded-source assignment. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
