pabloem opened a new issue, #25140: URL: https://github.com/apache/beam/issues/25140
### What happened? The transform `GenerateSequence.from(0).to(...).withRate(...)` that is expanded into a streaming source (that is bounded, as there's a maximum number of records read) is broken after it was migrated from an `UnboundedSource` based implementaion. For smaller data sizes, the source works as expected, and a sequence is generated properly. For larger data sizes, the transform causes workers to OOM and never make progress. Reasons: - Firstly, the maximum number of initial splits is 100: https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/BoundedReadFromUnboundedSource.java#L157-L161 - This makes it such that each individual split can be very large for medium data sizes (over 100k elements per split). - Secondly, the consumption is implemented as a simple DoFn that consumes the whole source without checkpointing halfway: https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/BoundedReadFromUnboundedSource.java#L189-L212 This is what causes ooms for workers before they can make any progress. (here's the expansion for GenerateSequence: https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/GenerateSequence.java#L242-L258) I found that the expansion always became a `BoundedReadFromUnboundedSource` if we had `from` and `to` parameters. ### Issue Priority Priority: 1 (data loss / total loss of function) ### Issue Components - [ ] Component: Python SDK - [X] Component: Java SDK - [ ] Component: Go SDK - [ ] Component: Typescript SDK - [X] Component: IO connector - [ ] Component: Beam examples - [ ] Component: Beam playground - [ ] Component: Beam katas - [ ] Component: Website - [ ] Component: Spark Runner - [ ] Component: Flink Runner - [ ] Component: Samza Runner - [ ] Component: Twister2 Runner - [ ] Component: Hazelcast Jet Runner - [X] Component: Google Cloud Dataflow Runner -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
