KafkaIO also supports autoscaling on Dataflow. In fact any unbounded source is supported as long as it implements 'getSplitBacklogBytes()' or 'getTotalBacklogBytes()' interface. E.g. PR 3551 <https://github.com/apache/beam/pull/3551> adds backlog support for KinesisIO.
On Tue, Sep 5, 2017 at 10:19 PM, Lukasz Cwik <[email protected]> wrote: > That is correct, autoscaling for streaming is only supported in Pubsub. > What sources were you interested in? > > On Mon, Sep 4, 2017 at 12:54 AM, Derek Hao Hu <[email protected]> > wrote: > >> I've used PubSubIO for autoscaling on a streaming pipeline and it seems >> to be working fine so far. >> >> I don't know if it is the **only** source that supports autoscaling on >> streaming pipelines though. >> >> On Mon, Sep 4, 2017 at 12:52 AM, Vilhelm von Ehrenheim < >> [email protected]> wrote: >> >>> Hi! >>> I cannot make my streaming job autoscale on Dataflow. To find an answer >>> to why I was looking around in the Dataflow docs and found this: >>> >>> > Currently, PubSubIO is the only source that supports autoscaling on >>> streaming pipelines. >>> >>> on the following page: https://cloud.google.com >>> /dataflow/service/dataflow-service-desc#autoscaling >>> >>> Does anyone know if this is true? I know this is not the forum for >>> Dataflow questions in general but I though someone else here might have >>> experience that support or contradict this. >>> >>> Thanks, >>> Vilhelm von Ehrenheim >>> >> >> >> >> -- >> Derek Hao Hu >> >> Software Engineer | Snapchat >> Snap Inc. >> > >
