Re: Auto Scaling in Flink
Hi Akash, The key difference between Pravega and Kafka is: Kafka is a messaging system, while Pravega is a streaming system.[1] The official documentation also statements their difference in their faq page.[2] [1]: https://siliconangle.com/2017/04/17/dell-emc-takes-on-streaming-storage-with-open-source-solution-pravega-ffsf17/ [2]: http://www.pravega.io/faq.html Best, Vino Akash Goel 于2019年12月4日周三 下午12:00写道： > Hi, > > If my application is already running on Kafka, then do I need to replace > with Pravega or can Pravega read directly from Kafka? > > I have also reached out to to Pravega Community but just checking here. > > Thanks, > Akash Goel > > On Fri, Nov 29, 2019 at 11:14 AM Caizhi Weng wrote: > >> Hi Akash, >> >> Flink doesn't support auto scaling in core currently, it may be supported >> in the future, when the new scheduling architecture is implemented >> https://issues.apache.org/jira/browse/FLINK-10407 . >> >> You can do it externally by cancel the job with a savepoint, update the >> parallelism, and restart the job, according to the rate of data. like what >> pravega suggests in the doc: >> http://pravega.io/docs/latest/key-features/#auto-scaling. >> >> vino yang 于2019年11月29日周五 上午11:12写道： >> >>> Hi Akash, >>> >>> You can use Pravega connector to integrate with Flink, the source code >>> is here[1]. >>> >>> In short, relying on its rescalable state feature[2] flink supports >>> scalable streaming jobs. >>> >>> Currently, the mainstream solution about auto-scaling is Flink + K8S, I >>> can share some resources with you[3]. >>> >>> Best, >>> Vino >>> >>> [1]: https://github.com/pravega/flink-connectors >>> [2]: >>> https://flink.apache.org/features/2017/07/04/flink-rescalable-state.html >>> [3]: >>> https://www.slideshare.net/FlinkForward/flink-forward-san-francisco-2019-scaling-a-realtime-streaming-warehouse-with-apache-flink-parquet-and-kubernetes-aditi-verma-ramesh-shanmugam >>> >>> Akash Goel 于2019年11月29日周五 上午9:52写道： >>> >>>> Hi, >>>> >>>> Does Flunk support auto scaling. I read that it is supported using >>>> pravega? Is it incorporated in any version. >>>> >>>> Thanks, >>>> Akash Goel >>>> >>>
Re: Auto Scaling in Flink
Hi, If my application is already running on Kafka, then do I need to replace with Pravega or can Pravega read directly from Kafka? I have also reached out to to Pravega Community but just checking here. Thanks, Akash Goel On Fri, Nov 29, 2019 at 11:14 AM Caizhi Weng wrote: > Hi Akash, > > Flink doesn't support auto scaling in core currently, it may be supported > in the future, when the new scheduling architecture is implemented > https://issues.apache.org/jira/browse/FLINK-10407 . > > You can do it externally by cancel the job with a savepoint, update the > parallelism, and restart the job, according to the rate of data. like what > pravega suggests in the doc: > http://pravega.io/docs/latest/key-features/#auto-scaling. > > vino yang 于2019年11月29日周五 上午11:12写道： > >> Hi Akash, >> >> You can use Pravega connector to integrate with Flink, the source code is >> here[1]. >> >> In short, relying on its rescalable state feature[2] flink supports >> scalable streaming jobs. >> >> Currently, the mainstream solution about auto-scaling is Flink + K8S, I >> can share some resources with you[3]. >> >> Best, >> Vino >> >> [1]: https://github.com/pravega/flink-connectors >> [2]: >> https://flink.apache.org/features/2017/07/04/flink-rescalable-state.html >> [3]: >> https://www.slideshare.net/FlinkForward/flink-forward-san-francisco-2019-scaling-a-realtime-streaming-warehouse-with-apache-flink-parquet-and-kubernetes-aditi-verma-ramesh-shanmugam >> >> Akash Goel 于2019年11月29日周五 上午9:52写道： >> >>> Hi, >>> >>> Does Flunk support auto scaling. I read that it is supported using >>> pravega? Is it incorporated in any version. >>> >>> Thanks, >>> Akash Goel >>> >>
Re: Auto Scaling in Flink
Hi Akash, Flink doesn't support auto scaling in core currently, it may be supported in the future, when the new scheduling architecture is implemented https://issues.apache.org/jira/browse/FLINK-10407 . You can do it externally by cancel the job with a savepoint, update the parallelism, and restart the job, according to the rate of data. like what pravega suggests in the doc: http://pravega.io/docs/latest/key-features/#auto-scaling. vino yang 于2019年11月29日周五 上午11:12写道： > Hi Akash, > > You can use Pravega connector to integrate with Flink, the source code is > here[1]. > > In short, relying on its rescalable state feature[2] flink supports > scalable streaming jobs. > > Currently, the mainstream solution about auto-scaling is Flink + K8S, I > can share some resources with you[3]. > > Best, > Vino > > [1]: https://github.com/pravega/flink-connectors > [2]: > https://flink.apache.org/features/2017/07/04/flink-rescalable-state.html > [3]: > https://www.slideshare.net/FlinkForward/flink-forward-san-francisco-2019-scaling-a-realtime-streaming-warehouse-with-apache-flink-parquet-and-kubernetes-aditi-verma-ramesh-shanmugam > > Akash Goel 于2019年11月29日周五 上午9:52写道： > >> Hi, >> >> Does Flunk support auto scaling. I read that it is supported using >> pravega? Is it incorporated in any version. >> >> Thanks, >> Akash Goel >> >
Re: Auto Scaling in Flink
Hi Akash, You can use Pravega connector to integrate with Flink, the source code is here[1]. In short, relying on its rescalable state feature[2] flink supports scalable streaming jobs. Currently, the mainstream solution about auto-scaling is Flink + K8S, I can share some resources with you[3]. Best, Vino [1]: https://github.com/pravega/flink-connectors [2]: https://flink.apache.org/features/2017/07/04/flink-rescalable-state.html [3]: https://www.slideshare.net/FlinkForward/flink-forward-san-francisco-2019-scaling-a-realtime-streaming-warehouse-with-apache-flink-parquet-and-kubernetes-aditi-verma-ramesh-shanmugam Akash Goel 于2019年11月29日周五 上午9:52写道： > Hi, > > Does Flunk support auto scaling. I read that it is supported using > pravega? Is it incorporated in any version. > > Thanks, > Akash Goel >
Auto Scaling in Flink
Hi, Does Flunk support auto scaling. I read that it is supported using pravega? Is it incorporated in any version. Thanks, Akash Goel
