Re: How choose between YARN/Mesos/StandAlone Flink
Ok I understand standalone mode it will be sufficient, but for my thesis I would like to setup a well performed ready-to-use infrastructure. My workload it's not heavy, about 35 millions of messages a day (35 gb) but it should be easily expandable and running for many days... due to this I would like to setup Flink on top of a cluster manager. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/How-choose-between-YARN-Mesos-StandAlone-Flink-tp13793p13829.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: How choose between YARN/Mesos/StandAlone Flink
Hello, just quickly chiming in for clarification/correction purpose: Flink can work in multi-node environments without yarn/mesos. If you are only starting out, or have short-lived workloads (i.e. a job that does not have to run for days/weeks), I would recommend standalone mode for easy-of-entry. Also, Flink does not use resource managers to distribute tasks, but to manage Job- and TaskManagers, which in standalone mode you have to do yourself. On 16.06.2017 15:37, Biplob Biswas wrote: Hi Andrea, If you are using Flink for research and/or testing purpose, standalone Flink is more or less sufficient. Although if you have a huge amount of data, it may take forever to process data with only one node/machine and that's where a cluster would be needed. A yarn and mesos cluster could provide you high availability and fault tolerance so that you don't lose your data if something happens to one of the nodes in your cluster setup. Also, AFAIK, flink relies on a resource manager like Yarn or Mesos to distribute the task between multiple nodes so that you don't have to worry about that distribution. For the rest, I would like the experts here to correct me and add more info here. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/How-choose-between-YARN-Mesos-StandAlone-Flink-tp13793p13801.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: How choose between YARN/Mesos/StandAlone Flink
Hi Andrea, If you are using Flink for research and/or testing purpose, standalone Flink is more or less sufficient. Although if you have a huge amount of data, it may take forever to process data with only one node/machine and that's where a cluster would be needed. A yarn and mesos cluster could provide you high availability and fault tolerance so that you don't lose your data if something happens to one of the nodes in your cluster setup. Also, AFAIK, flink relies on a resource manager like Yarn or Mesos to distribute the task between multiple nodes so that you don't have to worry about that distribution. For the rest, I would like the experts here to correct me and add more info here. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/How-choose-between-YARN-Mesos-StandAlone-Flink-tp13793p13801.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
