Thank you very much for the help, Xintong. Best, Ethan
> On Oct 7, 2019, at 9:18 PM, Xintong Song <tonysong...@gmail.com> wrote: > > I don't think using zookeeper should cause any problem on starting multiple > TMs per node. > > For standalone cluster, having one TM per node is usually the easiest way. It > is easy to config (simply config the TM resource to the available resource of > the node), and it is more efficient (less TM framework overhead, no isolation > between JVMs of different TMs). But that doesn't mean Flink can not have > multiple TMs per node, and given your requirements of job isolation, I think > you should do that. You just need to carefully config the TM number and > resources to make sure they do not exceed the node's available resource. > > I don't have much experience running standalone clusters with multiple TMs > per node. We mainly run Flink on Yarn in our production, with multiple TMs > (Yarn containers) per node (Yarn NM), and it works fine for us. > > Thank you~ > Xintong Song > > > On Mon, Oct 7, 2019 at 9:05 PM Ethan Li <ethanopensou...@gmail.com > <mailto:ethanopensou...@gmail.com>> wrote: > Thank you very much Xintong for the information. > > I am currently using zookeeper. I guess starting another task manager will > just work? But I am not sure if multiple TaskManagers per node is good idea > to use Flink. Do you have any experience on this? Thanks! > > Best, > Ethan > > >> On Oct 5, 2019, at 3:43 AM, Xintong Song <tonysong...@gmail.com >> <mailto:tonysong...@gmail.com>> wrote: >> >> For having multiple task managers on the same host, you can put multiple >> duplicated lines with the target host in '<your-flink-home>/conf/slaves'. >> The task managers on the same host will share the same config files on the >> host. >> >> Thank you~ >> Xintong Song >> >> >> On Sat, Oct 5, 2019 at 5:02 AM Ethan Li <ethanopensou...@gmail.com >> <mailto:ethanopensou...@gmail.com>> wrote: >> Hello, >> >> Does/did anyone try to set up a standalone cluster with multiple >> TaskManagers per node? We are working on moving to flink-on-yarn solution. >> But before that happens, I am thinking about the following setup to get >> jobs isolated from each other >> >> 1) multiple taskmanager per host >> 2) 1 taskSlot per TaskManager >> >> >> Currently we have 1 TaskManger per node and many taskSlot per TM, tasks from >> different job will be scheduled into one JVM process and it’s basically >> impossible to debug. One bad job will kill the whole cluster. >> >> Could you please share if you have any experience on this and what’re the >> problems that might have? >> >> Thank you very much. Really appreciate it. >> >> Best, >> Ethan >
