Looks like what I am currently doing, or at least close.No need to copy the big
file on every node.Copy on one node. Read the data, and send it to a Kafka
cluster using KafkaProducer() object.Use KafkaIO() (in case its a Beam app).
Deploy to the node where the JM is running.It will be executed in s distributed
fashion across all nodes.If that help, I can privately help you how the
logistics & the code may look like + a loooooooooooot of tricksI have learned
the hard way LOL!Thanks.
Amir-
From: Mariano Gonzalez <[email protected]>
To: [email protected]
Sent: Thursday, September 29, 2016 3:07 PM
Subject: Re: Use specific Task Manager for heavy computations
Any ideas?
--
View this message in context:
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Use-specific-Task-Manager-for-heavy-computations-tp13747p13771.html
Sent from the Apache Flink Mailing List archive. mailing list archive at
Nabble.com.
