Hi, Terence, Thank you for description! Should I assign this issue to me?
Sergey сб, 1 июл. 2017 г. в 21:23, Terence Yim <cht...@gmail.com>: > Hi Sergey, > > I think you are talking about TWILL-147 ( > https://issues.apache.org/jira/browse/TWILL-147), right? The idea for that > is we don't need to start EmbeddedKafkaServer in AM at all, but rather it > just take a configuration (via TwillPreparer, which can have a default > value in the Configuration object passed to YarnTwillRunnerService), which > the configuration specifies the Kafka broker list and topic that the AM > will publish to. > > Since under this model, application logs from different application may > send to the same Kafka topic (depends on the configuration), the LogEntry > needs to be modified to carry the application and run id, so that the > TwillController can filter based on it on the client side. > > Terence > > > > On Sat, Jul 1, 2017 at 1:46 AM, Сергей Филиппов <role...@gmail.com> wrote: > > > Hello, > > I would like to implement possibility to use external kafka server for > log > > aggregation. > > Now twill uses EmbededKafkaServer for that. I think implementation would > > look like this: > > 1. Add ZK path where kafka zk connection string will be stored. There > > should should be only one such path per ApplicationMaster > > 2. Use this path in ApplicationKafkaService while creating > > EmbededKafkaService, if there on brokers right now > > 3. For log aggregation there should be additional nodes in ZK for each > > instance with kafka topic's name in it. Something like > > "test-app-{UUID}-log". So publisher will send to this topic and consumer > > will consume log messages on the job submission machine. > > > > What would you say? Is this sounds ok? > > > > Sergey > > >
