you're probably better off using Apache Nifi at that point, Kafka Nifi is better integrated today than Oozie and Kafka. Probably wrong mailing list to push Nifi but it's a common use case in that realm.
Thanks On Mon, Dec 18, 2017 at 7:04 PM, Serega Sheypak <[email protected]> wrote: > Hi, I know default coordinator functionality, but it's limited (almost) to > HDFS. > Kafka (any other pub/sub or queue like rabbitMQ, whatever-MQ) makes > integration contract much more flexible. > I could have traceability, debuggability, transparency, throttling, > concurrency of oozie and push coordinator job on demand. And I'm not > limited to strict HDFS path pattern. > > > 2017-12-18 18:14 GMT+01:00 Andras Piros <[email protected]>: > > > Hi Serega, > > > > not to my knowledge. Would be interested on your use case, though. > > > > Would start w/ *Coordinator Input Events / Datasets > > <https://oozie.apache.org/docs/4.3.0/CoordinatorFunctionalSpec. > > html#a5._Dataset>* > > . > > > > Andras > > > > On Sat, Dec 16, 2017 at 2:54 PM, Serega Sheypak < > [email protected]> > > wrote: > > > > > Hi, did anyone try to integrate oozie coordinator with kafka? > > > use case: > > > > > > System publishes message to kafka topic (sample message) > > > - cluster: hdfs://prod-cluster > > > - path: /my/input/data > > > - format: avro > > > > > > Oozie coordinator listens to kafka topic, consumes message and starts > > > workflow. > > > > > >
