Hi, I know default coordinator functionality, but it's limited (almost) to HDFS. Kafka (any other pub/sub or queue like rabbitMQ, whatever-MQ) makes integration contract much more flexible. I could have traceability, debuggability, transparency, throttling, concurrency of oozie and push coordinator job on demand. And I'm not limited to strict HDFS path pattern.
2017-12-18 18:14 GMT+01:00 Andras Piros <[email protected]>: > Hi Serega, > > not to my knowledge. Would be interested on your use case, though. > > Would start w/ *Coordinator Input Events / Datasets > <https://oozie.apache.org/docs/4.3.0/CoordinatorFunctionalSpec. > html#a5._Dataset>* > . > > Andras > > On Sat, Dec 16, 2017 at 2:54 PM, Serega Sheypak <[email protected]> > wrote: > > > Hi, did anyone try to integrate oozie coordinator with kafka? > > use case: > > > > System publishes message to kafka topic (sample message) > > - cluster: hdfs://prod-cluster > > - path: /my/input/data > > - format: avro > > > > Oozie coordinator listens to kafka topic, consumes message and starts > > workflow. > > >
