As an alternative approach, is it conceivable to write a Custom Processor and run that in the MiNiFi 'pipeline' without connecting back to NiFi (Custom Processor -> Some Connector with socket / buffer input -> PublishToKafka) ? This wouldn't hide the MiNiFi installation / runtime from the user, but will keep the 'agent' independent of the NiFi backend.
On Wed, Sep 27, 2017 at 6:50 AM, p pathiyil <[email protected]> wrote: > Thanks for the pointer Joe. Let me look at Kafka Connect and see if that > can be used to address this use case. > > On Wed, Sep 27, 2017 at 6:35 AM, Joe Witt <[email protected]> wrote: > >> Praveen >> >> I think the direction of MiNiFi and where it can head will support >> this case nicely. Today though I dont think we offer the simple >> library model you're looking for. Fortunately, for Apache Kafka >> specifically their community has developed Kafka Connect [1] which >> sounds like it could be just what you need. >> >> Take a look at that and if that gets you where you want to be then >> great. If not, or I've misunderstood the ask please let us know. >> >> [1] https://kafka.apache.org/documentation/#connect >> >> Thanks >> Joe >> >> On Tue, Sep 26, 2017 at 8:50 PM, p pathiyil <[email protected]> wrote: >> > Hi Aldrin, >> > >> > I am looking at a couple of different flows, but to take the most simple >> > scenario, I would like to leverage the PublishToKafka_0_10 processor of >> > MiNiFi alone. The application will obtain some data, send it to a >> Processor >> > that has a simple upstream connection interface like a socket or in >> memory >> > buffer (so that conversion to FlowFiles can be taken care of in that >> > Processor) and then use PublishToKafka_0_10 to directly publish to a >> Kafka >> > cluster. Will that be easy enough to do ? >> > >> > Thanks, >> > Praveen. >> > >> > >> > >> > >> > On Wed, Sep 27, 2017 at 12:24 AM, Aldrin Piri <[email protected]> >> wrote: >> >> >> >> Hello, >> >> >> >> Neither NiFi nor MiNiFi (Java) is currently packaged as such. Would >> you >> >> be able to expand upon your use case of publishing to Kafka and/or what >> >> facets of MiNiFi you are looking to utilize? There may be other >> avenues >> >> that get you toward your solution. >> >> >> >> Thanks, >> >> Aldrin >> >> >> >> On Tue, Sep 26, 2017 at 1:03 PM, p pathiyil <[email protected]> >> wrote: >> >>> >> >>> Hi, >> >>> >> >>> I am starting to look at MiNiFi for a use case that involves >> publishing >> >>> to Kafka. Is it feasible to use MiNiFi as a library (or set of >> libraries) >> >>> from within another Java program ? The primary reason for looking at >> such an >> >>> approach is the desire to hide the MiNiFi installation / process >> management >> >>> from the users of my application. >> >>> >> >>> Thanks. >> >>> >> >> >> > >> > >
