I understand your desire for a bundled SDK for data routing, but NiFi/MiNiFi isn’t designed for that right now. As the code is open source, you are welcome to extract pieces you find useful to build your own. Kafka Connect or some other libraries are probably more relevant to what you are trying to do now. I’m not saying that in the future, MiNiFi libraries won’t be exposed or made accessible to other clients in order to expand the reach, but currently that is not how the software is designed.
Andy LoPresto [email protected] [email protected] PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Sep 27, 2017, at 9:54 AM, p pathiyil <[email protected]> wrote: > > Thanks for that detailed response Andy. > > I was not sure if it is advisable to use MiNiFi without connecting it back to > NiFi, so good to get clarity on that. > > To take a stab at providing a reasoning for the 'MiNiFi as a library' use > case: > > - I am looking to provide a library that exposes an SDK to clients, through > which they will pump in some data. I was hoping to avoid users having to run > a separate program (MiNiFi agent) in addition to the application that they > will build using the SDK. These two will have to run on client machines. > > - The second has more to do with the ease of deployment for an application > that the user will build with an SDK. If all the dependencies come packaged > with the SDK, users do not have to worry about deploying 2 applications to > their target systems. > > On Wed, Sep 27, 2017 at 11:14 AM, Andy LoPresto <[email protected] > <mailto:[email protected]>> wrote: > I'm not sure I understand this proposal. MiNiFi can be installed completely > separately from NiFi (and usually they are not co-located on the same > machine). If your requirements are to connect an arbitrary Java program with > Kafka to produce data that is published to a Kafka topic, MiNiFi is a > potential (but probably not ideal) tool, while its interdependency with NiFi > is only relevant if you wanted to push from MiNiFi to NiFi to Kafka. > > Please let me know if I am misunderstanding, but to go from your client to > Kafka, I would propose the following flow: > > Your code: persist or stream data somewhere (CSV/XML/JSON file, HTTP > endpoint, TCP packet, DB, whatever) > > MiNiFi: processor to read that format/location -> processor(s) to manipulate > what is read in as flowfile content to massage it to expected Kafka form -> > PublishKafka processor > > This allows you to decouple your application from the Kafka format while > using MiNiFi as a "glue layer" that can be updated by changing a single > processor should the source or destination change in the future. > > MiNiFi would provide you with the queuing features like ordering, > prioritization, and backpressure without requiring you to code that yourself. > > Hopefully this helps and if not, at least clarifies the capabilities and uses > of NiFi and MiNiFi. > > Andy LoPresto > [email protected] <mailto:[email protected]> > [email protected] <mailto:[email protected]> > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > On Sep 26, 2017, at 20:04, p pathiyil <[email protected] > <mailto:[email protected]>> wrote: > >> As an alternative approach, is it conceivable to write a Custom Processor >> and run that in the MiNiFi 'pipeline' without connecting back to NiFi >> (Custom Processor -> Some Connector with socket / buffer input -> >> PublishToKafka) ? This wouldn't hide the MiNiFi installation / runtime from >> the user, but will keep the 'agent' independent of the NiFi backend. >> >> >> >> >> >> On Wed, Sep 27, 2017 at 6:50 AM, p pathiyil <[email protected] >> <mailto:[email protected]>> wrote: >> Thanks for the pointer Joe. Let me look at Kafka Connect and see if that can >> be used to address this use case. >> >> On Wed, Sep 27, 2017 at 6:35 AM, Joe Witt <[email protected] >> <mailto:[email protected]>> wrote: >> Praveen >> >> I think the direction of MiNiFi and where it can head will support >> this case nicely. Today though I dont think we offer the simple >> library model you're looking for. Fortunately, for Apache Kafka >> specifically their community has developed Kafka Connect [1] which >> sounds like it could be just what you need. >> >> Take a look at that and if that gets you where you want to be then >> great. If not, or I've misunderstood the ask please let us know. >> >> [1] https://kafka.apache.org/documentation/#connect >> <https://kafka.apache.org/documentation/#connect> >> >> Thanks >> Joe >> >> On Tue, Sep 26, 2017 at 8:50 PM, p pathiyil <[email protected] >> <mailto:[email protected]>> wrote: >> > Hi Aldrin, >> > >> > I am looking at a couple of different flows, but to take the most simple >> > scenario, I would like to leverage the PublishToKafka_0_10 processor of >> > MiNiFi alone. The application will obtain some data, send it to a Processor >> > that has a simple upstream connection interface like a socket or in memory >> > buffer (so that conversion to FlowFiles can be taken care of in that >> > Processor) and then use PublishToKafka_0_10 to directly publish to a Kafka >> > cluster. Will that be easy enough to do ? >> > >> > Thanks, >> > Praveen. >> > >> > >> > >> > >> > On Wed, Sep 27, 2017 at 12:24 AM, Aldrin Piri <[email protected] >> > <mailto:[email protected]>> wrote: >> >> >> >> Hello, >> >> >> >> Neither NiFi nor MiNiFi (Java) is currently packaged as such. Would you >> >> be able to expand upon your use case of publishing to Kafka and/or what >> >> facets of MiNiFi you are looking to utilize? There may be other avenues >> >> that get you toward your solution. >> >> >> >> Thanks, >> >> Aldrin >> >> >> >> On Tue, Sep 26, 2017 at 1:03 PM, p pathiyil <[email protected] >> >> <mailto:[email protected]>> wrote: >> >>> >> >>> Hi, >> >>> >> >>> I am starting to look at MiNiFi for a use case that involves publishing >> >>> to Kafka. Is it feasible to use MiNiFi as a library (or set of libraries) >> >>> from within another Java program ? The primary reason for looking at >> >>> such an >> >>> approach is the desire to hide the MiNiFi installation / process >> >>> management >> >>> from the users of my application. >> >>> >> >>> Thanks. >> >>> >> >> >> > >> >> >
signature.asc
Description: Message signed with OpenPGP using GPGMail
