About tcp listener, its true there's a nifi listener, but we step back as its not 100% a resilient solution.. we wsnted to implement in nifi a listener for jira webhook events which are sent only once, so 0% margin loosing msgs. I wonder what nifi more seniors experienced advise for this type of scenario?
Some options in aws we talked: - AWS API Gateway as tcp listener invoking lambda function to put msg on a persistent queue (ex aws sqs, kinesis..). - custom EC2 machine. In reality theres no other direct tcp listener/servers in aws .. all aws services always need a client using an api/lib to put msgs on a service.. We found this hilarious, when the perception from all jargon online seems all options exists, all type of services exists and.. well one find out "nope no such thing as a simpler tcp server for you". Taking in account theres basically 2 data flows patterns "push" and "pull", seems nifi at current state is mostly a pull only.. so for push use case one must struggle using additional service, and then find out at least in aws theres really no lots of options as like i mentioned above (and im challenging here a bit, hoping im wrong) one must always implement a listener manually uding a lib/api to then be able to interact with services like putmsg() on aws sqs queue.. its not possible to have an aws sqs queue directly listening. Thanks, Emanuel O. On Sat 15 Feb 2020, 19:47 Mike Thomsen, <[email protected]> wrote: > Yes, very much so. The program I'm on uses NiFi for everything from > processing one off Excel spreadsheets, to database records (MySQL, > Postgres, Oracle, etc.), to log files. We also have some custom use cases I > can't go into that are very much off the beaten path and it still excels at > its job. > > With regard to your third use case, there's TCP listening support, but you > might want to consider creating a simple TCP listener that pushes the data > into Apache Kafka. The reason for that is because Kafka is really good for > organizing and routing events going into a system like NiFi. > > Thanks, > > Mike > > On Fri, Feb 14, 2020 at 7:36 AM Kan Wong <[email protected]> wrote: > > > Hi Nifi team, > > > > My company which is a bank in asia looking for a data integration > solution > > to collect all distributed data across fragmented window and linux > servers > > environment into a centralized data warehouse. > > > > We just came across Nifi and Minifi which seem to be capable for the > > integration and see what collaboration that we could come up with. I'm > > unable to reveal the name yet but very soon.. > > > > Here the data we would like to consolidate from distributed environments. > > > > - Order, Trade, Execution Messages across different systems (File and > > database Record (MSSSQL and Oracle)) > > - Audit Logs file (flat file and database record (MSSQL and Oracle) > > - Real time market database stream from third party via TCP > > - Fragmented historical data (flat file, multiple database instances in > > MSSQL) > > > > We would like to consolidate all into a single data warehouse and > integrate > > all for ongoing collection. > > > > is Nifi will fit the application? > > > > Kan > > >
