"Brian Candler" <b.cand...@pobox.com> wrote: > I can't answer those questions, but if I were writing a flow collector > to process a million or more flows per second I'd probably be using > Apache Kafka as the queue and data distribution mechanism.
Brian, my challenge was minimal time spend between read calls on the UDP socket handle. The more time spend elsewhere, the bigger the likelihood of a UDP packet being dropped. Not sure how Kafka would address this. The inherent problem with a single collector (UDP reader) process is that it is serialised. So to address this I would think one need to look at a main thread reading UDP data from the socket, writing it into a memory buffer, and additional threads to empty the buffer to disk. And this gets complex as one needs to manage access to memory to ensure that the writer does not spin on a memory latch, or that a reader process does not read a partial buffer, or does not read a buffer busy being read and written to disk by another thread, and so on. regards, Billy ------------------------------------------------------------------------------ Site24x7 APM Insight: Get Deep Visibility into Application Performance APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month Monitor end-to-end web transactions and take corrective actions now Troubleshoot faster and improve end-user experience. Signup Now! http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 _______________________________________________ Nfdump-discuss mailing list Nfdump-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfdump-discuss
