The European Spallation Source  seems to be using it for this case .
I am also using this code , but only for visualization in another "data
From: Hannes Petri <hannespe...@gmail.com>
Sent: Monday, March 12, 2018 1:37:29 PM
Subject: Using Kafka to build a streaming platform for a research facility?
I work at a research facility where numerous hi-res detectors produce thousands
of GB of data every day. We want to build a highly flexible and performant
streaming platform for storing, transmitting and routing the data. For example,
detector output needs to end up:
1. In permanent storage systems
2. In realtime or semi-realtime visualization software
3. In post-processing and analysis software
4. In metrics software
...and possibly more. Now I'm exploring Kafka as an option to back such a
Would Kafka be a good fit? The reason I'm asking is because among the use
cases, I've mostly seen Kafka being used with more lightweight data, geared
towards business events and high frequency streams of text and scalars. In
other words, *more* but *smaller* messages.
In my situation, we'd be looking at low frequency but huge files (typically
these detectors produce one large file at a time). In order not to flood the
storage, raw data topics would need to have a very short retention time (hours
Does anybody have experience in using Kafka in a similar scenario? What are
your thoughts about the situation I describe? Would we benefit from using Kafka?
I highly appreciate any input on this. Many thanks in advance.
Sent from my iPhone