The European Spallation Source [1] seems to be using it for this case [2].

I am also using this code [2], but only for visualization in another "data 



Thank you!


From: Hannes Petri <>
Sent: Monday, March 12, 2018 1:37:29 PM
Subject: Using Kafka to build a streaming platform for a research facility?


I work at a research facility where numerous hi-res detectors produce thousands 
of GB of data every day. We want to build a highly flexible and performant 
streaming platform for storing, transmitting and routing the data. For example, 
detector output needs to end up:

1. In permanent storage systems
2. In realtime or semi-realtime visualization software
3. In post-processing and analysis software
4. In metrics software

...and possibly more. Now I'm exploring Kafka as an option to back such a 

Would Kafka be a good fit? The reason I'm asking is because among the use 
cases, I've mostly seen Kafka being used with more lightweight data, geared 
towards business events and high frequency streams of text and scalars. In 
other words, *more* but *smaller* messages.

In my situation, we'd be looking at low frequency but huge files (typically 
these detectors produce one large file at a time). In order not to flood the 
storage, raw data topics would need to have a very short retention time (hours 
to days).

Does anybody have experience in using Kafka in a similar scenario? What are 
your thoughts about the situation I describe? Would we benefit from using Kafka?

I highly appreciate any input on this. Many thanks in advance.

Best regards

Sent from my iPhone

Reply via email to