Yury,
Well thanks for sharing the insight of kafka partition distribution.
Well I am more of a concerned about the throughtput that kafka-storm can
collaborative give so as to event process.
Currently I am having around a 30 Gb file with around .2 Billion events,
this number is soon gonna rise
Hi,
In short, I don't see Kafka having problems with those numbers. Logstash
will have a harder time, I believe.
That said, it all depends on how you tune things an what kind of / how much
hardware you use.
2B or 200B events, yes, big numbers, but how quickly do you need to process
those? in 1
This is a quote from Kafka documentation:
The routing decision is influenced by the kafka.producer.Partitioner.
interface PartitionerT {
int partition(T key, int numPartitions);
}
The partition API uses the key and the number of available broker
partitions to return a partition id. This id is
