Apologies. Misinterpreted your question. I am curious to know if people have faced issues with single partition on node in practice. Or because most uses do not keep logs forever, this is generally not an issue? Was this faced at LinkedIn? On 1 Jun 2016 6:28 p.m., "VG" <vlin...@gmail.com> wrote:
You are wrong Unmesh. Kafka design forces a partition to be on a single node only. My question is around the scalability of the partition itself. How to overcome the restriction of a single node for a partition ? Any clues anyone... On Wed, Jun 1, 2016 at 5:24 PM, Unmesh Joshi <unmeshjo...@gmail.com> wrote: > I do not see why this is a limitation. Any data storage application you use > will be limited by physical capacity of the nodes. > Distributed applications like Kafka (Distributed message broker), HDFS ( > Distributed file system), Cassandra ( distributed key value dB), by design > allow to store huge amount of data by partitioning it on multiple machine. > 'Multiple' here means 'Cloud Scale'. Tens of thousands of machines spanning > across data centres. > This actually has no limit on data storage capacity then. > > As far as duration for persisting per node log, there is no reason why you > can not store it for ever. > On 1 Jun 2016 9:07 a.m., "VG" <vlin...@gmail.com> wrote: > > > Hi, > > > > There are number of messages floating on the internet suggesting that > Kafka > > cannot persist messages infinitely ? > > Primarily that Kafka partitions are pinned to a node and they can’t > > outgrow the storage capacity of a node.. > > > > Can someone help me understand this limitation and how it can be > overcome ? > > > > Regards, > > Vipul > > >