You are wrong Unmesh.  Kafka design forces a partition to be on a single
node only.
My question is around the scalability of the partition itself.
How to overcome the restriction of a single node for a partition  ?

Any clues anyone...

On Wed, Jun 1, 2016 at 5:24 PM, Unmesh Joshi <unmeshjo...@gmail.com> wrote:

> I do not see why this is a limitation. Any data storage application you use
> will be limited by physical capacity of the nodes.
> Distributed applications like Kafka (Distributed message broker), HDFS (
> Distributed file system), Cassandra ( distributed key value dB), by design
> allow to store huge amount of data by partitioning it on multiple machine.
> 'Multiple' here means 'Cloud Scale'. Tens of thousands of machines spanning
> across data centres.
> This actually has no limit on data storage capacity then.
>
> As far as duration for persisting per node log, there is no reason why you
> can not store it for ever.
> On 1 Jun 2016 9:07 a.m., "VG" <vlin...@gmail.com> wrote:
>
> > Hi,
> >
> > There are number of messages floating on the internet suggesting that
> Kafka
> > cannot persist messages infinitely ?
> > Primarily that Kafka partitions are pinned to a node and they can’t
> > outgrow the storage capacity of a node..
> >
> > Can someone help me understand this limitation and how it can be
> overcome ?
> >
> > Regards,
> > Vipul
> >
>

Reply via email to