Apologies. Misinterpreted your question. I am curious to know if people
have faced issues with single partition on node in practice. Or because
most uses do not keep logs forever, this is generally not an issue? Was
this faced at LinkedIn?
On 1 Jun 2016 6:28 p.m., "VG" <vlin...@gmail.com> wrote:

You are wrong Unmesh.  Kafka design forces a partition to be on a single
node only.
My question is around the scalability of the partition itself.
How to overcome the restriction of a single node for a partition  ?

Any clues anyone...

On Wed, Jun 1, 2016 at 5:24 PM, Unmesh Joshi <unmeshjo...@gmail.com> wrote:

> I do not see why this is a limitation. Any data storage application you
use
> will be limited by physical capacity of the nodes.
> Distributed applications like Kafka (Distributed message broker), HDFS (
> Distributed file system), Cassandra ( distributed key value dB), by design
> allow to store huge amount of data by partitioning it on multiple machine.
> 'Multiple' here means 'Cloud Scale'. Tens of thousands of machines
spanning
> across data centres.
> This actually has no limit on data storage capacity then.
>
> As far as duration for persisting per node log, there is no reason why you
> can not store it for ever.
> On 1 Jun 2016 9:07 a.m., "VG" <vlin...@gmail.com> wrote:
>
> > Hi,
> >
> > There are number of messages floating on the internet suggesting that
> Kafka
> > cannot persist messages infinitely ?
> > Primarily that Kafka partitions are pinned to a node and they can’t
> > outgrow the storage capacity of a node..
> >
> > Can someone help me understand this limitation and how it can be
> overcome ?
> >
> > Regards,
> > Vipul
> >
>

Reply via email to