aarti gupta created KAFKA-3154:
----------------------------------
Summary: Ability to shard Kafka based cluster
Key: KAFKA-3154
URL: https://issues.apache.org/jira/browse/KAFKA-3154
Project: Kafka
Issue Type: Wish
Components: core
Affects Versions: 0.9.0.0
Reporter: aarti gupta
Use Case:
A Kafka cluster with subsets of data residing on each shard.
The current replication and partitioning schemes are not well suited, because
all messages are routed to all nodes.
Can we route messages to a shard, based on a shard key ?
While partitioning on a topic is possible today, the entire topic is replicated
to n nodes where n is the replication factor for that topic.
Having shading built into existing Kafka clustering model, (as opposed to as a
proxy on top of it), providing a single logical cluster , with plug-ability of
shard keys, would be ideal. This would be similar to the model adopted by
MongoDB https://docs.mongodb.org/manual/core/sharding-introduction/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
