Re: refactoring ZK so it is plugable, would this make sense?
You can run with a single node zookeeper cluster also. See http://zookeeper.apache.org/doc/r3.3.4/zookeeperStarted.html#sc_InstallingSingleMode Cheers, Erik. Op 9 okt. 2014, om 22:52 heeft S Ahmed het volgende geschreven: > I want kafka features (w/o the redundancy) but don't want to have to run 3 > zookeeper instances to save $$. > > On Thu, Oct 9, 2014 at 2:59 PM, Jun Rao wrote: > >> This may not be easy since you have to implement things like watcher >> callbacks. What's your main concern with the ZK dependency? >> >> Thanks, >> >> Jun >> >> On Thu, Oct 9, 2014 at 8:20 AM, S Ahmed wrote: >> >>> Hi, >>> >>> I was wondering if the zookeeper library (zkutils.scala etc) was designed >>> in a more modular way, would it make it possible to run a more "lean" >>> version of kafka? >>> >>> The idea is I want to run kafka but with a less emphasis on it being >>> durable with failover and more on it being a replacement for a standard >>> queue like kestrel. >>> >>> This way you could take advantage of how the other aspects of Kafka >>> (permanent log, etc etc.) >>> >>> I was just thinking if the zookeeper access was wrapped in something >> like: >>> >>> class DiscoverService >>> >>> def electLeader .. >>> def getFollower ... >>> >>> (I'm just making those methods up, but you get the point they are simply >>> the same calls zkutils etc. will be making to connect to zookeeper) >>> >>> Now the idea is, if you don't want to dedicate 3 servers to run >> zookeeper, >>> you could create your own implementation that e.g. returns data based on >> a >>> configuration file that is static and not a discover service like >>> zookeeper. >>> >>> Would wrapping the zookeper calls into a plugable/swapable service make >>> sense and allow you to still use Kakfa at a smaller scale or would this >> not >>> work for other reasons that I am overlooking? >>> >>
Re: refactoring ZK so it is plugable, would this make sense?
I want kafka features (w/o the redundancy) but don't want to have to run 3 zookeeper instances to save $$. On Thu, Oct 9, 2014 at 2:59 PM, Jun Rao wrote: > This may not be easy since you have to implement things like watcher > callbacks. What's your main concern with the ZK dependency? > > Thanks, > > Jun > > On Thu, Oct 9, 2014 at 8:20 AM, S Ahmed wrote: > > > Hi, > > > > I was wondering if the zookeeper library (zkutils.scala etc) was designed > > in a more modular way, would it make it possible to run a more "lean" > > version of kafka? > > > > The idea is I want to run kafka but with a less emphasis on it being > > durable with failover and more on it being a replacement for a standard > > queue like kestrel. > > > > This way you could take advantage of how the other aspects of Kafka > > (permanent log, etc etc.) > > > > I was just thinking if the zookeeper access was wrapped in something > like: > > > > class DiscoverService > > > >def electLeader .. > >def getFollower ... > > > > (I'm just making those methods up, but you get the point they are simply > > the same calls zkutils etc. will be making to connect to zookeeper) > > > > Now the idea is, if you don't want to dedicate 3 servers to run > zookeeper, > > you could create your own implementation that e.g. returns data based on > a > > configuration file that is static and not a discover service like > > zookeeper. > > > > Would wrapping the zookeper calls into a plugable/swapable service make > > sense and allow you to still use Kakfa at a smaller scale or would this > not > > work for other reasons that I am overlooking? > > >
Re: refactoring ZK so it is plugable, would this make sense?
This may not be easy since you have to implement things like watcher callbacks. What's your main concern with the ZK dependency? Thanks, Jun On Thu, Oct 9, 2014 at 8:20 AM, S Ahmed wrote: > Hi, > > I was wondering if the zookeeper library (zkutils.scala etc) was designed > in a more modular way, would it make it possible to run a more "lean" > version of kafka? > > The idea is I want to run kafka but with a less emphasis on it being > durable with failover and more on it being a replacement for a standard > queue like kestrel. > > This way you could take advantage of how the other aspects of Kafka > (permanent log, etc etc.) > > I was just thinking if the zookeeper access was wrapped in something like: > > class DiscoverService > >def electLeader .. >def getFollower ... > > (I'm just making those methods up, but you get the point they are simply > the same calls zkutils etc. will be making to connect to zookeeper) > > Now the idea is, if you don't want to dedicate 3 servers to run zookeeper, > you could create your own implementation that e.g. returns data based on a > configuration file that is static and not a discover service like > zookeeper. > > Would wrapping the zookeper calls into a plugable/swapable service make > sense and allow you to still use Kakfa at a smaller scale or would this not > work for other reasons that I am overlooking? >
RE: refactoring ZK so it is plugable, would this make sense?
Hi, I was wondering if the zookeeper library (zkutils.scala etc) was designed in a more modular way, would it make it possible to run a more "lean" version of kafka? The idea is I want to run kafka but with a less emphasis on it being durable with failover and more on it being a replacement for a standard queue like kestrel. This way you could take advantage of how the other aspects of Kafka (permanent log, etc etc.) I was just thinking if the zookeeper access was wrapped in something like: class DiscoverService def electLeader .. def getFollower ... (I'm just making those methods up, but you get the point they are simply the same calls zkutils etc. will be making to connect to zookeeper) Now the idea is, if you don't want to dedicate 3 servers to run zookeeper, you could create your own implementation that e.g. returns data based on a configuration file that is static and not a discover service like zookeeper. Would wrapping the zookeper calls into a plugable/swapable service make sense and allow you to still use Kakfa at a smaller scale or would this not work for other reasons that I am overlooking?
