Re: Kafka mirror maker help

2018-04-27 Thread Peter Bukowinski
I run instances of Mirror Maker as supervisord tasks (http://supervisord.org <http://supervisord.org/>). I’d recommend looking into it. In addition to letting you sidestep the service issue, supervisord watches the processes and can auto-restart them if they stop for any reason. —

Re: multi-disk brokers data replication

2018-05-10 Thread Peter Bukowinski
, assuming you're running 1.0+. If you're not running 1.0+, then a single disk failure on a broker configured with JBOD will bring down the broker. Hope this helps, Peter Bukowinski On Thu, May 10, 2018 at 1:49 AM, Andrian Jardan <andrianjar...@gmail.com> wrote: > Hello everyo

Re: multi-disk brokers data replication

2018-05-10 Thread Peter Bukowinski
Oops, sorry about the name misspelling, Andrian. (spell-check just tried to correct it again). On Thu, May 10, 2018 at 11:41 AM, Peter Bukowinski <pmb...@gmail.com> wrote: > Adrian, > > Replicas are *always* assigned to different brokers. You cannot, for > example, depl

Re: multi-disk brokers data replication

2018-05-10 Thread Peter Bukowinski
version <1, the whole broker dies ? > > What happens when the disk is replaced then ? > > > On May 10, 2018, at 20:42, Peter Bukowinski <pmb...@gmail.com> wrote: > > > > Oops, sorry about the name misspelling, Andrian. (spell-check just tried > to > > c

Re: How frequent does the follower replica issue a fetch request to leader replica?

2018-04-11 Thread Peter Bukowinski
in the num.replica.fetchers broker parameter). For one of my small 5-broker clusters, my follower request rates range from 130 per second on the high end to 3 per second on the low end. I’ve configured 16 fetchers. -- Peter Bukowinski > On Apr 10, 2018, at 10:54 PM, Yu Watanabe <yu.w.ten...@gma

Re: log retention bytes and log segment bytes

2018-04-12 Thread Peter Bukowinski
Hi Amit, This is from the broker config section of the very good documentation on the kafka web site: https://kafka.apache.org/0100/documentation.html#brokerconfigs log.segment.bytes: The maximum size of a single log file (default 1GB) log.retention.bytes: The maximum size of the log before

Re: Default kafka log.dir /tmp | tmp-file-cleaner process

2018-04-18 Thread Peter Bukowinski
for config parameters. That being said, perhaps this particular config parameter should be highlighted for update when moving to production. -- Peter Bukowinski > On Apr 18, 2018, at 11:27 AM, adrien ruffie <adriennolar...@hotmail.fr> wrote: > > Hi Marc, > > > I t

Re: Using Kafka CLI without specifying the URLs every single time?

2018-04-20 Thread Peter Bukowinski
call the wrapper scripts by passing the name of the cluster as an argument and then passing the standard kafka options, e.g. ktopics --cluster my_cluster --list -- Peter Bukowinski > On Apr 20, 2018, at 3:23 AM, Horváth Péter Gergely > <horvath.peter.gerg...@gmail.com> wrote: &g

Re: Kafka Brokers compatibility with Jumbo Frames option.

2018-03-31 Thread Peter Bukowinski
is not set to jumbo frames, you will most likely see dropped or fragmented packets. — Peter Bukowinski > On Mar 31, 2018, at 6:51 AM, Pena Quijada Alexander <a.penaquij...@reply.it> > wrote: > > Hi all, > > My name is Alexander, a Linux System Administrator. I'm using K

Re: Number of partitions for offsets topic cannot be changed

2018-03-25 Thread Peter Bukowinski
cluster data from zookeepers, then starting brokers again with your desired server properties settings. -- Peter Bukowinski > On Mar 25, 2018, at 1:01 PM, Anu P <iamkafkau...@gmail.com> wrote: > > Thanks Swapnil. > > I changed *offsets.topic.num.partitions* in server.properties

Re: Visual tool for kafka?

2018-10-19 Thread Peter Bukowinski
https://github.com/yahoo/kafka-manager This does all that, and has admin features (which you can disable) that allow you to change or create topics, do partition reassignment and preferred leader election. — Peter > On Oct 18, 2018, at 11:52 PM,

Re: When Kafka stores group information in zookeeper?

2018-10-22 Thread Peter Bukowinski
It all depends on which type of consumer you are using. If you use an old (original) consumer, you must specify one or more zookeepers since group management info is stored in zookeeper. If you use a new consumer, group management is handled by the kafka cluster itself so you must specify one

Re: Open Source Schema Registry

2018-10-23 Thread Peter Bukowinski
Have a look at https://github.com/confluentinc/schema-registry > On Oct 23, 2018, at 9:28 AM, chinchu chinchu wrote: > > Hi folks, > We are looking to use open source schema registry with apache kafka 1.0.1 > and avro. *Do we need to write a