Re: Recommended number of partitions on each broker

300 partitions is at lower side, surely that won't be the root cause. How about usage of network bandwidth for your nodes? Are they reachable to zookeeper? Are you executing some partition-rebalancing jobs (or re-assigning partitions) in parallel? On 2 March 2017 at 10:57, Jun MA

Re: Re: when will the messsages be sent to broker

Hi Matthias, I check my testing procedure again and again. I find a problem in that. I start a producer in debug mode and the breakpoint is suspended to all thread. It maybe block producing thread. After change to suspend to the main thread, the message is sent out immediately. Thanks.

Recommended number of partitions on each broker

Hi, I’m curious what’s the recommended number of partitions running on each individual broker? We have a 3 nodes clusters running 0.9.0.1, each one has 24 cores, 1.1T ssd, 48G ram, 10G NIC. There’s about 300 partitions running on each broker and the resource usage is pretty low (5% cpu, 50%

Re: when will the messsages be sent to broker

That's weird. If batch.size=0 and linger.ms=0 records should get send out immediately. buffer.memory is an upper bound for the overall memory. If you write to multiple partitions, there will be a buffer for each partitions and buffer.memory is shared over all buffers. Furthermore, if you exceed

Re: Consumption on a explicitly (dynamically) created topic has a 5 minute delay

For future reference - I asked this question on dev mailing list and based on the discussion there was able to come up with a workaround to get this working. Details here https://www.mail-archive.com/dev@kafka.apache.org/msg67613.html -Jaikiran On Wednesday 22 February 2017 01:16 PM,

Re: Re: when will the messsages be sent to broker

Hi, Records are not sent out immediately, even if configing batch.size=0 and linger.ms=0, They are buffered in the memory(buffer.memory). I don't want to do sync writes by future.get, which performance is not good. Have any configure can increase sending frequency? config buffer.memory=0?

Re: Securing Multi-Node single broker kafka instance

Hi Harsha , Just looked at the URL you shared . I have ensured that zookeeper.properties file is same across all nodes . Just like it's shown here . As i stated earlier , its working well for quite sometime . tickTime=2000 dataDir=/var/zookeeper/ clientPort=2181 initLimit=5 syncLimit=2

Re: Securing Multi-Node single broker kafka instance

Sure Harsha . I shall follow recommended method . However , i would like to add to the discussion that current deployment worked just fine . People were using it for quite sometime with no security . Do i need to create topics and all again if am enabling security ? On Thu, Mar 2, 2017 at 3:03

Re: Securing Multi-Node single broker kafka instance

Here is the recommended way to setup a 3-node Kafka cluster. Its always recommended to keep zookeeper nodes on different set of nodes than the one you are running Kafka. To go with your current 3-node installation. 1. Install 3-node zookeeper make sure they are forming the quorum

Re: Securing Multi-Node single broker kafka instance

Hi Harsha , Thanks a lot . Let me explain where am i stuck , i have three machines on which i am running apache kafka with single broker but zookeeper of each machine is configured with other machine. Example : node1=zk1,zk2,zk3 node2=zk1,zk2,zk3

Re: Securing Multi-Node single broker kafka instance

For inter broker communication over SSL all you need is to add security.inter.broker.protocol to SSL. "How do i make zookeeper talk to each other and brokers?" Not sure I understand the question. You need to make sure zookeeper hosts and port are reachable from your broker nodes. -Harsha On Wed,

Securing Multi-Node single broker kafka instance

Hi Team , Can you please help me understand , 1. How can i secure multi-node (3 machine) single broker (1 broker ) Apache Kafka deployment secure using SSL ? i tried to follow instructions here but found pretty confusing . https://www.confluent.io/blog/apache-kafka-security-authoriz

Problem connecting on Kafka cluster through a VPN IPSec

Hi All, Does everyone knows if is it possible to connect on Kafka brokers remotely using IPSec VPN ? If I connect through Internet it works fine, but inside an IPSec tunnel does not work. I´m implementing a scenario that the producers and consumers will be located on a Datacenter A, and the

Re: Chatty StreamThread commit messages

Thank you everyone for the feedback! I filed https://issues.apache.org/jira/browse/KAFKA-4829 > On Mar 1, 2017, at 8:22 AM, Michael Noll wrote: > > Good point, Steven. +1 here. > > On Wed, Mar 1, 2017 at 8:52 AM, Damian Guy wrote: > >> +1 >> On

Re: Subscribe to user mailing list

It's self service. See: http://kafka.apache.org/contact -Matthias On 3/1/17 8:48 AM, Mina Aslani wrote: > Hi, > > I would like to subscribe to user mailing list. > > Best regards, > Mina > signature.asc Description: OpenPGP digital signature

Re: Kafka Streams - ordering grouped messages

Just wanted to add, that there is always the potential about late arriving records, and thus, ordering by timestamp will never be perfect... You should rather try to design you application in a way such that it can handle out-of-order data gracefully and try to avoid the necessity of ordering

Re: when will the messsages be sent to broker

There is also linger.ms parameter that is an upper bound how long a (not yet filled) buffer is hold before sending it even if it's not full. Furthermore, you can do sync writes and block until producer received all acks. But it might have a performance penalty.

Messages rejected by broker due to size

Hello, I'm looking for a bit of clarification on the outcome of a RecordTooLargeException in the broker that was not evident from the documentation. First, what happens to a large record that is rejected? Is it ignored by the broker or is there a way to recover a rejected message? Second, is

Subscribe to user mailing list

Hi, I would like to subscribe to user mailing list. Best regards, Mina

Re: error in kafka producer

It means that your request has timed out or could not be sent within request timeout ms, also check max block ms. You will have to tune/increase throughout (search threads) or the timeouts. On Feb 28, 2017 2:22 PM, "shyla deshpande" wrote: >

Re: Kafka Streams vs Spark Streaming

Steven, I guess my last answer was not completely correct. You might start with a new store, if the task gets moved to a different machine. Otherwise, we don't explicitly wipe out the store, but just reuse it in whatever state it is on restart. -Matthias On 2/28/17 2:19 PM, Matthias J. Sax

new mail archive service from Apache

Hi, Just want to pass along this to the community. There is a new mail archive service https://lists.apache.org. It's beta, but is the long-term solution for official ASF mail archives, and offers much better searching/threading than mail-archives.a.o does. Thanks, Jun

Re: Chatty StreamThread commit messages

Good point, Steven. +1 here. On Wed, Mar 1, 2017 at 8:52 AM, Damian Guy wrote: > +1 > On Wed, 1 Mar 2017 at 07:15, Guozhang Wang wrote: > > > Hey Steven, > > > > That is a good question, and I think your proposal makes sense. Could you > > file a JIRA

Re: Kafka Streams - ordering grouped messages

You could implement your own based sorting algorithm using the low level processor api, i.e, you have a processor that keeps a sorted list of records and then, periodically, perhaps on punctuate, it emits the sorted messages downstream. You could do something like:

Re: when will the messsages be sent to broker

By default producer batches records (into a buffer) until batch.size is reached, then records will be sent out. Try lowering batch.size in your producer config. You might want to look at the producer javadoc here:

kafka-connect-salesforce

Hi All, I am trying to use kafka-connect-salesforce . topic is created but there is no data in it. should i have to start producer also to send data ? Thank you.

Re: Kafka Streams - ordering grouped messages

Is there any way to sort grouped records before sending them to the reducer? *Ofir Sharony* BackEnd Tech Lead Mobile: +972-54-7560277 | ofir.shar...@myheritage.com | www.myheritage.com MyHeritage Ltd., 3 Ariel Sharon St., Or Yehuda 60250, Israel

Re: Kafka Streams - ordering grouped messages

Hi, The TimestampExtractor won't effect the order the records arrive in. It just provides a way for developers to use a timestamp other than the default. Thanks, Damian On Wed, 1 Mar 2017 at 12:34 Ofir Sharony wrote: > Hi, > > I have the following code on a

Kafka Streams - ordering grouped messages

Hi, I have the following code on a stream: .selectKey(...) .groupByKey(...) .reduce(...) The records arrived to the Reducer function in the same order they were consumed from Kafka I have implemented a TimestampExtractor, extracting the wanted timestamp from each record, but unfortunately this

when will the messsages be sent to broker

Hi all, When will the messsages be sent in kafka0.10.0?If I use KafkaProducer.send to send one message, the messsages isn't sent immediately except invoke flush or close. Thanks. Yuanjia Li