Re: Per topic MBeans leaking

Thanks Alberto. On Fri, Jan 13, 2017 at 9:02 AM, wrote: > Sure, done: KAFKA-4629 > > Thanks, > Alberto > > > -Original Message- > From: Guozhang Wang [mailto:wangg...@gmail.com] > Sent: 13 January 2017 16:32 > To: users@kafka.apache.org >

Brokers stuck trying to shrink ISR down to themselves

Hey folks - we had a blip on one of our smaller clusters last night (3 nodes), around the same time as network maintenance in the DC that it was operating in. This caused a partition in the network between the brokers and Zk, as well as between the brokers themselves. The partition was for approx.

Re: Error finding consumer coordinators after restart

Just wanted to close the loop on this. It seems the consumer offset logs might have been corrupted by the system restart. Deleting the topic logs and restarting the Kafka service cleared up the problem. Thanks, Dave On 1/12/17, 2:29 PM, "Dave Hamilton" wrote:

Re: Kafka delay

batch.size, linger.ms, max.in.flight.requests.per.connection, etc all effect the trade off between latency and throughput. Your throughput is very low so you should be able to tune for much better latency. Very easy to get 1-2 ms of latency out of Kafka. Here is a video on Tuning Kafka for low

Kafka Connect requestTaskReconfiguration

Hey kafka-users, Is it normal for a Kafka Connect source connector that calls requestTaskReconfiguration to cause all the connectors on the kafka-connect distributed system to be stopped and started? One of my three connectors (2x source, 1x sink) runs a background thread that will occasionally

RE: Per topic MBeans leaking

Sure, done: KAFKA-4629 Thanks, Alberto -Original Message- From: Guozhang Wang [mailto:wangg...@gmail.com] Sent: 13 January 2017 16:32 To: users@kafka.apache.org Subject: Re: Per topic MBeans leaking * " This message originates from outside

Re: Taking a long time to roll a new log segment (~1 min)

Yep, that's exactly what the output is showing. On Fri, Jan 13, 2017 at 11:25 AM, Tauzell, Dave < dave.tauz...@surescripts.com> wrote: > I haven't used dtrace, but is it possible to have it running and recording > the ftruncate64 times? Then when you see one of these long roll times look > at

Re: Kafka Streams: from a KStream, aggregating records with the same key and updated metrics ?

Hi Nicolas, There is a lot here, so let's try to split the concerns around some themes: 1. The Processor API is flexible and can definitely do what you want, but as you mentioned, at the cost of you having to manually craft the code. 2. Why are the versions used? I sense there is concern about

Re: Per topic MBeans leaking

The way you used admin tools to create and delete topics looks reasonable to me. I suspect there may be some race conditions causing the metrics to not be de-registered. Could you file a JIRA to track this issue so we can investigate? Guozhang On Fri, Jan 13, 2017 at 8:12 AM,

RE: Taking a long time to roll a new log segment (~1 min)

I haven't used dtrace, but is it possible to have it running and recording the ftruncate64 times? Then when you see one of these long roll times look at the dtrace log to see if it was that call? -Dave -Original Message- From: Stephen Powis [mailto:spo...@salesforce.com] Sent: Friday,

Per topic MBeans leaking

Hi, In our application we create and delete topics dynamically. In our dev/test environment, where we create and delete a lot of topics, I've noticed that the memory usage of the Kafka brokers slowly increases and then noticed the huge amount of MBeans related to topics that should have been

Re: Kafka delay

Can you clarify what you mean by "process time"? Is this the time it takes to have a message sent from the producer appear in a consumer? You may be seeing the result of batching. The producer batches messages so if you are testing sending a single message at a time you will get different

Re: Taking a long time to roll a new log segment (~1 min)

So the underlying system call is ftruncate64, logged using dtrace. # BEGIN stack trace for ftruncate, call took 34160541200ns: > args== > 0x7f5f9a1134d7 : ftruncate64+0x7/0x30 [/lib64/libc-2.12.so] > 0x7f5f987b80c3 : 0x7f5f987b80c3 >

RE: First time implimention of publish and subscribe

When you say "high transaction" what are your estimates? ActiveMQ will be easier to get started with. If you design your code to hide the queuing it won't be too hard to switch between them. Kafka provides (I think) more scalability and HA but is going to more complicated to operate if you

First time implimention of publish and subscribe

I am being tasked to implement a publish and subscribe database system for patient data and am looking at apache to perform this but am unclear as to which product would be best to use. It looks like apache kafka but I have found other apache products that provide this function as well. Is

Kafka delay

Hi, we are developing a kafka environment in order to be able to process up hundreds of messages per second. The point is that if we send in different partitions a lot of messages, the time of the process is very good in ms, however if we send just a single message, the minimum time required

Re: Taking a long time to roll a new log segment (~1 min)

Hi Stephen, I think it's worth capturing this in a JIRA so that it doesn't get lost. Ismael On Thu, Jan 12, 2017 at 7:22 PM, Stephen Powis wrote: > I've further narrowed it down to this particular line: >

Re: Kafka Streams: consume 6 months old data VS windows maintain durations

That should work. Thanks Eno > On 12 Jan 2017, at 21:21, Nicolas Fouché wrote: > > Thanks Eno ! > > My intention is to reprocess all the data from the beginning. And we'll > reset the application as documented in the Confluent blog. > We don't want to keep the previous

Re: java.lang.OutOfMemoryError: Java heap space while running kafka-consumer-perf-test.sh

Perhaps the default heap options aren't sufficient for your particular use of the tool. The consumer perf test defaults to 512MB. I'd simply try increasing the max heap usage: KAFKA_HEAP_OPTS="-Xmx1024M" to bump it up a bit. -Ewen On Wed, Jan 11, 2017 at 2:59 PM, Check Peck

Re: Taking a long time to roll a new log segment (~1 min)

That line is truncating a file. If the underlying implementation is synchronous, it could potentially require multiple disk writes to remove blocks from the file, update the list of free blocks, etc. That could potentially cause the latency. I'm not sure that's actually where the cost is coming