Re: Best ways to unit test consumer that is based on simpleConsumer

Additional information. We currently using kafka 0.8 and client will be written in java. On Wed, Mar 12, 2014 at 10:41 PM, Vadim Keylis vkeylis2...@gmail.comwrote: Good evening. I am writing consumer that is based on SimpleConsumer. I would like to write integration/unit tests that simulate

Getting logs in kakfaServer-gc.log even if Log level is set to OFF in log4j.properties

Hi I am using kafka_2.8.0-0.8.0.tar.gz bundle. I have set Log level as OFF in log4j.properties file and according to this log level,the logging is off(there should be no log generation in any .log file). But Kafka is generating logs in kafkaServer-gc.log file.Please tell if logging of this

Re: Getting logs in kakfaServer-gc.log even if Log level is set to OFF in log4j.properties

That kafkaServer-gc.log contains GC logs produced by the JVM. The JVM GC logging options are set in kafka-run-class.sh script from Kafka distribution. GC logging is enabled by default. You should be able to override the default behavior with KAFKA_GC_LOG_OPTS variable (never tried it myself

consumer reaction to auto-create topics (0.8.1)

Hi there, A few days ago i have written about an issue (0.8.0) when auto-create topic is enabled and consumer starts before producer (consumer never sees any message and they seem to be lost even by the brokers) Now i have done the same test with 0.8.1 and the issue is partially fixed but now

Re: What's new in Kafka 0.8.1?

Thanks for the writeup. In 0.8.1, we also significantly improved the partition reassignment tool. Jun On Wed, Mar 12, 2014 at 8:57 PM, Jay Kreps jay.kr...@gmail.com wrote: Hi guys, I wrote up a quick blog post on the new stuff in 0.8.1:

Re: consumer reaction to auto-create topics (0.8.1)

Hello David, This is a known issue with consumer starting before the topic is created: https://issues.apache.org/jira/browse/KAFKA-1006 And we are still figuring the right way to fix it. Could you describe your use cases where you need to start the consumer beforehand? Guozhang On Thu, Mar

Re: consumer reaction to auto-create topics (0.8.1)

One system I am working on we have something similar however we are creating topics like how the TopicCommand does but different. We keep those operations persisted in a meta store so consumers that have to-do something know that everything is all ready for them (as often it is not just Kafka)

Re: What's new in Kafka 0.8.1?

Thanks, Jay. Great write up. I noticed a bad link to the docs for basic operations ( http://localhost/documentation.html#basic_ops). It's in the paragraph that starts with We also improved a lot of operational activities Roger On Wed, Mar 12, 2014 at 8:57 PM, Jay Kreps

Re: consumer reaction to auto-create topics (0.8.1)

Hi David, This issue should only happen if the following conditions are satisfied 1) The consumer does not have any offset stored in ZK for this topic when it first fetches messages for it. 2) The consumer's default offset reset value is latest. 3) After the topic is created, it will take a bit

Broker side consume-request filtering

Hi, I have a use case that might be relevant to the new consumer API but that would require most of the work on the broker. I would be surprised if it had not been discussed before but I was not able to find any directly related thread. Has there been any discussion about providing broker side

Broker plugins

Hi, I have a use case for which it would be useful with pluggable processing functions in the broker. We have some data containing sensitive information which is legally ok to transmit over the internal network to the Kafka brokers and keep in volatile memory but not to flush to disk

Re: Broker plugins

In general, the preference has been to avoid having user code run on the brokers since that just opens a can of worms where the broker logic get's complicated trying to deal with errors that the user code can throw. The suggestion is to push any user specific processing to the client side. In this

Re: Best ways to unit test consumer that is based on simpleConsumer

Currently, we don't have a mock class for the server. The SimpleConsumer actually doesn't do much with the fetch request and response. Take a look at the fetch() API on SimpleConsumer. It merely returns the response back to the user. Thanks, Neha On Wed, Mar 12, 2014 at 11:58 PM, Vadim Keylis

Re: consumer reaction to auto-create topics (0.8.1)

I think it is worth fixing https://issues.apache.org/jira/browse/KAFKA-1006since the behavior for new topic discovery is broken in the new consumer. Even if the consumer sets auto.offset.reset=largest, it is very unintuitive to apply that behavior to the discovery of new topics. I think that new

Re: Broker plugins

Or introduce an app layer between the producers and kafka that does the processing without changes/load to the producers. On Thu, Mar 13, 2014 at 1:18 PM, Neha Narkhede neha.narkh...@gmail.comwrote: In general, the preference has been to avoid having user code run on the brokers since that

Re: Best ways to unit test consumer that is based on simpleConsumer

Neha. That is correct SimpleConsumer does not do much, but my code that is calling it need to react properly at error response and I wanted to test that. On Thu, Mar 13, 2014 at 1:21 PM, Neha Narkhede neha.narkh...@gmail.comwrote: Currently, we don't have a mock class for the server. The

Re: Best ways to unit test consumer that is based on simpleConsumer

I see. In that case, it might be easier to write a MockSimpleConsumer and have it return different FetchResponse objects with the error codes you want to test. We don't have a MockSimpleConsumer as well. If you happen to write one, you could use it with EasyMock to get the fetch() API to return

Re: Broker plugins

Thanks for the response, indeed the encryption/concealment should ideally be done on the producer side but it's just not feasible in some of our applications so a middle layer would be needed. So far our thoughts have been around using Flume interceptors but that means introducing another moving

Invalid topic metadata response

After reassigning a topic to 5 new brokers, the topic metadata returned by the brokers for the topic is causing an exception when a producer tries to read it: java.util.NoSuchElementException: key not found: 7 ... at kafka.api.PartitionMetadata$.readFrom(TopicMetadata.scala:104) The reassignment

Re: Invalid topic metadata response

There are some known bugs with the partition reassignment tool in 0.8. We fixed a lot of those, tested it on large set of partitions and found it to be stable in 0.8.1. Could you give 0.8.1 a spin instead? On Thu, Mar 13, 2014 at 4:05 PM, Ryan Berdeen rberd...@hubspot.com wrote: After

Re: consumer reaction to auto-create topics (0.8.1)

If you set auto.offset.reset to smallest, the consumer should pick up the first message on new topics. Thanks, Jun On Thu, Mar 13, 2014 at 2:42 AM, David Morales de Frías dmora...@paradigmatecnologico.com wrote: Hi there, A few days ago i have written about an issue (0.8.0) when

zookeeper exception

Hi all i use 0.8.1 release to do test, and get following error logs [hadoop@nelo76 libs]$ [2014-03-14 12:11:44,310] INFO Partition [nelo2-normal-logs,0] on broker 0: Shrinking ISR for partition [nelo2-normal-logs,0] from 0,1 to 0 (kafka.cluster.Partition) [2014-03-14 12:11:44,313]