Re: Does Kafka 0.9 can guarantee not loss data

In order to satisfy a produce response, there are two conditions: A. The leader's high watermark should be higher than the requiredOffset (max offset in that produce request of that partition) B. The number of in sync replica is greater than min.isr. The ultimate goal here is to make sure at

Re: Does Kafka 0.9 can guarantee not loss data

@wangguozhang，could you give me some advices. > 在 2016年9月22日，下午6:56，Kafka 写道： > > Hi all, > in terms of topic， we create a topic with 6 partition，and each with 3 > replicas. >in terms of producer，when we send message with ack -1 using sync > interface. >

Re: Kafka Streams Python client?

Hello Samuel, We are indeed going beyond Java languages in Kafka Streams now, and Python is one of the candidates. We do not have any active developments on that, and have not decided if it will be based on librdkafka; and personally I would love to check Dana's branch built on kafka-python when

Re: Exception while deserializing in kafka streams

Hi Walter, I downloaded the 0.10.0 jar and verified that the configure() function is auto-triggered when you get the serde classes from `context.keySerde / valueSerde`, which is auto-triggered if you use the DSL. And your Scala code is the same as to our examples code:

Re: Migrating old consumer offsets to new consumer

Hello Nick, You would need to upgrade your Kafka Server first if you have not done so. After that you can first follow this as you mentioned: https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowdowemigratetocommittingoffsetstoKafka(ratherthanZookeeper)in0.8.2 ? to move the committing

Does Kafka Sync/persist every message from a publisher by default?

Does Kafka Sync/persist every message from a publisher by default? If not, What settings should I change so I Sync every message?

Re: Kafka Streams Python client?

Yeah, I thought they might be; hopefully someone there can comment. Please do let me know if/when you push that up! I'm happy to help if I can. :) Thanks! On Thu, Sep 22, 2016 at 3:08 PM, Dana Powers wrote: > I am not aware of any active development. I did some initial

Re: Benchmarking kafka performance

Good afternoon. Any suggestions regarding benchmark tool would be greatly appreciated. Thanks On Mon, Sep 19, 2016 at 8:18 AM, Vadim Keylis wrote: > Good morning. Which benchmarking tools we should use to compare > performance of 0.8 and 0.10 versions? Which metrics

Re: Error kafka-stream method punctuate in context.forward()

Hello Hamza, Returning the same instance of the processor is not recommended in Kafka Streams, since when you have multiple tasks or threads within the same instance, they will call the topology builder, hence the processor supplier to construct a topology for each task; if the same processor is

Re: Kafka Streams Python client?

I am not aware of any active development. I did some initial work on a branch on my laptop w/ basic functionality built on kafka-python. I'm happy to ping you if/when I push to github. I expect that Confluent may also be preparing something with their python client wrapper around librdkafka, but I

Kafka Streams Python client?

Hi all, Is there active development on a Kafka Streams-esque API for Python? I'm interested in working on such a project. Jay Kreps' article "Introducing Kafka Streams" implies that introducing the API in other languages is upcoming, but I can't find anything about a Python-specific effort

Re: why did Kafka choose pull instead of push for a consumer ?

@Gerard Thanks for this. It looks good any benchmarks on this throughput wise? On Thu, Sep 22, 2016 7:45 AM, Gerard Klijs gerard.kl...@dizzit.com wrote: We have a simple application producing 1 msg/sec, and did nothing to optimise the performance and have about a 10 msec delay between

Re: Exception while deserializing in kafka streams

Guozhang, I tried your suggestion. Below is the log from Serde, Serializer & Deserializer. Confirmed that KafkaAvroDeserializer.configure does get invoked. Line 379: 16/09/22 15:28:46 WARN GenericAvroSerdeWithSchemaRegistry: In > configure {num.standby.replicas=1, replication.factor=3, >

Re: why did Kafka choose pull instead of push for a consumer ?

We have a simple application producing 1 msg/sec, and did nothing to optimise the performance and have about a 10 msec delay between consumer and producer. When low latency is important, maybe pulsar is a better fit, https://www.datanami.com/2016/09/07/yahoos-new-pulsar-kafka-competitor/ . On

Does Kafka 0.9 can guarantee not loss data

Hi all, in terms of topic， we create a topic with 6 partition，and each with 3 replicas. in terms of producer，when we send message with ack -1 using sync interface. in terms of brokers，we set min.insync.replicas to 2. after we review the kafka broker’s code，we know that

Kafka Consumer group (High level consumer)

Hi, I'm looking for *consumer group* related settings of the Kafka server/cluster. - how can we tell the server to delete a consumer group if it has been inactive longer than a specific time ? - can this period be infinite ? - can this setting be specific to a consumer group ? - can there be a