Re: Which is True? Kafka site vs Confluent 3.2 site upgrade doc details contradiction regarding 0.10.2 clients backward compatible to resp. 0.10.0 vs 0.10.1?

The reason Streams API 0.10.2 is not backward compatible to 0.10.0 broker is not related to Producer/Consumer API. Streams API (as of 0.10.2) uses a new AdminClient (cf. KIP-4) for topic management that is not supported by 0.10.0 brokers. 0.10.0 broker topics are managed via ZK client only -- and

Re: Difference between with and without state store cleanup streams startup

Yes. That's correct. -Matthias On 3/31/17 9:32 PM, Sachin Mittal wrote: > Hi, > Ok so basically what I understand is that there are no global offset > maintained from changelog topic at broker level. > Every local state store maintains the offset under a local checkpoint file. > > And in order

Re: How to increase network throughput of Kafka cluster?

Then you will need even more parallel producers to saturate a 10 GigE network (if you don't hit you disk I/O limit first) -hans > On Apr 1, 2017, at 3:15 PM, Archie wrote: > > My replication factor is 1. > > Thanks, > Archie > >> On Sat, Apr 1, 2017 at 3:51 PM,

Re: How to increase network throughput of Kafka cluster?

My replication factor is 1. Thanks, Archie On Sat, Apr 1, 2017 at 3:51 PM, Hans Jespersen wrote: > What replication factor are you using? If you have a default replication > factor = 3 then a publish rate of 1.4 Gbps is actually 1.4 Gbps *3 = 4.2 > Gbps of network traffic.

kafka connection problem.

Hi! for last few days i am try to connect spark and kafka using direct streaming but I am not able to do so. I am facing following problem: http://stackoverflow.com/q/42991780/2315294 http://stackoverflow.com/q/43050654/2315294 . So,please tell me what else I can do? Thanks. Angshuman

Re: How to increase network throughput of Kafka cluster?

What replication factor are you using? If you have a default replication factor = 3 then a publish rate of 1.4 Gbps is actually 1.4 Gbps *3 = 4.2 Gbps of network traffic. If you are also consuming at the same time then it’s actually 4.2 Gbps + 1.4 Gbps = 5.6 Gbps. You would completely

Re: Kafka Streams: Is it possible to pause/resume consuming a topic?

Perfect, thanks so much Eno! On Sat, Apr 1, 2017 at 12:44 PM, Eno Thereska wrote: > Sure, here is an example of pausing: > https://github.com/apache/kafka/blob/trunk/streams/src/ > main/java/org/apache/kafka/streams/processor/internals/ > StreamTask.java#L165

Re: How to increase network throughput of Kafka cluster?

Producer performance also depends on disk I/O. You can try tuning batch.size, compression.type configs. also, add more partitions for more throughput. Useful presentation: https://www.slideshare.net/JiangjieQin/producer-performance-tuning-for-apache-kafka-63147600 On Sat, Apr 1, 2017 at 10:55

How to increase network throughput of Kafka cluster?

I have set up my kafka cluster in a network with 9.3 Gbps links. And am using the bin/kafka-producer-perf-test.sh script to test the throughput performance of kafka for a topic which has 1 partition. Right now I am only able to use 1.4 Gbps of network link, i.e. the script returns a throughput of

Re: Kafka Streams: Is it possible to pause/resume consuming a topic?

Sure, here is an example of pausing: https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamTask.java#L165

Re: Kafka Streams: Is it possible to pause/resume consuming a topic?

Hi Eno, Could you point to me where in code this is happening please? Thanks Tianji On Sat, Apr 1, 2017 at 11:45 AM, Eno Thereska wrote: > Tianji, > > You shouldn’t have to worry about pausing and resuming the consumer, since > that happens internally automatically. >

Re: Kafka Streams: Is it possible to pause/resume consuming a topic?

Tianji, You shouldn’t have to worry about pausing and resuming the consumer, since that happens internally automatically. Eno > On Apr 1, 2017, at 3:26 PM, Tianji Li wrote: > > Hi there, > > Say a processor that is consuming topic A and producing into topic B, and >

Kafka Streams: Is it possible to pause/resume consuming a topic?

Hi there, Say a processor that is consuming topic A and producing into topic B, and somehow the processing takes long time, is it possible to pause the consuming from topic A, and later on resume? Or does it make sense to do so? If not, what are the options to resolve this issue? Thanks Tianji

How many levels can Message Set nest recursively?

I am thinking about that: 1 If a Producer compresses a Message set which is nested more than 2 levels records recursively and sends it to broker, how does broker know which offset should be allocated to this message set without uncompressed all levels and getting the all records? 2

Re: Which is True? Kafka site vs Confluent 3.2 site upgrade doc details contradiction regarding 0.10.2 clients backward compatible to resp. 0.10.0 vs 0.10.1?

They are both true. The Apache text is talking about the compatibility of the Producer/Consumer API and the Confluent text is talking about the Streams API. -hans > On Mar 31, 2017, at 11:46 PM, Roger Vandusen > wrote: > > Read below and answer: So which is

Re: Kafka streams 0.10.2 Producer throwing exception eventually causing streams shutdown

Hi Sachin, In the bug fix release for 0.10.2 (and in trunk) we have now set max.poll.interval to infinite since from our experience with streams this should not be something that users set: https://github.com/apache/kafka/pull/2770/files .

Which is True? Kafka site vs Confluent 3.2 site upgrade doc details contradiction regarding 0.10.2 clients backward compatible to resp. 0.10.0 vs 0.10.1?

Read below and answer: So which is the source of truth ? Is 0.10.2.0 compatible to 0.10.0 or 0.10.1? Which site needs correction? From current Kafka docs. Statement from Kafka site: https://kafka.apache.org/documentation/#upgrade Starting with version 0.10.2, Java clients (producer and