kafka0.10 spark2.10

hi,all: when I run stream application for a few minutes ,I got this error : 17/06/22 10:34:56 INFO ConsumerCoordinator: Revoking previously assigned partitions [comment-0, profile-1, profile-3, cwb-3, bizs-1, cwb-1, weibocomment-0, bizs-2, pages-0, bizs-4, pages-2, weibo-0, pages-4, weibo-4,

RE: Kafka MirrorMaker - not replicating messages after being brought up

Hello, I have the same problem with Kafka 0.10.1.0, but MirrorMaker is not replicating anything, without any error message. I’ve been scratching my head for a demi-dozen of hours now, and I can’t think of what’s going on with my setup, my hundreds of topics keep unmirrored to my destination

Re: Kafka MirrorMaker - not replicating messages after being brought up

Karan, have you got auto.offset.reset in your consumer.properties? https://kafka.apache.org/documentation/#newconsumerconfigs On Thu, Jun 22, 2017 at 2:00 AM, karan alang wrote: > Hi All, > > I've 2 Kafka clusters (Kafka 10) & I'm trying to test the MirrorMaker >

Re: [DISCUSS] Streams DSL/StateStore Refactoring

I have been thinking about reducing all these overloaded functions for stateful operations (there are some other places that introduces overloaded functions but let's focus on these only in this discussion), what I used to have is to use some "materialize" function on the KTables, like:

Re: [DISCUSS]: KIP-161: streams record processing exception handlers

Thanks for the updated KIP, some more comments: 1.The config name is "default.deserialization.exception.handler" while the interface class name is "RecordExceptionHandler", which is more general than the intended purpose. Could we rename the class name accordingly? 2. Could you describe the full

Re: Handling 2 to 3 Million Events before Kafka

Getting good concurrency in a webapp is more than doable. Check out these benchmarks: https://www.techempower.com/benchmarks/#section=data-r14=ph=db I linked to the single query one because thats closest to a single operation like you will be doing. I'd also note if the data delivery does not

Kafka MirrorMaker - not replicating messages after being brought up

Hi All, I've 2 Kafka clusters (Kafka 10) & I'm trying to test the MirrorMaker functionality. Here is what i did : 1) I have identical topics Topic1 on 2 Kafka clusters - Cluster1 & Cluster2 2) On Cluster1, I publish 100 messages on Topic1 3) I've 2 consumers reading messages from the 2 topics

Re: Max message size and compression

If you are compressing messages than size of "compressed" message should be less than what's specified in these parameters. On Sat, Jun 17, 2017 at 7:46 PM, Eli Jordan wrote: > Hi > > max.message.bytes controls the maximum message size the kafka server will > process >

Re: [DISCUSS] KIP-163: Lower the Minimum Required ACL Permission of OffsetFetch

I appreciate everyone's feedback so far on this KIP. Before starting a vote, I'd like to also ask for feedback on the "Additional Food for Thought" section in the KIP:

Re: [VOTE] 0.11.0.0 RC1

That looks better than mine, nice! I think the tooling matters a lot to the usability of the product we're shipping, being able to test out Kafka's features on your own hardware/setup is very important to knowing if it can work. On Wed, Jun 21, 2017 at 8:01 PM, Apurva Mehta

Re: [VOTE] 0.11.0.0 RC1

Hi Tom, I actually made modifications to the produce performance tool to do real transactions earlier this week as part of our benchmarking (results published here: bit.ly/kafka-eos-perf). I just submitted that patch here: https://github.com/apache/kafka/pull/3400/files I think my version is

Re: Kafka MirrorMaker - errors/warning

Hi All - here is the update on this. I was able to fix the following warnings - 1) WARN Property bootstrap.servers is not valid (kafka.utils.VerifiableProperties) -> removed bootstrap.servers from mmConsumer.config (IT IS REQUIRED ONLY IN mmProducer.config) 2) zk.connectiontimeout.ms is

New Kafka Producer or the Old One ???

Hello All - I've *Kafka 0.9* & I'm running this command to publish records to Kafka topics - $KAFKA_HOME/bin/kafka-verifiable-producer.sh --topic mmtopic1 --max-messages 500 --broker-list localhost:9092,localhost:9093,localhost:9094,localhost:9095 --producer.config

RE: Handling 2 to 3 Million Events before Kafka

I’m not really familiar with Netty so I won’t be of much help. Maybe try posting on a Netty forum to see what they think? -Dave From: SenthilKumar K [mailto:senthilec...@gmail.com] Sent: Wednesday, June 21, 2017 10:28 AM To: Tauzell, Dave Cc: users@kafka.apache.org; senthilec...@apache.org;

Re: [VOTE] 0.11.0.0 RC1

Hi there, I'm -1 (non-binding) on shipping this RC. Heroku has carried on performance testing with 0.11 RC1. We have updated our test setup to use 0.11.0.0 RC1 client libraries. Without any of the transactional features enabled, we get slightly better performance than 0.10.2.1 with 10.2.1 client

Re: [DISCUSS] Streams DSL/StateStore Refactoring

To make it clear, it’s outlined by Damian, I just copy pasted what he told me in person :) Eno > On Jun 21, 2017, at 4:40 PM, Bill Bejeck wrote: > > +1 for the approach outlined above by Eno. > > On Wed, Jun 21, 2017 at 11:28 AM, Damian Guy wrote: >

Re: [DISCUSS] Streams DSL/StateStore Refactoring

+1 for the approach outlined above by Eno. On Wed, Jun 21, 2017 at 11:28 AM, Damian Guy wrote: > Thanks Eno. > > Yes i agree. We could apply this same approach to most of the operations > where we have multiple overloads, i.e., we have a single method for each > operation

Re: Handling 2 to 3 Million Events before Kafka

So netty would work for this case ? I do have netty server and seems to be i'm not getting the expected results .. here is the git https://github.com/senthilec566/netty4-server , is this right implementation ? Cheers, Senthil On Wed, Jun 21, 2017 at 7:45 PM, Tauzell, Dave

Re: [DISCUSS] Streams DSL/StateStore Refactoring

Thanks Eno. Yes i agree. We could apply this same approach to most of the operations where we have multiple overloads, i.e., we have a single method for each operation that takes the required parameters and everything else is specified as you have done above. On Wed, 21 Jun 2017 at 16:24 Eno

Re: [DISCUSS] Streams DSL/StateStore Refactoring

(cc’ing user-list too) Given that we already have StateStoreSuppliers that are configurable using the fluent-like API, probably it’s worth discussing the other examples with joins and serdes first since those have many overloads and are in need of some TLC. So following your example, I guess

RE: Handling 2 to 3 Million Events before Kafka

I see. 1. You don’t want the 100k machines sending directly to kafka. 2. You can only have a small number of web servers People certainly have web-servers handling over 100k concurrent connections. See this for some examples: https://github.com/smallnest/C1000K-Servers . It

Re: Handling 2 to 3 Million Events before Kafka

Thanks Jeyhun. Yes http server would be problematic here w.r.t network , memory .. Hi Dave , The problem is not with Kafka , it's all about how do you handle huge data before kafka. I did a simple test with 5 node Kafka Cluster which gives good result ( ~950 MB/s ) ..So Kafka side i dont see a

RE: Handling 2 to 3 Million Events before Kafka

What are your configurations? - production - brokers - consumers Is the problem that web servers cannot send to Kafka fast enough or your consumers cannot process messages off of kafka fast enough? What is the average size of these messages? -Dave -Original Message- From: SenthilKumar

Re: Handling 2 to 3 Million Events before Kafka

Hi, With kafka you can increase overall throughput by increasing the number of nodes in a cluster. I had a similar issue, where we needed to ingest vast amounts of data to streaming system. In our case, kafka was a bottleneck, because of disk I/O. To solve it, we implemented (simple) distributed

Handling 2 to 3 Million Events before Kafka

Hi Team , Sorry if this question is irrelevant to Kafka Group ... I have been trying to solve problem of handling 5 GB/sec ingestion. Kafka is really good candidate for us to handle this ingestion rate .. 100K machines > { Http Server (Jetty/Netty) } --> Kafka Cluster.. I see the problem

Re: [DISCUSS]: KIP-161: streams record processing exception handlers

Thanks Guozhang, I’ve updated the KIP and hopefully addressed all the comments so far. In the process also changed the name of the KIP to reflect its scope better: https://cwiki.apache.org/confluence/display/KAFKA/KIP-161%3A+streams+deserialization+exception+handlers

Re: ticketing system Design

If your business flow involves human actions, personally I would look at a business process engine like the open source camunda. Even if you don't choose to use it in production, you can use it to prototype and evolve your design at the inception stage. There's a simple to run example that