Re: Federated Alerting for Prometheus, Elasticsearch and Apache Kafka systems

On Fri, Aug 18, 2017 at 1:15 AM, Martin Gainty wrote: > > > > From: Lucas Ponce > Sent: Thursday, August 17, 2017 10:18 AM > To: users@kafka.apache.org > Subject: Federated Alerting for Prometheus, Elasticsearch and Apache Kafka > systems > > Hello, > > > From Ha

Re: SASL_PLAINTEXT impact on throughput and packets

Hi, I think it is a good option to log denials at WARN level. Pls raise JIRA for this. On Fri, Aug 18, 2017 at 3:47 AM, Phillip Walker wrote: > The problem turns out to be logging in > kafka.security.auth.SimpleAclAuthorizor. We had logging on because we need > to log denied authorization atte

Re: Avro With Kafka

You can send avro record to kafka and consume it without schema registry. In my approach, avro schema file avsc must be in the classpath on both producer and consumer side. On producer side, first write value avro serializer and set the properties of key.serializer and value.serializer to kafka p

Re: Different Schemas on same Kafka Topic

Thank you Stephen for a very detailed write up. Really helpful. I was stuck in a concept of one schema per topic. Let me try this in my use case. Thank you very much. And thank you Svante. Regards, Sreejith On 18-Aug-2017 12:17 am, "Stephen Durfey" wrote: > There's a lot to unpack here, so I

Re: Federated Alerting for Prometheus, Elasticsearch and Apache Kafka systems

From: Lucas Ponce Sent: Thursday, August 17, 2017 10:18 AM To: users@kafka.apache.org Subject: Federated Alerting for Prometheus, Elasticsearch and Apache Kafka systems Hello, >From Hawkular team we are working to create a lightweight alerting component that

Re: SASL_PLAINTEXT impact on throughput and packets

The problem turns out to be logging in kafka.security.auth.SimpleAclAuthorizor. We had logging on because we need to log denied authorization attempts; all logging in that class is at debug level with no way to log only denials, so the volume is huge. With logging turned on, especially on clusters

Re: Continue to consume messages when exception occurs in Kafka Stream

Hi Duy, What kind of exception are you getting? With KIP-161 (checked in trunk) we allow log-and-skip type exception handlers for deserialization errors: https://cwiki.apache.org/confluence/display/KAFKA/KIP-161%3A+streams+deserialization+exception+handlers

Under replicated partitions for specific leader

I'm running into this error while writing to the topics: Caused by: org.apache.kafka.common.errors.NotEnoughReplicasException: Messages are rejected since there are fewer in-sync replicas than required. The topics and the internal `__consumer_offsert` topic have a replication factor set to 3. Wh

Continue to consume messages when exception occurs in Kafka Stream

Hi everyone, My kafka stream app has an exception (my business exception), and then it doesn't consume messages anymore. Is there any way to make my app continues consume messages when the exception occurs? Thanks -- *Duy Truong*

Re: Different Schemas on same Kafka Topic

There's a lot to unpack here, so I'll do my best to answer. How ?. You are always registering a schema against a topic using the > topicname and schema registry is assiging a unique id across the registry > cluster. Where is the global unique schema id here ? When I say globally unique, I mean t

Re: Different Schemas on same Kafka Topic

Well, the purpose of the schema registry is to map a 16 bit id to a avro schema. with or without rules on how you may update a schema with a given name. To decode avro you need a schema. Either you "know" whats in a given topic and then you can hardcode it. Or you prepend it with something. ie the

Re: Different Schemas on same Kafka Topic

Hi Stephen, Thank you very much. Please give clarity on the statement. "each unique avro schema has a unique id associated with it. That id can be used across multiple different topics. The enforcement of which schemas are allowed in a particular topic comes down to the combination of the subjec

Re: Different Schemas on same Kafka Topic

There is a little nuance to this topic (hehe). When it comes down to it, yes, each unique avro schema has a unique id associated with it. That id can be used across multiple different topics. The enforcement of which schemas are allowed in a particular topic comes down to the combination of the sub

RE: Different Schemas on same Kafka Topic

Hmm, I think you are right that you cannot have multiple schemas on the same topic. -Dave -Original Message- From: Sreejith S [mailto:srssreej...@gmail.com] Sent: Thursday, August 17, 2017 11:42 AM To: users@kafka.apache.org Subject: RE: Different Schemas on same Kafka Topic Hi Dave,

RE: Different Schemas on same Kafka Topic

Hi Dave, Would like to get a clarity on one thing. If i register more than one schema for a topic, i am providing topic-key, topic-value to the schema registry. Id is created by schema registry and it will create different version of different schema. Still all schema have same id. Am i right ?

RE: Different Schemas on same Kafka Topic

> How does consumer know A is the avro class when there could be other classes > like B,C and D denoting different schemas?. There isn't a good way. One option is to have an avro wrapper that contains type, version and data fields. Then you wrap everything. Another option is to do what Ka

RE: Different Schemas on same Kafka Topic

Thanks Dave. We may not want to start using schema registery immediately . We would have java producers and consumers . I might also go with using byte messages but when consumer de serilize how can they map the byte[] to the correct Avro object For example: KafkaConsumer consumer = new Kaf

Re: Few questions about how Kafka Streams manages tasks

Yes. I'm doing that. But in this case the container orchestration system is killing it due to OOM or loss of node (I am simulating various failure scenarios). But yeah, the fact that kafka streams only commits offsets of messages that have been through the entire pipeline kind has enabled me to tak

RE: Different Schemas on same Kafka Topic

It does. The way it works is that the Avro serializer precedes each message with a two-byte integer that references a schema id in the confluent schema registry. The Avro deserializer looks at this value to determine which schema to de-serialize with. In order for this to work you need to u

Federated Alerting for Prometheus, Elasticsearch and Apache Kafka systems

Hello, >From Hawkular team we are working to create a lightweight alerting component that can combine alerts from several systems like Prometheus, Elasticsearch, Apache Kafka and others. We would like to get feedback about these features and how we can improve them to be useful for the Apache K