Re: Kafka node liveness check

Thanks Elias for sharing On Mon, 29 Feb 2016 at 22:23 Elias Abacioglu < elias.abacio...@deltaprojects.com> wrote: > Crap, forgot to remove my signature.. I guess my e-mail will now get > spammed forever :( > > > > > > On Mon, Feb 29, 2016 at 3:14 PM, Elias Abacioglu < >

kafka snappy compression

hi,all: why i send a messageset of 872 bytes snappy-compressed data to kafka,and the messageSet stored in kafka was of 3940 payloadsize? I used sarama AsyncProducer using snappy compression to send messageSet(5 messages per messageSet),I found that the origin data were 3940

About bootstrap.servers

Hi All: I want to known why use "bootstrap.servers" to establish the initial connection to the Kafka cluster when I initialize a Producer or Consumer？ Why not let producer or consumer connect to the zookeeper to get the broker's ip and port? I think this is one way to decouple the

Re: due to invalid request: Request of length

Hello, What is the size of the message you are sending. If it is 1937006964 then you should set message.max.bytes and replica.fetch.max.bytes to values larger than the size of the message you are trying to send. If it isnt, then its most likely an encoding issue on your side. The kafka message

larger than the maximum size

Hi, Anybody know how to fix the following error? I didn't send any large size message, seems the system was sending a large message itself: [2016-02-26 20:33:43,025] INFO Closing socket connection to /x due to invalid request: Request of length 1937006964 is not valid, it is larger than the

due to invalid request: Request of length

[2016-02-26 20:33:43,025] INFO Closing socket connection to /x due to invalid request: Request of length 1937006964 is not valid, it is larger than the maximum size of 104857600 bytes. (kafka.network.Processor) [2016-02-26 20:33:43,047] INFO Closing socket connection to /x due to invalid request:

About the number of partitions

Hello. I have questions about how many partitions are optimal while using kafka. As far as I know, even if there are multiple consumers that belong to a consumer group, say *group_A*, only one consumer can receive a kafka message produced by a producer if there is a partition. So, as a result,

Unavailable partitions (Leader: -1 and ISR is empty) and we can't figure out how to get them back online

Hi, We have 44 partitions in our cluster that are unavailable. kafka-topics.sh is reporting them with Leader: -1, and with no brokers in the ISR. Zookeeper says that broker 5 should be the partition leader for this topic partition. These are topics with replication-factor 1. Most of the topics

Re: Kafka 0.9 Spout

Ankit, We've one in the making and getting to ready to be merged in . You can follow the PR here https://github.com/apache/storm/pull/1131 . Any storm related questions please send an email at these lists http://storm.apache.org/getting-help.html . Thanks,

Re: Need to understand the graph

Hi Alexis, Thanks for the detailed response. A few questions 1. What is the response that is sent back by Kafka. Does it block the client till the message is successfully published/replicated to the servers ? A client here is a REST service which just pushes data to a kafka topic 2. Is

Re: Kafka Rest Proxy

Jan, I don't use the rest proxy, but Confluent has a mailing list where you can probably get more info: Here's the direct link: https://groups.google.com/forum/#!forum/confluent-platform And it is linked off of here: http://www.confluent.io/developer#documentation -James > On Mar 1, 2016,

Kafka 0.9 Spout

Hi All, We would need to use the SSL feature of Kafka and that would require the Kafka Spout upgrade from version 0.8.x to 0.9.x as the SSL is only supported by new consumer API. Please share any URL for the same. -- Thanks, Ankit

Fwd: Is Apache Kafka ready for deployment in IPv6/IPv4 or IPv6-only networks?

Regards to all the list. I sent a question at Twitter yesterday asking this, and Neha Narkhede, from Confluent said to me that the best place to ask this is here, so here are my questions: - Is Kafka ready to be deployed in dual stack networks where you could create an Kafka´s architecture using

Large Size Error even when the message is small

[2016-02-26 20:33:42,997] INFO Closing socket connection to /x due to invalid request: Request of length 1937006964 is not valid, [2016-02-26 20:33:42,997] INFO Closing socket connection to /10.224.146.58 due to invalid request: Request of length 1937006964 is not valid, it is larger than the

Re: Need to understand the graph

Also the metrics kafka.network.RequestChannel.RequestQueueSize and ResponseQueueSize will give you the saturation of the network and IO threads. On Tue, Mar 1, 2016 at 9:21 AM Alexis Midon wrote: > "request queue time" is the time it takes for IO threads to pick up the

Re: Need to understand the graph

"request queue time" is the time it takes for IO threads to pick up the request. As you increase the load on your broker, it makes sense to see higher queue time. Here are more details on the request/response model in a Kafka broker (0.8.2). All your requests and responses are belong to

Re: Spark Streaming + Kafka 0.9

These two issues track progress of Kafka consumer 0.9. https://github.com/apache/spark/pull/10953 https://github.com/apache/spark/pull/11143 Marko Bonaći Monitoring | Alerting | Anomaly Detection | Centralized Log Management Solr & Elasticsearch Support Sematext | Contact

Spark Streaming + Kafka 0.9

Has anyone built the Spark Streaming kafka receiver for Kafka 0.9? Asking since spark master branch is still using 0.8.2.1 -- Adam Kunicki StreamSets | Field Engineer mobile: 415.890.DATA (3282) | linkedin

Kafka Rest Proxy

Hey guys, Is someone using the kafka rest proxy from confluent? We have an issue, that all messages for a certain topic end up in the same partition. Has anyone faced this issue before? We're not using a custom partitioner class, so it's using the default partitioner. We're sending

Re: Turn off retention.ms

A very high retention time (e.g. 50 years in milliseconds) would effectively accomplish this. On 3/1/16, 8:39 AM, "Roshan Punnoose" wrote: >Hi, > >Is it possible to set up a topic with only a size limit (retention.bytes) >and not a time limit (retention.ms)? > >Roshan

Is Apache Kafka ready for deployment in IPv6/IPv4 or IPv6-only networks?

Regards to all the list. I sent a question at Twitter yesterday asking this, and Neha Narkhede, from Confluent said to me that the best place to ask this is here, so here are my questions: - Is Kafka ready to be deployed in dual stack networks where you could create an Kafka´s architecture using

Re: monitor rebalanace events

This for the new consumer? or old high-level consumer? For the old high-level consumer, rebalance happens on the consumer side for the following events: * Opening new streams * ZK Session expiration * When partition information changes or consumer information changes I don't think there is a

Turn off retention.ms

Hi, Is it possible to set up a topic with only a size limit (retention.bytes) and not a time limit (retention.ms)? Roshan

Re: KafkaConsumer poll() problems

Producer : https://gist.github.com/r0perice/9ce2bece76dd4113a44a Consumer : https://gist.github.com/r0perice/8dcee160017ccd779d59 Console : https://gist.github.com/r0perice/5a8e2b2939651b1ac893 2016-03-01 14:50 GMT+01:00 craig w : > Can you try posting your code into a Gist

Re: KafkaConsumer poll() problems

Can you try posting your code into a Gist (gist.github.com) or Pastebin, so it's formatted and easier to read? On Tue, Mar 1, 2016 at 8:49 AM, Péricé Robin wrote: > Hello everybody, > > I'm having troubles using KafkaConsumer 0.9.0.0 API. My Consumer class > doesn't

KafkaConsumer poll() problems

Hello everybody, I'm having troubles using KafkaConsumer 0.9.0.0 API. My Consumer class doesn't consumer messages properly. - | *Consumer*.java | - public final void run() { try { *consumer.subscribe(Collections.singletonList(topicName));* boolean

monitor rebalanace events

Is there some way to monitor when a rebalance occurs other than at the consumer level?