Re: WARN Error while fetching metadata with correlation id 1 : {MY_TOPIC?=INVALID_TOPIC_EXCEPTION} (org.apache.kafka.clients.NetworkClient)

Sorry my bad..I should not add "?" at the end On 6 April 2016 at 11:01, Ratha v <vijayara...@gmail.com> wrote: > Hi all; > when I run the following command with kafka 0.9.0.1 i get this > warnings[1]. Can you please tell me what is wrong with my topics? (I'm > talki

What is the best way to publish and consume different type of messages?

Hi all; Im using kafka 0.0.8V. I want to publish /consume byte[] objects, java bean objects, serializable objects and much more.. What is the best way to define a publisher and consumer for this type scenario? When I consume a message from the consumer iterator, I do not know what type of the

Why my consumer sometimes not listening messages?

I use older kafka consumer 0.8V. *Steps* - Starting listener - Send 10 messages .Listener listens around 4 messages. - Send single message. Listener not listening - Again single message published. Listener is not listening. Can anyone explain this behaviour? -- -Ratha

Apache kafka listener listens less messages than produced

I use old kafka consumer and producer(v 0.8x) When i produce 10 messages i get only 6 messages..Is there anything wrong in my listener .properties? I start listener first then producing .. *Listener properties* enable.auto.commit=true auto.commit.interval.ms=101

WARN Error while fetching metadata with correlation id 1 : {MY_TOPIC?=INVALID_TOPIC_EXCEPTION} (org.apache.kafka.clients.NetworkClient)

Hi all; when I run the following command with kafka 0.9.0.1 i get this warnings[1]. Can you please tell me what is wrong with my topics? (I'm talking to the kafka broker which runs in ec2) *#./kafka-console-consumer.sh --new-consumer --bootstrap-server kafka.xx.com:9092

Re: Is there any behavioural change to connect local server and remote server?

(or > advertised.host and advertised.port on older brokers). This is because EC2 > has internal and external addresses for each instance. > > -Ewen > > On Tue, Apr 5, 2016 at 5:13 AM, Ratha v <vijayara...@gmail.com> wrote: > > > Hi all; > > Is there any different connect

Re: How new consumer get to know about zookeeper URL?

Thanks Ian On 6 April 2016 at 13:26, Ian Wrigley <i...@confluent.io> wrote: > Hi Ratha > > New Consumers don’t use ZooKeeper; all offsets are stored in a Kafka topic. > > Regards > > Ian. > > > On Apr 5, 2016, at 10:20 PM, Ratha v <vijayara...@gmail.com>

Re: Is there any behavioural change to connect local server and remote server?

nts to use. If this is not set, # it will publish the same port that the broker binds to. advertised.port=9092 # The number of threads handling network requests num.network.threads=3 # The number of threads doing disk I/O num.io.threads=8 On 6 April 2016 at 10:08, Ratha v <vijayara...@gmail.com&

How new consumer get to know about zookeeper URL?

Hi all; Im using kafka 0.9.0.1 V with new consume APIs. I would like to know how new consumers get to know about the zookeeper's URL? Thanks -- -Ratha http://vvratha.blogspot.com/

How to publish/consume java bean objects to Kafka 2.11 version?

Hi all; Im a newbie to kafka. Im trying to publish my java object to kafka topic an try to consume. I see there are some API changes in the latest version of the kafka. can anybody point some samples for how to publish and consume java objects? I have written my own data serializer, but could not

Re: How to publish/consume java bean objects to Kafka 2.11 version?

have to maintain you > own de(serialiser). > > On Tue, Mar 22, 2016 at 3:38 AM Ratha v <vijayara...@gmail.com> wrote: > > > Hi all; > > Im a newbie to kafka. Im trying to publish my java object to kafka topic > an > > try to consume. > > I see

Re: kafka.consumer.ConsumerTimeoutException

Forgot to mention, I'm using kafka 2.11 version On 24 March 2016 at 16:55, Ratha v <vijayara...@gmail.com> wrote: > Hi all; > > I'm new to kafka and wrote a simple multithreaded kafka consumer. when try > to consume the messages,It continuously throwing timeoutexception..H

kafka.consumer.ConsumerTimeoutException

Hi all; I'm new to kafka and wrote a simple multithreaded kafka consumer. when try to consume the messages,It continuously throwing timeoutexception..How can i get rid of this? I have multiple topics. *Executor* public class MessageListener { private Properties properties; private

ProducerConfig - The configuration serializer.class = org.apache.kafka.common.serialization.StringSerializer was supplied but isn't a known config

Hi all; I'm getting above exception when i try to produce simple string messages using kafka 0.9.01 version. I could not produce messages now. The warning i get is; buffer.memory = 33554432 timeout.ms = 3 key.serializer = class org.apache.kafka.common.serialization.StringSerializer

If my producer producing, then why the consumer couldn't consume? it stuck @ poll()

Hi all; Im publishing to the remote kafka server and try to consume messages from that remote server. (Kafka v 0.90.1) Publishing works fine but nor the consuming. *Publisher* package org.test; *import java.io.IOException;* *import java.util.Properties;* *import

Re: ProducerConfig - The configuration serializer.class = org.apache.kafka.common.serialization.StringSerializer was supplied but isn't a known config

Sorry my bad..my kafka docker instance was down.. thanks On 4 April 2016 at 10:09, Ratha v <vijayara...@gmail.com> wrote: > > Hi all; > I'm getting above exception when i try to produce simple string messages > using kafka 0.9.01 version. I could not produce messages now. >

Does kafka version 0.9.0x use zookeeper?

I'm not seeing such parameter as an input for consumer. Does version 0.9.x use zookeeper? -- -Ratha http://vvratha.blogspot.com/

Re: New consumer API waits indefinitely

I too face same issue:( On 4 April 2016 at 15:51, Lohith Samaga M wrote: > HI, > Good morning. > > I am new to Kafka. So, please bear with me. > I am using the new Producer and Consumer API with Kafka > 0.9.0.1 running

Re: Does kafka version 0.9.0x use zookeeper?

Thanks Gerad On 4 April 2016 at 15:49, Gerard Klijs <gerard.kl...@dizzit.com> wrote: > Yes, but only via the broker you connect to. > > On Mon, Apr 4, 2016, 07:10 Ratha v <vijayara...@gmail.com> wrote: > > > I'm not seeing such parameter as an input for consumer.

Re: If my producer producing, then why the consumer couldn't consume? it stuck @ poll()

using kafka tool (UI tool to view topics, consumer..) But I could not figure out what blocks my consumer to poll() the message. Anyone can figure out my issue? On 4 April 2016 at 10:58, Ratha v <vijayara...@gmail.com> wrote: > > Hi all; > Im publishing to the remote kafka

Re: New consumer API waits indefinitely

sam...@mphasis.com> wrote: > Thanks for letting me know. > > Is there any work around? A fix? > > Which set of API is recommended for production use? > > Best regards / Mit freundlichen Grüßen / Sincères salutations > M. Lohith Samaga > > > > > -Original

Re: New consumer API waits indefinitely

This is the same logs i get with my local kafka server, that works fine.. On 5 April 2016 at 10:20, Ratha v <vijayara...@gmail.com> wrote: > HI Niko; > I face this issue with linux systems.. > I changed the logging level to debug and when I start and stop my consumer > (st

Re: New consumer API waits indefinitely

HI Niko; I face this issue with linux systems.. I changed the logging level to debug and when I start and stop my consumer (stopping the program) I get same exception. What is the cause here? [2016-04-05 00:01:08,784] DEBUG Connection with /192.xx.xx.248 disconnected

Is there any behavioural change to connect local server and remote server?

Hi all; Is there any different connection mechanism for local and remote (ec2 instance) server. Im asking because, my consumer is not working with remote server. -- -Ratha http://vvratha.blogspot.com/

Re: Custom serializer/deserializer for kafka 0.9.x version

are for older clients. > > Example code: > https://github.com/omkreddy/kafka-example > s/tree/master/consumer/src/main/java/kafka/examples/common/serialization > > On Tue, Mar 29, 2016 at 8:24 AM, Ratha v <vijayara...@gmail.com> wrote: > > > Hi all; > > > > I ha

Consumer thread is waiting forever, not returning any objects

Hi all; I publish a java object and try to consume it.. I have a poll method to consume objects, but it never returns any objects..My program runs forever.(?) *ConsumerThread* public void run() { try { consumer.subscribe(topics); Iterator> it =

java.lang.IllegalArgumentException: Heartbeat must be set lower than the session timeout

Hi all; When i try to create my consumer from the properties, i get above exception. Can anybody point me what property i set wrong? kafka 0.9.x version. *code* import org.apache.kafka.clients.consumer.KafkaConsumer; public ListenerThread(List topics, Properties properties) { this.consumer

Custom serializer/deserializer for kafka 0.9.x version

Hi all; I have written my custom serialiser/deserializer to publish/consume my java bean objects. However i get class nor found exception for "kafka.serializer" packages. Can someone point me which class i have to use to implement my custom serializer in kafka 0.9.x? import

Re: java.lang.IllegalArgumentException: Heartbeat must be set lower than the session timeout

I overcame this issue with the property "heartbeat.interval.ms".. Thanks. On 29 March 2016 at 14:54, Ratha v <vijayara...@gmail.com> wrote: > Hi all; > When i try to create my consumer from the properties, i get above > exception. > Can anybody point me what prope

Re: Consumer thread is waiting forever, not returning any objects

one > thing at the time. Use your Java producer, in conjunction with console > consumer. This will help to narrow down the problem (e.g., issue with > producer that may not be actually sending). Then hopefully if you receive > successfully then you know the problem is in the consumer and so

Re: Why my consumer does not print any messages? help..

messages before you start the consumer, not that consumers > by default start from the end of a queue. This is to prevent consumers from > being overwhelmed with old messages when they start. The auto.offset.reset > setting of the consumer can influence this behavior. > > On Tue, Ma

Re: Consumer thread is waiting forever, not returning any objects

t 4:19 PM, Ratha v <vijayara...@gmail.com> wrote: > > > 9092 > > > Are your brokers listening at this port? > > -- > Shrijeet > -- -Ratha http://vvratha.blogspot.com/

Re: Consumer thread is waiting forever, not returning any objects

console listener and with that I am going to assume > it works providing message is on the topic. And since nothing happened then > you probably have issue in producer. > > Could you post producer code. > > Oleg > > > On Mar 29, 2016, at 6:24 PM, Ratha v <vijayara..

Re: Consumer thread is waiting forever, not returning any objects

ssues.apache.org/jira/browse/KAFKA-1894. > Essentially poll(timeout) doesn't always respect the timeout :-| > > -- > Shrijeet > > On Tue, Mar 29, 2016 at 3:42 PM, Ratha v <vijayara...@gmail.com> wrote: > > > Hi oleg; > > > >1. The consumer shell (Th

Re: Consumer thread is waiting forever, not returning any objects

wrote: > The consumer script is using old consumer which connects via zookeeper, so > its not the same. The reason I asked because I recall poll(timeout) getting > stuck forever if it can't reach broker. > > -- > Shrijeet > > On Tue, Mar 29, 2016 at 4:51 PM, Ratha v <vijayara...@

Re: Consumer thread is waiting forever, not returning any objects

ent.send(request, now); iterator.remove(); requestsSent = true; } } } return requestsSent; On 30 March 2016 at 11:41, Ratha v <vijayara...@gmail.com> wrote: > And no error logs at kafka server..I exp

Re: Consumer thread is waiting forever, not returning any objects

And no error logs at kafka server..I expect any exception from my end, but I'm not getting any exception :( On 30 March 2016 at 11:39, Ratha v <vijayara...@gmail.com> wrote: > Hi Shrijeet ; > I checked the kafka communication layer,( tcp dumps) and I can see > some packets are re

org.apache.kafka.common.errors.ApiException: The session timeout is not within an acceptable range.

Hi all; I get above exception in my consumer with kafka 0.9.0.1..According to this thread[1] i do not see the "group.max.session.timeout.ms" property in the server.properties file. Any clue? [1]http://comments.gmane.org/gmane.comp.apache.kafka.user/12426 -- -Ratha http://vvratha.blogspot.com/

Re: org.apache.kafka.common.errors.ApiException: The session timeout is not within an acceptable range.

I added that property to the server.properties to a longer value than the session timeout value in the consumer.properties. But still i see same issue.. Any help? On 30 March 2016 at 13:28, Ratha v <vijayara...@gmail.com> wrote: > Hi all; > I get above exception in my consume

Re: org.apache.kafka.common.errors.ApiException: The session timeout is not within an acceptable range.

I overcame this issue by setting "session.timeout.ms" at consumer properties > group.min.session.timeout.ms @ server.properties Thanks. On 30 March 2016 at 13:41, Ratha v <vijayara...@gmail.com> wrote: > I added that property to the server.properties to a longer va

Re: Consumer thread is waiting forever, not returning any objects

Thanks all for the help.. On 30 March 2016 at 12:06, Ratha v <vijayara...@gmail.com> wrote: > Looks like this particular block runs forever.. > > @AbstractCoordinator.class > > /** > > * Block until the coordinator for this group is known. > > */

Iterator is in failed state -kafka 0.8x

Hi all; I have deployed my consumer as war file in wildfly 9.X. This simple consumer works fine in a standalone jar application. When deploying the war file, i get, *java.lang.IllegalStateException: Iterator is in failed state [1]* Exception.Why is that? *My consumer;* *public void run() {* *

JBOSS gives org.apache.kafka.common.KafkaException: auth.conf cannot be read

When I deploy war of my simple kafka project (which works fine as a jar ) in wildfly v 10 , i get some zookeeper connection exception[1].This occurs when kafka listener starts to connect with zookeeper [1] 15:21:58,531 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 82)

Re: How to define multiple serializers in kafka?

t you could configure it to use multiple non-compatible schema's > in one topic. Each object will be saved with a schema id, making it > possible to get back the original object. > > On Tue, May 3, 2016 at 1:52 AM Ratha v <vijayara...@gmail.com> wrote: > > > What is the best way fo

Re: How to define multiple serializers in kafka?

the > client with KafkaAvroDeserializerConfig.SPECIFIC_AVRO_READER_CONFIG="true" > to > get back the object, deserialized with the same version of the schema the > object was sent with. > > On Tue, May 3, 2016 at 12:06 PM Ratha v <vijayara...@gmail.com> wrote: > > > I plan t

How to define multiple serializers in kafka?

Hi all; Say, I publish and consume different type of java objects.For each I have to define own serializer implementations. How can we provide all implementations in the kafka consumer/producer properties file under the "serializer.class" property? -- -Ratha http://vvratha.blogspot.com/

Re: How to define multiple serializers in kafka?

fter you serialize, to be able to deserialize using the correct methods. > The custom serializer would always return an Object, which you could cast > when needed in the poll loop of the consumer. I think this is de > cleanest/best way, but maybe someone has a different idea? > > On

Why the consumer not listening to the first message?

Hi all; In my test program,I start listener. Then sending messages in a loop. If i send one message, it is not listning that message. If i send 2 messages, it listens one message.If I send 3 , it listens 2 messages..Why is that? *Producer* *KeyedMessage message = new

Re: Why the consumer not listening to the first message?

I fixed this by setting following property in my producer. request.required.acks=1 On 3 May 2016 at 09:50, Ratha v <vijayara...@gmail.com> wrote: > Hi all; > In my test program,I start listener. Then sending messages in a loop. > If i send one message, it is not listning tha

Re: How to deserialize the object without avro schema?

d a schema repository to lookup the matching schema > based on the id. > > Have a look at confluent.io. They offer a schema repo among other Kafka > related tools. > > Regards > > Jan > > > > Sent from my iPhone > > On 19 Apr 2016, at 08:02, Ratha v <vijayara...@

How to deserialize the object without avro schema?

Hi all; I try to publish/consume my java objects to kafka. I use Avro schema. My basic program works fine. In my program i use my schema in the producer (for encoding) and consumer (decoding). If i publish different objects to different topics( eg: 100 topics)at the receiver, i do not know,

Re: Need suggestion on sending XML files via kafka

This Sample program may help you? http://vvratha.blogspot.com.au/2016/07/sample-kafka-producer-and-consumer.html On 14 February 2017 at 06:36, Prashanth Venkatesan < prashanth.181...@gmail.com> wrote: > Hi Team, > > I just started using Kafka. I have a usecase to send XML file or Document >

kafka.common.ConsumerRebalanceFailedException: can't rebalance after 4

HI all; Im using 0.8 consumer with kafka 0.10.0.1. When i run my consumer app (within wildfly 10.0.x) Im getting follwoing exception and consumer is not listning any messages; I tried increasing *"rebalance.backoff.ms ", "zookeeper.session.timeout.ms

Re: How to merge two topics in kafka?

consume message from that new topic via second process. Note > you need to create all three topics in zookeeper first. > > On 10 Oct 2016 5:19 a.m., "Ratha v" <vijayara...@gmail.com> wrote: > > Hi all; > > I have two topics in the broker. Without con

{Kafka V 0.10.1}Producing message takes approximately 1 minute if any error occurs at async producer

Hi; Im using asynchronous producer to send messages to the broker. If there is no any error in my callback, i see it takes approximately 0.3 seconds to produce message. But when I get below error [1], I see it takes 60 seconds to produce message. But I do not see any message loss . All messages

Re: How can I delete a topic programatically?

Thanks..Which bash script I need to run? On 12 October 2016 at 15:20, Ali Akhtar <ali.rac...@gmail.com> wrote: > The last time I tried, I couldn't find a way to do it, other than to > trigger the bash script for topic deletion programatically. > > On Wed, Oct 12, 2016 a

Re: KafkaStream Merging two topics is not working fro custom datatypes

Serde, targetTopic); > > > > You need to override the default serdes whenever the data must be written > > with, well, non-default serdes. > > > > I'd recommend reading http://docs.confluent.io/ > current/streams/developer- > > guide.html#data-types-and-s

Re: KafkaStream Merging two topics is not working fro custom datatypes

Sorry my fault, In the kafkaConsumer I messed with 'value.deserializer' property.. Now things are working fine.. Thanks a lot. On 12 October 2016 at 14:10, Ratha v <vijayara...@gmail.com> wrote: > HI Michael; > Sorry , after setting "auto.offset.reset" to 'earliest' ,

Re: KafkaStream Merging two topics is not working fro custom datatypes

?:1.8.0_66] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_66] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_66] On 12 October 2016 at 13:19, Ratha v <vijayara...@gmail.com> wrote: > HI Michael; > > Really appreciate for the clear explanati

How can I delete a topic programatically?

Hi all; I have two topics(source and target). I do some processing on the message available in the source topic and i merge both topic. That is; builder.stream(sourceTopic).to(targetTopic) Once merged I no longer require the sourceTopic. I want to delete it. How can I do that programatically

Re: How can I delete a topic programatically?

is in good shape before and after deleting a topic before > working on next one. > > Regards, > > -- Jianbin > > > On Oct 11, 2016, at 9:26 PM, Ratha v <vijayara...@gmail.com> wrote: > > > > Thanks..Which bash script I need to run? > > > >> On 12 O

Re: kafka-topics.sh ---delete --topic ''x'' is not working for kafka V 0.10.1

I overcame the issue. Issue was I had "/chroot" at the end.That is how mentioned in the documentationkafka.apache.org/documentation.html#basic_ops_modify_topic. Thank you. On 14 October 2016 at 12:25, Ratha v <vijayara...@gmail.com> wrote: > HI all; > > I try to d

kafka-topics.sh ---delete --topic ''x'' is not working for kafka V 0.10.1

HI all; I try to delete a existing topic (I checked using kafka management console) using following command; #./kafka-topics.sh --zookeeper zookeeper.xx.com:2181/chroot --delete --topic testTopic But it says topic not available in zookeeper.[1] I create those topics at runtime . (I think it is

Re: How can I delete a topic programatically?

$.main(TopicCommand.scala:69)* * at kafka.admin.TopicCommand.main(TopicCommand.scala)* bin/kafka-topics.sh --zookeeper zk_host:port/chroot --delete --topic my_topic_name On 12 October 2016 at 17:02, Ratha v <vijayara...@gmail.com> wrote: > Thank you.. > > On 12 October 2016

How to merge two topics in kafka?

Hi all; I have two topics in the broker. Without consuming from one topic, I want to merge both topics, and will consume messages from the second topic. It is because, I have two processes, one process, pushes messages to topic A. And the second process once finished processing, it wants to

Re: How to merge two topics in kafka?

10 October 2016 at 13:10, Ratha v <vijayara...@gmail.com> wrote: > Thank you.I'll try the solution. > But in the highlevel consumer API, topics will be created automatically? , > We are not using zookeeper? > > On 10 October 2016 at 12:34, Sachin Mittal <sjmit...@gmail.c

Re: How to merge two topics in kafka?

ght be the cause? On 10 October 2016 at 17:54, Sachin Mittal <sjmit...@gmail.com> wrote: > Check this example > https://github.com/apache/kafka/blob/trunk/streams/ > examples/src/main/java/org/apache/kafka/streams/examples/ > pipe/PipeDemo.java > > On Mon, Oct 10, 2016 at

KafkaStream Merging two topics is not working fro custom datatypes

Hi all; I have custom datatype defined (a pojo class). I copy messages from one topic to another topic. I do not see any messages in my target topic. This works fro string messages, but not for my custom message. Waht might be the cause? I followed this sample [1] [1]

Re: How to merge two topics in kafka?

I checked my targetTopic for available messages, but it says '0' . What might cause issue here to merge two topics with custom type messages ? On 11 October 2016 at 14:44, Ratha v <vijayara...@gmail.com> wrote: > Thanks, this demo works perfectly for string messages. > >

Re: KafkaStream Merging two topics is not working fro custom datatypes

Topics);* * kafkaPayloadStream.to(targetTopic);* * streams = new KafkaStreams(builder, properties);* * streams.start();* Why do I see classcast exception when consuming the message? On 11 October 2016 at 15:19, Ratha v <vijayara...@gmail.com> wrote: > Hi all; > I have custom datatype defined (