Re: Announcing Confluent
Best of Luck..keep rocking... On Fri, Nov 7, 2014 at 1:30 AM, Joe Brown brownjoe...@gmail.com wrote: Best of luck!!! J On 6 Nov 2014, at 18:28, Jay Kreps jay.kr...@gmail.com wrote: Hey all, I’m happy to announce that Jun Rao, Neha Narkhede and I are creating a company around Kafka called Confluent. We are planning on productizing the kind of Kafka-based real-time data platform we built out at LinkedIn. We are doing this because we think this is a really powerful idea and we felt there was a lot to do to make this idea really take root. We wanted to make that our full time mission and focus. There is a blog post that goes into a little more depth here: http://blog.confluent.io/ LinkedIn will remain a heavy Kafka user and contributor. Combined with our additional resources from the funding of the company this should be a really good thing for the Kafka development effort. Especially when combined with the increasing contributions from the rest of the development community. This is great news, as there is a lot of work to do. We'll need to really focus on scaling this distributed development in a healthy way. One thing I do want to emphasize is that the addition of a company in the Kafka ecosystem won’t mean meddling with open source. Kafka will remain 100% open source and community focused, as of course is true of any Apache project. I have been doing open source for a long time and strongly believe it is the right model for infrastructure software development. Confluent is just getting off the ground now. We left LinkedIn, raised some money, and we have an office (but no furniture yet!). None the less, f you are interested in finding out more about the company and either getting help with your Kafka usage or joining us to help build all this, by all means reach out to us, we’d love to talk. Wish us luck! -Jay -- Thanks, Pankaj Ojha
Read a specific number of messages using kafka
Hi, My requirement is to read a specific number of messages from kafka topic which contains data in json format and after reading number of messges, i need to write that in a file and then stop. How can I count number of messages read by my consumer code(either simpleconsumer or high level) ? Please help. -- Thanks, Pankaj Ojha
Re: Read a specific number of messages using kafka
Thank You. I will try this out.
On Thu, Sep 25, 2014 at 10:01 PM, Gwen Shapira gshap...@cloudera.com
wrote:
Using high level consumer and assuming you already created an iterator:
while (msgCount maxMessages it.hasNext()) {
bytes = it.next().message();
eventList.add(bytes);
}
(See a complete example here:
https://github.com/apache/flume/blob/trunk/flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSource.java
)
Gwen
On Thu, Sep 25, 2014 at 9:15 AM, pankaj ojha pankajojh...@gmail.com
wrote:
Hi,
My requirement is to read a specific number of messages from kafka topic
which contains data in json format and after reading number of messges, i
need to write that in a file and then stop. How can I count number of
messages read by my consumer code(either simpleconsumer or high level) ?
Please help.
--
Thanks,
Pankaj Ojha
--
Thanks,
Pankaj Ojha
Re: Which is better?
Thank You Philip for helping me out in this. Thanks, Pankaj On Wed, Aug 20, 2014 at 9:54 PM, Philip O'Toole philip.oto...@yahoo.com.invalid wrote: If you have studied the docs yet, you should, as this is a broad question which needs background to understand the answer. But in summary, the high-level Consumer does more for you, and importantly, provides balancing between Consumers. The SimpleConsumer does less for you, but gives you more control. A common pattern when first working with Kafka is to develop your systems using the HighLevel Consumer and then, once you understand Kafka better, move to a SimpleConsumer-based design for more control (if you determine you need it -- the HighLevel Consumer is enough for many, if not most, people). Philip - http://www.philipotoole.com On Wednesday, August 20, 2014 8:35 AM, pankaj ojha pankajojh...@gmail.com wrote: Hi All, My requirement is to consume data from kafka using consumer. Which should be better to use? SimpleConsumer or HighLevel Consumer? Thanks, Pankaj Ojha
Which is better?
Hi All, My requirement is to consume data from kafka using consumer. Which should be better to use? SimpleConsumer or HighLevel Consumer? Thanks, Pankaj Ojha
Re: Issue in using kafka SimpleConsumer code
Hi Team, Can you please provide any information on my above problem? Thanks Regards, Pankaj Ojha On Wed, Aug 6, 2014 at 8:04 PM, pankaj ojha pankajojh...@gmail.com wrote: Hi Team, I have a requirement of reading real time data using kafka and write to cassandra. For this I am using SimpleConsumer to read data from Kafka topics and writing into Cassandra. I am maintaining offsets of topics in my log files. The issue is that after few days like 3-4 days my cosumer code does not read data from kafka topics and produce below log output : 20:01:17,068 INFO NposKafkaConsumer:48 - Taking partition from application properties 20:01:17,482 DEBUG BlockingChannel:52 - Created socket with SO_TIMEOUT = 10 (requested 10), SO_RCVBUF = 65536 (requested 65536), SO_SNDBUF = 64512 (requested -1). 20:01:17,545 DEBUG SimpleConsumer:52 - Disconnecting from IP address:9092 20:01:17,578 DEBUG NposKafkaConsumer:113 - NposKafkaConsumer.run() method Inside while loop :: Value of max_reads::1 20:01:17,662 DEBUG BlockingChannel:52 - Created socket with SO_TIMEOUT = 10 (requested 10), SO_RCVBUF = 65536 (requested 65536), SO_SNDBUF = 64512 (requested -1). 20:01:17,804 DEBUG NposKafkaConsumer:193 - numRead::0 Sleeping 20:01:18,804 DEBUG NposKafkaConsumer:113 - NposKafkaConsumer.run() method Inside while loop :: Value of max_reads::1 20:01:18,826 DEBUG NposKafkaConsumer:193 - numRead::0 Sleeping 20:01:19,827 DEBUG NposKafkaConsumer:113 - NposKafkaConsumer.run() method Inside while loop :: Value of max_reads::1 20:01:19,852 DEBUG NposKafkaConsumer:193 - numRead::0 Sleeping NposKafkaConsumer is my main SimpleCosumer class. But, when I restart the kafka process by incrementing the offset by one then again my code starts running fine for next few days. Can you please help me how I can solve this and where I am going wrong ? Thanks Regards, Pankaj Ojha
Re: Issue in using kafka SimpleConsumer code
Hi Harsha, Thank you for replying. There is no error in the kafka logs. The output I had pasted in my previous mail is from kafka logs only. But when I restart the kafka process after incrementing the offset by 1, then it runs fine. Is there any known issue with the SimpleConsumer code? Thanks, Pankaj On Thu, Aug 7, 2014 at 7:17 PM, Harsha ka...@harsha.io wrote: Hi Pankaj, Do you notice any errors in kafka logs when your consumer stops reading data. Attaching those logs would be helpful in finding the issue. -Harsha On Thu, Aug 7, 2014, at 06:12 AM, pankaj ojha wrote: Hi Team, Can you please provide any information on my above problem? Thanks Regards, Pankaj Ojha On Wed, Aug 6, 2014 at 8:04 PM, pankaj ojha pankajojh...@gmail.com wrote: Hi Team, I have a requirement of reading real time data using kafka and write to cassandra. For this I am using SimpleConsumer to read data from Kafka topics and writing into Cassandra. I am maintaining offsets of topics in my log files. The issue is that after few days like 3-4 days my cosumer code does not read data from kafka topics and produce below log output : 20:01:17,068 INFO NposKafkaConsumer:48 - Taking partition from application properties 20:01:17,482 DEBUG BlockingChannel:52 - Created socket with SO_TIMEOUT = 10 (requested 10), SO_RCVBUF = 65536 (requested 65536), SO_SNDBUF = 64512 (requested -1). 20:01:17,545 DEBUG SimpleConsumer:52 - Disconnecting from IP address:9092 20:01:17,578 DEBUG NposKafkaConsumer:113 - NposKafkaConsumer.run() method Inside while loop :: Value of max_reads::1 20:01:17,662 DEBUG BlockingChannel:52 - Created socket with SO_TIMEOUT = 10 (requested 10), SO_RCVBUF = 65536 (requested 65536), SO_SNDBUF = 64512 (requested -1). 20:01:17,804 DEBUG NposKafkaConsumer:193 - numRead::0 Sleeping 20:01:18,804 DEBUG NposKafkaConsumer:113 - NposKafkaConsumer.run() method Inside while loop :: Value of max_reads::1 20:01:18,826 DEBUG NposKafkaConsumer:193 - numRead::0 Sleeping 20:01:19,827 DEBUG NposKafkaConsumer:113 - NposKafkaConsumer.run() method Inside while loop :: Value of max_reads::1 20:01:19,852 DEBUG NposKafkaConsumer:193 - numRead::0 Sleeping NposKafkaConsumer is my main SimpleCosumer class. But, when I restart the kafka process by incrementing the offset by one then again my code starts running fine for next few days. Can you please help me how I can solve this and where I am going wrong ? Thanks Regards, Pankaj Ojha
Issue in using kafka SimpleConsumer code
Hi Team, I have a requirement of reading real time data using kafka and write to cassandra. For this I am using SimpleConsumer to read data from Kafka topics and writing into Cassandra. I am maintaining offsets of topics in my log files. The issue is that after few days like 3-4 days my cosumer code does not read data from kafka topics and produce below log output : 20:01:17,068 INFO NposKafkaConsumer:48 - Taking partition from application properties 20:01:17,482 DEBUG BlockingChannel:52 - Created socket with SO_TIMEOUT = 10 (requested 10), SO_RCVBUF = 65536 (requested 65536), SO_SNDBUF = 64512 (requested -1). 20:01:17,545 DEBUG SimpleConsumer:52 - Disconnecting from IP address:9092 20:01:17,578 DEBUG NposKafkaConsumer:113 - NposKafkaConsumer.run() method Inside while loop :: Value of max_reads::1 20:01:17,662 DEBUG BlockingChannel:52 - Created socket with SO_TIMEOUT = 10 (requested 10), SO_RCVBUF = 65536 (requested 65536), SO_SNDBUF = 64512 (requested -1). 20:01:17,804 DEBUG NposKafkaConsumer:193 - numRead::0 Sleeping 20:01:18,804 DEBUG NposKafkaConsumer:113 - NposKafkaConsumer.run() method Inside while loop :: Value of max_reads::1 20:01:18,826 DEBUG NposKafkaConsumer:193 - numRead::0 Sleeping 20:01:19,827 DEBUG NposKafkaConsumer:113 - NposKafkaConsumer.run() method Inside while loop :: Value of max_reads::1 20:01:19,852 DEBUG NposKafkaConsumer:193 - numRead::0 Sleeping NposKafkaConsumer is my main SimpleCosumer class. But, when I restart the kafka process by incrementing the offset by one then again my code starts running fine for next few days. Can you please help me how I can solve this and where I am going wrong ? Thanks Regards, Pankaj Ojha
