oom of kafka

Hi, I am using kafka_2.11-0.9.0.1, there are multiple of warnings occur in kafkaServer.out: java.lang.OutOfMemoryError: Java heap space. - the jstat output like this: # jstat -gc 28591 S0CS1CS0US1U EC EUOC OU MC MUCCSC CCSU YGC YGCT

Re: New Kafka Connector

Chris, I think you meant to link to https://github.com/wepay/kafka-connect-bigquery :) - Samuel On Mon, Aug 22, 2016 at 4:24 PM, Chris Egerton wrote: > Hi there, > > We've recently open-sourced a BigQuery sink connector and would like to > request that it be added to

New Kafka Connector

Hi there, We've recently open-sourced a BigQuery sink connector and would like to request that it be added to the Kafka Connector Hub ( https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Connector+Hub). The project can be found at https://github.com/wepay/kafka-connect-biquery, and the

RE: 0.10 Metrics Reporter Suggestions

Take a look at jmxtrans: https://github.com/jmxtrans. There are a number of options, including some ones you can ebmed into your java application. -Dave -Original Message- From: Lawrence Weikum [mailto:lwei...@pandora.com] Sent: Monday, August 22, 2016 2:32 PM To:

0.10 Metrics Reporter Suggestions

Hello, We’re looking into the functionality of Metrics Reporters for producers and consumers in Kafka 0.10. Are there any projects that can be recommended that seem promising; specifically involving sending metrics to either StatsD or Graphite? As always, thank you for your help! Lawrence

Re: Prevent broker from leading topic partitions

Hi, Jason This scenario is supported. Just set config option auto.leader.rebalance.enable=false And use tool kafka-preferred-replica-election.sh If you want to move leader from one host to another use tool kafka-reassign-partitions.sh with same replica list but other order 22.08.2016, 20:36,

Re: Kafka Connect - how to deal with multiple formats in Kafka?

Thank you, Glad to see it is addressed. On Mon, Aug 22, 2016 at 8:36 PM, Dustin Cote wrote: > Hi Michael, > > You'd probably be interested in the discussion for this KIP: > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 75+-+Add+per-connector+Converters > > For

Re: Pausing a partition in Kafka Consumer

Hi Kamal, Thanks for reporting this. I think we can carry the paused partitions info across rebalances so that if the partitions are assigned to the consumers again they are still automatically paused (currently we reset such information after rebalance). Guozhang On Sun, Aug 21, 2016 at

Kafka Connect - how to deal with multiple formats in Kafka?

I am looking into Kafka Connect and Confluent HDFSSinkConnector. The goal is to save data from various topics to HDFS. We have at least two different formats of the data in Kafka - raw data (JSON) - that we want to save as SequenceFile and normalized data (Protobuf) that we want to save as

Re: KIP-33 Opt out from Time Based indexing

Can you describe the behavior you saw that you didn't like? -Jay On Mon, Aug 22, 2016 at 12:24 AM, Jan Filipiak wrote: > Hello everyone, > > I stumbled across KIP-33 and the time based index, while briefly checking > the wiki and commits, I fail to find a way to opt

RE: Offsets getting lost if no messages sent for a long time

Hi, Can anybody provide any guidance on the following: 1. Given a limited set of groups and consumers, will increasing 'offsets.retention.minutes' to a high value (say 30 days) cause the __consumer_offsets topic to bloat unnecessarily or will compaction ensure that the entries for each key