Re: best python library to use?

Hi I heard that Pykakfa can create a balanced consumer. And there should be no other big difference. Best Regards > 在 2016年1月9日，08:58，Dana Powers 写道： > > Hi Doug, > > The differences are fairly subtle. kafka-python is a community-backed > project that aims to be consistent w/ the official

Memory records is not writable in MirrorMaker

Hi, I found a problem that says Memory records is not writable in MirrorMaker when producer sending out messages: [2016-01-08 18:52:30,729] FATAL [mirrormaker-thread-32] Mirror maker thread failure due to (kafka.tools.MirrorMaker$MirrorMakerThread) java.lang.IllegalStateException: Memory records

Re: best python library to use?

Hi Doug, The differences are fairly subtle. kafka-python is a community-backed project that aims to be consistent w/ the official java client; pykafka is sponsored by parse.ly and aims to provide a pythonic interface. whichever you go with, I would love to hear your specific feedback on kafka-pyth

best python library to use?

we're using kafka-python, weighing pykafka, and wondering if there's another that is bettor to use. does confluent endorse or recommend a particular python package (psorry for the alliteration)? doug

Re: kafka-producer-perf-test.sh - 0.8.2.1

It is single threaded in the sense that you can not request that multiple threads be used to call producer.send(). However, the producer has its own internal thread for doing network IO. When you have such a simple producer, depending on the size of messages you can saturate a 1Gbps link with a sin

Re: Kafka cluster

Hi Liquan: The application server is using both producer and consumer, the bootstrap.servers are configured with the 8 kafka servers. Thanks, Fang On Fri, Jan 8, 2016 at 3:04 PM, Liquan Pei wrote: > Hi Fang, > > Are you using the producer or consumer in your application server? You need > to c

Re: Bizarre crash when creating a consumer

Just for posterity: what happened here was an issue with the hostname ERROR 2016-01-08 22:02:09,675 [main] [none] c.k.messaging.kafka.ConsumerGroup: ip-10-100-102-52: ip-10-100-102-52: unknown error ! java.net.UnknownHostException: ip-10-100-102-52: unknown error ! at java.net.Inet4AddressImpl.lo

Re: Kafka cluster

Hi Fang, Are you using the producer or consumer in your application server? You need to config the bootstrap.servers so that it includes the servers that are up. Thanks, Liquan On Fri, Jan 8, 2016 at 2:53 PM, Fang Wong wrote: > Hi, > > I have a kafka cluster of 8 servers, but only 4 of them a

Kafka cluster

Hi, I have a kafka cluster of 8 servers, but only 4 of them are up, the other 4 are down. when I started my app server, some time it still tried to connect to the kafka servers which are down, since the kafka server is down, my app server couldn't start. Not sure why it doesn't just try to connect

Re: Consumer group disappears and consumers loops

We have an active KIP which aims to give better options for avoiding this problem: https://cwiki.apache.org/confluence/display/KAFKA/KIP-41%3A+KafkaConsumer+Max+Records. If you have any feedback, the discussion is on the dev list. The pause/resume trick should have also worked. The consumer should

Re: Problems using ssl

For reference, here's the PR with the fixes: https://github.com/apache/kafka/pull/745 Ismael On Thu, Jan 7, 2016 at 7:51 PM, Ismael Juma wrote: > Hi Mats, > > There's a bug in the documentation where quotes are used in the properties > file even though they should not. I'll submit a PR to fix

Re: kafka-producer-perf-test.sh - 0.8.2.1

Thanks Ewen. Do you know if kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance is single threaded? Or is there any way to specify number of threads? On Fri, Jan 8, 2016 at 1:24 PM, Ewen Cheslack-Postava wrote: > Ah, sorry, I missed the version number in your title. I think thi

Re: kafka-producer-perf-test.sh - 0.8.2.1

Ah, sorry, I missed the version number in your title. I think this tool saw some rearrangement in 0.9.0 and I was looking at the latest version. Unfortunately it doesn't look like the old kafka.tools.ProducerPerformance that is used in kafka-producer-perf-test.sh in 0.8.2.1 supports passing in addi

RE: committing particular offset with high level consumer?

Thanks, Ewen. I ended up migrating to 0.9.0.0 and using the new consumer's KafkaConsumer.commitSync(offsets) function, which looks like it's working perfectly. Nice feature. -J -Original Message- From: Ewen Cheslack-Postava [mailto:e...@confluent.io] Sent: 07 January 2016 20:15 To: use

Re: Leader not available (initially)

Hi yes, was actually just writing a reply to that :) Found the rest of the context: "Alternatively, instead of manually creating topics you can also configure your brokers to auto-create topics when a non-existent topic is published to.” I tried this: commented out my code that creates the topi

Re: Leader not available (initially)

I think that the attempt to write a message to a non-existent topic creates that topic (when auto.create is set to true). If it's set to false you get back error. Have you tried that? Marko Bonaći Monitoring | Alerting | Anomaly Detection | Centralized Log Management Solr & Elasticsearch Support S

Re: Upgraded from 0.8.2 to 0.9.0 and FetchFollower request time has increased

Great! On 8 Jan 2016 13:53, "Stephen Powis" wrote: > Hey Ismael, > > It did, just took a few hours :) It seems like its just hit where it was > before the upgrade. > > > > > Thanks! > Stephen > > On Fri, Jan 8, 2016 at 8:03 PM, Ismael Juma wrote: > > > Hi Stephen, > > > > Have things settled d

Re: Determine the number of consumers

Right, and I presume AdminUtils didn’t catch up with that change yet or something. Cheers Cos On Friday, 8 January 2016 at 17:09, Marko Bonaći wrote: > Hello again Cosmin :) > I think this is because offsets are kept in a special _consumer_offsets > topic, as opposed to ZK previously. > Take

Re: kafka-producer-perf-test.sh - 0.8.2.1

Ewen, I tried that before like this: ./kafka-producer-perf-test.sh --broker-list test:9092 --topics test8-3 --messages 200 --new-producer --message-size 200 --show-detailed-stats max.request.size=1000 and it does not work. It comletly ignore this option. And --producer-props is not

Re: Determine the number of consumers

Hello again Cosmin :) I think this is because offsets are kept in a special _consumer_offsets topic, as opposed to ZK previously. Take a look here: http://search-hadoop.com/m/uyzND1T1i3BNkRFM1&subj=Re+Kafka+0+8+2+1+how+to+read+from+__consumer_offsets+topic+ Marko Bonaći Monitoring | Alerting | Ano

RE: Kafka-Rest question

That cool, I will check that out but it makes sense -Original Message- From: Ewen Cheslack-Postava [mailto:e...@confluent.io] Sent: Thursday, January 07, 2016 8:14 PM To: users@kafka.apache.org Subject: Re: Kafka-Rest question The REST proxy doesn't currently handle topic creation, so yo

Re: kafka-producer-perf-test.sh - 0.8.2.1

Andrew, kafka-producer-perf-test.sh is just a wrapper around orga.apache.kafka.clients.tools.ProducerPerformance and all command line options should be forwarded. Can you just pass a --producer-props to set max.request.size to a larger value? -Ewen On Fri, Jan 8, 2016 at 7:51 AM, Andrej Vladimir

Re: Leader not available (initially)

Hi Dana I am actually creating the topic manually from our code. This is mainly because I couldn’t tell from the documentation when exactly is the topic auto-created - maybe you can shed some light on that. The docs (http://kafka.apache.org/documentation.html) only mention auto.create.topics.en

Re: Leader not available (initially)

That is expected when a topic is first created. Once the topic and all partitions are initialized, the leader(s) will be available and you can produce messages. Note that if you disable topic auto creation, you should get an UnknownTopicOrPartitionError, and you would need to create the topic manu

kafka-producer-perf-test.sh - 0.8.2.1

Hi! I am testing Kafka's performance with large messages and would like to specify maximum request size when I run kafka-producer-perf-test.sh: ./kafka-producer-perf-test.sh --broker-list "test1:9092" --topics test8-3 --messages 100 --new-producer --msage-size 110 --show-detailed-stats I

Re: MirrorMaker --whitelist option in 0.9.0.0

Hmm so after I sent my email to the list, I realized that I forgot to include.. I started with --whitelist ".*" but after that didn't work swapped to "*" after running across the docs here: https://kafka.apache.org/090/ops.html#basic_ops_mirror_maker Neither were working for me :/ On Fri, Jan 8,

Determine the number of consumers

Hi, Is there a reliable way to determine the number of consumers for a topic (or at least the number of consumer groups). I’m using Kafka 0.9.0.0 and both the ones below return empty sets zkUtils.getAllConsumerGroupsForTopic(topic); OR zkUtils.getConsumerGroups(); Thanks in advance Cos

Re: MirrorMaker --whitelist option in 0.9.0.0

Mirror maker white list uses Java pattern https://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html. In you case the whitelist is ".*" On Fri, 8 Jan 2016 at 20:54 Stephen Powis wrote: > Hey! > > I'm having a little trouble with mirror maker, if I supply --whitelist "*" > it will not

Re: Upgraded from 0.8.2 to 0.9.0 and FetchFollower request time has increased

Hey Ismael, It did, just took a few hours :) It seems like its just hit where it was before the upgrade. Thanks! Stephen On Fri, Jan 8, 2016 at 8:03 PM, Ismael Juma wrote: > Hi Stephen, > > Have things settled down since? > > Thanks, > Ismael > > On Fri, Jan 8, 2016 at 7:39 AM, Stephen Po

Re: Upgraded from 0.8.2 to 0.9.0 and FetchFollower request time has increased

Hi Stephen, Have things settled down since? Thanks, Ismael On Fri, Jan 8, 2016 at 7:39 AM, Stephen Powis wrote: > Hey! > > So I upgraded our production kafka cluster from 0.8.2 to 0.9.0 this > morning, and it seems like everything went smoothly. > > As the last step when I changed the inter.br

MirrorMaker --whitelist option in 0.9.0.0

Hey! I'm having a little trouble with mirror maker, if I supply --whitelist "*" it will not consume any topics. If do --whitelist "TopicName" it works, for the specified topic. Any idea why my wildcard does not work? Thanks Stephen

Re: Error while fetching metadata partition 0 leader: none replicas: isr: isUnderReplicated:

I am not on aws i am on digital ocean and my advertiser.host.name is set to the ip address. From the below docs I dont have that issue When a broker starts up, it registers its ip/port in ZK. You need to make sure the registered ip is consistent with what's listed in metadata.broker.list in th

Leader not available (initially)

Hi guys I’ve been seeing this in our logs a few times, when initially bootstrapping the system. WARN 2016-01-08 10:59:24,232 [pool-3-thread-11] [none] o.a.kafka.clients.NetworkClient: Error while fetching metadata with correlation id 3 : {mytopic=LEADER_NOT_AVAILABLE} It doesn’t seem to br

Re: Error while fetching metadata partition 0 leader: none replicas: isr: isUnderReplicated:

Hi David, Do you have advertised.host.name set in kafka properties? On AWS (might be the same for DO) we had to set this to the public DNS of the box (you might be able to get away with setting it to the public IP) Thanks, Ben On Fri, Jan 8, 2016 at 10:52 AM, David Montgomery wrote: > Hi,

Re: Bizarre crash when creating a consumer

Hi Marko, this seems to have solved this. Dealing with another issue now, which I’ll report separately. Thank you for your help! Cheers Cos On Friday, 8 January 2016 at 09:27, Cosmin Marginean wrote: > Hi Marko, I will migrate the code and also change the timeout. thanks for > your suggestio

Error while fetching metadata partition 0 leader: none replicas: isr: isUnderReplicated:

Hi, I am on Digi Ocean FYI and not dealing with the issue of private and public IP address. I killed all kafka servers and rebuilt but have the same zk cluster. One would think on kafka 0.8.2 that if I rebuild kafka, register with ZK all would be right with the world but guess not. I have new b

Re: Bizarre crash when creating a consumer

Hi Marko, I will migrate the code and also change the timeout. thanks for your suggestions. Will post a status once I’ve tested. Cheers Cos On Thursday, 7 January 2016 at 22:59, Marko Bonaći wrote: > Actually, why don't you use the same code as outlined here (that includes > timeout in props)