Re: SSL between Kafka and Spark Streaming API

SSL is supported for new producer and consumer api and old api (simple consumer and high-level consumer) is not supported. I think spark uses simple consumer? if so its not supported. Thanks, Harsha On August 28, 2015 at 11:00:30 AM, Cassa L (lcas...@gmail.com) wrote: Hi, I was going through

Re: SSL between Kafka and Spark Streaming API

        version0.8.2.1/version I believe this DirectStream API uses SimpleConsumer API. Can someone from Spark community confirm too? Thanks, LCassa. On Fri, Aug 28, 2015 at 11:12 AM, Sriharsha Chintalapani ka...@harsha.io wrote: SSL is supported for new producer and consumer api and old api (simple

Re: Issue when enabling SSL on broker

the keystores. Thanks, Harsha On August 25, 2015 at 8:56:24 PM, Sriharsha Chintalapani (ka...@harsha.io) wrote: Hi Xiang,          Did you try following the instructions here  https://cwiki.apache.org/confluence/display/KAFKA/Deploying+SSL+for+Kafka . Whats the output of openssl s_client and which

Re: Kafka Cluster behind Proxy

Ankit,        Did you try using advertised.host.name and advertised.port by specifying proxy host port details. --  Harsha Sent with Airmail On August 17, 2015 at 12:28:19 AM, Ankit Jain (ankitjainc...@gmail.com) wrote: Hi All, We want to deploy the Kafka cluster behind the proxy. We are

Re: Controlled Shutdown Tool?

have to tell the broker to shutdown nicely?  All I really want to do is tell the controller to move leadership to other replicas, so I can shutdown the broker without clients getting all confused. On Jul 27, 2015, at 14:48, Sriharsha Chintalapani ka...@harsha.io wrote: You can set

Re: Controlled Shutdown Tool?

You can set controlled.shutdown.enable to true in kafka’s server.properties  , this is enabled by default in 0.8.2 on wards and also you can set max retries using controlled.shutdown.max.retries defaults to 3 . Thanks, Harsha On July 27, 2015 at 11:42:32 AM, Andrew Otto (ao...@wikimedia.org)

Re: Implementing a custom partitioner

Hi,      Are you using the latest trunk for Producer API?.  Did you implement the interface here  https://cwiki.apache.org/confluence/display/KAFKA/KIP-+22+-+Expose+a+Partitioner+interface+in+the+new+producer --  Harsha On July 21, 2015 at 2:27:05 PM, JIEFU GONG (jg...@berkeley.edu) wrote: Hi

Re: Implementing a custom partitioner

= config.getConfiguredInstance(ProducerConfig.PARTITIONER_CLASS_CONFIG, Partitioner.class); does this mean I cannot use my own partitioner in v 0.8.2? On Tue, Jul 21, 2015 at 2:48 PM, Sriharsha Chintalapani ka...@harsha.io wrote: Hi,      Are you using the latest trunk for Producer API?.  Did you

Re: Dropping support for Scala 2.9.x

I am +1 on dropping 2.9.x support. Thanks,  Harsha On July 8, 2015 at 7:08:12 AM, Ismael Juma (mli...@juma.me.uk) wrote: Hi, The responses in this thread were positive, but there weren't many. A few months passed and Sriharsha encouraged me to reopen the thread given that the 2.9 build

Re: latest kafka consumer api maven location

Sushant,      You are using kafka clients new consumer api. It looks like you want to use high-level consumer api?. If so you need use following kafka core lib as the dependency dependency groupIdorg.apache.kafka/groupId artifactIdkafka_2.10/artifactId

Re: Log file of server start up error

       It looks like you have another process running for kafka broker. Stop that broker and start a new one. --  Harsha On May 24, 2015 at 11:05:27 PM, Sanjay Mistry (mistrysanja...@gmail.com) wrote: Hello, When run this command bin/kafka-console-producer.sh --broker-list localhost:9092

Re: Delete topic pending

May be thats a bug. We should probably alert the user saying the topic doesn’t exist rather than go and add it to the zookeeper for deletion trigger. Also to delete any topic you need to set “delete.topic.enable” to true --  Harsha On May 18, 2015 at 10:16:46 AM, Dillian Murphey

RE: Support https or ssl

is that we write an agent that sends message via https and then may be a servlet using Kafka produce class to authenticate with kafka server on ssl port and so on When will the full implementation be available for GA? Thanks again. Jamie -Original Message- From: Sriharsha

Re: Support https or ssl

Hi Jamie,         I am currently working on providing ssl support for kafka. Here are the iras https://issues.apache.org/jira/browse/KAFKA-1690 and  https://issues.apache.org/jira/browse/KAFKA-1684 . If you are using REST api to front kafka producer than you can probably make that http server to

Re: [KIP-DISCUSSION] KIP-22 Expose a Partitioner interface in the new producer

in the Partitioner api). Cheers, -Jay On Thu, Apr 23, 2015 at 6:55 AM, Sriharsha Chintalapani ka...@harsha.io wrote: Hi, Here is the KIP for adding a partitioner interface for producer. https://cwiki.apache.org/confluence/display/KAFKA/KIP-+22+-+Expose+a+Partitioner+interface

[KIP-DISCUSSION] KIP-22 Expose a Partitioner interface in the new producer

Hi, Here is the KIP for adding a partitioner interface for producer. https://cwiki.apache.org/confluence/display/KAFKA/KIP-+22+-+Expose+a+Partitioner+interface+in+the+new+producer There is one open question about how interface should look like. Please take a look and let me know if you

Re: [DISCUSS] KIP-12 - Kafka Sasl/Kerberos implementation

be in acceptor) and we have multiple processors thread. Do we still see this as an issue if it happens on the same thread as processor? .  --  Harsha Sent with Airmail On April 22, 2015 at 7:18:17 AM, Sriharsha Chintalapani (harsh...@fastmail.fm) wrote: Hi Rajini,        Thanks for the details

Re: [DISCUSS] KIP-12 - Kafka Sasl/Kerberos implementation

, Apr 21, 2015 at 11:13 PM, Sriharsha Chintalapani harsh...@fastmail.fm wrote: Hi Jay, Thanks for the review. 1. Isn't the blocking handshake going to be a performance concern? Can we do the handshake non-blocking instead? If anything that causes connections to drop can incur

Re: [DISCUSS] KIP-12 - Kafka Sasl/Kerberos implementation

, Sriharsha Chintalapani ka...@harsha.io wrote: Hi, I updated the KIP-12 with more details. Please take a look https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=51809888 Thanks, Harsha On February 11, 2015 at 10:02:43 AM, Harsha (ka...@harsha.io) wrote

Re: Topic existance Java API

Mingtao,           I think you are looking at scala version 2.8.2 . Check your kafka version by kafka_2.8.2-${version} . In 0.8.2  we’ve AdminUtils.topicExists(zkClient, topic) . Thanks, Harsha On April 20, 2015 at 2:20:27 PM, Mingtao Zhang (mail2ming...@gmail.com) wrote: Hi, I am on

Re: [DISCUSS] KIP-12 - Kafka Sasl/Kerberos implementation

one? Thanks, Jun On Mon, Apr 20, 2015 at 12:42 PM, Sriharsha Chintalapani ka...@harsha.io wrote: Hi, I updated the KIP-12 with more details. Please take a look https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=51809888 Thanks, Harsha On February

Re: [DISCUSS] KIP-12 - Kafka Sasl/Kerberos implementation

Hi,      I updated the KIP-12 with more details. Please take a look   https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=51809888 Thanks, Harsha On February 11, 2015 at 10:02:43 AM, Harsha (ka...@harsha.io) wrote: Thanks Joe. It will be part of KafkaServer and will run on its own