Re: [EXTERNAL] kafka

2016-11-02 Thread Raja . Aravapalli 

Thanks a lot for the response Pramod.

Our kafka environment is NOT Kerberized! But the kafka topic is secure via SSL 
certificates!

Only hosts with valid SSL certificates can only connect! Any thoughts in this 
regard will be really helpful!!

Thanks a lot!!


Regards,
Raja.

From: Pramod Immaneni <pra...@datatorrent.com>
Reply-To: "users@apex.apache.org" <users@apex.apache.org>
Date: Tuesday, November 1, 2016 at 3:24 AM
To: "users@apex.apache.org" <users@apex.apache.org>
Subject: Re: [EXTERNAL] kafka

Hi Raja,

Yes we have had success reading from kerberized secure Kafka. If I remember 
correctly we had to set a few attributes like container jvm options in Apex. Is 
your setup Kerberos secured Kafka?

Thanks

On Oct 30, 2016, at 10:56 PM, Raja.Aravapalli 
<raja.aravapa...@target.com<mailto:raja.aravapa...@target.com>> wrote:

Hi Team,

Can someone pls help me with below requested information ?

Does apache apex have any inbuilt kafka input operator to read from Kafka 0.9 
secure kafka topics?

Thanks a lot.

Regards,
Raja.

From: "Raja.Aravapalli" 
<raja.aravapa...@target.com<mailto:raja.aravapa...@target.com>>
Reply-To: "users@apex.apache.org<mailto:users@apex.apache.org>" 
<users@apex.apache.org<mailto:users@apex.apache.org>>
Date: Monday, October 24, 2016 at 2:29 PM
To: "users@apex.apache.org<mailto:users@apex.apache.org>" 
<users@apex.apache.org<mailto:users@apex.apache.org>>
Subject: [EXTERNAL] kafka


Hi,

Do we have any kaka operator readily available to consume messages from secure 
kafka topics in kafka 0.9 clusters?

Thanks a lot.


Regards,
Raja.

Re: [EXTERNAL] kafka

2016-11-01 Thread Thomas Weise 
IMO this should be added here:

http://apex.apache.org/docs/malhar/operators/kafkaInputOperator/


On Tue, Nov 1, 2016 at 5:24 PM, hsy...@gmail.com <hsy...@gmail.com> wrote:

> Hey Raja,
>
> The setup for secure kafka input operator is not easy. You can follow
> these steps.
> 1. Follow kafka document to setup your brokers properly (
> https://kafka.apache.org/090/documentation.html#security_overview)
> 2. You have to manually create the client JAAS file (
> https://kafka.apache.org/090/documentation.html#security_overview)
>a sample file would look like this:
>  KafkaClient {
>
> com.sun.security.auth.module.Krb5LoginModule required
> useKeyTab=true
> storeKey=true
> keyTab="/etc/security/keytabs/kafka_client.keytab"
> principal="kafka-clien...@example.com";
> };
>
> 3. On the operator you have to set the attribute JVM_OPTS
>  -Djava.security.auth.login.config=/etc/kafka/kafka_client_jaas.conf
>
> 4. On the operator you have to set the consumerProperties, for example
>
> set dt.operator.$your_operator_name.{consumerProps.security.protocol}  to
>  SASL or other security function you use
> set dt.operator.$your_operator_name.{consumerPrope.sasl.
> kerberos.service.name} to kafka
>
>
> Hope this would help you!
>
>
> Regards,
> Siyuan
>
>
>
> On Sun, Oct 30, 2016 at 10:56 PM, Raja.Aravapalli <
> raja.aravapa...@target.com> wrote:
>
>>
>>
>> Hi Team,
>>
>>
>>
>> Can someone pls help me with below requested information ?
>>
>>
>>
>> Does apache apex have any inbuilt kafka input operator to read from Kafka
>> 0.9 secure kafka topics?
>>
>>
>>
>> Thanks a lot.
>>
>>
>>
>> Regards,
>>
>> Raja.
>>
>>
>>
>> *From: *"Raja.Aravapalli" <raja.aravapa...@target.com>
>> *Reply-To: *"users@apex.apache.org" <users@apex.apache.org>
>> *Date: *Monday, October 24, 2016 at 2:29 PM
>> *To: *"users@apex.apache.org" <users@apex.apache.org>
>> *Subject: *[EXTERNAL] kafka
>>
>>
>>
>>
>>
>> Hi,
>>
>>
>>
>> Do we have any kaka operator readily available to consume messages from
>> secure kafka topics in kafka 0.9 clusters?
>>
>>
>>
>> Thanks a lot.
>>
>>
>>
>>
>>
>> Regards,
>>
>> Raja.
>>
>
>

Re: [EXTERNAL] kafka

2016-11-01 Thread hsy...@gmail.com 
Hey Raja,

The setup for secure kafka input operator is not easy. You can follow these
steps.
1. Follow kafka document to setup your brokers properly (
https://kafka.apache.org/090/documentation.html#security_overview)
2. You have to manually create the client JAAS file (
https://kafka.apache.org/090/documentation.html#security_overview)
   a sample file would look like this:
 KafkaClient {

com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
keyTab="/etc/security/keytabs/kafka_client.keytab"
principal="kafka-clien...@example.com";
};

3. On the operator you have to set the attribute JVM_OPTS
 -Djava.security.auth.login.config=/etc/kafka/kafka_client_jaas.conf

4. On the operator you have to set the consumerProperties, for example

set dt.operator.$your_operator_name.{consumerProps.security.protocol}  to
 SASL or other security function you use
set dt.operator.$your_operator_name.{consumerPrope.
sasl.kerberos.service.name} to kafka


Hope this would help you!


Regards,
Siyuan



On Sun, Oct 30, 2016 at 10:56 PM, Raja.Aravapalli <
raja.aravapa...@target.com> wrote:

>
>
> Hi Team,
>
>
>
> Can someone pls help me with below requested information ?
>
>
>
> Does apache apex have any inbuilt kafka input operator to read from Kafka
> 0.9 secure kafka topics?
>
>
>
> Thanks a lot.
>
>
>
> Regards,
>
> Raja.
>
>
>
> *From: *"Raja.Aravapalli" <raja.aravapa...@target.com>
> *Reply-To: *"users@apex.apache.org" <users@apex.apache.org>
> *Date: *Monday, October 24, 2016 at 2:29 PM
> *To: *"users@apex.apache.org" <users@apex.apache.org>
> *Subject: *[EXTERNAL] kafka
>
>
>
>
>
> Hi,
>
>
>
> Do we have any kaka operator readily available to consume messages from
> secure kafka topics in kafka 0.9 clusters?
>
>
>
> Thanks a lot.
>
>
>
>
>
> Regards,
>
> Raja.
>