Hi Michal,
The configuration in consumer.properties is not correct.
The 'sasl.kerberos.service.name' option expects the kerberos principal that
Kafka runs as.
In your case it should be '*sasl.kerberos.service.name
<http://sasl.kerberos.service.name/>=*kafka*' *
Can you please test using the Kafka Console Producer as well?
This will make sure that your kerberos setup is right.
Secondly, you must specify the same 'security.protocol' and '
sasl.kerberos.service.name' options in the Kafka Consumer stage in
StreamSets pipeline.
See attached snapshot that shows how to specify these properties.
Thanks
Hari.
*[image: Screen Shot 2016-03-04 at 10.32.42 AM.png]*
On Fri, Mar 4, 2016 at 7:16 AM Michał Kabocik <michal.kabo...@gmail.com>
wrote:
> Dear Hari,
>
> Thank you for your reply.
>
> Replying to your questions:
> Yes, I have all needed entries in etc/hosts and hosts can 'see' each other.
> I followed your suggestion and added mentioned entries in
> server.properties_krb5. Now when starting Kafka Broker I see:
> listeners = PLAINTEXT://:9092,SASL_PLAINTEXT://:9093
> advertised.listeners = PLAINTEXT://:9092,SASL_PLAINTEXT://:9093
> sasl.kerberos.service.name = kafka
> advertised.host.name = plx164h.xx.xxx.xx
>
> Unfortunately it didn't help. Error in StreamSets is the same. I've tried
> to use built-in kafka console consumer and also not succeded. Here is my
> config:
>
> On host A I have Kafka broker which is running with the config from
> previous email. On host B, I have another Kafka from which I used console
> consumer with following config:
>
> kafka_client_jaas.conf:
> KafkaClient {
> com.sun.security.auth.module.Krb5LoginModule required
> useKeyTab=true
> storeKey=true
> keyTab="/etc/security/keytabs/kafka_client.service.keytab"
> principal="client/10.xxx.xxx...@hdpcybersecacc.xx.xx";
> };
>
> consumer.properties:
> security.protocol=SASL_PLAINTEXT
> sasl.kerberos.service.name=client
>
> I'm starting console consumer with the command:
> ./bin/kafka-console-consumer.sh --bootstrap-server plx164h:9093 --topic
> streamsets2 --new-consumer --consumer.config consumer.properties
>
> When started, there is no error, console consumer seems to work fine, but
> when producing to this topic, no messages are read.
> From kerberos side everything looks correct:
>
> Mar 04 16:00:31 lxhnlxx.xxx.xx krb5kdc[16307](info): AS_REQ (4 etypes {18
> 17 16 23}) 10.xxx.xxx.72
> <http://www.google.com/url?q=http%3A%2F%2F10.111.159.72&sa=D&sntz=1&usg=AFQjCNGfeOUI5N_QC-VKyS_d9ouhNTfYpQ>:
> ISSUE: authtime 1457103631, etypes {rep=18 tkt=18 ses=18},
> client/10.xxx.xxxx...@hdpcybersecacc.xx.xx for
> krbtgt/hdpcybersecacc.xx...@hdpcybersecacc.xx.xx
> Mar 04 16:00:31 lxhnlxx.xxx.xx krb5kdc[16307](info): TGS_REQ (4 etypes {18
> 17 16 23}) 10.xxx.xxx.72
> <http://www.google.com/url?q=http%3A%2F%2F10.111.159.72&sa=D&sntz=1&usg=AFQjCNGfeOUI5N_QC-VKyS_d9ouhNTfYpQ>:
> ISSUE: authtime 1457103631, etypes {rep=18 tkt=18 ses=18},
> client/10.xxx.xxx...@hdpcybersecacc.xx.xx for
> client/plx164h.xx...@hdpcybersecacc.xx.xx
>
> Could you please a look at this? Maybe you see configuration error?
>
> Kind regards,
> Michal
>
> W dniu czwartek, 3 marca 2016 17:49:03 UTC+1 użytkownik Harikiran Nayak
> napisał:
>
>> Hi Michal,
>>
>> Can you please add the *advertised.listeners* and *advertised.host.name
>> <http://advertised.host.name>* properties in your kafka server config
>> file 'server.properties_krb5'?
>>
>> For example, I have the following configuration in my working setup
>>
>> listeners=SASL_PLAINTEXT://:9092
>> advertised.listeners=SASL_PLAINTEXT://:9092
>> host.name=kafka
>> advertised.host.name=kafka
>>
>> 'kafka' is the hostname on which the Kafka broker is running in my setup.
>> There is an entry for this host in '/etc/hosts' on the node where
>> StreamSets is running.
>>
>> Thanks
>> Hari.
>>
>> On Thu, Mar 3, 2016 at 8:19 AM Harikiran Nayak <ha...@streamsets.com>
>> wrote:
>>
> Hi Michal,
>>>
>>> Are you able to write and read from the kerberized Kafka setup using the
>>> Kafka Console Producer and Consumer?
>>>
>>> I am taking a look at your configuration files.
>>>
>>> Thanks
>>> Hari.
>>>
>> On Thu, Mar 3, 2016 at 8:09 AM Jonathan Natkins <na...@streamsets.com>
>>> wrote:
>>>
>> Hey Michal,
>>>>
>>>> I'm cc'ing the StreamSets user list, which might be able to get you
>>>> some better StreamSets-specific answers.
>>>>
>>>> Thanks!
>>>>
>>> Natty
>>>>
>>>
>>>>
>>>> On Thursday, March 3, 2016, Michał Kabocik <michal....@gmail.com>
>>>> wrote:
>>>>
>>>>> Dears,
>>>>>
>>>>> I’m Middleware Engineer and I’m trying to configure secure Kafka
>>>>> Cluster with SSL and Kerberos authentication with StreamSets, which will
>>>>> be
>>>>> used for data injection to HDP.
>>>>>
>>>>> I have two Kafka Clusters; one with SSL enabled and there I
>>>>> successfully connected StreamSets to Kafka with SSL authentication, and
>>>>> second one with Kerberos authentication and here I’m facing with the
>>>>> problem:
>>>>>
>>>>> Both Kafka (with Zookeeper) and StreamSets are configured to
>>>>> authenticate via Kerberos. When starting all of them, I see in the logs,
>>>>> that they are successfully authenticated (TGT granted etc.)
>>>>>
>>>>> I have two listeners defined in Kafka:
>>>>> listeners=PLAINTEXT://:9092,SASL_PLAINTEXT://:9093. When starting Kafka, I
>>>>> see Kafka listens on both, 9092 and 9093.
>>>>>
>>>>> When I connect StreamSets to Kafka on port 9092, everything works
>>>>> smooth. But when I try to connect to port 9093, error occurs:
>>>>>
>>>>> KAFKA_41 - Could not get partition count for topic 'streamsets5' :
>>>>> com.streamsets.pipeline.api.StageException: KAFKA_41 - Could not get
>>>>> caseition count for topic 'streamsets5' :
>>>>> org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
>>>>>
>>>>> I see no errors in Kafka, in the log of StreamSets, there is only
>>>>> above error visible. I attached major config files of Kafka, Zookeeper and
>>>>> StreamSets.
>>>>>
>>>>> Will greatly appreciate your help in solving this case!
>>>>>
>>>>> Kind regards,
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>> Jonathan "Natty" Natkins
>>>> StreamSets | Field Engineer
>>>>
>>> mobile: 609.577.1600[image: Auto Generated Inline Image 1]
>>>> <#msg-f:1527884890376799684_> | linkedin
>>>> <http://www.linkedin.com/in/nattyice>
>>>>
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "sdc-user" group.
>>>>
>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to sdc-user+u...@streamsets.com.
>>>
>>>
>>>> Visit this group at
>>>> https://groups.google.com/a/streamsets.com/group/sdc-user/.
>>>>
>>> --
> You received this message because you are subscribed to the Google Groups
> "sdc-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sdc-user+unsubscr...@streamsets.com.
> Visit this group at
> https://groups.google.com/a/streamsets.com/group/sdc-user/.
>
