[
https://issues.apache.org/jira/browse/FLUME-2792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14965171#comment-14965171
]
Alexander Bij commented on FLUME-2792:
--------------------------------------
*We have solved the problem*.
We can now use the Flume KafkaSink to send events to a Kerberos secured
KafkaCluster.
The kafka-jars distributed with flume
(/usr/hdp/2.3.0.0-2557/flume/lib/kafka_2.10-0.8.2-beta.jar +
kafka-clients-0.8.2-beta.jar) are not supporting kerberos. You must include the
/usr/hdp/2.3.0.0-2557/kafka/libs/kafka_2.10-0.8.2.2.3.0.0-2557.jar +
kafka-clients-0.8.2.2.3.0.0-2557.jar on the flume classpath.
In order to make it work we changed the flume classpath and removed the Flume
kafka jars from the /usr/hdp/2.3.0.0-2557/flume/lib/, because they are placed
infront of the classpath.
We also created the flume_client_jaas.conf as mentioned by Roshan Naik.
{noformat}
# Add hdp kafka jars
export
FLUME_CLASSPATH=/usr/hdp/2.3.0.0-2557/kafka/libs/kafka_2.10-0.8.2.2.3.0.0-2557.jar:/usr/hdp/2.3.0.0-2557/kafka/libs/kafka-clients-0.8.2.2.3.0.0-2557.jar:$FLUME_CLASSPATH
export JAVA_OPTS="-Xms1024m -Xmx4096m
-Djava.security.auth.login.config=/etc/flume/conf/flume_client_jaas.conf"
{noformat}
We hope this will be resolved in the next version of HDP.
> Flume Kafka Kerberos Support
> ----------------------------
>
> Key: FLUME-2792
> URL: https://issues.apache.org/jira/browse/FLUME-2792
> Project: Flume
> Issue Type: Bug
> Components: Configuration, Docs, Sinks+Sources
> Affects Versions: v1.6.0, v1.5.2
> Environment: HDP 2.3 fully kerberized including Kafka 0.8.2.2 + Flume
> 1.5.2 or Apache Flume 1.6 downloaded from apache.org
> Reporter: Hari Sekhon
> Priority: Blocker
>
> Following on from FLUME-2790 it appears as though Flume doesn't yet have
> support for Kafka + Kerberos as there are is no setting documented in the
> Flume 1.6.0 user guide under the Kafka source section to tell Flume to use
> plaintextsasl as the connection mechanism to Kafka and Kafka rejects
> unauthenticated plaintext mechanism:
> {code}15/09/10 16:51:22 INFO consumer.ConsumerFetcherManager:
> [ConsumerFetcherManager-1441903874830] Added fetcher for partitions
> ArrayBuffer()
> 15/09/10 16:51:22 WARN consumer.ConsumerFetcherManager$LeaderFinderThread:
> [flume_<custom_scrubbed>-1441903874763-abdc98ec-leader-finder-thread], Failed
> to find leader for Set([<custom_scrubbed>,0], [<custom_scrubbed>,1])
> kafka.common.BrokerEndPointNotAvailableException: End point PLAINTEXT not
> found for broker 0
> at kafka.cluster.Broker.getBrokerEndPoint(Broker.scala:140)
> at
> kafka.utils.ZkUtils$$anonfun$getAllBrokerEndPointsForChannel$1.apply(ZkUtils.scala:124)
> at
> kafka.utils.ZkUtils$$anonfun$getAllBrokerEndPointsForChannel$1.apply(ZkUtils.scala:124)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
> at
> scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
> at scala.collection.AbstractTraversable.map(Traversable.scala:105)
> at
> kafka.utils.ZkUtils$.getAllBrokerEndPointsForChannel(ZkUtils.scala:124)
> at
> kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66)
> at
> kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60){code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
