[
https://issues.apache.org/jira/browse/METRON-793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939104#comment-15939104
]
ASF GitHub Bot commented on METRON-793:
---------------------------------------
Github user mmiklavc commented on the issue:
https://github.com/apache/incubator-metron/pull/486
I just finished running some parser tests in a Kerberized environment to
confirm that this will address both kerberized and non-kerberized
configurations. Here is what I had to do to get this running. Note, I'm using
Ambari to configure everything for Hadoop, but not the Metron components. There
is separate work that will need to be done for Metron in the MPack for Ambari.
1. Spin up full dev as normal (ensure that the profile is correctly set to
-P HDP-2.5.0.0 in metron-deployment/roles/metron-builder/tasks/main.yml). As of
this writing, the current full-dev ansible install will do a complete build as
part of the deployment.
2. service stop monit and kill all the topologies. You can go ahead and
kill the other sensors per the commands from @cestella as well (bro, pycapa,
etc.). We'll spin up our own simple test topology and populate the kafka topic
manually to test this out.
3. Setup symlinks
```
ln -s /usr/metron/0.3.1/lib/metron-*-uber.jar
/usr/hdp/2.5.3.0-37/storm/extlib-daemon/
ln -s /usr/hdp/current/hadoop-client/conf/core-site.xml /etc/storm/conf
ln -s /usr/hdp/current/hbase-client/conf/hbase-site.xml /etc/storm/conf
ln -s
/usr/hdp/2.5.3.0-37/storm/contrib/storm-hbase/storm-hbase-1.0.1.2.5.3.0-37.jar
/usr/hdp/2.5.3.0-37/storm/extlib
ln -s
/usr/hdp/2.5.3.0-37/storm/contrib/storm-hdfs/storm-hdfs-1.0.1.2.5.3.0-37.jar
/usr/hdp/2.5.3.0-37/storm/extlib
```
4. Check that the jce security is setup in
/usr/jdk64/jdk1.8.0_77/jre/lib/security/. If not, then you'll want to run
through the following steps:
4.1 Download the jce policy from:
http://www.oracle.com/technetwork/java/javase/downloads/jce8-download-2133166.html
and copy it to your vm.
4.2 unzip -o -j -q /path-to-the-copied-zip/jce_policy-8.zip -d
/usr/jdk64/jdk1.8.0_77/jre/lib/security/
5. Configure MIT Kerberos
```
yum -y install krb5-server krb5-libs krb5-workstation
sed -i 's/kerberos.example.com/node1/g' /etc/krb5.conf
cp /etc/krb5.conf /var/lib/ambari-server/resources/scripts
kdb5_util create -s
/etc/rc.d/init.d/krb5kdc start
/etc/rc.d/init.d/kadmin start
chkconfig krb5kdc on
chkconfig kadmin on

```
6. Setup admin and personal user principals. You'll kinit as the "{your
user name}" user. For my tests, I did "addprinc mike". I just made the
passwords "password" for the sake of easy testing.
```
kadmin.local -q "addprinc admin/admin"
kadmin.local -q "addprinc {your user name}"
```
7. In Ambari, setup AutoHDFS by adding the following properties to
custom-storm-site
```
nimbus.autocredential.plugins.classes=['org.apache.storm.hdfs.common.security.AutoHDFS','org.apache.storm.hbase.security.AutoHBase']
nimbus.credential.renewers.classes=['org.apache.storm.hdfs.common.security.AutoHDFS','org.apache.storm.hbase.security.AutoHBase']
hdfs.keytab.file=/etc/security/keytabs/hdfs.headless.keytab
[email protected]
hbase.keytab.file=/etc/security/keytabs/hbase.headless.keytab
[email protected]
nimbus.credential.renewers.freq.secs=82800
```
8. Kerberize the cluster via Ambari. The wizard is fairly straightforward,
but further documentation can be found
[here](http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_security/content/_enabling_kerberos_security_in_ambari.html).
For this exercise, choose existing MIT KDC (this is what we setup and
installed in previous steps.) Realm is EXAMPLE.COM. The admin principal will
end up as admin/[email protected] when testing the KDC.
9. Let the cluster spin up, but don't worry about starting up Metron via
Ambari - we're going to run the parsers manually against the rest of the Hadoop
cluster Kerberized.
10. Kinit and provide the password you chose from earlier.
```
kinit <your username>
```
11. Create a kafka topic for the jsonMap parser.
```
/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper node1:2181
--create --topic jsonMap --replication-factor 1 --partitions 1
```
12. Setup the ACLs for the topic AND the consumer group. The consumer group
config is new in this API as far as I can tell because we didn't have to do
this in the "storm-kafka" version of the api. Note that the docs around kafka
0.10 will suggest using a "--consumer-group" option, but in HDP 2.5.x the
option needs to be "--group". Also make sure you use your username in
"--allow-principal User:<your username>" instead of "mike".
```
/usr/hdp/current/kafka-broker/bin/kafka-acls.sh --authorizer
kafka.security.auth.SimpleAclAuthorizer --authorizer-properties
zookeeper.connect=node1:2181 --add --allow-principal User:storm-metron_cluster
--allow-principal User:mike --topic jsonMap
/usr/hdp/current/kafka-broker/bin/kafka-acls.sh --authorizer-properties
zookeeper.connect=node1:2181 --group jsonMap_parser --list --topic jsonMap
```
13. Setup an additional config for the Kafka consumer security protocol
options. Create a file wherever is convenient and note the path. e.g. Here is
my config output from root home.
```
# cat ~/extra_config.json
{
"security.protocol":"PLAINTEXTSASL"
}
```
14. Configure the jsonMap parser to handle Kerberos. The parserConfig key
is the new piece required for Kerberos.
```
# cat /usr/metron/0.3.1/config/zookeeper/parsers/jsonMap.json
{
"parserClassName":"org.apache.metron.parsers.json.JSONMapParser",
"sensorTopic":"jsonMap",
"parserConfig" : {
"kafka.producerConfigs" : {
"security.protocol" : "PLAINTEXTSASL"
},
"error.kafka.producerConfigs" : {
"security.protocol" : "PLAINTEXTSASL"
}
}
}
```
15. Push the updated configs to Zookeeper
```
/usr/metron/0.3.1/bin/zk_load_configs.sh -z node1:2181 -m PUSH -i
/usr/metron/0.3.1/config/zookeeper
```
16. Start the topology
```
./start_parser_topology.sh -k node1:6667 -z node1:2181 -s jsonMap -esc
~/extra_config.json
```
17. Create a file with some test data that we'll push into the parser Kafka
topic
```
# cat ~/sampledata.txt
{ "string" : "bar", "number" : 2, "ignored" : [ "blah" ] }
{ "number" : 7 }
```
18. Push data into Kafa
```
cat ~/sampledata.txt |
/usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list
node1:6667 --security-protocol PLAINTEXTSASL --topic jsonMap
```
19. Verify our data comes out on the other end by looking at the
enrichments topic.
```
/usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper
node1:2181 --from-beginning --security-protocol PLAINTEXTSASL --topic
enrichments
```
The above process worked for me with this PR. That should be it.
> Migrate to storm-kafka-client kafka spout from storm-kafka
> ----------------------------------------------------------
>
> Key: METRON-793
> URL: https://issues.apache.org/jira/browse/METRON-793
> Project: Metron
> Issue Type: Improvement
> Reporter: Casey Stella
>
> In order to eventually support kerberos, the suggested path is to migrate to
> the new kafka spout (org.apache.storm:storm-kafka-client) which uses the new
> consumer API.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
