[
https://issues.apache.org/jira/browse/METRON-821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15968178#comment-15968178
]
ASF GitHub Bot commented on METRON-821:
---------------------------------------
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/510#discussion_r111482378
--- Diff: metron-deployment/vagrant/Kerberos-setup.md ---
@@ -87,147 +86,159 @@ sudo -u hdfs hdfs dfs -chmod 770 /user/metron
9. Setup Metron keytab
- ```
-kadmin.local -q "ktadd -k metron.headless.keytab [email protected]" && \
-cp metron.headless.keytab /etc/security/keytabs && \
-chown metron:hadoop /etc/security/keytabs/metron.headless.keytab && \
-chmod 440 /etc/security/keytabs/metron.headless.keytab
- ```
+ ```
+ kadmin.local -q "ktadd -k metron.headless.keytab [email protected]"
&& \
+ cp metron.headless.keytab /etc/security/keytabs && \
+ chown metron:hadoop /etc/security/keytabs/metron.headless.keytab && \
+ chmod 440 /etc/security/keytabs/metron.headless.keytab
+ ```
10. Kinit with the metron user
- ```
-kinit -kt /etc/security/keytabs/metron.headless.keytab [email protected]
- ```
+ ```
+ kinit -kt /etc/security/keytabs/metron.headless.keytab
[email protected]
+ ```
11. First create any additional Kafka topics you will need. We need to
create the topics before adding the required ACLs. The current full dev
installation will deploy bro, snort, enrichments, and indexing only. e.g.
- ```
-${HDP_HOME}/kafka-broker/bin/kafka-topics.sh --zookeeper ${ZOOKEEPER}:2181
--create --topic yaf --partitions 1 --replication-factor 1
- ```
+ ```
+ ${HDP_HOME}/kafka-broker/bin/kafka-topics.sh --zookeeper
${ZOOKEEPER}:2181 --create --topic yaf --partitions 1 --replication-factor 1
+ ```
12. Setup Kafka ACLs for the topics
- ```
-export KERB_USER=metron;
-for topic in bro enrichments indexing snort; do
-${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer
kafka.security.auth.SimpleAclAuthorizer --authorizer-properties
zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER}
--topic ${topic};
-done;
- ```
+ ```
+ export KERB_USER=metron;
+ for topic in bro enrichments indexing snort; do
+ ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer
kafka.security.auth.SimpleAclAuthorizer --authorizer-properties
zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER}
--topic ${topic}
+ done
+ ```
13. Setup Kafka ACLs for the consumer groups
- ```
-${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer
kafka.security.auth.SimpleAclAuthorizer --authorizer-properties
zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER}
--group bro_parser;
-${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer
kafka.security.auth.SimpleAclAuthorizer --authorizer-properties
zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER}
--group snort_parser;
-${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer
kafka.security.auth.SimpleAclAuthorizer --authorizer-properties
zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER}
--group yaf_parser;
-${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer
kafka.security.auth.SimpleAclAuthorizer --authorizer-properties
zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER}
--group enrichments;
-${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer
kafka.security.auth.SimpleAclAuthorizer --authorizer-properties
zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER}
--group indexing;
- ```
+ ```
+ ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer
kafka.security.auth.SimpleAclAuthorizer --authorizer-properties
zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER}
--group bro_parser
+ ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer
kafka.security.auth.SimpleAclAuthorizer --authorizer-properties
zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER}
--group snort_parser
+ ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer
kafka.security.auth.SimpleAclAuthorizer --authorizer-properties
zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER}
--group yaf_parser
+ ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer
kafka.security.auth.SimpleAclAuthorizer --authorizer-properties
zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER}
--group enrichments
+ ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer
kafka.security.auth.SimpleAclAuthorizer --authorizer-properties
zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER}
--group indexing
+ ```
14. Add metron user to the Kafka cluster ACL
- ```
-/usr/hdp/current/kafka-broker/bin/kafka-acls.sh --authorizer
kafka.security.auth.SimpleAclAuthorizer --authorizer-properties
zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER}
--cluster kafka-cluster
- ```
+ ```
+ ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer
kafka.security.auth.SimpleAclAuthorizer --authorizer-properties
zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER}
--cluster kafka-cluster
+ ```
15. We also need to grant permissions to the HBase tables. Kinit as the
hbase user and add ACLs for metron.
- ```
-kinit -kt /etc/security/keytabs/hbase.headless.keytab
[email protected]
-echo "grant 'metron', 'RW', 'threatintel'" | hbase shell
-echo "grant 'metron', 'RW', 'enrichment'" | hbase shell
- ```
+ ```
+ kinit -kt /etc/security/keytabs/hbase.headless.keytab
[email protected]
+ echo "grant 'metron', 'RW', 'threatintel'" | hbase shell
+ echo "grant 'metron', 'RW', 'enrichment'" | hbase shell
+ ```
16. Create a “.storm” directory in the metron user’s home directory and
switch to that directory.
- ```
-su metron && cd ~/
-mkdir .storm
-cd .storm
- ```
+ ```
+ su metron
+ mkdir ~/.storm
+ cd ~/.storm
+ ```
17. Create a custom client jaas file. This should look identical to the
Storm client jaas file located in /etc/storm/conf/client_jaas.conf except for
the addition of a Client stanza. The Client stanza is used for Zookeeper. All
quotes and semicolons are necessary.
- ```
-[metron@node1 .storm]$ cat client_jaas.conf
-StormClient {
- com.sun.security.auth.module.Krb5LoginModule required
- useTicketCache=true
- renewTicket=true
- serviceName="nimbus";
-};
-Client {
- com.sun.security.auth.module.Krb5LoginModule required
- useKeyTab=true
- keyTab="/etc/security/keytabs/metron.headless.keytab"
- storeKey=true
- useTicketCache=false
- serviceName="zookeeper"
- principal="[email protected]";
-};
-KafkaClient {
- com.sun.security.auth.module.Krb5LoginModule required
- useKeyTab=true
- keyTab="/etc/security/keytabs/metron.headless.keytab"
- storeKey=true
- useTicketCache=false
- serviceName="kafka"
- principal="[email protected]";
-};
- ```
+ ```
+ cat << EOF > client_jaas.conf
+ StormClient {
+ com.sun.security.auth.module.Krb5LoginModule required
+ useTicketCache=true
+ renewTicket=true
+ serviceName="nimbus";
+ };
+ Client {
+ com.sun.security.auth.module.Krb5LoginModule required
+ useKeyTab=true
+ keyTab="/etc/security/keytabs/metron.headless.keytab"
+ storeKey=true
+ useTicketCache=false
+ serviceName="zookeeper"
+ principal="[email protected]";
+ };
+ KafkaClient {
+ com.sun.security.auth.module.Krb5LoginModule required
+ useKeyTab=true
+ keyTab="/etc/security/keytabs/metron.headless.keytab"
+ storeKey=true
+ useTicketCache=false
+ serviceName="kafka"
+ principal="[email protected]";
+ };
+ EOF
+ ```
18. Create a storm.yaml with jaas file info. Set the array of nimbus hosts
accordingly.
- ```
-[metron@node1 .storm]$ cat storm.yaml
-nimbus.seeds : ['node1']
-java.security.auth.login.config : '/home/metron/.storm/client_jaas.conf'
-storm.thrift.transport :
'org.apache.storm.security.auth.kerberos.KerberosSaslTransportPlugin'
- ```
+ ```
+ cat << EOF > storm.yaml
+ nimbus.seeds : ['node1']
+ java.security.auth.login.config :
'/home/metron/.storm/client_jaas.conf'
+ storm.thrift.transport :
'org.apache.storm.security.auth.kerberos.KerberosSaslTransportPlugin'
+ EOF
+ ```
19. Create an auxiliary storm configuration json file in the metron user’s
home directory. Note the login config option in the file points to our custom
client_jaas.conf.
- ```
-cd /home/metron
-[metron@node1 ~]$ cat storm-config.json
-{
- "topology.worker.childopts" :
"-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf"
-}
- ```
+ ```
+ cat << EOF > ~/storm-config.json
+ {
+ "topology.worker.childopts" :
"-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf"
+ }
+ EOF
+ ```
20. Setup enrichment and indexing.
- a. Modify enrichment.properties -
`${METRON_HOME}/config/enrichment.properties`
-
+ a. Modify enrichment.properties as root located at
`${METRON_HOME}/config/enrichment.properties`
```
- kafka.security.protocol=PLAINTEXTSASL
-
topology.worker.childopts=-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf
+ if [[ $EUID -ne 0 ]]; then
+ echo -e "\nERROR:\tYou must be root to run these commands. You
may need to type exit."
+ else
+ sed -i
's/kafka.security.protocol=.*/kafka.security.protocol=PLAINTEXTSASL/'
${METRON_HOME}/config/enrichment.properties
+ sed -i
's/topology.worker.childopts=.*/topology.worker.childopts=-Djava.security.auth.login.config=\/home\/metron\/.storm\/client_jaas.conf/'
${METRON_HOME}/config/enrichment.properties
+ fi
```
- b. Modify elasticsearch.properties -
`${METRON_HOME}/config/elasticsearch.properties`
-
+ b. Modify elasticsearch.properties as root located at
`${METRON_HOME}/config/elasticsearch.properties`
```
- kafka.security.protocol=PLAINTEXTSASL
-
topology.worker.childopts=-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf
+ if [[ $EUID -ne 0 ]]; then
+ echo -e "\nERROR:\tYou must be root to run these commands. You
may need to type exit."
+ else
+ sed -i
's/kafka.security.protocol=.*/kafka.security.protocol=PLAINTEXTSASL/'
${METRON_HOME}/config/elasticsearch.properties
+ sed -i
's/topology.worker.childopts=.*/topology.worker.childopts=-Djava.security.auth.login.config=\/home\/metron\/.storm\/client_jaas.conf/'
${METRON_HOME}/config/elasticsearch.properties
+ fi
--- End diff --
I kind of prefer just showing the key=value so that it is clear to the user
what properties need to change.
I appreciate the fact that you've automated it, but since these are manual
instructions, maybe we should opt for making them as clear as possible. I could
be convinced otherwise though.
> Minor fixes in full dev kerberos setup instructions
> ---------------------------------------------------
>
> Key: METRON-821
> URL: https://issues.apache.org/jira/browse/METRON-821
> Project: Metron
> Issue Type: Bug
> Reporter: Jon Zeolla
> Assignee: Jon Zeolla
>
> There is at least one error in metron-deployment/vagrant/Kerberos-setup.md
> and some other very minor changes for clarity/accuracy.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
