tigerquoll edited a comment on issue #1519: METRON-2265: Update Kerberos settings URL: https://github.com/apache/metron/pull/1519#issuecomment-534891894 I do a full dev install and followed https://github.com/apache/metron/blob/429c5eac55554496c967ca9f6e935f6e0b2d4781/metron-deployment/Kerberos-manual-setup.md Used the following environment: ``` export BROKERLIST=node1:6667 export KAFKA_HOME=/usr/hdp/3.1.4.0-315/kafka export [email protected] export METRON_SERVICE_KEYTAB=/etc/security/keytabs/metron.headless.keytab export CLIENT_JAAS_ARG=/etc/kafka/conf/kafka_client_jaas.conf export KAFKA_SECURITY_PROTOCOL=SASL_PLAINTEXT export ELASTICSEARCH=node1:9200 export KAFKA_OPTS="-Djava.security.auth.login.config=$CLIENT_JAAS_ARG" ``` Please note the KAFKA_OPTS variable suggested in the guide is wrong, it is a copy from the HDP page which is most likely to be a typo. The check of the count in elasticsearch at the "Push Data" stage is misleading, as the guide does not delete the previous collection before enabling kerberos. Repeating the "add sample-bro.txt" stage does not increase the document count. Was utilising `${KAFKA_HOME}/bin/kafka-consumer-groups.sh` to monitor Kafka activity by utilising the following recipe: 1. Create file `/home/metron/kafka.command.config` with content of ``` security.protocol=SASL_PLAINTEXT ``` 2. Run command ``` ${KAFKA_HOME}/bin/kafka-consumer-groups.sh --command-config=/home/metron/kafka.command.config --bootstrap-server ${BROKERLIST} --describe --group bro_parser ``` This showed nothing consuming from the bro topic. ``` Consumer group 'bro_parser' has no active members. TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID bro 0 13536 13898 362 - - - [2019-09-25 04:45:03,310] WARN [Principal=null]: TGT renewal thread has been interrupted and will exit. (org.apache.kafka.common.security.kerberos.KerberosLogin) ``` Further investigations reveals that storm workers are unable to communicate with ZK? ``` 2019-2019-09-25 00:36:21.805 o.a.k.c.NetworkClient Thread-18-kafkaSpout-bro-executor[4 4] [WARN] [Consumer clientId=consumer-2, groupId=bro_parser] Connection to node 1001 could not be established. Broker may not be available.- 25 00:36:21.906 o.a.k.c.NetworkClient Thread-12-kafkaSpout-yaf-executor[6 6] [WARN] [Consumer clientId=consumer-1, groupId=yaf_parser] Connection to node 1001 could not be established. Broker may not be available. 2019-09-25 00:36:22.012 o.a.k.c.NetworkClient Thread-16-kafkaSpout-snort-executor[5 5] [WARN] [Consumer clientId=consumer-3, groupId=snort_parser] Connection to node 1001 could not be established. Broker may not be available. 2019-09-25 00:36:22.053 o.a.s.s.o.a.z.ClientCnxn main-SendThread(node1:2181) [INFO] Opening socket connection to server node1/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) 2019-09-25 00:36:22.053 o.a.s.s.o.a.z.ClientCnxn main-SendThread(node1:2181) [WARN] Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused ...java.lang.RuntimeException: ("Error when processing an event") at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341) [storm-core-1.2.1.3.1.4.0-315.jar:1.2.1.3.1.4.0-315] at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.7.0.jar:?] at org.apache.storm.daemon.worker$mk_halting_timer$fn__10399.invoke(worker.clj:259) [storm-core-1.2.1.3.1.4.0-315.jar:1.2.1.3.1.4.0-315] at org.apache.storm.timer$mk_timer$fn__1639$fn__1640.invoke(timer.clj:71) [storm-core-1.2.1.3.1.4.0-315.jar:1.2.1.3.1.4.0-315] at org.apache.storm.timer$mk_timer$fn__1639.invoke(timer.clj:42) [storm-core-1.2.1.3.1.4.0-315.jar:1.2.1.3.1.4.0-315] at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112] 2019-09-25 00:36:22.191 o.a.s.d.worker Thread-21 [INFO] Shutting down worker bro__snort__yaf-4-1569366291 a0046483-4363-4209-b4e0-d1ad9f6deea2 6701 2019-09-25 00:36:22.191 o.a.s.d.worker Thread-21 [INFO] Terminating messaging context 2019-09-25 00:36:22.191 o.a.s.d.worker Thread-21 [INFO] Shutting down executors ``` Tried back tracking through HDP/Storm/Kerberos docs to verify correct setup, but ran out of time.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
