Hi Jayesh, I'm sorry, but I can't change storm version. HDP2.5 includes 1.0.1.
Any workaround? Regards, Andrey > 18 марта 2017 г., в 5:53, Jayesh Senjaliya <[email protected]> написал(а): > > Hi Markovich, > > Eagle 0.5 is well supported with storm 0.9.3 , can you please try using that > ? > > or do you have to use it with storm 1.x ? > > - Jayesh > > >> On Fri, Mar 17, 2017 at 6:50 AM, Markovich <[email protected]> wrote: >> Hello eagle users and dev, >> >> I'm stuck on installing eagle on HDP2.5 cluster with jdk 1.8.0_101. >> >> Here is my service versions and my cluster is secured using kerberos + >> ranger. >> >> HDFS 2.7.3 >> Hive 1.2.1000 >> Storm 1.0.1 >> Kafka 0.10.0 >> Kerberos 1.10.3-10 >> >> Here is what I've done already: >> >> 1) Downloaded latest eagle from github (version 0.5 snapshot). >> 2) Builded it using mvn clean package -DskipTests >> [INFO] BUILD SUCCESS >> [INFO] >> ------------------------------------------------------------------------ >> [INFO] Total time: 16:11 min >> [INFO] Finished at: 2017-03-17T16:00:45+03:00 >> [INFO] Final Memory: 183M/1755M >> [INFO] >> ------------------------------------------------------------------------ >> >> 3) Moved tarball to /usr/hdp/current/ and extracted to eagle >> 4) Changed conf/eagle.conf for my cluster: >> zkQuorum >> zookeeperZnodeParent = "/hbase-secure" >> metadata -> jdbc -> user, pass and host >> nimbusHost >> 5) Launched bin/eagle-env.sh and bin/eagle-server start >> >> Eagle started on 9090 port. >> 6) In Web UI entered SiteId >> 7) Selected install on HDFS Audit Log Monitor and chenged General settings >> (Kafka hosts) and Advanced fs.defaultFS. Execution Mode - cluster >> 8) Created hdfs_audit_log_{SITE_ID}, hdfs_audit_log_enriched_{SITE_ID} >> 9) Launched Logstash to write to hdfs_audit_log_{SITE_ID} >> 10) Checked, logs are pushed in kafka >> 11) Appliaction is initialized >> 12) Tried to start this appliaction: >> >> INFO [2017-03-17 13:28:45,926] >> org.apache.eagle.dataproc.impl.storm.kafka.KafkaSpoutProvider: Use topic : >> hdfs_audit_log, zkConnection : localhost:2181 , fetchSize : 1048576 >> WARN [2017-03-17 13:28:46,216] >> org.apache.eagle.app.messaging.KafkaStreamProvider: Using default shared >> sink topic dataSinkConfig.topic: hdfs_audit_event >> INFO [2017-03-17 13:28:46,331] >> org.apache.eagle.app.environment.impl.StormExecutionRuntime: Starting >> HDFS_AUDIT_LOG_MONITOR_APP_MYSITE >> (org.apache.eagle.security.auditlog.HdfsAuditLogApplication), mode: CLUSTER >> INFO [2017-03-17 13:28:46,332] >> org.apache.eagle.app.environment.impl.StormExecutionRuntime: Overriding >> application.storm.nimbusHost = demo5 >> INFO [2017-03-17 13:28:46,332] >> org.apache.eagle.app.environment.impl.StormExecutionRuntime: Overriding >> application.storm.nimbusThriftPort = 6627 >> INFO [2017-03-17 13:28:46,332] >> org.apache.eagle.app.environment.impl.StormExecutionRuntime: Submitting as >> cluster mode ... >> >> Nothing in Storm UI. Nothing in logs. >> Tried to change KafkaSpoutProvider to normal with SITE_ID. -> Nothing >> changed. >> >> I think I need to enter some kerberos related configs for storm... >> >> 13) Ok, changed Execution Mode - Local. >> 14) Started. Status in UI changed to starting. In logs a lot of activity, >> but also errors: >> >> WARN [2017-03-17 13:33:40,952] storm.kafka.KafkaUtils: there are more tasks >> than partitions (tasks: 2; partitions: 1), some tasks will be idle >> INFO [2017-03-17 13:33:40,952] storm.kafka.KafkaUtils: Task [1/2] assigned >> [Partition{host=null:-1, partition=0}] >> INFO [2017-03-17 13:33:40,952] storm.kafka.ZkCoordinator: Task [1/2] >> Deleted partition managers: [] >> INFO [2017-03-17 13:33:40,952] storm.kafka.ZkCoordinator: Task [1/2] New >> partition managers: [Partition{host=null:-1, partition=0}] >> INFO [2017-03-17 13:33:40,990] storm.kafka.PartitionManager: Read partition >> information from: >> /consumers/hdfs_audit_log_tuskpro/eagleConsumer/partition_0 --> null >> ERROR [2017-03-17 13:33:41,047] backtype.storm.util: Async loop died! >> ! java.lang.NullPointerException: null >> ! at org.apache.kafka.common.utils.Utils.formatAddress(Utils.java:312) >> ~[eagle-topology-0.5.0-SNAPSHOT-assembly.jar:na] >> ! at >> kafka.consumer.SimpleConsumer$$anonfun$disconnect$1.apply(SimpleConsumer.scala:49) >> ~[eagle-topology-0.5.0-SNAPSHOT-assembly.jar:na] >> ! at >> kafka.consumer.SimpleConsumer$$anonfun$disconnect$1.apply(SimpleConsumer.scala:49) >> ~[eagle-topology-0.5.0-SNAPSHOT-assembly.jar:na] >> ! at kafka.utils.Logging$class.debug(Logging.scala:52) >> ~[eagle-topology-0.5.0-SNAPSHOT-assembly.jar:na] >> ! at kafka.consumer.SimpleConsumer.debug(SimpleConsumer.scala:30) >> ~[eagle-topology-0.5.0-SNAPSHOT-assembly.jar:na] >> ! at kafka.consumer.SimpleConsumer.disconnect(SimpleConsumer.scala:49) >> ~[eagle-topology-0.5.0-SNAPSHOT-assembly.jar:na] >> ! at kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:82) >> ~[eagle-topology-0.5.0-SNAPSHOT-assembly.jar:na] >> ! at >> kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:68) >> ~[eagle-topology-0.5.0-SNAPSHOT-assembly.jar:na] >> ! at >> kafka.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:127) >> ~[eagle-topology-0.5.0-SNAPSHOT-assembly.jar:na] >> ! at >> kafka.javaapi.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:79) >> ~[eagle-topology-0.5.0-SNAPSHOT-assembly.jar:na] >> ! at storm.kafka.KafkaUtils.getOffset(KafkaUtils.java:74) >> ~[eagle-topology-0.5.0-SNAPSHOT-assembly.jar:na] >> ! at storm.kafka.KafkaUtils.getOffset(KafkaUtils.java:64) >> ~[eagle-topology-0.5.0-SNAPSHOT-assembly.jar:na] >> ! at storm.kafka.PartitionManager.<init>(PartitionManager.java:89) >> ~[eagle-topology-0.5.0-SNAPSHOT-assembly.jar:na] >> ! at storm.kafka.ZkCoordinator.refresh(ZkCoordinator.java:98) >> ~[eagle-topology-0.5.0-SNAPSHOT-assembly.jar:na] >> ! ... 6 common frames omitted >> ! Causing: java.lang.RuntimeException: java.lang.NullPointerException >> ! at storm.kafka.ZkCoordinator.refresh(ZkCoordinator.java:103) >> ~[eagle-topology-0.5.0-SNAPSHOT-assembly.jar:na] >> ! at storm.kafka.ZkCoordinator.getMyManagedPartitions(ZkCoordinator.java:69) >> ~[eagle-topology-0.5.0-SNAPSHOT-assembly.jar:na] >> ! at storm.kafka.KafkaSpout.nextTuple(KafkaSpout.java:135) >> ~[eagle-topology-0.5.0-SNAPSHOT-assembly.jar:na] >> ! at >> backtype.storm.daemon.executor$fn__3373$fn__3388$fn__3417.invoke(executor.clj:565) >> ~[storm-core-0.9.3.jar:0.9.3] >> ! at backtype.storm.util$async_loop$fn__464.invoke(util.clj:463) >> ~[storm-core-0.9.3.jar:0.9.3] >> ! at clojure.lang.AFn.run(AFn.java:24) >> [eagle-topology-0.5.0-SNAPSHOT-assembly.jar:na] >> ! at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101] >> >> .... >> >> ERROR [2017-03-17 13:33:41,079] backtype.storm.util: Halting process: >> ("Worker died") >> ! java.lang.RuntimeException: ("Worker died") >> ! at backtype.storm.util$exit_process_BANG_.doInvoke(util.clj:325) >> [storm-core-0.9.3.jar:0.9.3] >> ! at clojure.lang.RestFn.invoke(RestFn.java:423) >> [eagle-topology-0.5.0-SNAPSHOT-assembly.jar:na] >> ! at backtype.storm.daemon.worker$fn__3808$fn__3809.invoke(worker.clj:452) >> [storm-core-0.9.3.jar:0.9.3] >> ! at >> backtype.storm.daemon.executor$mk_executor_data$fn__3274$fn__3275.invoke(executor.clj:240) >> [storm-core-0.9.3.jar:0.9.3] >> ! at backtype.storm.util$async_loop$fn__464.invoke(util.clj:473) >> [storm-core-0.9.3.jar:0.9.3] >> ! at clojure.lang.AFn.run(AFn.java:24) >> [eagle-topology-0.5.0-SNAPSHOT-assembly.jar:na] >> ! at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101] >> >> ERROR [2017-03-17 13:34:10,013] >> org.apache.eagle.security.enrich.DataEnrichJob: Fail to load sensitivity data >> ! java.net.ConnectException: Connection refused >> ! at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_101] >> ! at >> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) >> ~[na:1.8.0_101] >> ! at >> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) >> ~[na:1.8.0_101] >> ! at >> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) >> ~[na:1.8.0_101] >> ! at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) >> ~[na:1.8.0_101] >> ! at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_101] >> ! at sun.net.NetworkClient.doConnect(NetworkClient.java:175) ~[na:1.8.0_101] >> ! at sun.net.www.http.HttpClient.openServer(HttpClient.java:432) >> ~[na:1.8.0_101] >> ! at sun.net.www.http.HttpClient.openServer(HttpClient.java:527) >> ~[na:1.8.0_101] >> ! at sun.net.www.http.HttpClient.<init>(HttpClient.java:211) ~[na:1.8.0_101] >> ! at sun.net.www.http.HttpClient.New(HttpClient.java:308) ~[na:1.8.0_101] >> ! at sun.net.www.http.HttpClient.New(HttpClient.java:326) ~[na:1.8.0_101] >> ! at >> sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1169) >> ~[na:1.8.0_101] >> ! at >> sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1105) >> ~[na:1.8.0_101] >> ! at >> sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:999) >> ~[na:1.8.0_101] >> ! at >> sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:933) >> ~[na:1.8.0_101] >> ! at >> sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1513) >> ~[na:1.8.0_101] >> ! at >> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441) >> ~[na:1.8.0_101] >> ! at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) >> ~[na:1.8.0_101] >> ! at >> com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:253) >> ~[jersey-client-1.19.1.jar:1.19.1] >> ! at >> com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:153) >> ~[jersey-client-1.19.1.jar:1.19.1] >> ! ... 12 common frames omitted >> ! Causing: com.sun.jersey.api.client.ClientHandlerException: >> java.net.ConnectException: Connection refused >> ! at >> com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:155) >> ~[jersey-client-1.19.1.jar:1.19.1] >> ! at >> com.sun.jersey.api.client.filter.GZIPContentEncodingFilter.handle(GZIPContentEncodingFilter.java:123) >> ~[jersey-client-1.19.1.jar:1.19.1] >> ! at com.sun.jersey.api.client.Client.handle(Client.java:652) >> ~[jersey-client-1.19.1.jar:1.19.1] >> >> Ui wasn't accessable, but server was still running. >> It was failing again and again. >> >> 13) Restarted server >> 14) Nothing was saved. Again fresh install. UI asks for site_id. >> 15) Checked Hbase and MySQL Server - all is empty. >> >> >> So can someone please help me to get started with Eagle on my cluster? >> >> >> Also here is Logstash info: >> logstash-5.2.2 >> >> >> output { >> >> >> >> if [type] == "hdp-nn-audit" { >> >> kafka { >> >> codec => plain {format => "%{message}"} >> >> bootstrap_servers => "demo4:6667" >> >> topic_id => "hdfs_audit_log_tuskpro" >> >> security_protocol => "SASL_PLAINTEXT" >> >> sasl_kerberos_service_name => "kafka" >> >> jaas_path => >> "/usr/hdp/current/kafka-broker/config/kafka_client_jaas.conf" >> kerberos_config => "/etc/krb5.conf" >> >> client_id => "hdp-nn-audit" >> >> message_key => "%{user}" >> >> } >> >> } >> >> } >> >> >> Also I don't know why, but logstash is not using kafka_client_jaas. >> There is info about sasl_kerberos_service_name , but without this property >> logstash is not working. >> >> · >> KafkaClient { >> >> >> com.sun.security.auth.module.Krb5LoginModule required >> >> >> useTicketCache=true >> >> >> renewTicket=true >> >> >> serviceName="kafka"; >> >> >> }; >> >> >> Client { >> >> >> com.sun.security.auth.module.Krb5LoginModule required >> >> >> useTicketCache=true >> >> >> renewTicket=true >> >> >> serviceName="zookeeper"; >> >> >> }; >> >> >> >> >> Regards, >> Andrey >> >> >> >
