Hi, we do use eagle 0.5 in production although we dont use all the available hadoop applications.
EAGLE-968 <https://issues.apache.org/jira/browse/EAGLE-968> is a fix for email issue we found while our testing. should be merged soon after a rebase. @Colm, did you tried adding storage publisher (AlertEagleStorePlugin)? to see alerts on UI ? Thanks Jayesh On Wed, Jan 24, 2018 at 7:08 PM, Edward Zhang <yonzhang2...@gmail.com> wrote: > Eagle 0.5 was deployed in production as far as I know, but it may not be > exact the current version in master branch. > > Thanks for your investigation, seems there is still some bug in 0.5, but > this particular issue seems is due to dependent components version conflict. > > @Jayesh is this Jira ready for merge to master? https://issues.apache. > org/jira/browse/EAGLE-968 > > > Thanks > Edward > > On Tue, Jan 23, 2018 at 5:10 AM, Colm O hEigeartaigh <cohei...@apache.org> > wrote: > >> OK I've made some more progress. I wasn't seeing any email alerts due to >> https://issues.apache.org/jira/browse/EAGLE-968. Once I configure a Kafka >> alert, I can see the alerts flowing into my topic. It's still not clear to >> me however where the policy "output" is going. I also don't see any alerts >> in the UI window. >> >> Could I ask what the status of the project is in general? There have been >> no commits to master since November, so I'm not sure if there is any point >> in submitting Pull Requests for outstanding bugs? Are recent versions of >> Apache Eagle used in production? >> >> Colm. >> >> On Mon, Jan 22, 2018 at 1:07 PM, Colm O hEigeartaigh <cohei...@apache.org >> > >> wrote: >> >> > >> > I've done that but I'm not seeing any alerts, which is why I want to >> find >> > out what the "output" of a policy is and where I can check this. >> > >> > Colm. >> > >> > On Mon, Jan 22, 2018 at 1:05 PM, SUDHA JENSLIN <sjens...@gmail.com> >> wrote: >> > >> >> Create and add a publisher to see the output. >> >> >> >> >> >> >> >> Regards, >> >> Sudha jenslin >> >> >> >> On Jan 22, 2018 6:31 PM, "Colm O hEigeartaigh" <cohei...@apache.org> >> >> wrote: >> >> >> >> Thanks - the error was due to a problem running Storm with Java 1.8. >> I've >> >> abandoned the docker image for now, and I'm trying to get it working >> >> locally. >> >> >> >> There are two things I'm not clear on currently, if someone could fill >> me >> >> in: >> >> >> >> a) For the 'Hdfs Audit Log Monitor' application, the Kafka Consumer >> Topic >> >> is 'hdfs_audit_log_sandbox'. Under 'Kafka Topic for Auditlog Event >> Sink' >> >> it >> >> also specifies 'hdfs_audit_event_sandbox'. However the documentation >> for >> >> the application mentions 'hdfs_audit_log_enriched_sandbox'? >> >> >> >> When I click on "STREAMS", the "HDFS_AUDIT_LOG_ENRICHED_STREA >> M_SANDBOX" >> >> uses the topic "hdfs_audit_event_sandbox". And indeed when I run the >> >> application, I can see cleansed log data appearing in >> >> "hdfs_audit_event_sandbox". So I'm thinking here that >> >> 'hdfs_audit_log_enriched_sandbox' is not correct or necessary? >> >> >> >> b) It's unclear to me where the output data goes when you create a >> policy. >> >> E.g. say I have: >> >> >> >> from HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX[str:contains(src,'/ >> hbase')] >> >> select * group by user insert into hdfs_audit_log_enriched_stream_out >> >> >> >> Where is "hdfs_audit_log_enriched_stream_out" defined (is it a Kafka >> >> topic?). How can I check the output to make sure the policy is working >> >> correctly? >> >> >> >> Thanks, >> >> >> >> Colm. >> >> >> >> On Wed, Jan 17, 2018 at 10:32 PM, Edward Zhang <yonzhang2...@gmail.com >> > >> >> wrote: >> >> >> >> > There is a data preparation stage between data source(HDFS audit log) >> >> and >> >> > Alert Engine. This stage is running in Storm and transform the raw >> HDFS >> >> log >> >> > into something which can be alerted. >> >> > >> >> > The input for data preparation is hdfs_audit_log_sandbox topic and >> >> output >> >> > is >> >> > hdfs_audit_log_enriched_sandbox. >> >> > The input for Alert Engine is hdfs_audit_log_enriched_sandbox and >> >> output >> >> > is >> >> > hdfs_audit_log_alert_sandbox. >> >> > >> >> > Seems in your case, the data preparation staging is not working. We >> >> > probably need look at Storm console and figure out if that part is >> >> working. >> >> > >> >> > Thanks >> >> > Edward >> >> > >> >> > On Wed, Jan 17, 2018 at 7:19 AM, Colm O hEigeartaigh < >> >> cohei...@apache.org> >> >> > wrote: >> >> > >> >> > > Hi Jayesh, >> >> > > >> >> > > Many thanks for your feedback! I was able to make a little further >> >> > headway. >> >> > > There are two configuration problems with the official docker >> image: >> >> > > >> >> > > a) A mix of "sandbox.eagle.apache.org" and " >> server.eagle.apache.org" >> >> > (this >> >> > > only occurs in the instructions for running the docker image. The >> >> version >> >> > > that can be started via the script in the eagle source is OK). I'll >> >> > submit >> >> > > a PR to fix this once I get a basic use-case working. >> >> > > b) For the audit case, it automatically logs HDFS audit logs to the >> >> KAFKA >> >> > > topic sandbox_hdfs_audit_log instead of the expected >> >> > hdfs_audit_log_sandbox >> >> > > >> >> > > I've fixed these things locally and I can verify that everything is >> >> > started >> >> > > correctly in Ambari. I log into the docker container and create >> >> > > hdfs_audit_log_sandbox and hdfs_audit_log_enriched_sandbox topics, >> >> and >> >> > > verify that the HDFS audit logs are flowing into the first topic. >> >> Then in >> >> > > the UI I start the Alert Engine and then the HDFS Audit Log Monitor >> >> > > application (changing localhost:6667 to >> server.eagle.apache.org:6667 >> >> ). >> >> > > Both >> >> > > applications start up correctly and show "running". >> >> > > >> >> > > I then create a policy with an email alert along the lines of from >> >> > > "HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX[str:contains(src,'/ >> hbase')] >> >> > select >> >> > > * group by user insert into hdfs_audit_log_enriched_stream_out". >> >> However >> >> > > at >> >> > > this point I'm stuck - nothing appears in the alert window. Is >> there >> >> > > anything obvious I'm doing wrong, or how can I get access to logs >> to >> >> > figure >> >> > > out what the problem is? Other topics such as >> >> "hdfs_audit_event_sandbox" >> >> > > are mentioned in the streams window, but the documentation doesn't >> >> say to >> >> > > create them. >> >> > > >> >> > > The UI is buggy though on both Firefox and Chromium on Linux. What >> >> > > browser/platform are people using with the UI? >> >> > > >> >> > > Colm. >> >> > > >> >> > > On Wed, Jan 17, 2018 at 12:27 AM, Jayesh Senjaliya < >> jay...@apache.org >> >> > >> >> > > wrote: >> >> > > >> >> > > > Hi Colm, >> >> > > > >> >> > > > Please find my comments inline. >> >> > > > >> >> > > > a) The official docker image uses 0.5.0-SNAPSHOT and not the >> >> released >> >> > > > version. >> >> > > > - this is because we uploaded docker image before apache release. >> >> > > actually >> >> > > > this is same codebase apache-eagle-0.5, and it can be fixed >> easily >> >> by >> >> > > just >> >> > > > rebuilding docker image. there should not be any mismatch due to >> >> this. >> >> > > > >> >> > > > b) Aside from the above, the official docker image uses a mix of >> " >> >> > > > server.eagle.apache.org" and "sandbox.eagle.apache.org" as the >> host >> >> > > name. >> >> > > > The HBase service doesn't start by default in Ambari as a result. >> >> > > > - the only places it uses sandbox is in example script which you >> >> will >> >> > > have >> >> > > > to update anyway, which i agree that it would be good to keep it >> >> > > > consistent. >> >> > > > >> >> > > > c) The UI seems quite buggy. On both chromium and firefox, I only >> >> see >> >> > > > links to "Sandbox" and "Alert" on the left hand-side. Once I >> click >> >> on >> >> > > > "Alert" I have no way of going back to see the applications. I >> don't >> >> > see >> >> > > > the links to "integration" or "sites" as in the picture here: >> >> > > > http://eagle.apache.org/docs/latest/applications/#jmx-monitoring >> >> > > > - when hbase is as deep storage is used, and if eagle app has >> issue >> >> > > > connecting to hbase, the UI becomes unresponsive. >> >> > > > >> >> > > > d) In chromium, the button to create a new policy does not exist >> - I >> >> > can >> >> > > > only see it on Firefox. >> >> > > > - i have seen when you logged in, you will see admin actions. >> but if >> >> > this >> >> > > > still an issue, can you please file UI bug? >> >> > > > >> >> > > > e) I'm trying to get the "Hdfs Audit Log Monitor" use-case >> working, >> >> but >> >> > > it >> >> > > > seems to be stuck in "Initialized". >> >> > > > this eagle docs has example on how to setup the app. pls let us >> >> know if >> >> > > > you find any gaps. >> >> > > > >> >> > > > Thanks for trying out, and sharing your findings, >> >> > > > Jayesh >> >> > > > >> >> > > > >> >> > > > On Tue, Jan 16, 2018 at 3:34 AM, Colm O hEigeartaigh < >> >> > > cohei...@apache.org> >> >> > > > wrote: >> >> > > > >> >> > > >> Hi all, >> >> > > >> >> >> > > >> I'm trying to play around a bit with Apache Eagle 0.5.0 to no >> >> avail. >> >> > > Here >> >> > > >> are the problems I've run into so far: >> >> > > >> >> >> > > >> a) The official docker image uses 0.5.0-SNAPSHOT and not the >> >> released >> >> > > >> version. >> >> > > >> >> >> > > >> b) Aside from the above, the official docker image uses a mix >> of " >> >> > > >> server.eagle.apache.org" and "sandbox.eagle.apache.org" as the >> >> host >> >> > > >> name. The HBase service doesn't start by default in Ambari as a >> >> > result. >> >> > > >> >> >> > > >> c) The UI seems quite buggy. On both chromium and firefox, I >> only >> >> see >> >> > > >> links to "Sandbox" and "Alert" on the left hand-side. Once I >> click >> >> on >> >> > > >> "Alert" I have no way of going back to see the applications. I >> >> don't >> >> > see >> >> > > >> the links to "integration" or "sites" as in the picture here: >> >> > > >> http://eagle.apache.org/docs/latest/applications/#jmx-monito >> ring >> >> > > >> >> >> > > >> d) In chromium, the button to create a new policy does not >> exist - >> >> I >> >> > can >> >> > > >> only see it on Firefox. >> >> > > >> >> >> > > >> e) I'm trying to get the "Hdfs Audit Log Monitor" use-case >> working, >> >> > but >> >> > > >> it seems to be stuck in "Initialized". >> >> > > >> >> >> > > >> Could someone fill me in on what the "recommended" way is to >> start >> >> > > Apache >> >> > > >> Eagle so that I can play around with the functionality that it >> >> offers? >> >> > > >> Clearly the docker approach is buggy. Also, what browser should >> be >> >> > used? >> >> > > >> >> >> > > >> Thanks, >> >> > > >> >> >> > > >> Colm. >> >> > > >> >> >> > > >> >> >> > > >> -- >> >> > > >> Colm O hEigeartaigh >> >> > > >> >> >> > > >> Talend Community Coder >> >> > > >> http://coders.talend.com >> >> > > >> >> >> > > > >> >> > > > >> >> > > >> >> > > >> >> > > -- >> >> > > Colm O hEigeartaigh >> >> > > >> >> > > Talend Community Coder >> >> > > http://coders.talend.com >> >> > > >> >> > >> >> >> >> >> >> >> >> -- >> >> Colm O hEigeartaigh >> >> >> >> Talend Community Coder >> >> http://coders.talend.com >> >> >> >> >> >> >> > >> > >> > -- >> > Colm O hEigeartaigh >> > >> > Talend Community Coder >> > http://coders.talend.com >> > >> >> >> >> -- >> Colm O hEigeartaigh >> >> Talend Community Coder >> http://coders.talend.com >> > >
