Eagle 0.5 was deployed in production as far as I know, but it may not be exact the current version in master branch.
Thanks for your investigation, seems there is still some bug in 0.5, but this particular issue seems is due to dependent components version conflict. @Jayesh is this Jira ready for merge to master? https://issues.apache.org/jira/browse/EAGLE-968 Thanks Edward On Tue, Jan 23, 2018 at 5:10 AM, Colm O hEigeartaigh <cohei...@apache.org> wrote: > OK I've made some more progress. I wasn't seeing any email alerts due to > https://issues.apache.org/jira/browse/EAGLE-968. Once I configure a Kafka > alert, I can see the alerts flowing into my topic. It's still not clear to > me however where the policy "output" is going. I also don't see any alerts > in the UI window. > > Could I ask what the status of the project is in general? There have been > no commits to master since November, so I'm not sure if there is any point > in submitting Pull Requests for outstanding bugs? Are recent versions of > Apache Eagle used in production? > > Colm. > > On Mon, Jan 22, 2018 at 1:07 PM, Colm O hEigeartaigh <cohei...@apache.org> > wrote: > > > > > I've done that but I'm not seeing any alerts, which is why I want to find > > out what the "output" of a policy is and where I can check this. > > > > Colm. > > > > On Mon, Jan 22, 2018 at 1:05 PM, SUDHA JENSLIN <sjens...@gmail.com> > wrote: > > > >> Create and add a publisher to see the output. > >> > >> > >> > >> Regards, > >> Sudha jenslin > >> > >> On Jan 22, 2018 6:31 PM, "Colm O hEigeartaigh" <cohei...@apache.org> > >> wrote: > >> > >> Thanks - the error was due to a problem running Storm with Java 1.8. > I've > >> abandoned the docker image for now, and I'm trying to get it working > >> locally. > >> > >> There are two things I'm not clear on currently, if someone could fill > me > >> in: > >> > >> a) For the 'Hdfs Audit Log Monitor' application, the Kafka Consumer > Topic > >> is 'hdfs_audit_log_sandbox'. Under 'Kafka Topic for Auditlog Event Sink' > >> it > >> also specifies 'hdfs_audit_event_sandbox'. However the documentation for > >> the application mentions 'hdfs_audit_log_enriched_sandbox'? > >> > >> When I click on "STREAMS", the "HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX" > >> uses the topic "hdfs_audit_event_sandbox". And indeed when I run the > >> application, I can see cleansed log data appearing in > >> "hdfs_audit_event_sandbox". So I'm thinking here that > >> 'hdfs_audit_log_enriched_sandbox' is not correct or necessary? > >> > >> b) It's unclear to me where the output data goes when you create a > policy. > >> E.g. say I have: > >> > >> from HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX[str:contains(src,'/hbase')] > >> select * group by user insert into hdfs_audit_log_enriched_stream_out > >> > >> Where is "hdfs_audit_log_enriched_stream_out" defined (is it a Kafka > >> topic?). How can I check the output to make sure the policy is working > >> correctly? > >> > >> Thanks, > >> > >> Colm. > >> > >> On Wed, Jan 17, 2018 at 10:32 PM, Edward Zhang <yonzhang2...@gmail.com> > >> wrote: > >> > >> > There is a data preparation stage between data source(HDFS audit log) > >> and > >> > Alert Engine. This stage is running in Storm and transform the raw > HDFS > >> log > >> > into something which can be alerted. > >> > > >> > The input for data preparation is hdfs_audit_log_sandbox topic and > >> output > >> > is > >> > hdfs_audit_log_enriched_sandbox. > >> > The input for Alert Engine is hdfs_audit_log_enriched_sandbox and > >> output > >> > is > >> > hdfs_audit_log_alert_sandbox. > >> > > >> > Seems in your case, the data preparation staging is not working. We > >> > probably need look at Storm console and figure out if that part is > >> working. > >> > > >> > Thanks > >> > Edward > >> > > >> > On Wed, Jan 17, 2018 at 7:19 AM, Colm O hEigeartaigh < > >> cohei...@apache.org> > >> > wrote: > >> > > >> > > Hi Jayesh, > >> > > > >> > > Many thanks for your feedback! I was able to make a little further > >> > headway. > >> > > There are two configuration problems with the official docker image: > >> > > > >> > > a) A mix of "sandbox.eagle.apache.org" and "server.eagle.apache.org > " > >> > (this > >> > > only occurs in the instructions for running the docker image. The > >> version > >> > > that can be started via the script in the eagle source is OK). I'll > >> > submit > >> > > a PR to fix this once I get a basic use-case working. > >> > > b) For the audit case, it automatically logs HDFS audit logs to the > >> KAFKA > >> > > topic sandbox_hdfs_audit_log instead of the expected > >> > hdfs_audit_log_sandbox > >> > > > >> > > I've fixed these things locally and I can verify that everything is > >> > started > >> > > correctly in Ambari. I log into the docker container and create > >> > > hdfs_audit_log_sandbox and hdfs_audit_log_enriched_sandbox topics, > >> and > >> > > verify that the HDFS audit logs are flowing into the first topic. > >> Then in > >> > > the UI I start the Alert Engine and then the HDFS Audit Log Monitor > >> > > application (changing localhost:6667 to > server.eagle.apache.org:6667 > >> ). > >> > > Both > >> > > applications start up correctly and show "running". > >> > > > >> > > I then create a policy with an email alert along the lines of from > >> > > "HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX[str:contains(src,'/hbase')] > >> > select > >> > > * group by user insert into hdfs_audit_log_enriched_stream_out". > >> However > >> > > at > >> > > this point I'm stuck - nothing appears in the alert window. Is there > >> > > anything obvious I'm doing wrong, or how can I get access to logs to > >> > figure > >> > > out what the problem is? Other topics such as > >> "hdfs_audit_event_sandbox" > >> > > are mentioned in the streams window, but the documentation doesn't > >> say to > >> > > create them. > >> > > > >> > > The UI is buggy though on both Firefox and Chromium on Linux. What > >> > > browser/platform are people using with the UI? > >> > > > >> > > Colm. > >> > > > >> > > On Wed, Jan 17, 2018 at 12:27 AM, Jayesh Senjaliya < > jay...@apache.org > >> > > >> > > wrote: > >> > > > >> > > > Hi Colm, > >> > > > > >> > > > Please find my comments inline. > >> > > > > >> > > > a) The official docker image uses 0.5.0-SNAPSHOT and not the > >> released > >> > > > version. > >> > > > - this is because we uploaded docker image before apache release. > >> > > actually > >> > > > this is same codebase apache-eagle-0.5, and it can be fixed easily > >> by > >> > > just > >> > > > rebuilding docker image. there should not be any mismatch due to > >> this. > >> > > > > >> > > > b) Aside from the above, the official docker image uses a mix of " > >> > > > server.eagle.apache.org" and "sandbox.eagle.apache.org" as the > host > >> > > name. > >> > > > The HBase service doesn't start by default in Ambari as a result. > >> > > > - the only places it uses sandbox is in example script which you > >> will > >> > > have > >> > > > to update anyway, which i agree that it would be good to keep it > >> > > > consistent. > >> > > > > >> > > > c) The UI seems quite buggy. On both chromium and firefox, I only > >> see > >> > > > links to "Sandbox" and "Alert" on the left hand-side. Once I click > >> on > >> > > > "Alert" I have no way of going back to see the applications. I > don't > >> > see > >> > > > the links to "integration" or "sites" as in the picture here: > >> > > > http://eagle.apache.org/docs/latest/applications/#jmx-monitoring > >> > > > - when hbase is as deep storage is used, and if eagle app has > issue > >> > > > connecting to hbase, the UI becomes unresponsive. > >> > > > > >> > > > d) In chromium, the button to create a new policy does not exist > - I > >> > can > >> > > > only see it on Firefox. > >> > > > - i have seen when you logged in, you will see admin actions. but > if > >> > this > >> > > > still an issue, can you please file UI bug? > >> > > > > >> > > > e) I'm trying to get the "Hdfs Audit Log Monitor" use-case > working, > >> but > >> > > it > >> > > > seems to be stuck in "Initialized". > >> > > > this eagle docs has example on how to setup the app. pls let us > >> know if > >> > > > you find any gaps. > >> > > > > >> > > > Thanks for trying out, and sharing your findings, > >> > > > Jayesh > >> > > > > >> > > > > >> > > > On Tue, Jan 16, 2018 at 3:34 AM, Colm O hEigeartaigh < > >> > > cohei...@apache.org> > >> > > > wrote: > >> > > > > >> > > >> Hi all, > >> > > >> > >> > > >> I'm trying to play around a bit with Apache Eagle 0.5.0 to no > >> avail. > >> > > Here > >> > > >> are the problems I've run into so far: > >> > > >> > >> > > >> a) The official docker image uses 0.5.0-SNAPSHOT and not the > >> released > >> > > >> version. > >> > > >> > >> > > >> b) Aside from the above, the official docker image uses a mix of > " > >> > > >> server.eagle.apache.org" and "sandbox.eagle.apache.org" as the > >> host > >> > > >> name. The HBase service doesn't start by default in Ambari as a > >> > result. > >> > > >> > >> > > >> c) The UI seems quite buggy. On both chromium and firefox, I only > >> see > >> > > >> links to "Sandbox" and "Alert" on the left hand-side. Once I > click > >> on > >> > > >> "Alert" I have no way of going back to see the applications. I > >> don't > >> > see > >> > > >> the links to "integration" or "sites" as in the picture here: > >> > > >> http://eagle.apache.org/docs/latest/applications/#jmx-monitoring > >> > > >> > >> > > >> d) In chromium, the button to create a new policy does not exist > - > >> I > >> > can > >> > > >> only see it on Firefox. > >> > > >> > >> > > >> e) I'm trying to get the "Hdfs Audit Log Monitor" use-case > working, > >> > but > >> > > >> it seems to be stuck in "Initialized". > >> > > >> > >> > > >> Could someone fill me in on what the "recommended" way is to > start > >> > > Apache > >> > > >> Eagle so that I can play around with the functionality that it > >> offers? > >> > > >> Clearly the docker approach is buggy. Also, what browser should > be > >> > used? > >> > > >> > >> > > >> Thanks, > >> > > >> > >> > > >> Colm. > >> > > >> > >> > > >> > >> > > >> -- > >> > > >> Colm O hEigeartaigh > >> > > >> > >> > > >> Talend Community Coder > >> > > >> http://coders.talend.com > >> > > >> > >> > > > > >> > > > > >> > > > >> > > > >> > > -- > >> > > Colm O hEigeartaigh > >> > > > >> > > Talend Community Coder > >> > > http://coders.talend.com > >> > > > >> > > >> > >> > >> > >> -- > >> Colm O hEigeartaigh > >> > >> Talend Community Coder > >> http://coders.talend.com > >> > >> > >> > > > > > > -- > > Colm O hEigeartaigh > > > > Talend Community Coder > > http://coders.talend.com > > > > > > -- > Colm O hEigeartaigh > > Talend Community Coder > http://coders.talend.com >
