There is a data preparation stage between data source(HDFS audit log) and Alert Engine. This stage is running in Storm and transform the raw HDFS log into something which can be alerted.
The input for data preparation is hdfs_audit_log_sandbox topic and output is hdfs_audit_log_enriched_sandbox. The input for Alert Engine is hdfs_audit_log_enriched_sandbox and output is hdfs_audit_log_alert_sandbox. Seems in your case, the data preparation staging is not working. We probably need look at Storm console and figure out if that part is working. Thanks Edward On Wed, Jan 17, 2018 at 7:19 AM, Colm O hEigeartaigh <[email protected]> wrote: > Hi Jayesh, > > Many thanks for your feedback! I was able to make a little further headway. > There are two configuration problems with the official docker image: > > a) A mix of "sandbox.eagle.apache.org" and "server.eagle.apache.org" (this > only occurs in the instructions for running the docker image. The version > that can be started via the script in the eagle source is OK). I'll submit > a PR to fix this once I get a basic use-case working. > b) For the audit case, it automatically logs HDFS audit logs to the KAFKA > topic sandbox_hdfs_audit_log instead of the expected hdfs_audit_log_sandbox > > I've fixed these things locally and I can verify that everything is started > correctly in Ambari. I log into the docker container and create > hdfs_audit_log_sandbox and hdfs_audit_log_enriched_sandbox topics, and > verify that the HDFS audit logs are flowing into the first topic. Then in > the UI I start the Alert Engine and then the HDFS Audit Log Monitor > application (changing localhost:6667 to server.eagle.apache.org:6667). > Both > applications start up correctly and show "running". > > I then create a policy with an email alert along the lines of from > "HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX[str:contains(src,'/hbase')] select > * group by user insert into hdfs_audit_log_enriched_stream_out". However > at > this point I'm stuck - nothing appears in the alert window. Is there > anything obvious I'm doing wrong, or how can I get access to logs to figure > out what the problem is? Other topics such as "hdfs_audit_event_sandbox" > are mentioned in the streams window, but the documentation doesn't say to > create them. > > The UI is buggy though on both Firefox and Chromium on Linux. What > browser/platform are people using with the UI? > > Colm. > > On Wed, Jan 17, 2018 at 12:27 AM, Jayesh Senjaliya <[email protected]> > wrote: > > > Hi Colm, > > > > Please find my comments inline. > > > > a) The official docker image uses 0.5.0-SNAPSHOT and not the released > > version. > > - this is because we uploaded docker image before apache release. > actually > > this is same codebase apache-eagle-0.5, and it can be fixed easily by > just > > rebuilding docker image. there should not be any mismatch due to this. > > > > b) Aside from the above, the official docker image uses a mix of " > > server.eagle.apache.org" and "sandbox.eagle.apache.org" as the host > name. > > The HBase service doesn't start by default in Ambari as a result. > > - the only places it uses sandbox is in example script which you will > have > > to update anyway, which i agree that it would be good to keep it > > consistent. > > > > c) The UI seems quite buggy. On both chromium and firefox, I only see > > links to "Sandbox" and "Alert" on the left hand-side. Once I click on > > "Alert" I have no way of going back to see the applications. I don't see > > the links to "integration" or "sites" as in the picture here: > > http://eagle.apache.org/docs/latest/applications/#jmx-monitoring > > - when hbase is as deep storage is used, and if eagle app has issue > > connecting to hbase, the UI becomes unresponsive. > > > > d) In chromium, the button to create a new policy does not exist - I can > > only see it on Firefox. > > - i have seen when you logged in, you will see admin actions. but if this > > still an issue, can you please file UI bug? > > > > e) I'm trying to get the "Hdfs Audit Log Monitor" use-case working, but > it > > seems to be stuck in "Initialized". > > this eagle docs has example on how to setup the app. pls let us know if > > you find any gaps. > > > > Thanks for trying out, and sharing your findings, > > Jayesh > > > > > > On Tue, Jan 16, 2018 at 3:34 AM, Colm O hEigeartaigh < > [email protected]> > > wrote: > > > >> Hi all, > >> > >> I'm trying to play around a bit with Apache Eagle 0.5.0 to no avail. > Here > >> are the problems I've run into so far: > >> > >> a) The official docker image uses 0.5.0-SNAPSHOT and not the released > >> version. > >> > >> b) Aside from the above, the official docker image uses a mix of " > >> server.eagle.apache.org" and "sandbox.eagle.apache.org" as the host > >> name. The HBase service doesn't start by default in Ambari as a result. > >> > >> c) The UI seems quite buggy. On both chromium and firefox, I only see > >> links to "Sandbox" and "Alert" on the left hand-side. Once I click on > >> "Alert" I have no way of going back to see the applications. I don't see > >> the links to "integration" or "sites" as in the picture here: > >> http://eagle.apache.org/docs/latest/applications/#jmx-monitoring > >> > >> d) In chromium, the button to create a new policy does not exist - I can > >> only see it on Firefox. > >> > >> e) I'm trying to get the "Hdfs Audit Log Monitor" use-case working, but > >> it seems to be stuck in "Initialized". > >> > >> Could someone fill me in on what the "recommended" way is to start > Apache > >> Eagle so that I can play around with the functionality that it offers? > >> Clearly the docker approach is buggy. Also, what browser should be used? > >> > >> Thanks, > >> > >> Colm. > >> > >> > >> -- > >> Colm O hEigeartaigh > >> > >> Talend Community Coder > >> http://coders.talend.com > >> > > > > > > > -- > Colm O hEigeartaigh > > Talend Community Coder > http://coders.talend.com >
