I've done that but I'm not seeing any alerts, which is why I want to find out what the "output" of a policy is and where I can check this.
Colm. On Mon, Jan 22, 2018 at 1:05 PM, SUDHA JENSLIN <[email protected]> wrote: > Create and add a publisher to see the output. > > > > Regards, > Sudha jenslin > > On Jan 22, 2018 6:31 PM, "Colm O hEigeartaigh" <[email protected]> > wrote: > > Thanks - the error was due to a problem running Storm with Java 1.8. I've > abandoned the docker image for now, and I'm trying to get it working > locally. > > There are two things I'm not clear on currently, if someone could fill me > in: > > a) For the 'Hdfs Audit Log Monitor' application, the Kafka Consumer Topic > is 'hdfs_audit_log_sandbox'. Under 'Kafka Topic for Auditlog Event Sink' it > also specifies 'hdfs_audit_event_sandbox'. However the documentation for > the application mentions 'hdfs_audit_log_enriched_sandbox'? > > When I click on "STREAMS", the "HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX" > uses the topic "hdfs_audit_event_sandbox". And indeed when I run the > application, I can see cleansed log data appearing in > "hdfs_audit_event_sandbox". So I'm thinking here that > 'hdfs_audit_log_enriched_sandbox' is not correct or necessary? > > b) It's unclear to me where the output data goes when you create a policy. > E.g. say I have: > > from HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX[str:contains(src,'/hbase')] > select * group by user insert into hdfs_audit_log_enriched_stream_out > > Where is "hdfs_audit_log_enriched_stream_out" defined (is it a Kafka > topic?). How can I check the output to make sure the policy is working > correctly? > > Thanks, > > Colm. > > On Wed, Jan 17, 2018 at 10:32 PM, Edward Zhang <[email protected]> > wrote: > > > There is a data preparation stage between data source(HDFS audit log) and > > Alert Engine. This stage is running in Storm and transform the raw HDFS > log > > into something which can be alerted. > > > > The input for data preparation is hdfs_audit_log_sandbox topic and output > > is > > hdfs_audit_log_enriched_sandbox. > > The input for Alert Engine is hdfs_audit_log_enriched_sandbox and output > > is > > hdfs_audit_log_alert_sandbox. > > > > Seems in your case, the data preparation staging is not working. We > > probably need look at Storm console and figure out if that part is > working. > > > > Thanks > > Edward > > > > On Wed, Jan 17, 2018 at 7:19 AM, Colm O hEigeartaigh < > [email protected]> > > wrote: > > > > > Hi Jayesh, > > > > > > Many thanks for your feedback! I was able to make a little further > > headway. > > > There are two configuration problems with the official docker image: > > > > > > a) A mix of "sandbox.eagle.apache.org" and "server.eagle.apache.org" > > (this > > > only occurs in the instructions for running the docker image. The > version > > > that can be started via the script in the eagle source is OK). I'll > > submit > > > a PR to fix this once I get a basic use-case working. > > > b) For the audit case, it automatically logs HDFS audit logs to the > KAFKA > > > topic sandbox_hdfs_audit_log instead of the expected > > hdfs_audit_log_sandbox > > > > > > I've fixed these things locally and I can verify that everything is > > started > > > correctly in Ambari. I log into the docker container and create > > > hdfs_audit_log_sandbox and hdfs_audit_log_enriched_sandbox topics, and > > > verify that the HDFS audit logs are flowing into the first topic. Then > in > > > the UI I start the Alert Engine and then the HDFS Audit Log Monitor > > > application (changing localhost:6667 to server.eagle.apache.org:6667). > > > Both > > > applications start up correctly and show "running". > > > > > > I then create a policy with an email alert along the lines of from > > > "HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX[str:contains(src,'/hbase')] > > select > > > * group by user insert into hdfs_audit_log_enriched_stream_out". > However > > > at > > > this point I'm stuck - nothing appears in the alert window. Is there > > > anything obvious I'm doing wrong, or how can I get access to logs to > > figure > > > out what the problem is? Other topics such as > "hdfs_audit_event_sandbox" > > > are mentioned in the streams window, but the documentation doesn't say > to > > > create them. > > > > > > The UI is buggy though on both Firefox and Chromium on Linux. What > > > browser/platform are people using with the UI? > > > > > > Colm. > > > > > > On Wed, Jan 17, 2018 at 12:27 AM, Jayesh Senjaliya <[email protected]> > > > wrote: > > > > > > > Hi Colm, > > > > > > > > Please find my comments inline. > > > > > > > > a) The official docker image uses 0.5.0-SNAPSHOT and not the released > > > > version. > > > > - this is because we uploaded docker image before apache release. > > > actually > > > > this is same codebase apache-eagle-0.5, and it can be fixed easily by > > > just > > > > rebuilding docker image. there should not be any mismatch due to > this. > > > > > > > > b) Aside from the above, the official docker image uses a mix of " > > > > server.eagle.apache.org" and "sandbox.eagle.apache.org" as the host > > > name. > > > > The HBase service doesn't start by default in Ambari as a result. > > > > - the only places it uses sandbox is in example script which you will > > > have > > > > to update anyway, which i agree that it would be good to keep it > > > > consistent. > > > > > > > > c) The UI seems quite buggy. On both chromium and firefox, I only see > > > > links to "Sandbox" and "Alert" on the left hand-side. Once I click on > > > > "Alert" I have no way of going back to see the applications. I don't > > see > > > > the links to "integration" or "sites" as in the picture here: > > > > http://eagle.apache.org/docs/latest/applications/#jmx-monitoring > > > > - when hbase is as deep storage is used, and if eagle app has issue > > > > connecting to hbase, the UI becomes unresponsive. > > > > > > > > d) In chromium, the button to create a new policy does not exist - I > > can > > > > only see it on Firefox. > > > > - i have seen when you logged in, you will see admin actions. but if > > this > > > > still an issue, can you please file UI bug? > > > > > > > > e) I'm trying to get the "Hdfs Audit Log Monitor" use-case working, > but > > > it > > > > seems to be stuck in "Initialized". > > > > this eagle docs has example on how to setup the app. pls let us know > if > > > > you find any gaps. > > > > > > > > Thanks for trying out, and sharing your findings, > > > > Jayesh > > > > > > > > > > > > On Tue, Jan 16, 2018 at 3:34 AM, Colm O hEigeartaigh < > > > [email protected]> > > > > wrote: > > > > > > > >> Hi all, > > > >> > > > >> I'm trying to play around a bit with Apache Eagle 0.5.0 to no avail. > > > Here > > > >> are the problems I've run into so far: > > > >> > > > >> a) The official docker image uses 0.5.0-SNAPSHOT and not the > released > > > >> version. > > > >> > > > >> b) Aside from the above, the official docker image uses a mix of " > > > >> server.eagle.apache.org" and "sandbox.eagle.apache.org" as the host > > > >> name. The HBase service doesn't start by default in Ambari as a > > result. > > > >> > > > >> c) The UI seems quite buggy. On both chromium and firefox, I only > see > > > >> links to "Sandbox" and "Alert" on the left hand-side. Once I click > on > > > >> "Alert" I have no way of going back to see the applications. I don't > > see > > > >> the links to "integration" or "sites" as in the picture here: > > > >> http://eagle.apache.org/docs/latest/applications/#jmx-monitoring > > > >> > > > >> d) In chromium, the button to create a new policy does not exist - I > > can > > > >> only see it on Firefox. > > > >> > > > >> e) I'm trying to get the "Hdfs Audit Log Monitor" use-case working, > > but > > > >> it seems to be stuck in "Initialized". > > > >> > > > >> Could someone fill me in on what the "recommended" way is to start > > > Apache > > > >> Eagle so that I can play around with the functionality that it > offers? > > > >> Clearly the docker approach is buggy. Also, what browser should be > > used? > > > >> > > > >> Thanks, > > > >> > > > >> Colm. > > > >> > > > >> > > > >> -- > > > >> Colm O hEigeartaigh > > > >> > > > >> Talend Community Coder > > > >> http://coders.talend.com > > > >> > > > > > > > > > > > > > > > > > -- > > > Colm O hEigeartaigh > > > > > > Talend Community Coder > > > http://coders.talend.com > > > > > > > > > -- > Colm O hEigeartaigh > > Talend Community Coder > http://coders.talend.com > > > -- Colm O hEigeartaigh Talend Community Coder http://coders.talend.com
